From patchwork Wed Aug 10 15:49:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12941231 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 11CE4C25B0C for ; Thu, 11 Aug 2022 08:05:47 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.383693.619544 (Exim 4.92) (envelope-from ) id 1oM3C8-0004Yv-Jb; Thu, 11 Aug 2022 08:05:32 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 383693.619544; Thu, 11 Aug 2022 08:05:32 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oM3C8-0004Yj-E0; Thu, 11 Aug 2022 08:05:32 +0000 Received: by outflank-mailman (input) for mailman id 383693; Wed, 10 Aug 2022 15:51:39 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oLnzf-00016w-62 for xen-devel@lists.xenproject.org; Wed, 10 Aug 2022 15:51:39 +0000 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [2a00:1450:4864:20::42a]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 5186db06-18c4-11ed-924f-1f966e50362f; Wed, 10 Aug 2022 17:51:38 +0200 (CEST) Received: by mail-wr1-x42a.google.com with SMTP id l22so18237621wrz.7 for ; Wed, 10 Aug 2022 08:51:38 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:37 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 5186db06-18c4-11ed-924f-1f966e50362f DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=mveucMTZdZPLlN4D5K1kqfZw5UNWDwp9Ns7O7VRFK+I=; b=FeHdakLvXxtWgjpPF+aflE98THue/ZKUpEuSPYY/vwKS6pLky2/FycS8L1DaAOP7p2 gbmwq+ySuj1nbkojJhrktc86MeoeDdNZHmrmEXeui4TvX9dWZtL5o1pn/n2SFPu2UKqJ LwG8Bt1+6Z9HVd12TbOjrwFHHO7Xbx+0NZ4knAvVP00VnvWpTA1vdoX8qFWZUEU1a0Hc WTzV9hRGh/acAWBr/K4D/fYN2WXLMaChMzaEobWBKN01dp0GD77rhc2rXo7Zw1Rse8vI EoYL8E2KpOU5sN0NjoXacILSTjNTJslWSxRKaYX/is4eawuPkbRaWRlUK6vsWA5Q252i fHCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=mveucMTZdZPLlN4D5K1kqfZw5UNWDwp9Ns7O7VRFK+I=; b=EeFksYZ35i1qaPE28k27EygXi9Hk0amzquZl8wc9o0iOIXgXQymyoA3wd+YBcLMyCf LYoU+9zv7noUhKgoy3FbGRcTuPHbmb5jb5tZPpA3AQsfuXVndYoG6y8JQT0RhCrWZ5a5 M7rA4cCtB6pqxqawNGTeqwXoHFOHjLciPLpaFkPdRFZktgDdZ7LRMlz/SpBp3VF0/jOk pg7p+O0EqXrSBWakpb8YROAsDOBAg/lWE250apcV5bqE1tys6fDxcPeX3ymuC6ldA8nX Peuez7rvbi2ckMGCeJJlMSLBUxgHzBmAHmdAZONLF7I/TxY2wr8neuXvhyEk+PBSThDF IA6g== X-Gm-Message-State: ACgBeo1uwcLFx4gJmOQQ6iZaaN7fa7NAcz4iKUAojh38mEv6czDFFrcB WD4azZs9gnduOGFLdkZ9DQY= X-Google-Smtp-Source: AA6agR6IVj+Term37QFPdVaJx8duP8zGzs+p0efJL76vLIQCk5ivpP0E95/E2mMnqsqquEDgBKZAtg== X-Received: by 2002:a05:6000:3c6:b0:220:5efd:423c with SMTP id b6-20020a05600003c600b002205efd423cmr18135634wrg.214.1660146697602; Wed, 10 Aug 2022 08:51:37 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 01/11] net: introduce struct ubuf_info_msgzc Date: Wed, 10 Aug 2022 16:49:09 +0100 Message-Id: X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 We're going to split struct ubuf_info and leave there only mandatory fields. Users are free to extend it. Add struct ubuf_info_msgzc, which will be an extended version for MSG_ZEROCOPY and some other users. It duplicates of struct ubuf_info for now and will be removed in a couple of patches. Signed-off-by: Pavel Begunkov --- include/linux/skbuff.h | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index ca8afa382bf2..f8ac3678dab8 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -554,7 +554,28 @@ struct ubuf_info { } mmp; }; +struct ubuf_info_msgzc { + struct ubuf_info ubuf; + + union { + struct { + unsigned long desc; + void *ctx; + }; + struct { + u32 id; + u16 len; + u16 zerocopy:1; + u32 bytelen; + }; + }; + + struct mmpin mmp; +}; + #define skb_uarg(SKB) ((struct ubuf_info *)(skb_shinfo(SKB)->destructor_arg)) +#define uarg_to_msgzc(ubuf_ptr) container_of((ubuf_ptr), struct ubuf_info_msgzc, \ + ubuf) int mm_account_pinned_pages(struct mmpin *mmp, size_t size); void mm_unaccount_pinned_pages(struct mmpin *mmp); From patchwork Wed Aug 10 15:49:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12941235 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1443CC3F6B0 for ; Thu, 11 Aug 2022 08:05:50 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.383694.619551 (Exim 4.92) (envelope-from ) id 1oM3C8-0004h0-T5; Thu, 11 Aug 2022 08:05:32 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 383694.619551; Thu, 11 Aug 2022 08:05:32 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oM3C8-0004fU-NU; Thu, 11 Aug 2022 08:05:32 +0000 Received: by outflank-mailman (input) for mailman id 383694; Wed, 10 Aug 2022 15:51:40 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oLnzg-00016q-6c for xen-devel@lists.xenproject.org; Wed, 10 Aug 2022 15:51:40 +0000 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [2a00:1450:4864:20::42c]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 5259254d-18c4-11ed-bd2e-47488cf2e6aa; Wed, 10 Aug 2022 17:51:39 +0200 (CEST) Received: by mail-wr1-x42c.google.com with SMTP id p10so18205183wru.8 for ; Wed, 10 Aug 2022 08:51:39 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:38 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 5259254d-18c4-11ed-bd2e-47488cf2e6aa DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=RS63Hb0HKwQLj+yY8E+g8nWuxDYTXWs65j0+OKPH1OE=; b=ZQdOMbpO/VHyANe5mleak9CanXYG2CBuqoaXodVPqSxfN0jYX3Wpv/vcb6iubMBW1u 4Ble+mSyTPc7xm8K/FN13yhEm7K3hDV1my/RtaOdys3+jH8lzdROVrs3yday1zidVe+x bMEKgzVG6yfs6m4/lp21crX1YCY0qdX1LO/SN3r1GTpQB97R7NYdjhtr4vcyw0X1MhlZ Rsr3NqzS5od350DfBp0/rWIgXTPVwoJA57tT5sN5ssmtZG7YNPmWvIuqmqqLbLD3mFiz LM3JoK9cwamnuJVbkAxMoi+VNXdtwhLPXE2PCWt/23HgKhwIUuswHMgcOCy1QLV1mNAn Xu2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=RS63Hb0HKwQLj+yY8E+g8nWuxDYTXWs65j0+OKPH1OE=; b=Gr4KgvZMRV0/TgpcKcSPSZCBFXb/9Jbbn7b7AFXStaS7USHY2W6eD3SwJ8rKqENqKq 4oUwNrYY1RJ3kK6osWpfXqQChcE89Flq/2XZkIntU4TUwmeBZbikkVaHwHYHcKHT7pzb bP1Np8kF1kbfWkyIk+VgagaRg1JGRAC+h2/zVMxFdMcmXq4GNj5NGN0mfevvRV4Yrixm DsRsmaKX9JsmvcpJVpdxqXXYhGj5v+tXIEmKBzRIfGL0YuQM0PfOihK5F7a+xGopI1m8 SwDTXinAbE0Nn/40XcYhHnV+4SQz+2ocAl72M+rU2s2hjmpLzLd33LVzFAka6Gf5hxUI RhkA== X-Gm-Message-State: ACgBeo2h4bASfGJXdn4j8p3eflTRwDQXRH+e7Mw0Jaq6mrE1PRJuqTid trHk0AAi0e5aNLbJy5fj6AQ= X-Google-Smtp-Source: AA6agR4m7a1ZpBafnXxHxc3Qgb+mIUi9eJAFTWTIHDXvxrcaOuXHorpWp86zCAS5S/bxu6e6MTh00Q== X-Received: by 2002:a5d:64aa:0:b0:21e:be27:6dfb with SMTP id m10-20020a5d64aa000000b0021ebe276dfbmr18180217wrp.456.1660146698952; Wed, 10 Aug 2022 08:51:38 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 02/11] xen/netback: use struct ubuf_info_msgzc Date: Wed, 10 Aug 2022 16:49:10 +0100 Message-Id: <9d13d0d17d672ce55a3d5cfd06d5c521270a7d4d.1660124059.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 struct ubuf_info will be changed, use ubuf_info_msgzc instead. Signed-off-by: Pavel Begunkov --- drivers/net/xen-netback/common.h | 2 +- drivers/net/xen-netback/interface.c | 4 ++-- drivers/net/xen-netback/netback.c | 7 ++++--- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/drivers/net/xen-netback/common.h b/drivers/net/xen-netback/common.h index 8174d7b2966c..1545cbee77a4 100644 --- a/drivers/net/xen-netback/common.h +++ b/drivers/net/xen-netback/common.h @@ -62,7 +62,7 @@ struct pending_tx_info { * ubuf_to_vif is a helper which finds the struct xenvif from a pointer * to this field. */ - struct ubuf_info callback_struct; + struct ubuf_info_msgzc callback_struct; }; #define XEN_NETIF_TX_RING_SIZE __CONST_RING_SIZE(xen_netif_tx, XEN_PAGE_SIZE) diff --git a/drivers/net/xen-netback/interface.c b/drivers/net/xen-netback/interface.c index fb32ae82d9b0..e579ecd40b74 100644 --- a/drivers/net/xen-netback/interface.c +++ b/drivers/net/xen-netback/interface.c @@ -591,8 +591,8 @@ int xenvif_init_queue(struct xenvif_queue *queue) } for (i = 0; i < MAX_PENDING_REQS; i++) { - queue->pending_tx_info[i].callback_struct = (struct ubuf_info) - { .callback = xenvif_zerocopy_callback, + queue->pending_tx_info[i].callback_struct = (struct ubuf_info_msgzc) + { { .callback = xenvif_zerocopy_callback }, { { .ctx = NULL, .desc = i } } }; queue->grant_tx_handle[i] = NETBACK_INVALID_HANDLE; diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c index a256695fc89e..3d2081bbbc86 100644 --- a/drivers/net/xen-netback/netback.c +++ b/drivers/net/xen-netback/netback.c @@ -133,7 +133,7 @@ static inline unsigned long idx_to_kaddr(struct xenvif_queue *queue, /* Find the containing VIF's structure from a pointer in pending_tx_info array */ -static inline struct xenvif_queue *ubuf_to_queue(const struct ubuf_info *ubuf) +static inline struct xenvif_queue *ubuf_to_queue(const struct ubuf_info_msgzc *ubuf) { u16 pending_idx = ubuf->desc; struct pending_tx_info *temp = @@ -1228,11 +1228,12 @@ static int xenvif_tx_submit(struct xenvif_queue *queue) return work_done; } -void xenvif_zerocopy_callback(struct sk_buff *skb, struct ubuf_info *ubuf, +void xenvif_zerocopy_callback(struct sk_buff *skb, struct ubuf_info *ubuf_base, bool zerocopy_success) { unsigned long flags; pending_ring_idx_t index; + struct ubuf_info_msgzc *ubuf = uarg_to_msgzc(ubuf_base); struct xenvif_queue *queue = ubuf_to_queue(ubuf); /* This is the only place where we grab this lock, to protect callbacks @@ -1241,7 +1242,7 @@ void xenvif_zerocopy_callback(struct sk_buff *skb, struct ubuf_info *ubuf, spin_lock_irqsave(&queue->callback_lock, flags); do { u16 pending_idx = ubuf->desc; - ubuf = (struct ubuf_info *) ubuf->ctx; + ubuf = (struct ubuf_info_msgzc *) ubuf->ctx; BUG_ON(queue->dealloc_prod - queue->dealloc_cons >= MAX_PENDING_REQS); index = pending_index(queue->dealloc_prod); From patchwork Wed Aug 10 15:49:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12941226 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 14E4CC3F6B0 for ; Thu, 11 Aug 2022 08:05:47 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.383697.619556 (Exim 4.92) (envelope-from ) id 1oM3C9-0004nU-70; Thu, 11 Aug 2022 08:05:33 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 383697.619556; Thu, 11 Aug 2022 08:05:33 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oM3C9-0004lI-1j; Thu, 11 Aug 2022 08:05:33 +0000 Received: by outflank-mailman (input) for mailman id 383697; Wed, 10 Aug 2022 15:51:41 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oLnzh-00016w-HZ for xen-devel@lists.xenproject.org; Wed, 10 Aug 2022 15:51:41 +0000 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [2a00:1450:4864:20::32a]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 53273659-18c4-11ed-924f-1f966e50362f; Wed, 10 Aug 2022 17:51:40 +0200 (CEST) Received: by mail-wm1-x32a.google.com with SMTP id ay12so5185324wmb.1 for ; Wed, 10 Aug 2022 08:51:40 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:39 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 53273659-18c4-11ed-924f-1f966e50362f DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=zKsqv0p5Obkg1TLb6/kUNEYbyvoYoWpfWSOapsGCJxI=; b=DFTQlY8ddG8nxlheZEdY1AhOzyhFf8i+cYlGvYaQA2jXHoy604PEPidno8824FFofg YVuRVy3aoGvd+VxIlbyTojzVgA6ObPKrHOe3N+BtwnJ8/SiHLCzaUthyY809sB53Uy0k A80hQEmxtBOQP5ObevE2QDwC8UdnYAyiC/9NRGBeNLRJhgqxJeTUEIEq3lyZCycgRAKv YeRgjQWebysXI2hCFaZdjrhd9yqY/GnXi1hw7QfcxVhAyGjXJUcOFXn8ZTBLEhYDSjyB tkKzffcoleVU5wSv7BENj8Sif4EsQ6FQBYNVk3yJnkXWSuO8uva7e43fGJIcAy/AtU+c ghIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=zKsqv0p5Obkg1TLb6/kUNEYbyvoYoWpfWSOapsGCJxI=; b=RneziP7bwJVyggCDpAuRtNqwknNQjKBMYFIkqqdzDqtE7ZO+J44koX4b06y91fbK7G wNbrzkJghTn+pkUJm//5Tjy1WLC5HR8gJqfWU/dD44L4BitnKMWoxzHWpBOMHLAqg3Rd 9r1u/GFAYWhTs196Ix6+MPVc8LK7m8HW6uqlVTKxjUxBjx0rW9Jc3mYp9Idw4XXit+h1 a7Jhn97WaooYSMrrJFsWNZ7RAZ/s2PYG/MMxqYEHtmpilEAX2FmhoNYxFnASOgZggjDq L5Dx9NkkKEAKEe2GupiCrro0hbTNDEsEYYk5GUqKDhi610dWV7yz642jzjWqmJBBKxKw r/KQ== X-Gm-Message-State: ACgBeo2nod5sAkeIonE9ZVKx5K2POu1npfHndWD3RKMaJymgk4c6sAJs MlW4PcKRVulgwZpEovkOwfM= X-Google-Smtp-Source: AA6agR7LF0ZwQ1BlKaVDicKs5LglruTEZwAxo7zyYgmWusQVZHSM24Cst0gJV7pB4XoOeH/qUhDY2A== X-Received: by 2002:a05:600c:28cd:b0:3a5:4f45:b927 with SMTP id h13-20020a05600c28cd00b003a54f45b927mr2964921wmd.90.1660146700292; Wed, 10 Aug 2022 08:51:40 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 03/11] vhost/net: use struct ubuf_info_msgzc Date: Wed, 10 Aug 2022 16:49:11 +0100 Message-Id: <87cfb629defa270e5ed953c6e501a47278f916c9.1660124059.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 struct ubuf_info will be changed, use ubuf_info_msgzc instead. Signed-off-by: Pavel Begunkov --- drivers/vhost/net.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index 68e4ecd1cc0e..9b616536dd9e 100644 --- a/drivers/vhost/net.c +++ b/drivers/vhost/net.c @@ -118,7 +118,7 @@ struct vhost_net_virtqueue { /* Number of XDP frames batched */ int batched_xdp; /* an array of userspace buffers info */ - struct ubuf_info *ubuf_info; + struct ubuf_info_msgzc *ubuf_info; /* Reference counting for outstanding ubufs. * Protected by vq mutex. Writers must also take device mutex. */ struct vhost_net_ubuf_ref *ubufs; @@ -288,7 +288,7 @@ static int vhost_net_set_ubuf_info(struct vhost_net *n) n->vqs[i].ubuf_info = kmalloc_array(UIO_MAXIOV, sizeof(*n->vqs[i].ubuf_info), - GFP_KERNEL); + GFP_KERNEL | __GFP_ZERO); if (!n->vqs[i].ubuf_info) goto err; } @@ -382,8 +382,9 @@ static void vhost_zerocopy_signal_used(struct vhost_net *net, } static void vhost_zerocopy_callback(struct sk_buff *skb, - struct ubuf_info *ubuf, bool success) + struct ubuf_info *ubuf_base, bool success) { + struct ubuf_info_msgzc *ubuf = uarg_to_msgzc(ubuf_base); struct vhost_net_ubuf_ref *ubufs = ubuf->ctx; struct vhost_virtqueue *vq = ubufs->vq; int cnt; @@ -871,7 +872,7 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock) size_t len, total_len = 0; int err; struct vhost_net_ubuf_ref *ubufs; - struct ubuf_info *ubuf; + struct ubuf_info_msgzc *ubuf; bool zcopy_used; int sent_pkts = 0; @@ -907,14 +908,14 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock) ubuf = nvq->ubuf_info + nvq->upend_idx; vq->heads[nvq->upend_idx].id = cpu_to_vhost32(vq, head); vq->heads[nvq->upend_idx].len = VHOST_DMA_IN_PROGRESS; - ubuf->callback = vhost_zerocopy_callback; ubuf->ctx = nvq->ubufs; ubuf->desc = nvq->upend_idx; - ubuf->flags = SKBFL_ZEROCOPY_FRAG; - refcount_set(&ubuf->refcnt, 1); + ubuf->ubuf.callback = vhost_zerocopy_callback; + ubuf->ubuf.flags = SKBFL_ZEROCOPY_FRAG; + refcount_set(&ubuf->ubuf.refcnt, 1); msg.msg_control = &ctl; ctl.type = TUN_MSG_UBUF; - ctl.ptr = ubuf; + ctl.ptr = &ubuf->ubuf; msg.msg_controllen = sizeof(ctl); ubufs = nvq->ubufs; atomic_inc(&ubufs->refcount); From patchwork Wed Aug 10 15:49:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12941236 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 845C1C2BB41 for ; Thu, 11 Aug 2022 08:05:50 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.383699.619564 (Exim 4.92) (envelope-from ) id 1oM3C9-0004wj-Jv; Thu, 11 Aug 2022 08:05:33 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 383699.619564; Thu, 11 Aug 2022 08:05:33 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oM3C9-0004sw-D4; Thu, 11 Aug 2022 08:05:33 +0000 Received: by outflank-mailman (input) for mailman id 383699; Wed, 10 Aug 2022 15:51:43 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oLnzj-00016q-8T for xen-devel@lists.xenproject.org; Wed, 10 Aug 2022 15:51:43 +0000 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [2a00:1450:4864:20::435]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 541ecaca-18c4-11ed-bd2e-47488cf2e6aa; Wed, 10 Aug 2022 17:51:42 +0200 (CEST) Received: by mail-wr1-x435.google.com with SMTP id n4so16465056wrp.10 for ; Wed, 10 Aug 2022 08:51:42 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:41 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 541ecaca-18c4-11ed-bd2e-47488cf2e6aa DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=k7b9egffbhRjX5/OjtOx/by90hRcurZmtiI07zTQ9r0=; b=DO5S08zBi3vihZLd+TPnKuLvUBVq1/APvJ0DNR/ZVpJjz4DlQk8pVoCf++lDJQ+qCe J5GiO2q7rRcXgkXHqvH+Xa7gnatlgZJV5TwqmNquXIn+gL91mG56ATFSsq9BpXpc7zr0 /1S04KGFgf42+WhssmeRXUhlovG5xP8Cx3T/uzDl82B5aLUk0pPU9HQVxh+wNlScY3LK CgQH74Kc4vSw87U7lXPsLH2C2ORUZZBxNzu/uVo2SSLsJLJNwcM+/ACew/XMVXYxRLdn VTO2yH9kAmRsQtleJsYuYUQbcBz0OVWz31KhDO6+31seXNLpFkbd/CzwI333S10XzntZ GzzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=k7b9egffbhRjX5/OjtOx/by90hRcurZmtiI07zTQ9r0=; b=AdwYvUhrPJImmIpqUfsocOuARRDqwfyvRtRncLpbURVVdKOZLNq9N5ezLo0959NMLs 0ozbQamR9jbCuqHCfvZijusXUUVeW2DkQM/kD9UNs8N+o614rOWUeAHb5WnmbqR4Romi igyZwOsG0a1rfEd8hl27uZ5dJUezxT92c2AYZ2vjHk+wOOKHLIn3EvEqYiUpNIpwjdOK M4fB64sSf5MqdMwYKQBVdiawTMnwYVX0JPh8vY/vLQbOMBTxLuNUBET83/eJCRbv9fZK JguvxSGFLaI2esAetCw44+N4lgCxL3fopxdxGW8DO6HKL6tvAP94Qq+cmO07L3Qup5UF pwuQ== X-Gm-Message-State: ACgBeo0zFu5Hj2c1tFt37Ws+mF33dkLqWk+FMCqsp292xDKgI9lEjEaa xi4TEYxPPLY9OriLvHdeyQ8= X-Google-Smtp-Source: AA6agR6suRfvAaYpMQRv+C0nU04QPftK1r02sWI7OLgKf0C0mEfWd53MX+qfOk7MeFvdNRw94olKdA== X-Received: by 2002:adf:db8f:0:b0:220:6059:5284 with SMTP id u15-20020adfdb8f000000b0022060595284mr18041360wri.33.1660146701918; Wed, 10 Aug 2022 08:51:41 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 04/11] net: shrink struct ubuf_info Date: Wed, 10 Aug 2022 16:49:12 +0100 Message-Id: <1e2d0bef44cd68b0e3dd106283f4536645c2a124.1660124059.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 We can benefit from a smaller struct ubuf_info, so leave only mandatory fields and let users to decide how they want to extend it. Convert MSG_ZEROCOPY to struct ubuf_info_msgzc and remove duplicated fields. This reduces the size from 48 bytes to just 16. Signed-off-by: Pavel Begunkov --- include/linux/skbuff.h | 22 ++++------------------ net/core/skbuff.c | 38 +++++++++++++++++++++----------------- net/ipv4/ip_output.c | 2 +- net/ipv4/tcp.c | 2 +- net/ipv6/ip6_output.c | 2 +- 5 files changed, 28 insertions(+), 38 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index f8ac3678dab8..afd7400d7f62 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -533,25 +533,8 @@ enum { struct ubuf_info { void (*callback)(struct sk_buff *, struct ubuf_info *, bool zerocopy_success); - union { - struct { - unsigned long desc; - void *ctx; - }; - struct { - u32 id; - u16 len; - u16 zerocopy:1; - u32 bytelen; - }; - }; refcount_t refcnt; u8 flags; - - struct mmpin { - struct user_struct *user; - unsigned int num_pg; - } mmp; }; struct ubuf_info_msgzc { @@ -570,7 +553,10 @@ struct ubuf_info_msgzc { }; }; - struct mmpin mmp; + struct mmpin { + struct user_struct *user; + unsigned int num_pg; + } mmp; }; #define skb_uarg(SKB) ((struct ubuf_info *)(skb_shinfo(SKB)->destructor_arg)) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 974bbbbe7138..b047a773acd7 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1183,7 +1183,7 @@ EXPORT_SYMBOL_GPL(mm_unaccount_pinned_pages); static struct ubuf_info *msg_zerocopy_alloc(struct sock *sk, size_t size) { - struct ubuf_info *uarg; + struct ubuf_info_msgzc *uarg; struct sk_buff *skb; WARN_ON_ONCE(!in_task()); @@ -1201,19 +1201,19 @@ static struct ubuf_info *msg_zerocopy_alloc(struct sock *sk, size_t size) return NULL; } - uarg->callback = msg_zerocopy_callback; + uarg->ubuf.callback = msg_zerocopy_callback; uarg->id = ((u32)atomic_inc_return(&sk->sk_zckey)) - 1; uarg->len = 1; uarg->bytelen = size; uarg->zerocopy = 1; - uarg->flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; - refcount_set(&uarg->refcnt, 1); + uarg->ubuf.flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; + refcount_set(&uarg->ubuf.refcnt, 1); sock_hold(sk); - return uarg; + return &uarg->ubuf; } -static inline struct sk_buff *skb_from_uarg(struct ubuf_info *uarg) +static inline struct sk_buff *skb_from_uarg(struct ubuf_info_msgzc *uarg) { return container_of((void *)uarg, struct sk_buff, cb); } @@ -1222,6 +1222,7 @@ struct ubuf_info *msg_zerocopy_realloc(struct sock *sk, size_t size, struct ubuf_info *uarg) { if (uarg) { + struct ubuf_info_msgzc *uarg_zc; const u32 byte_limit = 1 << 19; /* limit to a few TSO */ u32 bytelen, next; @@ -1237,8 +1238,9 @@ struct ubuf_info *msg_zerocopy_realloc(struct sock *sk, size_t size, return NULL; } - bytelen = uarg->bytelen + size; - if (uarg->len == USHRT_MAX - 1 || bytelen > byte_limit) { + uarg_zc = uarg_to_msgzc(uarg); + bytelen = uarg_zc->bytelen + size; + if (uarg_zc->len == USHRT_MAX - 1 || bytelen > byte_limit) { /* TCP can create new skb to attach new uarg */ if (sk->sk_type == SOCK_STREAM) goto new_alloc; @@ -1246,11 +1248,11 @@ struct ubuf_info *msg_zerocopy_realloc(struct sock *sk, size_t size, } next = (u32)atomic_read(&sk->sk_zckey); - if ((u32)(uarg->id + uarg->len) == next) { - if (mm_account_pinned_pages(&uarg->mmp, size)) + if ((u32)(uarg_zc->id + uarg_zc->len) == next) { + if (mm_account_pinned_pages(&uarg_zc->mmp, size)) return NULL; - uarg->len++; - uarg->bytelen = bytelen; + uarg_zc->len++; + uarg_zc->bytelen = bytelen; atomic_set(&sk->sk_zckey, ++next); /* no extra ref when appending to datagram (MSG_MORE) */ @@ -1286,7 +1288,7 @@ static bool skb_zerocopy_notify_extend(struct sk_buff *skb, u32 lo, u16 len) return true; } -static void __msg_zerocopy_callback(struct ubuf_info *uarg) +static void __msg_zerocopy_callback(struct ubuf_info_msgzc *uarg) { struct sk_buff *tail, *skb = skb_from_uarg(uarg); struct sock_exterr_skb *serr; @@ -1339,19 +1341,21 @@ static void __msg_zerocopy_callback(struct ubuf_info *uarg) void msg_zerocopy_callback(struct sk_buff *skb, struct ubuf_info *uarg, bool success) { - uarg->zerocopy = uarg->zerocopy & success; + struct ubuf_info_msgzc *uarg_zc = uarg_to_msgzc(uarg); + + uarg_zc->zerocopy = uarg_zc->zerocopy & success; if (refcount_dec_and_test(&uarg->refcnt)) - __msg_zerocopy_callback(uarg); + __msg_zerocopy_callback(uarg_zc); } EXPORT_SYMBOL_GPL(msg_zerocopy_callback); void msg_zerocopy_put_abort(struct ubuf_info *uarg, bool have_uref) { - struct sock *sk = skb_from_uarg(uarg)->sk; + struct sock *sk = skb_from_uarg(uarg_to_msgzc(uarg))->sk; atomic_dec(&sk->sk_zckey); - uarg->len--; + uarg_to_msgzc(uarg)->len--; if (have_uref) msg_zerocopy_callback(NULL, uarg, true); diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index d7bd1daf022b..546897a4b4fa 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1043,7 +1043,7 @@ static int __ip_append_data(struct sock *sk, paged = true; zc = true; } else { - uarg->zerocopy = 0; + uarg_to_msgzc(uarg)->zerocopy = 0; skb_zcopy_set(skb, uarg, &extra_uref); } } diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 970e9a2cca4a..3152da8f4763 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1239,7 +1239,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) } zc = sk->sk_route_caps & NETIF_F_SG; if (!zc) - uarg->zerocopy = 0; + uarg_to_msgzc(uarg)->zerocopy = 0; } } diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 897ca4f9b791..6d4f01a0cf6e 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1568,7 +1568,7 @@ static int __ip6_append_data(struct sock *sk, paged = true; zc = true; } else { - uarg->zerocopy = 0; + uarg_to_msgzc(uarg)->zerocopy = 0; skb_zcopy_set(skb, uarg, &extra_uref); } } From patchwork Wed Aug 10 15:49:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12941225 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DB162C19F2D for ; Thu, 11 Aug 2022 08:05:46 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.383701.619576 (Exim 4.92) (envelope-from ) id 1oM3CA-000579-93; Thu, 11 Aug 2022 08:05:34 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 383701.619576; Thu, 11 Aug 2022 08:05:34 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oM3C9-00054h-RU; Thu, 11 Aug 2022 08:05:33 +0000 Received: by outflank-mailman (input) for mailman id 383701; Wed, 10 Aug 2022 15:51:44 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oLnzk-00016q-FG for xen-devel@lists.xenproject.org; Wed, 10 Aug 2022 15:51:44 +0000 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [2a00:1450:4864:20::334]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 54eb7c8b-18c4-11ed-bd2e-47488cf2e6aa; Wed, 10 Aug 2022 17:51:43 +0200 (CEST) Received: by mail-wm1-x334.google.com with SMTP id i128-20020a1c3b86000000b003a536d58f73so1211964wma.4 for ; Wed, 10 Aug 2022 08:51:43 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:42 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 54eb7c8b-18c4-11ed-bd2e-47488cf2e6aa DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=aYyDEkTXwWBCvoGHlIPqb+/BxSLKuFl8ndVR481kS1w=; b=hZ9U5hOp4VB5FaNjAUnUraKpOCA2Wpyj5HNp6Hhmv23m66TTZRgMWtl/dQOYhFz/E9 36lXEdGbdo0OOYp9ub05DxHU0JI2ixvQfkVCuWQmznlpIUTvzJs+XnGqaJ2CMBu+LqhP BnPg0B38v5clDnOrD+9TKJLttKJu9NRZchU1iQUWn8KW95x2YDQFqqAhiIbkPwedVKlK NGR7PWWBnUvj5HLI2Nv9zCRnQk5F7LOPjdp0zI7zbkzXr3sK9gOJiNthyiUh1Y08msvy 8rfhtGaZQIdqI/4wm9WgW+yQdrbS+/dsw22V6yBOrwzgjM4RRxcF9hYdLnEirsOA1u+p JGmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=aYyDEkTXwWBCvoGHlIPqb+/BxSLKuFl8ndVR481kS1w=; b=4FHtAh8TaKod+uiqFWHNTVsazBx+12jeeLzaQ1vkirwTNeDl4WCfRnPy0qP5gxlMRm MurSEKhwADLOH+HY5XnUjha1nqb5NjZAj0mNGvNuFdrygHw6q0JhwDsUcq93d3Ki4DYP F/ZWIU9MzX6kyujElJcdfDVXAm/Z/p3rwQTN/j1OpZczLD3gQa2aKAMcK+7cB1tpM9vB geJL9aNN4z5eAj6Gl63Dixh3zNxquXxhJlCc5yDhE9gqMGFlgrdt+vcEiXfFuFk96z6p Rq41MZaawb9MVoHY+2UkJS1awA/LI1kMeHlLiLVVze0V1oUQytjzWLa4900gNsncxDbw +53g== X-Gm-Message-State: ACgBeo2xYBBHDeNkgOjhZE5TEEibm7BKTKAcNvYc5yrOgrzoWNYulEP0 BZYCdgzrSeZAXLUCdv6wBR4= X-Google-Smtp-Source: AA6agR4pIC1FSo3wRp5p1MAJqhHvgQE9nDnk6Uk00rTTOcb22zy1nLrULEj+sKAA+/DezVNGQLcv4Q== X-Received: by 2002:a05:600c:1e8d:b0:3a5:74d:c61c with SMTP id be13-20020a05600c1e8d00b003a5074dc61cmr2953296wmb.70.1660146703329; Wed, 10 Aug 2022 08:51:43 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 05/11] net: rename ubuf_info's flags Date: Wed, 10 Aug 2022 16:49:13 +0100 Message-Id: <7d43d8dd1c7e7c9964d40202482f325ecd06642b.1660124059.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 ubuf_info::flags contains SKBFL_* flags that we copy into skbs, change the field name to stress that it keeps skb flags. Signed-off-by: Pavel Begunkov --- include/linux/skbuff.h | 4 ++-- io_uring/notif.c | 2 +- net/core/skbuff.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index afd7400d7f62..e749b5d3868d 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -534,7 +534,7 @@ struct ubuf_info { void (*callback)(struct sk_buff *, struct ubuf_info *, bool zerocopy_success); refcount_t refcnt; - u8 flags; + u8 skb_flags; }; struct ubuf_info_msgzc { @@ -1664,7 +1664,7 @@ static inline void net_zcopy_get(struct ubuf_info *uarg) static inline void skb_zcopy_init(struct sk_buff *skb, struct ubuf_info *uarg) { skb_shinfo(skb)->destructor_arg = uarg; - skb_shinfo(skb)->flags |= uarg->flags; + skb_shinfo(skb)->flags |= uarg->skb_flags; } static inline void skb_zcopy_set(struct sk_buff *skb, struct ubuf_info *uarg, diff --git a/io_uring/notif.c b/io_uring/notif.c index b5f989dff9de..97cb4a7e8849 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -65,7 +65,7 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx, nd = io_notif_to_data(notif); nd->account_pages = 0; - nd->uarg.flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; + nd->uarg.skb_flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; nd->uarg.callback = io_uring_tx_zerocopy_callback; /* master ref owned by io_notif_slot, will be dropped on flush */ refcount_set(&nd->uarg.refcnt, 1); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index b047a773acd7..40bb84986800 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1206,7 +1206,7 @@ static struct ubuf_info *msg_zerocopy_alloc(struct sock *sk, size_t size) uarg->len = 1; uarg->bytelen = size; uarg->zerocopy = 1; - uarg->ubuf.flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; + uarg->ubuf.skb_flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; refcount_set(&uarg->ubuf.refcnt, 1); sock_hold(sk); From patchwork Wed Aug 10 15:49:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12941227 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4D9E7C25B0D for ; Thu, 11 Aug 2022 08:05:47 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.383703.619584 (Exim 4.92) (envelope-from ) id 1oM3CA-0005Hq-NC; Thu, 11 Aug 2022 08:05:34 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 383703.619584; Thu, 11 Aug 2022 08:05:34 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oM3CA-0005Ev-CU; Thu, 11 Aug 2022 08:05:34 +0000 Received: by outflank-mailman (input) for mailman id 383703; Wed, 10 Aug 2022 15:51:45 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oLnzl-00016w-Ps for xen-devel@lists.xenproject.org; Wed, 10 Aug 2022 15:51:45 +0000 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [2a00:1450:4864:20::32a]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 55c66db5-18c4-11ed-924f-1f966e50362f; Wed, 10 Aug 2022 17:51:45 +0200 (CEST) Received: by mail-wm1-x32a.google.com with SMTP id a18-20020a05600c349200b003a30de68697so1852975wmq.0 for ; Wed, 10 Aug 2022 08:51:45 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:44 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 55c66db5-18c4-11ed-924f-1f966e50362f DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=I6HOvD/OHCC6U7VTSloH8Bzb7BhYYX9aNTArIgD6mXg=; b=CvZ+o5GyX9Cl58rR5XQmVQ/WnA/GRIRdZxP33Dzgj6r9DPlQRphs6yOhzWqdULKen9 gMmXHh/unAvJbQnQA9hJTCa2uvfKeDglP05vwYf7ACs1xU/sY79GVh0ja78nBVBuQos9 At2FAi3HtK9PCPmiPaAo3PGgSrGuJe4zvs8ZzOggBwh7U2mW5q2DaPPvVs8/T9wnfdXz 94FcpAUm2zJ0X95SveH+CL7kQAQJnVwYQOQrhReSHaP1tz8rIPU18lNRtSV+dGg7djft 87OokEhFNIW6wyCD9lMh9IYzKCch26fi5GrPIxaxnKIQEuRdYWpJJ4Lz0KAThc69LX2g ITgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=I6HOvD/OHCC6U7VTSloH8Bzb7BhYYX9aNTArIgD6mXg=; b=V5XjQqJpiDQraXhGnF08AHY+r4WI6XBblL97n9Mvt5SXCMRwJ0KVEpVK3OGyImPSye 6BzMy/dEFLyaiu8PmrY12e2PDX99FRpPyIH0BtU01iaHghRf2y0VzfxrpmwFagVA6D36 aDoThVhFFMzaWTUIlX1aCOw65vCCAbW0JXn1V+P/uKD46icE/vAInuGOa4vyD6AVdAl1 7xqvq9gjxnq2DvIUix9CdY6oJoZroPIFYLCRbnkGL4ZtMTNIIUccKiqJbbZL2f81dtyb AlHTm8r/bv+fePm/1G4qmBcphyx5zsUJf7WMJxwEbMhaIgHHc8+KoHJGua6DKMne8nJz VG7Q== X-Gm-Message-State: ACgBeo3Mx8JhYoFjJ1q0e3jNNMG7VxW3ul0iMXoBf91MNHpQKTn8+p4p w36nZ9GHFILZmwmTk7ezmuI= X-Google-Smtp-Source: AA6agR45he4dyQKI12poNM8AGSPq8F4/kGUNUT0s11gtoLe3WPA20HuKlateIe0Lr4uUnp3aBCdobA== X-Received: by 2002:a05:600c:1e05:b0:3a5:b441:e9c with SMTP id ay5-20020a05600c1e0500b003a5b4410e9cmr2221515wmb.24.1660146704771; Wed, 10 Aug 2022 08:51:44 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 06/11] net: add flags for controlling ubuf_info Date: Wed, 10 Aug 2022 16:49:14 +0100 Message-Id: <433971d77b5a757b11ce5683ef1d0377efcc8544.1660124059.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 There are already skb_flags in ubuf_info, which enhancing skbs. Also add flags controlling ubuf_info, mainly to hint about various referencing aspects of it, which will be introduced in later patches. Signed-off-by: Pavel Begunkov --- include/linux/skbuff.h | 1 + io_uring/notif.c | 1 + net/core/skbuff.c | 1 + 3 files changed, 3 insertions(+) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index e749b5d3868d..2b2e0020030b 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -535,6 +535,7 @@ struct ubuf_info { bool zerocopy_success); refcount_t refcnt; u8 skb_flags; + u8 flags; }; struct ubuf_info_msgzc { diff --git a/io_uring/notif.c b/io_uring/notif.c index 97cb4a7e8849..a2ba1e35a59f 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -66,6 +66,7 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx, nd = io_notif_to_data(notif); nd->account_pages = 0; nd->uarg.skb_flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; + nd->uarg.flags = 0; nd->uarg.callback = io_uring_tx_zerocopy_callback; /* master ref owned by io_notif_slot, will be dropped on flush */ refcount_set(&nd->uarg.refcnt, 1); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 40bb84986800..7e102373482c 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1207,6 +1207,7 @@ static struct ubuf_info *msg_zerocopy_alloc(struct sock *sk, size_t size) uarg->bytelen = size; uarg->zerocopy = 1; uarg->ubuf.skb_flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; + uarg->ubuf.flags = 0; refcount_set(&uarg->ubuf.refcnt, 1); sock_hold(sk); From patchwork Wed Aug 10 15:49:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12941229 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 91D9DC25B0E for ; Thu, 11 Aug 2022 08:05:48 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.383705.619589 (Exim 4.92) (envelope-from ) id 1oM3CB-0005Qe-2E; Thu, 11 Aug 2022 08:05:35 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 383705.619589; Thu, 11 Aug 2022 08:05:34 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oM3CA-0005On-RJ; Thu, 11 Aug 2022 08:05:34 +0000 Received: by outflank-mailman (input) for mailman id 383705; Wed, 10 Aug 2022 15:51:47 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oLnzn-00016q-Mc for xen-devel@lists.xenproject.org; Wed, 10 Aug 2022 15:51:47 +0000 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [2a00:1450:4864:20::42f]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 56d6cb5f-18c4-11ed-bd2e-47488cf2e6aa; Wed, 10 Aug 2022 17:51:47 +0200 (CEST) Received: by mail-wr1-x42f.google.com with SMTP id j1so18275492wrw.1 for ; Wed, 10 Aug 2022 08:51:46 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:46 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 56d6cb5f-18c4-11ed-bd2e-47488cf2e6aa DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=h9HHmdtmbDaBgAkYhJr0MaG2LsaIMYK3/tGmura3Q/c=; b=RkQDnrNrgozMvW1slfCx3exnfLZSPNL0SE3J0CmXcn4NPQmzKH1nUhVoCi4rJ/bGhQ qWkiTJzALDavZu1xxA3ht8UtZU4n6tL4iHm1Zx/Em0r6K4Qe5KsTNb1lHxoLPs9CxUrL ZNGihmk1LOvOohQDQ1AY3Ulw8RiLhtDxqyohg/rvq0SO7NQvrO7saW7gPAaH7++iBvJP Jl1azWCXpOvZRwom0TDdfqiZB33KjKODuKkdpzGyNypg8/uceMD9cO1ihA+jI9r5ZcC3 TYYoAUSpD/Fkw5BOO1iG7WSOK8lPFUPa6a5TlPYgWBB6P/+e3jyrNxZORTfBjtmz0zUS rYzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=h9HHmdtmbDaBgAkYhJr0MaG2LsaIMYK3/tGmura3Q/c=; b=gqrINwAblhQzlCwGHkavlNMWU+GHwUsvJ1tPHgZKZ9S6hpRhfrQtqdLGaNZP1dKJ8A BGB3wrC6cZl5URcgGGU85dsgmxtL8082aMGzME7+OmhBlQowzRQFeOIP4TGgQ2CrF5Cx 6DOviDb50fEQwfSL4EqM/2Mw2ghHJCan2kKlSySOYXiPHJG/tJFN0Uo+wvHYnfJRaNzR xypIDHSYootR/5ueXcr1xXTGp+jbhf1e6rH8/8s7cez429ndiuMSZ63llqNleMzHj8VX 0SSRXN+rh4XGluFNlxLemBxRNyJleI23rma3zwhbUtMO6ohbn/YGmmRA0Dvf8nQVo4dV y1OQ== X-Gm-Message-State: ACgBeo2CNKq43octZdG6Ec04vreyRn5eBNXAHS7gT2QkUEPwTbzbXn8o 1V9mvyNscg0XxUkSiGVJbPo= X-Google-Smtp-Source: AA6agR6ZNn9gZ3GD3qgxjdOVbL8JXumKhuxwUdsZoAGuCONXxJ2F64wM4uAcQJPM/4Jm+GN1m/z5nQ== X-Received: by 2002:a5d:634d:0:b0:220:5ff7:3969 with SMTP id b13-20020a5d634d000000b002205ff73969mr16857681wrw.709.1660146706405; Wed, 10 Aug 2022 08:51:46 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 07/11] net/tcp: optimise tcp ubuf refcounting Date: Wed, 10 Aug 2022 16:49:15 +0100 Message-Id: <9a42187cdc9ce034fd23179c7b31d7cc6a54bd45.1660124059.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 Add UARGFL_CALLER_PINNED letting protocols know that the caller holds a reference to the ubuf_info and so it doesn't need additional refcounting for purposes of keeping it alive. With that TCP can save a refcount put/get pair per send when used with ->msg_ubuf. Signed-off-by: Pavel Begunkov --- include/linux/skbuff.h | 7 +++++++ net/ipv4/tcp.c | 9 ++++++--- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 2b2e0020030b..45fe7f0648d0 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -522,6 +522,13 @@ enum { #define SKBFL_ALL_ZEROCOPY (SKBFL_ZEROCOPY_FRAG | SKBFL_PURE_ZEROCOPY | \ SKBFL_DONT_ORPHAN | SKBFL_MANAGED_FRAG_REFS) +enum { + /* The caller holds a reference during the submission so the ubuf won't + * be freed until we return. + */ + UARGFL_CALLER_PINNED = BIT(0), +}; + /* * The callback notifies userspace to release buffers when skb DMA is done in * lower device, the skb last reference should be 0 when calling this. diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 3152da8f4763..4925107de57d 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1229,7 +1229,8 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (msg->msg_ubuf) { uarg = msg->msg_ubuf; - net_zcopy_get(uarg); + if (!(uarg->flags & UARGFL_CALLER_PINNED)) + net_zcopy_get(uarg); zc = sk->sk_route_caps & NETIF_F_SG; } else if (sock_flag(sk, SOCK_ZEROCOPY)) { uarg = msg_zerocopy_realloc(sk, size, skb_zcopy(skb)); @@ -1455,7 +1456,8 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) tcp_push(sk, flags, mss_now, tp->nonagle, size_goal); } out_nopush: - net_zcopy_put(uarg); + if (uarg && !(uarg->flags & UARGFL_CALLER_PINNED)) + net_zcopy_put(uarg); return copied + copied_syn; do_error: @@ -1464,7 +1466,8 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (copied + copied_syn) goto out; out_err: - net_zcopy_put_abort(uarg, true); + if (uarg && !(uarg->flags & UARGFL_CALLER_PINNED)) + net_zcopy_put_abort(uarg, true); err = sk_stream_error(sk, flags, err); /* make sure we wake any epoll edge trigger waiter */ if (unlikely(tcp_rtx_and_write_queues_empty(sk) && err == -EAGAIN)) { From patchwork Wed Aug 10 15:49:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12941233 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 026A7C28B2B for ; Thu, 11 Aug 2022 08:05:50 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.383707.619603 (Exim 4.92) (envelope-from ) id 1oM3CC-0005fQ-C7; Thu, 11 Aug 2022 08:05:36 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 383707.619603; Thu, 11 Aug 2022 08:05:36 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oM3CB-0005cY-FN; Thu, 11 Aug 2022 08:05:35 +0000 Received: by outflank-mailman (input) for mailman id 383707; Wed, 10 Aug 2022 15:51:48 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oLnzo-00016w-NB for xen-devel@lists.xenproject.org; Wed, 10 Aug 2022 15:51:48 +0000 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [2a00:1450:4864:20::32a]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 579a2e61-18c4-11ed-924f-1f966e50362f; Wed, 10 Aug 2022 17:51:48 +0200 (CEST) Received: by mail-wm1-x32a.google.com with SMTP id a18-20020a05600c349200b003a30de68697so1853031wmq.0 for ; Wed, 10 Aug 2022 08:51:48 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:47 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 579a2e61-18c4-11ed-924f-1f966e50362f DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=itYGv2JZK8WBFRrJm44D9H1rA7HcrDCnR5GlKfK0jhs=; b=g4eoVr1fu36ewbAmfG/t6UJBFOF1oHPXDiKrqdGSPIInTe4GhXxac8aY/xl6qj4Ry6 hjJIrOjXgd6Yh8flP1MMruBaz19LaJGjliJaVdqyUgKhS5Iv3yZ5EXnaluD0BcKphXnA a5VMobfClu6mDyYL8AgAxvtUKvP55/JPpMBJHMu3wEu4WaWTzPCZhMYmHWz8b0MVDXhx GJ1Mj+/b/tEIWLH+pz8xBbXVEHEgTx42XMs5iv3QJxf8DSyr0HdmFD+Q1l3bx4FME1BL RC5BlkaAdauVJCjSBnJ19eyaQPimR9ZGZKGI4e8RTaSx6XcGE3jjRf3+QIMjbpvGpNmL Nr8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=itYGv2JZK8WBFRrJm44D9H1rA7HcrDCnR5GlKfK0jhs=; b=Eu1x/h1pKkKvMyI/hKC0fdssmeG+AuM7xF6yGrQkIUOxOYnVQhwoZt2wPPS48d2GTi Co7g1yHyqXGaD4bpYWEaYj8sj8HwwibAayborxLVKQZBrhnlceb5CD5nv4d0fCSPfBJ/ nVE5UPNWnndtegyGqyxVa9cfLZQta0JLyTRO2iBrXiqNZGu9Mk1E9IMv1y93wvVa5hsN uXtup5G9PMu9WS0GgzL/EaZv3J7gtKLDDD1X6VXGxips3JJ7MO1TQOXjC2KhSmaio6nI +xe8tVdU63qi4YHtc7ZC8d5jdBIu+PKHNnqhdphi4fYl31DdPmC6R9rzBSWjd4v5cx3V DjBg== X-Gm-Message-State: ACgBeo2L9Tv91Dc3U5ywlP7ZBlFZccwFmReyvj83G1OAOMCiArXaleyR 6E80RVGfQ3Je4SCOk5AzMkg= X-Google-Smtp-Source: AA6agR5tWoJNmIKULSv6qED4kYicFbD6aj95a2kTMgyfbOltADMEJMZMCY4KEhAPbmXEqnmelWqLpA== X-Received: by 2002:a05:600c:6009:b0:3a5:b069:5d34 with SMTP id az9-20020a05600c600900b003a5b0695d34mr2478342wmb.115.1660146707842; Wed, 10 Aug 2022 08:51:47 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 08/11] net: let callers provide ->msg_ubuf refs Date: Wed, 10 Aug 2022 16:49:16 +0100 Message-Id: <526fe4cb9cda287bedfc92b3888b48a4f3b0250b.1660124059.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 Some msg_ubuf providers like io_uring can keep elaborated ubuf_info reference batching and caching, so it will be of benefit to let the network layer to optionally steal some of the cached refs. Add UARGFL_GIFT_REF, if set the caller has at least one extra reference that it can gift away. If the network decides to take the ref it should clear the flag. Signed-off-by: Pavel Begunkov --- include/linux/skbuff.h | 14 ++++++++++++++ net/ipv4/ip_output.c | 1 + net/ipv6/ip6_output.c | 1 + 3 files changed, 16 insertions(+) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 45fe7f0648d0..972ec676e222 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -527,6 +527,11 @@ enum { * be freed until we return. */ UARGFL_CALLER_PINNED = BIT(0), + + /* The caller can gift one ubuf reference. The flag should be cleared + * when the reference is taken. + */ + UARGFL_GIFT_REF = BIT(1), }; /* @@ -1709,6 +1714,15 @@ static inline void net_zcopy_put(struct ubuf_info *uarg) uarg->callback(NULL, uarg, true); } +static inline bool net_zcopy_get_gift_ref(struct ubuf_info *uarg) +{ + bool has_ref; + + has_ref = uarg->flags & UARGFL_GIFT_REF; + uarg->flags &= ~UARGFL_GIFT_REF; + return has_ref; +} + static inline void net_zcopy_put_abort(struct ubuf_info *uarg, bool have_uref) { if (uarg) { diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 546897a4b4fa..9d42b6dd6b78 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1032,6 +1032,7 @@ static int __ip_append_data(struct sock *sk, paged = true; zc = true; uarg = msg->msg_ubuf; + extra_uref = net_zcopy_get_gift_ref(uarg); } } else if (sock_flag(sk, SOCK_ZEROCOPY)) { uarg = msg_zerocopy_realloc(sk, length, skb_zcopy(skb)); diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 6d4f01a0cf6e..8d8a8bbdb8df 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1557,6 +1557,7 @@ static int __ip6_append_data(struct sock *sk, paged = true; zc = true; uarg = msg->msg_ubuf; + extra_uref = net_zcopy_get_gift_ref(uarg); } } else if (sock_flag(sk, SOCK_ZEROCOPY)) { uarg = msg_zerocopy_realloc(sk, length, skb_zcopy(skb)); From patchwork Wed Aug 10 15:49:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12941230 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6D0A4C25B06 for ; Thu, 11 Aug 2022 08:05:47 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.383710.619610 (Exim 4.92) (envelope-from ) id 1oM3CC-0005oE-Qj; Thu, 11 Aug 2022 08:05:36 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 383710.619610; Thu, 11 Aug 2022 08:05:36 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oM3CB-0005jh-Tb; Thu, 11 Aug 2022 08:05:35 +0000 Received: by outflank-mailman (input) for mailman id 383710; Wed, 10 Aug 2022 15:51:50 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oLnzq-00016w-9S for xen-devel@lists.xenproject.org; Wed, 10 Aug 2022 15:51:50 +0000 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [2a00:1450:4864:20::32d]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 586c0349-18c4-11ed-924f-1f966e50362f; Wed, 10 Aug 2022 17:51:49 +0200 (CEST) Received: by mail-wm1-x32d.google.com with SMTP id q1-20020a05600c040100b003a52db97fffso1193011wmb.4 for ; Wed, 10 Aug 2022 08:51:49 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:48 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 586c0349-18c4-11ed-924f-1f966e50362f DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=P8aAKAj6QbUGM3qiLbxj6rCIuSDWisCzVUCts73nG7k=; b=iw4zMeJIq6Iduj5Ai23kLZLWv6lH2honhKvhFD/8RjX03J6CC6OT3GFZdBOkedBUdt n+4sEAA+jY/L/94w0veZ6+BigRGb/z+iiaiJDK2kbDqMnLkYpIKAVeEpBGeRkUhR8ZY4 3C2zmmMBUj+haew1vR4ZvQFcsNz2n5JD1clVwf0uHdPmnBF3QMz0ffFPSnpU76csdYvx WghdcQ+nasGK3chHwCDoPTmwBdCerSaeh8eCSRiiRqlv5NxV5e1DoaVA1cZ7GwqNoWTR Iu2g0zXQ4q7bhvEbKDfbxuVUgmKMV7FWlYgxIhokbOIyG0ObnwT0cf/lwdYDKKmbRxeE YrCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=P8aAKAj6QbUGM3qiLbxj6rCIuSDWisCzVUCts73nG7k=; b=GuZeUyiouTVg0qvaoa1ubwMRhZwwr1q0byzgSuSg4RZW2Qj6ctkiPBzQRI0nKaYGbJ dej5nyNcdEJLbMzbbmivnlgK9r01jEL9aHexV1rlMK9QA2I7ojzKQsG7wN50tNPiLUMu RLB+4N+X/BpRpAcgEb8iTRg8IHAPLvPuEQ/OiQclJRz+G5fVd0/fv6Kz1SpwcNWjNpTH Uqj16tjVNgUFAcNDzY2KXC0avuS8pz31u1nxQP3G24INXItUMt3wDF47PQgm/BRCsYJz nZH2zVvqh9d6n/r6DyVpJtiTv+6lh+k4bQeQz2cecvjgf4gm4ARpQNwohbBSAO/+GibQ xPpA== X-Gm-Message-State: ACgBeo2AG//HtraBIRmJhNM+wqvvoXecAiAIOWEHfExxPVfs1+XSTXH1 eaY6XadsbCqwnwC9M4EOPj+glHhCwpE= X-Google-Smtp-Source: AA6agR51LI/17gzAJrVldihDliUyILNuNHqftq15rGqHxVJN5cbfccCeChTUV2Xc8UnWpC6KUoJk0Q== X-Received: by 2002:a1c:f209:0:b0:3a4:f42c:9ffb with SMTP id s9-20020a1cf209000000b003a4f42c9ffbmr2964735wmc.62.1660146709173; Wed, 10 Aug 2022 08:51:49 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 09/11] io_uring/notif: add helper for flushing refs Date: Wed, 10 Aug 2022 16:49:17 +0100 Message-Id: X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 Add a helper for dropping notification references during flush. It's a preparation patch, currently it's only one master ref, but we're going to add ref caching. Signed-off-by: Pavel Begunkov --- io_uring/notif.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/io_uring/notif.c b/io_uring/notif.c index a2ba1e35a59f..5661681b3b44 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -73,6 +73,13 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx, return notif; } +static inline bool io_notif_drop_refs(struct io_notif_data *nd) +{ + int refs = 1; + + return refcount_sub_and_test(refs, &nd->uarg.refcnt); +} + void io_notif_slot_flush(struct io_notif_slot *slot) __must_hold(&ctx->uring_lock) { @@ -81,8 +88,7 @@ void io_notif_slot_flush(struct io_notif_slot *slot) slot->notif = NULL; - /* drop slot's master ref */ - if (refcount_dec_and_test(&nd->uarg.refcnt)) + if (io_notif_drop_refs(nd)) io_notif_complete(notif); } @@ -97,13 +103,11 @@ __cold int io_notif_unregister(struct io_ring_ctx *ctx) for (i = 0; i < ctx->nr_notif_slots; i++) { struct io_notif_slot *slot = &ctx->notif_slots[i]; struct io_kiocb *notif = slot->notif; - struct io_notif_data *nd; if (!notif) continue; - nd = io_kiocb_to_cmd(notif); slot->notif = NULL; - if (!refcount_dec_and_test(&nd->uarg.refcnt)) + if (!io_notif_drop_refs(io_kiocb_to_cmd(notif))) continue; notif->io_task_work.func = __io_notif_complete_tw; io_req_task_work_add(notif); From patchwork Wed Aug 10 15:49:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12941228 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E2F22C282E7 for ; Thu, 11 Aug 2022 08:05:48 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.383711.619616 (Exim 4.92) (envelope-from ) id 1oM3CD-00062o-Lg; Thu, 11 Aug 2022 08:05:37 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 383711.619616; Thu, 11 Aug 2022 08:05:37 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oM3CC-0005wT-SS; Thu, 11 Aug 2022 08:05:36 +0000 Received: by outflank-mailman (input) for mailman id 383711; Wed, 10 Aug 2022 15:51:51 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oLnzr-00016w-KA for xen-devel@lists.xenproject.org; Wed, 10 Aug 2022 15:51:51 +0000 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [2a00:1450:4864:20::333]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 59340c41-18c4-11ed-924f-1f966e50362f; Wed, 10 Aug 2022 17:51:51 +0200 (CEST) Received: by mail-wm1-x333.google.com with SMTP id 186-20020a1c02c3000000b003a34ac64bdfso1222800wmc.1 for ; Wed, 10 Aug 2022 08:51:51 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:50 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 59340c41-18c4-11ed-924f-1f966e50362f DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=KeuLRkJHac7T18EvJz7V2qjxDYwlB1JNeLqOJx59kSU=; b=NwaYFZmlXQkYLPFX3ZTPko8HcLgEHLwlrcyFvHq9hm7vQV2IwxrttH0/p8/BnnmZQl BTO0b3I01yD4gVwVuca1h/jbv+Cfysmr/pwkJd2AgCZmA4nNJaqDDMtHbPuEcJweTVo2 579S6P3KHIkfq4qhjgykgSQx3GEN7jkMGKOWCq1fqwahARcWVFoKAElwF2/ur8B6ClIo 4VUYL64JbAwxyO16ryqLgQ35TM4VOjpGgJE30dWaAoTd8zwN9ejtzD9yLFz3cxaGQo9r I+xFxR/ul8X0T8pEmed+ACyAEzgpzPI6KoqWeaF80317jz0w1JJi3GWEirlZLAj1Zjjz DcIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=KeuLRkJHac7T18EvJz7V2qjxDYwlB1JNeLqOJx59kSU=; b=u/Jrh0sBT780/L3Zu3KiTav3P/a5x4g4mEpqGuwdVeCdIsG04IB6tPO0Ec+hQB3Sbg brDxP3y8OXV70ZR1bD04+c2jObcRXwIlGoFnhB8kneooDTCUVUa6Nxbir0hBBUzZk1pY MgtuP+Tpw5Q2j0+tlGPrVUM2PCjT11BTMzvPolIzQFUaArMxnvbctbh1QFm7tSGl9lmc H4CxNp9egwW5V+5P1SW1KpH8eyjzKhuncyjGQ/brNIDenV2JUISrPJZufHM2tuY1R/zZ QwKVgE9c8tlF+GaggjdPBOGlBCLFkQauRuUleGuWvpfgiXp0Jw9MyJ9Mwg2pKQkkQZrZ aFyw== X-Gm-Message-State: ACgBeo0TuqNA424/8aNu+Airqp7Brmh0JQdB8w99EO96SMgy54rO2Dfs QgaIeJEeaOePRmXW0yg0Ngk= X-Google-Smtp-Source: AA6agR6Ld0dgt8VkL4NPznZl6LE5uIBV6RhDIhjeXAmILYOIjoMjQERHnQMZ5pvfGHOhFOpmDqlmgQ== X-Received: by 2002:a7b:c852:0:b0:3a5:407a:76df with SMTP id c18-20020a7bc852000000b003a5407a76dfmr2851591wml.101.1660146710481; Wed, 10 Aug 2022 08:51:50 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 10/11] io_uring/notif: mark notifs with UARGFL_CALLER_PINNED Date: Wed, 10 Aug 2022 16:49:18 +0100 Message-Id: <19cad183acc3a44b17b76c1719ad30c80aeff1ef.1660124059.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 We always keep references to active notifications and drop them only when we flush, so they're always pinned during sock_sendmsg() and we can add UARGFL_CALLER_PINNED. Signed-off-by: Pavel Begunkov --- io_uring/notif.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/notif.c b/io_uring/notif.c index 5661681b3b44..dd346ea67580 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -66,7 +66,7 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx, nd = io_notif_to_data(notif); nd->account_pages = 0; nd->uarg.skb_flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; - nd->uarg.flags = 0; + nd->uarg.flags = UARGFL_CALLER_PINNED; nd->uarg.callback = io_uring_tx_zerocopy_callback; /* master ref owned by io_notif_slot, will be dropped on flush */ refcount_set(&nd->uarg.refcnt, 1); From patchwork Wed Aug 10 15:49:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12941234 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 24619C25B0F for ; Thu, 11 Aug 2022 08:05:50 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.383713.619627 (Exim 4.92) (envelope-from ) id 1oM3CF-0006K3-1A; Thu, 11 Aug 2022 08:05:39 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 383713.619627; Thu, 11 Aug 2022 08:05:38 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oM3CE-0006Fc-7p; Thu, 11 Aug 2022 08:05:38 +0000 Received: by outflank-mailman (input) for mailman id 383713; Wed, 10 Aug 2022 15:51:53 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oLnzs-00016q-S2 for xen-devel@lists.xenproject.org; Wed, 10 Aug 2022 15:51:52 +0000 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [2a00:1450:4864:20::334]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 5a09326b-18c4-11ed-bd2e-47488cf2e6aa; Wed, 10 Aug 2022 17:51:52 +0200 (CEST) Received: by mail-wm1-x334.google.com with SMTP id i128-20020a1c3b86000000b003a536d58f73so1212195wma.4 for ; Wed, 10 Aug 2022 08:51:52 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:51 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 5a09326b-18c4-11ed-bd2e-47488cf2e6aa DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=8bvrycIBhC98AAS95EkRU1aUeIYv5cmtQ6g1fcH8Ls4=; b=NXRHpYfu0yj454B9TNqMSmsanOBuZYbEWiz+69QOiDC+DiJ2G+DWR8eQVlY70gQLu2 d0y7fh2jjsSTdciIvCwRVN4S9hece9UJOGa8XGh2v8vTiQdh6mNLdQQ8Fav5YfXJjcUU khtZG1kVvwRuto8hvHjfgINz8qFheZKolPC7tRow4X5InG3ks03RinhMEB69q/HPIghp RR52RfGpkiacBVtnyrW+hJzjBXI1YrXWzEp3tbUt8VcNjh69N5Ru/lSwr+rsyQJoHgS+ 05TXvir/g7FILT8dTPmjkGDy/QS+ouQWtNodRMedLwj3LN6GVEXfdRJ6j4sUIlZrGHUR ilkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=8bvrycIBhC98AAS95EkRU1aUeIYv5cmtQ6g1fcH8Ls4=; b=xkh+JqrfMo1fcX0a/DAeHtE/lamGUGXAZ/sD+m/msDPIXW3+vIijaEAgGMMHtOYhQm Qw6O/QUJUrD//CMqStYBmQw1JJDlGkK5QHxQy5DwxTqZ++2LDMVfTQ4PEUokPye2AEPU /pesd5FKjZjyJ2lrF3C43C7SXvEbR4b/szuhIbrDp36T+EMdhMYZMRxfyve2YQU91x9Q A+0WF74uI79DBynf7+lzJTBfVXUFVwgX5k8FNuQmaCLjYFx39XWn5CxmvuxS57qQbYP+ IUpRKBFTn4Hw9hBtanfKE2x6I0010KxSy12EX0wjZHbV+jo9F7zEEVCcN7aZLmy4os70 S7Qw== X-Gm-Message-State: ACgBeo3u9f4c0JgKDOC70WS8xA5679BKe+Kedpg1OSEAzsN3fDIkcH+7 8Zg9FdmyjZfOy2IORQds6HM= X-Google-Smtp-Source: AA6agR6ysliwRQwjHrbtgRjuvt8Ybcqj6h9fHpK3wukZ1ZPxN88XDbQ1aaLBOWU/0aq+IGK8xrnzTA== X-Received: by 2002:a05:600c:4ecd:b0:3a3:3eb0:9101 with SMTP id g13-20020a05600c4ecd00b003a33eb09101mr3007084wmq.49.1660146711936; Wed, 10 Aug 2022 08:51:51 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 11/11] io_uring/notif: add ubuf_info ref caching Date: Wed, 10 Aug 2022 16:49:19 +0100 Message-Id: <9514ddeea83902c36887ce764bc0306a518af73c.1660124059.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 Cache some active notifier references at the io_uring side and get them in batches, so the ammortised cost is low. Then these references can be given away to the network layer using UARGFL_GIFT_REF. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 8 +++++++- io_uring/notif.c | 6 ++++-- io_uring/notif.h | 22 +++++++++++++++++++++- 3 files changed, 32 insertions(+), 4 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index e6fc9748fbd2..bdaf9b10bd1b 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -949,6 +949,7 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) struct io_sendzc *zc = io_kiocb_to_cmd(req); struct io_notif_slot *notif_slot; struct io_kiocb *notif; + struct ubuf_info *ubuf; struct msghdr msg; struct iovec iov; struct socket *sock; @@ -1007,10 +1008,15 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) min_ret = iov_iter_count(&msg.msg_iter); msg.msg_flags = msg_flags; - msg.msg_ubuf = &io_notif_to_data(notif)->uarg; msg.sg_from_iter = io_sg_from_iter; + msg.msg_ubuf = ubuf = &io_notif_to_data(notif)->uarg; + ubuf->flags |= UARGFL_GIFT_REF; ret = sock_sendmsg(sock, &msg); + /* check if the send consumed an additional ref */ + if (likely(!(ubuf->flags & UARGFL_GIFT_REF))) + io_notif_consume_ref(notif); + if (unlikely(ret < min_ret)) { if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) return -EAGAIN; diff --git a/io_uring/notif.c b/io_uring/notif.c index dd346ea67580..73bbda5de07d 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -68,15 +68,17 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx, nd->uarg.skb_flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; nd->uarg.flags = UARGFL_CALLER_PINNED; nd->uarg.callback = io_uring_tx_zerocopy_callback; + nd->cached_refs = IO_NOTIF_REF_CACHE_NR; /* master ref owned by io_notif_slot, will be dropped on flush */ - refcount_set(&nd->uarg.refcnt, 1); + refcount_set(&nd->uarg.refcnt, IO_NOTIF_REF_CACHE_NR + 1); return notif; } static inline bool io_notif_drop_refs(struct io_notif_data *nd) { - int refs = 1; + int refs = nd->cached_refs + 1; + nd->cached_refs = 0; return refcount_sub_and_test(refs, &nd->uarg.refcnt); } diff --git a/io_uring/notif.h b/io_uring/notif.h index 0819304d7e00..2a263055a53b 100644 --- a/io_uring/notif.h +++ b/io_uring/notif.h @@ -9,11 +9,14 @@ #define IO_NOTIF_SPLICE_BATCH 32 #define IORING_MAX_NOTIF_SLOTS (1U << 10) +#define IO_NOTIF_REF_CACHE_NR 64 struct io_notif_data { struct file *file; - struct ubuf_info uarg; unsigned long account_pages; + /* extra uarg->refcnt refs */ + int cached_refs; + struct ubuf_info uarg; }; struct io_notif_slot { @@ -88,3 +91,20 @@ static inline int io_notif_account_mem(struct io_kiocb *notif, unsigned len) } return 0; } + +static inline void io_notif_consume_ref(struct io_kiocb *notif) + __must_hold(&ctx->uring_lock) +{ + struct io_notif_data *nd = io_notif_to_data(notif); + + nd->cached_refs--; + + /* + * Issue sends without looking at notif->cached_refs first, so we + * always have to have at least one ref cached + */ + if (unlikely(!nd->cached_refs)) { + refcount_add(IO_NOTIF_REF_CACHE_NR, &nd->uarg.refcnt); + nd->cached_refs += IO_NOTIF_REF_CACHE_NR; + } +}