From patchwork Thu Apr 28 10:58:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12830466 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE953C433F5 for ; Thu, 28 Apr 2022 11:00:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345164AbiD1LEA (ORCPT ); Thu, 28 Apr 2022 07:04:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47336 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343930AbiD1LDH (ORCPT ); Thu, 28 Apr 2022 07:03:07 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1121A9AE61; Thu, 28 Apr 2022 03:59:28 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id q23so6256678wra.1; Thu, 28 Apr 2022 03:59:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JJ/Jn2CaItPG2KrWE6z6rbp+t2EL7d23+IBwd5W6Kr8=; b=BTtmqWGCmABxPW53lwZmD/b/VAE84IqlAOJkfGCYrOQDB7rvCIti4hkbumbARY7ES6 nwbMp4prigAlU5k7f/bhzLS4eM9dQplfPndwW9fBjV/FENYRYDcBU5KpjmPfFOgZB7sI 6Jfu5492r8jqQ83K88JNvy3VjQRTa660KQ98gesnYPFtHJ/wzZK7JzLn0R9FNeHMndTH Ji6OKF16MkdxyTtImsBlElcq5s0KJYX9Cu5LzsAi0o3zE7akyA22VBYyLeQjeINg131f +ilEmO+xquKB3OLOvi1zdWlYXNaOIrlzZ6tW/FqWVRvDnFhdCXH8gtcj4e26Yhlr5z+Z MtJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JJ/Jn2CaItPG2KrWE6z6rbp+t2EL7d23+IBwd5W6Kr8=; b=brBadw7XtuxtfvWyqcXr51E20v6HU8wh5kfPLoR4NxhoejFGtYmPLH/5obk7yh89No qtVDZ5Inv147aZELjEWwXPE/KaJ+B26wmqezWqQrvj8pTZKGU+i4Tk4siKn0KDRStaxE UE01rWRAQK5im9lstKAuUucQ0BO3xVZHc5WxePy+j59CkuAARYYKJLIEIn1cjCoLMRi1 cw7ZarrGjJGunCP3ChMyn0dqYPV3lOIMTAeCoKgEYyWzkjoLQAZpGGQm4jNmJM0zKZfQ 0+wVZ9ADak+zlcy57Qp09kXnm6BBABvgpX06wNuPMiR0gQmBL1/KVT0430RjaihU7FWB rt/Q== X-Gm-Message-State: AOAM531pLqcGESGOq6mevOX7ODRQq5EEdvWsum89608fqWHlANrLPNEE lKb55gParjRbOiVW19ftrEcQRVDbzX8= X-Google-Smtp-Source: ABdhPJz8tngMyKNr78aheYziUbFINMy3fzZrKBXOBSG4c7opcdLd77kwyS2pdOJo43gK/lecZCf7Nw== X-Received: by 2002:adf:f0c6:0:b0:20a:d31b:6 with SMTP id x6-20020adff0c6000000b0020ad31b0006mr20672965wro.162.1651143566470; Thu, 28 Apr 2022 03:59:26 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-230-8.dab.02.net. [82.132.230.8]) by smtp.gmail.com with ESMTPSA id n10-20020a5d588a000000b002052e4aaf89sm16028895wrf.80.2022.04.28.03.59.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Apr 2022 03:59:26 -0700 (PDT) From: Pavel Begunkov To: netdev@vger.kernel.org, "David S . Miller" , Jakub Kicinski Cc: David Ahern , Eric Dumazet , linux-kernel@vger.kernel.org, Pavel Begunkov Subject: [PATCH net-next 1/5] net: inline sock_alloc_send_skb Date: Thu, 28 Apr 2022 11:58:44 +0100 Message-Id: <589966747301e2f71377f288faf4f8db78d72405.1651141755.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org sock_alloc_send_skb() is simple and just proxying to another function, so we can inline it and cut associated overhead. Signed-off-by: Pavel Begunkov --- include/net/sock.h | 10 ++++++++-- net/core/sock.c | 7 ------- 2 files changed, 8 insertions(+), 9 deletions(-) diff --git a/include/net/sock.h b/include/net/sock.h index a01d6c421aa2..df2e826f67ee 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -1824,11 +1824,17 @@ int sock_getsockopt(struct socket *sock, int level, int op, char __user *optval, int __user *optlen); int sock_gettstamp(struct socket *sock, void __user *userstamp, bool timeval, bool time32); -struct sk_buff *sock_alloc_send_skb(struct sock *sk, unsigned long size, - int noblock, int *errcode); struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, unsigned long data_len, int noblock, int *errcode, int max_page_order); + +static inline struct sk_buff *sock_alloc_send_skb(struct sock *sk, + unsigned long size, + int noblock, int *errcode) +{ + return sock_alloc_send_pskb(sk, size, 0, noblock, errcode, 0); +} + void *sock_kmalloc(struct sock *sk, int size, gfp_t priority); void sock_kfree_s(struct sock *sk, void *mem, int size); void sock_kzfree_s(struct sock *sk, void *mem, int size); diff --git a/net/core/sock.c b/net/core/sock.c index 29abec3eabd8..fbca35d3749a 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -2631,13 +2631,6 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, } EXPORT_SYMBOL(sock_alloc_send_pskb); -struct sk_buff *sock_alloc_send_skb(struct sock *sk, unsigned long size, - int noblock, int *errcode) -{ - return sock_alloc_send_pskb(sk, size, 0, noblock, errcode, 0); -} -EXPORT_SYMBOL(sock_alloc_send_skb); - int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg, struct sockcm_cookie *sockc) { From patchwork Thu Apr 28 10:58:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12830467 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54B85C433EF for ; Thu, 28 Apr 2022 11:01:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345296AbiD1LET (ORCPT ); Thu, 28 Apr 2022 07:04:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48940 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345298AbiD1LDL (ORCPT ); Thu, 28 Apr 2022 07:03:11 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 017859D4E8; Thu, 28 Apr 2022 03:59:32 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id r11-20020a05600c35cb00b0039409c1111bso1854598wmq.3; Thu, 28 Apr 2022 03:59:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=r5k1ZTMK7mSQH9mXWos/hmZJEpLUd2ex8bU7xnkW7WU=; b=BiCbQzJ1UxgfgSMP82tJgGSeQrQZBhu4ySl2LOi6qrXumvH2JtQWELv7YQxIqBnX4x 8tmxzOPLNtGKW/wYn8wGj5+HdNmnB20qh30CTdI67xC0DsqIbRLrqJNIezCj9qDPWuXk 5uli2jnAE6TAO0LGz3AYIP4LAv7QIhZ6k0hvp5HyZl6xthOnYi/FZ68Cgv65bFz/nWjo pew3BkE6RMRQE7CF+kGZqjDP0gPlZaB7QvwfL0ZASPkvuaUr9hv9QpzVZfYjf9mhEQJI Ily8Xebav4k8Ejsbfhv1Xx51e1dsWSugFbOMYMLuUQApcEfZChYzV1J2oAcNgPHUfOQR M4Dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=r5k1ZTMK7mSQH9mXWos/hmZJEpLUd2ex8bU7xnkW7WU=; b=Vn8tUkoZLRW7QaiF2ho/NsasvlQezC5l9Ltit66OzDHzBnqnuTG/pBuRc+04FWdS0I yFfyEqyLxiFfwwkNsIqjw6Ohu1GNC5Rt/dt1Fe2Eym6jOTfD2UGmqmAK4QZcoJgrqvHb LLoPNFUtHwTg6BCcKNgvPir+pZYAryNs2xQ3IVT6gyvVD4AiJsyPsj3sOOu0wyHqAoy4 0yZvcA3tFkncXJDaFVppasMA5J5xV8ZocERVgnH25axx68UGdJCrriaMJNfoQG4RYCRF tayMpMhvrVi/0CoYZCwKJHyXWa3taz+7f2hxe5Q9ThwduuVQ2QN197I7AFI033TN+Y2a ZxAg== X-Gm-Message-State: AOAM5307DPZLNiEjFvYgfeEdtsKmjvyEm0O443ER5R8MTTDJ2MgNZWbG 0mUdlOLBumvlx0IiyUz4EKrQ5C2S2dw= X-Google-Smtp-Source: ABdhPJybU48FPei64zykiGCI1HDY7wnNb5DGJdPnw67/1VDKwmUwYyv5obbnsCJ/5GqkULMBk/Izkg== X-Received: by 2002:a7b:c844:0:b0:38e:7c92:a9e3 with SMTP id c4-20020a7bc844000000b0038e7c92a9e3mr29617741wml.140.1651143570314; Thu, 28 Apr 2022 03:59:30 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-230-8.dab.02.net. [82.132.230.8]) by smtp.gmail.com with ESMTPSA id n10-20020a5d588a000000b002052e4aaf89sm16028895wrf.80.2022.04.28.03.59.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Apr 2022 03:59:30 -0700 (PDT) From: Pavel Begunkov To: netdev@vger.kernel.org, "David S . Miller" , Jakub Kicinski Cc: David Ahern , Eric Dumazet , linux-kernel@vger.kernel.org, Pavel Begunkov Subject: [PATCH net-next 2/5] net: inline skb_zerocopy_iter_dgram Date: Thu, 28 Apr 2022 11:58:45 +0100 Message-Id: <2159b3b4ff6bfe623e9836468aefc866ee3b4ab6.1651141755.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org skb_zerocopy_iter_dgram() is a small proxy function, inline it. For that, move __zerocopy_sg_from_iter into linux/skbuff.h Signed-off-by: Pavel Begunkov --- include/linux/skbuff.h | 36 ++++++++++++++++++++++-------------- net/core/datagram.c | 2 -- net/core/datagram.h | 15 --------------- net/core/skbuff.c | 7 ------- 4 files changed, 22 insertions(+), 38 deletions(-) delete mode 100644 net/core/datagram.h diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 84d78df60453..57182947cc80 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -684,20 +684,6 @@ struct ubuf_info { int mm_account_pinned_pages(struct mmpin *mmp, size_t size); void mm_unaccount_pinned_pages(struct mmpin *mmp); -struct ubuf_info *msg_zerocopy_alloc(struct sock *sk, size_t size); -struct ubuf_info *msg_zerocopy_realloc(struct sock *sk, size_t size, - struct ubuf_info *uarg); - -void msg_zerocopy_put_abort(struct ubuf_info *uarg, bool have_uref); - -void msg_zerocopy_callback(struct sk_buff *skb, struct ubuf_info *uarg, - bool success); - -int skb_zerocopy_iter_dgram(struct sk_buff *skb, struct msghdr *msg, int len); -int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb, - struct msghdr *msg, int len, - struct ubuf_info *uarg); - /* This data is invariant across clones and lives at * the end of the header data, ie. at skb->end. */ @@ -1676,6 +1662,28 @@ static inline void skb_set_end_offset(struct sk_buff *skb, unsigned int offset) } #endif +struct ubuf_info *msg_zerocopy_alloc(struct sock *sk, size_t size); +struct ubuf_info *msg_zerocopy_realloc(struct sock *sk, size_t size, + struct ubuf_info *uarg); + +void msg_zerocopy_put_abort(struct ubuf_info *uarg, bool have_uref); + +void msg_zerocopy_callback(struct sk_buff *skb, struct ubuf_info *uarg, + bool success); + +int __zerocopy_sg_from_iter(struct sock *sk, struct sk_buff *skb, + struct iov_iter *from, size_t length); + +static inline int skb_zerocopy_iter_dgram(struct sk_buff *skb, + struct msghdr *msg, int len) +{ + return __zerocopy_sg_from_iter(skb->sk, skb, &msg->msg_iter, len); +} + +int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb, + struct msghdr *msg, int len, + struct ubuf_info *uarg); + /* Internal */ #define skb_shinfo(SKB) ((struct skb_shared_info *)(skb_end_pointer(SKB))) diff --git a/net/core/datagram.c b/net/core/datagram.c index 70126d15ca6e..50f4faeea76c 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -62,8 +62,6 @@ #include #include -#include "datagram.h" - /* * Is a socket 'connection oriented' ? */ diff --git a/net/core/datagram.h b/net/core/datagram.h deleted file mode 100644 index bcfb75bfa3b2..000000000000 --- a/net/core/datagram.h +++ /dev/null @@ -1,15 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ - -#ifndef _NET_CORE_DATAGRAM_H_ -#define _NET_CORE_DATAGRAM_H_ - -#include - -struct sock; -struct sk_buff; -struct iov_iter; - -int __zerocopy_sg_from_iter(struct sock *sk, struct sk_buff *skb, - struct iov_iter *from, size_t length); - -#endif /* _NET_CORE_DATAGRAM_H_ */ diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 30b523fa4ad2..384c6098a5f8 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -80,7 +80,6 @@ #include #include -#include "datagram.h" #include "sock_destructor.h" struct kmem_cache *skbuff_head_cache __ro_after_init; @@ -1339,12 +1338,6 @@ void msg_zerocopy_put_abort(struct ubuf_info *uarg, bool have_uref) } EXPORT_SYMBOL_GPL(msg_zerocopy_put_abort); -int skb_zerocopy_iter_dgram(struct sk_buff *skb, struct msghdr *msg, int len) -{ - return __zerocopy_sg_from_iter(skb->sk, skb, &msg->msg_iter, len); -} -EXPORT_SYMBOL_GPL(skb_zerocopy_iter_dgram); - int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb, struct msghdr *msg, int len, struct ubuf_info *uarg) From patchwork Thu Apr 28 10:58:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12830468 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBFA3C433F5 for ; Thu, 28 Apr 2022 11:01:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345341AbiD1LEc (ORCPT ); Thu, 28 Apr 2022 07:04:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48846 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345261AbiD1LD3 (ORCPT ); Thu, 28 Apr 2022 07:03:29 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 158D79E9EB; Thu, 28 Apr 2022 03:59:35 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id n126-20020a1c2784000000b0038e8af3e788so2804186wmn.1; Thu, 28 Apr 2022 03:59:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2Zk8fXv1n2ViDSF6OIwgY/sl4ulWkkPTpvp0fu2VISY=; b=VnasJm98PgyBQzES3DFfd3IaxA6YTjNH/hj/4pevOxmkcfpxvQvn+zizcOxRCiDcrb 8lG5JjR3z13nNihnpZZcxyjRjsVk3X4achKutQBvZkbTAM/oHyXs/E346wGGQJXXvqU6 7Kkl9RQKhbpknyl1QwmimrUjpLOib/Te3Bkxck5FZRJ/ayU+pTzo7mls9L0gx2MwklSE Siik09sFZLNSuFTjQA77VeH5h+DVDelOf1RAdoRJTGzDpzouAwHj21WIOOCLEEmm1XZ/ GK53MhW/AijDH92KndyLFOCzxaFSRhu4K5adisuntYOgxlWbK04U+UD8ZZYfFHMoTYcE 9O8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2Zk8fXv1n2ViDSF6OIwgY/sl4ulWkkPTpvp0fu2VISY=; b=rwlPIc2S1eXcIe+mrOkAqMY2i7BTG10xWMmb2am/H7hoRureTs/qOnsY+Ue6JW6oqZ 71JxsxcVbprAYOLSqZki3Oirw055l6UZoI3RFixpHMwoPpQ+34mt2y1BjTrOFqaTnGWE wGOcMXsQI+A5rDu5sQ3EiloXeZyHtO+xQk0CdeNpwXLLJk6h3bqNgunp8TLsvR76F+qy T/wbaqe3V30P5JKTcWKxVjEVGczDxUcQ7WvMtSVAJNRKEkwEjiXQKok42TzOLjGaHpOY O+0yrwcq0SOiDYLtTa/gyEf7yWPw1yOfGZU+2dThNRoBd0dnHxlQtlFxmlsxIhnovtLi qNnQ== X-Gm-Message-State: AOAM53264tznohGeE6xnchs4lRQFy4xZKujOxCUQaG4af9RYyccW+BX/ bXzW77DqFUQYQhnlge1kA4Ahax6Md/Q= X-Google-Smtp-Source: ABdhPJwn3driW1HdF2RV7PlkZKel27bOnEq7CWM9SilzkhhHdIwXbpfiXcoSL76OG75gTrq14j6Zxg== X-Received: by 2002:a1c:f315:0:b0:381:1f6d:6ca6 with SMTP id q21-20020a1cf315000000b003811f6d6ca6mr40066156wmq.25.1651143574157; Thu, 28 Apr 2022 03:59:34 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-230-8.dab.02.net. [82.132.230.8]) by smtp.gmail.com with ESMTPSA id n10-20020a5d588a000000b002052e4aaf89sm16028895wrf.80.2022.04.28.03.59.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Apr 2022 03:59:33 -0700 (PDT) From: Pavel Begunkov To: netdev@vger.kernel.org, "David S . Miller" , Jakub Kicinski Cc: David Ahern , Eric Dumazet , linux-kernel@vger.kernel.org, Pavel Begunkov Subject: [PATCH net-next 3/5] net: inline dev_queue_xmit() Date: Thu, 28 Apr 2022 11:58:46 +0100 Message-Id: <99bf2b824f3a4e0428a8be81c8685afe78ffa207.1651141755.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Inline dev_queue_xmit() and dev_queue_xmit_accel(), they both are small proxy functions doing nothing but redirecting the control flow to __dev_queue_xmit(). Signed-off-by: Pavel Begunkov --- include/linux/netdevice.h | 14 ++++++++++++-- net/core/dev.c | 15 ++------------- 2 files changed, 14 insertions(+), 15 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 7dccbfd1bf56..3c3bd3381d38 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -2940,10 +2940,20 @@ u16 dev_pick_tx_zero(struct net_device *dev, struct sk_buff *skb, u16 dev_pick_tx_cpu_id(struct net_device *dev, struct sk_buff *skb, struct net_device *sb_dev); -int dev_queue_xmit(struct sk_buff *skb); -int dev_queue_xmit_accel(struct sk_buff *skb, struct net_device *sb_dev); +int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev); int __dev_direct_xmit(struct sk_buff *skb, u16 queue_id); +static inline int dev_queue_xmit(struct sk_buff *skb) +{ + return __dev_queue_xmit(skb, NULL); +} + +static inline int dev_queue_xmit_accel(struct sk_buff *skb, + struct net_device *sb_dev) +{ + return __dev_queue_xmit(skb, sb_dev); +} + static inline int dev_direct_xmit(struct sk_buff *skb, u16 queue_id) { int ret; diff --git a/net/core/dev.c b/net/core/dev.c index 4a77ebda4fb1..d160c35858df 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4111,7 +4111,7 @@ struct netdev_queue *netdev_core_pick_tx(struct net_device *dev, * the BH enable code must have IRQs enabled so that it will not deadlock. * --BLG */ -static int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev) +int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev) { struct net_device *dev = skb->dev; struct netdev_queue *txq = NULL; @@ -4235,18 +4235,7 @@ static int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev) rcu_read_unlock_bh(); return rc; } - -int dev_queue_xmit(struct sk_buff *skb) -{ - return __dev_queue_xmit(skb, NULL); -} -EXPORT_SYMBOL(dev_queue_xmit); - -int dev_queue_xmit_accel(struct sk_buff *skb, struct net_device *sb_dev) -{ - return __dev_queue_xmit(skb, sb_dev); -} -EXPORT_SYMBOL(dev_queue_xmit_accel); +EXPORT_SYMBOL(__dev_queue_xmit); int __dev_direct_xmit(struct sk_buff *skb, u16 queue_id) { From patchwork Thu Apr 28 10:58:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12830469 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CD6BC433EF for ; Thu, 28 Apr 2022 11:01:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345160AbiD1LEo (ORCPT ); Thu, 28 Apr 2022 07:04:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48924 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345117AbiD1LES (ORCPT ); Thu, 28 Apr 2022 07:04:18 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0BC449F3B0; Thu, 28 Apr 2022 03:59:41 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id v12so6218201wrv.10; Thu, 28 Apr 2022 03:59:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UWdt6/kj6CQanUvb/1U/JT2tCIyxFlbvE9RBa6NuQ9Y=; b=dxUoBWHq8/G/mTCjR4X/SBOP9c8cxgWUNkvq3di1dlDQujxsZtaLvTz5CUsxvYvD0C gzMVdBZj2AOBVzSkJ6CEnD9WsxuoDgTtsJ0k/uJCD9d+6lwOp0IKBR22I9LWHaXU911O JfaPqxh1AJhfQExmaQ1t1Wwc75im5LL4blvUjG2kPvn5FkBBB/3zlp1zKz7nE/38lg7Z xYr9y1/E9BYbnmBv6vX0AR3z3IAXL88wa1n4dDjDylARW73WySqFP33LCjBnSnzLwpKl IYTQa5a9oxihWpY/3vPNdnoKgh2SEyZ+lw01FJuwCoh6GGoQkdjPsTu2aRMaE6Rd2USy ksRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UWdt6/kj6CQanUvb/1U/JT2tCIyxFlbvE9RBa6NuQ9Y=; b=RG+eJRH1yinrLfnwQV0KeRuYrNCGmvMs7Hdexc6sBFrQM1DZ6RFkR7Q7D6kex9RdOG fGA/qsU/a0XTRIlhlFoiVaVBVBAJtydVRDZr8rZ57upV09EklOTfFXnW/V6RvkahXWxi CnwRcDEY6gUPuYISyHct+7WcxVNwhGJQxH+JCa4vPiT4VZdrZi20yXfGpUIKe9otE5OS Yd+0KPtLcIfAHg3+bs/8BFyHtbbysU16L8akIzZxz6+Kiiwz89NK6xBDd+GaIz0J1ozD 9TE0NDQv6vb5nUML/tT3KDVYefrwpFcbnykmokb9N9odcrYdlMitZL5ziGe/Bdd3LR4+ 6fUw== X-Gm-Message-State: AOAM532WvpFtMhQMCJ4fQjwRD4BD3d6j9YhR31khnN6iwQVd1I/mm238 CyKwmQgJUriK1VZnGlVM8XpjnQRo2iE= X-Google-Smtp-Source: ABdhPJwr7woSd2tVSZNGvcoKJW0Dug9oeN9orkvRFgM+O+58DLb5M1dyB3AGd1nKHUeIM0Q3Pz2mxw== X-Received: by 2002:a05:6000:718:b0:20a:e310:664f with SMTP id bs24-20020a056000071800b0020ae310664fmr11335462wrb.22.1651143579272; Thu, 28 Apr 2022 03:59:39 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-230-8.dab.02.net. [82.132.230.8]) by smtp.gmail.com with ESMTPSA id n10-20020a5d588a000000b002052e4aaf89sm16028895wrf.80.2022.04.28.03.59.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Apr 2022 03:59:38 -0700 (PDT) From: Pavel Begunkov To: netdev@vger.kernel.org, "David S . Miller" , Jakub Kicinski Cc: David Ahern , Eric Dumazet , linux-kernel@vger.kernel.org, Pavel Begunkov Subject: [PATCH net-next 4/5] ipv6: help __ip6_finish_output() inlining Date: Thu, 28 Apr 2022 11:58:47 +0100 Message-Id: X-Mailer: git-send-email 2.36.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org There are two callers of __ip6_finish_output(), both are in ip6_finish_output(). We can combine the call sites into one and handle return code after, that will inline __ip6_finish_output(). Note, error handling under NET_XMIT_CN will only return 0 if __ip6_finish_output() succeded, and in this case it return 0. Considering that NET_XMIT_SUCCESS is 0, it'll be returning exactly the same result for it as before. Signed-off-by: Pavel Begunkov --- net/ipv6/ip6_output.c | 1 - 1 file changed, 1 deletion(-) diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 1f3d777e7694..bda1d9f76f7e 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -198,7 +198,6 @@ static int ip6_finish_output(struct net *net, struct sock *sk, struct sk_buff *s ret = BPF_CGROUP_RUN_PROG_INET_EGRESS(sk, skb); switch (ret) { case NET_XMIT_SUCCESS: - return __ip6_finish_output(net, sk, skb); case NET_XMIT_CN: return __ip6_finish_output(net, sk, skb) ? : ret; default: From patchwork Thu Apr 28 10:58:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12830470 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEB4FC433F5 for ; Thu, 28 Apr 2022 11:01:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345088AbiD1LEt (ORCPT ); Thu, 28 Apr 2022 07:04:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48618 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345313AbiD1LEV (ORCPT ); Thu, 28 Apr 2022 07:04:21 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BCA75A5E97; Thu, 28 Apr 2022 03:59:44 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id w4so6212777wrg.12; Thu, 28 Apr 2022 03:59:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xjZmQddY+Qa40STFRzTaM10MN4/OwVjfFMoJFMnoO/U=; b=KI0wVcHP8w3j4ZlynXgywnOvvU2ta9F5GDA09NQQw0742Mmz7Z04vyz7e7tkE4zsjT WdSXBoMR6uQISplTlOvIkBZdnsAIoBp3DYBcsm+C68YUDG4grnyaauS70k7/oa4W50Gk NKP3MoIYBng9d/kpU1/PE+oZ1v8oVNY+Q5QfmgqDbNl1c/5Unx2+v2N3OD4iA8vjTo52 o1JDvQsb20S2/+KtDHhct8QbxE4htUj9705Nbg37mMq5rDAnCSDWMvjQl19hpfQAxZCk rJb1lOWfgcVsv03tHrFv6YIQshVrkElH9EcEuyABlvMPo8hD6R5MQTJvO6fJjQA2JcfJ 1Ijg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xjZmQddY+Qa40STFRzTaM10MN4/OwVjfFMoJFMnoO/U=; b=1d7RFdPScKwoepumPuHsdUYVjd3vfZpJ9IQpqLEJx0w332fsAApHezH0GUg1rWwUgw A+YogEcMNBfzqdT2lztezzKL0Z9e01ZJC8eht6G3pRXY35/yoE+/3LoNgSEUgOjxzuHR 9NpXhye/Lk4tNsBGvW2vp00ckeYxdzUxAF6+kbfSHPq/Sc0nyfRXB+2dnKxNfTkhDMal w7FEOS3AT9+vAz+nqURVQjQM5ZLCw0t3Jv/ExYyKFQfGU1pykmHsiCIlAjaQtSnTIO/Q xPHr0C3WUXchLniSiCzgQquqb9j8m7jyadFFiLCFq95I/hO3e4vcfXMjcW3j1MXqnqqk nMIA== X-Gm-Message-State: AOAM532PgaUmIjC86AkXQjFebH/cDyGNLpT3zis/KJTpwY0nfrjw0f1w Q7BpaU4TrSrSkQKySxpLXAW6xWlwjKw= X-Google-Smtp-Source: ABdhPJxrrImJGZcfa7m7BIGoNvOgrswftTi8ZrjfXNbHxBi4g1WMVQfAoSaHDHgSIXduwu/cKPoGFg== X-Received: by 2002:adf:e104:0:b0:206:109a:c90f with SMTP id t4-20020adfe104000000b00206109ac90fmr25718439wrz.259.1651143582963; Thu, 28 Apr 2022 03:59:42 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-230-8.dab.02.net. [82.132.230.8]) by smtp.gmail.com with ESMTPSA id n10-20020a5d588a000000b002052e4aaf89sm16028895wrf.80.2022.04.28.03.59.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Apr 2022 03:59:42 -0700 (PDT) From: Pavel Begunkov To: netdev@vger.kernel.org, "David S . Miller" , Jakub Kicinski Cc: David Ahern , Eric Dumazet , linux-kernel@vger.kernel.org, Pavel Begunkov Subject: [PATCH net-next 5/5] ipv6: refactor ip6_finish_output2() Date: Thu, 28 Apr 2022 11:58:48 +0100 Message-Id: X-Mailer: git-send-email 2.36.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Throw neigh checks in ip6_finish_output2() under a single slow path if, so we don't have the overhead in the hot path. Signed-off-by: Pavel Begunkov --- net/ipv6/ip6_output.c | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index bda1d9f76f7e..afa5bd4ad167 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -119,19 +119,21 @@ static int ip6_finish_output2(struct net *net, struct sock *sk, struct sk_buff * rcu_read_lock_bh(); nexthop = rt6_nexthop((struct rt6_info *)dst, daddr); neigh = __ipv6_neigh_lookup_noref(dev, nexthop); - if (unlikely(!neigh)) - neigh = __neigh_create(&nd_tbl, nexthop, dev, false); - if (!IS_ERR(neigh)) { - sock_confirm_neigh(skb, neigh); - ret = neigh_output(neigh, skb, false); - rcu_read_unlock_bh(); - return ret; + + if (unlikely(IS_ERR_OR_NULL(neigh))) { + if (unlikely(!neigh)) + neigh = __neigh_create(&nd_tbl, nexthop, dev, false); + if (IS_ERR(neigh)) { + rcu_read_unlock_bh(); + IP6_INC_STATS(net, idev, IPSTATS_MIB_OUTNOROUTES); + kfree_skb_reason(skb, SKB_DROP_REASON_NEIGH_CREATEFAIL); + return -EINVAL; + } } + sock_confirm_neigh(skb, neigh); + ret = neigh_output(neigh, skb, false); rcu_read_unlock_bh(); - - IP6_INC_STATS(net, idev, IPSTATS_MIB_OUTNOROUTES); - kfree_skb_reason(skb, SKB_DROP_REASON_NEIGH_CREATEFAIL); - return -EINVAL; + return ret; } static int