[00/14] udp optimisation

Message

Pavel Begunkov Jan. 11, 2022, 1:21 a.m. UTC

A mainly UDP/IPv6 optimisation patch set. Zerocopy io_uring benchmark over
dummy netdev (CPU bound) gives 2068992 -> 2166481 tx/s, which is ~4.7% or
over 5% of net layer overhead. Should give similar results for small
packet non-zerocopy.

- 1/14 and 9/14 remove a get/put dst pair each, so saving 4 atomics per
  corkless UDP send.
- Patches 3-8 optimise iflow handling, in particular removes one 88B
  memset and one 88B copy.
- 10-14 are random improvements, which are not UDP-specific but also
  beneficial to TCP and others.

Pavel Begunkov (14):
  ipv6: optimise dst referencing
  ipv6: shuffle up->pending AF_INET bits
  ipv6: remove daddr temp buffer in __ip6_make_skb
  ipv6: clean up cork setup/release
  ipv6: don't zero cork's flowi after use
  ipv6: pass full cork into __ip6_append_data()
  ipv6: pass flow in ip6_make_skb together with cork
  ipv6/udp: don't make extra copies of iflow
  ipv6: hand dst refs to cork setup
  skbuff: drop zero check from skb_zcopy_set
  skbuff: drop null check from skb_zcopy
  skbuff: optimise alloc_skb_with_frags()
  net: inline part of skb_csum_hwoffload_help
  net: inline sock_alloc_send_skb

 include/linux/netdevice.h |  16 +++++-
 include/linux/skbuff.h    |  45 +++++++++++++---
 include/net/ipv6.h        |   2 +-
 include/net/sock.h        |  10 +++-
 net/core/dev.c            |  15 ++----
 net/core/skbuff.c         |  34 +++++-------
 net/core/sock.c           |   7 ---
 net/ipv4/ip_output.c      |  10 ++--
 net/ipv4/tcp.c            |   5 +-
 net/ipv6/ip6_output.c     | 105 +++++++++++++++++++++-----------------
 net/ipv6/udp.c            | 103 ++++++++++++++++++-------------------
 11 files changed, 197 insertions(+), 155 deletions(-)