From patchwork Tue Sep 28 21:46:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Wang X-Patchwork-Id: 12523907 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56D74C433F5 for ; Tue, 28 Sep 2021 21:46:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3D2A761360 for ; Tue, 28 Sep 2021 21:46:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242664AbhI1Vsa (ORCPT ); Tue, 28 Sep 2021 17:48:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41564 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242556AbhI1Vs2 (ORCPT ); Tue, 28 Sep 2021 17:48:28 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 14385C06161C for ; Tue, 28 Sep 2021 14:46:49 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id b9-20020a5b07890000b0290558245b7eabso633786ybq.10 for ; Tue, 28 Sep 2021 14:46:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=3cXNsNOjcfE/nCmwonP8h/HYpTsOObliFungylaBWO4=; b=Mrljl0+vPVoTe1wFjthqq0Ty+D0+LTOxQsTu7uQYa40AgH1Teerf9M4nyIjXAVbNoL mN8zJc9tE9IXPgKzEWrISqnEDAvKq0pvUZIsCSeYf5p+cWLilGNiiK60CUA2hQendbYj +P2cyebyZ1IC8bCuMau6eWL8a5Ow7Dl+vhCYSQXUYYmP7lh29kL4M+RRwxGmVhStr36S gKgARv7ztV6iC/QGEZEjm2/Qk25Oq/MVdrPwxxet+/eGZlVqZH5aaVlK8sMaIglGIamG oZvm06bDJAU+/4AFzBDCnKv20PscqyBnzu0wKOxbQUCn9WEMfgcQVjRDZpJKgwG48Ac1 n3Pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=3cXNsNOjcfE/nCmwonP8h/HYpTsOObliFungylaBWO4=; b=VlTKS+afJt009YwSUC2M9FxWBu8uOLRsuJetn4KtI1kG14UYLiJie9CcamLfMjtDbo kDILnMgnWChO0r+zKHgvQg/CmGcTI7DdFqSMti3ebbYGYNaNf5+5yAAbAuEgmoD8Ou/v /HfEWITAOyRVPpLtd/xRLGkNvVgIyCq1OKtSfkIvP6clv9MQnY+f4w9n/cnuzfaw6zx3 h9W4IXTuEPDVEl2eIGA6YHRNB35/yolNWZeHiXDaPHu39Sdv8t++g5W/2f3eyG3wdFAs bIJhsPnk06c1Pr1IGZPVyBzjh9w1ypQaWOtByRzc8zz0Qgz+wAHBY0jypl0UlUJnWuy+ kn8g== X-Gm-Message-State: AOAM532C3p4tHq0MggWoNAOBdDIK4KnyNE743zzFr9ysD4ME9wIsWMek nRjBqsa0nV2o5wPF3L2MaMd5QsQe+Dw= X-Google-Smtp-Source: ABdhPJwNJ5OtIt2hjpz84UPAjQRDeat7S5G/RYM0Trmfj6zqFsth+OFAnqqBxYqqrF8dliiw1IEeVrqyjZM= X-Received: from weiwan.svl.corp.google.com ([2620:15c:2c4:201:5d42:81b8:4b7f:efe2]) (user=weiwan job=sendgmr) by 2002:a25:2c97:: with SMTP id s145mr1307923ybs.463.1632865608180; Tue, 28 Sep 2021 14:46:48 -0700 (PDT) Date: Tue, 28 Sep 2021 14:46:41 -0700 In-Reply-To: <20210928214643.3300692-1-weiwan@google.com> Message-Id: <20210928214643.3300692-2-weiwan@google.com> Mime-Version: 1.0 References: <20210928214643.3300692-1-weiwan@google.com> X-Mailer: git-send-email 2.33.0.685.g46640cef36-goog Subject: [PATCH v2 net-next 1/3] net: add new socket option SO_RESERVE_MEM From: Wei Wang To: Jakub Kicinski , "'David S . Miller'" , netdev@vger.kernel.org Cc: Shakeel Butt , Eric Dumazet Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org This socket option provides a mechanism for users to reserve a certain amount of memory for the socket to use. When this option is set, kernel charges the user specified amount of memory to memcg, as well as sk_forward_alloc. This amount of memory is not reclaimable and is available in sk_forward_alloc for this socket. With this socket option set, the networking stack spends less cycles doing forward alloc and reclaim, which should lead to better system performance, with the cost of an amount of pre-allocated and unreclaimable memory, even under memory pressure. Note: This socket option is only available when memory cgroup is enabled and we require this reserved memory to be charged to the user's memcg. We hope this could avoid mis-behaving users to abused this feature to reserve a large amount on certain sockets and cause unfairness for others. Signed-off-by: Wei Wang Signed-off-by: Eric Dumazet --- include/net/sock.h | 43 ++++++++++++++++--- include/uapi/asm-generic/socket.h | 2 + net/core/sock.c | 69 +++++++++++++++++++++++++++++++ net/core/stream.c | 2 +- net/ipv4/af_inet.c | 2 +- 5 files changed, 111 insertions(+), 7 deletions(-) diff --git a/include/net/sock.h b/include/net/sock.h index 879980de3dcd..0feb37c7f534 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -409,6 +409,7 @@ struct sock { #define sk_rmem_alloc sk_backlog.rmem_alloc int sk_forward_alloc; + u32 sk_reserved_mem; #ifdef CONFIG_NET_RX_BUSY_POLL unsigned int sk_ll_usec; /* ===== mostly read cache line ===== */ @@ -1511,20 +1512,49 @@ sk_rmem_schedule(struct sock *sk, struct sk_buff *skb, int size) skb_pfmemalloc(skb); } +static inline int sk_unused_reserved_mem(const struct sock *sk) +{ + int unused_mem; + + if (likely(!sk->sk_reserved_mem)) + return 0; + + unused_mem = sk->sk_reserved_mem - sk->sk_wmem_queued - + atomic_read(&sk->sk_rmem_alloc); + + return unused_mem > 0 ? unused_mem : 0; +} + static inline void sk_mem_reclaim(struct sock *sk) { + int reclaimable; + if (!sk_has_account(sk)) return; - if (sk->sk_forward_alloc >= SK_MEM_QUANTUM) - __sk_mem_reclaim(sk, sk->sk_forward_alloc); + + reclaimable = sk->sk_forward_alloc - sk_unused_reserved_mem(sk); + + if (reclaimable >= SK_MEM_QUANTUM) + __sk_mem_reclaim(sk, reclaimable); +} + +static inline void sk_mem_reclaim_final(struct sock *sk) +{ + sk->sk_reserved_mem = 0; + sk_mem_reclaim(sk); } static inline void sk_mem_reclaim_partial(struct sock *sk) { + int reclaimable; + if (!sk_has_account(sk)) return; - if (sk->sk_forward_alloc > SK_MEM_QUANTUM) - __sk_mem_reclaim(sk, sk->sk_forward_alloc - 1); + + reclaimable = sk->sk_forward_alloc - sk_unused_reserved_mem(sk); + + if (reclaimable > SK_MEM_QUANTUM) + __sk_mem_reclaim(sk, reclaimable - 1); } static inline void sk_mem_charge(struct sock *sk, int size) @@ -1536,9 +1566,12 @@ static inline void sk_mem_charge(struct sock *sk, int size) static inline void sk_mem_uncharge(struct sock *sk, int size) { + int reclaimable; + if (!sk_has_account(sk)) return; sk->sk_forward_alloc += size; + reclaimable = sk->sk_forward_alloc - sk_unused_reserved_mem(sk); /* Avoid a possible overflow. * TCP send queues can make this happen, if sk_mem_reclaim() @@ -1547,7 +1580,7 @@ static inline void sk_mem_uncharge(struct sock *sk, int size) * If we reach 2 MBytes, reclaim 1 MBytes right now, there is * no need to hold that much forward allocation anyway. */ - if (unlikely(sk->sk_forward_alloc >= 1 << 21)) + if (unlikely(reclaimable >= 1 << 21)) __sk_mem_reclaim(sk, 1 << 20); } diff --git a/include/uapi/asm-generic/socket.h b/include/uapi/asm-generic/socket.h index 1f0a2b4864e4..c77a1313b3b0 100644 --- a/include/uapi/asm-generic/socket.h +++ b/include/uapi/asm-generic/socket.h @@ -126,6 +126,8 @@ #define SO_BUF_LOCK 72 +#define SO_RESERVE_MEM 73 + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 || (defined(__x86_64__) && defined(__ILP32__)) diff --git a/net/core/sock.c b/net/core/sock.c index 512e629f9780..0ecb8590e043 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -947,6 +947,53 @@ void sock_set_mark(struct sock *sk, u32 val) } EXPORT_SYMBOL(sock_set_mark); +static void sock_release_reserved_memory(struct sock *sk, int bytes) +{ + /* Round down bytes to multiple of pages */ + bytes &= ~(SK_MEM_QUANTUM - 1); + + WARN_ON(bytes > sk->sk_reserved_mem); + sk->sk_reserved_mem -= bytes; + sk_mem_reclaim(sk); +} + +static int sock_reserve_memory(struct sock *sk, int bytes) +{ + long allocated; + bool charged; + int pages; + + if (!mem_cgroup_sockets_enabled || !sk->sk_memcg) + return -EOPNOTSUPP; + + if (!bytes) + return 0; + + pages = sk_mem_pages(bytes); + + /* pre-charge to memcg */ + charged = mem_cgroup_charge_skmem(sk->sk_memcg, pages, + GFP_KERNEL | __GFP_RETRY_MAYFAIL); + if (!charged) + return -ENOMEM; + + /* pre-charge to forward_alloc */ + allocated = sk_memory_allocated_add(sk, pages); + /* If the system goes into memory pressure with this + * precharge, give up and return error. + */ + if (allocated > sk_prot_mem_limits(sk, 1)) { + sk_memory_allocated_sub(sk, pages); + mem_cgroup_uncharge_skmem(sk->sk_memcg, pages); + return -ENOMEM; + } + sk->sk_forward_alloc += pages << SK_MEM_QUANTUM_SHIFT; + + sk->sk_reserved_mem += pages << SK_MEM_QUANTUM_SHIFT; + + return 0; +} + /* * This is meant for all protocols to use and covers goings on * at the socket level. Everything here is generic. @@ -1367,6 +1414,23 @@ int sock_setsockopt(struct socket *sock, int level, int optname, ~SOCK_BUF_LOCK_MASK); break; + case SO_RESERVE_MEM: + { + int delta; + + if (val < 0) { + ret = -EINVAL; + break; + } + + delta = val - sk->sk_reserved_mem; + if (delta < 0) + sock_release_reserved_memory(sk, -delta); + else + ret = sock_reserve_memory(sk, delta); + break; + } + default: ret = -ENOPROTOOPT; break; @@ -1733,6 +1797,10 @@ int sock_getsockopt(struct socket *sock, int level, int optname, v.val = sk->sk_userlocks & SOCK_BUF_LOCK_MASK; break; + case SO_RESERVE_MEM: + v.val = sk->sk_reserved_mem; + break; + default: /* We implement the SO_SNDLOWAT etc to not be settable * (1003.1g 7). @@ -2045,6 +2113,7 @@ struct sock *sk_clone_lock(const struct sock *sk, const gfp_t priority) newsk->sk_dst_pending_confirm = 0; newsk->sk_wmem_queued = 0; newsk->sk_forward_alloc = 0; + newsk->sk_reserved_mem = 0; atomic_set(&newsk->sk_drops, 0); newsk->sk_send_head = NULL; newsk->sk_userlocks = sk->sk_userlocks & ~SOCK_BINDPORT_LOCK; diff --git a/net/core/stream.c b/net/core/stream.c index 4f1d4aa5fb38..e09ffd410685 100644 --- a/net/core/stream.c +++ b/net/core/stream.c @@ -202,7 +202,7 @@ void sk_stream_kill_queues(struct sock *sk) WARN_ON(!skb_queue_empty(&sk->sk_write_queue)); /* Account for returned memory. */ - sk_mem_reclaim(sk); + sk_mem_reclaim_final(sk); WARN_ON(sk->sk_wmem_queued); WARN_ON(sk->sk_forward_alloc); diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c index 40558033f857..2fc6074583a4 100644 --- a/net/ipv4/af_inet.c +++ b/net/ipv4/af_inet.c @@ -135,7 +135,7 @@ void inet_sock_destruct(struct sock *sk) __skb_queue_purge(&sk->sk_receive_queue); __skb_queue_purge(&sk->sk_error_queue); - sk_mem_reclaim(sk); + sk_mem_reclaim_final(sk); if (sk->sk_type == SOCK_STREAM && sk->sk_state != TCP_CLOSE) { pr_err("Attempt to release TCP socket in state %d %p\n", From patchwork Tue Sep 28 21:46:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Wang X-Patchwork-Id: 12523909 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C29E8C433EF for ; Tue, 28 Sep 2021 21:46:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A75246127A for ; Tue, 28 Sep 2021 21:46:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242899AbhI1Vsh (ORCPT ); Tue, 28 Sep 2021 17:48:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41582 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242729AbhI1Vsb (ORCPT ); Tue, 28 Sep 2021 17:48:31 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 35B98C06161C for ; Tue, 28 Sep 2021 14:46:51 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id q8-20020a056902150800b005b640f67812so649166ybu.8 for ; Tue, 28 Sep 2021 14:46:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=8m0XVgut/HJbKZ+90MngCURWWhcQtIfU/0941XQ2oPg=; b=DruefNoCb6AKgp3SJRBM0hnk3lvTkECdqXzXZ7UFgbHowvRrgljpuz7mBGPDS915WG tCtKgHKimhQ+Pmw/+iES0d2sRQPBlsOnWo5MNwH1SZvft6m9EGBxshNaj/l1Be3fMDBG aQLPrR0UW9O4D3VCQ4tf6tJYz8lEM/JIoJ56HwsTAyNLfgu8ov6wJsnubmwQutjAxGvf Pgk8FxHz/HfsgurKRLR4DNQY0ZII79Oqn7wzi8rvvxOyF7BotxsTdfLtk5HbKQbsx6d1 KJGecZiplLJI7FrE5RbAM7p+cDWxGuAznqS70PXPaB7t+ZTIAmXXrVgRysmHqbbuYkcq ZrAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=8m0XVgut/HJbKZ+90MngCURWWhcQtIfU/0941XQ2oPg=; b=Y3Nmpswmcbbi0AOXhmrWhO7QbddmeoxbLQDRiDMRj/ZyoO2VJMWEtH6Ed0Twx8YcD6 dOuY12nC3bFuHKZCwi1pb1yyBsTzZSQejpbBOwBKgkQzvO2ngePb6YndjYB+yaBPdgC7 DVczq2AqfsY/ig93+8tnQ6L8VmBlipZKhYxlNHScpWo571plBKJpIuoVJxfATmH19ZWi 0siUoiqr7VqYrICOO+EVgD2mXQdj3fv4/QXJKu2A32p8kXVXl/lKTTGe6L3CrwPJpdrh jiMJbYMzduFJibUNrt4MrNxOES7efVmd1Z7aehaFcJqNKilFigIuMrd2TRr83za133n1 33Mg== X-Gm-Message-State: AOAM5312QvnqL1VuaJOlnWt1Jq+K4PdX6fOaQbSyisF2q66zkcVEn45h vvUpeOjHr80J8WbiCIfO8FrEklrKSpo= X-Google-Smtp-Source: ABdhPJyWjqpLYszv+in+YOlf3aeh28anjdsEE1OKHJ6TKhPBFy7lDpK5IUk7QSis6RePwmXR6AJnasBkbGY= X-Received: from weiwan.svl.corp.google.com ([2620:15c:2c4:201:5d42:81b8:4b7f:efe2]) (user=weiwan job=sendgmr) by 2002:a5b:f03:: with SMTP id x3mr9746554ybr.546.1632865610437; Tue, 28 Sep 2021 14:46:50 -0700 (PDT) Date: Tue, 28 Sep 2021 14:46:42 -0700 In-Reply-To: <20210928214643.3300692-1-weiwan@google.com> Message-Id: <20210928214643.3300692-3-weiwan@google.com> Mime-Version: 1.0 References: <20210928214643.3300692-1-weiwan@google.com> X-Mailer: git-send-email 2.33.0.685.g46640cef36-goog Subject: [PATCH v2 net-next 2/3] tcp: adjust sndbuf according to sk_reserved_mem From: Wei Wang To: Jakub Kicinski , "'David S . Miller'" , netdev@vger.kernel.org Cc: Shakeel Butt , Eric Dumazet Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org If user sets SO_RESERVE_MEM socket option, in order to fully utilize the reserved memory in memory pressure state on the tx path, we modify the logic in sk_stream_moderate_sndbuf() to set sk_sndbuf according to available reserved memory, instead of MIN_SOCK_SNDBUF, and adjust it when new data is acked. Signed-off-by: Wei Wang Signed-off-by: Eric Dumazet --- include/net/sock.h | 1 + net/ipv4/tcp_input.c | 14 ++++++++++++-- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/include/net/sock.h b/include/net/sock.h index 0feb37c7f534..df3b7a45a57e 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -2377,6 +2377,7 @@ static inline void sk_stream_moderate_sndbuf(struct sock *sk) return; val = min(sk->sk_sndbuf, sk->sk_wmem_queued >> 1); + val = max_t(u32, val, sk_unused_reserved_mem(sk)); WRITE_ONCE(sk->sk_sndbuf, max_t(u32, val, SOCK_MIN_SNDBUF)); } diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index 53675e284841..06020395cc8d 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -5380,7 +5380,7 @@ static int tcp_prune_queue(struct sock *sk) return -1; } -static bool tcp_should_expand_sndbuf(const struct sock *sk) +static bool tcp_should_expand_sndbuf(struct sock *sk) { const struct tcp_sock *tp = tcp_sk(sk); @@ -5391,8 +5391,18 @@ static bool tcp_should_expand_sndbuf(const struct sock *sk) return false; /* If we are under global TCP memory pressure, do not expand. */ - if (tcp_under_memory_pressure(sk)) + if (tcp_under_memory_pressure(sk)) { + int unused_mem = sk_unused_reserved_mem(sk); + + /* Adjust sndbuf according to reserved mem. But make sure + * it never goes below SOCK_MIN_SNDBUF. + * See sk_stream_moderate_sndbuf() for more details. + */ + if (unused_mem > SOCK_MIN_SNDBUF) + WRITE_ONCE(sk->sk_sndbuf, unused_mem); + return false; + } /* If we are under soft global TCP memory pressure, do not expand. */ if (sk_memory_allocated(sk) >= sk_prot_mem_limits(sk, 0)) From patchwork Tue Sep 28 21:46:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Wang X-Patchwork-Id: 12523911 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE9F1C433F5 for ; Tue, 28 Sep 2021 21:47:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AF9366127A for ; Tue, 28 Sep 2021 21:47:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242992AbhI1Vsl (ORCPT ); Tue, 28 Sep 2021 17:48:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41606 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242929AbhI1Vsf (ORCPT ); Tue, 28 Sep 2021 17:48:35 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 62594C061753 for ; Tue, 28 Sep 2021 14:46:53 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id q8-20020a056902150800b005b640f67812so649291ybu.8 for ; Tue, 28 Sep 2021 14:46:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=CvGJ4Z6Bljun4/eGYY7fseGY0GSY3h9OiPiupWWA5tM=; b=Lze0PfYSKoHzEG5gYm69AnCJ3QP3NVt3LpHVCa4RCR3dMtiWBjwf3IfyxCDEtPmCS7 QWQDaraGXxbClPJIVDR/SYrtLwULRVmEJPm7bDPuTs7DGs94f0wunIccg5wrPuy2N3s9 X0IbcEoDTlN7gnDTm4vQoze7yAvjKDWhTUOzffjVtNrobnJGlSMyho3Y1IsRe/hZQkQK FQL/Rnwtdwk6a0xNgiCgTOg/ge4nQ1m6B1GBkSiy7RakrlPZcN1TnG+udIG+0d6Dl0rt ClEMtnUfzLJUYKl86TwZNCb5BoZWyMY2lzjyc3VrsxFC2a20qh3X+P9EixaEmog0vC0y rojg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=CvGJ4Z6Bljun4/eGYY7fseGY0GSY3h9OiPiupWWA5tM=; b=LmwS2rg5MM/zvFPsbQwwzf96TUJUfXzszjmT5DckYdy2+aj1RfCFwsZBMANsuTGuwD 3xeDjp4mjoz6WmR9xtV9ib0rRCtD64vb3g2w/16BpG19ZSR/iEaiKs6VmwQeWZcKVqbf qf75Wi1ZGi2fBpE9rAb1Ym8MpFBMShaCAHoBxikheKdBpxYePX/vaq98Ro+61256SbgI 4L3vzUcB+dJZ2eB0aJCw2dNDii+Y4zVhAd121aNXU0XGEkW4HeBn5Syjoz3kbqqnIEMb t9DIUXNb/el0WFmdTgyLWyNks/Mre04EU/NM4kpilI0c+XWjD0IRLP9eLXyc6RGFLs6k oCBw== X-Gm-Message-State: AOAM533ZT2+MCN9gpGWlfuoB6VnzjAM5nW4pC+Oq9bsYUz+PVNdewIPo HgveROodUpg8GOT2KJcc2mBKpvyNg90= X-Google-Smtp-Source: ABdhPJxI61vdVZfTksJ4YAXnqjd6cddULBil+rmmgVNujyBhFt/DE0YXfmyLqzv9UK1LGmDiEDtzA8zAkrc= X-Received: from weiwan.svl.corp.google.com ([2620:15c:2c4:201:5d42:81b8:4b7f:efe2]) (user=weiwan job=sendgmr) by 2002:a05:6902:1141:: with SMTP id p1mr10035886ybu.0.1632865612675; Tue, 28 Sep 2021 14:46:52 -0700 (PDT) Date: Tue, 28 Sep 2021 14:46:43 -0700 In-Reply-To: <20210928214643.3300692-1-weiwan@google.com> Message-Id: <20210928214643.3300692-4-weiwan@google.com> Mime-Version: 1.0 References: <20210928214643.3300692-1-weiwan@google.com> X-Mailer: git-send-email 2.33.0.685.g46640cef36-goog Subject: [PATCH v2 net-next 3/3] tcp: adjust rcv_ssthresh according to sk_reserved_mem From: Wei Wang To: Jakub Kicinski , "'David S . Miller'" , netdev@vger.kernel.org Cc: Shakeel Butt , Eric Dumazet Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org When user sets SO_RESERVE_MEM socket option, in order to utilize the reserved memory when in memory pressure state, we adjust rcv_ssthresh according to the available reserved memory for the socket, instead of using 4 * advmss always. Signed-off-by: Wei Wang Signed-off-by: Eric Dumazet --- include/net/tcp.h | 11 +++++++++++ net/ipv4/tcp_input.c | 12 ++++++++++-- net/ipv4/tcp_output.c | 3 +-- 3 files changed, 22 insertions(+), 4 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index 32cf6c01f403..4c2898ac6569 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -1421,6 +1421,17 @@ static inline int tcp_full_space(const struct sock *sk) return tcp_win_from_space(sk, READ_ONCE(sk->sk_rcvbuf)); } +static inline void tcp_adjust_rcv_ssthresh(struct sock *sk) +{ + int unused_mem = sk_unused_reserved_mem(sk); + struct tcp_sock *tp = tcp_sk(sk); + + tp->rcv_ssthresh = min(tp->rcv_ssthresh, 4U * tp->advmss); + if (unused_mem) + tp->rcv_ssthresh = max_t(u32, tp->rcv_ssthresh, + tcp_win_from_space(sk, unused_mem)); +} + void tcp_cleanup_rbuf(struct sock *sk, int copied); /* We provision sk_rcvbuf around 200% of sk_rcvlowat. diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index 06020395cc8d..246ab7b5e857 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -500,8 +500,11 @@ static void tcp_grow_window(struct sock *sk, const struct sk_buff *skb, room = min_t(int, tp->window_clamp, tcp_space(sk)) - tp->rcv_ssthresh; + if (room <= 0) + return; + /* Check #1 */ - if (room > 0 && !tcp_under_memory_pressure(sk)) { + if (!tcp_under_memory_pressure(sk)) { unsigned int truesize = truesize_adjust(adjust, skb); int incr; @@ -518,6 +521,11 @@ static void tcp_grow_window(struct sock *sk, const struct sk_buff *skb, tp->rcv_ssthresh += min(room, incr); inet_csk(sk)->icsk_ack.quick |= 1; } + } else { + /* Under pressure: + * Adjust rcv_ssthresh according to reserved mem + */ + tcp_adjust_rcv_ssthresh(sk); } } @@ -5345,7 +5353,7 @@ static int tcp_prune_queue(struct sock *sk) if (atomic_read(&sk->sk_rmem_alloc) >= sk->sk_rcvbuf) tcp_clamp_window(sk); else if (tcp_under_memory_pressure(sk)) - tp->rcv_ssthresh = min(tp->rcv_ssthresh, 4U * tp->advmss); + tcp_adjust_rcv_ssthresh(sk); if (atomic_read(&sk->sk_rmem_alloc) <= sk->sk_rcvbuf) return 0; diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index fdc39b4fbbfa..3a01e5593a17 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -2967,8 +2967,7 @@ u32 __tcp_select_window(struct sock *sk) icsk->icsk_ack.quick = 0; if (tcp_under_memory_pressure(sk)) - tp->rcv_ssthresh = min(tp->rcv_ssthresh, - 4U * tp->advmss); + tcp_adjust_rcv_ssthresh(sk); /* free_space might become our new window, make sure we don't * increase it due to wscale.