From patchwork Sat Oct 10 10:45:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 11830117 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 61919139F for ; Sat, 10 Oct 2020 10:45:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A9877221FF for ; Sat, 10 Oct 2020 10:45:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=bytedance-com.20150623.gappssmtp.com header.i=@bytedance-com.20150623.gappssmtp.com header.b="TGfDZ8tS" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A9877221FF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AA38D6B005C; Sat, 10 Oct 2020 06:45:36 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A53D16B005D; Sat, 10 Oct 2020 06:45:36 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 940AB900002; Sat, 10 Oct 2020 06:45:36 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0134.hostedemail.com [216.40.44.134]) by kanga.kvack.org (Postfix) with ESMTP id 63C5B6B005C for ; Sat, 10 Oct 2020 06:45:36 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id DACA5362D for ; Sat, 10 Oct 2020 10:45:35 +0000 (UTC) X-FDA: 77355684630.17.pig99_4f0f812271e8 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id C2F03180D0181 for ; Sat, 10 Oct 2020 10:45:35 +0000 (UTC) X-Spam-Summary: 1,0,0,7fd5395773711382,d41d8cd98f00b204,songmuchun@bytedance.com,,RULES_HIT:1:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1437:1515:1605:1730:1747:1777:1792:2196:2199:2393:2559:2562:2636:2693:2904:3138:3139:3140:3141:3142:3865:3866:3867:3868:3871:3874:4250:4321:4385:4605:5007:6119:6261:6653:6737:7875:8603:9592:10004:11026:11233:11473:11658:11914:12043:12048:12114:12291:12296:12297:12438:12517:12519:12555:12679:12683:12895:12986:13869:13894:14096:14394:14915:21080:21444:21451:21611:21627:21972:21987:21990:30005:30012:30034:30054:30070:30083,0,RBL:209.85.210.195:@bytedance.com:.lbl8.mailshell.net-66.100.201.201 62.2.0.100;04yfunxootyoqbynuqorcf5xnscdgyc6eqtkiunau9sqai5zr76355dsc94kruz.zfkik6fxg786ow567jt5rt3tbyqub5hnamc6us9bw1gkcpr6dgiziw7fwj9uhyr.w-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: pig99_4f0f812271e8 X-Filterd-Recvd-Size: 14272 Received: from mail-pf1-f195.google.com (mail-pf1-f195.google.com [209.85.210.195]) by imf41.hostedemail.com (Postfix) with ESMTP for ; Sat, 10 Oct 2020 10:45:35 +0000 (UTC) Received: by mail-pf1-f195.google.com with SMTP id a200so9104087pfa.10 for ; Sat, 10 Oct 2020 03:45:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=7kDyicVnemf+97vSMnHND6q/0cC3Faf1x64wHvnAiYU=; b=TGfDZ8tSvIy6s6P5gNA6zgwo5FFiBrWoF2vxRAiZn7HCZ7OZYpifmSrWAviXxnIkpI c5Y81F5fc4Znxu6s5nR+c+uiPeZR75Vga+byGDz+SKCwQsDISBCpXpbcBYDjlr8deIPP xlhn/dQzRe3381p3Iq5u7T5ELgIo2IGl+EZSjaoTosWKeP338QoEWm4dLdCpEpU46rap VbD3NIS9+BoSBKXR/F45VvwGJj5upbp2iGylge1QfCBpGP4MdV98f2ygzKCYXgBErq7I pB+6aHbDPQoVfvNysor5hVC2kCiocrHCdKn1RuHaPB6JSPOIZDXQ+Bar48GSN0cTm4ou Uj6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=7kDyicVnemf+97vSMnHND6q/0cC3Faf1x64wHvnAiYU=; b=Xrv5U0gAFzksUDbbS7M1juQUuxyLwd9sEBI8n5EH1Xnruq8z3bF7/LLEn3xTLSxLpw dtN1dgMBDUc0ScXyo2wUoLyDFp2mHFAqOJqJDOJlEmOfny2sgHk4oT2Qyjgm714Chf2p QJ9OJWijJDy+ufxvGyc55GmSpAKtfclg1BWwk50j2g4eCGJaGFF20q43bCd793wgDy2V FtEa4x1imCGv4cEhJ739f0sgUpI5OTXmwrl4eAFj02XkDWARdNMoyubcPzyBwFpkfYs7 /paCmlTwi/u7pZYA5TnWP1wrVtORxdRlqXpy57tskxPi+9uoE/LLOYZZE3Vop9vF+rwF Vucg== X-Gm-Message-State: AOAM530EYTOxef3Dwtj+zPqxYPQuwwMseDwXgnD1ir2FoquO6UlqWS+s QYFMjYz+a/jkKnaKZ4ZPiANIIg== X-Google-Smtp-Source: ABdhPJwW+myip0mhMxOVtzbe4oquIgi1etPLX4AHjdj0ApW1SIKkEvVWACBRFWgZrck8ueAf4VQlnw== X-Received: by 2002:a17:90a:1bc3:: with SMTP id r3mr9767091pjr.196.1602326733948; Sat, 10 Oct 2020 03:45:33 -0700 (PDT) Received: from Smcdef-MBP.local.net ([103.136.220.73]) by smtp.gmail.com with ESMTPSA id p4sm11096891pgp.83.2020.10.10.03.45.25 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Oct 2020 03:45:33 -0700 (PDT) From: Muchun Song To: davem@davemloft.net, kuba@kernel.org, edumazet@google.com, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, akpm@linux-foundation.org, kuznet@ms2.inr.ac.ru, yoshfuji@linux-ipv6.org, shakeelb@google.com, guro@fb.com, iamjoonsoo.kim@lge.com, laoar.shao@gmail.com, chris@chrisdown.name, daniel@iogearbox.net, kafai@fb.com, ast@kernel.org, jakub@cloudflare.com, linmiaohe@huawei.com, keescook@chromium.org Cc: linux-kernel@vger.kernel.org, netdev@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH] mm: memcontrol: localize mem_cgroup_sockets_enabled() check Date: Sat, 10 Oct 2020 18:45:21 +0800 Message-Id: <20201010104521.67262-1-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move the mem_cgroup_sockets_enabled() checks into memcg socket charge or uncharge functions, so the users don't have to explicitly check that condition. This is purely code cleanup patch without any functional change. But move the sk_memcg member of the sock structure to the CONFIG_MEMCG scope. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 78 ++++++++++++++++++++++++++------- include/net/sock.h | 5 ++- include/net/tcp.h | 3 +- mm/memcontrol.c | 43 +++++++++++++----- net/core/sock.c | 15 +++---- net/ipv4/inet_connection_sock.c | 6 +-- net/ipv4/tcp_output.c | 3 +- 7 files changed, 111 insertions(+), 42 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 924177502479..a1395b584947 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1460,21 +1460,48 @@ static inline void mem_cgroup_flush_foreign(struct bdi_writeback *wb) #endif /* CONFIG_CGROUP_WRITEBACK */ struct sock; -bool mem_cgroup_charge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages); -void mem_cgroup_uncharge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages); #ifdef CONFIG_MEMCG +bool __mem_cgroup_charge_skmem(struct sock *sk, unsigned int nr_pages); +void __mem_cgroup_uncharge_skmem(struct sock *sk, unsigned int nr_pages); + extern struct static_key_false memcg_sockets_enabled_key; -#define mem_cgroup_sockets_enabled static_branch_unlikely(&memcg_sockets_enabled_key) -void mem_cgroup_sk_alloc(struct sock *sk); + +static inline bool mem_cgroup_sockets_enabled(void) +{ + return static_branch_unlikely(&memcg_sockets_enabled_key); +} + +static inline bool mem_cgroup_charge_skmem(struct sock *sk, + unsigned int nr_pages) +{ + if (mem_cgroup_sockets_enabled()) + return __mem_cgroup_charge_skmem(sk, nr_pages); + return true; +} + +static inline void mem_cgroup_uncharge_skmem(struct sock *sk, + unsigned int nr_pages) +{ + if (mem_cgroup_sockets_enabled()) + __mem_cgroup_uncharge_skmem(sk, nr_pages); +} + +void __mem_cgroup_sk_alloc(struct sock *sk); + +static inline void mem_cgroup_sk_alloc(struct sock *sk) +{ + if (mem_cgroup_sockets_enabled()) + __mem_cgroup_sk_alloc(sk); +} + void mem_cgroup_sk_free(struct sock *sk); -static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) + +bool __mem_cgroup_under_socket_pressure(const struct sock *sk); + +static inline bool mem_cgroup_under_socket_pressure(const struct sock *sk) { - if (!cgroup_subsys_on_dfl(memory_cgrp_subsys) && memcg->tcpmem_pressure) - return true; - do { - if (time_before(jiffies, memcg->socket_pressure)) - return true; - } while ((memcg = parent_mem_cgroup(memcg))); + if (mem_cgroup_sockets_enabled()) + return __mem_cgroup_under_socket_pressure(sk); return false; } @@ -1483,10 +1510,31 @@ extern int memcg_expand_shrinker_maps(int new_id); extern void memcg_set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); #else -#define mem_cgroup_sockets_enabled 0 -static inline void mem_cgroup_sk_alloc(struct sock *sk) { }; -static inline void mem_cgroup_sk_free(struct sock *sk) { }; -static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) +static inline bool mem_cgroup_sockets_enabled(void) +{ + return false; +} + +static inline bool mem_cgroup_charge_skmem(struct sock *sk, + unsigned int nr_pages) +{ + return true; +} + +static inline void mem_cgroup_uncharge_skmem(struct sock *sk, + unsigned int nr_pages) +{ +} + +static inline void mem_cgroup_sk_alloc(struct sock *sk) +{ +} + +static inline void mem_cgroup_sk_free(struct sock *sk) +{ +} + +static inline bool mem_cgroup_under_socket_pressure(const struct sock *sk) { return false; } diff --git a/include/net/sock.h b/include/net/sock.h index 7dd3051551fb..f486a121fb99 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -502,7 +502,9 @@ struct sock { void *sk_security; #endif struct sock_cgroup_data sk_cgrp_data; +#ifdef CONFIG_MEMCG struct mem_cgroup *sk_memcg; +#endif void (*sk_state_change)(struct sock *sk); void (*sk_data_ready)(struct sock *sk); void (*sk_write_space)(struct sock *sk); @@ -1311,8 +1313,7 @@ static inline bool sk_under_memory_pressure(const struct sock *sk) if (!sk->sk_prot->memory_pressure) return false; - if (mem_cgroup_sockets_enabled && sk->sk_memcg && - mem_cgroup_under_socket_pressure(sk->sk_memcg)) + if (mem_cgroup_under_socket_pressure(sk)) return true; return !!*sk->sk_prot->memory_pressure; diff --git a/include/net/tcp.h b/include/net/tcp.h index e85d564446c6..c2efcf3c1495 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -257,8 +257,7 @@ extern unsigned long tcp_memory_pressure; /* optimized version of sk_under_memory_pressure() for TCP sockets */ static inline bool tcp_under_memory_pressure(const struct sock *sk) { - if (mem_cgroup_sockets_enabled && sk->sk_memcg && - mem_cgroup_under_socket_pressure(sk->sk_memcg)) + if (mem_cgroup_under_socket_pressure(sk)) return true; return READ_ONCE(tcp_memory_pressure); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4d722f4eaba8..331d7cd92e3d 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -7012,13 +7012,10 @@ void mem_cgroup_migrate(struct page *oldpage, struct page *newpage) DEFINE_STATIC_KEY_FALSE(memcg_sockets_enabled_key); EXPORT_SYMBOL(memcg_sockets_enabled_key); -void mem_cgroup_sk_alloc(struct sock *sk) +void __mem_cgroup_sk_alloc(struct sock *sk) { struct mem_cgroup *memcg; - if (!mem_cgroup_sockets_enabled) - return; - /* Do not associate the sock with unrelated interrupted task's memcg. */ if (in_interrupt()) return; @@ -7041,18 +7038,39 @@ void mem_cgroup_sk_free(struct sock *sk) css_put(&sk->sk_memcg->css); } +bool __mem_cgroup_under_socket_pressure(const struct sock *sk) +{ + struct mem_cgroup *memcg = sk->sk_memcg; + + if (!memcg) + return false; + + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys) && memcg->tcpmem_pressure) + return true; + do { + if (time_before(jiffies, memcg->socket_pressure)) + return true; + } while ((memcg = parent_mem_cgroup(memcg))); + return false; +} +EXPORT_SYMBOL(__mem_cgroup_under_socket_pressure); + /** - * mem_cgroup_charge_skmem - charge socket memory - * @memcg: memcg to charge + * __mem_cgroup_charge_skmem - charge socket memory + * @sk: socket to charge * @nr_pages: number of pages to charge * * Charges @nr_pages to @memcg. Returns %true if the charge fit within * @memcg's configured limit, %false if the charge had to be forced. */ -bool mem_cgroup_charge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages) +bool __mem_cgroup_charge_skmem(struct sock *sk, unsigned int nr_pages) { + struct mem_cgroup *memcg = sk->sk_memcg; gfp_t gfp_mask = GFP_KERNEL; + if (!memcg) + return true; + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) { struct page_counter *fail; @@ -7079,12 +7097,17 @@ bool mem_cgroup_charge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages) } /** - * mem_cgroup_uncharge_skmem - uncharge socket memory - * @memcg: memcg to uncharge + * __mem_cgroup_uncharge_skmem - uncharge socket memory + * @sk: socket to uncharge * @nr_pages: number of pages to uncharge */ -void mem_cgroup_uncharge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages) +void __mem_cgroup_uncharge_skmem(struct sock *sk, unsigned int nr_pages) { + struct mem_cgroup *memcg = sk->sk_memcg; + + if (!memcg) + return; + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) { page_counter_uncharge(&memcg->tcpmem, nr_pages); return; diff --git a/net/core/sock.c b/net/core/sock.c index ba9e7d91e2ef..5972d26f03ae 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -1901,9 +1901,10 @@ struct sock *sk_clone_lock(const struct sock *sk, const gfp_t priority) sock_reset_flag(newsk, SOCK_DONE); +#ifdef CONFIG_MEMCG /* sk->sk_memcg will be populated at accept() time */ newsk->sk_memcg = NULL; - +#endif cgroup_sk_clone(&newsk->sk_cgrp_data); rcu_read_lock(); @@ -2587,10 +2588,10 @@ int __sk_mem_raise_allocated(struct sock *sk, int size, int amt, int kind) { struct proto *prot = sk->sk_prot; long allocated = sk_memory_allocated_add(sk, amt); - bool charged = true; + bool charged; - if (mem_cgroup_sockets_enabled && sk->sk_memcg && - !(charged = mem_cgroup_charge_skmem(sk->sk_memcg, amt))) + charged = mem_cgroup_charge_skmem(sk, amt); + if (!charged) goto suppress_allocation; /* Under limit. */ @@ -2653,8 +2654,7 @@ int __sk_mem_raise_allocated(struct sock *sk, int size, int amt, int kind) sk_memory_allocated_sub(sk, amt); - if (mem_cgroup_sockets_enabled && sk->sk_memcg) - mem_cgroup_uncharge_skmem(sk->sk_memcg, amt); + mem_cgroup_uncharge_skmem(sk, amt); return 0; } @@ -2693,8 +2693,7 @@ void __sk_mem_reduce_allocated(struct sock *sk, int amount) { sk_memory_allocated_sub(sk, amount); - if (mem_cgroup_sockets_enabled && sk->sk_memcg) - mem_cgroup_uncharge_skmem(sk->sk_memcg, amount); + mem_cgroup_uncharge_skmem(sk, amount); if (sk_under_memory_pressure(sk) && (sk_memory_allocated(sk) < sk_prot_mem_limits(sk, 0))) diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c index b457dd2d6c75..06959f4ec70e 100644 --- a/net/ipv4/inet_connection_sock.c +++ b/net/ipv4/inet_connection_sock.c @@ -511,7 +511,7 @@ struct sock *inet_csk_accept(struct sock *sk, int flags, int *err, bool kern) out: release_sock(sk); - if (newsk && mem_cgroup_sockets_enabled) { + if (newsk && mem_cgroup_sockets_enabled()) { int amt; /* atomically get the memory usage, set and charge the @@ -525,8 +525,8 @@ struct sock *inet_csk_accept(struct sock *sk, int flags, int *err, bool kern) amt = sk_mem_pages(newsk->sk_forward_alloc + atomic_read(&newsk->sk_rmem_alloc)); mem_cgroup_sk_alloc(newsk); - if (newsk->sk_memcg && amt) - mem_cgroup_charge_skmem(newsk->sk_memcg, amt); + if (amt) + mem_cgroup_charge_skmem(newsk, amt); release_sock(newsk); } diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index ab79d36ed07f..ce235cc02b7e 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -3369,8 +3369,7 @@ void sk_forced_mem_schedule(struct sock *sk, int size) sk->sk_forward_alloc += amt * SK_MEM_QUANTUM; sk_memory_allocated_add(sk, amt); - if (mem_cgroup_sockets_enabled && sk->sk_memcg) - mem_cgroup_charge_skmem(sk->sk_memcg, amt); + mem_cgroup_charge_skmem(sk, amt); } /* Send a FIN. The caller locks the socket for us.