From patchwork Tue Feb 9 17:46:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12078675 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4AC4C433E0 for ; Tue, 9 Feb 2021 17:47:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 44D0064EB8 for ; Tue, 9 Feb 2021 17:47:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 44D0064EB8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C7A756B0005; Tue, 9 Feb 2021 12:47:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C042A6B006C; Tue, 9 Feb 2021 12:47:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA4C36B006E; Tue, 9 Feb 2021 12:47:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0018.hostedemail.com [216.40.44.18]) by kanga.kvack.org (Postfix) with ESMTP id 9487F6B0005 for ; Tue, 9 Feb 2021 12:47:06 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 588AD18600E4A for ; Tue, 9 Feb 2021 17:47:06 +0000 (UTC) X-FDA: 77799460452.05.loaf29_2a164e827609 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id 37DBA18604CFA for ; Tue, 9 Feb 2021 17:47:06 +0000 (UTC) X-HE-Tag: loaf29_2a164e827609 X-Filterd-Recvd-Size: 4236 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Tue, 9 Feb 2021 17:47:05 +0000 (UTC) Received: by mail-pf1-f174.google.com with SMTP id j12so12342307pfj.12 for ; Tue, 09 Feb 2021 09:47:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=aag6FmxopGJJBcIK0DInGICi96zOp3/PA6Y0cDLmFTU=; b=dLY9oFzxKgCzVXOsBpGLct8FvBj5Y9PyAp5mpJbD4YGXcrynLeTNufVZKj3H+LOfto z1r7RqIfCfzzl6MqyuqdeQlmLJaLyMPqU4rtp7rtUSXeETOoLp/ty8292Gmth69PxCUy nxERgg9VrahdeOJdbD5dXXlfxkfdg6RGbvkmkpPmfED26//MvpWPhbuqSUsg8uIn7O0M bRT8S5xJNkI5lLx6ArZ0E+Cif/roOzZShm0WVMP3RfmjbfPdJgoTT8G8M9c5aAhrDIGY kVGh4LdR3b9ojq8HLKHXd2YlKfrZkVwXvi3lJcD1NZfbsyPChLkyRkcTdZaTViZzeBx9 KZvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aag6FmxopGJJBcIK0DInGICi96zOp3/PA6Y0cDLmFTU=; b=uitJCcYuv/qgwENTdpOWeBrrfPMn34vdKASj9AsakGYWYLokbpJ868AWvT5hTk3KNf UHFvyRobT2MOpUsErduHwTiyYtMa+wvgvfPmwg42/AGXV8iHlaAvbnVW6hlq1B1z4Li2 Z0OplOOhiVd5xajiZZGv5DsehkcjDcVgzbmNJd6Qadys5CboQ0Q1hpTJKMpnV//j6YXV 6kVeTjOBfcJAvpu3V7QuCdM2AOV4SUGvGdmP/8vBbn1oibq5hcmKs8z70EKqgCY3qzfn bGMf0JvCk4Wufs7I3juYp4VhuRBPmVcc1gFuvwo9OEOuqodLXQDkaj6wVUT+hJYrGVra Ocrg== X-Gm-Message-State: AOAM533WsJwo5lFKclgFwZ5NCOKWuivxTd2spjucMeClICjdPohUuNGO RoolY+j42UwZuRo9T/wNRLM= X-Google-Smtp-Source: ABdhPJwv6VYuTil89X6G9OgCmNWSOdJhaHkhGxouDpYGR9ltvOy9KIVDT3TN3w/r7cCiZDmCsTqUjA== X-Received: by 2002:a63:1965:: with SMTP id 37mr23377015pgz.349.1612892824322; Tue, 09 Feb 2021 09:47:04 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id j1sm22260929pfr.78.2021.02.09.09.47.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Feb 2021 09:47:03 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v7 PATCH 01/12] mm: vmscan: use nid from shrink_control for tracepoint Date: Tue, 9 Feb 2021 09:46:35 -0800 Message-Id: <20210209174646.1310591-2-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210209174646.1310591-1-shy828301@gmail.com> References: <20210209174646.1310591-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The tracepoint's nid should show what node the shrink happens on, the start tracepoint uses nid from shrinkctl, but the nid might be set to 0 before end tracepoint if the shrinker is not NUMA aware, so the traceing log may show the shrink happens on one node but end up on the other node. It seems confusing. And the following patch will remove using nid directly in do_shrink_slab(), this patch also helps cleanup the code. Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai Signed-off-by: Yang Shi Reviewed-by: Shakeel Butt Acked-by: Roman Gushchin --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index b1b574ad199d..b512dd5e3a1c 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -535,7 +535,7 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, else new_nr = atomic_long_read(&shrinker->nr_deferred[nid]); - trace_mm_shrink_slab_end(shrinker, nid, freed, nr, new_nr, total_scan); + trace_mm_shrink_slab_end(shrinker, shrinkctl->nid, freed, nr, new_nr, total_scan); return freed; } From patchwork Tue Feb 9 17:46:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12078695 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B535C433DB for ; Tue, 9 Feb 2021 17:54:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7C81C64EC5 for ; Tue, 9 Feb 2021 17:54:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7C81C64EC5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0CF5D6B006E; Tue, 9 Feb 2021 12:54:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 081B86B0070; Tue, 9 Feb 2021 12:54:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E8A016B0071; Tue, 9 Feb 2021 12:54:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0225.hostedemail.com [216.40.44.225]) by kanga.kvack.org (Postfix) with ESMTP id CF1EF6B006E for ; Tue, 9 Feb 2021 12:54:18 -0500 (EST) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 95FBF1854CF59 for ; Tue, 9 Feb 2021 17:54:18 +0000 (UTC) X-FDA: 77799478596.29.drink66_0b17fbe27609 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id 64B9218604D1F for ; Tue, 9 Feb 2021 17:47:08 +0000 (UTC) X-HE-Tag: drink66_0b17fbe27609 X-Filterd-Recvd-Size: 15354 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by imf41.hostedemail.com (Postfix) with ESMTP for ; Tue, 9 Feb 2021 17:47:07 +0000 (UTC) Received: by mail-pf1-f182.google.com with SMTP id k13so5898982pfh.13 for ; Tue, 09 Feb 2021 09:47:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ajWXRmDdFksPKsCRUGly2hHv2FNuhh6NlOr49bN31iA=; b=YHyMDWSdq8bfTLioWVVNHIMaXRsSgqiH1AvCU+PedtPqEkU35DG7dkqh9tUzislTGC g3RzXqfZ5bMpXOmznj6ntZwA4IW2usCj27UJN0x4btENFQaI2EYdOZTEvzFRzsZ8vbJO xvIxse+H1+QseQODb4/ogrL6RQe2PPSigL2aRlJ/MnhKHLboMGJh1TZqokmF1BV1VET6 j9VcLV4prTualS3jW8jkrmaUuR7M1V+GPqqoeCnaVq5+mG9XgQ3J/x8g/JzKUy0+sCIO D6l7GSNf5Gs+O01xxlsnTM0S/sYdP7upehYmwdSpZQYEOKB8Dd0gP4iyyd5Ym4VN2Bef DF3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ajWXRmDdFksPKsCRUGly2hHv2FNuhh6NlOr49bN31iA=; b=tGo1UDthJDKUpzaP5sPe3/V6v6rdEvKy99PxTniI6dfaq/XYTUcmi8AjnKd+RFfi/p aWSXlIpIXCxDk/mUkvaLZ43k7ravyvdisTNWpYvvVze/kg0dHujl+jt0Yuza7xLYBdPE aNj8lKL7iuvs7+VEoVWfz1hU4vv3U5Skbu03nJgCW/r/Qa4k0jbHhZVHA1g294RVilXI +NN0WEu4Ekj28ymeKTivBedSrYf5BEdXLsHkEsc9XdBbm9mW6qEYpHfONMLsVFMhVnou /4JVyEoTXhpgTfKeJjrok8qv4q+J1oQ6rjMFhS8TgXceRf8ro85dsFhxu/ZX3sF2nMu4 mfKA== X-Gm-Message-State: AOAM531ruIQ1BgywNpSm4WCiy2WO+etGOi66xv42Jy7711DssrZEjmMX /whe6OSaWT8TG8RNAcASshE= X-Google-Smtp-Source: ABdhPJxmKDQQ/hSFhHVFUA+WfnufQTNnsoG6MILz39FqdJ5yX3HHtdQbWuBI4yWeNy61OPulZXG17g== X-Received: by 2002:a63:4d4e:: with SMTP id n14mr22810241pgl.37.1612892826512; Tue, 09 Feb 2021 09:47:06 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id j1sm22260929pfr.78.2021.02.09.09.47.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Feb 2021 09:47:05 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v7 PATCH 02/12] mm: vmscan: consolidate shrinker_maps handling code Date: Tue, 9 Feb 2021 09:46:36 -0800 Message-Id: <20210209174646.1310591-3-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210209174646.1310591-1-shy828301@gmail.com> References: <20210209174646.1310591-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The shrinker map management is not purely memcg specific, it is at the intersection between memory cgroup and shrinkers. It's allocation and assignment of a structure, and the only memcg bit is the map is being stored in a memcg structure. So move the shrinker_maps handling code into vmscan.c for tighter integration with shrinker code, and remove the "memcg_" prefix. There is no functional change. Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai Signed-off-by: Yang Shi Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt --- include/linux/memcontrol.h | 11 ++-- mm/huge_memory.c | 4 +- mm/list_lru.c | 6 +- mm/memcontrol.c | 129 +----------------------------------- mm/vmscan.c | 131 ++++++++++++++++++++++++++++++++++++- 5 files changed, 141 insertions(+), 140 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index eeb0b52203e9..1739f17e0939 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1581,10 +1581,9 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) return false; } -extern int memcg_expand_shrinker_maps(int new_id); - -extern void memcg_set_shrinker_bit(struct mem_cgroup *memcg, - int nid, int shrinker_id); +int alloc_shrinker_maps(struct mem_cgroup *memcg); +void free_shrinker_maps(struct mem_cgroup *memcg); +void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); #else #define mem_cgroup_sockets_enabled 0 static inline void mem_cgroup_sk_alloc(struct sock *sk) { }; @@ -1594,8 +1593,8 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) return false; } -static inline void memcg_set_shrinker_bit(struct mem_cgroup *memcg, - int nid, int shrinker_id) +static inline void set_shrinker_bit(struct mem_cgroup *memcg, + int nid, int shrinker_id) { } #endif diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9237976abe72..05190d7f32ae 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2823,8 +2823,8 @@ void deferred_split_huge_page(struct page *page) ds_queue->split_queue_len++; #ifdef CONFIG_MEMCG if (memcg) - memcg_set_shrinker_bit(memcg, page_to_nid(page), - deferred_split_shrinker.id); + set_shrinker_bit(memcg, page_to_nid(page), + deferred_split_shrinker.id); #endif } spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); diff --git a/mm/list_lru.c b/mm/list_lru.c index fe230081690b..628030fa5f69 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -125,8 +125,8 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item) list_add_tail(item, &l->list); /* Set shrinker bit if the first element was added */ if (!l->nr_items++) - memcg_set_shrinker_bit(memcg, nid, - lru_shrinker_id(lru)); + set_shrinker_bit(memcg, nid, + lru_shrinker_id(lru)); nlru->nr_items++; spin_unlock(&nlru->lock); return true; @@ -548,7 +548,7 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, if (src->nr_items) { dst->nr_items += src->nr_items; - memcg_set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); + set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); src->nr_items = 0; } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1bdb93ee8e72..f5c9a0d2160b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -397,129 +397,6 @@ DEFINE_STATIC_KEY_FALSE(memcg_kmem_enabled_key); EXPORT_SYMBOL(memcg_kmem_enabled_key); #endif -static int memcg_shrinker_map_size; -static DEFINE_MUTEX(memcg_shrinker_map_mutex); - -static void memcg_free_shrinker_map_rcu(struct rcu_head *head) -{ - kvfree(container_of(head, struct memcg_shrinker_map, rcu)); -} - -static int memcg_expand_one_shrinker_map(struct mem_cgroup *memcg, - int size, int old_size) -{ - struct memcg_shrinker_map *new, *old; - int nid; - - lockdep_assert_held(&memcg_shrinker_map_mutex); - - for_each_node(nid) { - old = rcu_dereference_protected( - mem_cgroup_nodeinfo(memcg, nid)->shrinker_map, true); - /* Not yet online memcg */ - if (!old) - return 0; - - new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); - if (!new) - return -ENOMEM; - - /* Set all old bits, clear all new bits */ - memset(new->map, (int)0xff, old_size); - memset((void *)new->map + old_size, 0, size - old_size); - - rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, new); - call_rcu(&old->rcu, memcg_free_shrinker_map_rcu); - } - - return 0; -} - -static void memcg_free_shrinker_maps(struct mem_cgroup *memcg) -{ - struct mem_cgroup_per_node *pn; - struct memcg_shrinker_map *map; - int nid; - - if (mem_cgroup_is_root(memcg)) - return; - - for_each_node(nid) { - pn = mem_cgroup_nodeinfo(memcg, nid); - map = rcu_dereference_protected(pn->shrinker_map, true); - kvfree(map); - rcu_assign_pointer(pn->shrinker_map, NULL); - } -} - -static int memcg_alloc_shrinker_maps(struct mem_cgroup *memcg) -{ - struct memcg_shrinker_map *map; - int nid, size, ret = 0; - - if (mem_cgroup_is_root(memcg)) - return 0; - - mutex_lock(&memcg_shrinker_map_mutex); - size = memcg_shrinker_map_size; - for_each_node(nid) { - map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); - if (!map) { - memcg_free_shrinker_maps(memcg); - ret = -ENOMEM; - break; - } - rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); - } - mutex_unlock(&memcg_shrinker_map_mutex); - - return ret; -} - -int memcg_expand_shrinker_maps(int new_id) -{ - int size, old_size, ret = 0; - struct mem_cgroup *memcg; - - size = DIV_ROUND_UP(new_id + 1, BITS_PER_LONG) * sizeof(unsigned long); - old_size = memcg_shrinker_map_size; - if (size <= old_size) - return 0; - - mutex_lock(&memcg_shrinker_map_mutex); - if (!root_mem_cgroup) - goto unlock; - - for_each_mem_cgroup(memcg) { - if (mem_cgroup_is_root(memcg)) - continue; - ret = memcg_expand_one_shrinker_map(memcg, size, old_size); - if (ret) { - mem_cgroup_iter_break(NULL, memcg); - goto unlock; - } - } -unlock: - if (!ret) - memcg_shrinker_map_size = size; - mutex_unlock(&memcg_shrinker_map_mutex); - return ret; -} - -void memcg_set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) -{ - if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { - struct memcg_shrinker_map *map; - - rcu_read_lock(); - map = rcu_dereference(memcg->nodeinfo[nid]->shrinker_map); - /* Pairs with smp mb in shrink_slab() */ - smp_mb__before_atomic(); - set_bit(shrinker_id, map->map); - rcu_read_unlock(); - } -} - /** * mem_cgroup_css_from_page - css of the memcg associated with a page * @page: page of interest @@ -5369,11 +5246,11 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) struct mem_cgroup *memcg = mem_cgroup_from_css(css); /* - * A memcg must be visible for memcg_expand_shrinker_maps() + * A memcg must be visible for expand_shrinker_maps() * by the time the maps are allocated. So, we allocate maps * here, when for_each_mem_cgroup() can't skip it. */ - if (memcg_alloc_shrinker_maps(memcg)) { + if (alloc_shrinker_maps(memcg)) { mem_cgroup_id_remove(memcg); return -ENOMEM; } @@ -5437,7 +5314,7 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) vmpressure_cleanup(&memcg->vmpressure); cancel_work_sync(&memcg->high_work); mem_cgroup_remove_from_trees(memcg); - memcg_free_shrinker_maps(memcg); + free_shrinker_maps(memcg); memcg_free_kmem(memcg); mem_cgroup_free(memcg); } diff --git a/mm/vmscan.c b/mm/vmscan.c index b512dd5e3a1c..96b08c79f18d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -185,6 +185,131 @@ static LIST_HEAD(shrinker_list); static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG + +static int memcg_shrinker_map_size; +static DEFINE_MUTEX(memcg_shrinker_map_mutex); + +static void free_shrinker_map_rcu(struct rcu_head *head) +{ + kvfree(container_of(head, struct memcg_shrinker_map, rcu)); +} + +static int expand_one_shrinker_map(struct mem_cgroup *memcg, + int size, int old_size) +{ + struct memcg_shrinker_map *new, *old; + int nid; + + lockdep_assert_held(&memcg_shrinker_map_mutex); + + for_each_node(nid) { + old = rcu_dereference_protected( + mem_cgroup_nodeinfo(memcg, nid)->shrinker_map, true); + /* Not yet online memcg */ + if (!old) + return 0; + + new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); + if (!new) + return -ENOMEM; + + /* Set all old bits, clear all new bits */ + memset(new->map, (int)0xff, old_size); + memset((void *)new->map + old_size, 0, size - old_size); + + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, new); + call_rcu(&old->rcu, free_shrinker_map_rcu); + } + + return 0; +} + +void free_shrinker_maps(struct mem_cgroup *memcg) +{ + struct mem_cgroup_per_node *pn; + struct memcg_shrinker_map *map; + int nid; + + if (mem_cgroup_is_root(memcg)) + return; + + for_each_node(nid) { + pn = mem_cgroup_nodeinfo(memcg, nid); + map = rcu_dereference_protected(pn->shrinker_map, true); + kvfree(map); + rcu_assign_pointer(pn->shrinker_map, NULL); + } +} + +int alloc_shrinker_maps(struct mem_cgroup *memcg) +{ + struct memcg_shrinker_map *map; + int nid, size, ret = 0; + + if (mem_cgroup_is_root(memcg)) + return 0; + + mutex_lock(&memcg_shrinker_map_mutex); + size = memcg_shrinker_map_size; + for_each_node(nid) { + map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); + if (!map) { + free_shrinker_maps(memcg); + ret = -ENOMEM; + break; + } + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); + } + mutex_unlock(&memcg_shrinker_map_mutex); + + return ret; +} + +static int expand_shrinker_maps(int new_id) +{ + int size, old_size, ret = 0; + struct mem_cgroup *memcg; + + size = DIV_ROUND_UP(new_id + 1, BITS_PER_LONG) * sizeof(unsigned long); + old_size = memcg_shrinker_map_size; + if (size <= old_size) + return 0; + + mutex_lock(&memcg_shrinker_map_mutex); + if (!root_mem_cgroup) + goto unlock; + + memcg = mem_cgroup_iter(NULL, NULL, NULL); + do { + if (mem_cgroup_is_root(memcg)) + continue; + ret = expand_one_shrinker_map(memcg, size, old_size); + if (ret) { + mem_cgroup_iter_break(NULL, memcg); + goto unlock; + } + } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); +unlock: + if (!ret) + memcg_shrinker_map_size = size; + mutex_unlock(&memcg_shrinker_map_mutex); + return ret; +} + +void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) +{ + if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { + struct memcg_shrinker_map *map; + + rcu_read_lock(); + map = rcu_dereference(memcg->nodeinfo[nid]->shrinker_map); + /* Pairs with smp mb in shrink_slab() */ + smp_mb__before_atomic(); + set_bit(shrinker_id, map->map); + rcu_read_unlock(); + } +} + /* * We allow subsystems to populate their shrinker-related * LRU lists before register_shrinker_prepared() is called @@ -212,7 +337,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) goto unlock; if (id >= shrinker_nr_max) { - if (memcg_expand_shrinker_maps(id)) { + if (expand_shrinker_maps(id)) { idr_remove(&shrinker_idr, id); goto unlock; } @@ -589,7 +714,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, * case, we invoke the shrinker one more time and reset * the bit if it reports that it is not empty anymore. * The memory barrier here pairs with the barrier in - * memcg_set_shrinker_bit(): + * set_shrinker_bit(): * * list_lru_add() shrink_slab_memcg() * list_add_tail() clear_bit() @@ -601,7 +726,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; else - memcg_set_shrinker_bit(memcg, nid, i); + set_shrinker_bit(memcg, nid, i); } freed += ret; From patchwork Tue Feb 9 17:46:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12078677 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0321CC433E0 for ; Tue, 9 Feb 2021 17:47:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A4D7D64EB6 for ; Tue, 9 Feb 2021 17:47:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A4D7D64EB6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3C6DD6B006C; Tue, 9 Feb 2021 12:47:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 352806B006E; Tue, 9 Feb 2021 12:47:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 241B06B0070; Tue, 9 Feb 2021 12:47:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0223.hostedemail.com [216.40.44.223]) by kanga.kvack.org (Postfix) with ESMTP id 0AB9C6B006C for ; Tue, 9 Feb 2021 12:47:11 -0500 (EST) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id C740452AB for ; Tue, 9 Feb 2021 17:47:10 +0000 (UTC) X-FDA: 77799460620.07.move07_43046dc27609 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id A3F2918030435 for ; Tue, 9 Feb 2021 17:47:10 +0000 (UTC) X-HE-Tag: move07_43046dc27609 X-Filterd-Recvd-Size: 6274 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Tue, 9 Feb 2021 17:47:09 +0000 (UTC) Received: by mail-pl1-f172.google.com with SMTP id z7so250699plk.7 for ; Tue, 09 Feb 2021 09:47:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lKgWNZLrDd0Eei0IRyAIkFAH1mH+Bykeng/kbWxOqQI=; b=ZHWKtzam//P32Zt7GlfUKQIgBqYXUgTyfJinorJosiN4SiLrzzOB5gFaNy9iJPTsPN iPNHM27PiAPGB5B5c2eRk1s2MuhbvZdKUF7MQW36PC643l1nz2cLprkX4h8spiJ8jFEM pBjDaHNEkGKNJNJqa4xSfuBzq5rTR8un/e/5j2cJ2ZsFtl63pVnkczp08X+J+M+G3oIF h/M88R6hBlAunlW+0VEWzipqpwJLeqbBNybrwkL7u1PCkv0XAs12IdNqhPQUC0oRvAxx xH8GEGROukGKQd17TyxiF6LNll/AACoe60iKnAGJsJxmhF1dMl5M6AGPZ4vC3siAzn+i SffQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lKgWNZLrDd0Eei0IRyAIkFAH1mH+Bykeng/kbWxOqQI=; b=j+ESXE4GXgbvrCu732x12REWgikBuVDLJqF7QqnVyGB+MjYPtHHspoFlQByEBANikm m5ACCPgWEyF4wIC39ZekpOmHLA5OWk49f8njhyQGUIpoX7HohL6Tk/2E1r93QQZO1eU+ HB7GR0VYtkc5k78U6PTld49/mGlkE039RjJjNG8pHJS0N+FH5Dl+nW4gTEhiOPj9YM8o BA2k4IuclX7GbxgefoM5547XlY3VSOGM9FzOt9391qqQe8+d3n5kvFEHacNYo9cZBgbx /3KvfRnzfYaraVIx7znrAiJutc1L6xGFRDe2jAarsism4qYiMD3qcdfQLRavmCPgIST0 udhg== X-Gm-Message-State: AOAM531iitm72PXBGj85rR3Z4X79aIacSG2NsHPocb/MUnbfNYeqtE0s RKHdMyDtbmaT5Jtt4SLOhMc= X-Google-Smtp-Source: ABdhPJyHwMvBQ3+/v8vjRYQfJSipk1mypCtSPFHjH6iMTYpJBuPnLWzUJFqrD/AfsPW7vbQFYGYRFA== X-Received: by 2002:a17:90a:ab17:: with SMTP id m23mr5057195pjq.0.1612892829115; Tue, 09 Feb 2021 09:47:09 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id j1sm22260929pfr.78.2021.02.09.09.47.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Feb 2021 09:47:07 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v7 PATCH 03/12] mm: vmscan: use shrinker_rwsem to protect shrinker_maps allocation Date: Tue, 9 Feb 2021 09:46:37 -0800 Message-Id: <20210209174646.1310591-4-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210209174646.1310591-1-shy828301@gmail.com> References: <20210209174646.1310591-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since memcg_shrinker_map_size just can be changed under holding shrinker_rwsem exclusively, the read side can be protected by holding read lock, so it sounds superfluous to have a dedicated mutex. Kirill Tkhai suggested use write lock since: * We want the assignment to shrinker_maps is visible for shrink_slab_memcg(). * The rcu_dereference_protected() dereferrencing in shrink_slab_memcg(), but in case of we use READ lock in alloc_shrinker_maps(), the dereferrencing is not actually protected. * READ lock makes alloc_shrinker_info() racy against memory allocation fail. alloc_shrinker_info()->free_shrinker_info() may free memory right after shrink_slab_memcg() dereferenced it. You may say shrink_slab_memcg()->mem_cgroup_online() protects us from it? Yes, sure, but this is not the thing we want to remember in the future, since this spreads modularity. And a test with heavy paging workload didn't show write lock makes things worse. Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai Signed-off-by: Yang Shi Acked-by: Roman Gushchin --- mm/vmscan.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 96b08c79f18d..e4ddaaaeffe2 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -187,7 +187,6 @@ static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG static int memcg_shrinker_map_size; -static DEFINE_MUTEX(memcg_shrinker_map_mutex); static void free_shrinker_map_rcu(struct rcu_head *head) { @@ -200,8 +199,6 @@ static int expand_one_shrinker_map(struct mem_cgroup *memcg, struct memcg_shrinker_map *new, *old; int nid; - lockdep_assert_held(&memcg_shrinker_map_mutex); - for_each_node(nid) { old = rcu_dereference_protected( mem_cgroup_nodeinfo(memcg, nid)->shrinker_map, true); @@ -249,7 +246,7 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) if (mem_cgroup_is_root(memcg)) return 0; - mutex_lock(&memcg_shrinker_map_mutex); + down_write(&shrinker_rwsem); size = memcg_shrinker_map_size; for_each_node(nid) { map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); @@ -260,7 +257,7 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) } rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); } - mutex_unlock(&memcg_shrinker_map_mutex); + up_write(&shrinker_rwsem); return ret; } @@ -275,9 +272,8 @@ static int expand_shrinker_maps(int new_id) if (size <= old_size) return 0; - mutex_lock(&memcg_shrinker_map_mutex); if (!root_mem_cgroup) - goto unlock; + goto out; memcg = mem_cgroup_iter(NULL, NULL, NULL); do { @@ -286,13 +282,13 @@ static int expand_shrinker_maps(int new_id) ret = expand_one_shrinker_map(memcg, size, old_size); if (ret) { mem_cgroup_iter_break(NULL, memcg); - goto unlock; + goto out; } } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); -unlock: +out: if (!ret) memcg_shrinker_map_size = size; - mutex_unlock(&memcg_shrinker_map_mutex); + return ret; } From patchwork Tue Feb 9 17:46:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12078679 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACC55C433E0 for ; Tue, 9 Feb 2021 17:47:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 30F0764EB6 for ; Tue, 9 Feb 2021 17:47:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 30F0764EB6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B904A6B006E; Tue, 9 Feb 2021 12:47:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B1A866B0070; Tue, 9 Feb 2021 12:47:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9BD2C6B0071; Tue, 9 Feb 2021 12:47:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0205.hostedemail.com [216.40.44.205]) by kanga.kvack.org (Postfix) with ESMTP id 875976B006E for ; Tue, 9 Feb 2021 12:47:13 -0500 (EST) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 4F24C8245571 for ; Tue, 9 Feb 2021 17:47:13 +0000 (UTC) X-FDA: 77799460746.12.baby59_5305c9127609 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id 31330181D13B3 for ; Tue, 9 Feb 2021 17:47:13 +0000 (UTC) X-HE-Tag: baby59_5305c9127609 X-Filterd-Recvd-Size: 5685 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf42.hostedemail.com (Postfix) with ESMTP for ; Tue, 9 Feb 2021 17:47:12 +0000 (UTC) Received: by mail-pl1-f178.google.com with SMTP id s15so10145773plr.9 for ; Tue, 09 Feb 2021 09:47:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jMTd+AsX3BHmetJX1pSMFbMyduyVUPUtb+au9yZ0BQ8=; b=fb+LVYBoz1sAntMs28140vW8hhrBN9G2hhhUfuOAnPJDxB2NSshgpMJ9J2i5x+VN/f BTgbNHrvx+hIDQtkNqkxrQpE3Tx5105FsVMdgmnt9RYit5nYtD5kpLh6YjiuKqaaouID yLqsMwMoqWVaFX9IdBU1mODZD3wpmOajuHzlG3sshZrFWqywgoUXOPNhXKCeA8LPVWvd jMn7avn64DoAZsd80tsvoI7buryToQAvZMMKvo/S+CLifa3FY7g46diyVh+iqrpww0SV MNpazR9nSH9XVe5Cn0ItFLN/oAKCPr6VyBQmvX7aQZvEhoMsuaagEYlQZpDpYa67c+US ptUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jMTd+AsX3BHmetJX1pSMFbMyduyVUPUtb+au9yZ0BQ8=; b=Cb7/OMrftimLLSmE44DXRyaSpiUopxAgqdi7OKgLjCBBfAE9d1LXS8/BLTYXJWOOJp dW0D7LftF5qLDtUNg9SpjtSJwSyr5LyDEAFOg8JkPZKJNzYdQGJjfk/lEc8yYWMHCETH BqXdhUPckh6gl8jFSRQyy4brhxQZpYeG3dh3oMjFLwtX1oAgwck+OLB9QYTIybb1dGTc /JM/HYf4UhYCm0u11X0lXdKl5KQl7RHz7lnKbiz0ocoVMmM4KPYi87W/AkgG5Gn33oOT t4VqPAN6rpUIthMKE76mOfvkiRAPgHN5Rf/4lzuYAL58RT+a8ShqA8tzq5yZWG27fDo/ /Phg== X-Gm-Message-State: AOAM531x6DSRGpxKv2IL8G5dJiI0SQIZKmW2z/RDpAWfDuAIlDTYK72B 2PgOHGXkoArmopt0BofykdI= X-Google-Smtp-Source: ABdhPJwaqafmo1fqnOIsjZW/57shfRny1c9eDctpoMFQE7YXYsnfdumP7ICN/Kp3/hMsG89JqypTSg== X-Received: by 2002:a17:902:a383:b029:e0:10e6:6ed7 with SMTP id x3-20020a170902a383b02900e010e66ed7mr22633037pla.5.1612892831486; Tue, 09 Feb 2021 09:47:11 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id j1sm22260929pfr.78.2021.02.09.09.47.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Feb 2021 09:47:10 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v7 PATCH 04/12] mm: vmscan: remove memcg_shrinker_map_size Date: Tue, 9 Feb 2021 09:46:38 -0800 Message-Id: <20210209174646.1310591-5-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210209174646.1310591-1-shy828301@gmail.com> References: <20210209174646.1310591-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Both memcg_shrinker_map_size and shrinker_nr_max is maintained, but actually the map size can be calculated via shrinker_nr_max, so it seems unnecessary to keep both. Remove memcg_shrinker_map_size since shrinker_nr_max is also used by iterating the bit map. Acked-by: Kirill Tkhai Signed-off-by: Yang Shi Acked-by: Vlastimil Babka --- mm/vmscan.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index e4ddaaaeffe2..641077b09e5d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -185,8 +185,10 @@ static LIST_HEAD(shrinker_list); static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG +static int shrinker_nr_max; -static int memcg_shrinker_map_size; +#define NR_MAX_TO_SHR_MAP_SIZE(nr_max) \ + (DIV_ROUND_UP(nr_max, BITS_PER_LONG) * sizeof(unsigned long)) static void free_shrinker_map_rcu(struct rcu_head *head) { @@ -247,7 +249,7 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) return 0; down_write(&shrinker_rwsem); - size = memcg_shrinker_map_size; + size = NR_MAX_TO_SHR_MAP_SIZE(shrinker_nr_max); for_each_node(nid) { map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); if (!map) { @@ -265,12 +267,13 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) static int expand_shrinker_maps(int new_id) { int size, old_size, ret = 0; + int new_nr_max = new_id + 1; struct mem_cgroup *memcg; - size = DIV_ROUND_UP(new_id + 1, BITS_PER_LONG) * sizeof(unsigned long); - old_size = memcg_shrinker_map_size; + size = NR_MAX_TO_SHR_MAP_SIZE(new_nr_max); + old_size = NR_MAX_TO_SHR_MAP_SIZE(shrinker_nr_max); if (size <= old_size) - return 0; + goto out; if (!root_mem_cgroup) goto out; @@ -287,7 +290,7 @@ static int expand_shrinker_maps(int new_id) } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); out: if (!ret) - memcg_shrinker_map_size = size; + shrinker_nr_max = new_nr_max; return ret; } @@ -320,7 +323,6 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) #define SHRINKER_REGISTERING ((struct shrinker *)~0UL) static DEFINE_IDR(shrinker_idr); -static int shrinker_nr_max; static int prealloc_memcg_shrinker(struct shrinker *shrinker) { @@ -337,8 +339,6 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) idr_remove(&shrinker_idr, id); goto unlock; } - - shrinker_nr_max = id + 1; } shrinker->id = id; ret = 0; From patchwork Tue Feb 9 17:46:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12078697 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F64DC433E0 for ; Tue, 9 Feb 2021 17:55:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0AC8864EC4 for ; Tue, 9 Feb 2021 17:54:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0AC8864EC4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 75E676B0005; Tue, 9 Feb 2021 12:54:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 735886B006C; Tue, 9 Feb 2021 12:54:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 64D9E6B0071; Tue, 9 Feb 2021 12:54:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0026.hostedemail.com [216.40.44.26]) by kanga.kvack.org (Postfix) with ESMTP id 4BD316B0005 for ; Tue, 9 Feb 2021 12:54:59 -0500 (EST) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 15D891854CE0C for ; Tue, 9 Feb 2021 17:54:59 +0000 (UTC) X-FDA: 77799480318.18.nut41_360a1b827609 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin18.hostedemail.com (Postfix) with ESMTP id 2D8841005B55D for ; Tue, 9 Feb 2021 17:47:16 +0000 (UTC) X-HE-Tag: nut41_360a1b827609 X-Filterd-Recvd-Size: 11896 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) by imf49.hostedemail.com (Postfix) with ESMTP for ; Tue, 9 Feb 2021 17:47:15 +0000 (UTC) Received: by mail-pf1-f170.google.com with SMTP id d26so11135543pfn.5 for ; Tue, 09 Feb 2021 09:47:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vwOSQaUT4eqAIpLfDrhyexJQexyMJOejb+LRmYHuY6c=; b=j0YRV8IUH+FkqKhTqgAIlgT5AIQdh7Nr1kMRazoEy9IsbMJy2alWCeqlrJ+n4imhAC J29Acu0k6ee2xWHSzzxoGSVp4ORaw95vbKhrI/jwgkkzhVv/MqpYHJCRR76aGvscG2Vn Z8Og9+I3ot99wuqu1DbRpFEapHurcBW5+rGWS0H/j6PY7+dSfPWreKW4POLIjpnmrpVO zApCo19K4x7OFfPURic1N7Lg+rbzeR+vMUwDyQPDZsvlI68uKehr8FZiD+9fgYOwMIMo 4ptW9wAbkGrQ4Z0Zk38KpNdMKhDNTvi78c2KorjzFA7v5qYRX6c6xzOj08n/tl95aUNd +v0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vwOSQaUT4eqAIpLfDrhyexJQexyMJOejb+LRmYHuY6c=; b=HrTjiw6+irO3WO+fLCgSXfEXQ4xgCJ0dGTncbSs7fdKzovPwel8l5+06j2XIxY8/og FZHA+fc5T8FfB6K2l6y9L91kKGg99mm8tuZ0PmjIzYuFM350zzcSrCwXqPaFLQNhG7ZL Qezm4XI/sc+e9HOlFF/8o5jUQgjqEUZmh5WeaU56ZvvIs3w0Lhg89Fg/8i/PoXGdvUZX LAzyhRGzFmuCNKmTmq/9Nh9bLDbOFpCh9NOZP2xu9VRVISLe0PUC0TOMug1mt0WMG3Pe FpfawCe6u6xZ7Ra6NG3HzBmUAWQi6bXTCfqnr5CcStluPgZMGb1kIRhWh3OaTSNDlH1K 1GVQ== X-Gm-Message-State: AOAM530zDlP4wT2XN503/pCu0ICEDAtOpSdP7m8YjhbMlmYnOU+lKYF1 AxIUS1Cr2dp+3j8meHzdlSA= X-Google-Smtp-Source: ABdhPJxG7QKgygmxt5X+HVb/ltlq4fnomtgaRuUAWRFvqhMw2KuOZdA6EUhUilMMRHJLGeO0m22ZZw== X-Received: by 2002:a65:6096:: with SMTP id t22mr3595043pgu.66.1612892834752; Tue, 09 Feb 2021 09:47:14 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id j1sm22260929pfr.78.2021.02.09.09.47.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Feb 2021 09:47:13 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v7 PATCH 05/12] mm: memcontrol: rename shrinker_map to shrinker_info Date: Tue, 9 Feb 2021 09:46:39 -0800 Message-Id: <20210209174646.1310591-6-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210209174646.1310591-1-shy828301@gmail.com> References: <20210209174646.1310591-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The following patch is going to add nr_deferred into shrinker_map, the change will make shrinker_map not only include map anymore, so rename it to "memcg_shrinker_info". And this should make the patch adding nr_deferred cleaner and readable and make review easier. Also remove the "memcg_" prefix. Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai Signed-off-by: Yang Shi Acked-by: Roman Gushchin --- include/linux/memcontrol.h | 8 ++--- mm/memcontrol.c | 6 ++-- mm/vmscan.c | 62 +++++++++++++++++++------------------- 3 files changed, 38 insertions(+), 38 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 1739f17e0939..4c9253896e25 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -96,7 +96,7 @@ struct lruvec_stat { * Bitmap of shrinker::id corresponding to memcg-aware shrinkers, * which have elements charged to this memcg. */ -struct memcg_shrinker_map { +struct shrinker_info { struct rcu_head rcu; unsigned long map[]; }; @@ -118,7 +118,7 @@ struct mem_cgroup_per_node { struct mem_cgroup_reclaim_iter iter; - struct memcg_shrinker_map __rcu *shrinker_map; + struct shrinker_info __rcu *shrinker_info; struct rb_node tree_node; /* RB tree node */ unsigned long usage_in_excess;/* Set to the value by which */ @@ -1581,8 +1581,8 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) return false; } -int alloc_shrinker_maps(struct mem_cgroup *memcg); -void free_shrinker_maps(struct mem_cgroup *memcg); +int alloc_shrinker_info(struct mem_cgroup *memcg); +void free_shrinker_info(struct mem_cgroup *memcg); void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); #else #define mem_cgroup_sockets_enabled 0 diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f5c9a0d2160b..f64ad0d044d9 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5246,11 +5246,11 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) struct mem_cgroup *memcg = mem_cgroup_from_css(css); /* - * A memcg must be visible for expand_shrinker_maps() + * A memcg must be visible for expand_shrinker_info() * by the time the maps are allocated. So, we allocate maps * here, when for_each_mem_cgroup() can't skip it. */ - if (alloc_shrinker_maps(memcg)) { + if (alloc_shrinker_info(memcg)) { mem_cgroup_id_remove(memcg); return -ENOMEM; } @@ -5314,7 +5314,7 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) vmpressure_cleanup(&memcg->vmpressure); cancel_work_sync(&memcg->high_work); mem_cgroup_remove_from_trees(memcg); - free_shrinker_maps(memcg); + free_shrinker_info(memcg); memcg_free_kmem(memcg); mem_cgroup_free(memcg); } diff --git a/mm/vmscan.c b/mm/vmscan.c index 641077b09e5d..9436f9246d32 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -190,20 +190,20 @@ static int shrinker_nr_max; #define NR_MAX_TO_SHR_MAP_SIZE(nr_max) \ (DIV_ROUND_UP(nr_max, BITS_PER_LONG) * sizeof(unsigned long)) -static void free_shrinker_map_rcu(struct rcu_head *head) +static void free_shrinker_info_rcu(struct rcu_head *head) { - kvfree(container_of(head, struct memcg_shrinker_map, rcu)); + kvfree(container_of(head, struct shrinker_info, rcu)); } -static int expand_one_shrinker_map(struct mem_cgroup *memcg, +static int expand_one_shrinker_info(struct mem_cgroup *memcg, int size, int old_size) { - struct memcg_shrinker_map *new, *old; + struct shrinker_info *new, *old; int nid; for_each_node(nid) { old = rcu_dereference_protected( - mem_cgroup_nodeinfo(memcg, nid)->shrinker_map, true); + mem_cgroup_nodeinfo(memcg, nid)->shrinker_info, true); /* Not yet online memcg */ if (!old) return 0; @@ -216,17 +216,17 @@ static int expand_one_shrinker_map(struct mem_cgroup *memcg, memset(new->map, (int)0xff, old_size); memset((void *)new->map + old_size, 0, size - old_size); - rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, new); - call_rcu(&old->rcu, free_shrinker_map_rcu); + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, new); + call_rcu(&old->rcu, free_shrinker_info_rcu); } return 0; } -void free_shrinker_maps(struct mem_cgroup *memcg) +void free_shrinker_info(struct mem_cgroup *memcg) { struct mem_cgroup_per_node *pn; - struct memcg_shrinker_map *map; + struct shrinker_info *info; int nid; if (mem_cgroup_is_root(memcg)) @@ -234,15 +234,15 @@ void free_shrinker_maps(struct mem_cgroup *memcg) for_each_node(nid) { pn = mem_cgroup_nodeinfo(memcg, nid); - map = rcu_dereference_protected(pn->shrinker_map, true); - kvfree(map); - rcu_assign_pointer(pn->shrinker_map, NULL); + info = rcu_dereference_protected(pn->shrinker_info, true); + kvfree(info); + rcu_assign_pointer(pn->shrinker_info, NULL); } } -int alloc_shrinker_maps(struct mem_cgroup *memcg) +int alloc_shrinker_info(struct mem_cgroup *memcg) { - struct memcg_shrinker_map *map; + struct shrinker_info *info; int nid, size, ret = 0; if (mem_cgroup_is_root(memcg)) @@ -251,20 +251,20 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) down_write(&shrinker_rwsem); size = NR_MAX_TO_SHR_MAP_SIZE(shrinker_nr_max); for_each_node(nid) { - map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); - if (!map) { - free_shrinker_maps(memcg); + info = kvzalloc_node(sizeof(*info) + size, GFP_KERNEL, nid); + if (!info) { + free_shrinker_info(memcg); ret = -ENOMEM; break; } - rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); return ret; } -static int expand_shrinker_maps(int new_id) +static int expand_shrinker_info(int new_id) { int size, old_size, ret = 0; int new_nr_max = new_id + 1; @@ -282,7 +282,7 @@ static int expand_shrinker_maps(int new_id) do { if (mem_cgroup_is_root(memcg)) continue; - ret = expand_one_shrinker_map(memcg, size, old_size); + ret = expand_one_shrinker_info(memcg, size, old_size); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; @@ -298,13 +298,13 @@ static int expand_shrinker_maps(int new_id) void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) { if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { - struct memcg_shrinker_map *map; + struct shrinker_info *info; rcu_read_lock(); - map = rcu_dereference(memcg->nodeinfo[nid]->shrinker_map); + info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); /* Pairs with smp mb in shrink_slab() */ smp_mb__before_atomic(); - set_bit(shrinker_id, map->map); + set_bit(shrinker_id, info->map); rcu_read_unlock(); } } @@ -335,7 +335,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) goto unlock; if (id >= shrinker_nr_max) { - if (expand_shrinker_maps(id)) { + if (expand_shrinker_info(id)) { idr_remove(&shrinker_idr, id); goto unlock; } @@ -664,7 +664,7 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, struct mem_cgroup *memcg, int priority) { - struct memcg_shrinker_map *map; + struct shrinker_info *info; unsigned long ret, freed = 0; int i; @@ -674,12 +674,12 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (!down_read_trylock(&shrinker_rwsem)) return 0; - map = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_map, - true); - if (unlikely(!map)) + info = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, + true); + if (unlikely(!info)) goto unlock; - for_each_set_bit(i, map->map, shrinker_nr_max) { + for_each_set_bit(i, info->map, shrinker_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -690,7 +690,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, shrinker = idr_find(&shrinker_idr, i); if (unlikely(!shrinker || shrinker == SHRINKER_REGISTERING)) { if (!shrinker) - clear_bit(i, map->map); + clear_bit(i, info->map); continue; } @@ -701,7 +701,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, ret = do_shrink_slab(&sc, shrinker, priority); if (ret == SHRINK_EMPTY) { - clear_bit(i, map->map); + clear_bit(i, info->map); /* * After the shrinker reported that it had no objects to * free, but before we cleared the corresponding bit in From patchwork Tue Feb 9 17:46:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12078699 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF8DDC433E0 for ; Tue, 9 Feb 2021 17:55:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2E8A064EC0 for ; Tue, 9 Feb 2021 17:55:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2E8A064EC0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id ADFBC6B006C; Tue, 9 Feb 2021 12:55:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AB6496B0071; Tue, 9 Feb 2021 12:55:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9CC6D6B0072; Tue, 9 Feb 2021 12:55:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0138.hostedemail.com [216.40.44.138]) by kanga.kvack.org (Postfix) with ESMTP id 876146B006C for ; Tue, 9 Feb 2021 12:55:04 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 53C9A18484A07 for ; Tue, 9 Feb 2021 17:55:04 +0000 (UTC) X-FDA: 77799480528.05.step50_2014d7a27609 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id 399FC1803F9CD for ; Tue, 9 Feb 2021 17:55:04 +0000 (UTC) X-HE-Tag: step50_2014d7a27609 X-Filterd-Recvd-Size: 5300 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) by imf37.hostedemail.com (Postfix) with ESMTP for ; Tue, 9 Feb 2021 17:55:03 +0000 (UTC) Received: by mail-pj1-f45.google.com with SMTP id t2so2118102pjq.2 for ; Tue, 09 Feb 2021 09:55:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LtIT/XRubfZjNifW2/Wx2dvhH4DsCPssoQQ3vaxaswY=; b=f3P7M8G1BnqPYIthcB/2g5SI/aoPY7wzYvH+tH6KFpKU5R0eDTwdhWH4K18kqBhQBm XbCd824CFY/j+gg3m2xEUdnUS5o6nQz6BWqdkgk3U9hKkri6rEZgztlxI2/o1pFw4QEV F1dkb13u/1T1/e3Zn4VRnoVyrGu40vkPWClywRVJWbo1dmwoh3pb5CjJLQ7tBGD9OaXG fcm+jIdzMevEaJ1fiTUGdiyu1euBYSgbAo9WOlqfqiDfrXkiJY1EH/7sLk04XoZcyVek LAUq4qdKGqd53x2AQKGySFtwoAsFh/Kuoy89afStR91jCBE5MGcOcO96HLpOVVsJUr+e NBeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LtIT/XRubfZjNifW2/Wx2dvhH4DsCPssoQQ3vaxaswY=; b=Zyxkne0yoMVhewAmrJNl+R52iWN0fwdnGjoV4e6jV6vd1hVoZWp6E5Y2KSxm6HoUGw DyLzFw55xLPV7P21pxRykLU69e2wpc5kzPZGqSWrpT56Eq1PfMVrd/tBwMtp9Se4Gu91 BIPnpyOAuVB2Akt/C0TN/yIHkEXce8Xg6R3aYl0zveuS+YUvyr2WvqNynlxKCaAWzv4w R8SLzkTrxIQZd71yCwfRjWJDzl+5VfXzJOHGoA4lcnEAQumv+QdSaotZMCsU7qQJz6AA rxKqYvihdWbjJDszRguL4sdZKYwTJvpAnYSI23pFg+rOYdLpuYY52fHfTsTorAc4mMM8 QmUA== X-Gm-Message-State: AOAM532HTrevtx2969o3iKpGJRGfwENAppx/rwH2LEQZwvNt+woPSFeT snuq7b45bl8pX7lQAKi6b08UTXN9lTcilw== X-Google-Smtp-Source: ABdhPJwn2hQAiH23fVtXQpzqorHAVfhxl9TB10b0gI+9q594PYZIw32xabvZJWQNOW1EMGNWJo+64A== X-Received: by 2002:a17:902:c404:b029:e2:cb8e:6b78 with SMTP id k4-20020a170902c404b02900e2cb8e6b78mr13868552plk.3.1612892837299; Tue, 09 Feb 2021 09:47:17 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id j1sm22260929pfr.78.2021.02.09.09.47.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Feb 2021 09:47:16 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v7 PATCH 06/12] mm: vmscan: add shrinker_info_protected() helper Date: Tue, 9 Feb 2021 09:46:40 -0800 Message-Id: <20210209174646.1310591-7-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210209174646.1310591-1-shy828301@gmail.com> References: <20210209174646.1310591-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The shrinker_info is dereferenced in a couple of places via rcu_dereference_protected with different calling conventions, for example, using mem_cgroup_nodeinfo helper or dereferencing memcg->nodeinfo[nid]->shrinker_info. And the later patch will add more dereference places. So extract the dereference into a helper to make the code more readable. No functional change. Signed-off-by: Yang Shi Acked-by: Roman Gushchin Acked-by: Kirill Tkhai Acked-by: Vlastimil Babka Reported-by: kernel test robot --- mm/vmscan.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 9436f9246d32..273efbf4d53c 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -190,6 +190,13 @@ static int shrinker_nr_max; #define NR_MAX_TO_SHR_MAP_SIZE(nr_max) \ (DIV_ROUND_UP(nr_max, BITS_PER_LONG) * sizeof(unsigned long)) +static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, + int nid) +{ + return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, + lockdep_is_held(&shrinker_rwsem)); +} + static void free_shrinker_info_rcu(struct rcu_head *head) { kvfree(container_of(head, struct shrinker_info, rcu)); @@ -202,8 +209,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, int nid; for_each_node(nid) { - old = rcu_dereference_protected( - mem_cgroup_nodeinfo(memcg, nid)->shrinker_info, true); + old = shrinker_info_protected(memcg, nid); /* Not yet online memcg */ if (!old) return 0; @@ -234,7 +240,7 @@ void free_shrinker_info(struct mem_cgroup *memcg) for_each_node(nid) { pn = mem_cgroup_nodeinfo(memcg, nid); - info = rcu_dereference_protected(pn->shrinker_info, true); + info = shrinker_info_protected(memcg, nid); kvfree(info); rcu_assign_pointer(pn->shrinker_info, NULL); } @@ -674,8 +680,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (!down_read_trylock(&shrinker_rwsem)) return 0; - info = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, - true); + info = shrinker_info_protected(memcg, nid); if (unlikely(!info)) goto unlock; From patchwork Tue Feb 9 17:46:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12078683 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC7BAC433E6 for ; Tue, 9 Feb 2021 17:47:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6CCA364EB6 for ; Tue, 9 Feb 2021 17:47:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6CCA364EB6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EEAE36B0071; Tue, 9 Feb 2021 12:47:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E766F6B0072; Tue, 9 Feb 2021 12:47:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BE2326B0073; Tue, 9 Feb 2021 12:47:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7D8F36B0071 for ; Tue, 9 Feb 2021 12:47:27 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 3645D10F1A for ; Tue, 9 Feb 2021 17:47:21 +0000 (UTC) X-FDA: 77799461082.25.spoon02_5102b3627609 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id D476A1802E8AD for ; Tue, 9 Feb 2021 17:47:20 +0000 (UTC) X-HE-Tag: spoon02_5102b3627609 X-Filterd-Recvd-Size: 7304 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf14.hostedemail.com (Postfix) with ESMTP for ; Tue, 9 Feb 2021 17:47:20 +0000 (UTC) Received: by mail-pl1-f171.google.com with SMTP id k22so5782055pll.6 for ; Tue, 09 Feb 2021 09:47:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NF6HThwQUpqhcxPrBlYzwJ+cqS/FPyEXXyhktWrJklc=; b=T25wb3ff0wno0w5hs/bushE7ZWa46bofCJTkQrTEctOaXfljMeIg4rk08r/Gf82LnK Oq6pBXT1NBenLp59ywUUlnHMQeD8wEnqUPeIsS2fD4ukwJpUFO+YFjlHQrsP+l8eNHE1 tjq5FS0ZAGfDK3mAgLsgUdrq/pcJMicB716nhoXTMufgMop3hVZis+ASnY93sI6qTdcr cSG7qRwkYmDGSf/fSWTmXXEB7i5FtYxSR+Qku19/pMaT8nS5HQs9KNkVWcuQE0MR210O iYwXE7yKlYIumUI0NpL/Z0k5wtdO89W0//D6EQcOBbmjnMMsgqnuhVKImYrIdC9DMFex +oQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NF6HThwQUpqhcxPrBlYzwJ+cqS/FPyEXXyhktWrJklc=; b=qtflz9hErokcimawPVqUutrs2NO/+Y0PLtL7nEU1LyHirnvA7HvnnAqxKVFISUh8fj ZpPhUchBkMY6HtmcbWHJNv8XFROEuxDpcKl9e7sSIENlj7iFsGnRupwtP52+FxUFSrQC ksMolc5rBC+nsPZdNys2XO6QNrLeCDIOU8ejd6IyBrDafH7MsC1skN6jS1C3TSXGi3Pa mJ79EY+S/o7JbxuuBYf75LkuOVi+Agqe3BfIKtVgfCACjuoFTUFeM+ECb38qRrqcFCr8 LOQ9TZ5zJTJlCoyCr6rT5LCEvfIq90dDN9LIoFesRXmpfLWyg3uhs6Xp4uuyzFTICg7j IEnA== X-Gm-Message-State: AOAM531FUX+78ao7mOh9nsrhSAdz2EVZhpK4yh1mGB3HGeBJ04LEzM9F YS7t8u17mqNt1klIND7xrog= X-Google-Smtp-Source: ABdhPJye6RRH94+CGlvBo7IaC/UfHgjz9ba5WHxA0haMxMB24SUnJVINCvkrXRLTHgG+sGs23jEyVw== X-Received: by 2002:a17:902:ee44:b029:e2:bb4a:9ffb with SMTP id 4-20020a170902ee44b02900e2bb4a9ffbmr18582560plo.39.1612892839666; Tue, 09 Feb 2021 09:47:19 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id j1sm22260929pfr.78.2021.02.09.09.47.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Feb 2021 09:47:18 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v7 PATCH 07/12] mm: vmscan: use a new flag to indicate shrinker is registered Date: Tue, 9 Feb 2021 09:46:41 -0800 Message-Id: <20210209174646.1310591-8-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210209174646.1310591-1-shy828301@gmail.com> References: <20210209174646.1310591-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently registered shrinker is indicated by non-NULL shrinker->nr_deferred. This approach is fine with nr_deferred at the shrinker level, but the following patches will move MEMCG_AWARE shrinkers' nr_deferred to memcg level, so their shrinker->nr_deferred would always be NULL. This would prevent the shrinkers from unregistering correctly. Remove SHRINKER_REGISTERING since we could check if shrinker is registered successfully by the new flag. Acked-by: Kirill Tkhai Signed-off-by: Yang Shi Acked-by: Vlastimil Babka --- include/linux/shrinker.h | 7 ++++--- mm/vmscan.c | 31 +++++++++---------------------- 2 files changed, 13 insertions(+), 25 deletions(-) diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h index 0f80123650e2..1eac79ce57d4 100644 --- a/include/linux/shrinker.h +++ b/include/linux/shrinker.h @@ -79,13 +79,14 @@ struct shrinker { #define DEFAULT_SEEKS 2 /* A good number if you don't know better. */ /* Flags */ -#define SHRINKER_NUMA_AWARE (1 << 0) -#define SHRINKER_MEMCG_AWARE (1 << 1) +#define SHRINKER_REGISTERED (1 << 0) +#define SHRINKER_NUMA_AWARE (1 << 1) +#define SHRINKER_MEMCG_AWARE (1 << 2) /* * It just makes sense when the shrinker is also MEMCG_AWARE for now, * non-MEMCG_AWARE shrinker should not have this flag set. */ -#define SHRINKER_NONSLAB (1 << 2) +#define SHRINKER_NONSLAB (1 << 3) extern int prealloc_shrinker(struct shrinker *shrinker); extern void register_shrinker_prepared(struct shrinker *shrinker); diff --git a/mm/vmscan.c b/mm/vmscan.c index 273efbf4d53c..a047980536cf 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -315,19 +315,6 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) } } -/* - * We allow subsystems to populate their shrinker-related - * LRU lists before register_shrinker_prepared() is called - * for the shrinker, since we don't want to impose - * restrictions on their internal registration order. - * In this case shrink_slab_memcg() may find corresponding - * bit is set in the shrinkers map. - * - * This value is used by the function to detect registering - * shrinkers and to skip do_shrink_slab() calls for them. - */ -#define SHRINKER_REGISTERING ((struct shrinker *)~0UL) - static DEFINE_IDR(shrinker_idr); static int prealloc_memcg_shrinker(struct shrinker *shrinker) @@ -336,7 +323,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) down_write(&shrinker_rwsem); /* This may call shrinker, so it must use down_read_trylock() */ - id = idr_alloc(&shrinker_idr, SHRINKER_REGISTERING, 0, 0, GFP_KERNEL); + id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -499,10 +486,7 @@ void register_shrinker_prepared(struct shrinker *shrinker) { down_write(&shrinker_rwsem); list_add_tail(&shrinker->list, &shrinker_list); -#ifdef CONFIG_MEMCG - if (shrinker->flags & SHRINKER_MEMCG_AWARE) - idr_replace(&shrinker_idr, shrinker, shrinker->id); -#endif + shrinker->flags |= SHRINKER_REGISTERED; up_write(&shrinker_rwsem); } @@ -522,13 +506,16 @@ EXPORT_SYMBOL(register_shrinker); */ void unregister_shrinker(struct shrinker *shrinker) { - if (!shrinker->nr_deferred) + if (!(shrinker->flags & SHRINKER_REGISTERED)) return; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) - unregister_memcg_shrinker(shrinker); + down_write(&shrinker_rwsem); list_del(&shrinker->list); + shrinker->flags &= ~SHRINKER_REGISTERED; up_write(&shrinker_rwsem); + + if (shrinker->flags & SHRINKER_MEMCG_AWARE) + unregister_memcg_shrinker(shrinker); kfree(shrinker->nr_deferred); shrinker->nr_deferred = NULL; } @@ -693,7 +680,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, struct shrinker *shrinker; shrinker = idr_find(&shrinker_idr, i); - if (unlikely(!shrinker || shrinker == SHRINKER_REGISTERING)) { + if (unlikely(!shrinker || !(shrinker->flags & SHRINKER_REGISTERED))) { if (!shrinker) clear_bit(i, info->map); continue; From patchwork Tue Feb 9 17:46:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12078689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5A8FC433E0 for ; Tue, 9 Feb 2021 17:47:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 32D1264E37 for ; Tue, 9 Feb 2021 17:47:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 32D1264E37 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BB7946B0074; Tue, 9 Feb 2021 12:47:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B679C6B0075; Tue, 9 Feb 2021 12:47:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A309C8D0001; Tue, 9 Feb 2021 12:47:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0090.hostedemail.com [216.40.44.90]) by kanga.kvack.org (Postfix) with ESMTP id 8A1666B0074 for ; Tue, 9 Feb 2021 12:47:35 -0500 (EST) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 57B7B824556B for ; Tue, 9 Feb 2021 17:47:35 +0000 (UTC) X-FDA: 77799461670.20.stone51_1e176da27609 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id 68FA118294F12 for ; Tue, 9 Feb 2021 17:47:23 +0000 (UTC) X-HE-Tag: stone51_1e176da27609 X-Filterd-Recvd-Size: 9840 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by imf42.hostedemail.com (Postfix) with ESMTP for ; Tue, 9 Feb 2021 17:47:22 +0000 (UTC) Received: by mail-pj1-f51.google.com with SMTP id lw17so1816273pjb.0 for ; Tue, 09 Feb 2021 09:47:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Kvzc/fCpMbH2LLBrDe6dDllXNRKftSCiNU4YaBKuDgQ=; b=NI1m2W7cVpizXt/YOZ3aDRsQVWr4IYX5p9aD6+6+XgUoIYzb6GW6FioLXQ05nlLODN m4SgoGE924CgLaMOXVFTEv8jqYxyK763M3IambPUZEIzZ/whMXhcOBYwotyFA42Ato5s gxWXCs/rEq5fLXZpphOmpMor+mo1J++/G2OT/9MtNAaGHEDRUXXXxEHYg9EdyppMpfgL Nj5Z9QLYxsA8Q+41bU1QIPsT1vn28e97Namh2dh/auEu/Z9K6zgNWo/MW8x0LCO+XMbv 0M6qz6IOpYlTXWVjRmtbx5ee29GEjPpMnFWF9nKAfotl55jUVnbDAOShYJ1pml5Ik2H8 e/6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Kvzc/fCpMbH2LLBrDe6dDllXNRKftSCiNU4YaBKuDgQ=; b=ft8l1QYeNREvI9WPUybdQNZMOBv/AWMzHcb8cj2ocpFhowQoK3YCnuyc9N6vrqJnP9 HFPAg25051tR7l629DCDuMQzD7pZA5wGUfx0KE0YtlhJXMnen9bSAyWAl0bjYgbVyN29 La6MANb4hNpqCPb+uJJfPbicpguSZ6GhOdfthot7PN5AT7jkEdEiCbERPkl0fDXMVllQ ZaygZLnUgtAZugOBsuFB9aZKuyejIXPCjv4Do1kFZCG7zmzf8W2RUXa4YdUGFpuG2MFo /93PY1BW9OttvUhM68FLA1y7eOJBEWHV1+5kaDAWJX2i2PbEUZx2JERRDv57Ffqm0RnN w4ig== X-Gm-Message-State: AOAM532TfSxAoTs6Tet5GsL359Ure9KQw4/t2CrlJ4zXlLe2sNuQdlJS RuD3JzrGMKkvvR0nWZ1lFQo= X-Google-Smtp-Source: ABdhPJzihVj/26Ztz6MC1D4xaprmi29GfhKEOFTBPgwhSVN8uJmhzX5xRx6MPxMnkXCcAFzo5Meb8w== X-Received: by 2002:a17:90a:3e01:: with SMTP id j1mr5093096pjc.138.1612892842219; Tue, 09 Feb 2021 09:47:22 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id j1sm22260929pfr.78.2021.02.09.09.47.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Feb 2021 09:47:21 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v7 PATCH 08/12] mm: vmscan: add per memcg shrinker nr_deferred Date: Tue, 9 Feb 2021 09:46:42 -0800 Message-Id: <20210209174646.1310591-9-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210209174646.1310591-1-shy828301@gmail.com> References: <20210209174646.1310591-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently the number of deferred objects are per shrinker, but some slabs, for example, vfs inode/dentry cache are per memcg, this would result in poor isolation among memcgs. The deferred objects typically are generated by __GFP_NOFS allocations, one memcg with excessive __GFP_NOFS allocations may blow up deferred objects, then other innocent memcgs may suffer from over shrink, excessive reclaim latency, etc. For example, two workloads run in memcgA and memcgB respectively, workload in B is vfs heavy workload. Workload in A generates excessive deferred objects, then B's vfs cache might be hit heavily (drop half of caches) by B's limit reclaim or global reclaim. We observed this hit in our production environment which was running vfs heavy workload shown as the below tracing log: <...>-409454 [016] .... 28286961.747146: mm_shrink_slab_start: super_cache_scan+0x0/0x1a0 ffff9a83046f3458: nid: 1 objects to shrink 3641681686040 gfp_flags GFP_HIGHUSER_MOVABLE|__GFP_ZERO pgs_scanned 1 lru_pgs 15721 cache items 246404277 delta 31345 total_scan 123202138 <...>-409454 [022] .... 28287105.928018: mm_shrink_slab_end: super_cache_scan+0x0/0x1a0 ffff9a83046f3458: nid: 1 unused scan count 3641681686040 new scan count 3641798379189 total_scan 602 last shrinker return val 123186855 The vfs cache and page cache ratio was 10:1 on this machine, and half of caches were dropped. This also resulted in significant amount of page caches were dropped due to inodes eviction. Make nr_deferred per memcg for memcg aware shrinkers would solve the unfairness and bring better isolation. When memcg is not enabled (!CONFIG_MEMCG or memcg disabled), the shrinker's nr_deferred would be used. And non memcg aware shrinkers use shrinker's nr_deferred all the time. Signed-off-by: Yang Shi --- include/linux/memcontrol.h | 7 +++--- mm/vmscan.c | 49 +++++++++++++++++++++++++------------- 2 files changed, 37 insertions(+), 19 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 4c9253896e25..c457fc7bc631 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -93,12 +93,13 @@ struct lruvec_stat { }; /* - * Bitmap of shrinker::id corresponding to memcg-aware shrinkers, - * which have elements charged to this memcg. + * Bitmap and deferred work of shrinker::id corresponding to memcg-aware + * shrinkers, which have elements charged to this memcg. */ struct shrinker_info { struct rcu_head rcu; - unsigned long map[]; + atomic_long_t *nr_deferred; + unsigned long *map; }; /* diff --git a/mm/vmscan.c b/mm/vmscan.c index a047980536cf..d4b030a0b2a9 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -187,9 +187,13 @@ static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG static int shrinker_nr_max; +/* The shrinker_info is expanded in a batch of BITS_PER_LONG */ #define NR_MAX_TO_SHR_MAP_SIZE(nr_max) \ (DIV_ROUND_UP(nr_max, BITS_PER_LONG) * sizeof(unsigned long)) +#define NR_MAX_TO_SHR_DEF_SIZE(nr_max) \ + (round_up(nr_max, BITS_PER_LONG) * sizeof(atomic_long_t)) + static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, int nid) { @@ -203,10 +207,12 @@ static void free_shrinker_info_rcu(struct rcu_head *head) } static int expand_one_shrinker_info(struct mem_cgroup *memcg, - int size, int old_size) + int m_size, int d_size, + int old_m_size, int old_d_size) { struct shrinker_info *new, *old; int nid; + int size = m_size + d_size; for_each_node(nid) { old = shrinker_info_protected(memcg, nid); @@ -218,9 +224,15 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, if (!new) return -ENOMEM; - /* Set all old bits, clear all new bits */ - memset(new->map, (int)0xff, old_size); - memset((void *)new->map + old_size, 0, size - old_size); + new->nr_deferred = (atomic_long_t *)(new + 1); + new->map = (void *)new->nr_deferred + d_size; + + /* map: set all old bits, clear all new bits */ + memset(new->map, (int)0xff, old_m_size); + memset((void *)new->map + old_m_size, 0, m_size - old_m_size); + /* nr_deferred: copy old values, clear all new values */ + memcpy(new->nr_deferred, old->nr_deferred, old_d_size); + memset((void *)new->nr_deferred + old_d_size, 0, d_size - old_d_size); rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, new); call_rcu(&old->rcu, free_shrinker_info_rcu); @@ -235,9 +247,6 @@ void free_shrinker_info(struct mem_cgroup *memcg) struct shrinker_info *info; int nid; - if (mem_cgroup_is_root(memcg)) - return; - for_each_node(nid) { pn = mem_cgroup_nodeinfo(memcg, nid); info = shrinker_info_protected(memcg, nid); @@ -250,12 +259,13 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) { struct shrinker_info *info; int nid, size, ret = 0; - - if (mem_cgroup_is_root(memcg)) - return 0; + int m_size, d_size = 0; down_write(&shrinker_rwsem); - size = NR_MAX_TO_SHR_MAP_SIZE(shrinker_nr_max); + m_size = NR_MAX_TO_SHR_MAP_SIZE(shrinker_nr_max); + d_size = NR_MAX_TO_SHR_DEF_SIZE(shrinker_nr_max); + size = m_size + d_size; + for_each_node(nid) { info = kvzalloc_node(sizeof(*info) + size, GFP_KERNEL, nid); if (!info) { @@ -263,6 +273,8 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) ret = -ENOMEM; break; } + info->nr_deferred = (atomic_long_t *)(info + 1); + info->map = (void *)info->nr_deferred + d_size; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); @@ -274,10 +286,16 @@ static int expand_shrinker_info(int new_id) { int size, old_size, ret = 0; int new_nr_max = new_id + 1; + int m_size, d_size = 0; + int old_m_size, old_d_size = 0; struct mem_cgroup *memcg; - size = NR_MAX_TO_SHR_MAP_SIZE(new_nr_max); - old_size = NR_MAX_TO_SHR_MAP_SIZE(shrinker_nr_max); + m_size = NR_MAX_TO_SHR_MAP_SIZE(new_nr_max); + d_size = NR_MAX_TO_SHR_DEF_SIZE(new_nr_max); + size = m_size + d_size; + old_m_size = NR_MAX_TO_SHR_MAP_SIZE(shrinker_nr_max); + old_d_size = NR_MAX_TO_SHR_DEF_SIZE(shrinker_nr_max); + old_size = old_m_size + old_d_size; if (size <= old_size) goto out; @@ -286,9 +304,8 @@ static int expand_shrinker_info(int new_id) memcg = mem_cgroup_iter(NULL, NULL, NULL); do { - if (mem_cgroup_is_root(memcg)) - continue; - ret = expand_one_shrinker_info(memcg, size, old_size); + ret = expand_one_shrinker_info(memcg, m_size, d_size, + old_m_size, old_d_size); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; From patchwork Tue Feb 9 17:46:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12078681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 773FFC433DB for ; Tue, 9 Feb 2021 17:47:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0DD3164E37 for ; Tue, 9 Feb 2021 17:47:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0DD3164E37 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9E1D96B0070; Tue, 9 Feb 2021 12:47:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9914A6B0071; Tue, 9 Feb 2021 12:47:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 881BF6B0072; Tue, 9 Feb 2021 12:47:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0098.hostedemail.com [216.40.44.98]) by kanga.kvack.org (Postfix) with ESMTP id 6CC616B0070 for ; Tue, 9 Feb 2021 12:47:26 -0500 (EST) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 3445B186117F1 for ; Tue, 9 Feb 2021 17:47:26 +0000 (UTC) X-FDA: 77799461292.14.stage58_3610eff27609 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id 130EF1826F486 for ; Tue, 9 Feb 2021 17:47:26 +0000 (UTC) X-HE-Tag: stage58_3610eff27609 X-Filterd-Recvd-Size: 7474 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Tue, 9 Feb 2021 17:47:25 +0000 (UTC) Received: by mail-pl1-f181.google.com with SMTP id x9so10161538plb.5 for ; Tue, 09 Feb 2021 09:47:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gLHMO0CJlDGDLfBj7qUuWKf63jRhI9bAFy0uZYnSIgU=; b=iLutBmpw9q9DSfaNwTjqN17qLywyO9FvLxIpnRaQ/yPDHlc/FJfxq/D3q6Uh1uNd09 UgWHri/Fa1exto/A08UNGi1MQj0V037ASVjoYKgaja/cyLEdjWuYyC4bBZIXwQ2qGBoe a2KUM2IO5ZncBQNRTNqNcrkt8gyfro++XD3B76BYTp3/yPZ1drQ9ZWwXObYsIKexa7vT K5WPUDPiWvQU4yQd6kXTC+pW8GmJEJKwZ3hnTH6Fls8d6tY5fCshtlWb+75eZuVDOZyn /1h0jJqZCRxfqhIP1MH7OB8Q4ND25rnwElP+wCjZ9sHY+j/XQHZQLhtQ+w+nRUVb4rLF sYYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gLHMO0CJlDGDLfBj7qUuWKf63jRhI9bAFy0uZYnSIgU=; b=Fw7dyXTMNrdDm0v56xXec0a5rpw4sdO0/3LSm4eqnBcIlJPYGQuCjxOOPyPuk0+oIG T+hP1FmbzNasIsR5hBPkcgu2/gxtpCnGiGOI+bReT/KJ141tjbagAnsIdVpv94JY4uZh 9Uq39EGDYlOlPB9BuW09sDqb6FxF0Y0wkEsUBsX7iCtdzF4zgY9Edt+qFXVyCsF4c1ea 1S4OAfsTwDvitT4KFLYFhwuaCSsZvp4MKCJtVwDpSnX3LIw2/UHUvSnIgwnT83N3Tkv+ D9gFuXL2jcX3GBS4+ZGbnjWM/rObEMPLzYAD2UhThFEIZeFQcv80ssmOUp7o6mnh5LlN Bg1w== X-Gm-Message-State: AOAM533c1AMbh1uN1nFc9PQbU8mOUXCy2xqPLngOQxvOumLSI6gs7pME lWyiKMR4zF1EUmB7/tsiJsY= X-Google-Smtp-Source: ABdhPJzLr+e2VbMOyew26Jy1J13A+S62eVR/KNdvufscmPTbomzdCkSx5wFdCzpD0Ky2wyEMrbpB4A== X-Received: by 2002:a17:90a:a22:: with SMTP id o31mr4925486pjo.221.1612892844684; Tue, 09 Feb 2021 09:47:24 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id j1sm22260929pfr.78.2021.02.09.09.47.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Feb 2021 09:47:23 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v7 PATCH 09/12] mm: vmscan: use per memcg nr_deferred of shrinker Date: Tue, 9 Feb 2021 09:46:43 -0800 Message-Id: <20210209174646.1310591-10-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210209174646.1310591-1-shy828301@gmail.com> References: <20210209174646.1310591-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use per memcg's nr_deferred for memcg aware shrinkers. The shrinker's nr_deferred will be used in the following cases: 1. Non memcg aware shrinkers 2. !CONFIG_MEMCG 3. memcg is disabled by boot parameter Signed-off-by: Yang Shi --- mm/vmscan.c | 78 ++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 66 insertions(+), 12 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index d4b030a0b2a9..748aa6e90f83 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -368,6 +368,24 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) up_write(&shrinker_rwsem); } +static long count_nr_deferred_memcg(int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + struct shrinker_info *info; + + info = shrinker_info_protected(memcg, nid); + return atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); +} + +static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + struct shrinker_info *info; + + info = shrinker_info_protected(memcg, nid); + return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); +} + static bool cgroup_reclaim(struct scan_control *sc) { return sc->target_mem_cgroup; @@ -406,6 +424,18 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) { } +static long count_nr_deferred_memcg(int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + return 0; +} + +static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + return 0; +} + static bool cgroup_reclaim(struct scan_control *sc) { return false; @@ -417,6 +447,39 @@ static bool writeback_throttling_sane(struct scan_control *sc) } #endif +static long count_nr_deferred(struct shrinker *shrinker, + struct shrink_control *sc) +{ + int nid = sc->nid; + + if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) + nid = 0; + + if (sc->memcg && + (shrinker->flags & SHRINKER_MEMCG_AWARE)) + return count_nr_deferred_memcg(nid, shrinker, + sc->memcg); + + return atomic_long_xchg(&shrinker->nr_deferred[nid], 0); +} + + +static long add_nr_deferred(long nr, struct shrinker *shrinker, + struct shrink_control *sc) +{ + int nid = sc->nid; + + if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) + nid = 0; + + if (sc->memcg && + (shrinker->flags & SHRINKER_MEMCG_AWARE)) + return add_nr_deferred_memcg(nr, nid, shrinker, + sc->memcg); + + return atomic_long_add_return(nr, &shrinker->nr_deferred[nid]); +} + /* * This misses isolated pages which are not accounted for to save counters. * As the data only determines if reclaim or compaction continues, it is @@ -549,14 +612,10 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, long freeable; long nr; long new_nr; - int nid = shrinkctl->nid; long batch_size = shrinker->batch ? shrinker->batch : SHRINK_BATCH; long scanned = 0, next_deferred; - if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) - nid = 0; - freeable = shrinker->count_objects(shrinker, shrinkctl); if (freeable == 0 || freeable == SHRINK_EMPTY) return freeable; @@ -566,7 +625,7 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, * and zero it so that other concurrent shrinker invocations * don't also do this scanning work. */ - nr = atomic_long_xchg(&shrinker->nr_deferred[nid], 0); + nr = count_nr_deferred(shrinker, shrinkctl); total_scan = nr; if (shrinker->seeks) { @@ -657,14 +716,9 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, next_deferred = 0; /* * move the unused scan count back into the shrinker in a - * manner that handles concurrent updates. If we exhausted the - * scan, there is no need to do an update. + * manner that handles concurrent updates. */ - if (next_deferred > 0) - new_nr = atomic_long_add_return(next_deferred, - &shrinker->nr_deferred[nid]); - else - new_nr = atomic_long_read(&shrinker->nr_deferred[nid]); + new_nr = add_nr_deferred(next_deferred, shrinker, shrinkctl); trace_mm_shrink_slab_end(shrinker, shrinkctl->nid, freed, nr, new_nr, total_scan); return freed; From patchwork Tue Feb 9 17:46:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12078685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BC28C433E0 for ; Tue, 9 Feb 2021 17:47:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A4B5164EB6 for ; Tue, 9 Feb 2021 17:47:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A4B5164EB6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 308F96B0072; Tue, 9 Feb 2021 12:47:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 26B026B0073; Tue, 9 Feb 2021 12:47:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 06DD06B0074; Tue, 9 Feb 2021 12:47:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0198.hostedemail.com [216.40.44.198]) by kanga.kvack.org (Postfix) with ESMTP id E62746B0072 for ; Tue, 9 Feb 2021 12:47:29 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id A5A8618015CCA for ; Tue, 9 Feb 2021 17:47:29 +0000 (UTC) X-FDA: 77799461418.25.tramp06_3804a3527609 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id 6C7D818583370 for ; Tue, 9 Feb 2021 17:47:29 +0000 (UTC) X-HE-Tag: tramp06_3804a3527609 X-Filterd-Recvd-Size: 5819 Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) by imf25.hostedemail.com (Postfix) with ESMTP for ; Tue, 9 Feb 2021 17:47:28 +0000 (UTC) Received: by mail-pg1-f169.google.com with SMTP id j5so4129166pgb.11 for ; Tue, 09 Feb 2021 09:47:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AX6V0BvcTn1weE8/ZJGdvy6BSiCFsYPbqmrEZf2HeZY=; b=W6NXL6IpNXDaVEQv0X3zKJCU+Kq98htBiQpelM2gUTQC23iqlOjgk6/gYz7AMZT+Yf w1nA0oEMpjHsgmyphYdVEsYcfSJzQHy9gL0Cu15+YcUWyEtNSIYBbUQSWSjWTtvkWfWT GX62nAW5Xioiot59VRDfH4/zt64d42Su0yrsC8WtgpWrbdM8RfE7KBJRhY1U/PK5siSF 7ftHsRhzoUEUythqNJ7tpo1ldvn6tsT4XIBI7MajdCqHageGD95Uk639qEjbisuiWWTy Bhfvl8lfacdTm5CojxrlCIi9BXV0Lqzw8lXrebFQPctNYnotIqsSgcfGB6VCgqy0Ssxp DRfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AX6V0BvcTn1weE8/ZJGdvy6BSiCFsYPbqmrEZf2HeZY=; b=iIpv8cdtNYEAPaspg6I5qnujz8caVq+d0g0+s3XUbNRMWluV7TV35M4gZx9oYnsCcB 7kPl2tRxfQKnFGAgnzwcPnmnouZqywuiPIMbIhb2JlgvYLeFgwjEkmagAWBrJ8qc5IoQ Ck/8PG6m9waJ5EeW6zKrpVaJBRSRbs362ADplKINx0OZlbGSbUJonHDQVnw3YZ1oFu6Y nmjP9V5/GsIbmT15sY24V1UrNP2p40xEU+TO601LupLvAhMrLRFJn6chtapPC4yCs9EW bGZO/gFK5llSdXF2magiA8NqC0vUVVeq84rUvB4NjkNxT9hj63mwfwaq9fsirXSt5gqU qVDA== X-Gm-Message-State: AOAM531L8ZtjubkDvboiuxLAzAz2Bs76hSx206vg7Zm3scMLTcj4axe0 4yTdBhu5Fr5oexLoKfKR5Ng= X-Google-Smtp-Source: ABdhPJylfkR4RNuMaBkK1DV4MxLb32qE/adS2oaKsPC5eDZCdrkTWopj1y1vlm7v79AqsCXNLunWhw== X-Received: by 2002:aa7:8485:0:b029:1dd:bf6a:a9ec with SMTP id u5-20020aa784850000b02901ddbf6aa9ecmr9988270pfn.47.1612892847324; Tue, 09 Feb 2021 09:47:27 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id j1sm22260929pfr.78.2021.02.09.09.47.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Feb 2021 09:47:25 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v7 PATCH 10/12] mm: vmscan: don't need allocate shrinker->nr_deferred for memcg aware shrinkers Date: Tue, 9 Feb 2021 09:46:44 -0800 Message-Id: <20210209174646.1310591-11-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210209174646.1310591-1-shy828301@gmail.com> References: <20210209174646.1310591-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now nr_deferred is available on per memcg level for memcg aware shrinkers, so don't need allocate shrinker->nr_deferred for such shrinkers anymore. The prealloc_memcg_shrinker() would return -ENOSYS if !CONFIG_MEMCG or memcg is disabled by kernel command line, then shrinker's SHRINKER_MEMCG_AWARE flag would be cleared. This makes the implementation of this patch simpler. Acked-by: Vlastimil Babka Reviewed-by: Kirill Tkhai Signed-off-by: Yang Shi Acked-by: Roman Gushchin --- mm/vmscan.c | 33 ++++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 748aa6e90f83..dfde6e7fd7f5 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -338,6 +338,9 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) { int id, ret = -ENOMEM; + if (mem_cgroup_disabled()) + return -ENOSYS; + down_write(&shrinker_rwsem); /* This may call shrinker, so it must use down_read_trylock() */ id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); @@ -417,7 +420,7 @@ static bool writeback_throttling_sane(struct scan_control *sc) #else static int prealloc_memcg_shrinker(struct shrinker *shrinker) { - return 0; + return -ENOSYS; } static void unregister_memcg_shrinker(struct shrinker *shrinker) @@ -528,8 +531,18 @@ unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru, int zone */ int prealloc_shrinker(struct shrinker *shrinker) { - unsigned int size = sizeof(*shrinker->nr_deferred); + unsigned int size; + int err; + if (shrinker->flags & SHRINKER_MEMCG_AWARE) { + err = prealloc_memcg_shrinker(shrinker); + if (err != -ENOSYS) + return err; + + shrinker->flags &= ~SHRINKER_MEMCG_AWARE; + } + + size = sizeof(*shrinker->nr_deferred); if (shrinker->flags & SHRINKER_NUMA_AWARE) size *= nr_node_ids; @@ -537,26 +550,16 @@ int prealloc_shrinker(struct shrinker *shrinker) if (!shrinker->nr_deferred) return -ENOMEM; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - if (prealloc_memcg_shrinker(shrinker)) - goto free_deferred; - } return 0; - -free_deferred: - kfree(shrinker->nr_deferred); - shrinker->nr_deferred = NULL; - return -ENOMEM; } void free_prealloced_shrinker(struct shrinker *shrinker) { - if (!shrinker->nr_deferred) - return; - - if (shrinker->flags & SHRINKER_MEMCG_AWARE) + if (shrinker->flags & SHRINKER_MEMCG_AWARE) { unregister_memcg_shrinker(shrinker); + return; + } kfree(shrinker->nr_deferred); shrinker->nr_deferred = NULL; From patchwork Tue Feb 9 17:46:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12078687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10673C433E6 for ; Tue, 9 Feb 2021 17:47:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 94D1864EB8 for ; Tue, 9 Feb 2021 17:47:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 94D1864EB8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9E2A86B0073; Tue, 9 Feb 2021 12:47:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 96BC56B0074; Tue, 9 Feb 2021 12:47:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 834408D0001; Tue, 9 Feb 2021 12:47:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0035.hostedemail.com [216.40.44.35]) by kanga.kvack.org (Postfix) with ESMTP id 68F4B6B0073 for ; Tue, 9 Feb 2021 12:47:31 -0500 (EST) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 3661A1DF9 for ; Tue, 9 Feb 2021 17:47:31 +0000 (UTC) X-FDA: 77799461502.28.wound49_070f85127609 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id 10BF5F01D for ; Tue, 9 Feb 2021 17:47:31 +0000 (UTC) X-HE-Tag: wound49_070f85127609 X-Filterd-Recvd-Size: 5726 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by imf36.hostedemail.com (Postfix) with ESMTP for ; Tue, 9 Feb 2021 17:47:30 +0000 (UTC) Received: by mail-pg1-f177.google.com with SMTP id z21so12949077pgj.4 for ; Tue, 09 Feb 2021 09:47:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0vxSQstCm0rFqXMJSzTOrbGMjufuS8TCQJNnJKNC7TU=; b=kWhwJCn2HSplrJKmcFdUT8xE5+TrhsIvBVUakU3FxNw0V+tHRbkhT9QhFviWZpXngw O6YPg2xJbS+4A7+wI4BykkTC/dfls4ro1MyqlcWXEt7q+G83CsFdata9Mmc25DFWYEbi HyIGwQJUnNONeORegAXbOu0pWHI0/jWS0cEEFzSu30Nr0SftX+v2gBHde0oymvzY2Az9 Scfy1oEoB/gGsaxcDcKKiKleTcg1o+YfHeGd2wv/OX/ugFUI9/6xc8cepfppZsLWfj4i 3JsngITGO7n9ZlSdRCLEd6CUuCtvosTId5gZo1cxNtaM82gjTd/dQhC6trwkZerSrsdP 3GWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0vxSQstCm0rFqXMJSzTOrbGMjufuS8TCQJNnJKNC7TU=; b=supOeA9ygeJ5J4u2PtS7pd59lOB9i4bcoE0iDKXJNEkLjNpWVGAGc1veKquwtoBeBx JxvyV80JStKpeMH1vwAQJhRAtmMMorWWoAQnFPIckzaYRA2tI/pai0Sfy/Esz/NGctpD 7bx7mcJa97eoL+rPu1Daomq7yP4qoXKerOW0p4Gw0bt4e6ogUFnEuBlWBpCG21rkefvs tcw3UL3voURbIYIIuZOTr6V4S0fdnL6HAIdD3RP2DA+zk7Rz6eJ1+Q+xEy1ILS3vrIHh Zuj5ptazzzxWIVawj/B+2EHOOHy6hIDnB8lLDfbng43XO6SkjpY4rzBxoOXwHdgbv/9A vl2w== X-Gm-Message-State: AOAM532RV2UlbRuhImZ0LCRbxqviL2gEelbvVfmUtS/xluKUpGHQUmap zU3Cw8TpxDzuELso737eyaA= X-Google-Smtp-Source: ABdhPJyb1163li3Zilu1BpE4OXhPrvUfIC+orUtZkhePoodhjE4yilOtjb1DQgGcYyBK0qRpqo9ZZw== X-Received: by 2002:a62:b60c:0:b029:1dd:f110:b27d with SMTP id j12-20020a62b60c0000b02901ddf110b27dmr9796973pff.42.1612892849699; Tue, 09 Feb 2021 09:47:29 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id j1sm22260929pfr.78.2021.02.09.09.47.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Feb 2021 09:47:28 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v7 PATCH 11/12] mm: memcontrol: reparent nr_deferred when memcg offline Date: Tue, 9 Feb 2021 09:46:45 -0800 Message-Id: <20210209174646.1310591-12-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210209174646.1310591-1-shy828301@gmail.com> References: <20210209174646.1310591-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now shrinker's nr_deferred is per memcg for memcg aware shrinkers, add to parent's corresponding nr_deferred when memcg offline. Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai Signed-off-by: Yang Shi Acked-by: Roman Gushchin --- include/linux/memcontrol.h | 1 + mm/memcontrol.c | 1 + mm/vmscan.c | 24 ++++++++++++++++++++++++ 3 files changed, 26 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index c457fc7bc631..e1c4b93889ad 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1585,6 +1585,7 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) int alloc_shrinker_info(struct mem_cgroup *memcg); void free_shrinker_info(struct mem_cgroup *memcg); void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); +void reparent_shrinker_deferred(struct mem_cgroup *memcg); #else #define mem_cgroup_sockets_enabled 0 static inline void mem_cgroup_sk_alloc(struct sock *sk) { }; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f64ad0d044d9..21f36b73f36a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5282,6 +5282,7 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) page_counter_set_low(&memcg->memory, 0); memcg_offline_kmem(memcg); + reparent_shrinker_deferred(memcg); wb_memcg_offline(memcg); drain_all_stock(memcg); diff --git a/mm/vmscan.c b/mm/vmscan.c index dfde6e7fd7f5..66163082cc6f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -389,6 +389,30 @@ static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); } +void reparent_shrinker_deferred(struct mem_cgroup *memcg) +{ + int i, nid; + long nr; + struct mem_cgroup *parent; + struct shrinker_info *child_info, *parent_info; + + parent = parent_mem_cgroup(memcg); + if (!parent) + parent = root_mem_cgroup; + + /* Prevent from concurrent shrinker_info expand */ + down_read(&shrinker_rwsem); + for_each_node(nid) { + child_info = shrinker_info_protected(memcg, nid); + parent_info = shrinker_info_protected(parent, nid); + for (i = 0; i < shrinker_nr_max; i++) { + nr = atomic_long_read(&child_info->nr_deferred[i]); + atomic_long_add(nr, &parent_info->nr_deferred[i]); + } + } + up_read(&shrinker_rwsem); +} + static bool cgroup_reclaim(struct scan_control *sc) { return sc->target_mem_cgroup; From patchwork Tue Feb 9 17:46:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12078693 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9C9EC433DB for ; Tue, 9 Feb 2021 17:53:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 40ABD64EC4 for ; Tue, 9 Feb 2021 17:53:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 40ABD64EC4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8C14F6B0005; Tue, 9 Feb 2021 12:53:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8708E6B006C; Tue, 9 Feb 2021 12:53:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 760C76B006E; Tue, 9 Feb 2021 12:53:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0210.hostedemail.com [216.40.44.210]) by kanga.kvack.org (Postfix) with ESMTP id 5D03E6B0005 for ; Tue, 9 Feb 2021 12:53:35 -0500 (EST) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 21EF618041E74 for ; Tue, 9 Feb 2021 17:53:35 +0000 (UTC) X-FDA: 77799476790.09.start78_1a0a2f827609 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin09.hostedemail.com (Postfix) with ESMTP id 07C451809B373 for ; Tue, 9 Feb 2021 17:53:35 +0000 (UTC) X-HE-Tag: start78_1a0a2f827609 X-Filterd-Recvd-Size: 6042 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Tue, 9 Feb 2021 17:53:34 +0000 (UTC) Received: by mail-pg1-f177.google.com with SMTP id b21so12930411pgk.7 for ; Tue, 09 Feb 2021 09:53:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kgP7rVV9XOms9KmN6ZlTZi1sz4j8/Qcyt5+yMhudraQ=; b=txhUUvGbr6Et8P+qJstH1wN52HTIP3XvuXU6oQ/9/6VheYliy+7dIEw6ZrrxZnE3W5 8+KfztaNaJd32HRpr3VEsVNJVwTusKrTFLnbcTBWaDE6/cIbVDXhU5ipcpYUHcOzM+yt k9Z0nErlYMG7CPMHK3Pi0lZMM1vv5WTNH6iwzY/toOZvVUlk+Dekpk5VKNNFA/b5Wx3T 5Sr97cG005c+mFUoa9ajLmq+nQAGDNLqnOCjho1uL+LIwdur9WgZGXXwRDsqFnNGtGup KBcut14qm2csTIyHLQaxIz6xanoMmyaX3RZ+jrqmzP612y7tdyguUMidEMT8XzWW2uiu fqVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kgP7rVV9XOms9KmN6ZlTZi1sz4j8/Qcyt5+yMhudraQ=; b=LkdNloQj492GpHnv8dToORU99yKywwnMcRjoIU8g9aFKEV65VX/6Fb+q5LJjh31WV5 99q31/BEcFkIJ5hSHt4fegMykzy8r9UmfyGidgsBL63gog2RYGNakbb/nFpKQWURJOMl MzUNTlVppGsx3BJMTIaf5hKDtKu0LUUfaCcSzetlnMj4Cp0k5do4jMwz1GfZ4f925syx tFanBTBiTYcp3W26yx7zcjlC+S1JOuK5YlITBLA39cGNggJc6y577bPckhyLA9Z2LAtC WKFJWOZ/LO1hchz5BCAjJmgXY52AljUGcbVca10lsdxe8ghyGsUKuAfFEW8zH2n5Eah1 DnWA== X-Gm-Message-State: AOAM5338/QUrp07atfMDxgO1D7hE2PDgHp3ukFXGBoPdQNj716R6ZfKu S22KxTj8RoCQH8ns+rlp/EcRbZv5rrKINA== X-Google-Smtp-Source: ABdhPJxSxqPLVfiBze/4jfixH2sN5dj1At9bov1ap2/6OLfH0or2smCvE1k6LYh2/CXBR93M629enA== X-Received: by 2002:a65:6384:: with SMTP id h4mr22820090pgv.76.1612892852199; Tue, 09 Feb 2021 09:47:32 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id j1sm22260929pfr.78.2021.02.09.09.47.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Feb 2021 09:47:31 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v7 PATCH 12/12] mm: vmscan: shrink deferred objects proportional to priority Date: Tue, 9 Feb 2021 09:46:46 -0800 Message-Id: <20210209174646.1310591-13-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210209174646.1310591-1-shy828301@gmail.com> References: <20210209174646.1310591-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The number of deferred objects might get windup to an absurd number, and it results in clamp of slab objects. It is undesirable for sustaining workingset. So shrink deferred objects proportional to priority and cap nr_deferred to twice of cache items. The idea is borrowed from Dave Chinner's patch: https://lore.kernel.org/linux-xfs/20191031234618.15403-13-david@fromorbit.com/ Tested with kernel build and vfs metadata heavy workload in our production environment, no regression is spotted so far. Signed-off-by: Yang Shi --- mm/vmscan.c | 40 +++++----------------------------------- 1 file changed, 5 insertions(+), 35 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 66163082cc6f..d670b119d6bd 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -654,7 +654,6 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, */ nr = count_nr_deferred(shrinker, shrinkctl); - total_scan = nr; if (shrinker->seeks) { delta = freeable >> priority; delta *= 4; @@ -668,37 +667,9 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, delta = freeable / 2; } + total_scan = nr >> priority; total_scan += delta; - if (total_scan < 0) { - pr_err("shrink_slab: %pS negative objects to delete nr=%ld\n", - shrinker->scan_objects, total_scan); - total_scan = freeable; - next_deferred = nr; - } else - next_deferred = total_scan; - - /* - * We need to avoid excessive windup on filesystem shrinkers - * due to large numbers of GFP_NOFS allocations causing the - * shrinkers to return -1 all the time. This results in a large - * nr being built up so when a shrink that can do some work - * comes along it empties the entire cache due to nr >>> - * freeable. This is bad for sustaining a working set in - * memory. - * - * Hence only allow the shrinker to scan the entire cache when - * a large delta change is calculated directly. - */ - if (delta < freeable / 4) - total_scan = min(total_scan, freeable / 2); - - /* - * Avoid risking looping forever due to too large nr value: - * never try to free more than twice the estimate number of - * freeable entries. - */ - if (total_scan > freeable * 2) - total_scan = freeable * 2; + total_scan = min(total_scan, (2 * freeable)); trace_mm_shrink_slab_start(shrinker, shrinkctl, nr, freeable, delta, total_scan, priority); @@ -737,10 +708,9 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, cond_resched(); } - if (next_deferred >= scanned) - next_deferred -= scanned; - else - next_deferred = 0; + next_deferred = max_t(long, (nr - scanned), 0) + total_scan; + next_deferred = min(next_deferred, (2 * freeable)); + /* * move the unused scan count back into the shrinker in a * manner that handles concurrent updates.