From patchwork Tue Mar 7 06:55:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13162866 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C6C5C678D5 for ; Tue, 7 Mar 2023 06:56:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CADAC6B0072; Tue, 7 Mar 2023 01:56:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C5C526B0073; Tue, 7 Mar 2023 01:56:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B2434280001; Tue, 7 Mar 2023 01:56:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A60266B0072 for ; Tue, 7 Mar 2023 01:56:32 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7D38D120BA7 for ; Tue, 7 Mar 2023 06:56:32 +0000 (UTC) X-FDA: 80541193824.25.4F1A0FE Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by imf30.hostedemail.com (Postfix) with ESMTP id B64AF80006 for ; Tue, 7 Mar 2023 06:56:30 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="e8fFg/1Z"; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf30.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678172190; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ER53Y4QROl37259eb6APCiUTawgi9lN6yGxvXXCwSQY=; b=qfGWYmo5iduobyOh6qSBTz7VUtNsVnPhcBVCwBa1r8lZXP27JksdNthZ2YeoiCkltVDCFy H3e+IwkNGNq9uviy4qAEMQYD5jUrQCRzPHBF4fZBI3xFPdmK14J7aBqf1MP9wYS03DUWBQ lp8CUvIoUI8mltrF1JU3g8wdjENJavk= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="e8fFg/1Z"; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf30.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678172190; a=rsa-sha256; cv=none; b=VC2QDhFEHqSwf/ojnqCKv46GPvcmtvgbDIlhvNVszcjWC3wTP/SqcaQ85yIgzgO/xAje7J N4Oo5oF+xZE0l7BmzAg1XKQRFqijjP61oaYd70yhCgphz0kuCax1Mf5cdE81Aw+oa74vgQ GtqMx7NRzL4U15aJRVprdoZ8Oh6r7bw= Received: by mail-pj1-f47.google.com with SMTP id u3-20020a17090a450300b00239db6d7d47so10994819pjg.4 for ; Mon, 06 Mar 2023 22:56:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678172189; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ER53Y4QROl37259eb6APCiUTawgi9lN6yGxvXXCwSQY=; b=e8fFg/1ZMWrDBYFiJGMfpaUyPuJWpCLpWWZq+9/2MnqY7SWb0mfveVeBZnGqQEScRg SRpcjengd7bibS0+sCjtTNAacG5ZTh6UzMHGYHGhLesxbisy2K8z9u5WBafxqENJ3cle +aareyjuZRIYh2a2Es/Zb9QV30TYYKVFPL1aymJscHSeoeSE0VJP3FXWjmFFNrBEf2bt qFIAa0MZmcpqbFGrop17AAgZicTZfc+oz5iTO0ILfWsXzt07POPSqui0chCoPVnuTfpa jEpmE0LmM4BFlo/QSmO5S+SEmZWa4v5rkEHKIznIKzxalv0FQT/NqtWCuLU5TuTPr1k7 kJ+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678172189; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ER53Y4QROl37259eb6APCiUTawgi9lN6yGxvXXCwSQY=; b=UBxYVPA8UO8AcHjaNW7JhJx8eQcVO+zj0twF55ZlRm1xKFyO2yCrdqfGy5t8xNkF4E iFd4r1S6XxM49d3eOV0b6IN30xbTNWGMAKXvI02o7EQ9UAi4mQTrTsSFDYg7+xjYPjW9 oQJm9YfNVbEugVY2LgFnkqGH48Z5CX2Ghysjj9rZRVt4Ea4EwhcJ+7/cQqC7vPXrdWmo 61OtSkMSSnIiXv4rxSjulCMRNMBAUAvZ/bd01ldiF/pX7Q1ISbwR/6tZaae4FuOmBqG8 Kn6v51Cffn6JuKxZw9WSTGJwjSZ2wIR4LSSLO3u235LdxcaabrMJpScAks1A8H9T6Men JL+Q== X-Gm-Message-State: AO0yUKVQZKwq5L8ZWuF6qgRPEcZ+hag9U3pl8H1VRi0VrKN1vxZohGmk +jQ5z0bv1LkaHdyNiHS3ETRlrg== X-Google-Smtp-Source: AK7set93IUuefTEJlKAupAfnJncofKj6xEUjv3QuTx2o0at5hbUln/n/Dabc1bi7Ww06YH0cGT/21Q== X-Received: by 2002:a05:6a20:8404:b0:c0:2875:9e8c with SMTP id c4-20020a056a20840400b000c028759e8cmr16929564pzd.1.1678172189271; Mon, 06 Mar 2023 22:56:29 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id e5-20020a62ee05000000b00608dae58695sm7230854pfi.209.2023.03.06.22.56.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Mar 2023 22:56:28 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com, rppt@kernel.org Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v4 1/8] mm: vmscan: add a map_nr_max field to shrinker_info Date: Tue, 7 Mar 2023 14:55:58 +0800 Message-Id: <20230307065605.58209-2-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230307065605.58209-1-zhengqi.arch@bytedance.com> References: <20230307065605.58209-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: B64AF80006 X-Stat-Signature: onn448m8zzrhyhbawsxanmwdq7wsn541 X-HE-Tag: 1678172190-905890 X-HE-Meta: U2FsdGVkX19erKAtSbyPILWoqu9iq94kSVvXiqyhbhZY0HM5s36AaOq0AMjzoHpr7ajsrP0EEKY48JRcmyIto8DtJomxJVPhppw+Dpc5jC2wMPj3ZzxKNMcgDVBOLNs8hKv2IgkkIa6BfWVMgd3DWuJaBphLUF3WwosRs14X5jWPnszDUyCsL/PzB1zXsO+eBlen/7TC30x9l5KgOqMH0LA24MJ5V6ZEkIgdApZgCqw+KgK1bkuo8zRwKkllPHzvLL/toOCxQJE+jwMrT33OascZyCi6y0+m6DVlaj+FOBKhE++nV9vQv46/iqCgxoHIBSTTXcLUWLovKhst+DDbfsacTqxBvKcPcxhCn6X9dLsd6K0m0rnW3gGFat+x5qWM2dGU/SHrkhUcmCZSoWqObzr5GLAxqWaaNvVzD+v9zIP92/rK9+j36Iz3/RcRR9LNDfBqsi8JNWQjeeymFsnBZjH7CwyqX7GJuFlrCj03AYaX/nc7s8X6j38/f1242wEbZgUITy+At08gzeclsUPLGn5xG5jktl8frhu6pQ2pzFeGzQCGMDwLoZhAurClZKwpBkk7eWT8sihvcECvnT7Br8EFDVCjvfF7lY3SvINZb4bwXyefcYWrKWdmv/QHu7sAFeE79CTOCko0XlKcLSoE0casRzrnftBSpFPiX0F+A45Fm5fqryPFdsp1M8tvSepJ1MlcVW1tYxCpSst9RMFlug2k2dO34lHOgchcqtPqWIS4lPgdFPqmJaEfIK6f98nV5msEz7NagEmtI4EvBvUi8c7skUsGG8X89QOQwUK49U5hAR70fXQlgoxCz037V+hgMJIzEFjOLCX1Aex/oXsvqsrfk4HXqX+FYSc1gF2L7/sVIJHKpHBOdDTg3HW29SceIYluN0JK+Nq408oBwn82gMJMPSNvxRJ/cHpgk3abzKBS/sLkVCTGocngNWFEVPGaF9n7cD+b+AruBWzldKz Qr+92sSc f3nL/50LMlcEAEO+OHEMJljuoYfPbEje5V3F6pYlLLWibt3A7DK3R3bQa+ljjB1Bn64l9rtg+6liybJ2kaqUbXSE5lgDWoOAccB9z2apcSkVOzBkyCun2jyqhGhynsGbP4EAyE+ui7Pgv0h5c58cTw5MUBlERG4+2h80g2BQZMmt+WB1ZElqEg70ZFEUDbghsto9TTVQ49vHVyDGJSE8r+HkthbO86sF6yfxnSVksRefLud7y89oIlEmavSiGhPSz0BwqRzPIlLe1xGvDLqdwgP2pWvWqknfCsmEFRgEKcMN/6TlrM5Rpu1LpYruBg5zz9N6qqyYNJnzeoANvXGYO3o+HiW7stpaYowKIuZm5jbVw8yzZFvTzgpXa/fAkkzKVnyx9k6hmw5UxuYI1hReR7nPNzf2zJulG3fuifE4a3D3aWTY80iTEiPjQtBm61FU7sB7zRHx36JKqwspXWDjni0Se28++u6X0jd+imXzGEqZ5MP4ESx4yv5xH68HMYLYwrCufuXKEp0JP2UwFzpvHL9Jm1w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To prepare for the subsequent lockless memcg slab shrink, add a map_nr_max field to struct shrinker_info to records its own real shrinker_nr_max. Suggested-by: Kirill Tkhai Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka --- include/linux/memcontrol.h | 1 + mm/vmscan.c | 41 ++++++++++++++++++++++---------------- 2 files changed, 25 insertions(+), 17 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6eda2ab205d..aa69ea98e2d8 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -97,6 +97,7 @@ struct shrinker_info { struct rcu_head rcu; atomic_long_t *nr_deferred; unsigned long *map; + int map_nr_max; }; struct lruvec_stats_percpu { diff --git a/mm/vmscan.c b/mm/vmscan.c index 9414226218f0..2dcc01682026 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -224,9 +224,16 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, lockdep_is_held(&shrinker_rwsem)); } +static inline bool need_expand(int new_nr_max, int old_nr_max) +{ + return round_up(new_nr_max, BITS_PER_LONG) > + round_up(old_nr_max, BITS_PER_LONG); +} + static int expand_one_shrinker_info(struct mem_cgroup *memcg, int map_size, int defer_size, - int old_map_size, int old_defer_size) + int old_map_size, int old_defer_size, + int new_nr_max) { struct shrinker_info *new, *old; struct mem_cgroup_per_node *pn; @@ -240,12 +247,17 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, if (!old) return 0; + /* Already expanded this shrinker_info */ + if (!need_expand(new_nr_max, old->map_nr_max)) + continue; + new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); if (!new) return -ENOMEM; new->nr_deferred = (atomic_long_t *)(new + 1); new->map = (void *)new->nr_deferred + defer_size; + new->map_nr_max = new_nr_max; /* map: set all old bits, clear all new bits */ memset(new->map, (int)0xff, old_map_size); @@ -295,6 +307,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) } info->nr_deferred = (atomic_long_t *)(info + 1); info->map = (void *)info->nr_deferred + defer_size; + info->map_nr_max = shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); @@ -302,23 +315,14 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) return ret; } -static inline bool need_expand(int nr_max) -{ - return round_up(nr_max, BITS_PER_LONG) > - round_up(shrinker_nr_max, BITS_PER_LONG); -} - static int expand_shrinker_info(int new_id) { int ret = 0; - int new_nr_max = new_id + 1; + int new_nr_max = round_up(new_id + 1, BITS_PER_LONG); int map_size, defer_size = 0; int old_map_size, old_defer_size = 0; struct mem_cgroup *memcg; - if (!need_expand(new_nr_max)) - goto out; - if (!root_mem_cgroup) goto out; @@ -332,7 +336,8 @@ static int expand_shrinker_info(int new_id) memcg = mem_cgroup_iter(NULL, NULL, NULL); do { ret = expand_one_shrinker_info(memcg, map_size, defer_size, - old_map_size, old_defer_size); + old_map_size, old_defer_size, + new_nr_max); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; @@ -352,9 +357,11 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) rcu_read_lock(); info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); - /* Pairs with smp mb in shrink_slab() */ - smp_mb__before_atomic(); - set_bit(shrinker_id, info->map); + if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) { + /* Pairs with smp mb in shrink_slab() */ + smp_mb__before_atomic(); + set_bit(shrinker_id, info->map); + } rcu_read_unlock(); } } @@ -432,7 +439,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); - for (i = 0; i < shrinker_nr_max; i++) { + for (i = 0; i < child_info->map_nr_max; i++) { nr = atomic_long_read(&child_info->nr_deferred[i]); atomic_long_add(nr, &parent_info->nr_deferred[i]); } @@ -899,7 +906,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (unlikely(!info)) goto unlock; - for_each_set_bit(i, info->map, shrinker_nr_max) { + for_each_set_bit(i, info->map, info->map_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid,