From patchwork Tue Mar 7 06:55:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13162866 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C6C5C678D5 for ; Tue, 7 Mar 2023 06:56:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CADAC6B0072; Tue, 7 Mar 2023 01:56:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C5C526B0073; Tue, 7 Mar 2023 01:56:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B2434280001; Tue, 7 Mar 2023 01:56:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A60266B0072 for ; Tue, 7 Mar 2023 01:56:32 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7D38D120BA7 for ; Tue, 7 Mar 2023 06:56:32 +0000 (UTC) X-FDA: 80541193824.25.4F1A0FE Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by imf30.hostedemail.com (Postfix) with ESMTP id B64AF80006 for ; Tue, 7 Mar 2023 06:56:30 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="e8fFg/1Z"; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf30.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678172190; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ER53Y4QROl37259eb6APCiUTawgi9lN6yGxvXXCwSQY=; b=qfGWYmo5iduobyOh6qSBTz7VUtNsVnPhcBVCwBa1r8lZXP27JksdNthZ2YeoiCkltVDCFy H3e+IwkNGNq9uviy4qAEMQYD5jUrQCRzPHBF4fZBI3xFPdmK14J7aBqf1MP9wYS03DUWBQ lp8CUvIoUI8mltrF1JU3g8wdjENJavk= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="e8fFg/1Z"; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf30.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678172190; a=rsa-sha256; cv=none; b=VC2QDhFEHqSwf/ojnqCKv46GPvcmtvgbDIlhvNVszcjWC3wTP/SqcaQ85yIgzgO/xAje7J N4Oo5oF+xZE0l7BmzAg1XKQRFqijjP61oaYd70yhCgphz0kuCax1Mf5cdE81Aw+oa74vgQ GtqMx7NRzL4U15aJRVprdoZ8Oh6r7bw= Received: by mail-pj1-f47.google.com with SMTP id u3-20020a17090a450300b00239db6d7d47so10994819pjg.4 for ; Mon, 06 Mar 2023 22:56:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678172189; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ER53Y4QROl37259eb6APCiUTawgi9lN6yGxvXXCwSQY=; b=e8fFg/1ZMWrDBYFiJGMfpaUyPuJWpCLpWWZq+9/2MnqY7SWb0mfveVeBZnGqQEScRg SRpcjengd7bibS0+sCjtTNAacG5ZTh6UzMHGYHGhLesxbisy2K8z9u5WBafxqENJ3cle +aareyjuZRIYh2a2Es/Zb9QV30TYYKVFPL1aymJscHSeoeSE0VJP3FXWjmFFNrBEf2bt qFIAa0MZmcpqbFGrop17AAgZicTZfc+oz5iTO0ILfWsXzt07POPSqui0chCoPVnuTfpa jEpmE0LmM4BFlo/QSmO5S+SEmZWa4v5rkEHKIznIKzxalv0FQT/NqtWCuLU5TuTPr1k7 kJ+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678172189; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ER53Y4QROl37259eb6APCiUTawgi9lN6yGxvXXCwSQY=; b=UBxYVPA8UO8AcHjaNW7JhJx8eQcVO+zj0twF55ZlRm1xKFyO2yCrdqfGy5t8xNkF4E iFd4r1S6XxM49d3eOV0b6IN30xbTNWGMAKXvI02o7EQ9UAi4mQTrTsSFDYg7+xjYPjW9 oQJm9YfNVbEugVY2LgFnkqGH48Z5CX2Ghysjj9rZRVt4Ea4EwhcJ+7/cQqC7vPXrdWmo 61OtSkMSSnIiXv4rxSjulCMRNMBAUAvZ/bd01ldiF/pX7Q1ISbwR/6tZaae4FuOmBqG8 Kn6v51Cffn6JuKxZw9WSTGJwjSZ2wIR4LSSLO3u235LdxcaabrMJpScAks1A8H9T6Men JL+Q== X-Gm-Message-State: AO0yUKVQZKwq5L8ZWuF6qgRPEcZ+hag9U3pl8H1VRi0VrKN1vxZohGmk +jQ5z0bv1LkaHdyNiHS3ETRlrg== X-Google-Smtp-Source: AK7set93IUuefTEJlKAupAfnJncofKj6xEUjv3QuTx2o0at5hbUln/n/Dabc1bi7Ww06YH0cGT/21Q== X-Received: by 2002:a05:6a20:8404:b0:c0:2875:9e8c with SMTP id c4-20020a056a20840400b000c028759e8cmr16929564pzd.1.1678172189271; Mon, 06 Mar 2023 22:56:29 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id e5-20020a62ee05000000b00608dae58695sm7230854pfi.209.2023.03.06.22.56.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Mar 2023 22:56:28 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com, rppt@kernel.org Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v4 1/8] mm: vmscan: add a map_nr_max field to shrinker_info Date: Tue, 7 Mar 2023 14:55:58 +0800 Message-Id: <20230307065605.58209-2-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230307065605.58209-1-zhengqi.arch@bytedance.com> References: <20230307065605.58209-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: B64AF80006 X-Stat-Signature: onn448m8zzrhyhbawsxanmwdq7wsn541 X-HE-Tag: 1678172190-905890 X-HE-Meta: U2FsdGVkX19erKAtSbyPILWoqu9iq94kSVvXiqyhbhZY0HM5s36AaOq0AMjzoHpr7ajsrP0EEKY48JRcmyIto8DtJomxJVPhppw+Dpc5jC2wMPj3ZzxKNMcgDVBOLNs8hKv2IgkkIa6BfWVMgd3DWuJaBphLUF3WwosRs14X5jWPnszDUyCsL/PzB1zXsO+eBlen/7TC30x9l5KgOqMH0LA24MJ5V6ZEkIgdApZgCqw+KgK1bkuo8zRwKkllPHzvLL/toOCxQJE+jwMrT33OascZyCi6y0+m6DVlaj+FOBKhE++nV9vQv46/iqCgxoHIBSTTXcLUWLovKhst+DDbfsacTqxBvKcPcxhCn6X9dLsd6K0m0rnW3gGFat+x5qWM2dGU/SHrkhUcmCZSoWqObzr5GLAxqWaaNvVzD+v9zIP92/rK9+j36Iz3/RcRR9LNDfBqsi8JNWQjeeymFsnBZjH7CwyqX7GJuFlrCj03AYaX/nc7s8X6j38/f1242wEbZgUITy+At08gzeclsUPLGn5xG5jktl8frhu6pQ2pzFeGzQCGMDwLoZhAurClZKwpBkk7eWT8sihvcECvnT7Br8EFDVCjvfF7lY3SvINZb4bwXyefcYWrKWdmv/QHu7sAFeE79CTOCko0XlKcLSoE0casRzrnftBSpFPiX0F+A45Fm5fqryPFdsp1M8tvSepJ1MlcVW1tYxCpSst9RMFlug2k2dO34lHOgchcqtPqWIS4lPgdFPqmJaEfIK6f98nV5msEz7NagEmtI4EvBvUi8c7skUsGG8X89QOQwUK49U5hAR70fXQlgoxCz037V+hgMJIzEFjOLCX1Aex/oXsvqsrfk4HXqX+FYSc1gF2L7/sVIJHKpHBOdDTg3HW29SceIYluN0JK+Nq408oBwn82gMJMPSNvxRJ/cHpgk3abzKBS/sLkVCTGocngNWFEVPGaF9n7cD+b+AruBWzldKz Qr+92sSc f3nL/50LMlcEAEO+OHEMJljuoYfPbEje5V3F6pYlLLWibt3A7DK3R3bQa+ljjB1Bn64l9rtg+6liybJ2kaqUbXSE5lgDWoOAccB9z2apcSkVOzBkyCun2jyqhGhynsGbP4EAyE+ui7Pgv0h5c58cTw5MUBlERG4+2h80g2BQZMmt+WB1ZElqEg70ZFEUDbghsto9TTVQ49vHVyDGJSE8r+HkthbO86sF6yfxnSVksRefLud7y89oIlEmavSiGhPSz0BwqRzPIlLe1xGvDLqdwgP2pWvWqknfCsmEFRgEKcMN/6TlrM5Rpu1LpYruBg5zz9N6qqyYNJnzeoANvXGYO3o+HiW7stpaYowKIuZm5jbVw8yzZFvTzgpXa/fAkkzKVnyx9k6hmw5UxuYI1hReR7nPNzf2zJulG3fuifE4a3D3aWTY80iTEiPjQtBm61FU7sB7zRHx36JKqwspXWDjni0Se28++u6X0jd+imXzGEqZ5MP4ESx4yv5xH68HMYLYwrCufuXKEp0JP2UwFzpvHL9Jm1w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To prepare for the subsequent lockless memcg slab shrink, add a map_nr_max field to struct shrinker_info to records its own real shrinker_nr_max. Suggested-by: Kirill Tkhai Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka --- include/linux/memcontrol.h | 1 + mm/vmscan.c | 41 ++++++++++++++++++++++---------------- 2 files changed, 25 insertions(+), 17 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6eda2ab205d..aa69ea98e2d8 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -97,6 +97,7 @@ struct shrinker_info { struct rcu_head rcu; atomic_long_t *nr_deferred; unsigned long *map; + int map_nr_max; }; struct lruvec_stats_percpu { diff --git a/mm/vmscan.c b/mm/vmscan.c index 9414226218f0..2dcc01682026 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -224,9 +224,16 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, lockdep_is_held(&shrinker_rwsem)); } +static inline bool need_expand(int new_nr_max, int old_nr_max) +{ + return round_up(new_nr_max, BITS_PER_LONG) > + round_up(old_nr_max, BITS_PER_LONG); +} + static int expand_one_shrinker_info(struct mem_cgroup *memcg, int map_size, int defer_size, - int old_map_size, int old_defer_size) + int old_map_size, int old_defer_size, + int new_nr_max) { struct shrinker_info *new, *old; struct mem_cgroup_per_node *pn; @@ -240,12 +247,17 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, if (!old) return 0; + /* Already expanded this shrinker_info */ + if (!need_expand(new_nr_max, old->map_nr_max)) + continue; + new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); if (!new) return -ENOMEM; new->nr_deferred = (atomic_long_t *)(new + 1); new->map = (void *)new->nr_deferred + defer_size; + new->map_nr_max = new_nr_max; /* map: set all old bits, clear all new bits */ memset(new->map, (int)0xff, old_map_size); @@ -295,6 +307,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) } info->nr_deferred = (atomic_long_t *)(info + 1); info->map = (void *)info->nr_deferred + defer_size; + info->map_nr_max = shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); @@ -302,23 +315,14 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) return ret; } -static inline bool need_expand(int nr_max) -{ - return round_up(nr_max, BITS_PER_LONG) > - round_up(shrinker_nr_max, BITS_PER_LONG); -} - static int expand_shrinker_info(int new_id) { int ret = 0; - int new_nr_max = new_id + 1; + int new_nr_max = round_up(new_id + 1, BITS_PER_LONG); int map_size, defer_size = 0; int old_map_size, old_defer_size = 0; struct mem_cgroup *memcg; - if (!need_expand(new_nr_max)) - goto out; - if (!root_mem_cgroup) goto out; @@ -332,7 +336,8 @@ static int expand_shrinker_info(int new_id) memcg = mem_cgroup_iter(NULL, NULL, NULL); do { ret = expand_one_shrinker_info(memcg, map_size, defer_size, - old_map_size, old_defer_size); + old_map_size, old_defer_size, + new_nr_max); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; @@ -352,9 +357,11 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) rcu_read_lock(); info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); - /* Pairs with smp mb in shrink_slab() */ - smp_mb__before_atomic(); - set_bit(shrinker_id, info->map); + if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) { + /* Pairs with smp mb in shrink_slab() */ + smp_mb__before_atomic(); + set_bit(shrinker_id, info->map); + } rcu_read_unlock(); } } @@ -432,7 +439,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); - for (i = 0; i < shrinker_nr_max; i++) { + for (i = 0; i < child_info->map_nr_max; i++) { nr = atomic_long_read(&child_info->nr_deferred[i]); atomic_long_add(nr, &parent_info->nr_deferred[i]); } @@ -899,7 +906,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (unlikely(!info)) goto unlock; - for_each_set_bit(i, info->map, shrinker_nr_max) { + for_each_set_bit(i, info->map, info->map_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, From patchwork Tue Mar 7 06:55:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13162867 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33C36C678D5 for ; Tue, 7 Mar 2023 06:56:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C5DA06B0073; Tue, 7 Mar 2023 01:56:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C0E026B0074; Tue, 7 Mar 2023 01:56:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD594280001; Tue, 7 Mar 2023 01:56:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9E7466B0073 for ; Tue, 7 Mar 2023 01:56:39 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 64438A0A7C for ; Tue, 7 Mar 2023 06:56:39 +0000 (UTC) X-FDA: 80541194118.14.82D26A0 Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by imf07.hostedemail.com (Postfix) with ESMTP id 7FF3940017 for ; Tue, 7 Mar 2023 06:56:37 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=gwtF2j6j; spf=pass (imf07.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678172197; a=rsa-sha256; cv=none; b=Q2vT5S2E8odBM7KziIiFh48/MVmqx49WEJYY4KycVjJGmGDP6s33RLGNjQlwFO0U+WZWUC Eu6AKHwcfmi1Bf1uFto1iv4WtuDvLcceacsTSyElasz3JhGbvxiXeMi/5V9olcTu1EJRAQ ZhWmxoIozdjFyEu9YXFC28ByF7zk7Og= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=gwtF2j6j; spf=pass (imf07.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678172197; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ASSD7BghNw/J/hUaiXgWQCknh8Fl8R4TjAxXoZe8rSo=; b=BG2usaR8bt6MPpaxKzaLO+Zv2JSu5iutDN+uGp3AKQWxWZ+4jeVyv3HiCJgUILkQMGGQlC z6VB5EVbbyf+EY5T498a1R30wGasU8UiGK8i/6ECbuMqgG7cVrin3+vDMTWZYLUHd9OV8K BA/OiS5aNoslyvKNmkvh2L9JRuOBWVU= Received: by mail-pf1-f171.google.com with SMTP id ce7so7391848pfb.9 for ; Mon, 06 Mar 2023 22:56:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678172196; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ASSD7BghNw/J/hUaiXgWQCknh8Fl8R4TjAxXoZe8rSo=; b=gwtF2j6jEh4RIfI6JeZzkRF/pMyUNSc09ILbLx3nVVaFUO9uEdhj33JRDIszQXQGC7 tFr1nfLXPXzxu61Qs4pZfvZ+lgsXotnOK4kxSA6ErTDJ3W9Yv9udOYJFj30nHSvBFyCK 6zXZzdciINciHjYiRTh9V7nvLCLygsVJ/+eAkyvaE7v2gDCPhNda8nR9vHnymWdta9XX nom7SbNGo2amnoXsU0yzWSGLVguPni1/Zb7foDqZHxvqZtJ3jEqF2qkhQsumFjGyA+Oj mW773gxFLzs4+Pd46Q3RYSLslTYbgRVmbhEIspK60CCkH26JKY6i5r6Y+hVwWdew41+A rdQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678172196; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ASSD7BghNw/J/hUaiXgWQCknh8Fl8R4TjAxXoZe8rSo=; b=A8wk/DMibdp1wrViE+eP/K54fhbVwUyTVadN4O8ABvbOTBbJugY756SIPQ7I5rxV7A G4IIgCqXqnZSlJ5BUaArvuQxMzoUFUHSmjpw/mgbxbZm8XOxwci9KHYJa1mvu9mDWHlE lEiEDeJ6YDhmsrIXMHLxLsROFfMHMWiD2xEN7P1647zZGxlltRZQeCT+nsIJNZs+wxvL AEpYvNUnmDjhP2Q2W86ZTWTTwlD5UmWaNsxtPiaQI06S877acOq+QoZrxY5+IrH/cvcK IwJF02Aglz8XuSVcW1aWzllg98Qa3fsnu9pO6apbVlj/QVUkOrbKlzFrTjCPGaXQHhkr Rj9w== X-Gm-Message-State: AO0yUKXym8z3tH/qws2w3hSJ2u7+wpLAWgTQfp7cTxH/kgFpbqD61M56 hPSE0MgEJXWXFyl4M+OMSQj6FQ== X-Google-Smtp-Source: AK7set8ycLeV4ug3Hh8JCVS9nXkkF3OpTneeqDALEmgAnAPJ0FJlAs/ofqR1TK404Yuh4hIX7jHwLw== X-Received: by 2002:a05:6a00:2d89:b0:5e2:3086:f977 with SMTP id fb9-20020a056a002d8900b005e23086f977mr13567615pfb.2.1678172196363; Mon, 06 Mar 2023 22:56:36 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id e5-20020a62ee05000000b00608dae58695sm7230854pfi.209.2023.03.06.22.56.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Mar 2023 22:56:35 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com, rppt@kernel.org Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v4 2/8] mm: vmscan: make global slab shrink lockless Date: Tue, 7 Mar 2023 14:55:59 +0800 Message-Id: <20230307065605.58209-3-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230307065605.58209-1-zhengqi.arch@bytedance.com> References: <20230307065605.58209-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 7FF3940017 X-Rspamd-Server: rspam01 X-Stat-Signature: ngqyququqbwqagt6643nkxo1gitunchc X-HE-Tag: 1678172197-42126 X-HE-Meta: U2FsdGVkX1/z7maYK+KYA/o2jmpjZaA2XX7eY+ZSuLzkzTIG6b1/N1CVgrY5k6oqUCzPg/yWkuA+YqVFIiJJ7ZMuf7ZmR/Yh0aXhD8LsIZYe/RMsS4ATQvIHklvdj8aMIxtua0tr0fR37u35/IRmhYkA1HYUw0hi+xvPhz/SA7/pxhlNe9dgUrVGrGE6udxH6Zc5prLk4VogAtsQFHSef9AJSXeUo6sVm8WJuGOEAvD8oDfBAYpTMZq1+9vzWUES788e9yT4JQMSX7zMDjt7V5lMB2UwpZTQuvYXsmaR5AEaerU2fZpgR5tQYMAYFHG0djzItxhkpyJn+5qFj+yvOeVcarYhJ8jhBkijsF9wwHT5AIzdUtI463B76j9TX6Dt92HfJBaCwxwS4n3ME86z56q40R9qXsmgIddA7tdnkbxwik6ZdEm+iGitnYlO1qTIVKNr5GoTnQPxzydjkRHihp0FLt66zJ+QnaJaRz9zxlotWM7jf25eD6mHjrC+OwOy2tiyaRA4wDVCjxaNFLNYgPyE7+kEcmjWt2szAXLdKXbgMYSxUAZ4bl7wlBzt4NumPFf83BTyEZg5v8cvJEADVkpiyMfx/EbIZmFWsxWRmmw8M1B/+27BOxudVN6ZHdmJeYPr9s+ntq2QUcoB5BXO3P/JRNoH10hFoFHhKHAUI3YJDsZ3GUtgUAasYCHs506OlLMIpha13abCEezaLmQdFPJNBHKEQRPOOMksoXm8rg9hd+ITYrRJmP/Y1j/CDd293nExS48JuSbq5kJVOPAUkhuyIFXDjQ/FV3KL+Tua2ZMTd76E+c4WbwtV8Y616ZlzyYxHLFHItw0cfvBrXrbcbr2gJYU/vwpanpzU7l5h1p3szK+qys8dPsjn+YJW8UHjhUPTemwpBbJPBlJRiu9aXM2EmGLPe060OsVQ+vuvMITli0nclUkDGNNMYKfezpow33FEj4HrRrbyGCtBvdC 9xt52HaW +DcTwKR8FRbkcnVCmLPyOG1N0FzZBCdBJrX8FBKktL9HT3l1MOXEy0b7nN1YVSCdLfjPgZ6HpI4/59bJiXn7Y21Dup/Db+qEjKZjbu9jsJ392OE66QlxzgkHhuIH4q781WwEXSRO939JuZ8uByMmL2kRSHs2CDBgDfJyZ/SKRHzN4uk63fsvtDOi5ysBHG7XA4wkpb8cL1W7LNiTRnWzmFKfQmKXxTdQT3iQRqcRZNRr1IuFYGWvh3UmJPfl6QQuWDWLfB4hYuCuWDqxXKu146bgcv26O8igEwKuDMrn3uHmNNFn457Khk4FcjB3Jy2jpM27R7oPVE3cdRjGzOx5eWhGQ53hdNhNUwB4Ivu/kzaRnHbWRVMRwd3kucJMLGMzOwya6L4C/tN1LwSKIwdCQ0u24wcx/fr6enYJgnlHYMXyrZbT/OC5yQtaTSLERmIWgce9mKcq87suy679ExwWbrSxMgdGHNi8678sb6vgpfMc36yHYBZ5+ULZTRHqnOGmIq0QJV5qyalWHwL/smV6GknTq9FMxPAUAkIZW172cH6FMcO3bRzP5BoYTu5cJ9EUvNgNa3gTT7Dh/JaaEfWbAUslyf2ERLFqnigZlaxkbLpnq6LYjDv0PafH/tzwc6akQbr7Hqr+D4/eM0rXHZ/dSDJnLyt0A3p7xp8U3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The shrinker_rwsem is a global read-write lock in shrinkers subsystem, which protects most operations such as slab shrink, registration and unregistration of shrinkers, etc. This can easily cause problems in the following cases. 1) When the memory pressure is high and there are many filesystems mounted or unmounted at the same time, slab shrink will be affected (down_read_trylock() failed). Such as the real workload mentioned by Kirill Tkhai: ``` One of the real workloads from my experience is start of an overcommitted node containing many starting containers after node crash (or many resuming containers after reboot for kernel update). In these cases memory pressure is huge, and the node goes round in long reclaim. ``` 2) If a shrinker is blocked (such as the case mentioned in [1]) and a writer comes in (such as mount a fs), then this writer will be blocked and cause all subsequent shrinker-related operations to be blocked. Even if there is no competitor when shrinking slab, there may still be a problem. If we have a long shrinker list and we do not reclaim enough memory with each shrinker, then the down_read_trylock() may be called with high frequency. Because of the poor multicore scalability of atomic operations, this can lead to a significant drop in IPC (instructions per cycle). So many times in history ([2],[3],[4],[5]), some people wanted to replace shrinker_rwsem trylock with SRCU in the slab shrink, but all these patches were abandoned because SRCU was not unconditionally enabled. But now, since commit 1cd0bd06093c ("rcu: Remove CONFIG_SRCU"), the SRCU is unconditionally enabled. So it's time to use SRCU to protect readers who previously held shrinker_rwsem. This commit uses SRCU to make global slab shrink lockless, the memcg slab shrink is handled in the subsequent patch. [1]. https://lore.kernel.org/lkml/20191129214541.3110-1-ptikhomirov@virtuozzo.com/ [2]. https://lore.kernel.org/all/1437080113.3596.2.camel@stgolabs.net/ [3]. https://lore.kernel.org/lkml/1510609063-3327-1-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp/ [4]. https://lore.kernel.org/lkml/153365347929.19074.12509495712735843805.stgit@localhost.localdomain/ [5]. https://lore.kernel.org/lkml/20210927074823.5825-1-sultan@kerneltoast.com/ Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai --- mm/vmscan.c | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 2dcc01682026..8515ac40bcaf 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -202,6 +202,7 @@ static void set_task_reclaim_state(struct task_struct *task, LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); +DEFINE_SRCU(shrinker_srcu); #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -706,7 +707,7 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { down_write(&shrinker_rwsem); - list_add_tail(&shrinker->list, &shrinker_list); + list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |= SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); up_write(&shrinker_rwsem); @@ -760,13 +761,15 @@ void unregister_shrinker(struct shrinker *shrinker) return; down_write(&shrinker_rwsem); - list_del(&shrinker->list); + list_del_rcu(&shrinker->list); shrinker->flags &= ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry = shrinker_debugfs_remove(shrinker); up_write(&shrinker_rwsem); + synchronize_srcu(&shrinker_srcu); + debugfs_remove_recursive(debugfs_entry); kfree(shrinker->nr_deferred); @@ -786,6 +789,7 @@ void synchronize_shrinkers(void) { down_write(&shrinker_rwsem); up_write(&shrinker_rwsem); + synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); @@ -996,6 +1000,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, { unsigned long ret, freed = 0; struct shrinker *shrinker; + int srcu_idx; /* * The root memcg might be allocated even though memcg is disabled @@ -1007,10 +1012,10 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (!mem_cgroup_disabled() && !mem_cgroup_is_root(memcg)) return shrink_slab_memcg(gfp_mask, nid, memcg, priority); - if (!down_read_trylock(&shrinker_rwsem)) - goto out; + srcu_idx = srcu_read_lock(&shrinker_srcu); - list_for_each_entry(shrinker, &shrinker_list, list) { + list_for_each_entry_srcu(shrinker, &shrinker_list, list, + srcu_read_lock_held(&shrinker_srcu)) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -1021,19 +1026,9 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; freed += ret; - /* - * Bail out if someone want to register a new shrinker to - * prevent the registration from being stalled for long periods - * by parallel ongoing shrinking. - */ - if (rwsem_is_contended(&shrinker_rwsem)) { - freed = freed ? : 1; - break; - } } - up_read(&shrinker_rwsem); -out: + srcu_read_unlock(&shrinker_srcu, srcu_idx); cond_resched(); return freed; } From patchwork Tue Mar 7 06:56:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13162868 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39E88C678DB for ; Tue, 7 Mar 2023 06:56:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C5DBE6B0075; Tue, 7 Mar 2023 01:56:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C0E1C6B0078; Tue, 7 Mar 2023 01:56:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AAE9B280001; Tue, 7 Mar 2023 01:56:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 99A9C6B0075 for ; Tue, 7 Mar 2023 01:56:46 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 679CCA0A7C for ; Tue, 7 Mar 2023 06:56:46 +0000 (UTC) X-FDA: 80541194412.02.8B75694 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by imf06.hostedemail.com (Postfix) with ESMTP id 83FCD180013 for ; Tue, 7 Mar 2023 06:56:44 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=iOzGoBBF; spf=pass (imf06.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678172204; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9rhvfVxo/VIqlZiNKGwl0zk05JZT0WS+du3pcLhwX24=; b=nfavgqlLPaol/a4jsFyvSeMTQnWrYdGU6Oe08hVKgsT0apayFN6fW7Otc79Vf7GPwc9ToA Rq/+rIS7i98ooNsxzEy/1OiHDQY3uPrpvENz4bDAzaPzZDV0zPIJmQNTCQZHDOTpC3Uzaf CPLP14QlAox6lk9IFa8tJKcQIaQ0B1A= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=iOzGoBBF; spf=pass (imf06.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678172204; a=rsa-sha256; cv=none; b=p0oncvSAamrGQODZ+3R5SOJSvObyx9KtkjW5JKrpgG/8PduKMxbhNdX1XNCdFv8Ai/Yz00 i/94h6AzXrJDh3uqfCUX9ChQpO2Vp2gY8fu75AP5oSlrc0QlpIsjUXvDtvlsPl+J4BTnRl +McE/zA8pDByn3cLMehpMgFOVOHUmcA= Received: by mail-pj1-f41.google.com with SMTP id kb15so12306312pjb.1 for ; Mon, 06 Mar 2023 22:56:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678172203; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9rhvfVxo/VIqlZiNKGwl0zk05JZT0WS+du3pcLhwX24=; b=iOzGoBBFwi7lf9XHmfcZfShTltMjKW7fB7JTbgosxQmnrGMtoxbAgVo1GO+pB9geGD ux0EoYY0pV+ERjF8F18TEfHq9gvIyT4dhtCOozSdJ5U3Uc7KDQZjV/RZ0bw9vjZZGL8h 6AvKxfXxIsouFF4LnTb66UYjTB0h3F16LVfPmemtGR2hTzHJvTn0cjtGnyb6+IXsRvFv yWUb+9Dses9zqWd2+QitUQbqI7oCCIQFPmCp3hOV3Kywr901CALSY+b2ZE0aQz5c+ubl QFR4X7Ka+3ROzvlIIttem0P+mhUUNQnXkpmli1fLe9fgOOGOH4R00OxZ5OvRzTgdpGxM QZow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678172203; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9rhvfVxo/VIqlZiNKGwl0zk05JZT0WS+du3pcLhwX24=; b=dYQGNldMiTudQYje6ZhnO2bW99fIJkfOtllsU5zWfe66+zXn7XZMRqaty7xTNwpEAu b+Cx92N+M9/s3wOXEh81OK9H420gC3ga1mMY1d2O5gQRl8JAn6vV0LsNbDkBvd2cXlKi tnWY7b5TqcFr0WYjDkdnOnX83D3vJdKcARKBOlsn7yRy7X2a2NhZO3QFg99EHAGl8oMx eYrZZdBwG0YdBIEP9dbG011fC9netzOTxaHJ7q4MuUwBv0A2Z9XU+/kzQeFnbKl8DQbL csK7UW0Ft/jMYjcezRgG3KdR2q4g9ePFEhcQtYFIK0HD1b4+ema0qu81+K55Su6sH3bh QFFw== X-Gm-Message-State: AO0yUKWr+WwBx+C2dryf0xmoemZiORn0bS4CZCO8NUyib7UzdSOygDGW veZ3643EJiBAud//9W6PDqFtLdj7oUyzyN7uQJ8= X-Google-Smtp-Source: AK7set/e/2R01sRD84O4F/1HOwtTPn1OBW88UmvnVs6OndVpcmZVVmumCYjm8VTdPHpr5b8MxHJzjA== X-Received: by 2002:a05:6a20:1585:b0:cb:c276:58bb with SMTP id h5-20020a056a20158500b000cbc27658bbmr14626109pzj.4.1678172203481; Mon, 06 Mar 2023 22:56:43 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id e5-20020a62ee05000000b00608dae58695sm7230854pfi.209.2023.03.06.22.56.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Mar 2023 22:56:43 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com, rppt@kernel.org Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v4 3/8] mm: vmscan: make memcg slab shrink lockless Date: Tue, 7 Mar 2023 14:56:00 +0800 Message-Id: <20230307065605.58209-4-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230307065605.58209-1-zhengqi.arch@bytedance.com> References: <20230307065605.58209-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 83FCD180013 X-Stat-Signature: 8akomjt3gcoouyky19r7idckq7rofzye X-HE-Tag: 1678172204-870328 X-HE-Meta: U2FsdGVkX18nyAPIUiFR0WWaLb/WGXUArSbJMkWSMtMN64MfXTx71AgHGDBqMLxrHmk7YZCsranQJt0TnnfRBgmWKwfjAQRe8oT9aGlq2yYIe/s90pqXVMo2qR5d7c6EgWKCZkltm+GlhkYnmxVYqnUbbqLvVX2BeK+zVDyWR7JeQGAnNY8aQdfj+YVldXSKH7bylwbhlsJMSMu41lo/zDi/2ae70U+Unh481FIwDd6mxhqOVWJp6p0eLUIWUE3R3dXXagfesY6czyyrEXhMOrY9aapU9O0gD8zGqjVRqvOv/DOmvoT7QK5bPYKiChjW5N8txpSTbWOHIFpGE8s4BIwHfzPX9WzRWx33kk9rpYblTOfQ+swnq1WPL4ecSKVBgg0bTc15YjVG6M8tNXJwuWO2x2HXv78wzUcpcuktHK15dEeetQAA6JAbaNc+G5O1Cc54SrZYrtXdeoqTercr5UgjO3UVtSm2qW9Vv9SFRnORqaANpQUfCxJAwbUMfqBAS07+1sfNK8nsppypQ03nByPWoFk/cQ8+KPGCepuH9vQIbVs04xYGRNee6+lXd6t50fxJsB+3NYQCdYn+4wTtocK46IA8DsQFkQb8XEoptOvRh9vMW8HI53X8L5T25eY92IGaOTF5oWtQ+cMpEYbKPa+efVAQaxBaHVnnKRp3bJ9APpiDHuMeL3+5iHrtGJQM5iYdTI0QSNg9/bwvRacccEdIypxMdCXSrkfqV7U+UZtywGDvKaP/+8xabP5fOeDnUKTh9glvsn0bw1TcyyJIMByEyrDeZghGdedjD0GwFjtdHrKxqnmQVQlE88A60lC+HNBcNswvjvfgNyavP71y06jcBxUScOKMtxutFE+KsyYfnPSF284S/9Rtc3xfFo+B5BE/lZafsmvQoI+nHMWAjGjJzDgRDxRhN8TPpRoERIlRVAmP7VxbOGrTk8KgS3RGJf9HpyET31n/ifngPQl Gw15cedk u4Dct1o8wVJIgOw+mblElAA3gxnr6yG4l3M2q4EEngf+xb6QSfYrBG+XJy7gi6dzUlnvs06cr3pVOmSvg6E6pcqto8ysoOmu2pN53uXakSdG+MwQdowT8QCmPhl03TxZDNfWJ+nv1ipvojidnnz2PVBDqN/GB5+nY5cVbfj52+cY+l433I2UlHIL6b/azOEByCkmoquy6ugIGrSEJF1AE0vTTO6HfRzNJwfw+kaTmuyefSzo+XiV4lkzf2jSO78WVFSHYIzS02u1M4OW2oRxLEJK9iFlHdBDB6olCmBpDxo/b8HsqWytBQpsIO45gfsy6mHhrhQb+bkhg9iu5CO6n3r8sacfXniGM+krqHxMAxrMNdV5GRH3hNlSdIx76XG4akcYyUrM5aQJAV+X9dMDYk437vFptcFbKerN6cs7Z3jx/Wjc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Like global slab shrink, this commit also uses SRCU to make memcg slab shrink lockless. We can reproduce the down_read_trylock() hotspot through the following script: ``` DIR="/root/shrinker/memcg/mnt" do_create() { mkdir -p /sys/fs/cgroup/memory/test mkdir -p /sys/fs/cgroup/perf_event/test echo 4G > /sys/fs/cgroup/memory/test/memory.limit_in_bytes for i in `seq 0 $1`; do mkdir -p /sys/fs/cgroup/memory/test/$i; echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; echo $$ > /sys/fs/cgroup/perf_event/test/cgroup.procs; mkdir -p $DIR/$i; done } do_mount() { for i in `seq $1 $2`; do mount -t tmpfs $i $DIR/$i; done } do_touch() { for i in `seq $1 $2`; do echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; echo $$ > /sys/fs/cgroup/perf_event/test/cgroup.procs; dd if=/dev/zero of=$DIR/$i/file$i bs=1M count=1 & done } case "$1" in touch) do_touch $2 $3 ;; test) do_create 4000 do_mount 0 4000 do_touch 0 3000 ;; *) exit 1 ;; esac ``` Save the above script, then run test and touch commands. Then we can use the following perf command to view hotspots: perf top -U -F 999 1) Before applying this patchset: 32.31% [kernel] [k] down_read_trylock 19.40% [kernel] [k] pv_native_safe_halt 16.24% [kernel] [k] up_read 15.70% [kernel] [k] shrink_slab 4.69% [kernel] [k] _find_next_bit 2.62% [kernel] [k] shrink_node 1.78% [kernel] [k] shrink_lruvec 0.76% [kernel] [k] do_shrink_slab 2) After applying this patchset: 27.83% [kernel] [k] _find_next_bit 16.97% [kernel] [k] shrink_slab 15.82% [kernel] [k] pv_native_safe_halt 9.58% [kernel] [k] shrink_node 8.31% [kernel] [k] shrink_lruvec 5.64% [kernel] [k] do_shrink_slab 3.88% [kernel] [k] mem_cgroup_iter At the same time, we use the following perf command to capture IPC information: perf stat -e cycles,instructions -G test -a --repeat 5 -- sleep 10 1) Before applying this patchset: Performance counter stats for 'system wide' (5 runs): 454187219766 cycles test ( +- 1.84% ) 78896433101 instructions test # 0.17 insn per cycle ( +- 0.44% ) 10.0020430 +- 0.0000366 seconds time elapsed ( +- 0.00% ) 2) After applying this patchset: Performance counter stats for 'system wide' (5 runs): 841954709443 cycles test ( +- 15.80% ) (98.69%) 527258677936 instructions test # 0.63 insn per cycle ( +- 15.11% ) (98.68%) 10.01064 +- 0.00831 seconds time elapsed ( +- 0.08% ) We can see that IPC drops very seriously when calling down_read_trylock() at high frequency. After using SRCU, the IPC is at a normal level. Signed-off-by: Qi Zheng Acked-by: Kirill Tkhai Acked-by: Vlastimil Babka --- mm/vmscan.c | 46 +++++++++++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 19 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 8515ac40bcaf..1de9bc3e5aa2 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include @@ -221,8 +222,21 @@ static inline int shrinker_defer_size(int nr_items) static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, int nid) { - return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, - lockdep_is_held(&shrinker_rwsem)); + return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu, + lockdep_is_held(&shrinker_rwsem)); +} + +static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, + int nid) +{ + return srcu_dereference(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu); +} + +static void free_shrinker_info_rcu(struct rcu_head *head) +{ + kvfree(container_of(head, struct shrinker_info, rcu)); } static inline bool need_expand(int new_nr_max, int old_nr_max) @@ -269,7 +283,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, defer_size - old_defer_size); rcu_assign_pointer(pn->shrinker_info, new); - kvfree_rcu(old, rcu); + call_srcu(&shrinker_srcu, &old->rcu, free_shrinker_info_rcu); } return 0; @@ -355,15 +369,16 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) { if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { struct shrinker_info *info; + int srcu_idx; - rcu_read_lock(); - info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); + srcu_idx = srcu_read_lock(&shrinker_srcu); + info = shrinker_info_srcu(memcg, nid); if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) { /* Pairs with smp mb in shrink_slab() */ smp_mb__before_atomic(); set_bit(shrinker_id, info->map); } - rcu_read_unlock(); + srcu_read_unlock(&shrinker_srcu, srcu_idx); } } @@ -377,7 +392,6 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) return -ENOSYS; down_write(&shrinker_rwsem); - /* This may call shrinker, so it must use down_read_trylock() */ id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -411,7 +425,7 @@ static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, { struct shrinker_info *info; - info = shrinker_info_protected(memcg, nid); + info = shrinker_info_srcu(memcg, nid); return atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); } @@ -420,7 +434,7 @@ static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, { struct shrinker_info *info; - info = shrinker_info_protected(memcg, nid); + info = shrinker_info_srcu(memcg, nid); return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); } @@ -898,15 +912,14 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, { struct shrinker_info *info; unsigned long ret, freed = 0; + int srcu_idx; int i; if (!mem_cgroup_online(memcg)) return 0; - if (!down_read_trylock(&shrinker_rwsem)) - return 0; - - info = shrinker_info_protected(memcg, nid); + srcu_idx = srcu_read_lock(&shrinker_srcu); + info = shrinker_info_srcu(memcg, nid); if (unlikely(!info)) goto unlock; @@ -956,14 +969,9 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, set_shrinker_bit(memcg, nid, i); } freed += ret; - - if (rwsem_is_contended(&shrinker_rwsem)) { - freed = freed ? : 1; - break; - } } unlock: - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); return freed; } #else /* CONFIG_MEMCG */ From patchwork Tue Mar 7 06:56:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13162869 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A2B9C678DB for ; Tue, 7 Mar 2023 06:56:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 901EF6B0078; Tue, 7 Mar 2023 01:56:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8B3566B007B; Tue, 7 Mar 2023 01:56:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 752FA6B007D; Tue, 7 Mar 2023 01:56:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 660BB6B0078 for ; Tue, 7 Mar 2023 01:56:54 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2F63CA09BD for ; Tue, 7 Mar 2023 06:56:54 +0000 (UTC) X-FDA: 80541194748.25.E95ABF0 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) by imf09.hostedemail.com (Postfix) with ESMTP id 6AFB3140011 for ; Tue, 7 Mar 2023 06:56:52 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=j8xEkJaY; spf=pass (imf09.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.170 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678172212; a=rsa-sha256; cv=none; b=rps57CaVR/DVSgRRRL6uioG0htF1aOl7Y3Pnw27+HozjpdmJFhdYZqs55e3ulsax89FgR7 B309hIgWd2u/j1b9DObdDVn2VDxSZZLnLqmQXS6bOuD8l71V/n1WVybkjKEb/o3ll0yiHA 0dtzSrdCu0PSb3RVDsWabARK2mQ+sSY= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=j8xEkJaY; spf=pass (imf09.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.170 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678172212; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hlfbC5hLUOyZ5eiKWKe3FNliR+/5Zth66owFIX3duxQ=; b=t+qM2SX0KBo3d9ft4YSEu9RhUHyJW0H9RNR9wk8v3HKYxGXNXDp0+ARrK+kbL6mMp37JP/ m/6SUbP6e9+4XxfNNMDx+XIdGPfChM/5CkqSVCeKnITNI5d414b2NqBXBEoEczBFQtMl7V 4IkfWyUFoVln0SxZUQ0E14mXSUhRQTw= Received: by mail-pg1-f170.google.com with SMTP id 132so6987438pgh.13 for ; Mon, 06 Mar 2023 22:56:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678172211; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hlfbC5hLUOyZ5eiKWKe3FNliR+/5Zth66owFIX3duxQ=; b=j8xEkJaYof6JnjV2P0RsveJBbsnpcn9RY4V+ifKMT79Be93zvpgebNY5nSfWdFD0NU 1JW1AYTHaO+N+RFx2VLXc9tAFRn/DsFZYM5L65TiSMr1hMRgmru3xvORasp/75YD6J/y rGn2vVrH8HJoacjH3PbF9GnOpdyo3gX0mQEUZgRXnTBovRgY8MVxTw8BI9uz52Q/LQHG E2dcdZEf3BV3Hh0v3X7btzqm0okyXgWthyJJzVeextpZ9F1g0oAYjtHaaqZ/iEWplil2 10Sb5xodkY2XripisufbYdb8ro/zRPnyPWGLDA9/6bcUB7x768fm0jxge+41ve9sBsI8 3UMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678172211; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hlfbC5hLUOyZ5eiKWKe3FNliR+/5Zth66owFIX3duxQ=; b=emGMswa8tKOkusl3M+KkmiJ8dty+UlexHot+kd09uEy2rCUbi//r3t7OHaAOnIwRHm 8TcAEo1lcDWoV3drJrb1j8Djh+bxJFcOvmTIjLNpA3CUZ3mNJLk4MdsOXrmquqV4B8wR +uEXGEi1tFYCLN7/HBQTBNCn8tNigb0J5a2okVYu3egzz4fHiuoD771cpugzV4/844dC saoST2CwhUflDCO64hDUPQCTHf6hCyUjsf44nUR5ZsJD2VosUDefomEmZqmxZwtP78tI 9Y9uOybflr2EG/PD3zHJtFcQNxvSZCahbGFTw3/uDyzYc6Fv2UbWvnZMgv4e8c4Ws+tI ynog== X-Gm-Message-State: AO0yUKWh+UOfCwwPKVnBIFrC8ohku/N3NLyGuj+5dkU/KNoiswVLt3EP kQU4EaCcx+ILU5zNaNfRUxLEig== X-Google-Smtp-Source: AK7set9MiEUf3QenLOBMQ3cdZKmZL5CffIDhT1D/uEa1g/R5V5cbMebfmciE6mPU6pFS2nArJyvZ4A== X-Received: by 2002:a05:6a00:4087:b0:5d1:f76:d1d7 with SMTP id bw7-20020a056a00408700b005d10f76d1d7mr14022848pfb.1.1678172211105; Mon, 06 Mar 2023 22:56:51 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id e5-20020a62ee05000000b00608dae58695sm7230854pfi.209.2023.03.06.22.56.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Mar 2023 22:56:50 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com, rppt@kernel.org Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v4 4/8] mm: vmscan: add shrinker_srcu_generation Date: Tue, 7 Mar 2023 14:56:01 +0800 Message-Id: <20230307065605.58209-5-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230307065605.58209-1-zhengqi.arch@bytedance.com> References: <20230307065605.58209-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 6AFB3140011 X-Rspamd-Server: rspam01 X-Stat-Signature: tzjx17nfg9y8k3wxj91wrerns5wy33ik X-HE-Tag: 1678172212-693236 X-HE-Meta: U2FsdGVkX18U9xAg/4/582HTKRUbMkGrYyPIOR/FpS4QJ2xm6BzymIDMXMsL5FWJHPhmsAbaAleLOfR/6yLQP0WU+CGhL9T6OtFwsU295bZbuzhk4MJE1LB8ivKN8HNHMnc9iq0tbxG+rpRfTYTA9VzbA9MyiHo2clFhtwq1XlJknBaP9GzihcPrQ+Rev9Bl048woR9M9FHbOGZIGnVg3nD0tK5yuerUtojLdrtTTV7slZi21m4tb2qxf/hpTPM8bbIKDKLibAB4h2vYSMCU8aTcRNduTOkam71qyhQTuV+F1NlbndOVgZ0OC0mB/+yiYbD5RBH50ZrcFlZ92NfDzF+/J4m8XMQH2GsMCWiYF2Qa0Bys5/RNl2u8RanYmrTeccaTzIWarTTU5ofYBlchDcu/x5xkn1hV/Fiq8IcY5E0bExbjvZnMP58D7PZIw7oPi18IsQ1NUtQ77UsX/CMdzjvNUcxvnA1vxELF+Gnl8ECcEwFSoKjmEVJp0kJ5sTazfw/K0ja0Ytuv3DG7SLtbqxfox45ATHC0YupBjmAKsTzaQhNtEvAoWghm7mK2nN0hCe3N7WD4B7JCCllUNtULgf9NFH5GQd5w6kF0ckHQSxzlFzXf2yNY/N0jJIQ1R4C84TZNSswqlK4qLLOpauhL9f6+Qkzovkw0BmpArEimkEpR8svpdrkFxeWEn3tlddZxkdrWpCMnTMFyn4P9C65Dg+M1rU6HuBGYKK2PrKu7JNdPfM77VaKvwobowsadScB3akXgH/ATz4AOmKty+YQ1elHvyLXi4jwy+CQnltHKe+DR3a38z3JrZWg/XoAI+S7XWjgJHUlQ9PZP7uLA+UkI5l/311L3uR59aTFPOAZlUdG7iWrKqwvBHzmo+ibkTsK+4WYRNRZzOm1TcUkH/yVHqTYuQoMfE88+MQtkABQCbM/o5bYp98HwVZ5dE4Gl++3ko9z2b7gjsPXHBIYHKVT 9BufsMXK YftrP2jtx3B7pkiS28VcYOrkFIgv8eQwWop0LFs8+2DgOpTFgb35WhfRXZMNaSD0VcUAOMgDzjIm97lmyiBQlFWGupA31YvRCfxJZQhgzet9BMrISzSR2FZjRVK6vE1Ncp03uK/4vbbJ7CVoVzpoXHbUZYH7ZaWC5qEdsQA/DOKFtJRiYblr883cfkcDQHd6+zZNSSKmvVdLSZLCXTgFJ79AdaJfnPJb0UNGqh05di7LYe/77A/wafNvW63R/5Tn6nOjcgaCmUC29u4DYjqgeT4VmKwRHylUsqZoXdF139fTCyes/Gq0C++Rx2K2JlBDIqatY8SQfcOgENuXJfeJTww5PgpbMEflnqhONPGvrXulV2RaDxXS2SgougksJvt7tl5CqbzXjumMmYwtmaKnKyi8kGQc7tNvwuBwF55znYBmZy3rcThp3fOpkDvAs2Txfj4ynqMDkP1WnienxhSGgMXnD8C3wwmBrWsw6zWmWYOjv4jMF0pTzpivHkNRJCEd0U+873mUn8wFKpYkQoVWpyy3UIA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kirill Tkhai After we make slab shrink lockless with SRCU, the longest sleep unregister_shrinker() will be a sleep waiting for all do_shrink_slab() calls. To aviod long unbreakable action in the unregister_shrinker(), add shrinker_srcu_generation to restore a check similar to the rwsem_is_contendent() check that we had before. And for memcg slab shrink, we unlock SRCU and continue iterations from the next shrinker id. Signed-off-by: Kirill Tkhai Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka --- mm/vmscan.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 1de9bc3e5aa2..9a5a3da5c8b5 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -204,6 +204,7 @@ static void set_task_reclaim_state(struct task_struct *task, LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); DEFINE_SRCU(shrinker_srcu); +static atomic_t shrinker_srcu_generation = ATOMIC_INIT(0); #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -782,6 +783,7 @@ void unregister_shrinker(struct shrinker *shrinker) debugfs_entry = shrinker_debugfs_remove(shrinker); up_write(&shrinker_rwsem); + atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); debugfs_remove_recursive(debugfs_entry); @@ -803,6 +805,7 @@ void synchronize_shrinkers(void) { down_write(&shrinker_rwsem); up_write(&shrinker_rwsem); + atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); @@ -912,18 +915,20 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, { struct shrinker_info *info; unsigned long ret, freed = 0; - int srcu_idx; - int i; + int srcu_idx, generation; + int i = 0; if (!mem_cgroup_online(memcg)) return 0; +again: srcu_idx = srcu_read_lock(&shrinker_srcu); info = shrinker_info_srcu(memcg, nid); if (unlikely(!info)) goto unlock; - for_each_set_bit(i, info->map, info->map_nr_max) { + generation = atomic_read(&shrinker_srcu_generation); + for_each_set_bit_from(i, info->map, info->map_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -969,6 +974,11 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, set_shrinker_bit(memcg, nid, i); } freed += ret; + if (atomic_read(&shrinker_srcu_generation) != generation) { + srcu_read_unlock(&shrinker_srcu, srcu_idx); + i++; + goto again; + } } unlock: srcu_read_unlock(&shrinker_srcu, srcu_idx); @@ -1008,7 +1018,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, { unsigned long ret, freed = 0; struct shrinker *shrinker; - int srcu_idx; + int srcu_idx, generation; /* * The root memcg might be allocated even though memcg is disabled @@ -1022,6 +1032,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, srcu_idx = srcu_read_lock(&shrinker_srcu); + generation = atomic_read(&shrinker_srcu_generation); list_for_each_entry_srcu(shrinker, &shrinker_list, list, srcu_read_lock_held(&shrinker_srcu)) { struct shrink_control sc = { @@ -1034,6 +1045,11 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; freed += ret; + + if (atomic_read(&shrinker_srcu_generation) != generation) { + freed = freed ? : 1; + break; + } } srcu_read_unlock(&shrinker_srcu, srcu_idx); From patchwork Tue Mar 7 06:56:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13162870 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA5E0C678DB for ; Tue, 7 Mar 2023 06:57:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8DAEA6B007D; Tue, 7 Mar 2023 01:57:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 88AFC6B007E; Tue, 7 Mar 2023 01:57:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 72C9B280001; Tue, 7 Mar 2023 01:57:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 612356B007D for ; Tue, 7 Mar 2023 01:57:02 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 3CB4EA0A1B for ; Tue, 7 Mar 2023 06:57:02 +0000 (UTC) X-FDA: 80541195084.11.1223D80 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf03.hostedemail.com (Postfix) with ESMTP id 6F1382000F for ; Tue, 7 Mar 2023 06:57:00 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=PS4W+FgT; spf=pass (imf03.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678172220; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4ra+jS9qPdJDcLL3tCNVjDgmNt8q5h7NH0ipc1XhyKg=; b=T0ikg4rwBhq1CUb2ytk85hBCIjAmD2yMD/p7kYSJMppKpk+e/0U7lCLpJAH6M5k50yPhQu Sy8hA5vV7l1+4xb2oUf2i+P9rXreuN481VsCDvlNAybZJ07aBq4U7uYyLEaeRgvybUc/O7 rZyjltvTd9XIebCWrbMcvSg6jmKNCrc= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=PS4W+FgT; spf=pass (imf03.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678172220; a=rsa-sha256; cv=none; b=oo9fPSjhiNsG3bWlo0la7NbgJ8nsGvgYitzktZShQ/nMH7CzjwTs/IdZVRGHbtcWXyhGuo U75/vLODupEs2ram276fCmru5oMHaxBLNz/eJliwwjgL129NmF3GZQrgvgx2ByOZ+ulPd9 T12TVNcd6OHuTK6fx67l+9HjMBfJlHE= Received: by mail-pl1-f169.google.com with SMTP id x11so8554764pln.12 for ; Mon, 06 Mar 2023 22:57:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678172219; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4ra+jS9qPdJDcLL3tCNVjDgmNt8q5h7NH0ipc1XhyKg=; b=PS4W+FgTgUCjDbGtg+QwCoSMgK+PZFy9BY2F1ZThxrjZVSL14yAr0wpqGWIeHjNUI0 MjcnS5KpgGlvLOSQCUAdkAjQZHluL/aF5sSdv42dRLNv4KIIzh+aWWZDjamM2HQABfg/ ke15HkP04BOxxAYCBWXNW3bEgB1hjXjR2Uv8/YCxNcS8GL93fqqgDCLTaxFq1I/kUgHl LQ/t9QNRUeTUcPEAjMgUEh4v5iqcEyjJI/gJPK3velHfAXIwIh9N1aKawGtD5vTgD5rp 0q/DlCg3cZIA9QSknjY+zItjUDbIe0idSmifezJtWhrwh7Hi0S+Ju/qHW2bubpQjjP41 H5IQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678172219; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4ra+jS9qPdJDcLL3tCNVjDgmNt8q5h7NH0ipc1XhyKg=; b=D8AcUvMJseHMLt2uCwEMTur+sQJsaSXGtE0/dzuSZcZlJtlDNQKus9YNYzidGwz9vw tHJQ8y5nd8rpdDw2V6szmn1ZoWLF5MaCVV5Uju18vlxMI8BuQe8kpSadZhiRRZTYPz30 jT499bAFc2lT6XO2ZXGkJyNSmC0P34GBevIbI5DB48duo9MPXC5IP0bp55HXXqTmdEh4 LD5hxlmrbGrepyNjrLbY48WED/2P5qOuGGrBJXFDF5JVzYJ+f/pSLEXOGZBE9HQ9zEbz 6i8Km9ImRuVGsR1e+u78BECNBD9JRzeto3o3LyfsutAYbX0i6BzPz0bmCup7GPxWGayi 6E4g== X-Gm-Message-State: AO0yUKUc1QUiLMKgFfgx1eFgOewKMAmoKs/wATwMPijRYIsljUxczGlp m41EVkXPE+C/0blUAk2IH/8bnQ== X-Google-Smtp-Source: AK7set8QBMQzgGkmZ61qFh+1VNZI1JIL768E7/TAERxebGQIVfkBE4dBJCz38jB1/9PqBut3D4RTmw== X-Received: by 2002:a05:6a20:6914:b0:cc:4118:65c4 with SMTP id q20-20020a056a20691400b000cc411865c4mr14603250pzj.5.1678172219371; Mon, 06 Mar 2023 22:56:59 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id e5-20020a62ee05000000b00608dae58695sm7230854pfi.209.2023.03.06.22.56.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Mar 2023 22:56:58 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com, rppt@kernel.org Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v4 5/8] mm: shrinkers: make count and scan in shrinker debugfs lockless Date: Tue, 7 Mar 2023 14:56:02 +0800 Message-Id: <20230307065605.58209-6-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230307065605.58209-1-zhengqi.arch@bytedance.com> References: <20230307065605.58209-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 6F1382000F X-Rspam-User: X-Stat-Signature: dzeqpsi59kux4q5wc6np8ipg8d6f5gxi X-HE-Tag: 1678172220-295656 X-HE-Meta: U2FsdGVkX1+tsJxsIgl1kY5+NgrPUWcJ8Wj+ynpNiK8Vxs4LksxeU+3jc7gkRn/iAyejPoOY42uSNFfdZthcQzNQXwAJ2j1SUqvu+NHby+oK3jW/QhgYSqD6ZLSl2Rz5s4MPwkGHBViOQpv+k7Nf2IP0l3LUoAEY/9DS31fgpHY8KG3KilW4baniV3GxP/fZ509hE3/TWKiPtg/CNDTENZB2SCe/7xxkVt42EDuFNRm50Rry9IFDGDb7iMoQEYRuesfy8EqBi0B9VNlG+fhH9NldZRvtIn15TuFT1+dVChptZ+OG2y8PO2IqNY2itXbn67NGfmKLiVRjP+A4RN3PqDyTuRSyQYjnCpvlm4Zf/ND68/DhQIniheb9TYPUXVZFrfvFGi+BXmceYJnV1X9SV1qDr3sibQzjg3cOxxmqIzpwWlP10wt1s5qXBlqJfZOSUW8tWuApLQPi5lljELXr+xa7gIVTyF2XOKu5lZS+fi3zOtNw97atVyb0QcPpDvIzksfGBOc4mhtgmHapMpyH6CTNBvDfFlJ2Ec+6IxEFf4Bb+AOTqaC/sD472ZhTfsl5uixWuigEPaNe7+Rkob2ijnZ/2tBt+TxC9z+BaXxMRSUHRIFFClBj19TpJHuu0l1VF1za3EnDuCr0X5m+Q4fTL+KzS5CvEh1vtlHWM/e/VFzdEtLaEQDiOKYC8NcP1TCBBDbuWSR17nQKE6kexGpK4/eLl26C/AxKoJmWdhOdN1Gfmu7gIuKWKHotvVFozhpTY1rdhOV1MypsbvWz/3DYQsP44+WW4xF/yCS+qajNwET6Wxb1EM6vYNdzZiUscmfbWHP+z/CPsjLCwly04p/QJiQeUtU74aGy5TtajW3c/R8aSUU9K9Kk6vXRjUxKYhLbVRCtphE52sXZFCxDKRbsUyV0qo8q5ZUwAg/jNkSimxKGP2Z83hCNCYcXgmm2wJUhqEPtXyQd4oHOZMnLCj8 Gck+ak50 oyXq1aiGPmlAFgMAgVsvHiIRrsy0TKpb7e2sl9eeTdv7IE9xqNsYP12yLr8P62zyzAvb0dE8qR6n2hzTlCa61aCcXadhWDoemC0yA8f9OOjVmI6RPHnXe+jYmIw3s+xUVwbyZypvrPpTx/X/Kv1I4LZfx5X9cT3uCWBZTjCw+8kPc4YaWDsn0C9OyehYK3x0AtNOeLh3OzjeXrcjeIHz9ZpqYbleXwdqX9a6DRd8cYxrByHSALhnDrgfYAWKkBGwpX9EShgu1lVVWJardZ15YzFjP8xfs+KNrm7fx32fdpxe/jTHr6dvtQNq6zT9EHpx4agQ/bprIOR9sp+0Crtu+gQbKSIoVYWKe1SPD6O19YNsDNFZ5yAzcudzHsMDd+9gFvY1WnQsReWt6Ha4piZBhdQl0VJfiX5pcX+weanTyPWxXbMI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Like global and memcg slab shrink, also use SRCU to make count and scan operations in memory shrinker debugfs lockless. Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai --- mm/shrinker_debug.c | 24 +++++++----------------- 1 file changed, 7 insertions(+), 17 deletions(-) diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 39c3491e28a3..6aa7a7ec69da 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -9,6 +9,7 @@ /* defined in vmscan.c */ extern struct rw_semaphore shrinker_rwsem; extern struct list_head shrinker_list; +extern struct srcu_struct shrinker_srcu; static DEFINE_IDA(shrinker_debugfs_ida); static struct dentry *shrinker_debugfs_root; @@ -49,18 +50,13 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v) struct mem_cgroup *memcg; unsigned long total; bool memcg_aware; - int ret, nid; + int ret = 0, nid, srcu_idx; count_per_node = kcalloc(nr_node_ids, sizeof(unsigned long), GFP_KERNEL); if (!count_per_node) return -ENOMEM; - ret = down_read_killable(&shrinker_rwsem); - if (ret) { - kfree(count_per_node); - return ret; - } - rcu_read_lock(); + srcu_idx = srcu_read_lock(&shrinker_srcu); memcg_aware = shrinker->flags & SHRINKER_MEMCG_AWARE; @@ -91,8 +87,7 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v) } } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); - rcu_read_unlock(); - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); kfree(count_per_node); return ret; @@ -115,9 +110,8 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, .gfp_mask = GFP_KERNEL, }; struct mem_cgroup *memcg = NULL; - int nid; + int nid, srcu_idx; char kbuf[72]; - ssize_t ret; read_len = size < (sizeof(kbuf) - 1) ? size : (sizeof(kbuf) - 1); if (copy_from_user(kbuf, buf, read_len)) @@ -146,11 +140,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, return -EINVAL; } - ret = down_read_killable(&shrinker_rwsem); - if (ret) { - mem_cgroup_put(memcg); - return ret; - } + srcu_idx = srcu_read_lock(&shrinker_srcu); sc.nid = nid; sc.memcg = memcg; @@ -159,7 +149,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, shrinker->scan_objects(shrinker, &sc); - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); mem_cgroup_put(memcg); return size; From patchwork Tue Mar 7 06:56:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13162871 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25107C6FD1B for ; Tue, 7 Mar 2023 06:57:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BC0F76B007E; Tue, 7 Mar 2023 01:57:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B71256B0080; Tue, 7 Mar 2023 01:57:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A390F280001; Tue, 7 Mar 2023 01:57:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 967FE6B007E for ; Tue, 7 Mar 2023 01:57:09 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 6AAF8140C8F for ; Tue, 7 Mar 2023 06:57:09 +0000 (UTC) X-FDA: 80541195378.30.C610303 Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by imf11.hostedemail.com (Postfix) with ESMTP id 9978040003 for ; Tue, 7 Mar 2023 06:57:07 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=W4VHauOi; spf=pass (imf11.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678172227; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6RpTkxvvrfdHRVC7O4Q9cB0zRrVMaNNv+4vslhRNXIw=; b=rv7VwmdvwSlnnC8wZAw3vW73mPVS2K0quoP8YpybWD8mnOnUunU/sVWjOzC5XjNAxsqLpj afiB0JkLVQP+tYwc8rvg2bpTYxhSHLGr4m1Ay+PdSDC+xn5kZZeDVBm6ESx6ikpDMWSw2M /kyK9SCTKr+BjT/ADBG1QJH4KlZdRqs= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=W4VHauOi; spf=pass (imf11.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678172227; a=rsa-sha256; cv=none; b=Ua433X8CSI5MEztpNcqacr85WpgtyhKNMxaUHfiuegqqJ3zW1iyMXLR7cT00lC+BEZEnml NReat9QHsaoIreNwn9CIRsUzQ4ZQ2osOPozuRJNZ3JmeEWkCLRIY43sbGv9qLQ/C8KdIZB qM3paKCS+sIpnBDxm6hM5nbcAJwxtCE= Received: by mail-pf1-f179.google.com with SMTP id bd34so7419663pfb.3 for ; Mon, 06 Mar 2023 22:57:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678172226; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6RpTkxvvrfdHRVC7O4Q9cB0zRrVMaNNv+4vslhRNXIw=; b=W4VHauOiBpt34sXN0NGeRnCvFFOrK6bGxqZvOkEMVUkrE3vG6k1a/43ATXBWF9oL4D HYBUo7vHZe/W31enhNJ7CVLqdmu5Y02HPFFflxIozl0b/QDd6RN4k6EwvF+awGPNHWe4 Rx/0HE2KR+Qq7m9hXa35Ea3oyuplOTrjAbd44mvSt3lLuSOT9OuJKMHLt7IsuNAw2GaM bzcYuW9uMvkR+Z6LM4ysopbsVwXnebzGgDNnTsrwb/NRdLlO7ExqCFm1QRYlGeJTOVKA QV50Vul9k2ren/2UFSJ3/jtJDhKNqxK+2HgjtMnrdUW17iEio32nEMoMmRRjN2BUDePl lK1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678172226; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6RpTkxvvrfdHRVC7O4Q9cB0zRrVMaNNv+4vslhRNXIw=; b=SpaEj2/xBBnk0KPJU02Lk7crtFg0RFvdthlsLNAcjGdy28O38onkAshc4d7HHHiTfF hsIWKVEhqXuj1HC61goSaGaLY2+Q5fpx/qXstt533X4ZP48CloQ20HRiUSHKXJJWwK1P wQS4oBoPT5UizScS87HCbolJPNYDcH2aZlOOa4JM5v4QcskZodi+MApdqEBqBZPlppTT 8+el3kDtEc3St112BqjGWXJU02Cov8oE/z+yHpZYT1BS5cWoSb0b1Kpt046CKc+ZSvm7 HCg3DzzSqjjr9sknxYDxzio2CZTj5Fhs/5jJnCO4Kk8wjcxdU1iO3WAy9p5sVwejrmX4 Ni7A== X-Gm-Message-State: AO0yUKXeyV2SusRjJVfn7yl5IbFcfqU0vWM1lzm/SELhP0qA6611G6u9 t3V0dzYWpsXHD3BEsw90rQtMDg== X-Google-Smtp-Source: AK7set+ub7UgmiK7qg8lpFjlj2RQzhmm/aIy9dWEY6nXtBdGfq78UWJjhko94G0h2+ASvBEfmacZXA== X-Received: by 2002:aa7:8892:0:b0:5d1:bbd2:7604 with SMTP id z18-20020aa78892000000b005d1bbd27604mr12474224pfe.2.1678172226558; Mon, 06 Mar 2023 22:57:06 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id e5-20020a62ee05000000b00608dae58695sm7230854pfi.209.2023.03.06.22.56.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Mar 2023 22:57:06 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com, rppt@kernel.org Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v4 6/8] mm: vmscan: hold write lock to reparent shrinker nr_deferred Date: Tue, 7 Mar 2023 14:56:03 +0800 Message-Id: <20230307065605.58209-7-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230307065605.58209-1-zhengqi.arch@bytedance.com> References: <20230307065605.58209-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: ks7hh6dz8ftmza4465apo9ng43rwiqg8 X-Rspamd-Queue-Id: 9978040003 X-HE-Tag: 1678172227-100868 X-HE-Meta: U2FsdGVkX1+nJyN4SfCSEAzOCHIhqK8w+T16ykHXNLMnQR5L6htjUO9wc98WogT6lIKB/OcthCRTMUY5AflB4Or6tFq1H7KzMCud/Y4QVTdb5sknA2ejJNKEOs7kazr7yYJMewiBkl1DwLhAMgnHzbFw53/KP2ZBvhoRSdAVM6VuIQIbQrP2nTKqGcSZMAMsnt/LvyzejmdMyT+Icyzs7mKTAZ8+G7poVH3CHdYGFctqG9x0T62nqtDI1sPiZNemULmOHqtMR/KwFmpt+gnwuoNYX7btwP0M2vc0s2FR+Zfp/5zl13CjXTIiNO/EUTfaWN25/nrcbXqoD81lQqgWeDL2b9tUQeQPVw68XM9V71+90MWlBh++ALyzHTUa8rtxs37aMrWGpyqmgi+qIlOiAP+c1HdTqnIemUdfuHBSL/UEVl6jF/wKgG3bHn5mfd+68O/1cVNpVRiAKnpuBos8M9eFYgqOWyyg+bX/khFhShSeS3HW1FDx4knnOphSMPUz/nAdFBxEECAbZ/BEOM85uxdcaE+GfJ6k8fISmesikQvu3UjmgIhUP/a3hMIWKftvrt7mf4n50xwO3GXMajkVE7xr11nDV2c+glAEKt013LYR/kdgkOahlDTedrlJEdg5gdc/QtjjMhj8sR8+hSI8cmcauCiYWTay6rp1Nqm1nutal4j+O08YzU0ExGdYnBjkxr399ytt74Y47C4qfKdGfPD3xAfAvEI0JTlBqo3RS6iXwWTXjk2/mjahNGGlXToUCxkmuXceetHg4Fjiq/ygXXjSAXF3pUUbrLaLuSr6SwAz4ezg/FyfnkqSOSRn397mxoqUWD0pz+xVPXT8oV1XlfpTbU9/Drdwd2jkvz1SwFFoB4xN+TmmVKadqTo2FHkkQ0Rd23NubSM6cvJ8f8PaGH5Lg+Zmg2mF25wuDV9/JnvKUakzrqH61cPeK9iYAkk6U524mQ+40fhModacxPO imhBuz7x 49jQ1EAEmZC/ACTKdVHKjiDpUZPVHknDi0VQiW/1Cs2SxsGgpIUWJQJcWWwov5qjcOfG4ri6ZYilrFF6Gyf1MqYrd05HNsiYhL3SDMcS6SfYCH0v0wEJxzVtfVUctb/7EVQJ7u3S6Ig8ssh0+Fdot5cLNLxXYuKwocuCwi+qIzDUZZTbS+zqY1wV539SnMTX3Cr9P1kP6wyiuOJdfhJoJtraWz6oVoD2x+K3n+PqcJFHHrujsd2VDjYN3k36tnrpvyjNVzJlMzdEvFk1T0QX/b8LE3njauQs92K+4mv8FC/JpszKG48UAwSbs6X42YFw1S9c3ahfPsTeABzzBleNM95NTeiTwxgmfLCwD0gYk03SguHPkb/Tft6P4KVZ9c5XknAFMXqHmcLP3Xk+OHn5T14nDayCdzACbPKhy2KQ/6q7FFh74EETmNUpdHj0YXkMKc824IyueylspPicpeSCrWLxce5PJT5GpnOsB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For now, reparent_shrinker_deferred() is the only holder of read lock of shrinker_rwsem. And it already holds the global cgroup_mutex, so it will not be called in parallel. Therefore, in order to convert shrinker_rwsem to shrinker_mutex later, here we change to hold the write lock of shrinker_rwsem to reparent. Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka --- mm/vmscan.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 9a5a3da5c8b5..7aaf6f94ac1b 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -451,7 +451,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) parent = root_mem_cgroup; /* Prevent from concurrent shrinker_info expand */ - down_read(&shrinker_rwsem); + down_write(&shrinker_rwsem); for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); @@ -460,7 +460,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_read(&shrinker_rwsem); + up_write(&shrinker_rwsem); } static bool cgroup_reclaim(struct scan_control *sc) From patchwork Tue Mar 7 06:56:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13162872 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71649C678DB for ; Tue, 7 Mar 2023 06:57:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 076546B0071; Tue, 7 Mar 2023 01:57:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 026516B0081; Tue, 7 Mar 2023 01:57:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E30A3280001; Tue, 7 Mar 2023 01:57:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D22BC6B0071 for ; Tue, 7 Mar 2023 01:57:16 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B548CAB135 for ; Tue, 7 Mar 2023 06:57:16 +0000 (UTC) X-FDA: 80541195672.13.6E6328E Received: from mail-pg1-f174.google.com (mail-pg1-f174.google.com [209.85.215.174]) by imf27.hostedemail.com (Postfix) with ESMTP id E32DA40003 for ; Tue, 7 Mar 2023 06:57:14 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=MyoQeF+Z; spf=pass (imf27.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678172234; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pKI1kuAHKypUXeareR/09n2feCgpJViSdOybhZI+EsM=; b=C/pJ5LWaspXE1VHM+GQgJupWc8bLbaWZmm8tWFQVzji1ly6PbTldduXigPDTf1VfoOzKcI dI9LQSKEM9flLUREO+loXAaniTO84FyXscS3vH2T09T9/l+9zmnZ/5yyxcHDIZFVyK9i6w zlbIT64rPown3JPMGzfU61687tImKII= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=MyoQeF+Z; spf=pass (imf27.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678172234; a=rsa-sha256; cv=none; b=af985lj4jeySWggiHUbTlffaRGJ6bhp0baYBN5o8wJIfGVdjLhFtE/89LvtCHXMotQRFVk 66Wlx5406xy390/L00dsV4zE+HOqrEDccmI3nnjSpJ5tZS5eqqrLjxC8bjjxCVUs1/Qvil XE45WlXMyX8ih2sJ+hgVvEYoWquW8/k= Received: by mail-pg1-f174.google.com with SMTP id y19so7013899pgk.5 for ; Mon, 06 Mar 2023 22:57:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678172234; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pKI1kuAHKypUXeareR/09n2feCgpJViSdOybhZI+EsM=; b=MyoQeF+Z1Ozq9bE9/PbK9injww7cXqExwhY7rvSLxR4g/4O+Nfc9fGmLGzFYJMpAHD Zob+jZZ+kbE9jLOG+OEUynRAhk/1wDg68aB229TqJDeviGbMSvnk+L9gn1ZEZC2xzZew OV+vxUHqrr+Tv0KeVOcVBlszBPmxncShiPqOZhplHv4c4JKf7I0IqXVPOMWMvDJaZt4R LtsYlahwTiomz3DM5CScou3sBZL/2DaB/HRpvPb940m2H7IkHsGjeco1KuyTRwCNdWya 0gnRlmOrxjy+r8Q8lkkhv4vWG6f46ZgtO0AxZHLWg6LCI26AiYHKnM1ulIrPiJzRhtno V4Qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678172234; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pKI1kuAHKypUXeareR/09n2feCgpJViSdOybhZI+EsM=; b=rkKXlNo/qSWJXl78hSkw/spoLF/1NLdYk8rhBg+nyR3ftplIIxrBrgvUIIhrLEI8/B 1xRakpGc1Z5/jJeTyUDqjyFnwH5llRYqHg0p0aNt+Euzai6lylM+XTAnL/S7U8ukOsZ0 fuFlL6JCVWmC79xPMyVE6BvpZ5nfEZUsCJ8tFYiz5B4ApcMsTwB/xzQMpYvSdbHfztJq OirYHnanAkvgCDJy3ou3RjCHgfUe/ocgPenABSnA9tO5PbTfp9E7ausHfhEb2eVaOYsN aRfCl5tAtjfQIglGyuNnZpfMO6L+lRAunVujslxt6fh1evMN9RiYkXAuqOwLjiTrYCRi 5iYA== X-Gm-Message-State: AO0yUKUohduxBxJ/F0EhN75RYxha/teDBXDs0WtIboUaU20rxYq/EzIu 3FHWDNVt3HOJYvHQ968Gt006Pg== X-Google-Smtp-Source: AK7set/l9M6WgXBhpjojFQaIgGb5t8qipqn824NRUYVF7PhtZRZwJeaPHfCe3DAy+Lx024T0Tr2DWg== X-Received: by 2002:aa7:8892:0:b0:5d1:bbd2:7604 with SMTP id z18-20020aa78892000000b005d1bbd27604mr12474542pfe.2.1678172233918; Mon, 06 Mar 2023 22:57:13 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id e5-20020a62ee05000000b00608dae58695sm7230854pfi.209.2023.03.06.22.57.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Mar 2023 22:57:13 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com, rppt@kernel.org Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v4 7/8] mm: vmscan: remove shrinker_rwsem from synchronize_shrinkers() Date: Tue, 7 Mar 2023 14:56:04 +0800 Message-Id: <20230307065605.58209-8-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230307065605.58209-1-zhengqi.arch@bytedance.com> References: <20230307065605.58209-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: E32DA40003 X-Rspam-User: X-Stat-Signature: od79azyw1pzqd9z7ouu93uskkxxb8ou5 X-HE-Tag: 1678172234-874086 X-HE-Meta: U2FsdGVkX1937o0jy3v5XOdv37eUc2QBnViAlpwMU3/hhJko7ems21sv5UJ/k0BmsWknNI05jkFr5fFvdk9PoL+q36opyXAQPYieMeNrJTFeJ8SIwEhlQFRsW4oe9A1woPFfbZBDp7VLTCNScPprNAn+eWYW4Hmchk9spxpRKKi8UsNlIs0sQqbpm1Kb/XGMAhKN6JUViNfj4NAMEpr0J2d8ZBwEEv9uQLB6P23sDU8osI5cXUcGjkgPvXXzjNUkgmxFNP/iUr1VLaBfZJVmenFonIJ4wiiUXcnLZHk7lsyDmVY4tS247TD4MRz2hQ0xcjaR1bEd+TAJZTZyZiysIJR4w98SdNetAiB9zpIJOqmeNqoFi+4LHqNYflIrn14fQVqSYicCSPwFnGpT3PsxwfmSUtJlUIWTYqXnyWutLu3sVSqnmB9nOwmBkdu/6YNIeqA8lXJsVAfPNvxKS3pOPdqYpi2BVY/qsFEky90EIBFh1vvo6Ti0HkXe9kYfUdnMac1XKkM2aJh4xKHLA1ohDSBC/fTxyCL0us97dNtcNGtu2iDby0O8o0YTAmO/oQ1iNusb2ReUh2mH6of29cf34Ix6GvCzcH0kRAhf75/py8gQz7sAKYWKUXAO/WnIwNcYERL4F8QlFp3hozRo80N7w9S6ZLJXjGoRK/LrPqnZo/foj+OdN8U/a+Oaja6lzH1MiewikjKX6rGs81DhYbeEqOChpv5aG+6fvTIIKMPGDtffErke9mxCsYnb29YX4wMykKFbiQyY00Zg9C+2rP4O5zAC+020krARcZ7TQRIJQvOiP8pjn80U4cNlU85s6zulCMdmvn6x7QxV0dYzmIBBKrvztq3cJk3KqTvd35GOfbS5c3eAu8v4qpsOK52kWhqzQ0C2Xj6ioc6gHB/9Ou1wcd95tfaoqyN/X0XdU4U8m43mPKzHY0XCpHMXclt9BiuZG1Hid2/2nUabsgJRf1O PWvzXZvm Yu0ZEFPilOxGCM+cGMnnLcZVh6CEjggI7wXJQFIGNicX8GJ6IeDDOgcjFZl2A8xXSSn6sGflCw86+VraDasYfj/0mNxk6pqHUa5IbE0vyndf486Vp54KczveF/9ld9KdHUPihMiX82aQ4hF9aTYQVq7s56dNDVBu7gauZIBtIPEYro0t8F5Eef2i1pqjcTUV60VYt0akEGeGuNESoT5bMuQ7KumNo++dxWZNSGhLqBMBuYb8bTUpPcSu/U85HezIEtT23ZmoP6vQdiOk4kJX0yCju+rqhVlYh0IT0ddy+HaUuOoc8wcDdPwf+AbUfuM6QKuO7tqOJiOOfOc5YkkKFRNbsx6iv9xCFxyS+FZzdAg4Uw7OEofbGj0CTF494TPd5dVSk6xEiQeNaPjkpHs0aITXm0zaI9qlFd4g0lz1iWOp79cKmvlv8+3AS4r2xutXV23gBM19MAXyxrO5aO9+FRSdV48FmuNOrzy8F X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now there are no readers of shrinker_rwsem, so synchronize_shrinkers() does not need to hold the writer of shrinker_rwsem to wait for all running shinkers to complete, synchronize_srcu() is enough. Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai --- mm/vmscan.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 7aaf6f94ac1b..ac7ab4aa344f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -796,15 +796,11 @@ EXPORT_SYMBOL(unregister_shrinker); /** * synchronize_shrinkers - Wait for all running shrinkers to complete. * - * This is equivalent to calling unregister_shrink() and register_shrinker(), - * but atomically and with less overhead. This is useful to guarantee that all - * shrinker invocations have seen an update, before freeing memory, similar to - * rcu. + * This is useful to guarantee that all shrinker invocations have seen an + * update, before freeing memory. */ void synchronize_shrinkers(void) { - down_write(&shrinker_rwsem); - up_write(&shrinker_rwsem); atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); } From patchwork Tue Mar 7 06:56:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13162873 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1943C678DB for ; Tue, 7 Mar 2023 06:57:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 643B16B0075; Tue, 7 Mar 2023 01:57:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5F2546B0082; Tue, 7 Mar 2023 01:57:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 493CE280001; Tue, 7 Mar 2023 01:57:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 394C96B0075 for ; Tue, 7 Mar 2023 01:57:24 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0F3CC1C4BD9 for ; Tue, 7 Mar 2023 06:57:24 +0000 (UTC) X-FDA: 80541196008.06.9A530DD Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) by imf24.hostedemail.com (Postfix) with ESMTP id 437B918000A for ; Tue, 7 Mar 2023 06:57:22 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Rz+OlQxX; spf=pass (imf24.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678172242; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JEwsIZrv65iqCvc8ubXlIP3Z3XNAqh1J9iA+XAih3cE=; b=V/mXHrMYW0wC9da1OmQdlNNbDI3a5XQydY8EZFw5Dzplb9QO/Ygk5IL+gI6AUH1Mpkk69b mKNxJZM7DwPkDs7I5s1CbOaf4ICtIf5O8nOlS5VJ9rkF4PXD7r0XM0d53N+hHD+L79/zBY VEmBosTamRN0xi9oySyBAMsYD4gyW30= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Rz+OlQxX; spf=pass (imf24.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678172242; a=rsa-sha256; cv=none; b=k/QLj4VA+Zx7vXOPqqxhi6KB5OBndk84E/R7Tos35zc+SvJHHSMOjuBJpelc/DJW/OxTAz 3RNnHYvO+YSloryVCQ54KDIFNhSPpTkrB0Gr399i/Nn/XpSfaICr4z2bizf855e2X3Xckf pz4KA/1FdY82m3UJTB1psQIOO4LRxYc= Received: by mail-pg1-f181.google.com with SMTP id d10so6997494pgt.12 for ; Mon, 06 Mar 2023 22:57:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678172241; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JEwsIZrv65iqCvc8ubXlIP3Z3XNAqh1J9iA+XAih3cE=; b=Rz+OlQxXJ+H/lFVEkuWxvVRfLx2yy/2tPDqB+C/WLB56VLSKXLTwwrE98jNSXgcCeH Wp6ocsD+lSOvkeaVrQZW5FS2yg4o9s1dz1qiHMRxFASBIpc2JVftANvxB++8sPANbxME twinknB2RKFoFIRTXehrwocBnnddLU0nmhVWDA0ZICfv6A2YIPyUIpqw70fmmMDpC9LO go9gjPkLef7kHYIg+lwI+6Lb//VWhFFJMJNWrvhgTYNSkuf3AGI3BRtPFYG9oGd6YtoV hEYb4BtRHKy7EJ9lOblKA1RqHDTg9Y++t4SZN/OXDN6aiH/uovJaGTEGR8hEEHJgMKni koTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678172241; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JEwsIZrv65iqCvc8ubXlIP3Z3XNAqh1J9iA+XAih3cE=; b=LKkdhIWKqOyq6FycvvvEjTR7Ent30NMlMHQyNHrxnguwnuaKkiooyVJgwJEjECWxBJ TbthVTvjbxlmC8E+EHMMPmV6MXr7kC6ldwKn8Kqg3Jkk9lqwHk+xqtgP70GRF30ht7Px /1YLuPBpapUh20RZkwf2evST08vdENot1I4k/cYZhV2P0k/l0GSIjqhZrUJ65jrh4zDL QDHbGt8mRlra0mwF9UcFgy/MfJHbKhIUsUiwIgUfWRzF/VxNV4qwCK63od9Wcb19Bd2I jmRaHuCvUeldasUlspuYrxCs2eNjAdbY4nf9+9NCPwNQdTcCsJWrhye3FhHlSU6E22Sz 6mZw== X-Gm-Message-State: AO0yUKVVGYnU8PbT4Cja3msbQawWQpfBaVIDriBNJ1BdaXrjJpNEL3BU 3SlxC0MPzyctFr6vzXwXpzZZEQ== X-Google-Smtp-Source: AK7set/4ILAb+soAd7SXbXVfua+6ZlNnI8iHb30j7PkgDES5mxQFn1us7VDLALBB1adEQpVhd+mogw== X-Received: by 2002:a05:6a00:1d8c:b0:61a:50f:cad1 with SMTP id z12-20020a056a001d8c00b0061a050fcad1mr9376180pfw.3.1678172241246; Mon, 06 Mar 2023 22:57:21 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id e5-20020a62ee05000000b00608dae58695sm7230854pfi.209.2023.03.06.22.57.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Mar 2023 22:57:20 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com, rppt@kernel.org Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v4 8/8] mm: shrinkers: convert shrinker_rwsem to mutex Date: Tue, 7 Mar 2023 14:56:05 +0800 Message-Id: <20230307065605.58209-9-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230307065605.58209-1-zhengqi.arch@bytedance.com> References: <20230307065605.58209-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 437B918000A X-Stat-Signature: tygqw5cedhwwhwegdtd48iikiwcehwwo X-Rspam-User: X-HE-Tag: 1678172242-306017 X-HE-Meta: U2FsdGVkX18bFwvREVUr7xyN3LQn27JE9wpH5F9jLqVFTG6DfXn+bWAdkYQnyd9wuGagzt7Yp3CPmTrvVCziX16IRM3PHooWNukUZw+VoAD4Z4B6oo4JqXdGUyE5V1qihQ9SNzSzx2B/EV5EG1VwqGmUCcDpqUSXnHLRpc7ict850ucuh9k20qrWsizhampUrIjgJ+bm5PSBOVx1CZl6KKh9F6PotiPKs3wWU8fxsXBz6ynySH5r9Z8SQVjZmsscE+pM3S31GcoFPMQjrOUisGDqR1xpf66lH3tB7Jl99V7hZmfsOZUpRj3JsIQ2Uzpp+h7RrfG5DiqLooYvHFZBo+vnyFNpkt/1HYAdx0a2FHVKjPi3QOiaUG9oqQzvX88zdx9222YJ9O8BmaRX/Blb5SQXo8mWJX1RNbQkUApoUIEZ4IRY/4jjisOvz8eIVDva6++7ijSr8MZdjJ9vigpF80QePLzy6ZxA8B08QDL8DyfWOJ0WGcKxqLzahBZNavu/ieV8dGtA7s/ZiZ+CsqsBm+LMTc6OuG4uLLnCne3VOn/L9T1KSDUg+e9IjJLJnh7Qd8noP22wRR20e0wneqwp1aGvyTStEEI/goC6X5ojZTDISsgrQuiTz8XdV4ci1O3IB45g7FtCy/01BY9nipbMzWiKvqLUjNLYAHitbH0lCsCSx+SWzHsGSgTBYrLW9umBzJri2ScgCvNC6EMhD5ThaZiA9Dibqcq1wpUWToLkI18kLU59fpWTib54Yh1elU6zAEM+qrAuExj1FwLrr2lX+c59y585Uj0gZ7NdZrjzP9SkeSVAJ4zeQDdw/lar6HBm9Xt1jI5VSgbYvjq8F+SYI3P0ykBGepDFKU/ogWLb0tITusBSbyKBjcZCPaaPJmHWDL1pKJjDH8jI+cHWqV/qz/Hiq3/WtgAcvjKAhVUaWmA+MsDhXZdaNJEhJlQYq1+8aJ8h2T5Tk/Ra/GWs0w+ mygqm+Vu D8NMMI1NwgIxaE7Z8WDblmF1Kl8L3xCm2nNbQ7QZ8apFUHcJ2QVnWeWd/ioNjJhXL+BiEeX2DxjOOB/MSjJAaAbuFu6VgSHXM6z+M1uKUputSZ5qMjAWhF8t53fGg7uQ/iOsXikyxeyswzCPXw4snWUq0MsmG2smvPaDwABVBVn6J8MDjig7/DdsZtD8tznYf1SpVjbsTnIRsPQqZ2er05N3IYqK5rFqDv/PN2Lxmbo7br98Yo0I2b73bbCnWLFuhD3jYF07BeUGdj25dAvJbhPwborF/YyDBBfZftqspshYSOV3TScywHFzjA1JbZ+WFHJBandj36vLw+jqTQTrY3jdNO3bkcy77mRyIcN541g71St5vX8XtCiZ+s0UYZqmN3fvMRuLAdQgfhmvAICEXyxBM/pqDCzIWuCUQuzQSHKme6B+iNqeiFpFZzRltOYL5FY2j5GSn8WHPPIT4LPPVt7WStJzwyYZknzDI X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now there are no readers of shrinker_rwsem, so we can simply replace it with mutex lock. Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai --- drivers/md/dm-cache-metadata.c | 2 +- drivers/md/dm-thin-metadata.c | 2 +- fs/super.c | 2 +- mm/shrinker_debug.c | 14 +++++++------- mm/vmscan.c | 34 +++++++++++++++++----------------- 5 files changed, 27 insertions(+), 27 deletions(-) diff --git a/drivers/md/dm-cache-metadata.c b/drivers/md/dm-cache-metadata.c index acffed750e3e..9e0c69958587 100644 --- a/drivers/md/dm-cache-metadata.c +++ b/drivers/md/dm-cache-metadata.c @@ -1828,7 +1828,7 @@ int dm_cache_metadata_abort(struct dm_cache_metadata *cmd) * Replacement block manager (new_bm) is created and old_bm destroyed outside of * cmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of * shrinker associated with the block manager's bufio client vs cmd root_lock). - * - must take shrinker_rwsem without holding cmd->root_lock + * - must take shrinker_mutex without holding cmd->root_lock */ new_bm = dm_block_manager_create(cmd->bdev, DM_CACHE_METADATA_BLOCK_SIZE << SECTOR_SHIFT, CACHE_MAX_CONCURRENT_LOCKS); diff --git a/drivers/md/dm-thin-metadata.c b/drivers/md/dm-thin-metadata.c index fd464fb024c3..9f5cb52c5763 100644 --- a/drivers/md/dm-thin-metadata.c +++ b/drivers/md/dm-thin-metadata.c @@ -1887,7 +1887,7 @@ int dm_pool_abort_metadata(struct dm_pool_metadata *pmd) * Replacement block manager (new_bm) is created and old_bm destroyed outside of * pmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of * shrinker associated with the block manager's bufio client vs pmd root_lock). - * - must take shrinker_rwsem without holding pmd->root_lock + * - must take shrinker_mutex without holding pmd->root_lock */ new_bm = dm_block_manager_create(pmd->bdev, THIN_METADATA_BLOCK_SIZE << SECTOR_SHIFT, THIN_MAX_CONCURRENT_LOCKS); diff --git a/fs/super.c b/fs/super.c index 84332d5cb817..91a4037b1d95 100644 --- a/fs/super.c +++ b/fs/super.c @@ -54,7 +54,7 @@ static char *sb_writers_name[SB_FREEZE_LEVELS] = { * One thing we have to be careful of with a per-sb shrinker is that we don't * drop the last active reference to the superblock from within the shrinker. * If that happens we could trigger unregistering the shrinker from within the - * shrinker path and that leads to deadlock on the shrinker_rwsem. Hence we + * shrinker path and that leads to deadlock on the shrinker_mutex. Hence we * take a passive reference to the superblock to avoid this from occurring. */ static unsigned long super_cache_scan(struct shrinker *shrink, diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 6aa7a7ec69da..b0f6aff372df 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -7,7 +7,7 @@ #include /* defined in vmscan.c */ -extern struct rw_semaphore shrinker_rwsem; +extern struct mutex shrinker_mutex; extern struct list_head shrinker_list; extern struct srcu_struct shrinker_srcu; @@ -167,7 +167,7 @@ int shrinker_debugfs_add(struct shrinker *shrinker) char buf[128]; int id; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); /* debugfs isn't initialized yet, add debugfs entries later. */ if (!shrinker_debugfs_root) @@ -210,7 +210,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) if (!new) return -ENOMEM; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); old = shrinker->name; shrinker->name = new; @@ -228,7 +228,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) shrinker->debugfs_entry = entry; } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); kfree_const(old); @@ -240,7 +240,7 @@ struct dentry *shrinker_debugfs_remove(struct shrinker *shrinker) { struct dentry *entry = shrinker->debugfs_entry; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); kfree_const(shrinker->name); shrinker->name = NULL; @@ -265,14 +265,14 @@ static int __init shrinker_debugfs_init(void) shrinker_debugfs_root = dentry; /* Create debugfs entries for shrinkers registered at boot */ - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_for_each_entry(shrinker, &shrinker_list, list) if (!shrinker->debugfs_entry) { ret = shrinker_debugfs_add(shrinker); if (ret) break; } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } diff --git a/mm/vmscan.c b/mm/vmscan.c index ac7ab4aa344f..c00302fabc3d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -35,7 +35,7 @@ #include #include #include -#include +#include #include #include #include @@ -202,7 +202,7 @@ static void set_task_reclaim_state(struct task_struct *task, } LIST_HEAD(shrinker_list); -DECLARE_RWSEM(shrinker_rwsem); +DEFINE_MUTEX(shrinker_mutex); DEFINE_SRCU(shrinker_srcu); static atomic_t shrinker_srcu_generation = ATOMIC_INIT(0); @@ -225,7 +225,7 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, { return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, &shrinker_srcu, - lockdep_is_held(&shrinker_rwsem)); + lockdep_is_held(&shrinker_mutex)); } static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, @@ -310,7 +310,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) int nid, size, ret = 0; int map_size, defer_size = 0; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); map_size = shrinker_map_size(shrinker_nr_max); defer_size = shrinker_defer_size(shrinker_nr_max); size = map_size + defer_size; @@ -326,7 +326,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) info->map_nr_max = shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } @@ -342,7 +342,7 @@ static int expand_shrinker_info(int new_id) if (!root_mem_cgroup) goto out; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); map_size = shrinker_map_size(new_nr_max); defer_size = shrinker_defer_size(new_nr_max); @@ -392,7 +392,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) if (mem_cgroup_disabled()) return -ENOSYS; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -406,7 +406,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) shrinker->id = id; ret = 0; unlock: - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } @@ -416,7 +416,7 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) BUG_ON(id < 0); - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); idr_remove(&shrinker_idr, id); } @@ -451,7 +451,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) parent = root_mem_cgroup; /* Prevent from concurrent shrinker_info expand */ - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); @@ -460,7 +460,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); } static bool cgroup_reclaim(struct scan_control *sc) @@ -709,9 +709,9 @@ void free_prealloced_shrinker(struct shrinker *shrinker) shrinker->name = NULL; #endif if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); unregister_memcg_shrinker(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return; } @@ -721,11 +721,11 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |= SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); } static int __register_shrinker(struct shrinker *shrinker) @@ -775,13 +775,13 @@ void unregister_shrinker(struct shrinker *shrinker) if (!(shrinker->flags & SHRINKER_REGISTERED)) return; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_del_rcu(&shrinker->list); shrinker->flags &= ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry = shrinker_debugfs_remove(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu);