From patchwork Thu Feb 23 13:27:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13150327 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E687C636D6 for ; Thu, 23 Feb 2023 13:28:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DC8D26B007D; Thu, 23 Feb 2023 08:28:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D79656B007E; Thu, 23 Feb 2023 08:28:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C4B7B6B0080; Thu, 23 Feb 2023 08:28:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B5C196B007D for ; Thu, 23 Feb 2023 08:28:10 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 63F371A015F for ; Thu, 23 Feb 2023 13:28:10 +0000 (UTC) X-FDA: 80498635140.20.A29112C Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by imf12.hostedemail.com (Postfix) with ESMTP id 92ED140019 for ; Thu, 23 Feb 2023 13:28:08 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=hQ0T37qt; spf=pass (imf12.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677158888; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Syv0drMPggMMkttRMMkloSH5R6RYZ1YZAM5MesH77qc=; b=zMDrofz9kFfp6O/oCeQaAOnmtKwFCdk1VUTe+6ppObCNRg7i6h40dsRSSE2rEZeJGrKCBQ 41mx8PuFg8qWwpBfaQpKpwVyaZvzv/sZknRhf7VQxczriOsKQfofOwrB/3cj9uczdO7kKQ KLhvrsCREXBVhWQM5bmD41oJNEYI7Z4= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=hQ0T37qt; spf=pass (imf12.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677158888; a=rsa-sha256; cv=none; b=GA3zqLmSS3/UK41UhtGSUk2qD7H4nJ7au8cQpc/OIOEllMB98I4mkecAV5eFfUAHEPs5iy HZesfS+8Pak8yjL3WTNFhGiddKVEj5IBEnuMoHXT6K3BV6t3056RFx1LCJn3BZRpoAvVS0 UC70B9xBU4ZHHfEjFycnZ7CeugWczk4= Received: by mail-pg1-f177.google.com with SMTP id q189so5576449pga.9 for ; Thu, 23 Feb 2023 05:28:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1677158887; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Syv0drMPggMMkttRMMkloSH5R6RYZ1YZAM5MesH77qc=; b=hQ0T37qts3rJeuZtmDQARv10wVciADPxTYE7nHXMnr9/N5MAW0787FBKg4eYhKDGIR D+o6nrvNrTCJZgauHW1A4MDrBHjxksREY4lTnxwUa84Fsm5l7e+fbi03X46BgEoXdVQ+ dOAn3WrLOyf4bs99Rq8TgWRjbgf9v7Q1uHlPOpcxV2y0uxirjEy5+OIZMhdR26f/63fU P2Pmq2VOLIEelMQtA+O9p5Iee99sRG2fY+Tl1/DItQIUTAgSBqbuxd7az9ZKW/udBx0b cWC6J1VhJL/7sSkSUyQ5U+ZKGWW3NEEgGBmU1h341ArROsLUeQb5G50uiU2GeEJsA+K+ hEqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677158887; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Syv0drMPggMMkttRMMkloSH5R6RYZ1YZAM5MesH77qc=; b=thIT5dxUJqwjFXJYxJRJractDXLHe9Int8iDEnx87qdVd4zLjo/NWFecbGPzXspspf XtFTbvElr5sqS5QaTp7aq7FD3H2sAhfw3DtriKjg6Vi3fAIE1MkuzGN/L1hURdU7jqaW 0xyP+/mA3lfAarf038TVC1+Us1iRsvLKZkMBpsXcx/vUGB9AoG+jKuEE1uXW/513RMak 6QBAnQkgnCEnbrL6xbRo8nfLXCWz6sgXlE/yORWTi87wCTu2uoaFE5qxiHsRyYHDoKdN neuobFTTbmdoKrMw3umD1nwAsVzjMtbJDkiPl4dGxcwqk5CTEF+rWstiXA5h3Dv8J58b 2big== X-Gm-Message-State: AO0yUKVG656hHp/a6cZOQGRcoteQ06qJ+VCSzhsqjzxDNHPUxHRbnJv7 JYz7knj1JobI9h0BxtVRro7lyA== X-Google-Smtp-Source: AK7set/3KhLevmt8IbsCAmbb8GxI559C7KBiMEGLPkGOAY2YkyfrjWjj15z30QZAy/t94RwKaPBlcQ== X-Received: by 2002:aa7:90cf:0:b0:5a9:c2d5:136c with SMTP id k15-20020aa790cf000000b005a9c2d5136cmr12399865pfk.3.1677158887363; Thu, 23 Feb 2023 05:28:07 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id g18-20020aa78752000000b005a9bf65b591sm3848591pfo.135.2023.02.23.05.28.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 05:28:07 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 3/7] mm: vmscan: make memcg slab shrink lockless Date: Thu, 23 Feb 2023 21:27:21 +0800 Message-Id: <20230223132725.11685-4-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230223132725.11685-1-zhengqi.arch@bytedance.com> References: <20230223132725.11685-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 92ED140019 X-Stat-Signature: y4geenyznqg3ptttmaertsipibm1zdkq X-HE-Tag: 1677158888-914044 X-HE-Meta: U2FsdGVkX18AfGViE2tXGNb7+wsWsMGh3FBi7yvULj2K4DlnZFJxZmT3iZFMWfKVC/4QXFHrzh3NK8ynJvqMyGfLb8kJAw6RHKujHBqMT62E3C04udq7qSu2+TSFuD4Zij4VPuZMH6szlBXOp49+edcFS6OgZcCCP6ReicAzChDprulylc4JeE6MWPmXi0d37+EKWdgCoqw1MaX8B77KRNuUxja7ArlUyXXRfXGkac0+MhoVEzyJ3nEciNpGNDA4K7tryFqGGit1bCcRWyubZK0QxrEWDFrfvNKEN7VJ2TI1z6EPkQij2J1aAXFJGQ4K/MgIcvwUaEDdZeWaz6MTxNDe77JlCLBOIXaOqPPBHxewqsAMQfIxKLy2vcTwsQX+NzbZd5h/t+gKyDmSettl3TUyqwQEXPN2DUBBngBmGQ3cSHwNhw/MrPNVvpTP3FCidQDCgXlLhfUYzmIfTXVw9+gjE35oX7Dwyvd48PlMYXR8EYSt2nacoO0QwazSU7L13+2luACOIMGC/X/cRuOPKw2wuVnWtcVIwVT1gB/Y5LZKsftmS/OJ0Y8/v4H8Z4UXhHl+ZT4pvQSbkyLBlwv62Sse35wZeA3m96B9mK/IT+HqkT0khm4Yc/U5wUZvRCxERKQQNOSVO5+LUPevh1Jxumn5zHGUNoddMU0kAOqfxr2M1FLi93YC/OlWeeqXI8rtT0m0/JQlyN+p4MHg1w34bUhU6He7DUE8ZGibZUs4+H89f1ZcrhK7rhWbNE6z6xCtQ6y4emYFvBwf2xIAg3f8uztXuaZKLjqKpwWEAaWJMDUtrCo/4TUsDyhr01LZU8beHHQY0g2iFPaZmwzQXQSnpbuotTJzejZaDyz56ea7Obc0fC29qlydJMVqhviXYD6NsqNmVMh6HIXVCey9KNauOyL4L3dRVDLsFOb4tTCJ6pzCLlgF7P/4RXwkFF64+D5tpwk1EO+9vc/22ByEiar wA1tn4vQ g4DevlWVUY69fXxpFwzElFnxxK4GMciNrA2aYVJhFlmiJGoCevU5okw29TYNdwEv51/XixJLOblNBwD9T8Pc1QAR1DI+PPhGdvdu0QSggXtvtx+3HQ6r2HOLWO93KHr92ANXb7Jk9rAeD3VHI6gOdLQ3j3aczb7ddxEldEyYeldgdWmc95mLUrdUihl+HhZtlEmt62M2juBHPwLNDC46uciGmYJuiv/NdmL+Q6n5Q/oqO29nTatu8eQvZkpDjnSUwQ/virN/Czw6tWCHO5XbJFZjdmOfhVvpjak58Tn8jfd4d63Xv5KCScAsklUlnHRdg4XeoGoHlo/cMODTKDjlU5Xe3wb+5rt0tLW8ta0c0zfj8cNEpy2EFdhWCLATgbJrpQdXrVjqtQPH6yTds5frHrvsbnuLROt/+A+MAKr5LvhnjG0k= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Like global slab shrink, since commit 1cd0bd06093c ("rcu: Remove CONFIG_SRCU"), it's time to use SRCU to protect readers who previously held shrinker_rwsem. We can test with the following script: ``` DIR="/root/shrinker/memcg/mnt" do_create() { mkdir /sys/fs/cgroup/memory/test echo 200M > /sys/fs/cgroup/memory/test/memory.limit_in_bytes for i in `seq 0 $1`; do mkdir /sys/fs/cgroup/memory/test/$i; echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; mkdir -p $DIR/$i; done } do_mount() { for i in `seq $1 $2`; do mount -t tmpfs $i $DIR/$i; done } do_touch() { for i in `seq $1 $2`; do echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; dd if=/dev/zero of=$DIR/$i/file$i bs=1M count=1 & done } do_create 2000 do_mount 0 2000 do_touch 0 1000 ``` Before applying: 46.60% [kernel] [k] down_read_trylock 18.70% [kernel] [k] up_read 15.44% [kernel] [k] shrink_slab 4.37% [kernel] [k] _find_next_bit 2.75% [kernel] [k] xa_load 2.07% [kernel] [k] idr_find 1.73% [kernel] [k] do_shrink_slab 1.42% [kernel] [k] shrink_lruvec 0.74% [kernel] [k] shrink_node 0.60% [kernel] [k] list_lru_count_one After applying: 19.53% [kernel] [k] _find_next_bit 14.63% [kernel] [k] do_shrink_slab 14.58% [kernel] [k] shrink_slab 11.83% [kernel] [k] shrink_lruvec 9.33% [kernel] [k] __blk_flush_plug 6.67% [kernel] [k] mem_cgroup_iter 3.73% [kernel] [k] list_lru_count_one 2.43% [kernel] [k] shrink_node 1.96% [kernel] [k] super_cache_count 1.78% [kernel] [k] __rcu_read_unlock 1.38% [kernel] [k] __srcu_read_lock 1.30% [kernel] [k] xas_descend We can see that the readers is no longer blocked. Signed-off-by: Qi Zheng --- mm/vmscan.c | 46 +++++++++++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 19 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 02987a6f95d1..25a4a660e45f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include @@ -221,8 +222,21 @@ static inline int shrinker_defer_size(int nr_items) static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, int nid) { - return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, - lockdep_is_held(&shrinker_rwsem)); + return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu, + lockdep_is_held(&shrinker_rwsem)); +} + +static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, + int nid) +{ + return srcu_dereference(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu); +} + +static void free_shrinker_info_rcu(struct rcu_head *head) +{ + kvfree(container_of(head, struct shrinker_info, rcu)); } static inline bool need_expand(int new_nr_max, int old_nr_max) @@ -268,7 +282,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, defer_size - old_defer_size); rcu_assign_pointer(pn->shrinker_info, new); - kvfree_rcu(old, rcu); + call_srcu(&shrinker_srcu, &old->rcu, free_shrinker_info_rcu); } return 0; @@ -357,13 +371,14 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) { if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { struct shrinker_info *info; + int srcu_idx; - rcu_read_lock(); - info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); + srcu_idx = srcu_read_lock(&shrinker_srcu); + info = shrinker_info_srcu(memcg, nid); /* Pairs with smp mb in shrink_slab() */ smp_mb__before_atomic(); set_bit(shrinker_id, info->map); - rcu_read_unlock(); + srcu_read_unlock(&shrinker_srcu, srcu_idx); } } @@ -377,7 +392,6 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) return -ENOSYS; down_write(&shrinker_rwsem); - /* This may call shrinker, so it must use down_read_trylock() */ id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -411,7 +425,7 @@ static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, { struct shrinker_info *info; - info = shrinker_info_protected(memcg, nid); + info = shrinker_info_srcu(memcg, nid); return atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); } @@ -420,7 +434,7 @@ static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, { struct shrinker_info *info; - info = shrinker_info_protected(memcg, nid); + info = shrinker_info_srcu(memcg, nid); return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); } @@ -898,15 +912,14 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, { struct shrinker_info *info; unsigned long ret, freed = 0; + int srcu_idx; int i; if (!mem_cgroup_online(memcg)) return 0; - if (!down_read_trylock(&shrinker_rwsem)) - return 0; - - info = shrinker_info_protected(memcg, nid); + srcu_idx = srcu_read_lock(&shrinker_srcu); + info = shrinker_info_srcu(memcg, nid); if (unlikely(!info)) goto unlock; @@ -956,14 +969,9 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, set_shrinker_bit(memcg, nid, i); } freed += ret; - - if (rwsem_is_contended(&shrinker_rwsem)) { - freed = freed ? : 1; - break; - } } unlock: - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); return freed; } #else /* CONFIG_MEMCG */