From patchwork Mon Sep 11 09:44:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13379119 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF998EE57DF for ; Mon, 11 Sep 2023 09:51:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 67C606B026F; Mon, 11 Sep 2023 05:51:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 62D266B0270; Mon, 11 Sep 2023 05:51:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F54C6B0271; Mon, 11 Sep 2023 05:51:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 412DD6B026F for ; Mon, 11 Sep 2023 05:51:41 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 1A2C9B3A48 for ; Mon, 11 Sep 2023 09:51:41 +0000 (UTC) X-FDA: 81223849602.07.C2138E1 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by imf28.hostedemail.com (Postfix) with ESMTP id F21D5C0017 for ; Mon, 11 Sep 2023 09:51:38 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=j6JdpKd6; spf=pass (imf28.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694425899; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=02oN8W08zjTxUZl21U6+DV55wPZPOOOcomDUkgbjeeM=; b=kq4+DArMuE+eZP3VfAGVreKFA4KskGlISo+sM1ecfyYVJLiBp+THy0HYSA1zppyw6VPrG0 4uVu+dp9JA2vKlhilNNkpIaZXsmJc4O6ggIY60KYa+WDFZw9mlz3PEeanwB3peFy1auvsl WV5J6AL0j4U49WA0ydZadBz3HJRYb+Y= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694425899; a=rsa-sha256; cv=none; b=uVDEABLUBdiRBn+Z/wNZlO9Mn/+unfFqZrQJlHgiKE86FeiqhximT/b1S/gkWzdiKaPHBT pvNSOnSwMz9IkespnCXqdJWMzUyB9nLeN0v0UgLL67vn4fZJViMSxyAeqGVsIlQsUn/u5s FX5gxondMvbJ4Z8IOsMZI+uridIijqc= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=j6JdpKd6; spf=pass (imf28.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-68cc1b70e05so1081910b3a.1 for ; Mon, 11 Sep 2023 02:51:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1694425898; x=1695030698; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=02oN8W08zjTxUZl21U6+DV55wPZPOOOcomDUkgbjeeM=; b=j6JdpKd6UsTI8cDVRs0lYVO+RuBR53yWmwMluSx5s2QGRPIJMkuIylhY5ntBJDkEbA ZN65D7RxNKbVdO/H/gENOpVv2eBefG+i/gqxf3qlHqd1uAgYfaYrmLyiwkK//Kmvn/15 vtJ4Hr3R0tfeM8gxMfDKfIIBWPg9MhgJ6nd5lXsMxH+UhzPuldaP6ddev8T42F24eDnE 6ODTISLL4jrOCgEqGOyTv+BxPklSwkeXEd7aFGuISdNWyMvK+QTJ6XdJJbajSrw0FovZ rLNf/Tn8ka/XgqxWXnECJygjhwOUv6hIS0T3ILcos2mZ2RADAgccnjoJ0kbM/Q3Rz6cS gYIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694425898; x=1695030698; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=02oN8W08zjTxUZl21U6+DV55wPZPOOOcomDUkgbjeeM=; b=jLt1cX9GWWTnnvrhnrUkbcxAfXL+VnbJg2ewqejwldcysSCAXuslLLH996EmMCy9BX BH+t/uZw7cPysfMEjc0h90B4jilOnoaS2LRug3/jyS/V8dlwt0k86J/dtJA+cEVT8b1u tZevbi32KsAs29tVxy99niaZLrMMzwkh4TrjiX1adDK17SB7jQAblnTXCHTcqYVyMdnI k4S4h7rOC7nVPzAxTM9jbYxdFLGPGTxm9jb3+kF7XIQdFNeusVBqdHFF8zzS1cc/XEza 2kfSd7VvFK+qzEOCA02pXR/qVk7WfYCHSJRhwpSsdwfN2nCx2/7rdHdcrTzI94iARb4E PdiQ== X-Gm-Message-State: AOJu0Yx2Ek3xr8slhdREj0MCmtSM8OzzdVuY8YAMNwLKIhyqpFuYyZpy +SwaUA8n/HAEy1791OQeQ9s0wA== X-Google-Smtp-Source: AGHT+IH97cG96KOmpoktz2XA7SklxuW0ceeXqXRZ1DysovZgTlaqQRndvB65TuUAaXsE5amMaJ1n9w== X-Received: by 2002:a05:6a21:6da4:b0:137:3941:17b3 with SMTP id wl36-20020a056a216da400b00137394117b3mr12294593pzb.6.1694425897928; Mon, 11 Sep 2023 02:51:37 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.146]) by smtp.gmail.com with ESMTPSA id az7-20020a170902a58700b001bdc2fdcf7esm5988188plb.129.2023.09.11.02.51.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Sep 2023 02:51:37 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu, steven.price@arm.com, cel@kernel.org, senozhatsky@chromium.org, yujie.liu@intel.com, gregkh@linuxfoundation.org, muchun.song@linux.dev Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Qi Zheng Subject: [PATCH v6 43/45] mm: shrinker: make memcg slab shrink lockless Date: Mon, 11 Sep 2023 17:44:42 +0800 Message-Id: <20230911094444.68966-44-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230911094444.68966-1-zhengqi.arch@bytedance.com> References: <20230911094444.68966-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: F21D5C0017 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: u1foz6x3xbzagubewqpjacton9wetdpb X-HE-Tag: 1694425898-886365 X-HE-Meta: U2FsdGVkX18hYZXGoISnIPoIMjrdbzgmeIKspTYxsB1IEht4Nx++DiC7hz2B+VWKTxJ9pQ23DvIcaUBShM++bXz+0zxImdoy8bHezdWD6oGEz3a4AFmZbn4Hxsqy8C4jR/+DSG1L6ESXk1hIMDYe0PVWmuWtns1Dgrgp5mEn5YT4SbczDelV1oft75hKbZMYhkwU4FGMSMC1oG+wMhbD4qfOx8Kh7rFJktiZCJAxWCP6uSmJX1awSErQ/rd+g3OdjVV3YdAM2KQb3HzJkJBLb/8a5wkZLO/TZ0K+L2xJKKnz3tbF16Ft7hxKik7D2nQvb8gpSWvLVdv0MwBGZVVYOUIwpFVZZNZVmNZo/9DtSohLexPpa8bTIxrBslChyVG3Ste34lmfJImlSt+2yuw7yIwUc2IMbZRHb3y/vVUKxq9HHkcj1cXGKvNyWp1un2fovtIVR9yaZTzHcxF4idB9V9ohwXjebTH1Ej+5ZskrpqgC+8y4MciD6Ji7bCr9P3PdLHFpvnbrLJj449ZTu35d5q/Oc5vr6Bmwp/n07hWvLUF9GEu0ahK716S2zsT+7+e6A8BCdfvuKvK341Ea9ygYwJLxd9QA+nB/UKL5q3AArpIzhjpVsVW9U/dZmcA/dcbpv5UD39B5oudaFktvsz3UihZ/vmRrfrcWoveNHkkK0eXGyW9hhr4zIA6VT2f2tSKW1ZJcGhd/81HPu+aeXaWSDQuLC5vw3LnupmyRffQoLCO7ob6graKlQzVCQdh9qT37jhJoyFnfzHN/Hk6hWgq03ghUWqJSmnHcnqqf6uH5ISquWyGa5c+xC/eNyCeRtOfAXan0/Wq0mmQ8+oJy20ulPFz8jGH91iMH0oqe/Sf+rKxGkdAB7ZGv0NJ0sxMtavlgQEHwpfhyDRT1xe9Cf9fsiHKQzlxQRzQ1x4n7qK45mUgCmh1OaYMmeEdFBRjkjd8/Z9g+dEVO4gXHsykScR7 /JK/Usuc Iax+SjNFVFiYzI0J+XIAvB2z1Srtxk2EnrCxr0LaAnCosnP3BNGG8J4Z8RjJR8AmCjH814+Ir9E25nS+DzA/U8zGULp70zUUnd6YYlldQepIV6oCMguRUqj0ZApmtEiETsKuz2gbBDdxTJn0TE9iKcpxmjDKto4aZuwmVy7HqqwNpIIklKGLaLMtx3y3O8D1gPuHpKRBmkG4P+rv9A4is67tHP5cz5UuGpzCjsEwuj3pOzZa6WtTsDg7rd76Q074J3DAR5OGvtQ2QpinD80iuA/yIHlZAI9kPApyA/yCuPPbR3PeTOu3CknZfVmyz0aFrMgwyEt4HnEtkgCrFnr+WNu+jT833alBsLMV6NUbyYraveLP7nStWOe5Jwi26FJyjWKy6l46IrVGYbIZpsfWDXOjpOhNWou3ZnTLk X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Like global slab shrink, this commit also uses refcount+RCU method to make memcg slab shrink lockless. Use the following script to do slab shrink stress test: ``` DIR="/root/shrinker/memcg/mnt" do_create() { mkdir -p /sys/fs/cgroup/memory/test echo 4G > /sys/fs/cgroup/memory/test/memory.limit_in_bytes for i in `seq 0 $1`; do mkdir -p /sys/fs/cgroup/memory/test/$i; echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; mkdir -p $DIR/$i; done } do_mount() { for i in `seq $1 $2`; do mount -t tmpfs $i $DIR/$i; done } do_touch() { for i in `seq $1 $2`; do echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; dd if=/dev/zero of=$DIR/$i/file$i bs=1M count=1 & done } case "$1" in touch) do_touch $2 $3 ;; test) do_create 4000 do_mount 0 4000 do_touch 0 3000 ;; *) exit 1 ;; esac ``` Save the above script, then run test and touch commands. Then we can use the following perf command to view hotspots: perf top -U -F 999 1) Before applying this patchset: 33.15% [kernel] [k] down_read_trylock 25.38% [kernel] [k] shrink_slab 21.75% [kernel] [k] up_read 4.45% [kernel] [k] _find_next_bit 2.27% [kernel] [k] do_shrink_slab 1.80% [kernel] [k] intel_idle_irq 1.79% [kernel] [k] shrink_lruvec 0.67% [kernel] [k] xas_descend 0.41% [kernel] [k] mem_cgroup_iter 0.40% [kernel] [k] shrink_node 0.38% [kernel] [k] list_lru_count_one 2) After applying this patchset: 64.56% [kernel] [k] shrink_slab 12.18% [kernel] [k] do_shrink_slab 3.30% [kernel] [k] __rcu_read_unlock 2.61% [kernel] [k] shrink_lruvec 2.49% [kernel] [k] __rcu_read_lock 1.93% [kernel] [k] intel_idle_irq 0.89% [kernel] [k] shrink_node 0.81% [kernel] [k] mem_cgroup_iter 0.77% [kernel] [k] mem_cgroup_calculate_protection 0.66% [kernel] [k] list_lru_count_one We can see that the first perf hotspot becomes shrink_slab, which is what we expect. Signed-off-by: Qi Zheng --- mm/shrinker.c | 85 +++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 66 insertions(+), 19 deletions(-) diff --git a/mm/shrinker.c b/mm/shrinker.c index 82dc61133c5b..ad64cac5248c 100644 --- a/mm/shrinker.c +++ b/mm/shrinker.c @@ -218,7 +218,6 @@ static int shrinker_memcg_alloc(struct shrinker *shrinker) return -ENOSYS; down_write(&shrinker_rwsem); - /* This may call shrinker, so it must use down_read_trylock() */ id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -252,10 +251,15 @@ static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, { struct shrinker_info *info; struct shrinker_info_unit *unit; + long nr_deferred; - info = shrinker_info_protected(memcg, nid); + rcu_read_lock(); + info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); unit = info->unit[shrinker_id_to_index(shrinker->id)]; - return atomic_long_xchg(&unit->nr_deferred[shrinker_id_to_offset(shrinker->id)], 0); + nr_deferred = atomic_long_xchg(&unit->nr_deferred[shrinker_id_to_offset(shrinker->id)], 0); + rcu_read_unlock(); + + return nr_deferred; } static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, @@ -263,10 +267,16 @@ static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, { struct shrinker_info *info; struct shrinker_info_unit *unit; + long nr_deferred; - info = shrinker_info_protected(memcg, nid); + rcu_read_lock(); + info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); unit = info->unit[shrinker_id_to_index(shrinker->id)]; - return atomic_long_add_return(nr, &unit->nr_deferred[shrinker_id_to_offset(shrinker->id)]); + nr_deferred = + atomic_long_add_return(nr, &unit->nr_deferred[shrinker_id_to_offset(shrinker->id)]); + rcu_read_unlock(); + + return nr_deferred; } void reparent_shrinker_deferred(struct mem_cgroup *memcg) @@ -463,18 +473,54 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (!mem_cgroup_online(memcg)) return 0; - if (!down_read_trylock(&shrinker_rwsem)) - return 0; - - info = shrinker_info_protected(memcg, nid); + /* + * lockless algorithm of memcg shrink. + * + * The shrinker_info may be freed asynchronously via RCU in the + * expand_one_shrinker_info(), so the rcu_read_lock() needs to be used + * to ensure the existence of the shrinker_info. + * + * The shrinker_info_unit is never freed unless its corresponding memcg + * is destroyed. Here we already hold the refcount of memcg, so the + * memcg will not be destroyed, and of course shrinker_info_unit will + * not be freed. + * + * So in the memcg shrink: + * step 1: use rcu_read_lock() to guarantee existence of the + * shrinker_info. + * step 2: after getting shrinker_info_unit we can safely release the + * RCU lock. + * step 3: traverse the bitmap and calculate shrinker_id + * step 4: use rcu_read_lock() to guarantee existence of the shrinker. + * step 5: use shrinker_id to find the shrinker, then use + * shrinker_try_get() to guarantee existence of the shrinker, + * then we can release the RCU lock to do do_shrink_slab() that + * may sleep. + * step 6: do shrinker_put() paired with step 5 to put the refcount, + * if the refcount reaches 0, then wake up the waiter in + * shrinker_free() by calling complete(). + * Note: here is different from the global shrink, we don't + * need to acquire the RCU lock to guarantee existence of + * the shrinker, because we don't need to use this + * shrinker to traverse the next shrinker in the bitmap. + * step 7: we have already exited the read-side of rcu critical section + * before calling do_shrink_slab(), the shrinker_info may be + * released in expand_one_shrinker_info(), so go back to step 1 + * to reacquire the shrinker_info. + */ +again: + rcu_read_lock(); + info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); if (unlikely(!info)) goto unlock; - for (; index < shrinker_id_to_index(info->map_nr_max); index++) { + if (index < shrinker_id_to_index(info->map_nr_max)) { struct shrinker_info_unit *unit; unit = info->unit[index]; + rcu_read_unlock(); + for_each_set_bit(offset, unit->map, SHRINKER_UNIT_BITS) { struct shrink_control sc = { .gfp_mask = gfp_mask, @@ -484,12 +530,14 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, struct shrinker *shrinker; int shrinker_id = calc_shrinker_id(index, offset); + rcu_read_lock(); shrinker = idr_find(&shrinker_idr, shrinker_id); - if (unlikely(!shrinker || !(shrinker->flags & SHRINKER_REGISTERED))) { - if (!shrinker) - clear_bit(offset, unit->map); + if (unlikely(!shrinker || !shrinker_try_get(shrinker))) { + clear_bit(offset, unit->map); + rcu_read_unlock(); continue; } + rcu_read_unlock(); /* Call non-slab shrinkers even though kmem is disabled */ if (!memcg_kmem_online() && @@ -522,15 +570,14 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, set_shrinker_bit(memcg, nid, shrinker_id); } freed += ret; - - if (rwsem_is_contended(&shrinker_rwsem)) { - freed = freed ? : 1; - goto unlock; - } + shrinker_put(shrinker); } + + index++; + goto again; } unlock: - up_read(&shrinker_rwsem); + rcu_read_unlock(); return freed; } #else /* !CONFIG_MEMCG */