From patchwork Fri Jun 9 08:15:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13273415 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEF53C7EE2F for ; Fri, 9 Jun 2023 08:17:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 85C308E0009; Fri, 9 Jun 2023 04:17:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7E3A88E0003; Fri, 9 Jun 2023 04:17:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 65D758E0009; Fri, 9 Jun 2023 04:17:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4F26D8E0003 for ; Fri, 9 Jun 2023 04:17:07 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 17C7A4018E for ; Fri, 9 Jun 2023 08:17:07 +0000 (UTC) X-FDA: 80882504094.15.0F5B8C8 Received: from out-15.mta0.migadu.com (out-15.mta0.migadu.com [91.218.175.15]) by imf03.hostedemail.com (Postfix) with ESMTP id 0A9942000D for ; Fri, 9 Jun 2023 08:17:04 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="VJW/sTjl"; spf=pass (imf03.hostedemail.com: domain of qi.zheng@linux.dev designates 91.218.175.15 as permitted sender) smtp.mailfrom=qi.zheng@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686298625; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=50XF484pkGZZlvaX6wjD3S7ullpcs6Au3jWyE87o9lo=; b=hG4Ou7zNALQoZdDr/NlBj9Hd9gCPWH7jZYWQHeaRUB4rkJTJeEmKavMKsbHxJZA1YsHLYq jd54NCG9aLVxtMAQ+Fei488u2XHKoPT+Rurx4BKlDeIxQp2Or2ZE+6lBI362iVJK04ymfz FJUmMM8vsUlpP6tkIAgABp5m/wpZVy0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686298625; a=rsa-sha256; cv=none; b=fNnE5JALdJwDXeJMI9aX0wGU5mUNayy6te0ys3NorULWxe2U+WeGqAynlvpl0yvJujGKVK 4JoWOoI1NLwgJai151jcogz8i0N31HG8AmgOhdbHdNiQMRIgWM3CYx4ryZ24jtdb7aubCr rY305Zxfbr+IL+RcKB69EyRlOl0B9MM= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="VJW/sTjl"; spf=pass (imf03.hostedemail.com: domain of qi.zheng@linux.dev designates 91.218.175.15 as permitted sender) smtp.mailfrom=qi.zheng@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686298623; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=50XF484pkGZZlvaX6wjD3S7ullpcs6Au3jWyE87o9lo=; b=VJW/sTjlrsD06VeMs5MWZHOINSVeIuvUVV3XeMcdlaZgU6yHH1KDKeoDlQ0uNJ8ow7jI5i C2f1mQy0VShusDu7ScqyZuVT65khmY5y8dLoApSOfxCjNpnx2aeqmPB13WMTVIUTOMDiTn dvzcgxnTuwP9thnsK4Ff5TbiUeHu3ak= From: Qi Zheng To: akpm@linux-foundation.org Cc: david@fromorbit.com, tkhai@ya.ru, roman.gushchin@linux.dev, vbabka@suse.cz, muchun.song@linux.dev, yujie.liu@intel.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 1/7] Revert "mm: shrinkers: convert shrinker_rwsem to mutex" Date: Fri, 9 Jun 2023 08:15:12 +0000 Message-Id: <20230609081518.3039120-2-qi.zheng@linux.dev> In-Reply-To: <20230609081518.3039120-1-qi.zheng@linux.dev> References: <20230609081518.3039120-1-qi.zheng@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 0A9942000D X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: xbxckhqnfwbomf8xdkafqtwixhwsef1a X-HE-Tag: 1686298624-61193 X-HE-Meta: U2FsdGVkX18MJVuYb0vrqUB+uyNRPe0UGxU9uYQdNUCk/iDuT2KGVC+NHJUhV/kwnNQf3SFOcv8LvT+jzIgM3wt9M0MqXMMazg/tNyhyWJaM7tuZIvsZDz3aA8VguctRo6tHvDASe8AQpIaUZynoTi90RoWyNpoR2/H1i+7TXWxoPSRuCj2xmQijr6NWyJoumD5HBW2yfZ+oQfz1bAZNf6VC5ZE7mrtnu/NuPIEsBy6mon9L/nrdJ1J6lVzMc7rrnenK+BDMfXtoJFhQIUuxr+qrlwparBMoRvgG2lp1Nh0Z3W8iB7SEU88lwqlQmthT2c2K7Rk4luptdY7zH9Sk9x18J27lTuSjv1TkJ2+LbjJfjG77hGF+mHkPIcWytRXEUsDw4qEZ2ZOy2vmGagbVkPEbTn53OZJDzsjTa5obXYayfRMPKuvr3M9Moz5BAYkz6gAXOp1Z14Pye4MzPPQtg6bdewBwfJgYls2u8rGYFkCRfBbqsfcio1iv+qfCyROJduoeg8M4x45KSMZiMsSvmggXTHvw2soEpTFUbLot/lcddEP++DX6iMPYwSIZCXsTWkjXa8l67NDXdjek4a/k6Hu2GIkbhcf1isAS9U0mu601PrI6ySbPQWzp3XyxpA00/+2drg/iYtm0O0baQ6cJM9fOd/fQedOp/oMgrk48PPb7dDa944KBXNsNxSeAOXju6uy+K9+vYs460quXHOjP9KmDz7LwDpbaefwijp4Pe4zLtXah8x/dr5mHkXqANWzqSZtq5FT1XUIEp1TGrEnhsHlYQoG8oPH8ow2BKZBT2YOYM1V43hNMw4cDPOcpeI0+aBnX+YYWk3k5iXoG6uJ4REbwLFyc92ga4fsfZdarzDGCqs1jOe7cyXWInYKKqsqYq5wdrUtssbA3lVJCqU258X5GjMxjYEzpgTysiXhLfHCEoa/bSYh2lroZMhOwMY1NC2MEcfO9O9h3dKwsBqE bXLPv4Wt OKxD0dQWy9YsJTkA6TKWMygeKuYbaNOMi2OaECfxpS6rdNmYIrdX/K1myyzoRUooEtO895wdyZ9J6rIbIzHUv51yALyYrRPvb97GP/VkF5eaRwv/JpVSAR7n/0TeCy7xxtG4W1PkfvRkaJpODe4xozwtpkwThp+fTkpWqXLL7ibr0OLmAGEONum+KnMDPvc+wPNIRUkEFkeJ64EnYeqFVgnByT5iZB+t7Cw5lfq+nYLJNsVAU6II7i2lzI9SW/+Q+KRqMhH9VnU2/aMo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Qi Zheng This reverts commit cf2e309ebca7bb0916771839f9b580b06c778530. Kernel test robot reports -88.8% regression in stress-ng.ramfs.ops_per_sec test case [1], which is caused by commit f95bdb700bc6 ("mm: vmscan: make global slab shrink lockless"). The root cause is that SRCU has to be careful to not frequently check for SRCU read-side critical section exits. Therefore, even if no one is currently in the SRCU read-side critical section, synchronize_srcu() cannot return quickly. That's why unregister_shrinker() has become slower. After discussion, we will try to use the refcount+RCU method [2] proposed by Dave Chinner to continue to re-implement the lockless slab shrink. So revert the shrinker_mutex back to shrinker_rwsem first. [1]. https://lore.kernel.org/lkml/202305230837.db2c233f-yujie.liu@intel.com/ [2]. https://lore.kernel.org/lkml/ZIJhou1d55d4H1s0@dread.disaster.area/ Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202305230837.db2c233f-yujie.liu@intel.com Signed-off-by: Qi Zheng --- drivers/md/dm-cache-metadata.c | 2 +- drivers/md/dm-thin-metadata.c | 2 +- fs/super.c | 2 +- mm/shrinker_debug.c | 14 +++++++------- mm/vmscan.c | 34 +++++++++++++++++----------------- 5 files changed, 27 insertions(+), 27 deletions(-) diff --git a/drivers/md/dm-cache-metadata.c b/drivers/md/dm-cache-metadata.c index 9e0c69958587..acffed750e3e 100644 --- a/drivers/md/dm-cache-metadata.c +++ b/drivers/md/dm-cache-metadata.c @@ -1828,7 +1828,7 @@ int dm_cache_metadata_abort(struct dm_cache_metadata *cmd) * Replacement block manager (new_bm) is created and old_bm destroyed outside of * cmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of * shrinker associated with the block manager's bufio client vs cmd root_lock). - * - must take shrinker_mutex without holding cmd->root_lock + * - must take shrinker_rwsem without holding cmd->root_lock */ new_bm = dm_block_manager_create(cmd->bdev, DM_CACHE_METADATA_BLOCK_SIZE << SECTOR_SHIFT, CACHE_MAX_CONCURRENT_LOCKS); diff --git a/drivers/md/dm-thin-metadata.c b/drivers/md/dm-thin-metadata.c index 9f5cb52c5763..fd464fb024c3 100644 --- a/drivers/md/dm-thin-metadata.c +++ b/drivers/md/dm-thin-metadata.c @@ -1887,7 +1887,7 @@ int dm_pool_abort_metadata(struct dm_pool_metadata *pmd) * Replacement block manager (new_bm) is created and old_bm destroyed outside of * pmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of * shrinker associated with the block manager's bufio client vs pmd root_lock). - * - must take shrinker_mutex without holding pmd->root_lock + * - must take shrinker_rwsem without holding pmd->root_lock */ new_bm = dm_block_manager_create(pmd->bdev, THIN_METADATA_BLOCK_SIZE << SECTOR_SHIFT, THIN_MAX_CONCURRENT_LOCKS); diff --git a/fs/super.c b/fs/super.c index 34afe411cf2b..04bc62ab7dfe 100644 --- a/fs/super.c +++ b/fs/super.c @@ -54,7 +54,7 @@ static char *sb_writers_name[SB_FREEZE_LEVELS] = { * One thing we have to be careful of with a per-sb shrinker is that we don't * drop the last active reference to the superblock from within the shrinker. * If that happens we could trigger unregistering the shrinker from within the - * shrinker path and that leads to deadlock on the shrinker_mutex. Hence we + * shrinker path and that leads to deadlock on the shrinker_rwsem. Hence we * take a passive reference to the superblock to avoid this from occurring. */ static unsigned long super_cache_scan(struct shrinker *shrink, diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index fe10436d9911..2be15b8a6d0b 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -8,7 +8,7 @@ #include /* defined in vmscan.c */ -extern struct mutex shrinker_mutex; +extern struct rw_semaphore shrinker_rwsem; extern struct list_head shrinker_list; extern struct srcu_struct shrinker_srcu; @@ -168,7 +168,7 @@ int shrinker_debugfs_add(struct shrinker *shrinker) char buf[128]; int id; - lockdep_assert_held(&shrinker_mutex); + lockdep_assert_held(&shrinker_rwsem); /* debugfs isn't initialized yet, add debugfs entries later. */ if (!shrinker_debugfs_root) @@ -211,7 +211,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) if (!new) return -ENOMEM; - mutex_lock(&shrinker_mutex); + down_write(&shrinker_rwsem); old = shrinker->name; shrinker->name = new; @@ -229,7 +229,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) shrinker->debugfs_entry = entry; } - mutex_unlock(&shrinker_mutex); + up_write(&shrinker_rwsem); kfree_const(old); @@ -242,7 +242,7 @@ struct dentry *shrinker_debugfs_detach(struct shrinker *shrinker, { struct dentry *entry = shrinker->debugfs_entry; - lockdep_assert_held(&shrinker_mutex); + lockdep_assert_held(&shrinker_rwsem); kfree_const(shrinker->name); shrinker->name = NULL; @@ -271,14 +271,14 @@ static int __init shrinker_debugfs_init(void) shrinker_debugfs_root = dentry; /* Create debugfs entries for shrinkers registered at boot */ - mutex_lock(&shrinker_mutex); + down_write(&shrinker_rwsem); list_for_each_entry(shrinker, &shrinker_list, list) if (!shrinker->debugfs_entry) { ret = shrinker_debugfs_add(shrinker); if (ret) break; } - mutex_unlock(&shrinker_mutex); + up_write(&shrinker_rwsem); return ret; } diff --git a/mm/vmscan.c b/mm/vmscan.c index 6d0cd2840cf0..4730dba253c8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -35,7 +35,7 @@ #include #include #include -#include +#include #include #include #include @@ -190,7 +190,7 @@ struct scan_control { int vm_swappiness = 60; LIST_HEAD(shrinker_list); -DEFINE_MUTEX(shrinker_mutex); +DECLARE_RWSEM(shrinker_rwsem); DEFINE_SRCU(shrinker_srcu); static atomic_t shrinker_srcu_generation = ATOMIC_INIT(0); @@ -213,7 +213,7 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, { return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, &shrinker_srcu, - lockdep_is_held(&shrinker_mutex)); + lockdep_is_held(&shrinker_rwsem)); } static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, @@ -292,7 +292,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) int nid, size, ret = 0; int map_size, defer_size = 0; - mutex_lock(&shrinker_mutex); + down_write(&shrinker_rwsem); map_size = shrinker_map_size(shrinker_nr_max); defer_size = shrinker_defer_size(shrinker_nr_max); size = map_size + defer_size; @@ -308,7 +308,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) info->map_nr_max = shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } - mutex_unlock(&shrinker_mutex); + up_write(&shrinker_rwsem); return ret; } @@ -324,7 +324,7 @@ static int expand_shrinker_info(int new_id) if (!root_mem_cgroup) goto out; - lockdep_assert_held(&shrinker_mutex); + lockdep_assert_held(&shrinker_rwsem); map_size = shrinker_map_size(new_nr_max); defer_size = shrinker_defer_size(new_nr_max); @@ -374,7 +374,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) if (mem_cgroup_disabled()) return -ENOSYS; - mutex_lock(&shrinker_mutex); + down_write(&shrinker_rwsem); id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -388,7 +388,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) shrinker->id = id; ret = 0; unlock: - mutex_unlock(&shrinker_mutex); + up_write(&shrinker_rwsem); return ret; } @@ -398,7 +398,7 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) BUG_ON(id < 0); - lockdep_assert_held(&shrinker_mutex); + lockdep_assert_held(&shrinker_rwsem); idr_remove(&shrinker_idr, id); } @@ -433,7 +433,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) parent = root_mem_cgroup; /* Prevent from concurrent shrinker_info expand */ - mutex_lock(&shrinker_mutex); + down_write(&shrinker_rwsem); for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); @@ -442,7 +442,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - mutex_unlock(&shrinker_mutex); + up_write(&shrinker_rwsem); } static bool cgroup_reclaim(struct scan_control *sc) @@ -743,9 +743,9 @@ void free_prealloced_shrinker(struct shrinker *shrinker) shrinker->name = NULL; #endif if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - mutex_lock(&shrinker_mutex); + down_write(&shrinker_rwsem); unregister_memcg_shrinker(shrinker); - mutex_unlock(&shrinker_mutex); + up_write(&shrinker_rwsem); return; } @@ -755,11 +755,11 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { - mutex_lock(&shrinker_mutex); + down_write(&shrinker_rwsem); list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |= SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); - mutex_unlock(&shrinker_mutex); + up_write(&shrinker_rwsem); } static int __register_shrinker(struct shrinker *shrinker) @@ -810,13 +810,13 @@ void unregister_shrinker(struct shrinker *shrinker) if (!(shrinker->flags & SHRINKER_REGISTERED)) return; - mutex_lock(&shrinker_mutex); + down_write(&shrinker_rwsem); list_del_rcu(&shrinker->list); shrinker->flags &= ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry = shrinker_debugfs_detach(shrinker, &debugfs_id); - mutex_unlock(&shrinker_mutex); + up_write(&shrinker_rwsem); atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); From patchwork Fri Jun 9 08:15:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13273416 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEF5AC7EE2E for ; Fri, 9 Jun 2023 08:17:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 797788E000B; Fri, 9 Jun 2023 04:17:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 721078E0003; Fri, 9 Jun 2023 04:17:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C23B8E000B; Fri, 9 Jun 2023 04:17:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 481078E0003 for ; Fri, 9 Jun 2023 04:17:11 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 19FF314013C for ; Fri, 9 Jun 2023 08:17:11 +0000 (UTC) X-FDA: 80882504262.14.7D01E26 Received: from out-58.mta0.migadu.com (out-58.mta0.migadu.com [91.218.175.58]) by imf30.hostedemail.com (Postfix) with ESMTP id 421EB80015 for ; Fri, 9 Jun 2023 08:17:09 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=wuAhbqfY; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf30.hostedemail.com: domain of qi.zheng@linux.dev designates 91.218.175.58 as permitted sender) smtp.mailfrom=qi.zheng@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686298629; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=h+9C3yhXhe+HsjvGtX6Jxz3ecYmnsu4DemmWgRFXfj8=; b=sy4foSZhKGBqlgVeHf+1yGa1w70GzwvF80dT3th7U5ZOl1dss3bh9CGs9cEtk4vx9pqZY0 +dGvowywQmgX5yYjMgWsO1Zg2sxk5sbAyStB2XwzA/VfwrY15z94ttWymMc961Xo0fb9G0 GMHK8RWJptQnhf3Y/f3QCjqGY0TWPfk= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=wuAhbqfY; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf30.hostedemail.com: domain of qi.zheng@linux.dev designates 91.218.175.58 as permitted sender) smtp.mailfrom=qi.zheng@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686298629; a=rsa-sha256; cv=none; b=VjCEEkJtklfF/1VLSkQpGnvRON8osgfpStt5PUl+1G6ELSCP8k8mpzQK3kKyGo374x2Im5 VASBWgKSsj89VwLjEfv6gGmk4JVpynLjS0pRz+7F/6qsLwknSYHjIV63VYPshR1ieHCLaT M56UmmwsWDYyfV0UXHkj+XmWTo0sQu4= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686298627; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h+9C3yhXhe+HsjvGtX6Jxz3ecYmnsu4DemmWgRFXfj8=; b=wuAhbqfYpfLg4xQIqqPuVD2Rfwpmfb6tYYJSrXiOZnhHIakZm73KLHKdJ6ypEmmwJn/sYE EYRhoFMKrGVqbr6Pte8aRyp7U9WXdtvlr2I/JW1Y/9T0TEaerWHAcw46nLvdes6f6TOMB6 jHd4kdRFfCPCSHABsGW0lzVmG/1Xkxw= From: Qi Zheng To: akpm@linux-foundation.org Cc: david@fromorbit.com, tkhai@ya.ru, roman.gushchin@linux.dev, vbabka@suse.cz, muchun.song@linux.dev, yujie.liu@intel.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 2/7] Revert "mm: vmscan: remove shrinker_rwsem from synchronize_shrinkers()" Date: Fri, 9 Jun 2023 08:15:13 +0000 Message-Id: <20230609081518.3039120-3-qi.zheng@linux.dev> In-Reply-To: <20230609081518.3039120-1-qi.zheng@linux.dev> References: <20230609081518.3039120-1-qi.zheng@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 421EB80015 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: cgraope53uq8ck7119osa18kjmfqjdsb X-HE-Tag: 1686298629-885771 X-HE-Meta: U2FsdGVkX18mKukhXxzMuoolExkb5/ESLZ9IE7vg2DW3Trhkp1qurh2NHEtaEOVhPu5k8lfNjTfkMZ2vvr4qfKbyWQkP/WlgdQ5/Dv2xFKV795q98+m3DG7d9tGX4KWpZ7Nac+fu6/4wsp4qgGaZgkj7U2t9Y5TmBDU4FtPa1uieT89lN8l4j+D8tyS1Qaj3E9Tu2sz7jSCHbk+xTMtCmx2FSaf3YzRwSW4ottDN3n0GxeA4y0W6ph/8VBEdW5gIUpiJ2wGWqSC/74AJjjI1i74rvPM5St0sCI6OFZovpDqt46GxE39ZU5aPHbWrV2kXGk4SzlKJ1735yXf6pBb0cck+/XYAZLszUOgdb0c1foqi8clVwDQ8Fbrigd5wIawn17YFpZi5g1i96hUBXgXF1/+SSsYAv2tVIB+SkGV2cPgB9CaJecLAaX0bmWQQ9c47EoWf/9qTtvj4ea6rmMsvvUpoEcxPI/7bH8HtqI+3jUP0XzFB7EFJQ4OS2sQ8XHQ38152rh9MDuYhjOhRZ8fB8uS9Edw076rinAZDLjnqGeN/3Qpth4ZKZ6iGZGHlFxcvzm1aD8PL42uEC8XGX1lb5Uv6hnFQIH9MFoO/ubdQS5o8ZcwDwYGZ3vDlhnb6z3KnBZyCU9DjkUUTsTIZb6o7nt8RTxBfByqaqrI2c7GJPKpLenJ/sETJ/yqC7Q3YdKwcwXncfTmmc9orUK3b4WHvpM88x9z9DBjIgWHQuXTCak4Tss7nn0qXlv37lk74XhXdMs9MFatsm/8MzNypdIaSLdpaytG+yEtfQkI841E2XeF/lHDb5uKpyytx9++RtsPrSqfxl3fw1J0z4+NHdSSYtatA1iX++BEAoZd+y4vZB6cWyI+uAu0OuTHXOt6GuS4lEngLWSVGdIhPigr2Pw2xIdkbSHR9C8e3yALd7Hpq/VBmO0eFvIzoPGJ/jHgZwokq7sPdyItOsEt5hE0ZdBL 3iZA8PdJ R14XeSlEKPiYIjP9yUvy/ZL66csQxENpTAyt34DSfeHUACR3sNcyIcFTYFJoHPy45zKCjT16rhCPN/LfqcZvVesRGo+2z/viUEZTr3A6f5QI40QjD19P7Jxnq4pB5whzcTIdJ7gGePwO8HZ8tZ1xFGpNCoVfOzAoXr3QHQryTEt8xGP6BhjtbevmWKJhvkfMm2KLBUIVRK9v1dTDlZo+ikPzFSYmKPLcVttSpeFstUICSyqUaTkEQoskBE1usE8d+AdNlyttrv5VBvE4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Qi Zheng This reverts commit 1643db98d9b314e0a592d152603094fbf7ab906e. Kernel test robot reports -88.8% regression in stress-ng.ramfs.ops_per_sec test case [1], which is caused by commit f95bdb700bc6 ("mm: vmscan: make global slab shrink lockless"). The root cause is that SRCU has to be careful to not frequently check for SRCU read-side critical section exits. Therefore, even if no one is currently in the SRCU read-side critical section, synchronize_srcu() cannot return quickly. That's why unregister_shrinker() has become slower. We will try to use the refcount+RCU method [2] proposed by Dave Chinner to continue to re-implement the lockless slab shrink. So we still need shrinker_rwsem in synchronize_shrinkers() after reverting the shrinker_srcu related changes. [1]. https://lore.kernel.org/lkml/202305230837.db2c233f-yujie.liu@intel.com/ [2]. https://lore.kernel.org/lkml/ZIJhou1d55d4H1s0@dread.disaster.area/ Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202305230837.db2c233f-yujie.liu@intel.com Signed-off-by: Qi Zheng --- mm/vmscan.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 4730dba253c8..0ba0e1180f3f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -831,11 +831,15 @@ EXPORT_SYMBOL(unregister_shrinker); /** * synchronize_shrinkers - Wait for all running shrinkers to complete. * - * This is useful to guarantee that all shrinker invocations have seen an - * update, before freeing memory. + * This is equivalent to calling unregister_shrink() and register_shrinker(), + * but atomically and with less overhead. This is useful to guarantee that all + * shrinker invocations have seen an update, before freeing memory, similar to + * rcu. */ void synchronize_shrinkers(void) { + down_write(&shrinker_rwsem); + up_write(&shrinker_rwsem); atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); } From patchwork Fri Jun 9 08:15:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13273417 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A161C7EE25 for ; Fri, 9 Jun 2023 08:17:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B40DB8E000C; Fri, 9 Jun 2023 04:17:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ACCFE8E0003; Fri, 9 Jun 2023 04:17:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 96B298E000C; Fri, 9 Jun 2023 04:17:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 81ECF8E0003 for ; Fri, 9 Jun 2023 04:17:16 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4CD4E140173 for ; Fri, 9 Jun 2023 08:17:16 +0000 (UTC) X-FDA: 80882504472.27.8E1DE0F Received: from out-33.mta0.migadu.com (out-33.mta0.migadu.com [91.218.175.33]) by imf21.hostedemail.com (Postfix) with ESMTP id 95FAA1C0006 for ; Fri, 9 Jun 2023 08:17:14 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=jXFgIXzL; spf=pass (imf21.hostedemail.com: domain of qi.zheng@linux.dev designates 91.218.175.33 as permitted sender) smtp.mailfrom=qi.zheng@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686298634; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=z1jd6DS57tmtOuv1f+tPHUMa7LOeeEWjdKx6Y4gNtNw=; b=GNXSa6G5dXHS9DWtQLwJSSW4yC2yrPZHnvG2fM96ewAd3JH/ZfxCyYcWIeLfA/8NQDUQG9 OSykR3a43KNITyWNiBt52VxAUxJ7xkvyYpKTffvea89/JyS5fBTdf0Q8OjU3BvpB9so28p 4nzWZaaxdn1D/fHDzqsU9/7UUYpPfyI= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=jXFgIXzL; spf=pass (imf21.hostedemail.com: domain of qi.zheng@linux.dev designates 91.218.175.33 as permitted sender) smtp.mailfrom=qi.zheng@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686298634; a=rsa-sha256; cv=none; b=yAfC0BKdvg0Y88Z5feuoqwbVLRMfFEbJL26PLA/tfsxeLdnkEWd3T5glQo8HFJfmfGY9o9 gXv646ILQ8Jh74iFPTRHGKHjucnsuJm/7tKZZY7tRgW1le68i80w2Dk97OYMb/+F7mfJla ccST9fM+YCHlb4Gc0vVwPjznjaCCg5w= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686298633; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z1jd6DS57tmtOuv1f+tPHUMa7LOeeEWjdKx6Y4gNtNw=; b=jXFgIXzLdDJojoKAItNEv34otXyCWjyar4fawUm43LXwyCFBcR8kY9Gpt4WbtUzDcH3jkd /70kJs6ZiU9T/QhpzSJOCDGSO1yj+OA/JWTW7Qjwmfr5MuKM15sXtrYty/nAw5lTGHgi5v rbaXvSWMUUp8lu38yF2ylUnJkp1peeY= From: Qi Zheng To: akpm@linux-foundation.org Cc: david@fromorbit.com, tkhai@ya.ru, roman.gushchin@linux.dev, vbabka@suse.cz, muchun.song@linux.dev, yujie.liu@intel.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 3/7] Revert "mm: vmscan: hold write lock to reparent shrinker nr_deferred" Date: Fri, 9 Jun 2023 08:15:14 +0000 Message-Id: <20230609081518.3039120-4-qi.zheng@linux.dev> In-Reply-To: <20230609081518.3039120-1-qi.zheng@linux.dev> References: <20230609081518.3039120-1-qi.zheng@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 95FAA1C0006 X-Rspam-User: X-Stat-Signature: jeioyo9ywg1qknium6hgnu9areneehuz X-Rspamd-Server: rspam01 X-HE-Tag: 1686298634-597412 X-HE-Meta: U2FsdGVkX18vPuNtGKLeFEPAKY6lcLguNhP6jLr/J6kx5o19HImebA8NFgKDrDI410E79ivq7cfMj0GJhEef77DJJZXlRMnN2QGW3vfwtxzxiLUYqYFo2WfMzsP2mzfiB81dSCEQqgqw3NH0Y1osyCRINFbH+FC40ipuTRdyyheZW/MYY8x59+oq5Mmy/Pyb14EFKJ0fSjmUX+tZTX4XQ0dZ3oL6fY4lHyUD3nTC0jUxYS/vJFy9w6YNAmZyXXphX5Rt/YcKRW7PFqlTXCyvT7AV+Tt2nHWocx9ZXkb4TEy4nQqkRbmc6GFikK08IqU7gBC33jtzhfcJ0r98rDaclstYBXCYE8xBGpQMQLRwtBiJh/bJTmmCDPcGC4/yWfPM5oezyru0aEhLDA6n2ufppzns2m9pJ89YRtNdsT7rH2mD7sJ+9TQgupzsd1B58qKie1uZUr0CiOC1/Iu3zgHcmtBNb3BzDNJprl6Xh402ftPMvNf/eLFtIFbBVNk4tHiTlJE5v9wmRC9B5R1bt5SoPm1nqa6T25/iVrO67ahdhWYqe5NYCZnSlFzmUYD+RNyCwL5u7hRM6CNQ7p+/lfkxDmChD3eeTthBeLgup708ODsHVfBZWbf6e68RdmReQm9U5gjNU2Zn+kbIfKNxoMDy2D5OinqhOoFM8U0wdjsiexyt6tHniM0AEE1J4n3WMKGPcLrsx7ssAM/94FV4q2M0m3YVAUSWrA6EvyhKg31XNdejIoPypkAvZgwKND8KMuVnTYwPG0XP5EG4iw/Zbh439mgbWZtiSpqvHptpENXt/O+w5wkGNRcTQSkunnOHkn1ssx92AcAzjlq/K0z96lm+KUYD6xWNFc2nlQFDvLuLBU5SbBcKioeoGd0pXUhWAFYByi24e619eGG5rUtf8rsPyrAtPhiRJJQMoLi6NTiTf5XB7eg7AAhDYKx0xq9KHe/ujIWo9tTggVEo5/af1uM tIwy7aaW 92QUx2idjnbqbAq3VJtlVtZ9j5fGjtmBD9WJJg8YQ3ozlHA/VA2ehaA/bze/mVrXro+iNaJY1UypcMVPy7o2mtmeJo7Q/IcV6mWS9Mpj8qbuZgagQd7BEUPyqrNZNtuUeUovY1SO8whkTwi1ugYhL1iDfv7vFncpQC9cVlo5yPhHkoUba/c8DwlHhBOeL1f2GvLTOspFYl+60uHN19KaipulnKjfgJ/Ux/WpgHREi8IyQbC5pRQ/Q1+u7iXqErb4Cx/9akakoMYtKfSc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Qi Zheng This reverts commit b3cabea3c9153fd42fe5cb851ac58b51ea2b32b8. Kernel test robot reports -88.8% regression in stress-ng.ramfs.ops_per_sec test case [1], which is caused by commit f95bdb700bc6 ("mm: vmscan: make global slab shrink lockless"). The root cause is that SRCU has to be careful to not frequently check for SRCU read-side critical section exits. Therefore, even if no one is currently in the SRCU read-side critical section, synchronize_srcu() cannot return quickly. That's why unregister_shrinker() has become slower. We will try to use the refcount+RCU method [2] proposed by Dave Chinner to continue to re-implement the lockless slab shrink. Because there will be other readers after reverting the shrinker_srcu related changes, so it is better to restore to hold read lock to reparent shrinker nr_deferred. [1]. https://lore.kernel.org/lkml/202305230837.db2c233f-yujie.liu@intel.com/ [2]. https://lore.kernel.org/lkml/ZIJhou1d55d4H1s0@dread.disaster.area/ Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202305230837.db2c233f-yujie.liu@intel.com Signed-off-by: Qi Zheng --- mm/vmscan.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 0ba0e1180f3f..d1d309fc3212 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -433,7 +433,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) parent = root_mem_cgroup; /* Prevent from concurrent shrinker_info expand */ - down_write(&shrinker_rwsem); + down_read(&shrinker_rwsem); for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); @@ -442,7 +442,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_write(&shrinker_rwsem); + up_read(&shrinker_rwsem); } static bool cgroup_reclaim(struct scan_control *sc) From patchwork Fri Jun 9 08:15:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13273418 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A3F6C7EE2E for ; Fri, 9 Jun 2023 08:17:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 092958E000D; Fri, 9 Jun 2023 04:17:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 01C158E0003; Fri, 9 Jun 2023 04:17:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DFF058E000D; Fri, 9 Jun 2023 04:17:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CDC268E0003 for ; Fri, 9 Jun 2023 04:17:21 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 9CECEA0147 for ; Fri, 9 Jun 2023 08:17:21 +0000 (UTC) X-FDA: 80882504682.21.0F3E2CC Received: from out-13.mta0.migadu.com (out-13.mta0.migadu.com [91.218.175.13]) by imf06.hostedemail.com (Postfix) with ESMTP id C6733180005 for ; Fri, 9 Jun 2023 08:17:19 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=tddMOX9h; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf06.hostedemail.com: domain of qi.zheng@linux.dev designates 91.218.175.13 as permitted sender) smtp.mailfrom=qi.zheng@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686298640; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fW5UuGuy06FRX8Px/N9YG/QTNF0coGlqdNSj4AuBaaM=; b=Hg2YXWSmoFWaxrjVrrB0VQJny2k+PBPUv2ak6n8K2W5kEwiw4Fnj14pNqPXCxQl9E2qiJw 6QXKX4SDDCvhy5gSYVy+WtcSXgI3db1hjy+tsBAUzkvbXEfHc2s8516qbk5ykyonqqNLDq G5Xieoncdrn8MqkU57VT4pbrzeyoKwY= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=tddMOX9h; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf06.hostedemail.com: domain of qi.zheng@linux.dev designates 91.218.175.13 as permitted sender) smtp.mailfrom=qi.zheng@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686298640; a=rsa-sha256; cv=none; b=dpf9CQvfJAZe6mfG9nDngb6s+W/LTfbbuXN1xyWqiq47C2301lMxVQGqYNvZi94DYO4jAb WC1kwwOIT569Do3/IXKsfIDUpLvEIRMeVytXXR2lpRzZOx5JwzlUu+wYqXlKFkXsyTH9lR WJxK7D00bgt4xJXsXep1UysuvPo58Xo= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686298638; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fW5UuGuy06FRX8Px/N9YG/QTNF0coGlqdNSj4AuBaaM=; b=tddMOX9hrmBcUy1TaN6PYmjZr8N6htWv1AKIwBbCiMwwdMPB0GeJMYgG8Z8I93aOe/HBvd 5BXFnyv6bfT5hursOmsbSE208IL5wiQKL+bwZlti5+T99GzbUZ4tm7iuHx6iPsFe8QHCGt jtNfOoIRqSFtiW109WCZ6dTaNr7m36M= From: Qi Zheng To: akpm@linux-foundation.org Cc: david@fromorbit.com, tkhai@ya.ru, roman.gushchin@linux.dev, vbabka@suse.cz, muchun.song@linux.dev, yujie.liu@intel.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 4/7] Revert "mm: shrinkers: make count and scan in shrinker debugfs lockless" Date: Fri, 9 Jun 2023 08:15:15 +0000 Message-Id: <20230609081518.3039120-5-qi.zheng@linux.dev> In-Reply-To: <20230609081518.3039120-1-qi.zheng@linux.dev> References: <20230609081518.3039120-1-qi.zheng@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: C6733180005 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 36dg36no37mjtsw8br36uziu6ruzy9bx X-HE-Tag: 1686298639-478269 X-HE-Meta: U2FsdGVkX1+RtJnq2x7WaFPOZcqu/K3sGhuG8SpAwxfDl8cv3hTovMklcRwN2oZhu3qyWn+F69y5b5ZkMiiI7RblKJA0F0dKclpPcQVn0GE4qWL9QjVsqnLe5Q1/9Qa8TJVYjrKzkx5577athAog1MLxS0xe7q6bij+vdStOKGxv0YDOZGW/dYAC6Vss2ea6fILnBM7Rlxm9RDYcuWaPS4/XJ1eKhiB/ypCkgZbm6P24Nv2GWLYmcmbwNXCW7gzAD4qUBTi3TFA921GLUDV0XoVXa5dgka2vE6+CVbtCmGrnO5PIeAv9YvbA5IjPZIrOq4vYDwxwA+7Fhxb3atdkFWK9Xv3rfq/534k3WtblDge8egQWKmvVS5PT4qQV8dsX8nGo+smNzRNt2aCbds+T94wEE0sZB0XfmgNeMsKEI5fCZwUPJ7c6DAThActmqHgLEf0zhTYFZUI82wuOons5W23RUCZxlQPpRwgI+gGl0hBNyIAHTvfo+b3V/G3yhWSTFkyORUAiDwut5g5LMwyNPhfFPu5X85+/a6/x81d8Pl2/a9lhk8HBMOB6f7PzAl4oqrHXSW+kKS4izyw5G7875EpZJqgKuhqnnpJn8H1wCjCvDNYsbMpmykVFhXw/B2hjaL+GlyvL9yeOz7YwnZWFQhqGWJe/+RomcSlkicp+nGdyMrL//xXU0l8PFnOow0SKatN51bLd/3+KYO5CWOgjJ6nc72tU5N6GedCn13prXj43T5140YDXJ9GfFFixwSe/pEI7+lTbDCYIN/DBD3jA/n4lOSk1517QqYBE8FT/69c2gl+Pkin9jMa83SmttisTTCFu+UGQmH3oeKi+5tUXQIRqNGoVAavBMjUaWhPv+2xLePXZ+1IBgeXyq+/jL3JMIxhu2XGHJBAfFYrSG8FBHR/Ztm9XGecpufQmrhSrgAtRGcwLdqTUtx3eTduL9xsPG3ry5qQyi/gQ4vO0LB9 BMC3+xTy tGWkRZKGytwS64N8TjpZXr32KlsOpUpF+i4MBbWBUyayEPFunaQNT2tmuVPxLjPl4HlJmhSXAdKXtLUGmICcmwRcp4jelpt3L05gNPx+oI1B9px6Ofl1ymzuiGIm3bhxSm9C6DWEa/7vcm8aR1L1eJubzhifLGYpAe5p/WLPfX+KJanPjG1K3qsNYUMqvLkwsrj4rif7+3IQ+eBc67xGY77CYxzfDpZT5dU3i X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Qi Zheng This reverts commit 20cd1892fcc3efc10a7ac327cc3790494bec46b5. Kernel test robot reports -88.8% regression in stress-ng.ramfs.ops_per_sec test case [1], which is caused by commit f95bdb700bc6 ("mm: vmscan: make global slab shrink lockless"). The root cause is that SRCU has to be careful to not frequently check for SRCU read-side critical section exits. Therefore, even if no one is currently in the SRCU read-side critical section, synchronize_srcu() cannot return quickly. That's why unregister_shrinker() has become slower. We will try to use the refcount+RCU method [2] proposed by Dave Chinner to continue to re-implement the lockless slab shrink. So revert the shrinker_srcu related changes first. [1]. https://lore.kernel.org/lkml/202305230837.db2c233f-yujie.liu@intel.com/ [2]. https://lore.kernel.org/lkml/ZIJhou1d55d4H1s0@dread.disaster.area/ Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202305230837.db2c233f-yujie.liu@intel.com Signed-off-by: Qi Zheng --- mm/shrinker_debug.c | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 2be15b8a6d0b..3ab53fad8876 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -5,12 +5,10 @@ #include #include #include -#include /* defined in vmscan.c */ extern struct rw_semaphore shrinker_rwsem; extern struct list_head shrinker_list; -extern struct srcu_struct shrinker_srcu; static DEFINE_IDA(shrinker_debugfs_ida); static struct dentry *shrinker_debugfs_root; @@ -51,13 +49,18 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v) struct mem_cgroup *memcg; unsigned long total; bool memcg_aware; - int ret = 0, nid, srcu_idx; + int ret, nid; count_per_node = kcalloc(nr_node_ids, sizeof(unsigned long), GFP_KERNEL); if (!count_per_node) return -ENOMEM; - srcu_idx = srcu_read_lock(&shrinker_srcu); + ret = down_read_killable(&shrinker_rwsem); + if (ret) { + kfree(count_per_node); + return ret; + } + rcu_read_lock(); memcg_aware = shrinker->flags & SHRINKER_MEMCG_AWARE; @@ -88,7 +91,8 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v) } } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); - srcu_read_unlock(&shrinker_srcu, srcu_idx); + rcu_read_unlock(); + up_read(&shrinker_rwsem); kfree(count_per_node); return ret; @@ -111,8 +115,9 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, .gfp_mask = GFP_KERNEL, }; struct mem_cgroup *memcg = NULL; - int nid, srcu_idx; + int nid; char kbuf[72]; + ssize_t ret; read_len = size < (sizeof(kbuf) - 1) ? size : (sizeof(kbuf) - 1); if (copy_from_user(kbuf, buf, read_len)) @@ -141,7 +146,11 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, return -EINVAL; } - srcu_idx = srcu_read_lock(&shrinker_srcu); + ret = down_read_killable(&shrinker_rwsem); + if (ret) { + mem_cgroup_put(memcg); + return ret; + } sc.nid = nid; sc.memcg = memcg; @@ -150,7 +159,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, shrinker->scan_objects(shrinker, &sc); - srcu_read_unlock(&shrinker_srcu, srcu_idx); + up_read(&shrinker_rwsem); mem_cgroup_put(memcg); return size; From patchwork Fri Jun 9 08:15:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13273419 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B5D6C7EE2E for ; Fri, 9 Jun 2023 08:17:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CA21C8E000E; Fri, 9 Jun 2023 04:17:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C2A848E0003; Fri, 9 Jun 2023 04:17:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ACB208E000E; Fri, 9 Jun 2023 04:17:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 98CA28E0003 for ; Fri, 9 Jun 2023 04:17:28 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 70331AE67A for ; Fri, 9 Jun 2023 08:17:28 +0000 (UTC) X-FDA: 80882504976.11.7F0117A Received: from out-33.mta0.migadu.com (out-33.mta0.migadu.com [91.218.175.33]) by imf05.hostedemail.com (Postfix) with ESMTP id A369C100011 for ; Fri, 9 Jun 2023 08:17:26 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=dUWb27UU; spf=pass (imf05.hostedemail.com: domain of qi.zheng@linux.dev designates 91.218.175.33 as permitted sender) smtp.mailfrom=qi.zheng@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686298646; a=rsa-sha256; cv=none; b=ayDpNUkPc/8OIsZSn+z+8BlQhXaWMwsobjxt2NyKhsieLG9vbpQgd0KuD9/1RRmjhEwgT0 ywp44iz+9i2MkqYfCFEaIET8STPAezev3OoxchrzIGdgfLXA2ONf9euxStI97kyQxonbVZ rUVkmx5Bbj0XxnS50xXd9sqwWpMpcuA= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=dUWb27UU; spf=pass (imf05.hostedemail.com: domain of qi.zheng@linux.dev designates 91.218.175.33 as permitted sender) smtp.mailfrom=qi.zheng@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686298646; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ptGmpfMk/kHVim/COA1N68hD8MGHSDZD7U0gMdC3pxY=; b=q1CRHfAlqbhEDqFQAuygHB7CqdF3DPIkQUWH5ej2/geJsSVRnQSSFRcM/oI2WzhlBQOFc6 sinImrAVXZ0lAz8xxienyyxLr8SbkrT3vLQKH4M/YYWNxieQ52+z9t4Ib28yXOwCrJyilo OoqBAd1sP09U7zRUM+aKJbe0ws2Plx0= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686298645; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ptGmpfMk/kHVim/COA1N68hD8MGHSDZD7U0gMdC3pxY=; b=dUWb27UUXt+8QDbQIc9tustr3Ex0BabLxTJee5D+fXjeHZBQ4qcFjuXfmS9deSWg9pnPCR 9dTR+kP4JvZRsxWTBrikjSS0f2Fm0CUyHJiy5VG+bhRcgClpEyEjDlHhnKlTs1O4PWyCUu 6wpdq+dene7opH5wdk+agSCOHXRmYcg= From: Qi Zheng To: akpm@linux-foundation.org Cc: david@fromorbit.com, tkhai@ya.ru, roman.gushchin@linux.dev, vbabka@suse.cz, muchun.song@linux.dev, yujie.liu@intel.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 5/7] Revert "mm: vmscan: add shrinker_srcu_generation" Date: Fri, 9 Jun 2023 08:15:16 +0000 Message-Id: <20230609081518.3039120-6-qi.zheng@linux.dev> In-Reply-To: <20230609081518.3039120-1-qi.zheng@linux.dev> References: <20230609081518.3039120-1-qi.zheng@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: A369C100011 X-Stat-Signature: mq8cthua1f7q7opck9tr7o65jwpjmmuu X-Rspam-User: X-HE-Tag: 1686298646-312968 X-HE-Meta: U2FsdGVkX1/XtFe1W3PWmW+Mk9vf+ubXsxdsz1gxPcIzYkgn6mYxsivQsrRVXwS3JO4pNPzLTCwVrGYKwrBeNKNkoORGwJv/nOWujt4JubGsr7naO3hVYCo7MgylIjaWO6kJ4t0r0MrtorS2f4NwQi1ZDqwPv8qfCvsL90/CH+5qQaUgslhnVeLqdIJ+RqQserQMOyQRheLpY2IQfE5/j211DlRXf0sC9PjGQuHGMjeNaxyJ/hCBoxChOlahikY93TQcxGgU1iH697iAxpiI0H04szom7cEhslUfml5lW0vlxJ2srHRMk+nU4xaQT8YpBuzbR2NNFfRZdDxKrumw6FcW7Ut3YnkkS/W7KypjdXlzmIruseUGXDMV3uSqsgb7NIixQ0pXZu2F2IMV3wJGBHnIaVZiO5Xx5zCjYQ/NktlVAulPxaXDhJWm+ZQ54uhxtpqrrKxzQa3uY7Om2xESNqIBFnDmGjOUnUXLmCXBTSF2Veti8dQCXTlp23s5AWOvff2uVPTzaaoKD/z6oJ7pWfXLkv37g5QKHsoy9voigeVFopOiOkv7n8lfUoeW71xQVEU20+s32VRdGGZgm00CKYbYN3DmQXJiqN2+0mE36yVFVvFqPipE8REn/SfRaD4VQ86URurlkeJKME+IVxAy+GfpoYCwJqOhxdKS9vJoonW+nET5m9PxyZdyG+4q9p/S+JbaS2purHB9D3Jebkir4tLmEwve2gSr0wWJ/fHJwfio9vWzmHUXcmD2M0qpPjpBw6esd1Tdc1+CR3kdusuNbhvG7JFuO42ESL7MBsgUgMvAOsBodEpEfSttCNroJ7CaCR9ff2YkmVeVwXoiik6L5/TEzRdIkv/gGykTVZGdLtapC13OmATOM95cBVFAod54IYWuF1xB9Fq84QShEkeQ0S4RBjfoC2c8CO6hhUcIOYWea4ugeOyjASS6D/ztsl9xbdnl8IxF1SUiOiv1BGF hvnLM9z2 a95BdboFbD+9iEm+A2dkj3XOgpoinBikyIKAuwraKYUm4BipqHAJB/xPHaUH9/h7HG7zgwPg7VN8b/bL5sZcP5Y6BDpf5b8gEG9QelesXtxAwib8UkK+0z2w2JHlhKCZ6f6Z/GAEmHkCIRrM+gvUvuGBjgcNHOv4ThJzm1hE0Q1ng16qwxv9Fw5pSfNkDCFmqJ8clkP1pmjHzVG0aHKO3ysc5VwKClfYIhkO12SL5cFICmc9iJmuhzmka8eyDcZnphtB02eLSCnJggKg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Qi Zheng This reverts commit 475733dda5aedba9e086379aafe6b5ffd53e8f5e. Kernel test robot reports -88.8% regression in stress-ng.ramfs.ops_per_sec test case [1], which is caused by commit f95bdb700bc6 ("mm: vmscan: make global slab shrink lockless"). The root cause is that SRCU has to be careful to not frequently check for SRCU read-side critical section exits. Therefore, even if no one is currently in the SRCU read-side critical section, synchronize_srcu() cannot return quickly. That's why unregister_shrinker() has become slower. We will try to use the refcount+RCU method [2] proposed by Dave Chinner to continue to re-implement the lockless slab shrink. So revert the shrinker_srcu related changes first. [1]. https://lore.kernel.org/lkml/202305230837.db2c233f-yujie.liu@intel.com/ [2]. https://lore.kernel.org/lkml/ZIJhou1d55d4H1s0@dread.disaster.area/ Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202305230837.db2c233f-yujie.liu@intel.com Signed-off-by: Qi Zheng --- mm/vmscan.c | 24 ++++-------------------- 1 file changed, 4 insertions(+), 20 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index d1d309fc3212..50775b73d0c7 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -192,7 +192,6 @@ int vm_swappiness = 60; LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); DEFINE_SRCU(shrinker_srcu); -static atomic_t shrinker_srcu_generation = ATOMIC_INIT(0); #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -818,7 +817,6 @@ void unregister_shrinker(struct shrinker *shrinker) debugfs_entry = shrinker_debugfs_detach(shrinker, &debugfs_id); up_write(&shrinker_rwsem); - atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); shrinker_debugfs_remove(debugfs_entry, debugfs_id); @@ -840,7 +838,6 @@ void synchronize_shrinkers(void) { down_write(&shrinker_rwsem); up_write(&shrinker_rwsem); - atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); @@ -950,20 +947,18 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, { struct shrinker_info *info; unsigned long ret, freed = 0; - int srcu_idx, generation; - int i = 0; + int srcu_idx; + int i; if (!mem_cgroup_online(memcg)) return 0; -again: srcu_idx = srcu_read_lock(&shrinker_srcu); info = shrinker_info_srcu(memcg, nid); if (unlikely(!info)) goto unlock; - generation = atomic_read(&shrinker_srcu_generation); - for_each_set_bit_from(i, info->map, info->map_nr_max) { + for_each_set_bit(i, info->map, info->map_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -1009,11 +1004,6 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, set_shrinker_bit(memcg, nid, i); } freed += ret; - if (atomic_read(&shrinker_srcu_generation) != generation) { - srcu_read_unlock(&shrinker_srcu, srcu_idx); - i++; - goto again; - } } unlock: srcu_read_unlock(&shrinker_srcu, srcu_idx); @@ -1053,7 +1043,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, { unsigned long ret, freed = 0; struct shrinker *shrinker; - int srcu_idx, generation; + int srcu_idx; /* * The root memcg might be allocated even though memcg is disabled @@ -1067,7 +1057,6 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, srcu_idx = srcu_read_lock(&shrinker_srcu); - generation = atomic_read(&shrinker_srcu_generation); list_for_each_entry_srcu(shrinker, &shrinker_list, list, srcu_read_lock_held(&shrinker_srcu)) { struct shrink_control sc = { @@ -1080,11 +1069,6 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; freed += ret; - - if (atomic_read(&shrinker_srcu_generation) != generation) { - freed = freed ? : 1; - break; - } } srcu_read_unlock(&shrinker_srcu, srcu_idx); From patchwork Fri Jun 9 08:15:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13273420 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0639EC83005 for ; Fri, 9 Jun 2023 08:17:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 91D9F8E000F; Fri, 9 Jun 2023 04:17:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8A8058E0003; Fri, 9 Jun 2023 04:17:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 747FA8E000F; Fri, 9 Jun 2023 04:17:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 61C1A8E0003 for ; Fri, 9 Jun 2023 04:17:32 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3345440173 for ; Fri, 9 Jun 2023 08:17:32 +0000 (UTC) X-FDA: 80882505144.17.8D35FD0 Received: from out-1.mta0.migadu.com (out-1.mta0.migadu.com [91.218.175.1]) by imf22.hostedemail.com (Postfix) with ESMTP id 5B4A4C0015 for ; Fri, 9 Jun 2023 08:17:30 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=Hp4UWFqQ; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf22.hostedemail.com: domain of qi.zheng@linux.dev designates 91.218.175.1 as permitted sender) smtp.mailfrom=qi.zheng@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686298650; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=B/E+4G7xduNcnh08om90Yauw78sQGFjRZ4mRQx21A2Q=; b=VqijiapVpBPs1ho4J2FBOyhvz3/pezexzZnYcz8o43s4o8MS1AuCDB90McMCB4VhQa0Y+K UuSHpDf3NcKGAEnWeYQgPhwXsd1JzKj74IGzCZ30Pt+0aJ80mUEGjC/Gb4jnUBBMNN5eY0 NW23TdQzlvdpVR1McyYHv0L62UfhBNs= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=Hp4UWFqQ; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf22.hostedemail.com: domain of qi.zheng@linux.dev designates 91.218.175.1 as permitted sender) smtp.mailfrom=qi.zheng@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686298650; a=rsa-sha256; cv=none; b=syiS3x4tckveo6Z+0xWCURu3zrXaoIyfMoxcMbu6SA/sAN7RDX4caa9yfuYKW5w6ATCMd0 bNT1Ukr2T/BtGF2omsQhj6yAWf08QFj2FoyeMMWbVPcBMjpBuz+ltkbwAFigARg/iZ8nT7 Ns7eAZa9ntevd6t2v6wyRK8CaykfclU= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686298649; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=B/E+4G7xduNcnh08om90Yauw78sQGFjRZ4mRQx21A2Q=; b=Hp4UWFqQJil8Kf9Kbk7PpaCcvuRjBeSALwQccnwRZWKqDBGAxb1HaJdVYPnL+2SqJ79gWL Dvz4OkZmXEAisnQB6TAIkcV1qOdjUk0ksJQqHC9jb/4iwoI78cHfc6hLpTA+6ksOQfKmVx x7esjKuYGnPxyQBoj7U7f+XOMRKIYW8= From: Qi Zheng To: akpm@linux-foundation.org Cc: david@fromorbit.com, tkhai@ya.ru, roman.gushchin@linux.dev, vbabka@suse.cz, muchun.song@linux.dev, yujie.liu@intel.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 6/7] Revert "mm: vmscan: make memcg slab shrink lockless" Date: Fri, 9 Jun 2023 08:15:17 +0000 Message-Id: <20230609081518.3039120-7-qi.zheng@linux.dev> In-Reply-To: <20230609081518.3039120-1-qi.zheng@linux.dev> References: <20230609081518.3039120-1-qi.zheng@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 5B4A4C0015 X-Stat-Signature: ktjn5khhax3y6x111mizxzzwjeoozkxw X-Rspam-User: X-HE-Tag: 1686298650-627302 X-HE-Meta: U2FsdGVkX1/AJSE4Oszvxh66ztkJnJxGX6vEIDFAZV3OBPfDh1/iWwcmkYgj1KK0TH9zXBzaqHOtXei1nebJEOjmpDpXYu96UcQMP8cx8Rudvm8Sa7OZB5TmedoSx0mcNS2Hm/LwV5BHo3BM8zTv5uci7xATs5//4fDBQPyn40bbsjau8bTdCImzwgF6L/ATSDnndedXeDn/U3Cffw2zSeQNfd6VQEZzXe6kel5ftX6WPAsZQ4DTGdGAiX4G+OJU9feR39wZ4gAi8XSUQOMvWDFjpfNW6qFZ+ogo69xm9YXc7iuOV46LCepAQQf0GEmbWl5eNuNMSmR8NKGWQOBapnVHDx24H5Staj7IR0ptCCbCnB0WZPRS7E0o+NDz+u/O6hFHyMdRayawyfjsoq2NHxGbo/FRctAz7C4s7WydYYMyFDcwPa89qPj1NQxnKEtFF3onUfDpwYg/A4uMt4GyD7qfz8YjoVCdrTXCA83iG9dxNAHsnKIST5nCu4z9iOx+ZZrRhVbyzYLn4cEbQxkTLixM6kAzjUsHmMnFL3MeRiKnkVAZq3i2q6AHmavawdEnIamzsedCiyN2V3zAtxLasG32ip/L8GnDr4nibdFuLQuHJlc3FX3W81/3mO2nt0ZWzsS+u+lsbyRvPJ5+qNLFNqBHenYBsHWOzUnfHHWTUqqxYGeGVs40M9OJpCkagDZ5PAwp9jgF4j2D2u3pzT2ZJptNsD6CwQwU1COzrC/c1Rvj7co+NwPMW9HU/XgILfAbfXSgwjOQs3/ILV+s5PIyLBjQ0j6SOhband/yWSlvZ0zE3+B/cBVeZwQ81m7ig3b9j9HYxmylWtKC+R8RjVPSjKYyqjRxShqIVSzCe/DRbrl02awCQJvEIT1LFP1lU6wHiEp6jzs6xOSYuN4UoNQ77H3YMYCaZ6FQmK9KFWw3MbKUoINJEruFGW4Sp4vQtaqIiTPxGKrivLwo5IJJF9o ezptxVLd Z84VzvlJ189Ay9LStKGS9GUvQJyfWtroZajbcBsoaSsCFMrer0xmznPyTYsAejzXyu5oJZ/eCmWvDP536Rfk17KCC0QT1M8rRzlnQ/zmzC1BqLEUlFjkdzLAoxfu7+RAXaEb+5d5epRCsJnJIJhY7oFotXyCISG7WARgqs2iGunHGLf+4tamc89uV9DV6/iKmig54hvu8AOBebbP+l+y5qMkrgODFl2yEfbWb X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Qi Zheng This reverts commit caa05325c9126c77ebf114edce51536a0d0a9a08. Kernel test robot reports -88.8% regression in stress-ng.ramfs.ops_per_sec test case [1], which is caused by commit f95bdb700bc6 ("mm: vmscan: make global slab shrink lockless"). The root cause is that SRCU has to be careful to not frequently check for SRCU read-side critical section exits. Therefore, even if no one is currently in the SRCU read-side critical section, synchronize_srcu() cannot return quickly. That's why unregister_shrinker() has become slower. After discussion, we will try to use the refcount+RCU method [2] proposed by Dave Chinner to continue to re-implement the lockless slab shrink. So revert the shrinker_srcu related changes first. [1]. https://lore.kernel.org/lkml/202305230837.db2c233f-yujie.liu@intel.com/ [2]. https://lore.kernel.org/lkml/ZIJhou1d55d4H1s0@dread.disaster.area/ Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202305230837.db2c233f-yujie.liu@intel.com Signed-off-by: Qi Zheng --- mm/vmscan.c | 45 +++++++++++++++++++-------------------------- 1 file changed, 19 insertions(+), 26 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 50775b73d0c7..a008d7f2d0fc 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -210,21 +210,8 @@ static inline int shrinker_defer_size(int nr_items) static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, int nid) { - return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, - &shrinker_srcu, - lockdep_is_held(&shrinker_rwsem)); -} - -static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, - int nid) -{ - return srcu_dereference(memcg->nodeinfo[nid]->shrinker_info, - &shrinker_srcu); -} - -static void free_shrinker_info_rcu(struct rcu_head *head) -{ - kvfree(container_of(head, struct shrinker_info, rcu)); + return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, + lockdep_is_held(&shrinker_rwsem)); } static int expand_one_shrinker_info(struct mem_cgroup *memcg, @@ -265,7 +252,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, defer_size - old_defer_size); rcu_assign_pointer(pn->shrinker_info, new); - call_srcu(&shrinker_srcu, &old->rcu, free_shrinker_info_rcu); + kvfree_rcu(old, rcu); } return 0; @@ -351,16 +338,15 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) { if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { struct shrinker_info *info; - int srcu_idx; - srcu_idx = srcu_read_lock(&shrinker_srcu); - info = shrinker_info_srcu(memcg, nid); + rcu_read_lock(); + info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) { /* Pairs with smp mb in shrink_slab() */ smp_mb__before_atomic(); set_bit(shrinker_id, info->map); } - srcu_read_unlock(&shrinker_srcu, srcu_idx); + rcu_read_unlock(); } } @@ -374,6 +360,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) return -ENOSYS; down_write(&shrinker_rwsem); + /* This may call shrinker, so it must use down_read_trylock() */ id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -407,7 +394,7 @@ static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, { struct shrinker_info *info; - info = shrinker_info_srcu(memcg, nid); + info = shrinker_info_protected(memcg, nid); return atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); } @@ -416,7 +403,7 @@ static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, { struct shrinker_info *info; - info = shrinker_info_srcu(memcg, nid); + info = shrinker_info_protected(memcg, nid); return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); } @@ -947,14 +934,15 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, { struct shrinker_info *info; unsigned long ret, freed = 0; - int srcu_idx; int i; if (!mem_cgroup_online(memcg)) return 0; - srcu_idx = srcu_read_lock(&shrinker_srcu); - info = shrinker_info_srcu(memcg, nid); + if (!down_read_trylock(&shrinker_rwsem)) + return 0; + + info = shrinker_info_protected(memcg, nid); if (unlikely(!info)) goto unlock; @@ -1004,9 +992,14 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, set_shrinker_bit(memcg, nid, i); } freed += ret; + + if (rwsem_is_contended(&shrinker_rwsem)) { + freed = freed ? : 1; + break; + } } unlock: - srcu_read_unlock(&shrinker_srcu, srcu_idx); + up_read(&shrinker_rwsem); return freed; } #else /* CONFIG_MEMCG */ From patchwork Fri Jun 9 08:15:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13273421 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F23EC83005 for ; Fri, 9 Jun 2023 08:17:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B196F8E0009; Fri, 9 Jun 2023 04:17:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AA3058E0003; Fri, 9 Jun 2023 04:17:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 944198E0009; Fri, 9 Jun 2023 04:17:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7E34F8E0003 for ; Fri, 9 Jun 2023 04:17:38 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 49FB31C7318 for ; Fri, 9 Jun 2023 08:17:38 +0000 (UTC) X-FDA: 80882505396.02.CDA3FC4 Received: from out-10.mta0.migadu.com (out-10.mta0.migadu.com [91.218.175.10]) by imf15.hostedemail.com (Postfix) with ESMTP id 8525AA0012 for ; Fri, 9 Jun 2023 08:17:36 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=QnAzENdG; spf=pass (imf15.hostedemail.com: domain of qi.zheng@linux.dev designates 91.218.175.10 as permitted sender) smtp.mailfrom=qi.zheng@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686298656; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EMn+msnIJXvC+06grZ542lhB8pLG9zNhZSiyiHhRbaA=; b=1m4S4jmGqGd7GkjvrjHv6AyHX6UXj8jcIgKyoszZdxji0AUAEBbNe7vxo4YYhtnp760ANI OeRZnQ8lfGpnbOgZFuwQnqMDqDZeRgNTOCJ6yWX0eugRswap/+lj0034vFwPYwaWlz/xei tjbbAFmWZ5TPTu0pQDF4AUfNXe+nUmw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686298656; a=rsa-sha256; cv=none; b=pss/dAZHRU+r0/xmmWwKi0b67GFj0EqhN1H9cooshLjQli4as2E2qd1AtXBR2vmtIUofLe Zs/VLpcCEM/5pfN3QedhSQmm59CC2946Uuqz1e30ENTZYUzB4Plj5dOS0a4Kx4jBV9BYn7 ZwdVm1DDAtHu+S7quSaAuOAOP8fl6YI= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=QnAzENdG; spf=pass (imf15.hostedemail.com: domain of qi.zheng@linux.dev designates 91.218.175.10 as permitted sender) smtp.mailfrom=qi.zheng@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686298655; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EMn+msnIJXvC+06grZ542lhB8pLG9zNhZSiyiHhRbaA=; b=QnAzENdGYWeYO57Spgjhk1adZxJY7gan2t4wYWPeZIaItdmJZDM8xOICqPyq/QELcJD9OY hG30tKtjBO4CWu/+8SDMmqFoQybaFEZQceY8dbZCLKigPc6SX9awgQJwPEusZAlihLEw5W Hw8FqnopH5nBR0yG+Weoe/3z4ob2VOY= From: Qi Zheng To: akpm@linux-foundation.org Cc: david@fromorbit.com, tkhai@ya.ru, roman.gushchin@linux.dev, vbabka@suse.cz, muchun.song@linux.dev, yujie.liu@intel.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 7/7] Revert "mm: vmscan: make global slab shrink lockless" Date: Fri, 9 Jun 2023 08:15:18 +0000 Message-Id: <20230609081518.3039120-8-qi.zheng@linux.dev> In-Reply-To: <20230609081518.3039120-1-qi.zheng@linux.dev> References: <20230609081518.3039120-1-qi.zheng@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 8525AA0012 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: f95aq6hz41eohe493gbgekdo3xeni8ab X-HE-Tag: 1686298656-325687 X-HE-Meta: U2FsdGVkX1+VwDh9F/OQDS/6C1ppcSWY+97vzqb4XDuwiFQKjhCdCQoJ8XE+0ElLIlWWBzzAsHLntcPsNeSKAQIdGBlqBp3E3k3AYUFuQWFnagOgPqnzdsD+rwO/oVVtPgnquC9VH0LbofTvSLsEUfEi5/vG4j7vfvod7/UGZAu6mL4K8eAiBlA4inzF4zgxJeXXEh6iNtZC+r2yVqiYGrSU2VXbYfyXWEECgT3bBm9CnlLe0bXX2B3l13uUMd8TMuhJ+75lb3KWt0iEq1Bmj7UzFB/oP2BJ+s/C76vWizuqgwTozd9ZkFkgaakuIiQ0fh1IiHqif1dE8BCOHjCT0xnECIRPTF/a2sdlQLi9mwvSq01ULgLiWn3PGcgowTidpMK5ZPjZLp2zrduYA5PrEe4hxY+XACgnu5UM1bpcdX+ZezNp/usrQQZ+GoNGt1NEuogOHefdTUNffzO+m8vn36nfOdC7mVCWpt3iYK8dfZ7X45W69G9ulYFPZh58+hF48D28E0tGfvrXXNXeuHu3mGGWjnfjmuVU2GMVH33ZEhvUr2psrUBNzmDGl/d9faUz0u6hk4J29pYeSpNiHc1WM4BdndAQu959+QHCRkdvCkYvyorWIM5O2SaGy1r0qNfIxess1SdDiSp52eIOXc+djHrtU5etBG5lFY2MhKQxOjY4MKR/4/lXqJNxVnUaa/xPBqAQRiIVy4jG/hZUb6s5nY+Abq7wMVsYdgAdu4oQh1CTLhP7tX6fxVomV2jQXE6vm3TlGglxy88oObes3U9ayhLRu3F54fmP5ptGTHNwt9UYiSP8Q5AdvGoz0T2Am7Eg4GSr6z+tlhdtwIWiUIc/6NSvgQhBAxNL7IlAejlkNO9oiQNL1v5fpBczMSjE2nrf7JGwT4opQENmACbJXJkDXkxLO5xnTqA3qh5XAdRcoGt8FlqrqhIrCx8xTv4IU+EcBWxXTGUU/N3Dh9PmiFK WiMIUzhE gYJqmPayWhe+i6bf+WkBRfb3O5hq8n9hG4yOAE9zftJ/lzaxtqBf9aPBnoPsDnFfnW45bMR1caeSf487iThAzAdSXQh8VHPWJC8EpOnNOp2OoI8y1NcsVqeQIX2J04fThD8jqiAElgK6FTU38evGJEXkua2El4SuE7uixJO4mhH6ujG2E1n3s7+EXrnie3AzSND7L0Q/kkYVLxMfkUsYrhgixmPITZSQDddgM X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Qi Zheng This reverts commit f95bdb700bc6bb74e1199b1f5f90c613e152cfa7. Kernel test robot reports -88.8% regression in stress-ng.ramfs.ops_per_sec test case [1], which is caused by commit f95bdb700bc6 ("mm: vmscan: make global slab shrink lockless"). The root cause is that SRCU has to be careful to not frequently check for SRCU read-side critical section exits. Therefore, even if no one is currently in the SRCU read-side critical section, synchronize_srcu() cannot return quickly. That's why unregister_shrinker() has become slower. After discussion, we will try to use the refcount+RCU method [2] proposed by Dave Chinner to continue to re-implement the lockless slab shrink. So revert the shrinker_srcu related changes first. [1]. https://lore.kernel.org/lkml/202305230837.db2c233f-yujie.liu@intel.com/ [2]. https://lore.kernel.org/lkml/ZIJhou1d55d4H1s0@dread.disaster.area/ Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202305230837.db2c233f-yujie.liu@intel.com Signed-off-by: Qi Zheng --- mm/vmscan.c | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index a008d7f2d0fc..5bf98d0a22c9 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -57,7 +57,6 @@ #include #include #include -#include #include #include @@ -191,7 +190,6 @@ int vm_swappiness = 60; LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); -DEFINE_SRCU(shrinker_srcu); #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -742,7 +740,7 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { down_write(&shrinker_rwsem); - list_add_tail_rcu(&shrinker->list, &shrinker_list); + list_add_tail(&shrinker->list, &shrinker_list); shrinker->flags |= SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); up_write(&shrinker_rwsem); @@ -797,15 +795,13 @@ void unregister_shrinker(struct shrinker *shrinker) return; down_write(&shrinker_rwsem); - list_del_rcu(&shrinker->list); + list_del(&shrinker->list); shrinker->flags &= ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry = shrinker_debugfs_detach(shrinker, &debugfs_id); up_write(&shrinker_rwsem); - synchronize_srcu(&shrinker_srcu); - shrinker_debugfs_remove(debugfs_entry, debugfs_id); kfree(shrinker->nr_deferred); @@ -825,7 +821,6 @@ void synchronize_shrinkers(void) { down_write(&shrinker_rwsem); up_write(&shrinker_rwsem); - synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); @@ -1036,7 +1031,6 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, { unsigned long ret, freed = 0; struct shrinker *shrinker; - int srcu_idx; /* * The root memcg might be allocated even though memcg is disabled @@ -1048,10 +1042,10 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (!mem_cgroup_disabled() && !mem_cgroup_is_root(memcg)) return shrink_slab_memcg(gfp_mask, nid, memcg, priority); - srcu_idx = srcu_read_lock(&shrinker_srcu); + if (!down_read_trylock(&shrinker_rwsem)) + goto out; - list_for_each_entry_srcu(shrinker, &shrinker_list, list, - srcu_read_lock_held(&shrinker_srcu)) { + list_for_each_entry(shrinker, &shrinker_list, list) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -1062,9 +1056,19 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; freed += ret; + /* + * Bail out if someone want to register a new shrinker to + * prevent the registration from being stalled for long periods + * by parallel ongoing shrinking. + */ + if (rwsem_is_contended(&shrinker_rwsem)) { + freed = freed ? : 1; + break; + } } - srcu_read_unlock(&shrinker_srcu, srcu_idx); + up_read(&shrinker_rwsem); +out: cond_resched(); return freed; }