From patchwork Mon Mar 13 11:28:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13172281 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19E84C6FD19 for ; Mon, 13 Mar 2023 11:29:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6DBE96B0072; Mon, 13 Mar 2023 07:29:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 663EC6B0074; Mon, 13 Mar 2023 07:29:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4DD856B0075; Mon, 13 Mar 2023 07:29:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3F1AB6B0072 for ; Mon, 13 Mar 2023 07:29:46 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id DFCE1C0427 for ; Mon, 13 Mar 2023 11:29:45 +0000 (UTC) X-FDA: 80563655130.16.D97A580 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf13.hostedemail.com (Postfix) with ESMTP id 0FB622001C for ; Mon, 13 Mar 2023 11:29:43 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=cMAG05TH; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678706984; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cHjpkGxbHVFFSyOk4f8KxselJo2cIaqjL6okRMIx/Nw=; b=MET7vGG3FzDKtshy/srczu7gNh+3jd8AAu6Pl8T1fk3Nucx/ToFg52QghruF++cxIkdQPi OvClvHe9ijoMKQTmcvNRmCQ9Gr9NDytle2rA4YwS/UknWGmQ3OwFk5ih3t/BrE/zkmUcYw BWOPmkO+T2NOWlOvxixvwZ9eOz2azks= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=cMAG05TH; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678706984; a=rsa-sha256; cv=none; b=pfbvdsp6B+ldllVsP+agXAQlf1CrKh7Jj4lc1Xm3zDPmbwQulrMtU21+ozQSwD8rHA6Pmf NoTaRMp/fjxsxS0c9Ul5upHctTIsNoeWxVZ65v213ybyCAvumWfXW3VH3yft3BngPUjRuW HzSrK1D9JMAugssEcNsWxsxh5xGIzVo= Received: by mail-pf1-f181.google.com with SMTP id z11so7388922pfh.4 for ; Mon, 13 Mar 2023 04:29:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678706983; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cHjpkGxbHVFFSyOk4f8KxselJo2cIaqjL6okRMIx/Nw=; b=cMAG05THHEL45EY/ZCrlGRIbq1kt4I6LPUMAH/dQjouL+NMY60brEhRYxofz4Sc3yO jVllerX4euE/VSk8gxVy2qydl3B1I0ZvizXWnB/k4AlXoCTSsvfCFO9lUUclwXD/c7zx +Lstf6Kcx2x9pLpDuB/8MwyscxtFe5rDeDGAP/r3rkvK8Nru8nZQojqHvbsNe7a2Ab7/ gUPJ2C1u8cgdUyoJ+NnEXAlZ1izrfSpHumWDpFe9DMlhF0rV7W8iS0Gr8SVDtkxfdNr/ N3obCyhq6rnOWhxjxKbMgLykNy2bkGDKgby8e4YMGNLWFieQ1ksal3K4NhZ3AUVdAOK+ UsUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678706983; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cHjpkGxbHVFFSyOk4f8KxselJo2cIaqjL6okRMIx/Nw=; b=B7APolSBC2gob31OPk99du0dGYEKkaxXIBdpNaY45l/3G01yhsCGjoyMfFb2Pm1HGv 8p44ShRYX1yJfASxibQ967u2dPew62I0DQERw5Y8EkJU7icU7ZyYzym+IO9OyzYQbtCe 7fO5BzvqoPCahlHNYg1hHOj4IxeltqahMmf0vGfJiijh9qLkf5XOFeFWY7GRH5QAbU4A iQmi6VuwuXUVuoEP2vscmzP4KNmsx06Uti+lySDu4VHrSB8xKgcjTBRl0ugkAiPo6L8Z xzdwiz79Ugrh5QqjxGmL2UHDogE8D1HWAAzRaTzWkLwgQ2P5Mv4FL+t95yMZKDCRTF3I WRvA== X-Gm-Message-State: AO0yUKW4+3xa1IjPfPBZWhzbkKz14nltVUciMpFbFClZSk7kXsrckwCS 922lix/Rzn0PsaoRvd3jaAWxDg== X-Google-Smtp-Source: AK7set/7+CC2e3gj8ocPsHr26haOK3QREhbFYzVHIRff/+7vh0RhPeq9x+b/EZ+X+KieTYLWMp1SeQ== X-Received: by 2002:a05:6a00:2148:b0:606:d3d1:4cc4 with SMTP id o8-20020a056a00214800b00606d3d14cc4mr10842399pfk.3.1678706982825; Mon, 13 Mar 2023 04:29:42 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id n2-20020a654882000000b0050300a7c8c2sm4390827pgs.89.2023.03.13.04.29.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 04:29:42 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, vbabka@suse.cz, christian.koenig@amd.com, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v5 1/8] mm: vmscan: add a map_nr_max field to shrinker_info Date: Mon, 13 Mar 2023 19:28:12 +0800 Message-Id: <20230313112819.38938-2-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230313112819.38938-1-zhengqi.arch@bytedance.com> References: <20230313112819.38938-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: g6k4dur6qfk37ysamyrnt8donnsj3u3i X-Rspamd-Queue-Id: 0FB622001C X-HE-Tag: 1678706983-214151 X-HE-Meta: U2FsdGVkX1/JEzp+Xm/FdwfJi4XVYO5k1vEnxVFt7XGPLyt2ekizSaoypHjJXJ7xl5B2ohvM2ylV03lJUZDJfgSBc/iZWrE9BhBSiGmO/xlFoLZCpenFu7Cb4F09A9KtMWaip/gEyR8EpxwGhcINaj5PthDd7h81J6bkRUy/8BovLpZrEYETOAUlW907UVpt5NfdpJCJUXLmrg5tj66T5AI5CBkv3ypXJK5hwVYTX8KMxzLZMsmtXGlj0nbS+VEGuNOzZrVEsf7qxz3pBWgt+Hc2WyUn3v3zrE82NsDAAfPlcvKqllm4vI6YQCbYwAYHD2/MbbSwppi4n12xSsd4Bh7nzI2Fs6PkH6gnWt0DK31mNVAuE72DrHQt9viyWDrAoCtE+RcftQ9RvVXuvKu/JFuhJrBiNhAGugR+hFYfBAPOZWQq22cHYtfPkUzVV6HRf6DLzd1tQdrPtLcdQYCG23houXcMI9JBowvViffGEHmaiokZzKnyMXw15VzBxK4jaGEl0A5D1wVs9tWhcU7tUoT6f4LCw9Bhn5NGz1xXOOTMYZOPGe+uchEKlNyoXRgspl8rD0+eseSAFouEILdGWlEb2HEHylb7TOBoLHjWcu/NxsJSZSSP9YCyRSlcNnPE6Kf52OP9z4LwxrhuhD6cCkH/+NI0y2asCNLwkuAU3T6s+MsmpPjgijXklqd8kAy2FbbjC02pJM2vLSnP+FO9cubUim9MiddK3vQR9eTOYHZfxD4arG0GHP/Kzpz4dZIG4jOpFcoe+lfIIDoPLAzJi6yvdpahT+7W7A8EEg/tEeq7id+mH7YTAwurpFgfaFveE9YZSVLfuGKgOXj7UKrVPU3+QAbNc+0rF1AoxmvQJzUaBVMlkFX3qlv3FXRQCAdwhmi6ffjJfF/qKH18hzEzFhdmRvSA2dQLJsupw3WNrxXPyuckBPRwevmCUwZ89r6U5MLVvT52wUvCIW6NFLy Pq1y8axd dwWhMjz43CeL92hOd+BsIVn9/AbPX6sPYe0+lf0w9xqrQRkq+eSkLofci/nr4kMSJvkGobgCo/BBzMPjKIInTT6fllARC8zf8jYaQ9VjE4QKP8uypoev1IxrsmUjrpZFA97mTyuOcErlw8ADg+xK5sV7LX06l/IsEM91ub3uUNCFxFmCz7fuzDnKFJQUQACj9wQ/f6qWbkUP9RwiXUWVfCvO38+Kuf4cer2Df+8VTRRjKu9G9ftCGSSC91GW3Qi3+VG4qUBAI73nxcse3lTEGcWS5DE5kCiG0+VSIJanWMXCY+aw8JXHlO5pH4ln7uuwqmaOt7kwgmBjzU6Ddd0JQ3wUhoDibplWHwdFGJe5x26mraBTWU8Iu5KOU/ojFvOAPRLMXSu0I85gGknfGkQqiRSeXPBY+e/R0DzRus3m7KoWiKziQhyWEB83JW2eKxmW8CniGWEgY1c53LEUSxGTvy5Dd8bgs+PWU2SIBH6moS0O79lmn2xhb5iaulWc4OngRvHfath5yv2hciS4bg2UUNzjPFg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To prepare for the subsequent lockless memcg slab shrink, add a map_nr_max field to struct shrinker_info to records its own real shrinker_nr_max. Signed-off-by: Qi Zheng Suggested-by: Kirill Tkhai Acked-by: Vlastimil Babka Acked-by: Roman Gushchin --- include/linux/memcontrol.h | 1 + mm/vmscan.c | 35 ++++++++++++++++++----------------- 2 files changed, 19 insertions(+), 17 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6eda2ab205d..aa69ea98e2d8 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -97,6 +97,7 @@ struct shrinker_info { struct rcu_head rcu; atomic_long_t *nr_deferred; unsigned long *map; + int map_nr_max; }; struct lruvec_stats_percpu { diff --git a/mm/vmscan.c b/mm/vmscan.c index 9414226218f0..9a2a6301052c 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -226,7 +226,8 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, static int expand_one_shrinker_info(struct mem_cgroup *memcg, int map_size, int defer_size, - int old_map_size, int old_defer_size) + int old_map_size, int old_defer_size, + int new_nr_max) { struct shrinker_info *new, *old; struct mem_cgroup_per_node *pn; @@ -240,12 +241,17 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, if (!old) return 0; + /* Already expanded this shrinker_info */ + if (new_nr_max <= old->map_nr_max) + continue; + new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); if (!new) return -ENOMEM; new->nr_deferred = (atomic_long_t *)(new + 1); new->map = (void *)new->nr_deferred + defer_size; + new->map_nr_max = new_nr_max; /* map: set all old bits, clear all new bits */ memset(new->map, (int)0xff, old_map_size); @@ -295,6 +301,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) } info->nr_deferred = (atomic_long_t *)(info + 1); info->map = (void *)info->nr_deferred + defer_size; + info->map_nr_max = shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); @@ -302,23 +309,14 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) return ret; } -static inline bool need_expand(int nr_max) -{ - return round_up(nr_max, BITS_PER_LONG) > - round_up(shrinker_nr_max, BITS_PER_LONG); -} - static int expand_shrinker_info(int new_id) { int ret = 0; - int new_nr_max = new_id + 1; + int new_nr_max = round_up(new_id + 1, BITS_PER_LONG); int map_size, defer_size = 0; int old_map_size, old_defer_size = 0; struct mem_cgroup *memcg; - if (!need_expand(new_nr_max)) - goto out; - if (!root_mem_cgroup) goto out; @@ -332,7 +330,8 @@ static int expand_shrinker_info(int new_id) memcg = mem_cgroup_iter(NULL, NULL, NULL); do { ret = expand_one_shrinker_info(memcg, map_size, defer_size, - old_map_size, old_defer_size); + old_map_size, old_defer_size, + new_nr_max); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; @@ -352,9 +351,11 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) rcu_read_lock(); info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); - /* Pairs with smp mb in shrink_slab() */ - smp_mb__before_atomic(); - set_bit(shrinker_id, info->map); + if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) { + /* Pairs with smp mb in shrink_slab() */ + smp_mb__before_atomic(); + set_bit(shrinker_id, info->map); + } rcu_read_unlock(); } } @@ -432,7 +433,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); - for (i = 0; i < shrinker_nr_max; i++) { + for (i = 0; i < child_info->map_nr_max; i++) { nr = atomic_long_read(&child_info->nr_deferred[i]); atomic_long_add(nr, &parent_info->nr_deferred[i]); } @@ -899,7 +900,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (unlikely(!info)) goto unlock; - for_each_set_bit(i, info->map, shrinker_nr_max) { + for_each_set_bit(i, info->map, info->map_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, From patchwork Mon Mar 13 11:28:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13172282 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D8E5C6FD19 for ; Mon, 13 Mar 2023 11:29:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CCD8D6B0074; Mon, 13 Mar 2023 07:29:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C56C16B0075; Mon, 13 Mar 2023 07:29:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD08D6B0078; Mon, 13 Mar 2023 07:29:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 9C5706B0074 for ; Mon, 13 Mar 2023 07:29:53 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 772CD160433 for ; Mon, 13 Mar 2023 11:29:53 +0000 (UTC) X-FDA: 80563655466.05.3F08148 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf19.hostedemail.com (Postfix) with ESMTP id A06B31A0014 for ; Mon, 13 Mar 2023 11:29:51 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=LKdBuqoP; spf=pass (imf19.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678706991; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=o+PIjzMrqqus+vCKR4kW46vOjsCmmZLpdvQvuCO0Xe0=; b=MozoCzXEZnyB7T1e1cBzMhI2ARjeKA3pAbCh2jr8DeM5VzA8g9MUGSgSsHocfW1HU2pw65 WQiBOKvc1uWckcfyVBRQF0tInqiTRFTYg8j4v6w+fRUnf2w0RnA2+rfqMTcaCbZXb/sM8y b2e1rXVhvsAltin+QqOcrwCf6A8+LrY= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=LKdBuqoP; spf=pass (imf19.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678706991; a=rsa-sha256; cv=none; b=xU9/vv1K73fR9Zt1sY7mdkyBlRIMq6WcvnDbh4VUZaSPmynW7dP+2u7HrZGoZyYsrIRD5D ckwVHIYuP5PR+kRCeydlp754F7+qm8IuX2EPYXYn7W27XtPaZbJmE/gSy8CDGdH4t4WsbN h7NtJ781pNFx9OhjdLccQbzb7bNDYl8= Received: by mail-pl1-f179.google.com with SMTP id p6so12609085plf.0 for ; Mon, 13 Mar 2023 04:29:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678706990; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=o+PIjzMrqqus+vCKR4kW46vOjsCmmZLpdvQvuCO0Xe0=; b=LKdBuqoPZ3Eoe3IWRve5czAmJNAuSXHJEZyt6qMgQ2QWE0FfdegRHxqdr14XQLU0rx PsAdOS3EITkjvH5yWgAVHjIGJIgtokVqDvCFC26F4koRE2Dg75XvyXjyLjnvaxEhaWs0 dhQ9CFD461CllT/ivbb0KEKQLcI9xDxnAgIIgqPeywEBe7ZlcK3Kpq2x4YNZJlXiWmJW wFSsnsQNtCw4pUPGf8nkmUQNQWO9giNg3PiXQrgZK9dUZR0DabpxIf6V5RpwknTkHH+s 11as/+u7W1eLQqbscYrAfCE12YP+G4kZxz1LGruS5fzUaeaeMeP5o6+X9fPn1O+TsKdH kbvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678706990; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=o+PIjzMrqqus+vCKR4kW46vOjsCmmZLpdvQvuCO0Xe0=; b=z6ndaRwvYThYuHaEQw1pSEh9T3T6zRICTg7SGB0u9blBxQm2JwDbNVSKM9IfobNcPM Tp8nyULjwtGznjFThMspF3kfD3pSVjrnngjO3HxzJzc75YE0gh6PaXoBrrtNFHSiIuyN uRiKZdL3wTWTPRVQapS3tbYbDZsvgt9g3oYzFCGOj6dNBNk/x4vqZpnG4i6y7A8KAYHe VJ8bdPhvkNgVpfxXi7z2Niw4jHt9SyEz+BzvHC/iUXIYfoA7ApsYLAv2l7nZyFGGiUXk bnf9JXNE5+0vxkuI7zyB+vpdf7rIbwzK4SUemJR5nW4C/vojufmnL0jjF56LM66noBVM sUjw== X-Gm-Message-State: AO0yUKW6GzWlnpS5dGJjmZnuoEnkWfMNBSSYKlIYlW3HrNuwiBqfwJ3r a0CQPwniZ4/hKgAeQ4l+4VFEyw== X-Google-Smtp-Source: AK7set9cSLNe/LRJSehPejbnqQcEtWGFrHHPC3Q/DqoKx4Ii0aTTtc55w7QWCxr0TLSi4GrfoJETMQ== X-Received: by 2002:a05:6a20:841c:b0:c7:af88:4199 with SMTP id c28-20020a056a20841c00b000c7af884199mr17859458pzd.6.1678706990517; Mon, 13 Mar 2023 04:29:50 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id n2-20020a654882000000b0050300a7c8c2sm4390827pgs.89.2023.03.13.04.29.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 04:29:50 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, vbabka@suse.cz, christian.koenig@amd.com, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v5 2/8] mm: vmscan: make global slab shrink lockless Date: Mon, 13 Mar 2023 19:28:13 +0800 Message-Id: <20230313112819.38938-3-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230313112819.38938-1-zhengqi.arch@bytedance.com> References: <20230313112819.38938-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: A06B31A0014 X-Stat-Signature: 85ez69oa86izef9dzoyw3o9tdijfwu9j X-Rspam-User: X-HE-Tag: 1678706991-565083 X-HE-Meta: U2FsdGVkX19QWQXfj6noh/QiePAcWrFGCwINBGhtL8h86cqPtGjRlwb6QgpPF5pDGrgdZo8m/D/HJIgh/5/OZATAK7LQLWdcH0aF6SdzsSw6Ojaf/IZaleXhsidE+Y/FeSkcFQQXrV3XBGRlk2RVVdWigEdSC+2nVnOK4KuDlv1pfzUMR5EF+6UKr1BEHGBZcMVwmlXI56CUb7KSjUke+5Zy3eaRNCq0xf7Lo8rYU10jmcfWbiIE2LXnqkG5hOnfnixPiX8Lq/6Qy66/ZKoViy8UgKfbKX+oXVMUZGI+wuFyRGs9kYFKWADhyLl2NjrERLOyZ346EkdPznx8NIrUU0WdtQ1N4GpVFSK2ry3RB1F67wzp/v5pRAERPDiSuRbrR9gIu2SLWBz69gSDEN7EkSB4rZiZc4QXzx15cPMvWlqCgGzeCcpcAYxZhC62jFwprnbqAGdDxb1w5ALeZOTue84TiCBwtO7pJePLXeSPJQtm/uCCs1xEhTKp0oj4BmVtdR2UNmbGtrmTa+1WgwCY0fBOh8Hc/oGP5v/HbnXHblOva3W/uf3JFkvQiPnwj3j3O9mD3tv56/BfnQLZNsSe14DZ9hwusHtfIf2ogeSf0gac+bLjUm01KrAwpTi7RzJfZqr9e7KPZEUocn1cU90ZCfPezJlPsgYtwbWjUjCzucjdRu4ouDGcSWjIXNg5S7Jn7k9pbVfPG3sK0JCoiQnldgmnyUJZbcOkO+3HDfyo8EiZAMDLbaF4XVxDJ+f2cl3VfB8Vjb8nioQGPQ5bo35PzM/xosZSNQwailEGsWUqs0sWxdyRTEAywVNzp2J5v9QW9/7aYNONApmp0mqAc1Xv7oR2vkzd46+rfoEyXP0tQSOZ+xffQPnVvL4stdeOBNFC3Uaq+kGlJlY7By7UPmt1QmTjHl7mfAEjG4kwYrA9vO9ql+htYa44khOQ5WFFKIdrdfm+COYD1l8XPyLRgFG UfLESami e/4AAqs4eKWAwGnl/5bj1mp6mPFpQd6E1OqV/84jVztuQRM0KkUEx84j7UrNSRdCac7YffOEBI8kFjqG6xaVIVg0ntycZqAu4acXmyk+fRMVDYOjES9QQhC5n3OsgegL2EOBI8jCfOQxBmjRkWxCZX+Dof/DWM1eg7fWDU2V6ADN3aG8dv2TvMY/52vP2/S+/LeTykll+h+WxpY8nIfYjxBoG55Yrs9NyE25lCiRWc+CUicFOU1VNZ/l2pYNOsN6pphg7gpeRn9aV+a6DILeGHlXtrjMyI/NyOG9tKJxrFFuLwQnWTvoWnLMgZjlJJbAEoU+Hl5Fr3tG//gvwRBd04o6UP9GXjIYaJAZ53N01rGFIi1eEKJ52NTIcdRi4Sosj/Cae4ROjTn5boJeMJpNUkh1pEaWfoyHXajASs3NEZ9Lf4xYxWsVdvhf9b7wHsEE5eKFq7t3xpDAttumjRYE85yVD8GoI3zy3HazvejjnFN2+jl46vHSwrXt0zBaPdBGXjkrfyNVjAgKYvk/qWWkw4KDH40BwaFgPxgt1E2XRKBh4tCiUH5esIQZTRCdRBwiQRT8Y1AogsjoS70TvzVGuJADameMDUkWipXrrfgfxoHEV9DNhklRDzccTOKoJqgaeszFDINitC/O5G1cg/IrpHpVxW/TlJD2rWQ6pF7UhyQMVIn9mUbAkCe/H6OzjNM9HkX6J1rwualOLdP8ELuRUGPL+8g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The shrinker_rwsem is a global read-write lock in shrinkers subsystem, which protects most operations such as slab shrink, registration and unregistration of shrinkers, etc. This can easily cause problems in the following cases. 1) When the memory pressure is high and there are many filesystems mounted or unmounted at the same time, slab shrink will be affected (down_read_trylock() failed). Such as the real workload mentioned by Kirill Tkhai: ``` One of the real workloads from my experience is start of an overcommitted node containing many starting containers after node crash (or many resuming containers after reboot for kernel update). In these cases memory pressure is huge, and the node goes round in long reclaim. ``` 2) If a shrinker is blocked (such as the case mentioned in [1]) and a writer comes in (such as mount a fs), then this writer will be blocked and cause all subsequent shrinker-related operations to be blocked. Even if there is no competitor when shrinking slab, there may still be a problem. If we have a long shrinker list and we do not reclaim enough memory with each shrinker, then the down_read_trylock() may be called with high frequency. Because of the poor multicore scalability of atomic operations, this can lead to a significant drop in IPC (instructions per cycle). So many times in history ([2],[3],[4],[5]), some people wanted to replace shrinker_rwsem trylock with SRCU in the slab shrink, but all these patches were abandoned because SRCU was not unconditionally enabled. But now, since commit 1cd0bd06093c ("rcu: Remove CONFIG_SRCU"), the SRCU is unconditionally enabled. So it's time to use SRCU to protect readers who previously held shrinker_rwsem. This commit uses SRCU to make global slab shrink lockless, the memcg slab shrink is handled in the subsequent patch. [1]. https://lore.kernel.org/lkml/20191129214541.3110-1-ptikhomirov@virtuozzo.com/ [2]. https://lore.kernel.org/all/1437080113.3596.2.camel@stgolabs.net/ [3]. https://lore.kernel.org/lkml/1510609063-3327-1-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp/ [4]. https://lore.kernel.org/lkml/153365347929.19074.12509495712735843805.stgit@localhost.localdomain/ [5]. https://lore.kernel.org/lkml/20210927074823.5825-1-sultan@kerneltoast.com/ Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai --- mm/vmscan.c | 28 ++++++++++++---------------- 1 file changed, 12 insertions(+), 16 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 9a2a6301052c..db2ed6e08f67 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include @@ -202,6 +203,7 @@ static void set_task_reclaim_state(struct task_struct *task, LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); +DEFINE_SRCU(shrinker_srcu); #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -700,7 +702,7 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { down_write(&shrinker_rwsem); - list_add_tail(&shrinker->list, &shrinker_list); + list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |= SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); up_write(&shrinker_rwsem); @@ -754,13 +756,15 @@ void unregister_shrinker(struct shrinker *shrinker) return; down_write(&shrinker_rwsem); - list_del(&shrinker->list); + list_del_rcu(&shrinker->list); shrinker->flags &= ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry = shrinker_debugfs_remove(shrinker); up_write(&shrinker_rwsem); + synchronize_srcu(&shrinker_srcu); + debugfs_remove_recursive(debugfs_entry); kfree(shrinker->nr_deferred); @@ -780,6 +784,7 @@ void synchronize_shrinkers(void) { down_write(&shrinker_rwsem); up_write(&shrinker_rwsem); + synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); @@ -990,6 +995,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, { unsigned long ret, freed = 0; struct shrinker *shrinker; + int srcu_idx; /* * The root memcg might be allocated even though memcg is disabled @@ -1001,10 +1007,10 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (!mem_cgroup_disabled() && !mem_cgroup_is_root(memcg)) return shrink_slab_memcg(gfp_mask, nid, memcg, priority); - if (!down_read_trylock(&shrinker_rwsem)) - goto out; + srcu_idx = srcu_read_lock(&shrinker_srcu); - list_for_each_entry(shrinker, &shrinker_list, list) { + list_for_each_entry_srcu(shrinker, &shrinker_list, list, + srcu_read_lock_held(&shrinker_srcu)) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -1015,19 +1021,9 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; freed += ret; - /* - * Bail out if someone want to register a new shrinker to - * prevent the registration from being stalled for long periods - * by parallel ongoing shrinking. - */ - if (rwsem_is_contended(&shrinker_rwsem)) { - freed = freed ? : 1; - break; - } } - up_read(&shrinker_rwsem); -out: + srcu_read_unlock(&shrinker_srcu, srcu_idx); cond_resched(); return freed; } From patchwork Mon Mar 13 11:28:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13172283 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7510EC6FD19 for ; Mon, 13 Mar 2023 11:30:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 134DF6B0075; Mon, 13 Mar 2023 07:30:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0BE476B0078; Mon, 13 Mar 2023 07:30:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E53176B007B; Mon, 13 Mar 2023 07:29:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D2C076B0075 for ; Mon, 13 Mar 2023 07:29:59 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A4FCCC0162 for ; Mon, 13 Mar 2023 11:29:59 +0000 (UTC) X-FDA: 80563655718.14.E7A1FBB Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by imf16.hostedemail.com (Postfix) with ESMTP id C4E9D180005 for ; Mon, 13 Mar 2023 11:29:57 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=VL291Bsm; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf16.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678706997; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CwHKJa6QSzn+fFUbr3GdLYPgv95cnerwTX8zoVbLwNQ=; b=shkSTsTE9pi4dIqsn+4zj/MxNrr4W/naT04SjsYVIDkEur/4SruLiXVF9raqnMbBl4Drdu p+wJZ9q6eyGRijSwLGiUY4XgEbw5xzGDDHLqML3jWuidBH8p+Gr8pid3VIqxBWvFj/dWZk nW18r6jNcuz8kSB/GQi8EjZXAga4OZ0= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=VL291Bsm; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf16.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678706997; a=rsa-sha256; cv=none; b=X77UZyyFhClcU8EkqiG0pR20crV05Czhkwvn8kXgnf54WUbXyVRme5koklTECmugdaVcIF L9U92b1wjwF8XLfx5UZu3zoLC0RF+323db8gImSYzimY4QCYKH8R9K5iyAlhwkf3Hv6n5D MH7Li4aAYuzTsuJYfmClAg4OSDWFL38= Received: by mail-pj1-f42.google.com with SMTP id fy10-20020a17090b020a00b0023b4bcf0727so3861475pjb.0 for ; Mon, 13 Mar 2023 04:29:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678706997; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CwHKJa6QSzn+fFUbr3GdLYPgv95cnerwTX8zoVbLwNQ=; b=VL291BsmdjOKqQ02b55ZkiNe1Gxiw0B9HIG72ICowrQQIz60/2k76pkbV6ohUH0jkm Jox9Y09fVi77XloTu403ELBMBr76X8dzxeQ3haW2GgqK0Esu9ZkoXpWzHaSH6tAaMoDV tmDAGCPgOSqsKKkamJFDdXSO6nIaXZBF40p+RTjEGc53dohjfIGG8/C+G7GvyIESgath WeikExSTqhrXJdcJWm/xMTEGyKADTPQDEs+Ees4T5pfkypk5jFJZpd/7M1ecIk1aLgTn YzvxNO/avEaPXMEeDvs/G5Ff4veN37dFXuRX65vfKGciMuehn4VVeyzAB73LmFAqM9LG sxIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678706997; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CwHKJa6QSzn+fFUbr3GdLYPgv95cnerwTX8zoVbLwNQ=; b=0r+UOTMVaWuxqOfCsWlCFS3M51/fh4hlrquZ3tla7WA3aHMxVMffYT9mUwJXWgOStd 8nHAVoQ0UNyMsuJEskKAgrpn1jdGj0UaBiLLyv+uefm+t3ZlfvuvERGu9SZ4fBx6UVIP mWPZsMAlKmJDrnyRJVxGY0HBODZOryN88w2jNYzd1YRzPld1how3TKCczD3kq3EhLjMm NzLY2snOLOpXqHSQVnmvDq3mf8i0YcNRXFjPdsKo/n/jNRA579J5P1MWM1tLB6oGCo3Y sUzt1TrvM1dqijhORvnspgDJY+AqgzQ8s9H7UtQ4i0VIce8Ww/GgHtz7Rl7R2u1ik7DE 2/wg== X-Gm-Message-State: AO0yUKWbWrPHughHFFmRoXGInPY9YAHiz4FyaskUW1NA5sA4/PzrTrDi x8oEVhXhe0h8zJGMatCm2u98Pg== X-Google-Smtp-Source: AK7set9JyLNsrTQk+ldmpaXssouDn0mkoIHdAxrcwhQZfqJh+G9zdIKa6tn/z3NoTm4Hps375eI0IQ== X-Received: by 2002:a17:90a:990e:b0:233:a836:15f4 with SMTP id b14-20020a17090a990e00b00233a83615f4mr12917519pjp.1.1678706996712; Mon, 13 Mar 2023 04:29:56 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id n2-20020a654882000000b0050300a7c8c2sm4390827pgs.89.2023.03.13.04.29.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 04:29:56 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, vbabka@suse.cz, christian.koenig@amd.com, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , Vlastimil Babka Subject: [PATCH v5 3/8] mm: vmscan: make memcg slab shrink lockless Date: Mon, 13 Mar 2023 19:28:14 +0800 Message-Id: <20230313112819.38938-4-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230313112819.38938-1-zhengqi.arch@bytedance.com> References: <20230313112819.38938-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: C4E9D180005 X-Stat-Signature: us4d86kj84qu8zy6pirg66eq5nwq1ijs X-HE-Tag: 1678706997-11673 X-HE-Meta: U2FsdGVkX1/WzSkV6gPX+tLONRhApLIvT+8dUP8AA4cuXpi/YKWZUgjr6p5J+xRR+3aMiPikWrCefEGlhyE0w1QMWqmv+RPEfwRVjAzfCwA2IbSPSQKRa7JSSIkLsJHx6WxSKt67yDvTggfExP4Js11Gn+/WiYNWDwydCF9WQh7Yq8RLX8WbW6i6jMIbkHtzMr6lcqn9SnfAB0hDLyRMoA+G04edg632w4LoEKGGW4XCwa6XL0He7JRDWA92GdDT2bjgmHmFxw6JvrpVR36eEvqUd1c9GCtTZgn1ZZfD4YCfmPyTj2/EZmnKtrdyUfp05elGpwyFNw5f7b9zSF2HD65ARuPta/QlluEZYtJgAYMKNBbvAW79v5skPmMAiS7Yt1/xrDeUV+2DGkbL7zjwoM1r7e8vBzwQKI42eQjDY6e95lAIY3ui4zymm/B019bxNhln5e4ScInzVGbo+nugz7mHqZGEQByE8ZO48Hx5BhZVmQGeiPgYAPnQBfr8lKhLlHQRW+A/3xo+jNHeOo877PXE5eJjrZqLcO5Hbp+Z8XfmBieUyUMtT2Eqgj287GpjyY8Z1pDsxh72K8mYcnhKsqjmqCW0wLKO3d1ApRn3ivEsnBWw/dY2km9jPksNIhSY5nwgkQqhd8r62/zQHQnY9vQQoySLSPaxCRujvqUdjyPntNoIkFQk0w+PHhFd6bGeAZXFGvh2EjISzD+/h3wx2V07zuLU49zwVM3Ij0kxWJoiddP2pio57BASXJKkKldDQJ8Q8U6xpPhhkrM/a9HW2lMDIi+xLkHQE7o/CE8LWIzuYQdM1lMY102qEwQMcMXMVwUq0fPGoMyVQz+DFchVBVpy6IWAnOvL6hIaFfRXmXuMui5x2bvWSVpSUk2MGtbdWy+nl6Wo9d2GqU+CDe13xirP4/lRgj8a4v9CspXILhFaNJvTeJ0oPoj3e/n2KjlHlfNSbTNW7NCvkn2CW6X 1mcbrrut SErNFs7jt7uwqnVv0idMonxA6JUee1A9PnrGTWr6OXYgvE8eQp0H13GLjy3B14vGH7dh+gu+l5UZO51SNYsN06m39s2rdefoxN9a5NpuvpHvzhw3gcwHd3p9JW5zJUFb7ZXDDUkG2NbtRHGM82oBJeZo4tZKMm5RMAuKeNeDjJXP2oE4GVnlslnxOlLfOs9iDSscBFJvURuHJAEPUfSAwMQlVwK1VSDUfVrpkgdKrA3wVeSKHgTS0rscx3ocOduCf5bnYS1HOFl70c9ud7I/yOVOsm8Rls3Yi+DxlHpsnGDjrswMaOegLFNLnotXPK+3DHU2mjF1RHzNrrKVT83EoF/DObVWgSIWGUIk0P26cJc/+QlzaaH/Ne/gVLu5yWVu0bdbBqLc04OhEaseOZtc09MWPO2owonijsUL7tRaW1z2M0th3SeXymHxG3god06LWXxo/49JeXC8HeEHczFYXPpxqKIUhXYy3YWpwoxK5hH4F0Ys= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Like global slab shrink, this commit also uses SRCU to make memcg slab shrink lockless. We can reproduce the down_read_trylock() hotspot through the following script: ``` DIR="/root/shrinker/memcg/mnt" do_create() { mkdir -p /sys/fs/cgroup/memory/test mkdir -p /sys/fs/cgroup/perf_event/test echo 4G > /sys/fs/cgroup/memory/test/memory.limit_in_bytes for i in `seq 0 $1`; do mkdir -p /sys/fs/cgroup/memory/test/$i; echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; echo $$ > /sys/fs/cgroup/perf_event/test/cgroup.procs; mkdir -p $DIR/$i; done } do_mount() { for i in `seq $1 $2`; do mount -t tmpfs $i $DIR/$i; done } do_touch() { for i in `seq $1 $2`; do echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; echo $$ > /sys/fs/cgroup/perf_event/test/cgroup.procs; dd if=/dev/zero of=$DIR/$i/file$i bs=1M count=1 & done } case "$1" in touch) do_touch $2 $3 ;; test) do_create 4000 do_mount 0 4000 do_touch 0 3000 ;; *) exit 1 ;; esac ``` Save the above script, then run test and touch commands. Then we can use the following perf command to view hotspots: perf top -U -F 999 1) Before applying this patchset: 32.31% [kernel] [k] down_read_trylock 19.40% [kernel] [k] pv_native_safe_halt 16.24% [kernel] [k] up_read 15.70% [kernel] [k] shrink_slab 4.69% [kernel] [k] _find_next_bit 2.62% [kernel] [k] shrink_node 1.78% [kernel] [k] shrink_lruvec 0.76% [kernel] [k] do_shrink_slab 2) After applying this patchset: 27.83% [kernel] [k] _find_next_bit 16.97% [kernel] [k] shrink_slab 15.82% [kernel] [k] pv_native_safe_halt 9.58% [kernel] [k] shrink_node 8.31% [kernel] [k] shrink_lruvec 5.64% [kernel] [k] do_shrink_slab 3.88% [kernel] [k] mem_cgroup_iter At the same time, we use the following perf command to capture IPC information: perf stat -e cycles,instructions -G test -a --repeat 5 -- sleep 10 1) Before applying this patchset: Performance counter stats for 'system wide' (5 runs): 454187219766 cycles test ( +- 1.84% ) 78896433101 instructions test # 0.17 insn per cycle ( +- 0.44% ) 10.0020430 +- 0.0000366 seconds time elapsed ( +- 0.00% ) 2) After applying this patchset: Performance counter stats for 'system wide' (5 runs): 841954709443 cycles test ( +- 15.80% ) (98.69%) 527258677936 instructions test # 0.63 insn per cycle ( +- 15.11% ) (98.68%) 10.01064 +- 0.00831 seconds time elapsed ( +- 0.08% ) We can see that IPC drops very seriously when calling down_read_trylock() at high frequency. After using SRCU, the IPC is at a normal level. Signed-off-by: Qi Zheng Acked-by: Kirill Tkhai Acked-by: Vlastimil Babka --- mm/vmscan.c | 45 ++++++++++++++++++++++++++------------------- 1 file changed, 26 insertions(+), 19 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index db2ed6e08f67..ce7834030f75 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -222,8 +222,21 @@ static inline int shrinker_defer_size(int nr_items) static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, int nid) { - return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, - lockdep_is_held(&shrinker_rwsem)); + return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu, + lockdep_is_held(&shrinker_rwsem)); +} + +static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, + int nid) +{ + return srcu_dereference(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu); +} + +static void free_shrinker_info_rcu(struct rcu_head *head) +{ + kvfree(container_of(head, struct shrinker_info, rcu)); } static int expand_one_shrinker_info(struct mem_cgroup *memcg, @@ -264,7 +277,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, defer_size - old_defer_size); rcu_assign_pointer(pn->shrinker_info, new); - kvfree_rcu(old, rcu); + call_srcu(&shrinker_srcu, &old->rcu, free_shrinker_info_rcu); } return 0; @@ -350,15 +363,16 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) { if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { struct shrinker_info *info; + int srcu_idx; - rcu_read_lock(); - info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); + srcu_idx = srcu_read_lock(&shrinker_srcu); + info = shrinker_info_srcu(memcg, nid); if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) { /* Pairs with smp mb in shrink_slab() */ smp_mb__before_atomic(); set_bit(shrinker_id, info->map); } - rcu_read_unlock(); + srcu_read_unlock(&shrinker_srcu, srcu_idx); } } @@ -372,7 +386,6 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) return -ENOSYS; down_write(&shrinker_rwsem); - /* This may call shrinker, so it must use down_read_trylock() */ id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -406,7 +419,7 @@ static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, { struct shrinker_info *info; - info = shrinker_info_protected(memcg, nid); + info = shrinker_info_srcu(memcg, nid); return atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); } @@ -415,7 +428,7 @@ static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, { struct shrinker_info *info; - info = shrinker_info_protected(memcg, nid); + info = shrinker_info_srcu(memcg, nid); return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); } @@ -893,15 +906,14 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, { struct shrinker_info *info; unsigned long ret, freed = 0; + int srcu_idx; int i; if (!mem_cgroup_online(memcg)) return 0; - if (!down_read_trylock(&shrinker_rwsem)) - return 0; - - info = shrinker_info_protected(memcg, nid); + srcu_idx = srcu_read_lock(&shrinker_srcu); + info = shrinker_info_srcu(memcg, nid); if (unlikely(!info)) goto unlock; @@ -951,14 +963,9 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, set_shrinker_bit(memcg, nid, i); } freed += ret; - - if (rwsem_is_contended(&shrinker_rwsem)) { - freed = freed ? : 1; - break; - } } unlock: - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); return freed; } #else /* CONFIG_MEMCG */ From patchwork Mon Mar 13 11:28:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13172284 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6ADCC61DA4 for ; Mon, 13 Mar 2023 11:30:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 701246B0078; Mon, 13 Mar 2023 07:30:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 695476B007B; Mon, 13 Mar 2023 07:30:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4DE0C6B007D; Mon, 13 Mar 2023 07:30:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 405816B0078 for ; Mon, 13 Mar 2023 07:30:06 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 194BE1C6243 for ; Mon, 13 Mar 2023 11:30:06 +0000 (UTC) X-FDA: 80563656012.21.8EB96A0 Received: from mail-pg1-f175.google.com (mail-pg1-f175.google.com [209.85.215.175]) by imf12.hostedemail.com (Postfix) with ESMTP id 4363940020 for ; Mon, 13 Mar 2023 11:30:03 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=gxoAK0RJ; spf=pass (imf12.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678707004; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=joJBORpGAhUrTa0QaRmWtbHEppwsJHnBnzSS2GFBqpE=; b=JCj7Qg63cVZlrRYRaZ1eRUYUj35DZEhVUXRmauLwTTSuy1KB+2PrvUkOukovnFPZVDNXMw o24lwpHPz5iQ2hALwZBGUTYqbITtcp3UkdXEUcf0037ZH6ZsCuJF9+vWvLgA3P7vQhmG7v pi4Oy9d2xIU9AjDxY47ma87SDXmYDBg= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=gxoAK0RJ; spf=pass (imf12.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678707004; a=rsa-sha256; cv=none; b=eG5aePxdnckjopw818xjo+XW3QsJJ4zm0bF8gbJm++CKs0XMbBEHGehKdnaR059ksOYc6g z6HF8t1ZwmaYiCRP/C1Tv8PU6cIFke3iMO0Iks4B4Lm4A7R1+owM3XzfJy7EE2HGFS5862 pcud38/xN27iyKWoBDerlDKaJIGRUYc= Received: by mail-pg1-f175.google.com with SMTP id h31so6682433pgl.6 for ; Mon, 13 Mar 2023 04:30:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678707003; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=joJBORpGAhUrTa0QaRmWtbHEppwsJHnBnzSS2GFBqpE=; b=gxoAK0RJLeAUG9DMQX3AJ19hEgYUC73gn5qDU+xe20VchIqQl4+4Bs8TMoR+xUoHKj ND9SsfWlrkOJpCkSW9SdC0rMfbp2pkoC+L+Zsl8WbNt1tGBTAG2LXwZTOyMQ1iLcCwQd GxK8VSLfc4QaraebduBNkUss03AkdhgD0s4yXld1S43G6GFxDxgVa30i3fDKmLe2+vsn L0khGXfZPxYQ4maITGQLyGizgnKgpvbt2obFC6FA/0459J8cOE4CnMELpfc1XfocweG3 UCt1lbaasgjVAeNzd0lchgP8FL0nLDv/KuKuOjPDpVlt6/LjMfKic6NSPuZGnknpbpu6 jdFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678707003; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=joJBORpGAhUrTa0QaRmWtbHEppwsJHnBnzSS2GFBqpE=; b=2D/ZFpjCjbrcf1ZThaW/vqhGBNUY9dOeCSStYLts+a219a/Tdca72gr6eY+BHkJ2xl PV0koLDXRSLtMlGJJJeYDG92XnRIpryFNVZqLXso9qq/gNXMX7RD6Vl8eu7Beit+3QRU nROhXWRh8M21r5ILc3T+SwGnyeAtSwOGSKPqYP3b1uVRWLBHVCsfltqRJIVf4m8cuTv5 0Z7+NrexbJnt85n8Z/OADLbKvvWO4sWY6hRCkHpKDdy4nRwgp1aSNvxRYp4qjLz29fhI la6nKZSkbrjp2oWaqfcJHQeG8M0zhME0vLje7PG2GZiy7bogk8By3BWI25BC3mCotSdt OALw== X-Gm-Message-State: AO0yUKV+3qzCfegRRjOKPqp1NwBxQJRRSlgdi9MzHa6yzX8rNnTfoYgE QhaZlKEeXYJkQZb80wZoxz74GA== X-Google-Smtp-Source: AK7set8YFjpOWP7oOMMANTE81hjqmcZGdje1Alhj8F/KolKsPGx7n3vdYEg4ks+PCVoaRcuIXePCBQ== X-Received: by 2002:a05:6a00:2148:b0:606:d3d1:4cc4 with SMTP id o8-20020a056a00214800b00606d3d14cc4mr10843183pfk.3.1678707003039; Mon, 13 Mar 2023 04:30:03 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id n2-20020a654882000000b0050300a7c8c2sm4390827pgs.89.2023.03.13.04.29.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 04:30:02 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, vbabka@suse.cz, christian.koenig@amd.com, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v5 4/8] mm: vmscan: add shrinker_srcu_generation Date: Mon, 13 Mar 2023 19:28:15 +0800 Message-Id: <20230313112819.38938-5-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230313112819.38938-1-zhengqi.arch@bytedance.com> References: <20230313112819.38938-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 4363940020 X-Stat-Signature: 9igc1q44w5yw984rrmwnig4hdtd48qme X-Rspam-User: X-HE-Tag: 1678707003-343851 X-HE-Meta: U2FsdGVkX1+7uAUxDMXE4caw6icRqlyBxEztB2NK7Evm5bBLn8D03FNPmSioJNOtnUsmqHippAqBedwBKJYvmJH/PYe57T6IpH22hrUy2/Mw62noUIPckxx6ct0QD2KbJpz+KYtAE/hhXbbA401Bq543mnBXjeya6jT3+KzNV2kjzfC50xfRLUAijD+LBXmmzt1qQEQuwmwZFUADPXtmIMc6r1839u7rAM4G2CMgO2xzfzCtXpIm/mRaqoYwGot6DQItugaW1SYa9Wfb91bFQIRUlX/BznTx4tRzOn/YjPM1BHT5ciFi+lsnhLF3WrUhTGKhwobQuOjSH4Uw9bBVkZYlE4oZFe+p5SsO/yf2U+jGOdQfAJj0CyoLpgYhdtUPvdu571eNHlYz2+KBmyM41sBrjVp2fjHTY2VqsWXugJD6esn1lMOccjWyqwaTJ5kZqQUYsJP5Hp4EcZhL01wg60rVM8A4lcnLzDQZgr8hlZyFUH+Ag/X1jRheOq5i+bQ1cIwm5CXMpxqIeVv0ejBtZHOk3YWI+fSm+JeL1r8BoO7O7bXzHCUr/mZhCOyg6j5V+bUgq2+80jkViMIOnirnTYSj8lZJ7S+cx464pHT7PuEJH5Lyxdiow6aat3kM5lVdH1gWFLjKLEU/6/Zz0ZfG4mGr72oKmqbj6HE2UdvadIvs6wIuk2lMIf5mhT5HMu56HMCcA/rZZMXISJunrGDb4U/iytDvkw9nsZzDQDyWSUiWt/wT9w/bOI3oCm1iQU4atPRMiDOUPSO8/D55UDV9rq5C1ZA752bynm2R5ys9Icfd/sMPf1BlcoslsdU4L8uOJmu94MJ+0v/kVjexSLDnaZOSmaYkr3xKCnlUCu8/11h539Zh7EhpMfKGuwaD0OF3CjeensclRqgVuzwaQnY7QmbYNWSNSKTkGnnXmfuo+0jm5jvTNagJB8B1xsT1XvW2GK+jqSifCEHADCCIf6R khUmMt9F seIag3dWj0OdN2En3nIM/KIp82HmDNXwP4a6joQKYPTTkNnywvbfYnGbEhGTWNvGWHuI/BFP0X2drh2njRNog5gNmpYjTng4LH5md7xhNLM79cdcV8M4NoOKQ0+1TOsjVcGBLHUGVSp+pp/cfcDETy3TwWGF/888m7joG/CclEBthJ4LPBpodyOgn7rtLX+dc3ZO7jXKnfelmjgeab5BpBoK89BxyCg1WRe7Q3G/OBMbjQ8+7NR77GmgEy+sugDenVR2V1HTTUhJWFtUHSXnDpZ/gNOSlGUGVvC6+WhgD7LhUSlyl8yZ49cTXAfRQYCzgyT8UWBoKOKg8GWKKnnJ0DsFBnpx0g4enKbKng/lu5cBfHkBNtFiwARwGKKxyovZ6ffRbfg+6u5pfUIpolzymQzuyaiWgXTD3fXo7VUZYH+eIhVm6IeD/IOwPmnQtvPNX3QlA+5yNrzuJItF4Qjwxd2lx6ypOcvljJXyzaqYmQeRe/Hagjc3KTTGdnMZQrQFVcc41Twwt0qhxxQQTMmMKVO1Kow== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kirill Tkhai After we make slab shrink lockless with SRCU, the longest sleep unregister_shrinker() will be a sleep waiting for all do_shrink_slab() calls. To avoid long unbreakable action in the unregister_shrinker(), add shrinker_srcu_generation to restore a check similar to the rwsem_is_contendent() check that we had before. And for memcg slab shrink, we unlock SRCU and continue iterations from the next shrinker id. Signed-off-by: Kirill Tkhai Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka --- mm/vmscan.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index ce7834030f75..5c2a22454320 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -204,6 +204,7 @@ static void set_task_reclaim_state(struct task_struct *task, LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); DEFINE_SRCU(shrinker_srcu); +static atomic_t shrinker_srcu_generation = ATOMIC_INIT(0); #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -776,6 +777,7 @@ void unregister_shrinker(struct shrinker *shrinker) debugfs_entry = shrinker_debugfs_remove(shrinker); up_write(&shrinker_rwsem); + atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); debugfs_remove_recursive(debugfs_entry); @@ -797,6 +799,7 @@ void synchronize_shrinkers(void) { down_write(&shrinker_rwsem); up_write(&shrinker_rwsem); + atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); @@ -906,18 +909,20 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, { struct shrinker_info *info; unsigned long ret, freed = 0; - int srcu_idx; - int i; + int srcu_idx, generation; + int i = 0; if (!mem_cgroup_online(memcg)) return 0; +again: srcu_idx = srcu_read_lock(&shrinker_srcu); info = shrinker_info_srcu(memcg, nid); if (unlikely(!info)) goto unlock; - for_each_set_bit(i, info->map, info->map_nr_max) { + generation = atomic_read(&shrinker_srcu_generation); + for_each_set_bit_from(i, info->map, info->map_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -963,6 +968,11 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, set_shrinker_bit(memcg, nid, i); } freed += ret; + if (atomic_read(&shrinker_srcu_generation) != generation) { + srcu_read_unlock(&shrinker_srcu, srcu_idx); + i++; + goto again; + } } unlock: srcu_read_unlock(&shrinker_srcu, srcu_idx); @@ -1002,7 +1012,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, { unsigned long ret, freed = 0; struct shrinker *shrinker; - int srcu_idx; + int srcu_idx, generation; /* * The root memcg might be allocated even though memcg is disabled @@ -1016,6 +1026,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, srcu_idx = srcu_read_lock(&shrinker_srcu); + generation = atomic_read(&shrinker_srcu_generation); list_for_each_entry_srcu(shrinker, &shrinker_list, list, srcu_read_lock_held(&shrinker_srcu)) { struct shrink_control sc = { @@ -1028,6 +1039,11 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; freed += ret; + + if (atomic_read(&shrinker_srcu_generation) != generation) { + freed = freed ? : 1; + break; + } } srcu_read_unlock(&shrinker_srcu, srcu_idx); From patchwork Mon Mar 13 11:28:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13172285 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21DE3C6FD19 for ; Mon, 13 Mar 2023 11:30:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B47286B007B; Mon, 13 Mar 2023 07:30:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF6986B007D; Mon, 13 Mar 2023 07:30:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 949FA6B007E; Mon, 13 Mar 2023 07:30:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8768B6B007B for ; Mon, 13 Mar 2023 07:30:12 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 679D31C627E for ; Mon, 13 Mar 2023 11:30:12 +0000 (UTC) X-FDA: 80563656264.15.5BD8CF0 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf27.hostedemail.com (Postfix) with ESMTP id 759D240021 for ; Mon, 13 Mar 2023 11:30:10 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=MvlGryeR; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf27.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678707010; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Oxd2ThxTJ6UijCJ4SDDL7FUC7F7cO/btWP9US9pLSzo=; b=55Hv2ndpJNIZ08A6D+MLlPeeZtCiU8BfEX8KYVk0Y0M8MXItZWSAgkm+FBGvJaFCK0TeIW 9ResIJxCi/LS3Kuhb6sSH6sZ9BiVycUIBYwMfuNih4MAjjTfsduE8FsH3oeh7a4U8G57Nw EEiLvMamEVbGUpKUGT4+8G+n3GdC/AM= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=MvlGryeR; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf27.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678707010; a=rsa-sha256; cv=none; b=jInDQx89UlEnzRqj6ww9MIpTV6C/eC+PnHQnOzJolVFMKREF6gR0+TvkcaqYR6+jQ1K7Zg yDGBZOhQ0l0ftVUSvWmrPNcXg7NI4WtrNUIEWcPUXML7AmiZrzLBg9hIvwdWPd/iiaXSWR pkvl5qTj1lZP5QrL3HAWw/5Eq6nrTK8= Received: by mail-pl1-f174.google.com with SMTP id a2so12563895plm.4 for ; Mon, 13 Mar 2023 04:30:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678707009; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Oxd2ThxTJ6UijCJ4SDDL7FUC7F7cO/btWP9US9pLSzo=; b=MvlGryeRy82G7VkgqEKuMpQfzAezLp2xXrHbHKrQM/eEt81Oq/AfYVdlB/GIobl8It 7j2/egtK7yEFhJCwQO1OEf29hlspWlTFr6UYJKW30dRAjzdPoQtwo+yeUuFCO5XjM1fR QTpRAX/3KhabhS2FMYkh8JFMbk32znCcnVB0I8hQW5KZXN6S0Oxh6LXvEwjcTlYALXpV G6Kh9bDRNBabSRjM+xuJvnmQWOWsrkeBNmVU+B+cq0i32zf+AmsgqKMTYjYBMPOnHkL9 Scfe6Zs4rWOLcpW74stS1wV1wpoCCzVwzmigR2g2N6q5dgesopZRoP+7gE0A+6gqZQ3p 5mjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678707009; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Oxd2ThxTJ6UijCJ4SDDL7FUC7F7cO/btWP9US9pLSzo=; b=v+WPOcu9Q8GRTVuZK8WbxVtpmBVY/1bLm6rvPIrqesC9/H2TYFJOGMD1+H3eva4dYc HPD8QK/8i1zXGy8V6g4MiJjcVsRKXcM9X+GNCVbAkIC2XmdOmM5nhR3uyND547lz6Om6 JoCW/vFv7O2nyK5PfsVbWNzBPLD7NYd87T3SmjHYbxiEaGXiPUuYdOkI2docfdx72y/e o/6qu/goRweNahdc3ScbQOGlnAb0Wt12sUtnmFqNynI7b2GfuZ++8SWJnzOhlrAtiGTh SXGxQr03PacEFb7wGItn7DTYyYtks5Wbg0L5Ajk1UEINrAWKr1rhCGr00DqyylIDctIt OKMg== X-Gm-Message-State: AO0yUKVQU+iOLVz/Dz5QssoR9kkvAUu8Oj3SOfS+AczANTGHZ3tsFzt0 dR9sKI/ISUU2rzSbfzRkVXj20g== X-Google-Smtp-Source: AK7set/YpeTiNPrxyPdk7ef1TN6nco5qoqim2zvGi95z+KNjYPKTOUxtpQzpzKaJth1w+KxBldYOuQ== X-Received: by 2002:a05:6a20:6914:b0:cc:4118:65c4 with SMTP id q20-20020a056a20691400b000cc411865c4mr12735774pzj.5.1678707009311; Mon, 13 Mar 2023 04:30:09 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id n2-20020a654882000000b0050300a7c8c2sm4390827pgs.89.2023.03.13.04.30.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 04:30:08 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, vbabka@suse.cz, christian.koenig@amd.com, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v5 5/8] mm: shrinkers: make count and scan in shrinker debugfs lockless Date: Mon, 13 Mar 2023 19:28:16 +0800 Message-Id: <20230313112819.38938-6-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230313112819.38938-1-zhengqi.arch@bytedance.com> References: <20230313112819.38938-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 759D240021 X-Stat-Signature: ufhggeshqscatzygiqeqqnx6xnbuu77w X-HE-Tag: 1678707010-780958 X-HE-Meta: U2FsdGVkX1//2jSfLwA+w1rK3kAvvv2ycaiHNX49MXV4vr6BS5lkGZW/9YjoeXjB9mFeLpnjd3tD00EvCnoYgwAzE60oI7x4DjvG1wO32nXk98TLlcgzemn4GO564gWElwz+DC9pmr761GK5BjEU09T4+Eu9KjOksBNFpho7Sbr3JNoNdPgXlJ5QOQ3dva6DLjqbD3Og1sINzrdu4FJwi2YvKq8jJJWbqbWLqTzTbaPa5DMrX1Hhq7VuJ4i+VZ/wgMAzj8wbvBUBJqPKhbTIzKLv2ZShZP82YswQ1I/Cl4u7tDBvqFZgYaZLg9Fb9NjXSthINONdo2NsWHXrbZiVAOerHg+vLNzwfJ9vx9iEzoXvOHfRoYundBOYB1mHB8dJTGNnvWSmbUn/9UPN1rJ5uT4Hfcgzh23Bg3R44KIqP7euEwYGgBUNT+elYwXty9AXRft76pbp6XKJGjZ6SFlcatnElGMG9tTVcM8QRwMkJ35AzLfyRV7UQPSD4vw+GxxtKN0P2GzLL+CptBNPPpIDxqx1L7lN7sdGtzsMo9Mcd4blsn9WpDTMPraKp+N23JEzgXAcyjeA/pTnTG4BgSUxElz9W1BzAD58po7iCjQqGaUtu1X8W6eaFkqTnzxnN892iyAXwTTMKJ0o3VCjsIVuuBM7ZnJA5nFmdC4tK6q7TI8CE345xZhfd/fPK9FC4d1vv1scPW24cTQyX4kQ3do0s3G/N6/bqYz3LVZj9fkPMAey8RjvrNC8m3a8AJtN3GdbHgEDkbNH8LK+tXUUuNA+Hepb8o4ZXlSSn+VPSJm6RJ4tJzbuXSe7OPXYTDuzO5BezgyB81FmD4iQvVxwlx+0AgsS2j5H7maxmHebIJEadtdW8IBzJ7/MT/Fxxo3PDW5C420cKCTJOA4iLih1MnDtJuf6uO2t33NREvk4FWOLVzYc78Dj/LN2R5dBZmYKEC4wVsw2mkirXfrX4jUwIal IZaU4hwa LOt62SJ4lc1roGzxCZpReYILI2ike9/G33pfJ0r+60mg2flnpsqEtAtTLz+oapGWkBbYXuSVBVhAD4gsM9Q24xxsM2igq6h2AMkiZFOkKkF8uwM9fYxD11d9dlQJmBagl3BLg1uBwWiTVdkzMfzhzRQY0HgCYPV1KP9tKm2Tdc24LfJuJ4IRCyL5aZmQnAbb/AdFLfgeyiP74MWG6V/O29TPlo2+WpQ09stJ8S9//OjRVkPsG6BybTPiUSOQ+qMvdYbV4qV/fyhZlKM15R+RkrtwS+SJpOwD4NGpeMtQafgij83CkupIuHurfsYeFXUIkdaZE7dsQoGGLEy+e1zlBRW6fa6skSF91K0SYKKix/Dfp1u6xO4VZFRo59Hct7Rys9bwaH0mAg5cMCartx8+aapH13wrbbYfdIWKXqsIwkAUAdAn0t34AgjpYn//Km7s0XNuNgBDnCmkilzU8S5KteR6T8I2kOr5NJqYeq5GzlxG3TfM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Like global and memcg slab shrink, also use SRCU to make count and scan operations in memory shrinker debugfs lockless. Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai --- mm/shrinker_debug.c | 25 ++++++++----------------- 1 file changed, 8 insertions(+), 17 deletions(-) diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 39c3491e28a3..37d54d037495 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -5,10 +5,12 @@ #include #include #include +#include /* defined in vmscan.c */ extern struct rw_semaphore shrinker_rwsem; extern struct list_head shrinker_list; +extern struct srcu_struct shrinker_srcu; static DEFINE_IDA(shrinker_debugfs_ida); static struct dentry *shrinker_debugfs_root; @@ -49,18 +51,13 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v) struct mem_cgroup *memcg; unsigned long total; bool memcg_aware; - int ret, nid; + int ret = 0, nid, srcu_idx; count_per_node = kcalloc(nr_node_ids, sizeof(unsigned long), GFP_KERNEL); if (!count_per_node) return -ENOMEM; - ret = down_read_killable(&shrinker_rwsem); - if (ret) { - kfree(count_per_node); - return ret; - } - rcu_read_lock(); + srcu_idx = srcu_read_lock(&shrinker_srcu); memcg_aware = shrinker->flags & SHRINKER_MEMCG_AWARE; @@ -91,8 +88,7 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v) } } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); - rcu_read_unlock(); - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); kfree(count_per_node); return ret; @@ -115,9 +111,8 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, .gfp_mask = GFP_KERNEL, }; struct mem_cgroup *memcg = NULL; - int nid; + int nid, srcu_idx; char kbuf[72]; - ssize_t ret; read_len = size < (sizeof(kbuf) - 1) ? size : (sizeof(kbuf) - 1); if (copy_from_user(kbuf, buf, read_len)) @@ -146,11 +141,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, return -EINVAL; } - ret = down_read_killable(&shrinker_rwsem); - if (ret) { - mem_cgroup_put(memcg); - return ret; - } + srcu_idx = srcu_read_lock(&shrinker_srcu); sc.nid = nid; sc.memcg = memcg; @@ -159,7 +150,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, shrinker->scan_objects(shrinker, &sc); - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); mem_cgroup_put(memcg); return size; From patchwork Mon Mar 13 11:28:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13172286 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F48FC61DA4 for ; Mon, 13 Mar 2023 11:30:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CA7DD6B007D; Mon, 13 Mar 2023 07:30:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C56876B007E; Mon, 13 Mar 2023 07:30:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD0776B0080; Mon, 13 Mar 2023 07:30:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9C4CC6B007D for ; Mon, 13 Mar 2023 07:30:18 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6E727AADEB for ; Mon, 13 Mar 2023 11:30:18 +0000 (UTC) X-FDA: 80563656516.01.5ED5AF2 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf02.hostedemail.com (Postfix) with ESMTP id 86FDF8000C for ; Mon, 13 Mar 2023 11:30:16 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ItJnDFmM; spf=pass (imf02.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678707016; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=H5UXC4Y8L+sYixm/ZiEHHormamWIB6IrtYM6DyQ3d6c=; b=R85LvhbwJEdOltRDE/kQc9CV2i0d2p4HeNDSyjcAiEboZWC1mVImcuHPkVwVGe0Qw6Y2Wi yjiYKgHVCXH19XQF1Rn891vNtEJZI0yTO5j8wADb/R8suckPmX0aJyqa+zvq8Of8iWes72 ssBBu47WRzzHLtd51Y5fS2WfmwqItho= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ItJnDFmM; spf=pass (imf02.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678707016; a=rsa-sha256; cv=none; b=uAdX9awmjXUM5Lgv5YM+52gjwWY6BzI12CGRIP3PbNwXdImVV4Vf6vuZzvzFMq6AM0gzo1 ASIm3+LHfV3eNP1XpWZK+B94fX0kKLfurPzahl013DEwdIvlJ7MordwDVSCgtwkmVXB2Zk lD/sNLIK/Xq0VcdWg6Zdk8ozE3aG3bA= Received: by mail-pl1-f173.google.com with SMTP id h8so12534774plf.10 for ; Mon, 13 Mar 2023 04:30:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678707015; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H5UXC4Y8L+sYixm/ZiEHHormamWIB6IrtYM6DyQ3d6c=; b=ItJnDFmMsgazPNOaXu8i9CtcCnSzNN57aJj3idjpUOYKHDhuP27hoj7ePdkQNEOby+ riTdnuWQLKyP3nQPylZlMs6mnXVeCSnfHP6Q7AeVCEJSYIy+l6HW9Tbx7RDnuCzBTX04 lg/9ublTRynkmyfmxHDT8KqSD1ZrXcBSzOxYBDdTI0PM8nEOM7P/oyB09c6XGMeI9FY0 ZynsjAfzuUGtByb52mixStZA+f5WzLTbVCttXhkplGLfpxGeI3fKJtOb5yyJciSuqDaa kNFAiFqnIfkQtLxoqnTuMDedbMN6EiT6dScRhSmKgSpp0QGmx1ct8f81dbBiredODvuA ZyxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678707015; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H5UXC4Y8L+sYixm/ZiEHHormamWIB6IrtYM6DyQ3d6c=; b=q5XHvUABAF7lrXzqpVLVHmtH6yelZTwWOJS6D9Os3i3IxEwszQvLyyXFwcjndjJdiI x/v//WMb7ZsNrQwdbji2Wbw+uozuehqVZFHN1S8/ll9bEbOw6M9RbwHB+akv+TbA8iTs T0nbuDcvWn9E513o1vXd2eU+Vv3PkwPoWb0AvStKRb4D+WnAN5EiaUAEBMNIyYcPwxqK cizG8j0VmCozbWWwOQuPkTqjnH7H8rU+xl697D5GVe7Sah/5yH9OmTUbvgyRbwod8LG/ a8vRlZ7x+nnlxN9WOMgC8pnjTu/T/awLDLR1NKSp54mAOZ7p4EdD9hEdLprtuFVa5py/ TwOQ== X-Gm-Message-State: AO0yUKVmTH0kKmqlOoXCI+RJIYqAclrgC2QWwFeYgU6vZJkNubkOwmuo 8ugnT27cVmzJiZxj9zqgEd1F5A== X-Google-Smtp-Source: AK7set+yBkkIFIpn/7Eo+7xlIbSqY5w2muehss8s+sQ+FSFjAoN1HxOMAwRaBpjTqB+jL1sgGBo1BA== X-Received: by 2002:a05:6a20:440d:b0:cd:18a2:f6cc with SMTP id ce13-20020a056a20440d00b000cd18a2f6ccmr13960885pzb.3.1678707015460; Mon, 13 Mar 2023 04:30:15 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id n2-20020a654882000000b0050300a7c8c2sm4390827pgs.89.2023.03.13.04.30.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 04:30:15 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, vbabka@suse.cz, christian.koenig@amd.com, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v5 6/8] mm: vmscan: hold write lock to reparent shrinker nr_deferred Date: Mon, 13 Mar 2023 19:28:17 +0800 Message-Id: <20230313112819.38938-7-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230313112819.38938-1-zhengqi.arch@bytedance.com> References: <20230313112819.38938-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 86FDF8000C X-Stat-Signature: pw1xzjtisx7ts58aymqxwgsoi5o5d1c3 X-Rspam-User: X-HE-Tag: 1678707016-872228 X-HE-Meta: U2FsdGVkX1/yvRlCvXRuQwRwIRVQQJqb73wokbSrZEGNgAQMovHyjYjpos9QfXBeaQSHr7CiMCT62Ms190y09DcRYlLXsUYfv8vyvAWvcKLa3AtqlZPpMiB2SafEcMx4qglLh8SzKRfLRgpA4n88xp7XN66iLvY8y9vVYBrJTamIELrL2EkxtSbqnCn/aiN8+a9N7gkUQUdH+0weMKEO2dvqrlUspHAci5xVlmJaR0VDF2d4DLuLcRAvCS5fp9+r2GVeSWCDx7pfKQ0FMH8XUb2jB1Kvg7QsuPvkVVKeU9lh4f8X6AmbBLOUbOtY5IsYiTNxNh0AfKPOdoarxtMaGo6/ZTM54N/c9+QrKTcqFPKv7aNDjnJyLImRGe5NKCQIh7JE26yjVlAw+NJeXfZ8KlUuWtT13hnkDEYYraCyCMQW/yjO1souZTaNTBtjAjESF7O8n5Sif05S+BSnZ0HH+JLfoLYPsp4WdYQhnjrGydDmHvSoXIo2uuIxZc0mhq5a6m2nllGLYOsJgyyD6RRPHnFysOMzMqQ1FxB9DpWskB27Dd3lgfvbVr4x99N0JBymhAnhxnk1EFrx/KSNeLCzurPSK4X1tqY27YCZld0kjlhnjp3IN2cKKldFbehGFsihdC9XHrHiakpNL3APDKez5Rom1H2iq3MZa3tkdb2AfbN2Ud9MFwcYoo/nUu1uWNaQvC7SS3jpuo0tM8efgeGG6N1hwYPIY0ecDT7fEsOJLykMM8RwE6QSqX9Du4892CGvw/aASM/NbdCoTG575FpwH5txxYrpD8lc55iEVXUelBaj40N9+c8ttMtj6Ah+F28lRYUgE2L5UD2h/f0vo7pSLfG7THCHZrNcj2hBfsXG0SVyY7z7FKhmyOcS/xLA73ce+roNgNp6/xWHWbXJukRwH7FIw2ztUJJhdSk9fDin/cA/Pls/In3rxXlQ8uVTR/vRGRXms2ptPdRUjagAaeZ VT6O2QTO m+/U8plOBxvNXBMqzo/BuWlptQGvG9qUFNAriS24/93fUE0ymsbCfLps5rSP/HlBmWDvlQSEy7iKo7cOCLKe9j4c+wHCBbJQZGX0XRB2KEIb5tm73VzZhvkzyGmpkBKcy1f4XKlhl1/sHa2KxL9nD9Yj/Jrl50m+fZSnC45yRcx9CyEe/daDB6PSzOp8f6aJ1bvsJANbsY6WxTatFQGceR5A3bddZk4Zq+pv7i7EEfral2TxdDOYWfKytRzCMdT8ENjThU0gJ7UUoTBIzsvAIHbut0XxqmJtstQa9Mjha56vyBdVDUHQ6TCggwkrTOXjGHgBDa0PqTeoZtMD6ZH4BXojQmN1fIHMqn10fjeG0DGCtoYlFSruBjg1LLuuWdax2GjFpbFOEVU0XLzmVbWTSn9QdhcrjX1wPhY1Makjk3wd8c80sQS+cEA/z5VeeuoBnix7tKAz0olzEhcmnC0o/aE6fZUgVF53oKl6PTnKXjiOpudAOpzt714dVQIbap/LcuEdOUSvhQG82DrYzi7lNAPuwAg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For now, reparent_shrinker_deferred() is the only holder of read lock of shrinker_rwsem. And it already holds the global cgroup_mutex, so it will not be called in parallel. Therefore, in order to convert shrinker_rwsem to shrinker_mutex later, here we change to hold the write lock of shrinker_rwsem to reparent. Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai --- mm/vmscan.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 5c2a22454320..8c1ae7ea8dea 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -445,7 +445,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) parent = root_mem_cgroup; /* Prevent from concurrent shrinker_info expand */ - down_read(&shrinker_rwsem); + down_write(&shrinker_rwsem); for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); @@ -454,7 +454,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_read(&shrinker_rwsem); + up_write(&shrinker_rwsem); } static bool cgroup_reclaim(struct scan_control *sc) From patchwork Mon Mar 13 11:28:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13172287 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B0DFC6FD19 for ; Mon, 13 Mar 2023 11:30:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 15B9E6B007E; Mon, 13 Mar 2023 07:30:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0E5436B0080; Mon, 13 Mar 2023 07:30:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC8316B0081; Mon, 13 Mar 2023 07:30:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DF3676B007E for ; Mon, 13 Mar 2023 07:30:24 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id B7C671A04A2 for ; Mon, 13 Mar 2023 11:30:24 +0000 (UTC) X-FDA: 80563656768.12.B824EFA Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by imf13.hostedemail.com (Postfix) with ESMTP id CF5DB20026 for ; Mon, 13 Mar 2023 11:30:22 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=LueY7xOn; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678707022; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0Z4Tof/PnwUFj5i0NNzN56+j5kNnB7gWDICsVdyoruc=; b=BtZGBfPownkHrmPFEmN5xLuFNV/St/M/akaqPSOz8LNlNlVQ64kDS5yYecJpmjfWeEx1MG hxKcn05+GE+eJ/YaHA8SFWVDzCFeM7n6CEeonp+EGZrZQJreBulEgg6AXOd85fLNkO03NM Wu7nOzSFo95QRG4njGMHPCUw+PXcLCo= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=LueY7xOn; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678707022; a=rsa-sha256; cv=none; b=3VX9s/BiAD+FuqCetWIacH5/TTr9P/HRlfQqR5uhq8tjLOoXq6ks8puIp4YMgQesW/GaNZ K1wkxJIfDIG1shf5RQKkCrQiPu1UCMS1rkJa3qAKv/e/2VjGfvuWcgSB2oWYoW81JBvpFU h7JIG1DrRSjt0n79+G0HGsfvOgHVJ30= Received: by mail-pf1-f180.google.com with SMTP id bd34so7390980pfb.3 for ; Mon, 13 Mar 2023 04:30:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678707022; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0Z4Tof/PnwUFj5i0NNzN56+j5kNnB7gWDICsVdyoruc=; b=LueY7xOnIXHIbNor8k5oEpAk+tX93l8rCkl4eUPXSjj/FR0coKA+VxIcbga8n7x+3/ UVYQ+xwM5v7aAKibviyLXFIf9yfA5DqdUrbSgL59K0+yWXOscDZ9kE0PPEbSKSAKGOeW fYdcX0uAb27i11e16ubA0DvK2mQmR+nk8oZAtv85shH1l8grZ50ghPaV1UKNldaTWetj vt+eouSQMr0HwGA/8pBbk59dg1Pgdhd0cKvtaYLaKLCdPxJ8k+oWK21Mgq9NtpfoT2OG W0HAgtKswNDuPDIH/Ey2qVGkFfAIGkpwaECFhXVuXnSl/4V4k4XFXHrDTRlXi/isujEx xkxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678707022; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0Z4Tof/PnwUFj5i0NNzN56+j5kNnB7gWDICsVdyoruc=; b=EZ8exOAUf0OC63ucP9ILP8+iesI0ovdNwJqRi8H9cQqvOlUTC8lw8w8b/vxpK77Du2 8TgciJdJGbYeQsTFmvrkPQImG6rk4wvNAfciXGcTfkd0Fd4Tb/ktvMGtvpQuXb+Fx97g 1w0H2+d8I9sVpQhYx32JGVb1rS0CGhjaRqrVy74Gg4Pr3PVle1Yfksi5vl+JVhlhFzvm LqM17D3xHNvXdZfbdD4m9vyiGWsz/qTlRkMNqC53Wn1j/2L3NVMuCoPDMgfX2ilGDvkP l5RQOsesCcS2oBiyEbB4XlYfVYxvAZbsG+w2X4mCOD0mEoUHaLekqJmlZWMtTkGMYZAp B25A== X-Gm-Message-State: AO0yUKUvqfSMJpG35WnEtlHxaAPGU2q0uWjRToCVtDaWyftHdjXoHVUn YFmTZC32aGWS2ZxttdrNWR2Dbw== X-Google-Smtp-Source: AK7set/YWnhAf+YFhv2OXt5fozTrmg2G/24NbjxXRH9lebvV7mE5dh40UrMVfpfJ10aHjkOkvNuTfA== X-Received: by 2002:a05:6a00:2148:b0:606:d3d1:4cc4 with SMTP id o8-20020a056a00214800b00606d3d14cc4mr10844065pfk.3.1678707021778; Mon, 13 Mar 2023 04:30:21 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id n2-20020a654882000000b0050300a7c8c2sm4390827pgs.89.2023.03.13.04.30.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 04:30:21 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, vbabka@suse.cz, christian.koenig@amd.com, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v5 7/8] mm: vmscan: remove shrinker_rwsem from synchronize_shrinkers() Date: Mon, 13 Mar 2023 19:28:18 +0800 Message-Id: <20230313112819.38938-8-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230313112819.38938-1-zhengqi.arch@bytedance.com> References: <20230313112819.38938-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Stat-Signature: a6f37uqo8jy3hqqdzubdi5dx87n4641f X-Rspam-User: X-Rspamd-Queue-Id: CF5DB20026 X-Rspamd-Server: rspam06 X-HE-Tag: 1678707022-362590 X-HE-Meta: U2FsdGVkX18KzHd694Xqcf36jrR0388YIjnjukaivzS1Ho3F6s6rx4q7WW63FY7NVaxV9UA/9rGqEXPXM0cwQOD9HBKfxhuUU57SasMBC6I+W16jTaEOGkMgCufzKOb68XNqrQpnuU6MwT4s0qwwwCJC4nzUvviZmzqilKoBuCAjfS9wl8orZ1yfVWc5ptLwa0ulNFHoOo9//k3sGw9/VLlAwlov8EHoQMN1hQVpPg9hTE8y3TUUu/20WT98b1RCmuDClq+US/HDV/Q7+uZ3PSUgACoxRpXvnbESr4b/a8kU3Iro/0Mbthw30kZzPU62kaQMpdDelm3C0gzcRTCZne+BgdhfUz+VtgAo0hDaaunmhAs/2Liyv4DWmfz8qA33Y3/WXZAyKgToCKXedOhRZadpIk2dcAt9o0TfBAD9+PeOr55X/20aUZAOUa7QdjHodJkNmnmKJOYvNtT5qNkloj4arAcY+gOz6/W98MypyEb0SnoZzS8EkTBP719hgAgufELloe30meLSVXJoJ3Q9LcyfJHRuZD/5l5NxaAB1GgRU6PtrSZQVxlcZ6rgJS/jbOywvsgp7lo3iy7aHAw7Jfj8vIixND1+ltE4qGKa4J82Z201YvrbMbmqT6NGmqgWCAccoHdsv9ecN+rPS2kmADoadHxV2FzwNvTBOH+nIUn3aliApCNVF8iGXN776YR3ez015NNjQm9AxcuxKafzm9+H6u2mL/iMCsC/5vkAIQXzuKtqiaClQ5VSqBSCn8lw5/yvzOvjxEYTmBhFnQ1TzjB27deqi3P8DrfH+enHkEOBtlMQw/tWNlJZS4WEjL0IEAbfImQGxpyxR2hYGVAII6wvjLf7FsNNLq2I2xi3/b569snPC8pFQULmS12LtpGwxmWBEAKQgTH7juIh4QBDA8qXID919NvdK+RwanR1PGBxH5oREAfSnfl2WPFzSqm0j9kvnH5h5OEOGNp6dpCX A5R0WGFV mftqHl4CLku1JT2Aq6fN26W+6R3Q4CgBQXKCUkCotMrM9iKXzHHW6sgKVRmtUhq0NtemKXHx0wlKTzSKWSM2k8Lgi9O/7mEz8Nu8rSSxP1Q47GHcSn2uzVW8RrFzsPegekNauag0FuOQVukCZRYPk6rraUGvtILsLK89TiURCGwYtIi8GsP1EVyEwMuu75d+YHXrS1AENfRk61uz/l27WGrLYst2SlGxcq5xcNJQ20krLXrHQP/M+KyZO6AAtuS6BgjsnoUAv9tRphlWhf3DsWDoQijpchb60kJsFxDxDd+eADsfDR4gtVqI7SgMwbIKeGCOSURqBKp6sqyu+MRPALeMRjBYwLym5Rr9tesKvg17YTX+2er7wvieqceESORj1MSMjCHh/wsXTTvmCNfudHMWNuMHxk9znq+tWgFDMWPXayqnBDTK3TLp+ajh46itGGQJcoszbcf2nssrJyChR20CTwHSqWHVcg6iDUwnx5CIhHjENTuaceCehtQKT9fTQyudRXXPsRytte1x0iP5ek8TlWw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, the synchronize_shrinkers() is only used by TTM pool. It only requires that no shrinkers run in parallel, and doesn't care about registering and unregistering of shrinkers. Since slab shrink is protected by SRCU, synchronize_srcu() is sufficient to ensure that no shrinker is running in parallel. So the shrinker_rwsem in synchronize_shrinkers() is no longer needed, just remove it. Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai --- mm/vmscan.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 8c1ae7ea8dea..2b22a42d83c4 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -790,15 +790,11 @@ EXPORT_SYMBOL(unregister_shrinker); /** * synchronize_shrinkers - Wait for all running shrinkers to complete. * - * This is equivalent to calling unregister_shrink() and register_shrinker(), - * but atomically and with less overhead. This is useful to guarantee that all - * shrinker invocations have seen an update, before freeing memory, similar to - * rcu. + * This is useful to guarantee that all shrinker invocations have seen an + * update, before freeing memory. */ void synchronize_shrinkers(void) { - down_write(&shrinker_rwsem); - up_write(&shrinker_rwsem); atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); } From patchwork Mon Mar 13 11:28:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13172288 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2CA3C61DA4 for ; Mon, 13 Mar 2023 11:30:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7503E6B0071; Mon, 13 Mar 2023 07:30:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6D87F6B0080; Mon, 13 Mar 2023 07:30:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 52B3C6B0081; Mon, 13 Mar 2023 07:30:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4030C6B0071 for ; Mon, 13 Mar 2023 07:30:31 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0CAE0160389 for ; Mon, 13 Mar 2023 11:30:31 +0000 (UTC) X-FDA: 80563657062.16.708C8F1 Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) by imf13.hostedemail.com (Postfix) with ESMTP id 1B49B2001E for ; Mon, 13 Mar 2023 11:30:28 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="SVZ/kd4y"; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678707029; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=veHvRh5W1aQDA0P3DcAtGH917iTj6D7LjZJelB9m0SE=; b=0ou6CSSyfFsjR76iSZEH1uQhXSF84g+qYdad1CQOrCW+/F08/MurLpxNuHq9jY+m3Knah7 UFvIr6u8UODm1Z1DPuAubfR4+iEcwq7Q+gWrzIiNYNhhrF7TnUDjoWGHiY9Bc1SfGTO7Gu WcKfa0quvQtGyk1MRpgV2ACCyz37Zn4= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="SVZ/kd4y"; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678707029; a=rsa-sha256; cv=none; b=AH5S6QZkoCUCevLVSduuuhrNAlv60d82ULnc3Q8qEEckJ5OB6E3qG7i3c5dUKHyB0UzGbx PkV9iULfW+cmJTTRg+iZn6Gy2+dt7oz5EKg3iBnWTGK6gEcvs3DdaccXfnq/f7Qw9ueNse xViAFoasXg/pJFLD+t8X9zbwelJXHTA= Received: by mail-pg1-f171.google.com with SMTP id s17so6681708pgv.4 for ; Mon, 13 Mar 2023 04:30:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678707028; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=veHvRh5W1aQDA0P3DcAtGH917iTj6D7LjZJelB9m0SE=; b=SVZ/kd4yVajGRKXhMfnj08cBdN48skJmJEGIspTu2+CDNIdJP4iVJq7QNuWbryFTFQ 5WArVBLbgGw6BT+81/ZTiH5DRhGi3JDvekIZehLu2+/q199ZQJ1MzMGsH3B6ep70Nh2j O9CDfWe6oBvsg9PWBDzLHKCNF/NITz+vSqLJI6Zvlx3SIOWybhEFZsRLhi+R8hbquZII EF5ApH6pEdxse23guW/P/4ktZI7CDraqOYOVYGnfra1lYuYfsl2CmxifRbGwgg74BxXU wCwmiL2GpNFoGQ1kXBpQ/7VvFCZVNNvKkhzQxx4qZOb24QFhzFh+GYU9RthjQ0RmrpAP KQEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678707028; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=veHvRh5W1aQDA0P3DcAtGH917iTj6D7LjZJelB9m0SE=; b=D5mG73c25R6UHcUBqiFaJevAyi6eyYV7SKC9ABM4GANZ/62TYIaBMyRbIvzfw2GTFq mMuzDRR31I6LJEZmobEjqgPvSCC7Crfy5paDWEeXlGUIQ+dSHzeRuCFjasN0VQuGeuTo PneZaSXNcJlj7dpaHnc/2QNbMgAW71uGAyACCZ0F0DJ1k2U20BBVLZk+BrVu1dbdirFo sKhnhazp64tlx3t7ZfZSYQ8vDhg97TVdjuf8cLLkdi2OXYospC6KRfQfCueYHEnjYCuz zX2fgyrDolb1Jdr0hunk24/SfVFSzon3zqiEbrGb/dGmdSpsfmfMBYlIvcYC1Zg9IDB9 ZgUQ== X-Gm-Message-State: AO0yUKWQdVutI86dYhSmy/r/44Cp7rnwOS/PAOcoHxspFsKBn6xCLE0o 8AOWARiSF6WELtcupkrLNlim7g== X-Google-Smtp-Source: AK7set8i2b/48hUTIPTL5nz8DyCQ8zcW2aEytP3igjsldvVyv3Zw/RxUP33QJvrhhgkOlxsg3YqeKQ== X-Received: by 2002:a62:8784:0:b0:623:77a8:8f65 with SMTP id i126-20020a628784000000b0062377a88f65mr4740537pfe.3.1678707027920; Mon, 13 Mar 2023 04:30:27 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id n2-20020a654882000000b0050300a7c8c2sm4390827pgs.89.2023.03.13.04.30.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 04:30:27 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, vbabka@suse.cz, christian.koenig@amd.com, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v5 8/8] mm: shrinkers: convert shrinker_rwsem to mutex Date: Mon, 13 Mar 2023 19:28:19 +0800 Message-Id: <20230313112819.38938-9-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230313112819.38938-1-zhengqi.arch@bytedance.com> References: <20230313112819.38938-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 1B49B2001E X-Stat-Signature: cwr336dexkxqjyr7cmpjq6djfthtzg4a X-HE-Tag: 1678707028-928833 X-HE-Meta: U2FsdGVkX1/qZj2Cr1Yz4esWdGWlsGd9vmh4J5NzxcI4QBtHAYG08VFjyM90mh8hW0hCTrduSS+WfBp2rim6bKcxhmrC8gJqra7ulvYgO834LzGC/AU/XibkYDp0IABExidsOGdTGMdYHj7rg4nm6DHoCbeU2lfb5YtrwifYNQqUYPbC8ugY232Nf9/53dBX3SViSucwZCY87RiW0a3iWKKqf8z7fbibfOAjQedh+9Hdky4/dQC2pUCIA5zu4AJj4ywayRWmTnCXliKENeKy9Pm2G10rXZe8I7IaBZQjhg0agKwVPmDS0oQfp2gre8ERuafia+Rfacn4OmYGmQjj1hMjVulpSPlGYIQIrrmwXPAvjxwKzjZRfKO+7xu6cWttoDhSawJ4InBpI3x6O+XHMZBV8LdZHpAVHTIMaU/r/yj/poQtJMSffdGP4hO7FzorWStkwP/1+SHtJtqrRy5ZwglClNOygTJbJhq7ODZaSfbY4fiBlqFTa+lHUVu8ovldCEtIaJKBgYIirfsIaXlYpeCtA07GWtPNPp6qvj+OCT8D3FzJ7JDbR7q3QsGJJz6mtZqNpwVisRBQMKR7YcPKsKA8Pfzo/qgVKUkDki5y9eIc2YM8BVbm7X5kyL9c+BjXMTc7wELJzAhZWjBIywrDD6lg+iJrPMBSTLdd1xKwvE/sE4kIwbUThjNP02IMX/jZ0WDxpUbYQZXBdc0Xl2fI/ZtwOdL8etPAPLfU6QGLylJHOnDBh+Sf68XcVOCSKkFjLzr8oZluBszbfj9JGxW7rc8C95BBsasBnMQxlLdjWHuTImtGtmuBvohGnSpCFpE1s4DayLXUVaILe+xS7BEfizWpnP+Zt5wZiU5kI2vl4Ll0kuBSxkneAf0zqguCHfIGxMX4pxGIgRg8KqPV/Jbqb+B30ZA5SqqrjAIy3WPhBeaVXIL8ydjrl460qCNdAiw1DQ3lbBHw1vu8iOLhF64 KmWmR6he fqJbATtj7zSR6vz4K9S8MOzCEbNK9aEcarsSWHi9vX2XCHxPs7SZuZn1SrdaPPnWpHq77w7IiqeQwSH1LnQHeM1EwYbeRGvjZqNeDICt7/H0uRNAH8LB+QH4h2YN82uHaHi8v/tUqqCLODIa89A8o4flfU5IFr0xVj/pwTUBD5yfl0Gx2DmPovJjGNr2ZtFF0Fisk1LR80INbRnHmSELDOdTo0AhaMPBstvmnYCwcQThTG8Tou7vL3f1xCzyKHKHjNth4gLR9FiyaNXxlMkRfbM7aQsLZXW4U4fOPDOgndy4XhIsqnDDc2sFx+wgiaXDpTYmXdPV4upyBUkUp3jBmXvcOQDJvu6OHX6ak5zT81Ne5RylHMYRtJN/vN39ZF7lLu+x8kMDxAf2sejwsv3z/MblRMCAxEXW2nY2LXrDcAY132G670zh44bOodCx4UQvmlaF7kVTA2Ky5lVF3FPumzbDEau2nWbkmKpZ0V0OQ9IHZU96Fm5Qq6IsTjlGJLU7LhfcbwZSvK/u8EdXUcgnG4D2aFQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now there are no readers of shrinker_rwsem, so we can simply replace it with mutex lock. Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai --- drivers/md/dm-cache-metadata.c | 2 +- drivers/md/dm-thin-metadata.c | 2 +- fs/super.c | 2 +- mm/shrinker_debug.c | 14 +++++++------- mm/vmscan.c | 34 +++++++++++++++++----------------- 5 files changed, 27 insertions(+), 27 deletions(-) diff --git a/drivers/md/dm-cache-metadata.c b/drivers/md/dm-cache-metadata.c index acffed750e3e..9e0c69958587 100644 --- a/drivers/md/dm-cache-metadata.c +++ b/drivers/md/dm-cache-metadata.c @@ -1828,7 +1828,7 @@ int dm_cache_metadata_abort(struct dm_cache_metadata *cmd) * Replacement block manager (new_bm) is created and old_bm destroyed outside of * cmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of * shrinker associated with the block manager's bufio client vs cmd root_lock). - * - must take shrinker_rwsem without holding cmd->root_lock + * - must take shrinker_mutex without holding cmd->root_lock */ new_bm = dm_block_manager_create(cmd->bdev, DM_CACHE_METADATA_BLOCK_SIZE << SECTOR_SHIFT, CACHE_MAX_CONCURRENT_LOCKS); diff --git a/drivers/md/dm-thin-metadata.c b/drivers/md/dm-thin-metadata.c index fd464fb024c3..9f5cb52c5763 100644 --- a/drivers/md/dm-thin-metadata.c +++ b/drivers/md/dm-thin-metadata.c @@ -1887,7 +1887,7 @@ int dm_pool_abort_metadata(struct dm_pool_metadata *pmd) * Replacement block manager (new_bm) is created and old_bm destroyed outside of * pmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of * shrinker associated with the block manager's bufio client vs pmd root_lock). - * - must take shrinker_rwsem without holding pmd->root_lock + * - must take shrinker_mutex without holding pmd->root_lock */ new_bm = dm_block_manager_create(pmd->bdev, THIN_METADATA_BLOCK_SIZE << SECTOR_SHIFT, THIN_MAX_CONCURRENT_LOCKS); diff --git a/fs/super.c b/fs/super.c index 84332d5cb817..91a4037b1d95 100644 --- a/fs/super.c +++ b/fs/super.c @@ -54,7 +54,7 @@ static char *sb_writers_name[SB_FREEZE_LEVELS] = { * One thing we have to be careful of with a per-sb shrinker is that we don't * drop the last active reference to the superblock from within the shrinker. * If that happens we could trigger unregistering the shrinker from within the - * shrinker path and that leads to deadlock on the shrinker_rwsem. Hence we + * shrinker path and that leads to deadlock on the shrinker_mutex. Hence we * take a passive reference to the superblock to avoid this from occurring. */ static unsigned long super_cache_scan(struct shrinker *shrink, diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 37d54d037495..fdd155fd35ed 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -8,7 +8,7 @@ #include /* defined in vmscan.c */ -extern struct rw_semaphore shrinker_rwsem; +extern struct mutex shrinker_mutex; extern struct list_head shrinker_list; extern struct srcu_struct shrinker_srcu; @@ -168,7 +168,7 @@ int shrinker_debugfs_add(struct shrinker *shrinker) char buf[128]; int id; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); /* debugfs isn't initialized yet, add debugfs entries later. */ if (!shrinker_debugfs_root) @@ -211,7 +211,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) if (!new) return -ENOMEM; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); old = shrinker->name; shrinker->name = new; @@ -229,7 +229,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) shrinker->debugfs_entry = entry; } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); kfree_const(old); @@ -241,7 +241,7 @@ struct dentry *shrinker_debugfs_remove(struct shrinker *shrinker) { struct dentry *entry = shrinker->debugfs_entry; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); kfree_const(shrinker->name); shrinker->name = NULL; @@ -266,14 +266,14 @@ static int __init shrinker_debugfs_init(void) shrinker_debugfs_root = dentry; /* Create debugfs entries for shrinkers registered at boot */ - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_for_each_entry(shrinker, &shrinker_list, list) if (!shrinker->debugfs_entry) { ret = shrinker_debugfs_add(shrinker); if (ret) break; } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } diff --git a/mm/vmscan.c b/mm/vmscan.c index 2b22a42d83c4..8faac4310cb5 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -35,7 +35,7 @@ #include #include #include -#include +#include #include #include #include @@ -202,7 +202,7 @@ static void set_task_reclaim_state(struct task_struct *task, } LIST_HEAD(shrinker_list); -DECLARE_RWSEM(shrinker_rwsem); +DEFINE_MUTEX(shrinker_mutex); DEFINE_SRCU(shrinker_srcu); static atomic_t shrinker_srcu_generation = ATOMIC_INIT(0); @@ -225,7 +225,7 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, { return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, &shrinker_srcu, - lockdep_is_held(&shrinker_rwsem)); + lockdep_is_held(&shrinker_mutex)); } static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, @@ -304,7 +304,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) int nid, size, ret = 0; int map_size, defer_size = 0; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); map_size = shrinker_map_size(shrinker_nr_max); defer_size = shrinker_defer_size(shrinker_nr_max); size = map_size + defer_size; @@ -320,7 +320,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) info->map_nr_max = shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } @@ -336,7 +336,7 @@ static int expand_shrinker_info(int new_id) if (!root_mem_cgroup) goto out; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); map_size = shrinker_map_size(new_nr_max); defer_size = shrinker_defer_size(new_nr_max); @@ -386,7 +386,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) if (mem_cgroup_disabled()) return -ENOSYS; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -400,7 +400,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) shrinker->id = id; ret = 0; unlock: - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } @@ -410,7 +410,7 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) BUG_ON(id < 0); - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); idr_remove(&shrinker_idr, id); } @@ -445,7 +445,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) parent = root_mem_cgroup; /* Prevent from concurrent shrinker_info expand */ - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); @@ -454,7 +454,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); } static bool cgroup_reclaim(struct scan_control *sc) @@ -703,9 +703,9 @@ void free_prealloced_shrinker(struct shrinker *shrinker) shrinker->name = NULL; #endif if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); unregister_memcg_shrinker(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return; } @@ -715,11 +715,11 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |= SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); } static int __register_shrinker(struct shrinker *shrinker) @@ -769,13 +769,13 @@ void unregister_shrinker(struct shrinker *shrinker) if (!(shrinker->flags & SHRINKER_REGISTERED)) return; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_del_rcu(&shrinker->list); shrinker->flags &= ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry = shrinker_debugfs_remove(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu);