From patchwork Mon Mar 13 11:28:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13172281 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19E84C6FD19 for ; Mon, 13 Mar 2023 11:29:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6DBE96B0072; Mon, 13 Mar 2023 07:29:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 663EC6B0074; Mon, 13 Mar 2023 07:29:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4DD856B0075; Mon, 13 Mar 2023 07:29:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3F1AB6B0072 for ; Mon, 13 Mar 2023 07:29:46 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id DFCE1C0427 for ; Mon, 13 Mar 2023 11:29:45 +0000 (UTC) X-FDA: 80563655130.16.D97A580 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf13.hostedemail.com (Postfix) with ESMTP id 0FB622001C for ; Mon, 13 Mar 2023 11:29:43 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=cMAG05TH; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678706984; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cHjpkGxbHVFFSyOk4f8KxselJo2cIaqjL6okRMIx/Nw=; b=MET7vGG3FzDKtshy/srczu7gNh+3jd8AAu6Pl8T1fk3Nucx/ToFg52QghruF++cxIkdQPi OvClvHe9ijoMKQTmcvNRmCQ9Gr9NDytle2rA4YwS/UknWGmQ3OwFk5ih3t/BrE/zkmUcYw BWOPmkO+T2NOWlOvxixvwZ9eOz2azks= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=cMAG05TH; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678706984; a=rsa-sha256; cv=none; b=pfbvdsp6B+ldllVsP+agXAQlf1CrKh7Jj4lc1Xm3zDPmbwQulrMtU21+ozQSwD8rHA6Pmf NoTaRMp/fjxsxS0c9Ul5upHctTIsNoeWxVZ65v213ybyCAvumWfXW3VH3yft3BngPUjRuW HzSrK1D9JMAugssEcNsWxsxh5xGIzVo= Received: by mail-pf1-f181.google.com with SMTP id z11so7388922pfh.4 for ; Mon, 13 Mar 2023 04:29:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678706983; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cHjpkGxbHVFFSyOk4f8KxselJo2cIaqjL6okRMIx/Nw=; b=cMAG05THHEL45EY/ZCrlGRIbq1kt4I6LPUMAH/dQjouL+NMY60brEhRYxofz4Sc3yO jVllerX4euE/VSk8gxVy2qydl3B1I0ZvizXWnB/k4AlXoCTSsvfCFO9lUUclwXD/c7zx +Lstf6Kcx2x9pLpDuB/8MwyscxtFe5rDeDGAP/r3rkvK8Nru8nZQojqHvbsNe7a2Ab7/ gUPJ2C1u8cgdUyoJ+NnEXAlZ1izrfSpHumWDpFe9DMlhF0rV7W8iS0Gr8SVDtkxfdNr/ N3obCyhq6rnOWhxjxKbMgLykNy2bkGDKgby8e4YMGNLWFieQ1ksal3K4NhZ3AUVdAOK+ UsUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678706983; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cHjpkGxbHVFFSyOk4f8KxselJo2cIaqjL6okRMIx/Nw=; b=B7APolSBC2gob31OPk99du0dGYEKkaxXIBdpNaY45l/3G01yhsCGjoyMfFb2Pm1HGv 8p44ShRYX1yJfASxibQ967u2dPew62I0DQERw5Y8EkJU7icU7ZyYzym+IO9OyzYQbtCe 7fO5BzvqoPCahlHNYg1hHOj4IxeltqahMmf0vGfJiijh9qLkf5XOFeFWY7GRH5QAbU4A iQmi6VuwuXUVuoEP2vscmzP4KNmsx06Uti+lySDu4VHrSB8xKgcjTBRl0ugkAiPo6L8Z xzdwiz79Ugrh5QqjxGmL2UHDogE8D1HWAAzRaTzWkLwgQ2P5Mv4FL+t95yMZKDCRTF3I WRvA== X-Gm-Message-State: AO0yUKW4+3xa1IjPfPBZWhzbkKz14nltVUciMpFbFClZSk7kXsrckwCS 922lix/Rzn0PsaoRvd3jaAWxDg== X-Google-Smtp-Source: AK7set/7+CC2e3gj8ocPsHr26haOK3QREhbFYzVHIRff/+7vh0RhPeq9x+b/EZ+X+KieTYLWMp1SeQ== X-Received: by 2002:a05:6a00:2148:b0:606:d3d1:4cc4 with SMTP id o8-20020a056a00214800b00606d3d14cc4mr10842399pfk.3.1678706982825; Mon, 13 Mar 2023 04:29:42 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id n2-20020a654882000000b0050300a7c8c2sm4390827pgs.89.2023.03.13.04.29.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 04:29:42 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, vbabka@suse.cz, christian.koenig@amd.com, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v5 1/8] mm: vmscan: add a map_nr_max field to shrinker_info Date: Mon, 13 Mar 2023 19:28:12 +0800 Message-Id: <20230313112819.38938-2-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230313112819.38938-1-zhengqi.arch@bytedance.com> References: <20230313112819.38938-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: g6k4dur6qfk37ysamyrnt8donnsj3u3i X-Rspamd-Queue-Id: 0FB622001C X-HE-Tag: 1678706983-214151 X-HE-Meta: U2FsdGVkX1/JEzp+Xm/FdwfJi4XVYO5k1vEnxVFt7XGPLyt2ekizSaoypHjJXJ7xl5B2ohvM2ylV03lJUZDJfgSBc/iZWrE9BhBSiGmO/xlFoLZCpenFu7Cb4F09A9KtMWaip/gEyR8EpxwGhcINaj5PthDd7h81J6bkRUy/8BovLpZrEYETOAUlW907UVpt5NfdpJCJUXLmrg5tj66T5AI5CBkv3ypXJK5hwVYTX8KMxzLZMsmtXGlj0nbS+VEGuNOzZrVEsf7qxz3pBWgt+Hc2WyUn3v3zrE82NsDAAfPlcvKqllm4vI6YQCbYwAYHD2/MbbSwppi4n12xSsd4Bh7nzI2Fs6PkH6gnWt0DK31mNVAuE72DrHQt9viyWDrAoCtE+RcftQ9RvVXuvKu/JFuhJrBiNhAGugR+hFYfBAPOZWQq22cHYtfPkUzVV6HRf6DLzd1tQdrPtLcdQYCG23houXcMI9JBowvViffGEHmaiokZzKnyMXw15VzBxK4jaGEl0A5D1wVs9tWhcU7tUoT6f4LCw9Bhn5NGz1xXOOTMYZOPGe+uchEKlNyoXRgspl8rD0+eseSAFouEILdGWlEb2HEHylb7TOBoLHjWcu/NxsJSZSSP9YCyRSlcNnPE6Kf52OP9z4LwxrhuhD6cCkH/+NI0y2asCNLwkuAU3T6s+MsmpPjgijXklqd8kAy2FbbjC02pJM2vLSnP+FO9cubUim9MiddK3vQR9eTOYHZfxD4arG0GHP/Kzpz4dZIG4jOpFcoe+lfIIDoPLAzJi6yvdpahT+7W7A8EEg/tEeq7id+mH7YTAwurpFgfaFveE9YZSVLfuGKgOXj7UKrVPU3+QAbNc+0rF1AoxmvQJzUaBVMlkFX3qlv3FXRQCAdwhmi6ffjJfF/qKH18hzEzFhdmRvSA2dQLJsupw3WNrxXPyuckBPRwevmCUwZ89r6U5MLVvT52wUvCIW6NFLy Pq1y8axd dwWhMjz43CeL92hOd+BsIVn9/AbPX6sPYe0+lf0w9xqrQRkq+eSkLofci/nr4kMSJvkGobgCo/BBzMPjKIInTT6fllARC8zf8jYaQ9VjE4QKP8uypoev1IxrsmUjrpZFA97mTyuOcErlw8ADg+xK5sV7LX06l/IsEM91ub3uUNCFxFmCz7fuzDnKFJQUQACj9wQ/f6qWbkUP9RwiXUWVfCvO38+Kuf4cer2Df+8VTRRjKu9G9ftCGSSC91GW3Qi3+VG4qUBAI73nxcse3lTEGcWS5DE5kCiG0+VSIJanWMXCY+aw8JXHlO5pH4ln7uuwqmaOt7kwgmBjzU6Ddd0JQ3wUhoDibplWHwdFGJe5x26mraBTWU8Iu5KOU/ojFvOAPRLMXSu0I85gGknfGkQqiRSeXPBY+e/R0DzRus3m7KoWiKziQhyWEB83JW2eKxmW8CniGWEgY1c53LEUSxGTvy5Dd8bgs+PWU2SIBH6moS0O79lmn2xhb5iaulWc4OngRvHfath5yv2hciS4bg2UUNzjPFg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To prepare for the subsequent lockless memcg slab shrink, add a map_nr_max field to struct shrinker_info to records its own real shrinker_nr_max. Signed-off-by: Qi Zheng Suggested-by: Kirill Tkhai Acked-by: Vlastimil Babka Acked-by: Roman Gushchin --- include/linux/memcontrol.h | 1 + mm/vmscan.c | 35 ++++++++++++++++++----------------- 2 files changed, 19 insertions(+), 17 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6eda2ab205d..aa69ea98e2d8 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -97,6 +97,7 @@ struct shrinker_info { struct rcu_head rcu; atomic_long_t *nr_deferred; unsigned long *map; + int map_nr_max; }; struct lruvec_stats_percpu { diff --git a/mm/vmscan.c b/mm/vmscan.c index 9414226218f0..9a2a6301052c 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -226,7 +226,8 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, static int expand_one_shrinker_info(struct mem_cgroup *memcg, int map_size, int defer_size, - int old_map_size, int old_defer_size) + int old_map_size, int old_defer_size, + int new_nr_max) { struct shrinker_info *new, *old; struct mem_cgroup_per_node *pn; @@ -240,12 +241,17 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, if (!old) return 0; + /* Already expanded this shrinker_info */ + if (new_nr_max <= old->map_nr_max) + continue; + new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); if (!new) return -ENOMEM; new->nr_deferred = (atomic_long_t *)(new + 1); new->map = (void *)new->nr_deferred + defer_size; + new->map_nr_max = new_nr_max; /* map: set all old bits, clear all new bits */ memset(new->map, (int)0xff, old_map_size); @@ -295,6 +301,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) } info->nr_deferred = (atomic_long_t *)(info + 1); info->map = (void *)info->nr_deferred + defer_size; + info->map_nr_max = shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); @@ -302,23 +309,14 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) return ret; } -static inline bool need_expand(int nr_max) -{ - return round_up(nr_max, BITS_PER_LONG) > - round_up(shrinker_nr_max, BITS_PER_LONG); -} - static int expand_shrinker_info(int new_id) { int ret = 0; - int new_nr_max = new_id + 1; + int new_nr_max = round_up(new_id + 1, BITS_PER_LONG); int map_size, defer_size = 0; int old_map_size, old_defer_size = 0; struct mem_cgroup *memcg; - if (!need_expand(new_nr_max)) - goto out; - if (!root_mem_cgroup) goto out; @@ -332,7 +330,8 @@ static int expand_shrinker_info(int new_id) memcg = mem_cgroup_iter(NULL, NULL, NULL); do { ret = expand_one_shrinker_info(memcg, map_size, defer_size, - old_map_size, old_defer_size); + old_map_size, old_defer_size, + new_nr_max); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; @@ -352,9 +351,11 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) rcu_read_lock(); info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); - /* Pairs with smp mb in shrink_slab() */ - smp_mb__before_atomic(); - set_bit(shrinker_id, info->map); + if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) { + /* Pairs with smp mb in shrink_slab() */ + smp_mb__before_atomic(); + set_bit(shrinker_id, info->map); + } rcu_read_unlock(); } } @@ -432,7 +433,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); - for (i = 0; i < shrinker_nr_max; i++) { + for (i = 0; i < child_info->map_nr_max; i++) { nr = atomic_long_read(&child_info->nr_deferred[i]); atomic_long_add(nr, &parent_info->nr_deferred[i]); } @@ -899,7 +900,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (unlikely(!info)) goto unlock; - for_each_set_bit(i, info->map, shrinker_nr_max) { + for_each_set_bit(i, info->map, info->map_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid,