From patchwork Sun Feb 26 14:46:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13152422 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D976C64ED6 for ; Sun, 26 Feb 2023 14:48:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6866B6B0073; Sun, 26 Feb 2023 09:48:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6364B6B0074; Sun, 26 Feb 2023 09:48:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D67E6B0075; Sun, 26 Feb 2023 09:48:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3FA846B0073 for ; Sun, 26 Feb 2023 09:48:19 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E3BC51403C3 for ; Sun, 26 Feb 2023 14:48:18 +0000 (UTC) X-FDA: 80509723476.10.1C0679B Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by imf09.hostedemail.com (Postfix) with ESMTP id 18B4314000D for ; Sun, 26 Feb 2023 14:48:16 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=OEQgF8XF; spf=pass (imf09.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677422897; a=rsa-sha256; cv=none; b=s8kPeYpcBRk/H8Ubtn/lZfcfnLmNzpBhHMuMjXdnNVQYZ1k3Nv6XqvcRCsZ+YRxI9ejV3p 3Qg68QTGCUhUIPj9Mu87/T45NUWUkmmiN3O3mNFFdvo7AmhrDEUXaockXrRZLV71eVgcLW O1YAYO3EXfBDftz5a57NNEPdPGLk4Mw= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=OEQgF8XF; spf=pass (imf09.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677422897; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jZz7Wrj6RpcNJkd0HlCRkRoarWFliGItBtXoMhl1VyQ=; b=mGPxFrXdwWfn3woPO9P7Og9LskcfzNs6Sc8xXAf02DPpCoB2oQZ5k+DTP0RpWBY5CIWAjn nMShLWJFi5fzH/44xEHtqxUS+YsNRe+4S44YEv3GWnwOZdjNrZDbPMx6cCtGNomOGf6i5V Ac3zc/Th60HNNWeei1ALPOuFIXr6Vys= Received: by mail-pj1-f44.google.com with SMTP id m8-20020a17090a4d8800b002377bced051so7612895pjh.0 for ; Sun, 26 Feb 2023 06:48:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jZz7Wrj6RpcNJkd0HlCRkRoarWFliGItBtXoMhl1VyQ=; b=OEQgF8XFhS1YnbfOQOJh1IFogO5l8aVUQrCwR9nyy7oN1K9Ji93rp0LItnd8h4XE6q oXvlgHNvYZY7Sru38EHw5XpbSsjVDCpErYLQOtN774iKL22Mm3meSyNn/DoDpseqe7VK M148czmxLrasp9EktCEGPZBaRaJCIL/h+oxW/Len8VuNVSZLJgVQPTgQZl2H0oXCxaKT th31uEB6vb+9QTQp2lgTemj0Wo5Eo3476YMBJNBoEXl1eXjK1lRElSTkvEo/pHe8O8zz kKJciJ2YT4YMKnGfyaACr4ey76nUPX2Xt475p0Ss+/Ot3sZ5uU3wsAqqCutxvYDzlxUi SM6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jZz7Wrj6RpcNJkd0HlCRkRoarWFliGItBtXoMhl1VyQ=; b=Zf+mSkUQWA1xrsSW77UqmIulXV8hkZnw0hza/DbYrYESeOCahzpl/kYjLNsErBt2OY qUBxoktfWZxpUyuoNxu4RuofvLuBkLKTzhsP+ZmPljBCSj+Mpl4BigFERDK5VFiK2W04 fPF7C5gRHNWGWQZbNcUKel6bkbOEc47UivQ8qpbXWKFaDaTJmqvOZuYwGlKIe7N4yih1 r4rnXZMDckGQa1AHatKKaxtNskp0/NZWIXCrfjuEMfbw0FkP8nw4oxIcQSlgn8h1F/F3 ff9Ju6+ihJsBCvDG1JNhWL+FK9uDQbs/D0PLRTXM0iwTYj9mhRPLKi5CIxtva8BeGs+k su0w== X-Gm-Message-State: AO0yUKXvfcHDkZz/msihjMf5vDrsLLPNHR5tVnPN6smECN89EILRaJqL z1KIAHx0g/Mw996rpWLAU5mR8A== X-Google-Smtp-Source: AK7set+gayem3mXhkOe0NYaFyVrW/tJ1qNYdSQjiJZ8SAHMh4YRVx23yElXFyFWK9z6fB2W3nd2NCA== X-Received: by 2002:a17:902:d4c8:b0:19a:7217:32af with SMTP id o8-20020a170902d4c800b0019a721732afmr23248669plg.5.1677422896084; Sun, 26 Feb 2023 06:48:16 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:15 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 1/8] mm: vmscan: add a map_nr_max field to shrinker_info Date: Sun, 26 Feb 2023 22:46:48 +0800 Message-Id: <20230226144655.79778-2-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 18B4314000D X-Rspamd-Server: rspam01 X-Stat-Signature: k9o8p15k7b9a9zedwqn63969sajg3rcc X-HE-Tag: 1677422896-220529 X-HE-Meta: U2FsdGVkX19bMJOdpEJR6tj46x+e6Fm9gfNz96OwbrKjE13wQJunl4PcDANIz8cBI/YVteq+TB2PHNGDN2Gx7Hc+VlKTSDa9zjrqLhrNEBms0RTQftgzo7Myl7pWQsJ6BrtPsaamWghZbA/UBacW54jPa15xHm5PKlppEcH3FoCyKXFNVtiRU8+dPsLSgcGAtwimpC6heJ4c6SDYO2dOPaEGQQoCjMSeSjivlkFoapbfJe1cAcJGn9CUs6lGS5Pbaaet7j+ESx8gr3BN1ymvY7h/vw1NaIlg3sV/zs/74NjhY0VNUnEOLMioU6203eQ8CSsWbcClkXjqvyKqDppwulk7ULZaO/IuEbHjIEqTI4CXY0PbEE+BoW4WZ0atNK2/CM8WCv/LcavqUzFHbmAZqz6FHrI8EVdWgjnIGJNWCckC+/3Xwy7ZLLBQ4ypCWpvAaGWoVWwzTmYh6q6or7ZfxCkiSS/QcpFuui45Hov8vf/HIxcN48ngfHu6YOwxx6Khnt7MUIiAZt+P592fumAgXQi0q28lLqCl84v12QrqF0B7L0Axruw7SdXK8HC33/swlfqsvWtN9O2KUOHCM7fT3x4HsKjeeua8RQs6EBg8PFdimQVg7QOFoPzHwNa1zcAAxfN1dp36s+m1bAaHghThQh/8hlgzMRZxjd7ae5j+R163HZU3RDMpZ65ntduzOaJWWsUbsU0OqjGmX+K06M6pgw4knuHa5HCGJJvhAtEszowNQbwKywvjmAY1HkR2gDc7IUWNeSC3yXprY9ZZpkNXt4tOfx6VU8O5WOCHIPIO3xuYSdXEllY9tc9olHzM8uQvvcZkEcSg5PYtF+diUBG8mBA+BGFIa1oj1cEegq6Bcl+02291CTT6BiSja5WQDemUZH/wHzJClLKn3dwSV8+vQnUOHCY9I4doLAVg5mZpX3XjjfkIIqQhpC3Vzdv4Vvcqc86dV210c/GX4bO4nU4 L/CAXYqs UVs+YtBCaxMUyIDifa+u4ByvTeS1k12vOz3iwnDwv9sxhy+oZuIfGpKkqZeLCSscadEkctl2CAbDL8vJTEYVlkKvynp4v1N7WhGZP8KtI2vapmngTvCC35b8D1XKNTzfPlSJqhrnj5DaFPXzZiqIZXk3KJhMITSGQ1eT1EOLjtW7DU99oODxIQe2kS3Gmq5xyBMgEXHsSEvzqHfmtzKQyEQOhGNzOCgtq9JSQskquXii0HP9LM9CoKMdn+hGUtyKJgdWYT4u7VH/MQIN6Cw51cT3gXxx8CvXQI9cLR6GVAZUb9FOPRBEgJO2zy3rMYw1fgfwfmlvcEdapjsHn1vjzmLqOBqvsa+419sTBt9I7omqz289bI7rLSSiTwAMer7P/pd8vAouqpZ7t1D0710G30TPfK5o/k7BLKkcixSbVXVT48hKuARoeUe2KlcQHRJgdCkdGaATIC8x6sZ9ssTe0ojONPqG8PWff+OUCuvkcog3ZUH4Uko/lrjvMXhHFYUlMBkC2hZ7dFzKpg7SYYtF8UvuDMg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To prepare for the subsequent lockless memcg slab shrink, add a map_nr_max field to struct shrinker_info to records its own real shrinker_nr_max. Suggested-by: Kirill Tkhai Signed-off-by: Qi Zheng --- include/linux/memcontrol.h | 1 + mm/vmscan.c | 41 ++++++++++++++++++++++---------------- 2 files changed, 25 insertions(+), 17 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6eda2ab205d..aa69ea98e2d8 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -97,6 +97,7 @@ struct shrinker_info { struct rcu_head rcu; atomic_long_t *nr_deferred; unsigned long *map; + int map_nr_max; }; struct lruvec_stats_percpu { diff --git a/mm/vmscan.c b/mm/vmscan.c index 9c1c5e8b24b8..546c07ccb3bc 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -224,9 +224,16 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, lockdep_is_held(&shrinker_rwsem)); } +static inline bool need_expand(int new_nr_max, int old_nr_max) +{ + return round_up(new_nr_max, BITS_PER_LONG) > + round_up(old_nr_max, BITS_PER_LONG); +} + static int expand_one_shrinker_info(struct mem_cgroup *memcg, int map_size, int defer_size, - int old_map_size, int old_defer_size) + int old_map_size, int old_defer_size, + int new_nr_max) { struct shrinker_info *new, *old; struct mem_cgroup_per_node *pn; @@ -240,12 +247,17 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, if (!old) return 0; + /* Already expanded this shrinker_info */ + if (!need_expand(new_nr_max, old->map_nr_max)) + return 0; + new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); if (!new) return -ENOMEM; new->nr_deferred = (atomic_long_t *)(new + 1); new->map = (void *)new->nr_deferred + defer_size; + new->map_nr_max = new_nr_max; /* map: set all old bits, clear all new bits */ memset(new->map, (int)0xff, old_map_size); @@ -295,6 +307,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) } info->nr_deferred = (atomic_long_t *)(info + 1); info->map = (void *)info->nr_deferred + defer_size; + info->map_nr_max = shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); @@ -302,23 +315,14 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) return ret; } -static inline bool need_expand(int nr_max) -{ - return round_up(nr_max, BITS_PER_LONG) > - round_up(shrinker_nr_max, BITS_PER_LONG); -} - static int expand_shrinker_info(int new_id) { int ret = 0; - int new_nr_max = new_id + 1; + int new_nr_max = round_up(new_id + 1, BITS_PER_LONG); int map_size, defer_size = 0; int old_map_size, old_defer_size = 0; struct mem_cgroup *memcg; - if (!need_expand(new_nr_max)) - goto out; - if (!root_mem_cgroup) goto out; @@ -332,7 +336,8 @@ static int expand_shrinker_info(int new_id) memcg = mem_cgroup_iter(NULL, NULL, NULL); do { ret = expand_one_shrinker_info(memcg, map_size, defer_size, - old_map_size, old_defer_size); + old_map_size, old_defer_size, + new_nr_max); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; @@ -352,9 +357,11 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) rcu_read_lock(); info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); - /* Pairs with smp mb in shrink_slab() */ - smp_mb__before_atomic(); - set_bit(shrinker_id, info->map); + if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) { + /* Pairs with smp mb in shrink_slab() */ + smp_mb__before_atomic(); + set_bit(shrinker_id, info->map); + } rcu_read_unlock(); } } @@ -432,7 +439,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); - for (i = 0; i < shrinker_nr_max; i++) { + for (i = 0; i < child_info->map_nr_max; i++) { nr = atomic_long_read(&child_info->nr_deferred[i]); atomic_long_add(nr, &parent_info->nr_deferred[i]); } @@ -899,7 +906,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (unlikely(!info)) goto unlock; - for_each_set_bit(i, info->map, shrinker_nr_max) { + for_each_set_bit(i, info->map, info->map_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid,