From patchwork Sun Feb 26 14:46:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13152422 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D976C64ED6 for ; Sun, 26 Feb 2023 14:48:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6866B6B0073; Sun, 26 Feb 2023 09:48:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6364B6B0074; Sun, 26 Feb 2023 09:48:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D67E6B0075; Sun, 26 Feb 2023 09:48:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3FA846B0073 for ; Sun, 26 Feb 2023 09:48:19 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E3BC51403C3 for ; Sun, 26 Feb 2023 14:48:18 +0000 (UTC) X-FDA: 80509723476.10.1C0679B Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by imf09.hostedemail.com (Postfix) with ESMTP id 18B4314000D for ; Sun, 26 Feb 2023 14:48:16 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=OEQgF8XF; spf=pass (imf09.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677422897; a=rsa-sha256; cv=none; b=s8kPeYpcBRk/H8Ubtn/lZfcfnLmNzpBhHMuMjXdnNVQYZ1k3Nv6XqvcRCsZ+YRxI9ejV3p 3Qg68QTGCUhUIPj9Mu87/T45NUWUkmmiN3O3mNFFdvo7AmhrDEUXaockXrRZLV71eVgcLW O1YAYO3EXfBDftz5a57NNEPdPGLk4Mw= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=OEQgF8XF; spf=pass (imf09.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677422897; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jZz7Wrj6RpcNJkd0HlCRkRoarWFliGItBtXoMhl1VyQ=; b=mGPxFrXdwWfn3woPO9P7Og9LskcfzNs6Sc8xXAf02DPpCoB2oQZ5k+DTP0RpWBY5CIWAjn nMShLWJFi5fzH/44xEHtqxUS+YsNRe+4S44YEv3GWnwOZdjNrZDbPMx6cCtGNomOGf6i5V Ac3zc/Th60HNNWeei1ALPOuFIXr6Vys= Received: by mail-pj1-f44.google.com with SMTP id m8-20020a17090a4d8800b002377bced051so7612895pjh.0 for ; Sun, 26 Feb 2023 06:48:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jZz7Wrj6RpcNJkd0HlCRkRoarWFliGItBtXoMhl1VyQ=; b=OEQgF8XFhS1YnbfOQOJh1IFogO5l8aVUQrCwR9nyy7oN1K9Ji93rp0LItnd8h4XE6q oXvlgHNvYZY7Sru38EHw5XpbSsjVDCpErYLQOtN774iKL22Mm3meSyNn/DoDpseqe7VK M148czmxLrasp9EktCEGPZBaRaJCIL/h+oxW/Len8VuNVSZLJgVQPTgQZl2H0oXCxaKT th31uEB6vb+9QTQp2lgTemj0Wo5Eo3476YMBJNBoEXl1eXjK1lRElSTkvEo/pHe8O8zz kKJciJ2YT4YMKnGfyaACr4ey76nUPX2Xt475p0Ss+/Ot3sZ5uU3wsAqqCutxvYDzlxUi SM6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jZz7Wrj6RpcNJkd0HlCRkRoarWFliGItBtXoMhl1VyQ=; b=Zf+mSkUQWA1xrsSW77UqmIulXV8hkZnw0hza/DbYrYESeOCahzpl/kYjLNsErBt2OY qUBxoktfWZxpUyuoNxu4RuofvLuBkLKTzhsP+ZmPljBCSj+Mpl4BigFERDK5VFiK2W04 fPF7C5gRHNWGWQZbNcUKel6bkbOEc47UivQ8qpbXWKFaDaTJmqvOZuYwGlKIe7N4yih1 r4rnXZMDckGQa1AHatKKaxtNskp0/NZWIXCrfjuEMfbw0FkP8nw4oxIcQSlgn8h1F/F3 ff9Ju6+ihJsBCvDG1JNhWL+FK9uDQbs/D0PLRTXM0iwTYj9mhRPLKi5CIxtva8BeGs+k su0w== X-Gm-Message-State: AO0yUKXvfcHDkZz/msihjMf5vDrsLLPNHR5tVnPN6smECN89EILRaJqL z1KIAHx0g/Mw996rpWLAU5mR8A== X-Google-Smtp-Source: AK7set+gayem3mXhkOe0NYaFyVrW/tJ1qNYdSQjiJZ8SAHMh4YRVx23yElXFyFWK9z6fB2W3nd2NCA== X-Received: by 2002:a17:902:d4c8:b0:19a:7217:32af with SMTP id o8-20020a170902d4c800b0019a721732afmr23248669plg.5.1677422896084; Sun, 26 Feb 2023 06:48:16 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:15 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 1/8] mm: vmscan: add a map_nr_max field to shrinker_info Date: Sun, 26 Feb 2023 22:46:48 +0800 Message-Id: <20230226144655.79778-2-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 18B4314000D X-Rspamd-Server: rspam01 X-Stat-Signature: k9o8p15k7b9a9zedwqn63969sajg3rcc X-HE-Tag: 1677422896-220529 X-HE-Meta: U2FsdGVkX19bMJOdpEJR6tj46x+e6Fm9gfNz96OwbrKjE13wQJunl4PcDANIz8cBI/YVteq+TB2PHNGDN2Gx7Hc+VlKTSDa9zjrqLhrNEBms0RTQftgzo7Myl7pWQsJ6BrtPsaamWghZbA/UBacW54jPa15xHm5PKlppEcH3FoCyKXFNVtiRU8+dPsLSgcGAtwimpC6heJ4c6SDYO2dOPaEGQQoCjMSeSjivlkFoapbfJe1cAcJGn9CUs6lGS5Pbaaet7j+ESx8gr3BN1ymvY7h/vw1NaIlg3sV/zs/74NjhY0VNUnEOLMioU6203eQ8CSsWbcClkXjqvyKqDppwulk7ULZaO/IuEbHjIEqTI4CXY0PbEE+BoW4WZ0atNK2/CM8WCv/LcavqUzFHbmAZqz6FHrI8EVdWgjnIGJNWCckC+/3Xwy7ZLLBQ4ypCWpvAaGWoVWwzTmYh6q6or7ZfxCkiSS/QcpFuui45Hov8vf/HIxcN48ngfHu6YOwxx6Khnt7MUIiAZt+P592fumAgXQi0q28lLqCl84v12QrqF0B7L0Axruw7SdXK8HC33/swlfqsvWtN9O2KUOHCM7fT3x4HsKjeeua8RQs6EBg8PFdimQVg7QOFoPzHwNa1zcAAxfN1dp36s+m1bAaHghThQh/8hlgzMRZxjd7ae5j+R163HZU3RDMpZ65ntduzOaJWWsUbsU0OqjGmX+K06M6pgw4knuHa5HCGJJvhAtEszowNQbwKywvjmAY1HkR2gDc7IUWNeSC3yXprY9ZZpkNXt4tOfx6VU8O5WOCHIPIO3xuYSdXEllY9tc9olHzM8uQvvcZkEcSg5PYtF+diUBG8mBA+BGFIa1oj1cEegq6Bcl+02291CTT6BiSja5WQDemUZH/wHzJClLKn3dwSV8+vQnUOHCY9I4doLAVg5mZpX3XjjfkIIqQhpC3Vzdv4Vvcqc86dV210c/GX4bO4nU4 L/CAXYqs UVs+YtBCaxMUyIDifa+u4ByvTeS1k12vOz3iwnDwv9sxhy+oZuIfGpKkqZeLCSscadEkctl2CAbDL8vJTEYVlkKvynp4v1N7WhGZP8KtI2vapmngTvCC35b8D1XKNTzfPlSJqhrnj5DaFPXzZiqIZXk3KJhMITSGQ1eT1EOLjtW7DU99oODxIQe2kS3Gmq5xyBMgEXHsSEvzqHfmtzKQyEQOhGNzOCgtq9JSQskquXii0HP9LM9CoKMdn+hGUtyKJgdWYT4u7VH/MQIN6Cw51cT3gXxx8CvXQI9cLR6GVAZUb9FOPRBEgJO2zy3rMYw1fgfwfmlvcEdapjsHn1vjzmLqOBqvsa+419sTBt9I7omqz289bI7rLSSiTwAMer7P/pd8vAouqpZ7t1D0710G30TPfK5o/k7BLKkcixSbVXVT48hKuARoeUe2KlcQHRJgdCkdGaATIC8x6sZ9ssTe0ojONPqG8PWff+OUCuvkcog3ZUH4Uko/lrjvMXhHFYUlMBkC2hZ7dFzKpg7SYYtF8UvuDMg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To prepare for the subsequent lockless memcg slab shrink, add a map_nr_max field to struct shrinker_info to records its own real shrinker_nr_max. Suggested-by: Kirill Tkhai Signed-off-by: Qi Zheng --- include/linux/memcontrol.h | 1 + mm/vmscan.c | 41 ++++++++++++++++++++++---------------- 2 files changed, 25 insertions(+), 17 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6eda2ab205d..aa69ea98e2d8 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -97,6 +97,7 @@ struct shrinker_info { struct rcu_head rcu; atomic_long_t *nr_deferred; unsigned long *map; + int map_nr_max; }; struct lruvec_stats_percpu { diff --git a/mm/vmscan.c b/mm/vmscan.c index 9c1c5e8b24b8..546c07ccb3bc 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -224,9 +224,16 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, lockdep_is_held(&shrinker_rwsem)); } +static inline bool need_expand(int new_nr_max, int old_nr_max) +{ + return round_up(new_nr_max, BITS_PER_LONG) > + round_up(old_nr_max, BITS_PER_LONG); +} + static int expand_one_shrinker_info(struct mem_cgroup *memcg, int map_size, int defer_size, - int old_map_size, int old_defer_size) + int old_map_size, int old_defer_size, + int new_nr_max) { struct shrinker_info *new, *old; struct mem_cgroup_per_node *pn; @@ -240,12 +247,17 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, if (!old) return 0; + /* Already expanded this shrinker_info */ + if (!need_expand(new_nr_max, old->map_nr_max)) + return 0; + new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); if (!new) return -ENOMEM; new->nr_deferred = (atomic_long_t *)(new + 1); new->map = (void *)new->nr_deferred + defer_size; + new->map_nr_max = new_nr_max; /* map: set all old bits, clear all new bits */ memset(new->map, (int)0xff, old_map_size); @@ -295,6 +307,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) } info->nr_deferred = (atomic_long_t *)(info + 1); info->map = (void *)info->nr_deferred + defer_size; + info->map_nr_max = shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); @@ -302,23 +315,14 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) return ret; } -static inline bool need_expand(int nr_max) -{ - return round_up(nr_max, BITS_PER_LONG) > - round_up(shrinker_nr_max, BITS_PER_LONG); -} - static int expand_shrinker_info(int new_id) { int ret = 0; - int new_nr_max = new_id + 1; + int new_nr_max = round_up(new_id + 1, BITS_PER_LONG); int map_size, defer_size = 0; int old_map_size, old_defer_size = 0; struct mem_cgroup *memcg; - if (!need_expand(new_nr_max)) - goto out; - if (!root_mem_cgroup) goto out; @@ -332,7 +336,8 @@ static int expand_shrinker_info(int new_id) memcg = mem_cgroup_iter(NULL, NULL, NULL); do { ret = expand_one_shrinker_info(memcg, map_size, defer_size, - old_map_size, old_defer_size); + old_map_size, old_defer_size, + new_nr_max); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; @@ -352,9 +357,11 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) rcu_read_lock(); info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); - /* Pairs with smp mb in shrink_slab() */ - smp_mb__before_atomic(); - set_bit(shrinker_id, info->map); + if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) { + /* Pairs with smp mb in shrink_slab() */ + smp_mb__before_atomic(); + set_bit(shrinker_id, info->map); + } rcu_read_unlock(); } } @@ -432,7 +439,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); - for (i = 0; i < shrinker_nr_max; i++) { + for (i = 0; i < child_info->map_nr_max; i++) { nr = atomic_long_read(&child_info->nr_deferred[i]); atomic_long_add(nr, &parent_info->nr_deferred[i]); } @@ -899,7 +906,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (unlikely(!info)) goto unlock; - for_each_set_bit(i, info->map, shrinker_nr_max) { + for_each_set_bit(i, info->map, info->map_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, From patchwork Sun Feb 26 14:46:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13152423 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1F9DC7EE23 for ; Sun, 26 Feb 2023 14:48:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 833826B0074; Sun, 26 Feb 2023 09:48:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7E3146B0075; Sun, 26 Feb 2023 09:48:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6ABB36B0078; Sun, 26 Feb 2023 09:48:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 5C02F6B0074 for ; Sun, 26 Feb 2023 09:48:25 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 24E44401FB for ; Sun, 26 Feb 2023 14:48:25 +0000 (UTC) X-FDA: 80509723770.28.347063C Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf12.hostedemail.com (Postfix) with ESMTP id 4E8924001A for ; Sun, 26 Feb 2023 14:48:23 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ALVQXXOf; spf=pass (imf12.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677422903; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xJVL9LTM3YR7R2xJV6pOC5vrFajMNWUdaUGoIRklN4g=; b=uk+dBilwpsiKPQEjwSK+l0z3d9TGc8NzPXlHJZmb86ARJeuSOc4l7KXVT2eeYzuw9J5/AP EYOCzHD0mpNnOewNQuzX18SFeEiJDP3mZWnnm+q6Is+Azfysoiru6R2Rn2ktQdEXbv40lp +O8M2tPU5+TxfeETXxFp48ZchNEuuj4= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ALVQXXOf; spf=pass (imf12.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677422903; a=rsa-sha256; cv=none; b=P4OPlvQNZ4g8mJHv4lhY31C+dFI9/xvG49/D4QAaOApqrdjj4ywqaQJbwSXPihVWzKJ9Lm rUZs2b3ro8sh1QMRknsaTS1B9MJxHFWEAY6cwAlI/Vv1RFT/+8uHY1kldVCJhsZv0RtLzS +y0sm1ncrooRJh7qoY6wTZoUi11Scm8= Received: by mail-pl1-f180.google.com with SMTP id u5so805788plq.7 for ; Sun, 26 Feb 2023 06:48:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1677422902; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xJVL9LTM3YR7R2xJV6pOC5vrFajMNWUdaUGoIRklN4g=; b=ALVQXXOfu8Q3gVhZC/7gcT/As/x0PQNyX2nqdY6UDPAFzmCXukW/onK5TzeE9pcwvk vXJzQkiBP+sXbTYXsw+HpB7JRRbrwmsjkOEq5DQ1S1Eac1N0S/0/DsgoqOjYjHQYnPSR P4R9Pfo0jggo+dcJDr8XJLDD9doWZVqghSIkea6uqcKjvoOktQF/G7VDIKPmpqIX0uaq CUp5L8HF8li82mResfb29HBRedCwzzg7BH3UMRf7q7wMBQ/7+ybQtKL36gQDJZu0gwHP FCqtYPLddEJNMwfVTwEbd49hBEqsrqinMleX5XcFcj1s7BFSCNs7Wfig3bKPX0H4mz4V BKmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677422902; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xJVL9LTM3YR7R2xJV6pOC5vrFajMNWUdaUGoIRklN4g=; b=J76qHeRUx7h7EkuagN5iYbfDAaBdoFgPeODGJCb8VZ8gMUABMhxCF3TL708Tkty+ur hQCEMQJNHWIBDYSUX/b7RoqgQN0c+IMXf4XHFTtPLjoSCJRVWktL1HBhEVA0QM6hrHwt E4I0sPlmmbSvRnglGEezU7uhMWm1ViMKbdTNzZ5OX52MFShDx4fW6AyfnpWDE+I/uhgT T6hRG5T17eacsDe9rUzZCaYUiqWzHLhyBbhe0T1Oz4vuAELDexBGjTyVZHbtU1IQZdb2 aCwfB81U+7uotX/qPjlbbACT7LuCDX0Nf5RtqGXZqvbKmeHbi0CTDCqdHe6GC+cYkwpg 42JA== X-Gm-Message-State: AO0yUKXqX+ie9swIq0E1uumLt8GseFV+aBhSNWLV/METMR594CSyqNFE zcr76oZUV0IhWKswRNX/7lozkQ== X-Google-Smtp-Source: AK7set+bbch+q47tvYRNxM7CkTvsssKUNlySDXHnmC0iMbwnLjHUXiyS9TSOg6OMzWfUw8IIF2O97w== X-Received: by 2002:a17:902:e84b:b0:19a:7439:3e98 with SMTP id t11-20020a170902e84b00b0019a74393e98mr23301977plg.4.1677422901796; Sun, 26 Feb 2023 06:48:21 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:21 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 2/8] mm: vmscan: make global slab shrink lockless Date: Sun, 26 Feb 2023 22:46:49 +0800 Message-Id: <20230226144655.79778-3-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 4E8924001A X-Stat-Signature: owrgpr1xhyin16iod6wow8fgx4amfymr X-HE-Tag: 1677422903-117693 X-HE-Meta: U2FsdGVkX19oW6pdF7b0a3FNefYXQzbj5oFkJeyWsn+16YhjpZkWCWKlX3OWs72JOFr9drwJwIdUpBrOP5NqBVYIe9ZyufTZTOj/WOfxRfPIxfhm/H6O0gX3fcgivNshgwaapRrontWoj4ofQxzd8Sk1X9W2GuO3xXGyJJKJql81QZOU+fgKAwB+z//ChAFpuVu0HtdJ6oA6Fbp20Aqyul4uLgFGyEjDVcvICUQTZSL1eRho51LqNIJgOygPxtExJ1W3647w21jLg5N17g6x/JaWOZvs+8DEP/W4tYMPXXqKKXHj7DaE8RBacW0R6JoL+CvuBLDEHBSS98pv4BS16DFa61E0DJP28GOzKmnpSJkNTgpd9c8baYfAq1UDJeAL+qxwhQr5lvHSQdOVze/e/cKPRlFf/e7hOLf4q362cjmEaEpTx1t/UHzs1DdJabhfBiny+M37+OMiAK6Smo7PQLOlIHYMHFDFzP5RlizJ9sj67i8lvK8T04iNwaZDUxZzu9hw2VFtwm0KjHCUeIV1+EYzDMxWbs/01Ii+5Df64Eg6kmKdLloDEofFcprqgPackh7k8Pi6P25bCLW5oOW56f98rYmiNbfklc8W5a0hFQkXJXVj0/aaNKuz+sJ1jZbdBmwGtV4YKROzOrU0tYNKc8pxhn12IQgfN7DDrOaeJzoiLnVPh0HQYJGylHHlEK8OZisJUnsVTac9BYH7YOJ72s1TZ5ss1Jg4O4rDmww8w97i5gWI0VDs9P3s0Tr/PKSdYByBlfDtjMtIDtlYExOwZjK6AYBuzPEDDsbbZOxw/sJ0tPn+OM0P6pLttOT2XfOGJK2Te5S6VyVsAIi0jXfnAuuh30MNJKaZARClwkVez0uRp6lMJLw+6WlPqI5aNE80ds33S3qdCfqfH8DptEmslhYsNxkFjsC62bAgwZTOJpuMwL2WqHEsAlNiSBD22kJF3bYwpszp8MJt+kKYPLi 811m5ojl Tz3AElOn2haAivi30KfvraSMEi0ZKo+L9TmrrUxoFbmyjINWb7nKBBWtt8albpBIzK0yeGo9DLwsAJe871yQ+3cZYK9mJ9ZzMgtZ/Q2EUNb8dqsPjFLob5L2XUCLiXN553WFoZzS+OCDKI4DG0+fV2frYfq9Nk/6L3w7LKf+2QL/DZsN38UhF+ZbKnZTBlRxrhwTkfqakItm49DlR7+svX2B3leAgDo6Ioxe2TlsWQcL3zJW5/wsqlJwpa7BCHnizvlkN4s1I3MU6kCQoqYWxYP3bcJU4KoVqGyL/mTXpL8pNiSl+T2MNaathMrVigzUl6OC5Sif5JL64e1Tbznillmpd1NSRU4QqYuY2MOQVGOC9o3+yQqJ3kz5e90z4Fo/1xN2XLagU7bI8f3XUZ5JRJXw/pFjzzhn9cg2GjVTvdlNw2M0P4VVM/oHES0UqAVMZw8DqcML5bNE6qO0c+maX1QS2cdcMF8uPaP4Ff8r5ZhAlUUY78aVewLRjm2SZd5stZdOBrbKRI8hqTf8K7MvL9G5HjIYJZ4eA0jTohr3R7vrQpdBzaW6UJT8tAsnX8kR5ZeHrPlJEpOfuf1UzA5NmjQkWdngqYZ4hQpmH7YP7VyJgtohLef9GC9ZQlnvhr2Yy0OyafUcxfpDhcuyUwS6nkmM5fm8coa00dQ+8 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The shrinker_rwsem is a global lock in shrinkers subsystem, it is easy to cause blocking in the following cases: a. the write lock of shrinker_rwsem was held for too long. For example, there are many memcgs in the system, which causes some paths to hold locks and traverse it for too long. (e.g. expand_shrinker_info()) b. the read lock of shrinker_rwsem was held for too long, and a writer came at this time. Then this writer will be forced to wait and block all subsequent readers. For example: - be scheduled when the read lock of shrinker_rwsem is held in do_shrink_slab() - some shrinker are blocked for too long. Like the case mentioned in the patchset[1]. Therefore, many times in history ([2],[3],[4],[5]), some people wanted to replace shrinker_rwsem reader with SRCU, but they all gave up because SRCU was not unconditionally enabled. But now, since commit 1cd0bd06093c ("rcu: Remove CONFIG_SRCU"), the SRCU is unconditionally enabled. So it's time to use SRCU to protect readers who previously held shrinker_rwsem. [1]. https://lore.kernel.org/lkml/20191129214541.3110-1-ptikhomirov@virtuozzo.com/ [2]. https://lore.kernel.org/all/1437080113.3596.2.camel@stgolabs.net/ [3]. https://lore.kernel.org/lkml/1510609063-3327-1-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp/ [4]. https://lore.kernel.org/lkml/153365347929.19074.12509495712735843805.stgit@localhost.localdomain/ [5]. https://lore.kernel.org/lkml/20210927074823.5825-1-sultan@kerneltoast.com/ Signed-off-by: Qi Zheng --- mm/vmscan.c | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 546c07ccb3bc..2a21a84d3db1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -202,6 +202,7 @@ static void set_task_reclaim_state(struct task_struct *task, LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); +DEFINE_SRCU(shrinker_srcu); #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -706,7 +707,7 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { down_write(&shrinker_rwsem); - list_add_tail(&shrinker->list, &shrinker_list); + list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |= SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); up_write(&shrinker_rwsem); @@ -760,13 +761,15 @@ void unregister_shrinker(struct shrinker *shrinker) return; down_write(&shrinker_rwsem); - list_del(&shrinker->list); + list_del_rcu(&shrinker->list); shrinker->flags &= ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry = shrinker_debugfs_remove(shrinker); up_write(&shrinker_rwsem); + synchronize_srcu(&shrinker_srcu); + debugfs_remove_recursive(debugfs_entry); kfree(shrinker->nr_deferred); @@ -786,6 +789,7 @@ void synchronize_shrinkers(void) { down_write(&shrinker_rwsem); up_write(&shrinker_rwsem); + synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); @@ -996,6 +1000,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, { unsigned long ret, freed = 0; struct shrinker *shrinker; + int srcu_idx; /* * The root memcg might be allocated even though memcg is disabled @@ -1007,10 +1012,10 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (!mem_cgroup_disabled() && !mem_cgroup_is_root(memcg)) return shrink_slab_memcg(gfp_mask, nid, memcg, priority); - if (!down_read_trylock(&shrinker_rwsem)) - goto out; + srcu_idx = srcu_read_lock(&shrinker_srcu); - list_for_each_entry(shrinker, &shrinker_list, list) { + list_for_each_entry_srcu(shrinker, &shrinker_list, list, + srcu_read_lock_held(&shrinker_srcu)) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -1021,19 +1026,9 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; freed += ret; - /* - * Bail out if someone want to register a new shrinker to - * prevent the registration from being stalled for long periods - * by parallel ongoing shrinking. - */ - if (rwsem_is_contended(&shrinker_rwsem)) { - freed = freed ? : 1; - break; - } } - up_read(&shrinker_rwsem); -out: + srcu_read_unlock(&shrinker_srcu, srcu_idx); cond_resched(); return freed; } From patchwork Sun Feb 26 14:46:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13152424 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 725AEC64ED6 for ; Sun, 26 Feb 2023 14:48:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C78E96B0075; Sun, 26 Feb 2023 09:48:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C29076B0078; Sun, 26 Feb 2023 09:48:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AF18E6B007B; Sun, 26 Feb 2023 09:48:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 9F5176B0075 for ; Sun, 26 Feb 2023 09:48:31 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 7DC41A0251 for ; Sun, 26 Feb 2023 14:48:30 +0000 (UTC) X-FDA: 80509723980.21.EA8DD23 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf16.hostedemail.com (Postfix) with ESMTP id B6A0018000B for ; Sun, 26 Feb 2023 14:48:28 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=LzQ+kcdg; spf=pass (imf16.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677422908; a=rsa-sha256; cv=none; b=n1o5/SzSOD1caF4pifW4Z9QaIXZjxpqFiutuQCl0D064fzZAlTY6gw0JhrcOA6cIlSQysX uNKmgcFx9TkSpkAefbksxAFUymGnQTmUu6wo6UTtAvZDP9EAEnNdlUUco5q9NYKN+AUVxO spTQe3pTb93MZ3Qxsx2zDRmJqgXij8Q= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=LzQ+kcdg; spf=pass (imf16.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677422908; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gvRso6AuWsz0X9JCAZwWH9GIT0TN5B8gocPlJ++Lup8=; b=0T/49CSlavG5e4Fpj3YL3pY4pOpVV1KFK1Z9G+p1ZQ0PKSRNG8uP2ibVw1PS0ZiML5hLy2 AmREJjVvbxklP/I8D5jgLL3o9qt5SYQdS+6DlFbHrLNXH+FN04qrb9NGlbIZq5hZ6q1EFK OmskC2mID8nZ6OMGVaFPnAk1sz6tAhY= Received: by mail-pl1-f179.google.com with SMTP id i10so4189949plr.9 for ; Sun, 26 Feb 2023 06:48:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gvRso6AuWsz0X9JCAZwWH9GIT0TN5B8gocPlJ++Lup8=; b=LzQ+kcdgxQjvYV6/aBTUiIdXty1SduBzej96ViyOvdlzB3Fsg4Lcdgbz3j4hmKO+8+ TIcLX9NYqDrIQ5/ezPI3RaFfYwQ1ZEDeoEpqbn5P8z2NLz+savfhhRBcSPVq7BwrsDXr R1O+OCmiRS291b9X4XNwI5b5W0qVttXgY3Oqixpog+eVUFyly+Rm9YGT9YU8Ro/3F5Pw d/nQ47A2O7bUl+GBn4gAU0PulKhyQ7Bu/CfMAW5zYIkO2kY76GczKehRR3kCEzOW262q ug1kCR6iRSBx0q+cNBVn+mB+nddqScXB+yCOQ55hN5exNnBS/N7LGAnn4tefuskRqhyI gJZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gvRso6AuWsz0X9JCAZwWH9GIT0TN5B8gocPlJ++Lup8=; b=2GgtU9TCgqsIkdQj5+4R3ToCt3zU9lx+4jm13fHlP2tk4T71o6SHj73ipat88Jdpj9 lcS+UJZEUTBVVzPZO8Wzj1J7+2kCKDpwR6oyxtLaROTzPFvYPZRDG1c8+u4GgYfSWfOS 3ilRx5W9rvr+Hjz3wCtoy+mcR7rRUkZBwzsd0M0HMrQ0RCkOM6mSFMjdso3S5Jq4cRm/ 7zrWFihXv9gp5ZQORZCYFTa0VMf4MvZyoLRRsNthdXAj4UzhYBJk8KdYDQlswW5JAC0d AQM268SJIDlhGa/Zt5hUN2YluOYBtnZlUa6n6kgEI8c/WTp1IvCRiDLiNKZAWUiKoQxA Kt+g== X-Gm-Message-State: AO0yUKW4uSVzj2PtzCpaGE8ETRsIJQEfUAvxIbhZ24jo+Ww6im6CI1+w A/sEQYGvwm0pRxf9oVOZrR6sHQ== X-Google-Smtp-Source: AK7set8spgroQQIMXk8gdX51DNtELr9O76gOJSC+OLLzgfCw5s0Vzk98NSKahnR1+3e2KLwZ3nhiFQ== X-Received: by 2002:a17:903:230f:b0:19a:7060:948 with SMTP id d15-20020a170903230f00b0019a70600948mr21481855plh.1.1677422907679; Sun, 26 Feb 2023 06:48:27 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:27 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 3/8] mm: vmscan: make memcg slab shrink lockless Date: Sun, 26 Feb 2023 22:46:50 +0800 Message-Id: <20230226144655.79778-4-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: B6A0018000B X-Rspamd-Server: rspam01 X-Stat-Signature: t7hcsaeyy3b1ys6dqt4io149pbsxrok9 X-HE-Tag: 1677422908-156519 X-HE-Meta: U2FsdGVkX1+yV6cBmh6jp0ppyNNvJuVOvx8ixVDwz2XylhaJk6FBz9aLYb3Q2YWR8Yg0J6+XYvQ3hGeigTSGsKicKkj9vQnhn1K8j/EIwGioqoWAC/W48aNLsDegdOkewMKmFaQPcHdNqE0xntgYXGPg0AwEujyOyg+QKHPGIT/ot1FRbuNloe8BWFkvtRqr1ZLO6M8X2fDyzVppVZdpVKcZPTBXXSSpLrDYBAVuRq2X6Bi/5EQqFtOcQRcZysKsm/JHqgRebA4hU2UlZ/F/oDz7cbTfH6maCmlcOKyrNkv5kXHY8PN/+J7S12DIfvwoDrJ4ydfBpF/UKkD78lTTbDb7ldlhGA3w/4ni8G1dAEGRaVe/LSTCjVSJPYVUPEMcCtH9yJdLXE0h4fFWE9dBZNTHfwoVy6aEKaJu1DmMaVgbotN4VF+tIcuw8NkRvGyymhinalRXVxzkhi36XPKj5cCkoLcALXSMXxWITBap1Igoz32IOsqpDyQppoHFV7DfwcG3Mz1cfcrkKec5Rno/qv64/PqsRm8/9nwY+iUH/TTeUZ5rQyJl8510nWnoq1wFif/sQzZx1PJqOqf7yGnmVDFPe2B7mWZZjUUUAC7+IFa69SSqCMAXPcXwMBbqTZSK96hRusk0rADBzyPkUfJwWoSX+u4f+SkAXLNKn36UyDyabFbKy2EBN8N8cOfm4sZ8OE9akdNZoaNffdJsiuMECeqDwjVI3YyTvVrXqeLTWSQ3Ejt+cdUT5qbcdwidKpsq+jVYnQ4zeUIgxzwiV8/CGieOgBRzd677IygaW3+gSIjOfmf8i2NVUHi6OtKixpymQDOohiz+l4dxOrn6oP38wh9x0HDpk1rYew1hDsaFt/Amht8xt2A75Nv3n/cyzLSMW1U2Zd4WyIeM8cGyhoYjtw63sv11SV42k+cAxa/+/NK3O+EKDxIDO8bu93vPmO49R7BoorOv2VMauIWKhy2 KEbkymNW epg6g9X/6DBYH3pf6xwVMZUknSPEyh1NSu+jz+PxUpbkYrtxmNDQ5lmSWHa95Ozs1IADnlccMw7Q9er7seaQQvGj1+3KBGlwUGZ7Gokj18xm4uDJHHbuozvn7sH0WnTyXPtRY3nGNj2+iCL4/a44E4d4golktFyiyvBb04uioC+hpmsxXnP7siUD+ceH4eTroJTGCQ05CzSXN8DWx9PItNkG8Nnyhucw6BR6lLdr2iuaURi29R8niapmTuPE1OXqkXVOEvdqmzsi0Q3KZyzbOyGr8VM8+dGg1/z9NtbFltY0vNyYe5tiLuDbilIK7U7cwML9ydndbFz4tCWQTfvd5PjgMOZlg6Auo1c8GnuJA4Bfnk7WhXGY2aObpsarw/JL6342KgxkAFPxobONC8sw9yNPfZiFBHplq45gDWrzV5sQmYtI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Like global slab shrink, since commit 1cd0bd06093c ("rcu: Remove CONFIG_SRCU"), it's time to use SRCU to protect readers who previously held shrinker_rwsem. We can test with the following script: ``` DIR="/root/shrinker/memcg/mnt" do_create() { mkdir /sys/fs/cgroup/memory/test echo 200M > /sys/fs/cgroup/memory/test/memory.limit_in_bytes for i in `seq 0 $1`; do mkdir /sys/fs/cgroup/memory/test/$i; echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; mkdir -p $DIR/$i; done } do_mount() { for i in `seq $1 $2`; do mount -t tmpfs $i $DIR/$i; done } do_touch() { for i in `seq $1 $2`; do echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; dd if=/dev/zero of=$DIR/$i/file$i bs=1M count=1 & done } do_create 2000 do_mount 0 2000 do_touch 0 1000 ``` Before applying: 46.60% [kernel] [k] down_read_trylock 18.70% [kernel] [k] up_read 15.44% [kernel] [k] shrink_slab 4.37% [kernel] [k] _find_next_bit 2.75% [kernel] [k] xa_load 2.07% [kernel] [k] idr_find 1.73% [kernel] [k] do_shrink_slab 1.42% [kernel] [k] shrink_lruvec 0.74% [kernel] [k] shrink_node 0.60% [kernel] [k] list_lru_count_one After applying: 19.53% [kernel] [k] _find_next_bit 14.63% [kernel] [k] do_shrink_slab 14.58% [kernel] [k] shrink_slab 11.83% [kernel] [k] shrink_lruvec 9.33% [kernel] [k] __blk_flush_plug 6.67% [kernel] [k] mem_cgroup_iter 3.73% [kernel] [k] list_lru_count_one 2.43% [kernel] [k] shrink_node 1.96% [kernel] [k] super_cache_count 1.78% [kernel] [k] __rcu_read_unlock 1.38% [kernel] [k] __srcu_read_lock 1.30% [kernel] [k] xas_descend We can see that the readers is no longer blocked. Signed-off-by: Qi Zheng --- mm/vmscan.c | 46 +++++++++++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 19 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 2a21a84d3db1..490764f8e085 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include @@ -221,8 +222,21 @@ static inline int shrinker_defer_size(int nr_items) static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, int nid) { - return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, - lockdep_is_held(&shrinker_rwsem)); + return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu, + lockdep_is_held(&shrinker_rwsem)); +} + +static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, + int nid) +{ + return srcu_dereference(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu); +} + +static void free_shrinker_info_rcu(struct rcu_head *head) +{ + kvfree(container_of(head, struct shrinker_info, rcu)); } static inline bool need_expand(int new_nr_max, int old_nr_max) @@ -269,7 +283,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, defer_size - old_defer_size); rcu_assign_pointer(pn->shrinker_info, new); - kvfree_rcu(old, rcu); + call_srcu(&shrinker_srcu, &old->rcu, free_shrinker_info_rcu); } return 0; @@ -355,15 +369,16 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) { if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { struct shrinker_info *info; + int srcu_idx; - rcu_read_lock(); - info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); + srcu_idx = srcu_read_lock(&shrinker_srcu); + info = shrinker_info_srcu(memcg, nid); if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) { /* Pairs with smp mb in shrink_slab() */ smp_mb__before_atomic(); set_bit(shrinker_id, info->map); } - rcu_read_unlock(); + srcu_read_unlock(&shrinker_srcu, srcu_idx); } } @@ -377,7 +392,6 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) return -ENOSYS; down_write(&shrinker_rwsem); - /* This may call shrinker, so it must use down_read_trylock() */ id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -411,7 +425,7 @@ static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, { struct shrinker_info *info; - info = shrinker_info_protected(memcg, nid); + info = shrinker_info_srcu(memcg, nid); return atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); } @@ -420,7 +434,7 @@ static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, { struct shrinker_info *info; - info = shrinker_info_protected(memcg, nid); + info = shrinker_info_srcu(memcg, nid); return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); } @@ -898,15 +912,14 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, { struct shrinker_info *info; unsigned long ret, freed = 0; + int srcu_idx; int i; if (!mem_cgroup_online(memcg)) return 0; - if (!down_read_trylock(&shrinker_rwsem)) - return 0; - - info = shrinker_info_protected(memcg, nid); + srcu_idx = srcu_read_lock(&shrinker_srcu); + info = shrinker_info_srcu(memcg, nid); if (unlikely(!info)) goto unlock; @@ -956,14 +969,9 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, set_shrinker_bit(memcg, nid, i); } freed += ret; - - if (rwsem_is_contended(&shrinker_rwsem)) { - freed = freed ? : 1; - break; - } } unlock: - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); return freed; } #else /* CONFIG_MEMCG */ From patchwork Sun Feb 26 14:46:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13152425 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C2D8C7EE2F for ; Sun, 26 Feb 2023 14:48:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC73B6B0078; Sun, 26 Feb 2023 09:48:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C77A16B007B; Sun, 26 Feb 2023 09:48:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B18336B007D; Sun, 26 Feb 2023 09:48:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A38256B0078 for ; Sun, 26 Feb 2023 09:48:36 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8979CC02A1 for ; Sun, 26 Feb 2023 14:48:36 +0000 (UTC) X-FDA: 80509724232.13.E739144 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf16.hostedemail.com (Postfix) with ESMTP id AA35C18000B for ; Sun, 26 Feb 2023 14:48:34 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=CsO5aMnV; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf16.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677422914; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zCpviF+GWpW2q6LnTDjAWSvloYoZWztn7l/3EVg4xkQ=; b=7jpWM81KKdpNNv4x1HcMu/aRxbM77GRAxWm7K7x6N5rv/RJgadcNsN/0M827ple9YRISay EqsKrA6e7/5mGlKDL0JsL2m72AToFo/5hUsTqiQvnxEWbPJ9gD/sAsL2ja/kEb/X5rkvEv mUy8cZDC2pXAJrE5QJkGQHn7QcdQKpU= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=CsO5aMnV; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf16.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677422914; a=rsa-sha256; cv=none; b=3WNag/o5Hf2WyK2T9djFSGuIU0oHwRLNZIYMDG2BaTKNH5p7ccuksrgs1An/3l6t8juJp8 3x0h36X2t89f/72ht9oG31AhOFiJzW6nYmTE2mm60Y0/hgArsQ9G1RnSFVTv/SiBzTayD0 MZ/V99ZkyQx6qxXds2NYcQhrJc3yhsc= Received: by mail-pl1-f174.google.com with SMTP id v11so680843plz.8 for ; Sun, 26 Feb 2023 06:48:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zCpviF+GWpW2q6LnTDjAWSvloYoZWztn7l/3EVg4xkQ=; b=CsO5aMnVYOQmnyjcQDWffm4ibQqMe8oamD5qA31APfwGf7gzHtJWrrI9olR/qCLH3m +Dg9XEOXL03hNqlNAICq5ya5pc+4BAUaxlKS89rulXGKJbAvILD5lY4IpjVkqVKRPLvh 40gqi3qq8AOZ1mGSmaZCdTm8hikTcJ9NRlhGcNGYHgwL8+XrC0lLnzZo9UmwdvnTP54e eUYL2ktptCc5CxNqWcNGg2KrPSG53OlDew6ZxZjtTpBl8B2IbscuG/h4GalyWHIQrMzw 3OBbkZBqaEHnKb1ujMBfyfX5Hxby7rTMu2I2Ehi1U62/G1skp5O5yL+5jSowBJMa0FAX IEvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zCpviF+GWpW2q6LnTDjAWSvloYoZWztn7l/3EVg4xkQ=; b=SckMsIKK+qF6YKg8O0YQKfudtNxKfDPhrIXWHcgYTnx/WnoGPEL4sJY5A8rki+k5WG MBadb1JAaEe+0xM7dwo0uQ1ihXD3d+tyio3E2iP6giBhalWpee8c8IqGCOBw3uBAAiIK oaGYdQvoZi2tzbv0AfujQ7NJ6wXtoVbZajJlBzQ+qgcNdxZGovXTVCbQ6oOfCcRaWE4I 346UG3mDrT0vd7U3Bb3CHt+DIn7bOYdc6DXIuXi3ciduudt5aaW1ls1U27jKH7cPa3fi JnRTdlpAr+JuepYN0o5qRUThIIOa1pYajx8ZWxTv/kYHeo89ryeWAms+KUyCgs7gptFu 6qHQ== X-Gm-Message-State: AO0yUKWyiK5uAHjcENofaAmqYqi5JrjDqCCUObpWggjsU82pzAPhX9Iv QOvunMHAUap6wC/cxYVNu8XlUQ== X-Google-Smtp-Source: AK7set+c04U6J0wwMClaGVyTvpOvZFG7I93CD+bD7JQJKcXtMBNQuebr82PnrpOoKfjQ0YfQoyG7og== X-Received: by 2002:a17:902:ab0c:b0:19a:b1ac:45d4 with SMTP id ik12-20020a170902ab0c00b0019ab1ac45d4mr23569774plb.3.1677422913584; Sun, 26 Feb 2023 06:48:33 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:33 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 4/8] mm: vmscan: add shrinker_srcu_generation Date: Sun, 26 Feb 2023 22:46:51 +0800 Message-Id: <20230226144655.79778-5-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: AA35C18000B X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: a899m5ou9dwm5ucs6ykrpanz1zosejhw X-HE-Tag: 1677422914-261083 X-HE-Meta: U2FsdGVkX1/22NQmJFXvb/3nrwNQb3xoSPzlTcvUgebT8bCUrGg+XlgXxIcnAUt9R7HejjcUcWZ8Qt08y5LgEdAMrGqm4xFsMV9kk+4q/Qw68kuIQtPdO0SqXUnWDAQfzP8NBmIMl7gX7Rm2If/K360j2Gn8lwHrjHepm4tBLf1Nji0vHNwFzpnvWeOO+GuNWTG7x8tY1HzfD8WxgcxPyLQqIwG97lJQhM66aqceMbndgqRvXEz7xVPfsCQt2jR8UiMxr4YaIHAGhxLcUX7Sg93dHYqffcubQLQwAo71KrLUehtO84/4nfXYep0ljTiknrsrafRId7Htnm4ZICW+6FUG8R2nqdRmORW3+ex319p0j5P0Bwh0wev1Kd0EfDnUaX4muNYDb8/XHk/Mn3MetLqZYqqm3SFICXCQ8XYJCsyqILJ930I9iFtP6J4vU6+BSmL3d5/GbMbmCrRBmWC9TOOVR1tzFMDBEpeGw+LHm6FjDjWdtsV/ZRrSZ/tExpmFL3RafggA3qz9qed49d/tK3n51ZbXYfD56+qa1rSZIHx44vMxF9SDWs1HfV3abiue40o2dviXFdT8i9lHW5vhdsAKBZLR7/baa6dt2K4fHf49QiuiCTXyHi/5z525fJoFYJc+c1YlXUhb8VVNDPUMPGQ1mhB05lbeDgcJO7E19Oa1t6f2u6xXMwY9u1T1uAeq+EWdfyVI8+VcLK2DPSL0yGVqv8ceN8pIjq40sJ+KmyfJf7lL8nCb9a7NCLwXmWXRV9NM0FFcIh0UiQconVjRdrf66rm0cuyptL2Y5ffO3HG8deEnuUJ6L0ACCoUHGEuaFJ83zQtzRtqb7hEEW80qPHSsQdkCQVpnxEsphDVUQBQpQWcWypz3aWsJ25pUfhzFs5a9mg2f5ER1XfhIFy5tqK03mFhe+1dxhKO2V6CeXtJUlf9Op1uny/D5nPv+AkprXvGf8R3IVrXlbNvKJQA gk5Gq78k ulE40R8/W+jkfTT7B3HETNE48EUU/3cdVaxniiccp7Fowwp5tPQaPOMA4SOMtapaPRpv09IedGJxeedARJyucHepGKKzIKeYHV7LcjUHrD+ssAQjqaJnXrmhYWwRA/PrBrUe5S0NT+Nq/m78w/vMrslgLCEo/UMrAp2iKaQdqBcxljA1pmrDAFU1X+cSa4L5BAXioTFW/nmk4SdMld3+2dPCpMIcz0EclbciEcAeduX5HlgzEcMSDuFIiU+OaKSgXI3kA3O2a/2rf2Yz5pgR9YGi/kBCg+O7dbJk+9Zp7FZsfKOJFfFD6fHgsKFyDMb8RUisJqzwY2snhoMQNoG2a2f88+z3yOiyZ28AAijlQr7+sZIWAkUgdRGJBg7RHCcgKW61qstaZKhVnYJSkwaEIVLnXys++FP+svJZTzDxdprw+Q/4KoEDK+6kE1Si4yLVY4LtoMEGf+qqvt2HaGOicH5Mhbp1Ah4FUnVBXaNOmQ/Uso2o2K0j9OjJxonmQomRhPbxX6oFPQu3VeWeMEEezrPqI0g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kirill Tkhai After we make slab shrink lockless with SRCU, the longest sleep unregister_shrinker() will be a sleep waiting for all do_shrink_slab() calls. To aviod long unbreakable action in the unregister_shrinker(), add shrinker_srcu_generation to restore a check similar to the rwsem_is_contendent() check that we had before. And for memcg slab shrink, we unlock SRCU and continue iterations from the next shrinker id. Signed-off-by: Kirill Tkhai Signed-off-by: Qi Zheng --- mm/vmscan.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 490764f8e085..99e852c0ab9e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -204,6 +204,7 @@ static void set_task_reclaim_state(struct task_struct *task, LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); DEFINE_SRCU(shrinker_srcu); +static atomic_t shrinker_srcu_generation = ATOMIC_INIT(0); #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -782,6 +783,7 @@ void unregister_shrinker(struct shrinker *shrinker) debugfs_entry = shrinker_debugfs_remove(shrinker); up_write(&shrinker_rwsem); + atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); debugfs_remove_recursive(debugfs_entry); @@ -803,6 +805,7 @@ void synchronize_shrinkers(void) { down_write(&shrinker_rwsem); up_write(&shrinker_rwsem); + atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); @@ -912,18 +915,20 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, { struct shrinker_info *info; unsigned long ret, freed = 0; - int srcu_idx; - int i; + int srcu_idx, generation; + int i = 0; if (!mem_cgroup_online(memcg)) return 0; +again: srcu_idx = srcu_read_lock(&shrinker_srcu); info = shrinker_info_srcu(memcg, nid); if (unlikely(!info)) goto unlock; - for_each_set_bit(i, info->map, info->map_nr_max) { + generation = atomic_read(&shrinker_srcu_generation); + for_each_set_bit_from(i, info->map, info->map_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -969,6 +974,11 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, set_shrinker_bit(memcg, nid, i); } freed += ret; + if (atomic_read(&shrinker_srcu_generation) != generation) { + srcu_read_unlock(&shrinker_srcu, srcu_idx); + i++; + goto again; + } } unlock: srcu_read_unlock(&shrinker_srcu, srcu_idx); @@ -1008,7 +1018,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, { unsigned long ret, freed = 0; struct shrinker *shrinker; - int srcu_idx; + int srcu_idx, generation; /* * The root memcg might be allocated even though memcg is disabled @@ -1022,6 +1032,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, srcu_idx = srcu_read_lock(&shrinker_srcu); + generation = atomic_read(&shrinker_srcu_generation); list_for_each_entry_srcu(shrinker, &shrinker_list, list, srcu_read_lock_held(&shrinker_srcu)) { struct shrink_control sc = { @@ -1034,6 +1045,11 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; freed += ret; + + if (atomic_read(&shrinker_srcu_generation) != generation) { + freed = freed ? : 1; + break; + } } srcu_read_unlock(&shrinker_srcu, srcu_idx); From patchwork Sun Feb 26 14:46:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13152426 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3681C64ED6 for ; Sun, 26 Feb 2023 14:48:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 573CA6B007B; Sun, 26 Feb 2023 09:48:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 523FA6B007D; Sun, 26 Feb 2023 09:48:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C4CE6B007E; Sun, 26 Feb 2023 09:48:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2AEE06B007B for ; Sun, 26 Feb 2023 09:48:42 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id ECE6B1201C3 for ; Sun, 26 Feb 2023 14:48:41 +0000 (UTC) X-FDA: 80509724442.02.7B39B76 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf16.hostedemail.com (Postfix) with ESMTP id 1B113180013 for ; Sun, 26 Feb 2023 14:48:39 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=PLf5B69J; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf16.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677422920; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4ra+jS9qPdJDcLL3tCNVjDgmNt8q5h7NH0ipc1XhyKg=; b=pjffzugiV5UuYrmeZca6BHuPM5hmk97pyhikScRAaqrW0iqr7xHKz6/AglfdJ+SdoWe9bf jfeJnF7cnI6PXMAYpOLfjgnfNO81NSuR34LGLpdMTSKqldau7rEt68F5PyAgC5QFkxPnJZ ckPYTCXA7KKT73g8w7rtaTbbKwqTRnQ= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=PLf5B69J; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf16.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677422920; a=rsa-sha256; cv=none; b=DKIYlzfMqFREAyppsc8PPvOS0D63cKrKG84efH8KgKmSQVKtYZdbrODbFRpXMY9N7YXDs/ xMs/GNks45e4drWm6GsjmFhq90xcRhjmB4CgULFF12S34Pwu4lC6RCZHyz141XOXpSI+8K A4zDsI29U3Hx4grcU0jRY9NJzbck884= Received: by mail-pl1-f174.google.com with SMTP id v11so680953plz.8 for ; Sun, 26 Feb 2023 06:48:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4ra+jS9qPdJDcLL3tCNVjDgmNt8q5h7NH0ipc1XhyKg=; b=PLf5B69Jh3pz5GXkfJ0kzyAzNE9QmfDh24Eym4+RIsIY+kh7/HFsEQ2eahiZrlAtR+ 4MxgpRyO+JR9bXbQmyfDQ7Xzvmlu8kCxHtggdFCMFTGHU/zrqq+iyrlQZ1x2SIvb3hxE GprjsegftQOiKVGJiPih2MUwmk/ZVPeSzgftTgUUilU4HagO6HlGsAjToabMn30Ye2jJ D1sW5UHlLVaMC0PQVaPxtdHsFbPDqUB7VsgFSalXcyA5wqm6EJXO+viNO6MMi5fGoOK4 u4SJ8zlHcmMYd0X0NrZ/p1mkOI0zz70UPWZaATCxnS4NbaHp2WXdx+JDlArqTdLRnVOn lPsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4ra+jS9qPdJDcLL3tCNVjDgmNt8q5h7NH0ipc1XhyKg=; b=3PX8k5mKPN46p7RZuFStb2FGavsKXe8CoRwv8olpifHWF02vnhof9Z8e2D0WH/cmlv O0ZVNfrZk+ZhdfL5hzoLWtQJAP9Row9zmvaEP3ZFyB2+HaNFEyua6ZXI01ukGLMVRayf abooQ6C/QmeRCKKBSfvV4D1wcWOMfOvIM1f6eXZwVX+Ss6EdiXHUjunjBF+Xgr0M0n7B b8p2/hkUx1z3qghuCR3SuiU6F7bMqYmJeAMPn7dtMBEQ1xC/+S99wv/K6Yt9P8aUcX+P JU/3EouYdFHM8tXQh9vXtnF2zubQd2ficf6nPYQnfPDaFY+0UOXfBgjd4PVdpXr/gyvg h6Bw== X-Gm-Message-State: AO0yUKUfnyh26n7ZEQeC+03PbjiI1BuV3XZWFono6QrGea6e0FdqouOj C8wANTVJC2d7yUDcjgaS90JPkQ== X-Google-Smtp-Source: AK7set+f72fXg+fOOH45vAOIpM4RdqUXZxaFOCsUrACUIANqZsAcaZRsNmxNPLzIzBDfISXFBkJnQw== X-Received: by 2002:a17:903:2291:b0:19a:8202:2dad with SMTP id b17-20020a170903229100b0019a82022dadmr23250501plh.2.1677422919354; Sun, 26 Feb 2023 06:48:39 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:39 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 5/8] mm: shrinkers: make count and scan in shrinker debugfs lockless Date: Sun, 26 Feb 2023 22:46:52 +0800 Message-Id: <20230226144655.79778-6-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1B113180013 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: gxhqj7g94qxwj13humb5h6gwje4q8ta9 X-HE-Tag: 1677422919-36286 X-HE-Meta: U2FsdGVkX18y8t0cXSbuuzdF+UOTqIfG57CcBs/vhYScDxv4cTh6+/kYzvrY+MPskkzqqXBtMCKoReL/RzilwJYVomSKqv+fUpIYf6rmjQGJjxHGrFBWX1BFxjUJm4UwJgmQa5vS879K2+73thsCVM87RLrCGRKl6Wu+GNRlB1iWAik01seVFpRdAPjT6XoiGyzsrnTlrsipl6xTSl1bRrlSdlBY8J5YnmfqsaA1jkzn5H9GDJ/dlPeDbgxHIdtNlWv+KJWcnFR6UjvZ8yrGWKTJhgm318YaYO3Pgx4ukQ98oxvrZ9he2e2kCp292sclVxrOnOHoMzppi7Ld6Vh7JF3qfaNFiDG4k36wz9Av8A9+w5Bck74vR7qX6ATElxmYul5TcpfFZctTEmoNA2qe2y0ZYRO4VMj5uQEZnhN5vxFLpJySjf3VVi33IyUthIEZ98z9VY9MXLjAwnydb0YGkuE76U6orGQ447e7ECSliVlCgHEzwKlaTHIOjRG0Na0sxaR6hL1XX9q9E6mADbX/2cwrIM61OtLZ6bdZdwXqtJq1L/g+bqq9YwjDRQnzuhXoS6MvOXZx5T9wSapSm10l1Dvq6GPn6qltZDE/3CUbgTTVS5/My6tnq0XdFDtlDpcqQaVzlmAM34RiBzEf3KPGFuPAvfnI3FtVPwWNK4m2FwIpfoQBJ7phv5S4g756GVLmA/HKDrZ6S8WYAvUh6JE8MiNWGVhBG98dggXCO1dZJ4sRR9pRyJUJdKZdAL8KL+lyMxGZgfPsbXEt3e6JnJv8GCPr6nGMUdPG37y8JZXm0W+2EM8c6qIpID1lIpfg1gLSF2JwQ8SKhWDVZ4fYOAE/uyFMPeWybcd5EGbyVWgpz1I/35OlvnbJ/9VqNh/RG51ErZxtqXOQLjaEnExaEP5drxqj83ekqKbwC0EaYQHMgwCXO/wiWncmvROKdp+C2AIprxMqp1e5g1yV9d7ZcsG Xm1yIHU9 IdS5TRvqVmwcuynp5yvgeyF6MKq5ENkCO2Kf+fQIGnicmYFQlsnSxuVUN0g4NZkOFeWjGMnrUT49pvE1O7gA96rk1eeOW9Pfp8W5JqPU0k7EfD4eoMKbj0JiJn1AwY4qoIp4tkv/csbtJky8xLslAHwyEI++3gvo3ulzCblvHZl9Xy4I6Dlh2E5oxMZk4Xg48hOjsoiqFqB4EPEUFaObv4k6eZqaCiO7wJhyPVeg0tq2M+Sbnesg+K2HYQkjUGO5+iH4bP2Lv3dcr9uH3ORKug2knJRqJjryHdz/R3NDAvuGHbQ/gAeelxocBgH9pE0dHgBZLQh132OdIC6V1ker7pN7Fpjg0Yv58GU+FUOI82z0YIFPtMZcFCPT/3f0beAnv4ct0o5t2LnME1ox4AsLLhoTXHxYcMSj0JWk2r16+SrysyjE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Like global and memcg slab shrink, also use SRCU to make count and scan operations in memory shrinker debugfs lockless. Signed-off-by: Qi Zheng --- mm/shrinker_debug.c | 24 +++++++----------------- 1 file changed, 7 insertions(+), 17 deletions(-) diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 39c3491e28a3..6aa7a7ec69da 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -9,6 +9,7 @@ /* defined in vmscan.c */ extern struct rw_semaphore shrinker_rwsem; extern struct list_head shrinker_list; +extern struct srcu_struct shrinker_srcu; static DEFINE_IDA(shrinker_debugfs_ida); static struct dentry *shrinker_debugfs_root; @@ -49,18 +50,13 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v) struct mem_cgroup *memcg; unsigned long total; bool memcg_aware; - int ret, nid; + int ret = 0, nid, srcu_idx; count_per_node = kcalloc(nr_node_ids, sizeof(unsigned long), GFP_KERNEL); if (!count_per_node) return -ENOMEM; - ret = down_read_killable(&shrinker_rwsem); - if (ret) { - kfree(count_per_node); - return ret; - } - rcu_read_lock(); + srcu_idx = srcu_read_lock(&shrinker_srcu); memcg_aware = shrinker->flags & SHRINKER_MEMCG_AWARE; @@ -91,8 +87,7 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v) } } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); - rcu_read_unlock(); - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); kfree(count_per_node); return ret; @@ -115,9 +110,8 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, .gfp_mask = GFP_KERNEL, }; struct mem_cgroup *memcg = NULL; - int nid; + int nid, srcu_idx; char kbuf[72]; - ssize_t ret; read_len = size < (sizeof(kbuf) - 1) ? size : (sizeof(kbuf) - 1); if (copy_from_user(kbuf, buf, read_len)) @@ -146,11 +140,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, return -EINVAL; } - ret = down_read_killable(&shrinker_rwsem); - if (ret) { - mem_cgroup_put(memcg); - return ret; - } + srcu_idx = srcu_read_lock(&shrinker_srcu); sc.nid = nid; sc.memcg = memcg; @@ -159,7 +149,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, shrinker->scan_objects(shrinker, &sc); - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); mem_cgroup_put(memcg); return size; From patchwork Sun Feb 26 14:46:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13152427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93C39C64ED6 for ; Sun, 26 Feb 2023 14:48:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 39F716B007D; Sun, 26 Feb 2023 09:48:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 34F486B007E; Sun, 26 Feb 2023 09:48:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 219BB6B0080; Sun, 26 Feb 2023 09:48:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0EC646B007D for ; Sun, 26 Feb 2023 09:48:48 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E12221C5D79 for ; Sun, 26 Feb 2023 14:48:47 +0000 (UTC) X-FDA: 80509724694.25.0BADB9A Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf21.hostedemail.com (Postfix) with ESMTP id 1B8141C000D for ; Sun, 26 Feb 2023 14:48:45 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=lIKiq8Er; spf=pass (imf21.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677422926; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=c6cuoJEo92PScDqgnKjc0MgPFLiMUUPUvBDJj7SgxHo=; b=XgpIithzxhJvVsUarXMz5TL2uhJbvFUQK5uHALkILAs64Jc39hkEn+y/bwRCwEsAud0jbd rgVKLrhzwCzdN5g8nh2wYWt3ULQltznGsyQB0WAHrCQDu5rb6MnYK3+MS0oeVuMpOE4L65 I8HMa3WmWOI5eeh2ntqSjlVuIdJOim8= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=lIKiq8Er; spf=pass (imf21.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677422926; a=rsa-sha256; cv=none; b=ZUlrke+CKLvI46mjxeDKk9WNlAp07iKAAg5gNMfCfiWDW1/HvN4KUnLz9c4+JgA5TKTIkx 9piylUEbl3UgrjEsNkF4t3RbO06ZrQ1ipzBxnaHNMKDPz+PrqHGpIFS57FoSqNkv9VahGC nxOAYcYsg8gRk7dBQjX8naxVV3nb2X0= Received: by mail-pl1-f171.google.com with SMTP id bh1so4177179plb.11 for ; Sun, 26 Feb 2023 06:48:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1677422925; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=c6cuoJEo92PScDqgnKjc0MgPFLiMUUPUvBDJj7SgxHo=; b=lIKiq8ErtDRmfz+i/ano83moR805w7pQEVFBs5DWz8iogOWtMgCKlWMtTjjG4CUYGf qZUEa/ayLAYcSmEePRMBNnD4UMS+HxFoUzmlroe0ZVHuqZ3z2bv2HGbh7Yq0wKCvHk4k QE8yyj88IWGiNyUTVZhSPdvEoYldgr4C6pMLd3y42cOtg65TmytTMhFeDnjLzWUSuqD4 9aNiX3gjRzbrv4MYxQIZGaPfntfaShxVFiNdV3/73U+gNlaqjtHmoW98ZiAC2kGLB21U 7G4hnPw+a24oAWIJ7kfT7KLIQYmuet0PscCXlnmSZhQ2rIyHh9ik/cyvUbqhLd+kEnv8 Uwjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677422925; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=c6cuoJEo92PScDqgnKjc0MgPFLiMUUPUvBDJj7SgxHo=; b=LMpmzeks2mwr4szAExhGREinpguTYvQcypAqo+UMiekncWnKlTjZ3lncbuChz5YnNL OyXcgm9TLR8qaqmKUbyGbmSaGbfk1U8hTWqQ15cqyBPXMliYcBweyR5RumRMJWSXdSqM Jg1ue9bW0+w430RqjvcPrMpLafg+AzjrJVoEK6r7WYEGBYrWjjD8Qq1PKsiem5SsQO/3 dxp34fGsAeriNKh3ZgWLVOIJHsRmgFMQaLAvEQT/fCFPaRcAzOk8OC6q3gYC7pZ0zCQG X1UuTHiy5sm8/Qy8MZIcaYbHCal2EqRBtZvnFC72/J5dkpqcJGAYhFZXB8VAVzXh/ssq ujNg== X-Gm-Message-State: AO0yUKWC1Zr4O7orkOBPztBgLXT1y3dQ5UrlTTtlY2gX5oNsfZeJd/Td Cr/zPaD+uZ1j/DqfG0hnvc1leA== X-Google-Smtp-Source: AK7set+XvuciT55+1IMf9hAHsturuZDzsZyl61PmuY/3HoeWXaqHrVSGXMTK34S3WOCoi78lnQD8RA== X-Received: by 2002:a17:902:bb0f:b0:197:8e8e:f15 with SMTP id im15-20020a170902bb0f00b001978e8e0f15mr23385229plb.6.1677422925086; Sun, 26 Feb 2023 06:48:45 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:44 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 6/8] mm: vmscan: hold write lock to reparent shrinker nr_deferred Date: Sun, 26 Feb 2023 22:46:53 +0800 Message-Id: <20230226144655.79778-7-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 1B8141C000D X-Stat-Signature: orkrbd64gfdooxndueqycj14crjnhytf X-HE-Tag: 1677422925-457733 X-HE-Meta: U2FsdGVkX1+R+KJlXjHhk8iROShnD9RGKMbxJDLWzsNdd1Plnw/QTymCNZuPVICh4gaFTKrkV8APFkAMkpMWklQD6s4cfxRW0H8ry3sR2FwUGza0XKt+g7vIKBp+KNZSYEPjAoj1zMj14DVwYIyofyz/umJtyUYDjRXetCHH8+mZr6zMWtfc2ebyKHSEGzG/lP3YjAssIwmAooKRAqAhpGr4M8yoBSyksXHjoGgWUQ6K6sTlMJQyZoy9iLjMhShhdT+HHMQPv31PyoQ506JI2VMdgM2/7KndWNS2Xs1SOhOV8cWP7tpqfFOxkGY2AuXI5HdBKTpcDTN25spDR8V0vJxa1zdB+RLn5CLo/epiPajZd8oCslsGAK66d3/gUvv9Yq2lJIXioa6bQrHTz9gf6t6rtNBQgGE8bN7Zia3+MMrtq49vpfmHbnO1jU9qpwE6i0DwNXscNCfOhyK0/b+bTOfSHJLwdHkTxVeJZoNOCVKT7BzVqZws8lUxnn39Ib7bTBS3/ih5gMW1hDDbMuofOVdBemu0BCHLbVK6tBE7syCwzHy4Ege5z5wEW4KxrbO472+T85Tqm+xLtnTdiu+eAo/rDKhxcKe3RAIduV5gdVETtZsYLzFRGd5UZKp1EQAMNJ5wW5WWoQOkFAAOlzIAuhUgoOyGSRJxCLUUP0UT6X6Qni58gdSfrK43KawSdUIIDcF9spfcmOoNNC11pEnyJB6DdHvllKmD+Hyvnxka1o+ryypSHfTo6NsRRWf9mwhdYANylRAHF+JXXfHb0DEpwzAZTj4rVEqdwH1j5D+IXy8aTNtHnUPnaPa19ZcE09f2nvAqCVmXvzBUQ2jdfKxpCKfqv/YPi3Xyoi17s34BZegeU8mRA3kVLhmke6bmdiB3dFbdDFGZyyOPYNSvAZP8kF3RrCP5gAAqAobBceiXLeIIhbLWaTQOt0etY4WGuNL+HnLOsdNtMtCp2TH3bhV /OW5zR0e RFU4hAT4t481fLGh0MKbEgGFtDgkYh6AIf4wqQhN3ro56kB57lj6hRvonVwWsR5RCMv6+qvDLIWBkcEzJx6ATj2TJd+NINn6X2ULPCG85LTTmkhsSq1rIXlzsnH3qp8kQlU6qfIGaeHys9UBib+KY2rrRoJBUAyW+cjVp0hLJgkM/dfWy7DGb5vegzRW7Xorm5bLJf5Nr1hwb2FztggG/M8dbqnnw6RKeuHtdcNcrF+rqes86+o9MLjPlHvi0FJu5Cd0OjFvFmEWz1w4lRescJsLMji5JS3uRF+OGl6isqzUhZcnrmxhPFy03sF7NmaMSvUUlgZzVJ96GQbJGx66M+reW2L1LHIujysidWj0XJwdGPg5DI4zfyFE+U5qhObDNdb3fPpVyZXD5lX/cnbHScuyfL+HgjTej+rKEqlS0Hh4RqI9RCyU9Qub3ajVM7N1Yfx0Nti1dDXkdmrbQ/gZl3kS2n2zmgGinRm5y X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For now, reparent_shrinker_deferred() is the only holder of read lock of shrinker_rwsem. And it already holds the global cgroup_mutex, so it will not be called in parallel. Therefore, in order to convert shrinker_rwsem to shrinker_mutex later, here we change to hold the write lock of shrinker_rwsem to reparent. Signed-off-by: Qi Zheng --- mm/vmscan.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 99e852c0ab9e..16ff64813175 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -451,7 +451,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) parent = root_mem_cgroup; /* Prevent from concurrent shrinker_info expand */ - down_read(&shrinker_rwsem); + down_write(&shrinker_rwsem); for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); @@ -460,7 +460,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_read(&shrinker_rwsem); + up_write(&shrinker_rwsem); } static bool cgroup_reclaim(struct scan_control *sc) From patchwork Sun Feb 26 14:46:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13152428 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24692C64ED6 for ; Sun, 26 Feb 2023 14:48:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B8EB76B007E; Sun, 26 Feb 2023 09:48:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B3E496B0080; Sun, 26 Feb 2023 09:48:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A07836B0081; Sun, 26 Feb 2023 09:48:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 92B836B007E for ; Sun, 26 Feb 2023 09:48:53 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 76DEA1C5D41 for ; Sun, 26 Feb 2023 14:48:53 +0000 (UTC) X-FDA: 80509724946.11.3A56D93 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf15.hostedemail.com (Postfix) with ESMTP id BA4F9A0010 for ; Sun, 26 Feb 2023 14:48:51 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=RLNAX82e; spf=pass (imf15.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677422931; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rPbK2mA8SftO30bweBpSZUhXCxYptD2QUCquMlUkzZQ=; b=bF0Y1KO1gaPuODxxmAQpZkkphBWVScYHqRCGohn79em1BT4oBcaxFSeXub+II1csjnPDOZ dlW/v2yDOilFWHFJihEEaVpI1sdjJwKjLouhIypMNDBoV9WVeuQFLwW/w55+RK0wE+XlZw GhXPevff0OmZeCOqpm/1M1PV1QSElv8= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=RLNAX82e; spf=pass (imf15.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677422931; a=rsa-sha256; cv=none; b=h7dhUxmmGlh9XHbTxpKekzIdJeF0eMBu5R6YEpoRSoINkRkOZc7CbQB4GTnLuknoTdH82i qjKmLzLZu6m9U3CdlBW3StgGMnnEXOgPReGjFsHFefQNuw9FkxB0jXQLw+5Nyq/EnxCqLS LqEBnxlGpRwJEzN/0qBiEw2tg/mdzQE= Received: by mail-pl1-f175.google.com with SMTP id i5so2515827pla.2 for ; Sun, 26 Feb 2023 06:48:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rPbK2mA8SftO30bweBpSZUhXCxYptD2QUCquMlUkzZQ=; b=RLNAX82ej6WiOp+gJxxrYUevqQBzdDGYpvRp2WltPYHzsFagaq47ozL06y+Hy6OgmK JK9iGaHN7dJvbFey7hQXB7aig8jRb8Fzh2ihtgS4KPi3JJgVeer/QP403MbnYAJfXhak bYN0OJJ/krh591ZGhC6ZmW4Bo9d/0ELhp3VjgtJ7eyFa0hScQMuwE81nRfrktkjFE6Os OjbYs65uBBhgwi4YlDp3YuXDzBBPVqJDlkDh/2welq1ly/HLAfshGinj1f1s9zkkezz/ 2xhknARP+taXDeqDm4xAjtZ8aw0C7dGDxIicSpRhb2gQC9so4gEZ84pp+H7E0riZtKLc Tpsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rPbK2mA8SftO30bweBpSZUhXCxYptD2QUCquMlUkzZQ=; b=m/TByve51pX6VvSmkFj8ml4a75wZKvJMnTrFJOUvgHjUZDzloAysPPnHtdeBug5BLb qQixXApGniRfvq+Dundj0dFK5adjvXEYJnJbpq46+jzSEdw65xgK1QXaZ3w8ynvp+SNc GJeAuYTvtYBN6Xa72i0mHj8z852jpOxkjwKxb7xCnqyIzlCKMk81pWxSFMbGWXaDrM/p UNPFlcQovI0vqdP3dGePeuUAD5jDsEAD27PgA/Dg8dgUi1tKfo132f7XREN2qILn6U07 e3HWlbHE2j2+u/dd3ZlHdfR+lIL82/Iiyw+L2UB/c9kCTOSxQ11WFr3ZxCEmOMnwBRJz eDKA== X-Gm-Message-State: AO0yUKXhIU97m+oloWRpLoLue13wlbVSonDV1zhYK45G+8h2V4o94tUu Mg8QAC4q5slNRNPi7U0teXgZCw== X-Google-Smtp-Source: AK7set8jSrtu5jvJSft27Jtxn5UB2T4NTHUh1NNGGVH4kZw+rpTeROnHO4aYE0DylXNd7RBZ0PVGQw== X-Received: by 2002:a17:902:d4c8:b0:19a:7217:32af with SMTP id o8-20020a170902d4c800b0019a721732afmr23249925plg.5.1677422930708; Sun, 26 Feb 2023 06:48:50 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:50 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 7/8] mm: vmscan: remove shrinker_rwsem from synchronize_shrinkers() Date: Sun, 26 Feb 2023 22:46:54 +0800 Message-Id: <20230226144655.79778-8-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 93ob46a3fpmntoq7ik8h3t5tem861hkh X-Rspamd-Queue-Id: BA4F9A0010 X-HE-Tag: 1677422931-13535 X-HE-Meta: U2FsdGVkX1+WxauKIwAzELU0vEeUfse2cHf9dy4GiQxuU1O4yIDDBLRJZh0wiNT/0yJwZLAbl/9GuEyJOTE0bS3vXWM+SG1z0fE4gJ0UcpBbgXQCWbM809Mybxq5Ma6jnoDS57EIelFepbXX79a1DoSLfxBSzscnN1rPYsWQkeSqbmVuynGL2y/W0osJMbWXKsT2cimbKfTxN4K1zOEk6P4RnSh2+bO7b1WUOosDJTo9dmZwvVO3nq07hTCoKhiwHCowTRBjD0XyMBm6WpFSd0XXdUy9PoGKNUd8JfTZiPrLB9XJvfr+GZAWoXlLVVtV47Jxu91EzMGfUsLEgp0AF7WcpVSaIjLT4hU7PP+PmGh+/rRW73e4V9HDBrPTwc8/ScDEY1LepD0SA08LE4BseMOtoxwW1vHIXhH7NGqyVDcNRN+C6c4TJspU9ursTGTXfR9qOkD7P1JivtcMMHsPskMJg0w5Sv2FKQeb8USpw7MYKL40VxYH+KhlplApXRMcvZcMVcLBqfM9mSFIBOO2jinSl6wQVmroCxEtGb0nW5E4DdrFxnXZkhTJgQDr6+ldlG7SM4MuDpES7uwly8/pSxwVeRc3IQsiGSYXFU+YOqx9aWHLW0w+SZPZ+cM+T4jkW8iu1/7Ajpj6n5A7pnQBkAKp8AnzNy6rfIziG/ikQa9P7F4AryjsArdzzMqkGM8fFXG4RzAEcVA3qkabAa9LDT++/TEmwkKsz1ZzqIc21bN6F0NIwUI968s6rSXEKwJVM+lAC5P28LCbdNZEr7M3sCgzRFYt5FjvUA8zG5il5UpBPuvOaaT9zW1MGlMupeJapxqwAAxON9ObuKE2x7lFa+o/ExTnztBUwKbXuFd9kBLBdtql1xUqAXYNzqyWtGJC9zeRcPmNl7GML5f5NoP7dqS/e5WZqZhXLM/vdcVEh9dZy+Y7Bf50AD9ko+t4Iod7q+6dog0C8m1SuFolq/u VTQl3clQ iAxM7UEk7HMCPbF/OhqOLbeC9IxvpC5ltQDFfQZo8LfFncqy51/sq5+6X8UcqoUttLLU6yFIy8U0KzaQ9+GwVC86k6I847dhBw4pB6HK5maERpgFRTvhqJ/sSyY68NlOF+GfWuOSBVlkSE7aZVUFkMPTdupFVxc0E0uvREl9U5nopglMbUzNOCAsZlFtTiGlY8nco+Zt0U1d9lC7yF4UODHFYXhPnIFEX6v6So0gz1ZttvLz3EF6FJBuysGqbgT5O5bv5NuleC9+2B9XURyV3W7HHXCQZ/f0zwsHTftJTZcpBJ0NvQ/WD5AJc/8h3iRnatvzxo4aHiWK1AtZUZWEWikhaaYC2+MwXNVVDhw2RGaXn5Q9bXk8Hki1zi0XjvkO81l+rVkF4sYVJrZv+AFDsqqndrKRobVbbH71/NAFmTNRcSh28o7vkpI0KJ2Hqo06Ag/XnNAk8JtVv2pcfw+WsEO9GijxFPOwisKLn X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now there are no readers of shrinker_rwsem, so synchronize_shrinkers() does not need to hold the writer of shrinker_rwsem to wait for all running shinkers to complete, synchronize_srcu() is enough. Signed-off-by: Qi Zheng --- mm/vmscan.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 16ff64813175..2d71fd565c78 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -796,15 +796,11 @@ EXPORT_SYMBOL(unregister_shrinker); /** * synchronize_shrinkers - Wait for all running shrinkers to complete. * - * This is equivalent to calling unregister_shrink() and register_shrinker(), - * but atomically and with less overhead. This is useful to guarantee that all - * shrinker invocations have seen an update, before freeing memory, similar to - * rcu. + * This is useful to guarantee that all shrinker invocations have seen an + * update, before freeing memory. */ void synchronize_shrinkers(void) { - down_write(&shrinker_rwsem); - up_write(&shrinker_rwsem); atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); } From patchwork Sun Feb 26 14:46:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13152431 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31DE9C64ED6 for ; Sun, 26 Feb 2023 14:49:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C517F6B0080; Sun, 26 Feb 2023 09:48:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C01BC6B0081; Sun, 26 Feb 2023 09:48:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AF2EF6B0082; Sun, 26 Feb 2023 09:48:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 9F0A26B0080 for ; Sun, 26 Feb 2023 09:48:59 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 6E3261A01B9 for ; Sun, 26 Feb 2023 14:48:59 +0000 (UTC) X-FDA: 80509725198.06.8FE3218 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by imf22.hostedemail.com (Postfix) with ESMTP id 9ECE9C000A for ; Sun, 26 Feb 2023 14:48:57 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=DU9ImCfz; spf=pass (imf22.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677422937; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=f5xm65H9BLTSDcpU/nYsAyAok8lJrBUsJgeNL2FDrMI=; b=lFW1dqedn6nuYvv7nBJf8enpbsGC2KNFsa/W8NaOUWDe2QyS3qK6kOtcGZlYEHzdbW6XA9 68ypTDkTeXXfrC2fazWJPJ7jJbHtDqCST4wD15CoaL4+UKCteIzJk9OUYikJuTodn8Vt9n PXdnKqa0HnZ18h9rgc8QNy/baPw6aFo= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=DU9ImCfz; spf=pass (imf22.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677422937; a=rsa-sha256; cv=none; b=UavgSBFkploGFAeMcE60ub7iwWPxrfEkvXVa5GnHCLXg1hMuXvDegbr/aXPHkqeC8WN+Bq D2ApbH6KTn5hFqh2HvEdToGOrb/d9WNfIJlfOIGErRepKHnBUP37TsqnAPBrUqt2DPOq+3 tUNjw5xfoQU3gWjINmJMwzaUzBjWseQ= Received: by mail-pj1-f44.google.com with SMTP id nw10-20020a17090b254a00b00233d7314c1cso7524967pjb.5 for ; Sun, 26 Feb 2023 06:48:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=f5xm65H9BLTSDcpU/nYsAyAok8lJrBUsJgeNL2FDrMI=; b=DU9ImCfzPDt6r5qsdIJgFlMqPyv1XNzrFdxX1MYevySp7rV/vBy4oHtHlXM0ehu+/q e7Sjxf8qsqyQGdUSKu80ZuyH8gv9rpDyqCA+6sYlagLU/RVK4FI8pl7xZiXV+Ojb41zx ezPpwq3uJvNV5TVTXIF7WKeiEmB772qpgZU/rJEebve9b47zOunZTUMe/f+gk7PISSjc Ehcf7cS8nj1sKVkD/JveUqUJzsUCxQfCPh/rzpzR2amvQmZsujUa/Epz4Eonu2OG9utb VbC/USw9zVaaeY/TFpc3MuKTAHI4ssQxadvh9v+878R2uidsPviikdk/mxseIFk0IHSC +N3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=f5xm65H9BLTSDcpU/nYsAyAok8lJrBUsJgeNL2FDrMI=; b=a43bkBQHW8yfcBzHHxnb1YqklU6eWW4yG7sx7yVrCZzdhPCyAWm2OQ6GG0vfl6q6zv cEuJXSQECxInHXJ0KuM1vYrQfiWX2G4Em2eTn47erO6BamoBclhK9xyzXdFafdlzM1mj zfxA3YVo+0BUito59oiszno7CiN0HXQsCrgnVOYxidq88mwwZ2XSslzz2rOvIvFRrBcZ jOOXpvI82F4dG1rDY+beLYBrCpLiGa7Tt6EJMS1yH2WouBe4+Dk0o0WDHjKNkesR9mdn jEtPn12fuX9W78t20qXFV4u/17Nf3u1+WDAnEyPZ7QKQJPQnZrEGzTQb+drHE1vvUClG WdeQ== X-Gm-Message-State: AO0yUKUxZuj6a5hr9PyiiTiMNsu8qiovMRy42ZRM3yMacVJWexuXu7yo MIPdBhzjav/dAfda82Jpv+HQjw== X-Google-Smtp-Source: AK7set9OLUlQVvloHLhW8LBiRt1OW+15leJOMZQHQ9/zIbOmm9Z4KelXtK4HEXm+zbHSQlGGRoJ3MA== X-Received: by 2002:a17:902:d4c8:b0:19a:7217:32af with SMTP id o8-20020a170902d4c800b0019a721732afmr23250125plg.5.1677422936561; Sun, 26 Feb 2023 06:48:56 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:56 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 8/8] mm: shrinkers: convert shrinker_rwsem to mutex Date: Sun, 26 Feb 2023 22:46:55 +0800 Message-Id: <20230226144655.79778-9-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 9ECE9C000A X-Stat-Signature: ndf64fu4noso7pbpji3yzk6x8bw13igm X-Rspam-User: X-HE-Tag: 1677422937-12708 X-HE-Meta: U2FsdGVkX18ZFPAq80UjTx5C6+iac2IIlwFJ1URX3gw4jGIgVsLVTZrBrfel+ccUlVMzFZ/dzCGNDpnxIXfb8y1lN6eU96lf/EDhA8JxKcyhT6eMEFerzPretvQmtFzH9fmpT8vDHyzQFk6GZk1sYGY/WNt0mpwvdRRHyHdRICaDvPnzZJVINHeobTpVru1vCDiKre3VT05KSG7xIILxBSiq9AeeQaIoUTHSMGRHtgjMj+s94xgSjmJ+GN1S7OO4SAY8WYBYuGhrLohLFmsfZoJpjMzpEXrJVZG9tVMEwW9AJHr9TZezIVn6LSZfsMfdSsiYjs1uAW1j1kBIUjOmVU5S1NpJ0teuHGFzs52TXToQwyqjYQ7EwDqnmnL3RizgDd314TBRh6+3PvalfIonDRQZe2fcNY58lwTzpG+8xuRCtQAeyL7uugkLnBpTSpIKns45dxWgsDzeQ0LIavANVBT5oub6iFQIgWVauN0m5Auf40c67G1RRQS2pwYnchZovuNUECrmS6YnB9qvecRlkirpYzT28VoqnTR3Kp3uX0h2tXJa2JlME2ngCAH3E3y6nw4CXo8Y/e+YVY8VStPUDA+btGFt1b9RtO/KgRn10oCmtUusAOG4NjNexvv2pj0ljbw61IcSu5N4yPSmrbbGycE+QWoSs60/CzueCylJbNFHsUOqkmvoyGtEQWOKTO3yVZVXNQ3d5TFBs9v7q6JopgYtZM7uqwwv+ufQkeAwM1ddpJxGLmgqSifDuqHM6A3fv4ZGhx8MGFrDhVpZV6PPhhdujISvBFeAX+L5Cdssgp6KA8/WDSFTyoVfEc5oZpRJP5AG4V7cC3uFFezzow3AlB9wqcPOCyUqZ+KSK/sIjKl2pyoCKy5P1bkDFqd/2v6YWKHzSD8hk/G9wJYnaYt1i/kuXGxpOnSVSHBvE7kEUMkBo0li1L+1VWaS3XI6C7cA4DxOSYG8Tu0NurU1NuN lFUyQIdK 3BNT09H/f0ZnXLpGYe/DLd+ttTSUUXgEyLOz0XQCmmEuJaQG+fXVaZ+30W75DccIRhp2N4lEa4bZZ8tQYF1S1UFJRvmw0iEbhhzRSgtpGvaLIcQ5D0qbRFI7e3J2+C2IjFDK7OsBc8biR6lh6dB1myJduZ9DSVPH8WBhYscQjDhL3Ze9U2AjMbIV3FZ4pNq7UEYx4Y6hfnijSaD1T/PxYCDM6kH5FIMaxVG78kfQz7PkavNXbdPYyIOC/ngUrP4zOMDnqZj3k/jLa91aNy2+AmkMB8ERDPtAmfgCyOayXx2vsoxE73ootaNMKiVxBbGT+hc3BnwxeSRdvWK7tSmAJpDwv6y2XslSAUsCsDH9qN5dtWB0PZ2RvEsk7VuoFWraDXwdEDZYDwlkfxv4T4qt0etdNqm8sqe93ml1Zi2dpgUOr43luyARajdOzn9pMIDgaQYpx33PAx7yS8IuH8XHa2yekStKkQAqpgWpl X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now there are no readers of shrinker_rwsem, so we can simply replace it with mutex lock. Signed-off-by: Qi Zheng --- drivers/md/dm-cache-metadata.c | 2 +- drivers/md/dm-thin-metadata.c | 2 +- fs/super.c | 2 +- mm/shrinker_debug.c | 14 +++++++------- mm/vmscan.c | 34 +++++++++++++++++----------------- 5 files changed, 27 insertions(+), 27 deletions(-) diff --git a/drivers/md/dm-cache-metadata.c b/drivers/md/dm-cache-metadata.c index acffed750e3e..9e0c69958587 100644 --- a/drivers/md/dm-cache-metadata.c +++ b/drivers/md/dm-cache-metadata.c @@ -1828,7 +1828,7 @@ int dm_cache_metadata_abort(struct dm_cache_metadata *cmd) * Replacement block manager (new_bm) is created and old_bm destroyed outside of * cmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of * shrinker associated with the block manager's bufio client vs cmd root_lock). - * - must take shrinker_rwsem without holding cmd->root_lock + * - must take shrinker_mutex without holding cmd->root_lock */ new_bm = dm_block_manager_create(cmd->bdev, DM_CACHE_METADATA_BLOCK_SIZE << SECTOR_SHIFT, CACHE_MAX_CONCURRENT_LOCKS); diff --git a/drivers/md/dm-thin-metadata.c b/drivers/md/dm-thin-metadata.c index fd464fb024c3..9f5cb52c5763 100644 --- a/drivers/md/dm-thin-metadata.c +++ b/drivers/md/dm-thin-metadata.c @@ -1887,7 +1887,7 @@ int dm_pool_abort_metadata(struct dm_pool_metadata *pmd) * Replacement block manager (new_bm) is created and old_bm destroyed outside of * pmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of * shrinker associated with the block manager's bufio client vs pmd root_lock). - * - must take shrinker_rwsem without holding pmd->root_lock + * - must take shrinker_mutex without holding pmd->root_lock */ new_bm = dm_block_manager_create(pmd->bdev, THIN_METADATA_BLOCK_SIZE << SECTOR_SHIFT, THIN_MAX_CONCURRENT_LOCKS); diff --git a/fs/super.c b/fs/super.c index 84332d5cb817..91a4037b1d95 100644 --- a/fs/super.c +++ b/fs/super.c @@ -54,7 +54,7 @@ static char *sb_writers_name[SB_FREEZE_LEVELS] = { * One thing we have to be careful of with a per-sb shrinker is that we don't * drop the last active reference to the superblock from within the shrinker. * If that happens we could trigger unregistering the shrinker from within the - * shrinker path and that leads to deadlock on the shrinker_rwsem. Hence we + * shrinker path and that leads to deadlock on the shrinker_mutex. Hence we * take a passive reference to the superblock to avoid this from occurring. */ static unsigned long super_cache_scan(struct shrinker *shrink, diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 6aa7a7ec69da..b0f6aff372df 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -7,7 +7,7 @@ #include /* defined in vmscan.c */ -extern struct rw_semaphore shrinker_rwsem; +extern struct mutex shrinker_mutex; extern struct list_head shrinker_list; extern struct srcu_struct shrinker_srcu; @@ -167,7 +167,7 @@ int shrinker_debugfs_add(struct shrinker *shrinker) char buf[128]; int id; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); /* debugfs isn't initialized yet, add debugfs entries later. */ if (!shrinker_debugfs_root) @@ -210,7 +210,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) if (!new) return -ENOMEM; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); old = shrinker->name; shrinker->name = new; @@ -228,7 +228,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) shrinker->debugfs_entry = entry; } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); kfree_const(old); @@ -240,7 +240,7 @@ struct dentry *shrinker_debugfs_remove(struct shrinker *shrinker) { struct dentry *entry = shrinker->debugfs_entry; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); kfree_const(shrinker->name); shrinker->name = NULL; @@ -265,14 +265,14 @@ static int __init shrinker_debugfs_init(void) shrinker_debugfs_root = dentry; /* Create debugfs entries for shrinkers registered at boot */ - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_for_each_entry(shrinker, &shrinker_list, list) if (!shrinker->debugfs_entry) { ret = shrinker_debugfs_add(shrinker); if (ret) break; } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } diff --git a/mm/vmscan.c b/mm/vmscan.c index 2d71fd565c78..6c5d21ba0c9a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -35,7 +35,7 @@ #include #include #include -#include +#include #include #include #include @@ -202,7 +202,7 @@ static void set_task_reclaim_state(struct task_struct *task, } LIST_HEAD(shrinker_list); -DECLARE_RWSEM(shrinker_rwsem); +DEFINE_MUTEX(shrinker_mutex); DEFINE_SRCU(shrinker_srcu); static atomic_t shrinker_srcu_generation = ATOMIC_INIT(0); @@ -225,7 +225,7 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, { return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, &shrinker_srcu, - lockdep_is_held(&shrinker_rwsem)); + lockdep_is_held(&shrinker_mutex)); } static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, @@ -310,7 +310,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) int nid, size, ret = 0; int map_size, defer_size = 0; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); map_size = shrinker_map_size(shrinker_nr_max); defer_size = shrinker_defer_size(shrinker_nr_max); size = map_size + defer_size; @@ -326,7 +326,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) info->map_nr_max = shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } @@ -342,7 +342,7 @@ static int expand_shrinker_info(int new_id) if (!root_mem_cgroup) goto out; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); map_size = shrinker_map_size(new_nr_max); defer_size = shrinker_defer_size(new_nr_max); @@ -392,7 +392,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) if (mem_cgroup_disabled()) return -ENOSYS; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -406,7 +406,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) shrinker->id = id; ret = 0; unlock: - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } @@ -416,7 +416,7 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) BUG_ON(id < 0); - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); idr_remove(&shrinker_idr, id); } @@ -451,7 +451,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) parent = root_mem_cgroup; /* Prevent from concurrent shrinker_info expand */ - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); @@ -460,7 +460,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); } static bool cgroup_reclaim(struct scan_control *sc) @@ -709,9 +709,9 @@ void free_prealloced_shrinker(struct shrinker *shrinker) shrinker->name = NULL; #endif if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); unregister_memcg_shrinker(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return; } @@ -721,11 +721,11 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |= SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); } static int __register_shrinker(struct shrinker *shrinker) @@ -775,13 +775,13 @@ void unregister_shrinker(struct shrinker *shrinker) if (!(shrinker->flags & SHRINKER_REGISTERED)) return; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_del_rcu(&shrinker->list); shrinker->flags &= ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry = shrinker_debugfs_remove(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu);