From patchwork Mon Feb 20 09:16:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13146178 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C35BAC6379F for ; Mon, 20 Feb 2023 09:17:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4C1E66B0072; Mon, 20 Feb 2023 04:17:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4715A6B0073; Mon, 20 Feb 2023 04:17:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3124E6B0074; Mon, 20 Feb 2023 04:17:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 229896B0072 for ; Mon, 20 Feb 2023 04:17:07 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E6A76160F41 for ; Mon, 20 Feb 2023 09:17:06 +0000 (UTC) X-FDA: 80487116052.09.DE9D78C Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf30.hostedemail.com (Postfix) with ESMTP id 20EA780013 for ; Mon, 20 Feb 2023 09:17:04 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=neiTcExI; spf=pass (imf30.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676884625; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OSu6LoDi3bt+On14RPrDb/428OOXZNPo77LIFm1Bi28=; b=8icQKIjl7/+TByCDbVV0/CN7SXIMNfEwqoZsBGOKLZrinFpxzV4ipGWFW+QTGT3lbmiXwh wyIadoAFxQHJnziq4NBsiyTCiOA4DFtg8a3fZr89ManfZl+WY5V/l8OppAzhljzAQYvfgR XOCXba8ZnoEaFTA69Z1N6Ziy4PDCAEw= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=neiTcExI; spf=pass (imf30.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676884625; a=rsa-sha256; cv=none; b=KIcZ/M9iSo+KRf5j63HVHS5tUPLHbAPNMBh5arjMwky6Vp8YRMe5zUKsk3yWIBGZEYuKVI EXQaZ3WMMe5FeEN7rkvn/BBmI+8ywyJC7LXmfCzYHRFRgjwpz6DDxs6EcSfSw0lQpJl38N c0rJKhrH+fuXTTrxejtXjPtIyXIYvm4= Received: by mail-pl1-f174.google.com with SMTP id z2so750782plf.12 for ; Mon, 20 Feb 2023 01:17:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; t=1676884624; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OSu6LoDi3bt+On14RPrDb/428OOXZNPo77LIFm1Bi28=; b=neiTcExIi57fl2R7QOvc3r3ELyzAtnMxW9GBNNCltkuMHmkXl5ZK+7Q0xCR6QpS+bt hP0XX6ZlR6iZxJOKblWlpeGKlK0XFXTJ6YTh8xf01BgvCpVSzBAdsU1eNWAu/zmWpkI3 9p1TCdObRxhi5YMuGSLQ1T3HSRXNro+TSsI76hsUHs5DW20YstK9espDPHnwdEXtao3l lajZuuPnoZK/Rm/vt/YViwwLTzZW/RtdXkdVPp+DvpozOWqhFL7zaeOqjL7CkS5IB/Uq XEBgboArtxfsBttqiHviM+ZB06ohsgBSRsBrlb3EsBlg+YZoeTyyZ3T9K3z2UfsI7PLd kbmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1676884624; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OSu6LoDi3bt+On14RPrDb/428OOXZNPo77LIFm1Bi28=; b=nv6rGaYszQDymq52HuwtNo23SNtEoqL5HECYllCrs5uMZAT7d+S25mtad6sJzAhZLR hw2oSaAGQar4qquBicabeWVZKUL376LeBzZhtKrnNR8VGti/wuXtABNzWmlJo+BWeda9 WEfj511maK/TrWI9seXCHPycbVDhIdBRpphCmnVSO//m4qkqM8c8/l/f3PwcXoL4OlML m7QlhclL9U3FWmehPY8Qyv7mUOnWnURwDlx9piNLEK7JvM6HVbcDOjFHiplVGyciQVBW Wc/i45bKyfWToHkpVDFmCzPwnY3Z0x9kFJiMcj6/LDjt/fDj8z53t/M3mYyTC/UQ9o9P nJcQ== X-Gm-Message-State: AO0yUKWX2wpCqc/KywFkUmZS8dRKOflQOHNHalFOcYLriH97t7wzOl08 DrNw518wGRqttQqcOncMENmRIw== X-Google-Smtp-Source: AK7set/Qfx+rPSFCLub0/LUO6j0gUaT7gq5PlaRfKArK9fwOowwRmHnhYd2zrA9jqT1wfk9fQyhMBA== X-Received: by 2002:a17:90a:bf0d:b0:233:a836:15f4 with SMTP id c13-20020a17090abf0d00b00233a83615f4mr528696pjs.1.1676884623852; Mon, 20 Feb 2023 01:17:03 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id fs13-20020a17090af28d00b002339195a47bsm432382pjb.53.2023.02.20.01.16.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Feb 2023 01:17:03 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: tkhai@ya.ru, sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 1/5] mm: vmscan: make global slab shrink lockless Date: Mon, 20 Feb 2023 17:16:33 +0800 Message-Id: <20230220091637.64865-2-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230220091637.64865-1-zhengqi.arch@bytedance.com> References: <20230220091637.64865-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 20EA780013 X-Stat-Signature: kw9k8qeq6hoxbbzd5ypx8i5skabpqf3r X-HE-Tag: 1676884624-264074 X-HE-Meta: U2FsdGVkX1/8tWpjXpApI2cjVEwDbk6zsu4EN6QIPHQPfp8DtqASypVvNyCLkEv+2emVoetupmQZuLBgoDNoh/bJY3W2rLDeNFjDRhnjmxX0ffJUP1V/Y/T5UtGJeIYAT8/qYw6TszQquWNNO1k4EkkaQIJc+U3v+sJvyPrqqcJY8n6yTBoXlS0LXp7mzeYnHVBmYSWnnyz6nwTfcFw+XTj5bD2Ip6F0WCyS1ygfrHe/3WZMVieY/C/bDLS0y/ybw8c7yAmsUXN5PXfZj00jpnTLkyoDmlxRRDDVUBQmL4EX41Rk7VOkGXrdykU9VvnTJ+JAOEfdpCmGHhkQZqU/N8jvPDOUTASaoemuDg36PkH8FByFGAnudwpw1p5ugK2ruK8Pg9onX5FRqkjdvZIOAAnh7Mt5qRIHJ7erNsNWCfGDBdL9lm+k/lwAgEJHzzaijXxoxgBdXktWPmmlmVH08Wgh86WVOx9s3cQEFeNRL/USZa4rMLH6ZipfOcPz1OXthDDRFVNQ6PRxi4CakDMCjGP55k4hT32yoeGo522GYW0DzG+6lBuRQoK78hXcZiWrD5i/dyjtBbfDcsuHZB6OV0gvsxEy8f18UCT+iN105lRoLVq5dhyn9NqwLdg0h+29Uhk6QR04VNBhhoWMlbo6PiBcDITK9pcf0T2qoS5iSCGHjvm5Ut/y15DLCyn67GvJ0BDwv4AWgZw/uS3bptg40TckLILIjNQOZIe91tAzgGK/xYSWzmpkReVnxFKP9Ptzo0HrL6fNAcvJkrePJl76FO7NDpq84GXvGfNb3MPSvw0oN0O7jcVuSyo+486+54dg/SeF8JPLuVyCd2dt20NW9ALiFmDy5PnPpUmJGCBNAXSnVXALR2EKIZNKc8TPbsS4H7tPtic+ss16j+tNOpWoL8+rMc2Y2Vl1tmEp3GlQLnys6FvRonY7uo/12glf30vwpoX0vM8vnpC6Ua5RCvJ iwRgJ2z6 wDovYohnaitPmLneYBGESRMZRIwquGeO8jKlRvuJLb2ooM1wn2y/Bj3hhjTG9fw+waTu9U8j06yeWFI2kg14WwJC1qGLwTLEOkAJgVpfJDXbEnjtlXBn/XRBr1EiRdQCI0suKYZnf58GqsfiCYTz4pv0jYS42tLvxMz1tM7hMTCNKmXTjKgTo0NcJSDlXf3MyQw/jkP2IVubeeCvp3GQkMUxkwRYu9JNMCrmeWzxMCk1UJ4SQB40qbaFvmhmQjyxK6Co37KIAPMyw+TXslp6/1icUeZEk8dhXjsobDeSN8Ekt6ZHUO3a3NC/qNzOL86OPs1bhpY3sGffNEsiGOF/5U3f7/0wAEIkUpn4ix6fZ3IwqiMIRGfOz1yCKGr4Y5Q8VmF6gfWkTnAtNnDjkYKS8vOu4uw+I+7koO06U0MOvUqCZlsqflPa9Naj/2hPBOGTQkGYyM6TnB+T0cCkgh1iPzWmcuGSuko0YD/si+eG2Fou/e+Sr3LrRNe6SLVpXAutk/ihZUeWm0XFrCxQBxJonRFLteKlY1HtvvfTwrDmC84KmpcycTyNsxgqWdSqLQJl8+PYtXneO8rWU0QhpcQDGh03JGIjAqVcJqiM54EfSWKiP9K3k/HI110mgtYIr7+Z9PYf7VuMvE4YhrYupgbLvW75ztZ8gi9nRCNmXxXr+75LPdcZBVa4w8TnTzLbCeOto8LWepikAk8UeqhrFCdt2O0Yrkq0DttKZD/Rp X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The shrinker_rwsem is a global lock in shrinkers subsystem, it is easy to cause blocking in the following cases: a. the write lock of shrinker_rwsem was held for too long. For example, there are many memcgs in the system, which causes some paths to hold locks and traverse it for too long. (e.g. expand_shrinker_info()) b. the read lock of shrinker_rwsem was held for too long, and a writer came at this time. Then this writer will be forced to wait and block all subsequent readers. For example: - be scheduled when the read lock of shrinker_rwsem is held in do_shrink_slab() - some shrinker are blocked for too long. Like the case mentioned in the patchset[1]. Therefore, many times in history ([2],[3],[4],[5]), some people wanted to replace shrinker_rwsem reader with SRCU, but they all gave up because SRCU was not unconditionally enabled. But now, since commit 1cd0bd06093c ("rcu: Remove CONFIG_SRCU"), the SRCU is unconditionally enabled. So it's time to use SRCU to protect readers who previously held shrinker_rwsem. [1]. https://lore.kernel.org/lkml/20191129214541.3110-1-ptikhomirov@virtuozzo.com/ [2]. https://lore.kernel.org/all/1437080113.3596.2.camel@stgolabs.net/ [3]. https://lore.kernel.org/lkml/1510609063-3327-1-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp/ [4]. https://lore.kernel.org/lkml/153365347929.19074.12509495712735843805.stgit@localhost.localdomain/ [5]. https://lore.kernel.org/lkml/20210927074823.5825-1-sultan@kerneltoast.com/ Signed-off-by: Qi Zheng --- mm/vmscan.c | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 9c1c5e8b24b8..95a3d6ddc6c1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -202,6 +202,7 @@ static void set_task_reclaim_state(struct task_struct *task, LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); +DEFINE_SRCU(shrinker_srcu); #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -699,7 +700,7 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { down_write(&shrinker_rwsem); - list_add_tail(&shrinker->list, &shrinker_list); + list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |= SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); up_write(&shrinker_rwsem); @@ -753,13 +754,15 @@ void unregister_shrinker(struct shrinker *shrinker) return; down_write(&shrinker_rwsem); - list_del(&shrinker->list); + list_del_rcu(&shrinker->list); shrinker->flags &= ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry = shrinker_debugfs_remove(shrinker); up_write(&shrinker_rwsem); + synchronize_srcu(&shrinker_srcu); + debugfs_remove_recursive(debugfs_entry); kfree(shrinker->nr_deferred); @@ -779,6 +782,7 @@ void synchronize_shrinkers(void) { down_write(&shrinker_rwsem); up_write(&shrinker_rwsem); + synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); @@ -989,6 +993,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, { unsigned long ret, freed = 0; struct shrinker *shrinker; + int srcu_idx; /* * The root memcg might be allocated even though memcg is disabled @@ -1000,10 +1005,10 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (!mem_cgroup_disabled() && !mem_cgroup_is_root(memcg)) return shrink_slab_memcg(gfp_mask, nid, memcg, priority); - if (!down_read_trylock(&shrinker_rwsem)) - goto out; + srcu_idx = srcu_read_lock(&shrinker_srcu); - list_for_each_entry(shrinker, &shrinker_list, list) { + list_for_each_entry_srcu(shrinker, &shrinker_list, list, + srcu_read_lock_held(&shrinker_srcu)) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -1014,19 +1019,9 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; freed += ret; - /* - * Bail out if someone want to register a new shrinker to - * prevent the registration from being stalled for long periods - * by parallel ongoing shrinking. - */ - if (rwsem_is_contended(&shrinker_rwsem)) { - freed = freed ? : 1; - break; - } } - up_read(&shrinker_rwsem); -out: + srcu_read_unlock(&shrinker_srcu, srcu_idx); cond_resched(); return freed; } From patchwork Mon Feb 20 09:16:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13146179 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71837C636CC for ; Mon, 20 Feb 2023 09:17:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EAC126B0073; Mon, 20 Feb 2023 04:17:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E5CB76B0074; Mon, 20 Feb 2023 04:17:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D245E6B0075; Mon, 20 Feb 2023 04:17:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C50366B0073 for ; Mon, 20 Feb 2023 04:17:13 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 9F2C71C41F0 for ; Mon, 20 Feb 2023 09:17:13 +0000 (UTC) X-FDA: 80487116346.21.1AE36A6 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf12.hostedemail.com (Postfix) with ESMTP id C834C40002 for ; Mon, 20 Feb 2023 09:17:11 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=dzq60rTh; spf=pass (imf12.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676884631; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Hay0adYlVwY1IjdRfk5CKFeKCnG2WTWsVt3L/E4Vszc=; b=TsumGqMkVlIM10OEgPtG2xwfgLTlMcTce0fBo26caPpM4ehrqVdBhBGX64yCqfH30xx1cl Sb+/K1pxHv/H8hQCiQTzAEKXmILvDI4xvAvqIikEgAgpyHwmrEGR2DXBG688VBnlJQw9YX uvGFtC34BnV3L58zuI0JjhHNVfBjAMo= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=dzq60rTh; spf=pass (imf12.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676884631; a=rsa-sha256; cv=none; b=mMGLTk28gJIT9bBg+BEK8cy2nbYeGj8AUh82TC59PSiCCK36L1IFZ5HitvH+a3frUN7PRB xRz5Go00MuwMEzIywfCE+Kpg5FL/r4Jnc4gtXfDwtU9dHKqHPhwTMfh9Lml0L6scjqNXmJ Uf9M64gcgM87gk8k+twMn+4d3z0574w= Received: by mail-pl1-f171.google.com with SMTP id h14so754299plf.10 for ; Mon, 20 Feb 2023 01:17:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; t=1676884631; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Hay0adYlVwY1IjdRfk5CKFeKCnG2WTWsVt3L/E4Vszc=; b=dzq60rThnG1eHPZmbaqIAdKnvx11jcx+9c7+Xz+Fvb8jzUNuPvxrvvoH9rxiY7bseY MwBIqKmGnnRLy9oD7KNU0IP05L8avR9WGkGCuO0KBPHzABJt+tBiqxGIQYJRYUEQ7wdc +CCtYi1ncziYhQ42cmwAoqIuU1fou5tShKZ0YV8xXDToTIOwyrRgdIk32smyOhodVlyW 8GS5Jc0N9MoRt/38dCyYzVz+zoLJ6UwCIwnPMbV9kDCk/1kPdWTnY7K7PljNojAlUkBH R//DTbH1un7fzgI87WSVUXn8bEViUtn4XHEVe68NYl5dLaXaPf0wAFPKmBBOQiZAvtXk j0mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1676884631; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Hay0adYlVwY1IjdRfk5CKFeKCnG2WTWsVt3L/E4Vszc=; b=IgtvR5TWASmSL3UcDA2r9IBHVblUsay9FFe3+/UV0od7GzE0Y5LVAz+rhzS1aZuW6M jGl4dadIrpre+xnDIfd7eubfHNV9VWzCiphs2w3m/vsL3LUoLS5GfX6/N9ycs1yZhCJW 4u8Y7ZXqCFamADr+lc8Q7gQDG+ewni8+n8BgIYQzEa9FUF4c5yISIBEX3W4RmsGkskgD tRxtCynwWnYphBKMgChtDOZmUr5BlEvZ/K7HC/ie0ykIwF/HfA5j1GwwI+aUfrHi7L8f CGJ1Bh234dCI4N+yM8AWvj4O/GuvDFtUISGyOJsaPZXpdQFPqXRkAeM+Y7kRsL902n8e qJpQ== X-Gm-Message-State: AO0yUKVnquRSIXY2uqiryvoqiRYJWV+IFTU/xn+sjs14ujz0MVXO+8// 4RCStUQt8R1ErYUAJgR0WC1wMQ== X-Google-Smtp-Source: AK7set+RFNh5qhT4uR8QO+C6MsDJpFzqsI6Ran0kA2qMXuWKiMKjLinzDtbzLeMxRFj33Xvv1RqpTg== X-Received: by 2002:a17:90a:ac0a:b0:230:9ae4:b5e2 with SMTP id o10-20020a17090aac0a00b002309ae4b5e2mr566508pjq.0.1676884630706; Mon, 20 Feb 2023 01:17:10 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id fs13-20020a17090af28d00b002339195a47bsm432382pjb.53.2023.02.20.01.17.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Feb 2023 01:17:10 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: tkhai@ya.ru, sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 2/5] mm: vmscan: make memcg slab shrink lockless Date: Mon, 20 Feb 2023 17:16:34 +0800 Message-Id: <20230220091637.64865-3-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230220091637.64865-1-zhengqi.arch@bytedance.com> References: <20230220091637.64865-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: sftgu3r3toz49tsphjhz983n8xgghb9q X-Rspamd-Queue-Id: C834C40002 X-HE-Tag: 1676884631-223200 X-HE-Meta: U2FsdGVkX19qhBR3q9ILE8X9Nu+AX3gHXMLu99N2jbZL8XpRH7JRhAaRzgp0nn0qOuAfPoOSeVdSpM6Ea+EBUTO8Es7qJTvBfmMcLU0QeiW9SJ9B9xrnDuu1MUFua2BUspSkT/xmK0wE1TSxSlOCyxtR6RNgfDoCgCXkUBI8KR32UdwkNB59wc5FOcmcvxQSDAevMLjcj4GtV5ru9FxBMyz2pHmmONtrmveH6DZYh5o/uHXEq0V/+xHW6GpRUrsXXIPLaBi13h8UX0B3md1MUOnotHyxnJ2GGqT6rNBcUk/8ohIiu4Z3wTMTIL8OskGlr6+8ERxH1QcSvEraG+CdBebi6A4J6Vcl/8LxYrdiPLzA7dg9fuuTBB5QNplgPifN/Qps5/lsZcNx0KyKz9GoyqUOQCvODkV8hX4GNvLhB3FFqa7mCcBhB1t3AYkcinvVK20VGN9xtVxcBv3cbd1+c+Rn8BjdkdM6jSrTtmn5zHTjJe8nBUWeiRZZ0YqWur8vDpvTUo2JXD+QF84FqvASNOkt8WnEWWVxYmRJN6NTzKCTMzeGTyzqDdneRDyv0hupxdolKXu/EpkFuEpsg60f3pQS3g6J6ceMe/0dRKogJjsYyX91GSI+ICOD/ugCjtYzCJcKVYCkS4+661y2TbdFLsd/JPMoeQ0ymuHHksB6hsPx71NW3AFcAyeEXNQapzN78UqMBkHL/4pKfAq1JD3d7dfb1zxzv5Tga/6Z2BnwMUvsySFStl/nhUvP3JXdX1BIcrKPMz/yOpN9EgkOrEiWLIb3D3XxSnJX9yE7yGCGXv+TJWJc+Ly0eCTGZpIpSzGLI8tvrGOq+exM/NmGjmVvilFSjIWTr/Zzs/BuIsJx8sLUo3wHig5V8NzPeS2m3LFHWInOZ9pirgipR2jQHPDHCnevCpIqw2a1p17Op/VptbC5tfygy9Wu3IUgf4TdKmJzPPElzJChj8VTebjO0my Gew0piio xwOrnTfVrXDzAn74elRee1TJPmGOSPiUJ8yrA4M+vVGzzwVCyaNeaEmHUBhGmT75R7DGNw1jvJmPoan18h8AhGLox+SUnGVQbOgvWrxPA0Fk382gilbvaYuaLFgx80BKfsiGq/lYV330wKLjTEpYGpTJoku97DvgaX4dwbWxOAd4EpSSSSKsOop50so1NrBLtMVqbbZKWuObbseOjW1wM+xF5Kk8AZMy9Egm5TRYyXeOSagTfmKOXnpCBC0R//PWWI5GMQegeS45q7KB/vMNPUx6pfxdjyJVBkYlebwEPGbGpvdc0ifNc/TFSFOOeFhVHWsieSVr7BFsCb8QxOB1kLQ+WOsk6OspnBuNLo4Fou3QzYCouibciV43fOTGpD8ZUTFgBNXsKipAOFCbh6k/WUu/AvSDtL6QzR8EyR4g4Ktct5boBvfydSAdDV0s8Kkw156z8eM/V5GtCGfXGLN1QE1SRLl3rNFvUb5qWKwo82ijTYRg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Like global slab shrink, since commit 1cd0bd06093c ("rcu: Remove CONFIG_SRCU"), it's time to use SRCU to protect readers who previously held shrinker_rwsem. We can test with the following script: ``` DIR="/root/shrinker/memcg/mnt" do_create() { mkdir /sys/fs/cgroup/memory/test echo 200M > /sys/fs/cgroup/memory/test/memory.limit_in_bytes for i in `seq 0 $1`; do mkdir /sys/fs/cgroup/memory/test/$i; echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; mkdir -p $DIR/$i; done } do_mount() { for i in `seq $1 $2`; do mount -t tmpfs $i $DIR/$i; done } do_touch() { for i in `seq $1 $2`; do echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; dd if=/dev/zero of=$DIR/$i/file$i bs=1M count=1 & done } do_create 2000 do_mount 0 2000 do_touch 0 1000 ``` Before applying: 46.60% [kernel] [k] down_read_trylock 18.70% [kernel] [k] up_read 15.44% [kernel] [k] shrink_slab 4.37% [kernel] [k] _find_next_bit 2.75% [kernel] [k] xa_load 2.07% [kernel] [k] idr_find 1.73% [kernel] [k] do_shrink_slab 1.42% [kernel] [k] shrink_lruvec 0.74% [kernel] [k] shrink_node 0.60% [kernel] [k] list_lru_count_one After applying: 19.53% [kernel] [k] _find_next_bit 14.63% [kernel] [k] do_shrink_slab 14.58% [kernel] [k] shrink_slab 11.83% [kernel] [k] shrink_lruvec 9.33% [kernel] [k] __blk_flush_plug 6.67% [kernel] [k] mem_cgroup_iter 3.73% [kernel] [k] list_lru_count_one 2.43% [kernel] [k] shrink_node 1.96% [kernel] [k] super_cache_count 1.78% [kernel] [k] __rcu_read_unlock 1.38% [kernel] [k] __srcu_read_lock 1.30% [kernel] [k] xas_descend We can see that the readers is no longer blocked. Signed-off-by: Qi Zheng --- mm/vmscan.c | 56 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 32 insertions(+), 24 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 95a3d6ddc6c1..dc47396ecd0e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include @@ -221,8 +222,21 @@ static inline int shrinker_defer_size(int nr_items) static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, int nid) { - return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, - lockdep_is_held(&shrinker_rwsem)); + return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu, + lockdep_is_held(&shrinker_rwsem)); +} + +static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, + int nid) +{ + return srcu_dereference(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu); +} + +static void free_shrinker_info_rcu(struct rcu_head *head) +{ + kvfree(container_of(head, struct shrinker_info, rcu)); } static int expand_one_shrinker_info(struct mem_cgroup *memcg, @@ -257,7 +271,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, defer_size - old_defer_size); rcu_assign_pointer(pn->shrinker_info, new); - kvfree_rcu(old, rcu); + call_srcu(&shrinker_srcu, &old->rcu, free_shrinker_info_rcu); } return 0; @@ -350,13 +364,14 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) { if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { struct shrinker_info *info; + int srcu_idx; - rcu_read_lock(); - info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); + srcu_idx = srcu_read_lock(&shrinker_srcu); + info = shrinker_info_srcu(memcg, nid); /* Pairs with smp mb in shrink_slab() */ smp_mb__before_atomic(); set_bit(shrinker_id, info->map); - rcu_read_unlock(); + srcu_read_unlock(&shrinker_srcu, srcu_idx); } } @@ -370,7 +385,6 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) return -ENOSYS; down_write(&shrinker_rwsem); - /* This may call shrinker, so it must use down_read_trylock() */ id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -404,7 +418,7 @@ static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, { struct shrinker_info *info; - info = shrinker_info_protected(memcg, nid); + info = shrinker_info_srcu(memcg, nid); return atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); } @@ -413,13 +427,13 @@ static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, { struct shrinker_info *info; - info = shrinker_info_protected(memcg, nid); + info = shrinker_info_srcu(memcg, nid); return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); } void reparent_shrinker_deferred(struct mem_cgroup *memcg) { - int i, nid; + int i, nid, srcu_idx; long nr; struct mem_cgroup *parent; struct shrinker_info *child_info, *parent_info; @@ -429,16 +443,16 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) parent = root_mem_cgroup; /* Prevent from concurrent shrinker_info expand */ - down_read(&shrinker_rwsem); + srcu_idx = srcu_read_lock(&shrinker_srcu); for_each_node(nid) { - child_info = shrinker_info_protected(memcg, nid); - parent_info = shrinker_info_protected(parent, nid); + child_info = shrinker_info_srcu(memcg, nid); + parent_info = shrinker_info_srcu(parent, nid); for (i = 0; i < shrinker_nr_max; i++) { nr = atomic_long_read(&child_info->nr_deferred[i]); atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); } static bool cgroup_reclaim(struct scan_control *sc) @@ -891,15 +905,14 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, { struct shrinker_info *info; unsigned long ret, freed = 0; + int srcu_idx; int i; if (!mem_cgroup_online(memcg)) return 0; - if (!down_read_trylock(&shrinker_rwsem)) - return 0; - - info = shrinker_info_protected(memcg, nid); + srcu_idx = srcu_read_lock(&shrinker_srcu); + info = shrinker_info_srcu(memcg, nid); if (unlikely(!info)) goto unlock; @@ -949,14 +962,9 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, set_shrinker_bit(memcg, nid, i); } freed += ret; - - if (rwsem_is_contended(&shrinker_rwsem)) { - freed = freed ? : 1; - break; - } } unlock: - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); return freed; } #else /* CONFIG_MEMCG */ From patchwork Mon Feb 20 09:16:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13146180 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB729C636CC for ; Mon, 20 Feb 2023 09:17:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7F5046B0074; Mon, 20 Feb 2023 04:17:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7A59D6B0075; Mon, 20 Feb 2023 04:17:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 694D36B0078; Mon, 20 Feb 2023 04:17:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5A8406B0074 for ; Mon, 20 Feb 2023 04:17:20 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 289FEA0EEF for ; Mon, 20 Feb 2023 09:17:20 +0000 (UTC) X-FDA: 80487116640.28.DBDA568 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) by imf28.hostedemail.com (Postfix) with ESMTP id 4BA05C0002 for ; Mon, 20 Feb 2023 09:17:18 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b="pwzSI/vf"; spf=pass (imf28.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.49 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676884638; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3zoWZenRiYoNEE8cLwabKUnZ2EMcSBm37bQ/nay5M2Q=; b=BTaTBjI350efGhMWsVHyQ/pLeebrDxGTURtN68apjiGzJjj+cX9sEfLyG2IvciCvRb/hcD MkYw0x1syCcIh9ctUjob+PbW9G4eDAL1deYgqIKjOhjAjGYKb/dAGItGLM0Qo/bYmNc2qg Es47pav0oHxpygkiKT7f7Is11gbwIlY= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b="pwzSI/vf"; spf=pass (imf28.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.49 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676884638; a=rsa-sha256; cv=none; b=n5E6g0KmtGVKD6fHHrlnDpgjiQB0fd0eyi1kheO7iU9eYmLvMND+DeeUrvf59bDub3fIZk 0o7A4SZjhxbrPPnWpN/X1mOxGl20jYd5JzjV5bVyaZEBfEqMU9kBs7bXpN7PfAX4C0MKZE VZx9yE3Aa+6UFOq1s6Q8elpSIVf3EfY= Received: by mail-pj1-f49.google.com with SMTP id pt11so1278302pjb.1 for ; Mon, 20 Feb 2023 01:17:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; t=1676884637; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3zoWZenRiYoNEE8cLwabKUnZ2EMcSBm37bQ/nay5M2Q=; b=pwzSI/vf4Vj+GnZxNmyMXkgptBLDA76ZJ1o4OmWFlr6dQarsPx2C++HYAz6Dpn2Aht AJqCW7TXaRjGCwXrIkRHBWZUa/9Uq4ShcEkudpvCqCUfsPPUCgYIFXYWtN9++5OxpAqO gLMuad7DyF/pXHMjpwmBdYV9XSfe7U12nWp4ShfpW3jKBm3KNO+XTZT5cAaCcrQzzIV1 oqisd022sPfT5wczUD241WMl4IN/0R3a2LXx3Bxnlvt0dMus9chI20ZmPMJtAtZCf//h 6YyC4HjkwgSHI06kMieLeE16TKJuxDd6RvwRYYtadIFV1TOrOtPRFHlt9t2tMxB1hTiB KYrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1676884637; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3zoWZenRiYoNEE8cLwabKUnZ2EMcSBm37bQ/nay5M2Q=; b=L+JEpRDD+bv6IGylhSEQrx7DQimXOKN/k8MFMkJtynsFwKcvdkPEO6DCTFEBB4/3Xm AZgkfhwi5LtCoOGEUqC37xXitpZ7JDRFGf6Fy6BnR2p0cWvs4Nk80RYpXJXAc6/FNBWL myHeSf7pvTwP88D6E/u5aYjR8V39ZuPvGRRvIFICvJ701gjVodRCLVYEBpYJpMPuEJDU 1PsSiXRDuFCjj3sqk5lFhlF2du5ISEUrMXrjGMQ7LjcPZVAEeA7c96Al2BeLNwcUbpNh spLgmJUAAx22QTPHibg6CGwf7KHsnKGnYRukt54lEUQ71WzyxS9UC5rZy376VOm25mgv y9wg== X-Gm-Message-State: AO0yUKUHa5SYQJv7NciZnAe8LEB+Upg940pvUCJ72v8zA8otSCqLVrJc paobGcxHtXzc3CBfq03bnZOreQ== X-Google-Smtp-Source: AK7set/u/nMGrcmsUwC5AW5uACSHbng2msaMYYygQeSDbSQ22MPUA9tjyQkhFmQoA6dgXKjSY0gUIw== X-Received: by 2002:a17:90a:c915:b0:234:175:839a with SMTP id v21-20020a17090ac91500b002340175839amr660167pjt.4.1676884637069; Mon, 20 Feb 2023 01:17:17 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id fs13-20020a17090af28d00b002339195a47bsm432382pjb.53.2023.02.20.01.17.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Feb 2023 01:17:16 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: tkhai@ya.ru, sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 3/5] mm: shrinkers: make count and scan in shrinker debugfs lockless Date: Mon, 20 Feb 2023 17:16:35 +0800 Message-Id: <20230220091637.64865-4-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230220091637.64865-1-zhengqi.arch@bytedance.com> References: <20230220091637.64865-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Stat-Signature: ncrb45sg7ocf5myezi1jsfmd45thrxis X-Rspam-User: X-Rspamd-Queue-Id: 4BA05C0002 X-Rspamd-Server: rspam06 X-HE-Tag: 1676884638-172083 X-HE-Meta: U2FsdGVkX19/fL/JqZ0P1tfP4d5bniLP9rrSdZjdomEk2JNgKvZH0T6ick5c9xgy+CByFEbqPn+3z8ZCupd8UiuRaSWZ6aIvXoGCZlipHu5CTYFJn/JFAaDA2bp2cYbpE6IWEPu0pM6RGfgKlJ5WULBhzfSTiYEqcX1Kb/9mUKdXF2285+QsrxCuje9EXOE0Dk0OCg/xS3A6YAUvEJdkEEvFr7qC+m5uPNy0+PS7Sf0XCAHn/tl/t6xAe9YgolRyYWnMgmdIv6IvXw+C+DFeH/mX5bZ1k6NlMDBT905svaiITxjne2WScW+CcKm7pGAZ6QI3dcjD/avO2/PXkujFpvV33OQ6h6655hAcTsO4URuierXXhkNN5XLbJ5Owf7yGjLo166oT4kDTllHFnZGtg7cs0ot0HCaKEiPp1NhVmhM0OYls3ILA2DoMeEknUm6cX1vwcr4ygDCWbyGsB5aL83pc1cRNd1ZFg3cHL5wNBDtzwudoM942vv0mYraLlxqqLr1rlidt2jbbpjf2g/QOTV1oZh8SuTRbLY3rukOwD0wSfS+iv2WXnvQQa1Sp0o2mcETcdAP2W0NPhpHBPij72QKzjX2XO1Q7OcC/zjPBlXAfuptVzMYkId4x79LVorolb619GLxJliqUHH3SP+OEXE7iT4ORXAw9fLSHCdCg7pkjYCQ+85ZZPKGG2yZALL6XqjMWX7coOgP3VtMW635xS3fJno8WXVHMi2h8wdmLd6xHYfHymmlWjtmAIj0QlceTWMPdizRq5oHFiQbxfDZGN7ocKv7QQ5C/GGj4ZMHSx12aQ4P62rG177tOu6Wa+MHXeB2XDAPLBygK06IQGyNhK2o0sj8FmkFy6PNXZU2OV5D8wNLRGqIGqoqKu7NV0xaCNxKoIh9P2GZyFv7I0CNr/nA/zDm3LEOxpWvvDMp5ahl8HSP1pbykMi7JU+ACyXg1sf81hKe2Qctmwo0ijNL OblsGDQ7 ejdzfb/jvbuReh1VReL1Jua5WGH/VPlIWRFbsukxXWzTw4pF8ozUR8lt8a834iijP7BzG/zkMuSsDHCkZzQ/tTc71DkwHZH9YvZhiozqknjtL3hsMSkQHyECPN38+hAhNYBSCRczN+R3y5vU9QWkuUeZyFJE3fiBSW08zCHoWCAE7U4KBgfy12Eaamz6CblBVdHAsEb/7P/+iwqQan2xgAkm/ycLXsmAU+BBDhihdeOh593H61U7Ez8mwVQDY/NiBuW6fLYRdvnWpFWjqQnscKADkM6FjejZutncVu1P8cZDJyxOcJRkcaP/UFMo8Vzl/aoTeV18pUhFIL5E3uxC76yOUZ0BZ0AvmuEkDVewSJ9Gf0JVBS7ZztHuGM/0lUe0mIs7sJuVTmp0jlW68qGWIyAlOjFQQCORjgirtTWVd3ZSBWsql3s2XGM5tyrSjU0SmXlGF5+aQplPNOhcnSlhLpUa/Rc48PD8Q87icBq4AXAVVSts= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Like global and memcg slab shrink, also use SRCU to make count and scan operations in memory shrinker debugfs lockless. Signed-off-by: Qi Zheng --- mm/shrinker_debug.c | 24 +++++++----------------- 1 file changed, 7 insertions(+), 17 deletions(-) diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 39c3491e28a3..6a26e8ac40aa 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -9,6 +9,7 @@ /* defined in vmscan.c */ extern struct rw_semaphore shrinker_rwsem; extern struct list_head shrinker_list; +extern struct srcu_struct shrinker_srcu; static DEFINE_IDA(shrinker_debugfs_ida); static struct dentry *shrinker_debugfs_root; @@ -49,18 +50,13 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v) struct mem_cgroup *memcg; unsigned long total; bool memcg_aware; - int ret, nid; + int ret, nid, srcu_idx; count_per_node = kcalloc(nr_node_ids, sizeof(unsigned long), GFP_KERNEL); if (!count_per_node) return -ENOMEM; - ret = down_read_killable(&shrinker_rwsem); - if (ret) { - kfree(count_per_node); - return ret; - } - rcu_read_lock(); + srcu_idx = srcu_read_lock(&shrinker_srcu); memcg_aware = shrinker->flags & SHRINKER_MEMCG_AWARE; @@ -91,8 +87,7 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v) } } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); - rcu_read_unlock(); - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); kfree(count_per_node); return ret; @@ -115,9 +110,8 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, .gfp_mask = GFP_KERNEL, }; struct mem_cgroup *memcg = NULL; - int nid; + int nid, srcu_idx; char kbuf[72]; - ssize_t ret; read_len = size < (sizeof(kbuf) - 1) ? size : (sizeof(kbuf) - 1); if (copy_from_user(kbuf, buf, read_len)) @@ -146,11 +140,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, return -EINVAL; } - ret = down_read_killable(&shrinker_rwsem); - if (ret) { - mem_cgroup_put(memcg); - return ret; - } + srcu_idx = srcu_read_lock(&shrinker_srcu); sc.nid = nid; sc.memcg = memcg; @@ -159,7 +149,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, shrinker->scan_objects(shrinker, &sc); - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); mem_cgroup_put(memcg); return size; From patchwork Mon Feb 20 09:16:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13146181 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BFBEC05027 for ; Mon, 20 Feb 2023 09:17:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6208A6B0075; Mon, 20 Feb 2023 04:17:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5D0B06B0078; Mon, 20 Feb 2023 04:17:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 499126B007B; Mon, 20 Feb 2023 04:17:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 3BFFD6B0075 for ; Mon, 20 Feb 2023 04:17:27 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id F35D4160F42 for ; Mon, 20 Feb 2023 09:17:26 +0000 (UTC) X-FDA: 80487116892.15.3760E6C Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) by imf02.hostedemail.com (Postfix) with ESMTP id 27EEB8000C for ; Mon, 20 Feb 2023 09:17:24 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=JxT0r83N; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf02.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.50 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676884645; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+SuOsoPKf5aCTrdN4oo5ZtKUiUHbFHnkKOWfb3ggCgM=; b=WCRSpCtACx7v52npU1m16J4Emc/d6AllV5gtpuArF1Glp3fhhGkH5KNYUoWueqT1atu9aO XpjV80gFELzStMcMbVd0R9/SBamOhTsyInfAUJtaLzZXWEz6b8GN+XytrbzDJZv7m/o8Xg OBVCcFcJP5/bJFD2NHHf4TKFTk5u5II= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=JxT0r83N; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf02.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.50 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676884645; a=rsa-sha256; cv=none; b=TMvRGkXExYf6bXjqlmxhrh66/sthLpGjUzf0NyMTH6ODP8gq3m8YVJAlSgrDyuGmqGcAp4 Ojqi+sR9gUSi3w/AucqPu8vEKyXL4Jjbup5D3YBY6K/svmDey0GRzhvBki7aVBt8+wiJrN sCO+b3IEfAe4usn7EW3ZIRnLxbfWlSo= Received: by mail-pj1-f50.google.com with SMTP id d1-20020a17090a3b0100b00229ca6a4636so787683pjc.0 for ; Mon, 20 Feb 2023 01:17:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; t=1676884644; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+SuOsoPKf5aCTrdN4oo5ZtKUiUHbFHnkKOWfb3ggCgM=; b=JxT0r83NllZWDrv7JUbi57ga92yM+aV9REzj4acVWyDEC0qP6CkyEbEtUoBJBuS9q0 EmuU2mv0gSa+rVWs9ezOVY7JDMeIZ43k4AXpUSAQB6AdIxCSGzjOaPZ6AwD8FZdu+xj8 KqBK8u5KJuURh/jBiBPwgQcLVYo8QdvgjooE8iFX5K/C+5c1neFrb20RGiwO7pLYhkpx Pm7zBDLdADM1l0+KQy41Ws2iuIjA3OuXyqw27+xYBk0HctZU8tWkikX0xOU4A6qvcPuI IeHxUsqC/EryGaQnmdCJLqENq3+l53MZEuHQ8WxAIARks9MMf6XnteE6OVTM6echH0ty jkKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1676884644; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+SuOsoPKf5aCTrdN4oo5ZtKUiUHbFHnkKOWfb3ggCgM=; b=DMJkYxOXCOFKMGtaH2+MmnA51Rw76Snbx55dMXB2IiwF63TEgeduNW62BAyFCeq7bj KNXx++P6KhptYHjPQDrBFp1d4Fy9/Fu8eTJOu2ddAXtq4GaC+g9nPKaZ1CW2o2uJKTLy DYHbOWhLviIgjPlSSs0cPB1SbfnwBCrFqiOkv8y2ANNSi+DezKF4WJNAkfXIrWu8CLLT 54aCI5BWE5hYyEBacJ8s6VZUAlfoTd0x0oEBhGABCKh3JyDiQB0JOPW7nXddYCjidopF HLGL9S5CuKAJWWe9Ebqaym0lcFBEu0mm18ERQ8kKj1NkfTyqOE+owROmA/vfpfHFn2VT 8sbQ== X-Gm-Message-State: AO0yUKWwLmiTn9Qs6VOfvSG9PzXTrYCruVobzuVqF3sg6Rl4ybnAIDw5 4pyXJcosN4CK69hMVQiuvmHoNg== X-Google-Smtp-Source: AK7set9Jss6nYpleTWMKLk37JYg5U2t/GE2b3ogkN+P0mI3dj50eWKwdpAoiMz4LEqWxjmoXlXyuDQ== X-Received: by 2002:a17:90a:a516:b0:234:9d30:84ff with SMTP id a22-20020a17090aa51600b002349d3084ffmr950101pjq.3.1676884643904; Mon, 20 Feb 2023 01:17:23 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id fs13-20020a17090af28d00b002339195a47bsm432382pjb.53.2023.02.20.01.17.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Feb 2023 01:17:23 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: tkhai@ya.ru, sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 4/5] mm: vmscan: remove shrinker_rwsem from synchronize_shrinkers() Date: Mon, 20 Feb 2023 17:16:36 +0800 Message-Id: <20230220091637.64865-5-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230220091637.64865-1-zhengqi.arch@bytedance.com> References: <20230220091637.64865-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 27EEB8000C X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: xbt7b7mgrhdccwcobb9fxipby7fgmnzo X-HE-Tag: 1676884644-416537 X-HE-Meta: U2FsdGVkX18WolTxhBsK+QrCO9KPuk02UdqmQU+Rae5WYF8ca/EqPsZ8XlHwQ9Clv+77tviV6a/o4EunOTK/1jMOxZizb/hP5lrHHl4OhQ15Tz6hh9iO3hfPxqEAPyX8N/1M9wIWuPRmwqARRr/zAI2fkjDA9MWrov/bGkwqgvzjdnX1VjfwDltKVe6CgeFhmqGOLGp7N2YIXNj11sbcm2V2ObKYx1GUvAd1YBb0UpW/EepUakKLAyKUXL03jKTBY18Y+kXU1UGyauW0ShezPuJ2tEpUHFeNOPlbnsFoLbTDODaysppD7QphjVxZRyKNmxOdq/M+fqk0mG4Bxydp42tyCjLgsLvnPhA+Ci+lYKn/l3LzGbcVVLKpEkm1fuSu0QfkR7eGDQLFTu1B26z2/GH3w5AMxuDea9ZKednaGpaUneGayaphRco1o7MZVVqdfQF92xbL9bkaFVFWcaQHpJeahxwJDHJGeXr3a24sT1cCKircSSY8tTAoZV5bjTFu+kCF2a/T0yV902T7pXT+4hF1yom9ffU8wujxrRTUVkG34EBrpR5N/C40k+dlwLH1dyNRCSKrv2ABDLbjQFj0tk3wp4DQbT9BHOeKhoeLkmLop0rt91iRLhnJP4L24Ey7x489wNaJTKEWioPl0FDqWZVnRIbo3Ztq5p0o9j/za0LvOzfPMkGeY02KaDMgu+KhQa2ogWn2s89rHYS9nPucgAnyER7+jqyTuTpHbvm+51cQtGVxuE5C4WK3f81CfJ2jn5gPe3htD3kNtT9IyKtlluM7PdMQJQHFLEy8QhEUGKv802ClPho5PdCl8z2qaVJkohiNX0mjkvfza8JphzCjehee1xBH9Dwe+27jUE0dIDXm33XhhTiXmhbMmBIhEC6UvVXdEwlOHTZrobkM1+mpNYQzWRoNvFdijULPVTPvJcs9S2jN+HYmV+gcYGSPAb8Z3bQVJupxXxpWbuRKxV8 GcKhUO9R zeuDlVs8KL4KKAs8cSGZazS/HAT206gpX8GvoVXa9K4oC2cdLvqY9synI16tUQpzoloVsatfh7/weN6FZ1oaNIu7DqN1TsiGO91YqlJHC7BvUUs3FGqQdpSzGkx1mURvWCFLxCqlr62ZTcxJoVn2ruCGJHPWLUuu5dWcjjmglisFns7+9roWx/8dHdlLKqlc/QXK4eO4FKhHcPWIp7gyKpvUyNi331sfIwfh36R6mYqywirXYc5exXa2HGX9BaaRPYBKRRX6c0HoybeD+ccTpeN8tLF71U9WLkZsI/aDyoBICMVKcUq406409vpdjC46wiKSBWYD3cU11aeghUlavc+u6O20vTtFrfMATiBNCkDw9As7eAKCZUBUWoC2WjC/dMSROCR1omEze40H7kNg0Yo7sRZIn7k+nhoIUyboG0VH371re6pRKLDTsU3b1vMSC3Xq1GOFlS+r6NJgZgHic+m7y6k9bSPSoHLquee6PQlFduFaGTESA1b9JC8SU8JPy/ncJSDDG2LomVWgliYqlQ2+Hsw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000372, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now there are no readers of shrinker_rwsem, so synchronize_shrinkers() does not need to hold the writer of shrinker_rwsem to wait for all running shinkers to complete, synchronize_srcu() is enough. Signed-off-by: Qi Zheng --- mm/vmscan.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index dc47396ecd0e..b8da56beba96 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -787,15 +787,11 @@ EXPORT_SYMBOL(unregister_shrinker); /** * synchronize_shrinkers - Wait for all running shrinkers to complete. * - * This is equivalent to calling unregister_shrink() and register_shrinker(), - * but atomically and with less overhead. This is useful to guarantee that all - * shrinker invocations have seen an update, before freeing memory, similar to - * rcu. + * This is useful to guarantee that all shrinker invocations have seen an + * update, before freeing memory. */ void synchronize_shrinkers(void) { - down_write(&shrinker_rwsem); - up_write(&shrinker_rwsem); synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); From patchwork Mon Feb 20 09:16:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13146182 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C81F7C6379F for ; Mon, 20 Feb 2023 09:17:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6963B6B0078; Mon, 20 Feb 2023 04:17:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 646BC6B007B; Mon, 20 Feb 2023 04:17:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 50E6E6B007D; Mon, 20 Feb 2023 04:17:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3F9606B0078 for ; Mon, 20 Feb 2023 04:17:33 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 18450A0EF0 for ; Mon, 20 Feb 2023 09:17:33 +0000 (UTC) X-FDA: 80487117186.08.56CA113 Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by imf22.hostedemail.com (Postfix) with ESMTP id 49B45C0014 for ; Mon, 20 Feb 2023 09:17:31 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=4puz4Jgz; spf=pass (imf22.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676884651; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3B7sNafZtBlgHmh5fpLAa14mnyUBnuiEHhgxOSu4EUY=; b=th/A/KhYDdIjYxa842mBSOQb3FnFLStgBQ6ST1Al3/GabuSkJyZn9nvguoBKhEM8LAmUki fvYf0J1BBCzWROeW6ud1j8Ctth7RCro9jEaYAOaWkC08vf6Y0Bpg4Nn2sul76pgpKsicS2 d8vu97AtKyelQkTabllCYtL1KodrWTo= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=4puz4Jgz; spf=pass (imf22.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676884651; a=rsa-sha256; cv=none; b=he7GmOHYzXfRwAlgbWIenT1FvtT294YKL0PVOb3HgyV6SeZnhMwdAEN8f0IHT6rgi98vmi l2l07QM2AU2vEMXC1XgSTS0xDu2YITZ1cx0g3bCuXeTlHz3qhXl0dAJuyH+2LznanUu0uA LHr6Y8gEOQkcQuqPTxmJ+RcSa0cPiSo= Received: by mail-pj1-f42.google.com with SMTP id i1-20020a17090ad34100b00234463de251so454801pjx.3 for ; Mon, 20 Feb 2023 01:17:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3B7sNafZtBlgHmh5fpLAa14mnyUBnuiEHhgxOSu4EUY=; b=4puz4Jgz93txeceBh3fQ+lEdkKu9A4lOtC1kYgX78UKROjsP3O5+Y11jVMsovlB8oU sXuaRzKA6g9H9MXb3TdaF5sp9f/QMmV6iVKEx444zjffMm5LRNXcvYaal2t+iybJaLjY dOAt6AzfkgtaTHsJmUt79SpRjjglW8OOcC+KtEe+U5VqwR2qSZOga5VUjMCAWJRD00IM JG/qWE7eie2G7sZ4cJOwULL9sO4k8JkKSdEghpO73ZZLKfjRUL1R74gz0k7djhVwirUQ TM5T4chfi9rC4wta62Q+8DdRwAl1M2SQDAocd2oBCkh1qD+vWeFfRIBKkH20GK8x1S4Z r++w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3B7sNafZtBlgHmh5fpLAa14mnyUBnuiEHhgxOSu4EUY=; b=hcP/vn/jDzLz0dfCV40SdWdV/6ty91yGoZ65+/Kll0sAgItxSgtTozRygPSWL48wMa tzpesY0KfXPFyzLXf1PAgYTvutyf24B2jSzXpDC96URp9TquJZLX6jo3Wz87ySARPMhT K447tVrAS2eCaYOWIfi3eo8tJqE7yzptAUEm64ozpX2RG4PkM4UsU8I2Iz7hqIhtY+kp sghZ4kCl97CDbrQApi9a+PkMtg+NZMbOpGLp4HATfjoaeDbQO20vMnlsDWidlPuI41Vp YfHTB3Y2+XGFH3b/16q0ivC4IW8fGIp4+OUq5PAFW4xMfgsd56gybWkbcQvnFY+ulL9V 46nA== X-Gm-Message-State: AO0yUKUvY2NjNtieAyiRvvfUg705306PWRpbglip26QMeHIFM4AC0Ooq 4PvaMyCIGtYVohgqcBDL/1vNoA== X-Google-Smtp-Source: AK7set/FrMA+DzzJjfgDbDVBnD4DP8OP+6IttlJ5FihjDu+ykoY0DseCdfCDQOT7VAaqhwcLBAHAYg== X-Received: by 2002:a17:90b:948:b0:236:7f67:8338 with SMTP id dw8-20020a17090b094800b002367f678338mr365975pjb.4.1676884650199; Mon, 20 Feb 2023 01:17:30 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id fs13-20020a17090af28d00b002339195a47bsm432382pjb.53.2023.02.20.01.17.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Feb 2023 01:17:29 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: tkhai@ya.ru, sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 5/5] mm: shrinkers: convert shrinker_rwsem to mutex Date: Mon, 20 Feb 2023 17:16:37 +0800 Message-Id: <20230220091637.64865-6-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230220091637.64865-1-zhengqi.arch@bytedance.com> References: <20230220091637.64865-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: umgon37h7b1mmjm93zzonrx8gerxjz68 X-Rspamd-Queue-Id: 49B45C0014 X-HE-Tag: 1676884651-940082 X-HE-Meta: U2FsdGVkX1+kXmOIFxj9pgNXmdgh8LFcEdanlbv6uFA0kn2bqBBZrjcVG4fk84WA2PEW84ijW7qZTuMnD67mYWbA2YoTuigovs0iGw4bIBVtyFblyDoNG6uiFIngdOfDfrrPrtLeVnH3r3n/2cEvEBe9MUbZmsSrumkcTFNFB0MShWIgku8EFDywpu73aD06aiTQnI0XPZpqHLQWb9byGsv3CuGib5DNmKog1Yn7Swit5wYAqZK7JYZgxxs+Uni3AiPS/QKdnjufQWNu4XlYVAyBWxMTlh7h03JW3vWjUuuDe9BAH1Qe/v19ERdl+wh5H0Qkx4VuaxZlIBgLaGv3u52d+05mJmepe7IJT62Q6tnD/VE6rFKlOBi7RmXt5YEM8Wluw+aPqUU6MvvVyM/zswFdl/GYWa+/GLtqWzr4n2ui+jhQyvmb1fJG7JYvXKOmA2e+xSHjCztey1FMG3bRi/SsKtb7ZpicZPIKfZRJVpkaU3D1WjtZ2uJfevRpk9kR4izpajXTM8TlgFGoRHM9APTlggLFf8DA6La6/epA9ogpJuvkz/7TxR+Diy7HsYBAda9Ej+YPEBRH4SGekEcUTjYngec4qz1lcMQTzaXIgRP3ie9cEU+nRRoy+MyvHrxBpZKoJecKZE9M40aKG41GG4pzKRG/RaEyFKipoZA6Jh2bXLJo65OG9wOsCRBYKsNFH0D86xo3Escxv1X7yXkFAxBebFp5Nf1/wbcfotuU83Gwi1HewWBgVjAdobJ+DZ/zcvEt9wBnvfuznfjAZOFN9CdZrz8XjAq1meDj5AGGO5aYmSMhqNOrrXSIOh8JGCjRudlHj7n7+QLuN4J6PiezSfPrSO7kjfHbNREi47UVNEvEjJp32DAGX+G76NIeePEJMuWR109dAH9aNxj+U4tyKqHkDtkqprj0mWWWoJ459mG14kDwvjPSlMKVoAYa8FOWz1sRqUSgvH28T1cYPfA nIOOhxg5 UrGrC3rvrF/l4xRGLRmELMcQzufuz+tlACHw0MmoanCiUU1zcdlWBvmJjknDteQAa6L+rnGapV3VmqgTwLkbrwo7MYZrZTSFbp/3abCKveOxVG19HNO9j8fj/xtkfXle//ij0OOAnlpO11H1KDzPb7e7DBeyeG5+OtVMP8etxO1Me0lM5xmtZbBfA71sDCP+grzU+x/DA6BWN+i6xOMs4alEQ+lA4gw4U4047vfOp07dOA8UQFsSPHI9pj+QXQlfDFBRuL7veiRJ9UNko0Spg0GpMb8jqTtzfrLlLOtczht1KKQ+XYiSPtaRQfF45AaVQvdasvbJGr0GzbPrQPcw/Pf6tIs9vc1NP2RbRtfREXqER6JpscHAEueAIcIhOmKSnVWakWB3+oxF0g6EkgrWETfFHoHTGERYTitOz06zGd4yoqJGfoTD5KuBDjzYnQvZwm/mHsg9MhGGRy97evX1YgDaZ+LR+JmbRDJ2Sbk5N05LIwWtdUngXknyTXK379iY9+jU6cYaXXrzRIhsapCWT8uYJhRwXtf6zfOzc X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now there are no readers of shrinker_rwsem, so we can simply replace it with mutex lock. Signed-off-by: Qi Zheng --- drivers/md/dm-cache-metadata.c | 2 +- drivers/md/dm-thin-metadata.c | 2 +- fs/super.c | 2 +- mm/shrinker_debug.c | 14 +++++++------- mm/vmscan.c | 30 +++++++++++++++--------------- 5 files changed, 25 insertions(+), 25 deletions(-) diff --git a/drivers/md/dm-cache-metadata.c b/drivers/md/dm-cache-metadata.c index acffed750e3e..9e0c69958587 100644 --- a/drivers/md/dm-cache-metadata.c +++ b/drivers/md/dm-cache-metadata.c @@ -1828,7 +1828,7 @@ int dm_cache_metadata_abort(struct dm_cache_metadata *cmd) * Replacement block manager (new_bm) is created and old_bm destroyed outside of * cmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of * shrinker associated with the block manager's bufio client vs cmd root_lock). - * - must take shrinker_rwsem without holding cmd->root_lock + * - must take shrinker_mutex without holding cmd->root_lock */ new_bm = dm_block_manager_create(cmd->bdev, DM_CACHE_METADATA_BLOCK_SIZE << SECTOR_SHIFT, CACHE_MAX_CONCURRENT_LOCKS); diff --git a/drivers/md/dm-thin-metadata.c b/drivers/md/dm-thin-metadata.c index fd464fb024c3..9f5cb52c5763 100644 --- a/drivers/md/dm-thin-metadata.c +++ b/drivers/md/dm-thin-metadata.c @@ -1887,7 +1887,7 @@ int dm_pool_abort_metadata(struct dm_pool_metadata *pmd) * Replacement block manager (new_bm) is created and old_bm destroyed outside of * pmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of * shrinker associated with the block manager's bufio client vs pmd root_lock). - * - must take shrinker_rwsem without holding pmd->root_lock + * - must take shrinker_mutex without holding pmd->root_lock */ new_bm = dm_block_manager_create(pmd->bdev, THIN_METADATA_BLOCK_SIZE << SECTOR_SHIFT, THIN_MAX_CONCURRENT_LOCKS); diff --git a/fs/super.c b/fs/super.c index 84332d5cb817..91a4037b1d95 100644 --- a/fs/super.c +++ b/fs/super.c @@ -54,7 +54,7 @@ static char *sb_writers_name[SB_FREEZE_LEVELS] = { * One thing we have to be careful of with a per-sb shrinker is that we don't * drop the last active reference to the superblock from within the shrinker. * If that happens we could trigger unregistering the shrinker from within the - * shrinker path and that leads to deadlock on the shrinker_rwsem. Hence we + * shrinker path and that leads to deadlock on the shrinker_mutex. Hence we * take a passive reference to the superblock to avoid this from occurring. */ static unsigned long super_cache_scan(struct shrinker *shrink, diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 6a26e8ac40aa..4c3a5d9afb5c 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -7,7 +7,7 @@ #include /* defined in vmscan.c */ -extern struct rw_semaphore shrinker_rwsem; +extern struct mutex shrinker_mutex; extern struct list_head shrinker_list; extern struct srcu_struct shrinker_srcu; @@ -167,7 +167,7 @@ int shrinker_debugfs_add(struct shrinker *shrinker) char buf[128]; int id; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); /* debugfs isn't initialized yet, add debugfs entries later. */ if (!shrinker_debugfs_root) @@ -210,7 +210,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) if (!new) return -ENOMEM; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); old = shrinker->name; shrinker->name = new; @@ -228,7 +228,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) shrinker->debugfs_entry = entry; } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); kfree_const(old); @@ -240,7 +240,7 @@ struct dentry *shrinker_debugfs_remove(struct shrinker *shrinker) { struct dentry *entry = shrinker->debugfs_entry; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); kfree_const(shrinker->name); shrinker->name = NULL; @@ -265,14 +265,14 @@ static int __init shrinker_debugfs_init(void) shrinker_debugfs_root = dentry; /* Create debugfs entries for shrinkers registered at boot */ - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_for_each_entry(shrinker, &shrinker_list, list) if (!shrinker->debugfs_entry) { ret = shrinker_debugfs_add(shrinker); if (ret) break; } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } diff --git a/mm/vmscan.c b/mm/vmscan.c index b8da56beba96..f94bfe540675 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -35,7 +35,7 @@ #include #include #include -#include +#include #include #include #include @@ -202,7 +202,7 @@ static void set_task_reclaim_state(struct task_struct *task, } LIST_HEAD(shrinker_list); -DECLARE_RWSEM(shrinker_rwsem); +DEFINE_MUTEX(shrinker_mutex); DEFINE_SRCU(shrinker_srcu); #ifdef CONFIG_MEMCG @@ -224,7 +224,7 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, { return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, &shrinker_srcu, - lockdep_is_held(&shrinker_rwsem)); + lockdep_is_held(&shrinker_mutex)); } static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, @@ -297,7 +297,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) int nid, size, ret = 0; int map_size, defer_size = 0; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); map_size = shrinker_map_size(shrinker_nr_max); defer_size = shrinker_defer_size(shrinker_nr_max); size = map_size + defer_size; @@ -312,7 +312,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) info->map = (void *)info->nr_deferred + defer_size; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } @@ -337,7 +337,7 @@ static int expand_shrinker_info(int new_id) if (!root_mem_cgroup) goto out; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); map_size = shrinker_map_size(new_nr_max); defer_size = shrinker_defer_size(new_nr_max); @@ -384,7 +384,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) if (mem_cgroup_disabled()) return -ENOSYS; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -398,7 +398,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) shrinker->id = id; ret = 0; unlock: - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } @@ -408,7 +408,7 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) BUG_ON(id < 0); - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); idr_remove(&shrinker_idr, id); } @@ -701,9 +701,9 @@ void free_prealloced_shrinker(struct shrinker *shrinker) shrinker->name = NULL; #endif if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); unregister_memcg_shrinker(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return; } @@ -713,11 +713,11 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |= SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); } static int __register_shrinker(struct shrinker *shrinker) @@ -767,13 +767,13 @@ void unregister_shrinker(struct shrinker *shrinker) if (!(shrinker->flags & SHRINKER_REGISTERED)) return; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_del_rcu(&shrinker->list); shrinker->flags &= ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry = shrinker_debugfs_remove(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); synchronize_srcu(&shrinker_srcu);