From patchwork Mon Nov 4 17:52:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13861757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F010DD1BDD5 for ; Mon, 4 Nov 2024 17:55:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6E2E16B008C; Mon, 4 Nov 2024 12:55:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 691C26B0092; Mon, 4 Nov 2024 12:55:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 50BF86B0093; Mon, 4 Nov 2024 12:55:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 28CA96B008C for ; Mon, 4 Nov 2024 12:55:04 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 4A6781C1B37 for ; Mon, 4 Nov 2024 17:55:03 +0000 (UTC) X-FDA: 82749163350.28.B8AFE1B Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) by imf15.hostedemail.com (Postfix) with ESMTP id 9307EA0005 for ; Mon, 4 Nov 2024 17:54:28 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="Tr7Jj/Rx"; spf=pass (imf15.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.182 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730742737; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gRTR4u71/5k0GKWZY0aR2dLAZ3MdIKwjTz5ojnEJCYA=; b=Ims7YZMs9uKovP6LBcpWTda8xf6CRaSV+htaeiaqW2KEslu1VRyQbIzfKgILfj74hwXdbJ 4qyKvEx4Hb+ZGp4gthsMs5AJXsQ98WKos3C7lPg6PjO9m6bie4gqCuGEiNw2UVsvwSJIfo 91Z5WZpbUtk/KVzeR6tJdFTDVrTRQAU= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="Tr7Jj/Rx"; spf=pass (imf15.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.182 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730742737; a=rsa-sha256; cv=none; b=QDlv9j5dokG8LW7ITvLNjSSmJ/coFBlSA5gmw5f3rUzbETN5Nns3jnWlMwIHmJyGZQv+sq 7Vro6BwVGORogmcgLjrAWL6pCGApo0kiNM+heUjgn/Tj1NkAh6Ym+PnHMB+KL3zNCq+Dtz r09egjAFc473FX5bjsG/V2hVlWNO4CY= Received: by mail-pg1-f182.google.com with SMTP id 41be03b00d2f7-7eae96e6624so3290199a12.2 for ; Mon, 04 Nov 2024 09:55:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730742900; x=1731347700; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=gRTR4u71/5k0GKWZY0aR2dLAZ3MdIKwjTz5ojnEJCYA=; b=Tr7Jj/RxjLryYkMX3Vc5ijdL+c0V1122rlbyp0NTL3lESgX2B4Bc2Rr2pW+N66ETV5 2013OT1LMwvTVCGmwdM9StJ1aR/bVriGp1c7BIgyjqQsbTdCs9YreOL+o0Q8zm4dH71z QNQw1n+7VrTL2NSb5Xc//j4T/LWrrlAXAJcjc5SugxURRb0ONw3uKIecO1TVq6pPZXEK jpz6dSPZeY5PKWWdzLMmjpdKPdDOu5FlyhHIhHYxDCxnwS6K4IXEstFQzOfKkVc32A8M MivudrCw45ySNe037wTRXg6OsrLHwKdZ00zJRIhYpIwEhOWZ/Cnvk7yJyQeaJlHH/tMF i8aQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730742900; x=1731347700; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=gRTR4u71/5k0GKWZY0aR2dLAZ3MdIKwjTz5ojnEJCYA=; b=BxAyTv82v/JcuLUyIpAEsVzLEFGGTfm1DdM2Fs12wRrGH+iP5Ui1P95G19WxdcJQ7I HKkpgAs38kWjiLCHv7thT6KLvnWVPGRVYaYNjtv4hxeUXEqWd4lTl+D286XNHWNYgjxO mZplT9ya6/xH77ZaXafjyj+18ejtSvrgHfDeqDtmj3JZFrbcd10tcj/O8h5DI3kIBA5O YGHCFUhTbbSMRT05G6qWsCRectj/zv54NzMe6pWGHhHU3Ew5YTaufOOpG2PScbcGNUXt bCMq+z53mKnmotX7JevcUMwsht5IFxI4cyjLqfT+HB7s2vLBFOMA+At4kcWxIGWMWb/h AWqA== X-Gm-Message-State: AOJu0YzArAZV9e7FYxH4MG3kuF7tVcSUoz8MP4tgxePw6n/5pFTsLlJe FXCuJHcNcAV85NpJdfjWn84NThcZ1rYKluQ6JbCGv1ZOWlALqoLVSYgZtnbXldE= X-Google-Smtp-Source: AGHT+IEH6inVkjKT4+groMvSWdnpVy/1uu73SUBRjO1KRAszhe1IQSdHDoafHlMnPuy75/JgZF9wUQ== X-Received: by 2002:a05:6a20:b925:b0:1d9:6a6b:faf4 with SMTP id adf61e73a8af0-1d9a83d6639mr32856124637.15.1730742899565; Mon, 04 Nov 2024 09:54:59 -0800 (PST) Received: from KASONG-MC4.tencent.com ([106.37.121.39]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-720bc1e5f5bsm7777039b3a.55.2024.11.04.09.54.55 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 04 Nov 2024 09:54:59 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , Johannes Weiner , Roman Gushchin , Waiman Long , Shakeel Butt , Michal Hocko , Chengming Zhou , Qi Zheng , Muchun Song , Kairui Song Subject: [PATCH v3 1/6] mm/list_lru: don't pass unnecessary key parameters Date: Tue, 5 Nov 2024 01:52:52 +0800 Message-ID: <20241104175257.60853-2-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241104175257.60853-1-ryncsn@gmail.com> References: <20241104175257.60853-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 9307EA0005 X-Stat-Signature: z9hagwitdf87qxbfu469o3yzrjb8w49j X-Rspam-User: X-HE-Tag: 1730742868-462590 X-HE-Meta: U2FsdGVkX1+xjnii0uNltr/UmlJZmkDCN1ggaIGDqkZfByLzyT8bYpU0FitLpw0zBBzPHeWeOm56+8wJtJxGpt3H38eWz/4lD/W/btuRUFBScCqbbJ1OjE5+Cky+Tz8erfDN2eKe1yzPr8HZ4vQ+bwlr3wWEoVSdxExllDWNqKt4cgcbHAAtPpgyS47dfpuxZWvMHNoJnBHmlWsmTHOWPYUQtOFlapa98oABNg71EC8nuQ3856PLEhDOnPQx2+kiLPAi1zVjkTv8bbikmbpLcCndr5zefxEgViA4rZt/3t5MjEqWvIIzoVrbphrO87xS7zkQHS25R6j0CI+WmqYWHhV+CueYb8TOIuP6/gAbldcEra/jrwWmtukQIekH3U4q9tns56iAqA5TjqFB6GHwl0U/IUWeNNB+e3I4crwCDXAX+CVUmkROcfk7hScirwO1Q2OIrXcL7hpSjvsux9bJSaSvIW4pQzh1tkCaaS9Iw8FX8N89vDihjGw29gGX1O/roJCQ5MfhR4Dj3sTVY+rME8um3+fsWhENGkfrHctye36PFKGVSyAWQCiw4UxSxNXVTkk0DNOvM990ysYKOKkhMklhzBXFi1f+jxq7pCNODt1lQkiWRnOVEkmuZlrgL2hovycECUFuYnZ0/KZXagHe/OcZc/svQ6dSyofOgbiZcFdAE0f7FyC/6RI3PYJHx+XrrumR7Sf7/FEOjt9rRe22ugRRCnXrkS6YK1OD1B305QgljOZUsiIY9L8Yp5CxNyqM5IKq3tYZkT4RZ8RfczcnFN3LJdQSuOPkPsk/D6tenilzJxOI9KFSNGNdY5UJRksQz6NDdBPZgf5blbHs3jlSaAU0XULJFoTDREJOdlqDllHse3+2dHxxi1OdCXjXhctlaugVdmoUTM/WjXAslQfEUMYM+7PrPEfx3Me1lufGwaQNwKhUAym33BssrsbUDLJWPUYTis51m3TFGVEN79r K1uLupPl +FhDTkl2fhWL++lFjBFvUQwpFEzatrdezS2mydUP6BhGySto/M8rm4lk1HjlNHLGn+Jt28wBgjUZ5ioIyIw1f6OHSzwwvK7iomQL+pJzKesLR8e97TFc40tZImpaQTJyfhxqhrRySLziEE77+7DFIWQyLKJ3UGnGhmkTLhphNz0A5H5TMF1JzhNqloGhtB4hKxGZ4QLpKWIMt5nQwuwq/JMDzaWhwTg6oWYC8DzUY2Znu1QELrP70L0hlRbq/HVer8E8OhQvlHER7S8QdGbQ8VpJiRfSC5Azj5qo+TJ86Zt66q/L6L7T0KBgMYpjTmJJ0jlk67CRYO1DmPManfijmCpgPbWIa9cU9I7lUEMeHti6xlTZQHxS7HVE68iMvH/yySISuc5Wu7ckl97mvn2lmRHYLMuDD7mTNPSpHqWCElP6REl2jOraBoSp15PuvdC7eae/UlhWm+EEpYLThMdcPBT0+6909gG4wwlg2mUDanyz8pp45q3qZ2FAF5r7g/Hgits4b4Sks8rKlddH9jjjtoyCOMKBFgJO/DHh9WHTRHC0DgxMLE6BZvS2V2pa4AI8TD0tC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song When LOCKDEP is not enabled, lock_class_key is an empty struct that is never used. But the list_lru initialization function still takes a placeholder pointer as parameter, and the compiler cannot optimize it because the function is not static and exported. Remove this parameter and move it inside the list_lru struct. Only use it when LOCKDEP is enabled. Kernel builds with LOCKDEP will be slightly larger, while !LOCKDEP builds without it will be slightly smaller (the common case). Signed-off-by: Kairui Song Acked-by: Shakeel Butt --- include/linux/list_lru.h | 18 +++++++++++++++--- mm/list_lru.c | 9 +++++---- mm/workingset.c | 4 ++-- 3 files changed, 22 insertions(+), 9 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index 5099a8ccd5f4..eba93f6511f3 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -56,16 +56,28 @@ struct list_lru { bool memcg_aware; struct xarray xa; #endif +#ifdef CONFIG_LOCKDEP + struct lock_class_key *key; +#endif }; void list_lru_destroy(struct list_lru *lru); int __list_lru_init(struct list_lru *lru, bool memcg_aware, - struct lock_class_key *key, struct shrinker *shrinker); + struct shrinker *shrinker); #define list_lru_init(lru) \ - __list_lru_init((lru), false, NULL, NULL) + __list_lru_init((lru), false, NULL) #define list_lru_init_memcg(lru, shrinker) \ - __list_lru_init((lru), true, NULL, shrinker) + __list_lru_init((lru), true, shrinker) + +static inline int list_lru_init_memcg_key(struct list_lru *lru, struct shrinker *shrinker, + struct lock_class_key *key) +{ +#ifdef CONFIG_LOCKDEP + lru->key = key; +#endif + return list_lru_init_memcg(lru, shrinker); +} int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, gfp_t gfp); diff --git a/mm/list_lru.c b/mm/list_lru.c index 9b7ff06e9d32..ea7dc9fa4d05 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -562,8 +562,7 @@ static void memcg_destroy_list_lru(struct list_lru *lru) } #endif /* CONFIG_MEMCG */ -int __list_lru_init(struct list_lru *lru, bool memcg_aware, - struct lock_class_key *key, struct shrinker *shrinker) +int __list_lru_init(struct list_lru *lru, bool memcg_aware, struct shrinker *shrinker) { int i; @@ -583,8 +582,10 @@ int __list_lru_init(struct list_lru *lru, bool memcg_aware, for_each_node(i) { spin_lock_init(&lru->node[i].lock); - if (key) - lockdep_set_class(&lru->node[i].lock, key); +#ifdef CONFIG_LOCKDEP + if (lru->key) + lockdep_set_class(&lru->node[i].lock, lru->key); +#endif init_one_lru(&lru->node[i].lru); } diff --git a/mm/workingset.c b/mm/workingset.c index a2b28e356e68..df3937c5eedc 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -823,8 +823,8 @@ static int __init workingset_init(void) if (!workingset_shadow_shrinker) goto err; - ret = __list_lru_init(&shadow_nodes, true, &shadow_nodes_key, - workingset_shadow_shrinker); + ret = list_lru_init_memcg_key(&shadow_nodes, workingset_shadow_shrinker, + &shadow_nodes_key); if (ret) goto err_list_lru; From patchwork Mon Nov 4 17:52:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13861758 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B172D1BDD5 for ; Mon, 4 Nov 2024 17:55:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 905166B0093; Mon, 4 Nov 2024 12:55:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8C0B96B0095; Mon, 4 Nov 2024 12:55:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 72E9F6B0096; Mon, 4 Nov 2024 12:55:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 541766B0093 for ; Mon, 4 Nov 2024 12:55:07 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 021811A06C4 for ; Mon, 4 Nov 2024 17:55:06 +0000 (UTC) X-FDA: 82749163518.14.91425CD Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) by imf03.hostedemail.com (Postfix) with ESMTP id 506ED20013 for ; Mon, 4 Nov 2024 17:54:50 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=KgjLReEo; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf03.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.170 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730742738; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qzULfnxx/PY2zPKAoHEw7ZgfeSZIMyVl8FxRbLFN7FU=; b=2gWgdBc/QlvJ/sBIyCw6v2nu0IMxvD+hRiywny0c2hAu991xXF17mviJ1t2ycOtbYQHh6x E9YkvfyJLg+sfzN43OnSrvEHUpzHTbx5y2ff06xbjvTty4jEderZ339ULaFH4lKO82sO2a wuxTvyOp9g1sPySOqeYLj6jMLRoeLr8= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=KgjLReEo; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf03.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.170 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730742738; a=rsa-sha256; cv=none; b=dN7m2muPLpImQjO72fTNjxrnz5GAtoVlKKgjF4mkky62SeeM1yvrfpdTL8OOsLPJFN3d5r fY1ZZRx9J58V/RB6ZM/SKK01aA4Gkmt+FVV0D+2zacKO5OomtOEj1OFMyb0MeWJ8iaWw72 He4uJe8ge/0io/4gnLZXKfyDJRxJgBY= Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-71e592d7f6eso3544739b3a.3 for ; Mon, 04 Nov 2024 09:55:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730742903; x=1731347703; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=qzULfnxx/PY2zPKAoHEw7ZgfeSZIMyVl8FxRbLFN7FU=; b=KgjLReEoT3S05uJXjIl6gPLbeWW351RSaNbtiIqBydLejNQ1jopkWVD8MnmcqihvYI mZeL5ESX04y0CIzSvN3sSPI0DzZbiKdAYPMzErFaGSJNsICS6K0a+TLHotN2YqJ9rp4t W6fmf9OpD2yefz2PVIveGGD5nFJLzs3VSzl/lfBVjmbuzSpykVIHhHoKJ/9ygEgFIgKz VRll3RKf5Dtf9BEaXfP86HfJF0j2l0J1j9yzZUI0hfOwWgIZJnNyEGaNAlU31NkCFhsb n4hNguD490sjpwn7PE7A7Q5LMOk1KrXzTiFRxYVbGGhg1dQiOj3C+FdVque9JXVS+vBL QCKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730742903; x=1731347703; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=qzULfnxx/PY2zPKAoHEw7ZgfeSZIMyVl8FxRbLFN7FU=; b=HoRIwuk4OqqiMSuhGSYkwHZeNZw3xYwUYcn2LaZfLPUAGqPXJ30M+4PuIdc3rmvSwC ciM6Oy5819YOivl/WyAbaXyREOd6L0DJhztmhWu/6/zOKmGkVRgKkiKEVi4AIzot/WFO coEJIWw1rVHxkKRzALyrddJ24pbVejbzHp6jUsYW9+KQIl9nDsCiPYn6cGNBSepgNgpK t/v1xHTuWVZ0lHHBiyn3qMCO/0WEmAAABEj3ws9f8qEowYSWU/i85fz5Euom4dvdEJvo nt8R3rIaOPDvY1ij+3sD8OmKzm6Cuw98mh/JEKQwqQQtBqtPlTdnJsRdGLknIJTZvQwq v36A== X-Gm-Message-State: AOJu0YzOm8ZChKaadzkAv8qnI2pwzgeZQWmoWIClRpdQVmQFKEXJikuI HMjp3uxzUjWFyRxOckySvUKUfTGEO56JoTpyptTufg8/zBAIb/bBf/x37ZPqpe8= X-Google-Smtp-Source: AGHT+IFHDG/ffQeflGvIVKhJ59EpVHezCKKz7j32EsQ6iF8cb47lZX1SxGzgWXQovDhSMjTSEj1uTw== X-Received: by 2002:a05:6a00:b47:b0:718:ea3c:35c3 with SMTP id d2e1a72fcca58-720c99111ebmr17962004b3a.15.1730742903327; Mon, 04 Nov 2024 09:55:03 -0800 (PST) Received: from KASONG-MC4.tencent.com ([106.37.121.39]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-720bc1e5f5bsm7777039b3a.55.2024.11.04.09.54.59 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 04 Nov 2024 09:55:02 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , Johannes Weiner , Roman Gushchin , Waiman Long , Shakeel Butt , Michal Hocko , Chengming Zhou , Qi Zheng , Muchun Song , Kairui Song Subject: [PATCH v3 2/6] mm/list_lru: don't export list_lru_add Date: Tue, 5 Nov 2024 01:52:53 +0800 Message-ID: <20241104175257.60853-3-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241104175257.60853-1-ryncsn@gmail.com> References: <20241104175257.60853-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 506ED20013 X-Stat-Signature: 7frrmpy3hipwwtj38ghy6pcohurzhxrd X-Rspam-User: X-HE-Tag: 1730742890-145749 X-HE-Meta: U2FsdGVkX1+d2BRgnx18ebP3lF41LfUpeHtH8mcJ70Yx4W69RkwsGe8tdKBAV24p/gheQ+nLfZuY9uKljqNqy40zezMrGne636kCD2Ydq0wTWuZEak4AhkxiaPjqzb5OnJUY+DpaJpMNcSVT6YkSHvWUHP0FqOo9581eqq5QDInZfNCkNzT7yN9BHR9LIBN5tVulamT//0NGYsQHCMgcrCLgYg9hGsoZHcc9TS558UrTJUPgRjgM6OHpWAHyvkKPPQ1cRRqLpa3AP9yBrZHfcDMS0xZU6ERSfOqrYC669pfLKHjUBlU/kktcylI/mh+oMU0gftQIxkQWuA7/uUFF6lDwPWfV5V9yok/QXegTdM6CkQ7rjBgRc2W1dx2KDbA6B4mee1NpEinKl04sOAbBT4wK1vO+yto3OvTlw2GpGF2L0vr6INt7DcReSwP+SdtVB6Z5J4xSet3Sez3uALkG4Eh8IkF/HVNJJ9R93LidPreF7P6QrgG5L6k2K44g2O+Kj7f2jJJ0Msx1nq+f5J+z7F8xorbVbcM4INz33kp0ieZNhMCt0UstAnx8w4JE4EEOAeXDA42KNHD326v7KgzjEFsqTCmokqceJFnHJy8KrvS+qdoesC8pkNaCYb4JQ9ZBHi51IStuo0Ov5kfh1Lb3QEL0rF0bIbIbys1rxuiPPBFl3vc7Rc1JXI/8NvtdZzSQUC5uTj3ggr8oqtrGrwdXpu/B1fA4y+NQU5p/HWqUZjUpjtaZKItiIm/ozInkM4L6+C3xpD6Lr1EcC2KNjppBbfl0HvF/jBLGPH/Yx3QvMJDuW1j/zxP4BhPDvxPf+QMPGvBCa3hYWrtq/wxrqWibtkXhRn2qBJblyXRBMkAjXW10O1+/8exeYyz2ZVXEQS4Stz2m6pJBpEpqhl7EE7RYk+mJ2PaWgDZtQgQshnJTAyLp10/xNaIZ0dFnT2JknG/s75Doghsognn9r6vRWYo A9TRINl+ S2mT64zXLNSM84a1CurMBVkwsd4JJd9jIGvT2iU3BrpzAaTkIclZ5dJRoJ5mcsKKZw/mVYVFCqouEf8YPtAWLCismzUBjRHIvf1l7Rnom8qdp8AjDUT5XThAMSeKnVItuntzHDR/MeIIQmpWd/rQ+sl2Z69vIqH2EaKLcaZ746mHEO/a1+nb+S+z0392Of7YVNEUl/53P2mOUgpR8/0KnY9SWZEUZEMRZsG6qC1zCFhk9XzfSwIAVn0zCHdgGWnhuV8o2lLJJF2PHBnNizbQRVg51vXKWM0w5fENBFyr6WXeyeNflx4YUZwzCBVaOW2samHSPTr6oGb0PNE5yQdaei68d9YQ5lNV5kwTLqVKrDhS5zo+nPbTksiCfLUc18N2ILA4E4A6e81qPJWEbaKpSCjnAxWg+UjU6hm5gmARy29Eb7KlVkeocGrpuFelZtxP+4/8mseVskodWP2k+Gb3CI4JBD0XO83cS9YQ5WYtSxQciy/2cu167Qq4cz64FvmLlQcmz+iiGg42egx/l2KOSwfNRIWSdBaGHbGqlhqy9gHlxV58DqXgwo0Ubjk/F3/Y+2K4LxQYDiTFutfpSV+mAqvDhu07SbFelPcCy X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song It's no longer used by any module, just remove it. Signed-off-by: Kairui Song Reviewed-by: Muchun Song Acked-by: Shakeel Butt --- mm/list_lru.c | 1 - 1 file changed, 1 deletion(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index ea7dc9fa4d05..a798e7624f69 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -106,7 +106,6 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid, spin_unlock(&nlru->lock); return false; } -EXPORT_SYMBOL_GPL(list_lru_add); bool list_lru_add_obj(struct list_lru *lru, struct list_head *item) { From patchwork Mon Nov 4 17:52:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13861759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA7ABD1BDD4 for ; Mon, 4 Nov 2024 17:55:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 449836B0096; Mon, 4 Nov 2024 12:55:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3FA046B0098; Mon, 4 Nov 2024 12:55:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 29A336B0099; Mon, 4 Nov 2024 12:55:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0C6866B0096 for ; Mon, 4 Nov 2024 12:55:12 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B4ADCAC238 for ; Mon, 4 Nov 2024 17:55:11 +0000 (UTC) X-FDA: 82749162426.13.D986EC1 Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by imf30.hostedemail.com (Postfix) with ESMTP id 10CD080011 for ; Mon, 4 Nov 2024 17:54:08 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=h8BWhuhy; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730742851; a=rsa-sha256; cv=none; b=XZxNkcuq49ff8co84DV/jzik5wrifYp8WoOS1gFWNOksi/cV2shMjJ+od/oB/0Afm/fFvS DiChzvqqOaU5K1GSeBKjzoNd6B9anCLZ7tR4rQVTF5gLywxzIoassBdhI8Wfk/X/zRvdhw h0II08ZkVOYYl3pfvpvH9wEWCXsVozA= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=h8BWhuhy; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730742851; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iL9IvtjIkFQSQS96IExA8P6PV6Z5oBDu+RSPWvdKwUw=; b=VUutWM70C+tknrq72p5H38/1HNMKdyFJv6iYgD/ZH5ydaUFGwPsGDTVi9ZrJI+FjGg4Qq7 E+gB7/qu8fgMbpjYN52iZEwSRvWRuhl0zUArVNrFOqXSeSCE3nDEIH5Suy25GLW2iD/ZxN wkSUIsSAXLGkcx/FxVJWhsOoHVcE++E= Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-720be27db27so3610959b3a.2 for ; Mon, 04 Nov 2024 09:55:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730742908; x=1731347708; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=iL9IvtjIkFQSQS96IExA8P6PV6Z5oBDu+RSPWvdKwUw=; b=h8BWhuhy4FSbsRl57yki79iHbMvmphFuZT7e6V5qbzyo15iKP5xQ1hSbNyJDCKj0Br CqOo4XY5foZViV7ojrBTCO8wQ6RMtKn1svKSpKQBRbpfmIc5HSiqYRnH99sZfAaZgPv3 mM9mwwE0v1+nacpebN4UGZuZLLMA85kZ5ldaSYFF7CgS5F2A1KXUaCsVQ/jJn//YDxCf zjl5BS1ziWaDwRi4rcCBaZZR7/JD9RwQmueWOMK7UZHde4Q79Q07BnI4Y7XUaU4dgxOg Ga/nYl66zmnRIseYGi0rWqzKuEspjA87ZwdCTuKq9o9jvPE/AziuHWGGyVXwABmY1X0u 0d3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730742908; x=1731347708; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=iL9IvtjIkFQSQS96IExA8P6PV6Z5oBDu+RSPWvdKwUw=; b=TspsGmA0KJeXp/Hy9fQBwXl77fznGQ05eIm2idYd4pAPspSz4oV41awaU1nFSFfTMA TddRQbSo63PfKKNsHxGPQUXdEr0XXU7ZcJeGZP/j6iH64G1ZvpLy0Yz7zpqKhwgW1fRu go6CMgaa/euaVG5coLqLjwFfdHPRxui8M9ALY5wZfJesFJoCatzCEToYFUvze1WXzJhQ ZU58XYNxV5Ppm4KEEppPmkYMKLoLn6NG3l2ofZ+PHTfwo1SF9Rc23LA+AndsYMzkzG/W tiicpbOpfKKXok7fO+fDSPgXsOghK8OMkpULru6GklcAmavszrRSqtJ0XZIAFRWbJxxC 0dnQ== X-Gm-Message-State: AOJu0YwaM16coqMOnVvN/7DX/NgWcD7bzCSZ0NGSuzumi+J2A+fgufG9 rxcV0VmMLo0IdnDccFrhlJmzU7RECdAnH0/xDwna7TrJzLapwOxc3E+oN2eS6E8= X-Google-Smtp-Source: AGHT+IFcJ2lqm5wdX7MS5DNufJZ3E89XUP2EPJ1Z5Isz/XIavZ/J0/34TdR7wCXBdSTpRahIjuGfzA== X-Received: by 2002:a05:6a00:2191:b0:71e:44f6:6900 with SMTP id d2e1a72fcca58-720ab3f39d8mr28676270b3a.16.1730742907924; Mon, 04 Nov 2024 09:55:07 -0800 (PST) Received: from KASONG-MC4.tencent.com ([106.37.121.39]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-720bc1e5f5bsm7777039b3a.55.2024.11.04.09.55.03 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 04 Nov 2024 09:55:07 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , Johannes Weiner , Roman Gushchin , Waiman Long , Shakeel Butt , Michal Hocko , Chengming Zhou , Qi Zheng , Muchun Song , Kairui Song Subject: [PATCH v3 3/6] mm/list_lru: code clean up for reparenting Date: Tue, 5 Nov 2024 01:52:54 +0800 Message-ID: <20241104175257.60853-4-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241104175257.60853-1-ryncsn@gmail.com> References: <20241104175257.60853-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 10CD080011 X-Rspamd-Server: rspam01 X-Stat-Signature: gyjxrxk6japa8bqds7khcoxexmx6sbox X-HE-Tag: 1730742848-640512 X-HE-Meta: U2FsdGVkX1//E1CS4CMoI8xQti2oX+SvYZqz1hoXoGbOHbbjuEr9wa7JDEqzCV4SOEXARGlV1ZyWVw+ozLqA+TXHPkX6q3LlNkJGgwrMKbxO423v9y5hZxNKmAC8Mr0QT+JhdjgLiCT3giiaXyOxIVzbSAnTG7WKI/nvfSmKLhH2WjlgmIunMrA1rM8O/qNkGqCs9/w88SMRV29JXY7PxX0thl35JftSxEIFq9PnR7V5T8jTdnaI5pPQhnzmsmURZYdWfaAiyvgbpzelV8aTfaiemA7pES1Agww32q3rQl3WS1PpBalnpflyNFaFjrFfkFUuILpuzEZMfMMjqaWkEkZY23jmG9V67J7vdVjkK+RmUI4uIwcJHyi6tE4au5OOxjKDtv4pJwq28RVT48AOw1HBnC7w13KMsPHqgzlesPMFPvsCyWGvn5AglkGtQ21MPg6DZAH/4sg1pL0V0x6A4F4tTm92v6s50P0HHdo3SBJ9duq+wxX5dJ0yaA9J/PtvhlEf0w+6K7DNHjDXCFGZaR7BcHaVPnUAM1DyaWaYa3fUB5Qe+iX1zXJ+xM1ne+1CracF7/ZnFidkhSY8zc9uiAw2VIzgpsjUvMJ7ljfBipJf7aOtH5HDo+3DnGHF8x0dlN63JsAyD8C8qUZII+eM4klmF1p4LIhIVQPT9FS5z6rGGa+O5Ullh/vXzgLaVHB+39UfjkS9hbtzDONRJLE+BlmBOWW9alOYoPIkzlntrrRF9PtEZpO+x745hXLRCr/hwxXP7l7o9HEzbDitY08+GVCAliRDHTFughvxDtol5fxkwWyS0lp5ylJ8dnWG7U3qwSunsjp5uQlCycreTBFPbr4+2oIL2FSSKyp/cFk+fZvK2RWEXJAoHreIsvCctVcYLI/CG4A35nqJ5Mmxn6GA5QzeFbUG5Rvr1BgxVZpy45cHWXKv2pcg/bDkFNU1FD/yfwzPlEhmf5vSjouxNGZ fW2c0+Me czDo/8yHqRlPc2C4hcMuDTXTN/1efVldVMB0wRZMsJmPlV1PavDoTFKNlgF7soOb4LPlH96nfeMsspbDyQpKM4sdE0TWCRZW7iCPzDdilLPmCZKe2WtI24K4DnKDjwLRTX8WB3VuV6042dvAUYZUM09cwsXqnxOvRA6v9xZw11wOcYURopF/5FqYrqyGsJmh2b0dlln+3OrJxOVCxh3VqIQ3cs4ksC0cJ1PZwdoVHB+Coh3eph4T6p3ClzljcwP8N5Sl2n+UnxKi5cMygNe1SN0V8j5FDY9OmIaZbv0A7chMt0RE3HTmbCzwKxO3j9QOmoKy6jx1dg1i9a/uq/p/RbYCAGUNoLAOy3IE+ceSkk23bIeP59xErbeJrEsttF+q9jlDJmmhkUPg3cin7+WDxydsS2Xp3RCoy6N81d5ZyKZLXBWD/UiPG9/+0PT2UQn6SRsqxieuxor050LxAcdBhLQBvTe0FYqg5B6YNiZ8pfwUdy88A26DPZblRZGkco/KqGlJ9JG9v8kYOGuVgWjgfJDco3kp+YIlAnmMtCKKb/bRskxWuuYRnJj1XxmlharzcWt1DJtshQDN4vJxX3N0F4jMR/iuUPo9G2xkSamkFixklgYI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song No feature change, just change of code structure and fix comment. The list lrus are not empty until memcg_reparent_list_lru_node() calls are all done, so the comments in memcg_offline_kmem were slightly inaccurate. Signed-off-by: Kairui Song Reviewed-by: Muchun Song Acked-by: Shakeel Butt --- mm/list_lru.c | 39 +++++++++++++++++---------------------- mm/memcontrol.c | 7 ------- 2 files changed, 17 insertions(+), 29 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index a798e7624f69..b54f092d4d65 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -421,35 +421,16 @@ static void memcg_reparent_list_lru_node(struct list_lru *lru, int nid, spin_unlock_irq(&nlru->lock); } -static void memcg_reparent_list_lru(struct list_lru *lru, - int src_idx, struct mem_cgroup *dst_memcg) -{ - int i; - - for_each_node(i) - memcg_reparent_list_lru_node(lru, i, src_idx, dst_memcg); - - memcg_list_lru_free(lru, src_idx); -} - void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent) { struct cgroup_subsys_state *css; struct list_lru *lru; - int src_idx = memcg->kmemcg_id; + int src_idx = memcg->kmemcg_id, i; /* * Change kmemcg_id of this cgroup and all its descendants to the * parent's id, and then move all entries from this cgroup's list_lrus * to ones of the parent. - * - * After we have finished, all list_lrus corresponding to this cgroup - * are guaranteed to remain empty. So we can safely free this cgroup's - * list lrus in memcg_list_lru_free(). - * - * Changing ->kmemcg_id to the parent can prevent memcg_list_lru_alloc() - * from allocating list lrus for this cgroup after memcg_list_lru_free() - * call. */ rcu_read_lock(); css_for_each_descendant_pre(css, &memcg->css) { @@ -460,9 +441,23 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren } rcu_read_unlock(); + /* + * With kmemcg_id set to parent, holding the lock of each list_lru_node + * below can prevent list_lru_{add,del,isolate} from touching the lru, + * safe to reparent. + */ mutex_lock(&list_lrus_mutex); - list_for_each_entry(lru, &memcg_list_lrus, list) - memcg_reparent_list_lru(lru, src_idx, parent); + list_for_each_entry(lru, &memcg_list_lrus, list) { + for_each_node(i) + memcg_reparent_list_lru_node(lru, i, src_idx, parent); + + /* + * Here all list_lrus corresponding to the cgroup are guaranteed + * to remain empty, we can safely free this lru, any further + * memcg_list_lru_alloc() call will simply bail out. + */ + memcg_list_lru_free(lru, src_idx); + } mutex_unlock(&list_lrus_mutex); } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7845c64a2c57..8e90aa026c47 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3099,13 +3099,6 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) parent = root_mem_cgroup; memcg_reparent_objcgs(memcg, parent); - - /* - * After we have finished memcg_reparent_objcgs(), all list_lrus - * corresponding to this cgroup are guaranteed to remain empty. - * The ordering is imposed by list_lru_node->lock taken by - * memcg_reparent_list_lrus(). - */ memcg_reparent_list_lrus(memcg, parent); } From patchwork Mon Nov 4 17:52:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13861760 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FBDBD1BDD4 for ; Mon, 4 Nov 2024 17:55:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A0476B0099; Mon, 4 Nov 2024 12:55:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 050636B009A; Mon, 4 Nov 2024 12:55:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E0C4C6B009B; Mon, 4 Nov 2024 12:55:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id BAECE6B0099 for ; Mon, 4 Nov 2024 12:55:16 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 80E1E1606E4 for ; Mon, 4 Nov 2024 17:55:16 +0000 (UTC) X-FDA: 82749162636.01.7A20DCE Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) by imf13.hostedemail.com (Postfix) with ESMTP id 68C5B2000F for ; Mon, 4 Nov 2024 17:54:41 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Sx8oToSU; spf=pass (imf13.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.176 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730742831; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1Bqi64CkTqyBVVmYN5m+bzgJB4o8XYhPi7RPO5VmPnE=; b=TDU34X1tFHtBQH8VS3EMDGauFf3ByUd9FILTtCgf8yli2kCO7qbLPP0goMEiKyAz0g3lgT QB5K4Y7FvxwFOwj4op0j/A5mtC1KcJTix0lnLDjurC2qCyQceUheRH0st8S2YK6D5whSo5 yR/hSVnee8tvQKGaW9FonzvJiouoiU0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730742831; a=rsa-sha256; cv=none; b=gVxo8/MC4HJ4xFEgMNdSS6hk8bjHqHLt7Sanj/nEa1e41uJxPzskKSKRBpbqvnwL7NnEZ0 KTaY1JCTFMqrk8lDZnaSLo3+cXgZmv9BipVyOdMMLu/SNOdHP/vlS0RyIwjySrcsy6w16t fa8QVnzgWz91s3GK1r1clGa51bwvxGQ= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Sx8oToSU; spf=pass (imf13.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.176 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-720be2b27acso3702907b3a.0 for ; Mon, 04 Nov 2024 09:55:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730742913; x=1731347713; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=1Bqi64CkTqyBVVmYN5m+bzgJB4o8XYhPi7RPO5VmPnE=; b=Sx8oToSUB/20LT/oZzuQ1WjHk3sdpl8zYTBswl4JSUWvvwfEqbILXDLYylqrdGxuSg RsFYUqCIM8KSDQRjwc/VoLmt2jYLKe5P6ccFA9Ggj1hpIouWBQGxZ9yQ094+Sdmv7SMJ vVs21ibkpAQV7O1VfK/E1XadReqADce0RH8sBfh1FZi4hvZdPM7hmfhdD2CrW5UQz/w4 CT5kCZS1t1z3DtaRoQO0emqP2m1pHvSvzJcu2c284xZTY865OcjO9hZPX8BVPd7Q0s2g TS7Ws2twvxwp/89bgV/GA9BPYRtTcBDoRUh0yTz9EgJyF5iTs/dK4G+DrdcDSTZ1wdrf 8P6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730742913; x=1731347713; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=1Bqi64CkTqyBVVmYN5m+bzgJB4o8XYhPi7RPO5VmPnE=; b=VPgEaVFsbEvgvWLA98PLjtprGdj3CLFY6TGZT1yCz9wGHKICy9KRumTtoQ88wBIJsn nmlW7fIwCp8A1/6j5KA4Lan8qtSvdtMtvm1gd2reU6bCOtbV/PqgaIt8rbYXV0bHYvK5 8TRFumDquDnf0kl0IrVfJ1fT74scQjQIVRXqW/TRYuRzT3m3Tz/WW/TKp3AJAFL9+OjO +NVmqsl8QRkXugDzKnKaDQKO0XgHNAtliLL9nZafRcKLyzQjxYWKebbuBSTdUtWSZI+I 7j5Gakeb+XlfD0kQ1ma3rGiwKC/C7dvkoL6hpNKe4mKXDUXTHkaAj0urtjRk+7wzHjeo fS7w== X-Gm-Message-State: AOJu0YzRMB/cGmO5Ep5Tr+X26GLZj7z0iVA5EDK0LB4/6+qhAP8vYbUt OcMFJ6n83Cxm3aVBLcbcN2vahgJDcp8kdMf/ASx/CWLkurr+NahEVbgpJWUoECI= X-Google-Smtp-Source: AGHT+IGj7VI5BwEUr5hbEO/itHelyNmhDHbTJ/8gnX12vkBNDrFawHLAhiENlXFv3OWd1YMx7bUMrw== X-Received: by 2002:a05:6a20:6f0a:b0:1d9:c4e0:140c with SMTP id adf61e73a8af0-1db91d44042mr22904744637.1.1730742912765; Mon, 04 Nov 2024 09:55:12 -0800 (PST) Received: from KASONG-MC4.tencent.com ([106.37.121.39]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-720bc1e5f5bsm7777039b3a.55.2024.11.04.09.55.08 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 04 Nov 2024 09:55:12 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , Johannes Weiner , Roman Gushchin , Waiman Long , Shakeel Butt , Michal Hocko , Chengming Zhou , Qi Zheng , Muchun Song , Kairui Song Subject: [PATCH v3 4/6] mm/list_lru: simplify reparenting and initial allocation Date: Tue, 5 Nov 2024 01:52:55 +0800 Message-ID: <20241104175257.60853-5-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241104175257.60853-1-ryncsn@gmail.com> References: <20241104175257.60853-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Stat-Signature: 6ugpc34jyrmt9uhsm4ewbx6m4kzcsrfm X-Rspamd-Queue-Id: 68C5B2000F X-Rspam-User: X-HE-Tag: 1730742881-806167 X-HE-Meta: U2FsdGVkX1+CA5R0CjSa+F3B3kKsDU/+WHH2ilorJMF5uxvGbnd0bG9dbE/Qk+MYawd3WwfXLoXAQ+QHyW3+aig7JzAjHOJ1/Aw8DAk7+MD4PsecejyjUXSBW51FX4v0KThrEIqr65p9fsFbc20WInNaz1em4HtQMQ/8L+6NTvy3oe/PvT+YpO70R0IHI8403oh2Lk1Tmb9Q2E66IHKIdHhY2SieovZWcLFtCM972TS+JC90KACavoUbORd9LJvRKJO8n42WdB3W7QVhEngukuCWK/RtmJ38o0teZSU/8EOzGUDvOT/4DECWi+tjdLALl3Gu7fhNQ0L42YCBiHGlpzhwteQFl0ORbOICqH4Z4ADDewIsVWRQ1D1ktQkJP7+yupwL/630jJ4jhzMQ9j4ndwgitngZ/3f43AYECzn7YG7HC2q2m6VmRJ+WYTSy+vuFEXCRwIKRmiCE5P87sNVF8Z7r3HMqXx5Z7qv4C7q2uAXDDB8fbGtT540+nJH9bkDQdk2uCRHImmBCaFvdQsZqVn6y7Fl8+/jmnGZjWN5KN+q7NcR0BkhM6QEDhjwhSe58FJXsj81GwHsUactPCOS4myN+hAy7HohFJKhLODsafGh4pABfj2UvNDUDB2D61h0kXh4u7zKLUtaIaIsrK2zocvDeA1izdVOKTXjwD85dNWQcA6W0jFQirZEOjGHQm5jhM8wG/hKGWPsXDdAoIGpQEmESZ3OwzvTmm47qCxDxXTTFh6hq9N5zDtFMnMA+m0srrzCiCAJ2kjQ19yjsWDTuqFJdjvzW44lVKwZ+7y29QU5ZyvXuu1qJsvu7gD47fiet8/DOOn8Pb7mtR4Z2nznAIp9i55FUGMpFnVl7kP61wyWR7DpKiF4rox5QWQ7TRLZ2ko/+nKCW10Q7xU6SD4umJQGNac+pm+ZV5s2pKpjR5qp6Tzgbx+Sqi12Onb2nuTym2ZQ0aQUGlbCeaiXmYbB 1pRvcJ8N dTPNLL5wIdjvBijmWbGoNLpd3gwJwEwxvT0Gv1F1FX7Rii8AveGiCDMZpBJ1/TOcRIid79k8o5rmAAVIC7VNUGF3Cs16Bp5wFPHv6wkuTUG+tdi1ekT/fAf4xap0q9AKVxyDISeS76mEUQl42zApg+/EQ3M4gUgVs35sVNDSG/qVPufTQ8Gtqq+PB3ypbmfGpIUvpePJVdZ87hmCXvC7+SB51gAM2tORAEnKbyADnF/L6jFvVSvcwtX9YjNCkuIMlD5OmgeY4K2bsk9mpnvXQ2A1Bgd2lmgfm81IL0vBn5pHTnXGvsYjd0hsI08LMMLItMf+a+QFAOIXD+owJ5yRNRZKF+gV4iUJ8kMoqcJdptN0TLnN37yqZ+xth6xz9PX95bTGf2YWLMATUsBBt8kdms6PS+PLi7/sz/PFbB1uagbRSsDELb+3v1Z4HnVPIiZTkrtJF2EA2MctPIS5i2t/AfbbsyIgWf030Y1oDG08f/6mU7+Wv2ksCopWWJadbvXFrOAp3EKoTgkDR/C5B/vk7Jj9TAQQgJyFMwJQa+PHeV+eEcI5WiI2eCDCimZJYUwh4GTBB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Currently, there is a lot of code for detecting reparent racing using kmemcg_id as the synchronization flag. And an intermediate table is required to record and compare the kmemcg_id. We can simplify this by just checking the cgroup css status, skip if cgroup is being offlined. On the reparenting side, ensure no more allocation is on going and no further allocation will occur by using the XArray lock as barrier. Combined with a O(n^2) top-down walk for the allocation, we get rid of the intermediate table allocation completely. Despite being O(n^2), it should be actually faster because it's not practical to have a very deep cgroup level, and in most cases the parent cgroup should have been allocated already. This also avoided changing kmemcg_id before reparenting, making cgroups have a stable index for list_lru_memcg. After this change it's possible that a dying cgroup will see a NULL value in XArray corresponding to the kmemcg_id, because the kmemcg_id will point to an empty slot. In such case, just fallback to use its parent. As a result the code is simpler, following test also showed a very slight performance gain (12 test runs): prepare() { mkdir /tmp/test-fs modprobe brd rd_nr=1 rd_size=16777216 mkfs.xfs -f /dev/ram0 mount -t xfs /dev/ram0 /tmp/test-fs for i in $(seq 10000); do seq 8000 > "/tmp/test-fs/$i" done mkdir -p /sys/fs/cgroup/system.slice/bench/test/1 echo +memory > /sys/fs/cgroup/system.slice/bench/cgroup.subtree_control echo +memory > /sys/fs/cgroup/system.slice/bench/test/cgroup.subtree_control echo +memory > /sys/fs/cgroup/system.slice/bench/test/1/cgroup.subtree_control echo 768M > /sys/fs/cgroup/system.slice/bench/memory.max } do_test() { read_worker() { mkdir -p "/sys/fs/cgroup/system.slice/bench/test/1/$1" echo $BASHPID > "/sys/fs/cgroup/system.slice/bench/test/1/$1/cgroup.procs" read -r __TMP < "/tmp/test-fs/$1"; } read_in_all() { for i in $(seq 10000); do read_worker "$i" & done; wait } echo 3 > /proc/sys/vm/drop_caches time read_in_all for i in $(seq 1 10000); do rmdir "/sys/fs/cgroup/system.slice/bench/test/1/$i" &>/dev/null done } Before: real 0m3.498s user 0m11.037s sys 0m35.872s real 1m33.860s user 0m11.593s sys 3m1.169s real 1m31.883s user 0m11.265s sys 2m59.198s real 1m32.394s user 0m11.294s sys 3m1.616s real 1m31.017s user 0m11.379s sys 3m1.349s real 1m31.931s user 0m11.295s sys 2m59.863s real 1m32.758s user 0m11.254s sys 2m59.538s real 1m35.198s user 0m11.145s sys 3m1.123s real 1m30.531s user 0m11.393s sys 2m58.089s real 1m31.142s user 0m11.333s sys 3m0.549s After: real 0m3.489s user 0m10.943s sys 0m36.036s real 1m10.893s user 0m11.495s sys 2m38.545s real 1m29.129s user 0m11.382s sys 3m1.601s real 1m29.944s user 0m11.494s sys 3m1.575s real 1m31.208s user 0m11.451s sys 2m59.693s real 1m25.944s user 0m11.327s sys 2m56.394s real 1m28.599s user 0m11.312s sys 3m0.162s real 1m26.746s user 0m11.538s sys 2m55.462s real 1m30.668s user 0m11.475s sys 3m2.075s real 1m29.258s user 0m11.292s sys 3m0.780s Which is slightly faster in real time. Signed-off-by: Kairui Song --- mm/list_lru.c | 178 +++++++++++++++++++++----------------------------- mm/zswap.c | 7 +- 2 files changed, 77 insertions(+), 108 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index b54f092d4d65..172b16146e15 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -59,6 +59,20 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) } return &lru->node[nid].lru; } + +static inline struct list_lru_one * +list_lru_from_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg) +{ + struct list_lru_one *l; +again: + l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); + if (likely(l)) + return l; + + memcg = parent_mem_cgroup(memcg); + VM_WARN_ON(!css_is_dying(&memcg->css)); + goto again; +} #else static void list_lru_register(struct list_lru *lru) { @@ -83,6 +97,12 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) { return &lru->node[nid].lru; } + +static inline struct list_lru_one * +list_lru_from_memcg(struct list_lru *lru, int nid, int idx) +{ + return &lru->node[nid].lru; +} #endif /* CONFIG_MEMCG */ /* The caller must ensure the memcg lifetime. */ @@ -94,7 +114,7 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid, spin_lock(&nlru->lock); if (list_empty(item)) { - l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); + l = list_lru_from_memcg(lru, nid, memcg); list_add_tail(item, &l->list); /* Set shrinker bit if the first element was added */ if (!l->nr_items++) @@ -133,7 +153,7 @@ bool list_lru_del(struct list_lru *lru, struct list_head *item, int nid, spin_lock(&nlru->lock); if (!list_empty(item)) { - l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); + l = list_lru_from_memcg(lru, nid, memcg); list_del_init(item); l->nr_items--; nlru->nr_items--; @@ -355,20 +375,6 @@ static struct list_lru_memcg *memcg_init_list_lru_one(gfp_t gfp) return mlru; } -static void memcg_list_lru_free(struct list_lru *lru, int src_idx) -{ - struct list_lru_memcg *mlru = xa_erase_irq(&lru->xa, src_idx); - - /* - * The __list_lru_walk_one() can walk the list of this node. - * We need kvfree_rcu() here. And the walking of the list - * is under lru->node[nid]->lock, which can serve as a RCU - * read-side critical section. - */ - if (mlru) - kvfree_rcu(mlru, rcu); -} - static inline void memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) { if (memcg_aware) @@ -393,22 +399,18 @@ static void memcg_destroy_list_lru(struct list_lru *lru) } static void memcg_reparent_list_lru_node(struct list_lru *lru, int nid, - int src_idx, struct mem_cgroup *dst_memcg) + struct list_lru_one *src, + struct mem_cgroup *dst_memcg) { struct list_lru_node *nlru = &lru->node[nid]; - int dst_idx = dst_memcg->kmemcg_id; - struct list_lru_one *src, *dst; + struct list_lru_one *dst; /* * Since list_lru_{add,del} may be called under an IRQ-safe lock, * we have to use IRQ-safe primitives here to avoid deadlock. */ spin_lock_irq(&nlru->lock); - - src = list_lru_from_memcg_idx(lru, nid, src_idx); - if (!src) - goto out; - dst = list_lru_from_memcg_idx(lru, nid, dst_idx); + dst = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(dst_memcg)); list_splice_init(&src->list, &dst->list); @@ -417,46 +419,43 @@ static void memcg_reparent_list_lru_node(struct list_lru *lru, int nid, set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); src->nr_items = 0; } -out: spin_unlock_irq(&nlru->lock); } void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent) { - struct cgroup_subsys_state *css; struct list_lru *lru; - int src_idx = memcg->kmemcg_id, i; - - /* - * Change kmemcg_id of this cgroup and all its descendants to the - * parent's id, and then move all entries from this cgroup's list_lrus - * to ones of the parent. - */ - rcu_read_lock(); - css_for_each_descendant_pre(css, &memcg->css) { - struct mem_cgroup *child; - - child = mem_cgroup_from_css(css); - WRITE_ONCE(child->kmemcg_id, parent->kmemcg_id); - } - rcu_read_unlock(); + int i; - /* - * With kmemcg_id set to parent, holding the lock of each list_lru_node - * below can prevent list_lru_{add,del,isolate} from touching the lru, - * safe to reparent. - */ mutex_lock(&list_lrus_mutex); list_for_each_entry(lru, &memcg_list_lrus, list) { + struct list_lru_memcg *mlru; + XA_STATE(xas, &lru->xa, memcg->kmemcg_id); + + /* + * Lock the Xarray to ensure no on going list_lru_memcg + * allocation and further allocation will see css_is_dying(). + */ + xas_lock_irq(&xas); + mlru = xas_store(&xas, NULL); + xas_unlock_irq(&xas); + if (!mlru) + continue; + + /* + * With Xarray value set to NULL, holding the lru lock below + * prevents list_lru_{add,del,isolate} from touching the lru, + * safe to reparent. + */ for_each_node(i) - memcg_reparent_list_lru_node(lru, i, src_idx, parent); + memcg_reparent_list_lru_node(lru, i, &mlru->node[i], parent); /* * Here all list_lrus corresponding to the cgroup are guaranteed * to remain empty, we can safely free this lru, any further * memcg_list_lru_alloc() call will simply bail out. */ - memcg_list_lru_free(lru, src_idx); + kvfree_rcu(mlru, rcu); } mutex_unlock(&list_lrus_mutex); } @@ -472,77 +471,48 @@ static inline bool memcg_list_lru_allocated(struct mem_cgroup *memcg, int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, gfp_t gfp) { - int i; unsigned long flags; - struct list_lru_memcg_table { - struct list_lru_memcg *mlru; - struct mem_cgroup *memcg; - } *table; + struct list_lru_memcg *mlru; + struct mem_cgroup *pos, *parent; XA_STATE(xas, &lru->xa, 0); if (!list_lru_memcg_aware(lru) || memcg_list_lru_allocated(memcg, lru)) return 0; gfp &= GFP_RECLAIM_MASK; - table = kmalloc_array(memcg->css.cgroup->level, sizeof(*table), gfp); - if (!table) - return -ENOMEM; - /* * Because the list_lru can be reparented to the parent cgroup's * list_lru, we should make sure that this cgroup and all its * ancestors have allocated list_lru_memcg. */ - for (i = 0; memcg; memcg = parent_mem_cgroup(memcg), i++) { - if (memcg_list_lru_allocated(memcg, lru)) - break; - - table[i].memcg = memcg; - table[i].mlru = memcg_init_list_lru_one(gfp); - if (!table[i].mlru) { - while (i--) - kfree(table[i].mlru); - kfree(table); - return -ENOMEM; + do { + /* + * Keep finding the farest parent that wasn't populated + * until found memcg itself. + */ + pos = memcg; + parent = parent_mem_cgroup(pos); + while (!memcg_list_lru_allocated(parent, lru)) { + pos = parent; + parent = parent_mem_cgroup(pos); } - } - - xas_lock_irqsave(&xas, flags); - while (i--) { - int index = READ_ONCE(table[i].memcg->kmemcg_id); - struct list_lru_memcg *mlru = table[i].mlru; - xas_set(&xas, index); -retry: - if (unlikely(index < 0 || xas_error(&xas) || xas_load(&xas))) { - kfree(mlru); - } else { - xas_store(&xas, mlru); - if (xas_error(&xas) == -ENOMEM) { - xas_unlock_irqrestore(&xas, flags); - if (xas_nomem(&xas, gfp)) - xas_set_err(&xas, 0); - xas_lock_irqsave(&xas, flags); - /* - * The xas lock has been released, this memcg - * can be reparented before us. So reload - * memcg id. More details see the comments - * in memcg_reparent_list_lrus(). - */ - index = READ_ONCE(table[i].memcg->kmemcg_id); - if (index < 0) - xas_set_err(&xas, 0); - else if (!xas_error(&xas) && index != xas.xa_index) - xas_set(&xas, index); - goto retry; + mlru = memcg_init_list_lru_one(gfp); + if (!mlru) + return -ENOMEM; + xas_set(&xas, pos->kmemcg_id); + do { + xas_lock_irqsave(&xas, flags); + if (!xas_load(&xas) && !css_is_dying(&pos->css)) { + xas_store(&xas, mlru); + if (!xas_error(&xas)) + mlru = NULL; } - } - } - /* xas_nomem() is used to free memory instead of memory allocation. */ - if (xas.xa_alloc) - xas_nomem(&xas, gfp); - xas_unlock_irqrestore(&xas, flags); - kfree(table); + xas_unlock_irqrestore(&xas, flags); + } while (xas_nomem(&xas, gfp)); + if (mlru) + kfree(mlru); + } while (pos != memcg && !css_is_dying(&pos->css)); return xas_error(&xas); } diff --git a/mm/zswap.c b/mm/zswap.c index 162013952074..6910c37cb8ec 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -703,12 +703,11 @@ static void zswap_lru_add(struct list_lru *list_lru, struct zswap_entry *entry) /* * Note that it is safe to use rcu_read_lock() here, even in the face of - * concurrent memcg offlining. Thanks to the memcg->kmemcg_id indirection - * used in list_lru lookup, only two scenarios are possible: + * concurrent memcg offlining: * - * 1. list_lru_add() is called before memcg->kmemcg_id is updated. The + * 1. list_lru_add() is called before list_lru_memcg is erased. The * new entry will be reparented to memcg's parent's list_lru. - * 2. list_lru_add() is called after memcg->kmemcg_id is updated. The + * 2. list_lru_add() is called after list_lru_memcg is erased. The * new entry will be added directly to memcg's parent's list_lru. * * Similar reasoning holds for list_lru_del(). From patchwork Mon Nov 4 17:52:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13861761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28005D1BDD5 for ; Mon, 4 Nov 2024 17:55:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A8C666B009B; Mon, 4 Nov 2024 12:55:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A3B206B009C; Mon, 4 Nov 2024 12:55:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8B46D6B009D; Mon, 4 Nov 2024 12:55:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 66B946B009B for ; Mon, 4 Nov 2024 12:55:21 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 26D4DABC4E for ; Mon, 4 Nov 2024 17:55:21 +0000 (UTC) X-FDA: 82749163770.16.DE09C37 Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by imf11.hostedemail.com (Postfix) with ESMTP id 9336440018 for ; Mon, 4 Nov 2024 17:54:39 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="X0oM/0P2"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf11.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.173 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730742861; a=rsa-sha256; cv=none; b=pC3rtyI+7KXfZ9MmcPvq791gMql4V8qbUFC9tfHw5PSlz/rviEoahwdj9XSHs1njpq1j5G x+H+xTK7AZC3sF75eXUqJAXZOQdcKo8FUgpT0CglOuKLDZOoHx7BvADjxVqZkdXT83Vq2Y Wylulws12baxpCiJnlnhL14+Wb6OqH0= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="X0oM/0P2"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf11.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.173 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730742861; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xm4RIN86YOYAhUhI1xEjsw7JOOQXhtR2ucgLJFq4m24=; b=jTiQcIA51/fDCLDyZNdbmdAgCF+fgdGKe5EWSQycNevv1WgJj2OC1kbRODuxXGZCp6mFlA hOteDcfMenYCO1r30LAe/7z+8xkGMfIA9AbSz/JQ3ipSwp5Je0iTROKFlNocLNWCgw7NU7 7yVq74MwQ34SrIdJQ8PnAVGCuUaTx3I= Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-71e681bc315so3219059b3a.0 for ; Mon, 04 Nov 2024 09:55:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730742917; x=1731347717; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=xm4RIN86YOYAhUhI1xEjsw7JOOQXhtR2ucgLJFq4m24=; b=X0oM/0P2GHjEVa18wM8O4upPCvyHToZlbsbSytjIGsPRbIqfIIXQ7kFgcV6H8xxOAU ZZfVC3hxPH2dYDxsioZkVdumqodYt/ScpLAruWnOoSGjb9Z1NGYyNcgndYODG9Sj+f3f tVZlHRMbjnRWfeVPJlP8o4oP5D5DENlP/IWox4jOkIBn9bs61N+1czVag1MXlD9HniKk whwoVzcLuZ4D/C5Lh2DsWQRfI3aBKzIMLczluvn4/OC3LBchablpfAABJmeCrh+yHcc9 JxhKNZVChqBxlymLlO+/ff9MW9NczrTBJ+j56FyG/XU42KvsAa798VYYS6iFXh5nVNUc yUoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730742917; x=1731347717; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=xm4RIN86YOYAhUhI1xEjsw7JOOQXhtR2ucgLJFq4m24=; b=KMUCi9SgJQnolMBV9mM+5YfugOVLVGHJ68w5pvpZFg1L6UPf1xTKYeaMY5ZzgiWxjo IouqExypbAD+n7PkMYCwLrq35H1xa2uIJY492hTeNDDuJ6Vd0NbDsGsfdwlMbj7kn9zk JnaUADhs1xMTDka0XU4+pbHKZPVWiNosnAfApRoecoPTf0Lh3uooGA0YeMFzyJQcjmaP HB8yo1EFfj8VwVp4rAUAoSaSY0JmAG4ba1tURI/dHIFQNx3uY1g5hv6EYx4Fxnoefuwp jVmuh0DVnzwxHI7qn7zYZx1HEacWcHwTc8CuM3F42qOmfS/Ot/MC3DFPgfzCPghWlp/P P8GQ== X-Gm-Message-State: AOJu0Yy3UfXtAcYLrGoFtgzfB0XecwjboySVkJxBLlxqQx3GjTYXHeDp MOneTlBEReH+E1bTJMDxQSQ/xH0vCAvlIXApF1JUMdUND3oE7ja8Ne2iiX0Ejfg= X-Google-Smtp-Source: AGHT+IEyZs30XY5LXAe3Hc+AOGvBBxpW1Z4EzxPHloLB1gLaZvcPnT7vF7muuYamNGaBGUWVp0Ot+g== X-Received: by 2002:a05:6a00:790:b0:71e:52f0:1c28 with SMTP id d2e1a72fcca58-720c945c433mr18882000b3a.0.1730742917340; Mon, 04 Nov 2024 09:55:17 -0800 (PST) Received: from KASONG-MC4.tencent.com ([106.37.121.39]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-720bc1e5f5bsm7777039b3a.55.2024.11.04.09.55.13 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 04 Nov 2024 09:55:16 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , Johannes Weiner , Roman Gushchin , Waiman Long , Shakeel Butt , Michal Hocko , Chengming Zhou , Qi Zheng , Muchun Song , Kairui Song Subject: [PATCH v3 5/6] mm/list_lru: split the lock to per-cgroup scope Date: Tue, 5 Nov 2024 01:52:56 +0800 Message-ID: <20241104175257.60853-6-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241104175257.60853-1-ryncsn@gmail.com> References: <20241104175257.60853-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 9336440018 X-Rspamd-Server: rspam11 X-Stat-Signature: 7hohc5cmycoj76me5i5b7pcinjt5kue5 X-HE-Tag: 1730742879-969172 X-HE-Meta: U2FsdGVkX19S3U+aLKGaHcMKumFF8I89vRhUtBGIoTOnWkXrtGkojbRws069TT0ZL0LVlWDzKmxTcxWjzJf7DDq/62Miawjw+8mFF5/WtF9M8jkMIrP42M/zVnx1w43aE2IzyRJMZEpG75rtqHFaLKPwdUqout9/CTki4BQpZ6vWwBwNGZ3A3U55zhfJWfVKgkG2PxI8Ned1qb1D8ABXgs8ocE6IeeNGysl655d47kitc1+oZ9Ke+6ZV/HW4MNN6YPMGxEq6IxxZ6fgKopVlC4NM9693mGMihShCBN1aykJbjPxGtyNYd2rKVLDgr/fR1aT0XGWfa0zGavNQPLUqILkRbGeJVJA7m2UtQevPtgjJMePLShmcYsxtnxQ5Q8IELmKfbrivAfp87mRecDSsGetDDnDDtMfjbNxqb26GP1CHsvHT+ioI2jNLuBjBLOrT7z6q2h8GK/h+6JebY9U4sAN13ZqT1SE9KBJGWPtkF+FAWOcaEemiDEjZOg+U261L0jFUDeo4p+07zuN08kMghSwDvjYsvYpx2g6LK6oD4xPG6eOHbnK25mlT/Ncb5DWMlv69srEq5BizH96xZJiUM466jLMfmsMZKohpiTflk0qk/bitCgVcFJDUoiAyDNpt/APF9K3MidlFxAH+xtNTUKBW3ysQBwau+b55Tygh4Z6alQpkQwrJ8rALdhg4v9ONg4og0J4jAT0X1WWiISjcVnsos6FFhlo5osslDFtroomBJpNlZatibhcP5tC2ZQaL/ogj2pkfoaUgNuCeCXcN3/8KRwVEZ+90FlDh1o3ZfP3J0Xou4wGKwoMbcsfTAmkU9I2Sh/JhVQ9HYkDtb+7n04A79G5uBs8CSJR51TYRQbybICGBBlzjnrkmFvD+1yMRZPKrYxeyx9V7kw/E1HG6+HD/FQdVmXsWbjVG4cQArHmW4Aeg1HFyXC+Z0XgZ+XH7qHnWsZOj2TGyHttRNeG Wx7Dw8WE wKQuqw45nIihhpQm5xlw4AGfZhqtntCVp3ZwCho+1BdxrpqA9CNMS39gfws3X7dtFzwpkhWQqwWnwaRiUsz+aw5e0HGFAZpGXdhxFTVYODMOxq+Pvwgul9aF94iILyKYrqQYnX6KqOVJcPD0wDUchJkvydo1dZq1TVLwZJEbMZIXH1P9xO2rk2L4ROJHT7u4gNfecFggmWJrCe7qEzvosC2CJTpdRcCHJ6Snjt8D+AO95T3aXn8jqTAJUdZH4q7tXND/BGaGvY9bd1Lj9O8fHw+MCAVchKXFn+aC95qRpvQz6kIpl2hVEs3Ve6WCX623o6IbcHxQpCCkLU1/buP59K4mGFhsiWHlTG80vHIFgxbbWFR0irhwfsCVGlrjAZE7nezK7EkJV85QzaaQmHOTfH5NDcViF3gm3H4U8yH3emLwjqYyFAsLT6WSATjvdFD7snd35Uh/ZrrFtKZMkO3mQyVoL9gTun9qdehQyCy7xLTEwMc5D+t0bMY2pUWCMGq9OvRviUpvqk0sUJAb7yWia9FAsvCZ+MjTmYMnHcC7TnOWGMDuEMy5koZS0L98+2PwgBjhM X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Currently, every list_lru has a per-node lock that protects adding, deletion, isolation, and reparenting of all list_lru_one instances belonging to this list_lru on this node. This lock contention is heavy when multiple cgroups modify the same list_lru. This lock can be split into per-cgroup scope to reduce contention. To achieve this, we need a stable list_lru_one for every cgroup. This commit adds a lock to each list_lru_one and introduced a helper function lock_list_lru_of_memcg, making it possible to pin the list_lru of a memcg. Then reworked the reparenting process. Reparenting will switch the list_lru_one instances one by one. By locking each instance and marking it dead using the nr_items counter, reparenting ensures that all items in the corresponding cgroup (on-list or not, because items have a stable cgroup, see below) will see the list_lru_one switch synchronously. Objcg reparent is also moved after list_lru reparent so items will have a stable mem cgroup until all list_lru_one instances are drained. The only caller that doesn't work the *_obj interfaces are direct calls to list_lru_{add,del}. But it's only used by zswap and that's also based on objcg, so it's fine. This also changes the bahaviour of the isolation function when LRU_RETRY or LRU_REMOVED_RETRY is returned, because now releasing the lock could unblock reparenting and free the list_lru_one, isolation function will have to return withoug re-lock the lru. prepare() { mkdir /tmp/test-fs modprobe brd rd_nr=1 rd_size=33554432 mkfs.xfs -f /dev/ram0 mount -t xfs /dev/ram0 /tmp/test-fs for i in $(seq 1 512); do mkdir "/tmp/test-fs/$i" for j in $(seq 1 10240); do echo TEST-CONTENT > "/tmp/test-fs/$i/$j" done & done; wait } do_test() { read_worker() { sleep 1 tar -cv "$1" &>/dev/null } read_in_all() { cd "/tmp/test-fs" && ls for i in $(seq 1 512); do (exec sh -c 'echo "$PPID"') > "/sys/fs/cgroup/benchmark/$i/cgroup.procs" read_worker "$i" & done; wait } for i in $(seq 1 512); do mkdir -p "/sys/fs/cgroup/benchmark/$i" done echo +memory > /sys/fs/cgroup/benchmark/cgroup.subtree_control echo 512M > /sys/fs/cgroup/benchmark/memory.max echo 3 > /proc/sys/vm/drop_caches time read_in_all } Above script simulates compression of small files in multiple cgroups with memory pressure. Run prepare() then do_test for 6 times: Before: real 0m7.762s user 0m11.340s sys 3m11.224s real 0m8.123s user 0m11.548s sys 3m2.549s real 0m7.736s user 0m11.515s sys 3m11.171s real 0m8.539s user 0m11.508s sys 3m7.618s real 0m7.928s user 0m11.349s sys 3m13.063s real 0m8.105s user 0m11.128s sys 3m14.313s After this commit (about ~15% faster): real 0m6.953s user 0m11.327s sys 2m42.912s real 0m7.453s user 0m11.343s sys 2m51.942s real 0m6.916s user 0m11.269s sys 2m43.957s real 0m6.894s user 0m11.528s sys 2m45.346s real 0m6.911s user 0m11.095s sys 2m43.168s real 0m6.773s user 0m11.518s sys 2m40.774s Signed-off-by: Kairui Song --- drivers/android/binder_alloc.c | 1 - fs/inode.c | 1 - fs/xfs/xfs_qm.c | 1 - include/linux/list_lru.h | 6 +- mm/list_lru.c | 216 +++++++++++++++++++-------------- mm/memcontrol.c | 7 +- mm/workingset.c | 1 - mm/zswap.c | 5 +- 8 files changed, 135 insertions(+), 103 deletions(-) diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index b3acbc4174fb..86bbe40f4bcd 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -1106,7 +1106,6 @@ enum lru_status binder_alloc_free_page(struct list_head *item, mmput_async(mm); __free_page(page_to_free); - spin_lock(lock); return LRU_REMOVED_RETRY; err_invalid_vma: diff --git a/fs/inode.c b/fs/inode.c index 8dabb224f941..442cb4fc09b2 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -934,7 +934,6 @@ static enum lru_status inode_lru_isolate(struct list_head *item, mm_account_reclaimed_pages(reap); } inode_unpin_lru_isolating(inode); - spin_lock(lru_lock); return LRU_RETRY; } diff --git a/fs/xfs/xfs_qm.c b/fs/xfs/xfs_qm.c index 7e2307921deb..665d26990b78 100644 --- a/fs/xfs/xfs_qm.c +++ b/fs/xfs/xfs_qm.c @@ -496,7 +496,6 @@ xfs_qm_dquot_isolate( trace_xfs_dqreclaim_busy(dqp); XFS_STATS_INC(dqp->q_mount, xs_qm_dqreclaim_misses); xfs_dqunlock(dqp); - spin_lock(lru_lock); return LRU_RETRY; } diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index eba93f6511f3..10ba9a54d42c 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -32,6 +32,8 @@ struct list_lru_one { struct list_head list; /* may become negative during memcg reparenting */ long nr_items; + /* protects all fields above */ + spinlock_t lock; }; struct list_lru_memcg { @@ -41,11 +43,9 @@ struct list_lru_memcg { }; struct list_lru_node { - /* protects all lists on the node, including per cgroup */ - spinlock_t lock; /* global list, used for the root cgroup in cgroup aware lrus */ struct list_lru_one lru; - long nr_items; + atomic_long_t nr_items; } ____cacheline_aligned_in_smp; struct list_lru { diff --git a/mm/list_lru.c b/mm/list_lru.c index 172b16146e15..c139202e27f7 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -61,18 +61,51 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) } static inline struct list_lru_one * -list_lru_from_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg) +lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg, + bool irq, bool skip_empty) { struct list_lru_one *l; + long nr_items; + + rcu_read_lock(); again: l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); - if (likely(l)) - return l; - - memcg = parent_mem_cgroup(memcg); + if (likely(l)) { + if (irq) + spin_lock_irq(&l->lock); + else + spin_lock(&l->lock); + nr_items = READ_ONCE(l->nr_items); + if (likely(nr_items != LONG_MIN)) { + WARN_ON(nr_items < 0); + rcu_read_unlock(); + return l; + } + if (irq) + spin_unlock_irq(&l->lock); + else + spin_unlock(&l->lock); + } + /* + * Caller may simply bail out if raced with reparenting or + * may iterate through the list_lru and expect empty slots. + */ + if (skip_empty) { + rcu_read_unlock(); + return NULL; + } VM_WARN_ON(!css_is_dying(&memcg->css)); + memcg = parent_mem_cgroup(memcg); goto again; } + +static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off) +{ + if (irq_off) + spin_unlock_irq(&l->lock); + else + spin_unlock(&l->lock); +} #else static void list_lru_register(struct list_lru *lru) { @@ -99,31 +132,48 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) } static inline struct list_lru_one * -list_lru_from_memcg(struct list_lru *lru, int nid, int idx) +lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg, + bool irq, bool skip_empty) { - return &lru->node[nid].lru; + struct list_lru_one *l = &lru->node[nid].lru; + + if (irq) + spin_lock_irq(&l->lock); + else + spin_lock(&l->lock); + + return l; +} + +static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off) +{ + if (irq_off) + spin_unlock_irq(&l->lock); + else + spin_unlock(&l->lock); } #endif /* CONFIG_MEMCG */ /* The caller must ensure the memcg lifetime. */ bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid, - struct mem_cgroup *memcg) + struct mem_cgroup *memcg) { struct list_lru_node *nlru = &lru->node[nid]; struct list_lru_one *l; - spin_lock(&nlru->lock); + l = lock_list_lru_of_memcg(lru, nid, memcg, false, false); + if (!l) + return false; if (list_empty(item)) { - l = list_lru_from_memcg(lru, nid, memcg); list_add_tail(item, &l->list); /* Set shrinker bit if the first element was added */ if (!l->nr_items++) set_shrinker_bit(memcg, nid, lru_shrinker_id(lru)); - nlru->nr_items++; - spin_unlock(&nlru->lock); + unlock_list_lru(l, false); + atomic_long_inc(&nlru->nr_items); return true; } - spin_unlock(&nlru->lock); + unlock_list_lru(l, false); return false; } @@ -146,24 +196,23 @@ EXPORT_SYMBOL_GPL(list_lru_add_obj); /* The caller must ensure the memcg lifetime. */ bool list_lru_del(struct list_lru *lru, struct list_head *item, int nid, - struct mem_cgroup *memcg) + struct mem_cgroup *memcg) { struct list_lru_node *nlru = &lru->node[nid]; struct list_lru_one *l; - - spin_lock(&nlru->lock); + l = lock_list_lru_of_memcg(lru, nid, memcg, false, false); + if (!l) + return false; if (!list_empty(item)) { - l = list_lru_from_memcg(lru, nid, memcg); list_del_init(item); l->nr_items--; - nlru->nr_items--; - spin_unlock(&nlru->lock); + unlock_list_lru(l, false); + atomic_long_dec(&nlru->nr_items); return true; } - spin_unlock(&nlru->lock); + unlock_list_lru(l, false); return false; } -EXPORT_SYMBOL_GPL(list_lru_del); bool list_lru_del_obj(struct list_lru *lru, struct list_head *item) { @@ -220,25 +269,24 @@ unsigned long list_lru_count_node(struct list_lru *lru, int nid) struct list_lru_node *nlru; nlru = &lru->node[nid]; - return nlru->nr_items; + return atomic_long_read(&nlru->nr_items); } EXPORT_SYMBOL_GPL(list_lru_count_node); static unsigned long -__list_lru_walk_one(struct list_lru *lru, int nid, int memcg_idx, +__list_lru_walk_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg, list_lru_walk_cb isolate, void *cb_arg, - unsigned long *nr_to_walk) + unsigned long *nr_to_walk, bool irq_off) { struct list_lru_node *nlru = &lru->node[nid]; - struct list_lru_one *l; + struct list_lru_one *l = NULL; struct list_head *item, *n; unsigned long isolated = 0; restart: - l = list_lru_from_memcg_idx(lru, nid, memcg_idx); + l = lock_list_lru_of_memcg(lru, nid, memcg, irq_off, true); if (!l) - goto out; - + return isolated; list_for_each_safe(item, n, &l->list) { enum lru_status ret; @@ -250,19 +298,19 @@ __list_lru_walk_one(struct list_lru *lru, int nid, int memcg_idx, break; --*nr_to_walk; - ret = isolate(item, l, &nlru->lock, cb_arg); + ret = isolate(item, l, &l->lock, cb_arg); switch (ret) { + /* + * LRU_RETRY, LRU_REMOVED_RETRY and LRU_STOP will drop the lru + * lock. List traversal will have to restart from scratch. + */ + case LRU_RETRY: + goto restart; case LRU_REMOVED_RETRY: - assert_spin_locked(&nlru->lock); fallthrough; case LRU_REMOVED: isolated++; - nlru->nr_items--; - /* - * If the lru lock has been dropped, our list - * traversal is now invalid and so we have to - * restart from scratch. - */ + atomic_long_dec(&nlru->nr_items); if (ret == LRU_REMOVED_RETRY) goto restart; break; @@ -271,20 +319,13 @@ __list_lru_walk_one(struct list_lru *lru, int nid, int memcg_idx, break; case LRU_SKIP: break; - case LRU_RETRY: - /* - * The lru lock has been dropped, our list traversal is - * now invalid and so we have to restart from scratch. - */ - assert_spin_locked(&nlru->lock); - goto restart; case LRU_STOP: - assert_spin_locked(&nlru->lock); goto out; default: BUG(); } } + unlock_list_lru(l, irq_off); out: return isolated; } @@ -294,14 +335,8 @@ list_lru_walk_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg, list_lru_walk_cb isolate, void *cb_arg, unsigned long *nr_to_walk) { - struct list_lru_node *nlru = &lru->node[nid]; - unsigned long ret; - - spin_lock(&nlru->lock); - ret = __list_lru_walk_one(lru, nid, memcg_kmem_id(memcg), isolate, - cb_arg, nr_to_walk); - spin_unlock(&nlru->lock); - return ret; + return __list_lru_walk_one(lru, nid, memcg, isolate, + cb_arg, nr_to_walk, false); } EXPORT_SYMBOL_GPL(list_lru_walk_one); @@ -310,14 +345,8 @@ list_lru_walk_one_irq(struct list_lru *lru, int nid, struct mem_cgroup *memcg, list_lru_walk_cb isolate, void *cb_arg, unsigned long *nr_to_walk) { - struct list_lru_node *nlru = &lru->node[nid]; - unsigned long ret; - - spin_lock_irq(&nlru->lock); - ret = __list_lru_walk_one(lru, nid, memcg_kmem_id(memcg), isolate, - cb_arg, nr_to_walk); - spin_unlock_irq(&nlru->lock); - return ret; + return __list_lru_walk_one(lru, nid, memcg, isolate, + cb_arg, nr_to_walk, true); } unsigned long list_lru_walk_node(struct list_lru *lru, int nid, @@ -332,16 +361,21 @@ unsigned long list_lru_walk_node(struct list_lru *lru, int nid, #ifdef CONFIG_MEMCG if (*nr_to_walk > 0 && list_lru_memcg_aware(lru)) { struct list_lru_memcg *mlru; + struct mem_cgroup *memcg; unsigned long index; xa_for_each(&lru->xa, index, mlru) { - struct list_lru_node *nlru = &lru->node[nid]; - - spin_lock(&nlru->lock); - isolated += __list_lru_walk_one(lru, nid, index, + rcu_read_lock(); + memcg = mem_cgroup_from_id(index); + if (!mem_cgroup_tryget(memcg)) { + rcu_read_unlock(); + continue; + } + rcu_read_unlock(); + isolated += __list_lru_walk_one(lru, nid, memcg, isolate, cb_arg, - nr_to_walk); - spin_unlock(&nlru->lock); + nr_to_walk, false); + mem_cgroup_put(memcg); if (*nr_to_walk <= 0) break; @@ -353,14 +387,19 @@ unsigned long list_lru_walk_node(struct list_lru *lru, int nid, } EXPORT_SYMBOL_GPL(list_lru_walk_node); -static void init_one_lru(struct list_lru_one *l) +static void init_one_lru(struct list_lru *lru, struct list_lru_one *l) { INIT_LIST_HEAD(&l->list); + spin_lock_init(&l->lock); l->nr_items = 0; +#ifdef CONFIG_LOCKDEP + if (lru->key) + lockdep_set_class(&l->lock, lru->key); +#endif } #ifdef CONFIG_MEMCG -static struct list_lru_memcg *memcg_init_list_lru_one(gfp_t gfp) +static struct list_lru_memcg *memcg_init_list_lru_one(struct list_lru *lru, gfp_t gfp) { int nid; struct list_lru_memcg *mlru; @@ -370,7 +409,7 @@ static struct list_lru_memcg *memcg_init_list_lru_one(gfp_t gfp) return NULL; for_each_node(nid) - init_one_lru(&mlru->node[nid]); + init_one_lru(lru, &mlru->node[nid]); return mlru; } @@ -398,28 +437,27 @@ static void memcg_destroy_list_lru(struct list_lru *lru) xas_unlock_irq(&xas); } -static void memcg_reparent_list_lru_node(struct list_lru *lru, int nid, - struct list_lru_one *src, - struct mem_cgroup *dst_memcg) +static void memcg_reparent_list_lru_one(struct list_lru *lru, int nid, + struct list_lru_one *src, + struct mem_cgroup *dst_memcg) { - struct list_lru_node *nlru = &lru->node[nid]; + int dst_idx = dst_memcg->kmemcg_id; struct list_lru_one *dst; - /* - * Since list_lru_{add,del} may be called under an IRQ-safe lock, - * we have to use IRQ-safe primitives here to avoid deadlock. - */ - spin_lock_irq(&nlru->lock); - dst = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(dst_memcg)); + spin_lock_irq(&src->lock); + dst = list_lru_from_memcg_idx(lru, nid, dst_idx); + spin_lock_nested(&dst->lock, SINGLE_DEPTH_NESTING); list_splice_init(&src->list, &dst->list); - if (src->nr_items) { dst->nr_items += src->nr_items; set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); - src->nr_items = 0; } - spin_unlock_irq(&nlru->lock); + /* Mark the list_lru_one dead */ + src->nr_items = LONG_MIN; + + spin_unlock(&dst->lock); + spin_unlock_irq(&src->lock); } void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent) @@ -448,7 +486,7 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren * safe to reparent. */ for_each_node(i) - memcg_reparent_list_lru_node(lru, i, &mlru->node[i], parent); + memcg_reparent_list_lru_one(lru, i, &mlru->node[i], parent); /* * Here all list_lrus corresponding to the cgroup are guaranteed @@ -497,7 +535,7 @@ int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, parent = parent_mem_cgroup(pos); } - mlru = memcg_init_list_lru_one(gfp); + mlru = memcg_init_list_lru_one(lru, gfp); if (!mlru) return -ENOMEM; xas_set(&xas, pos->kmemcg_id); @@ -544,14 +582,8 @@ int __list_lru_init(struct list_lru *lru, bool memcg_aware, struct shrinker *shr if (!lru->node) return -ENOMEM; - for_each_node(i) { - spin_lock_init(&lru->node[i].lock); -#ifdef CONFIG_LOCKDEP - if (lru->key) - lockdep_set_class(&lru->node[i].lock, lru->key); -#endif - init_one_lru(&lru->node[i].lru); - } + for_each_node(i) + init_one_lru(lru, &lru->node[i].lru); memcg_init_list_lru(lru, memcg_aware); list_lru_register(lru); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 8e90aa026c47..f421dfcfe8a1 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3098,8 +3098,13 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) if (!parent) parent = root_mem_cgroup; - memcg_reparent_objcgs(memcg, parent); memcg_reparent_list_lrus(memcg, parent); + + /* + * Objcg's reparenting must be after list_lru's, make sure list_lru + * helpers won't use parent's list_lru until child is drained. + */ + memcg_reparent_objcgs(memcg, parent); } #ifdef CONFIG_CGROUP_WRITEBACK diff --git a/mm/workingset.c b/mm/workingset.c index df3937c5eedc..8c4b6738dcad 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -777,7 +777,6 @@ static enum lru_status shadow_lru_isolate(struct list_head *item, ret = LRU_REMOVED_RETRY; out: cond_resched(); - spin_lock_irq(lru_lock); return ret; } diff --git a/mm/zswap.c b/mm/zswap.c index 6910c37cb8ec..63bcd94dc2cb 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -705,9 +705,9 @@ static void zswap_lru_add(struct list_lru *list_lru, struct zswap_entry *entry) * Note that it is safe to use rcu_read_lock() here, even in the face of * concurrent memcg offlining: * - * 1. list_lru_add() is called before list_lru_memcg is erased. The + * 1. list_lru_add() is called before list_lru_one is dead. The * new entry will be reparented to memcg's parent's list_lru. - * 2. list_lru_add() is called after list_lru_memcg is erased. The + * 2. list_lru_add() is called after list_lru_one is dead. The * new entry will be added directly to memcg's parent's list_lru. * * Similar reasoning holds for list_lru_del(). @@ -1172,7 +1172,6 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o zswap_written_back_pages++; } - spin_lock(lock); return ret; } From patchwork Mon Nov 4 17:52:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13861762 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BA40D1BDD5 for ; Mon, 4 Nov 2024 17:55:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A16766B009D; Mon, 4 Nov 2024 12:55:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9A0106B009E; Mon, 4 Nov 2024 12:55:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8406E6B009F; Mon, 4 Nov 2024 12:55:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5A56C6B009D for ; Mon, 4 Nov 2024 12:55:27 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 0A4741205BE for ; Mon, 4 Nov 2024 17:55:27 +0000 (UTC) X-FDA: 82749163728.24.BDE8867 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) by imf23.hostedemail.com (Postfix) with ESMTP id 1034D140004 for ; Mon, 4 Nov 2024 17:55:03 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ti8JuirC; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf23.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.175 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730742758; a=rsa-sha256; cv=none; b=SDQGQD2eH15p0JY2rYIJwgQZdN+kfGcAYybBbCdcfNrIwa7Ck0K+58xkAnDebZEoUPT/mu W7+Ik0kX92FlU6SHpXDKI16+8z70kPDRKmFax9Ms2o8dGf8O42jMS+oxQRjlP0d4Kbvtvi fkEpZ8vvqK8vYV5XEUvfslTL1UbFeQk= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ti8JuirC; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf23.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.175 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730742758; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EnhhoKxkODIhYY9nEI9AbHmj0CiiAcnFinSSL27vrKc=; b=hk95ZxElmzmOBSKhHRgYBB+8E3ErOWUTSi7my5NeLLo8VupOcee9z+Oofned7nT/Kaypu7 qXsjjIaB3F+lnP+Qk6j163Xl0nPpBxgTaZuQSMSvNWP1zZfEv+vio/N0H4pScIhU1H82R3 vLD/PgZNnD3K+leRpUTgc7wLcHLLLtQ= Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-72097a5ca74so4298182b3a.3 for ; Mon, 04 Nov 2024 09:55:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730742923; x=1731347723; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=EnhhoKxkODIhYY9nEI9AbHmj0CiiAcnFinSSL27vrKc=; b=Ti8JuirCYnZ0Ssb2dz0L+P4Adnl89q5gLCS06ZTpKLLnksgWCMTdzlfgCnCp1DU5cz xRn9djG8Lv5Vfct41hR/9PeqiUEA1VgDt6B1dA1t7U882V4B2d9L337jh80zAwOSua44 kpfHYeydSWg1S6QYbTlhdtMzW4/PMi06Z66mjMrnS2P4CNhc3upGHRNff+wD1g8+AbkW tl+r2U93VblmRzhBQJ/ODVk0i2zISmyKIeFA7DPFFkAlB6T4VEQel9v2mVZnWxh0Uj+/ UmRUz1DR7msvBQxblU9g87ptlnTIKlTdJ3cy3o3oWx89g8wOrh6zG5NSio4g4IDTqb1J lbsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730742923; x=1731347723; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=EnhhoKxkODIhYY9nEI9AbHmj0CiiAcnFinSSL27vrKc=; b=looK5K2biBEcKdMHJUXIsK9sjKa5fXUypbgh/y0HEuBO1E30+gZJpseN/hEpgvb9Cn 5ISSfL05TTc6ypV6teMla9GqdlPdX4j6V4YsquPSRJFSEN0Kkt1Tph0aYRD8ikxU+PBk 8/qYBynwXyRZHX+AuDoX2D49/giM+rFCAIV8d3aAVUr34VF+GzvtoT+GorVxLvLG7Tt9 eCYBVm3AT0liHojHYVIPP0hZuuO8zIQkKv8zjI7z4jPlkCnAPR1+crZ47vmqZhBeHDlh mA+nxpcZeRvcruBCyNfgdwv7mtkGCTszDWM1FTXCzAlojrdS8Fhyly/GPCUejB2kFXOR P+BQ== X-Gm-Message-State: AOJu0YybCzuV8Q8vxhn2Dktr4vYRJ21CgSZYQekmTUaDU/jbZ/ST+ipC grfjpeE5rFTXesvOOUo858sEccuB6/dIKsJAuok3t2m1DhgVWyWnEfhAsOOceuM= X-Google-Smtp-Source: AGHT+IEXhVUoQHVQ9mj0BiltiQonsY9Pm1Uh5kuFjg7mCL6RFv4KpU1NPGiVjmgKBw1fK7XGmymJqw== X-Received: by 2002:a05:6a21:3386:b0:1db:ed17:dd7e with SMTP id adf61e73a8af0-1dbed17deb9mr1166678637.16.1730742921721; Mon, 04 Nov 2024 09:55:21 -0800 (PST) Received: from KASONG-MC4.tencent.com ([106.37.121.39]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-720bc1e5f5bsm7777039b3a.55.2024.11.04.09.55.17 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 04 Nov 2024 09:55:21 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , Johannes Weiner , Roman Gushchin , Waiman Long , Shakeel Butt , Michal Hocko , Chengming Zhou , Qi Zheng , Muchun Song , Kairui Song Subject: [PATCH v3 6/6] mm/list_lru: Simplify the list_lru walk callback function Date: Tue, 5 Nov 2024 01:52:57 +0800 Message-ID: <20241104175257.60853-7-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241104175257.60853-1-ryncsn@gmail.com> References: <20241104175257.60853-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Stat-Signature: s6yckkxoddesg57zjqox6uwweasomksx X-Rspamd-Queue-Id: 1034D140004 X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1730742903-28061 X-HE-Meta: U2FsdGVkX18XIR1whs/wLxC5ZWi3yFv0d70+yK4W+Nl3gP4gCCXYBp2A1c56RNMH9F9IS1nd7LX5ZWdqjbJyKYmJV/rQ8rUI1YCA+9QxdC0SYktheNRwL/U+hh7AoohegLX5f6WDyvrAxJGJQNNkb9nDTp7EqmuJmkXHKJY1O46/RlZhkBQ9AFYRsqf2tQ4kJvaNkDdf75qQOkQYuqobPLAGbxdvvqsHl7c4yDJVjSVAxCYiUIr35DjK27Ji4HYG4VdNFH87Aj/QQuUYGfWDnrgRczXgQ6Kl29uHHYAow4AoBVSn/KmuLs+crrR3QNK7ll/Y2g3xWUG1JBTGGYpkGtHh147xpm6gz256wrjxsZ5J/u3w4adZXksFhcF1vlj45CPLTqDA6XeE0KoCHhw71AlHUCt+ADkp7U5xzuSjSmLablm4dwlqlj8L5eJuq3Hx7ivifSbYQbye0/tNHFx4x9vu4BrghNFzIy2+uR5HwLW4M3DbxkixybDo43NEq2MOcf5dA9dzDuGVKpVPUeliieQviw9iCcjn+yOYx/JD+fOvHeBtxuDMEaGSRIUg51GWgIElkoxUmBVHNsprJt5u0RSp261Ro/R4sDI4hWUwi1AnBX4G4J3s1mlBSWP4/rEWPHVRAiZSk7YRqKpPH/nDEhkxnmrBGuJFxLGx9uwT8RxRIzkRIGxN2e8MUMQO9J5Y/FaK1Jeh0aStW9bdThhI/T1gCdkwLSSN8el1/g3Lc5ixf6HR0mK2GJTD1IvYSWHDZrCly4tZWqwjUdle5BBvIor/GmvLEJAQYOgdrMNscE9+S1w5sg+oZ/Ay2Y+mWCap7mhnxFGKq1PSob1qymzx2PcrTrqIMrB1NeWUnFemcp7/HdrBmZQZTM8xCWyJyi0IwKO/Ewre1jy6dnxAEK9id1eCrwB0QlJsyxM4SmJkNIoNjzR3au8j8xnPZ5CNEg5EWFgbasI8BZYsQ8NlGtY mHLFgEbE PZsu2aMZpU8iN969+fwLDXw5Fbva8ZoBkq/pBki23RUxvsVBlb7GXvrtkoG5pk7pkN2JWKKcKAqVrsx848CMN6s9NaukIZ9pkmfqjnSLYnridQIMsgfNQkzxJVSzKa50RSkeNq/+tnXF2/+P6a56qT+2Vvi+jm2WN1ItpLNGmM4wjQkAdAoz8MxKPf2b7WxrZQLyzGFSxixJvjX2f2TwHshTOOx4LGQFMm9avKFc7fXtsVRzO0zikoCRnAcQ9hpi8e1qn/awK0A63ZEMHfXnXfcYD+PPbRNsfifRouFXZYi6DVw8J+8HlO1bKzOhsLsJ+oZcpzkQKGxXriVYMAl3o/hX3MmA6L+XPBs6iyC10fIrOiJ4z+EjppP4iplX8qKQ/sqwWd+KnYt96KXQOkMjDiTRo4PONO2FiPdmVmV+2vmFzS0/JfPQWer/rgGL3C6P+L3k0Lx5MDUz0toXb1aAuL6nsClvxyHQ+/n6atq11IJi8S/XqEhtTjKmxe47pBzUOWkknNNUML6saMD7OKMHjpJ/IGmWGZKJo5yct7v58Ijljxp1S13/wCkF/gL8Yq//4WZss X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Now isolation no longer takes the list_lru global node lock, only use the per-cgroup lock instead. And this lock is inside the list_lru_one being walked, no longer needed to pass the lock explicitly. Signed-off-by: Kairui Song --- drivers/android/binder_alloc.c | 7 +++---- drivers/android/binder_alloc.h | 2 +- fs/dcache.c | 4 ++-- fs/gfs2/quota.c | 2 +- fs/inode.c | 4 ++-- fs/nfs/nfs42xattr.c | 4 ++-- fs/nfsd/filecache.c | 5 +---- fs/xfs/xfs_buf.c | 2 -- fs/xfs/xfs_qm.c | 5 ++--- include/linux/list_lru.h | 2 +- mm/list_lru.c | 2 +- mm/workingset.c | 15 +++++++-------- mm/zswap.c | 4 ++-- 13 files changed, 25 insertions(+), 33 deletions(-) diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 86bbe40f4bcd..a738e7745865 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -1047,7 +1047,7 @@ void binder_alloc_vma_close(struct binder_alloc *alloc) /** * binder_alloc_free_page() - shrinker callback to free pages * @item: item to free - * @lock: lock protecting the item + * @lru: list_lru instance of the item * @cb_arg: callback argument * * Called from list_lru_walk() in binder_shrink_scan() to free @@ -1055,9 +1055,8 @@ void binder_alloc_vma_close(struct binder_alloc *alloc) */ enum lru_status binder_alloc_free_page(struct list_head *item, struct list_lru_one *lru, - spinlock_t *lock, void *cb_arg) - __must_hold(lock) + __must_hold(&lru->lock) { struct binder_lru_page *page = container_of(item, typeof(*page), lru); struct binder_alloc *alloc = page->alloc; @@ -1092,7 +1091,7 @@ enum lru_status binder_alloc_free_page(struct list_head *item, list_lru_isolate(lru, item); spin_unlock(&alloc->lock); - spin_unlock(lock); + spin_unlock(&lru->lock); if (vma) { trace_binder_unmap_user_start(alloc, index); diff --git a/drivers/android/binder_alloc.h b/drivers/android/binder_alloc.h index 70387234477e..c02c8ebcb466 100644 --- a/drivers/android/binder_alloc.h +++ b/drivers/android/binder_alloc.h @@ -118,7 +118,7 @@ static inline void binder_selftest_alloc(struct binder_alloc *alloc) {} #endif enum lru_status binder_alloc_free_page(struct list_head *item, struct list_lru_one *lru, - spinlock_t *lock, void *cb_arg); + void *cb_arg); struct binder_buffer *binder_alloc_new_buf(struct binder_alloc *alloc, size_t data_size, size_t offsets_size, diff --git a/fs/dcache.c b/fs/dcache.c index 0f6b16ba30d0..d7f6866f5f52 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -1089,7 +1089,7 @@ void shrink_dentry_list(struct list_head *list) } static enum lru_status dentry_lru_isolate(struct list_head *item, - struct list_lru_one *lru, spinlock_t *lru_lock, void *arg) + struct list_lru_one *lru, void *arg) { struct list_head *freeable = arg; struct dentry *dentry = container_of(item, struct dentry, d_lru); @@ -1170,7 +1170,7 @@ long prune_dcache_sb(struct super_block *sb, struct shrink_control *sc) } static enum lru_status dentry_lru_isolate_shrink(struct list_head *item, - struct list_lru_one *lru, spinlock_t *lru_lock, void *arg) + struct list_lru_one *lru, void *arg) { struct list_head *freeable = arg; struct dentry *dentry = container_of(item, struct dentry, d_lru); diff --git a/fs/gfs2/quota.c b/fs/gfs2/quota.c index 2e6bc77f4f81..72b48f6f5561 100644 --- a/fs/gfs2/quota.c +++ b/fs/gfs2/quota.c @@ -149,7 +149,7 @@ static void gfs2_qd_list_dispose(struct list_head *list) static enum lru_status gfs2_qd_isolate(struct list_head *item, - struct list_lru_one *lru, spinlock_t *lru_lock, void *arg) + struct list_lru_one *lru, void *arg) { struct list_head *dispose = arg; struct gfs2_quota_data *qd = diff --git a/fs/inode.c b/fs/inode.c index 442cb4fc09b2..46fbd5b23482 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -881,7 +881,7 @@ void invalidate_inodes(struct super_block *sb) * with this flag set because they are the inodes that are out of order. */ static enum lru_status inode_lru_isolate(struct list_head *item, - struct list_lru_one *lru, spinlock_t *lru_lock, void *arg) + struct list_lru_one *lru, void *arg) { struct list_head *freeable = arg; struct inode *inode = container_of(item, struct inode, i_lru); @@ -923,7 +923,7 @@ static enum lru_status inode_lru_isolate(struct list_head *item, if (inode_has_buffers(inode) || !mapping_empty(&inode->i_data)) { inode_pin_lru_isolating(inode); spin_unlock(&inode->i_lock); - spin_unlock(lru_lock); + spin_unlock(&lru->lock); if (remove_inode_buffers(inode)) { unsigned long reap; reap = invalidate_mapping_pages(&inode->i_data, 0, -1); diff --git a/fs/nfs/nfs42xattr.c b/fs/nfs/nfs42xattr.c index b6e3d8f77b91..37d79400e5f4 100644 --- a/fs/nfs/nfs42xattr.c +++ b/fs/nfs/nfs42xattr.c @@ -802,7 +802,7 @@ static struct shrinker *nfs4_xattr_large_entry_shrinker; static enum lru_status cache_lru_isolate(struct list_head *item, - struct list_lru_one *lru, spinlock_t *lru_lock, void *arg) + struct list_lru_one *lru, void *arg) { struct list_head *dispose = arg; struct inode *inode; @@ -867,7 +867,7 @@ nfs4_xattr_cache_count(struct shrinker *shrink, struct shrink_control *sc) static enum lru_status entry_lru_isolate(struct list_head *item, - struct list_lru_one *lru, spinlock_t *lru_lock, void *arg) + struct list_lru_one *lru, void *arg) { struct list_head *dispose = arg; struct nfs4_xattr_bucket *bucket; diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c index 2e6783f63712..09c444eb944f 100644 --- a/fs/nfsd/filecache.c +++ b/fs/nfsd/filecache.c @@ -487,7 +487,6 @@ void nfsd_file_net_dispose(struct nfsd_net *nn) * nfsd_file_lru_cb - Examine an entry on the LRU list * @item: LRU entry to examine * @lru: controlling LRU - * @lock: LRU list lock (unused) * @arg: dispose list * * Return values: @@ -497,9 +496,7 @@ void nfsd_file_net_dispose(struct nfsd_net *nn) */ static enum lru_status nfsd_file_lru_cb(struct list_head *item, struct list_lru_one *lru, - spinlock_t *lock, void *arg) - __releases(lock) - __acquires(lock) + void *arg) { struct list_head *head = arg; struct nfsd_file *nf = list_entry(item, struct nfsd_file, nf_lru); diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c index aa4dbda7b536..43b914c1f621 100644 --- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -1857,7 +1857,6 @@ static enum lru_status xfs_buftarg_drain_rele( struct list_head *item, struct list_lru_one *lru, - spinlock_t *lru_lock, void *arg) { @@ -1956,7 +1955,6 @@ static enum lru_status xfs_buftarg_isolate( struct list_head *item, struct list_lru_one *lru, - spinlock_t *lru_lock, void *arg) { struct xfs_buf *bp = container_of(item, struct xfs_buf, b_lru); diff --git a/fs/xfs/xfs_qm.c b/fs/xfs/xfs_qm.c index 665d26990b78..8413ac368042 100644 --- a/fs/xfs/xfs_qm.c +++ b/fs/xfs/xfs_qm.c @@ -412,9 +412,8 @@ static enum lru_status xfs_qm_dquot_isolate( struct list_head *item, struct list_lru_one *lru, - spinlock_t *lru_lock, void *arg) - __releases(lru_lock) __acquires(lru_lock) + __releases(&lru->lock) __acquires(&lru->lock) { struct xfs_dquot *dqp = container_of(item, struct xfs_dquot, q_lru); @@ -460,7 +459,7 @@ xfs_qm_dquot_isolate( trace_xfs_dqreclaim_dirty(dqp); /* we have to drop the LRU lock to flush the dquot */ - spin_unlock(lru_lock); + spin_unlock(&lru->lock); error = xfs_qm_dqflush(dqp, &bp); if (error) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index 10ba9a54d42c..05c166811f6b 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -184,7 +184,7 @@ void list_lru_isolate_move(struct list_lru_one *list, struct list_head *item, struct list_head *head); typedef enum lru_status (*list_lru_walk_cb)(struct list_head *item, - struct list_lru_one *list, spinlock_t *lock, void *cb_arg); + struct list_lru_one *list, void *cb_arg); /** * list_lru_walk_one: walk a @lru, isolating and disposing freeable items. diff --git a/mm/list_lru.c b/mm/list_lru.c index c139202e27f7..f93ada6a207b 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -298,7 +298,7 @@ __list_lru_walk_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg, break; --*nr_to_walk; - ret = isolate(item, l, &l->lock, cb_arg); + ret = isolate(item, l, cb_arg); switch (ret) { /* * LRU_RETRY, LRU_REMOVED_RETRY and LRU_STOP will drop the lru diff --git a/mm/workingset.c b/mm/workingset.c index 8c4b6738dcad..4b58ef535a17 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -712,8 +712,7 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker, static enum lru_status shadow_lru_isolate(struct list_head *item, struct list_lru_one *lru, - spinlock_t *lru_lock, - void *arg) __must_hold(lru_lock) + void *arg) __must_hold(lru->lock) { struct xa_node *node = container_of(item, struct xa_node, private_list); struct address_space *mapping; @@ -722,20 +721,20 @@ static enum lru_status shadow_lru_isolate(struct list_head *item, /* * Page cache insertions and deletions synchronously maintain * the shadow node LRU under the i_pages lock and the - * lru_lock. Because the page cache tree is emptied before - * the inode can be destroyed, holding the lru_lock pins any + * &lru->lock. Because the page cache tree is emptied before + * the inode can be destroyed, holding the &lru->lock pins any * address_space that has nodes on the LRU. * * We can then safely transition to the i_pages lock to * pin only the address_space of the particular node we want - * to reclaim, take the node off-LRU, and drop the lru_lock. + * to reclaim, take the node off-LRU, and drop the &lru->lock. */ mapping = container_of(node->array, struct address_space, i_pages); /* Coming from the list, invert the lock order */ if (!xa_trylock(&mapping->i_pages)) { - spin_unlock_irq(lru_lock); + spin_unlock_irq(&lru->lock); ret = LRU_RETRY; goto out; } @@ -744,7 +743,7 @@ static enum lru_status shadow_lru_isolate(struct list_head *item, if (mapping->host != NULL) { if (!spin_trylock(&mapping->host->i_lock)) { xa_unlock(&mapping->i_pages); - spin_unlock_irq(lru_lock); + spin_unlock_irq(&lru->lock); ret = LRU_RETRY; goto out; } @@ -753,7 +752,7 @@ static enum lru_status shadow_lru_isolate(struct list_head *item, list_lru_isolate(lru, item); __dec_node_page_state(virt_to_page(node), WORKINGSET_NODES); - spin_unlock(lru_lock); + spin_unlock(&lru->lock); /* * The nodes should only contain one or more shadow entries, diff --git a/mm/zswap.c b/mm/zswap.c index 63bcd94dc2cb..0e29a0b0db71 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1095,7 +1095,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, * for reclaim by this ratio. */ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_one *l, - spinlock_t *lock, void *arg) + void *arg) { struct zswap_entry *entry = container_of(item, struct zswap_entry, lru); bool *encountered_page_in_swapcache = (bool *)arg; @@ -1151,7 +1151,7 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o * It's safe to drop the lock here because we return either * LRU_REMOVED_RETRY or LRU_RETRY. */ - spin_unlock(lock); + spin_unlock(&l->lock); writeback_result = zswap_writeback_entry(entry, swpentry);