From patchwork Tue Apr 15 02:45:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051354 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7924C369B2 for ; Tue, 15 Apr 2025 02:46:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4948E28011F; Mon, 14 Apr 2025 22:46:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 418E12800C2; Mon, 14 Apr 2025 22:46:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2B9FF28011F; Mon, 14 Apr 2025 22:46:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0B0842800C2 for ; Mon, 14 Apr 2025 22:46:04 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 6445CC0B07 for ; Tue, 15 Apr 2025 02:46:04 +0000 (UTC) X-FDA: 83334738648.28.09CC8A7 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf21.hostedemail.com (Postfix) with ESMTP id 7FB4E1C0008 for ; Tue, 15 Apr 2025 02:46:02 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=DEnPhcsb; spf=pass (imf21.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685162; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=M8KzBx499Eh3TwCUOfkq+U0TMdvhQzDUhsoXtCDxKvc=; b=3j0kS1LCTzfnvpnQDdiBiqVMsc8+5+IiaXuFO0KMcVq+/Cuup6TyH15ypsjCUn1Updq7hl OlXQJYIAvTKYW8UEhkJrUGieE4DnqlersgKjdhhrwGWI0eDhcXJQE3OrxIsLdYf2DmWKyC WlKOtI0Iu3lSCLFVVtD5UBpwD/XZegU= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=DEnPhcsb; spf=pass (imf21.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685162; a=rsa-sha256; cv=none; b=yHTExIA962XEkGcrrKtCBMNiHOqwIchIjuCra1/2Xj2JQz7Jd0oA/zNE8vfryZakIGg2DD hrMQppVrTFX23dUHvbz9vWXVq1QTv5i80L4aW84cRDqhYX0aFUjxXwY7umtgsBbWsqo9fC EHKa4UbfXAsjvBOblpUgDsVTJMKsmqQ= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-224191d92e4so48464995ad.3 for ; Mon, 14 Apr 2025 19:46:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685161; x=1745289961; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=M8KzBx499Eh3TwCUOfkq+U0TMdvhQzDUhsoXtCDxKvc=; b=DEnPhcsb5xpx9PpjvyTaX2FmqZo1zB7pVD+1crqDpdTkQxfnyQjL/tMRkggsKma1xH ZXAqtpwKy40OgStUrxeXjR3q9XvfP34rC8ovlmkgwKAjwcpyFIvCKmE2yYgJ9Htm+GFa B+4uT8DkWw0rgdXQLLJVq0bdKM48j3SbE15m68lGS6reX6znP1DP40kf6YAEEZ30nnux 3eDe3N2buABk9ulSzQ1c1l3s0e+cq7LPpZdzONtXQM4OHBLinWEtDTs4IYvrCutpj8Gd RVOAoWEviyV1BeAXRkibsEa0kzEd1iLogfgckPDyt7RixE8Z/YFsT8WkB0cYsVa1Mc5l lqiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685161; x=1745289961; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=M8KzBx499Eh3TwCUOfkq+U0TMdvhQzDUhsoXtCDxKvc=; b=FfXVPqro7WYB/KEKzSqznFGuBZ8rPGMVsqfGKEK4lNPA4qzLyxmHNCf7jleGYrSfqr ivRR7kTiWlM+mNhyMsN6HC9rbw6RqMCJLBctM+lLJzhCR3qe3jiWHegnaMCH7XmqBHJd 8oWIlYxh5pq9gf6ntshjoPQWfyWohKxn/BbQZ6RtdWyFJs6tfXVAHtqdueWVVglFR+cF KTgznyuLc+QQhOTKUIygE5Z6A+kV2X1v6qCso//wadevQ6vNAKo78mDQOoVOGo6jAnfg 18InEBxZ0Xuz2qJEOTM6dVuD+hzKzbYDWpjvOSRnijoiG8dTsWieyOvhMuMc9P21XeLH poXA== X-Forwarded-Encrypted: i=1; AJvYcCXziR8N/qe8hr/SlcfSvBtoYcjzqwQo/3DoR57l4337FnCq+MROw0WtWMB4Jf5AGB5C/x8bt+W/Kw==@kvack.org X-Gm-Message-State: AOJu0YwkPybJIY0msQeF5/kyK8Tc5vBWONqULMbt+pSYy3vt7vbYawxh 6Snt4sPbBYx4lOOaFnvuCciVoLfIS6h1T2I6T9+BDTO0jWqWux00KuDtkN1z4Kw= X-Gm-Gg: ASbGncsgMLiCFamAoXXZXEuTtS4N5+fFUbODNG2vzOy6aKZvhJ8OEmCZ9PqPKBlvwPV 3PNFB6Yic1L5m8Dj1sW7fXuPYJUxlAY9lf/KNwSRBn3A0JDXOTyAogzwlR6vMkf/hJJ8Eys75mQ 6MDHvFfkinvJkVLtKVVHrRlaWORrhEUf0rFd1hm0qfcWEQZkcLuDu5C5crGeJR/CJBOKg54zUZi P+BertORlfsuLNAvcWstsb68SHx59KgcKmdxh54wrcxeZQZiv2320c0AEhDcVKFtrkZG239lwtA bcahnM7pAew8yG3HPvyb7vm+eZduXQ2E38M4E3g2tqUsfyVLvoYK8vbu0FYkYv6CJiKBwIt7 X-Google-Smtp-Source: AGHT+IGjusvGNjKHuKsmMzFyHvfJ71DX0fIRnpQJPK83MJ7aTNHlxqG7qwINGqRKJ0KYDTG88Uch2Q== X-Received: by 2002:a17:902:c40b:b0:223:88af:2c30 with SMTP id d9443c01a7336-22bea4b3c03mr161252765ad.16.1744685161415; Mon, 14 Apr 2025 19:46:01 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.45.56 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:46:00 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 01/28] mm: memcontrol: remove dead code of checking parent memory cgroup Date: Tue, 15 Apr 2025 10:45:05 +0800 Message-Id: <20250415024532.26632-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 7FB4E1C0008 X-Stat-Signature: gwhhbtf6b67trbca7thqxicurr9pf7dk X-HE-Tag: 1744685162-368520 X-HE-Meta: U2FsdGVkX18SG5LPXsPeXlPNu8jyuOHIEY1mWaf8o84j8KvEgTf53tXNnUeCq2vuXjUVZ6Vo3t8mlvFgC2BMy8e1Edi09adweGob8y117/u9KJitEctfr8/zeubdIxl/6se3yqN77Jn/E0uCY+w6NerbyfGlIDO+1aaQp70bq86SJhRTKdZCFa1P8rGfylIKMdAEf0ggFYgRfA04ObA95sL85bp045/1eR3FO7IqExH6/uxOZqjLTWbm2XC3TZpvvM654ihWOh12KiIxCRLpR3o3noZAYMiJviCof2kGZn+hNJa7MJS5nRd+JJzGyR1k8CZEF4odX1wqSc/IQNgav0IqRNJ/Li0jSmhm6xqa4tXEXeCCWmMHfUyrOdQb+5nz1LialpVJGGSozUux9JqD604VkIU8i/+m3N6JcguXYvpfC2KodmElqIpUmmC80yhXDbwPAjasmLnNfp7w8UVUFRa57iKIVDoBXKY7WhBJRtyUWW9ZKBNeK7w6gr7d55atN1aX9LPmYPYHw31A09D6SSVHJbteYips4y/ZQAsJ65nXYMwJe6wJLfV/Aq0sobLJ+4Kq2tfEyM3NkmKI/wXfrGYc80AXPXwO/yND82E8t3GCLc2x02G0cLatH5zrzh3N+jUQ+7TRiJyf6fLWh1Xj/aR+SDHXrlzKvWJfVN9vPanz0w533sU6M/XLYrGRo59qY/V/+tM4g9jpNIxTzP+MnaCh5xVnjnJXSCw9ekS4f0wm7tl0cteo8YKzz5/+Xp+6ZXXunV2THQs/C5HI0UD6X7I17xNfoAmmNO8qrAiwtGc9VM7+/9GqxnwLdYRkUiw9TQB5bCnX/OE+36QxSJdJJW8EWfAKH1YYmxA6kALlKxaPtYlQ9SaA7sQop7B9WIF9cRC2CiYnnInLlUXg6cbda+hrDuAwHkY7Dyc9/AqOgsb4uhREhaFbwOs/4vkJTjEscuPd71upDAgigievP1t Ybw37p7/ c3RFnXsPFSge0n0hp94C1vsysdUHTet6ssNoMTu1u15mU7QqzVzOAWux+MGRybmX+WJTOjAuc47QL280jS92KJrP0gxHFk+QUCHJxdpmkzm+U9WP6jRMRoL47cnCdKJqWg/SuS9ZJkO9G3Ls6C7vadSDv7FN4TP9GjBXZMeZ39p560c8N0Nl2KHsRJcQ2qN75Xrn3CyGjFHZm4HrL1jGHY6zwe/Jomr0Tn9sPDjDZtIde8sNVvcpreJdxZ3WWhJ+OdZyxUnTgKhjLoF/MgTsvtD0GGhdAHLXG4iaKQn36/6pc/2BGFHxJhUDc+Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Since the no-hierarchy mode has been deprecated after the commit: commit bef8620cd8e0 ("mm: memcg: deprecate the non-hierarchical mode"). As a result, parent_mem_cgroup() will not return NULL except when passing the root memcg, and the root memcg cannot be offline. Hence, it's safe to remove the check on the returned value of parent_mem_cgroup(). Remove the corresponding dead code. Signed-off-by: Muchun Song Acked-by: Roman Gushchin Acked-by: Johannes Weiner --- mm/memcontrol.c | 5 ----- mm/shrinker.c | 6 +----- 2 files changed, 1 insertion(+), 10 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 421740f1bcdc..61488e45cab2 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3196,9 +3196,6 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) return; parent = parent_mem_cgroup(memcg); - if (!parent) - parent = root_mem_cgroup; - memcg_reparent_list_lrus(memcg, parent); /* @@ -3489,8 +3486,6 @@ struct mem_cgroup *mem_cgroup_id_get_online(struct mem_cgroup *memcg) break; } memcg = parent_mem_cgroup(memcg); - if (!memcg) - memcg = root_mem_cgroup; } return memcg; } diff --git a/mm/shrinker.c b/mm/shrinker.c index 4a93fd433689..e8e092a2f7f4 100644 --- a/mm/shrinker.c +++ b/mm/shrinker.c @@ -286,14 +286,10 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) { int nid, index, offset; long nr; - struct mem_cgroup *parent; + struct mem_cgroup *parent = parent_mem_cgroup(memcg); struct shrinker_info *child_info, *parent_info; struct shrinker_info_unit *child_unit, *parent_unit; - parent = parent_mem_cgroup(memcg); - if (!parent) - parent = root_mem_cgroup; - /* Prevent from concurrent shrinker_info expand */ mutex_lock(&shrinker_mutex); for_each_node(nid) { From patchwork Tue Apr 15 02:45:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051355 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47F30C369B8 for ; Tue, 15 Apr 2025 02:46:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E3E47280127; Mon, 14 Apr 2025 22:46:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DEB832800C2; Mon, 14 Apr 2025 22:46:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C8B38280127; Mon, 14 Apr 2025 22:46:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A85752800C2 for ; Mon, 14 Apr 2025 22:46:09 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3A6AA5C53D for ; Tue, 15 Apr 2025 02:46:10 +0000 (UTC) X-FDA: 83334738900.09.2B2CA07 Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) by imf01.hostedemail.com (Postfix) with ESMTP id 57C6B4000A for ; Tue, 15 Apr 2025 02:46:08 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Y6vWKurA; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf01.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685168; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gvWIGpHR+GW9zZlduzgMx/Do9egOoQOFfamShBDZoJg=; b=L4efDpwFbJZ+kzOaOQO/Udl2j4CQy5IDmd7/CwXZHl3mBekuya9bylFH6XMeRCD6M9WjSp 7UgaedkKQMKVXgTvtxZKJPKNd7u218im5OXmzN2VG7OAcOFldicU/ERLX1dL843IPSzEYx mvlrZacbb2AyqpZLD54y5DYrRciV/Kc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685168; a=rsa-sha256; cv=none; b=DZ8Su/O1ZhEWmTPy1oAkABXnDSkajqi/Pr3/oVE2hQO4rM3O4WhjwAYIFAFhNFXycysmgw YN6CNuo3ZsWLDZxTxiOEe72ZAXSNB1YH6OAbV6vwLx7GUxZaoIWBSk2o6/XIXUozcvsgaM Vw3x+syww8h3N7vUFb3x9bjwyfOUlWc= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Y6vWKurA; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf01.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com Received: by mail-pg1-f171.google.com with SMTP id 41be03b00d2f7-aee79a0f192so3336902a12.3 for ; Mon, 14 Apr 2025 19:46:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685167; x=1745289967; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gvWIGpHR+GW9zZlduzgMx/Do9egOoQOFfamShBDZoJg=; b=Y6vWKurAbNg9oVhs6HMZc8ZtTitjvrAezF8tix5ppdT68+wlYbHoNEfsRDAqvA0Yd7 bqBTniPdNy3ta7rkcXR1yEBXVhsvtvtJ7zfHvZbpn9+Tge9xsRFttDUI4t5rwQqM3JMN Oil15WIKBsMgpo/Xr7PDFg0tHimxNj5DhDFzBVAY5wSlt4zxVIdkLoF8UB2Mz9PRq8tN tGZ+aJwOR8Ju3gXK/T89YFWPfUYyP1u0PXTOKfrrBgaqP52iUqAQqTEE78uhM31eN8yO K3Pgg5iM/E9W/2vi46WsrpUSDOVPPqEC8ZVRfKm+1fdT74n00LjVbHP3ywSMo+4+tEKv 5tuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685167; x=1745289967; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gvWIGpHR+GW9zZlduzgMx/Do9egOoQOFfamShBDZoJg=; b=j1o/wLax3qZR5MTQJJyf1pj0kiav3AWqnEuLeujozsShF7waTCRUJyh7Rx5nqMkx13 Rrd6D2n5wrCTqaZPEDFkLSiRdsPNcNkDHQzlD2qS4hNZCwXLbbyorXUkbS1rGnWhXGEO Zix3pXhoNmsyL9lFFGDuUROS8OrIMzP/UjQjP7JEKcb50XI9jlnNF4tWKVdBhh/T0jkl v8y3uTic2BVcKBabSLKjV7+zwkV2uMgCpb39KEHT6O9rqJQPOlkbPkV0KBphmKpKWCnv yQsMxzyILiXY1aAV+k11Qim70lAWhsXsduStjkiJM+kb5fQlSJ0Tdya1VXTxIKODABYY ZtqA== X-Forwarded-Encrypted: i=1; AJvYcCXWR8navABySznjYA6j8ZTbLCAGZsZwaZeVpmGL5TS+HYEUABV1UUhhnm6lwMiFfoEeVHaf/kK1Lw==@kvack.org X-Gm-Message-State: AOJu0YxkKMPBk/KnZA2SlkFZpqZMV/n8gY0MMfz6N5zqwh0lBvHzCMQU kBlFzC4iwxbfrV3zpED1x5gZ+ikezgqSg6tk9p/bDVTxHMGkHgYXJegXwBV3S+k= X-Gm-Gg: ASbGncsbfTHDS8/KTlxdTsNi8IUnKlVoS88qzIGi/OxLyLzrEkBZX1mSpnTY+JO7l0B e6EsgcTGyEfq8VGhlDJLPrqJonXofWzJ+mmDpaeCBaqwlPjkAQo9FSQQbe/yJNq+pAmB0jsGwS5 nShvhZYyLCvSnHe5WjCMm3gLt8czqJexQ5XUuyLDlAo0Uhtb1ws9HE2j65APaKEGRpcG5qRC6JW 2sD5mospcNC50uROPaUh3g196thQ/AWCPACFKrZGs1f+85CTi6OO5ivedsJ6A922BY8kePCW4mC wntKx3ZC611cQBfqFyfJI/Kywvhqhypi6gs4EjDQTk0MLWW0NmHv8SQri0TO2FDUmb52hm5R X-Google-Smtp-Source: AGHT+IH8Y9q5TPLiC8967x/6BZ0skn8uSLHYwF2c4d9LznyXXEA3aGsij5atdYTxBM4bP93XVTR95w== X-Received: by 2002:a17:90a:c105:b0:2fa:157e:c790 with SMTP id 98e67ed59e1d1-30823676374mr19082580a91.5.1744685166974; Mon, 14 Apr 2025 19:46:06 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.46.01 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:46:06 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 02/28] mm: memcontrol: use folio_memcg_charged() to avoid potential rcu lock holding Date: Tue, 15 Apr 2025 10:45:06 +0800 Message-Id: <20250415024532.26632-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 57C6B4000A X-Stat-Signature: jypon1kudt9x7qw4espjup13ay1p8fei X-Rspam-User: X-HE-Tag: 1744685168-602253 X-HE-Meta: U2FsdGVkX1+YAayjvA+6jlEnxhx8tNUaY7+/zbpDKuIMO8ac1yNWJD+51cXb2swk5+xrYzP+F+X61M4FPJAHIdeqlbIuyBDGqmyJlwYYKO1CquNoYvUI0mwnRB94v13nMMmufSoQaRU8jjTvLUHC+FJV+5ZzE3pDXb4Hsepe1LJrPs3NHSscS6J3Lm216sc4uFFKLb7FA3tJoEBPJZdc8HDVNB/r23qi9TyyDr1eap3N4/gwf+frZNzmrDae+MVJYCYSqZSQYJeqTBes0BAYx+Rjfyj4X9guJQhYBLz/o9ijRS9JrNoh430rDOazHKrKd3jmi7cID+yHRROg4rX/CuGQWWbO6HFQxRbLkaK5ViRtBqm00Rp/0FJT5DS4AfL3O3NFt+6CBN2hc480hpWW3Ah5WkwD3tVjxPZ2QDX9u/RyBEXxVwZ0wpeb9H/XeohvUCeI8KrQ2BhlQuuS6ijOd1kB3FTyBOprvwXEXCgHsqnDUmOtgNnAlmlVeSSZSRP+qgL1ztsnlMgakOwxjTZ3FT/z8kum+jouo7PjsKVF1lJJws8xNfXTkL61+KN6jwvwvEjyU3ZfMui73RwFTPALWyZU5YH3ZYl6rT6Rv20JlIDBrpXPRn/lqdjvS1/cFdnPbxoAepW7d6KBF+E3AF0nzIvPgrQpMobY6rGkLZfqw9gu8h0lPierDWzaf0AeBUd/guLpUhtMPV0NyZBynG/EMMMEgsEWoFUGhxiuKizqwhyLNreKpcX8izyX1z/Xv81/OdlBq8AqT1M+ygxwQEp+cWC8ggDTW+T8hQCjY8tG/f+fF/Pm/21NEhXr72bQwonhkd+5tJNMfGKekia5M4rMDk3iqiG2U2crpJk/heltIUiJ8xbEO6s15YC6i3U+Arn4mXfQN9U6bMwOPLRgCoPuY4l1y/ghJuFBpm8V8U02WspB1LmyiUo4RHtmCzaQ6mK2vX9EwZMfNKrc+AdcGf2 yF8K9caq 83guXtb0WKWPpSL9RheQNuc+nNaZZVj7/fjJZVTogzl116KH1kAYQNnpljH/tzpHM2t3qe6vcwzXABWusxJsnRIuXZNFX1BcFmUeoghbJ9R4XqqYSeFqG2lxRtrDqso7pdBEEO/XWHziBCnfwQOhlKc+dFR2g41KrX5qH8Y9MtLuN8A3X80jVKckwTKDJEXwSBiFGRpsnFV58fUEHsUXcYr9c7OfDXh77gZgAMn64Rlm0x20B06RNSs28yvB4L/EOdvycXKLL2aCYvc6FU9+/FR/Cs06Vp7cPvMjr7k6jlH1VnDhxvqO+eRFcw0FYtU47MqsJxVnrVy7umbMVTno/x73YwuecA/5Jt8wRqNhI372ZTLM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If a folio isn't charged to the memory cgroup, holding an rcu read lock is needless. Users only want to know its charge status, so use folio_memcg_charged() here. Signed-off-by: Muchun Song --- mm/memcontrol.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 61488e45cab2..0fc76d50bc23 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -797,20 +797,17 @@ void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, void __lruvec_stat_mod_folio(struct folio *folio, enum node_stat_item idx, int val) { - struct mem_cgroup *memcg; pg_data_t *pgdat = folio_pgdat(folio); struct lruvec *lruvec; - rcu_read_lock(); - memcg = folio_memcg(folio); - /* Untracked pages have no memcg, no lruvec. Update only the node */ - if (!memcg) { - rcu_read_unlock(); + if (!folio_memcg_charged(folio)) { + /* Untracked pages have no memcg, no lruvec. Update only the node */ __mod_node_page_state(pgdat, idx, val); return; } - lruvec = mem_cgroup_lruvec(memcg, pgdat); + rcu_read_lock(); + lruvec = mem_cgroup_lruvec(folio_memcg(folio), pgdat); __mod_lruvec_state(lruvec, idx, val); rcu_read_unlock(); } From patchwork Tue Apr 15 02:45:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051356 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1627C369B4 for ; Tue, 15 Apr 2025 02:46:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 45F7628012A; Mon, 14 Apr 2025 22:46:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3E81A2800C2; Mon, 14 Apr 2025 22:46:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C7A028012A; Mon, 14 Apr 2025 22:46:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id EAB8B2800C2 for ; Mon, 14 Apr 2025 22:46:14 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7FCA780B38 for ; Tue, 15 Apr 2025 02:46:15 +0000 (UTC) X-FDA: 83334739110.22.CF6A980 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf09.hostedemail.com (Postfix) with ESMTP id 8C3A2140004 for ; Tue, 15 Apr 2025 02:46:13 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Y8NWWgGB; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf09.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685173; a=rsa-sha256; cv=none; b=4cnBMfgTKDXHsb+BbkBWlSBLIMp6CNv+M72A83IAySToUChUshnlO5HbBK+Vz8+q4PiW01 0kQvYYtoug0zqt6u9dwP5mh9fFapVtD/5dBWk4jxhZyuDW6W60y8jCDgZp6F+DiIRlWQMZ xHxglNjKfRL0ww8A+ZbKwmkuSbHNEMc= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Y8NWWgGB; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf09.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685173; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WxEqDGFZMUmXVsS40Ac89DQGDSUswXt+C8ij+v7lQJA=; b=f4wUUt63sgYX45EMN25oRuZJl0h0q+N+oZHHmTaZjlOFKqDR0HyDFyTwYDSIXqCwVmuGz8 oUM/ujjztj1+OlAsVN+bxlPvIl8Blz5d1VXytiGvM2x3St83njyfHN3hXFwrrH+y4Cx6rX xN0VT9pEUNLzRu8qK+AP57UXdeU714Q= Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-22435603572so46947495ad.1 for ; Mon, 14 Apr 2025 19:46:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685172; x=1745289972; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WxEqDGFZMUmXVsS40Ac89DQGDSUswXt+C8ij+v7lQJA=; b=Y8NWWgGB1R/NgZrF95C0ydQ75wKz7Kl6TDcrl9/NutcWiHBcOcSeamTTtg0knCRWcH gGc6zYMuRMIn1jebIqbxfSvr+jk+3FDACk82CPaYfKTjyCH3eWmnLltyAmATW6ObK6v/ WPG6LJTiceI/DR3uH6tAGSjELuNNgKAZyYVejGZuQbjxbenZIqjeCeJELVIAgHpZAZ8V A3pSp6KB0m2+G6L1/mXVR4aUpphnIjFZqusMiaI7EUr4WP5NkrUYAlrYLMGup7jmHSWm 2n/mGEL9GzbZAt/GaXjGt+wkjWcpqvHT+wO8DzOksjzLIXBgtSp/x0LFxm4lA/NhW1zz y+7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685172; x=1745289972; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WxEqDGFZMUmXVsS40Ac89DQGDSUswXt+C8ij+v7lQJA=; b=QYB52kS2hLxvzj4SiEjkcBP0GHYn3To6Zc2bOCKU3qSGPknBs+OhS4R5rQI+YAEAr7 EbAZNu2X6yQbs+RUB+rNjRiuQy9cm8VvhvO4Dtx3JTczrkEJ1fHQEp3Y6CK41F3okuB/ y3BdXKdvroNd+R6nfZuA7quxcGY2hADiI1nWtBPND7j4Y9jWNIH/yStB3D4KlB6o6oG9 njTWvXHfqugIUOS+X2v56ExBk+CRowN8M2k5egDO12dRuwrqh2EW7C3CTZpvmdM7lbWQ 8NVTqzCwSfykQEcPOxYGXe4+VKlWOgQp2Igtx+QqDA2ORZNYWs62C/Tjq9C92DkQmONl MsXw== X-Forwarded-Encrypted: i=1; AJvYcCXvNr768hkN2+vG0rueziNvzjwK84hbjJuVRq/jqo2psASD+8Po6NDI2e3Stm4OeHnYmiCGsbtzag==@kvack.org X-Gm-Message-State: AOJu0YyNJonyk5vzo10urkPwCHHlk0+yfpVrZy6tCcG3QJI59ZX0Xu3v snZ7wc6Y1ChK79EgpyP36FlGNur//+YM8p2TNwuDi5NrJtRF3lS+cFwp2u7tj3g= X-Gm-Gg: ASbGncvShwtlnC4RqHD7DlMQ8F8n1cJN5W7YYx7R77NlT6rVekxwj8k4Vp4XYeBJ8Vi LEJZ2abnDjRo8wZbVdOmLK4YTzZlPiLgRfoiWKuLA5VP+NfbZkgbQfFhFFOJd/ieU9eq4EpNRqN DbHhrwCZMsexlXqfAXF7+GQ60Piz0xeLN4I4neERUmuSmvqQaZUO05VjuQ0AD+mObpddJotQlaI ZHxgpLcuQcvgmILP/yfaGDqix43yT7A9hFhYRdq8gEPU0nI9LmQbkfWZ0HHgTDvberG0NKMBZ9+ j4m0r1HHPseT+ac+vIC97d7TWWsWfQmBN+V8pDnEIQqi1qmgWr0myhfIFf4T5dumTFZdFpxc X-Google-Smtp-Source: AGHT+IEIXidh/WytIqsVeM73H2bfcV0T/EuPMcTn2oTZb/fVqTSwur1YIJZD8NROyg3NAghPl0x8VQ== X-Received: by 2002:a17:903:228f:b0:220:bcc5:2845 with SMTP id d9443c01a7336-22bea49548fmr180452435ad.7.1744685172229; Mon, 14 Apr 2025 19:46:12 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.46.07 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:46:11 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 03/28] mm: workingset: use folio_lruvec() in workingset_refault() Date: Tue, 15 Apr 2025 10:45:07 +0800 Message-Id: <20250415024532.26632-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8C3A2140004 X-Stat-Signature: o7f6cmewu41cnoixxiedgnucbkp69mb9 X-Rspam-User: X-Rspamd-Server: rspam06 X-HE-Tag: 1744685173-337429 X-HE-Meta: U2FsdGVkX1+O1vsMmGJUWNha1yIYF0FuJrWST5jDMCoY0ZdATjw2EE3xVKuPX63BfiP44dII3xwaFLzCoGnoGViONpxxCIcUoTRtAN2ULkHc+7tCzYi7fYQIiFKtP9PAqpkn8DHirJ2kJm8sFPBOVT/4wNFsAvSoWWGFKdW0f7uneRXnlalpX9n1h/X/2t/LltyUwGzIyQBTsp/3HiwlV2255MWEI12e3ybO9G12H8znCA44/HJbwYjEkJvH7buFm5/BQoPRkViOdlqukTnDchp+rl/pHS0j9UBSFkw15am0+E3bqOVizvnbKIXVKe67xjXFIux3KKadrOIFjarNHbL2YEai2BCACXph4omBq3UYNIY2XDJlq5Hk4/BlysNlhDj8SVNOz5JGuCnJdhARO941oNiOK0ciwlAN6hmGL8Fj5U+1wpjnzRE3ojo+shv0PyGV7/Jlr74tUto8fqJy8/z0KmCUXuZagVQwbYlkMAWN8dOjHflNjmbNVHMNzputHhiE07wFCgKi5emQ5vIRMH4TueGUQ3qNO7JeZaWQuot9QN8+fcQfyFAUoP4vaejnyeEjIJbuubad/A9a1fvgpBsmOgo3yFUXvjlMjbIQVeVOfc5lnBUXk1oDm+rjucDxV8vSqtqYOzJFBtDCFWvs9jJQ4hZ7jyEo4E1d4EdaNwNMvc7NbJgeSkbSplp7pH97V7BO2q3VT6+L6hq2+38E/He9eSLaY6u1gM6ehLgLpS1kNdPe6yhraiDi7+NsDIBsKx9eY233D/d5rCzrRfCrNCpFr14eu5cnIrTIOAOKhnZ2A2DotdDfahP9QsF9qYbfjcYIiZwAbC+a5RA+k3ay4YULa/ID8U8XpY5lWA/0ObSH0tCOLRC/Fw92YcMEqoH9Ivv8A5XO/0+Xv4YWvZ1eBwxLXpXaE2mu84wo25dHkWtCc9SFkrA7zns0hFmhvjK+zAMLMVpfUB9tehTiSqp x378MCn6 vjsQEdveAym4lgefuXRCmMT1FCDmjE2uapdAiF4QueFzW6K3C+W3hNxY2E7MC+tscC4+8go6V8rsohInPIXRLAtODkjk9LP8/vQ60rGpSqf+S/RClhTdbA3uZo8gTdKE31FfPZWiSCjzP75Jpi98IBPJ5bbpkjyUdDwTeq+hBg+A3UsAqCsGyFHb8AsdOWYLsJ1l3NNsZqOdJwBnXrLJ2rl1BoWt+H6xTwOKvAJENeHOdSyfbRskT1rKyGrgF12e7Idb9uAKd3V1zwnkdnLrxNPU4RAKLm1shAklIvRIbB9Vc1QMu04Au4lkehxcfu1zIzvuo0Y12dQs0zop42RPdLhWSPR0WlAv5dh+yjwyHb9Z7uxc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000492, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use folio_lruvec() to simplify the code. Signed-off-by: Muchun Song Acked-by: Johannes Weiner --- mm/workingset.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/mm/workingset.c b/mm/workingset.c index 4841ae8af411..ebafc0eaafba 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -534,8 +534,6 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset, void workingset_refault(struct folio *folio, void *shadow) { bool file = folio_is_file_lru(folio); - struct pglist_data *pgdat; - struct mem_cgroup *memcg; struct lruvec *lruvec; bool workingset; long nr; @@ -557,10 +555,7 @@ void workingset_refault(struct folio *folio, void *shadow) * locked to guarantee folio_memcg() stability throughout. */ nr = folio_nr_pages(folio); - memcg = folio_memcg(folio); - pgdat = folio_pgdat(folio); - lruvec = mem_cgroup_lruvec(memcg, pgdat); - + lruvec = folio_lruvec(folio); mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); if (!workingset_test_recent(shadow, file, &workingset, true)) From patchwork Tue Apr 15 02:45:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051357 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68930C369B4 for ; Tue, 15 Apr 2025 02:46:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 18592280137; Mon, 14 Apr 2025 22:46:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 10CFE2800C2; Mon, 14 Apr 2025 22:46:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA180280137; Mon, 14 Apr 2025 22:46:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C76A22800C2 for ; Mon, 14 Apr 2025 22:46:19 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 6694C1CC3F4 for ; Tue, 15 Apr 2025 02:46:20 +0000 (UTC) X-FDA: 83334739320.26.32C61CE Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf23.hostedemail.com (Postfix) with ESMTP id 80537140004 for ; Tue, 15 Apr 2025 02:46:18 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=IkA0jX7U; spf=pass (imf23.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685178; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=brxtynYJLZUnqxef6aOOLSxck7i4XWxbG91ObFKf1mY=; b=seWYNCT91D0DNqJptXgTLedH3bSW0dzzSA+t4wygjhFphbkb3HEMsi0fBBtrOpWdTTnQMP m3KbEE77b5TEi4dvRV+tc474MNV+kofBHMqju8W/IyRqQvW05Y5mvGv1Jkjks06CJOM+zh AN/K2VvyXC3Pv+HlUInjMULL+BRjS7c= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=IkA0jX7U; spf=pass (imf23.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685178; a=rsa-sha256; cv=none; b=LOMbWA5z3CWg/qMQpXjU6QqMnBozOg0/SzzjPydREl0R47n+RouZt4oA1e9odqfNRRDXch /zGfuLCIonPQ/87OD2n9GXekrF4RVNiaHREVXJIFchvq4N1GT2zol1Oq0e6hhtVn2RCMc4 cYdjtE6h7YYq2jACCgPt47d/CEeyO0c= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2264aefc45dso74524015ad.0 for ; Mon, 14 Apr 2025 19:46:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685177; x=1745289977; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=brxtynYJLZUnqxef6aOOLSxck7i4XWxbG91ObFKf1mY=; b=IkA0jX7UgdL1DpukE2xiBODK/J4yGRXtc4JUqDe3KzW1kNJEx0Qvhp81wAi28QyxrT csZXZUHSd72CalV6U5efLG9w8jFv6JK9Ht7VKmmX9/Nml9sCJ3RhPBZUMHCZp88Bw0a0 NXaDGeoSukt4q9jytPPl3ldaDXGPzo5hIum9oBhUpS9o4+6n+6aF9PmYI/KskwdgKVX6 KWPakQEmpLMsjGMk1nHPKJ39Bq0smINpwGzrh7IUt6zhdaNrfRbGUi3zRxz0qNIQwhLt BcyhHpfLpCemSLFv/HjRgsMDFyxpy5QcGVe6t5PnnEDaHfBPZmnyXqg5gz4QPjS+q53D 8EhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685177; x=1745289977; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=brxtynYJLZUnqxef6aOOLSxck7i4XWxbG91ObFKf1mY=; b=HKp8XW0PqJ1odU1FjmNrXyTpFGspGb0BagdS6+DNfpD+DAJTbAH+ZfXJ7dtF0bH/80 UPy5HsHOGZFWmJ7O984/UpMVQWnYHEmLHpuecMqPKpRPpRdQYj9if9Umn0smDImnwJY9 rirfoj6vExfK9yvVYQOz4yY12yeDlHHzYo5UH6Rj18GKq9qwYcfCOyKyRJhzptG2qU5h 4+5nGz+zMbq6OYQwNDX+AE1E0uji4ks62Aq0DB4uodSv+uFUeb01nNpz2cJKj15HTDk7 1H3lo9wCxIOHbzIW7da0vOJYSCMO1TpA7wDnbDaEFIqUOKY5aFRG4rBqj1WDDMWHIg9+ wVyw== X-Forwarded-Encrypted: i=1; AJvYcCV90Sar0QbsWLxbdQ/Po4j71+tZUPf4wzl9r36PLOg7fXsrI7L/WkU2LW0Oj/NHRid46fygl8ZDEA==@kvack.org X-Gm-Message-State: AOJu0YwK5KWrIjv5JIi0MUM2nR2wt3Jnot2Oogv9tv9I9AJ8KcwFEBiK 0mbbWEt32GlNmZaudu84GCo4V+U9koQtq1a4Fgpz+wtbUP1upfh0fcfK6OBZ4MI= X-Gm-Gg: ASbGncuJvzVmEGYMMkZOvb49v0gDHlqkbTZjds/6ESwXYYOrPbgz8qUEV5lMPirP+8T z5x4Q0qkDc6Pma6nSLTo5JPSb9LRGcOcmOb4sjHlOQbz4D5hNhj31kctHxLZFD96IYQVOMWUHZg FrHC55YwAAc5mckSluZ2mz9BKe9HwmVhBUuKKfsSL2401fAOH26gKkCcAHvJzYtIaYG9GNsSSOP IO1kei0lDTmSqEsANSw+wl2ssXSITAWXj7Y7b+s7D6xytM6tBonyjbSF3yE6zlt/boVMicUUbhx qjpziMCZJiMoswsaphhKCMCVyVoRZedasXpWkCM2QSataAP6T79Z5z4x2yn1F7t/oRwjOO7m X-Google-Smtp-Source: AGHT+IHyyAeyjZ49bEZ0MTKd+BqhH6rBa5vP3SZXknSqiAeNZVb3i/S33ow8Ug4oRyTybyiTWON2CQ== X-Received: by 2002:a17:902:db0b:b0:21f:164d:93fe with SMTP id d9443c01a7336-22bea50832bmr206493445ad.53.1744685177428; Mon, 14 Apr 2025 19:46:17 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.46.12 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:46:16 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 04/28] mm: rename unlock_page_lruvec_irq and its variants Date: Tue, 15 Apr 2025 10:45:08 +0800 Message-Id: <20250415024532.26632-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 80537140004 X-Rspamd-Server: rspam05 X-Rspam-User: X-Stat-Signature: x18zi4akg6e5krxm9itdpbrxqt4mky4b X-HE-Tag: 1744685178-690638 X-HE-Meta: U2FsdGVkX19UpdmBYNjjjhkLwoxVjQjaYQ02z74qK76P4xOwpx2hFpPulZbVpj1oujKW/52wH3cTRE3ch38SfuyWoeNJLObr++qfX4aTL1iOy5q4Sh8uAlKA10ocBDBxBS8wim8TzuDddWYjJtbNUO67wa3/sxWtjaiXf8rmbo2JiAd5y4t71/sBmVkCqdTc4paG9c3BXWdbzEVFha20+SH33Yo7eQV7azLI9mfVOcctyE0pS2Psr4rqu1Blyq8oIShTcTlIEav33X780+vRISfYOoAoyZPZOefdaIleu/3/fFOOD512JPMiu8OC6q4gVt4ViAgojNy+2h37LuC7voBn3vTd6zItQKZNLsJGdIoAgMLp7x/V3OsoYVqloR0HZ1KBhdqnBMvIj1+rLfmftwSL/2sDBoH1twaUuoRjLKCtirZPKu2Z0E6PJt9vQRg1pCLWa3KSNAvaUwfp5dA+olOhIBChI75gaIkNOgdpIxT/WcG745BpVa5zAbPtp6KkxGeXreKe/UIFQ8/moWT2vnfS/k0Cy/02qR63CNDD533sFDOdmWLMptQVUQlKYZ35jVP4oBiiIG7jo8g4rUlqWihZRgnNLeT4qHIq7NQa5HY3Zy3NSsvT3UGRh8M8Qb1TnxYvd3ywKxf/pPFatjIz1RkITtzRSsXmLL5ePt3mEGgWaCv2qpHdo1nQ8STB1ru0AnJAnWNUQJ7KaCHFhlKNWFza0LYGZFx3TwNIwxCaTqDX3ZFO7f2ozg2q3cVpki3XshLceLPkq5RihOfEE2oGQ4eK8gTiX0g2/G3mi7m7oqg6cwNcC8JBtIqYOe0Jds+aUtgo03oYNodHcYvHjYvm1tBVcx6AqL8Wqhwjuf4AI993DC5fIDFED/xJykIbTnT8e1VppLsBubYYjHUzi+Wfa4yScx6EMqxeRWsd2bugLlApljgQfZ9ZvyWmiBAFWo6pZc1lMbKy/9ZsYySY0vH rn3K26hm jIXEPmcoQ8dFiFW6V9DbVRoCnfYDijBgY90KOOm07YN91Nk4CPhxJT8p75LvFukW8T84sAo5hqiFwDksfgPNp4NEyZOC1DoVZAyZemr1BoVtbX1b4s0siGDWu+ctOWBnQfLkG9P6mHOColWWzAJ2lFTgE/k6eQ5ENkz5PyxeXzbvvT1RiM5fnNuaE1+UnmUsedCD/jC1qSdVv352Ebj+tjOKUzZkp86DNzsxP/Mr+pqyojggQLah57REQh1Qg1JWivtNFEFINiLU0qlVaTWGpugY6ZbbW13o4j+fAZmW0llTMOOgGQgvGcM4iySooRkonJYGcmV8EV7RKPahqPOUpexIPHNzqKTU6gCUO/cMjbctIF8Q= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: It is inappropriate to use folio_lruvec_lock() variants in conjunction with unlock_page_lruvec() variants, as this involves the inconsistent operation of locking a folio while unlocking a page. To rectify this, the functions unlock_page_lruvec{_irq, _irqrestore} are renamed to lruvec_unlock{_irq, _irqrestore}. Signed-off-by: Muchun Song Acked-by: Roman Gushchin Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 10 +++++----- mm/compaction.c | 14 +++++++------- mm/huge_memory.c | 2 +- mm/mlock.c | 2 +- mm/swap.c | 12 ++++++------ mm/vmscan.c | 4 ++-- 6 files changed, 22 insertions(+), 22 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 53364526d877..a045819bcf40 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1510,17 +1510,17 @@ static inline struct lruvec *parent_lruvec(struct lruvec *lruvec) return mem_cgroup_lruvec(memcg, lruvec_pgdat(lruvec)); } -static inline void unlock_page_lruvec(struct lruvec *lruvec) +static inline void lruvec_unlock(struct lruvec *lruvec) { spin_unlock(&lruvec->lru_lock); } -static inline void unlock_page_lruvec_irq(struct lruvec *lruvec) +static inline void lruvec_unlock_irq(struct lruvec *lruvec) { spin_unlock_irq(&lruvec->lru_lock); } -static inline void unlock_page_lruvec_irqrestore(struct lruvec *lruvec, +static inline void lruvec_unlock_irqrestore(struct lruvec *lruvec, unsigned long flags) { spin_unlock_irqrestore(&lruvec->lru_lock, flags); @@ -1542,7 +1542,7 @@ static inline struct lruvec *folio_lruvec_relock_irq(struct folio *folio, if (folio_matches_lruvec(folio, locked_lruvec)) return locked_lruvec; - unlock_page_lruvec_irq(locked_lruvec); + lruvec_unlock_irq(locked_lruvec); } return folio_lruvec_lock_irq(folio); @@ -1556,7 +1556,7 @@ static inline void folio_lruvec_relock_irqsave(struct folio *folio, if (folio_matches_lruvec(folio, *lruvecp)) return; - unlock_page_lruvec_irqrestore(*lruvecp, *flags); + lruvec_unlock_irqrestore(*lruvecp, *flags); } *lruvecp = folio_lruvec_lock_irqsave(folio, flags); diff --git a/mm/compaction.c b/mm/compaction.c index 139f00c0308a..ce45d633ddad 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -946,7 +946,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, */ if (!(low_pfn % COMPACT_CLUSTER_MAX)) { if (locked) { - unlock_page_lruvec_irqrestore(locked, flags); + lruvec_unlock_irqrestore(locked, flags); locked = NULL; } @@ -997,7 +997,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, } /* for alloc_contig case */ if (locked) { - unlock_page_lruvec_irqrestore(locked, flags); + lruvec_unlock_irqrestore(locked, flags); locked = NULL; } @@ -1089,7 +1089,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, if (unlikely(__PageMovable(page)) && !PageIsolated(page)) { if (locked) { - unlock_page_lruvec_irqrestore(locked, flags); + lruvec_unlock_irqrestore(locked, flags); locked = NULL; } @@ -1194,7 +1194,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, /* If we already hold the lock, we can skip some rechecking */ if (lruvec != locked) { if (locked) - unlock_page_lruvec_irqrestore(locked, flags); + lruvec_unlock_irqrestore(locked, flags); compact_lock_irqsave(&lruvec->lru_lock, &flags, cc); locked = lruvec; @@ -1262,7 +1262,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, isolate_fail_put: /* Avoid potential deadlock in freeing page under lru_lock */ if (locked) { - unlock_page_lruvec_irqrestore(locked, flags); + lruvec_unlock_irqrestore(locked, flags); locked = NULL; } folio_put(folio); @@ -1278,7 +1278,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, */ if (nr_isolated) { if (locked) { - unlock_page_lruvec_irqrestore(locked, flags); + lruvec_unlock_irqrestore(locked, flags); locked = NULL; } putback_movable_pages(&cc->migratepages); @@ -1310,7 +1310,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, isolate_abort: if (locked) - unlock_page_lruvec_irqrestore(locked, flags); + lruvec_unlock_irqrestore(locked, flags); if (folio) { folio_set_lru(folio); folio_put(folio); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 2a47682d1ab7..df66aa4bc4c2 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3605,7 +3605,7 @@ static int __split_unmapped_folio(struct folio *folio, int new_order, folio_ref_unfreeze(origin_folio, 1 + ((mapping || swap_cache) ? folio_nr_pages(origin_folio) : 0)); - unlock_page_lruvec(lruvec); + lruvec_unlock(lruvec); if (swap_cache) xa_unlock(&swap_cache->i_pages); diff --git a/mm/mlock.c b/mm/mlock.c index 3cb72b579ffd..86cad963edb7 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -205,7 +205,7 @@ static void mlock_folio_batch(struct folio_batch *fbatch) } if (lruvec) - unlock_page_lruvec_irq(lruvec); + lruvec_unlock_irq(lruvec); folios_put(fbatch); } diff --git a/mm/swap.c b/mm/swap.c index 77b2d5997873..ee19e171857d 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -91,7 +91,7 @@ static void page_cache_release(struct folio *folio) __page_cache_release(folio, &lruvec, &flags); if (lruvec) - unlock_page_lruvec_irqrestore(lruvec, flags); + lruvec_unlock_irqrestore(lruvec, flags); } void __folio_put(struct folio *folio) @@ -171,7 +171,7 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) } if (lruvec) - unlock_page_lruvec_irqrestore(lruvec, flags); + lruvec_unlock_irqrestore(lruvec, flags); folios_put(fbatch); } @@ -343,7 +343,7 @@ void folio_activate(struct folio *folio) lruvec = folio_lruvec_lock_irq(folio); lru_activate(lruvec, folio); - unlock_page_lruvec_irq(lruvec); + lruvec_unlock_irq(lruvec); folio_set_lru(folio); } #endif @@ -953,7 +953,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs) if (folio_is_zone_device(folio)) { if (lruvec) { - unlock_page_lruvec_irqrestore(lruvec, flags); + lruvec_unlock_irqrestore(lruvec, flags); lruvec = NULL; } if (folio_ref_sub_and_test(folio, nr_refs)) @@ -967,7 +967,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs) /* hugetlb has its own memcg */ if (folio_test_hugetlb(folio)) { if (lruvec) { - unlock_page_lruvec_irqrestore(lruvec, flags); + lruvec_unlock_irqrestore(lruvec, flags); lruvec = NULL; } free_huge_folio(folio); @@ -981,7 +981,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs) j++; } if (lruvec) - unlock_page_lruvec_irqrestore(lruvec, flags); + lruvec_unlock_irqrestore(lruvec, flags); if (!j) { folio_batch_reinit(folios); return; diff --git a/mm/vmscan.c b/mm/vmscan.c index b620d74b0f66..a76b3cee043d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1847,7 +1847,7 @@ bool folio_isolate_lru(struct folio *folio) folio_get(folio); lruvec = folio_lruvec_lock_irq(folio); lruvec_del_folio(lruvec, folio); - unlock_page_lruvec_irq(lruvec); + lruvec_unlock_irq(lruvec); ret = true; } @@ -7681,7 +7681,7 @@ void check_move_unevictable_folios(struct folio_batch *fbatch) if (lruvec) { __count_vm_events(UNEVICTABLE_PGRESCUED, pgrescued); __count_vm_events(UNEVICTABLE_PGSCANNED, pgscanned); - unlock_page_lruvec_irq(lruvec); + lruvec_unlock_irq(lruvec); } else if (pgscanned) { count_vm_events(UNEVICTABLE_PGSCANNED, pgscanned); } From patchwork Tue Apr 15 02:45:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051358 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9FB32C369B4 for ; Tue, 15 Apr 2025 02:46:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 403D728014C; Mon, 14 Apr 2025 22:46:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 38B5C2800C2; Mon, 14 Apr 2025 22:46:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2075628014C; Mon, 14 Apr 2025 22:46:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id F025F2800C2 for ; Mon, 14 Apr 2025 22:46:24 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 994885A4AA for ; Tue, 15 Apr 2025 02:46:25 +0000 (UTC) X-FDA: 83334739530.20.6599C84 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf19.hostedemail.com (Postfix) with ESMTP id AD02D1A000B for ; Tue, 15 Apr 2025 02:46:23 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="HUeAd/sI"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf19.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685183; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HknBZVJEEwldsJF0essNl26exDom/3YV3flikzCNEqI=; b=A6xKTj/TTWMpU4DnoRGbC4tkwTG1ARsXX4cY0Iuyg+1b734Xq7wdctK1Ww57thhGDvzhka rb+gLHYUpoMOLsBaGWpolgaBvmzXCP9i9YY9YtxDsna9SkI652GiGD8mkmDIX165xf4gfH itR59Algt+kMAfjhxkzU/Czwtr7Kk6g= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="HUeAd/sI"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf19.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685183; a=rsa-sha256; cv=none; b=114Hwf/tDpjb4kDJyvHf46DdOVgULPcGShnw5M3mOLEuhcpJ+92PQdWbVxNLVQYxIKD2MA rKfy/fy4dwrStG/CQTl5JP2vsFgPY77RdLM2kYx3SMI5wlaAFaozMe51fqmaQ7fYT9HU/f Nbsrb7NGA8hvUzz3lOx3Dd9oTUjPFaQ= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-224171d6826so70694915ad.3 for ; Mon, 14 Apr 2025 19:46:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685182; x=1745289982; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HknBZVJEEwldsJF0essNl26exDom/3YV3flikzCNEqI=; b=HUeAd/sI0j8vqOOTYGjI1P//bNlKJ2xyDQ7lcFv16hfKUHu+gWu8MU79/5JPatmLT1 5jDnJ1vRVThv2ID8R06WbfjArTH+y+8H2mjW9HuzM7FcJbreVvCCUJeV6tK23zQMSpns MagoyxQC3pK/8H6MeDX7KKHRk9mdUWvZBkpfO/TWwh9yxmVIOfd8usMYRl0svJXNNUee R98lYjz8XP+JBl35sJlCNoRe0VDY89XY4an/52lG0sVqr/DqL+yNdW5Bzo5chEEK4IHL v+1RvaLeR4LNi0NTYfxHrItrAQOPxTNv+JX+B8QB4TzMN5REZPBRrcb8az4DiD8svEBx Z4KA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685182; x=1745289982; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HknBZVJEEwldsJF0essNl26exDom/3YV3flikzCNEqI=; b=cSQvIFsAe+IQDC7BPq9P1pjEIzekpD9OfHaX4c681gz8ZIcl6Bvab2G+mLTaMPe3D8 kdMfyYizNhGHxP7kmyfVhZxJLc1vIRivxX3kevZSFIcWxfk+jAfc/yB29ibWboLpei9K DORvTgz7rvH01lydODIq7uIfJSups7Y/Zw/R25VpiGtMy+JHIVshN1KLX89iR8OsRDEL g17Dby6lkHSRfED0WxTKldmFRkEomRiwfvJQ41gLllB32oU3DgHPdbMCWuijJF/7tTlF uenkwfG8eIfIosktHK/dDFXPtULM6hMvAPJPLRL9hjKPuxbeml95iNBD3j/jGkvmo0wh Iw8w== X-Forwarded-Encrypted: i=1; AJvYcCXCdMmKkjbYXxyRWQyT1LGAsnKqewKyd5z/7UeT3MgZt9TlYl8pKwqNWbAQ/QIHfCHsIkvgxjEQSQ==@kvack.org X-Gm-Message-State: AOJu0Yzkn8XKTVfSxYsSa7zLY9sJXsMMJHQiHXsFayW5Ni8+1C/f9qDN YtY3eZ2sAdlAO52XP8ctxHNBtnInKm/vHHvOLyDqqRENQiXehoOSzYPqtX0R4ok= X-Gm-Gg: ASbGncvNV1Jjm5t3Lve+7iaA5FXHXmmlX3e9xgqDyl5jDFdV3FbQ7OTOlaEJ2fjz1Qc uWWXBgbs3lSfNe74OQNrVWSSf8XFCwIcJI6HtEbyy0EF1p9S4Zkz55Zj5aWyj4NPEtXqKfDXaTn pGzr51dJHj6HHI/vNGrRn3GhBw5nHGq5l5WtUr0LGGl0ClEFjmVbebcvRpKj5tYcWCSUMYVtJcd UiE6LmkvBtxz7KJs0aOqC3Nx07qurkLZC94u/Gl1Psxfr+Tk5cpOrSOviz+i7StZyJ1paE1Vc9e BHVkjMAXOZ1tNqAc1tfNgtN6xqicb98V+Alg0TK0GERCa7p3uk4SUqHoMMz7m23mwPunYRY8 X-Google-Smtp-Source: AGHT+IF9USZi2XhF57yhEEp9osrqi/g5DdA8y8Taoi4iL31Fzh+/3YgFBVfEBC6Bksa7uUzxyxne4g== X-Received: by 2002:a17:903:3c44:b0:223:f408:c3e2 with SMTP id d9443c01a7336-22bea4b6136mr205639305ad.14.1744685182592; Mon, 14 Apr 2025 19:46:22 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.46.17 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:46:22 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 05/28] mm: thp: replace folio_memcg() with folio_memcg_charged() Date: Tue, 15 Apr 2025 10:45:09 +0800 Message-Id: <20250415024532.26632-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Stat-Signature: nmpmw811q7s3aarj756n9is8y4kuppqm X-Rspam-User: X-Rspamd-Queue-Id: AD02D1A000B X-HE-Tag: 1744685183-962604 X-HE-Meta: U2FsdGVkX19KEAFzzV9eMpgjjvk1wAv2LsMXrgLwhty3VuJYpNhnZXAEw0Qli2iMT6zQYubSycYjd80TGE3Ofn9JzQUzbR5fvQknxdcD8k3ESZWtlAM1zfH0ur65Lwjr/b9KL4t54hmN/PWlEm1fdI34C+1MkZGKxtYFgdNMIzD5zGXVGAivXmofXmzPqq5XMy36ctBgI0g2MVV/TbKx6chwtPpQnieciBvkEH821+CgUiIQPiStIslNiDsTvq1usBAhLXc2bofdW3AJSk8HMyCZsAdMsuGxvYP91+xlFGxq9L19Ordl3m/bYKPrMTxJ5YgxhCaD1zEaBFlILpmiMPvgQ48tltoxs54m3e5/F7CABlrSQHyLje/oqRjH6UniZXTHQn7SPjJcAhg3lvWKxAnYbwYuGooL8TvnV5DY5xcuQLNEx3S6QOTTWfTbTwsrM/n6BMOV7U5vUQjSPUdmwMHp/FB8fR7ID0Pe2CQTycEppeS60CoENOP8Q6Hn5dvUnfvF4YrTvANgjFKqSNQ5NpNPB91BUHFmFUfuSIGYhQMArgxJhJv6cqYkN33tggqbVwW1c1mC/Zps8XG4WloioKNBuNefp+eXeKic6D9kehjxI07GOD/tElU6mhx3yiy2oBVKSUOEJWEW6O3GLmVs0yWYmxraMImmMBSybo/KHB6J4sj8QG5RtdjAxnUX0HIlc3VXEW0P/IHlTgZXL9UZddihIc7Xjz/LXpE1XvZANOgznK9nb8V/LqiWlnnUjWwtDKqYUXRbwAdeML24nYXpMtWSp5e+osiSmsg5pdwY0StUrGG3m9WrHXJr7zQ8PDQppV9RySCiCkxZ7A1nGqJh1LKORp/BrJH+Bmm7Ki8p/iSewKkxbJqfdawaczw3tP2SmjIKvO2BlHWqU/6gxxFi2X9vKNc6eoNE36WT50O9X+8qZrCW2qCMwHW3VMarYgTTqC9VCqgxH62n3ewyzh5 nlS275Y4 mVCIjN8Rom9rq8xtr/zDl4XR/m/jQcPH+s4Xt93NNSN58dH0k8QtO8hIhZBHt0lKeshzhwaI754oCxYIndc+7u7VLx6U0Uv6lsInmxSEtfwD8RNoaelDj+0b4OM/nZhaewcCum5qBD/LeaF93Ejos8ZJ6DuQ0GA0j6hM38G92PjLcZUTnRuWuZDA6S+iSmcEPENDEGOjFo3aIUs1/ilGIUTgmYf2AquSnqEaZUNSrwd2PKfNdKm+FnJ5vPyb0HT8f5hr+t73tKR/yTd+HARVlSw3jb3VUoVDQgcva2CnrEbbBVzld3JnSoNFbcDuDZxj9W4UF9v1IyrjxOLTXcuHhrI/8MxTAQAKanUVCq5hs1DSgdIs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: folio_memcg_charged() is intended for use when the user is unconcerned about the returned memcg pointer. It is more efficient than folio_memcg(). Therefore, replace folio_memcg() with folio_memcg_charged(). Signed-off-by: Muchun Song Acked-by: Johannes Weiner --- mm/huge_memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index df66aa4bc4c2..a81e89987ca2 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -4048,7 +4048,7 @@ bool __folio_unqueue_deferred_split(struct folio *folio) bool unqueued = false; WARN_ON_ONCE(folio_ref_count(folio)); - WARN_ON_ONCE(!mem_cgroup_disabled() && !folio_memcg(folio)); + WARN_ON_ONCE(!mem_cgroup_disabled() && !folio_memcg_charged(folio)); ds_queue = get_deferred_split_queue(folio); spin_lock_irqsave(&ds_queue->split_queue_lock, flags); From patchwork Tue Apr 15 02:45:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051359 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D05E6C369B4 for ; Tue, 15 Apr 2025 02:46:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 769CC280190; Mon, 14 Apr 2025 22:46:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 71B8F2800C2; Mon, 14 Apr 2025 22:46:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5BCFD280190; Mon, 14 Apr 2025 22:46:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 3F00B2800C2 for ; Mon, 14 Apr 2025 22:46:30 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id CF90EC0B76 for ; Tue, 15 Apr 2025 02:46:30 +0000 (UTC) X-FDA: 83334739740.04.ED2D93C Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf22.hostedemail.com (Postfix) with ESMTP id 00715C0003 for ; Tue, 15 Apr 2025 02:46:28 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=JPGM+rRr; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf22.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685189; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SVQwnJXLTzf5IbKZLnXsEzmufmqfYmZThSJSPJR5bRg=; b=DxGSvHAzNgupC2ouxek8re0dp4Q7O5pMf3V8ajv1ra085cyhiF8lloPQcfODNwgBnb/V/g XmMf5W/W2B6KGOXOqomnN3i05Ncv79dkdv3xiENEkwLSJj0C6bPcFofyUYuTBFgqRc/xz5 e/Zkm45NBrj8DTrxVyKubGwzvTCnIDg= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=JPGM+rRr; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf22.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685189; a=rsa-sha256; cv=none; b=vM/iOK3f8NrPGvHdGCipMx9yrEh4cq0QlmTWYdSyedhenQ1v+B9UYi79sJl1lQni8XQr79 RPNY87w4F2pHV+AJNQyeCb6760UXrjhFFFZp+cvwZDblxJNS8iDX0mtdlAKl45t2KvDGTA zTLsvU9GBGNOZP3JZ6db9wdwVH4RQFo= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-22435603572so46948265ad.1 for ; Mon, 14 Apr 2025 19:46:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685188; x=1745289988; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SVQwnJXLTzf5IbKZLnXsEzmufmqfYmZThSJSPJR5bRg=; b=JPGM+rRrfGLGnHCx+9yN4NgK/c0Ar7qWq+JCRbs3Tz1bU0R0ZuGAkw31SVklijxfAI IlB5DKuiremml3uDNt8lb2t+sU6ga/nmeFP+XmN8KJUnPeCgCtyboJ//tjAs4YF5qpw7 fVqSk+i8SELen5Z8+/w6eNd5j78Ad8W8TU4H9BJa6cdhpKwxWEAQAJOOsNUDWZ1pvHq1 2xda68xr4xgaFsYcsOqcfTdedHe491YB1QGwIUZXB91awJnhtpYSLmAY0k/S241iZxt8 l/0omgzsYiEKCmm0xkr4JAJZ+yIAorKi3lTf12NTaVhTDQBuQePUmKSh8ucdSqeinHHK qjLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685188; x=1745289988; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SVQwnJXLTzf5IbKZLnXsEzmufmqfYmZThSJSPJR5bRg=; b=W/YulNNxvyn72bR8Rb4kl6SGx8Tr8P1ZrHPakytMeXjvWTNSYJ8J47CX1L38yuDIU7 KW9XodIMyCq4JYNugKozkSxvTptp8Ha3dR7Nxnj/8Ws51BisDpiLw8/V+CzXPtQr2ml3 8UWvVHmlxNvW2XloxyMwmkcvlwnzRLaCgMQE9qBW0NLtQaVRaKz8V5KFwGfh3jKcMirj kNVOwDwsXdJj0pSKb343TPJ9cLa5h+h2OwPPGDXteD8xe+/EsVGt7X3eaqy4IldeEECk kuYaeLFnFGbmh56gAM112heZa9eN3ayfAZPm4MgG50iLLMQgnh9T7ry7eApCdSFSs15F WXcw== X-Forwarded-Encrypted: i=1; AJvYcCXpLtyFhzI5vzb3H1Q7TIstB/KOn6ebLJbmr4TLkJjKoFgMw4J6tWTY66VFsgAvzgDhjl/mEbuNzg==@kvack.org X-Gm-Message-State: AOJu0YwytmuiPis1TmMUiT6KLV2MTpST4euNX3WafDLJ/hP7hQcrYZIr D/20iUvW6TUSJIEXT0C3u2fx5eX1cFhOKhLa3r6YiJsCC+Im9Hf83uyCuvJRJG0= X-Gm-Gg: ASbGnctVBA+bgcpBpTBERw63PTr/tugdOVboLn03ymA5kH0j2F3n8pzPcWY84CL+5d7 xP18GKryN88F7B/kZOJwlHKwjlkIsmwbm9zwS9rJS7e+nmFCAm4o72J8IUh36EZZU/o2PtQ8brQ NVtWiQhpsZTlsXdAmxG7izBWqPfxE+MMi1jXTb3nw8MvJLmdP9ji7ojDKSJjY7VwfJ5DlU1wrul gjnzBpYQbd7f1FF8V9jo3SID+RD5Vnv/OS7Kpb+wEvjLiTQNj58LFyV4qbQr8eAle2vpBmkMGom PDmXDiK5qd+896CVi7YEfxtiOLn9qC1YxHoh4je3fSnJXJKoq1itSR1f5j9NXXPqVCzUybZT X-Google-Smtp-Source: AGHT+IH0uXKr0KJ6xOw01dclaGRwapxC+cZ57mqT8VHJVXzMXdBqWXQQ8E+M2n4/a2Vb6fIXE8woqw== X-Received: by 2002:a17:903:98b:b0:224:1781:a950 with SMTP id d9443c01a7336-22bea4aba54mr197670155ad.14.1744685187749; Mon, 14 Apr 2025 19:46:27 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.46.22 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:46:27 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 06/28] mm: thp: introduce folio_split_queue_lock and its variants Date: Tue, 15 Apr 2025 10:45:10 +0800 Message-Id: <20250415024532.26632-7-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 00715C0003 X-Stat-Signature: 6u9kinxmr5kdddsp7juqwtnc1sxcbq1d X-Rspam-User: X-HE-Tag: 1744685188-141582 X-HE-Meta: U2FsdGVkX1+21nYTqFA/8jM/IMzcTamTDrC0t1LWujrm6LdT2dYfGC5Xsbb5ebZ+NWG2IQEvGxGRjMETGyUcRpgN5Xnu/3TTwziHlOn4QBsXv3qcVhEd2wo3uwPmdOjTpEVMNRRtqzrA9DmKswEeCkmfybIwNJAa++geiVQhG81nI/AEvNwRCH9a3ZFZaDsJ0QMlzhb2XCtJppy4tQH4dR3MOp/oclXlHaNb8ZrBjhULwMpm2dr6Yfgpxm0ARHSMRjZq/m/y3j7GrnVgiUeOcn+0JGtrClFsuzpBogLVc9d5HEdW+Mp8npLXAOwoMykPPUQb4i2mWa4rJTUsM7IYKk8ylM1iZJdVotm0FpNEG/5ccg7x7Vpe39Ivr92dpl4UbWEB0n4J8S2OILBIzWL/IhBCJMwsmzP9Kz9Vy83oX3kBBVeoe4vkBMnDKDRvZpD3uKgQgA6+5c15RrtuFyIfs5xV5qeml6p8OaRKfvMrBUdy/K4ekM+vxg965LcS0nUSJrYaZctXTvww9dpMbhidW15tnfjK9nyVe6JBe21qbMtyPdX/Z1KIEXlXGffPUUIp1hi1Kb7Gg+IJwpuEtvcJDuqIR4cX2WI9FkkrCQiR3wI65ODXKYl2HyrBrWk/PuDv9OQZNl5yy1PV2dAlTn36MXq2c/GY1mRPdNjgqx9O3uIAplDPb/iDNgONtoaaoTsqiJtSG0VZr6ypn0wFcGsWDclTs5wGHECbvG+4ONUgR8jl6lllY2qLZXYOTrssaL+4wTgT2qDfi7ON4GXVuMO8xKp0Vbgut1PKeElJjllA4aPtHaY7paiiR0qIUR+WpQYNJp/2wFfmGrzSswKRqyVnhagslkSt2cSeRQyLJXiS91NUdfNUwIS0EnnofwRjWEjzoor+iJ1C6ukQLcKL1MdhXO70zgtrexSJ6mDz63inIF0QFHPlEtQ5gvufE19F/r/9e+EcJJGRRfncZmc4kp9 1ZO3FwRw 4MmY7iSuZAsKjxdGuJ+zPBzhcOY5RAn3mdyAx6WCvIl5NgnqaC/M3T7vuaMw1q9qnLhzHC4CfkiztFBG8zPvEQBbnqCLVQHPvJov92EEpQtjvN3DO9+zVxdW9EcCRKCHgXg07grhx+AMxxuv5EBTn15N++Zef/ZiEFG5+Qp3uN8h2hNKWrqEnCZHTtAU7ZY7ofshd5fFYPJop21Zn8FP0nwkyJgQsrqd9bNcDvkEso43n7yK5WS9uJo3xeJpGbnr9wFjpMl9IOoVSRPLl0DzfcYpWNU/2u2eHk+OpovpI0HU4SJt0OU8zn7Z8n6mwrxzPXV4ln6TPFgRFmN7ZsdJMql7JsxbAlJqAtlo3v0Y0sGmU8rM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In future memcg removal, the binding between a folio and a memcg may change, making the split lock within the memcg unstable when held. A new approach is required to reparent the split queue to its parent. This patch starts introducing a unified way to acquire the split lock for future work. It's a code-only refactoring with no functional changes. Signed-off-by: Muchun Song Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 10 ++++ mm/huge_memory.c | 100 +++++++++++++++++++++++++++---------- 2 files changed, 83 insertions(+), 27 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index a045819bcf40..bb4f203733f3 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1639,6 +1639,11 @@ int alloc_shrinker_info(struct mem_cgroup *memcg); void free_shrinker_info(struct mem_cgroup *memcg); void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); void reparent_shrinker_deferred(struct mem_cgroup *memcg); + +static inline int shrinker_id(struct shrinker *shrinker) +{ + return shrinker->id; +} #else #define mem_cgroup_sockets_enabled 0 static inline void mem_cgroup_sk_alloc(struct sock *sk) { }; @@ -1652,6 +1657,11 @@ static inline void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) { } + +static inline int shrinker_id(struct shrinker *shrinker) +{ + return -1; +} #endif #ifdef CONFIG_MEMCG diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a81e89987ca2..70820fa75c1f 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1059,26 +1059,75 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) #ifdef CONFIG_MEMCG static inline -struct deferred_split *get_deferred_split_queue(struct folio *folio) +struct mem_cgroup *folio_split_queue_memcg(struct folio *folio, + struct deferred_split *queue) +{ + if (mem_cgroup_disabled()) + return NULL; + if (&NODE_DATA(folio_nid(folio))->deferred_split_queue == queue) + return NULL; + return container_of(queue, struct mem_cgroup, deferred_split_queue); +} + +static inline struct deferred_split *folio_memcg_split_queue(struct folio *folio) { struct mem_cgroup *memcg = folio_memcg(folio); - struct pglist_data *pgdat = NODE_DATA(folio_nid(folio)); - if (memcg) - return &memcg->deferred_split_queue; - else - return &pgdat->deferred_split_queue; + return memcg ? &memcg->deferred_split_queue : NULL; } #else static inline -struct deferred_split *get_deferred_split_queue(struct folio *folio) +struct mem_cgroup *folio_split_queue_memcg(struct folio *folio, + struct deferred_split *queue) { - struct pglist_data *pgdat = NODE_DATA(folio_nid(folio)); + return NULL; +} - return &pgdat->deferred_split_queue; +static inline struct deferred_split *folio_memcg_split_queue(struct folio *folio) +{ + return NULL; } #endif +static struct deferred_split *folio_split_queue(struct folio *folio) +{ + struct deferred_split *queue = folio_memcg_split_queue(folio); + + return queue ? : &NODE_DATA(folio_nid(folio))->deferred_split_queue; +} + +static struct deferred_split *folio_split_queue_lock(struct folio *folio) +{ + struct deferred_split *queue; + + queue = folio_split_queue(folio); + spin_lock(&queue->split_queue_lock); + + return queue; +} + +static struct deferred_split * +folio_split_queue_lock_irqsave(struct folio *folio, unsigned long *flags) +{ + struct deferred_split *queue; + + queue = folio_split_queue(folio); + spin_lock_irqsave(&queue->split_queue_lock, *flags); + + return queue; +} + +static inline void split_queue_unlock(struct deferred_split *queue) +{ + spin_unlock(&queue->split_queue_lock); +} + +static inline void split_queue_unlock_irqrestore(struct deferred_split *queue, + unsigned long flags) +{ + spin_unlock_irqrestore(&queue->split_queue_lock, flags); +} + static inline bool is_transparent_hugepage(const struct folio *folio) { if (!folio_test_large(folio)) @@ -3723,7 +3772,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order, struct page *split_at, struct page *lock_at, struct list_head *list, bool uniform_split) { - struct deferred_split *ds_queue = get_deferred_split_queue(folio); + struct deferred_split *ds_queue; XA_STATE(xas, &folio->mapping->i_pages, folio->index); bool is_anon = folio_test_anon(folio); struct address_space *mapping = NULL; @@ -3857,7 +3906,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order, } /* Prevent deferred_split_scan() touching ->_refcount */ - spin_lock(&ds_queue->split_queue_lock); + ds_queue = folio_split_queue_lock(folio); if (folio_ref_freeze(folio, 1 + extra_pins)) { if (folio_order(folio) > 1 && !list_empty(&folio->_deferred_list)) { @@ -3875,7 +3924,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order, */ list_del_init(&folio->_deferred_list); } - spin_unlock(&ds_queue->split_queue_lock); + split_queue_unlock(ds_queue); if (mapping) { int nr = folio_nr_pages(folio); @@ -3896,7 +3945,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order, split_at, lock_at, list, end, &xas, mapping, uniform_split); } else { - spin_unlock(&ds_queue->split_queue_lock); + split_queue_unlock(ds_queue); fail: if (mapping) xas_unlock(&xas); @@ -4050,8 +4099,7 @@ bool __folio_unqueue_deferred_split(struct folio *folio) WARN_ON_ONCE(folio_ref_count(folio)); WARN_ON_ONCE(!mem_cgroup_disabled() && !folio_memcg_charged(folio)); - ds_queue = get_deferred_split_queue(folio); - spin_lock_irqsave(&ds_queue->split_queue_lock, flags); + ds_queue = folio_split_queue_lock_irqsave(folio, &flags); if (!list_empty(&folio->_deferred_list)) { ds_queue->split_queue_len--; if (folio_test_partially_mapped(folio)) { @@ -4062,7 +4110,7 @@ bool __folio_unqueue_deferred_split(struct folio *folio) list_del_init(&folio->_deferred_list); unqueued = true; } - spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); + split_queue_unlock_irqrestore(ds_queue, flags); return unqueued; /* useful for debug warnings */ } @@ -4070,10 +4118,7 @@ bool __folio_unqueue_deferred_split(struct folio *folio) /* partially_mapped=false won't clear PG_partially_mapped folio flag */ void deferred_split_folio(struct folio *folio, bool partially_mapped) { - struct deferred_split *ds_queue = get_deferred_split_queue(folio); -#ifdef CONFIG_MEMCG - struct mem_cgroup *memcg = folio_memcg(folio); -#endif + struct deferred_split *ds_queue; unsigned long flags; /* @@ -4096,7 +4141,7 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped) if (folio_test_swapcache(folio)) return; - spin_lock_irqsave(&ds_queue->split_queue_lock, flags); + ds_queue = folio_split_queue_lock_irqsave(folio, &flags); if (partially_mapped) { if (!folio_test_partially_mapped(folio)) { folio_set_partially_mapped(folio); @@ -4111,15 +4156,16 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped) VM_WARN_ON_FOLIO(folio_test_partially_mapped(folio), folio); } if (list_empty(&folio->_deferred_list)) { + struct mem_cgroup *memcg; + + memcg = folio_split_queue_memcg(folio, ds_queue); list_add_tail(&folio->_deferred_list, &ds_queue->split_queue); ds_queue->split_queue_len++; -#ifdef CONFIG_MEMCG if (memcg) set_shrinker_bit(memcg, folio_nid(folio), - deferred_split_shrinker->id); -#endif + shrinker_id(deferred_split_shrinker)); } - spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); + split_queue_unlock_irqrestore(ds_queue, flags); } static unsigned long deferred_split_count(struct shrinker *shrink, @@ -4202,7 +4248,7 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, if (!--sc->nr_to_scan) break; } - spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); + split_queue_unlock_irqrestore(ds_queue, flags); list_for_each_entry_safe(folio, next, &list, _deferred_list) { bool did_split = false; @@ -4251,7 +4297,7 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, spin_lock_irqsave(&ds_queue->split_queue_lock, flags); list_splice_tail(&list, &ds_queue->split_queue); ds_queue->split_queue_len -= removed; - spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); + split_queue_unlock_irqrestore(ds_queue, flags); if (prev) folio_put(prev); From patchwork Tue Apr 15 02:45:11 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051360 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29ED4C369B4 for ; Tue, 15 Apr 2025 02:46:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C347328019A; Mon, 14 Apr 2025 22:46:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BE44D2800C2; Mon, 14 Apr 2025 22:46:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A84CB28019A; Mon, 14 Apr 2025 22:46:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 89B902800C2 for ; Mon, 14 Apr 2025 22:46:35 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 018E11A0B5B for ; Tue, 15 Apr 2025 02:46:35 +0000 (UTC) X-FDA: 83334739992.04.880A603 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf08.hostedemail.com (Postfix) with ESMTP id 1C9C7160002 for ; Tue, 15 Apr 2025 02:46:33 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=EK0tmV2E; spf=pass (imf08.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685194; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=y8vA4hA6IVI4psjQZDVnpF3qlQM7CohLDDJ5h+RUQ50=; b=5VgF0I2NRf64JDsRFvX3upSszg2j8ebYm3O8sR1zllyB/znD7KFNhFV6pvU+dF56nGyEdn w3eJI7sX0qLyfRJ8YJ7ojeqWzYqNiOQt4o16XXKI/i5dBt5n3I8jUd+2HEqLeS06AC1hHF SLhhVw+LZ208/AFhGtsBMRDJngqRiBY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685194; a=rsa-sha256; cv=none; b=q0ACGNbnD0YWE5glhlIhJ56lzqge8jsDk16EbMUzsEmBt0ZOGVW3BJFG0g6hX4PnSqpDhV MhUIxx4HtkVmvq09Ml7jTh512dQN6UKYd9j9aNUuWE0I9+okPhWufpbfZmTa9oMyO5dwwr zRCtP3ftoMSMw+zqHi7+qAkpc8s4XCA= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=EK0tmV2E; spf=pass (imf08.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-227b650504fso48139495ad.0 for ; Mon, 14 Apr 2025 19:46:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685193; x=1745289993; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=y8vA4hA6IVI4psjQZDVnpF3qlQM7CohLDDJ5h+RUQ50=; b=EK0tmV2EJ+kGNp5AZbhg89fTc+GgwNcZ8dO7HmzR2PqLqsAsZfHJyxr8GzMqCf5w1r yeCgThYth8qi0k7ZWrAYsxujxubvCYSaY+FllWopUC4sonxlq+e5XHHhP2QbtMqjdiCE x44nVPGc10bnfezWfMVt3f0SEqTvPYQsOFDCg2wrbmqhxd/LRMbVFVtN/T9yLULcq/oR xJ+exp0QKQonFDGwfN1kH6fkSyx3zilNWPgO/g+trc7cYFxbbs6FpE7yIpdH8MQ1Tt34 /n2TK2bKCyFbDuLcw27iyAYWP+FtMDTfh8383ai2BLYmR1JNwHeuKePmdcaaLyY7x9CR Vr3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685193; x=1745289993; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=y8vA4hA6IVI4psjQZDVnpF3qlQM7CohLDDJ5h+RUQ50=; b=rnHVBkA1H6Rfr7TozOkPy5GQuL+9Xx4GmA8EpWVpixa24VlaFHbdw1tj18myn5E9GH luPzhXixBjAe3vlM19pP5z5rv3i2PU34EROU4nmzyLxrS2ZwtvumLPaALGMlGWpjbI+b b5joUiNooJPUoSao7W5zNP8wtGnk/gS5J10Hh0NHTrWc337SXgaKeCFMWO5pIboORTvg TkZqgokaM0uLqUcaYj97nwJNFPdKgzChxNokRe8WU65Bx4+KOE/0Qq+f7pUY9IvoNf5n 0oazYJVau1Wai3xx7Ak1uESXLJfpTDW8p+csqf5tAIR92cbo9YdRVEmnzYZzT2fvOsqX r3Xw== X-Forwarded-Encrypted: i=1; AJvYcCXbDBKQ6D7oXhXqLeC6o5BAbZT4CkGikf7aPivuTbLbYR/+lCkTNwvy8XqxZC5tcUoQd27kOdnGGA==@kvack.org X-Gm-Message-State: AOJu0YzyY6Ep2rS3cwTQawyar28GjwL/mjiwAEocVCG4UtJ+u3fmPA0w s8eRbU2Jgr3lJCXa667z24Q8ijrwCD3NkKobR4LFEzravhdrCOc4JBlVZVZn6Hc= X-Gm-Gg: ASbGnctQarPjhpeZxf9RD1vlp5MxEJQf9IPDxBjDuLV4AURemv2YJjbF0GVPL54h5tK M5Q+AoDc0ya7YBYMbEd/XlyGL3pczih6Bso+87ifCzRBPF9RmsF0ILkBM066r9mPqGDpWBHIleS dLnfKTXRXyzeSAuDPHN8WZByO5qz5HLnN+TXv+L5+sErkpPX+ATP0BwMCpbzSQPW0AcTJp3cupR ddH4WhXqwpGZrXLII2eZPC0rVrxF9P+0W6H2AnduNzIIRDaQUWGaYhWVd+Qt4XMEnoF1gHUVvm3 r+jOXW4MSVqE2p6M0HJd3BUSGPy9Dt+D9eYPIltXITG21fWaIKq0hkLsXf94xRYLIXd1i1ik X-Google-Smtp-Source: AGHT+IGN/ZCUfpQSbaBS4W8bBOfntfSIoehjtu5uWTeww1FUk2wFfobrLS/nbSO2R96mT1mo6Vfdog== X-Received: by 2002:a17:902:e552:b0:223:66bc:f1de with SMTP id d9443c01a7336-22bea4afc0fmr177104965ad.21.1744685192967; Mon, 14 Apr 2025 19:46:32 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.46.28 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:46:32 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 07/28] mm: thp: use folio_batch to handle THP splitting in deferred_split_scan() Date: Tue, 15 Apr 2025 10:45:11 +0800 Message-Id: <20250415024532.26632-8-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 1C9C7160002 X-Stat-Signature: 4376j19neta7ru1b5ikuohrugcowqfyp X-HE-Tag: 1744685193-823830 X-HE-Meta: U2FsdGVkX1/UX3VLCAryOwZdzQJZfK+jKf/qRIxmI1HZ9kWNEC+wnH+Btqpia8TBNetQYMX/vSX89/VipwPtetN8zZEtkEUMIA1yBBIP59rpRill57Gr1omN6fbvbv3pyRpXh2qQYWfr8X7MdovOYfQOoQkhQXhEaXQW+ypsE+neL1a5Tr0WzCRoVSYWn8dZOoUOlLsY9gFbi5VR5T8rIzNnDdDahDtKcx8l9WXqi32SzMXHFN1YiQKjC4K2+wKYR1RBUEJliiN/EDcr3+DM0gzRUMv1il13B256vaNrevRjIZDW9HawTDKuCazEl5rPk3kyut+KLXzBb4Xy2A1orUUid6OUCWjXBg7dLrEmLBe+AMOYwddlv7Ybzcvv1c9jYTfHqizTd+uIpApMfCHzQvB6VyboSHgoduwanvnboQU/a7grnsS1OaXgMa0Tvj74/QOtjF/XcgPPyxAXYxEObgMq1aLtnSAy9re4TPc0jPe3AMRPScaocOfypfyb9nMPhjSfZ3v59MFteGFK6FXPOvV0IlwpyEbLqc8pTc9AKul7xa8ZtpQyKURYUqTevDyKwGRAZZD+bfcEC1At+c9bJLRguRaM5ZzJ8cfcJ0U23gvgfpAlEY+b5Br9ov0ytER5CjFFzMyU2Vs5pG8SnexqAr3MPv5qadOo4nyfvZMrValmV4yoWTUGd55ebECk9PeW6Y/DSGFHRFDQl73mcdKU2aEsub4GkRxfaR2+Qe9oNW4F261XuNuOoy1LMtingDr78VPLVtKIe/TdP0Uho7FsByE/I3AXduoQkywvNMau4xhXdNMvM9HkJGNtixH4iQcJP5nOgPRWXOsBBfwYYcX3CsLcajDjS8nkgKHXUn3dc/BUm7iluhSe5A66GStuHqgTImS93vqJyeVmNmq0qnnySJKlo5mqQ1v40ENas9svAAMXKPSSDQFbxWuQMcdNvuO00ozpVqJPxAfPnNLId0R PwWBfs8H j+c4j1jVEGgxGFs/VUXF0+EPIyef8Fv9g8rYVm63i0FNcaOfLqFQ9l7onIDu30KVDEgGWqRjNTACOnjLudJXRpsTQ68OiR+MEGgFBWqIw3S2VqDEXzVpjV95y8Umz6ixIjWqB7cVFzhHoGIy4rPkEpifmoAFQ3DN1mFl3KZ8yutL/hntgDh7xuxZ6g61QdZC65HZ7xqMkt7kKVEw5oK/rQB1MZWG+1cw2v5VXgaAcy4cvfHxxDpkUUF5QXphkPoX8XLVdp6j/cOFwbtnS+WFqWHzog0xJr5WXCLLgze3MSHaRD8Dv+wDHcKWPgFIgTTJuaBHz3Xp4l+8K2zyUGEMkA8GD8Pf54nQErNh1dpZEfzXaYHE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The maintenance of the folio->_deferred_list is intricate because it's reused in a local list. Here are some peculiarities: 1) When a folio is removed from its split queue and added to a local on-stack list in deferred_split_scan(), the ->split_queue_len isn't updated, leading to an inconsistency between it and the actual number of folios in the split queue. 2) When the folio is split via split_folio() later, it's removed from the local list while holding the split queue lock. At this time, this lock protects the local list, not the split queue. 3) To handle the race condition with a third-party freeing or migrating the preceding folio, we must ensure there's always one safe (with raised refcount) folio before by delaying its folio_put(). More details can be found in commit e66f3185fa04. It's rather tricky. We can use the folio_batch infrastructure to handle this clearly. In this case, ->split_queue_len will be consistent with the real number of folios in the split queue. If list_empty(&folio->_deferred_list) returns false, it's clear the folio must be in its split queue (not in a local list anymore). In the future, we aim to reparent LRU folios during memcg offline to eliminate dying memory cgroups. This patch prepares for using folio_split_queue_lock_irqsave() as folio memcg may change then. Signed-off-by: Muchun Song --- mm/huge_memory.c | 69 +++++++++++++++++++++--------------------------- 1 file changed, 30 insertions(+), 39 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 70820fa75c1f..d2bc943a40e8 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -4220,40 +4220,47 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, struct pglist_data *pgdata = NODE_DATA(sc->nid); struct deferred_split *ds_queue = &pgdata->deferred_split_queue; unsigned long flags; - LIST_HEAD(list); - struct folio *folio, *next, *prev = NULL; - int split = 0, removed = 0; + struct folio *folio, *next; + int split = 0, i; + struct folio_batch fbatch; + bool done; #ifdef CONFIG_MEMCG if (sc->memcg) ds_queue = &sc->memcg->deferred_split_queue; #endif - + folio_batch_init(&fbatch); +retry: + done = true; spin_lock_irqsave(&ds_queue->split_queue_lock, flags); /* Take pin on all head pages to avoid freeing them under us */ list_for_each_entry_safe(folio, next, &ds_queue->split_queue, _deferred_list) { if (folio_try_get(folio)) { - list_move(&folio->_deferred_list, &list); - } else { + folio_batch_add(&fbatch, folio); + } else if (folio_test_partially_mapped(folio)) { /* We lost race with folio_put() */ - if (folio_test_partially_mapped(folio)) { - folio_clear_partially_mapped(folio); - mod_mthp_stat(folio_order(folio), - MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1); - } - list_del_init(&folio->_deferred_list); - ds_queue->split_queue_len--; + folio_clear_partially_mapped(folio); + mod_mthp_stat(folio_order(folio), + MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1); } + list_del_init(&folio->_deferred_list); + ds_queue->split_queue_len--; if (!--sc->nr_to_scan) break; + if (folio_batch_space(&fbatch) == 0) { + done = false; + break; + } } split_queue_unlock_irqrestore(ds_queue, flags); - list_for_each_entry_safe(folio, next, &list, _deferred_list) { + for (i = 0; i < folio_batch_count(&fbatch); i++) { bool did_split = false; bool underused = false; + struct deferred_split *fqueue; + folio = fbatch.folios[i]; if (!folio_test_partially_mapped(folio)) { underused = thp_underused(folio); if (!underused) @@ -4269,39 +4276,23 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, } folio_unlock(folio); next: + if (did_split || !folio_test_partially_mapped(folio)) + continue; /* - * split_folio() removes folio from list on success. * Only add back to the queue if folio is partially mapped. * If thp_underused returns false, or if split_folio fails * in the case it was underused, then consider it used and * don't add it back to split_queue. */ - if (did_split) { - ; /* folio already removed from list */ - } else if (!folio_test_partially_mapped(folio)) { - list_del_init(&folio->_deferred_list); - removed++; - } else { - /* - * That unlocked list_del_init() above would be unsafe, - * unless its folio is separated from any earlier folios - * left on the list (which may be concurrently unqueued) - * by one safe folio with refcount still raised. - */ - swap(folio, prev); - } - if (folio) - folio_put(folio); + fqueue = folio_split_queue_lock_irqsave(folio, &flags); + list_add_tail(&folio->_deferred_list, &fqueue->split_queue); + fqueue->split_queue_len++; + split_queue_unlock_irqrestore(fqueue, flags); } + folios_put(&fbatch); - spin_lock_irqsave(&ds_queue->split_queue_lock, flags); - list_splice_tail(&list, &ds_queue->split_queue); - ds_queue->split_queue_len -= removed; - split_queue_unlock_irqrestore(ds_queue, flags); - - if (prev) - folio_put(prev); - + if (!done) + goto retry; /* * Stop shrinker if we didn't split any page, but the queue is empty. * This can happen if pages were freed under us. From patchwork Tue Apr 15 02:45:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051361 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7565EC369B4 for ; Tue, 15 Apr 2025 02:46:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E8D3E28019B; Mon, 14 Apr 2025 22:46:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E13742800C2; Mon, 14 Apr 2025 22:46:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C8F6728019B; Mon, 14 Apr 2025 22:46:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A665A2800C2 for ; Mon, 14 Apr 2025 22:46:40 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1D7381CC58F for ; Tue, 15 Apr 2025 02:46:41 +0000 (UTC) X-FDA: 83334740202.11.728F6CA Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf10.hostedemail.com (Postfix) with ESMTP id 3D368C0005 for ; Tue, 15 Apr 2025 02:46:39 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=FWkelz0H; spf=pass (imf10.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685199; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FoWdLHOi8MZDe7tFGtzR8WCw9Q33zyNjsUf4A8uDwIg=; b=3Rc+9p2VIYoUU3bqcmZuHeFED3KFqQd6xZdeOWsEZhSNfcRJzGNKQmrShPiWf5Ddh15Bho NLLPymWScpOcDQ6UMwidSWTRhrfW9/Lg5oidjZVz0bpXZS6e/SMSMqY2gImrZ5UyFG3nRb pjVvHHhgX584UPOgzn/snWq1UWBu1j0= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=FWkelz0H; spf=pass (imf10.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685199; a=rsa-sha256; cv=none; b=Vm8Uef5Ymr4n+z2oyBzZ5hfJ5t3IUxP1NoEh5BelfKAOfa1Bm/r8LFORhjfHWacwzROgbI 9fsHuDl0d7XUvYb3WHA3azf4/iY66WhT9XYVSmTe7F1kVPhH1T9L7xBOb+jXDGEhZvnm+m NqZ9xkBygoah9i5WwsdTzGa17WYclXA= Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-2ff799d99dcso4702081a91.1 for ; Mon, 14 Apr 2025 19:46:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685198; x=1745289998; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FoWdLHOi8MZDe7tFGtzR8WCw9Q33zyNjsUf4A8uDwIg=; b=FWkelz0HhiFzBlUnqNWAvRKamGKJUTvJZkwzHP3MLa+w3Qx02hk/KPtqrcmuWPI8Gp 271UMKupBbOPW8TNz/V5Whxv01OG1FbFK1zHIyTNefTTJvXB6WaKSC2sNEhgZ5WFhgzD 7j9PYjBN72WYch5/HPOWqGchtu1C1pIghsOvcGFNqhfZBT3hMcTlcNXJE39ptP7RGcnt un4WcCFoVTdbWPfFJSPewXs1GUCtzlh9eEnVh2TsbYA910xxVfnDT1P0I2n07NFnlf3M qGp1rCc1iP+UlIfgq2e7rOYwVPJ4oUnHlNJNx3728qggs34FvCvIutKBTVY/ZOKqERTl W3jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685198; x=1745289998; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FoWdLHOi8MZDe7tFGtzR8WCw9Q33zyNjsUf4A8uDwIg=; b=b9B9yKYsvFyEnz18p1xYymtljARzrWZjSRIagu6Mn1uJIG8d0ErohbLsJ8yrC3ECpK 74WRm39Ct3aPJuqIw6b75vSXKGp+hEZJfFVZG/z+YZsX6Om1QCFc5YQEia1OZKRF4kkC jJAB4OkIr74S9cSatfkSgbE+6YsPW324TgoWQoetOphFB0GCHNSEyViglBEo1fA/m35h NVr0BkMWVliVDbznEMSIr589mWZxO4UBduhzQ0euhszd6ast6/Nf0krUi5DOHtG/nYia 4UYIPb87K8cfJLms0Dzy5mirAb+JMIQFzI9jomnC7JHtDyOsF1QjzxKdBxDBVsrDk+mu y/ng== X-Forwarded-Encrypted: i=1; AJvYcCUjDenKR+qkNFnH+5x/zrRgLpVQZiMUF+fPtDzQOWUejPOHU8vPoa/27cki5b1PRKJUQ+8XoaCwrA==@kvack.org X-Gm-Message-State: AOJu0Yx3DFjPSe26+cMLwRug5udcSnVil9ofqkfuyR5mTAjTt+DnbfKJ 0EedbUgPxRN8Rc0UeIL00Gb6OBJ0FjzNSo8GovWLTPGhG4yxtZ1l+C079HIUCEE= X-Gm-Gg: ASbGncujO023DFscQESXNh9Y8qdXS8XEWsXsUNvr4NtwjJgpoM1MtCJQm4cYjrc/uLe +dEQzCXtMYQgPEAwt2wj3+IwuFuCQj6dVr6F/mEP2v8sU92K8YMrv4lEC8MUrzodCS13SsjsioW 0jtDUEwhu/CWBJBAX6INkRXHo+5wn2BpsfL6GtyKen3T+5IW8wQILfoB0cb0n0jsXhw+w6JPQwn ct2EDPh0QkNTx3IYLqyduByZRqOYfeTVFYim8ea/3Zk5mtmCFoLpTnvp7DjOZ10dvzV0g+EE2LN ll4dQtUU/6oe2LzWr0vpBrKMdSlqTFW2fWPIYZrzPNbvICpf1STY2c8oKuYiRfpZ1YHHlmmMiME nIFM3ehY= X-Google-Smtp-Source: AGHT+IEKT/drxov/S/IJkMzk/bPhvMbUiZuHxa6kWkMhcuX0SINcw3+8GdOiNf/WYjYHBQMzkgoxKg== X-Received: by 2002:a17:90b:2f0b:b0:2f6:f32e:90ac with SMTP id 98e67ed59e1d1-30823639726mr23171470a91.11.1744685198109; Mon, 14 Apr 2025 19:46:38 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.46.33 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:46:37 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 08/28] mm: vmscan: refactor move_folios_to_lru() Date: Tue, 15 Apr 2025 10:45:12 +0800 Message-Id: <20250415024532.26632-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 3D368C0005 X-Stat-Signature: pf7o9gnqb8tzkpd1a1a445ozjb4dfepu X-HE-Tag: 1744685199-934166 X-HE-Meta: U2FsdGVkX1+g+lDkhADyybp1UnU4U7Ug64HHqhe1c4j2MNxWyBDcV12+FXgj/cbvrrpyd+Et2Dkl2yysAgFadCWTRtkKuHrfQPQnqbBVjHTCvULgl4mEVT9Xgl8/lpkikzt+Xtuv3JkzcBp1wDHzAMC6yBAGY15XLIw7H3NIXjBgEn7+TjTUg0/iAManRqe/IEz4VJJv7PKiiVmo6RDuDzWvI/+j3lp6Jn6BokuTdE+5KiP7CLSgn5JmYuVUYPevm5t89s0MxkK4KubkQnA9JJsv2AXEbdPeGyqxbp55sZSYgZQhz+o7tMAH2BgTtU2oTWCwaZEE12HUKz59+nUxmEt37CLtdeKQ4ZeLu+M//WCfi2WIaVrIRpW/G9U1+4ZTbsSpWClnOLGkLt/qp6XAFXeMWPhhYlxLnaRpc2UM7DLMl6UDb3i3Q7hH78FXPeXzxDC2N2NBHVVOK4StZVshQ3blE/Y7O1HC55M+v0T2KERl+Bq/F4aZ0PPjBPHU6F+gpuU654k0TqCAO/nwmyOWhG7YSd7RMzqsKGe3BGtfCGAp7KR1M57SrsW263O7Mqgl7Bn3YmwCOEHRIm440C78PzMmDQl4Jxxuep3r+EQK53qgUNHnEUEUfSlIEG3m+a1DEWbT7R+mWJo+hxkcc7BH93VcUcTs3N87fs3xcQX6TnsTHpCyDpdmdPMob1URxZnHybn5+dFLU5LFv/3+YYhRC0LfI3iJd2QR7khn3uAMAkpYHORBwqzRlsgNjxJr3OCV3xXpIhZ1RQdjZBXDKBpMxDvmNid/xh953gmkBOgMRE1Ar/W6vOHoo90pSKdquVGK1+KOeyGF16M99xYwKpdGIXBz201U85kBa7yh/bbatmqN1Zj8aqIaATkc/GTuLuIn43mwPZlcz/ucyLIKgzvqu8ONPXLsn7yNcgCBSQYhn6Imtis5srjahslOR9MkUvUTRZjQZigUejzgSGqs8fJ g3ystd2V 10irJrwwzZvXFr4uU2YYOHVL0nQzcrthke9WyM6YjuCm7sL9WpTAkYFNVzjCCLto4pXXTBmnXqjnZgft1Rh6E0v9vUQd6CEHHwAyGFSZb4s+G17VuPMcSFGVHFfaqqKyW00kL/BUjxOlFlCxkm/nz3dZ6YkViLWTx2R98n3UDz5I4EVbyz6ZyINmHQE97m7xHHIEuqqaRpeNuHop+aD1Qvswxf27pAvrkOMGOtWkC5MrVNcLjsxUmgDFLCJD2jvRygB7AlSB71wkm5aSoxDSGX/BcMLwO8gQQIX+Cm7B65aTNOclGasuL8/PKWpcgpYoA4Cho6lir8ts/UFfr44MZtoFiikW2TGcXpjydsD4Y+uc6ulA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In a subsequent patch, we'll reparent the LRU folios. The folios that are moved to the appropriate LRU list can undergo reparenting during the move_folios_to_lru() process. Hence, it's incorrect for the caller to hold a lruvec lock. Instead, we should utilize the more general interface of folio_lruvec_relock_irq() to obtain the correct lruvec lock. This patch involves only code refactoring and doesn't introduce any functional changes. Signed-off-by: Muchun Song --- mm/vmscan.c | 51 +++++++++++++++++++++++++-------------------------- 1 file changed, 25 insertions(+), 26 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index a76b3cee043d..eac5e6e70660 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1901,24 +1901,27 @@ static bool too_many_isolated(struct pglist_data *pgdat, int file, /* * move_folios_to_lru() moves folios from private @list to appropriate LRU list. * - * Returns the number of pages moved to the given lruvec. + * Returns the number of pages moved to the appropriate lruvec. + * + * Note: The caller must not hold any lruvec lock. */ -static unsigned int move_folios_to_lru(struct lruvec *lruvec, - struct list_head *list) +static unsigned int move_folios_to_lru(struct list_head *list) { int nr_pages, nr_moved = 0; + struct lruvec *lruvec = NULL; struct folio_batch free_folios; folio_batch_init(&free_folios); while (!list_empty(list)) { struct folio *folio = lru_to_folio(list); + lruvec = folio_lruvec_relock_irq(folio, lruvec); VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); list_del(&folio->lru); if (unlikely(!folio_evictable(folio))) { - spin_unlock_irq(&lruvec->lru_lock); + lruvec_unlock_irq(lruvec); folio_putback_lru(folio); - spin_lock_irq(&lruvec->lru_lock); + lruvec = NULL; continue; } @@ -1940,19 +1943,15 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec, folio_unqueue_deferred_split(folio); if (folio_batch_add(&free_folios, folio) == 0) { - spin_unlock_irq(&lruvec->lru_lock); + lruvec_unlock_irq(lruvec); mem_cgroup_uncharge_folios(&free_folios); free_unref_folios(&free_folios); - spin_lock_irq(&lruvec->lru_lock); + lruvec = NULL; } continue; } - /* - * All pages were isolated from the same lruvec (and isolation - * inhibits memcg migration). - */ VM_BUG_ON_FOLIO(!folio_matches_lruvec(folio, lruvec), folio); lruvec_add_folio(lruvec, folio); nr_pages = folio_nr_pages(folio); @@ -1961,11 +1960,12 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec, workingset_age_nonresident(lruvec, nr_pages); } + if (lruvec) + lruvec_unlock_irq(lruvec); + if (free_folios.nr) { - spin_unlock_irq(&lruvec->lru_lock); mem_cgroup_uncharge_folios(&free_folios); free_unref_folios(&free_folios); - spin_lock_irq(&lruvec->lru_lock); } return nr_moved; @@ -2033,9 +2033,9 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, nr_reclaimed = shrink_folio_list(&folio_list, pgdat, sc, &stat, false); - spin_lock_irq(&lruvec->lru_lock); - move_folios_to_lru(lruvec, &folio_list); + move_folios_to_lru(&folio_list); + local_irq_disable(); __mod_lruvec_state(lruvec, PGDEMOTE_KSWAPD + reclaimer_offset(sc), stat.nr_demoted); __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); @@ -2044,7 +2044,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, __count_vm_events(item, nr_reclaimed); __count_memcg_events(lruvec_memcg(lruvec), item, nr_reclaimed); __count_vm_events(PGSTEAL_ANON + file, nr_reclaimed); - spin_unlock_irq(&lruvec->lru_lock); + local_irq_enable(); lru_note_cost(lruvec, file, stat.nr_pageout, nr_scanned - nr_reclaimed); @@ -2183,16 +2183,15 @@ static void shrink_active_list(unsigned long nr_to_scan, /* * Move folios back to the lru list. */ - spin_lock_irq(&lruvec->lru_lock); - - nr_activate = move_folios_to_lru(lruvec, &l_active); - nr_deactivate = move_folios_to_lru(lruvec, &l_inactive); + nr_activate = move_folios_to_lru(&l_active); + nr_deactivate = move_folios_to_lru(&l_inactive); + local_irq_disable(); __count_vm_events(PGDEACTIVATE, nr_deactivate); __count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate); __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); - spin_unlock_irq(&lruvec->lru_lock); + local_irq_enable(); if (nr_rotated) lru_note_cost(lruvec, file, 0, nr_rotated); @@ -4723,14 +4722,15 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap set_mask_bits(&folio->flags, LRU_REFS_FLAGS, BIT(PG_active)); } - spin_lock_irq(&lruvec->lru_lock); - - move_folios_to_lru(lruvec, &list); + move_folios_to_lru(&list); + local_irq_disable(); walk = current->reclaim_state->mm_walk; if (walk && walk->batched) { walk->lruvec = lruvec; + spin_lock(&lruvec->lru_lock); reset_batch_size(walk); + spin_unlock(&lruvec->lru_lock); } __mod_lruvec_state(lruvec, PGDEMOTE_KSWAPD + reclaimer_offset(sc), @@ -4741,8 +4741,7 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap __count_vm_events(item, reclaimed); __count_memcg_events(memcg, item, reclaimed); __count_vm_events(PGSTEAL_ANON + type, reclaimed); - - spin_unlock_irq(&lruvec->lru_lock); + local_irq_enable(); list_splice_init(&clean, &list); From patchwork Tue Apr 15 02:45:13 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051362 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53E38C369B4 for ; Tue, 15 Apr 2025 02:46:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E28702801B6; Mon, 14 Apr 2025 22:46:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DAF352800C2; Mon, 14 Apr 2025 22:46:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C25422801B6; Mon, 14 Apr 2025 22:46:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A20E52800C2 for ; Mon, 14 Apr 2025 22:46:45 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 3A285120B7F for ; Tue, 15 Apr 2025 02:46:46 +0000 (UTC) X-FDA: 83334740412.18.875B955 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf16.hostedemail.com (Postfix) with ESMTP id 7DB7A18000E for ; Tue, 15 Apr 2025 02:46:44 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=DgsDK6va; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf16.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685204; a=rsa-sha256; cv=none; b=Xngal/fi93ydGw4WVduRYc6z3QzfuIgD10WD1qWVVduCPl1ob0RbWayvW8MV8ITlJR61/Q BuOKd7bS6IpCJbP2woA2U/EUnNAAmyoVO1005P9Vlg07QrJosyVKYVpiJb0AqIb6C5FGYT XTXZiTFBsQR91EW+ltH9LQXFTDFrffg= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=DgsDK6va; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf16.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685204; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4255W6cWFY69GBJYbQ0IFQtoRNp4GanHOHXEYxbFWEk=; b=UnQvmP4bsx/xyE1NuzwyPsXbepdlWBvleMbIk1MVYwNGUKMOZr3t/r7momrIsUcO7hFwfu FAgWxQdTOOX6gtkI/VXsX57egC8RNd0pzufrlxioANjdOslV2P9C7A1HT6QZqL4Jp5w4sf KbfrOk9h5JP+zl8dec8HXMehOKOb0do= Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-22401f4d35aso56951615ad.2 for ; Mon, 14 Apr 2025 19:46:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685203; x=1745290003; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4255W6cWFY69GBJYbQ0IFQtoRNp4GanHOHXEYxbFWEk=; b=DgsDK6vaApP9E881qzPcNfrESS/S8w728oQk1So0gF5yX9Ms6WceN2um2N3JSrHI8B SEU1QMsLE8P76j0TA63krhR4mkTXIuwDAiEdbzpj55GyxibBimlpDHERhBbDQW+8jphq myqZusRMFIbIAmU7cZB4pzzCSeBabv0pCeku6hZWVbPl0Hj+Iuv27O2u8wgzJ8dYdjJ7 TEJWJzDWDhXcemlRDom462R1Dg37cQQ6Vfi2dpcYVr34umE+wVPGRdp+jTtMSjOX7Ysi 45MICFBeZ9I0/uBovRp62TK0uHyiDbwWpXYoD1rFMdudYgnn/PXfNDXfAQ5yhby9NWue 2GYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685203; x=1745290003; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4255W6cWFY69GBJYbQ0IFQtoRNp4GanHOHXEYxbFWEk=; b=ZWUdAJ6wNFSGdXemvkeilflFP48UzO+/XnA+8Mq8CZAvV5dmBvnsN9gbUbz9jZ1Rnl 8yvlHktEvuxgv6GXqESV3ICU61KLtLEh9N2CveOXJyQh6FsuxPMj7S5K4sWnV+TvFJOX RSk2qGljAEs1DvuQnF4lsIswkXzcrygtNaC5lY0G9j2OcxtiCGqWRjcUdyYK68WJbbYW d7+DVlWiFoMrsn4g7E/FW4UPcl9NTj1XaB9UTbR2emEy8lI2Cvo5yC9/RPbG4Tb3pVVM KehkQPf5pgB9uHPjtoKCvaEoI3dzzg3Hjm5N6Sgb4mYktoPJaTTku6fGfk+sPrm0XVL4 tJtA== X-Forwarded-Encrypted: i=1; AJvYcCWSk3Zn2cm8+a2XWfUHEy2hAtLl8qemTSn1K4MMJ+2+HCeKaa+5cwRb+XmHbTOahz84gOta6iG8Hw==@kvack.org X-Gm-Message-State: AOJu0Yw0etXM3wt1IOmcuHApoarswDsKsDyx7dWmmoaluKfnX3dVBBaa /xj+kUZeLhNbNse1Joj7K+H0aeGgQ/+YPXRPyM54f771SLFNdMmIxoCTzSWjEps= X-Gm-Gg: ASbGnctnOChq2pc1rtgzaqIMadOkmpv3LKnH6W1MfiuACU01bmLhiH6OnztSEVHM+sn MSpIqVy6gDzpZpMCDEd6LXCGla6TAI30lDhY5DWpKP71+bg0UuNbNwF+vbTqHFkrNwieQVJ+YwS XNtUeVef1jo1gP3DLeKbsUHEgbyvg6UHSbRwBioBf4PZcBmAKXWHfpzYIEBUTXMpoUhMUGufPyw GP1vTmHIqhtARSktnRrXgZCU0YLShUpR5MnRyv2NyHASflQcyfn2EN0830wShg/A0TBs4Yi9ZL9 iGpkXvLIiU/ksxctmbZIJJcQON5wOEDd3Jnq62/0+5jCoHaKqXkdlMl+boduFcHIRwhfEq7e X-Google-Smtp-Source: AGHT+IF1hz17w48BHL1psfGG5kRB5zOy7YB0gh0pPjQ8MypiC8y7ZHlHyktUWpSXGC6ZIkLQ5ghe2Q== X-Received: by 2002:a17:902:f546:b0:210:fce4:11ec with SMTP id d9443c01a7336-22bea49575fmr200724635ad.1.1744685203339; Mon, 14 Apr 2025 19:46:43 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.46.38 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:46:42 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 09/28] mm: memcontrol: allocate object cgroup for non-kmem case Date: Tue, 15 Apr 2025 10:45:13 +0800 Message-Id: <20250415024532.26632-10-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 7DB7A18000E X-Rspamd-Server: rspam04 X-Stat-Signature: iyt6z5pa17boajnwy31kgi1dgt1kcjmj X-HE-Tag: 1744685204-807141 X-HE-Meta: U2FsdGVkX19aspYqwt4x+zVmAGWjYm+rxGQ11E0QE8/dIN4JQZFqIyosJW9jQ/qxAwOV/KDCha/W1FyTijYc0FIaK7GQJ9LonPzAgQxCzUEGmetwhJ91mgM2dMHxq8MKvCtUH9gJWF07zIiK4JLTrpFoeQrzF5FWujCoMnYfjH/d9Af7WjdfATBoWjSwh7+7hfSwf1PEAx/v+xpjZ9JVJCGSjIKSldae8oGntKra/2bf+uoOYk/Syh0sRv/C3J2uU++lftc6bRnb2xTSgW59L9TxH/ywVJEux7R5mS4P20zzJYpIJqw1nS8M4ZxGZYqOgkcKqxly9VueIvQzLptFKyWqyBbXNJM154QFAmSc3NOVQFMBXXJgLlM/qTcDuH+OqSpUUHx529h73B4TAeJqz65CipSoBelVYbz8iZMu2KfL4Sc00JUQ4fFYf13wFepQZ66yVWyfHDATm98GY/Idq4gdj2Tyf3GCu3SEp0Eq4d7vOlQs0JyqcBUoWlKaooYz4D9uwuj+tx0bjenjkmARzCT3kvLFgv2ST6XMZo6GaX+El9/QC1fMDev2zU1Z8zaeHWY2rNYwBt0eZflQU5fqyEpRY5SqCUy0xNxYK+lXbg8htjLEOFkjCUpUo7bo3DztZo5kkE5trWWs3a0VyGE8b0i11qxJe2p7CRtjKbR1yPcUEKGjGeFWgjIjODmE6CEp0Ha/57k+rgVKCWIJzlZhyj1ie5wK2MZbTp3vvEm7Qas8yvUwYqDhOO+SyL3BAd8M9NpZq8kmoygWqo0BhFNoWbm1gfHQUfM3W+bT0klDZixTd4pFaFpXU2OOiRjE0x6u7qcVQUiggcWKQyUFT9PEPZXqNqglb+rElDCKBpyQoFj2QqAJHMo4mUDcAt3ofGlstcZF6j+9AEZdvP3aEUjPjplBTwYC3OcC42irk3rB4PR4jGTptTWkPvRuNkUaSjdDh7wJKfsan5uH/CszxV/ OfkSKEbJ RlvdIdDc9dpuXB+hSpo2PPCPWD8XSdyln8pA+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Pagecache pages are charged at allocation time and hold a reference to the original memory cgroup until reclaimed. Depending on memory pressure, page sharing patterns between different cgroups and cgroup creation/destruction rates, many dying memory cgroups can be pinned by pagecache pages, reducing page reclaim efficiency and wasting memory. Converting LRU folios and most other raw memory cgroup pins to the object cgroup direction can fix this long-living problem. As a result, the objcg infrastructure is no longer solely applicable to the kmem case. In this patch, we extend the scope of the objcg infrastructure beyond the kmem case, enabling LRU folios to reuse it for folio charging purposes. It should be noted that LRU folios are not accounted for at the root level, yet the folio->memcg_data points to the root_mem_cgroup. Hence, the folio->memcg_data of LRU folios always points to a valid pointer. However, the root_mem_cgroup does not possess an object cgroup. Therefore, we also allocate an object cgroup for the root_mem_cgroup. Signed-off-by: Muchun Song --- mm/memcontrol.c | 50 +++++++++++++++++++++++-------------------------- 1 file changed, 23 insertions(+), 27 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0fc76d50bc23..a6362d11b46c 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -193,10 +193,10 @@ static struct obj_cgroup *obj_cgroup_alloc(void) return objcg; } -static void memcg_reparent_objcgs(struct mem_cgroup *memcg, - struct mem_cgroup *parent) +static void memcg_reparent_objcgs(struct mem_cgroup *memcg) { struct obj_cgroup *objcg, *iter; + struct mem_cgroup *parent = parent_mem_cgroup(memcg); objcg = rcu_replace_pointer(memcg->objcg, NULL, true); @@ -3156,30 +3156,17 @@ unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) return val; } -static int memcg_online_kmem(struct mem_cgroup *memcg) +static void memcg_online_kmem(struct mem_cgroup *memcg) { - struct obj_cgroup *objcg; - if (mem_cgroup_kmem_disabled()) - return 0; + return; if (unlikely(mem_cgroup_is_root(memcg))) - return 0; - - objcg = obj_cgroup_alloc(); - if (!objcg) - return -ENOMEM; - - objcg->memcg = memcg; - rcu_assign_pointer(memcg->objcg, objcg); - obj_cgroup_get(objcg); - memcg->orig_objcg = objcg; + return; static_branch_enable(&memcg_kmem_online_key); memcg->kmemcg_id = memcg->id.id; - - return 0; } static void memcg_offline_kmem(struct mem_cgroup *memcg) @@ -3194,12 +3181,6 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) parent = parent_mem_cgroup(memcg); memcg_reparent_list_lrus(memcg, parent); - - /* - * Objcg's reparenting must be after list_lru's, make sure list_lru - * helpers won't use parent's list_lru until child is drained. - */ - memcg_reparent_objcgs(memcg, parent); } #ifdef CONFIG_CGROUP_WRITEBACK @@ -3711,9 +3692,9 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) static int mem_cgroup_css_online(struct cgroup_subsys_state *css) { struct mem_cgroup *memcg = mem_cgroup_from_css(css); + struct obj_cgroup *objcg; - if (memcg_online_kmem(memcg)) - goto remove_id; + memcg_online_kmem(memcg); /* * A memcg must be visible for expand_shrinker_info() @@ -3723,6 +3704,15 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) if (alloc_shrinker_info(memcg)) goto offline_kmem; + objcg = obj_cgroup_alloc(); + if (!objcg) + goto free_shrinker; + + objcg->memcg = memcg; + rcu_assign_pointer(memcg->objcg, objcg); + obj_cgroup_get(objcg); + memcg->orig_objcg = objcg; + if (unlikely(mem_cgroup_is_root(memcg)) && !mem_cgroup_disabled()) queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME); @@ -3745,9 +3735,10 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) xa_store(&mem_cgroup_ids, memcg->id.id, memcg, GFP_KERNEL); return 0; +free_shrinker: + free_shrinker_info(memcg); offline_kmem: memcg_offline_kmem(memcg); -remove_id: mem_cgroup_id_remove(memcg); return -ENOMEM; } @@ -3764,6 +3755,11 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) zswap_memcg_offline_cleanup(memcg); memcg_offline_kmem(memcg); + /* + * Objcg's reparenting must be after list_lru's above, make sure list_lru + * helpers won't use parent's list_lru until child is drained. + */ + memcg_reparent_objcgs(memcg); reparent_shrinker_deferred(memcg); wb_memcg_offline(memcg); lru_gen_offline_memcg(memcg); From patchwork Tue Apr 15 02:45:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051363 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A83F5C369B4 for ; Tue, 15 Apr 2025 02:46:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 446712801B8; Mon, 14 Apr 2025 22:46:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3CF4E2800C2; Mon, 14 Apr 2025 22:46:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 221842801B8; Mon, 14 Apr 2025 22:46:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id F0A662800C2 for ; Mon, 14 Apr 2025 22:46:50 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A13661CB74C for ; Tue, 15 Apr 2025 02:46:51 +0000 (UTC) X-FDA: 83334740622.02.2673679 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf20.hostedemail.com (Postfix) with ESMTP id C85AC1C000B for ; Tue, 15 Apr 2025 02:46:49 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=QT019ESb; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf20.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685209; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IuDX2yZAES6U0UQApE3qruOoIN30Kn4u1JZ0iDLJzNw=; b=uTnH+mFwcq2YYcqo9FC2+mFuyBR8CUH8ampFSYrbafgj6X6lD8Cy90aq6Nf5jTBYaDh0TH r88xIesb8mzi6ag+rPsNW9xg7i55hZXmuSY/E8BxYEp489H73/0Fh//I0Ulf8c8s3T0dXK yP4sh0YJoparVP/VrE78AD/fSLJXzw4= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=QT019ESb; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf20.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685209; a=rsa-sha256; cv=none; b=ORX48DMoEHwN6xHSwc8wnk925t5wzvvYtleNvrGe2q8rbSR4uyqqKlJ8V6wt7tdMIXH7Y8 vn+lgEWtgcJBL6ezaFFs9hJt2rgEa5wabzMnW7OTz0kFJvzpeQsqTGHTy9N4sxDQpuya70 3tcJDEG5wCXftzgc0TmV6rCqJLXLz2s= Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-2295d78b45cso70679555ad.0 for ; Mon, 14 Apr 2025 19:46:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685209; x=1745290009; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IuDX2yZAES6U0UQApE3qruOoIN30Kn4u1JZ0iDLJzNw=; b=QT019ESb26aVFd5QFXw0VsIPUB85Fk3iEXfVJtG88aV1J5aW4YUa7WsYgpkFAJ8R+H 0AnkAun+tUE9Llmn5kP6A+mvooJIIzup8ZvCVxZMJWzUcDHConiGSlXcSyl38nv+lp5E HF9aG7GE1eqEOgjActIUF1k3rjIsYkgkYknWOzac1aQcumw6wBDJGg7ter1OrEvCyHIl 0W+344vGUaIdxCPq70tNtRuQrpd/Z2TsbruLpj5xPVcOYqH0hdnphneHX8KT5tO5Fck8 NQMskFcnEWQI7G9Q4j2Ynp7RLpwgMYVP3z+Ve23PsZ9F5f3Y1dleVn9XeclztTLK72u/ Y0fQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685209; x=1745290009; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IuDX2yZAES6U0UQApE3qruOoIN30Kn4u1JZ0iDLJzNw=; b=qbBslc8JlTrjemG9SJ4wf2Bn0LV4DUv3RogBd1AdTZU9wOy7uTXJrLvzOh1wdbZaDa dRdnh568C1fXMCdTWqwl+EV+eyvKaF/GlHLPTwXtTP1vhfniCzKqxis+b8fh2FfhWGQw fqg4hFf+IUzpEwapbBZ7k/wOX6DtPckXdMbh4kHfPIUzChk6OS41RsXbhRV3O0fdInep rcMqAXIYexb9hVIBZfnZ6WYAZVCneBRuk7Np1FY4V9tUqkel/W2lP/8HuXY5s6zmRMPs QJ7/tTwttec2IcAQD7KQRPoWAIy091JyOmcTjwtoZqBPrGSbjncgI7cUCpHTMDfixo3o DpUg== X-Forwarded-Encrypted: i=1; AJvYcCViRj2Fhc2NIbMwKy+zzjSEvuSmAj3DbbhmX8Jn3tREXiPR5S1Syn/SpVVh3kItpumnOR5ytn+wOg==@kvack.org X-Gm-Message-State: AOJu0YzvBObYma3LRVAgOPhvBI7xifjlbc9oW9iWME6TSo6VKsdd/FlW JWDGBaGYgYzJrJ/paFH2f53Rr36Piiyvf3lZxKkLTvxqIWcCpe1x1B+HLkoShhc= X-Gm-Gg: ASbGncskFBI2tGZVz9r7S2gH525orjpiok8J+Q7JUHSoenjxbsXLVSHN15Ow3bazsbZ 95xARMQIEii7ShD8IW2TEwVrdhP53srje9bLU//q4YtO/l+kDnMB2/LHXncK5KMaOMI333t1bXR bn1xJpOwzDlWt6KU5D8FJ7kRBzzhb+jqsZBykRDs9pakyu8DPQGWqoxykkobTu27CtmDPMB1Kyq 6nDgFRjgzsuGQ36QtyI097Q2cKbArzHa3sdwF7RxA2HYf0hA0+iGOQfN3Uiulg4xK49UOPMQlC3 1ZInVw7bJbkJUwpoaMYVJbppaPO1nwgUyrKYidsYrfrfxcO2FZPAEPMFqfqky8sqT3Zmx4v7 X-Google-Smtp-Source: AGHT+IEiRiEAaC7KVEwbBorKdUbvn6lwcIuA3If1BEtGwUIR4XacRMxzFJ4y2u3Y4Nx1KigSul40og== X-Received: by 2002:a17:903:230e:b0:224:1234:5a3b with SMTP id d9443c01a7336-22bea50da98mr202894305ad.51.1744685208495; Mon, 14 Apr 2025 19:46:48 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.46.43 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:46:48 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 10/28] mm: memcontrol: return root object cgroup for root memory cgroup Date: Tue, 15 Apr 2025 10:45:14 +0800 Message-Id: <20250415024532.26632-11-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: C85AC1C000B X-Stat-Signature: 4i6wauap6ax57afmieio8ot89i4t5zic X-Rspam-User: X-HE-Tag: 1744685209-553020 X-HE-Meta: U2FsdGVkX1+MiJqDVOkI2ChZHFIvmQQ6mDU7iU0ENaqbNourSFJDix9qwBdUIQ2fzP/atjwHgUYJXpuKPcte3T0W0n8vSuTcyczYtMgPIyq0u23NsXuRcoCvHJkLMTDhjzF1HU4ZcZOeLG5NuqgcUtdS/56KEVSVoSK9fGZwnQa3NUTsjkD6zOMrMiN6Rf6+fR9+0lm69XAHrUo3wIZm93z1KlA+51Ie9idbKub3LzSgmQUXMwQR7HYXJxFU8VOGp9CAnOQ9sb8ksyRGlSrxjU2jcsa+k2mFKaCB71jtkzDVp4RCNz1BJzN2ZnH+1a05Nle1w1CjzuPsKDlpB4Pptc84R5nsE8/4UpdgiFn5VlZXI/tPxYK08I31KamQvwOZ1oieACh7/vp/KG64qeZ60wegt9PuLt9zdUg+C79IMgZPMCva6uFmvBjjHLDj//G56dFZJfISc+dIWv3HDuxuyIruJMD0/TJmx1c+7NUItIpa9FxZBH1u069jlM//cralHOPE4HUrac6i4Bp02sVAu8qV0RypzgWAG6IAfvtEJ8H3QPBzuZWeKNPZvsEZF70Rdwlj8K2lFCtxTWW+1p7q79WIpg15/BXJsyK1UcLNSlskrvN0G35CEex0Y4da6/PrHA7xfZy/Kf73zgs3yMsGwtMJ4RUhqscDvIK47aXbldK+QrEMb+nFol3dfNg+POh5Ubz5509hoBbsNKIij2XF8JQRMId1lUvLLesbsxLULGGFBvvrgpmTu7yjoN3+4YWCQV7rkY+xAVXLkFrSbqCKDFJc5sxFdJtprQ4uKKC8VYvALAieKbZ1ePekYEdgqH0eH2bCHF7nCBQARIYWnOUxVPK6g7aJ+IvukkmemvyLOfAN5pcLf/WquUX5dAjjEN9+CdvHiDNVbEFKQJxQAczNWCP+PWkCyB7W5bdWAJWw8un8s1rPK46tpVWZAb3rIguTAxDlsJrPlX7MqNaOCsF v/CGJqcu 5PvGYmT3kq+qnk5rIk+NZRNHGuFqRRiGjNr7njXK/A6h10WrE43eGBdGiGDGqwYIFe9OEDZNvHA3KaaDGsXjdLzIYGLNY5Sd69qBByVM18F9S85pXr4MlUdK/Y6nczvO+QRz3fljLy2U/YHnIMRq0zo86FDjtLcvxsXjf4YzJFy+PedM+FQoYjZ5Eiqf6d78aGvBhb32wbL7ImCFXS+eY5gMraceTAkl3iz00PmRIMVE53R0U2gH7qFOvYyAR46c3g4WujF61ixUiJZPN8R7jk0ZLbBXj06YjLKscfsDZV2SLFlxK0m6EDwzxKg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Memory cgroup functions such as get_mem_cgroup_from_folio() and get_mem_cgroup_from_mm() return a valid memory cgroup pointer, even for the root memory cgroup. In contrast, the situation for object cgroups has been different. Previously, the root object cgroup couldn't be returned because it didn't exist. Now that a valid root object cgroup exists, for the sake of consistency, it's necessary to align the behavior of object-cgroup-related operations with that of memory cgroup APIs. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 29 ++++++++++++++++++------- mm/memcontrol.c | 44 ++++++++++++++++++++------------------ mm/percpu.c | 2 +- 3 files changed, 45 insertions(+), 30 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index bb4f203733f3..e74922d5755d 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -319,6 +319,7 @@ struct mem_cgroup { #define MEMCG_CHARGE_BATCH 64U extern struct mem_cgroup *root_mem_cgroup; +extern struct obj_cgroup *root_obj_cgroup; enum page_memcg_data_flags { /* page->memcg_data is a pointer to an slabobj_ext vector */ @@ -528,6 +529,11 @@ static inline bool mem_cgroup_is_root(struct mem_cgroup *memcg) return (memcg == root_mem_cgroup); } +static inline bool obj_cgroup_is_root(const struct obj_cgroup *objcg) +{ + return objcg == root_obj_cgroup; +} + static inline bool mem_cgroup_disabled(void) { return !cgroup_subsys_enabled(memory_cgrp_subsys); @@ -752,23 +758,26 @@ struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){ static inline bool obj_cgroup_tryget(struct obj_cgroup *objcg) { + if (obj_cgroup_is_root(objcg)) + return true; return percpu_ref_tryget(&objcg->refcnt); } -static inline void obj_cgroup_get(struct obj_cgroup *objcg) +static inline void obj_cgroup_get_many(struct obj_cgroup *objcg, + unsigned long nr) { - percpu_ref_get(&objcg->refcnt); + if (!obj_cgroup_is_root(objcg)) + percpu_ref_get_many(&objcg->refcnt, nr); } -static inline void obj_cgroup_get_many(struct obj_cgroup *objcg, - unsigned long nr) +static inline void obj_cgroup_get(struct obj_cgroup *objcg) { - percpu_ref_get_many(&objcg->refcnt, nr); + obj_cgroup_get_many(objcg, 1); } static inline void obj_cgroup_put(struct obj_cgroup *objcg) { - if (objcg) + if (objcg && !obj_cgroup_is_root(objcg)) percpu_ref_put(&objcg->refcnt); } @@ -1101,6 +1110,11 @@ static inline bool mem_cgroup_is_root(struct mem_cgroup *memcg) return true; } +static inline bool obj_cgroup_is_root(const struct obj_cgroup *objcg) +{ + return true; +} + static inline bool mem_cgroup_disabled(void) { return true; @@ -1684,8 +1698,7 @@ static inline struct obj_cgroup *get_obj_cgroup_from_current(void) { struct obj_cgroup *objcg = current_obj_cgroup(); - if (objcg) - obj_cgroup_get(objcg); + obj_cgroup_get(objcg); return objcg; } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index a6362d11b46c..4aadc1b87db3 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -81,6 +81,7 @@ struct cgroup_subsys memory_cgrp_subsys __read_mostly; EXPORT_SYMBOL(memory_cgrp_subsys); struct mem_cgroup *root_mem_cgroup __read_mostly; +struct obj_cgroup *root_obj_cgroup __read_mostly; /* Active memory cgroup to use from an interrupt context */ DEFINE_PER_CPU(struct mem_cgroup *, int_active_memcg); @@ -2525,15 +2526,14 @@ struct mem_cgroup *mem_cgroup_from_slab_obj(void *p) static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) { - struct obj_cgroup *objcg = NULL; + for (; memcg; memcg = parent_mem_cgroup(memcg)) { + struct obj_cgroup *objcg = rcu_dereference(memcg->objcg); - for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) { - objcg = rcu_dereference(memcg->objcg); if (likely(objcg && obj_cgroup_tryget(objcg))) - break; - objcg = NULL; + return objcg; } - return objcg; + + return NULL; } static struct obj_cgroup *current_objcg_update(void) @@ -2604,18 +2604,17 @@ __always_inline struct obj_cgroup *current_obj_cgroup(void) * Objcg reference is kept by the task, so it's safe * to use the objcg by the current task. */ - return objcg; + return objcg ? : root_obj_cgroup; } memcg = this_cpu_read(int_active_memcg); if (unlikely(memcg)) goto from_memcg; - return NULL; + return root_obj_cgroup; from_memcg: - objcg = NULL; - for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) { + for (; memcg; memcg = parent_mem_cgroup(memcg)) { /* * Memcg pointer is protected by scope (see set_active_memcg()) * and is pinning the corresponding objcg, so objcg can't go @@ -2624,10 +2623,10 @@ __always_inline struct obj_cgroup *current_obj_cgroup(void) */ objcg = rcu_dereference_check(memcg->objcg, 1); if (likely(objcg)) - break; + return objcg; } - return objcg; + return root_obj_cgroup; } struct obj_cgroup *get_obj_cgroup_from_folio(struct folio *folio) @@ -2641,14 +2640,8 @@ struct obj_cgroup *get_obj_cgroup_from_folio(struct folio *folio) objcg = __folio_objcg(folio); obj_cgroup_get(objcg); } else { - struct mem_cgroup *memcg; - rcu_read_lock(); - memcg = __folio_memcg(folio); - if (memcg) - objcg = __get_obj_cgroup_from_memcg(memcg); - else - objcg = NULL; + objcg = __get_obj_cgroup_from_memcg(__folio_memcg(folio)); rcu_read_unlock(); } return objcg; @@ -2733,7 +2726,7 @@ int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) int ret = 0; objcg = current_obj_cgroup(); - if (objcg) { + if (!obj_cgroup_is_root(objcg)) { ret = obj_cgroup_charge_pages(objcg, gfp, 1 << order); if (!ret) { obj_cgroup_get(objcg); @@ -3036,7 +3029,7 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru, * obj_cgroup_get() is used to get a permanent reference. */ objcg = current_obj_cgroup(); - if (!objcg) + if (obj_cgroup_is_root(objcg)) return true; /* @@ -3708,6 +3701,9 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) if (!objcg) goto free_shrinker; + if (unlikely(mem_cgroup_is_root(memcg))) + root_obj_cgroup = objcg; + objcg->memcg = memcg; rcu_assign_pointer(memcg->objcg, objcg); obj_cgroup_get(objcg); @@ -5302,6 +5298,9 @@ void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size) if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) return; + if (obj_cgroup_is_root(objcg)) + return; + VM_WARN_ON_ONCE(!(current->flags & PF_MEMALLOC)); /* PF_MEMALLOC context, charging must succeed */ @@ -5329,6 +5328,9 @@ void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size) if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) return; + if (obj_cgroup_is_root(objcg)) + return; + obj_cgroup_uncharge(objcg, size); rcu_read_lock(); diff --git a/mm/percpu.c b/mm/percpu.c index b35494c8ede2..3e54c6fca9bd 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1616,7 +1616,7 @@ static bool pcpu_memcg_pre_alloc_hook(size_t size, gfp_t gfp, return true; objcg = current_obj_cgroup(); - if (!objcg) + if (obj_cgroup_is_root(objcg)) return true; if (obj_cgroup_charge(objcg, gfp, pcpu_obj_full_size(size))) From patchwork Tue Apr 15 02:45:15 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051364 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9CB9C369B4 for ; Tue, 15 Apr 2025 02:46:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6ED402801BB; Mon, 14 Apr 2025 22:46:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 69C372800C2; Mon, 14 Apr 2025 22:46:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 53CBD2801BB; Mon, 14 Apr 2025 22:46:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 33A8F2800C2 for ; Mon, 14 Apr 2025 22:46:56 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id C40B9BAD34 for ; Tue, 15 Apr 2025 02:46:56 +0000 (UTC) X-FDA: 83334740832.22.5909222 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf15.hostedemail.com (Postfix) with ESMTP id EB05FA0006 for ; Tue, 15 Apr 2025 02:46:54 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Tmhwsg7v; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf15.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685215; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GDFiTtsYE3vsXL9LKCv2J9wFa6MHITbi7MDnimw67RY=; b=pNcYvji6g+LEv1zRLc/m22aWT6qbzml7JyjlXBSo5asQNuze8x56cg0+ma2fJQcm2UIeUf R2tjdeVo9eN+0oD8Ka66fnSbr9UrdPsU0UgrV8I80qfnoCVLRmn91iKrQ9RylX7+jBYe0w mCnOyCt4S5xtZmTOzWxTLaAS1+ZGDFw= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Tmhwsg7v; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf15.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685215; a=rsa-sha256; cv=none; b=DFpWqJ/R7aWYytZh1kK2Z2JSPuDY4JUd8MbSoEX70/SqSbqhPBckmOWycKClZm0nMCpljY OONsU5Cngv61lJxJB8UhzoJl2GY5mpAk6hgG+1emLncbSChkra4kO0lbUkEmgPBC5zgOq5 iMlLQ64O17B4EVWsbMqg1WrJhMxBWOs= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-2295d78b45cso70680065ad.0 for ; Mon, 14 Apr 2025 19:46:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685214; x=1745290014; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GDFiTtsYE3vsXL9LKCv2J9wFa6MHITbi7MDnimw67RY=; b=Tmhwsg7vQTpZbHyndgfRAF09Nx5FXMjDtub+6+yC0AdO1R8XvvgpH0KQngNI5I9csd cCBSge8WQXEbmn4/gs5TdAlLZjIw844q0G4BCp9MKSV/dHHqWAOlnrRH082hw/KgYZ9+ vWYZ29If3C3WIiPTSnUlpy+OaTtIVfVUEZDpta4F5oYvxIEluThG5QZ4MDutxfELmlhz 9sUmXQUJQdBr6HtjrHoxfoH2i/q36B6TPzGVqoRw/P+SpBaI0MdBpP0WOstZJE6BOuhw 9zPHSz9O92tjUcvctcGXlOhZzgr1s151CRRrkEowht7YeOY20PU7RoJZWaA9vpibH+Fn c55g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685214; x=1745290014; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GDFiTtsYE3vsXL9LKCv2J9wFa6MHITbi7MDnimw67RY=; b=TYFnQJov5UVLlWZ+gGjiz5mhkaoHWXWZREIIEy6jehh4w+8+BjH5vrFkx9Ne7cxsug DY1AsAql5ubP5+L9HAXE47/R/k3ZBJmKGHUpGGe908JU7ypLE6FFeWB7GyNB/YJfD6ip NcTyC81U3y8olP1JHqbM94oiBItTviq6nM8TE7ibaMmc4m6cI7nKY8yNy6dBAZm443JV s2Hu2FYC8VDgAQGFJDpb8B1BVZeD3ecVS30EVz1yZGtuR1fVts/M9L1qaWcsOaOZlJB7 BKJrnA53Gd497/ldxZObsEKrEjfLWS3jgBDseXIEHHuBEJVAG3diDGmDHAgqc8BWw+gz EEMQ== X-Forwarded-Encrypted: i=1; AJvYcCV257Go2EcaHwfDgEmA5sJrvtQRj5nH2AxeyremQweGADvNMVADa19jxRak4mUK1WAjYbkHWjz2eA==@kvack.org X-Gm-Message-State: AOJu0YzRMVVWK0+v4vNdmYCqRP828/MEIYGAcmXmwXALj5M+XfxnW9hS Xh8QTqM4okc4vfhV7BPW0gV4rnBxziR5xFEuSIClL5/K6ruaZ1/U2qPJKm3IyAI= X-Gm-Gg: ASbGncua3GVMMsR1CcBSfH6W8IaLL5pcN161hiFXG7JUFYFpjpXgByz5RvEjbVTzS8U MHJdOOHzHlPeVQobf/IWQ4n3Yju0PM0SWVQ4iAvXNsmY1PiOuzzVKS7xlwFcvHBw2bZ4lT1mUIy /qsym77o7qmdKFAd26t2WPJkWxWqjipShVUOynMqSnbhorfJ9EDMfSmWjnHgrsUiymaA6jG8Saq AtOJsqP9z66Byhm824IzVNjfwxrZdgDv9oZL9JnRrb2PZUnRvE+O9WInMvQEHHPRXV2kTIEN21V CojcmVdo64EBLNzOfsUF0mEE7gQlEl/rWFGQEyzYN9tZTW8wynvqDJlc2FvJdoqhabJ0tD4RMA5 D5euIGlc= X-Google-Smtp-Source: AGHT+IFH7dR3wBmfihakVorT6Qxxwq/ncIuMTxmwaVROxMgkaq6cZIV7iNxi0oFb9UgfVvElafvYJA== X-Received: by 2002:a17:902:eb8a:b0:21f:6c81:f63 with SMTP id d9443c01a7336-22bea4b6141mr200724475ad.16.1744685213756; Mon, 14 Apr 2025 19:46:53 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.46.48 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:46:53 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 11/28] mm: memcontrol: prevent memory cgroup release in get_mem_cgroup_from_folio() Date: Tue, 15 Apr 2025 10:45:15 +0800 Message-Id: <20250415024532.26632-12-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Stat-Signature: uuhifw67ao5hj1dof9n8r3n8j5jtpcbk X-Rspam-User: X-Rspamd-Queue-Id: EB05FA0006 X-HE-Tag: 1744685214-743806 X-HE-Meta: U2FsdGVkX1/8wpnxd8pd6ykLjAfTY9+eiDFMO6JMphLpeJ/hnLNW3HMnpF8owNQKN8PwqoudrziTzlhp4NztSDy37QXDI1YFQNZy6Qw5T2i+lYl6PzBVLcket2/vNVlAzpY7SgkTWG3DLaSY5Kc5gXEZmTeGMrFwbLWSQKqR3o+sUoSlwxZ80nlBfs7pcHdMCPPgqzmodoKUh0vN18Jy0vDJStbk3SQaUyjmLp9l/2Ld0oezPBv8ZqS09TRf1VRr8dNvIW8F1PvpuAL1EvuGDShyD8arVyWoZNIqzbkfu2Vqlh8gCURhzxlTVsDwsMSCWwgYZ4Gqf9ZudBl+e4b6aCBpabR8p0bZiTZwcpnveU4v2yfhaiQSApkPWb7GI2fSdzvnDa4NXfnwUsomCmFd/C941JFP3ExEdp03bHetqGDnBOQUT1sS2nD5MDO+OscRmMlrfrsAZ6/Ri4M923tDuhiZphGdcStYn0H7oO1IIJfugYCnFrkRDgn4M9zC10YuoZ0HEyasKOEXCs1x9fhVDXVpx6C2T5Rm2xTECr1FGkhUaocpSggiPcOdOLLupkkOKJtfDTD2LzpivtjHIM5MdfhBJtf9cbpndJiamLVToI9tmJLoAXYSIPxKVpPquQmcUfa90beHrrTGs8/AV8rXaDRq61hf6fH1ywB6e1NuAyRGJqJs+HCVLY0SXYnGK6wnlcqDL9pJtwDXS/zcwkNb6xkkD1eQGOXI8RftifRywYs505U+cGi9VPMANCQEjaf/c4fqqn6cL7nxKCyUs2QEkSV8Wwt/treSNfF19eb0sqJkxOnnFmlbtZptTZCiQ9wQK4JG39lWfalYHZ5zwSihkX8rQ1OJXK08K0WrYILgF5ORarV/Uh0+5C1uFZ9L78l0GloOwf+/Z+8x4IaNIR3IucwZlrrMU6h+XEYIDTgdkaDc62DV4XjHhwDpAQzcPBEHEg41lPBwBo9/4z19TKi Nx/9r3+F C317+kz8jxBW64/2wKETV/j/pLqAi79x2P5NtpNYbd91M71qF4xcyYVFv89Ktu05jWD/yaya7vaUwRdrVB+VkufgY7U5GHzJ1fk/14nB5CtzZBCtMml6lSFPKqPHWaal0uJq2WC+PPF3Q77WmhFZPbkaAzBPsJmrZ3RKQfR1qXqg2fasawdaS3Cdn6eQVPLaOubh9JbGMp1hBSnYldoTZw2CwWGmx0bpFPhVG/TMdfL/k73CVz+zgzUAPVDBx5hy7YsQyNgmu5uZi+cQwZ5INV9TK0u6ot5xwvTwRsRer9uSORZowXK/hZgXLIbnUNfbtNoysHVBZgvChGG/GyBlZLb8ireNApEIceYe2ZAg3f88Wfuc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released. In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in get_mem_cgroup_from_folio(). This serves as a preparatory measure for the reparenting of the LRU pages. Signed-off-by: Muchun Song --- mm/memcontrol.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4aadc1b87db3..4802ce1f49a4 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -983,14 +983,19 @@ struct mem_cgroup *get_mem_cgroup_from_current(void) */ struct mem_cgroup *get_mem_cgroup_from_folio(struct folio *folio) { - struct mem_cgroup *memcg = folio_memcg(folio); + struct mem_cgroup *memcg; if (mem_cgroup_disabled()) return NULL; + if (!folio_memcg_charged(folio)) + return root_mem_cgroup; + rcu_read_lock(); - if (!memcg || WARN_ON_ONCE(!css_tryget(&memcg->css))) - memcg = root_mem_cgroup; +retry: + memcg = folio_memcg(folio); + if (unlikely(!css_tryget(&memcg->css))) + goto retry; rcu_read_unlock(); return memcg; } From patchwork Tue Apr 15 02:45:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051365 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACA97C369B4 for ; Tue, 15 Apr 2025 02:47:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 37B042801BC; Mon, 14 Apr 2025 22:47:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3274A2800C2; Mon, 14 Apr 2025 22:47:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1A2BA2801BC; Mon, 14 Apr 2025 22:47:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id EE90E2800C2 for ; Mon, 14 Apr 2025 22:47:01 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 943B5160B14 for ; Tue, 15 Apr 2025 02:47:02 +0000 (UTC) X-FDA: 83334741084.09.DD35DF2 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf14.hostedemail.com (Postfix) with ESMTP id AB8CE100007 for ; Tue, 15 Apr 2025 02:47:00 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=OHkBBD2B; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf14.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685220; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ubou74n5eLeoB4dEz2IgWAHxjcUZtEwt6QUiSiKGpOQ=; b=qCbgla8jTB17TbQOw6aVpC9VyrZqRf/AdlOtBL6n0UVF6qBW8eYzVurzXdT7kIA/XTtzr6 W1/zmMRqAYnb6Z9Cif/ig0VZaNf2q6WDauZ8KMLUDEk6XMNnCpXG5TzXipExwRc9/jrBUN RAjSHyrwrE2JD7A3WyACnIcrfAh3Nw8= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=OHkBBD2B; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf14.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685220; a=rsa-sha256; cv=none; b=EfEaj2m1LNyqUT10WXh/NOxjjuRPSZEM4phFjSKOsNLunrOq1SqsMLVtte1ZhN8E/kdn+u VQJULNGUcNiEYuI8xlp2jwtt51enMB4C39Jggj0WFXmlgL4iCvMHPxMcHDG4fdh4WKN8HL Kx8cAXNHpUJzl1Bz7mw+0ajt8yImLSc= Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-22409077c06so65950775ad.1 for ; Mon, 14 Apr 2025 19:47:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685219; x=1745290019; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ubou74n5eLeoB4dEz2IgWAHxjcUZtEwt6QUiSiKGpOQ=; b=OHkBBD2Be+1jVspW112/sJkYEB2wt9jSjEyqRzwCqp4qTwSLcdeOAxzMbYORoh/lYi xevLTsPJTlHNKTf1M2oe6rDx7oX9kjZ7G1ddySPeHE+Ewz1Xw4l2OVHSrdA4tPzAW5V0 5u+vcf/bIzvTgtUee6UYLzsrauEYgcurOyBgmGiHRSjVsXYaHg5YtpKx0oICPJCG+jOR bNBTSvAjfUrfCYFPy+jIZ+9zHFq6PjPfM9iIWYkZZRSB2YzMQEoA2l7yMWOiruG/ncKd 7Gs2405CQL+MaHwFG4Qc2Ub86bY3uKxyLZ0wVdYwCc62NNro41xLCY6nYfwZuweRIfXZ RWBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685219; x=1745290019; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ubou74n5eLeoB4dEz2IgWAHxjcUZtEwt6QUiSiKGpOQ=; b=DqKMpLj1xYmTw5gT+JUCgc/st53ol0408vjAobVRFQ3pUgH7rgkyoxKJscF4m7MDx7 qd8iZ3qlfPIihy9LkhClrkBXuEZNH8HXR740YIg1sKTeUIEQePLzgGUDrUeyYQNwkfDn Ue8e7CLoZqTzvV3J7x1BabN3ueUQJwa3K8NgE5PwtH/jw1rvo9GKSp3K9JecW14WonyA FtI/Gg1Uw73X9xDdvVsS5VDZ9yr0sxBl/54RrWu0oNYaxzasN8JTqZkKFEBWB8/tHXan Olim+rOusp+2wGGSed3XtB8MRt3HnPDy2sLRFRocjHrxJ3MzN+gcGBAHxvmxogplP8WN Ew/w== X-Forwarded-Encrypted: i=1; AJvYcCXveeDd/lLW1iP7Pxv0HKpthvfLzeFppeVKGmL8OxcltH0B+mHmfA/rP0+rDAQFmlI0s4nykdXC/g==@kvack.org X-Gm-Message-State: AOJu0YyGqgP3W6ImK357tWpkFZk6+khKm69c5Bg79N6N0TxOe+MlwU21 IgSsSjjB9VZiSm3W/97ZH0Ojv4ldB0VP5Fo3Vzpti4zdLuUftE5NsDSUSKnWod0= X-Gm-Gg: ASbGncssCh2O/B4sQQTw5wybTMQT/AFAKdQ+RmU8UA9sSCtSSjxqj2pTUuk+r+d14gd ulWcH7c6Ab4zFR8eXO+L+7+Eyrmu2T3bMd7af/+gZLEOnHpzTtoTHAVYKLOVTmoYi6VjV3/kkzc XBf4vEI6l85g4CY/L3gTug5lq9Ed+jJuZtUbq2RfZGFgIKdoyorYX8yW3GFTxZgVblLADa3hC9H B+35oPC6l7HWxL4FCbKcm1swLihcEW0YxlFplfLVldjKrWp9CF0t8GYeOhR3hGPyhrVPAje5lkl mPcESuCCR0uuowJcY5K0pD3bx4NWH1BYB2hvO9Z61YcYCZ0N/Mt9FLxx9PXHmLJlImD3Ajyl X-Google-Smtp-Source: AGHT+IFRGP6yJdpVqApS9S/aczCeDW7cEdxeVDyn5gX3MHZ0LzmgIuZG2begu5ay7gyJ3d0hK4lbIw== X-Received: by 2002:a17:902:db0e:b0:224:23be:c569 with SMTP id d9443c01a7336-22bea4adf49mr221463895ad.22.1744685219616; Mon, 14 Apr 2025 19:46:59 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.46.54 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:46:59 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 12/28] buffer: prevent memory cgroup release in folio_alloc_buffers() Date: Tue, 15 Apr 2025 10:45:16 +0800 Message-Id: <20250415024532.26632-13-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Stat-Signature: 69megweysirp1ym479mpnscp5fmqfj3f X-Rspam-User: X-Rspamd-Queue-Id: AB8CE100007 X-HE-Tag: 1744685220-927450 X-HE-Meta: U2FsdGVkX1+AJeSmDCfxdzB2LCk9CHptKo23qSQWaKSbvNJxItzJrjHBlgQtJiFkbCMaU1VdpVe3q++2EsGIHTflV+b5froyPYEO1m0VbdSngPLX+1yJ18nOVpXKuv0oqYw7Q2WI6aOom+vNvIISTicaXHC/++s4TBro9945dS6opgWxRbgSYGpexQcfvVJ/z4SIl0c4quqeOvV7XWNGugt1Z4g8kqPo7TUblnfxMEfLCbjzifuH8TTxda5251WKJrr9FC7Cu6UetQJY1aYpPM8fHXNbAkNTC7kT/FgSvHIsxxZ9vs+g/VKakmagOBbEVOKQPY8YYmGqod7PyTVlrZgFDYZLPsOBNXGklL93+c5aa8gLUdW598G/MwOmVp8Y1xyvgeRY6ObFzRCcyshj8wTZIqQKbL000wLwVEcW1lLJT6Jpo/Lwi0eOava+RDVSpetfjkVW+ckOI4/u6ifXzo7QJT2qDoy48V32bDaDjeYBr3hbBjxBWXAL9LWRFBJaiLl7LNPmB4e1VdB4of/NLWsWXYY/xf2ScpjmUkXztUJnYbeGeyEGyvI8lA9+DmOINE7KGQMCoxRxn4oehWGruaYYJ/mBmCnkzQght/DsuG/yyogvRfjg6oOOvZG1AA0Q38Ga+/m5/kfmlNkPd8PrlIBcN8xouiZ9tSqOPnTYMxYCFkoqgT1ySjo/fmCSy4I+1Th0rYSEtFg78mQcjY47Fdho417aXQSYlYcHW2hncomXFP7jUw7W5qGgN2puvXZ8jkI2FbUg7joQU4aaNB6OorJ1nSmBg4ew58IFzfQmFwsf7VTkzBqfLtIwtXRSG8xVnqTjG0Q18Viyr5mQnBIq4KXMPYeu2Te4zLIs9yJzxG9HI4gPwk7hjl3ykFl68UJ3rkWIcYJGA+oNRyjf4r0wuO6aIxefGqXjK1PHRBBctFKKyqeFkpTAxFVljVxp2ldlg74JzP0l8J2O4qERvHb n3+u+F+p fmZqTJEY1qeGj/tKuZdpKm8ebS21VPF5DcDikBSjVyhP7+M947KTv9kEsSUiSRBkBjbr3p9cvSXs8rMKDCSRqhOdHXGTXhU8U1AMZHLQ5VsVgglX2cICyO0DjCfs888soJ9S1Xlbi1V2ROTwZzzWGOrDvUm+L7RujBijdfYieYiMJJrGP9g2/svHtMgJQf42+SoBezAVqRxfnOT7ZnafZzKUBozi7gZLAtbOpfJVZGCPTDiWXpqfjwFtKrvPGUwvFK6y0GfR7TfBs4RctFrBbcaBe4R4BI+9JJ4GBoj1jrEhxCYn3YWRLZsGcFantZdfTius12l4vZo1HoXHV3JLUq79Llrj3AGl7od3oA8oECTZ12Tc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released. In the current patch, the function get_mem_cgroup_from_folio() is employed to safeguard against the release of the memory cgroup. This serves as a preparatory measure for the reparenting of the LRU pages. Signed-off-by: Muchun Song --- fs/buffer.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index c7abb4a029dc..d8dca9bf5e38 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -914,8 +914,7 @@ struct buffer_head *folio_alloc_buffers(struct folio *folio, unsigned long size, long offset; struct mem_cgroup *memcg, *old_memcg; - /* The folio lock pins the memcg */ - memcg = folio_memcg(folio); + memcg = get_mem_cgroup_from_folio(folio); old_memcg = set_active_memcg(memcg); head = NULL; @@ -936,6 +935,7 @@ struct buffer_head *folio_alloc_buffers(struct folio *folio, unsigned long size, } out: set_active_memcg(old_memcg); + mem_cgroup_put(memcg); return head; /* * In case anything failed, we just free everything we got. From patchwork Tue Apr 15 02:45:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051366 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C62DFC369B4 for ; Tue, 15 Apr 2025 02:47:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 50C432801BD; Mon, 14 Apr 2025 22:47:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 494E72800C2; Mon, 14 Apr 2025 22:47:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2C0F42801BD; Mon, 14 Apr 2025 22:47:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 07FD82800C2 for ; Mon, 14 Apr 2025 22:47:08 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id B5F8A140B7B for ; Tue, 15 Apr 2025 02:47:08 +0000 (UTC) X-FDA: 83334741336.19.A7F3545 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf10.hostedemail.com (Postfix) with ESMTP id D8557C0007 for ; Tue, 15 Apr 2025 02:47:06 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=A5XQwmYI; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf10.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685226; a=rsa-sha256; cv=none; b=XN6Yjy0MZANtFTjIEjar1lN6Q/NMoI7srFSxDkhTF+U+/Pcu+sE1lN6BL/k4dj/UJG92m1 VKqbxJbC/pGLhrn39i4L8y8iTQjOpVjnQvvZpq0xhSXB7IKfFyMOFDYExz7rzKDIBkQ77U bNQKEqg5G1PVAmbTk2XFuqlx7r6M3EA= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=A5XQwmYI; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf10.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685226; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=obht/6Kk0NnhGpMOiil8TzYARUGhrq+7EhlNkpnMaGs=; b=CxqTIW8L+mU9IZao1XFJFJdWHv7rGthrML7JylSbXOzPf+KMrNAiD6HfYiCQ1JGYd/Zcg5 xMR56qvVryadxQwRSkJbSdLWJENG/aaVzKzuPlduE5S7N8cFeZD6P6G8lmhERLMPNu+KFg w7SB6j7U8y0i1r6n1vzLCQvLQi9TaVM= Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-227914acd20so48110905ad.1 for ; Mon, 14 Apr 2025 19:47:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685226; x=1745290026; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=obht/6Kk0NnhGpMOiil8TzYARUGhrq+7EhlNkpnMaGs=; b=A5XQwmYIVd6L1YQH5Q7TTMiw74CF73O5kNQWGlpZylrZSsxXo/e102Y6iwo2wSPCVA B45av6dYNoDsuGvEPRcE3QUBtdRcAZVjWwcUbjR26Y5iO5001GUc4VVCcqJuje4WWcwL OPrlI23802Jc/Rt8MxLefyNJF3eTCPzREa20XSkOqsAsxleh73CE9AKLUAUJH5SYMXaf V3AGWrHKkIoB8BBtimRPzGSdQXdunvvX9oQEECFwDzpicHbPckdr+yOJqFK+LEeIjJtz pgOOtSXB9ox4FzRO7en33H0YH3cyPcKjpPiXPFGFiUAaXlwoufmKf284li9oapzmVbZM mA8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685226; x=1745290026; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=obht/6Kk0NnhGpMOiil8TzYARUGhrq+7EhlNkpnMaGs=; b=iKBLDYSiqqN5ONwwvAANE2rJ74wJKnY76AGdMK+nf9Dh3NeyDKiA9AkY5Rke8Ad7u0 NvPwaAJL7TQr/a4+Pi1+/m4PAdMY3R3Y6kAddWwM1+xKJrHxMCVf2gEfmUmrnz5ruICs qxhB5XnwVHDjQwBk0IKIs6qh2j/ZhSOrlOzh9OYPfhUnhQ/VFZ7Fq8pn3vR+9rGLQ5Y+ lwAZP3ncbr2F4Wnpj/bJyJrhelnwmG74MAimsuA0h0fdZhGHy9E9o+lGzzVqoHqwPNqO t00mTvJwGbVfZAWRJmRf0UMPjZW3Yv/1Ydx8q5KXjHZoYy6bBqzyTsxKnP4uV+9HUfDk AIUg== X-Forwarded-Encrypted: i=1; AJvYcCW/JGDJ1LXHmKN7B7TOv2vWvj7n0IB6QwBmeR1BB0Zgqs1Cp8tBKuBvlMuqt64NM2CFDJUv4c3nMw==@kvack.org X-Gm-Message-State: AOJu0YwITwS6Nx/LKJpepLJBT8WIqAPHpRybn562fuRJpulFC+DckxF1 fG3GlZ/Ce6Ce7X+fh4+xVzmDem5IV9h9X4UJSOyTFtpaFpOR5WCO/oA0AmN9sqQ= X-Gm-Gg: ASbGncsxy4CCbsPI1vYsEMivn/Q7o95yoMakDZ2kYTLPagxpAD0/096k+EOwwfoukdY +jaMVi6kznV5qtoiWtbRmY1p71+b8W4HhAm7JmyipOqgcOWE++S/qqBFf/Qt1FcEgF8UykYKDyE VvG26Dfu8CJLgZrHIVwvhHA9PteFnDgHjVkpk+fL1SH1CYF0skBf3Kmsup7FVU8/U1FpCStksfc VYehFEkgqtdyl1LRl+ZQtd5cKWdlzekyk7z4wJu/xDOqPe3180nxSkZjg1ITSszaTHLsPRSzEZV M85EhmBd9MKFcZXgyj54ctad+7W0GbBF6UPX2zraspDG5opf9Ipy2zUIgM+kUKSofIitKDQ1dPa kjb5IJro= X-Google-Smtp-Source: AGHT+IF1G++5FxihLJJNREqKO+5yf6OFv8YN+5YM0gPXaE7ee12PkODsP2jTee4zjVH6VXKddsDutQ== X-Received: by 2002:a17:902:d490:b0:226:3392:3704 with SMTP id d9443c01a7336-22c24984d71mr24086475ad.12.1744685225787; Mon, 14 Apr 2025 19:47:05 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.46.59 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:47:05 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 13/28] writeback: prevent memory cgroup release in writeback module Date: Tue, 15 Apr 2025 10:45:17 +0800 Message-Id: <20250415024532.26632-14-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: D8557C0007 X-Rspamd-Server: rspam04 X-Stat-Signature: djf71ed1e94urogazaiucagf9wqknf8o X-HE-Tag: 1744685226-690658 X-HE-Meta: U2FsdGVkX18mKZsf3yHwr7WvjDmvSmcGLr/xGysJix6UXVTxbzaqbINstNWyC4BlrAdf9W7eqQ1wOgravqbauwuWPHNixFMQAxCZ6nm1RqFeFKcmjqA122q/ZqJwxhQtXH9hm5iq6jMFWdA+ok4RPsQQ/4slUTRRSNoLWRyxAtc8Hyshz4YuI7hDX8QvNzZR6OjRyAHUtOlPNLrKVsyS2xJu9UoXPIS0V9Rdr5PmjVwDITPWgHRJtn70bLf0OGORRBB3zzyUb0LZRSeTNnVPWcvNAsQpYx11dxDhY+lFpM3x7TSwz8gdvE+TpAnr5/B4nDZHvxN8x8L/GUj03rEhxBcPvJtzZ6uW5Jkw3slCPFO+9E3Adi8sZ3QJHhrm2l/7IgnHL2/0Pct2EWyBegR1nEbAgdmORKNYEZ3iOgtlzbIEhnGg9CfJbb2g0Zw/vOYbynjXv5Nb22lRehuSwawNOmMDgUfIUqoUftjjaDB+EQUMD6o801yjLT44nPL3D7MJlA5Yqi1hvtKdLXPIyB7GqFQr2PJzmnmyKBagVhg5rLrrFhw3VmrIvu2dtijuxFM27LfzpkDLVNsp5Df8+XMUaESKuX2Y6DzJsGKoNLqOo5aLTAX40VauKUjTTfYorBLGrYpsZXuWjqFgzYf104I6GPgUD1vgxlfxwsaH4hDfx8PtGkTEF+16IYjnQfL+ApQt45R1/wOETZSPbjNI/qwa3kUp9/kuqW2B1ropzpWALd0Pw/+qEiDEbOu7ky5W5oBOgHSrVV7zrz+uFJBUk+pcZSPT+ZxmpXUnVD9R8pZ+iW9lgiffmJTX4ccPvQO5V+l8FpBoL17O5AbKGRhFqlAPwl6P/2XMYzujKgiKUCoODaM8dLlbYuy3T1cJmqL02U/imKbfCLDWfLvs66amZIa/foOq9UFaatF4XF09b6vo9mU19uecXdttuvMiZLJEyV6/QEtFB6uIjG5imjyY8Nt F6UfUW8m k1fxpmO7ox66AhUuFd8evI6/KDmii7aE2ckrsCDlStxump4uE7zxyfqbhxXs339YcHVjwzkE9sd0aeixL+5w8tYPBmW56ftAOiimz93yH5r9auKm4ta+1l7nT4+U+ORT1cEJrpmw/l1aITSaaTwnPQFh1BPYd1dX54TXFbXltCyfU+ct361pr1YS+kThd+um+nF9lrb4WtoRU2eeGqdb6iSWceWCvZwFMzn6EEc2qLZYTBNiMt7JZaIo0cbZHBidcc8q5xxxku6JG8JRiVNYImBYk7jW4ln2zsMNYMTJPWrDOj+zCg3vdboKzvA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released. In the current patch, the function get_mem_cgroup_css_from_folio() and the rcu read lock are employed to safeguard against the release of the memory cgroup. This serves as a preparatory measure for the reparenting of the LRU pages. Signed-off-by: Muchun Song --- fs/fs-writeback.c | 22 +++++++++++----------- include/linux/memcontrol.h | 9 +++++++-- include/trace/events/writeback.h | 3 +++ mm/memcontrol.c | 14 ++++++++------ 4 files changed, 29 insertions(+), 19 deletions(-) diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c index cc57367fb641..e3561d486bdb 100644 --- a/fs/fs-writeback.c +++ b/fs/fs-writeback.c @@ -269,15 +269,13 @@ void __inode_attach_wb(struct inode *inode, struct folio *folio) if (inode_cgwb_enabled(inode)) { struct cgroup_subsys_state *memcg_css; - if (folio) { - memcg_css = mem_cgroup_css_from_folio(folio); - wb = wb_get_create(bdi, memcg_css, GFP_ATOMIC); - } else { - /* must pin memcg_css, see wb_get_create() */ + /* must pin memcg_css, see wb_get_create() */ + if (folio) + memcg_css = get_mem_cgroup_css_from_folio(folio); + else memcg_css = task_get_css(current, memory_cgrp_id); - wb = wb_get_create(bdi, memcg_css, GFP_ATOMIC); - css_put(memcg_css); - } + wb = wb_get_create(bdi, memcg_css, GFP_ATOMIC); + css_put(memcg_css); } if (!wb) @@ -929,16 +927,16 @@ void wbc_account_cgroup_owner(struct writeback_control *wbc, struct folio *folio if (!wbc->wb || wbc->no_cgroup_owner) return; - css = mem_cgroup_css_from_folio(folio); + css = get_mem_cgroup_css_from_folio(folio); /* dead cgroups shouldn't contribute to inode ownership arbitration */ if (!(css->flags & CSS_ONLINE)) - return; + goto out; id = css->id; if (id == wbc->wb_id) { wbc->wb_bytes += bytes; - return; + goto out; } if (id == wbc->wb_lcand_id) @@ -951,6 +949,8 @@ void wbc_account_cgroup_owner(struct writeback_control *wbc, struct folio *folio wbc->wb_tcand_bytes += bytes; else wbc->wb_tcand_bytes -= min(bytes, wbc->wb_tcand_bytes); +out: + css_put(css); } EXPORT_SYMBOL_GPL(wbc_account_cgroup_owner); diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index e74922d5755d..a9ef2087c735 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -874,7 +874,7 @@ static inline bool mm_match_cgroup(struct mm_struct *mm, return match; } -struct cgroup_subsys_state *mem_cgroup_css_from_folio(struct folio *folio); +struct cgroup_subsys_state *get_mem_cgroup_css_from_folio(struct folio *folio); ino_t page_cgroup_ino(struct page *page); static inline bool mem_cgroup_online(struct mem_cgroup *memcg) @@ -1594,9 +1594,14 @@ static inline void mem_cgroup_track_foreign_dirty(struct folio *folio, if (mem_cgroup_disabled()) return; + if (!folio_memcg_charged(folio)) + return; + + rcu_read_lock(); memcg = folio_memcg(folio); - if (unlikely(memcg && &memcg->css != wb->memcg_css)) + if (unlikely(&memcg->css != wb->memcg_css)) mem_cgroup_track_foreign_dirty_slowpath(folio, wb); + rcu_read_unlock(); } void mem_cgroup_flush_foreign(struct bdi_writeback *wb); diff --git a/include/trace/events/writeback.h b/include/trace/events/writeback.h index 0ff388131fc9..99665c79856b 100644 --- a/include/trace/events/writeback.h +++ b/include/trace/events/writeback.h @@ -266,7 +266,10 @@ TRACE_EVENT(track_foreign_dirty, __entry->ino = inode ? inode->i_ino : 0; __entry->memcg_id = wb->memcg_css->id; __entry->cgroup_ino = __trace_wb_assign_cgroup(wb); + + rcu_read_lock(); __entry->page_cgroup_ino = cgroup_ino(folio_memcg(folio)->css.cgroup); + rcu_read_unlock(); ), TP_printk("bdi %s[%llu]: ino=%lu memcg_id=%u cgroup_ino=%lu page_cgroup_ino=%lu", diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4802ce1f49a4..09ecb5cb78f2 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -229,7 +229,7 @@ DEFINE_STATIC_KEY_FALSE(memcg_bpf_enabled_key); EXPORT_SYMBOL(memcg_bpf_enabled_key); /** - * mem_cgroup_css_from_folio - css of the memcg associated with a folio + * get_mem_cgroup_css_from_folio - acquire a css of the memcg associated with a folio * @folio: folio of interest * * If memcg is bound to the default hierarchy, css of the memcg associated @@ -239,14 +239,16 @@ EXPORT_SYMBOL(memcg_bpf_enabled_key); * If memcg is bound to a traditional hierarchy, the css of root_mem_cgroup * is returned. */ -struct cgroup_subsys_state *mem_cgroup_css_from_folio(struct folio *folio) +struct cgroup_subsys_state *get_mem_cgroup_css_from_folio(struct folio *folio) { - struct mem_cgroup *memcg = folio_memcg(folio); + struct mem_cgroup *memcg; - if (!memcg || !cgroup_subsys_on_dfl(memory_cgrp_subsys)) - memcg = root_mem_cgroup; + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) + return &root_mem_cgroup->css; - return &memcg->css; + memcg = get_mem_cgroup_from_folio(folio); + + return memcg ? &memcg->css : &root_mem_cgroup->css; } /** From patchwork Tue Apr 15 02:45:18 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051367 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4088C369B4 for ; Tue, 15 Apr 2025 02:47:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3E46F2801BE; Mon, 14 Apr 2025 22:47:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3947E2800C2; Mon, 14 Apr 2025 22:47:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2356E2801BE; Mon, 14 Apr 2025 22:47:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 025E72800C2 for ; Mon, 14 Apr 2025 22:47:13 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A79E7B0F5F for ; Tue, 15 Apr 2025 02:47:14 +0000 (UTC) X-FDA: 83334741588.21.9DCAF73 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf12.hostedemail.com (Postfix) with ESMTP id EB64F40002 for ; Tue, 15 Apr 2025 02:47:12 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=XFI+5lpB; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf12.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685233; a=rsa-sha256; cv=none; b=ZlVkWtlbuCnPqX0KwpkioTKPAcq5P2RoAtGOT8DP8DR/8jy96iPN9VIjxtU1xB/4aJE7u1 f1f6QE+QApJl/puzm2lFE7ajOATUY29IyCzOCcPC7DLn6ysbCRKnFbyXmzLEXwG/4aW9BU D8SmGINicHefZxXz0G6LSgoXkSPrEDg= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=XFI+5lpB; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf12.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685233; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xg/Q9PeJ3a0fPq4jQ400Y/OgBu3oFR0wAT2gCDp4D5Q=; b=btAPv/VoJnWAf9sHjfxIi4pvH3bggA/7LQxyNJ4ejxVv90/2YNp2KS7lZvj4xKcX/nU/yJ jGYoIWTmtpiJFu33s2B8LhrEmRsczghQfVNJs0qiIbkX5/69uXOOBgJ/jQmeTz3n9TfqIe VbRVIPZdRPD/esEo/6UUddHAsz9b+AM= Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-227aaa82fafso39527955ad.2 for ; Mon, 14 Apr 2025 19:47:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685232; x=1745290032; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xg/Q9PeJ3a0fPq4jQ400Y/OgBu3oFR0wAT2gCDp4D5Q=; b=XFI+5lpB2nzG03S3uJvAJdY77YlOpsxT0iKDY11gQ+aJPfTjZFK7sf+MNCwjYwubsX wwHtlrJO3202yVi2Tek2ZjE38/6kl3dqMqDIvJ/2CqL/T043glmLPfPwITEAFDgpXSON ygpxXCyES+OCm05NEVTOkicFzAYqxdbH+XKUP8MW0l9hEm0C5/qQfGUpVQPezQaO/cPu PPddKtdQz0ab64dTCqOAqpJpBFzSALED+F188BZkjSHRtOb83TdHI1NJV73hyoL/uE4s 7tKz638EVy1jsB6LIVMsLNXTJSNHQU/PIRzZDD8c0mgM22RX5kr0y2TXickhCcYm8SSW OAlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685232; x=1745290032; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xg/Q9PeJ3a0fPq4jQ400Y/OgBu3oFR0wAT2gCDp4D5Q=; b=MYlAaYTLCNwlru3gb0cHzYZyA5/l+ecNx73EKdM4QFCImg4TiPsD0JQU/OVKTobVMr IoZN/jgU3GxMoaKqVKwWGA7cAmhNRRdKzpnxoFdAM50sJsQhrY2ZOMhB25oqXcM4hgsT Djgdh/Jx9Enwd0aek4MSQ4lb38JjIXptaQ/kg7F/3wWeaXng2P4DwpqbaojjOWmxlejq AOFfxNnMznTrRLkAV4OFQwKthjY/Z2t+xiRNlWuIhN6Hr2uszfuk80qLWDuYZnS5wxuO XND3jihySY1u01+b9IkPGFZTo1GWxOJAngDLXYf3WiIgvcer4sKOKf0jv48lks486jYB aVQg== X-Forwarded-Encrypted: i=1; AJvYcCVqTN53Uuwd6TyglEflwiTxLX2JySzJk1RhaVXelnaAdiDFlbYx0+Dgj7oiDDCP1gDOU6gNk5Atyw==@kvack.org X-Gm-Message-State: AOJu0Yxfc3MdliLJKPVBm6+qza+9NDxFL9xuwBjcCSoYn/i0t2No3z8a cz7WAao7xkNQ0trTPJzpsO938VgdcGJvr72Xt306uese9OUqt8zcrYb0tyjRJUk= X-Gm-Gg: ASbGncu0OZTYegxR6pb2K6SrbncIYUYdeYr3QqkaB4ZuerlFsGyImpoHnigaueQiUYk pK434eLi/pIPn5ehxZbH6mLJSM3cn8b5ZfzSctiGbPp3azHZIpWZyGiLP2pkrni8vVLFIQMJRwp QgfM+7ri1kjMZtC6fBtxLDSggB/E3hJGZiQ6rLZUjQigPXoIbCld1wW3RYKB89/rjAf21wxUrlv nJ/n+zIGvboFImau/auLrFG1r83mqtFaAwoezuZ6YwwgxOcy/5GbX64gdp5AEJZ38vxff9SQFdd XDWPPvxObi8B5JFVI0hZltyXSQv0asJ4tvC5gzM1hKwU8i6T6cSclvy306Kv9Psz6kyScJDcKTz avd1qFpQ= X-Google-Smtp-Source: AGHT+IEzkWcKa2ct6W+m99WRXOgiOEymvJ30CBNBOfWjsGr2/iLsdVh+Li4JvemGEyYH2iRVm2MWrA== X-Received: by 2002:a17:902:d485:b0:21f:6fb9:9299 with SMTP id d9443c01a7336-22bea4bf561mr197033565ad.27.1744685231866; Mon, 14 Apr 2025 19:47:11 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.47.06 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:47:11 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 14/28] mm: memcontrol: prevent memory cgroup release in count_memcg_folio_events() Date: Tue, 15 Apr 2025 10:45:18 +0800 Message-Id: <20250415024532.26632-15-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: EB64F40002 X-Rspamd-Server: rspam04 X-Stat-Signature: 8w77bs6z84eyodmc8mq5qcxzd5z97p1z X-HE-Tag: 1744685232-370621 X-HE-Meta: U2FsdGVkX1/I7mpOWVJFdJ/rWeONKgS46IUBoaogBBskRO+lIwANV49+hKOSpge9opDjiX0dxfZBnRC+J644F0g1KoRyTpHMQMmQvpYOfWqJPD6xlDsSFG1tj6TVKxNoSqjir0uworfSBb7gdmOJAmwXivV55VBSnLBpf4IX/+D+jHPIv3CmDg/iJTYVMtE3PlVQkK8C45Tcvq4KfW3Qo4knxKEY9UDV+uHJsbjLNBUIr7es6MoU9M+WWGBkeZZuOxcy7RizUKTa8smRsRqBO3pjTvVAcjozs8Lbsy4oDlAOCHRqPq6iDIcKbdDQk2nSNSQBrvWNWuQCyXxQmxa5+8S6z3PivOjyNq1qStrX+k4bd7KL4YlMzUjbqJ9gulaiE0JHQZh8foYbBvXcW5J8n/61o0+W5Hac9Dl40uugvNouXDnB6PvTu4vDLjmPSxM7EuaNbNwpKURpAitlAK+q6ErIxigI2IILVwVDRsD5IWJzWkH77IZqLDsLaQnNS6KUxpJ3h0XzxCvgzT7zv2sqdrqhvmZfJFckjjfsYVd6UJpeAG7AL6RS4FZTOdDK4n5MOEniIvNL0E2F5Ffn9NG8X3OU3YVg/g9oMMdXJRzTqaZUC3hJqP3F3OFj19VmLo2IoqZWv7X0qJ/T5B+RPo6a2LuOb+u9+Ay5miyqbY61jeG5jJkUuoN3dfrEiNWq4ABKsrOg+x4MUxJ3IwssWr950jY9/4giVtiwJQ2mN8BQW4JAAQrEBVXlv8QJ18mhOC6MNLHTrKqHzbwSTg4EykexcC1vpvUduJmPk+mh1OwLFqlxAqNefcnve/Hymh9ITUpncQUmrGuNkzlaFYagoSwBSe/Pg0RE9aLrc36wfTZWBkWFl5UI1rdzM0wyYhC+JY6so3QEYSy1eDOPIYWWKTTgook4ke2YqOTjjT6Fboxbr3MS/GPDuFwDeA2vYmC+zDzyw0Xxnl+u7OpdVCJj9f3 FetfFZR3 6t3sHJ5B6xynjKXmlSQ4IAZY6AgxPHVkz5icEWwMtDbqqJy5D2mPUT/cWikqR2nXdE6oaM2gOs5M5DlKcaP/WQhIa7w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released. In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in count_memcg_folio_events(). This serves as a preparatory measure for the reparenting of the LRU pages. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index a9ef2087c735..01239147eb11 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -978,10 +978,15 @@ static inline void count_memcg_events(struct mem_cgroup *memcg, static inline void count_memcg_folio_events(struct folio *folio, enum vm_event_item idx, unsigned long nr) { - struct mem_cgroup *memcg = folio_memcg(folio); + struct mem_cgroup *memcg; - if (memcg) - count_memcg_events(memcg, idx, nr); + if (!folio_memcg_charged(folio)) + return; + + rcu_read_lock(); + memcg = folio_memcg(folio); + count_memcg_events(memcg, idx, nr); + rcu_read_unlock(); } static inline void count_memcg_events_mm(struct mm_struct *mm, From patchwork Tue Apr 15 02:45:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051368 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7720AC369B4 for ; Tue, 15 Apr 2025 02:47:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EA4B92801BF; Mon, 14 Apr 2025 22:47:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E4E472800C2; Mon, 14 Apr 2025 22:47:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CCCE62801BF; Mon, 14 Apr 2025 22:47:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AA20C2800C2 for ; Mon, 14 Apr 2025 22:47:19 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 61C6C1CA391 for ; Tue, 15 Apr 2025 02:47:20 +0000 (UTC) X-FDA: 83334741840.22.4B0AB03 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf11.hostedemail.com (Postfix) with ESMTP id 9C12740009 for ; Tue, 15 Apr 2025 02:47:18 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=JqiSrCgE; spf=pass (imf11.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685238; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=048a/COuUtmsXsXzHcibcjt0GyskIDx7SCGdBM7o194=; b=5C19c6k04a0MAHVOMkmb/RxchonQKBa+9cSUveCq/3tl0DcgrX3PBKXHlPfW8FL7BlwqkK Vp89mVIp9EQYPOA8x+XpD5O8RmF8YRKvaVQIgbupTc52guidSVBnCGlcwq/3qzIapwYzRB EinS4cXfDCV4fJrfzoQG7Z8w7L5CHSQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685238; a=rsa-sha256; cv=none; b=WTt8wruke0epotXmIkHMJJItNFjCiWE4Vh88BVsAX4wxjnoSjZy0MzWA8XWfitfcCUdSTm mK3KJhXNsUimf2WzwmwromxN1JpmwgsjMsB6RkYJk1a/PNlLlxpSmn4bhyp8/hrm8+4xjd Tq0c1bceOQkq+GHPFIdd3huDt7W71j4= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=JqiSrCgE; spf=pass (imf11.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-22423adf751so45811615ad.2 for ; Mon, 14 Apr 2025 19:47:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685237; x=1745290037; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=048a/COuUtmsXsXzHcibcjt0GyskIDx7SCGdBM7o194=; b=JqiSrCgELU/yBMeAxh0q7HxbGgUc1jy7mw2QJuCIlaL8ZgWKE+qNV5XYxjq11mdoOS si6uChl2JEfzM/uqWLUPuHaVjB7qgw7k9jljChWsO3wYFHxN3cWX0SdX9kiJx+YVOzc/ VmNzh3eX7jlrIHklicNf4PB0TRHnBcE+FsTtKVJtENAURIcnOE82r+BtG+TCPGTFdEXk Z3cMfbwfb1WY+mU9awW47KZlHDHn+tQzsd2edFVxZoumXkqcabGReKWjomDVtQ9WoAQ2 02mXBA03cY0+A89BclYCJV1oBTZ1gRigM6meSsaa4RdsRRYDck93ObfgbUZm2zub+3lv cinQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685237; x=1745290037; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=048a/COuUtmsXsXzHcibcjt0GyskIDx7SCGdBM7o194=; b=gKgRyDPv3udIVG/BzLCKmsN7XPK2cLaNwge2Q39PcnMzTSvpayIS+4gD30R/5mkLpp RNxgyeag3EfRMa44Wo6MrU/m5wIgeBdmedzZu7Fl5fm3qqsH8m0mUJbvn8eZ82vzqovT ICyaxMZvT+xVwJ4WXqGAy75+IgWjIgLPfBl4Y9VMmobjH93Bw2H84D55L4kAz4AQHtFQ 3f6inO6PA4iklIChXTjMutGg2916/3ZS8I+hpb53Er4MvjytRXhiwfQHekDnLqVfxt8g LuqpDbqluuAkyespot7mABY10Svjrv8w0QubPsOP6jGTuDUAzNfGCrh1fcCjYvpatqfd /9Xw== X-Forwarded-Encrypted: i=1; AJvYcCV1ezQgkj1G64T4dLOM3P9VF/4Pvg7Zn4acgcduFfME6ZX1KizgLefo/dLwbps7VyEJtACfWeXsbQ==@kvack.org X-Gm-Message-State: AOJu0YzbXfMzq1NYQn8VTMuK5PYIF1ACyflBxDveKaLwiG8ftop2F0XT AjfA/kfSluoTIb0/CgN6WnT96xhprNVAtUS4X1qu+eJKU7u70ph7KK7sNBumBdA= X-Gm-Gg: ASbGncuwOQoS3w1sS6V8EtAqj3xOxHhGhLg2QdhTfft/GXZWT4tVUmMD/wMVY5jCKrX sB9GAKirHM2/hObDFQ+zr0L+V2/yGNVU+gSCc+zAxzZA3B0WhkoK5Rf9J8H9Gc5mTaJTeubzULa Jnc/o0jyUiQd2pLjqKwuz9hymHSZykbP8wfNeEmeXKq48bq6B2Le0ZceIZLlg8j5PuyJ7ga9sf6 6KEgikvRf3cUuUEnRJpsIjgrp4cin2RwlM751rzLFyDP94uldrUxmGQTJ899KZ0LZLTIJXADHQW VsOm5G7cOb370RISJA+NdZ9BKeFpsdxr66QHiYz9z4A46xjbzK9JU2UUKT7mDS0m9ZVNC013 X-Google-Smtp-Source: AGHT+IH8mFGGtZG1eAt8IdVT2T6wuIxSclSFtYUMLbYnFcscVwUk704iXPYP1CTuJRMCglCtOnp9BA== X-Received: by 2002:a17:902:ccc5:b0:221:7b4a:476c with SMTP id d9443c01a7336-22bea4ab854mr201964985ad.18.1744685237616; Mon, 14 Apr 2025 19:47:17 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.47.12 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:47:17 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 15/28] mm: page_io: prevent memory cgroup release in page_io module Date: Tue, 15 Apr 2025 10:45:19 +0800 Message-Id: <20250415024532.26632-16-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 9C12740009 X-Stat-Signature: pehzmg3bdz5rherbuf6krtf9ikbn4yrw X-HE-Tag: 1744685238-218485 X-HE-Meta: U2FsdGVkX1+r2oCUIlxefBWVG3DV9TZ4NrbOTpe7Mjfdq/yT1z867fgBpVmLfTpEs3oNtyO7b60f5sHz0rsOclhpSo2sF2VzomQgkhxB1Q8JI5t0MsBYiIeXjer1y/9Znbcgn3dcZUVSk8UROycHUieAAn9KAGwdeMygch9SKj90hV3/hdT5/tVqrcvrDDHQVapuEi+AqybcMOCstvTqeJ3BLkCU2kynbYrIMXR+YXK0D5VdHRs86Dub50FAWWp0v7Y+2eJhttU+e/LVccbti2WjMOmuB4s0TJznSybdpnPrYCM+qf9+XI85/i+Vql6oMHaP/25wVa0gbKjavGR2wB2gkIotgKb32JFIXTe2ym6Jy3mBK0R5v0BRmMsO4PKWIrybk1jpb+ri5KEcJhbSJTIcjUyJbL6ctbyPrHeA9uSQ3v6RdqExEW35t3IcK26yEhsAJoTi4D0fDy1GmrxTiZRiac+50H9mTvaV8hHPIAXMpbvnLXjVPn4Oz4k/3346X+F5Bnu8AKLVvkr7pX6724K3fqdA/KkRrBL2NFClPSGn/wvjcRkToRQdbLKOD+2101c4idB72/2HN4NIyxEdRyH+wBZdrIdrpYPk2fM8BZEn1+xwz8Fk37YhQ+gRBXpnnHzeN4uw78Suk6gvFOn9hH55PFMUHR3dSSOTa36y9ZzS2X5WvpP+QTUNgu+VHvoWQAPg/x785bxOlVVrE+K3J7GJsgjc4iVKUF/enUniuns39u/hl7okBozLLrXnoxxuKsVOrBywfGnhkR7u00+/GbYPs7iJPgDC90IXB07MxVyEO5DHI1GjGw6YabAQIpFq1i1uZym3kaNtIksLHMGRMGQSAtBzVTs//Cq1Z2w7qd4NxBcEjwWpfBVnLliZOVlSc2LmuT6pMVBFTdElO/X38IWWGD40bCKckj7zd14Rn34r+naUIypKpVQ8LrozMIBO/hD+uNzQJvt2BkCwvsp D7mtcrga RnJhCmyeg3Of0V1Ysl6nokuDL+I+jlj6fKF7TYlsgeaLYht2jO8xUlTFQ/aj7McQVD/5shCymZ6ZI7gBtWpzyWfKMHw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released. In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in swap_writepage() and bio_associate_blkg_from_page(). This serves as a preparatory measure for the reparenting of the LRU pages. Signed-off-by: Muchun Song --- mm/page_io.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mm/page_io.c b/mm/page_io.c index 4bce19df557b..5894e2ff97ef 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -280,10 +280,14 @@ int swap_writepage(struct page *page, struct writeback_control *wbc) folio_unlock(folio); return 0; } + + rcu_read_lock(); if (!mem_cgroup_zswap_writeback_enabled(folio_memcg(folio))) { + rcu_read_unlock(); folio_mark_dirty(folio); return AOP_WRITEPAGE_ACTIVATE; } + rcu_read_unlock(); __swap_writepage(folio, wbc); return 0; @@ -308,11 +312,11 @@ static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio) struct cgroup_subsys_state *css; struct mem_cgroup *memcg; - memcg = folio_memcg(folio); - if (!memcg) + if (!folio_memcg_charged(folio)) return; rcu_read_lock(); + memcg = folio_memcg(folio); css = cgroup_e_css(memcg->css.cgroup, &io_cgrp_subsys); bio_associate_blkg_from_css(bio, css); rcu_read_unlock(); From patchwork Tue Apr 15 02:45:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051369 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA2D3C369B4 for ; Tue, 15 Apr 2025 02:47:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6C3372801C0; Mon, 14 Apr 2025 22:47:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 64E7E2800C2; Mon, 14 Apr 2025 22:47:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 476FC2801C0; Mon, 14 Apr 2025 22:47:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 222A12800C2 for ; Mon, 14 Apr 2025 22:47:26 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id BC6081A0B67 for ; Tue, 15 Apr 2025 02:47:26 +0000 (UTC) X-FDA: 83334742092.15.6DCFAB1 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf28.hostedemail.com (Postfix) with ESMTP id DA4C5C0007 for ; Tue, 15 Apr 2025 02:47:24 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="GR/8caci"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf28.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685244; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=M5EyQKk8+6fTeRtjB7wRmwigOpyCG4thZQUtLcFoNss=; b=cLq9hjdtRCVut1A9xV80b0WC14m86Eq6Cinj2XEBdtIajZM1m8PDh+Kp/GCflrW5fGX54C cp9E6lTe1jZ97VfW591v874QIATHXcssz9IOx9Rw4Nnu/dskH1z55aY5lmeQDadCcC/L2f yYIrimU8UcJSQrklKdV9rHDCBQjdKrk= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="GR/8caci"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf28.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685244; a=rsa-sha256; cv=none; b=dX/vrirxx31bd/PYFPzPX4X1qBC5oHCbCxbhWemT+7qSqGfwDqa0/1+4CNG3fNmvlfnI3M uXw2GMz2K3H4p5DErD5qP4IeWnu7470I+J8FrurXj/VfMiHMbupIrlOrfXQTodjeFRXtBa HTqa/nKOhjCEk4WanBw+Q7LPx1IHqpo= Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-224171d6826so70701945ad.3 for ; Mon, 14 Apr 2025 19:47:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685244; x=1745290044; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=M5EyQKk8+6fTeRtjB7wRmwigOpyCG4thZQUtLcFoNss=; b=GR/8caci1XSeLZHnuAKRtCBFQFFqQSkftcGgUoyuxzpCO3qzaMcQd3kDv6o2SGkJmB iNxe6/rKftJtlPS3NxlsCmILOeLqEbMRoXf360Dh+bOXDL1NuEgOU7njmQTFsLdwRsxW +kEtTER8W2MzY2UUVcrw9dxj6fMJ1TgtTB3/2oAphFeAP1RRr4GbO+1hd207HSkNHShY WYrLa6XQrs6Rw21V2P3H+Aivb3voqWh2qI01QqA4WQ5sKOpVg8NeaCt5Ger9Y2eGYDZI IeHSo8ffk0Lebllhy2WSMfpc8p57q6zRhbVM7HuIhszhlbVal7E4yK7LFZN/5nRqz5+u TRlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685244; x=1745290044; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=M5EyQKk8+6fTeRtjB7wRmwigOpyCG4thZQUtLcFoNss=; b=nuGLLLpG1a9Ga6x7MAjHRCF2ob7OjqDGB7AKnNadcLHaXObMjSxUBk04K8aWaAMmGn TvK2huM+sJ6bJYO1XAQO6EK5k2eABZZ6n8V45u/qzFvmWV+/C2kt0nMKkgf7UCm1Bfqc uOkVBP63XH6UJpi9g9mf6nCq2V+Br9AMc7zaWFsOsb1gW1XFxqSgfD8Z/pWFx3cEi6gP bz7Guu/iFnoKNHpjDElHDfssGGrDxUVzp3iJfKnZ742HwLLJG3GFbgXpaMYEbke+agmg aT6At4GJ06q9qRNgu65wg7bGe9+FhVoQ4Y1Es8DJG6qRqClHwSEUwCHvFjILYnYk9L5z JzMg== X-Forwarded-Encrypted: i=1; AJvYcCUTgsq9Jxhc+5WpaTuhjK1ZqYC0fjawkv1zPmflfFpPkeavKumvdz1eZT3aXlWg4FPKAp97hACmmw==@kvack.org X-Gm-Message-State: AOJu0YxYLyfRxlEs/cdGMOJ6SvK2jWUnZ3T/n7zby3DvWUIrnSa8+kZ5 dWd97uPTUbjPKzYPpdTMvDy71wcDnnBYTjRzLqBTUJC5EttclkifPm2V14ynrKo= X-Gm-Gg: ASbGncs+REUpfnYkie9dS+dnyYuVt8IReKHGQzS81kfa6l2sdNyegerpaTQj8vE1V9b TChAgxVpGfBLK5DWyMpOA3tp2veZyPVeWCrJqX7hEDjZk3Ls8Cv6ABVzKHjMIqDgbChk2bI3uO6 dEvn6rDE9JWsuuJsprWKULpOiaKVVzwwOvSbhqvtKw0irVMo2BJlOMOoW1qHGpqsgAotuDkjjOU Dx1hCljPWZVqJHmHoDhdIDEqdjSUGTTUXK/d5i+ktqa7otNyIyrGAHeFUkPepQKusbHYk00uyPr NcVh9RMNqOU8MhhAg4uixMeQsQBy0agm44pTr0Qf+fhg9Pu4yM8eBOIV43k6kYDTAO6HT1X/ X-Google-Smtp-Source: AGHT+IHMPzkkwvrvPUMjLd8RH358czmXgq5CB+tWd2lPWjwDGWGWGLk2G1vtiG4B5iGYuVvrrbfcCQ== X-Received: by 2002:a17:902:f70f:b0:224:5a8:ba29 with SMTP id d9443c01a7336-22bea50ba50mr238795615ad.43.1744685243640; Mon, 14 Apr 2025 19:47:23 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.47.18 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:47:23 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 16/28] mm: migrate: prevent memory cgroup release in folio_migrate_mapping() Date: Tue, 15 Apr 2025 10:45:20 +0800 Message-Id: <20250415024532.26632-17-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: DA4C5C0007 X-Stat-Signature: dpqax6h3wdx4hnsnr8iepfn4hwqdzz5s X-Rspam-User: X-HE-Tag: 1744685244-840651 X-HE-Meta: U2FsdGVkX1/rr5u1NvfIIkH214qDXjeY6NR5yBXcEXnnsiRwJdBunimUVtpJDUWMBQcGjN6g9S6P6Z20QoFk/pskvIjmks0/DMgHFunFGJZviDS7g7f32C6ZzDkALLfdP6JqN9ZbQSd4vULAfH6WIfHabbbw0IkhG7vlw885C+uTQ/Ydq/5s4uQCqp9L6li8xhwm65o8IVlCeOL3IkOmU/PW7TfXYuwYApY7NJIXe+MP/A4ykaFkeiKPUUHBEAEDGCV7LnFKeBuiFTHvXpJFjHr5K7eWYhrr093fi/R8mg1EXf4keE1+c19QztlOhKnsxUrz2xaVd+EqkBpbbcO02jcqO8qcUihfnMpJpv8Sof2xrwYf7XX684c6jde1ytiPEaJK1iyAi53ruzQhOdyFlPC8pGEK2R7EhXWgF8IK7Ufc9pJTdSa+hTAQSdNqlmvPRf6gRJT/lMuRcf0h6LNQKzQ+haQr+aTKmxGSZv6KQtwp3EQcnC3Q0//HZhrsEdaZsqd/suKrSZu8+Fpg/TxUNV0nE61GBX+b4zt5qRBAmjBwc86jmrVscPrTrkn3jUMp3gxLp8gwBO9fxglkDLbEHdZ9o2dknoZP0borpvoMEIVvx2lAGl+tFrhwrxgslzOrycMhOEfTW/uXU1tTanIYGiXK35OcZ7ygqW7nKcBxKBt4cwwmCNvNc4Xix49HuQEHT+UUEvtNmRBRomK5szakCBD1iInThDet/tH2Z/MOcFP6T6SjkQn0ISdQyQSQLVtLxvxl1oGRalHsWoWTA8LKnvaH8cGJCNUx6Dl5AikPozZxEGlwoSDeQa/ind7m/Nodv2IfGy8sY3S4VEFUGEVnwmIYA3b/GssfrJ0DM3vfb8b84aTT7wd4LXP68cKC2FGlFzQxtELlMHOGgk5j1v29alKPwbNOth1ICjhuHz75Wk3OGnc+Vl7YdXT//jIwd5LuMCnhsQFlG0p6+KHL44B jayzXypC jndP5PPdgAmYoXYAkL8PCVqY1iQNZI/UMXyqOm3SfdpW1JIgfh4TWVz7Nw//QHlbvku8LnRpUKLeSGnlVcUeZzqu90qwrYREc/84o3Ks9VeV/EIErdZj8ZACwitS9xMLGVLwcXrOMBLgeSu8IY2K3mGDlgEnyu6hmGnQ0H9xUYw8Vs520I+qWKJlVczCJ/WDnFI7s71547S3h0t0P2vl5p8itSuMcTuR0X77uYhojz2igUvrmQz0kW3ZEH2fDBJqgXvLY14hHUVDMiqvbvN165OyzMwWl5wsB81O8GoISqRef2mh9O70FM2I0GrRD8jV/u0gaZUyHETXuJKeqMm5uw+bDOEXPYBUx6jxziLtXsXShmXQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000770, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released. In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in folio_migrate_mapping(). This serves as a preparatory measure for the reparenting of the LRU pages. Signed-off-by: Muchun Song --- mm/migrate.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/migrate.c b/mm/migrate.c index f3ee6d8d5e2e..2ff1eaf39a9e 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -565,6 +565,7 @@ static int __folio_migrate_mapping(struct address_space *mapping, struct lruvec *old_lruvec, *new_lruvec; struct mem_cgroup *memcg; + rcu_read_lock(); memcg = folio_memcg(folio); old_lruvec = mem_cgroup_lruvec(memcg, oldzone->zone_pgdat); new_lruvec = mem_cgroup_lruvec(memcg, newzone->zone_pgdat); @@ -592,6 +593,7 @@ static int __folio_migrate_mapping(struct address_space *mapping, __mod_lruvec_state(new_lruvec, NR_FILE_DIRTY, nr); __mod_zone_page_state(newzone, NR_ZONE_WRITE_PENDING, nr); } + rcu_read_unlock(); } local_irq_enable(); From patchwork Tue Apr 15 02:45:21 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051371 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A341C369B4 for ; Tue, 15 Apr 2025 02:47:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7BD142801C1; Mon, 14 Apr 2025 22:47:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 769072800C2; Mon, 14 Apr 2025 22:47:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E4FE2801C1; Mon, 14 Apr 2025 22:47:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 3C6602800C2 for ; Mon, 14 Apr 2025 22:47:31 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id ECE26B0F59 for ; Tue, 15 Apr 2025 02:47:31 +0000 (UTC) X-FDA: 83334742302.12.79B88BD Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf29.hostedemail.com (Postfix) with ESMTP id 1D34512000E for ; Tue, 15 Apr 2025 02:47:29 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=gHJkNipC; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf29.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685250; a=rsa-sha256; cv=none; b=wgoqDCmSLAmNSqOulR8UoAw3brSeCWFeVhFxac5MnG46vcjHtMfd7dOgPs2OOljQxiSkG1 iVGkv8Iol6mlx8VkpAXJdg/U1dxo29B9kimaA0s9PBQrVF+wMsxVLMrP/+rwvIxacUMCjv ocQyMZjaFVywdqDVZs6iKYTWbMf67qo= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=gHJkNipC; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf29.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685250; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=P4yZw6tSBeNw48uMiPAV08H8+5zhu9HJtv5YMARqzOY=; b=KSdAZ+/wy1HdoMMYtaqWfazIs+6zX6K8I7ZhEMQ4ImnUQfLmolFX2esLQIRvKuK0XM6wc3 WWNmO9y1HqBZjc57aslDrTb786XNJN6/Ow7b1BElr8Ld7lvm1KkpHcjpY1+v3ffIQsGgav bkN59pZuGYHxuaqrDMJBBuOLELfrIPI= Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-22403cbb47fso53265335ad.0 for ; Mon, 14 Apr 2025 19:47:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685249; x=1745290049; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=P4yZw6tSBeNw48uMiPAV08H8+5zhu9HJtv5YMARqzOY=; b=gHJkNipCy0VV44eIVWLZUkvb2xuF/XWx3mmzjuHvT2giowXFussJWhLxaUbbfBhj90 PiLrHzHPxkQ8kqn5YRXhpJkOKPxXx6rfVF0rQ24MHp2c0XfArG+sMTKOGDDTAOite6r+ VUWnjvgsDmNlouRVyUIHt/XDtFcpwY0lOWH/ywqoZXDlD1H6ulea9GjTaufuVOfCJp+7 +pZkk8Ee92xFMdXJ0xjMTqygu27fSeMPaUi4ob+VxBb3fbT8nhuCP55bLunt2xiB/nrt /rAjH07Us5nc96P5EPF0B3gCD5ZQaBJH8VeiKhvotOmFHV1KYypOECFXwBYO1MkQE1JV rUrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685249; x=1745290049; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=P4yZw6tSBeNw48uMiPAV08H8+5zhu9HJtv5YMARqzOY=; b=w2kc+wuKrIKLcxZxuHuKGrkMKn1nmOhwqVG1U0Jct4leg0FxZCTTQgeWQfdMWdm3zZ LYmWuPrkwRHQ1sVQLM+rAiVJnX0xZXJ8CVf8uNGTDaBfNuEzRAqiT4WWZwAXuuHLbqXY YkD1y32nFYvkUfOO9UmN5uElitvL7pfFC1JubIXxjoTjPfNuDZRJHobQKkeegyox9GLu HiWBS3WAYNx2epXPkMgyGMUHDVgyfNGUYvF2TgpbIPJD1H1dWK79Wluea23YfO0dhxD4 jKHERkAIt2chTJnAFGGhIyGDDAtVxPkM2l3kYE1CsC+iBGa5xIZjxCpFZ+YU5IRQu5Yo xR3A== X-Forwarded-Encrypted: i=1; AJvYcCUM3vvJ+PW//uFK20oVWGRzMArNV3S7yweQVJelEc6cJixR1yqdHVxehjfnr7VFjD6h7sHZ+qzv8w==@kvack.org X-Gm-Message-State: AOJu0YwIW3KKv3FlyNBBpBPU3O1N+r6s1Hene6JQpNiiqQIS5KBFdIAb wD4ltnNm1o1/mV3EcN9qazC2dSeWnRa8xrUMF/379SX14+zdxZxKYLlPvZ79hgo= X-Gm-Gg: ASbGncsa3JOxA+8fH5CCMBFrE1mR13xZkaB4/2GnkYrfyp/ysaDJYarYFF5oIvx8Bkp PRVi0g8pF371yNcOvqGf4wdfGJFtxQiVBRFNY8Razv1VUYyC7TtvyxojCkekBYnxnfWAtc17fiQ F5VPZU74nng2S6yTeDxOYAXVR+IYHPrh6Sg4BwOSNMD8YRW/6NEteVvieRnZI9Oq7BoeeiPkJk2 l0WwJj2IfyDv5yyyfqj4iDlH8G4XRkihD/pBZWcGXzfS7cuBZziLI4X4UiL5dPQdNCPP3csQB2s nPOJ6wWJMipd0TZepOlbNCj1P+5duMjABoh08RfIHdT2W85DpSccbWn+CinyZnJiiutAqJFm X-Google-Smtp-Source: AGHT+IHc1pLuWcfx52pBjvHAkcC6la0P8uERYY4ckvTOHEzauARR1urHfVeCryhgzFD7fMiSaMQDhw== X-Received: by 2002:a17:903:2f86:b0:223:47b4:aaf8 with SMTP id d9443c01a7336-22bea50e1a5mr207262015ad.52.1744685249046; Mon, 14 Apr 2025 19:47:29 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.47.24 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:47:28 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 17/28] mm: mglru: prevent memory cgroup release in mglru Date: Tue, 15 Apr 2025 10:45:21 +0800 Message-Id: <20250415024532.26632-18-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 1D34512000E X-Stat-Signature: 3p55cshwt1zi9ddq5a19f4kzkgsfa3ca X-HE-Tag: 1744685249-963015 X-HE-Meta: U2FsdGVkX19KUUNe2l+iFk1s6lkoBJJaag0YkOr4HyqfR0lxJIU6Wa5O42BHNmCsqpH1QL+iegIMV15A0D60F6xlF2NwxC651a60FNSqkrac13wqR5fLq7nAf4pltiBPVFoDWz8+6Mk9IJD+SgSu9DoVPdCpzJedUtqtZ+9wJ6ekOfxGMSWfJHETYcmae8npTP9/dygbBfE2xeN4BUa7INgdJiD8xwq06AV9EW4jtJQTzM4HUj65cEyDe33f9TJ1bfdRqFAlQpqWZOlWKbqJNzRwxgAprEGk5thIYpI9zQa+1e8r8jKoKNJ+uPCdsW7l9rdDzAQOClM2r8dbPF4qin4huEVixWkZkVXDi3pxxn8mq5V/3qjOVd69GZkv+PL3LzXTYsYqshWu8lAn7U0fwhVs0r1CbcmMCYiXcmnOA5hfRYT7K3WhJ8CatFBJ4uF7wjsEPg1irMDtLHYHZWFbxoNEONhXGBioCPkiTSScw0y9Q66pBrfuTpBbxKJm871F6Qx+wHiLJKpjSiSGuolx/fzjBJJBP8OHkb7Exn18E/f+TLV5sh0zcxXxmBXVhIz8sQroVFalO4vVic+LzqG872rn1bjZs0QRwv9aqz1UJADtXrTrYwIac16muFanTPg+Qp+uN4h1MDukU5CuR06vrOXpDc0kCEkfDTP8dYIRXTCov/4IKoNT800qiU0YfASNAyjs4hbbqLomN477FDZED977+1BM4Z+OS+X+u1KHOgduLLtS/EcA4mG9hAaL8lGOp0jMKjJL3pVuHoGNVmlKyAMVKOj/Ev88V4qzMldKTrJW+fIWy/CRxB0XYBxMjTdtRUlIWVznXyRH1f/e792JZOhJWZEl28LVFcHg5b3oYecKz7uEm7T1Kdt7hhMdZsPCIUDYKz4aK8zO7/J9fQc/845EOrLkauJiwGhDFeEfDbTu8HMkfyjMd+XNzIi/V6V6fDfC/33gYkLY6aD50Y1 wTItfJKB ednc0KdIEtRwIH+sTyH5HhqUwuD6fjp7htHlEv4wsDTm8MFSeurolnfW8V++yO0/D0Nz2xKT2CznkjI+Yk37K+Wm2/BO9NH+WubV+OLkgUG5kFoAUSMsEcOHH7gTyMfGM0qq6ZUxSK43p+vWjea+wSx6zzV2UP4e8/4w6RAJPGrq+tDQek0Ossliloy/CLAmFFBMvf4E4L2XFp1BizOne7LDLn6oFBNbQuZio/BOup8vv2Yz2noty/b4yqajVDPBZYQSEbNoCrgqxibsHGTKiKM63XDZkJ0yUd2YCTv0z0W7TLm08tB8Z1tqOqQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released. In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in mglru. This serves as a preparatory measure for the reparenting of the LRU pages. Signed-off-by: Muchun Song --- mm/vmscan.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index eac5e6e70660..fbba14094c6d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3451,8 +3451,10 @@ static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg, if (folio_nid(folio) != pgdat->node_id) return NULL; + rcu_read_lock(); if (folio_memcg(folio) != memcg) - return NULL; + folio = NULL; + rcu_read_unlock(); return folio; } @@ -4194,10 +4196,10 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) unsigned long addr = pvmw->address; struct vm_area_struct *vma = pvmw->vma; struct folio *folio = pfn_folio(pvmw->pfn); - struct mem_cgroup *memcg = folio_memcg(folio); + struct mem_cgroup *memcg; struct pglist_data *pgdat = folio_pgdat(folio); - struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); - struct lru_gen_mm_state *mm_state = get_mm_state(lruvec); + struct lruvec *lruvec; + struct lru_gen_mm_state *mm_state; DEFINE_MAX_SEQ(lruvec); int gen = lru_gen_from_seq(max_seq); @@ -4234,6 +4236,11 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) } } + rcu_read_lock(); + memcg = folio_memcg(folio); + lruvec = mem_cgroup_lruvec(memcg, pgdat); + mm_state = get_mm_state(lruvec); + arch_enter_lazy_mmu_mode(); pte -= (addr - start) / PAGE_SIZE; @@ -4270,6 +4277,8 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) arch_leave_lazy_mmu_mode(); + rcu_read_unlock(); + /* feedback from rmap walkers to page table walkers */ if (mm_state && suitable_to_scan(i, young)) update_bloom_filter(mm_state, max_seq, pvmw->pmd); From patchwork Tue Apr 15 02:45:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051372 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 708B9C369B4 for ; Tue, 15 Apr 2025 02:47:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E7FC42801C3; Mon, 14 Apr 2025 22:47:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E075E2800C2; Mon, 14 Apr 2025 22:47:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA5D72801C3; Mon, 14 Apr 2025 22:47:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id AA8942800C2 for ; Mon, 14 Apr 2025 22:47:36 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 64DF958B63 for ; Tue, 15 Apr 2025 02:47:37 +0000 (UTC) X-FDA: 83334742554.04.4FABA28 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf25.hostedemail.com (Postfix) with ESMTP id 7A763A0002 for ; Tue, 15 Apr 2025 02:47:35 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=QmFvfgtB; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf25.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685255; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IXD1JHisUM2V0YMP9gwUIK79l/C4R22oOynHqMcLpgg=; b=rXsokhjU6Z8QjKDIAuN4qtDnv4tEexZYHilQRyM5KDzU6zyQ2lvUCsK+ITNy0mQLA/+OA7 CsnqytAHjHE9YpvNcnaCzZ9xGo2BBz+qTke2ayqwSPVwnhNg9CqFwVq/H/rKejrjcWrBRv 7z/KMzWBhnVSEQa7BcJ3yDaWTu4qwJ0= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=QmFvfgtB; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf25.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685255; a=rsa-sha256; cv=none; b=DaDTDkrm7Y9o1pirrtjbQLFk6ne/LuyXtu6AbtIqwfYcPpGD5iuxmu8Gct+fe00dUMcxTe /SYvvDnfkvhvtJCxQtfOePK/pDH/GRvVLTU0oy6HRDMKSrc+RQhQw14PbFVGsvPm2LlV0S ro2TQ/ZDWpQxTgOau6hAQ34YDY6To4g= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-223f4c06e9fso46672455ad.1 for ; Mon, 14 Apr 2025 19:47:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685254; x=1745290054; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IXD1JHisUM2V0YMP9gwUIK79l/C4R22oOynHqMcLpgg=; b=QmFvfgtB96Sov8GmMi120VWaMAVz2jLwPC4afDRbN+XkDjLbEksuzOx/eqAcwIK7PF 0mzW5xaM4UkUtycr+cNFcTGhVZw/ocgjceuMivPfJg7al4zcPM/6nbqE39LQj/kAU3Ah GKVPAtdAGSCrvN3uNI7VX3MpsWSnOFbbMvKgqGmx6SBCvrySTFgllRRLwtCwlPYi6NKY RR+nByT62ULw9hz0sBe1f1QpxWFitHBl72mvFf7y/9gYTRMNz/kUlWbY4wP/YRLNCbpy VSC1qSi+lNGUaADeIAH/27VVuMaqb92IZrkMHFZhYVo8mXXCP1EQQVggFMShOiOhEkoH li4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685254; x=1745290054; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IXD1JHisUM2V0YMP9gwUIK79l/C4R22oOynHqMcLpgg=; b=q+IheuL7tn7NV+aHkYQXM8WPmS807s3qD5OOF/jVk0zIC1zZEm+24WvnlleGAwVN/+ 38uKo3yEjXjS+DWImPgsqStp3pU9gIbBHDqJsdt4OfkvAuG5KT1Ujtp0658eFUb7u5tk dySd8aZ49nB41ef83GOeHSj6Ju1S9bRaEjqVFmcqdVS/qP8pHRNeP0pdMme1Xw0JgXly /X0Rrq81y/6xZMWU6LCAgZIO2XeSI/Y1c6pkgZI63BiOLOjax1+KKOEWP4h7eZbIOp7M nIbrHhHFk/W3z/jie3vUipMMZ4aOY5TpKOCIqujMtFsM1aQI3FNbOxSQX8dbfTWMd3+U lk5A== X-Forwarded-Encrypted: i=1; AJvYcCVMd1QeGC/hxPOkRPTVFfvk/NOZAFqLqhtZSD292szNk5EQbTO90IGsdW15eDQL/3RxkOg/4QiMyA==@kvack.org X-Gm-Message-State: AOJu0Yz7Z8nQ1Z2fXtjGwKZF0W0kwkTgkhJfh3DR+86wlP2+/XCfLnrA 1fuYqaYUOHyy8Mt+lfMCjGNRVqI8GCN3piNLlcuiV8KJOUf+NUiSs943blFrNUQ= X-Gm-Gg: ASbGnctbu2fp6Inj29r3ROqgKjEbufhpxjFRTKRC4ddL3dyTPIySMVnOXzz9ORn40OX Dp6F7zRGctF1o6v2SOUgMsZ7U/lNPIUqJX9mw6ImwTlI1P7P0Df03Gb6Nf/clL6RTmkJ0dJV7zD N/fMRyWcWTp4sIIzhtYBACdu/lAADwno0udIrQa97EJAfMI2d1mcl2deJvxXWTLQiXHoeGyU96j ePfdaGLKKZE3ELq2OXotpzoXAKBRkYYo3dh/GWfT0hrGPwJQVmAayccj7Ljvdy0Gx7NR8ynvSsq nUbA46pfJSRvz54XPREuRgTNVCFU0Xpe/8GiEJu/mRzd6LC3dSAFLjI34snZCgeB4D5XyVyY X-Google-Smtp-Source: AGHT+IGov3Ukjqy6KHI2Ta9mNCWkvrqJ5tO5c4YlNBGkmYhVlyky9mFum65/WmO/B3bYIoz2x2Rf2g== X-Received: by 2002:a17:902:c942:b0:215:b1e3:c051 with SMTP id d9443c01a7336-22c24984d66mr19690505ad.11.1744685254245; Mon, 14 Apr 2025 19:47:34 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.47.29 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:47:33 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 18/28] mm: memcontrol: prevent memory cgroup release in mem_cgroup_swap_full() Date: Tue, 15 Apr 2025 10:45:22 +0800 Message-Id: <20250415024532.26632-19-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Stat-Signature: 4mrkc7z4awrkhagnjqk6mfhj6mwxx13w X-Rspam-User: X-Rspamd-Queue-Id: 7A763A0002 X-HE-Tag: 1744685255-339708 X-HE-Meta: U2FsdGVkX1/DQhtq0lHk8wJfTqIioEgikB09aulwLu3D36wIWDpLgnszhTUSq/qOBdBfhsnRy/adJ3ZgJuJ07EC7zffKWP5j1wkeqR+JGtVmU2M+1K2WC0smFWJUDnAXRGvlCCG2YYhjsH9UTWJUhU+W07Mhiptc4EL9mTFEWT9TM4qqJCBrfhj/rrgDKfjDonCyVfgiAtn3myZmf2KAGZVaYHmi97Gc4ASEkMz4vMkg3wqp4woZnsUl8Z/lNgp/940InoNxTBjPZ41H53GXOFpQIfxevqwWNigRiBMe1HmXo9snI4WxVx0ntfW2O7Kuoen5WYYoymAWWLeCPhNH8X+R1ZRFcBrbTvw57C3eP8ZwC+O3l/6kEGL3ghf5zpKff/4iYIaj0EsMweC9XGHqU5PeyHPMtPkpcg+ttFBIBCoLmapQ3RxkiZITdBPW3Sst/RQ+ndY1Sh2N0wDUOPLnqwY8IMoAvVRFWa14qZ60g33y5iVLc4KTS+b+g5/+tYjX/hkuTtRIbC7EuC0VmXiY2hGPm1hj0ibtJxDwoEa6BcwmI2cYR+HlHYZtGn6RY5ebK9zXuB7ygJc8hDS0Z2nornJWYOLQbF3K3tj+5TNsnFtLXhZI53YE4j8OWuaGZH/IW80YxF74kyiUU2bIf0EmLZ0xV9aju+nKSbYbXSKzLq1sRGw1AmYnXIIka6xgFmB7WJxW1JrA2Vb6AleNRCnFHlk/QCILJaPgHIzfH2xlMtnodagmn8ep0sViWV7Iax64ygzyXDIJjhVGQ1w1RAV/GklKTQGtQ7NmKqD3w/1Tl0aHPewF1w8wqmH/mRDPPuKkjUNll4//4CthlpuMl8sqNmXo+fmaLd7sUTtuRArpCVG/eDTTaQGgUDt/Tka1HfI1Q2dIYfCDg3yLPsQpPPpxcRNQI9ok8mhzGXbZAQwtGvQGHHWHDVYUcRJFuQ5y31Fp4gRJxj8ebb0T67aNMUe bF8Od8l+ w77Ff/oXo3pS+ZCW9ui8qJt1w73id1NoordhmQJKgrDcx1YyweN9F7CXDr27FFUsqpfakHFFRDfBtK/KhlyAi2dFtuV/H5cNFAWOLh1ACndVuV7ljsvQ4h9uXr/OC4+tBnW3XaxMroPN8RratY50RdSltuIRA/LRofJlLVoUNH2wqXOYN7NrZn9HUB4Q9ma+93Loaerxd7yhqgVIfSqw3olrVCVP1EX2XlnzhlgfU329wON0Ogy7qsp6sgDoex+HqCSyksVytm3fCJCVgAccYJ+ElA0qkGLP3sGZbq9LB9yt6f3qoDGZw0pQrRtRuBy47boHkM1hd8EaQ1wJd6RX3nsas0fbyCmbygHBEATTmhP+zbcQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released. In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in mem_cgroup_swap_full(). This serves as a preparatory measure for the reparenting of the LRU pages. Signed-off-by: Muchun Song --- mm/memcontrol.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 09ecb5cb78f2..694f19017699 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5097,17 +5097,21 @@ bool mem_cgroup_swap_full(struct folio *folio) if (do_memsw_account()) return false; - memcg = folio_memcg(folio); - if (!memcg) + if (!folio_memcg_charged(folio)) return false; + rcu_read_lock(); + memcg = folio_memcg(folio); for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) { unsigned long usage = page_counter_read(&memcg->swap); if (usage * 2 >= READ_ONCE(memcg->swap.high) || - usage * 2 >= READ_ONCE(memcg->swap.max)) + usage * 2 >= READ_ONCE(memcg->swap.max)) { + rcu_read_unlock(); return true; + } } + rcu_read_unlock(); return false; } From patchwork Tue Apr 15 02:45:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051373 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A43FC369B4 for ; Tue, 15 Apr 2025 02:47:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 033C22801C6; Mon, 14 Apr 2025 22:47:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F239A2800C2; Mon, 14 Apr 2025 22:47:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E142A2801C6; Mon, 14 Apr 2025 22:47:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C17CA2800C2 for ; Mon, 14 Apr 2025 22:47:41 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 83506120B40 for ; Tue, 15 Apr 2025 02:47:42 +0000 (UTC) X-FDA: 83334742764.17.6D3E236 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf28.hostedemail.com (Postfix) with ESMTP id A1140C0003 for ; Tue, 15 Apr 2025 02:47:40 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=jfXUCRvA; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf28.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685260; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=il+WlAu7ECEYeHNnJ6w5CmqDQz43sJSf6crApj5k5Jc=; b=RBnIqnT+pILXa4ZEVHjkmZFfrLlqcVIBXzGLd5GcdvfbfT+Si4Mti+W8+80/wS6dehP9dy uP+kERMlCJ2mngkS3SC4McEKmHBomduhqSwdqY//GXRkykVDTm4ZS0CPUXHkuCyVo0aMOs /0lazdUut9uPkio/blC1dv/cE8mfa0o= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685260; a=rsa-sha256; cv=none; b=5UEfIGDC8vvTQlVayyH9W6V2R1IE64WpOtdfTld6BG//z5ViDRXPsSUXAXXSXYNKnKwTwG TBqzcnGtsKJIX/YlvnSVYh/DQZ5QqaNbqMjo34X8rRaZPhgB7ctVfwiduPNpCk3LYB9pj0 hWf0CT5UAmpEO+B86tkjQYJ4OTq7QPQ= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=jfXUCRvA; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf28.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-224019ad9edso65029965ad.1 for ; Mon, 14 Apr 2025 19:47:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685259; x=1745290059; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=il+WlAu7ECEYeHNnJ6w5CmqDQz43sJSf6crApj5k5Jc=; b=jfXUCRvAPkZgkdOq0BIc+CKeIPUE7roThzbavpiLdwyVP8qdgDn1WbfECyC3YUQWxS pGpruVos5E71MDJVH+b0O2uKAIu1/wqiWADfgCsaghutpShJOOZpJjj0546DQhsJhnDE E+qevkpAhpNICbQQghxYApJUeZH4RVSwSt07u/V3jA2agL2pcJWfEVmd5waGpltb1xyZ Tp2ndj7wT5anEUmKazYlwld00bm7S6IDFofJakV9eOeD65zlNgNG3FYj2qImwoZ/DTW0 00SOYDz/GOwXG8LFSTmH+PO+QRiFDZzKYwRQqazPlTMjBSrUWTJsZJYowSjgViyBHOVS fHPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685259; x=1745290059; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=il+WlAu7ECEYeHNnJ6w5CmqDQz43sJSf6crApj5k5Jc=; b=PmywVHkEPyn3wEA62Bp0l7E/hB/NHnWw5FCWmXel/Vlxi3ttPNFXL9WjGg59zydzr5 me3oOg1UlRU1MvnzN4FaliAU1neCWN29vNLUS3FZrYGKReVZZrjPP5vJt1XN2q3Cr+8h eWbc1YrCriznjVpExXbOhdEtrKLVWYZSaVyn7yMUBCEWXnprs1jZK7FP+ikiTDgXhUMl IJAkTM/isN3ykCEsar0KJUiXN9EovJRtTRFOaBcZ0v017TCH7dmU/I4JstbGu39gwvZ2 kAh+VjEwIQzYla8QNDdNS/kogoVoFWh+B7xRY7uvX/6QB9pndITGPrsUIvokwZa/Ioxh HZoQ== X-Forwarded-Encrypted: i=1; AJvYcCWzj8lJeXDAvsB+YphEiHPlWsTt8BsBQNE06Aa6+6I13wNB/hsUcPyp91ubZmF46gRt/RAryzBZwA==@kvack.org X-Gm-Message-State: AOJu0YyywSWybvbFJSiQdwiiqwnyJPK31bcb2Dz1EAFRfEcpFfCJ+jUR OWvQFZKjjE+Pmkvc8V58sJIbMNPhi8sH4ghFwRaHNh2GHNjo7CudwUK1Ya/RTbo= X-Gm-Gg: ASbGncu+eONehyAIFkQqoFBmtt29TY6IyUs9T83XORFSz9fmPV+HTaYDNRDBGyBL5+K P5zDKjlFsXoLUGk/kc1rFcCNpDAmU0v/rMIejRbKsjjKsVMU2pIBqkfOJ7I2qpxNbbM9gFlnH85 IbVxwRI/sLs/V9Li5+BeOMlwvP5z9KqWT26hHV51B0Ipxak3bGckKgFJKprlzd1QewsVAyhurlB ZCe1PlmG+zKhNHPXVc0jSf2WSe0XpXuePnejVJnLQg8fArNX+HihfHvtk4hsUSXq9lcE8lsLPYH v185oE9/XwRyq5FmnH4xiK9ukMEAei4P/pxrj0k/Tl6fuv8/xSYmFVaqDqYEF7ol4annef/6 X-Google-Smtp-Source: AGHT+IELTjYsxK3e4qqX4T5H1yTy2Tq0r1FbaCUPQTdjd2lEOKiB73ooWS5EpMEZdecB9fJL4bZ8Lg== X-Received: by 2002:a17:902:d591:b0:224:c76:5e57 with SMTP id d9443c01a7336-22bea4ef9e9mr221732775ad.39.1744685259610; Mon, 14 Apr 2025 19:47:39 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.47.34 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:47:39 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 19/28] mm: workingset: prevent memory cgroup release in lru_gen_eviction() Date: Tue, 15 Apr 2025 10:45:23 +0800 Message-Id: <20250415024532.26632-20-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: A1140C0003 X-Stat-Signature: 6d1tmm9aqe3zpqm1r9mno3ft87dceaxj X-Rspam-User: X-HE-Tag: 1744685260-968999 X-HE-Meta: U2FsdGVkX1+IxhwQpx9bC6gWAncL2M7isL0+b3FxJGpEIRaXpqDZ40UMmz7uu8dKNlqveEkKw/oSAAPXLnwRe2KQ3ZeazMMHcIJ3KD5gkmXGo0RDuAMqhEGMGoZ1zYbBMotj8Mnbwn6N+4WjuahQQFZdxhgCdBTBfuK6UB08PgkmpRAC9pYRhSRNWlszZ4YjqmmPKtQGGF6FogiKp0Z4ed8fbDcojFNKZJsnNf4Zdks8tGFTH+dyhbTc/jqJ58j6Ti2VdoOv3QgM+a4Iiw7WeqW8fnkTdaLGKjz2MXHYThb42Cr9CJ1fR6sY/9SK+YqrMAh2nwnzyImOqT1++l9SxssWfi+RGSE+dTiWQzguRsPqb4uujEB+h4pL2/poZXL+9p1vMr43U4eEcV/Mt4lZzo0/AzuqJkOWD3JEbfltq9P+7Y3ypWWIvlVGXcWS3iyoeCwq2oM12bvr8Hosi2hNWkNSO7WDG3JOZSethGrfLaAJCiqey/avTx90h2R6LX7Ig2HQrnS6qvDr92CSJ4M+DBo2zBxByruNviK4FK1mFl+Xbm1Do2kt19laLjBuFsPzfMRSbj5GecSviPmN2Y52Ixhgbz9InEa3n3oSVVsFf5fkWzT4Hg2AQLHLwVbs04of8BYU4TqlFkZ1VtrLUl2f2vdWVdbf7z+xAloHMHN4jm317HSE9Bafm1i+yoyVFoxJy3gZ7intSzzpNSF9lMZrcoStd4lyMYN4emqjm+LwpwPPBo6dYrraQntv6MYlhAskQbGqc7JX/SUcJ0yJpslL1aCAhwRpfIMR5HGxrtaMKPy3WlOVv9CzFcTQmyFjY1q6/c2K/Wzm2Ab42UXvXZl6gR3KAbOh6mU+wA4GJOia8spa4qrwQ2cn5EdIis5Pi52HGy1aWzd+/AiM00QnfdXfb43b4DSn7bcoUaUKuAGS+WUsytcpIqhdId9LcGpqypQ8jFRHuYjimgIT1VpPalI A6xbYRxf g6wpXszpIJAARreBlU6zYYdE3sRxrE1y4MsGLhBIL4o5P4Y7o3sQDFSp2A7Cb1wgL/N1qfAb7rSwqF5laVV4FparnZPrt01s8RaquiTqblCJrF2B2TvcRNbnIIkoOHAOjp5j0QiCZFygWCbJvJn4M/3Y2BGbYQaplG85RqlzrLyeFLuiyT8AImuAeiOfSzVaRSJXVA0JgMYQ5p6Yak1Wkw3u05iSRr5Mhdw8c4QbEBWtIUlB+ItPSUbwiCRVO4m1Kn5tVYowiiOGo5Sva+u6FovKAUjFbMN4AFp46YkajjM++7rNzoe1koeFO0FwrkWFrknTFmbWWAB941FN1MggnXr7Ybx3Y20qabhGDeSl6a5k+bjY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released. In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in lru_gen_eviction(). This serves as a preparatory measure for the reparenting of the LRU pages. Signed-off-by: Muchun Song --- mm/workingset.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/mm/workingset.c b/mm/workingset.c index ebafc0eaafba..e14b9e33f161 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -241,11 +241,14 @@ static void *lru_gen_eviction(struct folio *folio) int refs = folio_lru_refs(folio); bool workingset = folio_test_workingset(folio); int tier = lru_tier_from_refs(refs, workingset); - struct mem_cgroup *memcg = folio_memcg(folio); + struct mem_cgroup *memcg; struct pglist_data *pgdat = folio_pgdat(folio); + unsigned short memcg_id; BUILD_BUG_ON(LRU_GEN_WIDTH + LRU_REFS_WIDTH > BITS_PER_LONG - EVICTION_SHIFT); + rcu_read_lock(); + memcg = folio_memcg(folio); lruvec = mem_cgroup_lruvec(memcg, pgdat); lrugen = &lruvec->lrugen; min_seq = READ_ONCE(lrugen->min_seq[type]); @@ -253,8 +256,10 @@ static void *lru_gen_eviction(struct folio *folio) hist = lru_hist_from_seq(min_seq); atomic_long_add(delta, &lrugen->evicted[hist][type][tier]); + memcg_id = mem_cgroup_id(memcg); + rcu_read_unlock(); - return pack_shadow(mem_cgroup_id(memcg), pgdat, token, workingset); + return pack_shadow(memcg_id, pgdat, token, workingset); } /* From patchwork Tue Apr 15 02:45:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051374 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB0F5C369B4 for ; Tue, 15 Apr 2025 02:47:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 636072801C7; Mon, 14 Apr 2025 22:47:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 59700280126; Mon, 14 Apr 2025 22:47:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C0582801C7; Mon, 14 Apr 2025 22:47:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1743D280126 for ; Mon, 14 Apr 2025 22:47:47 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C1148160B07 for ; Tue, 15 Apr 2025 02:47:47 +0000 (UTC) X-FDA: 83334742974.04.F8ED3A2 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf18.hostedemail.com (Postfix) with ESMTP id D30781C0007 for ; Tue, 15 Apr 2025 02:47:45 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=h7Lnfwbu; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf18.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685265; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=63b5+F5fFo9uNyjbfST0FZ55aigVErZF90xX0QFU4a4=; b=RkoIL0a3kcbKFR+Ej5mww0MATRS3YjD8zb3tLcidfNyCpfUg72/9yNz3WtrQEFIZcord5m 3VMS45hsISUZekAkiFt+F7GvLdPpCBUWuwuigUyqvvnQ2Gj0fwol0O4sUzHx4pRfFKhz2H 1Exn6dsuEo66dvFM7CNOufzKvIz1vNo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685265; a=rsa-sha256; cv=none; b=HCXKFQ90Z5WyjCqPzNsEWBvnr7QaveMUP4pa/1l0OllcJClHkFwsk0lrrxroLN+0qdJnte 8PrD1W7z+r50M15D1Dw6KC8PrRImzPZCbuA1jn86ufyWQM7pO9ue7hqYb12Uo00MGgJJ0r Kz4Gg00BewxVQkoB9TK0HBXsvEiED/Y= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=h7Lnfwbu; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf18.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-223fb0f619dso53489145ad.1 for ; Mon, 14 Apr 2025 19:47:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685265; x=1745290065; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=63b5+F5fFo9uNyjbfST0FZ55aigVErZF90xX0QFU4a4=; b=h7LnfwbuyIKwAPNHJ2vBO3+px6savGKQlCn+ZNnhsnG9exO5pEOn8UvVTi4ddc3sNC WBgxJwTBYnYIXLfUgFu40jJQ98mP1fID/EqimfsZZK1PM/y3lUyFPzYqiao35be5wc79 Gptc8TLnfa89C5nCwwx8tQGxzoGIvVy2ekZPap4AIQb8fiE15a1Bi+lJoPh/xUN876cy A+6ttipzFJ8CHhR8za7H0XzJJd/KvRThiE7nZ5uCETXLy6Ihqu8gnADM2qdfpNL/RAPi wXXYMnGJIYrG03M9t1PgErTNG+5q+ImC5M+Gd0TdLRHA8u1sbeEh/tcA6VUXUYuARcja yH9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685265; x=1745290065; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=63b5+F5fFo9uNyjbfST0FZ55aigVErZF90xX0QFU4a4=; b=Ro2TNUUtd6IwbhSTTt+29PPeSNGL5ax8TIT2JxoyqL+VagsvvPE7x7m0M/6SQSjvUD tM2d9br7D9EznoNR5x/Z9+JRiTqyCP4/DK0166a+GKQnghL2rGNPfMkTk+roi5UQyn2g eDyxJufAQhLANT5FtLEuHh+li8/DWtOgB2rRoO/vF+HiWiqPydyZXv52Silk9+djsbEa Aqe+AZaS/h9J+wQH/m8dKc/u3hKYNszggcj6QK3ICgl+vMpu+OehUiTykkFDIw8qvYdg deYCn+Nuqt8IH/DpfwiHJmqFSakWP27Wn2o4nqUmWbVIaXgc0TokL8iZ/secdjmWDM50 R/8g== X-Forwarded-Encrypted: i=1; AJvYcCU26RRlEHjnI/tZ80ATZZ+pXu6RHafGX7vG7D1kVjn2KVkfPfHA9SYgRi/5M/xO39MNUnbhV2fLsA==@kvack.org X-Gm-Message-State: AOJu0YyoulvtNXVD4fNkRpokDfBQAxgxRFSnUHUvR+4aN6VC+ycbyi+T 9fgjQ/Kxo1lH9ubWKMOw4PBUYrJOaA57FBwlY/tm3jZEM4kWHuEoHwvmx3oPYUo= X-Gm-Gg: ASbGncvtpnIH3jldG5fp2JjTYnju4crBdUMzHnth+42DZveNq1wWu3Vo9+RDgHseh05 t5TqKb1ZX2zZ2KAHGK1H+hlJQQkTHseQ6u7oA9DIy2tZ6wL4sgldp0TdA7V5gcRJnrFNGMyydZi +JHncLI0m48T3aMyqf9zn2VeL5ObQH1swgcPc9eIvFXmplfWCw5K+XrqpelX4dE5g+plT4JBgXh Ufe1I6H5aVbdFl37vwfa+SInyh4jzZLqhu1Thx2sRw/8nypbocujWQE2JV6WqOka6fleS1xToai UN+LpNsP/uApuat2CzqF8kVCPcbAt4tsIGkQ/S0ZewADY5bZNq89gA+gSMMRgmi+J1Sq/VBG X-Google-Smtp-Source: AGHT+IFmIkyy7FOoYDJmjbvJNymfdfM84C+MAEagetSFHxG3fte/Kh1b2EKbddjsXQhUKdg0bAYVCg== X-Received: by 2002:a17:902:da91:b0:220:f7bb:842 with SMTP id d9443c01a7336-22bea4efafamr182908255ad.42.1744685264764; Mon, 14 Apr 2025 19:47:44 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.47.39 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:47:44 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 20/28] mm: workingset: prevent lruvec release in workingset_refault() Date: Tue, 15 Apr 2025 10:45:24 +0800 Message-Id: <20250415024532.26632-21-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D30781C0007 X-Stat-Signature: 8jr3kgqqd3nm5ktubgz6kg6gy1rg8kzt X-Rspam-User: X-HE-Tag: 1744685265-606559 X-HE-Meta: U2FsdGVkX180lqh1Ne/XprvIFopXt1Zr+Rgu4nFHLVWpPTNO7NMRKRpFKuIb8mtlzuPvBvpxF6kS7FRaMRTIerNok4B/6b7Bv23MtCmwYOoSlD+CbtWC3iF8ExrWqkDsU+8GHRhRP6L5C2asBKeDh+K3cGE9SuLDiP//hLceggkpRiJxXEE5o1cBPxxQgWOneySLobNsgeIPZoUy49TcQPpJZGnL4ANxdCXAI6Vw9dLPmClIgo+kWPIykZ11XXkLJ41FHEmbRJanFOBRQy8JZ9PaxOV17TQ6wYLI2b/UJHZ7h88molxKBIGQIlj/aF5sOTXPqW9Ptb6zCUggb7e0wtWPSiD4QTuRlMN4ANGgxTjHrpU109lLOdxyclgfuKAobEk3+lUKTt2XtXjBJvmwxLAo/Hx9BDIjm/xINvICKnoW+YdflBYWKa1UfO3Jrix1JSXhA3LHa6IKsUtdRlZRruNeFGYN6h/c7wI6V+Nd+EMjoyU/Gp+NO9iYAHFdzjYnQW73MD43DcsWUwLUxlKRSuGyQaIHuaVJvzr4qlXNhhaqpVAtO8jMnWUa7dhtQLCV1GGjH+EUPpIGvFnAQx8P8uZfFt/Eed907jG1AYPz6Cm+w8KKuAQw2pL8jyWYfcT+RiPIRJ65QnVZCEDn6dCC9aIllc29n3OvbZO5/IaLXeh7e+zQ4Y28yDJ+I5vspGe192i2YGu/15zQyxILGKi/e5XdHjEznl72usIJ+RRXOrxfeKwhFNqSwnMTjoqSYqmJ0l6Sq9UiZguwEfCP9C0A9xyB3nu7CWnh/bdY04abMr8z8CDZAuM+xiu9Fjo15H8mDvIQyjJG1wFcCGV/JYVCFx1FVNar8EqQSNtkssw09nHAR1ZKZ43/+WAKvSyjvhK1FybrFtOXmBZgiDwUhrKXkcRrL5oSogaWAVSgshV05cjpGhv3UAJjQK0fwI2iCW6hyRVztzfajt+btlkvkVw HV6DDTnR wrRJgfLs0teeRBUrk5e0WGXpjceNNHvr10vUyqkSQfwqtEpmcp0Ip3n/W8yqVaJR4+To9oex+PD5E8DSyMEQeRm+60iFYIiNYArRjmpsfqCRjhyksP17aB+fpJVGPF5QoZKL877D86UxZCCpFJC34wpdPRMBKjDFqKy5tKj6X859blOi+0Yn4zGsfeXZZxFrPNNWraTthlVo/sY7ZlYVwF43CvbT1wFA1+WWuh+MfuVdduuRJ38/KymtNG5Iu6meTNDuJMIl7bfxH7Ktq4bYOPH56lEz5F4Gc74Ny7mJIQOy/SelN07Mzp0NWFX/NoOs1R2k2+Sdz27QR2TE9YK+XqYdWQ0mLSe+sPECQAkTSU3N94Jw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the near future, a folio will no longer pin its corresponding memory cgroup. So an lruvec returned by folio_lruvec() could be released without the rcu read lock or a reference to its memory cgroup. In the current patch, the rcu read lock is employed to safeguard against the release of the lruvec in workingset_refault(). This serves as a preparatory measure for the reparenting of the LRU pages. Signed-off-by: Muchun Song --- mm/workingset.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/workingset.c b/mm/workingset.c index e14b9e33f161..ef89d18cb8cf 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -560,11 +560,12 @@ void workingset_refault(struct folio *folio, void *shadow) * locked to guarantee folio_memcg() stability throughout. */ nr = folio_nr_pages(folio); + rcu_read_lock(); lruvec = folio_lruvec(folio); mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); if (!workingset_test_recent(shadow, file, &workingset, true)) - return; + goto out; folio_set_active(folio); workingset_age_nonresident(lruvec, nr); @@ -580,6 +581,8 @@ void workingset_refault(struct folio *folio, void *shadow) lru_note_cost_refault(folio); mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + file, nr); } +out: + rcu_read_unlock(); } /** From patchwork Tue Apr 15 02:45:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051375 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EAC47C369B4 for ; Tue, 15 Apr 2025 02:47:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 79F292801C8; Mon, 14 Apr 2025 22:47:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7025C280126; Mon, 14 Apr 2025 22:47:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 57CF42801C8; Mon, 14 Apr 2025 22:47:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 317C9280126 for ; Mon, 14 Apr 2025 22:47:52 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id DE6B1C0B94 for ; Tue, 15 Apr 2025 02:47:52 +0000 (UTC) X-FDA: 83334743184.12.DEF2BB0 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf13.hostedemail.com (Postfix) with ESMTP id 06C7220009 for ; Tue, 15 Apr 2025 02:47:50 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=KExeryQ5; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf13.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685271; a=rsa-sha256; cv=none; b=YGRDvJBKSuMeR04LyyjviunXk3YAAWBl4cWrKfUFO5oc9wXTx7T36nbR5ppQwf3X8Gb5BF osbQyreLoHTiH40wR+kiwuGTOOszBzWaXwOU3R93PGAq/dY2kKCYqfiaSBXcw51QeMXdDA yKGLPzYdjRRXtF6hBJ2V2A9wonPB47Y= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=KExeryQ5; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf13.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685271; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Dhcsyw5OvFqI3eRSntntJlPhmFn+ppWjISpYyf0tCPE=; b=TraKhMmR+Npcc5WmKKIszk8TSULOl62oqRUaiLnDUQWsrZZk2MJsIidvPBmiduerT+Exab o++VdJdcGvj60KT7GaynN+sVXM9xnYuQYryxre4FX2qhkVCD53B4bmtjyeU2qP/Cq4mUly P7qT2yQD5V9s22oQL/DOoE2Wxai22LM= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-22423adf751so45814315ad.2 for ; Mon, 14 Apr 2025 19:47:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685270; x=1745290070; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Dhcsyw5OvFqI3eRSntntJlPhmFn+ppWjISpYyf0tCPE=; b=KExeryQ5BJdMtv+EdtUh/pWJDp6Ee13+3n6TrysbTHcScPwE3DE92QpyVt7SC1Dn9f yrs6DImoyF0NEOm2NU90C+PDRqNoYbGrkUsgmO9UcyHKjpEP/7yNZR9nmNHpyRuvItDB WF20klttRJ4Ev+cmRIHwaR0uwJHqHBu4aZHyXqLngobZOnkCUM4D5f3yQTXCA0D2qPQM Th0NdqfGlJSS3YjHpgE1Afi7PmedjKuxQ928EGvi4LQyBPhzHkvO++WUf7SmrFxBVbMB JahK2msZWg44gUtc1ShxfnRjBrpoOwgFWSsphMjBG9K0QKwjIeQUk5MW3wF7770Rqdxs h2mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685270; x=1745290070; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Dhcsyw5OvFqI3eRSntntJlPhmFn+ppWjISpYyf0tCPE=; b=DFLnl0Rfxi/Kn47R+rQ/E9NuTCZEVTwAjUIYtRJuUkXAx3QottEAtsouZ+XLRiMJhm yWE86X0heTppCXmAPc5K85mWYaVKpK9p+HFRTnktEzSOGJQd6oY+FQE2sCwygaBJ4Fvg McQzySFIqzNjIj9/esQLmx2PX3gGEonNbjQfu5EX63ypNdz7fn/8i6fO5Rd/MrijVXfu gzZVqgjxZTmATHXBFVRLaUq+7q1b+oVFTYY14BLVzCOiRzCx4ODIB5IrNdAp9ABIxFba U6Hk2GXb4BWNnfWXOAEuWZeZZNhkpdFjuGIiLOE+ShlRjEZO8XjjMxh6Z3P7/exWNYk0 aBEQ== X-Forwarded-Encrypted: i=1; AJvYcCUyy4FmWxt9djhUGlWPp13/ZDYn2886peIeDEmd/KEYADdmGonJFOE1G+6iIVnBT8v4VxZ5x18jGQ==@kvack.org X-Gm-Message-State: AOJu0YzdsEUZBq1jeafnARjYVUjvFibsHg4cPEQbXQsBSweYY5aAsuFN AwlzN3SogyTxbzIkYHfHx3U5XEzqOMW0GAi5172QTYRMJ2PolnL5uDrPujVisJI= X-Gm-Gg: ASbGnctwXLdZVT5Sqjjd0tUXM+saHIFSwvbcmxLOqL/aq3gYklWcLwESJPZ5iVuwTUL hp0QYAp2a0qtDQEgVgZZgF09lNyCmy2iNmIUCCsJIfEtWjfpNxh+F6xrqr2QQ3wMeknescUgjPU iBcfnmFFI7K2L4an0kdJAdgdLmtqGEFb8c8fdlfW/tsqCiPtD35ZGUTSXi+BKRgDFz85op27sfV v/ztU9mW47xwBQcELhM4B9v59i48J2oQSGhbN6s4ewKkcAPWUn1W178o+/D9N3IcPonwaDrRaZI +FdLszozB0TWM3LPyrvIPM5I9GHZXa+1h3vy5XQV0UXKf+JOE8BZrF3QXfVjN9fcvgSjSKyX X-Google-Smtp-Source: AGHT+IHQB5bi/wRWV63xYCSrz429tnAZcPPlisuUNzOS3Dpdg6ZYVtpDC/C2wxVGDE/gRRdOBX11FA== X-Received: by 2002:a17:902:ef4c:b0:224:76f:9e4a with SMTP id d9443c01a7336-22bea4ab7e3mr209530695ad.14.1744685269959; Mon, 14 Apr 2025 19:47:49 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.47.45 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:47:49 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 21/28] mm: zswap: prevent lruvec release in zswap_folio_swapin() Date: Tue, 15 Apr 2025 10:45:25 +0800 Message-Id: <20250415024532.26632-22-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 06C7220009 X-Rspamd-Server: rspam04 X-Stat-Signature: mh34frn35pppmfihq873498hs6yj9m1u X-HE-Tag: 1744685270-42003 X-HE-Meta: U2FsdGVkX18YDh5iqBHH3J/Wb3VgeRiqhOPuM28pm0+UY/qdmFcguctTEfCoBaFaloWFJ/evLX/01dxc65Oaipo6e5bilkm2+vhPWVduSEqvKnfl/Nky4sK4KE3shBQF6zoFtmpVkd6staVEOELWY38/AYvi5Ktd4VCCAqc3CLRsK1pjPLLDPVP1+sETv79JBIZ9/47kuQNDMOJqEuoPQoCFykq7lxMvAy9CLAG67hFLbsotKF1PEn+t/Fn7w7Njmq7JIUaTnrLAnchnVq16uGI/u98HLJDG5JimTAcq1Ro3Y7IXjMfZu96S8zTr2aDrQme1cR5+NkqgYdCTq6FU7Wf+lLVa39LMUzq09k8fkMfdQGIMRDNdbT0jV1V9kny2IBrJWxd4fD7DKU0pygkOJelsffOSnyljn+aH2ux9tBZvKmPuZoMkQFGK0CDSPdAmAIJ5OMNyuY10iTEosXsuHQ9m/BuS6qVGVr8ub81T650u1Ctp/y9NiUMVKIg1SjvFJQQmT/igkVkhiqULXyBdx8Fvdqze2QCWFWaCJsGYw5pvMvu0Zcmc13FgfjvN4CO3O5JTbdWf0d3dLZ7oGlulNM6bxb+litkkviNJt6Bc90Ss3vuv/6FThJJjwqoHnmH2CpxjtWVBFYG8HB7zyCa1f+DLGasRko3RdEUwEAq7igBAWZ/8xRRIWlIao1O84Wu78lGOOW/vC9EfP5ElAZ1yxPMBlsRC2rJKk9M8TQS+OdVjxS7b2EZ/1zYZ4TVRpHgfhPmaDp5wSgLIuwhOQr1lpUUtbYAlGlILUaz7zRBQq7ReaH/L7pXsl66CO6t2/7gsD9wNk8EKXsO2SZwDMOtU3VgkXHVSAJTPLd3kY4Xt7xHSarZgeLLJ3D0IQDam6WHssEG/VYKKCBtPe6M4qSRwEWDEYoarwj7MsFTaqp7AeYJDWd92M8dtNnzBxVeYWgsBAPf6r6+q0w5Fy21uwno DQDUSacj /m+lHVqh+jT+W7haQ8rFr7i2Fxa7ykxdP/n8RFuu+7eUNSeruapsMZvc9Y0HVW//IGeHSpHKaOMSHpOLTqNW5+guDWwloUo+evkqwP3sa2pgV9ZMTW7itBITGIoRMGc3A2wkn4qkW5YiDUaRqEvcB1DkmjwjFGlvKV2GxUJY4/addQNc6zpNM4jWU9xBFtwzwpYx/ZAzKH8EUvWUeOx635c+jyh14DyeczDiilyg/goJ02qisRHNTKfCV+zOKi/teHdBWhA9aYi41X0Q6uN2OI2D20wVQGOkhSJsrTCvFmr6PhNrsu49R09SOgGXJh54r0n+ddrQMKogqpQVqyKP5ACGK7aTTEFJlg6cNhbFZDexaegY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the near future, a folio will no longer pin its corresponding memory cgroup. So an lruvec returned by folio_lruvec() could be released without the rcu read lock or a reference to its memory cgroup. In the current patch, the rcu read lock is employed to safeguard against the release of the lruvec in zswap_folio_swapin(). This serves as a preparatory measure for the reparenting of the LRU pages. Signed-off-by: Muchun Song Acked-by: Nhat Pham Reviewed-by: Chengming Zhou --- mm/zswap.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/zswap.c b/mm/zswap.c index 204fb59da33c..4a41c2371f3d 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -752,8 +752,10 @@ void zswap_folio_swapin(struct folio *folio) struct lruvec *lruvec; if (folio) { + rcu_read_lock(); lruvec = folio_lruvec(folio); atomic_long_inc(&lruvec->zswap_lruvec_state.nr_disk_swapins); + rcu_read_unlock(); } } From patchwork Tue Apr 15 02:45:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051376 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15FB8C369B2 for ; Tue, 15 Apr 2025 02:47:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8BFC3280021; Mon, 14 Apr 2025 22:47:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 86C9B280020; Mon, 14 Apr 2025 22:47:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6C162280021; Mon, 14 Apr 2025 22:47:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 490C8280020 for ; Mon, 14 Apr 2025 22:47:57 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 12617BAFB9 for ; Tue, 15 Apr 2025 02:47:58 +0000 (UTC) X-FDA: 83334743436.19.CFF4D14 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf15.hostedemail.com (Postfix) with ESMTP id 3556AA0006 for ; Tue, 15 Apr 2025 02:47:56 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="A/cxQguc"; spf=pass (imf15.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685276; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lBa3KcBA5wxUNKsQ+K1hpla3o8QIUcc8xJR0inNtMu0=; b=U39UoQPHF5oRX8APWrTZBGpOISRRdSs0hEP8YJmIFTrkV3vv6vgamV/XmjZ4uQhM7uZp/c j0BZbPoiXTQs4EveO5W7BNeda5ZKn9pwmHDM1Abw8Ttaq3YQwFH/AoB/mTp77/l6VE6lqq 4d+ZRKoS1MSgidmF2fvsYBNLC2T4eZ4= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="A/cxQguc"; spf=pass (imf15.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685276; a=rsa-sha256; cv=none; b=uwuykOcE2B4p7fNn1PveBJU87y+ylhbnimz7plVutZVKkCaFWeLG96HyNvsBEkWFjQQp+M 9ROu45s2yHgcAMjc0drD58IS2USlrU57bQV8csMWdKe9ERYwksNpMVWKYgy2xfVEKQVtjA 9affLV6rt5qv06jO5K5q5YAj/0PSaQs= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-22548a28d0cso72999365ad.3 for ; Mon, 14 Apr 2025 19:47:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685275; x=1745290075; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lBa3KcBA5wxUNKsQ+K1hpla3o8QIUcc8xJR0inNtMu0=; b=A/cxQgucbb1fffVEBH+TO5kmI/qfe5bM/jRQnNEILcPUHVWUMq7KhVadQzPF6R42aG Jvmbag0QwmVEYJEN3N3yHvfItWd9EYyrgjjW/jXjRfLZgs3YXbxuWnDyF8dM4jjY88Tt HDOb2+f1rPahOdXnoyg79QG2eYFwTBEQI9FyatM6n3+loiI8eiesgJlHtPa85aZSgTQ4 zH4mFbDqPGp3bkdm+EMpj8U7Qjc3fEDxEVBw9PpRPnoCLcDf7QXKV9PQyIbj9WnOqCAC U4a1u1taoJfiL9bby7kI2xlMV+tEzE8s4v5GzaMaX9YVaU/xGBZvTVGaCdDjqgjClp7g TjXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685275; x=1745290075; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lBa3KcBA5wxUNKsQ+K1hpla3o8QIUcc8xJR0inNtMu0=; b=MAtbTKQnGuF21vCieEqJRC6VDx7wMkbLTO9VIeD8kNIe6lSp5Kp/p/nciClC3CHZRz nKf7cFxSpAO3AcxwI9E8mLDOx6SQar8/oflZ3EkS6sKK80rRZFc9VPhlBYGZ2VNHFkKA ISSAeCSxht9+HO0+y2PWF+SdFmAPSYSdB1Pg0iftmNzGdPmqnceUZPB+WvH2mqT/+CYA 9hFmdtkoS3UD4SsQTT6z4fFC0mLXd6fiusJNRafSb3a1ogN3Y6R97YG6LvrQcdA0R5Ww WOa0W14P+UqFipM8aOBvdpsBAOeLVtytNeUEPQPo7N0pLN883uVUxovgH5n7BayyDiR3 C1gA== X-Forwarded-Encrypted: i=1; AJvYcCXkgMDHY5CmCwUyFoSPxZvtgqvbhQoEFXoL7W3sgEFuTJ3BkhF8QMyniJ4QkEj3CM6Zs/sTPlaCqg==@kvack.org X-Gm-Message-State: AOJu0Yy4C5exXec43idP+Lvu8X6I/e9dfs8JkAegvjtNq0pSKCd9B5YW A2sQrRKFBieiItNAnKmDhnp4eOPCsc0qYGD1w3Tqd+cPkBgqRhVq7HxWx4hDdSo= X-Gm-Gg: ASbGncstE3rAxLtFTGyffLyBT6/Che4aDmjGsTmb/phq6OHM6WTXyjKhTx3ZTLJOvLT Gs3QnJKItzl/VOQ1kIBuHjczoz0HSXzszWwmCluhFtE3nHu3Q2yDn5B2ze8gqV12j6YbRYFRJjH dnYnNckQWXtLGPO8sUMgtU7JwRgpnci6aSrbp0Wjwn4IvzKknS6UNy3hUTrYRoX8iG0RqN77ykn +M9OCqSonItdEp/LzAzcoV6s1y+tmR4y8QaftV1GhzZKX50aWBKn2ab3cEkVkIVXWRFLK2vSn3+ IX9pPSXnu6fBRDnNyEjaWYsa7i5J+z/kEQ6bEcgMEIEZFV98M/3PGvqO+YFBjbnnV/1Qh2jb X-Google-Smtp-Source: AGHT+IHI4hqLb4XlutpOFIzQEdrRaemLqaUdJqfYPhQI+o4uvusfdbjBRgP5q4orxhE3PLyGhMO1fw== X-Received: by 2002:a17:903:3202:b0:224:1c41:a4bc with SMTP id d9443c01a7336-22bea4ab6c4mr253471365ad.12.1744685275144; Mon, 14 Apr 2025 19:47:55 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.47.50 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:47:54 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 22/28] mm: swap: prevent lruvec release in swap module Date: Tue, 15 Apr 2025 10:45:26 +0800 Message-Id: <20250415024532.26632-23-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 3556AA0006 X-Stat-Signature: nk5jrra3n43ojw85i3jmd3kwn7geunuf X-Rspam-User: X-Rspamd-Server: rspam12 X-HE-Tag: 1744685275-644915 X-HE-Meta: U2FsdGVkX1+UaqV9NMaDEE8+p976Y+q3FO6WB6B3GH37WhsjR02rH1of7zef9dd5a8BgYNE7jldZ7Xxly+2Xq2wYW4qSFG+V3Sg832SUGGXc2y5t2P5EB9H9TVBfnpXXqUCj2mSu1U7Elpm7iHzO2Qb9X4UzZ8QVUdM/MExdFNCgS0+XZFUioyxJuzP1mUpdnBiqNsXNpPRMm6X2Y8jzlu5DJa2Z6fAclj5vhfbRZa65FQE4iFlDflGesZYuZURb4tXtCvl8/yqA0Q+rmWcu9waHKmL4NVfadDxma/1bHnPqq9ITli8JHw9TzUDD4zN8pStpTHL/7CLBFR0A1rB9AEKCA+6vvPXrU3x6EWkIh5M6Qk8cA9Dmi804WjclQJT/9oEPw9H58T2CP9WYm5gOPZwuRcLiQFuP3mjOml6qkrtVDsQdsrmWNHsPAzzbfoxOTOXf2stBRxJvIJuvT/iY/4PNQc45upLVI29G0K1ttrvn2W8rgO8gF7SKyZW6M2x/9ouGNJfQMrHIq6Dhglezu0DN88KnZMOMWZRP19n1P0cNC4DmgbxbbxYS/lpt5WfnzdlJDuU2666+B6yiPKzKNechCD0Vh631pcyqbecpp8n5ZhvpPeV2uSvLOBhsJevoFJ4ch8WXMhjXHrBxFA/wcTRF3iYv376XhjXSZpfyASZJm7lgkHQs3EztgofInEq3QexA2/VH4aOIlvo/K9WnhYzD+I9OrxJRGsCnw9FPJP03Drz3yqnISwXM+s2Ub+5RDuzK1utsQgcPggU4VvWllibjr6Ukp7gXiQGpJpnRfymbCjMDCtb3SXR7C2slbbp+C9JdfH3aU1J6OghfT+yGeN1JGvvSfgruC0lKqffJcxhKDU7Q77zUm3F/QrpybE0GEfgvKjpQwAILyAI4wegFcytqbfax2pWTfxld/K6RoHve6qnPWGze3TL7J2DHXRXOMmllwGHcTeCOASPEy5C HXXzRRNO jwGr/uBLuJ/WG86mD2xJSGumQ+RtsY9UFnyJwqNMxRkyteJL6IqZZ3dI+zlPQhLls2wS+I6QYi20jVVyNgSLXh51Q8hYOX3r+l9xR/VFExnHfwNpXM679WBSvepovuUuydFnoGoYovwcPcc//XI/Jc5A+uYiwYntkYSB7G1tkCQlsI7Xvtrhth5+qVnyMOPrvg8r+89Z/Gg6N5wpXNmwN0e+3YIRg7QNjE/6+nUMeINGPZ1JNFdKO0V9vxZbZiaObhz72/nlgct7fXOCgbmDTeAjoLAIM718bfOrv2AurOBOdEK5Pd+N8ODPgHg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the near future, a folio will no longer pin its corresponding memory cgroup. So an lruvec returned by folio_lruvec() could be released without the rcu read lock or a reference to its memory cgroup. In the current patch, the rcu read lock is employed to safeguard against the release of the lruvec in lru_note_cost_refault() and lru_activate(). This serves as a preparatory measure for the reparenting of the LRU pages. Signed-off-by: Muchun Song --- mm/swap.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index ee19e171857d..fbf887578dbe 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -291,8 +291,10 @@ void lru_note_cost(struct lruvec *lruvec, bool file, void lru_note_cost_refault(struct folio *folio) { + rcu_read_lock(); lru_note_cost(folio_lruvec(folio), folio_is_file_lru(folio), folio_nr_pages(folio), 0); + rcu_read_unlock(); } static void lru_activate(struct lruvec *lruvec, struct folio *folio) @@ -406,18 +408,20 @@ static void lru_gen_inc_refs(struct folio *folio) static bool lru_gen_clear_refs(struct folio *folio) { - struct lru_gen_folio *lrugen; int gen = folio_lru_gen(folio); int type = folio_is_file_lru(folio); + unsigned long seq; if (gen < 0) return true; set_mask_bits(&folio->flags, LRU_REFS_FLAGS | BIT(PG_workingset), 0); - lrugen = &folio_lruvec(folio)->lrugen; + rcu_read_lock(); + seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]); + rcu_read_unlock(); /* whether can do without shuffling under the LRU lock */ - return gen == lru_gen_from_seq(READ_ONCE(lrugen->min_seq[type])); + return gen == lru_gen_from_seq(seq); } #else /* !CONFIG_LRU_GEN */ From patchwork Tue Apr 15 02:45:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051377 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A886FC369B4 for ; Tue, 15 Apr 2025 02:48:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2403C280022; Mon, 14 Apr 2025 22:48:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F257280020; Mon, 14 Apr 2025 22:48:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F3989280022; Mon, 14 Apr 2025 22:48:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D2213280020 for ; Mon, 14 Apr 2025 22:48:02 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 98FD7C0B93 for ; Tue, 15 Apr 2025 02:48:03 +0000 (UTC) X-FDA: 83334743646.30.D4A9192 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf10.hostedemail.com (Postfix) with ESMTP id AC012C0007 for ; Tue, 15 Apr 2025 02:48:01 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="M/tlHHOJ"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf10.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685281; a=rsa-sha256; cv=none; b=QlWxahxkj5DbemCfdjMAIWlzLqZdHJncUUz+IweDdt5YmRCF5KZ4rnqvjyG4VjbAyP/8bw Ibr9bStdRMwUGYSh6FT+LWkxm69hapBPjNKqMR0VpkcoVXlXmz8hKslnLiaDtrDWjHx5x/ AWJ+WhrWqV8EZ/OZpd9zuXZqYFfL3ps= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="M/tlHHOJ"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf10.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685281; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OUs32dtFutsOKBw24WRLrtykUD+fXNK5AQbCBXYyH+U=; b=WQzavzN17t8Jm6Z98S5VNKVE4aC0sdH0lotgsOQk15vjoONSN1nSNQDG/qINRxnKuVeZeP d5AuBJYs2BHw9TwkJjCDpAjfzVnd0Pp+2gUh+UFe55IEJhf2XtRkJHIcREdPbWgLhkcSWh 5pVd8TVZw7mi2Uru36L3CTXfSZ6QWLQ= Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-223fb0f619dso53490485ad.1 for ; Mon, 14 Apr 2025 19:48:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685280; x=1745290080; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OUs32dtFutsOKBw24WRLrtykUD+fXNK5AQbCBXYyH+U=; b=M/tlHHOJkkxWW4x9kW06/5dLKXQKRCIMSW+/oRXQS/7zLMn/P+X3Y99jbvDdlK3ytm DcCdZtql1QHWXoSUYI7EgY4knL9T0t0zfrwaDAqGFxh5XpDYcjAj5AekmLlkwLEDMhCe rsUIDNLPsXO3SxffSJpg+IGk4LZeM18E+qdM+9cmTX9gd192D8CTLkxOzhFcwMb6bPTf 2pazBLoNIvdWLT+zGTHoe+2CUjtwPKQ7vl04lKgwkI5NysG3dZef5M1ZsOKBn6zc6fTM kEFBZ+aRW/6okXekoy8zs05OIZ9yuoNm437cEauKs3xoc8grTwS8jGLOn73cqYlMojjG 3vjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685280; x=1745290080; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OUs32dtFutsOKBw24WRLrtykUD+fXNK5AQbCBXYyH+U=; b=PGOgpiggblSsKxIlpmxN9M6+1coGHUHGGcbj+/aUTh1nzbfI84diRkB4IOUzcEIYhq qa3doCu7MwhycG55YMKXGqKecjlKae1/mrpJh59sUs6Y8OxEGOtJkrYTk3cROnejA30s w/Rh4Jm2vhDrd17CBpcE68lPJaDp6FKmlkQAlQfGCFRyp49utIq8BzGSn1bf3lF0oGm7 MQ0cWQ6lzE/OMLSEMKVJkeRxJ2NeTrs9AWvmHu1nkL3PjAHv6xYmwH/eKSuyth5WB6YA Pp7eL7VzohetWtJigtlFqU43bQPUyW+qZATqNj6L+BKJKOHucrKM/R/61M2Mp0W9rWgb leVA== X-Forwarded-Encrypted: i=1; AJvYcCU1Kg8uF1yhfZ2qnnVGNF454rKRjWTTwS3Q2GTRuAHiC71j2Zva8D0NzhneZdGqB8VA6iFyo5HYbw==@kvack.org X-Gm-Message-State: AOJu0Yyybpm5Mp9mPl4whSOS/z5Cbh3FDilhO3PQwfPiruCYbdeQlGf+ hSc1CNcT5KIKl78uihTAvmiTFM2AIlBwZihi/Y5HRFrJRRUcLxa6h0vbdD3B2fo= X-Gm-Gg: ASbGncu2f+V+ghlOfz67rGc4EBxnGFhoCnoaVOKwJTzhvh7YSv5SRVIbsY7b3sxsups JQcVSYutXM8fPDZDkMowhbKh6AuuuJjYxfrT+kAPxFijB2n2RJ4KiF+Z+A7HEGQ7F2TzylSZ3ct vHucTvyZqOLPlJVPkgYVBK9/AoT36PZ2/ElecW/CoDKtZibVvoIdf9guGdYbhpPVRtwFRJNvBsM TOzc4WEGDhseZ3QaOJBZN3GNCP17odfLTRGF+wNqQ7Evejc7+sL/zRzbpUIQ7JdB+9XFHaL3BCV UgHLofBESSAKXcXxNOG07wGzz2o/IebihybwwudYVkm8RpRXyBo8wFhcbt/9yHedbDQQ+Fzh X-Google-Smtp-Source: AGHT+IEtjVtnb1BSndquKXoPm2m8j71DYv9c3rLhXw+JhVIvJMgE2spfc/6lyYJ7b6Mu11ismWhvkw== X-Received: by 2002:a17:902:e547:b0:220:c911:3f60 with SMTP id d9443c01a7336-22bea4fd182mr195573825ad.47.1744685280634; Mon, 14 Apr 2025 19:48:00 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.47.55 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:48:00 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 23/28] mm: workingset: prevent lruvec release in workingset_activation() Date: Tue, 15 Apr 2025 10:45:27 +0800 Message-Id: <20250415024532.26632-24-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: AC012C0007 X-Stat-Signature: fbdhmfdhifews83rch13iokhmz3pqfdx X-Rspam-User: X-Rspamd-Server: rspam06 X-HE-Tag: 1744685281-287272 X-HE-Meta: U2FsdGVkX1/5XYtA7bxt2RvYxzomdA5MMKGCi7bFYjoGFstGyGCGMeWwukeMb8lgcblqZl7MI2b7Rt17CRenAXNwggZFGhzwBr3hsK4wn2jpwV7Q0h13ctmhWQl1WTYFE1TpgbcSzgKSWDy+gdF9KvYBYv5qHnGxe5BP02RBOXetvdP0Qyqxhy94d5KcB2Dv5ACcNvbTuDxXaZlYai3WPkn10XmSXF2dpQBBwqT7Wq079snmVWDrrZgdq143zu/jGI0IAxSxVwXVFZm1O0gMS+cx/5F+r/+pHWCQfKSYlWU5hCQT6cGYOvhCNvtEwaWHj3fYBqJLtzLzLHV/HUytsulNZq89iifeq3KH5/AaH7QGLX4gnrwx+DlHWdEIo7CSadd5+5iy9N7lv5k94MaiEktwl8J2Lmvem/4KWj14GWgGOn0gr5yduXHxYcT38G8TMRtqb1nU3JuUJQMrmI1mSQpgLSCrLFOkX6EeL8TcPvjCVpZ1sthakVwmKOvekpEkKQ00k/xSLEpIrl0ZjZndwx5CePgxAR+3G1vfIGKRff69akIn/fFaBX+P4AP7OZLI6BnB0+fgcqejE+YMB2m9a6FFRxa+/2DlglCnVQrO8z1KHeFFZz2q9EYq8vpaQ/EhIlRuNm7Hr25zyGqNMh4BZM9ohg8jmiF/XQ27RjgU1tWdbME3I9RWyYvfFwxnbBiGjNJb3TfwiODn6fYVR+HuTsRTRnRkrMb6dbjmQXg5f0UHAd0aaaiRBqRrHW44c/oecbvjUggNQm2QL+K0CijxBB+YketYVx2XjJzCRmqbk5hAuXRpHwbg7xrrJJmS98NeSMjBXfppv0TZBv3HzaXrcvpTmEm3cJDha/4JysGfe8QpeOinsYdi0LZzYeGsf8H+NdPXGavdGswN24rpg4XDOrcwo7e6c3l3dFlyCI8znQhbkA58kZsHZS+XEfiBiiDt6aVOX87zxA6EQjUmAOO AjaG/Ci0 F1OCnIiaQn9QEaY/YcGt3KPQhJOoV5tKyZXzlG+HnixqCMcybOBdFrrqtBfD69MRF9RydYCNUsZW4nR87BUvZNKivhwvb6ohIY8XMP3yLryDm2k6MpTXWQ+ldfxsYOe9/UUgQlWAFEp3s2bsEcGkWiWPLkFnIUkWNEPsabsPpyqYARZ4Z9RxZ/e6QTE5EXjSEPIw1X2jfOM2wfP1YJrVp/kEKy0OJH42bRSQEaebBCvelTHXnH+/S2CTNCzphE8K9n0Xu5ai6e4okJeJ28ZiTPe0yIBSqA9VU/cS7U49f6UPFkRQy9ZIropXaoR1PrSR54+8O1wcWUKOEROizUVWOuQgvFxQiJh5T3u9KtWmjnmfCzoU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the near future, a folio will no longer pin its corresponding memory cgroup. So an lruvec returned by folio_lruvec() could be released without the rcu read lock or a reference to its memory cgroup. In the current patch, the rcu read lock is employed to safeguard against the release of the lruvec in workingset_activation(). This serves as a preparatory measure for the reparenting of the LRU pages. Signed-off-by: Muchun Song --- mm/workingset.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/workingset.c b/mm/workingset.c index ef89d18cb8cf..ec625eb7db69 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -595,8 +595,11 @@ void workingset_activation(struct folio *folio) * Filter non-memcg pages here, e.g. unmap can call * mark_page_accessed() on VDSO pages. */ - if (mem_cgroup_disabled() || folio_memcg_charged(folio)) + if (mem_cgroup_disabled() || folio_memcg_charged(folio)) { + rcu_read_lock(); workingset_age_nonresident(folio_lruvec(folio), folio_nr_pages(folio)); + rcu_read_unlock(); + } } /* From patchwork Tue Apr 15 02:45:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051378 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04CDFC369B2 for ; Tue, 15 Apr 2025 02:48:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6B84D280025; Mon, 14 Apr 2025 22:48:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 63F68280024; Mon, 14 Apr 2025 22:48:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4BADA280025; Mon, 14 Apr 2025 22:48:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 29B70280024 for ; Mon, 14 Apr 2025 22:48:08 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 00BC3B0FBC for ; Tue, 15 Apr 2025 02:48:08 +0000 (UTC) X-FDA: 83334743898.16.72FC457 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) by imf05.hostedemail.com (Postfix) with ESMTP id 0D4E9100003 for ; Tue, 15 Apr 2025 02:48:06 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=c4lF7jsJ; spf=pass (imf05.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.170 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685287; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CpSUB3YDfgeYGmfO/fpEZlFvo0fqwlF+YsT5eAHFtoQ=; b=iH1Vf9faKnv+BJnoxIKrIewZxjG6r95qoM9s+AcVf9EO09gIX3Y8rqtj74xLD3dTr8ipjy Hh5EkN+L6UT/sr4TeB5ZstxKfoVfT0KoIuBgIsQg88fG1uUjWpACg26/QQUOBNrXLajwzh YR4ZswLm6vQ2HGhvw92ZBqqrkS1PsBs= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=c4lF7jsJ; spf=pass (imf05.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.170 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685287; a=rsa-sha256; cv=none; b=Q7gtysIj5cx/vtgB7nKNO0yB57y2hLRSXVPNQPbiUjhx6fbba0gnqwa3+CfO+G3BmZzv1d oW5194athgPn56DRAtYtUq8q9jpy+Ol+AhfBhODKkIgr249Mf+iqdqIqcR8eyZLWNQTWu/ 5b1m8+Cb3XsBu10CC/1SwBlins7VNCQ= Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-af51596da56so4317187a12.0 for ; Mon, 14 Apr 2025 19:48:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685286; x=1745290086; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CpSUB3YDfgeYGmfO/fpEZlFvo0fqwlF+YsT5eAHFtoQ=; b=c4lF7jsJ0sKXnIJoQQl2sjYGlFsOZaJrx23jktbIOIOTLEbUm5BC6ZyWO2Rs3q5DGR MKxVh4tEbU//p+SRiCM30XxnOeThl6DyxKsEGVsMLT4J5uX6DuuEzX4kr2CIcSAaeYjR jQZ9zmiiCZDiY/V1++JAZgXCpDxLQZjEB2lX1MwoJUIT2KFdA8csRmYZv600TrimO+6U TxK7QTh3rQfA/Msa0zip5t6j57vn/xLryEieTQ2tTsB2s4opRenkTa7IHTEJB/A1iFyr 7qfaiIS1v1R5iKnXaTKGrA5g/xisiw0j6XIAqCrnKrd3qPb8eqrk2WLX4MgsLQ9K36i4 XEVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685286; x=1745290086; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CpSUB3YDfgeYGmfO/fpEZlFvo0fqwlF+YsT5eAHFtoQ=; b=aESQoNWiXgCEIJsGnhcHDhU1DFYDly1aW/wzelmr4d4F2QPwJ6QWWKDw608TK/LaHs vFpw7hSR8wt0zUHD93YR6C32+eSEfS5EgO7F/BeFoPfJWaipDC8gQO9xw0/f7gvcwZ1u 4yN60smsb+AwQb4aa5vP8Pf0qWLG6YAoCyRCwcuLV8ct/awua1+xpgfobEeyF4MifffT wKk++oHMAiKK6tVJ1rTA0AW4wnS4HDrIG++D7TACUHeWOfP7g+Z2zcar9onlwz+lR/S7 jO8lcwIkrhJIr3V0NaIKLU1VIzm4p3f9nQ8fEVUwwEerup3x3FREGJqOBWjcMr9k12HD iC3Q== X-Forwarded-Encrypted: i=1; AJvYcCWNfg4cK/1NIQEooobSB2iJcXJUdaL+GvcH+UCKrLZDNb0DvLjqxRj3NX5um3ICUulyDnMYKi6HeA==@kvack.org X-Gm-Message-State: AOJu0Yy67ZuuzVOHtLOawe2ZB/7NdYy+/z5BTPfbCf62/zj23m8dMwAS QIocl5YuN5dntKnOtmUR3VnQxLBNxKtKzqv26X4pe9RSKR3Qah9T81fnzOA2ASc= X-Gm-Gg: ASbGncuDZT2UcW9iHUJXnIvYHHOfBC4Ge6DahDongVLJhsAgrnRVoV0Fy/vcRRzuvNr JuX0LaZt6SS/u+/VaReZzeMcRjwCM4GWykEXzciPbKKxfI3YH4d6jZ16IDZdWg0GCG/dTF8hvh+ NLuuDBGwnmm5rySJFH0pg/1m1kZmh3A48Kl+oZ2gdt8dil0KxPKTZgYQvNRLTzQg4sUa82mZ25t w28GL2VLJzPcgCuO38rqg0vb/q5M0IVCDafATXruZe/HNIRoKHKHZZjvZ89Fytv8VmVuhjbGX7U 9B1RVcNMPPxxjMXcPfkXHfgKU4RASxG6OdhEh88mV/l/hwygUWjb5NQ18Ij0heRWvfVxDse5RzJ 0CR5b9i4= X-Google-Smtp-Source: AGHT+IHgqNDuzvTzNM38ht/Apai59kRy7TIM0/jwQoLjF9IBxI/3XCoQow31pfiiq7WdKcqDsqh+0g== X-Received: by 2002:a17:902:f78a:b0:224:1eab:97b5 with SMTP id d9443c01a7336-22bea49530dmr219890795ad.1.1744685285756; Mon, 14 Apr 2025 19:48:05 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.48.00 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:48:05 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 24/28] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock Date: Tue, 15 Apr 2025 10:45:28 +0800 Message-Id: <20250415024532.26632-25-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0D4E9100003 X-Rspamd-Server: rspam05 X-Rspam-User: X-Stat-Signature: xbwpjuy8y98pmj5fdmperwfshadafinm X-HE-Tag: 1744685286-526222 X-HE-Meta: U2FsdGVkX191qXDkcgZDGhwuCZk4V2k9OpejcQhQJcYHqsRuZANtLlDRGC96dRYWoNs+08YHrzo4tEw/EWeoXxwoowpvS4kCfKGukjOXMsKRvBO5BMUtIcMj2xK/Tyee4jqm7So+0H8Plvmw56KTI9kTva2Kq1xCP2htl/ZCdizImMOncArVksapnP8j4GEzTBnpDnIt6sWKqC+eYjf7yep0R5uV+5GSiw+xCMaBIUPDJFAmZ6meBWF65yioZIEPKEPRIr9Y64Sg0NddHODMWaqWXqNeGwIOQmKmcq8DcT/aHYgr+HHpVO88fia1xEqAKbrmj7QquD02Uqw6E2TPzadqT+9YZTHdyx8k+GoT8baAb/TZWPPKWRSMNYzMmBf/A0NFxtj1l2danIJn6nODVVhZtcaz2MjCh7AVcBIvWUjoyXsnfdWvg+VAL0l70I7zVKX94HSuAlPia1OZ0w678/xjYEDk8osBIUT+XSVqfzWr8xRFzubHFbmojzEbcYR6COID3fvdmp80A3lPJxFBv2zPGfTfL11SGPp+Fke9cEllWKZqb1C6ZO2+g2TE6jmVPu0a3O8AAxX7O3ikX1CmkS7yX5i3sJoBHw25wT/VqEDXMy5Jr0kWzS04fT5QimhY02K44nNk9CwrxqR4RuvMf8EFHujXhJHbr33Plnzb+HKg4v2S4xuLEyF/38x0MzM7vtRndnngEJ65T1zBcM2dgRF5PcWg2LK2FuFMMZDnMRpO8zOKSaGmrrhnqB2K3c+IIbeEnDKpkcQqyIrG23zoXo4/lBIv3nETLmPRm3dZ88pTzz81CR+QUsJaXdpnBHPU7p5qC83QuiIC9FknI1dAmKRdMOjtNflJlQ5SoZKI/CBNbREyck9PA59Jhh/VxYgiNOL+woZdTA/GkbDQavnyw30h8m6Cpl8GwWOhP311H8yp/9zW6VRATuHYnwPIA5T5TNOgeAeH6ymeijOtfZC Qul8Vr25 uWDA9jepjGd+R1M/CZJfW+JoBvSjZ99APuV4igsUlxTnSJKCZQVK+/MVMBYbJywSiJ6w4WQw6JBtcAyH7s4bYXe8vB5BOVdoCgpeNSuWTQpQX6KOX+bOJl6gOQ+bkQPEFmr4IZMq0XNzyhHVVT5u6Xlfh8HGV4KPAeBNyIMK6zc9JF0kK4h5/Qq9/4x9Zeab5HVX6yAu3yntFZyMY4MzabSx8eHr4nqymSfjFQh4yClx4kPiA2f809CnPeDTbTqEzC918FmcIgKFPCEA+EJ7DS30Psj+QY/fzQVI/jW/Qf+uzAn8WtzGwBxTiNw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The following diagram illustrates how to ensure the safety of the folio lruvec lock when LRU folios undergo reparenting. In the folio_lruvec_lock(folio) function: ``` rcu_read_lock(); retry: lruvec = folio_lruvec(folio); /* There is a possibility of folio reparenting at this point. */ spin_lock(&lruvec->lru_lock); if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) { /* * The wrong lruvec lock was acquired, and a retry is required. * This is because the folio resides on the parent memcg lruvec * list. */ spin_unlock(&lruvec->lru_lock); goto retry; } /* Reaching here indicates that folio_memcg() is stable. */ ``` In the memcg_reparent_objcgs(memcg) function: ``` spin_lock(&lruvec->lru_lock); spin_lock(&lruvec_parent->lru_lock); /* Transfer folios from the lruvec list to the parent's. */ spin_unlock(&lruvec_parent->lru_lock); spin_unlock(&lruvec->lru_lock); ``` After acquiring the lruvec lock, it is necessary to verify whether the folio has been reparented. If reparenting has occurred, the new lruvec lock must be reacquired. During the LRU folio reparenting process, the lruvec lock will also be acquired (this will be implemented in a subsequent patch). Therefore, folio_memcg() remains unchanged while the lruvec lock is held. Given that lruvec_memcg(lruvec) is always equal to folio_memcg(folio) after the lruvec lock is acquired, the lruvec_memcg_debug() check is redundant. Hence, it is removed. This patch serves as a preparation for the reparenting of LRU folios. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 23 ++++++----------- mm/compaction.c | 29 ++++++++++++++++----- mm/memcontrol.c | 53 +++++++++++++++++++------------------- 3 files changed, 58 insertions(+), 47 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 01239147eb11..27b23e464229 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -719,7 +719,11 @@ static inline struct lruvec *mem_cgroup_lruvec(struct mem_cgroup *memcg, * folio_lruvec - return lruvec for isolating/putting an LRU folio * @folio: Pointer to the folio. * - * This function relies on folio->mem_cgroup being stable. + * The user should hold an rcu read lock to protect lruvec associated with + * the folio from being released. But it does not prevent binding stability + * between the folio and the returned lruvec from being changed to its parent + * or ancestor (e.g. like folio_lruvec_lock() does that holds LRU lock to + * prevent the change). */ static inline struct lruvec *folio_lruvec(struct folio *folio) { @@ -742,15 +746,6 @@ struct lruvec *folio_lruvec_lock_irq(struct folio *folio); struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio, unsigned long *flags); -#ifdef CONFIG_DEBUG_VM -void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio); -#else -static inline -void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio) -{ -} -#endif - static inline struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){ return css ? container_of(css, struct mem_cgroup, css) : NULL; @@ -1211,11 +1206,6 @@ static inline struct lruvec *folio_lruvec(struct folio *folio) return &pgdat->__lruvec; } -static inline -void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio) -{ -} - static inline struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *memcg) { return NULL; @@ -1532,17 +1522,20 @@ static inline struct lruvec *parent_lruvec(struct lruvec *lruvec) static inline void lruvec_unlock(struct lruvec *lruvec) { spin_unlock(&lruvec->lru_lock); + rcu_read_unlock(); } static inline void lruvec_unlock_irq(struct lruvec *lruvec) { spin_unlock_irq(&lruvec->lru_lock); + rcu_read_unlock(); } static inline void lruvec_unlock_irqrestore(struct lruvec *lruvec, unsigned long flags) { spin_unlock_irqrestore(&lruvec->lru_lock, flags); + rcu_read_unlock(); } /* Test requires a stable folio->memcg binding, see folio_memcg() */ diff --git a/mm/compaction.c b/mm/compaction.c index ce45d633ddad..4abd1481d5de 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -551,6 +551,24 @@ static bool compact_lock_irqsave(spinlock_t *lock, unsigned long *flags, return true; } +static struct lruvec * +compact_folio_lruvec_lock_irqsave(struct folio *folio, unsigned long *flags, + struct compact_control *cc) +{ + struct lruvec *lruvec; + + rcu_read_lock(); +retry: + lruvec = folio_lruvec(folio); + compact_lock_irqsave(&lruvec->lru_lock, flags, cc); + if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) { + spin_unlock_irqrestore(&lruvec->lru_lock, *flags); + goto retry; + } + + return lruvec; +} + /* * Compaction requires the taking of some coarse locks that are potentially * very heavily contended. The lock should be periodically unlocked to avoid @@ -872,7 +890,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, { pg_data_t *pgdat = cc->zone->zone_pgdat; unsigned long nr_scanned = 0, nr_isolated = 0; - struct lruvec *lruvec; + struct lruvec *lruvec = NULL; unsigned long flags = 0; struct lruvec *locked = NULL; struct folio *folio = NULL; @@ -1189,18 +1207,17 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, if (!folio_test_clear_lru(folio)) goto isolate_fail_put; - lruvec = folio_lruvec(folio); + if (locked) + lruvec = folio_lruvec(folio); /* If we already hold the lock, we can skip some rechecking */ - if (lruvec != locked) { + if (lruvec != locked || !locked) { if (locked) lruvec_unlock_irqrestore(locked, flags); - compact_lock_irqsave(&lruvec->lru_lock, &flags, cc); + lruvec = compact_folio_lruvec_lock_irqsave(folio, &flags, cc); locked = lruvec; - lruvec_memcg_debug(lruvec, folio); - /* * Try get exclusive access under lock. If marked for * skip, the scan is aborted unless the current context diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 694f19017699..1f0c6e7b69cc 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1196,23 +1196,6 @@ void mem_cgroup_scan_tasks(struct mem_cgroup *memcg, } } -#ifdef CONFIG_DEBUG_VM -void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio) -{ - struct mem_cgroup *memcg; - - if (mem_cgroup_disabled()) - return; - - memcg = folio_memcg(folio); - - if (!memcg) - VM_BUG_ON_FOLIO(!mem_cgroup_is_root(lruvec_memcg(lruvec)), folio); - else - VM_BUG_ON_FOLIO(lruvec_memcg(lruvec) != memcg, folio); -} -#endif - /** * folio_lruvec_lock - Lock the lruvec for a folio. * @folio: Pointer to the folio. @@ -1222,14 +1205,20 @@ void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio) * - folio_test_lru false * - folio frozen (refcount of 0) * - * Return: The lruvec this folio is on with its lock held. + * Return: The lruvec this folio is on with its lock held and rcu read lock held. */ struct lruvec *folio_lruvec_lock(struct folio *folio) { - struct lruvec *lruvec = folio_lruvec(folio); + struct lruvec *lruvec; + rcu_read_lock(); +retry: + lruvec = folio_lruvec(folio); spin_lock(&lruvec->lru_lock); - lruvec_memcg_debug(lruvec, folio); + if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) { + spin_unlock(&lruvec->lru_lock); + goto retry; + } return lruvec; } @@ -1244,14 +1233,20 @@ struct lruvec *folio_lruvec_lock(struct folio *folio) * - folio frozen (refcount of 0) * * Return: The lruvec this folio is on with its lock held and interrupts - * disabled. + * disabled and rcu read lock held. */ struct lruvec *folio_lruvec_lock_irq(struct folio *folio) { - struct lruvec *lruvec = folio_lruvec(folio); + struct lruvec *lruvec; + rcu_read_lock(); +retry: + lruvec = folio_lruvec(folio); spin_lock_irq(&lruvec->lru_lock); - lruvec_memcg_debug(lruvec, folio); + if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) { + spin_unlock_irq(&lruvec->lru_lock); + goto retry; + } return lruvec; } @@ -1267,15 +1262,21 @@ struct lruvec *folio_lruvec_lock_irq(struct folio *folio) * - folio frozen (refcount of 0) * * Return: The lruvec this folio is on with its lock held and interrupts - * disabled. + * disabled and rcu read lock held. */ struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio, unsigned long *flags) { - struct lruvec *lruvec = folio_lruvec(folio); + struct lruvec *lruvec; + rcu_read_lock(); +retry: + lruvec = folio_lruvec(folio); spin_lock_irqsave(&lruvec->lru_lock, *flags); - lruvec_memcg_debug(lruvec, folio); + if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) { + spin_unlock_irqrestore(&lruvec->lru_lock, *flags); + goto retry; + } return lruvec; } From patchwork Tue Apr 15 02:45:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051379 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BFD4C369B2 for ; Tue, 15 Apr 2025 02:48:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 73EE1280027; Mon, 14 Apr 2025 22:48:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6EC69280024; Mon, 14 Apr 2025 22:48:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 515BC280027; Mon, 14 Apr 2025 22:48:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 307BD280024 for ; Mon, 14 Apr 2025 22:48:13 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id ED8C4BAF73 for ; Tue, 15 Apr 2025 02:48:13 +0000 (UTC) X-FDA: 83334744066.07.F4233EE Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) by imf25.hostedemail.com (Postfix) with ESMTP id 137A7A0010 for ; Tue, 15 Apr 2025 02:48:11 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=OgwrPZ8K; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf25.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.179 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685292; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vzey8goXEog+ZffEsOrcu6z/kqiEK8l86D0gwmtPc7Q=; b=fQO+GcDM1i/NRBtEeVn5tajlTgSGuu1eOQmvq5ikKRDVPqVebHD7NEyIMFhYnW8Wm7p6Eo P0EFPrftjZN2mLOyxnZ4O++NO1sZrtVijJCXwsn+0HXpFEXLYoqcB5N0bMj1YeMQ1g+HUE ufRm/IMCjKyhJt/v0LEg5bECFYgm8tk= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=OgwrPZ8K; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf25.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.179 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685292; a=rsa-sha256; cv=none; b=3zR2/1KXf7d5DCi79o3tkq2Ct+scscZjCSgPjzmHqIAipLemokj17upeBUIKxCG0FPKPaZ k854ncUYnZjCFp25HW97BrazY2nEuP9Zx6KPj5wF2Mp3O1gNAwEneeibBZOLP94OKrnmgc rHmSlDZRfBxEwRAvFBr35eqkEZUQIp0= Received: by mail-pg1-f179.google.com with SMTP id 41be03b00d2f7-879d2e419b9so4154114a12.2 for ; Mon, 14 Apr 2025 19:48:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685291; x=1745290091; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vzey8goXEog+ZffEsOrcu6z/kqiEK8l86D0gwmtPc7Q=; b=OgwrPZ8KJCuZoIxVpY3pVwLE0VLFFmjJot6QLs57KoK6MptANeGCXGcX6bcq1sRTmY NlgerXQ5DoTVhwDvSfuXreoSIBbRBWT6EyBFBjP0cYOXc1gxyRMGwgrFC6hp4hivoTFc x9DuMmG9wIaF0T9hiLa46ets2F19fIo5+9PLMlu6EMylXNmvi5YLv9DhyaHKFSOkPwIK 2qhKxXAU7ZQobpgC9S6qeezraTstgwwgtisKacFJEym6y4tQCWz5ynrYqAC+gscch0li L6JCfVo4ZgPtRh34NOHmXuSFTdoL0SFWd6HaSCx2Vu5/gfL4CDtQ+jh9ff68EshZEdMW WS2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685291; x=1745290091; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vzey8goXEog+ZffEsOrcu6z/kqiEK8l86D0gwmtPc7Q=; b=WUyb0uVrSUI0dJTnYWdkx+WLpRgewFMUNF539eV0td+lNh8MBQrpiOI5X/WjQinf9m 8zw5I5tusJ3cBbEpX0q9Eh8GKYkIi8G8pkfS4jgfcImlcTI3VfXk2ef1CaD5ZUn5Rln3 pwmWzeIsLR9gsBKrjP84Clans2U0ZR4FhFuG1moXm75q+Eo1NgrNxYgVCQ8keXZrCuJs B+LiNPf0XcbrTA+zXQsCpyhtfDlH23uui2MTORG2BWAqYf7viYtqbOQo7bKjPMqu2RKm hm3JJvlVaOiyhok5BGKs+wx4mWBB/cCcq5F3+B0xXDncx2+GLrKbLEyemdDQy1v/aHgx 6pfw== X-Forwarded-Encrypted: i=1; AJvYcCW0s66YJ44WV/qwQejOd5sMO+9Rb+KiyXjSNOTM/VVUH6ba+vZF/5tWlmBxMfVF8TQwZyItX421AA==@kvack.org X-Gm-Message-State: AOJu0Yy4Pt41gEy95d5MLiVwA/ixA/kb4Q5fKZDe5Te3NPCzCPkEnoUe teWMthDucvxTvaQKaeaHgw9GDlSqOqLHhYIy7dL+rnjpm5zI7kWabqn/CTTOW/g= X-Gm-Gg: ASbGnctxRcuZEiFC1LiEKK6/7LHvsdkcf+7I/rXixXZr/7VMN2f3k3lsg9lBcxmtf0D /w6G03d2u+GZQ9ZKUZ8Dx+bDXwP3h3fXipGrs44VxECJUxNGnz5Alcd5mhgvDQB1VecNVSlGVMT 87G/JhdMpewKu2fWF5/vfMKm7PN/mhaiLr/jvK3HXy66DYcWTUjDImSRlJLgPtvsoj7YAHZrtZ0 GkPuKDLVDN+Xx9ksTl5em5hp/kHJDuW1OT2t+s/e+kx7wHgELQ4xPuQ+khIaT4WwHYaOzBOSC/D 659f+iTboK5Kf2YjrOhrlkC2j8bHUsFBsPh3CjBcTCAvOaXx0DQ6bx7vg9bco+FT/VmQZUk40Ao tqre2NoA= X-Google-Smtp-Source: AGHT+IGKYbRxT9nU7SMJ0Zxzh0coEyzVb1MW3k+CSg2YANyZzMz0FPRUg+d3lPphVCyM8bkhwJIXdw== X-Received: by 2002:a17:902:eb8a:b0:220:c813:dfce with SMTP id d9443c01a7336-22bea4f273amr208688465ad.39.1744685290927; Mon, 14 Apr 2025 19:48:10 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.48.06 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:48:10 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 25/28] mm: thp: prepare for reparenting LRU pages for split queue lock Date: Tue, 15 Apr 2025 10:45:29 +0800 Message-Id: <20250415024532.26632-26-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 137A7A0010 X-Stat-Signature: tby8q37jrbg7esg5huafqy9ac53tu4fi X-Rspam-User: X-HE-Tag: 1744685291-720488 X-HE-Meta: U2FsdGVkX19TB3CohZl1ImkNcwn8AwTvt7OJXylA5FfvkAK5y3eAdm2t08wploAdvQJQ7h8hl6Sc4VKQ9dDbvoBVu8WJrXJ0dTSTIYvR+x5e467kp1vIxEVTmNvttjA+q7Zt7zE9WMjAHMtneR5xIb82iwcibCfx9hWnOqis3PkwlTIJOVmOzghYH4bPFwpQYmudnz38LdShsTomMqdmMCCv11N3pgBc5KO1mRzwWZkhR6vum4QVAn3HknHknKC6obPgX7AnWsJMD3oR4JiVDUV55+xdcI1zRDYsPi4VfkWj9xVbM45mQwfs/uh0RghrV93sVIW189eKCPTkpH8pAln4jGC30ajGx7EC5No+T+7zLRH16BJ7RiAeU9dwUPlXGmVoUnE/R3/JVOJ+lYNSjbUsc+mNKHbEjj5ksSkzm8QO04Re9pEtuxsOj6QTPDCb2rK7Lc5jXrmVLYa07dz0RWIVQJg6GTdSAe5S6ys1lkyCuvilSq/KCocsWSwpUNG9yLONArLOsQ5rdZp7x959gZ63cWt8cnpPfMJ/emhYFeQ3CsYTwSoBRNbwc3eWO2TJHnO0fTvHrQGrgLVFy5M/LXsMcumZc1VkX2n+XMYxjM5th6OtBWzNcBSfr27UtbfHlTUjTjr6MrR/cU01kB0sZQmhx+rsMcNoA2bJstQks7kE/L1EQmlrVfsiz1xS1OHNvKTkmuIKsTitwfb0Urug5uaR0beb8DGfo5mQIklEPjxWvld3H8poglM4yAF3XA4I6UB6MhBxEK/+KOk8jOWRryJ3fEsZuVeLUDg5wrMM/QbxG9CJ03Sxj4QYEjBPbWo++h4g35W8lEsgMkwTXS88mtJHJtUA0VwGtipHGKCPwXc7/u/Wy1aoauxTF6d7kSNuei2Lv+8e8o6SHozfK1q5+suJCohGGW5W+zD1kZyf836mZ7IF7664MjxNYbhuMhLs6mnX7kxtxe6VkQkzZB6 TqwQVVsQ MuRcVqubuL1QcN2/DfE9fDsxRVcTMuPh84V2GkUtpGoa4oQ9sBVaG14yf0/1Hzj0Rx35MNQEdFHY1I39Yk+PyUIZvs3dgeUW5QTOcDupQoajyTrz40kX1+kK/FwB5AiIY+MlAA8W8i1fN27lybZmSk/UDN8oOTfjKvRiQm4xtq+rlGajSjRkVE2zrXGDEnIxu5oEWjbYmt9un9paSWrlAPNY1GT5BRqkDTRClab4dLdTLTO2ghlN7ptyNtEqouN4qb5XiG8J6GU56jRQaGCWIV6K8qcgZwRhaCS/pyBBgenE2dowyw+krrVNzKLp9SirmbOef X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Analogous to the mechanism employed for the lruvec lock, we adopt an identical strategy to ensure the safety of the split queue lock during the reparenting process of LRU folios. Signed-off-by: Muchun Song --- mm/huge_memory.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d2bc943a40e8..813334994f84 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1100,8 +1100,14 @@ static struct deferred_split *folio_split_queue_lock(struct folio *folio) { struct deferred_split *queue; + rcu_read_lock(); +retry: queue = folio_split_queue(folio); spin_lock(&queue->split_queue_lock); + if (unlikely(folio_split_queue_memcg(folio, queue) != folio_memcg(folio))) { + spin_unlock(&queue->split_queue_lock); + goto retry; + } return queue; } @@ -1111,8 +1117,14 @@ folio_split_queue_lock_irqsave(struct folio *folio, unsigned long *flags) { struct deferred_split *queue; + rcu_read_lock(); +retry: queue = folio_split_queue(folio); spin_lock_irqsave(&queue->split_queue_lock, *flags); + if (unlikely(folio_split_queue_memcg(folio, queue) != folio_memcg(folio))) { + spin_unlock_irqrestore(&queue->split_queue_lock, *flags); + goto retry; + } return queue; } @@ -1120,12 +1132,14 @@ folio_split_queue_lock_irqsave(struct folio *folio, unsigned long *flags) static inline void split_queue_unlock(struct deferred_split *queue) { spin_unlock(&queue->split_queue_lock); + rcu_read_unlock(); } static inline void split_queue_unlock_irqrestore(struct deferred_split *queue, unsigned long flags) { spin_unlock_irqrestore(&queue->split_queue_lock, flags); + rcu_read_unlock(); } static inline bool is_transparent_hugepage(const struct folio *folio) From patchwork Tue Apr 15 02:45:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051380 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39DA6C369B2 for ; Tue, 15 Apr 2025 02:48:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B901280028; Mon, 14 Apr 2025 22:48:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 94017280024; Mon, 14 Apr 2025 22:48:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 79424280028; Mon, 14 Apr 2025 22:48:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 59254280024 for ; Mon, 14 Apr 2025 22:48:18 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 2BEB3BAEFF for ; Tue, 15 Apr 2025 02:48:19 +0000 (UTC) X-FDA: 83334744318.01.7422FA8 Received: from mail-pg1-f174.google.com (mail-pg1-f174.google.com [209.85.215.174]) by imf03.hostedemail.com (Postfix) with ESMTP id 38CE320009 for ; Tue, 15 Apr 2025 02:48:17 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=bQs+DKaT; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf03.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.174 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685297; a=rsa-sha256; cv=none; b=NRW816o3/6bwIZFjZYSwccifmSvhDn7TZANlgCiiBNa+g0Aw8RpFJz2ZWZ7V9Nzn86EfM5 dUuFLtH/R+cBFSwMiHr1AWsTlpN9lEWmbqWfxilVklHylXnBBuEH64iODo8zxRByllbfAr chPcbTolPCf4TUHDNASYoefgXpN90yk= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=bQs+DKaT; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf03.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.174 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685297; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=chHLX5G0loxHSohYhxyRHh7WVRRPsaZ78MDusal3hIs=; b=VUM/KRQsulnoHciah+vneKwIrq1bnWYl6rRsENLEaRv8Qme8Vf8cqZzdB8DuGdFJdVt2Mw LYOoPfBLC5t2Z9BnS5GYUu8D4i0QYi8rDN7miRzSa3/z/8Y2BC/G/q/w5umnxZEo7LYEFO QlNDdCyBVv4q3Z3zL7vGabpWA7Wtals= Received: by mail-pg1-f174.google.com with SMTP id 41be03b00d2f7-af590aea813so5551302a12.0 for ; Mon, 14 Apr 2025 19:48:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685296; x=1745290096; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=chHLX5G0loxHSohYhxyRHh7WVRRPsaZ78MDusal3hIs=; b=bQs+DKaTVdAZJvwf7ECYvjdnVlIzmsF23PHff10sZAFSN9+pCYNqNnkM/4QLLpG/3h B+N52oLTVtxleHypr63dwEkbGaVUy5pykPt8RBtslnlwZ2/YkREI4U8VmlI+pMcg9vwK 7obr35cfiHSXmYVoMr7GtEIeBnsmkwyYg/qHX5dICDeMYYmb97Id4irOHVATM1Y/+pyM nAgFdELrQkfYguMw5BarFZ1Yq197FYunt46g1cDP7gCMeuu/Psa+R0z4fweCcUuCD37J 1Vm3TymO851Bx1UxigjWUQ4inqglIGAuUJiwDG8QbEZzlFI0DuPICiQHUKFdE1/tEfog rsGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685296; x=1745290096; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=chHLX5G0loxHSohYhxyRHh7WVRRPsaZ78MDusal3hIs=; b=lvzoQ4Q7bTUZYorI5Khc9/AeY+kXvYgk5dNfCPVR9aszghtnKtGt6zx2pyVxDeLhHp 6vwDgKOlgHm/77HtqHogZ1Xkk/059D2tl537lKlLlvf4wd9x4i8c8+05l5h1OdbXoy74 fivI05CxcflsaQn6f7+UfEzRKAc5vwWyQ/S7eI8rGWNaYpAy34kIPwbWPyY3HvI6xqC2 FfGOi8V/xEeXNXy8c0rJkIJxQjONyTp4a7hiRpGyGUihEzuK0m1pejthxBpPmWW8qBuj Zfz2fR4a9LYIVRg18bIk8NWruq739U5EraSUT/Md9MCYfBNbmqgsogthcDBb5d+MNncn z/uA== X-Forwarded-Encrypted: i=1; AJvYcCWd/yMuw1y8RnQ5KoEdjZdP2MVg/ux6XKfEsBhobuoIdE3qZjXQX1ncdK9We4rwxZNIyG1+9SyM1g==@kvack.org X-Gm-Message-State: AOJu0YzYVSLu21vWHkFJT4NY08TdtZcDRh4VD0JreulEpnjgRLUM/fcz Kq8sZ1wIMLyh9T5ToVZ3lDyCF5m0fnoiaufw1okZt+M7DCe9C4zqK/ZwRwr3uUlO4+jVJIIZ0iP YSPqqYA== X-Gm-Gg: ASbGnctcDdqJ+g2yTO2fvodyY2b9e7MgZo3M972d0EevD6Yp+bl6CD87Q/Vbu37y9UF Ohz3/yIp6pGuFsxT+oiPUKIXv5Pm1z7PNnB06TJXdUvKXgU5wEJ5VlUp6wnxdc7E+cvzFBQDl82 fOTouQg4X1KlhFg5Vu3jhvPp5YjV7YOUyzc7/7KfbIShYEeuiaZzRLoz53orA7L+uUD9pwZWqrd NxkR2Kklw2/Fr4F9CoONlc+dqfsrVLZWXzTsQAv6Oq4b98B80X1ReHhPnspRNW6lmLWBlNuffvC i3ILUYMENOOM7f5ddSVXFegY3zHbzSRuMVBKb5vJ7HbxmwpABgUamSYpotw/EeFqpiBLJt0X X-Google-Smtp-Source: AGHT+IHfdYPb0mTyQBEggcIEtJXMdJHS98DVkuG5qNfvvUIjwBW9edwqUb5MNEzj4f0Vk/I+xU8TBA== X-Received: by 2002:a17:903:3203:b0:215:58be:3349 with SMTP id d9443c01a7336-22c24987312mr27942235ad.14.1744685296075; Mon, 14 Apr 2025 19:48:16 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.48.11 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:48:15 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 26/28] mm: memcontrol: introduce memcg_reparent_ops Date: Tue, 15 Apr 2025 10:45:30 +0800 Message-Id: <20250415024532.26632-27-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 38CE320009 X-Stat-Signature: 7nza646ycoyki9h81pyxy58sn5y573qa X-HE-Tag: 1744685297-12912 X-HE-Meta: U2FsdGVkX1/f1qE/g92qKe4kKTJorn1lRRCvwk++bkL8xs2pJqDP575M8wMQxHSIQtGhDpRAKdKa63Y2ER9cKfAAzgKi1H7KcF8wDkuDf78o4ZHBWUARGyTYTA+5e+n5VcSDZQn23TUwBfbrXJEfJnDecDurIAmHmgE07kkXOLrRLeiQ99uTTGBLHvHatNOMRoyX6jcTGu2sOZzQfT5cKjuTsvBFaCfTeMvu6Gtktakf+S81sSZ+SGsboboRE5ZF9vvjX5wOhcJT9w3gznLcrkRd+Cs+RiJmmQe+cJbJ70DwWkiQvT1vSr5yt/SpK60Seaa+xoQ8x+b/ldKl8w3T6CLAM2Q3fcWWE+cCIP4WzvmyLhhXe+mJfBK0/Nbe8JaRiS7ft1xlxmHXHvtoEsdkyM/SzfyQgbhpScggmDVD98hNH6SaelV/qa/FZCp1g0145pNczEZ4dSiCqVwQ/FKuwNCo0IJNErh8mOLxpNcxvlmHm/hLrI/a6N2T1CvjKBs6fZDhWiB2Ji9zDr5C8LLMzRD55TONFEFv0fPrbe4cJi8Qj54d3HDPJngsioFlBjzTrK5SaOHbFTPdqzBp2JHqpEyK0Qal4eHXk7BYtCDBAJ03xZTk36IaJar/1P9KEfARkYn3E37iPkXadKirfcWdIP8+qKxXlQE+IT/fgkFXPcmgy7YB6bxioI3vWAOT42HSoFruRkKo/N5LxqaEcP8+nIhWoSUdMygRHzLc+xhG0OVWkxC/yQhPAUVV+0ganZ3iPG1XjwxGZ0V1akWQSueC9UozNK/wfUZpsBPv4NnWmEquRyIgtx7Z5fo60m/+bkqP6NRna2VrUcRWnqPRPCklj8W6nzvLJCewTd3WDXgLIDaMt2W5rGzGl1s1msR+LX+M72d3KBYFKMpGjtuHcxZgXVPZh+rzrZe6HIPpvtPcia1HT40M92Mhi7Z1CUuYbh5CWdxbpknuRwlXJIi9JT5 lc6TY0XB YeEAoh8qvK+7Ro1Xf7eLJj0ox7UpYmUKZnf+ynntRahzOdviAuJ1j8XBGc4lB8Wl98Jq+5NEDUFfekegZrk93UkwlrPrRRpsL5UDkBi+Yb/XtZ3ggdNFGqBJbkErleex8bxiEDaV7S9bPJxuY442ziLp3i/3R/qGj3wcDXumx1tT2CnL/hn6uWbm9LOHx1NVCxKFSjobVFugTIlvJxCoyBe5u/Ppd+/E4L7ZalWNRL9hQ52L3ojXrJyVG+vdyvV0D5loDSgwPVXhJrsVgDcsheZKsAA3J63XFAeVW7MR+1IUaija+V6W5xUXBlK5YHnO5rWMtuKIlRY2ULYXJxibiF0eTDXsqsFpox2V7gdu6aDNk9vU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the previous patch, we established a method to ensure the safety of the lruvec lock and the split queue lock during the reparenting of LRU folios. The process involves the following steps: memcg_reparent_objcgs(memcg) 1) lock // lruvec belongs to memcg and lruvec_parent belongs to parent memcg. spin_lock(&lruvec->lru_lock); spin_lock(&lruvec_parent->lru_lock); 2) relocate from current memcg to its parent // Move all the pages from the lruvec list to the parent lruvec list. 3) unlock spin_unlock(&lruvec_parent->lru_lock); spin_unlock(&lruvec->lru_lock); In addition to the folio lruvec lock, the deferred split queue lock (specific to THP) also requires a similar approach. Therefore, we abstract the three essential steps from the memcg_reparent_objcgs() function. memcg_reparent_objcgs(memcg) 1) lock memcg_reparent_ops->lock(memcg, parent); 2) relocate memcg_reparent_ops->relocate(memcg, reparent); 3) unlock memcg_reparent_ops->unlock(memcg, reparent); Currently, two distinct locks (such as the lruvec lock and the deferred split queue lock) need to utilize this infrastructure. In the subsequent patch, we will employ these APIs to ensure the safety of these locks during the reparenting of LRU folios. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 20 ++++++++++++ mm/memcontrol.c | 62 ++++++++++++++++++++++++++++++-------- 2 files changed, 69 insertions(+), 13 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 27b23e464229..0e450623f8fa 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -311,6 +311,26 @@ struct mem_cgroup { struct mem_cgroup_per_node *nodeinfo[]; }; +struct memcg_reparent_ops { + /* + * Note that interrupt is disabled before calling those callbacks, + * so the interrupt should remain disabled when leaving those callbacks. + */ + void (*lock)(struct mem_cgroup *src, struct mem_cgroup *dst); + void (*relocate)(struct mem_cgroup *src, struct mem_cgroup *dst); + void (*unlock)(struct mem_cgroup *src, struct mem_cgroup *dst); +}; + +#define DEFINE_MEMCG_REPARENT_OPS(name) \ + const struct memcg_reparent_ops memcg_##name##_reparent_ops = { \ + .lock = name##_reparent_lock, \ + .relocate = name##_reparent_relocate, \ + .unlock = name##_reparent_unlock, \ + } + +#define DECLARE_MEMCG_REPARENT_OPS(name) \ + extern const struct memcg_reparent_ops memcg_##name##_reparent_ops + /* * size of first charge trial. * TODO: maybe necessary to use big numbers in big irons or dynamic based of the diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1f0c6e7b69cc..3fac51179186 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -194,24 +194,60 @@ static struct obj_cgroup *obj_cgroup_alloc(void) return objcg; } -static void memcg_reparent_objcgs(struct mem_cgroup *memcg) +static void objcg_reparent_lock(struct mem_cgroup *src, struct mem_cgroup *dst) +{ + spin_lock(&objcg_lock); +} + +static void objcg_reparent_relocate(struct mem_cgroup *src, struct mem_cgroup *dst) { struct obj_cgroup *objcg, *iter; - struct mem_cgroup *parent = parent_mem_cgroup(memcg); - objcg = rcu_replace_pointer(memcg->objcg, NULL, true); + objcg = rcu_replace_pointer(src->objcg, NULL, true); + /* 1) Ready to reparent active objcg. */ + list_add(&objcg->list, &src->objcg_list); + /* 2) Reparent active objcg and already reparented objcgs to dst. */ + list_for_each_entry(iter, &src->objcg_list, list) + WRITE_ONCE(iter->memcg, dst); + /* 3) Move already reparented objcgs to the dst's list */ + list_splice(&src->objcg_list, &dst->objcg_list); +} - spin_lock_irq(&objcg_lock); +static void objcg_reparent_unlock(struct mem_cgroup *src, struct mem_cgroup *dst) +{ + spin_unlock(&objcg_lock); +} - /* 1) Ready to reparent active objcg. */ - list_add(&objcg->list, &memcg->objcg_list); - /* 2) Reparent active objcg and already reparented objcgs to parent. */ - list_for_each_entry(iter, &memcg->objcg_list, list) - WRITE_ONCE(iter->memcg, parent); - /* 3) Move already reparented objcgs to the parent's list */ - list_splice(&memcg->objcg_list, &parent->objcg_list); - - spin_unlock_irq(&objcg_lock); +static DEFINE_MEMCG_REPARENT_OPS(objcg); + +static const struct memcg_reparent_ops *memcg_reparent_ops[] = { + &memcg_objcg_reparent_ops, +}; + +#define DEFINE_MEMCG_REPARENT_FUNC(phase) \ + static void memcg_reparent_##phase(struct mem_cgroup *src, \ + struct mem_cgroup *dst) \ + { \ + int i; \ + \ + for (i = 0; i < ARRAY_SIZE(memcg_reparent_ops); i++) \ + memcg_reparent_ops[i]->phase(src, dst); \ + } + +DEFINE_MEMCG_REPARENT_FUNC(lock) +DEFINE_MEMCG_REPARENT_FUNC(relocate) +DEFINE_MEMCG_REPARENT_FUNC(unlock) + +static void memcg_reparent_objcgs(struct mem_cgroup *src) +{ + struct mem_cgroup *dst = parent_mem_cgroup(src); + struct obj_cgroup *objcg = rcu_dereference_protected(src->objcg, true); + + local_irq_disable(); + memcg_reparent_lock(src, dst); + memcg_reparent_relocate(src, dst); + memcg_reparent_unlock(src, dst); + local_irq_enable(); percpu_ref_kill(&objcg->refcnt); } From patchwork Tue Apr 15 02:45:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051381 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91D37C369B2 for ; Tue, 15 Apr 2025 02:48:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 053F8280126; Mon, 14 Apr 2025 22:48:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 000EE280073; Mon, 14 Apr 2025 22:48:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D480E280126; Mon, 14 Apr 2025 22:48:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AAFC7280073 for ; Mon, 14 Apr 2025 22:48:23 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 749D31C6A78 for ; Tue, 15 Apr 2025 02:48:24 +0000 (UTC) X-FDA: 83334744528.10.ED3A138 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) by imf30.hostedemail.com (Postfix) with ESMTP id 836D48000B for ; Tue, 15 Apr 2025 02:48:22 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="OzDUP/92"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf30.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.52 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685302; a=rsa-sha256; cv=none; b=EQz5jft3snU8xgCFHiQ2BxHKfQbWxkRMpJnX7eWqx8ZtmjuHgLongtLTV/DiE2qtLZRHg1 vRElLx/atoNF9mVOF1MSt+Yca7sV1GSHdVc6h1ivKg9j+KIjqibwXWyCt96xYVxVvUTTan ysmWgy4djX0BfQX/oPEPIRAAwVT+uQ8= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="OzDUP/92"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf30.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.52 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685302; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/5hAkTeajZjUlhSwkD9bqooCoCo2zpB/VByckb93fwU=; b=1H5tTbKIaEwI61TfhJxmTcTNWHS6rPJiMBTBWkXfPBBf91IdS/EX0VZbwx7pvQg+f7h/oD lMRAVeYvXr0ug/hXQmwzg1LeByrcRSpEQ7GSpQG35vh/ckywyXwQLE6Ciznw5EynJlvaiy XmpP9g7CogJie05aiPf2zBJ6bAkz2BE= Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-2ff615a114bso5113039a91.0 for ; Mon, 14 Apr 2025 19:48:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685301; x=1745290101; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/5hAkTeajZjUlhSwkD9bqooCoCo2zpB/VByckb93fwU=; b=OzDUP/92c3ZBqEXnOl4W11zZRlQmmqK7Z8z5ewkTqol/PbJB/zfTtXTIN8WTtHvWGe myjsDGLN7BbfI0ygSTOsDNPy3h+5QnqdZrLRkJ1lalpzTd13xDBHHrhRKCEJ9ZgjWZXG Cff3+ggXW+JN1+5GurRH/Itf/z3EfHvlzLIj8GGa7xGh8f030MTqUCvEPD4iCxo2aEgI ihbmVFsGHQoKTcGiHl61otYadI+Fkgvt1Jl/6XgVX08wKsquBP9mGyLr6pZeYHpp15dZ 8v+8OO8v5LYHAk+HFugEerlBYg49AUPCel8TfToVDPPTsDu8aPHio2rCxqrMnvGOLUYY t/ow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685301; x=1745290101; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/5hAkTeajZjUlhSwkD9bqooCoCo2zpB/VByckb93fwU=; b=EwH/gLSlsZcF1xnRCQhjPmJbpeIrGCrohUDblYhYBTcS4CP7QSSCBjp5Hx/eGQoPOP vNyzfY+qdknlsobZJyfHmAvQ6+itkPgVG1SSInOsxdit58TjttQU/5CVSyEzSodMHWXR S29sgj9U4pChZqJfKc5AKoqICxjlTeGZnqe3LMnzzsC7nc1RlJLI4ay/SvBT8hejFjsu +m15mPTPag2PoBtEpUhEHloneyF5qRRTcsmzECBsj/rgkeaVIZPM/OW5krzBWN0HIVWe JQ8uO3yT2kgfRYaV8ZXdXj4fnqT4oBZFDBn6Ap3vZYjio/ou+h3x7vxDBBPFuBlqwfbi 7vew== X-Forwarded-Encrypted: i=1; AJvYcCW0pnMHXV0DbQs6hzbuBxfRRQd1RYB5T3TqRE6VaS07uzrACZ4tBt2yodyFNKfWG1U/W7mTLL7kCw==@kvack.org X-Gm-Message-State: AOJu0YyD4J5M9HiEZ4/bj5P0AS6v3qynnetGUcHoZ7CoA8oh2Ju04W9n qOmUsUqerechq0SIOIxKEbtKcMe438RFCFTKj7P56+BkCT4Zxgor7KKh1ph4k3I= X-Gm-Gg: ASbGncsY/o65UO4yWT43X6LNbBAjKQL4WR/N2qP2VhQS36vWhbQ0ObOjIeDSxAwG3lZ GGkU2MBgSccLgsyptlWoJfGLqrQLVVLLa75RdE7sd/zKzwS+RrsD/JEpgwrZEWyzT7LPxSedQrO nLmkR3OaMEevbZgU18jM5g9Vf4cfrVfdddoxTQf2ZUfgTlFhp8e9Wd78PKzPsaSSYyMInDXOBq/ nvUmCeyDkfEdrF8S6gcDxcy3PPtBqi9T+wbZc2yAcmVszo+S7w77+j6f0HAEsllNmzv2TfIIVPT DKAnvU1aqvFNpfdqj/KyzL9IvH133+sGxmUTdMIShbWGsh8HxyDbYUKWVEBOqTQOPfrYgr6/ X-Google-Smtp-Source: AGHT+IHCiWoDN+bE/e12Bjpe86h+z5c2l97IYpJx1vPLZfIiCxp0ezSHDN9cZFtMGTkakWuJkAZr7g== X-Received: by 2002:a17:90b:2dcd:b0:306:b593:455e with SMTP id 98e67ed59e1d1-3084f306f84mr2571647a91.1.1744685301315; Mon, 14 Apr 2025 19:48:21 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.48.16 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:48:20 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 27/28] mm: memcontrol: eliminate the problem of dying memory cgroup for LRU folios Date: Tue, 15 Apr 2025 10:45:31 +0800 Message-Id: <20250415024532.26632-28-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 836D48000B X-Rspamd-Server: rspam04 X-Stat-Signature: 1jubcpx9siy7y9363tzmqnubkgy9mm5o X-HE-Tag: 1744685302-708922 X-HE-Meta: U2FsdGVkX1+d96+Dg7BobaJi1CXd6h2pgN5FJBD3kSYTKVKLW5ijZ/1EWmsIqHsHFm1f0WR12H0/le//5dmUy8GlpYFMuBmGlNuilf/3mwFWITJqUw/0QiKdr/FChK4UoT/TVOMTt47g5MSr1Yamgx+PLkBRouJvsezf3m9yFnCOvZ6WlxvngtQoxmpPtH/QXiIouTF8nW+5sIXybvtrstGng+Y4tr7+SN/RzMxnGvLGjnjby3Raz4T3c6dNk0oiyNoEyAJXXc2L0GBAY212jzvUDRvyzYY7lmagxgD+aQDaiAkYAlHB1GI+Gm1WlW7/r2hxD7cPFY+GWGKfBMeUF3JuWm3/cerEAtQ4tpN7ImmkO7QqVnYpyj+uCt6b5T4R4Ez3n6OaM+ZD2N3zDJKk/s/7FCS7lZi6P6mMe9IbEvt6AQWXGEFFs5qjR8ii/mEBw2vqmCmQoX0mW5SQQp9bbRT1EfbM6rOSWhkPPBve0nUi+oU3lOuoyg8xrz4u6LIFQAcNwtlyERb/muG19isMeG0y/TvZGHaw6rYYin2krcnO6/F0e9ySAkxI5wN0rmq5fBSvMRUYa4CwTXzoeOYBUi1P0ssYoKsM1+MNUq7/6CexI/mm8NRfgVCLdTAMaVhBGUIdtvMrp61wcOttgVVrlCgaAgtKFzVryPldmqWZOapZHOLfgQz5v3BLloU37aQWo1A8VBOsnwZFre7Qnvf8BSbGzXTj1aWAa5MV8qj1zxDtgBcTQrlxbIIPADzJgx/euuMiX7ZArAY0XiH8KriD3GSS5eBz2Bn0nvNfujOz1eOh/xNZvsHEprJmDqAD9mPAIbkDHFPqfJTFJUdRb65DWQ7R/k/00JQN9OepwgcVfq+MDn6SIDyyBTHKql0mGFcTSmpLL+5gdy3nt4EQm3VKiJjyYQYGxHyU6ogD6oGd6B+cZtR4CNRP+9KQhCVafzdVeP7DQfb3qD8rOTsZ//3 JlcPHGck 6T6tD3kWBGos6m4G+P3wpzNagRgeP2zYWXGoPS3gJRzGEA305ynzwmOStLFu7MGd5Wj4dfgp6QkkrE6F82SpRoIF/fQQI7+69IMPaqv9COhKs125oGwNB4VnjYw7eDE4U+VRkSIpbRgNTMNoRisiqqvxl7h2PMqDnPRPIxFXgZ5t3PR0nxra6utBx2pHOjMvjdrS0wee+wbpGDHh/ZYkvmIzWvk8fYOAEOUtJj177VT20aJf7jGGCEVwhudw3Qhoai5sfZQNDBYAG5kTANtum3PK6Eoy/Qx4lUMFOYMh3hT3cr8Zm5kxvz35p5Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Pagecache pages are charged at allocation time and hold a reference to the original memory cgroup until reclaimed. Depending on memory pressure, page sharing patterns between different cgroups and cgroup creation/destruction rates, many dying memory cgroups can be pinned by pagecache pages, reducing page reclaim efficiency and wasting memory. Converting LRU folios and most other raw memory cgroup pins to the object cgroup direction can fix this long-living problem. Finally, folio->memcg_data of LRU folios and kmem folios will always point to an object cgroup pointer. The folio->memcg_data of slab folios will point to an vector of object cgroups. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 78 +++++-------- mm/huge_memory.c | 33 ++++++ mm/memcontrol-v1.c | 15 ++- mm/memcontrol.c | 228 +++++++++++++++++++++++++------------ 4 files changed, 222 insertions(+), 132 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 0e450623f8fa..7b1279963c0c 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -368,9 +368,6 @@ enum objext_flags { #define OBJEXTS_FLAGS_MASK (__NR_OBJEXTS_FLAGS - 1) #ifdef CONFIG_MEMCG - -static inline bool folio_memcg_kmem(struct folio *folio); - /* * After the initialization objcg->memcg is always pointing at * a valid memcg, but can be atomically swapped to the parent memcg. @@ -384,43 +381,19 @@ static inline struct mem_cgroup *obj_cgroup_memcg(struct obj_cgroup *objcg) } /* - * __folio_memcg - Get the memory cgroup associated with a non-kmem folio - * @folio: Pointer to the folio. - * - * Returns a pointer to the memory cgroup associated with the folio, - * or NULL. This function assumes that the folio is known to have a - * proper memory cgroup pointer. It's not safe to call this function - * against some type of folios, e.g. slab folios or ex-slab folios or - * kmem folios. - */ -static inline struct mem_cgroup *__folio_memcg(struct folio *folio) -{ - unsigned long memcg_data = folio->memcg_data; - - VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); - VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); - VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_KMEM, folio); - - return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); -} - -/* - * __folio_objcg - get the object cgroup associated with a kmem folio. + * folio_objcg - get the object cgroup associated with a folio. * @folio: Pointer to the folio. * * Returns a pointer to the object cgroup associated with the folio, * or NULL. This function assumes that the folio is known to have a - * proper object cgroup pointer. It's not safe to call this function - * against some type of folios, e.g. slab folios or ex-slab folios or - * LRU folios. + * proper object cgroup pointer. */ -static inline struct obj_cgroup *__folio_objcg(struct folio *folio) +static inline struct obj_cgroup *folio_objcg(struct folio *folio) { unsigned long memcg_data = folio->memcg_data; VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); - VM_BUG_ON_FOLIO(!(memcg_data & MEMCG_DATA_KMEM), folio); return (struct obj_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } @@ -434,21 +407,31 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio) * proper memory cgroup pointer. It's not safe to call this function * against some type of folios, e.g. slab folios or ex-slab folios. * - * For a non-kmem folio any of the following ensures folio and memcg binding - * stability: + * For a folio any of the following ensures folio and objcg binding stability: * * - the folio lock * - LRU isolation * - exclusive reference * - * For a kmem folio a caller should hold an rcu read lock to protect memcg - * associated with a kmem folio from being released. + * Based on the stable binding of folio and objcg, for a folio any of the + * following ensures folio and memcg binding stability: + * + * - cgroup_mutex + * - the lruvec lock + * - the split queue lock (only THP page) + * + * If the caller only want to ensure that the page counters of memcg are + * updated correctly, ensure that the binding stability of folio and objcg + * is sufficient. + * + * Note: The caller should hold an rcu read lock or cgroup_mutex to protect + * memcg associated with a folio from being released. */ static inline struct mem_cgroup *folio_memcg(struct folio *folio) { - if (folio_memcg_kmem(folio)) - return obj_cgroup_memcg(__folio_objcg(folio)); - return __folio_memcg(folio); + struct obj_cgroup *objcg = folio_objcg(folio); + + return objcg ? obj_cgroup_memcg(objcg) : NULL; } /* @@ -472,15 +455,10 @@ static inline bool folio_memcg_charged(struct folio *folio) * has an associated memory cgroup pointer or an object cgroups vector or * an object cgroup. * - * For a non-kmem folio any of the following ensures folio and memcg binding - * stability: + * The page and objcg or memcg binding rules can refer to folio_memcg(). * - * - the folio lock - * - LRU isolation - * - exclusive reference - * - * For a kmem folio a caller should hold an rcu read lock to protect memcg - * associated with a kmem folio from being released. + * A caller should hold an rcu read lock to protect memcg associated with a + * page from being released. */ static inline struct mem_cgroup *folio_memcg_check(struct folio *folio) { @@ -489,18 +467,14 @@ static inline struct mem_cgroup *folio_memcg_check(struct folio *folio) * for slabs, READ_ONCE() should be used here. */ unsigned long memcg_data = READ_ONCE(folio->memcg_data); + struct obj_cgroup *objcg; if (memcg_data & MEMCG_DATA_OBJEXTS) return NULL; - if (memcg_data & MEMCG_DATA_KMEM) { - struct obj_cgroup *objcg; - - objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK); - return obj_cgroup_memcg(objcg); - } + objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK); - return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); + return objcg ? obj_cgroup_memcg(objcg) : NULL; } static inline struct mem_cgroup *page_memcg_check(struct page *page) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 813334994f84..0236020de5b3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1075,6 +1075,39 @@ static inline struct deferred_split *folio_memcg_split_queue(struct folio *folio return memcg ? &memcg->deferred_split_queue : NULL; } + +static void thp_sq_reparent_lock(struct mem_cgroup *src, struct mem_cgroup *dst) +{ + spin_lock(&src->deferred_split_queue.split_queue_lock); + spin_lock_nested(&dst->deferred_split_queue.split_queue_lock, + SINGLE_DEPTH_NESTING); +} + +static void thp_sq_reparent_relocate(struct mem_cgroup *src, struct mem_cgroup *dst) +{ + int nid; + struct deferred_split *src_queue, *dst_queue; + + src_queue = &src->deferred_split_queue; + dst_queue = &dst->deferred_split_queue; + + if (!src_queue->split_queue_len) + return; + + list_splice_tail_init(&src_queue->split_queue, &dst_queue->split_queue); + dst_queue->split_queue_len += src_queue->split_queue_len; + src_queue->split_queue_len = 0; + + for_each_node(nid) + set_shrinker_bit(dst, nid, deferred_split_shrinker->id); +} + +static void thp_sq_reparent_unlock(struct mem_cgroup *src, struct mem_cgroup *dst) +{ + spin_unlock(&dst->deferred_split_queue.split_queue_lock); + spin_unlock(&src->deferred_split_queue.split_queue_lock); +} +DEFINE_MEMCG_REPARENT_OPS(thp_sq); #else static inline struct mem_cgroup *folio_split_queue_memcg(struct folio *folio, diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c index 8660908850dc..fb060e5c28ca 100644 --- a/mm/memcontrol-v1.c +++ b/mm/memcontrol-v1.c @@ -591,6 +591,7 @@ void memcg1_commit_charge(struct folio *folio, struct mem_cgroup *memcg) void memcg1_swapout(struct folio *folio, swp_entry_t entry) { struct mem_cgroup *memcg, *swap_memcg; + struct obj_cgroup *objcg; unsigned int nr_entries; VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); @@ -602,12 +603,13 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry) if (!do_memsw_account()) return; - memcg = folio_memcg(folio); - - VM_WARN_ON_ONCE_FOLIO(!memcg, folio); - if (!memcg) + objcg = folio_objcg(folio); + VM_WARN_ON_ONCE_FOLIO(!objcg, folio); + if (!objcg) return; + rcu_read_lock(); + memcg = obj_cgroup_memcg(objcg); /* * In case the memcg owning these pages has been offlined and doesn't * have an ID allocated to it anymore, charge the closest online @@ -625,7 +627,7 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry) folio_unqueue_deferred_split(folio); folio->memcg_data = 0; - if (!mem_cgroup_is_root(memcg)) + if (!obj_cgroup_is_root(objcg)) page_counter_uncharge(&memcg->memory, nr_entries); if (memcg != swap_memcg) { @@ -646,7 +648,8 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry) preempt_enable_nested(); memcg1_check_events(memcg, folio_nid(folio)); - css_put(&memcg->css); + rcu_read_unlock(); + obj_cgroup_put(objcg); } /* diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 3fac51179186..1381a9e97ec5 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -220,8 +220,78 @@ static void objcg_reparent_unlock(struct mem_cgroup *src, struct mem_cgroup *dst static DEFINE_MEMCG_REPARENT_OPS(objcg); +static void lruvec_reparent_lock(struct mem_cgroup *src, struct mem_cgroup *dst) +{ + int nid, nest = 0; + + for_each_node(nid) { + spin_lock_nested(&mem_cgroup_lruvec(src, + NODE_DATA(nid))->lru_lock, nest++); + spin_lock_nested(&mem_cgroup_lruvec(dst, + NODE_DATA(nid))->lru_lock, nest++); + } +} + +static void lruvec_reparent_lru(struct lruvec *src, struct lruvec *dst, + enum lru_list lru) +{ + int zid; + struct mem_cgroup_per_node *mz_src, *mz_dst; + + mz_src = container_of(src, struct mem_cgroup_per_node, lruvec); + mz_dst = container_of(dst, struct mem_cgroup_per_node, lruvec); + + if (lru != LRU_UNEVICTABLE) + list_splice_tail_init(&src->lists[lru], &dst->lists[lru]); + + for (zid = 0; zid < MAX_NR_ZONES; zid++) { + mz_dst->lru_zone_size[zid][lru] += mz_src->lru_zone_size[zid][lru]; + mz_src->lru_zone_size[zid][lru] = 0; + } +} + +static void lruvec_reparent_relocate(struct mem_cgroup *src, struct mem_cgroup *dst) +{ + int nid; + + for_each_node(nid) { + enum lru_list lru; + struct lruvec *src_lruvec, *dst_lruvec; + + src_lruvec = mem_cgroup_lruvec(src, NODE_DATA(nid)); + dst_lruvec = mem_cgroup_lruvec(dst, NODE_DATA(nid)); + + dst_lruvec->anon_cost += src_lruvec->anon_cost; + dst_lruvec->file_cost += src_lruvec->file_cost; + + for_each_lru(lru) + lruvec_reparent_lru(src_lruvec, dst_lruvec, lru); + } +} + +static void lruvec_reparent_unlock(struct mem_cgroup *src, struct mem_cgroup *dst) +{ + int nid; + + for_each_node(nid) { + spin_unlock(&mem_cgroup_lruvec(dst, NODE_DATA(nid))->lru_lock); + spin_unlock(&mem_cgroup_lruvec(src, NODE_DATA(nid))->lru_lock); + } +} + +static DEFINE_MEMCG_REPARENT_OPS(lruvec); + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +DECLARE_MEMCG_REPARENT_OPS(thp_sq); +#endif + +/* The lock order depends on the order of elements in this array. */ static const struct memcg_reparent_ops *memcg_reparent_ops[] = { &memcg_objcg_reparent_ops, + &memcg_lruvec_reparent_ops, +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + &memcg_thp_sq_reparent_ops, +#endif }; #define DEFINE_MEMCG_REPARENT_FUNC(phase) \ @@ -1018,6 +1088,8 @@ struct mem_cgroup *get_mem_cgroup_from_current(void) /** * get_mem_cgroup_from_folio - Obtain a reference on a given folio's memcg. * @folio: folio from which memcg should be extracted. + * + * The page and objcg or memcg binding rules can refer to folio_memcg(). */ struct mem_cgroup *get_mem_cgroup_from_folio(struct folio *folio) { @@ -2489,17 +2561,17 @@ static inline int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask, return try_charge_memcg(memcg, gfp_mask, nr_pages); } -static void commit_charge(struct folio *folio, struct mem_cgroup *memcg) +static void commit_charge(struct folio *folio, struct obj_cgroup *objcg) { VM_BUG_ON_FOLIO(folio_memcg_charged(folio), folio); /* - * Any of the following ensures page's memcg stability: + * Any of the following ensures page's objcg stability: * * - the page lock * - LRU isolation * - exclusive reference */ - folio->memcg_data = (unsigned long)memcg; + folio->memcg_data = (unsigned long)objcg; } static inline void __mod_objcg_mlstate(struct obj_cgroup *objcg, @@ -2580,6 +2652,17 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) return NULL; } +static inline struct obj_cgroup *get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) +{ + struct obj_cgroup *objcg; + + rcu_read_lock(); + objcg = __get_obj_cgroup_from_memcg(memcg); + rcu_read_unlock(); + + return objcg; +} + static struct obj_cgroup *current_objcg_update(void) { struct mem_cgroup *memcg; @@ -2677,17 +2760,10 @@ struct obj_cgroup *get_obj_cgroup_from_folio(struct folio *folio) { struct obj_cgroup *objcg; - if (!memcg_kmem_online()) - return NULL; - - if (folio_memcg_kmem(folio)) { - objcg = __folio_objcg(folio); + objcg = folio_objcg(folio); + if (objcg) obj_cgroup_get(objcg); - } else { - rcu_read_lock(); - objcg = __get_obj_cgroup_from_memcg(__folio_memcg(folio)); - rcu_read_unlock(); - } + return objcg; } @@ -3168,7 +3244,7 @@ void folio_split_memcg_refs(struct folio *folio, unsigned old_order, return; new_refs = (1 << (old_order - new_order)) - 1; - css_get_many(&__folio_memcg(folio)->css, new_refs); + obj_cgroup_get_many(folio_objcg(folio), new_refs); } unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) @@ -4616,16 +4692,20 @@ void mem_cgroup_calculate_protection(struct mem_cgroup *root, static int charge_memcg(struct folio *folio, struct mem_cgroup *memcg, gfp_t gfp) { - int ret; - - ret = try_charge(memcg, gfp, folio_nr_pages(folio)); - if (ret) - goto out; + int ret = 0; + struct obj_cgroup *objcg; - css_get(&memcg->css); - commit_charge(folio, memcg); + objcg = get_obj_cgroup_from_memcg(memcg); + /* Do not account at the root objcg level. */ + if (!obj_cgroup_is_root(objcg)) + ret = try_charge(memcg, gfp, folio_nr_pages(folio)); + if (ret) { + obj_cgroup_put(objcg); + return ret; + } + commit_charge(folio, objcg); memcg1_commit_charge(folio, memcg); -out: + return ret; } @@ -4711,7 +4791,7 @@ int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm, } struct uncharge_gather { - struct mem_cgroup *memcg; + struct obj_cgroup *objcg; unsigned long nr_memory; unsigned long pgpgout; unsigned long nr_kmem; @@ -4725,60 +4805,54 @@ static inline void uncharge_gather_clear(struct uncharge_gather *ug) static void uncharge_batch(const struct uncharge_gather *ug) { + struct mem_cgroup *memcg; + + rcu_read_lock(); + memcg = obj_cgroup_memcg(ug->objcg); if (ug->nr_memory) { - page_counter_uncharge(&ug->memcg->memory, ug->nr_memory); + page_counter_uncharge(&memcg->memory, ug->nr_memory); if (do_memsw_account()) - page_counter_uncharge(&ug->memcg->memsw, ug->nr_memory); + page_counter_uncharge(&memcg->memsw, ug->nr_memory); if (ug->nr_kmem) { - mod_memcg_state(ug->memcg, MEMCG_KMEM, -ug->nr_kmem); - memcg1_account_kmem(ug->memcg, -ug->nr_kmem); + mod_memcg_state(memcg, MEMCG_KMEM, -ug->nr_kmem); + memcg1_account_kmem(memcg, -ug->nr_kmem); } - memcg1_oom_recover(ug->memcg); + memcg1_oom_recover(memcg); } - memcg1_uncharge_batch(ug->memcg, ug->pgpgout, ug->nr_memory, ug->nid); + memcg1_uncharge_batch(memcg, ug->pgpgout, ug->nr_memory, ug->nid); + rcu_read_unlock(); /* drop reference from uncharge_folio */ - css_put(&ug->memcg->css); + obj_cgroup_put(ug->objcg); } static void uncharge_folio(struct folio *folio, struct uncharge_gather *ug) { long nr_pages; - struct mem_cgroup *memcg; struct obj_cgroup *objcg; VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); /* * Nobody should be changing or seriously looking at - * folio memcg or objcg at this point, we have fully - * exclusive access to the folio. + * folio objcg at this point, we have fully exclusive + * access to the folio. */ - if (folio_memcg_kmem(folio)) { - objcg = __folio_objcg(folio); - /* - * This get matches the put at the end of the function and - * kmem pages do not hold memcg references anymore. - */ - memcg = get_mem_cgroup_from_objcg(objcg); - } else { - memcg = __folio_memcg(folio); - } - - if (!memcg) + objcg = folio_objcg(folio); + if (!objcg) return; - if (ug->memcg != memcg) { - if (ug->memcg) { + if (ug->objcg != objcg) { + if (ug->objcg) { uncharge_batch(ug); uncharge_gather_clear(ug); } - ug->memcg = memcg; + ug->objcg = objcg; ug->nid = folio_nid(folio); - /* pairs with css_put in uncharge_batch */ - css_get(&memcg->css); + /* pairs with obj_cgroup_put in uncharge_batch */ + obj_cgroup_get(objcg); } nr_pages = folio_nr_pages(folio); @@ -4786,20 +4860,17 @@ static void uncharge_folio(struct folio *folio, struct uncharge_gather *ug) if (folio_memcg_kmem(folio)) { ug->nr_memory += nr_pages; ug->nr_kmem += nr_pages; - - folio->memcg_data = 0; - obj_cgroup_put(objcg); } else { /* LRU pages aren't accounted at the root level */ - if (!mem_cgroup_is_root(memcg)) + if (!obj_cgroup_is_root(objcg)) ug->nr_memory += nr_pages; ug->pgpgout++; WARN_ON_ONCE(folio_unqueue_deferred_split(folio)); - folio->memcg_data = 0; } - css_put(&memcg->css); + folio->memcg_data = 0; + obj_cgroup_put(objcg); } void __mem_cgroup_uncharge(struct folio *folio) @@ -4823,7 +4894,7 @@ void __mem_cgroup_uncharge_folios(struct folio_batch *folios) uncharge_gather_clear(&ug); for (i = 0; i < folios->nr; i++) uncharge_folio(folios->folios[i], &ug); - if (ug.memcg) + if (ug.objcg) uncharge_batch(&ug); } @@ -4840,6 +4911,7 @@ void __mem_cgroup_uncharge_folios(struct folio_batch *folios) void mem_cgroup_replace_folio(struct folio *old, struct folio *new) { struct mem_cgroup *memcg; + struct obj_cgroup *objcg; long nr_pages = folio_nr_pages(new); VM_BUG_ON_FOLIO(!folio_test_locked(old), old); @@ -4854,21 +4926,24 @@ void mem_cgroup_replace_folio(struct folio *old, struct folio *new) if (folio_memcg_charged(new)) return; - memcg = folio_memcg(old); - VM_WARN_ON_ONCE_FOLIO(!memcg, old); - if (!memcg) + objcg = folio_objcg(old); + VM_WARN_ON_ONCE_FOLIO(!objcg, old); + if (!objcg) return; + rcu_read_lock(); + memcg = obj_cgroup_memcg(objcg); /* Force-charge the new page. The old one will be freed soon */ - if (!mem_cgroup_is_root(memcg)) { + if (!obj_cgroup_is_root(objcg)) { page_counter_charge(&memcg->memory, nr_pages); if (do_memsw_account()) page_counter_charge(&memcg->memsw, nr_pages); } - css_get(&memcg->css); - commit_charge(new, memcg); + obj_cgroup_get(objcg); + commit_charge(new, objcg); memcg1_commit_charge(new, memcg); + rcu_read_unlock(); } /** @@ -4884,7 +4959,7 @@ void mem_cgroup_replace_folio(struct folio *old, struct folio *new) */ void mem_cgroup_migrate(struct folio *old, struct folio *new) { - struct mem_cgroup *memcg; + struct obj_cgroup *objcg; VM_BUG_ON_FOLIO(!folio_test_locked(old), old); VM_BUG_ON_FOLIO(!folio_test_locked(new), new); @@ -4895,18 +4970,18 @@ void mem_cgroup_migrate(struct folio *old, struct folio *new) if (mem_cgroup_disabled()) return; - memcg = folio_memcg(old); + objcg = folio_objcg(old); /* - * Note that it is normal to see !memcg for a hugetlb folio. + * Note that it is normal to see !objcg for a hugetlb folio. * For e.g, itt could have been allocated when memory_hugetlb_accounting * was not selected. */ - VM_WARN_ON_ONCE_FOLIO(!folio_test_hugetlb(old) && !memcg, old); - if (!memcg) + VM_WARN_ON_ONCE_FOLIO(!folio_test_hugetlb(old) && !objcg, old); + if (!objcg) return; - /* Transfer the charge and the css ref */ - commit_charge(new, memcg); + /* Transfer the charge and the objcg ref */ + commit_charge(new, objcg); /* Warning should never happen, so don't worry about refcount non-0 */ WARN_ON_ONCE(folio_unqueue_deferred_split(old)); @@ -5049,22 +5124,27 @@ int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry) unsigned int nr_pages = folio_nr_pages(folio); struct page_counter *counter; struct mem_cgroup *memcg; + struct obj_cgroup *objcg; if (do_memsw_account()) return 0; - memcg = folio_memcg(folio); - - VM_WARN_ON_ONCE_FOLIO(!memcg, folio); - if (!memcg) + objcg = folio_objcg(folio); + VM_WARN_ON_ONCE_FOLIO(!objcg, folio); + if (!objcg) return 0; + rcu_read_lock(); + memcg = obj_cgroup_memcg(objcg); if (!entry.val) { memcg_memory_event(memcg, MEMCG_SWAP_FAIL); + rcu_read_unlock(); return 0; } memcg = mem_cgroup_id_get_online(memcg); + /* memcg is pined by memcg ID. */ + rcu_read_unlock(); if (!mem_cgroup_is_root(memcg) && !page_counter_try_charge(&memcg->swap, nr_pages, &counter)) { From patchwork Tue Apr 15 02:45:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051382 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 809C9C369B2 for ; Tue, 15 Apr 2025 02:48:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DF3142801C9; Mon, 14 Apr 2025 22:48:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D790F280154; Mon, 14 Apr 2025 22:48:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BFE0E2801C9; Mon, 14 Apr 2025 22:48:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 9C5B4280154 for ; Mon, 14 Apr 2025 22:48:28 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6F470BAE8A for ; Tue, 15 Apr 2025 02:48:29 +0000 (UTC) X-FDA: 83334744738.06.60E0AEA Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) by imf08.hostedemail.com (Postfix) with ESMTP id 9B715160002 for ; Tue, 15 Apr 2025 02:48:27 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=RiiXBcTd; spf=pass (imf08.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.49 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685307; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UXOKqlBcK7wmw0vjxin9AD5vy+d1LV8LiFHk+wC2Q/0=; b=nkUSkprH4jv7lxEzxEfIolQDlpqtVaw3OlpQQ8jtCnv4thgDzcXIW1eHQBn5rb43Lf4VGI o0+ef10CFXxsGAM9h1gnHG51une7f6ZhAU7hf451ydbG27QwK77/JUYsP5/21p7VYoFz6x Tz0tVJRX5/SRi2Ad7hQ+RKlnSzZLvgM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685307; a=rsa-sha256; cv=none; b=EWwXYwBfvSe7dZ7Dp+34wLbj4xwrDjAtGmqcRWi7Q53wxcFAuimGUg0kltjrytMdmNaczF yGWeIZxUp9buEYy6hhMC/kcZ5lLhnCwdn7vIfE6NKNHTrK1tQ96Y9eZtcVT7Kyv9AucLvp FC89eLSfHlUvLyWZE5JoC7alsuPeOjg= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=RiiXBcTd; spf=pass (imf08.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.49 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-30820167b47so4172841a91.0 for ; Mon, 14 Apr 2025 19:48:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685306; x=1745290106; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UXOKqlBcK7wmw0vjxin9AD5vy+d1LV8LiFHk+wC2Q/0=; b=RiiXBcTdA4KjKcoEuzZ8g4oDOuIMiJfiwccbCZSUZijW3LDht2fEkibs1z8BunAnfR OfrfjMhJnE4/VB61k55dnIz1kwz7K1W7yfl8g47Jt5JIHh4b7lwOUEIK7qXjLaR96J5w s+FNmHg2vR+XGh/7LZj2ZN8sHnfdNxBVIZMfqg3WW6RQkBTsIP5YXte2vaOtJuHVXmCa QnNFQs3cBh11o5nZVIzJS3amq+EtJaSInldfHIKQvBvJKIwRSFdDMawyQcdjsLnfMuoT 9Cgoyaay+AMg8sYvUtVOW7RrytB+hMcWpYVro+xABrr5Q8K2Jy2MAx2ZUFPRg/DflkC9 6+jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685306; x=1745290106; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UXOKqlBcK7wmw0vjxin9AD5vy+d1LV8LiFHk+wC2Q/0=; b=q5rScQaMllNtFrZlM4UKjfHJiYrpYC4eZcqtx9swtfQG/hu3K87ZrVSop2MOL7TP4O ZLp3HTUdUEqdCAO3WMJMXcAp9eQnc0aQGcwLBQrXeH1L7w0efcfZS3yLziO1aN6Sp05u tVRvZoxiofzY3sCljJZ1S15AURGrITlAdHujoFuc6nBYYNdZ+e+TdeAnbFCVUNUZA9kI akwZV8faalU0MHKqlxZPay6sa4Xrfb/3mL16tTIHXvh6nhUY4fwhHx3cA1Hq2A0U/3fm E11u+ELuMiKmysd3FmdNjYxpbSJW9sCuCwJ6apKpaRbY1gYSBgIrqFskEsD29QctmaTF itqw== X-Forwarded-Encrypted: i=1; AJvYcCXX5J96QT5JZIbMGsHtY4Cq2iMC82w6SHwEsKRW5IKxUCCnQg58SUBxTLlmK5nFcj0jw4u1vgFf0A==@kvack.org X-Gm-Message-State: AOJu0YyOdYdMCQUaIEI79nSwAw7Hflqi4gyXWG7JKeCOK9tZT/uU/luY AbbTSF7N4RSgzKo/GKo/YbV3X9xGZHfzyJMWCaErhwTUjGE4lt++4NSj/WNEtIE= X-Gm-Gg: ASbGncsokAam/+73EnfFUTgFrKsyM+3FkK7KJh3txb1cMgZfklBH1IkXCkyqpp+4DAT 5CXvjZRdK/2c49qUjG+Pj5q02W/riiAeNyfc7GoayK7mG3gInkLOzv5mlyExpyTR9YMciG9u3mu nJf9cbn5asND2AtWg1cF+3mB8GvBNikiOpwr5yYjkZvkdVd4ogc4dPSU6FLDmoVgsKmHPGRGL+s gF88IIjvMo2btMlTE7Lwbg70YDhc5hmFKeJfhKtlpBlz6PhBak4h1D7C+YVgMnhQ3fAUM7uGkU4 u8fe87ZBd8xLl7PpWv6tBaP4L9ixIwis15kCCibTCQPIrDKf5YnXnQImQJxyi9u//x5QoXGvvPP E89sawJo= X-Google-Smtp-Source: AGHT+IH7xXBf+pSvn3kHgRy6uEQ3xLaVZ8ScWHSC0IXjydDRK6JttBy5tPzuwSPN3xTseySeCrXp4w== X-Received: by 2002:a17:90b:28d0:b0:2ef:ad48:7175 with SMTP id 98e67ed59e1d1-3084f3d2a3amr2508095a91.15.1744685306495; Mon, 14 Apr 2025 19:48:26 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.48.21 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:48:26 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 28/28] mm: lru: add VM_WARN_ON_ONCE_FOLIO to lru maintenance helpers Date: Tue, 15 Apr 2025 10:45:32 +0800 Message-Id: <20250415024532.26632-29-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Stat-Signature: qi8it359d6yremb43taja9ecc48dh7j9 X-Rspam-User: X-Rspamd-Queue-Id: 9B715160002 X-Rspamd-Server: rspam08 X-HE-Tag: 1744685307-102822 X-HE-Meta: U2FsdGVkX19ROhks+A/qfPnvuiEl8PIEJdBr3NOgfq7bFB/17nKLLh+i5ngpZLeDzdzi/bzHq4WK3oVyGIIPIPQk1bWWhFrDVYdToXh4F54iPh5u+cUy9VNKBeTBD4KWEGnsBsKIa4qmPoGr0Wt+LOvt1FwdVQuvLzMrhfy74Qi6HJIex8UPgJEXNA5o9nBtclyACPlm5ikpd1F7EYPywYXG3SbPkWs2Bcj0OJxfggjl+f3WG0/8aM2ZORHV/T0PKtTnOoXipyeZQDPSwusd+baqxKpK/E7qD7xsmKjYYvwEIxkqC97+F/5aqfC3EptT15bAMF9KwqL/4wp2auQ7cf1e9pJo2i8QtwpL5Sr2/EjGMvV0/iVgwTHo+99EVg1SlyHDMC/3jR+vrbVrc/gr73FABIxvuPuaKwbcuMJ5NKuEXXr79fyNE65AfwPiTTmUNBdIQygyvmZwdiAkt4IeklUU53BZ3k0SxArXtsdLjxWwlGw4XdhTjDhfiZyF4bX7E2d0J7vEcGRzE1nCo3aiVJKYGQTm9ds9It18joOhvnaikcVGMGGYWe+T5LzP43B5YkZrpNWMiEHOmNXPEg3BhCjJ0QVLb0qgfcSbcW1JQP28LaBp2M5OtoG4+qTMVZ6P35XFfJG/z6g1vim7F57IlxSL58ttO7CfmPlqI9smlz/vwUixGtUTj2eu6WpjaBPHnDQ2fvk0ePveGPn/rU5cSugMbXHFElda25TAC2c+UsMQo8jZEjlpa1FyOSyyuxYRjp/MvlHHstFdAWZL7SqOT374ASH9aBDYlcW2rFgJQ/Byf6dh3EOFwBCRL9o674dGjllqId+ZbOvVlcWBOf3FLtA9UhR1ddfAjm+6cYAVWMurTZuUeootS+HThucQF9/xURj9FGs3AozX/5+ewqZv1M8wquUxCREN1biSsUva9sC4mAkGtOAHqFZU9Frp+ewJqQwH73GUAM8ZFLTE8dl Tm+lHVol 9r97lAbXz1erAB5MXD8LAP+3gUjTYLwYW4rAhteD1jNn/jU04EJgEfdVD1oOrlykGvrqYqz6aJxo7fd7e6luyFB65aS6IoVdPx/A0tLC0FuLX7QFLkVH9uf/pTHafsHGz6Q1uTTCtz3W7YXMl6g+S6qReSP8zZttPUzIsay09xVkI+XrXOJtT8u+mUC42sbNkuYOGakm0TeAf5scIzaIM13vrKt5KA9dxNZ0Pdnf+JcEU1MKTaNrUH4SteW6hRiUGyBGxLqmvFjV+kJePS1/d53ZxRoAdwp0DzhhO1pTux6C2PbmzTnHqN86cdMstWkzSajC64WARfDwtzGFpheHdOiqiQcZmLNajCGEYCgU9UmZZ9wI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We must ensure the folio is deleted from or added to the correct lruvec list. So, add VM_WARN_ON_ONCE_FOLIO() to catch invalid users. The VM_BUG_ON_PAGE() in move_pages_to_lru() can be removed as add_page_to_lru_list() will perform the necessary check. Signed-off-by: Muchun Song Acked-by: Roman Gushchin --- include/linux/mm_inline.h | 6 ++++++ mm/vmscan.c | 1 - 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index f9157a0c42a5..f36491c42ace 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -341,6 +341,8 @@ void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio) { enum lru_list lru = folio_lru_list(folio); + VM_WARN_ON_ONCE_FOLIO(!folio_matches_lruvec(folio, lruvec), folio); + if (lru_gen_add_folio(lruvec, folio, false)) return; @@ -355,6 +357,8 @@ void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) { enum lru_list lru = folio_lru_list(folio); + VM_WARN_ON_ONCE_FOLIO(!folio_matches_lruvec(folio, lruvec), folio); + if (lru_gen_add_folio(lruvec, folio, true)) return; @@ -369,6 +373,8 @@ void lruvec_del_folio(struct lruvec *lruvec, struct folio *folio) { enum lru_list lru = folio_lru_list(folio); + VM_WARN_ON_ONCE_FOLIO(!folio_matches_lruvec(folio, lruvec), folio); + if (lru_gen_del_folio(lruvec, folio, false)) return; diff --git a/mm/vmscan.c b/mm/vmscan.c index fbba14094c6d..a59268bf4112 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1952,7 +1952,6 @@ static unsigned int move_folios_to_lru(struct list_head *list) continue; } - VM_BUG_ON_FOLIO(!folio_matches_lruvec(folio, lruvec), folio); lruvec_add_folio(lruvec, folio); nr_pages = folio_nr_pages(folio); nr_moved += nr_pages;