From patchwork Mon Nov 4 17:52:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13861756 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D5ECD1BDD5 for ; Mon, 4 Nov 2024 17:55:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 97A086B0085; Mon, 4 Nov 2024 12:55:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 929D36B008A; Mon, 4 Nov 2024 12:55:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F0F36B008C; Mon, 4 Nov 2024 12:55:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5C7D36B0085 for ; Mon, 4 Nov 2024 12:55:00 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 064A3406FD for ; Mon, 4 Nov 2024 17:54:59 +0000 (UTC) X-FDA: 82749162930.04.E012BBA Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by imf17.hostedemail.com (Postfix) with ESMTP id 5FB6F40012 for ; Mon, 4 Nov 2024 17:54:33 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=eXjwBNt1; spf=pass (imf17.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.173 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730742677; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=eE5gbgiwpgKM7it2M9ApiJUy+ihhoCu1oBqja8NNKhg=; b=pSsjDeJfNY4GFZtkmbaa8rBhkEBNUVfaxfRRFMBLdeOuaCDJ4OoceMuxdPMoz3G/9UpaXF vwrJ1Zk5ZIRzbsQAczwRxZkBsCcdvyDmjQEK7c1uZIXpzC8JkULV+yP8X6G5hOGhu8cjQ8 BGTX5sqbGjVtcuG9fLAew/jIH7UW20k= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=eXjwBNt1; spf=pass (imf17.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.173 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730742677; a=rsa-sha256; cv=none; b=faDR3pxdZMSyAlTVVY7efHEdjDVNCN2p1jXdgCD/GvG1CsTkiRf/a9yQejEbwPiyKoT2Et dxLzPkeaAmtQFGdPflbBYX1HW6vEXwdWc1E8PW2D+3wBcxGcflc6p6aDaLqf3BycdQX5Ns vNXdBnT6EsDDuqDgytqKak16lM+hM48= Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-71e49ad46b1so3893607b3a.1 for ; Mon, 04 Nov 2024 09:54:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730742896; x=1731347696; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:message-id:date :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=eE5gbgiwpgKM7it2M9ApiJUy+ihhoCu1oBqja8NNKhg=; b=eXjwBNt1NdML8fBgk2Givl6dra2o+SkSIQZ8u3R3/+fLckYRdZ9ITe9ZCBR23/6i2/ dGVn1/1bNcCf6GrmvSskXlJz8OCT/ZBH+WocbDH063I4cdYhgJ3O5MZatLScwqTxrVN9 nh9VfzcKGmVElXukhTk9N04gdm+oV6nLlHEr0OVsBvivu+wk8SuuK7xr5QFq9ckLnzjI 3Y3laX9577KECwvNNX1TuOSG6/PFETI9zTyWKbqwyKIR/9InjE4vD1t5jOwHJbdzQGcP Jj/GJgs4GDFeP1bmPV69DAi2rpHhv2GVfboRQKc+P0DlI+Z6+8hpCsHrRN+qBeAbciwt T7dA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730742896; x=1731347696; h=content-transfer-encoding:mime-version:reply-to:message-id:date :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=eE5gbgiwpgKM7it2M9ApiJUy+ihhoCu1oBqja8NNKhg=; b=nDiZ06YDLx4GWXErmw1UAct7dxrAHIW/8nwoTebWDsGDDz2Vf0zcvXfsr6bd0Llq8s x2SbYRaOcTw19gS5ZlLUJUfEz516WdABXUdmxicv+7dp4mUKo7He5Jdi7k3+11NLtiim CdvhHtwg/2PYHoXmCnXROsFVKp85CWCvU2r5nfs7sByUsmf43Jo+qS4thiZBdfD6Gzrd rlM9chWvkRy88jqBFodf92HdbtFw1J12vvJWEELVAMq6uGSAr6g263TvI0nRk7eYBXKd IabnRFz/CkNpj3A/uvvWsugCBblvFylNZ7kZdjrA7e+H0NqQ+F6p/cQ3qoHQO2b51IFa 1F0w== X-Gm-Message-State: AOJu0Yzmvuz2SCzYksIk6YnB/VNFIkqZ1we5iU8otRaM2yvgitYOmzLk YQNpVdct4UV7nLxYwX78gpyc2nA2u6kHkXGRBHabpgrHjLTkEGMyK5fc/Fze X-Google-Smtp-Source: AGHT+IESIE3hXphs59MaH+NQyfzxFLm7ACYxVDgi0VGCCnxYJQLFKiBukUKO1+UqKARrWA1ZFArLMQ== X-Received: by 2002:a05:6a00:4610:b0:71e:4fe4:282e with SMTP id d2e1a72fcca58-72063096385mr45615034b3a.28.1730742895659; Mon, 04 Nov 2024 09:54:55 -0800 (PST) Received: from KASONG-MC4.tencent.com ([106.37.121.39]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-720bc1e5f5bsm7777039b3a.55.2024.11.04.09.54.52 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 04 Nov 2024 09:54:55 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , Johannes Weiner , Roman Gushchin , Waiman Long , Shakeel Butt , Michal Hocko , Chengming Zhou , Qi Zheng , Muchun Song , Kairui Song Subject: [PATCH v3 0/6] mm/list_lru: Split list_lru lock into per-cgroup scope Date: Tue, 5 Nov 2024 01:52:51 +0800 Message-ID: <20241104175257.60853-1-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.0 Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 5FB6F40012 X-Stat-Signature: r4e49cu17e6eek86mm3i41zu86tiqm8i X-Rspam-User: X-HE-Tag: 1730742873-743865 X-HE-Meta: U2FsdGVkX1+sfOlaDyfasJqmb/f3ZwRoFcTd5nNr1ar0yWOiG6OFDbmjJsiylbOKn2Xyy/6VaUKJULagKsTxUaCmGeAKaWtZNmsaCN9H5JQZYefnarh3erLZRC/OwGE1heFu5A5uvi4f0QU7wDm7TClcYtfSuex2Fl/UOwuC8oHJVXpB6lXxsnDoRIQtmDz/RyBsh5ofxPb15WBeMr2+x4Aw196leIfprlPdovSLWSzcKl5E8uciOJzMQe600iiW3UN/6Ft4L5EnyBzi8rWlZys0OhZBv79b4vK0ktycYYmo9mLCV0Ai9Yed6TFLTyQKRaW0+wO5YFhUbbG6mJnbMQuQUn3RMRQ0wkkA7nzuu4jjSL1iiEEnsQm/6fG6AUcli4NdHvpT1alj3QjFbrHgckgSBhQrMbQOfSVkNYJ3+qiLyDnbwuv+50kDH137WazE3fllV4uH5KSML5Ph8C45eAmnSixvO3TCbAYMwN2Ep4ck01tXaGXqE8tUiNoXn8OrAFH+4s78bOYy07Ta6LeodTjafIi79UC6GSyYmggtQhW2cr7/zt+wT7bd11IKEi/phR7iMCiMcwbXTHfXtMjCw1d05ZSMQVVcrPz8dAxJWwnaMDsfoAy80DttXXJpYP56IalNboMQ/h3ygn5/bTHM/GZGtTo568J6S4qFpFeshYycHFgty6BdnSW2UxdA1Rcn/LCyM6iIJi50ZH4rgxrWYC082VhvGTBQSo95+3DuA/vN8Yy1eCaMBETganyJf5ydQHF7KJVyWCxtIl1gUygdG64ACb2KfKLPpHUqdbxY5GHO7ymE3b4feM2NgrgkEzcnvEkFk51S7p354EluZOQaefNHrCg0plcG59Ng6kC+U+V1u5NP+Cufrzs26BFgvjAS6mg3uKN6MCkKRHulUyR59Jmr1g+l6LEtZETsfxdGhn9DeMrBX7t103gOEP0AsT1jTeBPeTfSkyxUeC3AJ/D qTZOOwYO 5/QfBVtvGCzSv+2Me23kYUOrHzykOo4gldargb/ale2nosCfZetxI16CZo3ByfqBuaub/TwpVkiw4s0JYk6zI/3lYgrXD9pOvisdmXxRY3zySlb0n+7hDrR5ZQNizaPfvH2IWoHISX0eaksoh58MQCgMExKVioFwoWoTc5LxlI816HptzDjswnWTj3kXhu4kPad7518efQnUlQVf3RuK3olLP2rHd9xJefj9A4Vyjxo9KQ1kblBycUFKpqW0eOQnYJoduerC7n6Ru41EyFUwQTxFpbDA7Bhn7XwCVG4F6Pu2QNAacdULt+3FYdkHfWF6tlfEzHO3k+V+00PWLrpMSW1CNVKq3GYYoCAwZB7jm5pY6zQBU4IxRMN0vuFIjYdn8318fkepFrE9nnt2cPdfTpJBWFGgPgqpj870gwdytK1Rw7vtQfvVD+UEdkAQbN/n0AXmAnfyBm0XztXy3C1wG/n7Zt93RfWuhkSehUCux890EyBiufL0ERt3PWQD6H3tVw7NEeiehv0YJ4ahPja5MQg3qcXBhPA4WIsFMAVhB/wFnrXczcHBaiU/hnSBkX44vwmkeQhCvMgTcoPCxPHgjpX1vI+V792RInm9Wy8ar+xoM1CY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Currently, every list_lru has a per-node lock that protects adding, deletion, isolation, and reparenting of all list_lru_one instances belonging to this list_lru on this node. This lock contention is heavy when multiple cgroups modify the same list_lru. This can be alleviated by splitting the lock into per-cgroup scope. To achieve this, this series reworked and optimized the reparenting process step by step, making it possible to have a stable list_lru_one, and making it possible to pin the list_lru_one. Then split the lock into per-cgroup scope. The result is ~15% performance gain for simple multi-cgroup tar test of small files, and reduced LOC. See PATCH 5/6 for test details. V2: https://lore.kernel.org/linux-mm/20240925171020.32142-1-ryncsn@gmail.com/ Updates from V2: - Collect Acked-by. - Fix a WARN_ON issue caused by potential compiler optimization issue. [Dan Carpenter, Naresh Kamboju] https://lore.kernel.org/linux-mm/62a65418-2393-40ec-b462-151605a5efcf@stanley.mountain/ [Applied to "mm/list_lru: split the lock to per-cgroup scope"] - Fix a BUG_ON issue, V2 forgot to cover user of LRU_STOP. [Usama Arif] https://lore.kernel.org/linux-mm/CAMgjq7D_OA=vYf5SnNnKXjppPFhDqsbYF--6=cOayKiadxuwrQ@mail.gmail.com/ V1: https://lore.kernel.org/linux-mm/20240624175313.47329-1-ryncsn@gmail.com/ Updates from V1: - Collect Review-by. - Fix a race of initialization issue that may lead to mem leak [Muchun Song] - Drop a unrelated and incorrect fix [Shakeel Butt] - Use VM_WARN_ON instead of WARN_ON for several sanity checks. Kairui Song (6): mm/list_lru: don't pass unnecessary key parameters mm/list_lru: don't export list_lru_add mm/list_lru: code clean up for reparenting mm/list_lru: simplify reparenting and initial allocation mm/list_lru: split the lock to per-cgroup scope mm/list_lru: simplify the list_lru walk callback function drivers/android/binder_alloc.c | 8 +- drivers/android/binder_alloc.h | 2 +- fs/dcache.c | 4 +- fs/gfs2/quota.c | 2 +- fs/inode.c | 5 +- fs/nfs/nfs42xattr.c | 4 +- fs/nfsd/filecache.c | 5 +- fs/xfs/xfs_buf.c | 2 - fs/xfs/xfs_qm.c | 6 +- include/linux/list_lru.h | 26 ++- mm/list_lru.c | 383 ++++++++++++++++----------------- mm/memcontrol.c | 10 +- mm/workingset.c | 20 +- mm/zswap.c | 12 +- 14 files changed, 241 insertions(+), 248 deletions(-)