From patchwork Wed Sep 25 17:10:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13812336 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B372CCF9E7 for ; Wed, 25 Sep 2024 17:10:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 09C2D6B00AE; Wed, 25 Sep 2024 13:10:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0246F6B00AF; Wed, 25 Sep 2024 13:10:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E07376B00B2; Wed, 25 Sep 2024 13:10:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B8E706B00AE for ; Wed, 25 Sep 2024 13:10:31 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 49D121C4C17 for ; Wed, 25 Sep 2024 17:10:31 +0000 (UTC) X-FDA: 82603899462.15.AB4DF62 Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by imf23.hostedemail.com (Postfix) with ESMTP id 6D57E140020 for ; Wed, 25 Sep 2024 17:10:29 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LpDT+j6m; spf=pass (imf23.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727284108; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=lF9CILP+icRjVazHP8OoOt/TGPU91izFB8yriuuDyB0=; b=MnbWItqua1oKR4CZI0TCP428WnD0C1eyQRjhI7FjKuo90/xgJDoviLMwjOza7sGmz+xhHk pQm/uOAw9dxbA3egs9Sq298Rs77tS1qNHOyGrSrXRFHMHHAu1lb4h9WeNbfZX2ENeElXXX z0m/B3RuSb48By2BRCdS00dPoL6Ciy8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727284108; a=rsa-sha256; cv=none; b=p4YkU3/G5FwQ2oP9CRKUVuraaWXhxhZ7zSbJfSTtzlFOaMBCFWRYA8wQoKdvf5mrEDgCcF oJ+yQtB51KXOAwbgAJhDzPFGReYVtGMkR1fnn/4+2bNzpROVJ2ohQcnOtS38Rc0LKmwQiP +xSso3jUdQPtGn8PgC4QzpIOVrfSVeo= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LpDT+j6m; spf=pass (imf23.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-718e6299191so91402b3a.2 for ; Wed, 25 Sep 2024 10:10:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1727284227; x=1727889027; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:message-id:date :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=lF9CILP+icRjVazHP8OoOt/TGPU91izFB8yriuuDyB0=; b=LpDT+j6mGLwjBta3mv7DNcmgib3qeMd2llThRC5OKaKf0klfTmDzVIyrIVohIeqE+M ePCT+Q+HLQ8c1Ck07RkGXWRcJ5qAXpbnYdpB7w+2UyAOc0eWAy78hRBG9ilBM7BFlNyN 2YC3zT0nrvlGte/u3eGGL+rvD5jFEMDxpzgLJWmTpxr6HlzhzjgqlzpYX9C/DFpKqzYR 3uIJ2iUr7MHYHEF2vRupD4j0Pnw0mV8lQdZXrw72FOfxVPDNSoVcJtOhqWGB0gkxHox8 1fsXuTIVYrDJZzzGLKvUm5MzZLF8Lss+viK1P7tbLnnZbXt9lgEOid5R/Qf8OKyW6EK4 DcEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727284227; x=1727889027; h=content-transfer-encoding:mime-version:reply-to:message-id:date :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=lF9CILP+icRjVazHP8OoOt/TGPU91izFB8yriuuDyB0=; b=u9iILep3Ay8o2U6GgySMo8OJSNWiKiXitVEo9Ga18oaxAxOiDrRUiWtI97Wo667wlV ja4iGmqY2AA4iZ1yPVLfxJuwUkZW26hVusiAwY9R3Ja7152G0SbzVLb10JXY8RE8daSV VoMCwWsvkWCDq0Yz47jLXA27HNjOyOE1tOrcmf1GIS18kcIy3jKrMqSXOUhhzxtZj5Zy BgHNW6ysD8Nl+lWHeKv+++mWLzc3WtVsp2rpXsYNtwMeXdbt9YI4pxJisY0+liazEBrm NQCeSzzoPJdb9EN614ggzrrbYFmiGoMkZ2YeMfqwWWVyMk4GQqaZkdSKioDw7NgEs9Bo ZhVg== X-Gm-Message-State: AOJu0Yzo7eL78bftKeYkAhUTvA/CqJo9gPLDJ3jX5aRN5+FA7zQ0ZV6i XQ4yw5Ql+WNWSljp1BCD8JnMbZw7+l9MqCb7P54UkbT/BkWfJzMd11yuG4e4 X-Google-Smtp-Source: AGHT+IFZvjnkm0PpLMFljy7Rb6DNO+BU+ba5Sjp1xcjuovVVNjW7wMeQtfadIlBGMEUwiWyNpcNIRA== X-Received: by 2002:a05:6a00:928c:b0:717:85e2:28fa with SMTP id d2e1a72fcca58-71b0aadda9amr5188218b3a.16.1727284226118; Wed, 25 Sep 2024 10:10:26 -0700 (PDT) Received: from KASONG-MC4.tencent.com ([101.32.222.185]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71b032eb4f7sm2437900b3a.181.2024.09.25.10.10.22 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 25 Sep 2024 10:10:25 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , Johannes Weiner , Roman Gushchin , Waiman Long , Shakeel Butt , Michal Hocko , Chengming Zhou , Qi Zheng , Muchun Song , Kairui Song Subject: [PATCH v2 0/6] Split list_lru lock into per-cgroup scope Date: Thu, 26 Sep 2024 01:10:14 +0800 Message-ID: <20240925171020.32142-1-ryncsn@gmail.com> X-Mailer: git-send-email 2.46.1 Reply-To: Kairui Song MIME-Version: 1.0 X-Stat-Signature: myuzg4sk4jt1qyd7zga4jf7au4shxpj9 X-Rspamd-Queue-Id: 6D57E140020 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1727284229-782175 X-HE-Meta: U2FsdGVkX1+hl9YPcbIT8GgCL49ZKgyVdL8xboyVdzFnZFArIbPfVGXNPoJE8QV5/sEoryZQUAA8y6moxfdLyt2TcJYTor8jhxtfiNqzCd152TfTpWjTblh/CSGTRC4PRLFkYYaDIH3mzVdJuecfykG/TeE6SiCsz24SFEf3jHUSfZ8QcPSvfueYT43zcuyAN6vAtnUV8Iu8pPLZWNuzY+6ij6s00QE/QSaouzjguXvzd/7sCVn0JlI8SnDi74nbP/9BU4PR6lxsaXp7VUDfPz5DTk2cF7BO/Yhhk9gT2y4zrR7Ux/Q0w6mrr2Z+QVJlSU5+DE0wXoGsEn2vgM7gB7cNcuKTLZXyO0Y3fH8rM4165zE1G/EqdPDvSm8mV9oym2C8wZmdQ1NmVPbCsokEFWNBquy7so0+VFOnMTbLQR/itQ+NDhZV6Dy2DZkJx+pAwdsn6lNlmtKuhQW7mj/TjX3AshsYPMrfxqOOrsz/9B55zBC8nRgI8mbqcQlTSbWD17bVvECEA3wA1etBU5nFut1pIiqINIxo60hTnVZLXDzlfywW2pLDHqpZT1vO4RPLows+IvA6YKAidmYYMS5YBKU695XMTkSzN4BMWm5t6CpPOEJgJ9YXIDQKmSVUnGVEKhNCUWqi52Y6W1YijXvCmr1t2MEWSOw8LpNtMB1Kpk0s2t/nlOsmipsgwWa48ZAuJ9yblgW7PGYEHYQjMel0iaWnH054lZmAzcWEotfwkhbTGEZGKW/OkHQzWkFVyDc2emSghL/kgYENL68N0g+Uo4c3gEeFI33zFVi5QEuraPsV+ZqijZAJmUR9OU3O7/IZlamGc9XSexsR0a3wHMRHDP5/4jCs2gCpLM2FVSPXGKwHW1JzhDF3Hyr+OV1g+4VT0i1QYQxWet3S32joKP6/OeCfN4EfCi83+PzrDIPMJRZ4qdtXpfNE/7aVjYlXK65IWAdAxevQ8pGfcIb0CkX gsf0Okiz 4E9WYE3vC6w0+vRcStQnO/Eh3zXCcXINuH9WmgA4VUm33vfw3rK0LNaTYn6qFlqYFTmjEYb+LBHkxmlV8A+Kl1w9P+Hvyv8k/nkM9PtnE2hiVaye29PZNPMgUWinksx7WysFuHKEIqDq2Zc3Raatt7bbqFTA3/+cEGLfAHvWx7fFK3sAWA++LERE1/RWMhvtj9FiYkmqdIRTBe8j7mKHI/2Q1QNUYuJoCqwaZlN18JK7QOXFUU8/MTy6xVd06dtuLV2md7Fj9uo7KM6gvDvHBl58AiUbVC/uVTA86bK+M25QkGYR28pR4zqqDF9GlDq3JMUrWdQ8CitRrNj/tLzzEtEN6HvldFABfFs7agCaCWdGAtbbsNrMJ+TTvIzV188J2K5DrrrYv1OU81a45h7SdeF7oxEaSsPPqMoFM+UmVG0OO0tFfh9RTWSCpGcIHv6l0Djn5x8zMMjjDXBylxwxUz2ZSquojwvZkuzDjLJRGO38sruPwdIUAKgluitG+IogUxLUZetib9pDVT5mODxWgyb0k+npKuQ9A44CiOU4Zyo4nfmjjETlDkgVL3/N7mCcm3kSAeecsO+KdHeuhulxyEq0Fc7VMnedAquULRk3MynCT7FI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Currently, every list_lru has a per-node lock that protects adding, deletion, isolation, and reparenting of all list_lru_one instances belonging to this list_lru on this node. This lock contention is heavy when multiple cgroups modify the same list_lru. This can be alleviated by splitting the lock into per-cgroup scope. To achieve this, this series reworked and optimized the reparenting process step by step, making it possible to have a stable list_lru_one, and making it possible to pin the list_lru_one. Then split the lock into per-cgroup scope. The result is ~15% performance gain for simple multi-cgroup tar test of small files, and reduced LOC. See PATCH 5/6 for test details. V1: https://lore.kernel.org/linux-mm/20240624175313.47329-1-ryncsn@gmail.com/ Updates from V1: - Collect Review-by. - Fix a race of initialization issue that may lead to mem leak [Muchun Song] - Drop a unrelated and incorrect fix [Shakeel Butt] - Use VM_WARN_ON instead of WARN_ON for several sanity checks. Kairui Song (6): mm/list_lru: don't pass unnecessary key parameters mm/list_lru: don't export list_lru_add mm/list_lru: code clean up for reparenting mm/list_lru: simplify reparenting and initial allocation mm/list_lru: split the lock to per-cgroup scope mm/list_lru: Simplify the list_lru walk callback function drivers/android/binder_alloc.c | 8 +- drivers/android/binder_alloc.h | 2 +- fs/dcache.c | 4 +- fs/gfs2/quota.c | 2 +- fs/inode.c | 5 +- fs/nfs/nfs42xattr.c | 4 +- fs/nfsd/filecache.c | 5 +- fs/xfs/xfs_buf.c | 2 - fs/xfs/xfs_qm.c | 6 +- include/linux/list_lru.h | 26 ++- mm/list_lru.c | 382 ++++++++++++++++----------------- mm/memcontrol.c | 10 +- mm/workingset.c | 20 +- mm/zswap.c | 12 +- 14 files changed, 240 insertions(+), 248 deletions(-)