From patchwork Mon Feb 24 18:02:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13988661 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A541DC021A4 for ; Mon, 24 Feb 2025 18:02:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1F65828000E; Mon, 24 Feb 2025 13:02:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A62D28000C; Mon, 24 Feb 2025 13:02:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 01F9928000E; Mon, 24 Feb 2025 13:02:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D6F5F28000C for ; Mon, 24 Feb 2025 13:02:53 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 995BBA3D4E for ; Mon, 24 Feb 2025 18:02:53 +0000 (UTC) X-FDA: 83155609026.23.DB944AC Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf23.hostedemail.com (Postfix) with ESMTP id B4B4D14001B for ; Mon, 24 Feb 2025 18:02:51 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LDhcHt2A; spf=pass (imf23.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740420171; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=rFplMvPNGKU5wF32T8NlzWSeNewPtKyBr0rvnILieu4=; b=tV6LtjgZLmD1sBbJEsS6itseAfiXLAuxsj8Y0/4/3soplehzVy7yZoxxuNFwKPPgWSDVAw 9mrnwN4XQKQApaDaKCVI2j6iwMZTUZSr2Om3keYYcC4+dV1RzAeH6Y72jfMfmEEMlUwgVH OORevab1ZBOzkmlrG1V8HC5OBiSZqyg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740420171; a=rsa-sha256; cv=none; b=7QWTp6AeQcS/ZvGsgW0BwO2xZLnUmkCkM8b6XsJHaJHE0PUSM6xehYbdqvgLrloAeSlTux eKYFEzaBHjSlMjJQhE9zjr9f1w/9KLoeATGIHV/SjNlFfVwRild2LFxnpHyJ9ClLAWWsu0 YFSwTUFQI9Ji8E0gIvTIs6Br389xGOY= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LDhcHt2A; spf=pass (imf23.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-220f048c038so89550115ad.2 for ; Mon, 24 Feb 2025 10:02:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740420170; x=1741024970; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:message-id:date :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=rFplMvPNGKU5wF32T8NlzWSeNewPtKyBr0rvnILieu4=; b=LDhcHt2ATw4ixOE7w38bh0cx4h3X2JOPbZEPpC2aoFKsLyyfeMr0GBpog03RT49jUD gjbVpSA2Us4S8Q1+FLa7/oZ/aIXS8dH5YYef1cRFwe848BXnh1VRwbqus0rvNexQFLDi DrUj4HOUjrCm4VCIsW9MSIt67KvKT31kyHIcUlPppTbYlD84r2WKhJK5LYizFb56A/g4 04RaOy+GR+K0+YHxJZK4GzYQ7blQFBWuihP8xCGVcJTEUgG/Xgxy5CbxzgsczCiWgz/N Ayf3Yyceyl08TESGwtxw6Jo8oM/F4Hb+ByIppGGFubr5nmVRLQPSf1t/+U+vUqTjELi7 6kZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740420170; x=1741024970; h=content-transfer-encoding:mime-version:reply-to:message-id:date :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=rFplMvPNGKU5wF32T8NlzWSeNewPtKyBr0rvnILieu4=; b=uGYauVZce6T3qzRsKah/gd3sKxXjhXSbzREEaPYPMd7W79X+q+zxJqU0dF7wL0frJA Xl9P5Dna4lj62tW6bEfi2oLj2czHhv86F/hrNVKAvnMp4w+alKDLKWu0GNLX0/2slHSn 6L6OT3/ZD9qAxxFHD12Yh6b5znQt3nmk1g6vZJcSpuDHIFjpkj1Xf7vAVBmIiTfia7OS L/vcRb5CWPp0LamgB7r3zwUZFCLqlEoKLul5sR2qBgR4HMZkDQItnTGT4UKCc9X8ezlY UO4n34QMaJzYJO8lGeMXp0mbhTMS/OGLEzSVCu1gkL23gYzlSPy5fm7CNYcQhOyiQSlZ wZuw== X-Gm-Message-State: AOJu0YyAtO3ESHddR8Y6L4MSLRacumrmoQWMpZNdq3hFCCOqG1X6HtAU bL52rqUKhcB7OTYQc7bTFXWUkMYJ9Oz25br4Yn8+bZKapfVBr9Zg7eFtCZ4gHAQ= X-Gm-Gg: ASbGncvIEkunuPGb0jeCPXmiF0oT0RxE5RcEtZ/su8KFHrDbIaDmLez9VB9bhu0cZz4 VgJfxxeMR/Cjk4r3rtmKZRedGE0neBP4VbTd1pEkSi8/aTHh/TkOZcHY0LAsc0uCLQL6zOi/b8N C7AvMoxPiIOuozGPXCs8yPyHTesuEnAQGH1/0aY4Y5PTYseOfg3CpBkkS+d+F1hzB/1QpA9iHfQ EBZRxm3PdAdgr2i8QLAM4X9bIKsOPRuQNNjXOV16iJ5KNB00Fmdm1EWwfnTap9c5HG6XtivYKkZ rRSCO9is1E8xu1JGirJSSf5utrQyBKUasJYFGV0hQZEm X-Google-Smtp-Source: AGHT+IFjacj+DFQCh6G+bTApcerKaqlV3cTYZ+wt3sRTgERS9Hhv3HCDGIZqHizo19yPjQEhUpUnUw== X-Received: by 2002:a17:902:ced1:b0:21f:85ee:f2df with SMTP id d9443c01a7336-22307b496fcmr2055065ad.15.1740420169958; Mon, 24 Feb 2025 10:02:49 -0800 (PST) Received: from KASONG-MC4.tencent.com ([1.203.117.88]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d556e15esm184834695ad.190.2025.02.24.10.02.46 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 24 Feb 2025 10:02:49 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Barry Song , Hugh Dickins , Yosry Ahmed , "Huang, Ying" , Baoquan He , Nhat Pham , Johannes Weiner , Baolin Wang , Kalesh Singh , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 0/7] mm, swap: remove swap slot cache Date: Tue, 25 Feb 2025 02:02:05 +0800 Message-ID: <20250224180212.22802-1-ryncsn@gmail.com> X-Mailer: git-send-email 2.48.1 Reply-To: Kairui Song MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: B4B4D14001B X-Rspamd-Server: rspam07 X-Stat-Signature: 8jbe9kz9ixqh7bq6wwfifoxm74tttfg9 X-HE-Tag: 1740420171-983941 X-HE-Meta: U2FsdGVkX1/awWApDzBFULMoni06Pi4XlsZj5ZQeiWHQ4SDDJpkWAbATaFFEZeafd4pYgIWoJ6yV6Q/BLylm0F6zUhxG1WXO3T1PoBqOxluzblxa6oSVs+AaYavVNXqxzhyBh/8WjWFmHAMKDm76Ln3JxU8Hzvoj7MsjMVJ4qliKlNLdQwE9eL274y51GGn88fe7fVjV1nU7aw70MCBTT/RrBe9PWgod3NJBLN2+9imufSd4RxPdmA5gx9srqdRP7ZsAYIWUxGZ7rZKfgeRyL8kd3rHKME4ms4FDm83wywefDq5B6FStlERIAWmuhVw6S4JpSVwVVlwXlRtx7V5ruguEKnP7rB8TQiJUxenwOOcyDzddvIHKeT6n24QQ84Hgj5wNsy762dHd83Q2gGXQwQR2HyForfGm0VNvXhHCXGHGz3be79T61DTf/qPbOa96eYP94fth3CActi1S2fz+e8iG4bLaxd7SAZ8NmqqZHyNSsuXamceuMNe3Wa4M8+c+eYMAu79wIChrAgr0/lIpyw8Wkrf+OA2tXfhEaxYs4IVyL+NGGHAK10GEhW/tUIP9R3Z6qALuoEfA5YNvj6MwqKIrpZ9vzsBYdoHwxL9ZD356u69p9TFlwFsgEne49BIn/OcCGoNQPdeQIfmPMDFpASQs+RYKxy/L5lM+LU4s5gCxxo63WVsTh8JS2G58qX3FKq7RnCW6Thsfo9m/f728KWicCYsAkRAYI1P/sR5Bjm+a7jwCqouo50eT9B056qopyMBlpVfujKvTcPUkHL3djRyIzpBS1wWRfaLehK+1eIecGjcydy28h3brs25S96Gt1x27y/sbhW9HBAqnqU5g+YnCQ+A2XwoTvwMt/tM86RrrpUHBr5EytM4AAQTG62T2c3/lDNx2Bo4RCq9j13u/4D1J1EbYSSKavr7T9h/vo2J2aiVsWUSKTy54d7xz0Oj8KfB/ZHk+OML2SLucYIA ykSvfLJX Cz+ABaIKP7WmzWSYt42rT5JLmaLSXSsojxlNjpNAjBFHopIwjUpEm/gvWmQiOiJO4XsVLI+LtbfUDQTQn8wWzdWGSmDAVrpFx4zwXSG6PwRz+Rji3ATxwA30teAp0yQDcGFRI9hWN0T4vkE/Z+n7YMIvjTV9LJc9901WWZKs5cyd52QDE1peOFpW/4u1zggRT6wXqHIJ6HtXMv81Yxg6Acfk2QHoav6ymwVwvfS1UxOLAouLcNnLuDQ3Ha3U2Pz6LeGSUwK7dDIakqxftCU1caiUyLLWvU/nbboselQ/5BBr8kAr74AKsteIWFzBBSNlkotPGWxp5m/D+3zGX4Ggd2RRB3qnjZEjZfzfB6F1YoSp/wcxNKkkYhbvH1umhKyZ6Ey1d5b27p07J9jE1LEX/ozW/6N9eP4cVcU5W/+uoCVTzdnfjkx5aDv1VQjNRGerWxbaRTums9JhUuZ2lenfBAh2Z3JzW2OMUDTHtrJeCaE9KxtklNqMg+Z0L7/bzTB/XvP3rcOMMIom++iu9o1HjnnzBYCuk/BydaFf2Sjdh5f598qqWu8nGLoHJ2go+VXTb2Hp4gUfUDI3k+TT556yGpnQa2wZaQSSJ8Mcx3PPaKo3+SDw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Slot cache was initially introduced by commit 67afa38e012e ("mm/swap: add cache for swap slots allocation") to reduce the lock contention of si->lock. Previous series "mm, swap: rework of swap allocator locks" [1] removed swap slot cache for freeing path as freeing path no longer touches si->lock in most cased. Allocation path also have slight to none contention on si->lock since that series, but slot cache still helps to reduce other overheads, like counters and the plist. This series removes the slot cache from allocation path too, by using the cluster as allocation fast path and also reduce other overheads. Now slot cache is completely gone, the code is much simplified without obvious feature or performance change, also clean up related workaround. Also this should avoid other potential issues, e.g. the long pinning of swap slots: swap slot cache pins swap slots with HAS_CACHE, causing reclaim or allocation fail to use these slots on scanning. The only behavior change is the swap device allocation rotation mechanism, as explained in the patch "mm, swap: use percpu cluster as allocation fast path". Test results are looking good after deleting the swap slot cache: - vm-scalability with: `usemem --init-time -O -y -x -R -31 1G`, 12G memory cgroup using simulated pmem as SWAP (32G pmem, 32 CPUs), 16 test runs for each case, measuring the total throughput: Before (KB/s) (stdev) After (KB/s) (stdev) Random (4K): 424907.60 (24410.78) 414745.92 (34554.78) Random (64K): 163308.82 (11635.72) 167314.50 (18434.99) Sequential (4K, !-R): 6150056.79 (103205.90) 6321469.06 (115878.16) - Build linux kernel with make -j96, using 4K folio with 1.5G memory cgroup limit and 64K folio with 2G memory cgroup limit, on top of tmpfs, 12 test runs, measuring the system time: Before (s) (stdev) After (s) (stdev) make -j96 (4K): 6445.69 (61.95) 6408.80 (69.46) make -j96 (64K): 6841.71 (409.04) 6437.99 (435.55) The performance is unchanged, slightly better in some cases. [1] https://lore.kernel.org/linux-mm/20250113175732.48099-1-ryncsn@gmail.com/ --- V1: https://lore.kernel.org/linux-mm/20250214175709.76029-1-ryncsn@gmail.com/ Updates from V1: - Check the cluster with cluster_is_usable and cluster_is_empty in fast path too, improve performance and avoid fragmentation. - Fix a build warning and error for !SWAP build reported by test bot. - Global cluster array also record device for each order [Baoquan He] - Adjust of comments and function name [Baoquan He] - Collect Review-by [Baoquan He] - Minor function style improvement [Matthew Wilcox] - Retest didn't show performance beyond noise so just keep using the old performance test result. Signed-off-by: Kairui Song Kairui Song (7): mm, swap: avoid reclaiming irrelevant swap cache mm, swap: drop the flag TTRS_DIRECT mm, swap: avoid redundant swap device pinning mm, swap: don't update the counter up-front mm, swap: use percpu cluster as allocation fast path mm, swap: remove swap slot cache mm, swap: simplify folio swap allocation include/linux/swap.h | 21 +-- include/linux/swap_slots.h | 28 ---- mm/Makefile | 2 +- mm/shmem.c | 21 +-- mm/swap.h | 6 - mm/swap_slots.c | 295 -------------------------------- mm/swap_state.c | 79 +-------- mm/swapfile.c | 333 +++++++++++++++++++------------------ mm/vmscan.c | 16 +- mm/zswap.c | 6 + 10 files changed, 210 insertions(+), 597 deletions(-) delete mode 100644 include/linux/swap_slots.h delete mode 100644 mm/swap_slots.c