From patchwork Wed Jan 17 09:23:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13521516 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47238C47258 for ; Wed, 17 Jan 2024 09:23:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2FD966B00B9; Wed, 17 Jan 2024 04:23:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2AC6D6B00BA; Wed, 17 Jan 2024 04:23:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 175EA6B00BB; Wed, 17 Jan 2024 04:23:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0A8F86B00B9 for ; Wed, 17 Jan 2024 04:23:51 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id BC8E51A0AEA for ; Wed, 17 Jan 2024 09:23:50 +0000 (UTC) X-FDA: 81688265820.17.495B173 Received: from out-177.mta0.migadu.com (out-177.mta0.migadu.com [91.218.175.177]) by imf05.hostedemail.com (Postfix) with ESMTP id DE85410000D for ; Wed, 17 Jan 2024 09:23:48 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf05.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.177 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705483429; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JjwjCE9zbsUpeYC9mkWvuJBdlLHScaadhZbWI74mw28=; b=sl00zjAWbEEw6HGoOVcLqdW32s6PIG9anYdtqiNbaT36ykgd6uKsPnFDyi0+xt1GdW4o5v mtsjuFeH1jZT8LZRILRGkkeS3Mw9XGR8clBWZji5kz9eVA2UoqRAHK1gUrphpKq6L4stY+ Gyuw1i+nwQSjxUKcMIf5yCOmssHRTSg= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf05.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.177 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705483429; a=rsa-sha256; cv=none; b=4z/OL3Tii6vP2u2hfSsSZ7mNI2UUGuVU1YUJU5b/MrZoTmunTYdqp6SSXno5i+g56HBY/r BcVpceJoMqpOyMv6H5jC4fuV6iVB8/poH/Qwygo8CeUClh46HCXPHrzZ7ehRyzLZWMaW0M fTmKm1pLwfqAI7VP5H/r8cxWKAJQqT4= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Wed, 17 Jan 2024 09:23:18 +0000 Subject: [PATCH 1/2] mm/zswap: make sure each swapfile always have zswap rb-tree MIME-Version: 1.0 Message-Id: <20240117-b4-zswap-lock-optimize-v1-1-23f6effe5775@bytedance.com> References: <20240117-b4-zswap-lock-optimize-v1-0-23f6effe5775@bytedance.com> In-Reply-To: <20240117-b4-zswap-lock-optimize-v1-0-23f6effe5775@bytedance.com> To: Andrew Morton Cc: Yosry Ahmed , Chengming Zhou , linux-kernel@vger.kernel.org, Johannes Weiner , linux-mm@kvack.org, Chris Li , Nhat Pham X-Developer-Signature: v=1; a=ed25519-sha256; t=1705483421; l=3720; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=yTwBO7fuapSePdtIIuK9uUbgWqoR7phetxp9o7hUhmg=; b=wqYBYZNagUKyFwJS+dwLD0bCqzZPOw7UPxfqGXbE4C15ul1rQmFIQtctYAEyQZZMDuUOGgUJR jMdWmww0EZ1Dhn9po3/A3DMC2ZRN/ZOwGBaYiDPVRhAnmGPf1AlNK4W X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: DE85410000D X-Stat-Signature: q5kfkyb4mcb8sj8jddbrx9htp5obwxys X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Rspam: Yes X-HE-Tag: 1705483428-281930 X-HE-Meta: U2FsdGVkX19Edt3sU+6v74EZ5pi3N6FDIoiNiYg0EiE07gk7+v7eDe/815izLc9V3H+Y0xjRvukfg96zIMQQTDaewjD9roAI3ugnTapAisYdAaa2w6p8kQIejghEzIZ7ASRzqGspQI39oDwLbIBXAuuoXSYqe7+H5BXLcLVnq3MC/Cdh/iCznLtqfCA42K3ANc66Y7ymwkN0CmhrAocZE6lGqBF2NGP/s4mXAydPO297s+BPXsZyP8cErEPSTrgjGQWbjL3tdsFPWCsThzcir/0nb9yi/Qd+B+ym7TX1xRUwvQ+/yUJ2s3QRzWufyt2uaWuaI0z9f8v2eQo/ZNKRyqlmNWOfk0t2kCnvHWru7Sj38JR6mo73uEy7vN2nxdy2ltl5az2eqL+YwXz00a4aykEObrWNdtRY2hv+AkJDPYjkKC+JTjFG0AQ/ylVtUjOENYfSVLcsSWM8UJ/tU7BT+XMY148qyXyznW8FeOcScGdMdvn1QAM/UQYKWAraT0J4Pnndjl8eXpkXl37tgZ7vkITVzVmpdGMDq4WNr6VPgo8htVYee14ckWaZpUaCdU420bICEwoWm9B44GzuDXP0Tb4v5oLlES6FbQNxjkTjYpmUO1jH9HDLIwTYJut2M+BzguRCUntSosPEoBAJ3Ir5IEryGS5UMUiqq9upbUoqDx+kcfy674jwMbUIZ2kJ+HG33Zap3U+FbpAKjBVJKyYFfaS0r0gUP131cgwX8+hc7JG7BSVfZ+hXnkwRN7d8lRfeloz2mJ0i/D350mIbmTPLCDpb71RD8XCOZiUuqEJ8EocjcXQ83blgTmDm4urxG7vtb0zMKiYcRr0vwoBGW1/S9KwCPDyHspskad0su59jvELro62oiSb/SsbppXjjURJjQinIZ8YCHKV6UX3Ztr+mDsx3QvP+3M6KhIWAO3b+iWLXYHJ1K3EWpdig8Utk7aIoTdAiTvWuAIUKHLEWhy3 sAEu38YG hv64OFQkcaFGSbXu+qBp0NEsFzwdkCVfQsCKBNSL8eyr6bjf2GAvKWrUK3Hap19bXcKtBbOeBw062S1FRoyDl6a9zlJJuSyIJigYQGQaBh6JgyXa4nryUzEloyRCMbLsD3DbF29z9mtvVWho1fAFVMkNcS2xougjmabNY+I5L8QSP0XXIZ6LT1en4jhKtvq7mS045VXGJPyk6OTbp7uD60LmBgyXWW0Kt845t44f+5YgTf7MXIGz6eGGgxgueks4R5eCIVNTwK8YS0S45vfPs/TdP7paQ837cewGwduUtku85FUGkc7KTK6PLY4toKGPLYzOPVsCMpVdLBj0geIHlMYGPupaWGo6S1HZtzEJMDfg9baqujYelmzawgA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Not all zswap interfaces can handle the absence of the zswap rb-tree, actually only zswap_store() has handled it for now. To make things simple, we make sure each swapfile always have the zswap rb-tree prepared before being enabled and used. The preparation is unlikely to fail in practice, this patch just make it explicit. Signed-off-by: Chengming Zhou Acked-by: Johannes Weiner Acked-by: Yosry Ahmed Acked-by: Nhat Pham --- include/linux/zswap.h | 7 +++++-- mm/swapfile.c | 10 +++++++--- mm/zswap.c | 7 ++++--- 3 files changed, 16 insertions(+), 8 deletions(-) diff --git a/include/linux/zswap.h b/include/linux/zswap.h index 0b709f5bc65f..eca388229d9a 100644 --- a/include/linux/zswap.h +++ b/include/linux/zswap.h @@ -30,7 +30,7 @@ struct zswap_lruvec_state { bool zswap_store(struct folio *folio); bool zswap_load(struct folio *folio); void zswap_invalidate(int type, pgoff_t offset); -void zswap_swapon(int type); +int zswap_swapon(int type); void zswap_swapoff(int type); void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg); void zswap_lruvec_state_init(struct lruvec *lruvec); @@ -51,7 +51,10 @@ static inline bool zswap_load(struct folio *folio) } static inline void zswap_invalidate(int type, pgoff_t offset) {} -static inline void zswap_swapon(int type) {} +static inline int zswap_swapon(int type) +{ + return 0; +} static inline void zswap_swapoff(int type) {} static inline void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg) {} static inline void zswap_lruvec_state_init(struct lruvec *lruvec) {} diff --git a/mm/swapfile.c b/mm/swapfile.c index 3eec686484ef..6c53ea06626b 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -2347,8 +2347,6 @@ static void enable_swap_info(struct swap_info_struct *p, int prio, unsigned char *swap_map, struct swap_cluster_info *cluster_info) { - zswap_swapon(p->type); - spin_lock(&swap_lock); spin_lock(&p->lock); setup_swap_info(p, prio, swap_map, cluster_info); @@ -3166,6 +3164,10 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) if (error) goto bad_swap_unlock_inode; + error = zswap_swapon(p->type); + if (error) + goto free_swap_address_space; + /* * Flush any pending IO and dirty mappings before we start using this * swap device. @@ -3174,7 +3176,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) error = inode_drain_writes(inode); if (error) { inode->i_flags &= ~S_SWAPFILE; - goto free_swap_address_space; + goto free_swap_zswap; } mutex_lock(&swapon_mutex); @@ -3198,6 +3200,8 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) error = 0; goto out; +free_swap_zswap: + zswap_swapoff(p->type); free_swap_address_space: exit_swap_address_space(p->type); bad_swap_unlock_inode: diff --git a/mm/zswap.c b/mm/zswap.c index ca25b676048e..d88faea85978 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1519,7 +1519,7 @@ bool zswap_store(struct folio *folio) if (folio_test_large(folio)) return false; - if (!zswap_enabled || !tree) + if (!zswap_enabled) return false; /* @@ -1772,19 +1772,20 @@ void zswap_invalidate(int type, pgoff_t offset) spin_unlock(&tree->lock); } -void zswap_swapon(int type) +int zswap_swapon(int type) { struct zswap_tree *tree; tree = kzalloc(sizeof(*tree), GFP_KERNEL); if (!tree) { pr_err("alloc failed, zswap disabled for swap type %d\n", type); - return; + return -ENOMEM; } tree->rbroot = RB_ROOT; spin_lock_init(&tree->lock); zswap_trees[type] = tree; + return 0; } void zswap_swapoff(int type) From patchwork Wed Jan 17 09:23:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13521517 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CEE29C47258 for ; Wed, 17 Jan 2024 09:23:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 19D5B6B00BC; Wed, 17 Jan 2024 04:23:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 14F496B00BE; Wed, 17 Jan 2024 04:23:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E46F16B00BF; Wed, 17 Jan 2024 04:23:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D54776B00BC for ; Wed, 17 Jan 2024 04:23:53 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A442680A9B for ; Wed, 17 Jan 2024 09:23:53 +0000 (UTC) X-FDA: 81688265946.02.7617BAA Received: from out-182.mta0.migadu.com (out-182.mta0.migadu.com [91.218.175.182]) by imf08.hostedemail.com (Postfix) with ESMTP id 99A2F16000D for ; Wed, 17 Jan 2024 09:23:51 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf08.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.182 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705483431; a=rsa-sha256; cv=none; b=5AsycRrRqQAmD/aJe0D55ztMwI8kyYL6OGLIoHubWlS4TipdzBpiEtkg1etgcK4Qf9z2+i ZAgTPLBJyliPbH6In9yjsB0O+nU2SgWM+ejccKlISSrRnR+GbbIIKaDUn9p/dIQ1sdg2eO fco0/Y3YliLx8bQ7QiRmbYMZCDqfmUs= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf08.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.182 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705483431; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tNgj5IS1y8LEh6ku3tUlD1svZpUisJdFOiMEcoNzF/0=; b=lOIxDOWazie8iK1UQ/mU8GU7pp9ol7/hMgA2dYBepjsYurAi/MPn0MLLGy+zE+toN4WVHH 9SLRAoOkm408QlgVtINgRX0HRSLpxYjjy4a5JIFR6D6HsrMvaPgLxG0cR1cZIeJGvwbM8U Y0wLhpR2aZY5o0y/k6w8v27N0wLvFiU= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Wed, 17 Jan 2024 09:23:19 +0000 Subject: [PATCH 2/2] mm/zswap: split zswap rb-tree MIME-Version: 1.0 Message-Id: <20240117-b4-zswap-lock-optimize-v1-2-23f6effe5775@bytedance.com> References: <20240117-b4-zswap-lock-optimize-v1-0-23f6effe5775@bytedance.com> In-Reply-To: <20240117-b4-zswap-lock-optimize-v1-0-23f6effe5775@bytedance.com> To: Andrew Morton Cc: Yosry Ahmed , Chengming Zhou , linux-kernel@vger.kernel.org, Johannes Weiner , linux-mm@kvack.org, Chris Li , Nhat Pham X-Developer-Signature: v=1; a=ed25519-sha256; t=1705483421; l=6988; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=lWCmSuBadnYIWvHfK92ODIlIKaW1aWLiYyH2QfP4qS0=; b=Tp7gp92PMp6MGPWK007HHErLLXRRti2ol0tPfgAeuf/e7QHzpBZVxtF6qPxzxG+cOwOgSILPL u9CHXBpgp53AbUL8faDQ8nGv9D6JVfeVhKjhfFoPEK1liA2cepBCes0 X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 99A2F16000D X-Stat-Signature: 8ami577ioxnefnwx8oir7yd4963rhhsh X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Rspam: Yes X-HE-Tag: 1705483431-185887 X-HE-Meta: U2FsdGVkX19SftgQhOiPeN500jsSVDFUFB6OsgrAfNROLSTybJ0tPvKR0sQoLugT9N4Y7YM5EMU2pIDnhSe5c8zyJYV+jmdu/Mg2R5pc8cqiyRmfuV5pz6R3UYtyVA0J3tHtnTVn5/CvffaTfFaBG6tqTI2i36RtqPokpDhHvVWacIuCiYGOwu0kwMaEqQhQY8BsGax4TNWZJMu7cLRjaNq2sXdu5G3lDdRpH9Yx3FnjoPBJkmVmhXbXWN0yuL/iuVBpKRWbipux2SqB1+XIJfslkV55atzLtsofbR+2CpDVwj9bsVuXf5bCbPvlZ69IjvHRkZ3P9yQRy8YVVh2p6q4IacA9hi5Xz/0nlFkJD6yy08qCSPrUR9j3jJa5ve2F5hMEBpT9kJHXlpwl6qbR7EacCaeJczehKb941NSmdybFzSME+ShgT8G4JQTlqUi2ySb0IhKyLb9NewXBYkrJWkbuXQATcCDcfGWm9DiXWIEU09pze2C5pZMoLT+a/uwLul45a7cbNUpLY8TBvrdtZjVtO+dySE28/fio0CzECA19LmVL3QieEICw0ZBDNtrvqCnQ5yNtESSiwKwRhLLQInqhjGjamIvTc5q8RXhXCCPO+655agdZfwCFmx5TzvsA0z69/OooYBpWi2vvemC7XfqrWr4obaLiCY5GKvJ0NMZvzPNkfY/Rgv1L2JFWc0s2S2218HRLky7zWlu9vcjIMUSHachZD+T5fyBhy1lASP4pN9/QzQxt7eDNj7UdaAMShON2XoPVxdfeUB1eOczOzVC9Ttcj9HUgl3gMO2Xt5fx9RlgHoYm4F60glpU6G8EhbROj7myWu2RpWNvaiTZqW4OKkNWQOEj2h7lX2xghDGpo1CD68z+jZdsyzLefEg0yHFROtke7dSF7YubEodEWWsVq5aRm7TQrn7XcDu15/rG2kdZaLm3yuihoBhERj0IGS3e4k3ZtamaTrwdThld y5FAsoWM mG3ISqNzChBtV0Q6ecrgO7VJLOVHAZJ2Ncs9cZTYhjGWnaXptQEfksZS3BiadKAy07dzBXRj9MQdNhhLv65arUbTOoYnyGSzN5/iSARD20UqXFOt/2KA0dGALJdg0tdrtYLQoH57Rhs3Be/Bx5a8NSd0OAtlNEbNssXXaKcgGXohbRFgdUAujh8YU0jjX7WP4lZhMXj5f+p7oe6djWkMxXePNUuTu4bBKQt83ju5wPpNFllFeS+xHjh58JgMtmK1RM2WfEkZQEz/H4xbOA3KFi4BYuMjB23zPTAdvG5wvV9Xjp2Rt9czbkNkmABuDQdUTmen+xBsvhnRRyY4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Each swapfile has one rb-tree to search the mapping of swp_entry_t to zswap_entry, that use a spinlock to protect, which can cause heavy lock contention if multiple tasks zswap_store/load concurrently. Optimize the scalability problem by splitting the zswap rb-tree into multiple rb-trees, each corresponds to SWAP_ADDRESS_SPACE_PAGES (64M), just like we did in the swap cache address_space splitting. Although this method can't solve the spinlock contention completely, it can mitigate much of that contention. Below is the results of kernel build in tmpfs with zswap shrinker enabled: linux-next zswap-lock-optimize real 1m9.181s 1m3.820s user 17m44.036s 17m40.100s sys 7m37.297s 4m54.622s So there are clearly improvements. Signed-off-by: Chengming Zhou Acked-by: Johannes Weiner Acked-by: Nhat Pham --- include/linux/zswap.h | 4 +-- mm/swapfile.c | 2 +- mm/zswap.c | 69 ++++++++++++++++++++++++++++++++------------------- 3 files changed, 47 insertions(+), 28 deletions(-) diff --git a/include/linux/zswap.h b/include/linux/zswap.h index eca388229d9a..91895ce1fdbc 100644 --- a/include/linux/zswap.h +++ b/include/linux/zswap.h @@ -30,7 +30,7 @@ struct zswap_lruvec_state { bool zswap_store(struct folio *folio); bool zswap_load(struct folio *folio); void zswap_invalidate(int type, pgoff_t offset); -int zswap_swapon(int type); +int zswap_swapon(int type, unsigned long nr_pages); void zswap_swapoff(int type); void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg); void zswap_lruvec_state_init(struct lruvec *lruvec); @@ -51,7 +51,7 @@ static inline bool zswap_load(struct folio *folio) } static inline void zswap_invalidate(int type, pgoff_t offset) {} -static inline int zswap_swapon(int type) +static inline int zswap_swapon(int type, unsigned long nr_pages) { return 0; } diff --git a/mm/swapfile.c b/mm/swapfile.c index 6c53ea06626b..35aa17b2a2fa 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -3164,7 +3164,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) if (error) goto bad_swap_unlock_inode; - error = zswap_swapon(p->type); + error = zswap_swapon(p->type, maxpages); if (error) goto free_swap_address_space; diff --git a/mm/zswap.c b/mm/zswap.c index d88faea85978..4a6dbc620c7c 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -239,6 +239,7 @@ struct zswap_tree { }; static struct zswap_tree *zswap_trees[MAX_SWAPFILES]; +static unsigned int nr_zswap_trees[MAX_SWAPFILES]; /* RCU-protected iteration */ static LIST_HEAD(zswap_pools); @@ -265,6 +266,10 @@ static bool zswap_has_pool; * helpers and fwd declarations **********************************/ +#define swap_zswap_tree(entry) \ + (&zswap_trees[swp_type(entry)][swp_offset(entry) \ + >> SWAP_ADDRESS_SPACE_SHIFT]) + #define zswap_pool_debug(msg, p) \ pr_debug("%s pool %s/%s\n", msg, (p)->tfm_name, \ zpool_get_type((p)->zpools[0])) @@ -865,7 +870,7 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o * until the entry is verified to still be alive in the tree. */ swpoffset = swp_offset(entry->swpentry); - tree = zswap_trees[swp_type(entry->swpentry)]; + tree = swap_zswap_tree(entry->swpentry); list_lru_isolate(l, item); /* * It's safe to drop the lock here because we return either @@ -1494,10 +1499,9 @@ static void zswap_fill_page(void *ptr, unsigned long value) bool zswap_store(struct folio *folio) { swp_entry_t swp = folio->swap; - int type = swp_type(swp); pgoff_t offset = swp_offset(swp); struct page *page = &folio->page; - struct zswap_tree *tree = zswap_trees[type]; + struct zswap_tree *tree = swap_zswap_tree(swp); struct zswap_entry *entry, *dupentry; struct scatterlist input, output; struct crypto_acomp_ctx *acomp_ctx; @@ -1569,7 +1573,7 @@ bool zswap_store(struct folio *folio) src = kmap_local_page(page); if (zswap_is_page_same_filled(src, &value)) { kunmap_local(src); - entry->swpentry = swp_entry(type, offset); + entry->swpentry = swp; entry->length = 0; entry->value = value; atomic_inc(&zswap_same_filled_pages); @@ -1651,7 +1655,7 @@ bool zswap_store(struct folio *folio) mutex_unlock(&acomp_ctx->mutex); /* populate entry */ - entry->swpentry = swp_entry(type, offset); + entry->swpentry = swp; entry->handle = handle; entry->length = dlen; @@ -1711,10 +1715,9 @@ bool zswap_store(struct folio *folio) bool zswap_load(struct folio *folio) { swp_entry_t swp = folio->swap; - int type = swp_type(swp); pgoff_t offset = swp_offset(swp); struct page *page = &folio->page; - struct zswap_tree *tree = zswap_trees[type]; + struct zswap_tree *tree = swap_zswap_tree(swp); struct zswap_entry *entry; u8 *dst; @@ -1757,7 +1760,7 @@ bool zswap_load(struct folio *folio) void zswap_invalidate(int type, pgoff_t offset) { - struct zswap_tree *tree = zswap_trees[type]; + struct zswap_tree *tree = swap_zswap_tree(swp_entry(type, offset)); struct zswap_entry *entry; /* find */ @@ -1772,37 +1775,53 @@ void zswap_invalidate(int type, pgoff_t offset) spin_unlock(&tree->lock); } -int zswap_swapon(int type) +int zswap_swapon(int type, unsigned long nr_pages) { - struct zswap_tree *tree; + struct zswap_tree *trees, *tree; + unsigned int nr, i; - tree = kzalloc(sizeof(*tree), GFP_KERNEL); - if (!tree) { + nr = DIV_ROUND_UP(nr_pages, SWAP_ADDRESS_SPACE_PAGES); + trees = kvcalloc(nr, sizeof(*tree), GFP_KERNEL); + if (!trees) { pr_err("alloc failed, zswap disabled for swap type %d\n", type); return -ENOMEM; } - tree->rbroot = RB_ROOT; - spin_lock_init(&tree->lock); - zswap_trees[type] = tree; + for (i = 0; i < nr; i++) { + tree = trees + i; + tree->rbroot = RB_ROOT; + spin_lock_init(&tree->lock); + } + + nr_zswap_trees[type] = nr; + zswap_trees[type] = trees; return 0; } void zswap_swapoff(int type) { - struct zswap_tree *tree = zswap_trees[type]; - struct zswap_entry *entry, *n; + struct zswap_tree *trees = zswap_trees[type]; + unsigned int i; - if (!tree) + if (!trees) return; - /* walk the tree and free everything */ - spin_lock(&tree->lock); - rbtree_postorder_for_each_entry_safe(entry, n, &tree->rbroot, rbnode) - zswap_free_entry(entry); - tree->rbroot = RB_ROOT; - spin_unlock(&tree->lock); - kfree(tree); + for (i = 0; i < nr_zswap_trees[type]; i++) { + struct zswap_tree *tree = trees + i; + struct zswap_entry *entry, *n; + + /* walk the tree and free everything */ + spin_lock(&tree->lock); + rbtree_postorder_for_each_entry_safe(entry, n, + &tree->rbroot, + rbnode) + zswap_free_entry(entry); + tree->rbroot = RB_ROOT; + spin_unlock(&tree->lock); + } + + kvfree(trees); + nr_zswap_trees[type] = 0; zswap_trees[type] = NULL; }