From patchwork Sun Feb 4 03:05:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13544431 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DF48C4828D for ; Sun, 4 Feb 2024 03:06:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B51E86B0075; Sat, 3 Feb 2024 22:06:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AB32F6B0078; Sat, 3 Feb 2024 22:06:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 953216B007B; Sat, 3 Feb 2024 22:06:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 880226B0075 for ; Sat, 3 Feb 2024 22:06:21 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 41CD4C011F for ; Sun, 4 Feb 2024 03:06:21 +0000 (UTC) X-FDA: 81752632962.02.B53EDAA Received: from out-188.mta1.migadu.com (out-188.mta1.migadu.com [95.215.58.188]) by imf04.hostedemail.com (Postfix) with ESMTP id 7C4CE40008 for ; Sun, 4 Feb 2024 03:06:18 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=none; spf=pass (imf04.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.188 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707015978; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Csm8/8Af2EeUKCuq16yFfXjClpCr+YQkSwdEsQEV9IY=; b=2AJunHtpdg3AgMHOab3N1zYDyZ2IOrqguMJc5sxrjmF0i/9oQ3uO6Ur2eyCqf1yXqtgybH ZhDBAKCW6AK4k0q1irm9y07xT2nozm//Qtezw6HgzmWtjtwRsAFhQ6eD3i429GuHP+Yh1A UxAN+2EtSXo6ob8V3ef5f4imnQPZzwo= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=none; spf=pass (imf04.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.188 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707015978; a=rsa-sha256; cv=none; b=YAIF8F5tUl3B35UcDDHVLpR9qQn8kiH9ml+Ha1iwRpgRHXh2+ccwawiLjKPRHJGiACxPs9 W73lwPGnZ3wPGaRIC7nkpODTAAuUvNT5BzoYQSTMtssbPOYB3z2VOdZ0Mvy/4usSWk/M3D AS2Sir0KdjpLGySkgnTfGzyHmHgcKUE= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Sun, 04 Feb 2024 03:05:59 +0000 Subject: [PATCH v2 1/6] mm/zswap: add more comments in shrink_memcg_cb() MIME-Version: 1.0 Message-Id: <20240201-b4-zswap-invalidate-entry-v2-1-99d4084260a0@bytedance.com> References: <20240201-b4-zswap-invalidate-entry-v2-0-99d4084260a0@bytedance.com> In-Reply-To: <20240201-b4-zswap-invalidate-entry-v2-0-99d4084260a0@bytedance.com> To: Nhat Pham , Yosry Ahmed , Andrew Morton , Johannes Weiner Cc: linux-mm@kvack.org, Nhat Pham , Chengming Zhou , linux-kernel@vger.kernel.org, Yosry Ahmed , Johannes Weiner X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 7C4CE40008 X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Stat-Signature: 9toxb88n4bpbmc7koksr3t61t3iahds8 X-Rspamd-Server: rspam01 X-Rspam: Yes X-HE-Tag: 1707015978-421325 X-HE-Meta: U2FsdGVkX18TOZ9nQ9LY+roOPvgClu87BjrPVZACEnkR5CMpQR5x0GFv73H7oVywAQFDUaLhfqTAP9V+/XHTD+GFb6xqsGnCPlEOvMpIhYnXSL22/BDthLTFHwF3J9uKCA+21kVyvPrV/f2GhY2f43RYX+V/qW0O8WvM6R0Z8J4pqlIvJYZd3HHBoNhTCMmIaAoMSmLopM+v7BjEbPFyTq9e8WVopr73Efx1PU2hynjOuRP4tNkZyM68AwSanauaZTzmPKMdAZ78pWNBo1eeTlSbB1OFY4Kwt/R7OQ2L2xjYVRBbQ45ANjeVsF/dz9I6auNF1R93eG0QnDVJ894L57cOW8wPkss74ixRgEF2zDqWojpx9Ighctuj9YsQXkPtidR4q2g2x6NsvhXGwNYMBIaDU+RvuW474K26gJtNcS8EZWkeHD9nSHh/p+dMcjnHeENQ2aAlxJUjyMvw81HJzsjfhG3Z0Bzpao5UMG14NF0ND5u8STuhqtH/8B7hXfnYqhuQmm7eG8hB55IEXz1FIq7bdPb34ulP6x+4QPoXniV7JunYzmw4cGrqBhImCobB5o41lAC/pGJPjsmVrHLhx5yaLxCxc8WsA9AxOPDiWRiduXBrtjswxy1DMInPTB0V72NYIde5Stsi9h094mEq+E9/2Qp2WJmDfLgvab5HlTHHLgLZbgPxiSnsWf+hCPvSsDMiqGVmKGviAA/D/ZyBKJnTQh7uVoFgBbqjyOAsbGnLHbVwT/3v+b6mAVTv496SA8XxjZlcAVd8Kr3xAaNFHh2W6bDhwxiErXNT+cdUt93bj9Kj7X154ocqSQLkNz5SPB9TjiVOwY10wt2EMM2yX0DEuKMVVpg43pRke5WJEbtmT1ghSEpmlSebDNa7ddhNwF1eHU7DUUixIsXjpjJhVKRACDyrIhuOHXtx7hfOwtjRUGj0r6zDEL0yMeP/aR20LmUWvY7nyNinW5mcxWA 87N7Z9ll hlquOoMJc5dfptUAhJbP5OnqzWI3Tqt1CeSuFXjdczA6Eknu2ORrl3Iod7ht7TlNge1rcRR/HjO4NZ6tEW9ps8xJqqTalYIX+JuLEmTfH3toE/m3Yhr2vGpBLWNX0Rak4RcSyBLKXCP5pQFmMNq4WQOpYLCx3ROqLgOHNw9pZiWIonG0Kqr69ay0tuO9NQitBITqCAjKSdTjPRQwZVnhae8hXTy09kA9JLooCREekZQyLjc6/8jf6FRkwdn6wpgsnRD2BwITUxNij5yXIqS20++J9iXiV25fBpltCFO2bLQZnHfFwXVjfCaA/iqF9e54J/7IIjlsJTd+Rnk2XPV1U2MU37cByt6A9yAXiOkCQCIN2wRf6Ij8zPkoex6OLbfP/uNa0Mi5IEQ9OU4cFVpPwgAQ8o+jmmlee59XQIswfCB65dcpeem1CZkhEkq5WXpGjvwfSNHHdhjMIbbDBgiQleEqYRYlBZi/pAcppBX1sC5JpcAynr/b9OYyHRFbBxGBGOakK0q8ANHWINDE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add more comments in shrink_memcg_cb() to describe the deref dance which is implemented to fix race problem between lru writeback and swapoff, and the reason why we rotate the entry at the beginning. Also fix the stale comments in zswap_writeback_entry(), and add more comments to state that we only deref the tree after we get the swapcache reference. Suggested-by: Yosry Ahmed Suggested-by: Johannes Weiner Acked-by: Yosry Ahmed Reviewed-by: Nhat Pham Signed-off-by: Johannes Weiner Signed-off-by: Chengming Zhou --- mm/zswap.c | 43 ++++++++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 17 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 4aea03285532..735f1a6ef336 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1207,10 +1207,12 @@ static int zswap_writeback_entry(struct zswap_entry *entry, /* * folio is locked, and the swapcache is now secured against - * concurrent swapping to and from the slot. Verify that the - * swap entry hasn't been invalidated and recycled behind our - * backs (our zswap_entry reference doesn't prevent that), to - * avoid overwriting a new swap folio with old compressed data. + * concurrent swapping to and from the slot, and concurrent + * swapoff so we can safely dereference the zswap tree here. + * Verify that the swap entry hasn't been invalidated and recycled + * behind our backs, to avoid overwriting a new swap folio with + * old compressed data. Only when this is successful can the entry + * be dereferenced. */ tree = swap_zswap_tree(swpentry); spin_lock(&tree->lock); @@ -1263,22 +1265,29 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o int writeback_result; /* - * Rotate the entry to the tail before unlocking the LRU, - * so that in case of an invalidation race concurrent - * reclaimers don't waste their time on it. + * As soon as we drop the LRU lock, the entry can be freed by + * a concurrent invalidation. This means the following: * - * If writeback succeeds, or failure is due to the entry - * being invalidated by the swap subsystem, the invalidation - * will unlink and free it. + * 1. We extract the swp_entry_t to the stack, allowing + * zswap_writeback_entry() to pin the swap entry and + * then validate the zwap entry against that swap entry's + * tree using pointer value comparison. Only when that + * is successful can the entry be dereferenced. * - * Temporary failures, where the same entry should be tried - * again immediately, almost never happen for this shrinker. - * We don't do any trylocking; -ENOMEM comes closest, - * but that's extremely rare and doesn't happen spuriously - * either. Don't bother distinguishing this case. + * 2. Usually, objects are taken off the LRU for reclaim. In + * this case this isn't possible, because if reclaim fails + * for whatever reason, we have no means of knowing if the + * entry is alive to put it back on the LRU. * - * But since they do exist in theory, the entry cannot just - * be unlinked, or we could leak it. Hence, rotate. + * So rotate it before dropping the lock. If the entry is + * written back or invalidated, the free path will unlink + * it. For failures, rotation is the right thing as well. + * + * Temporary failures, where the same entry should be tried + * again immediately, almost never happen for this shrinker. + * We don't do any trylocking; -ENOMEM comes closest, + * but that's extremely rare and doesn't happen spuriously + * either. Don't bother distinguishing this case. */ list_move_tail(item, &l->list); From patchwork Sun Feb 4 03:06:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13544432 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DCC9C4828F for ; Sun, 4 Feb 2024 03:06:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A66AB6B007D; Sat, 3 Feb 2024 22:06:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A19E76B007E; Sat, 3 Feb 2024 22:06:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 906EA6B0080; Sat, 3 Feb 2024 22:06:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 804996B007D for ; Sat, 3 Feb 2024 22:06:24 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 069801A0796 for ; Sun, 4 Feb 2024 03:06:24 +0000 (UTC) X-FDA: 81752633088.26.3444704 Received: from out-178.mta1.migadu.com (out-178.mta1.migadu.com [95.215.58.178]) by imf25.hostedemail.com (Postfix) with ESMTP id 48F23A0003 for ; Sun, 4 Feb 2024 03:06:21 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=none; spf=pass (imf25.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.178 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707015981; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7MT4xJM6LG9fSA23gd9gvL3cDTs+SavPSKSov+Jw+Bs=; b=zj8lInWKPwW1iJ9J9kCeJKLUKnJss+00ai8i/Yv/8xuiN0bCWd96LJLFN94IoMdi8crY6d NI2t2NrXZzQbtWYW6iuaLjXgIGTOIsglGcTT4YeYHOQd/eKDEUQjPE1OCnZ6mrwb57mE/4 uDyqmbxnhm2jc2YciJChR8lkFjleEMM= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=none; spf=pass (imf25.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.178 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707015981; a=rsa-sha256; cv=none; b=678369NSP4DfeH6gfyXpjLeaiA2YC6z5WmBrzjs1wrcSwWcTHYLtdEdUHU9HwhB/u7TcTz TDp1NWV+beM2yJ1TDZcz6PhHGT1Eh2Phvmy5amQMijlSE03DCNKXvNr3w+YT6o53hNCPa8 hLDFIs6CvappTheHyesWJD8fZKuKVwY= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Sun, 04 Feb 2024 03:06:00 +0000 Subject: [PATCH v2 2/6] mm/zswap: invalidate zswap entry when swap entry free MIME-Version: 1.0 Message-Id: <20240201-b4-zswap-invalidate-entry-v2-2-99d4084260a0@bytedance.com> References: <20240201-b4-zswap-invalidate-entry-v2-0-99d4084260a0@bytedance.com> In-Reply-To: <20240201-b4-zswap-invalidate-entry-v2-0-99d4084260a0@bytedance.com> To: Nhat Pham , Yosry Ahmed , Andrew Morton , Johannes Weiner Cc: linux-mm@kvack.org, Nhat Pham , Chengming Zhou , linux-kernel@vger.kernel.org, Yosry Ahmed , Johannes Weiner X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 48F23A0003 X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Stat-Signature: g39ooh6w6kjht7kqm5n8ikd7fpyec5na X-Rspamd-Server: rspam01 X-Rspam: Yes X-HE-Tag: 1707015981-893331 X-HE-Meta: U2FsdGVkX1/SmdIT2/sMOKRDtTREXjdixaBVBDk+ALBMwuloXKCIYj2xI1rTF1EUJbc7DFfBtdbDbiW9VIIrDQ13Vq2t3Y3VpWjnq8y7i7hNDwcNzzyJY1dBdHBFuiZ0vrLVEUXVqIXRbsAEJvFW3XExCjXDfl//JVCrS/1TLD4XELR0gmp9l8njbYIP8zawSLvQ5+x5BPrZDpT01/xvzT6RittowVS6VxW4rVfdW1e/aLgFdNFq5jsk31RUkB0/tVeWZY2hYwKVJxMYoKH9klaLemwkHHnYswM6vzP/Tvsvo+/3UTGRz0HeTfKMKR8ExhngaV8LnhKFBphW08Mnf+uMHyecIn0XJoM9wcjGlz61gKHQTVBsWe/++NNaqBfkNfykUje7H2AdCROQA2AibFwsYKFilijLv3jAnSraMOP0GJVeXNKmmJN+k1FjJqOh+tMnHzZkpygmj+gXOmB4haiNVKYxo3jAHpZj/nCJX/vXsk++sWEAZGhFIcsrgJHO8IElKCBbooc/DfVgQkAEvqhyAaxDHPHMfy6VYiVfwV3EoXb0fffadNgB2x6OScNlFz4PR58yrM6El16qixtqL1FSMYlX4KqhF3ZcIvijJTq8HiGpV0Zaj5yGMuaiFiHHmm4a3rEI1df7LXNEo5tMlwBP1YPiUuo/mQexrGxhO5HRwVt6aPtxoyHUNbvx8PYQTvwb5FI0z/fMSdBKXBHWnDfgsPweHBtOegZELdznGzAcGGrslooTZvrA3vY6fjF9SUudl9hkToUOmCUH60/P0lQCi6FDnyYrzaV+hL+5mbmTtJkAQToB4TIjUxYD5kdSeokQnz32ba7Z5dL4QUPn0gpwbNZC+L3+FqR1PoDFXGbAFNH01MvdHQwTIll8UZ0QX3VcQokpBgP6+F1E54A9n8U5i5HyRYmTXU3Llqih3dLylEIOIVwooMDtl1ncnJCm4ZDJFo3RbKJbdRlKzv9 8RLaOJjc Mhu9AgYYfuOa50aazfpcK/GyNqn4jvuAtEaMjlzWWll9ZnZ8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: During testing I found there are some times the zswap_writeback_entry() return -ENOMEM, which is not we expected: bpftrace -e 'kr:zswap_writeback_entry {@[(int32)retval]=count()}' @[-12]: 1563 @[0]: 277221 The reason is that __read_swap_cache_async() return NULL because swapcache_prepare() failed. The reason is that we won't invalidate zswap entry when swap entry freed to the per-cpu pool, these zswap entries are still on the zswap tree and lru list. This patch moves the invalidation ahead to when swap entry freed to the per-cpu pool, since there is no any benefit to leave trashy zswap entry on the tree and lru list. With this patch: bpftrace -e 'kr:zswap_writeback_entry {@[(int32)retval]=count()}' @[0]: 259744 Note: large folio can't have zswap entry for now, so don't bother to add zswap entry invalidation in the large folio swap free path. Reviewed-by: Nhat Pham Acked-by: Johannes Weiner Signed-off-by: Chengming Zhou Acked-by: Yosry Ahmed --- include/linux/zswap.h | 4 ++-- mm/swap_slots.c | 3 +++ mm/swapfile.c | 1 - mm/zswap.c | 5 +++-- 4 files changed, 8 insertions(+), 5 deletions(-) diff --git a/include/linux/zswap.h b/include/linux/zswap.h index 91895ce1fdbc..341aea490070 100644 --- a/include/linux/zswap.h +++ b/include/linux/zswap.h @@ -29,7 +29,7 @@ struct zswap_lruvec_state { bool zswap_store(struct folio *folio); bool zswap_load(struct folio *folio); -void zswap_invalidate(int type, pgoff_t offset); +void zswap_invalidate(swp_entry_t swp); int zswap_swapon(int type, unsigned long nr_pages); void zswap_swapoff(int type); void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg); @@ -50,7 +50,7 @@ static inline bool zswap_load(struct folio *folio) return false; } -static inline void zswap_invalidate(int type, pgoff_t offset) {} +static inline void zswap_invalidate(swp_entry_t swp) {} static inline int zswap_swapon(int type, unsigned long nr_pages) { return 0; diff --git a/mm/swap_slots.c b/mm/swap_slots.c index 0bec1f705f8e..90973ce7881d 100644 --- a/mm/swap_slots.c +++ b/mm/swap_slots.c @@ -273,6 +273,9 @@ void free_swap_slot(swp_entry_t entry) { struct swap_slots_cache *cache; + /* Large folio swap slot is not covered. */ + zswap_invalidate(entry); + cache = raw_cpu_ptr(&swp_slots); if (likely(use_swap_slot_cache && cache->slots_ret)) { spin_lock_irq(&cache->free_lock); diff --git a/mm/swapfile.c b/mm/swapfile.c index 0580bb3e34d7..65b49db89b36 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -744,7 +744,6 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset, swap_slot_free_notify = NULL; while (offset <= end) { arch_swap_invalidate_page(si->type, offset); - zswap_invalidate(si->type, offset); if (swap_slot_free_notify) swap_slot_free_notify(si->bdev, offset); offset++; diff --git a/mm/zswap.c b/mm/zswap.c index 735f1a6ef336..d8bb0e06e2b0 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1738,9 +1738,10 @@ bool zswap_load(struct folio *folio) return true; } -void zswap_invalidate(int type, pgoff_t offset) +void zswap_invalidate(swp_entry_t swp) { - struct zswap_tree *tree = swap_zswap_tree(swp_entry(type, offset)); + pgoff_t offset = swp_offset(swp); + struct zswap_tree *tree = swap_zswap_tree(swp); struct zswap_entry *entry; spin_lock(&tree->lock); From patchwork Sun Feb 4 03:06:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13544433 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6BDAC48291 for ; Sun, 4 Feb 2024 03:06:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C5996B007E; Sat, 3 Feb 2024 22:06:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 276AF6B0080; Sat, 3 Feb 2024 22:06:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0EF116B0081; Sat, 3 Feb 2024 22:06:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 029DB6B007E for ; Sat, 3 Feb 2024 22:06:25 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id BAA0980151 for ; Sun, 4 Feb 2024 03:06:24 +0000 (UTC) X-FDA: 81752633088.15.39CD1D0 Received: from out-179.mta1.migadu.com (out-179.mta1.migadu.com [95.215.58.179]) by imf02.hostedemail.com (Postfix) with ESMTP id E0CB28000A for ; Sun, 4 Feb 2024 03:06:22 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf02.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.179 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707015983; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oehZngeZ2nrb2M/GFxupZyVJlDL4AGtR6M2RnEZZxsc=; b=e/Npf5InKAGIHLTmnMLiTC7JUVv4pqsAeHeInRTLKBJvxwz6ctb++xQt3kx5BnqbhsRhuY x8qWvwm8Hta+WSpdSoyjhheGGNNx2GoWV8ZnXQ4abVEiZVPbD4dUtwPFRo4x0k+PRxkp9h xqutBkNoPsitRxUEWPLmul44mCpAYL8= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf02.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.179 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707015983; a=rsa-sha256; cv=none; b=3BmHDraDc/4f1EWFFbj/Kllg7ZnMR06Utefr2dD+vKKXo81S7rh7azIeyvGBPANZK7NRUQ 9L3+UI14YZjDoACc4d8lRiS8TeInW7jXhPq8yjs7Yr9DfuoUttUzcCVgRYzrBne4SCH8SV 1BJKnNorL8NXDeFSNxHgRXA4ZUY2V6o= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Sun, 04 Feb 2024 03:06:01 +0000 Subject: [PATCH v2 3/6] mm/zswap: stop lru list shrinking when encounter warm region MIME-Version: 1.0 Message-Id: <20240201-b4-zswap-invalidate-entry-v2-3-99d4084260a0@bytedance.com> References: <20240201-b4-zswap-invalidate-entry-v2-0-99d4084260a0@bytedance.com> In-Reply-To: <20240201-b4-zswap-invalidate-entry-v2-0-99d4084260a0@bytedance.com> To: Nhat Pham , Yosry Ahmed , Andrew Morton , Johannes Weiner Cc: linux-mm@kvack.org, Nhat Pham , Chengming Zhou , linux-kernel@vger.kernel.org, Yosry Ahmed , Johannes Weiner X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Queue-Id: E0CB28000A X-Rspamd-Server: rspam12 X-Stat-Signature: thz6x1jef3cq39rcjwp4u837er1pqi9j X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam: Yes X-HE-Tag: 1707015982-652375 X-HE-Meta: U2FsdGVkX198cINser+BLonNTLZdIxB5SgSuMFDrTj3VtGYhoHTJ8R9DGc+kN7ZaiGNh/9z8c66TVh2v3pcqQAclJkQ0s3V2vd90givAJWfIVfEFdDXxV5hGmVW46meXRN9b94QScnpZC+opj+sw4bG4HgGJhpLb4/mejI5dsFJGXaCMhAspaUEQzG9iSr6HPAGZl054iUv9KomtzpbQbAeeRNbQERaw66eblOjullnm0gB5t2Z0tq3L0iGHVIx0i7wkTC8r78sEcheG8LRYVUg8KUDahF7U6e4VYjBZJttALzOHBDkfbO6UiJ8pm17Y8r/mAmLFM33+OvHy1i3SRsrc1It0aNXoIE1ep+afYccPvOdI83nVfcRtCADx0IotWZ67tNTW4o1Ck28DxRrFBS3jZ+RnbxEnfV4E7pr/HnGZFpGSu1B/uLt1ULNfIeNoZRnTZNewMNB4XLT3/koFjDAcJ9B8zSkKniPLvrdBTeeSYnnvus3wWPEoin4zz3kXCEXiw3KcttF7j1itT/x0c+BPCHZJHDHwRvq5/7rhOSi5zZjh9yax+aXcZAAIrs12YYlEwFfT2jxnyaox4iPKDZchq4dTI5oLRTP6ryC8UEuAERebYlJHSjU8j81KPplxfxnu/6RXb4PdDVODgHoAfHRfs3+gcAL0MnzPO/sHx2n+3FMnGfBc5x00a3/PgLppIvG7GAMZak5RZXOzpaN/L4Fmb85s2qRQwQpWUkbqZm5mqwMZMWAuoRPGKLx/MgEtPGLTZJDVQbbwBoBvDskLeGFgt6lNm5/wp019muyJ7jy96iwd0fZ5zwoV8/d9HMuFWcexQgvzHlFAKmq9y3RRFamBUhnKs5xKJ+xJnfygM7SyZVT2OFRd1LqQNsmzZhTYGIYdAIwg6Rt8/4OIonQzhCOla87p03LK1wr2wULQYv6nz7VyUk2yQpzK6tYiEdcpDdSC+NqUlJRflVcp+ch RVbuJfqz acmG2EDjiM1pHLfhqdOk/G9fgYZPE+d2+DOpjikdyHGctlhkhZ217oEUL5qcwblGG5JYzcVJHgum5qd9TdEdDiXwlsIthhHAs/5tPBnZ6j/N8NpNG0xrAcxDvubkP3FFLU2SKanFFGLL0sqiWi0im35qkH+jjTfBwZjq/LRGfd5l7LIhqycaOmx2o5YK9VeBY3uQaPyU4ClkrFfCUXOxAMKkZDDjNt2wEiQccY3sGJsmxQn70EkpB6UJMRKQ9azWR2DJz2TQ01LrXsZpYraqHadW6+Urno+/dkF5uutNVaO9d2doiM2DvVJiQqOTArIrTUXjkFVlfOtyKHg0twnzDUSCpFCoUXN4WHGaHL8K/OauASwlk+rCVpldmusC/gxs2Mz5GKZ93XRmcZgNO5fHjrF2jAjzRYg0jQb3Bw7EHMfr3AnLIh+PZ3iGd9XcI11hrRpduWY/6F89sL9HAOt4Y95WhIQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When the shrinker encounter an existing folio in swap cache, it means we are shrinking into the warmer region. We should terminate shrinking if we're in the dynamic shrinker context. This patch add LRU_STOP to support this, to avoid overshrinking. Acked-by: Johannes Weiner Acked-by: Nhat Pham Reviewed-by: Yosry Ahmed Signed-off-by: Chengming Zhou --- include/linux/list_lru.h | 2 ++ mm/list_lru.c | 3 +++ mm/zswap.c | 4 +++- 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index f2882a820690..792b67ceb631 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -24,6 +24,8 @@ enum lru_status { LRU_SKIP, /* item cannot be locked, skip */ LRU_RETRY, /* item not freeable. May drop the lock internally, but has to return locked. */ + LRU_STOP, /* stop lru list walking. May drop the lock + internally, but has to return locked. */ }; struct list_lru_one { diff --git a/mm/list_lru.c b/mm/list_lru.c index 61f3b6b1134f..3fd64736bc45 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -243,6 +243,9 @@ __list_lru_walk_one(struct list_lru *lru, int nid, int memcg_idx, */ assert_spin_locked(&nlru->lock); goto restart; + case LRU_STOP: + assert_spin_locked(&nlru->lock); + goto out; default: BUG(); } diff --git a/mm/zswap.c b/mm/zswap.c index d8bb0e06e2b0..4381b7a2d4d6 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1315,8 +1315,10 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o * into the warmer region. We should terminate shrinking (if we're in the dynamic * shrinker context). */ - if (writeback_result == -EEXIST && encountered_page_in_swapcache) + if (writeback_result == -EEXIST && encountered_page_in_swapcache) { + ret = LRU_STOP; *encountered_page_in_swapcache = true; + } } else { zswap_written_back_pages++; } From patchwork Sun Feb 4 03:06:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13544434 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D157C4828F for ; Sun, 4 Feb 2024 03:06:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3614E6B0082; Sat, 3 Feb 2024 22:06:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3135F6B0083; Sat, 3 Feb 2024 22:06:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 200E26B0085; Sat, 3 Feb 2024 22:06:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0A79A6B0082 for ; Sat, 3 Feb 2024 22:06:28 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B6856A03FC for ; Sun, 4 Feb 2024 03:06:26 +0000 (UTC) X-FDA: 81752633172.07.F6D6FD8 Received: from out-182.mta1.migadu.com (out-182.mta1.migadu.com [95.215.58.182]) by imf26.hostedemail.com (Postfix) with ESMTP id E7A75140009 for ; Sun, 4 Feb 2024 03:06:24 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=none; spf=pass (imf26.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.182 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707015985; a=rsa-sha256; cv=none; b=2jxpchsdpPTBr3vERN/6OpVlyAkTLU32tY13Hf74cX79g2p22/Hqj/Da4UhY7Zc7JI+N3N dAWBltxJmxx+ncrvOjzAXoHUxqKkBQyoND/83AdrNHmKCmcJAX+He71GqPR26gxrHe4fOS s4jf4fy1D5i2aL8ZpxKNZ500PXeY22c= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=none; spf=pass (imf26.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.182 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707015985; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YgqDFb0TLK4KgJhm0uuyYO3PUvleKCq5Wo0I0TWEijQ=; b=B8siCgRv7r5dbFLy4XmTxPmhFR9hiZ7IblLr72I6ujOJpARFqaDHGHGdU66ooR0aGEcf3Y Lw7uueru4urSwDfufI7VGTR0vY7ez8IP8t+0EWqpYJdk7RoEN+Rdj9Exq4UBKM1astYNC+ 9iD1IE7o1AdwqKHIogYrguIe6MyasUY= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Sun, 04 Feb 2024 03:06:02 +0000 Subject: [PATCH v2 4/6] mm/zswap: remove duplicate_entry debug value MIME-Version: 1.0 Message-Id: <20240201-b4-zswap-invalidate-entry-v2-4-99d4084260a0@bytedance.com> References: <20240201-b4-zswap-invalidate-entry-v2-0-99d4084260a0@bytedance.com> In-Reply-To: <20240201-b4-zswap-invalidate-entry-v2-0-99d4084260a0@bytedance.com> To: Nhat Pham , Yosry Ahmed , Andrew Morton , Johannes Weiner Cc: linux-mm@kvack.org, Nhat Pham , Chengming Zhou , linux-kernel@vger.kernel.org, Yosry Ahmed , Johannes Weiner X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: E7A75140009 X-Stat-Signature: yieb3zfyukit14wef1pc4pgbcud7nz5q X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam: Yes X-HE-Tag: 1707015984-918533 X-HE-Meta: U2FsdGVkX18GDWmAZDbEvOr1BmGsOEP8rlyFuxqwZqX1sbflQ2TFxTCx7mL6324rfomlVa8mCKzMU0TPkRY99DkvakDXQ5XjwO1NrKziY4X/+sTpwhtGez9MtzaY7Ur2Kov2o3incDvxnad+AOSuFWBsniujlh3ylffgu8Wmq1IIHIu5RVAY1fLo96WwMCxMoXPvEUgpxVjauYfHZP4r/D4YmhKzz9JpLfmEWl1Q2IJuM4zxhk2CeEQ7z3wl1k1qXz54paa7X94AvhUJ89iBJP47REV9rLinbqTh0sb8p/pm1ISmjDS/5x+PoA1dlN17lTK2x8FcP43a7SARnxV2FtHN3lKfFCsrXtPxt+LYYxZYz2e/8VUSpkLnoCQPcHEWQj7kbdiRhRkholA0v2uz/6C/dMUd7w8X67iRJezcBzLf48M+k9qRmT/fdn+YfUd81HgUBUnvKtn0xG6csnr0rSku3PwwEjpvcOQXb7kaFOYllCysQKRCE6rHKPlWU+lAi62ZXNriFit7rzx/Gx9ZSIh1lM23mRf4dBRISNKFSS3akKQ8N2tFgo+xdqv0ciPkoI+b01OwvkvZIXvpXK9RrLIzMmD0s+1LaQ60NlzvaOx5ZexbwsELqWXHGI0u+pts1sJ/BFbi9QQxznt2dNH/bhXbVHGn+j2lJHRDx6I/vbd+K3deXv9LhvwVGMueeKEN+Gs9ILQXB98YKxaHGCeLjuNlf5xlnb7onQGJ3QSTewAGHcp5LbizwdbMSdt3MQDwj+lajGm5UziJcr2w/sQbwI66DS4ERvYdPQi2wsq2l0fPoZd4KIyJVYRvngiSCN9q6FurcegJXpguPHriyuoERtaUU44NNTnmqn2KSNGdajBmQTI2WX21++9iXorYVsZBE83pR8z760IjP+5Sd8SCYHYKD/8rfjTn4Sj3ecZCDqOdRVYKwFTmZOi33IdlYfkcNQ9N7HBa9oRPk9puc++ TrM9KmJU cMfNTqRHE8EiJTlIiyEOvTUPNFf8nGb+CSWdC8VtWgNLmrc2iVz43vbA+UNMLLVFqdymGZbebSOXv0Ce6FCeOMWhIAsQ7qpSffJeQzMMLZfybD1V7WOlqza3v1K1DzkTBFXHGajUFBzG9sxDGE17zr6JlzKjQm8xFE76Q4illcCAWjj5kNwlyaQ1FlYneEEPpW7WfN53gOiCbIY3bczDxleVEdQZQ+uzmvdsNzGLdilPuuXfc4lRlhEdWDxWoHDgQMA4MsbQDXzwVAT+V8bYHXT+z2mPEj6B4Jad+SZWjZ4kCUlNRey2UYuzJKAGUQktHzMBxUFF7NrxibuKiUES/tYfpzBu7FyQvL5LAX09KsNNZzUHv6ZZJd32TPim3lGoXQww8L6LWJscOH++OSMYZBzLRrhsnvCSfbZXL4oOAPXqUzfQr/7gRBylt0D2kVoBCPk+gDP30LMAp6hnrOtJnWn2ftr01i/tT4WHOA0Gla5/CBXykEN/1gpPPq5EheghwGlufRt9y1lYIXfo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: cat /sys/kernel/debug/zswap/duplicate_entry 2086447 When testing, the duplicate_entry value is very high, but no warning message in the kernel log. From the comment of duplicate_entry "Duplicate store was encountered (rare)", it seems something goes wrong. Actually it's incremented in the beginning of zswap_store(), which found its zswap entry has already on the tree. And this is a normal case, since the folio could leave zswap entry on the tree after swapin, later it's dirtied and swapout/zswap_store again, found its original zswap entry. So duplicate_entry should be only incremented in the real bug case, which already have "WARN_ON(1)", it looks redundant to count bug case, so this patch just remove it. Acked-by: Johannes Weiner Reviewed-by: Nhat Pham Acked-by: Yosry Ahmed Signed-off-by: Chengming Zhou --- mm/zswap.c | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 4381b7a2d4d6..3fbb7e2c8b8d 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -71,8 +71,6 @@ static u64 zswap_reject_compress_poor; static u64 zswap_reject_alloc_fail; /* Store failed because the entry metadata could not be allocated (rare) */ static u64 zswap_reject_kmemcache_fail; -/* Duplicate store was encountered (rare) */ -static u64 zswap_duplicate_entry; /* Shrinker work queue */ static struct workqueue_struct *shrink_wq; @@ -1571,10 +1569,8 @@ bool zswap_store(struct folio *folio) */ spin_lock(&tree->lock); entry = zswap_rb_search(&tree->rbroot, offset); - if (entry) { + if (entry) zswap_invalidate_entry(tree, entry); - zswap_duplicate_entry++; - } spin_unlock(&tree->lock); objcg = get_obj_cgroup_from_folio(folio); if (objcg && !obj_cgroup_may_zswap(objcg)) { @@ -1661,7 +1657,6 @@ bool zswap_store(struct folio *folio) */ while (zswap_rb_insert(&tree->rbroot, entry, &dupentry) == -EEXIST) { WARN_ON(1); - zswap_duplicate_entry++; zswap_invalidate_entry(tree, dupentry); } if (entry->length) { @@ -1822,8 +1817,6 @@ static int zswap_debugfs_init(void) zswap_debugfs_root, &zswap_reject_compress_poor); debugfs_create_u64("written_back_pages", 0444, zswap_debugfs_root, &zswap_written_back_pages); - debugfs_create_u64("duplicate_entry", 0444, - zswap_debugfs_root, &zswap_duplicate_entry); debugfs_create_u64("pool_total_size", 0444, zswap_debugfs_root, &zswap_pool_total_size); debugfs_create_atomic_t("stored_pages", 0444, From patchwork Sun Feb 4 03:06:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13544435 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63A64C48291 for ; Sun, 4 Feb 2024 03:06:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E398B6B0083; Sat, 3 Feb 2024 22:06:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DC3BC6B0085; Sat, 3 Feb 2024 22:06:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B77F66B0087; Sat, 3 Feb 2024 22:06:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id AA4916B0083 for ; Sat, 3 Feb 2024 22:06:28 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 7D50B1603DC for ; Sun, 4 Feb 2024 03:06:28 +0000 (UTC) X-FDA: 81752633256.28.3A6D749 Received: from out-179.mta1.migadu.com (out-179.mta1.migadu.com [95.215.58.179]) by imf24.hostedemail.com (Postfix) with ESMTP id 9636B180005 for ; Sun, 4 Feb 2024 03:06:26 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=none; spf=pass (imf24.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.179 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707015986; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=34aQDIzLuFrxZZKDlUhIhZ5RNkMS8ZW/uWPkH0n/4kQ=; b=QTAPgVY0uJJzFcV/TpxcgxDd2liim6fcM7IXkDbZAPXh4eDAvWz1OweMYdqspeI1g9Wb+h ayGO0K/RKH3Wpu2ypdUN4GSEBM93uGkiR9q4SwgcTSS1NpwtKAYSNdC+1XNp8Z94+JMJev FcaMZmf8uXsDeZVS1OpTrtvGWw2/jgI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707015986; a=rsa-sha256; cv=none; b=p/ii2Ay/5jhC4CakJ63z61+LwlgfDPwmbCxbqdrzAIb6tOn4bbFK6GsqzsqQjW+28p7BCS HctXo95yx5aMowqYqQRcMH17VSkIDBDGctwKC56KgFeRUYRMgGVBU4tGieas4EJOF+ZvFV GIGJ2bsoBYbNcekClrmGFis9Vi2wfM8= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=none; spf=pass (imf24.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.179 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Sun, 04 Feb 2024 03:06:03 +0000 Subject: [PATCH v2 5/6] mm/zswap: only support zswap_exclusive_loads_enabled MIME-Version: 1.0 Message-Id: <20240201-b4-zswap-invalidate-entry-v2-5-99d4084260a0@bytedance.com> References: <20240201-b4-zswap-invalidate-entry-v2-0-99d4084260a0@bytedance.com> In-Reply-To: <20240201-b4-zswap-invalidate-entry-v2-0-99d4084260a0@bytedance.com> To: Nhat Pham , Yosry Ahmed , Andrew Morton , Johannes Weiner Cc: linux-mm@kvack.org, Nhat Pham , Chengming Zhou , linux-kernel@vger.kernel.org, Yosry Ahmed , Johannes Weiner X-Migadu-Flow: FLOW_OUT X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Rspamd-Queue-Id: 9636B180005 X-Rspamd-Server: rspam11 X-Stat-Signature: m9edhh4qbzfz56uqi5fwm775uheodnpj X-Rspam: Yes X-HE-Tag: 1707015986-910477 X-HE-Meta: U2FsdGVkX1/Uvg2OJIUjkoDcvPVtU11oTun2ZIjGAx5yaC1a+fBYI0sLzy/iRaQ+Inev8JZvHid0h3lFJIGWNaonnEaRzLlA9FQMUM+HnGlijzvSWjh5D7iLaP7pSwLYKV6bHqQTARYad7RWgOSk4+fjabshrJTyaGKRjpbR3mGIwDoAHfXPIzxf+BopYpL4jjJ6GoICW6lu70ahF5+8atgtAxritp6weyS0qZ57N2AwSWuy25/ouzdWhMwpxyPu0UaBFksCilW7X3iyiUXIwlYnC+sWdfsvzIyQywylSsWN0FotwJcOrV0gWBxTWPJd8/14KT46gr9nk4Pd/zs4YFcZPWYE7+gEJq14p3fUbvOSeEQaiaAFRDRa++IBaZI5C4vm01IDBnUmTIvQGDFwfRxDIL4sO9cwe8It4Al731fkiW+y2OGcNtyiqu1RlB7BpGhJeAFsj3gR22On7gJdNB0WVSlPZhD+fgjaeagPfaJk97HlM8NC6Sipf8EHDJXvkvfgoWbDWXEUSd7B+uPIwxkJCpTcJ4Tqajik2gdKk4a7LeEyyIaCiMTrqP4vRpTfb5D9MnpQNQ3GSfwoGP1oIFDi5RfU4k+AKttvAJ+iu2qejT+NxdzXXVGbwCM9/62tY90QLlQ/pu8thV8BoAME7qlZG14rucrKnAwoL1T5f7CKwKTa34dI4uopXQtawTp1ldWpbsGS3J93lz0PzhsFk6TUpeB2dsJGRf5vd1axG1NKuYL6guwyxvTqSXxprUKjaKNh49mVu9v0To4bxfAx1QyNEea1EzjoLyMbg/Q5UsehD3MMAGXoyJDggpr9SCg7fC0MRGsaWGhMXVKK8EXNFEQtzEUE6yVmYZvwLczrJTHrGx8Sepcv1yL3OggMzbIr3whM/hxPI+7GndiXv3R8Y05A9bSyQNHJMR+oFY2rhr+2t79E0+vydXiaA6nYjTG316I+dnFKLQX6DgI0IOZ KbqH2OYZ TY6FW3Wr5D0OzkgC7mggov4AGrHdA4Tv67EvYGCkTDADs+oLV/hZ/IPFGSottr+uDNgud+IJ4g+Mb+ieg457GbECt1sD0T9NegQ2gTian+uMI/qUBYBYyyxg+HWTBoZk2F5ZFx9KS7/d/0UAn10+6xtKHpxzr0g64KK31ss9zJ1VEqUaGMQOgC7qaY6YTHN3iXirMbmbFbG9BDlC1yflanyPWlsOMEkXM7HKau2BPFweNwy4qPvtKCTqlZnG5XJFq59rRvy9T7P82tbrmtNhFVNVDLpqyS+7uAEs/M+/qOWCFZSGAwBElNu1Fk/zjaF6exeO00b/uCq/DQtzwd3Jnkk0fMg7FTlx+jrAvZS4GfT9JFKPIEuBlK8S19OCssRQ+29wDYLDIHyqC7vA/ZhwNAtB+dh8VMAobsfyH4Z8YqJ780EXX1LG7pv/U+dJdnFXljJxRQ4UoHK1Q7grFKvtGPXEcf8VTW5tqgVFH2RHcjxoOW/fCijShZT9qXqIHoUdkqY4G3V9WQ3y6liTc02ADmIqQY/ptSkivruENMVRT5GlmCwA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The !zswap_exclusive_loads_enabled mode will leave compressed copy in the zswap tree and lru list after the folio swapin. There are some disadvantages in this mode: 1. It's a waste of memory since there are two copies of data, one is folio, the other one is compressed data in zswap. And it's unlikely the compressed data is useful in the near future. 2. If that folio is dirtied, the compressed data must be not useful, but we don't know and don't invalidate the trashy memory in zswap. 3. It's not reclaimable from zswap shrinker since zswap_writeback_entry() will always return -EEXIST and terminate the shrinking process. On the other hand, the only downside of zswap_exclusive_loads_enabled is a little more cpu usage/latency when compression, and the same if the folio is removed from swapcache or dirtied. More explanation by Johannes on why we should consider exclusive load as the default for zswap: Caching "swapout work" is helpful when the system is thrashing. Then recently swapped in pages might get swapped out again very soon. It certainly makes sense with conventional swap, because keeping a clean copy on the disk saves IO work and doesn't cost any additional memory. But with zswap, it's different. It saves some compression work on a thrashing page. But the act of keeping compressed memory contributes to a higher rate of thrashing. And that can cause IO in other places like zswap writeback and file memory. And the A/B test results of the kernel build in tmpfs with limited memory can support this theory: !exclusive exclusive real 63.80 63.01 user 1063.83 1061.32 sys 290.31 266.15 workingset_refault_anon 2383084.40 1976397.40 workingset_refault_file 44134.00 45689.40 workingset_activate_anon 837878.00 728441.20 workingset_activate_file 4710.00 4085.20 workingset_restore_anon 732622.60 639428.40 workingset_restore_file 1007.00 926.80 workingset_nodereclaim 0.00 0.00 pgscan 14343003.40 12409570.20 pgscan_kswapd 0.00 0.00 pgscan_direct 14343003.40 12409570.20 pgscan_khugepaged 0.00 0.00 Acked-by: Yosry Ahmed Acked-by: Johannes Weiner Reviewed-by: Nhat Pham Signed-off-by: Chengming Zhou --- mm/Kconfig | 16 ---------------- mm/zswap.c | 14 +++----------- 2 files changed, 3 insertions(+), 27 deletions(-) diff --git a/mm/Kconfig b/mm/Kconfig index ffc3a2ba3a8c..673b35629074 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -45,22 +45,6 @@ config ZSWAP_DEFAULT_ON The selection made here can be overridden by using the kernel command line 'zswap.enabled=' option. -config ZSWAP_EXCLUSIVE_LOADS_DEFAULT_ON - bool "Invalidate zswap entries when pages are loaded" - depends on ZSWAP - help - If selected, exclusive loads for zswap will be enabled at boot, - otherwise it will be disabled. - - If exclusive loads are enabled, when a page is loaded from zswap, - the zswap entry is invalidated at once, as opposed to leaving it - in zswap until the swap entry is freed. - - This avoids having two copies of the same page in memory - (compressed and uncompressed) after faulting in a page from zswap. - The cost is that if the page was never dirtied and needs to be - swapped out again, it will be re-compressed. - config ZSWAP_SHRINKER_DEFAULT_ON bool "Shrink the zswap pool on memory pressure" depends on ZSWAP diff --git a/mm/zswap.c b/mm/zswap.c index 3fbb7e2c8b8d..cbf379abb6c7 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -139,10 +139,6 @@ static bool zswap_non_same_filled_pages_enabled = true; module_param_named(non_same_filled_pages_enabled, zswap_non_same_filled_pages_enabled, bool, 0644); -static bool zswap_exclusive_loads_enabled = IS_ENABLED( - CONFIG_ZSWAP_EXCLUSIVE_LOADS_DEFAULT_ON); -module_param_named(exclusive_loads, zswap_exclusive_loads_enabled, bool, 0644); - /* Number of zpools in zswap_pool (empirically determined for scalability) */ #define ZSWAP_NR_ZPOOLS 32 @@ -1722,16 +1718,12 @@ bool zswap_load(struct folio *folio) count_objcg_event(entry->objcg, ZSWPIN); spin_lock(&tree->lock); - if (zswap_exclusive_loads_enabled) { - zswap_invalidate_entry(tree, entry); - folio_mark_dirty(folio); - } else if (entry->length) { - zswap_lru_del(&entry->pool->list_lru, entry); - zswap_lru_add(&entry->pool->list_lru, entry); - } + zswap_invalidate_entry(tree, entry); zswap_entry_put(entry); spin_unlock(&tree->lock); + folio_mark_dirty(folio); + return true; } From patchwork Sun Feb 4 03:06:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13544436 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D03EC4828D for ; Sun, 4 Feb 2024 03:06:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ED8DD6B0087; Sat, 3 Feb 2024 22:06:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E36F46B0088; Sat, 3 Feb 2024 22:06:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BC57E6B0089; Sat, 3 Feb 2024 22:06:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A0E4E6B0087 for ; Sat, 3 Feb 2024 22:06:30 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6217480151 for ; Sun, 4 Feb 2024 03:06:30 +0000 (UTC) X-FDA: 81752633340.25.D8A6B82 Received: from out-188.mta1.migadu.com (out-188.mta1.migadu.com [95.215.58.188]) by imf25.hostedemail.com (Postfix) with ESMTP id 83D3BA0007 for ; Sun, 4 Feb 2024 03:06:28 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf25.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.188 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707015988; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CpJJ+lmut35Sgt5qzijPizjCkrhWKWh5X/lhHB6j0D8=; b=Izk/QUd/52PFZesujlctVd+IHmbJk8wOEWGiRt7Lx68Z9cEy94c4WuXUKZxaSXECfwjPWa +RD6l1Xqa1X6zo6Em2jn0EtImR1iKz0bwBG1hbVlo+jzdnOyYBmJzcxvD4StrNgMMTWW7n c4bao0jAplFizzpY4gnWRF/keBRHS2g= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf25.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.188 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707015988; a=rsa-sha256; cv=none; b=zQ+fST17APbhN7bMAhkvjfCD1pxvFWtSTB5GJfshgD9UACc5YV13Q0deH/78kLQ+Ndfdx6 vr3QQxkMMKJcPO5xRO1XV2dhAXo0Svt/U4B3rg1UBlJVXveAXWVP4NqyZdHNlbUgZ71b0E 6kGz9nZJznxARXyybBn99z3qtqy8PL0= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Sun, 04 Feb 2024 03:06:04 +0000 Subject: [PATCH v2 6/6] mm/zswap: zswap entry doesn't need refcount anymore MIME-Version: 1.0 Message-Id: <20240201-b4-zswap-invalidate-entry-v2-6-99d4084260a0@bytedance.com> References: <20240201-b4-zswap-invalidate-entry-v2-0-99d4084260a0@bytedance.com> In-Reply-To: <20240201-b4-zswap-invalidate-entry-v2-0-99d4084260a0@bytedance.com> To: Nhat Pham , Yosry Ahmed , Andrew Morton , Johannes Weiner Cc: linux-mm@kvack.org, Nhat Pham , Chengming Zhou , linux-kernel@vger.kernel.org, Yosry Ahmed , Johannes Weiner X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 83D3BA0007 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: o4j81th4ytxcou3fsezsebq9pqeacccf X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam: Yes X-HE-Tag: 1707015988-957758 X-HE-Meta: U2FsdGVkX19Dz6mBu7MAwBh/nsrBq6dB5IADrTK73jjHnUX1o55E4UeGSjraANHcFWlneOTM4ZbXmMxHv5zhPiXieNvoUHJho8DtIoodwmhqXVB6VMu8PvtcqucPJFe5Lx/2WMV9rGxfi4wlehovh1fbuSf4inVu8sufQX+JrX8K9gSFBZsg2dqzRNUCFq8QLLudsrO+Lo6o/Jow6+oP44sK8DXGLQsLC32jok09j/2mG0CQ2+Bnb5IW/07USZNuhThYJ1XtXcOvaB/fCt7x2ETM92UmAi817RsCQDUzkLJbRtuz4rPkrzGDqsHvKSfD3kfGeuWafNUb4f605K6JE1CqMn1Q+v9wf+IcPpG9pDNfXlEa0+AT6JaXyxHNQ6LpN2cVOTyow3IOiu338yUV6gUjTzHBA2luhWQ1Thxo4rpVtuGWfuRlWPM7EOXKErcHqLVSlFzSogroAGu03fwm2AnT7vF9K/L+Ckam2AkQ6DJx7KKwLGEc5DK4fRR1yu4Np65f5U0TVHPFUHUOExsBWcjUPB/8NHD2bVFrh2nNnlcLySSdcISPYf0Hj/OJdbPBlqLZw32wjIlo3snwS59jnQatGJlORyZ0Y2JUTyxu+VoSlYsxOiVidHGatdbV5ImMqsJmjrM0mq/yj7DiUt7ECIJrhMdggeU6dyfj4hAYNRagOf0QwBjN7pz1668FfuvNQVDYs8U9GxqQQt7dN2PHPSErcUF6dXaIwPXJRh2Ox7argkNqJA5QvwUrjccJeiJRn8XqVDbYzUzcYq3blC0tNLn743/zGMNVLTH17UFjhFVmYEHPWkzJW3y/zAp2tCNVYTEAoTByaAVsq9lVkENnz++DtBoCovRnyn/BZawhfqnm8iHPy37K25AAW8thf+flKSPtK574Rlidtk2i6dCl0/gogN254hgMz8v6EGPeqX9qCEzlAUX0hNTnvNoFVmOEIztHqnL6wAo1QGkmDic 4cQYo8Dy nsIlFpaBQoBJtnRGLlyHH7c0WQSvIrZK7GrUGh+HXd6t2MxosqnjaRHXsR8Ge7x4iKkmHVUMqgMLlsrw8UT6iataKXjwVc7QyRuFXPAutmDuthjsia2Bs1GlcNZ2UV/VyXy9Vx6ol1XCxIBeecEQK1n2J8JkW9EhqeX8TIghKaDUAf/sB4abjEADSNroAD2i/2Yo5wJI+HVlbrkFfqu+s0c22GI4cCAHBy8eJrNkTaaViJrV+vBVydrPQS0S4ZW65kgw00GhKm47+4xy5bgSTChQT/WJdfcpjFU4Pyjdw1cuHJm19yKiWvrasziZk8w0T24z5F0ryLKNCez2AbhVPkEA+pBSvjFTMxffvJtnPnw0qViFjVeSpD8dce4VG5397bDVpv4+bNTUOaTizw7eaTgayPvCfQy9PJ1AggoEuX0e4s9U73PvXfpUgfQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Since we don't need to leave zswap entry on the zswap tree anymore, we should remove it from tree once we find it from the tree. Then after using it, we can directly free it, no concurrent path can find it from tree. Only the shrinker can see it from lru list, which will also double check under tree lock, so no race problem. So we don't need refcount in zswap entry anymore and don't need to take the spinlock for the second time to invalidate it. The side effect is that zswap_entry_free() maybe not happen in tree spinlock, but it's ok since nothing need to be protected by the lock. Reviewed-by: Nhat Pham Acked-by: Johannes Weiner Signed-off-by: Chengming Zhou --- mm/zswap.c | 63 +++++++++++--------------------------------------------------- 1 file changed, 11 insertions(+), 52 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index cbf379abb6c7..cd67f7f6b302 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -193,12 +193,6 @@ struct zswap_pool { * * rbnode - links the entry into red-black tree for the appropriate swap type * swpentry - associated swap entry, the offset indexes into the red-black tree - * refcount - the number of outstanding reference to the entry. This is needed - * to protect against premature freeing of the entry by code - * concurrent calls to load, invalidate, and writeback. The lock - * for the zswap_tree structure that contains the entry must - * be held while changing the refcount. Since the lock must - * be held, there is no reason to also make refcount atomic. * length - the length in bytes of the compressed page data. Needed during * decompression. For a same value filled page length is 0, and both * pool and lru are invalid and must be ignored. @@ -211,7 +205,6 @@ struct zswap_pool { struct zswap_entry { struct rb_node rbnode; swp_entry_t swpentry; - int refcount; unsigned int length; struct zswap_pool *pool; union { @@ -222,11 +215,6 @@ struct zswap_entry { struct list_head lru; }; -/* - * The tree lock in the zswap_tree struct protects a few things: - * - the rbtree - * - the refcount field of each entry in the tree - */ struct zswap_tree { struct rb_root rbroot; spinlock_t lock; @@ -890,14 +878,10 @@ static int zswap_rb_insert(struct rb_root *root, struct zswap_entry *entry, return 0; } -static bool zswap_rb_erase(struct rb_root *root, struct zswap_entry *entry) +static void zswap_rb_erase(struct rb_root *root, struct zswap_entry *entry) { - if (!RB_EMPTY_NODE(&entry->rbnode)) { - rb_erase(&entry->rbnode, root); - RB_CLEAR_NODE(&entry->rbnode); - return true; - } - return false; + rb_erase(&entry->rbnode, root); + RB_CLEAR_NODE(&entry->rbnode); } /********************************* @@ -911,7 +895,6 @@ static struct zswap_entry *zswap_entry_cache_alloc(gfp_t gfp, int nid) entry = kmem_cache_alloc_node(zswap_entry_cache, gfp, nid); if (!entry) return NULL; - entry->refcount = 1; RB_CLEAR_NODE(&entry->rbnode); return entry; } @@ -954,33 +937,15 @@ static void zswap_entry_free(struct zswap_entry *entry) zswap_update_total_size(); } -/* caller must hold the tree lock */ -static void zswap_entry_get(struct zswap_entry *entry) -{ - WARN_ON_ONCE(!entry->refcount); - entry->refcount++; -} - -/* caller must hold the tree lock */ -static void zswap_entry_put(struct zswap_entry *entry) -{ - WARN_ON_ONCE(!entry->refcount); - if (--entry->refcount == 0) { - WARN_ON_ONCE(!RB_EMPTY_NODE(&entry->rbnode)); - zswap_entry_free(entry); - } -} - /* - * If the entry is still valid in the tree, drop the initial ref and remove it - * from the tree. This function must be called with an additional ref held, - * otherwise it may race with another invalidation freeing the entry. + * The caller hold the tree lock and search the entry from the tree, + * so it must be on the tree, remove it from the tree and free it. */ static void zswap_invalidate_entry(struct zswap_tree *tree, struct zswap_entry *entry) { - if (zswap_rb_erase(&tree->rbroot, entry)) - zswap_entry_put(entry); + zswap_rb_erase(&tree->rbroot, entry); + zswap_entry_free(entry); } /********************************* @@ -1219,7 +1184,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, } /* Safe to deref entry after the entry is verified above. */ - zswap_entry_get(entry); + zswap_rb_erase(&tree->rbroot, entry); spin_unlock(&tree->lock); zswap_decompress(entry, &folio->page); @@ -1228,10 +1193,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, if (entry->objcg) count_objcg_event(entry->objcg, ZSWPWB); - spin_lock(&tree->lock); - zswap_invalidate_entry(tree, entry); - zswap_entry_put(entry); - spin_unlock(&tree->lock); + zswap_entry_free(entry); /* folio is up to date */ folio_mark_uptodate(folio); @@ -1702,7 +1664,7 @@ bool zswap_load(struct folio *folio) spin_unlock(&tree->lock); return false; } - zswap_entry_get(entry); + zswap_rb_erase(&tree->rbroot, entry); spin_unlock(&tree->lock); if (entry->length) @@ -1717,10 +1679,7 @@ bool zswap_load(struct folio *folio) if (entry->objcg) count_objcg_event(entry->objcg, ZSWPIN); - spin_lock(&tree->lock); - zswap_invalidate_entry(tree, entry); - zswap_entry_put(entry); - spin_unlock(&tree->lock); + zswap_entry_free(entry); folio_mark_dirty(folio);