From patchwork Mon Apr 7 23:42:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 14042005 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0EFB7C36010 for ; Mon, 7 Apr 2025 23:43:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 90D48280003; Mon, 7 Apr 2025 19:42:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 860FA280002; Mon, 7 Apr 2025 19:42:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5F3B4280003; Mon, 7 Apr 2025 19:42:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 16196280001 for ; Mon, 7 Apr 2025 19:42:35 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id DDB4B1217F7 for ; Mon, 7 Apr 2025 23:42:35 +0000 (UTC) X-FDA: 83308874670.28.0BA90D3 Received: from mail-yw1-f179.google.com (mail-yw1-f179.google.com [209.85.128.179]) by imf30.hostedemail.com (Postfix) with ESMTP id 2393580006 for ; Mon, 7 Apr 2025 23:42:33 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=OqLMffmE; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.128.179 as permitted sender) smtp.mailfrom=nphamcs@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744069354; a=rsa-sha256; cv=none; b=LgrZu+HEvCYvLItzXNPVcKfwM/GMvapG6334zzXpHAnQYrVV1RyPU7dpyZohfs2YENNc+i a/ifBlojnHPhfFRjugHN2KDrgjCPn5+hFmGi7vY14fwGcYcWAabtpiZo/PJwT+lvlr/Ppq R3ITBKCsZtorNl8AZGC7lgboK7Z9U98= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=OqLMffmE; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.128.179 as permitted sender) smtp.mailfrom=nphamcs@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744069354; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=u4vHOJqfqkXMoPP1dZtjw+0/K3krKLrN4KxVkeSTfDE=; b=4VsTdwx5AS4+GcZZeo1XRQhKOb0bkW+mclG/26UiAdkZ1k1Ka/lP9U1jbtuWqny5E/7naP TZmP6t/NOibnO9Ko7BXVC1gh4W97xnsns5VEQ1/DbMTDuFQmExlxBVMC7mPmeYTZGoEGu7 qW3Cr+lYqUe/sT8/4coAooP6kl3naAc= Received: by mail-yw1-f179.google.com with SMTP id 00721157ae682-703cd93820fso46073677b3.2 for ; Mon, 07 Apr 2025 16:42:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744069353; x=1744674153; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=u4vHOJqfqkXMoPP1dZtjw+0/K3krKLrN4KxVkeSTfDE=; b=OqLMffmEEC8k0OTRGiBq0PVENgw8OoNrdu71RiplWMoXjJCMYpN/gY9cqYGHUpq/on LG6iIxh3hUK6KEZPgWtVNz+J+Rzy/T3UhTgG07LYqK1ufaIU3xudBhEjCP8TDPGMJ8la pmYtfC0N2ZUPLij59gn/UvKF3rV4iQ+8HAYQi7W1xFvEybfAnAgYaP27nwK+mbA+lM1J DAF7kmVyh9QIJmZJboorZWjN8/ToLzUN7l0aeYGo399N3/t5dVcbb2xsGGECqgg8XjTE I5sKfgq2euuLtlh03vHPnutEJLWl7vChn9ZKsGhEMdM3mpcfA7YJrqgPqpyxv2Uvlp6b UvOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744069353; x=1744674153; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=u4vHOJqfqkXMoPP1dZtjw+0/K3krKLrN4KxVkeSTfDE=; b=oZRLF+jgGxYqdkeMvDwyUNkLD69Ak4pP0lGWhA8YZUSSzwmx95F1SnMMWVpomzmkXg r/PHucCzXuQ9rfnKAD3eC+APfkjUJwMWuqjho2IfhSOF7/7iGMXcw7a8VGLRCIUpiTd3 Hggk1khkvJ3Ju/wXIBXYvCsJRV7Im0V26lUdDm+qsgJaU+YghHGADZSvvi5H8wki0tEr GRh5zca3z0EUxaZ6FNg+dMzjc3u2rCpWgJtYkBs1xZ3ahEdPLgfXfFOahDqUoTT8+OLQ v7UKQueUa1pLW1co473OGtvsVT+U5UTJBp/3e0df5dIWxOP1xFj7DqcQDke8zLcAZDQZ S0rA== X-Gm-Message-State: AOJu0Yw1WiIP6ORX0beGljxCzLlfLb9dWjyYglL8GDv+enjLm+htQKAL BjzB4Da1hAjqIk7qKy5JQGPlolL24zragEa7g9Omjpb+G8FvPrUS8tGpVQ== X-Gm-Gg: ASbGncuP6PMgwY28DtmS7uuVSaLsqFeajwiryhFwt8410N7XOafYQoHE9JfuqE8tIjE Q+GDTcLxzYMbEAfgJn/2Mk7YZQyQWy6uqoQTtBSislRUi+Mz8FWMvfMRlcV9xxI9ISHriW/77w+ AGRGSv/p4PeB1/LsBatW8tQhfVUIZxIRxxQwpX6JLXf7fms8Y5bxR8UErZ7l/xGJiszOJYUu6M9 6uChopotj8HDIc4Xu2CIKCAs9i4m1bv5bBIGWgMWux7EYJVu0GcQNsHhbmsmqPSJjuMzxP6ul56 00xV+eP1IMJ00EG3SVdw3S8Vz5LvgllRyVQ= X-Google-Smtp-Source: AGHT+IHbZnYiY/BuJoWRHHRQ6L5BtEhuC3x+eZrXL9ezJSr88dEBvO1rKsdBYHgZC7piIso+1awHyQ== X-Received: by 2002:a05:690c:48c5:b0:6fb:9b8c:4b50 with SMTP id 00721157ae682-703e154ea64mr257126007b3.13.1744069353085; Mon, 07 Apr 2025 16:42:33 -0700 (PDT) Received: from localhost ([2a03:2880:25ff:1::]) by smtp.gmail.com with ESMTPSA id 00721157ae682-703d1e57c4csm27969137b3.49.2025.04.07.16.42.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Apr 2025 16:42:32 -0700 (PDT) From: Nhat Pham To: linux-mm@kvack.org Cc: akpm@linux-foundation.org, hannes@cmpxchg.org, hughd@google.com, yosry.ahmed@linux.dev, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, len.brown@intel.com, chengming.zhou@linux.dev, kasong@tencent.com, chrisl@kernel.org, huang.ying.caritas@gmail.com, ryan.roberts@arm.com, viro@zeniv.linux.org.uk, baohua@kernel.org, osalvador@suse.de, lorenzo.stoakes@oracle.com, christophe.leroy@csgroup.eu, pavel@kernel.org, kernel-team@meta.com, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-pm@vger.kernel.org Subject: [RFC PATCH 13/14] swap: simplify swapoff using virtual swap Date: Mon, 7 Apr 2025 16:42:14 -0700 Message-ID: <20250407234223.1059191-14-nphamcs@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250407234223.1059191-1-nphamcs@gmail.com> References: <20250407234223.1059191-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 2393580006 X-Stat-Signature: 5situd73qw9uyctdq6iwo3mtzur88os8 X-Rspam-User: X-HE-Tag: 1744069353-450499 X-HE-Meta: U2FsdGVkX1/YFwu1YxoAz95oVr9wXKU42ZqJcvq42MWRhbgLBu6SETu/SKduYQereDOg0HXjM2rxDXeIJ0VAr71PvzeRRcvf2JtUWoz00xDH6THAVHagEBGDHe1ppscvVnC2+W4ayGshicgBhvTRfXOOXJ7oE7GNR43RXBWN7/aj5wQwW97dUwS1mzuFrk861DHh6B1I0Le5pM9CmArFfrm1vOzFYeDNz9HqOqlPnkByj/El/+yBi8xibWtbzJqnYPo1pJ54mqeOi1YBxHbVPh9GTzUFGMeax7jqDCj/Jn20S7FUpW9u0OcWi4KgjI4fBgzsHbJVJA0LfXzC2/Cx5XdQENmvtZ8PvXOsy9P2yy/r6IuICNEAgbgg60xifQEEaWJXbz4s7Bp0vUf7j/ndFki0nOO1eEjKmgKnlV+AYIkuv/M5VqwMHurGzQd9M9GwDgEhlrVfndWJk8a65AMe2VJ9vpmIx8gLhdknXXF6VbgRpIgPeZc7gnWDERZ4FHBIDi1i+TtJ+uvNXH3PebnCHEYZBvn+MjTRSmg/MPC4mnunMJ927PSdBDN27VathfZXfzgpoYw/rtqAwc8lXh4dzk+D5UFrttQpeOsxyeM47jjPh1aA8Zbp5p+eqjMcvlGC8M3wrdMd0O3cz5zMF+VZ1QJ51oRE0ZtVOaIbzsg4RgQFpHnHC7Dr3S+BPT+h9RvtLSukkQ+6egLQTJxCqV8KrQIsE0EOqDTkLY3sTA7fP0qUQ+clBGUx7VadPi5zEK7XDTvx4OPwv0rjPZQapNiCEalvpPhG7dKNYddTchSWPPH5vY/kqj02fPkhPlU/XiDZzEAVwQH2yRqtUjL62f3V0JCR6qqgSabi29fUJgqqYXWEskGTL8EbMvdKW4j2aFQYUn1dhTH/MP6qLuhCTDp0nCTd307Ga9fMs1h7ZQH6YBElis99NC8R3ub3QB0tCsjm1wJHMgi6fHmSZQLHz7P hUY6hpZx ln8j0/UQ7nTQqesLcKxzX8t79zQYJ/drmT3onHzErYt/7FMtWFivU5VTXC+R5363tq5FwpgHdYV0qZN7o4EImGudvoOUENaeIOsCjSGGDyKGd6v5Ksgq/RO/vIwDteJkGKKlq8fRVipkAgN9DFQheBlZpo0cgey1hgmHA8Fq/4E5E/oa2cAYQPhPg1Kpcq5HYtj0rUxLjBhROWMUVOhAUNv5Fh5Gd8ffKm+tiyn1AAms3NIgmdVohDop04rlNk7wqXVJWc8DMPm0FV1+YHKJzS0jnjJAoo7SFi/Jlo3hCLT4ntDJUC5iAgS62AuQXp/DTM3v/UIY+ZzQW8Jkm5D1qPYwkN/psIBYxluxDUo+uND/N82lNpQMUwe3f5rDSi2LPBymY9eKdyJlMEPhkvegscds+4n7/w2rwnEnfRBiy4RyJx6JZD4JHleCIoOQ/GOnIFT08MmKohUGOhbbTNFMFGHiV6DUtRIUTVvXOrzizvVH4fKTZOx8zTaGHKg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch presents the second applications of virtual swap design - simplifying and optimizing swapoff. With virtual swap slots stored at page table entries and used as indices to various swap-related data structures, we no longer have to perform a page table walk in swapoff. Simply iterate through all the allocated swap slots on the swapfile, invoke the backward map and fault them in. This is significantly cleaner, as well as slightly more performant, especially when there are a lot of unrelated VMAs (since the old swapoff code would have to traverse through all of them). In a simple benchmark, in which we swapoff a 32 GB swapfile that is 50% full, and in which there is a process that maps a 128GB file into memory: Baseline: real: 25.54s user: 0.00s sys: 11.48s New Design: real: 11.69s user: 0.00s sys: 9.96s Disregarding the real time reduction (which is mostly due to more IO asynchrony), the new design reduces the kernel CPU time by about 13%. Signed-off-by: Nhat Pham --- include/linux/shmem_fs.h | 3 + include/linux/swap.h | 1 + mm/shmem.c | 2 + mm/swapfile.c | 189 ++++++++++++++++++++++++++++++++------- mm/vswap.c | 61 +++++++++++++ 5 files changed, 225 insertions(+), 31 deletions(-) diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h index 0b273a7b9f01..668b6add3b8f 100644 --- a/include/linux/shmem_fs.h +++ b/include/linux/shmem_fs.h @@ -108,7 +108,10 @@ extern void shmem_unlock_mapping(struct address_space *mapping); extern struct page *shmem_read_mapping_page_gfp(struct address_space *mapping, pgoff_t index, gfp_t gfp_mask); extern void shmem_truncate_range(struct inode *inode, loff_t start, loff_t end); + +#ifndef CONFIG_VIRTUAL_SWAP int shmem_unuse(unsigned int type); +#endif #ifdef CONFIG_TRANSPARENT_HUGEPAGE unsigned long shmem_allowable_huge_orders(struct inode *inode, diff --git a/include/linux/swap.h b/include/linux/swap.h index c3a10c952116..177f6640a026 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -764,6 +764,7 @@ void vswap_store_folio(swp_entry_t entry, struct folio *folio); void swap_zeromap_folio_set(struct folio *folio); void vswap_assoc_zswap(swp_entry_t entry, struct zswap_entry *zswap_entry); bool vswap_can_swapin_thp(swp_entry_t entry, int nr); +void vswap_swapoff(swp_entry_t entry, struct folio *folio, swp_slot_t slot); static inline bool trylock_swapoff(swp_entry_t entry, struct swap_info_struct **si) diff --git a/mm/shmem.c b/mm/shmem.c index 609971a2b365..fa792769e422 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1380,6 +1380,7 @@ static void shmem_evict_inode(struct inode *inode) #endif } +#ifndef CONFIG_VIRTUAL_SWAP static int shmem_find_swap_entries(struct address_space *mapping, pgoff_t start, struct folio_batch *fbatch, pgoff_t *indices, unsigned int type) @@ -1525,6 +1526,7 @@ int shmem_unuse(unsigned int type) return error; } +#endif /* CONFIG_VIRTUAL_SWAP */ /* * Move the page from the page cache to the swap cache. diff --git a/mm/swapfile.c b/mm/swapfile.c index 59b34d51b16b..d1251a9264fa 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -2053,6 +2053,163 @@ unsigned int count_swap_pages(int type, int free) } #endif /* CONFIG_HIBERNATION */ +/* + * Scan swap_map from current position to next entry still in use. + * Return 0 if there are no inuse entries after prev till end of + * the map. + */ +static unsigned int find_next_to_unuse(struct swap_info_struct *si, + unsigned int prev) +{ + unsigned int i; + unsigned char count; + + /* + * No need for swap_lock here: we're just looking + * for whether an entry is in use, not modifying it; false + * hits are okay, and sys_swapoff() has already prevented new + * allocations from this area (while holding swap_lock). + */ + for (i = prev + 1; i < si->max; i++) { + count = READ_ONCE(si->swap_map[i]); + if (count && swap_count(count) != SWAP_MAP_BAD) + break; + if ((i % LATENCY_LIMIT) == 0) + cond_resched(); + } + + if (i == si->max) + i = 0; + + return i; +} + +#ifdef CONFIG_VIRTUAL_SWAP +#define for_each_allocated_offset(si, offset) \ + while (swap_usage_in_pages(si) && \ + !signal_pending(current) && \ + (offset = find_next_to_unuse(si, offset)) != 0) + +static struct folio *pagein(swp_entry_t entry, struct swap_iocb **splug, + struct mempolicy *mpol) +{ + bool folio_was_allocated; + struct folio *folio = __read_swap_cache_async(entry, GFP_KERNEL, mpol, + NO_INTERLEAVE_INDEX, &folio_was_allocated, false); + + if (folio_was_allocated) + swap_read_folio(folio, splug); + return folio; +} + +static int try_to_unuse(unsigned int type) +{ + struct swap_info_struct *si = swap_info[type]; + struct swap_iocb *splug = NULL; + struct mempolicy *mpol; + struct blk_plug plug; + unsigned long offset; + struct folio *folio; + swp_entry_t entry; + swp_slot_t slot; + int ret = 0; + + if (!atomic_long_read(&si->inuse_pages)) + goto success; + + mpol = get_task_policy(current); + blk_start_plug(&plug); + + /* first round - submit the reads */ + offset = 0; + for_each_allocated_offset(si, offset) { + slot = swp_slot(type, offset); + entry = swp_slot_to_swp_entry(slot); + if (!entry.val) + continue; + + folio = pagein(entry, &splug, mpol); + if (folio) + folio_put(folio); + } + blk_finish_plug(&plug); + swap_read_unplug(splug); + lru_add_drain(); + + /* second round - updating the virtual swap slots' backing state */ + offset = 0; + for_each_allocated_offset(si, offset) { + slot = swp_slot(type, offset); +retry: + entry = swp_slot_to_swp_entry(slot); + if (!entry.val) + continue; + + /* try to allocate swap cache folio */ + folio = pagein(entry, &splug, mpol); + if (!folio) { + if (!swp_slot_to_swp_entry(swp_slot(type, offset)).val) + continue; + + ret = -ENOMEM; + pr_err("swapoff: unable to allocate swap cache folio for %lu\n", + entry.val); + goto finish; + } + + folio_lock(folio); + /* + * We need to check if the folio is still in swap cache. We can, for + * instance, race with zswap writeback, obtaining the temporary folio + * it allocated for decompression and writeback, which would be + * promply deleted from swap cache. By the time we lock that folio, + * it might have already contained stale data. + * + * Concurrent swap operations might have also come in before we + * reobtain the lock, deleting the folio from swap cache, invalidating + * the virtual swap slot, then swapping out the folio again. + * + * In all of these cases, we must retry the physical -> virtual lookup. + * + * Note that if everything is still valid, then virtual swap slot must + * corresponds to the head page (since all previous swap slots are + * freed). + */ + if (!folio_test_swapcache(folio) || folio->swap.val != entry.val) { + folio_unlock(folio); + folio_put(folio); + if (signal_pending(current)) + break; + schedule_timeout_uninterruptible(1); + goto retry; + } + + folio_wait_writeback(folio); + vswap_swapoff(entry, folio, slot); + folio_unlock(folio); + folio_put(folio); + } + +finish: + if (ret == -ENOMEM) + return ret; + + /* concurrent swappers might still be releasing physical swap slots... */ + while (swap_usage_in_pages(si)) { + if (signal_pending(current)) + return -EINTR; + schedule_timeout_uninterruptible(1); + } + +success: + /* + * Make sure that further cleanups after try_to_unuse() returns happen + * after swap_range_free() reduces si->inuse_pages to 0. + */ + smp_mb(); + return 0; +} +#else static inline int pte_same_as_swp(pte_t pte, pte_t swp_pte) { return pte_same(pte_swp_clear_flags(pte), swp_pte); @@ -2340,37 +2497,6 @@ static int unuse_mm(struct mm_struct *mm, unsigned int type) return ret; } -/* - * Scan swap_map from current position to next entry still in use. - * Return 0 if there are no inuse entries after prev till end of - * the map. - */ -static unsigned int find_next_to_unuse(struct swap_info_struct *si, - unsigned int prev) -{ - unsigned int i; - unsigned char count; - - /* - * No need for swap_lock here: we're just looking - * for whether an entry is in use, not modifying it; false - * hits are okay, and sys_swapoff() has already prevented new - * allocations from this area (while holding swap_lock). - */ - for (i = prev + 1; i < si->max; i++) { - count = READ_ONCE(si->swap_map[i]); - if (count && swap_count(count) != SWAP_MAP_BAD) - break; - if ((i % LATENCY_LIMIT) == 0) - cond_resched(); - } - - if (i == si->max) - i = 0; - - return i; -} - static int try_to_unuse(unsigned int type) { struct mm_struct *prev_mm; @@ -2474,6 +2600,7 @@ static int try_to_unuse(unsigned int type) smp_mb(); return 0; } +#endif /* CONFIG_VIRTUAL_SWAP */ /* * After a successful try_to_unuse, if no swap is now in use, we know diff --git a/mm/vswap.c b/mm/vswap.c index c09a7efc2aeb..c0da71d5d592 100644 --- a/mm/vswap.c +++ b/mm/vswap.c @@ -1289,6 +1289,67 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry) swapcache_clear(NULL, entry, nr); } +/** + * vswap_swapoff - unlink a range of virtual swap slots from their backing + * physical swap slots on a swapfile that is being swapped off, + * and associate them with the swapped in folio. + * @entry: the first virtual swap slot in the range. + * @folio: the folio swapped in and loaded into swap cache. + * @slot: the first physical swap slot in the range. + */ +void vswap_swapoff(swp_entry_t entry, struct folio *folio, swp_slot_t slot) +{ + int i = 0, nr = folio_nr_pages(folio); + struct swp_desc *desc; + unsigned int type = swp_slot_type(slot); + unsigned int offset = swp_slot_offset(slot); + + XA_STATE(xas, &vswap_map, entry.val); + + rcu_read_lock(); + xas_for_each(&xas, desc, entry.val + nr - 1) { + if (xas_retry(&xas, desc)) + continue; + + write_lock(&desc->lock); + /* + * There might be concurrent swap operations that might invalidate the + * originally obtained virtual swap slot, allowing it to be + * re-allocated, or change its backing state. + * + * We must re-check here to make sure we are not performing bogus backing + * store changes. + */ + if (desc->type != VSWAP_SWAPFILE || + swp_slot_type(desc->slot) != type) { + /* there should not be mixed backing states among the subpages */ + VM_WARN_ON(i); + write_unlock(&desc->lock); + break; + } + + VM_WARN_ON(swp_slot_offset(desc->slot) != offset + i); + + xa_erase(&vswap_rmap, desc->slot.val); + desc->type = VSWAP_FOLIO; + desc->folio = folio; + write_unlock(&desc->lock); + i++; + } + rcu_read_unlock(); + + if (i) { + /* + * If we update the virtual swap slots' backing, mark the folio as + * dirty so that reclaimers will try to page it out again. + */ + folio_mark_dirty(folio); + swap_slot_free_nr(slot, nr); + /* folio is in swap cache, so entries are guaranteed to be valid */ + mem_cgroup_uncharge_swap(entry, nr); + } +} + #ifdef CONFIG_MEMCG static unsigned short vswap_cgroup_record(swp_entry_t entry, unsigned short memcgid, unsigned int nr_ents)