From patchwork Tue Mar 26 18:50:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13604903 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01747C54E67 for ; Tue, 26 Mar 2024 19:11:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8A7CF6B0096; Tue, 26 Mar 2024 15:11:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8580B6B0098; Tue, 26 Mar 2024 15:11:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D1686B00A6; Tue, 26 Mar 2024 15:11:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5B9666B0096 for ; Tue, 26 Mar 2024 15:11:26 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 2E1FD1C0AEF for ; Tue, 26 Mar 2024 19:11:26 +0000 (UTC) X-FDA: 81940133772.15.203F45A Received: from mail-io1-f53.google.com (mail-io1-f53.google.com [209.85.166.53]) by imf24.hostedemail.com (Postfix) with ESMTP id 43A2F18001B for ; Tue, 26 Mar 2024 19:11:24 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="h1VDjW/7"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf24.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.166.53 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711480284; a=rsa-sha256; cv=none; b=w9NlZATY0pcdHZjITFGfetVrlA2MXw7j/B0a6c/LOcoeh/v9P1Mqoe29grytOglqlwdvp5 5xddJqspUy5Me1tr4/cJSSxf+et92s/1b58edIkd+MyP0UDPzJFKYfreovaDKEBDVW/zDO WhfCccbqBsCTQF2gX7O+sEkka9xtsrg= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="h1VDjW/7"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf24.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.166.53 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711480284; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ztDoqKIOUdyGA4S9aoIHNjHCO+9b1c8CvnSrz3vjQuA=; b=wKcJe4po8BM+7s5qPr6LZdTQU8N9443fG8gAYfZE6/hXAjjlPg/aQUWViRELEKX4LVpCCZ MBAEHlMMYOYOaqWBWUXfPO4/983LtuealMntqmc9wOhT+ebT/ZdDa9IXhKF3Hl99mYHWM2 OiHeBa7JHKE4iOVbzGW8JNVifLYbv6I= Received: by mail-io1-f53.google.com with SMTP id ca18e2360f4ac-7c8ae457b27so142592039f.2 for ; Tue, 26 Mar 2024 12:11:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711480283; x=1712085083; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=ztDoqKIOUdyGA4S9aoIHNjHCO+9b1c8CvnSrz3vjQuA=; b=h1VDjW/7GI1UVCE6OKVizbAIO1GSZSmwXiJ1oB0boVZ7cMW08Oj1V3PjAAo/Fe51gl lDh9rWE7uObrQxHB+i0F30c3DapFNyqHInnx6XbyaAliLnLRESdv7He68NMH3z+//dt6 +E4xBQptzm1xh9d5xwoQ28/GSMgu6M3AGoYIwBaDSBZ/7N2o4MyUlE0e5gHjFsK+ozNO C8LeA9Wb+WgZ4tcourR57UlQyRPbkFQIdDiYUf0f+WresLuepclS9oxWC3nU4FI1NvfP pp9uDZ4nIMoapMpf3y8RSUOlG2kDjw89XxT0GbnqIvSc3uFHvK5TDNfFlTUb4Rzx7MRW sMeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711480283; x=1712085083; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=ztDoqKIOUdyGA4S9aoIHNjHCO+9b1c8CvnSrz3vjQuA=; b=tcyUcG2yFGENPYCsRhq3VIV9Fs52Y/BJThwcPr8jYEPLSEmn8+8HdSeCdv5bmezscU 9VmIJSyB3Rw2r3P/Zan5DRdQ4RCRGcsrTau7yfuz0uZ8xfQ1kCAuT/HBMPBcVOT3fBHB mRQItfxzAxvEzRHTw0hNlNPub761+9cOKxIY5ObpjKxMX8k0Z8Q1/v+em6bvhMCx7NBQ 8YXqTKZkhDe4YEBIVA6nzYWEtXcTvjYcJjnfvhfa7ZTits+NlYJf05/2cT6NLh1BxbQ/ xCMfpm3d6OhMNazrQH5upNr3spZH39YO4NF+NXUSEVECl4YbD98NX+zLgbWX/IVSNLmr ITAg== X-Gm-Message-State: AOJu0YyQVGHLm7YFYLKE+ROs8sQulARlnxKbTpte/gWmBIhuOfFDlecV MQuRfJkOaFMTf03+WsnTSez5qniyGknsRQvNKsDgAY9v2WGRNvjkKxNqlD4nlC1pa7eo X-Google-Smtp-Source: AGHT+IFhZpRE3C2jmFm0uvP6wXg7vusrz0LzQTW8Jr4o04NUgO6I7cuBwzk1y02ru2ic2EfhCl9c0w== X-Received: by 2002:a05:6a00:21d2:b0:6e6:8df5:f77a with SMTP id t18-20020a056a0021d200b006e68df5f77amr2245769pfj.31.1711479883480; Tue, 26 Mar 2024 12:04:43 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([115.171.40.106]) by smtp.gmail.com with ESMTPSA id j14-20020aa783ce000000b006ea790c2232sm6298350pfn.79.2024.03.26.12.04.39 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 26 Mar 2024 12:04:42 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: "Huang, Ying" , Chris Li , Minchan Kim , Barry Song , Ryan Roberts , Yu Zhao , SeongJae Park , David Hildenbrand , Yosry Ahmed , Johannes Weiner , Matthew Wilcox , Nhat Pham , Chengming Zhou , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [RFC PATCH 06/10] mm/swap: switch to use multi index entries Date: Wed, 27 Mar 2024 02:50:28 +0800 Message-ID: <20240326185032.72159-7-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240326185032.72159-1-ryncsn@gmail.com> References: <20240326185032.72159-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 43A2F18001B X-Stat-Signature: qt5rh39oa4mj4kem6zqi4herecstj65f X-HE-Tag: 1711480284-378087 X-HE-Meta: U2FsdGVkX19yATz5d9bBlayouTIEfuvDv2+9GzZcfJbkz+R270q++ZgjvnQr0pKW5ygUsnV32aGFs1LJ0yefqCMgHQjf9zdnxUJdbXbIZMO8tHFa8bIk+/OE9Qi/S5jDt4iEEzWPAjiOUPrWTl4wdphIU++ys5Ab/xe2lOx8u32QgmVX875OVY8aWVBmRB8MyZgVSPrPqyDUVa/hp/fz6RokgvzCAMRgSjfYh6fBjka7ciysqLT+amM5EJAQcU6C8eQ3iMkrOtvs3eJDsTg5kCZ6eaBnw3XFWogeQZ7b02kHmD8eCNY8Ao7+XPBbW4P+S0RLNpvrNPRS1ct0rfyO6gyzUZ/xKQqiY4ZukDk9/JgQ9+d5mX+6AsjTUdB2vhbT0p+wZo5vq8AF/zo4bFm9Oo/hE+wEO9aQcVYlePuiOy1h4zwSeoywyWUSMWQvbP1STrrdjUNRTJXSjY2bWHhaxaDALkif1Do++Wj2Du8j9wHIWg+2ctLMxWGv86oxzYO4R1ijO/jy1cOzF0rEC/0dKfTdXTdoHtBR0L7F6Nnvquasbg82iUtofst7cpSoezPBn4y69W7XdOyPMd4M4XCx7/6ZKp3DDILf6Ei0LneUuUq/mpnq/82E4+sBSGni/pNHBjp6CQ3zQdgF3K+eaks3UVaxEI0w+kCPMrtG/nBuFPGLXv+5BR+BkgLbQLaHwAKBxVX6dDQACQLZg4ZVlMvMYAfByEyqx00/XXdcdRcInIDtI4vzdHYIxbBiaZVc3vH6JpneMEPTSZggqzHYb39sS3d+DgqqEg9nsGQxmmL3CBXQZDrik/0aQr4puSXrWmzCrTTOrgogihQ2ClG6ow+3Lf9LHjg7jh2WmU9uMMJrtSVqDsJ8SuR3iKlpwAT0rUmq7d7mbYjFvH42+EhgWfeZ0iVS5hhMstqXJg2z/iQFaRX5icbfQkp/DsAGplcOk8eDdoGqnMRGzIfk3qWvWuS 0rr8LZIu NGa1JW72Z6Bqpmw9txduhxFnKIYhpF9HAz1pQ4qlMk6E1NtMNgPEaEDWooujIoeEPTMMrtcxibqIDTSiX1lrsXmTGKy2fU0CmOk5AS/GwjLiu4lqEjAtc1nq4SpT6DD1Rz+826LcL4MISNBW/wLE6wN/zqh+6d5ZCWL/COJGbxmacjc3Mz0Vwa2+6+ij1sc6S9a/vwMl8WRBVwrQtRcVDyMejyS7+9oL/pAhUAh2WgB/NhWO4npsaBga5JwoatqXXBc1dleBDmblRMTNTRHZFFp+SKHfl4mV5UefBeFrlVorkg2PBVGFwAJHGc7YciUGMaaUPm8pMxOFLFPFsn3StqcR1lHknllTdUhsDU70CjPdj5hjLWqtWaF8Lf2N8y/qg0022GYQrjv6GTVuQijeGdW4g8EKNA3NnruPVyVFsTCQJp/D+bO8qaBfWcYFeAAkwMbnRL9GpDGScng3HXf7C4UqSuuLsaCkxhcitWfqOcO5K0OSLh6zh7T2giyu9D3X28bOMjFl6zszjPr6JmGeWPbZ0qyQUX9g9T5be62Ca8bDmcx8eQ2cZVtgki5TYslRs95vHfI0GNzl7/E5E6k274tVXPN/VtszISlXALsvNU9vrJIMqLdD68XxnOicdn06MbgIbTrHTjxzWRKLaHWUDxbZ84g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song From: Kairui Song Since now all explicit shadow clearing is gone and all swapin / swapout path is all using swap cache, switch swap cache to use multi index so swapping out of THP will be faster, also using less memory. Test result of sequential swapin/out of 30G zero page on ZRAM: Before (us) After (us) Swapout: 33648529 33713283 Swapin: 40667696 40954646 Swapout (THP): 7658664 6921176 (+9.7%) Swapin (THP) : 40602278 40891953 And after swapping out 30G with THP, the radix node usage dropped by a lot: Before: radix_tree_node 73728K After: radix_tree_node 7056K (-94%) Signed-off-by: Kairui Song --- mm/filemap.c | 27 +++++++++++++++++ mm/huge_memory.c | 77 +++++++++++++++++++----------------------------- mm/internal.h | 2 ++ mm/swap_state.c | 54 ++++++++++----------------------- 4 files changed, 75 insertions(+), 85 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 0ccdc9e92764..5e8e3fd26b8d 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -919,6 +919,33 @@ static int __filemap_lock_store(struct xa_state *xas, struct folio *folio, return xas_error(xas); } +int __filemap_add_swapcache(struct address_space *mapping, struct folio *folio, + pgoff_t index, gfp_t gfp, void **shadowp) +{ + XA_STATE_ORDER(xas, &mapping->i_pages, index, folio_order(folio)); + long nr; + int ret; + + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio); + mapping_set_update(&xas, mapping); + + nr = folio_nr_pages(folio); + folio_ref_add(folio, nr); + + ret = __filemap_lock_store(&xas, folio, index, gfp, shadowp); + if (likely(!ret)) { + mapping->nrpages += nr; + __node_stat_mod_folio(folio, NR_FILE_PAGES, nr); + __lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr); + xas_unlock_irq(&xas); + } else { + folio_put_refs(folio, nr); + } + + return ret; +} + noinline int __filemap_add_folio(struct address_space *mapping, struct folio *folio, pgoff_t index, gfp_t gfp, void **shadowp) { diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9859aa4f7553..4fd2f74b94a9 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2886,14 +2886,12 @@ static void __split_huge_page_tail(struct folio *folio, int tail, lru_add_page_tail(head, page_tail, lruvec, list); } -static void __split_huge_page(struct page *page, struct list_head *list, - pgoff_t end, unsigned int new_order) +static void __split_huge_page(struct address_space *mapping, struct page *page, + struct list_head *list, pgoff_t end, unsigned int new_order) { struct folio *folio = page_folio(page); struct page *head = &folio->page; struct lruvec *lruvec; - struct address_space *swap_cache = NULL; - unsigned long offset = 0; int i, nr_dropped = 0; unsigned int new_nr = 1 << new_order; int order = folio_order(folio); @@ -2902,12 +2900,6 @@ static void __split_huge_page(struct page *page, struct list_head *list, /* complete memcg works before add pages to LRU */ split_page_memcg(head, order, new_order); - if (folio_test_anon(folio) && folio_test_swapcache(folio)) { - offset = swp_offset(folio->swap); - swap_cache = swap_address_space(folio->swap); - xa_lock(&swap_cache->i_pages); - } - /* lock lru list/PageCompound, ref frozen by page_ref_freeze */ lruvec = folio_lruvec_lock(folio); @@ -2919,18 +2911,18 @@ static void __split_huge_page(struct page *page, struct list_head *list, if (head[i].index >= end) { struct folio *tail = page_folio(head + i); - if (shmem_mapping(folio->mapping)) + if (shmem_mapping(mapping)) nr_dropped++; else if (folio_test_clear_dirty(tail)) folio_account_cleaned(tail, - inode_to_wb(folio->mapping->host)); + inode_to_wb(mapping->host)); __filemap_remove_folio(tail, NULL); folio_put(tail); } else if (!PageAnon(page)) { - __xa_store(&folio->mapping->i_pages, head[i].index, + __xa_store(&mapping->i_pages, head[i].index, head + i, 0); - } else if (swap_cache) { - __xa_store(&swap_cache->i_pages, offset + i, + } else if (folio_test_swapcache(folio)) { + __xa_store(&mapping->i_pages, swp_offset(folio->swap) + i, head + i, 0); } } @@ -2948,23 +2940,17 @@ static void __split_huge_page(struct page *page, struct list_head *list, split_page_owner(head, order, new_order); /* See comment in __split_huge_page_tail() */ - if (folio_test_anon(folio)) { + if (mapping) { /* Additional pin to swap cache */ - if (folio_test_swapcache(folio)) { - folio_ref_add(folio, 1 + new_nr); - xa_unlock(&swap_cache->i_pages); - } else { - folio_ref_inc(folio); - } - } else { - /* Additional pin to page cache */ folio_ref_add(folio, 1 + new_nr); - xa_unlock(&folio->mapping->i_pages); + xa_unlock(&mapping->i_pages); + } else { + folio_ref_inc(folio); } local_irq_enable(); if (nr_dropped) - shmem_uncharge(folio->mapping->host, nr_dropped); + shmem_uncharge(mapping->host, nr_dropped); remap_page(folio, nr); if (folio_test_swapcache(folio)) @@ -3043,11 +3029,12 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, struct deferred_split *ds_queue = get_deferred_split_queue(folio); /* reset xarray order to new order after split */ XA_STATE_ORDER(xas, &folio->mapping->i_pages, folio->index, new_order); + struct address_space *mapping = folio_mapping(folio);; struct anon_vma *anon_vma = NULL; - struct address_space *mapping = NULL; int extra_pins, ret; pgoff_t end; bool is_hzp; + gfp_t gfp; VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); @@ -3079,7 +3066,6 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, } } - is_hzp = is_huge_zero_page(&folio->page); if (is_hzp) { pr_warn_ratelimited("Called split_huge_page for huge zero page\n"); @@ -3089,6 +3075,17 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, if (folio_test_writeback(folio)) return -EBUSY; + if (mapping) { + gfp = current_gfp_context(mapping_gfp_mask(mapping) & + GFP_RECLAIM_MASK); + + xas_split_alloc(&xas, folio, folio_order(folio), gfp); + if (xas_error(&xas)) { + ret = xas_error(&xas); + goto out; + } + } + if (folio_test_anon(folio)) { /* * The caller does not necessarily hold an mmap_lock that would @@ -3104,33 +3101,19 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, goto out; } end = -1; - mapping = NULL; anon_vma_lock_write(anon_vma); } else { - gfp_t gfp; - - mapping = folio->mapping; - /* Truncated ? */ if (!mapping) { ret = -EBUSY; goto out; } - gfp = current_gfp_context(mapping_gfp_mask(mapping) & - GFP_RECLAIM_MASK); - if (!filemap_release_folio(folio, gfp)) { ret = -EBUSY; goto out; } - xas_split_alloc(&xas, folio, folio_order(folio), gfp); - if (xas_error(&xas)) { - ret = xas_error(&xas); - goto out; - } - anon_vma = NULL; i_mmap_lock_read(mapping); @@ -3189,7 +3172,9 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, int nr = folio_nr_pages(folio); xas_split(&xas, folio, folio_order(folio)); - if (folio_test_pmd_mappable(folio) && + + if (!folio_test_anon(folio) && + folio_test_pmd_mappable(folio) && new_order < HPAGE_PMD_ORDER) { if (folio_test_swapbacked(folio)) { __lruvec_stat_mod_folio(folio, @@ -3202,7 +3187,7 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, } } - __split_huge_page(page, list, end, new_order); + __split_huge_page(mapping, page, list, end, new_order); ret = 0; } else { spin_unlock(&ds_queue->split_queue_lock); @@ -3218,9 +3203,9 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, if (anon_vma) { anon_vma_unlock_write(anon_vma); put_anon_vma(anon_vma); - } - if (mapping) + } else { i_mmap_unlock_read(mapping); + } out: xas_destroy(&xas); count_vm_event(!ret ? THP_SPLIT_PAGE : THP_SPLIT_PAGE_FAILED); diff --git a/mm/internal.h b/mm/internal.h index 7e486f2c502c..b2bbfd3c2b50 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1059,6 +1059,8 @@ struct migration_target_control { */ size_t splice_folio_into_pipe(struct pipe_inode_info *pipe, struct folio *folio, loff_t fpos, size_t size); +int __filemap_add_swapcache(struct address_space *mapping, struct folio *folio, + pgoff_t index, gfp_t gfp, void **shadowp); /* * mm/vmalloc.c diff --git a/mm/swap_state.c b/mm/swap_state.c index b84e7b0ea4a5..caf69696f47c 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -90,48 +90,22 @@ int add_to_swap_cache(struct folio *folio, swp_entry_t entry, { struct address_space *address_space = swap_address_space(entry); pgoff_t idx = swp_offset(entry); - XA_STATE_ORDER(xas, &address_space->i_pages, idx, folio_order(folio)); - unsigned long i, nr = folio_nr_pages(folio); - void *old; - - xas_set_update(&xas, workingset_update_node); + int ret; VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(folio_test_swapcache(folio), folio); VM_BUG_ON_FOLIO(!folio_test_swapbacked(folio), folio); - folio_ref_add(folio, nr); folio_set_swapcache(folio); folio->swap = entry; - do { - xas_lock_irq(&xas); - xas_create_range(&xas); - if (xas_error(&xas)) - goto unlock; - for (i = 0; i < nr; i++) { - VM_BUG_ON_FOLIO(xas.xa_index != idx + i, folio); - if (shadowp) { - old = xas_load(&xas); - if (xa_is_value(old)) - *shadowp = old; - } - xas_store(&xas, folio); - xas_next(&xas); - } - address_space->nrpages += nr; - __node_stat_mod_folio(folio, NR_FILE_PAGES, nr); - __lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr); -unlock: - xas_unlock_irq(&xas); - } while (xas_nomem(&xas, gfp)); - - if (!xas_error(&xas)) - return 0; + ret = __filemap_add_swapcache(address_space, folio, idx, gfp, shadowp); + if (ret) { + folio_clear_swapcache(folio); + folio->swap.val = 0; + } - folio_clear_swapcache(folio); - folio_ref_sub(folio, nr); - return xas_error(&xas); + return ret; } /* @@ -142,7 +116,6 @@ void __delete_from_swap_cache(struct folio *folio, swp_entry_t entry, void *shadow) { struct address_space *address_space = swap_address_space(entry); - int i; long nr = folio_nr_pages(folio); pgoff_t idx = swp_offset(entry); XA_STATE(xas, &address_space->i_pages, idx); @@ -153,11 +126,9 @@ void __delete_from_swap_cache(struct folio *folio, VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio); VM_BUG_ON_FOLIO(folio_test_writeback(folio), folio); - for (i = 0; i < nr; i++) { - void *entry = xas_store(&xas, shadow); - VM_BUG_ON_PAGE(entry != folio, entry); - xas_next(&xas); - } + xas_set_order(&xas, idx, folio_order(folio)); + xas_store(&xas, shadow); + folio->swap.val = 0; folio_clear_swapcache(folio); address_space->nrpages -= nr; @@ -252,6 +223,11 @@ void clear_shadow_from_swap_cache(swp_entry_t entry) xas_set_update(&xas, workingset_update_node); + /* + * On unmap, it may delete a larger order shadow here. It's mostly + * fine since not entirely mapped folios are spiltted on swap out + * and leaves shadows with order 0. + */ xa_lock_irq(&address_space->i_pages); if (xa_is_value(xas_load(&xas))) xas_store(&xas, NULL);