From patchwork Wed Feb 26 00:14:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13991157 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89118C021B2 for ; Wed, 26 Feb 2025 00:14:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 00E846B0083; Tue, 25 Feb 2025 19:14:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EFEFA280001; Tue, 25 Feb 2025 19:14:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D79216B0089; Tue, 25 Feb 2025 19:14:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B8C726B0083 for ; Tue, 25 Feb 2025 19:14:26 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 71BD5B0B11 for ; Wed, 26 Feb 2025 00:14:26 +0000 (UTC) X-FDA: 83160174132.21.A508421 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by imf14.hostedemail.com (Postfix) with ESMTP id A4388100006 for ; Wed, 26 Feb 2025 00:14:24 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=f7pGeaqb; spf=pass (imf14.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.216.54 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740528864; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=9DIRO3luX58Fb692QN92PavGqb9xzVHLwNZR8FZJ9W8=; b=J6EyuHGiQEkcJsHPHwv9boLXvlD/nUCkQd0KaItSltYx5Kn6uB1iIMS7SwDilf1jlY3Gr7 pSRyjNIaqNXcttZ62d3aHTIjFKxwdVheLXjDkCoexFwXcTfXGhSsCOoEXKp6Hkg+CiNmUi dXV8I0mi8KYpWjC+wyi7lCd+O32o6QU= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=f7pGeaqb; spf=pass (imf14.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.216.54 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740528864; a=rsa-sha256; cv=none; b=JamX39zJOp++KpQbuMQ63tn0QSWhpSEqu03ccmpdyhBS+sK4ti21hei+YSNRkoeECE0YPc JxH/y9cN2vRxcuLefTve9PoRcKFuh0Gpv9KZ7Uopa1znFc6yadLJShueEu/pkABnAgqjt5 S5zwYpjYEZa8W+gn8x0OVrK6rNVhaQA= Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-2fc33aef343so12503718a91.1 for ; Tue, 25 Feb 2025 16:14:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740528863; x=1741133663; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=9DIRO3luX58Fb692QN92PavGqb9xzVHLwNZR8FZJ9W8=; b=f7pGeaqbgx4Y/xCjtZiihnMJyR/LeVhMTVisS6eMQbx2cl8B9GgwsZCWld73enm2zb Kx5Te4rXsw6uD9+0zaUyq0fWegYlqNfHuajZ+zO37PktC/12sC/IkJkDcq37v2wVisVY 8XwOOxneNpvke14P7aWnxOGH7MM0I9Y7qRg5z/vgbVuc4b/yo458RM5uvPFv5udM5Wfd V7QXqvSg/Dh7+kzaM33jfHFm0TW/f8FfMiTMX0oatzlGhOvABU620L66TYrxU0CxR4BA r2IjvAWR+6gXzotECzK5oNZ+XNV+FhgiHJ3kJWhq/+Djimxjts1KqGLO88nscE4g/ymX 33dQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740528863; x=1741133663; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=9DIRO3luX58Fb692QN92PavGqb9xzVHLwNZR8FZJ9W8=; b=n4HjILoZ5imaB0nBS0Fsc8/LrueSvr0QfNgz4exIXcRXu4R2gaD4hMD6DCZWoCtEnc i2P5CtQPr2lOJY6ELx0bOtkOEHJfhEffuqNjAMB5eC6eTA4p5asELHlDDCv1zcmA6xAI hyQkkDz0pIf2Iy/aHkRp08F1LErgDyjGwomirfcClz88Bt1hSpmMdS+0cxohI56kN9wU XEzS988qoGryXQZupKTUlv6ArEnIcWuHANBf/A7YzYHAwEFXKI129k+y6MCr60qtU7ik dcdtlXXk7GHh33bzj2qsyWM37SRZIOzmIhreZmlqvnZv+Vf5PIHNX19gMtBlWDwTrdnR bVvA== X-Gm-Message-State: AOJu0YwTt/hflQZIBVUiMiAynIsqQbG1eRG1TrcWw0/00IpQqzCNLG9H mVK5jireOIHfECk1IdkCONNdA2s55LubxAboYWCgvOE2saUECVddKeCcGoYZ X-Gm-Gg: ASbGncsQd/zoqzX0u3Qb9JIRrLdqBNKm21aAM7hgP7D8RGqibPamJIDKMhC5I15l0yy 5VWwXv6Mgim0XWPIlghxPiYRJ2L4gLDiNQkyPe0blgzCIasIkK2wSB1MBlqVG2lSeYk6hIbFU2Q ZxPMCpvIsFZj2Rt/yACkt5mJfHAAnjNJCg1zThdnAlNSJyLbHd25LDyK8hk3Tt8teFdCWtxiBty tbbl00pTVsBySqCjQt6aE5aM7ET+LG5apJsUAX//DvZCpSTrGea6qStUxpq0/pHZD8QPvjAHI4D kv1rRCEg7ZYmxiP0yi7PukH0vd+QHkPvZbyQf4uuNA== X-Google-Smtp-Source: AGHT+IH+tcF/meu04KjAIJyWYXTnvqP6CN/RslpggeabdBy9bUmHRhVjt7A5yw3T3Iv9NtcRfUig6A== X-Received: by 2002:a17:90b:53c4:b0:2ee:70cb:a500 with SMTP id 98e67ed59e1d1-2fe7e2eb08emr2032332a91.1.1740528863104; Tue, 25 Feb 2025 16:14:23 -0800 (PST) Received: from Barrys-MBP.hub ([118.92.30.135]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2230a0b2975sm20057815ad.259.2025.02.25.16.14.14 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 25 Feb 2025 16:14:22 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, Barry Song , Andrea Arcangeli , Suren Baghdasaryan , Al Viro , Axel Rasmussen , Brian Geffon , Christian Brauner , David Hildenbrand , Hugh Dickins , Jann Horn , Kalesh Singh , "Liam R . Howlett" , Lokesh Gidra , Matthew Wilcox , Michal Hocko , Mike Rapoport , Nicolas Geoffray , Peter Xu , Ryan Roberts , Shuah Khan , ZhangPeng , Tangquan Zheng , stable@vger.kernel.org Subject: [PATCH v2] mm: Fix kernel BUG when userfaultfd_move encounters swapcache Date: Wed, 26 Feb 2025 13:14:00 +1300 Message-Id: <20250226001400.9129-1-21cnbao@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: A4388100006 X-Stat-Signature: di8ug5ocbx5gzcnuynfck8pq9e87mt6o X-HE-Tag: 1740528864-289249 X-HE-Meta: U2FsdGVkX1+iS8tAvA3l3F7pVGNteiKlMqbwzQoHm9uhYpYu/CCFWT5aG0DOse89Qpz2aT+w+yYsooPP/I/fTEKqHtJpnHzkht9F1So0Qdst4EtlqCEpgGxrnf9zmuTNDPrZSjrWu2nxJSqkX8Ozf1R0MlmamaBj6WR3nArpsglmXoVXI1ZLPkVA/rWr6BWtVEQq+35lbdt+5y0Odb+3Zd18SwkZmaVkp/fFa8QGVEBeuMfjGUEEKV5cBjnMKQ9/w7LMRr/wMoKc6oY1iJV/XN2mC5xhWXSxrz8LRzkYSGVhqNcP2X5rnm44OP5tkrj+QHwtQCSnQoiDNcXOpAQESK4XU1x2LC1DqzQ+s33IA2FVHThSteAGJ9PRH3UA52X/GyUyxrfD08FXc1/Yvy/7oP9fg6iSr9sD607JuWSPILNRQa43wQ1FGOKmXCUA/kOFr0PTJVEhSqg0puZz3MjQFxl8ac5RIMbDNhmEgV+Cnkrw08eagEEnwLyQ5ePTvNO30zJbbrkbKkzKKEJrXAgs0ul0La8pQ7Z71vG+AByY6sQq1t9TF1NaMbo5219hOzlLESJcvMLG4bng6lMGBnvhnN3o6GhjCyNHr0rCmrLWm90OMhEEL8Gg/+W+eFheaf5yueLeyZ+VrTL6fz6aeOestEBN+yKidqjQ60XU8BScrVFzAJoc6yyEmVR1sRAxonAhRECrTmhQwQMZgT04JBmk5Wc0jw618Csimgb7j6L5bvYqj+eSBYApKcScpzl7aJQ45CfUps4FZGfAvuWfoj0hg2IdfdgWeRja/yyHmbQd5VEXj8YtCJX3NCFU1lDAZuJUiQrvASoJCD8iGH383ctYvlJ1p8HlCLrdTwnkA617N/lu99BapkhSXHhcMKg4glP912fvGxj59SDuwLeqAinGyGyVbvDKc3KyxZVcGOSGNcVrcQQ4jSLlGi2VspTBcWWGzOL+I3APug+42a90FQl VmXpgR6a tU1E9k+WFmQBKaUJjRswE++oWxc7lkXZ0oZgjxjCwoO8nnDa5tHtRbkCGiqJSty2HaW2xfAg06zQvQNKAvKFnCpa1PE3MLfB2cntnEvmlr2rDrQ18bBkL4IZAGCrecsYleJM0SHl0NWxpmcb33iNUNKRRpl516DfhKQyLp9uS8c3MxjIqPja8P0F5n2aMma3lYs4L716GjBQ7zmSTUF/0w0oh6X7AE+soH2oA3/CAE0GHULowYxEZhOEAA9hMtn9+jFIVGChpySFJz9k379zcInNCnb8oK+Nd18QVs2WgV71i9bWN+aeA6I7IN4akhBLuJA9JNDlWtJ/QcI34z5hHCz20rG9ShnMBGxNWoLAidfqd/bNdWqLuPANfwjpB/JfLDfHhM5voDkBw+lR6zSxAhYxMLfn/73gDl5KKy2hhUazdTp1D81/wz6gU9C/y4RDgI0X5rlQq3JKSfRnIe10FR83efEQHAP6TNSV2/j2LPGQKcTSSxvBP56HC34AEPXCLWHZs/m/i+smHPrs8iHTyTRVGrnTst338Wywlh+l64K4Qtc+YzzbOtsv1wgZVYYblfsXSTSXF9nX2Bs53x9/JtZr9YNhyLha5I9kHiD2rVEgwuZPqKheggOWdcmZxmNbmaOEFsYXeg70b5RKDwEjjufV85PNuIHNevTCOhkOG+PFE27A1GSzJYtxXZTTkfvTKj1x1jONuFW7rj0KJSdW/SDhwAGN6ZSihZI3UmkS+MgETXjnw61tFDazm+eD8rNl3iIIKMfKNV8G62xOYsosKljKh5A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song userfaultfd_move() checks whether the PTE entry is present or a swap entry. - If the PTE entry is present, move_present_pte() handles folio migration by setting: src_folio->index = linear_page_index(dst_vma, dst_addr); - If the PTE entry is a swap entry, move_swap_pte() simply copies the PTE to the new dst_addr. This approach is incorrect because, even if the PTE is a swap entry, it can still reference a folio that remains in the swap cache. This creates a race window between steps 2 and 4. 1. add_to_swap: The folio is added to the swapcache. 2. try_to_unmap: PTEs are converted to swap entries. 3. pageout: The folio is written back. 4. Swapcache is cleared. If userfaultfd_move() occurs in the window between steps 2 and 4, after the swap PTE has been moved to the destination, accessing the destination triggers do_swap_page(), which may locate the folio in the swapcache. However, since the folio's index has not been updated to match the destination VMA, do_swap_page() will detect a mismatch. This can result in two critical issues depending on the system configuration. If KSM is disabled, both small and large folios can trigger a BUG during the add_rmap operation due to: page_pgoff(folio, page) != linear_page_index(vma, address) [ 13.336953] page: refcount:6 mapcount:1 mapping:00000000f43db19c index:0xffffaf150 pfn:0x4667c [ 13.337520] head: order:2 mapcount:1 entire_mapcount:0 nr_pages_mapped:1 pincount:0 [ 13.337716] memcg:ffff00000405f000 [ 13.337849] anon flags: 0x3fffc0000020459(locked|uptodate|dirty|owner_priv_1|head|swapbacked|node=0|zone=0|lastcpupid=0xffff) [ 13.338630] raw: 03fffc0000020459 ffff80008507b538 ffff80008507b538 ffff000006260361 [ 13.338831] raw: 0000000ffffaf150 0000000000004000 0000000600000000 ffff00000405f000 [ 13.339031] head: 03fffc0000020459 ffff80008507b538 ffff80008507b538 ffff000006260361 [ 13.339204] head: 0000000ffffaf150 0000000000004000 0000000600000000 ffff00000405f000 [ 13.339375] head: 03fffc0000000202 fffffdffc0199f01 ffffffff00000000 0000000000000001 [ 13.339546] head: 0000000000000004 0000000000000000 00000000ffffffff 0000000000000000 [ 13.339736] page dumped because: VM_BUG_ON_PAGE(page_pgoff(folio, page) != linear_page_index(vma, address)) [ 13.340190] ------------[ cut here ]------------ [ 13.340316] kernel BUG at mm/rmap.c:1380! [ 13.340683] Internal error: Oops - BUG: 00000000f2000800 [#1] PREEMPT SMP [ 13.340969] Modules linked in: [ 13.341257] CPU: 1 UID: 0 PID: 107 Comm: a.out Not tainted 6.14.0-rc3-gcf42737e247a-dirty #299 [ 13.341470] Hardware name: linux,dummy-virt (DT) [ 13.341671] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 13.341815] pc : __page_check_anon_rmap+0xa0/0xb0 [ 13.341920] lr : __page_check_anon_rmap+0xa0/0xb0 [ 13.342018] sp : ffff80008752bb20 [ 13.342093] x29: ffff80008752bb20 x28: fffffdffc0199f00 x27: 0000000000000001 [ 13.342404] x26: 0000000000000000 x25: 0000000000000001 x24: 0000000000000001 [ 13.342575] x23: 0000ffffaf0d0000 x22: 0000ffffaf0d0000 x21: fffffdffc0199f00 [ 13.342731] x20: fffffdffc0199f00 x19: ffff000006210700 x18: 00000000ffffffff [ 13.342881] x17: 6c203d2120296567 x16: 6170202c6f696c6f x15: 662866666f67705f [ 13.343033] x14: 6567617028454741 x13: 2929737365726464 x12: ffff800083728ab0 [ 13.343183] x11: ffff800082996bf8 x10: 0000000000000fd7 x9 : ffff80008011bc40 [ 13.343351] x8 : 0000000000017fe8 x7 : 00000000fffff000 x6 : ffff8000829eebf8 [ 13.343498] x5 : c0000000fffff000 x4 : 0000000000000000 x3 : 0000000000000000 [ 13.343645] x2 : 0000000000000000 x1 : ffff0000062db980 x0 : 000000000000005f [ 13.343876] Call trace: [ 13.344045] __page_check_anon_rmap+0xa0/0xb0 (P) [ 13.344234] folio_add_anon_rmap_ptes+0x22c/0x320 [ 13.344333] do_swap_page+0x1060/0x1400 [ 13.344417] __handle_mm_fault+0x61c/0xbc8 [ 13.344504] handle_mm_fault+0xd8/0x2e8 [ 13.344586] do_page_fault+0x20c/0x770 [ 13.344673] do_translation_fault+0xb4/0xf0 [ 13.344759] do_mem_abort+0x48/0xa0 [ 13.344842] el0_da+0x58/0x130 [ 13.344914] el0t_64_sync_handler+0xc4/0x138 [ 13.345002] el0t_64_sync+0x1ac/0x1b0 [ 13.345208] Code: aa1503e0 f000f801 910f6021 97ff5779 (d4210000) [ 13.345504] ---[ end trace 0000000000000000 ]--- [ 13.345715] note: a.out[107] exited with irqs disabled [ 13.345954] note: a.out[107] exited with preempt_count 2 If KSM is enabled, Peter Xu also discovered that do_swap_page() may trigger an unexpected CoW operation for small folios because ksm_might_need_to_copy() allocates a new folio when the folio index does not match linear_page_index(vma, addr). This patch also checks the swapcache when handling swap entries. If a match is found in the swapcache, it processes it similarly to a present PTE. However, there are some differences. For example, the folio is no longer exclusive because folio_try_share_anon_rmap_pte() is performed during unmapping. Furthermore, in the case of swapcache, the folio has already been unmapped, eliminating the risk of concurrent rmap walks and removing the need to acquire src_folio's anon_vma or lock. Note that for large folios, in the swapcache handling path, we directly return -EBUSY since split_folio() will return -EBUSY regardless if the folio is under writeback or unmapped. This is not an urgent issue, so a follow-up patch may address it separately. Fixes: adef440691bab ("userfaultfd: UFFDIO_MOVE uABI") Cc: Andrea Arcangeli Cc: Suren Baghdasaryan Cc: Al Viro Cc: Axel Rasmussen Cc: Brian Geffon Cc: Christian Brauner Cc: David Hildenbrand Cc: Hugh Dickins Cc: Jann Horn Cc: Kalesh Singh Cc: Liam R. Howlett Cc: Lokesh Gidra Cc: Matthew Wilcox (Oracle) Cc: Michal Hocko Cc: Mike Rapoport (IBM) Cc: Nicolas Geoffray Cc: Peter Xu Cc: Ryan Roberts Cc: Shuah Khan Cc: ZhangPeng Cc: Tangquan Zheng Cc: Signed-off-by: Barry Song Acked-by: Peter Xu Signed-off-by: Barry Song Reviewed-by: Suren Baghdasaryan --- mm/userfaultfd.c | 76 ++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 67 insertions(+), 9 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 867898c4e30b..2df5d100e76d 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -18,6 +18,7 @@ #include #include #include "internal.h" +#include "swap.h" static __always_inline bool validate_dst_vma(struct vm_area_struct *dst_vma, unsigned long dst_end) @@ -1072,16 +1073,14 @@ static int move_present_pte(struct mm_struct *mm, return err; } -static int move_swap_pte(struct mm_struct *mm, +static int move_swap_pte(struct mm_struct *mm, struct vm_area_struct *dst_vma, unsigned long dst_addr, unsigned long src_addr, pte_t *dst_pte, pte_t *src_pte, pte_t orig_dst_pte, pte_t orig_src_pte, pmd_t *dst_pmd, pmd_t dst_pmdval, - spinlock_t *dst_ptl, spinlock_t *src_ptl) + spinlock_t *dst_ptl, spinlock_t *src_ptl, + struct folio *src_folio) { - if (!pte_swp_exclusive(orig_src_pte)) - return -EBUSY; - double_pt_lock(dst_ptl, src_ptl); if (!is_pte_pages_stable(dst_pte, src_pte, orig_dst_pte, orig_src_pte, @@ -1090,10 +1089,20 @@ static int move_swap_pte(struct mm_struct *mm, return -EAGAIN; } + /* + * The src_folio resides in the swapcache, requiring an update to its + * index and mapping to align with the dst_vma, where a swap-in may + * occur and hit the swapcache after moving the PTE. + */ + if (src_folio) { + folio_move_anon_rmap(src_folio, dst_vma); + src_folio->index = linear_page_index(dst_vma, dst_addr); + } + orig_src_pte = ptep_get_and_clear(mm, src_addr, src_pte); set_pte_at(mm, dst_addr, dst_pte, orig_src_pte); - double_pt_unlock(dst_ptl, src_ptl); + double_pt_unlock(dst_ptl, src_ptl); return 0; } @@ -1137,6 +1146,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, __u64 mode) { swp_entry_t entry; + struct swap_info_struct *si = NULL; pte_t orig_src_pte, orig_dst_pte; pte_t src_folio_pte; spinlock_t *src_ptl, *dst_ptl; @@ -1318,6 +1328,8 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, orig_dst_pte, orig_src_pte, dst_pmd, dst_pmdval, dst_ptl, src_ptl, src_folio); } else { + struct folio *folio = NULL; + entry = pte_to_swp_entry(orig_src_pte); if (non_swap_entry(entry)) { if (is_migration_entry(entry)) { @@ -1331,9 +1343,53 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, goto out; } - err = move_swap_pte(mm, dst_addr, src_addr, dst_pte, src_pte, - orig_dst_pte, orig_src_pte, dst_pmd, - dst_pmdval, dst_ptl, src_ptl); + if (!pte_swp_exclusive(orig_src_pte)) { + err = -EBUSY; + goto out; + } + + si = get_swap_device(entry); + if (unlikely(!si)) { + err = -EAGAIN; + goto out; + } + /* + * Verify the existence of the swapcache. If present, the folio's + * index and mapping must be updated even when the PTE is a swap + * entry. The anon_vma lock is not taken during this process since + * the folio has already been unmapped, and the swap entry is + * exclusive, preventing rmap walks. + * + * For large folios, return -EBUSY immediately, as split_folio() + * also returns -EBUSY when attempting to split unmapped large + * folios in the swapcache. This issue needs to be resolved + * separately to allow proper handling. + */ + if (!src_folio) + folio = filemap_get_folio(swap_address_space(entry), + swap_cache_index(entry)); + if (!IS_ERR_OR_NULL(folio)) { + if (folio && folio_test_large(folio)) { + err = -EBUSY; + folio_put(folio); + goto out; + } + src_folio = folio; + src_folio_pte = orig_src_pte; + if (!folio_trylock(src_folio)) { + pte_unmap(&orig_src_pte); + pte_unmap(&orig_dst_pte); + src_pte = dst_pte = NULL; + /* now we can block and wait */ + folio_lock(src_folio); + put_swap_device(si); + si = NULL; + goto retry; + } + } + err = move_swap_pte(mm, dst_vma, dst_addr, src_addr, dst_pte, src_pte, + orig_dst_pte, orig_src_pte, dst_pmd, dst_pmdval, + dst_ptl, src_ptl, src_folio); } out: @@ -1350,6 +1406,8 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, if (src_pte) pte_unmap(src_pte); mmu_notifier_invalidate_range_end(&range); + if (si) + put_swap_device(si); return err; }