From patchwork Wed Jan 29 11:53:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13953674 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 629B1C02193 for ; Wed, 29 Jan 2025 11:54:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C777F280009; Wed, 29 Jan 2025 06:54:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C2737280007; Wed, 29 Jan 2025 06:54:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC85F280009; Wed, 29 Jan 2025 06:54:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 89C2A280007 for ; Wed, 29 Jan 2025 06:54:22 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2726580C0F for ; Wed, 29 Jan 2025 11:54:21 +0000 (UTC) X-FDA: 83060331522.26.86435C6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf04.hostedemail.com (Postfix) with ESMTP id CB9EF40008 for ; Wed, 29 Jan 2025 11:54:18 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OBptNJsb; spf=pass (imf04.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738151659; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=NTXcI2/JXi5J0nSE2GsUn3l1ydd00sgNvgVHjRNY4To=; b=DoZimGh//UjKClfqFiiOXFTsl/gSBc5AihyMAypcjZNYxhRRtuLAEEh57quvzht+vjR2zt n2d3IobAEGkG+poiuyBtiCmKnuvFPbBlKPKoAqnNFsXmkDzT4xjpPsbGqXvMYJk51gUMNC R4Z9CljcXt5BmNZoTMy2JVg8fDbDync= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738151659; a=rsa-sha256; cv=none; b=koP/GO0kHeK0NZR4OvBtDOZfEATH4+/HGojEcnAIkLgxIQ41p32vK3G0UGDa04z1lpseIC Xo0HPy1u+dPs9pCC9jZMX/3Tuy2z0gpyPCxnSQNtURkiowtk0cqrFmQmr2MD4t0Hva+Vqc 118x3W1yQfnIsaTXrW49n0V6+TmMQ5E= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OBptNJsb; spf=pass (imf04.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1738151658; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=NTXcI2/JXi5J0nSE2GsUn3l1ydd00sgNvgVHjRNY4To=; b=OBptNJsb2/SmiJ+isgPuV5/ZT61nCaND63BlIGLtddIcqcnccekWPu3ysucDoBM2nUc36a KCJC1of3f3YFJ7a7wE5XQf2olUaknw5XmsaK8CK0YBJITXL/j+SEa1iuPZADJkpnR2srTj bUL2vPcROkKPeJ7MnDsfQwOD7JK8bO4= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-617-w6ni47cWOEq49kM9SYmPJw-1; Wed, 29 Jan 2025 06:54:17 -0500 X-MC-Unique: w6ni47cWOEq49kM9SYmPJw-1 X-Mimecast-MFC-AGG-ID: w6ni47cWOEq49kM9SYmPJw Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-43624b08181so3397545e9.0 for ; Wed, 29 Jan 2025 03:54:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738151656; x=1738756456; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=NTXcI2/JXi5J0nSE2GsUn3l1ydd00sgNvgVHjRNY4To=; b=P/94qQdR8byr91ayfMq19Nw3pyZoheb3i1dWcJtgoJAxy6E4nmtaZJm8UtFD5KVRdu V3s54TIr14rNvOheJQn5MfP4jinK8ox0r08SEQZLnS7/h5iEVRSpVlpLKf2U4y++gYPg CRVcxQputfqgDbMAgsjX9Qpjj9TYbl0XvKRO8pNNDfZDiMSO0U/e3Uar0/A9NaPVvLqY zOxxYoK0ikWVWoYoHd4n9bucqyqDjToQmSPMWbzmEmTOgg8Cpxs46593Afb8ZgGZp5kB X+eqNj8RIFHGR3FOFJkPotZPf5GqY81klpN61w2NpzcxTnBo3IW+3uMX1GCQL4O8P5dC X3ig== X-Forwarded-Encrypted: i=1; AJvYcCWTpA9C5YuKhG6Eq1MhS915FWfi8JF7XA1lX6HUkc0woPuoDVx9b77amH1j7GCA0tKLM2xnkoGFbA==@kvack.org X-Gm-Message-State: AOJu0YyA8rP4QfF8OCVXGaG1nbUR9u1KeJI7KB/sNmdzbeFCQ3kgsjYu wfH/CJaCnLOhAvPLdoFfrBGyAqIUy8BYpWonURyInNirupnKQ/T2LlzPVd6SqDpIM4XvwtOM8oA zrAVosP+7sebqUMY9f1jJcYsvPFfpjAECwUPZRPpDbrfeYprQ X-Gm-Gg: ASbGncsYM1rTBjHqufaWLnqiP/0MyakoUVwQwcsPyexB5Oy+4mmkJZLMZHT0g+A8vcS 9J8ENex1KUmDCaZdypDOQJog2D9mWRouqE9n7KcOmooCOIsMqoylG4D96j6ZYLZPo1WnydUpjTI B/KPwJMGKONJQ8up5FryiHJWDPvveCB0uQLkoEYtO1WO+rWoNImC44yrMB1r8/GKjFjfUEz4ARX QUk/9YNktURQcqiTmU7ivcByXt1ub6g0OOOP3YUXS3ZgwCGIkrmppYAIjjKiROHxWAQ1gzpyxRG UXuh4QPaC3JYWvIM5g2hB74iPtTROd60rrFSCoyDfzEapX+OdMW838O6rkSBrdM8CQ== X-Received: by 2002:a05:600c:244:b0:436:1b77:b5aa with SMTP id 5b1f17b1804b1-438d5967dbfmr59308515e9.8.1738151654600; Wed, 29 Jan 2025 03:54:14 -0800 (PST) X-Google-Smtp-Source: AGHT+IHA1IH33KcDzR5tSVGfetIObwrZb+AzQ8Bt78Bds+XQKoxAw+M++eXXCVeUlojayhqR4Mp0lA== X-Received: by 2002:a05:600c:244:b0:436:1b77:b5aa with SMTP id 5b1f17b1804b1-438d5967dbfmr59308045e9.8.1738151654065; Wed, 29 Jan 2025 03:54:14 -0800 (PST) Received: from localhost (p200300cbc7053b0064b867195794bf13.dip0.t-ipconnect.de. [2003:cb:c705:3b00:64b8:6719:5794:bf13]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-438dcc2f73asm20350435e9.24.2025.01.29.03.54.12 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 03:54:12 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v1 00/12] mm: fixes for device-exclusive entries (hmm) Date: Wed, 29 Jan 2025 12:53:58 +0100 Message-ID: <20250129115411.2077152-1-david@redhat.com> X-Mailer: git-send-email 2.48.1 MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: zJTYxCm23Fj-0L49v0nTcH8pt7N-DOTFKKH0aNAyEJg_1738151656 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: CB9EF40008 X-Stat-Signature: ihfqdk9c7pk6i65gn1ujn77qzop9yogy X-Rspam-User: X-Rspamd-Server: rspam12 X-HE-Tag: 1738151658-474707 X-HE-Meta: U2FsdGVkX18mV429pMiEKz/y7jmrA+5wE7EqR0f+DK8QSCjcstSV7KuA6JdZI/3AerVV9RKodQRjyBbNLX9letbRH3UqfaQAK+FeA2Nhnhl3He9zB7JMW+iy3Lrr+St2slb5/Cj5j3R3xBDMOls/4ciCK50pVEsDKBSoFqMI3Cb8N3c/uGa0pj+17Ew1G5Gz1FQU57UMPIukyBjcft7BES3PIoS6DbMjI1UTFi8ksozlT+MyeFubXfv0YDyu2Fn6CG6DrVzUy19JdYfVkmplMd7BXTznDJMIQah1b2FWgCxSV+dqh5iWvpRHkogp//PBgs+1TJqHRnjj4Qs6kSmJ4jVvby5z1sE4autNnxqrZqpzn8Xj7E31dMjxxU7wT6HyY75DrQt5BFKyHvUpjdKbqdMQehEU94RqioebAnSo1b9FZ9fqFO0VcYEoKsMY3XnV2GlWmUeLg66PQnvA0Sp7Nmmv9ckyls1Gqz7j9o4eRMTundvS5OKvLEi7ecu3I+KAN0knoZ7t2YvUCtpkSQ2AkzVyi6yNd/Vj+WRUZ+0jX+W6KkSpNlNKhIgNY4Yc1sXMIczAzDGGXToFXI6TYccOn4LyQfaS6i8thS4Pv9MH//blB/HfQWK+HcdlzGmfDJPBiPZriTO5t4N4o3PfMzmPjRrf/UQWIfrrZ4ajM7Zybdf9MGANzV3f84LQLYX3Mqa7ZxU7SPP/GxrDG/HYAG1l8L5dBrVTl24hYwDAbFoXXX3Oyv8JRyisSa9FoRo/aYWcMjoB5cwQ68XCu3egcia6h9Mpa88un3Hr3Log+erPYuMFeC1cEzimr4fHCJBrCh9UcSgVi7AZSO15spAQTFQe2VX7PBU7dlpY20N7Ki3tMhnVO6hb1XoHW2y3ezL4V7MkFm0A7f1J7bSSj6/A8WyG1AHlpeDUafwt/ys6FeRrbDBdvoALUIIvyZppm3wS/vddCO+f62ttcV4YrT+aZSJ VTWmUm95 27jaVg5yM/h5HOl8/lmlK0acU9qRM/o7V707miuDxaluZZpDDQr6o5NZGRj0QKLMgAWAFrE0OIimPhsnmkWdHjSS70y8ZUSSCIKVgKi29cqzqJ0QivtjIs0WBEanK97WJW6apDEUQJ9YAFz9Nzi+b2VhyecUM7GKCJpzW9pLji9UpclO6Ta5yzzmDD27xa/Mujx91FPUD6m/KCeyyxegvJYK2+lwg0vD+MgKsmyXAScyR11KJzMEH2lTAYps8ZQly26YGOp4gvLKhuH7I58ojV1FwzgR/Hn115fFmudaLoFDFUigWK4KFuIvOlE3BkJ+sp5+ZplzaMHUb88hkuuyzTOeADQG0wvOIszBjFRsCcXFCmQBSdpnN1UqxvrFXsDF7nxyBFHmJ/Jp72jNcD4aGV6IyBeluDSQJM0oLW4TWkbO1P5bVD/c1kBu9rwosfWXGOMlLEnPCYMBkHMOPV/6ouQPkMoWBa9JCsMrpIA4BpU2xAO5N+cEAZRSON1QYJ0drgdk9WQTWyiOIlD3mt3fYW12Kb5qNKqZX53YYhAO6vz6Wb+myiZ8aF7i0PFFTfEZwo0VtKkJQLFo8rnH5WnhjOPF/vxjdvDv0G8ptUPNkycqQxUPlUPsF3LSOmmc2fxv712fI5Fp7vKmTgmTB2Kt6zUHOgsVjJXTuDAAyRzBHQ/NX2EA0Mv+DFezQHH95B0c6BUFPQJNCosAmFIEylb0PBu5Y/MWQS4NGz6KSI8YwfbODPy7pJt0OHSBziAJ5OUvaPT2qgGhyyeJXYTp7SOX0ouc4p1fh742ZY7W+sM6e9fMLOPM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Discussing the PageTail() call in make_device_exclusive_range() with Willy, I recently discovered [1] that device-exclusive handling does not properly work with THP, making the hmm-tests selftests fail if THPs are enabled on the system. Looking into more details, I found that hugetlb is not properly fenced, and I realized that something that was bugging me for longer -- how device-exclusive entries interact with mapcounts -- completely breaks migration and swapout of these folios while they have device-exclusive PTEs. The program below can be used to allocate 1 GiB worth of pages and making them device-exclusive on a kernel with CONFIG_TEST_HMM. Once they are device-exclusive, these folios cannot get swapped out (/proc/$pid/smaps_rollup will always indicate 1 GiB RSS no matter how much one forces memory reclaim), and when having a memory block onlined to ZONE_MOVABLE, trying to offline it will loop forever and complain about failed migration of a page that should be movable. # echo offline > /sys/devices/system/memory/memory136/state # echo online_movable > /sys/devices/system/memory/memory136/state # ./hmm-swap & ... wait until everything is device-exclusive # echo offline > /sys/devices/system/memory/memory136/state [ 285.193431][T14882] page: refcount:2 mapcount:0 mapping:0000000000000000 index:0x7f20671f7 pfn:0x442b6a [ 285.196618][T14882] memcg:ffff888179298000 [ 285.198085][T14882] anon flags: 0x5fff0000002091c(referenced|uptodate| dirty|active|owner_2|swapbacked|node=1|zone=3|lastcpupid=0x7ff) [ 285.201734][T14882] raw: ... [ 285.204464][T14882] raw: ... [ 285.207196][T14882] page dumped because: migration failure [ 285.209072][T14882] page_owner tracks the page as allocated [ 285.210915][T14882] page last allocated via order 0, migratetype Movable, gfp_mask 0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_COMP|__GFP_ZERO), id 14926, tgid 14926 (hmm-swap), ts 254506295376, free_ts 227402023774 [ 285.216765][T14882] post_alloc_hook+0x197/0x1b0 [ 285.218874][T14882] get_page_from_freelist+0x76e/0x3280 [ 285.220864][T14882] __alloc_frozen_pages_noprof+0x38e/0x2740 [ 285.223302][T14882] alloc_pages_mpol+0x1fc/0x540 [ 285.225130][T14882] folio_alloc_mpol_noprof+0x36/0x340 [ 285.227222][T14882] vma_alloc_folio_noprof+0xee/0x1a0 [ 285.229074][T14882] __handle_mm_fault+0x2b38/0x56a0 [ 285.230822][T14882] handle_mm_fault+0x368/0x9f0 ... This series fixes all issues I found so far. There is no easy way to fix without a bigger rework/cleanup. I'll send out some additional cleanups that are not strictly required separately on top. I wish we could just use some special present PROT_NONE PTEs instead of these (non-present, non-none) fake-swap entries; but that just results in the same problem we keep having (lack of spare PTE bits), and staring at other similar fake-swap entries, that ship has sailed. With this series, make_device_exclusive() doesn't actually belong into mm/rmap.c anymore, but I'll leave moving that for another day. I only tested this series with the hmm-tests selftests due to lack of HW, so I'd appreciate some testing, especially if the interaction between two GPUs wanting a device-exclusive entry works as expected. I was able to trigger the -EBUSY in the hmm-tests a couple of times; but it doesn't retry yet. We should look into converting the folio_try_lock() into a folio_lock(), and just retry GUP immediately if the folio_walk fails in loop. #include #include #include #include #include #include #include #include #include #include #define HMM_DMIRROR_EXCLUSIVE _IOWR('H', 0x05, struct hmm_dmirror_cmd) struct hmm_dmirror_cmd { __u64 addr; __u64 ptr; __u64 npages; __u64 cpages; __u64 faults; }; const size_t size = 1 * 1024 * 1024 * 1024ul; const size_t chunk_size = 2 * 1024 * 1024ul; int main(void) { struct hmm_dmirror_cmd cmd; size_t cur_size; int fd, ret; char *addr, *mirror; fd = open("/dev/hmm_dmirror1", O_RDWR, 0); if (fd < 0) { perror("open failed\n"); exit(1); } addr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (addr == MAP_FAILED) { perror("mmap failed\n"); exit(1); } madvise(addr, size, MADV_NOHUGEPAGE); memset(addr, 1, size); mirror = malloc(chunk_size); for (cur_size = 0; cur_size < size; cur_size += chunk_size) { cmd.addr = (uintptr_t)addr + cur_size; cmd.ptr = (uintptr_t)mirror; cmd.npages = chunk_size / getpagesize(); ret = ioctl(fd, HMM_DMIRROR_EXCLUSIVE, &cmd); if (ret) { perror("ioctl failed\n"); exit(1); } } pause(); return 0; } [1] https://lkml.kernel.org/r/25e02685-4f1d-47fa-be5b-01ff85bb0ce2@redhat.com Cc: Andrew Morton Cc: "Jérôme Glisse" Cc: Jonathan Corbet Cc: Alex Shi Cc: Yanteng Si Cc: Karol Herbst Cc: Lyude Paul Cc: Danilo Krummrich Cc: David Airlie Cc: Simona Vetter Cc: "Liam R. Howlett" Cc: Lorenzo Stoakes Cc: Vlastimil Babka Cc: Jann Horn Cc: Pasha Tatashin Cc: Peter Xu Cc: Alistair Popple Cc: Jason Gunthorpe David Hildenbrand (12): mm/gup: reject FOLL_SPLIT_PMD with hugetlb VMAs mm/rmap: reject hugetlb folios in folio_make_device_exclusive() mm/rmap: convert make_device_exclusive_range() to make_device_exclusive() mm/rmap: implement make_device_exclusive() using folio_walk instead of rmap walk mm/memory: detect writability in restore_exclusive_pte() through can_change_pte_writable() mm: use single SWP_DEVICE_EXCLUSIVE entry type mm/page_vma_mapped: device-private entries are not migration entries mm/rmap: handle device-exclusive entries correctly in try_to_unmap_one() mm/rmap: handle device-exclusive entries correctly in try_to_migrate_one() mm/rmap: handle device-exclusive entries correctly in folio_referenced_one() mm/rmap: handle device-exclusive entries correctly in page_vma_mkclean_one() mm/rmap: keep mapcount untouched for device-exclusive entries Documentation/mm/hmm.rst | 2 +- Documentation/translations/zh_CN/mm/hmm.rst | 2 +- drivers/gpu/drm/nouveau/nouveau_svm.c | 5 +- include/linux/mmu_notifier.h | 2 +- include/linux/rmap.h | 5 +- include/linux/swap.h | 7 +- include/linux/swapops.h | 27 +- lib/test_hmm.c | 45 +- mm/gup.c | 3 + mm/memory.c | 28 +- mm/mprotect.c | 8 - mm/page_table_check.c | 5 +- mm/page_vma_mapped.c | 3 +- mm/rmap.c | 457 +++++++++----------- 14 files changed, 246 insertions(+), 353 deletions(-) base-commit: 4845035bec80db8d716a94b80bb5593bf1d69270