From patchwork Mon Feb 10 19:37:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13968581 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9109C021A1 for ; Mon, 10 Feb 2025 19:38:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C93B280003; Mon, 10 Feb 2025 14:38:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 27947280001; Mon, 10 Feb 2025 14:38:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0CBC4280003; Mon, 10 Feb 2025 14:38:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E26E0280001 for ; Mon, 10 Feb 2025 14:38:16 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 77DAEC0537 for ; Mon, 10 Feb 2025 19:38:16 +0000 (UTC) X-FDA: 83105046192.03.725AD0E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 41906140007 for ; Mon, 10 Feb 2025 19:38:14 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=W859LszW; spf=pass (imf23.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739216294; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=h7zrLQOHNY2WMB4NJ6M+YTGeviHiiyTUNMzrBj8byNk=; b=VaWZsoXZaKlcNamz7FQXJpC/yWkVRiVToMxzUiyxMmakABUbWr7KqI9pWspg9NVcEGn0m4 QbAO5v0mabQfqe3d//tiuBn0+pH1R6DZTdGGKrzB0k0P4GMlAKM+5rdfoEnH7ayLMRyYM/ i1GEf3wu6xzheP4gOurWvRXj0alpMtc= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=W859LszW; spf=pass (imf23.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739216294; a=rsa-sha256; cv=none; b=Zt0b9Q+Qi5rJdcXbHGWMCUHW+tbelL++1PbWWlgJDBjTdSIwZQVeU38vZJk+PMpyaom3FN 27ISpiWlLrACT5DIU9h61oEl2ERFlkd15LBdzAaAwSBu4joMtKzYUFBLIrOg5IjPvYGGWX S+uzF5/vt1IgRqJG9GEhaaW/70wg/Sk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739216293; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h7zrLQOHNY2WMB4NJ6M+YTGeviHiiyTUNMzrBj8byNk=; b=W859LszW+34Ru2XgecNoxyt7JxZTOnDIylQHB4Tg5iRRKyo3KYCVRfiSzby8is3KUBx+WW k7525ISmS1iisnzJdUvI/UP7OZt2WIuo0pqOj7v2LfZuwaZyQBea0KP2wmVm67B4RMuY6P zVjouBohtThUVgJZ0e2Fy0JJx949sJ4= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-687-mZ6Awaw0ONmPZRO-oUz8Eg-1; Mon, 10 Feb 2025 14:38:10 -0500 X-MC-Unique: mZ6Awaw0ONmPZRO-oUz8Eg-1 X-Mimecast-MFC-AGG-ID: mZ6Awaw0ONmPZRO-oUz8Eg Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-43943bd1409so9692845e9.3 for ; Mon, 10 Feb 2025 11:38:10 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739216289; x=1739821089; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=h7zrLQOHNY2WMB4NJ6M+YTGeviHiiyTUNMzrBj8byNk=; b=WTYGeDt4P9vjqA9vDAhvvMnPY/GQNPEZ3rRDXfiH99xcJfoXmTozE7d+K5lN0csVvw Nv6bVshaGM2cWCyBpYsqhTrn7lBRnFlurFCVUBT0t2t9YWKnkuaI/69BSBez+TBI68dR pwKmOwMRfIzo1HNsMbj9BPclHEjjhyhaCQ2aHFw4FM7jSyw3oqw+mDvBubpeIcxMMNDB tElHgGM73Ux8CMkokcRbHO9VIFa+8Z/nVyk8ocjmpK1wtFmFXkOdAY5BFhrPc3k+2K8i KPS4qsJ+5iT/T3TlFpLShQFKhBn3VaCyO20F5RAri24ZJQUC6TVG2cpFCQY3ePyq6gS3 JqcQ== X-Forwarded-Encrypted: i=1; AJvYcCVkXnUogRFpgOgJ8NcB7afUDGI+MKJY372d9zcX6t6fz1P0w6v1J5qi5+fsXUcgM2oW94riJX7dOg==@kvack.org X-Gm-Message-State: AOJu0Ywjw0LdfsLFCdP1g11cSJJ9jf9vvwU6PX46UBUkcu/ygM12V4qC nyY14lwB36uvr9uynF7eQuvYOUyGElqWK6cbA1VDGDUijOvC0WblvpKjZPgqA0XyhQijvLSnsJA pIfiFSXfwrl0yQiIl4OP7eD+yQSWutrSdWeWUreDiG1v1Q4nT X-Gm-Gg: ASbGncsp15cJxNrKNgWEPx7yKhrVjUD935haJIb8XNZnTK6Ajs8gLB2ZWC7ac0joZOC B+qvqdEBvhXIr4Ow2wa/9uteiIRq8/UD7FRdAhVuZDeGbETLx6Lgz4zTM/d0+5CbdoWEdUjD3Jw zQK7XcP91HZ3T2LGyer21hoMd9VAcXeojVwuHA71wtLZaK/K2eywyn+e2Jluj6gfllEWAXNEsYJ zeqT3dqO9IgvBOESmQ3r8kKCZW/acSAI/K0WsOdblfcoE99NP0yOo0jIS1KcHd4ScmMQ4FRsMy2 nA4hVdIvlilRBQsBQx8/QCiuJJmd2vn00kl8Lr9YqdFP+Zgln4mqAEObq61AS+7BdQ== X-Received: by 2002:a05:600c:3482:b0:439:42c6:f108 with SMTP id 5b1f17b1804b1-43942c6f621mr54495905e9.6.1739216289156; Mon, 10 Feb 2025 11:38:09 -0800 (PST) X-Google-Smtp-Source: AGHT+IGBNHtYM4Npt0edDE/HafeE1RKAAghby0XrE6w/BdutPKy4T8is0nT67O6E9ZhRDeehOSIMcA== X-Received: by 2002:a05:600c:3482:b0:439:42c6:f108 with SMTP id 5b1f17b1804b1-43942c6f621mr54495475e9.6.1739216288775; Mon, 10 Feb 2025 11:38:08 -0800 (PST) Received: from localhost (p200300cbc734b80012c465cd348aaee6.dip0.t-ipconnect.de. [2003:cb:c734:b800:12c4:65cd:348a:aee6]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-439452533ecsm23523535e9.0.2025.02.10.11.38.06 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 10 Feb 2025 11:38:07 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, damon@lists.linux.dev, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Masami Hiramatsu , Oleg Nesterov , Peter Zijlstra , SeongJae Park , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe , John Hubbard , stable@vger.kernel.org Subject: [PATCH v2 01/17] mm/gup: reject FOLL_SPLIT_PMD with hugetlb VMAs Date: Mon, 10 Feb 2025 20:37:43 +0100 Message-ID: <20250210193801.781278-2-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250210193801.781278-1-david@redhat.com> References: <20250210193801.781278-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 1k9KqKSNz77jKob1q0cx_CFPZetIfN7e-a4gn-n3Ay0_1739216289 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Stat-Signature: kduzqzabrix5pbxab6hs5hquy647rudr X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 41906140007 X-HE-Tag: 1739216294-547764 X-HE-Meta: U2FsdGVkX19Hz7kYloDo2nmzi5KCIP9FplCZl7yt8CNqxHHIKTQYiuO2FmR/ttvazIadWzvpjLnxt3VRUDhDC4QNeLoyeFc8kjKSkQFt/xVZ1qfDyFKWHiIN3RbUu1PUAKvaB8c/atNX9howBq61fJjQwQfBhpoG4w0I3ZNwRA8XmzC/KHJkl1RS9/KANcD9tpsBAt6Xfc2sJ+0ksNPFDO6UHMWcTMyLraWIHvZIo2HV0lmt5IQu4+qfyxavVJ5I0JlU5c0+Q60oN0PJqfgPzHa7o9W0e6lRp/YO+OzJFB2thzDeiWUYJD0fZW+pf3I5afyX4g0hwD+mnoIQtwe8y6a/I+Y4W30RxVfnYGdDtaleZC9XslLnUwWgwgxaXajCM66e6+9uAIa0GycNoVzdVVgmt6o9zRL5yWi+0Q9opuujqWbG4IUPHfpHCRI3qP+Tpz5LdtDt+ojGRZcNWxPjCCeXarSHdqwC2u4E92yJexSDsSo/mrGEQImRonZcAE69gKAZt9NxC+RSSmwkO8aKp1xaHJpl3G3KAG597hZxA6LlWlHto/p6ng37o2N/EBapPhFznApZOK4PENtGPGQij9PhZ207de+tsQ+dwBZQnSmT/frbDP97PZSkAHuSAGCRrm0V2wClRLuTJ8ATwq5NYmcE7cbLyVpMXnCOYZYzUfBMXzLV6VYZnC8I87v+COKHedzbx+TRSJi/9OX2CysFWLVhsR5VfoZzQD0WR/6YdpZnEHeXYkSHTuDfxYizh2YzzybrjT7ADcTSlkWdpSOIAVVAsO0VDTXX5siznINEtkzJJN8DaaWFfsUtxu9TTd4VGG2pRf4NA8FDot/yc5V+k2U1mWgFolpNJwnqqJaYSV0eeMyB4SN9rxHT+TQYebjCw63G4rP7xZdpZvjDigLpsft94PuiD+/ng0R2ER14fiNVzZkIVq+MlYTSE7L153BdT9zssLB+c/fSW2toYWE lFnt4Ac6 6gEmk1Pabq3VmR/8+hPdTtDDrP2ORv+F4n4zhhLcXYqAB656uG4JIccDLA3s2OX0DLU29lqDGY5xs9eE8jyeQWAMA2I8arMMWCwqb1eWxQfBbWyvIw74FyOJBihJuCakQGg50/f4JX1YVqTNUpE0nyqnwbMbHuVTIdzQuMYAXkGS8UwJ738UWB0UxfatRZxF2NW/O3cOEDXQWD+Wz1Yg83a5mLkMuqShNrMXUfFWJ/nlk3e5NcM+lsBLmDiYWTru/SIHDx3XqEbiiBmF6pWbJOJ62IwPwWaiFqs2WWKQ+LIffmUWx/srHwhcnEI8skrMukrOEwOGo5pZossHTDs23PWpGMR8/VjshBwFrD5YM8CfoHyd+e3BjuWori35vnVRZTxviG6om8J0c/av1BcwY8jfElQPS68H6MsLW+qkkmXciaiekD6+KQDY79e8eFuaUmKqGnfwIHC/c+KNSuEIqs2JTbk7+tNr9bfzmrsBKwvgf2YjlKHB6g7akLP2r4n6TWORi2aPXjixQQ7VdzHoZkNb8kvHO7LrdwBRursyv56XDK+p9GQyp9dzHCln2NmdH1GPU+OlEbsplO1E= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We only have two FOLL_SPLIT_PMD users. While uprobe refuses hugetlb early, make_device_exclusive_range() can end up getting called on hugetlb VMAs. Right now, this means that with a PMD-sized hugetlb page, we can end up calling split_huge_pmd(), because pmd_trans_huge() also succeeds with hugetlb PMDs. For example, using a modified hmm-test selftest one can trigger: [ 207.017134][T14945] ------------[ cut here ]------------ [ 207.018614][T14945] kernel BUG at mm/page_table_check.c:87! [ 207.019716][T14945] Oops: invalid opcode: 0000 [#1] PREEMPT SMP KASAN NOPTI [ 207.021072][T14945] CPU: 3 UID: 0 PID: ... [ 207.023036][T14945] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-2.fc40 04/01/2014 [ 207.024834][T14945] RIP: 0010:page_table_check_clear.part.0+0x488/0x510 [ 207.026128][T14945] Code: ... [ 207.029965][T14945] RSP: 0018:ffffc9000cb8f348 EFLAGS: 00010293 [ 207.031139][T14945] RAX: 0000000000000000 RBX: 00000000ffffffff RCX: ffffffff8249a0cd [ 207.032649][T14945] RDX: ffff88811e883c80 RSI: ffffffff8249a357 RDI: ffff88811e883c80 [ 207.034183][T14945] RBP: ffff888105c0a050 R08: 0000000000000005 R09: 0000000000000000 [ 207.035688][T14945] R10: 00000000ffffffff R11: 0000000000000003 R12: 0000000000000001 [ 207.037203][T14945] R13: 0000000000000200 R14: 0000000000000001 R15: dffffc0000000000 [ 207.038711][T14945] FS: 00007f2783275740(0000) GS:ffff8881f4980000(0000) knlGS:0000000000000000 [ 207.040407][T14945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 207.041660][T14945] CR2: 00007f2782c00000 CR3: 0000000132356000 CR4: 0000000000750ef0 [ 207.043196][T14945] PKRU: 55555554 [ 207.043880][T14945] Call Trace: [ 207.044506][T14945] [ 207.045086][T14945] ? __die+0x51/0x92 [ 207.045864][T14945] ? die+0x29/0x50 [ 207.046596][T14945] ? do_trap+0x250/0x320 [ 207.047430][T14945] ? do_error_trap+0xe7/0x220 [ 207.048346][T14945] ? page_table_check_clear.part.0+0x488/0x510 [ 207.049535][T14945] ? handle_invalid_op+0x34/0x40 [ 207.050494][T14945] ? page_table_check_clear.part.0+0x488/0x510 [ 207.051681][T14945] ? exc_invalid_op+0x2e/0x50 [ 207.052589][T14945] ? asm_exc_invalid_op+0x1a/0x20 [ 207.053596][T14945] ? page_table_check_clear.part.0+0x1fd/0x510 [ 207.054790][T14945] ? page_table_check_clear.part.0+0x487/0x510 [ 207.055993][T14945] ? page_table_check_clear.part.0+0x488/0x510 [ 207.057195][T14945] ? page_table_check_clear.part.0+0x487/0x510 [ 207.058384][T14945] __page_table_check_pmd_clear+0x34b/0x5a0 [ 207.059524][T14945] ? __pfx___page_table_check_pmd_clear+0x10/0x10 [ 207.060775][T14945] ? __pfx___mutex_unlock_slowpath+0x10/0x10 [ 207.061940][T14945] ? __pfx___lock_acquire+0x10/0x10 [ 207.062967][T14945] pmdp_huge_clear_flush+0x279/0x360 [ 207.064024][T14945] split_huge_pmd_locked+0x82b/0x3750 ... Before commit 9cb28da54643 ("mm/gup: handle hugetlb in the generic follow_page_mask code"), we would have ignored the flag; instead, let's simply refuse the combination completely in check_vma_flags(): the caller is likely not prepared to handle any hugetlb folios. We'll teach make_device_exclusive_range() separately to ignore any hugetlb folios as a future-proof safety net. Fixes: 9cb28da54643 ("mm/gup: handle hugetlb in the generic follow_page_mask code") Reviewed-by: John Hubbard Reviewed-by: Alistair Popple Cc: Signed-off-by: David Hildenbrand --- mm/gup.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index 3883b307780ea..61e751baf862c 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1283,6 +1283,9 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags) if ((gup_flags & FOLL_LONGTERM) && vma_is_fsdax(vma)) return -EOPNOTSUPP; + if ((gup_flags & FOLL_SPLIT_PMD) && is_vm_hugetlb_page(vma)) + return -EOPNOTSUPP; + if (vma_is_secretmem(vma)) return -EFAULT; From patchwork Mon Feb 10 19:37:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13968582 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B649C021A2 for ; Mon, 10 Feb 2025 19:38:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1DE4F280004; Mon, 10 Feb 2025 14:38:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 18F0B280001; Mon, 10 Feb 2025 14:38:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0087F280004; Mon, 10 Feb 2025 14:38:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D5F13280001 for ; Mon, 10 Feb 2025 14:38:20 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 6425BA053A for ; Mon, 10 Feb 2025 19:38:20 +0000 (UTC) X-FDA: 83105046360.06.EB8581C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf15.hostedemail.com (Postfix) with ESMTP id 27EF6A0003 for ; Mon, 10 Feb 2025 19:38:17 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PvAMsqFV; spf=pass (imf15.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739216298; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sqeXk3kyzFMqBsxEiyyOZegmX6HzmCNZGUJC3XQma+8=; b=WXY6cY69eGU5iDUjWcgKDKaUvfbBzW1NfDlpuQB0bduVnV5wcRplFZHSgmeDVRvdL/tXpM TSf96yzH9MiE3xJQc4lJQ8Mp0CMQdq0N0TXB1EZnh5sKhLfzmgy3S8OV6W7KfTflaedJSs PMR1jGJi77usg64FIHI6RDD3fiVLBSI= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PvAMsqFV; spf=pass (imf15.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739216298; a=rsa-sha256; cv=none; b=284ltYT3svxwn5/0w1kEsWkx90sjTR0QQNWbU0oxfsHI9L3+eCNGknZ8tgan6xhDux6LaB wOUDHurnhMZROW0IE+G2ToPODdehyiwolngnev5pDXVWyjDY+u6NV6PQJMOIc0XRL/fDWo FmGUC7mCAJTj/t0ZATJUCjWIReZyJZE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739216297; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sqeXk3kyzFMqBsxEiyyOZegmX6HzmCNZGUJC3XQma+8=; b=PvAMsqFVqyLFimvCSyLi25m+PfFuhxxh7sQCbk7oBX/IZRsF/mA/OaixvocvpWHxkt0/jj uN4N5Ed4hb27y3wYZgS8gbxhGg/DrnxDnTt61OIqJVDA7zDdbLHrUP8LZaodkH4s5s6JCl 5Zk5iBgbWId32Js+IxueQOM7lYR3b80= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-582-ZD_950qMOpSKlku7iJP-YQ-1; Mon, 10 Feb 2025 14:38:14 -0500 X-MC-Unique: ZD_950qMOpSKlku7iJP-YQ-1 X-Mimecast-MFC-AGG-ID: ZD_950qMOpSKlku7iJP-YQ Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-43935bcec79so9251475e9.3 for ; Mon, 10 Feb 2025 11:38:13 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739216293; x=1739821093; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sqeXk3kyzFMqBsxEiyyOZegmX6HzmCNZGUJC3XQma+8=; b=sbQZLvq24pj6zYZ8eleTUpb4eoc02V8/SMT5L9BOnQ7bn9tO3luccWYPnzc1RSo2DH kHs4K35Cb6ElsXNn2HHvFMkFSQy/+/5M3KXBvnfu2dBhgm61qRO+Xrox9JTgPnAXPyYu RT4zfo8GpNDkHEro29BnTnyNb1VLg6BuutV7BsrKvaXBCrT/wAqAyEihpw8kzuXnVId6 GzgTqE3ogbxCXx2RL/eAHHDwEQwsjxiacxhsyXQs7+QTrmt9UttUZXMhvTBuSaKD7OQB 6GrMtFQlMDUo7wi2E5Fg0ZrgpFSHG7rSk6vSNiILm7BaGm4kqAJgC/60jkeY8+o7JgYm 0rTQ== X-Forwarded-Encrypted: i=1; AJvYcCVfGdMh6cn3SyaF5SS7TKW3YWkig1CkJlhj+nZ394pejIlIKQ7AlpJC1KbEH9cCHDyOdRqIlx89GQ==@kvack.org X-Gm-Message-State: AOJu0YywsDtFqw5j/IRmvWvUksCzc3dISZgRHqVNEepBsiFvVygr09Al 6+XQ9eTDEHqelcPE3RInhI7/Dqnw1rxEcXxMVi9anSoMvzwgNk66VmNEQSbWEsDZfocnWEkqNCk LeRmwwOL0xS59wzR6CYdbxvrA9XgyyOxH8BHc9bsoizUpvqwo X-Gm-Gg: ASbGnctCxvB6+wdAONJ+XfigNN8Th+7h+uRMqCTbJi4xobsGlN7dkCHDuoCL0hqzCKk jq0Jz+F5AzJrCBlBfiXd+VF4DfWJVw/bK6T4javuhv+52TtCoRj1+wBXVgsSUdpxQ1WK9Ko3a6d wVW7eqY+DmGPlb1WSphpUOK6iEHMYuy7oA9TfHF0ujkiWDbUzihb0DYMS2qF/c49YNut6BYe/uL rGMvXjiuNAi4No1AU+q4qIdC2d7cy2LBa4u0nhEoRgZcCvLYk/5mW13Qi3gZhuSoV/y2SXBiY/0 ebptKd1H5plS12r78A1gW5x5tKa+gKm9owy9VuEe7Z2zvrgr6JnZKqBP//8E7zRy9Q== X-Received: by 2002:a05:600c:34c4:b0:439:4b9e:461b with SMTP id 5b1f17b1804b1-4394b9e47dfmr21357985e9.14.1739216292943; Mon, 10 Feb 2025 11:38:12 -0800 (PST) X-Google-Smtp-Source: AGHT+IFz4p8WiTgagZxwnDzZ7MmocZCF9a1hUwQMJIvxH3cH5x6X+HJSMDirq9gM71FmkI7KhAXcpA== X-Received: by 2002:a05:600c:34c4:b0:439:4b9e:461b with SMTP id 5b1f17b1804b1-4394b9e47dfmr21357855e9.14.1739216292555; Mon, 10 Feb 2025 11:38:12 -0800 (PST) Received: from localhost (p200300cbc734b80012c465cd348aaee6.dip0.t-ipconnect.de. [2003:cb:c734:b800:12c4:65cd:348a:aee6]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-43947bdc5c4sm26937995e9.23.2025.02.10.11.38.09 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 10 Feb 2025 11:38:11 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, damon@lists.linux.dev, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Masami Hiramatsu , Oleg Nesterov , Peter Zijlstra , SeongJae Park , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe , stable@vger.kernel.org Subject: [PATCH v2 02/17] mm/rmap: reject hugetlb folios in folio_make_device_exclusive() Date: Mon, 10 Feb 2025 20:37:44 +0100 Message-ID: <20250210193801.781278-3-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250210193801.781278-1-david@redhat.com> References: <20250210193801.781278-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 9-kt09sEJ7Uz0wWLJvFvIFXv9mY2QoLd10Jmn4u2Cuo_1739216293 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Stat-Signature: yri3uyi4exhnqsmidd75in8y4p7yu35w X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 27EF6A0003 X-HE-Tag: 1739216297-61055 X-HE-Meta: U2FsdGVkX1/9o9WL97V2zYEKIK2Hy9GFDrotfEuvrMPd8aJ+h+eEx4QWJZhghgmW63DDJp9pffxydDIcnmM3oQEdhRxVBAkhebMHMOQdBhUAT1VoY5ywcvifa4J6qknaw0RHpyp5+3w0OMNTiyjS6LV9T0RozvHouO0O0f6UTOPuPNV+iPTe0vbotNSuYP+nQk/6zphvXXXXjXxnGTYb+mmAihBwo56nylv4NyAjcooCCAJm/J0C4FHEjQK+F7kuxKEOzIEY4te1de5nRsGWwaqRk51Z6vGTa9YEDNmYLHmdg0Vs9GcvMtIJ7hFqmQNDgDCJ1Z07WChFL3ZbFaan9DqK6HUXuQGUTOfliVpDYJqiNcZHhh+OoBi/FfpLW+B/WK1HwybcGmXfYhsphQI1bOKKQtcdjerKOr0EnYygv6RMuopJy9EvIH5NF/Cyg9KV/Qo3zWjnyrEM/N+DbYaxd2VhBA2RYALNEip6KWVHgmYGn6YsgbxBIhHEpWQ+MwbqQCNIYolQQ7TUQVpNqCqW9rUl+jExSev5w6XLpx3y5vvHTw8riq9A8RF1E3OYxknFldVrpim8RBSkaAUwtMX/Pjr+35QqVprS3wjpq3362sK13tmpK61l1dL9Y3Nc1n9tpT2hn410ZOFLC80caRVAuGMcjaAI9rJdcjP5fdwuDjPRyXvugoaMwrZxD/nClLecw0E+5I4IROjQ6xG6Se7BTYNkyvHrlnVWbAkyU5jK5BNbzu7BCe4uimsFMcuHrvtUD/F42xk/186nlqgnaMrSQDcAl4yupxu/4EHg+N29+/a4TNE33gCHJHQX4CaEmGC1ieFbv8Gc86oKplR8ZTDd+obEertvLGl0E//rYwtTTJrGgISaUjX8dLs7zjs2vditMKwvWf2XdJU/UZ98ImpLjYa4Ei06gpjN6bNahp6fxFYZecrAvoTUn+5Bz/aO5evKfRAY8xsQ7rj3FS3Sude Xwbb52hi ak43o8cul60VANf8A0AhdulJ7okjH4HlGynVcUYrCA3pbwwIS4FXG0oLBGFfHaJU7Am0g+tF8HF2VKWe+ZIQMIxzrD1y8ofilI9u3VzOreRITKTV875dIJANppL7f1TtefGDIgNajpK5g1Cldd37Qq2JgFkRFuIvJ/Ujxa+ubar5yal1730cWi/4FhNtmv2xhz7pc4yJOvIEeEWvxWNRoAWRdX6Gq8rr7dduCny3wAwJdjHq+Z+HlGG2+FZl5x3/vuEE1t/jnr7CQvCLHLHShXe4X3sT2ulE2i/9nREAqRZ+PDFUI0lUl4IcWL7EInh0sj0n7U7G+1jBOzzNQivfkhyrlE9h8cDbw9ZxAzTWLMhFTJKhoBUIfNYfin1qZKzAIeifr4SFpCd6WsF9HwcOQLp4Cqx0DFPtZjLdXAIrdEqVY6NbufaZpZ/eCsGwqGjEtL1ARpl3qX6Y9lwUuUklD/shEFbMbNiXVjZHvlxOOY/2emkIAQVqkKsYKxOhTNqO0eP8nwvr6lb9Gya6hsFVjOJO59Ax1U+v5J1yz/lCFlBys6WwvvvRfmc+jrWa3TB/hypPQ X-Bogosity: Ham, tests=bogofilter, spamicity=0.016494, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Even though FOLL_SPLIT_PMD on hugetlb now always fails with -EOPNOTSUPP, let's add a safety net in case FOLL_SPLIT_PMD usage would ever be reworked. In particular, before commit 9cb28da54643 ("mm/gup: handle hugetlb in the generic follow_page_mask code"), GUP(FOLL_SPLIT_PMD) would just have returned a page. In particular, hugetlb folios that are not PMD-sized would never have been prone to FOLL_SPLIT_PMD. hugetlb folios can be anonymous, and page_make_device_exclusive_one() is not really prepared for handling them at all. So let's spell that out. Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Reviewed-by: Alistair Popple Cc: Signed-off-by: David Hildenbrand --- mm/rmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/rmap.c b/mm/rmap.c index c6c4d4ea29a7e..17fbfa61f7efb 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2499,7 +2499,7 @@ static bool folio_make_device_exclusive(struct folio *folio, * Restrict to anonymous folios for now to avoid potential writeback * issues. */ - if (!folio_test_anon(folio)) + if (!folio_test_anon(folio) || folio_test_hugetlb(folio)) return false; rmap_walk(folio, &rwc); From patchwork Mon Feb 10 19:37:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13968583 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1B60C0219E for ; Mon, 10 Feb 2025 19:38:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7AB9C280005; Mon, 10 Feb 2025 14:38:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 75D04280001; Mon, 10 Feb 2025 14:38:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 53A31280005; Mon, 10 Feb 2025 14:38:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2CB6B280001 for ; Mon, 10 Feb 2025 14:38:23 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D353D804A0 for ; Mon, 10 Feb 2025 19:38:22 +0000 (UTC) X-FDA: 83105046444.16.53690C0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 9297E40012 for ; Mon, 10 Feb 2025 19:38:20 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cmQZ6yYB; spf=pass (imf07.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739216300; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rcUwV1yRdQas5qpumSgRDUB0x7445Uy9FJj+N197GyA=; b=FDFPQOhnDyRcJevVJtchm4gfE4GMQf/b2PQmQX7ecGesQeBC+ZFXe9xQQWQb03Ddiee56O 0yNm2vcO5MDlnCeI2Kz2NbHZCKD4PRxJ6QC7b/NiZCE2u32Rb6aLUnoBobOhVaVJhrNUdT nrZHEHuKfkbXU088QF+lnfDvtFqvtw8= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cmQZ6yYB; spf=pass (imf07.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739216300; a=rsa-sha256; cv=none; b=HSSG3kdUI8i99H2NLbHgCmS8pal0gSBxK5oekaUM5twvTrU3LyrvN4qUbII/bEOAW2HnTz e8xKbFbT8In5WvngKw4ahunjfIeVlalzpZelO+9U3kWFYagMtHvcDNgIuYvztK2hY4K5y7 vbbJXBFlhovPUs0X3sbpeSSqQt8h6KQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739216299; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rcUwV1yRdQas5qpumSgRDUB0x7445Uy9FJj+N197GyA=; b=cmQZ6yYBSUXdXDbl6NbX5AqxMQDU3CgeBVWG4wim0RIrKgLulckyOxTftDcIVvcLLmgAH7 amSpxAOhjhA3NfV2Ka6Hd8wA5QS9iz7ykfx/w8Df21dMejr4BF25TY//sEkH+UYOcaWgC3 s0akaaJAgLhykTjndXiTiv/Uf9ptTds= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-593-GJdYd6hSOSy8YNVJF-0t1g-1; Mon, 10 Feb 2025 14:38:18 -0500 X-MC-Unique: GJdYd6hSOSy8YNVJF-0t1g-1 X-Mimecast-MFC-AGG-ID: GJdYd6hSOSy8YNVJF-0t1g Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-4394c0a58e7so3345045e9.0 for ; Mon, 10 Feb 2025 11:38:18 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739216297; x=1739821097; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rcUwV1yRdQas5qpumSgRDUB0x7445Uy9FJj+N197GyA=; b=TwJee9hQE0Hv43JlWVERdYPG8NJo4a9Wtgoy0RNdTrVw/WZWabNSXMK8GpCqZl2UkC wCVLxt0vvt3tnINMWc4yefjUAtsCQXoTOzO3d0tFuQguJyBiAklMgvOEVAuEe/9P6cWO RDvQDRJbDxTU4updG/Qgbl7KO6p/kKdrHvkM1TBUITx2tNkjWrclTYNgTAQqO0mgBih2 5x7lMvBD2b2TbO+d965OWWcE1gC5tJ2bT5y7xLWrlXlatJErFM3NjIye+KHvAXZTxXUm 2outoISVYZFzvAdggQcf9lwPnuSJ9K7cqVUA3Z5Rn/+9wyJE8wjxK43n0KtD3r8hO6QB Ar2Q== X-Forwarded-Encrypted: i=1; AJvYcCXPVgwW1y9y1TMsTAXxLG0sACYh/pcrsT3r7DEuXybF+7ceNPur6QHLyyByWzjVeO7OssFMB8Cstg==@kvack.org X-Gm-Message-State: AOJu0Yx1wocTixlE26JqSBO/9Sjdgg0p4wC4KVzLdpVZy+CLhh0Co4cQ 94GbdqP/Aj8kNgPXvWDwaB0BSyqppz3zutv2gIxCYzJBR3aqJprMrcgQdrDU2oaoiBPahW+qm1C jfe0ETmkVpQB8E3a6gUvBV+xAK8nnuyr/Xna8srYdkNWIAukQ X-Gm-Gg: ASbGncujQ6dZoDwrd59ItcrCJpX5+Mt260kYsPqzg8Jl2BjVlBrM4lU8zV2DYtvs9Fv 2LZVQ/LoCBgqe+qFl1L1o5eQGjeVLNSH/hU8xPi9wZGYXu83t+dq50O375E3a7GJaGqqNF6vskW A3i7w8PREAjHBfaYE8P8VVDXFjxoVdZesS/mNBUanPOeEdP04Qw5Pb+rSd+PBbFpbFLNRnpTdH3 zznUYq0J2JBz7KmGOY77V9SYHCev56td4CZWExOL6IdOrCWmk5TIUuoh/1Xhcp8gHU4EhvVD+AJ q8y454JH9TGDMQL26EswjP1TF1+awS7NWv6zWeyWlYe4UPYwgiegokofOkzcLGT6mA== X-Received: by 2002:a05:600c:34ce:b0:439:3254:4bf1 with SMTP id 5b1f17b1804b1-43932544f7cmr76706485e9.8.1739216296988; Mon, 10 Feb 2025 11:38:16 -0800 (PST) X-Google-Smtp-Source: AGHT+IG+xVOX7QwrZYtM4UMSRPj8gjU3UfaLjkH/l+EqYjgLGDmsbA/lZxZrURGW8ZjfFBhlKrTOWA== X-Received: by 2002:a05:600c:34ce:b0:439:3254:4bf1 with SMTP id 5b1f17b1804b1-43932544f7cmr76706055e9.8.1739216296381; Mon, 10 Feb 2025 11:38:16 -0800 (PST) Received: from localhost (p200300cbc734b80012c465cd348aaee6.dip0.t-ipconnect.de. [2003:cb:c734:b800:12c4:65cd:348a:aee6]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-4390d94d802sm195253865e9.12.2025.02.10.11.38.13 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 10 Feb 2025 11:38:15 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, damon@lists.linux.dev, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Masami Hiramatsu , Oleg Nesterov , Peter Zijlstra , SeongJae Park , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe , Simona Vetter Subject: [PATCH v2 03/17] mm/rmap: convert make_device_exclusive_range() to make_device_exclusive() Date: Mon, 10 Feb 2025 20:37:45 +0100 Message-ID: <20250210193801.781278-4-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250210193801.781278-1-david@redhat.com> References: <20250210193801.781278-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: Q8JZLkrxZFZRicLBU1TO3G-CfMc9qfK76OfemVgs1eQ_1739216297 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Queue-Id: 9297E40012 X-Stat-Signature: fx99b7peqy3ue8gw58aggcwoga538qui X-Rspamd-Server: rspam03 X-HE-Tag: 1739216300-620633 X-HE-Meta: U2FsdGVkX1/iPTK2jgcML+irdtKz/TSFvvFgYJRFeM7JDEC0oCs/uMipQ/6mhqjXOvs7tK5iJ9kOr3JPoXiHd38yJN/BtWXfY9FL9xBGkxoNoLlEVABoBIs2L5AlWmutA4KAemOTKVcu0sGR5n1E/+ZspZUxiNAjKpqnhJfgPN9rt/rfC1bxqXRZJ0PtsbP45UA23nsSDXccrGvXFGuBI7DUwoMLugHtwdVgzMtmyojK/kP6IVgfjY5Akco3bVFNlmSWkR2LnSOPQhOxNRKCJHnyafP6T+9dAM9WUvAcc7om9EnwS9nFmpZ+TeNhCxQEnHSsn9raUDgOzj0OK7sLYEU5Gxf+nMx3WjGVKYjDRXX3HdvWmhFw03VZGVsd01hZx0qcD3pvKOhUq5nUBF+FGH7yZjUZzEvNy5il+KIuIOGZBx9XWGHDD0EhLe3sS9nfvhstwEUjr4lPcNbqh4n7abHcJtJ7wO2wGdFPsfpxO3ZL3OprgKOOnphjqU6LGL6BXGCCM1DTdJtnDKe6pj3wA0iYprjgy8IDXYyHTkYWAEIfiEJ1Mm4gJC9XyV1JTKQDWe1fJmou1rd0ZVva56+DI+iKRQvL2ebLxCFSrVLGMmmDPltUnKoHD8MJ6tAGazbFQD+YMg8Z6oRRd1t/6NYFjDd89pzgVSjZF5Jdei1h8/D5aXnnUy82P53hhbs6a9OI+fbdmE50e4k/v3cdYaFpydBAU/w/yfqIDg4JS04Vdp2uG64QKc6ZpAsGJJFOcZYt1hlB/B8EKTwWinO2TG+SjqeC03HTQIiicy2Zhq+DlymC6dUCZNO8Lifyi7x5GvGYjJFstBEOrzRw0sxm6bI73xsXON2jIUth6sA++VIT2iJ5ZSV+sfUdPRAsI3cNubsBBZc5exiQK6QAUtA5F6T5VI2Qtcj/v1zTNYdJwbCmDn3EbsUgRdq6DhtFJWsGUfW09LfO8p8OcLr2N6elQmJ 8kopGkMB 87CMsOYJsylIhMT++6jtyyd+tvkLcT47TV/QLPmloTZw2MwYml7XsvhCfHC9Bj7RoApdd/Tr3HvuOaZieL/+m+0Zt9lzI93kjJWjashtVd0pVibl5ttixwe4B9po0018DN9Ribn8TrNa+Q+UDl1IhsG0Hg5Fv0DPgFjkBLRKU8PObtRZa7nomiR+6dI4dGokhi6dNneAFSLguMaxBVNUE5fFdpyujlVNU1BjTn3Po5NdTERcwwhdYLhjiSFH5GT1CeZhYwhVPx4k0mCrAX4/fwAGc9kTvCB0xKvUIsr5uRlMJumO9Hhn1R40uOFVaxtMcFNV+qfU+CjnjKULrccDFKpftuPIDNlSvnOax7nQEhZ4HT/jbsnzuk+T2J1GuEMB6VhveB1UE7zAyP1R1Fj1aqnCxM4q3pEdlE9+6xWdjfK3Ojox5ncQmwqzMR7KWu4KL+jlZH9DGBx6vF+NEPimnfk0a7gAT8TanpPAkM9DhnQfofH933CXSPrs1g+oeQ/EM8h6Vv4s7xtOPAHsJkLPykOdWbbZLNz40PpDRLydtZbpXZx2x2RgiaUzIpuWUyHgBmxLgk5IQCHHbU1E= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The single "real" user in the tree of make_device_exclusive_range() always requests making only a single address exclusive. The current implementation is hard to fix for properly supporting anonymous THP / large folios and for avoiding messing with rmap walks in weird ways. So let's always process a single address/page and return folio + page to minimize page -> folio lookups. This is a preparation for further changes. Reject any non-anonymous or hugetlb folios early, directly after GUP. While at it, extend the documentation of make_device_exclusive() to clarify some things. Acked-by: Simona Vetter Reviewed-by: Alistair Popple Signed-off-by: David Hildenbrand Signed-off-by: David Hildenbrand --- Documentation/mm/hmm.rst | 2 +- Documentation/translations/zh_CN/mm/hmm.rst | 2 +- drivers/gpu/drm/nouveau/nouveau_svm.c | 5 +- include/linux/mmu_notifier.h | 2 +- include/linux/rmap.h | 5 +- lib/test_hmm.c | 41 +++----- mm/rmap.c | 103 ++++++++++++-------- 7 files changed, 83 insertions(+), 77 deletions(-) diff --git a/Documentation/mm/hmm.rst b/Documentation/mm/hmm.rst index f6d53c37a2ca8..7d61b7a8b65b7 100644 --- a/Documentation/mm/hmm.rst +++ b/Documentation/mm/hmm.rst @@ -400,7 +400,7 @@ Exclusive access memory Some devices have features such as atomic PTE bits that can be used to implement atomic access to system memory. To support atomic operations to a shared virtual memory page such a device needs access to that page which is exclusive of any -userspace access from the CPU. The ``make_device_exclusive_range()`` function +userspace access from the CPU. The ``make_device_exclusive()`` function can be used to make a memory range inaccessible from userspace. This replaces all mappings for pages in the given range with special swap diff --git a/Documentation/translations/zh_CN/mm/hmm.rst b/Documentation/translations/zh_CN/mm/hmm.rst index 0669f947d0bc9..22c210f4e94f3 100644 --- a/Documentation/translations/zh_CN/mm/hmm.rst +++ b/Documentation/translations/zh_CN/mm/hmm.rst @@ -326,7 +326,7 @@ devm_memunmap_pages() 和 devm_release_mem_region() 当资源可以绑定到 ``s 一些设备具有诸如原子PTE位的功能，可以用来实现对系统内存的原子访问。为了支持对一 个共享的虚拟内存页的原子操作，这样的设备需要对该页的访问是排他的，而不是来自CPU -的任何用户空间访问。 ``make_device_exclusive_range()`` 函数可以用来使一 +的任何用户空间访问。 ``make_device_exclusive()`` 函数可以用来使一 个内存范围不能从用户空间访问。 这将用特殊的交换条目替换给定范围内的所有页的映射。任何试图访问交换条目的行为都会 diff --git a/drivers/gpu/drm/nouveau/nouveau_svm.c b/drivers/gpu/drm/nouveau/nouveau_svm.c index b4da82ddbb6b2..39e3740980bb7 100644 --- a/drivers/gpu/drm/nouveau/nouveau_svm.c +++ b/drivers/gpu/drm/nouveau/nouveau_svm.c @@ -609,10 +609,9 @@ static int nouveau_atomic_range_fault(struct nouveau_svmm *svmm, notifier_seq = mmu_interval_read_begin(¬ifier->notifier); mmap_read_lock(mm); - ret = make_device_exclusive_range(mm, start, start + PAGE_SIZE, - &page, drm->dev); + page = make_device_exclusive(mm, start, drm->dev, &folio); mmap_read_unlock(mm); - if (ret <= 0 || !page) { + if (IS_ERR(page)) { ret = -EINVAL; goto out; } diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index e2dd57ca368b0..d4e7146618262 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -46,7 +46,7 @@ struct mmu_interval_notifier; * @MMU_NOTIFY_EXCLUSIVE: to signal a device driver that the device will no * longer have exclusive access to the page. When sent during creation of an * exclusive range the owner will be initialised to the value provided by the - * caller of make_device_exclusive_range(), otherwise the owner will be NULL. + * caller of make_device_exclusive(), otherwise the owner will be NULL. */ enum mmu_notifier_event { MMU_NOTIFY_UNMAP = 0, diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 683a04088f3f2..86425d42c1a90 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -663,9 +663,8 @@ int folio_referenced(struct folio *, int is_locked, void try_to_migrate(struct folio *folio, enum ttu_flags flags); void try_to_unmap(struct folio *, enum ttu_flags flags); -int make_device_exclusive_range(struct mm_struct *mm, unsigned long start, - unsigned long end, struct page **pages, - void *arg); +struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, + void *owner, struct folio **foliop); /* Avoid racy checks */ #define PVMW_SYNC (1 << 0) diff --git a/lib/test_hmm.c b/lib/test_hmm.c index 056f2e411d7b4..e4afca8d18802 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -780,10 +780,8 @@ static int dmirror_exclusive(struct dmirror *dmirror, unsigned long start, end, addr; unsigned long size = cmd->npages << PAGE_SHIFT; struct mm_struct *mm = dmirror->notifier.mm; - struct page *pages[64]; struct dmirror_bounce bounce; - unsigned long next; - int ret; + int ret = 0; start = cmd->addr; end = start + size; @@ -795,36 +793,27 @@ static int dmirror_exclusive(struct dmirror *dmirror, return -EINVAL; mmap_read_lock(mm); - for (addr = start; addr < end; addr = next) { - unsigned long mapped = 0; - int i; - - next = min(end, addr + (ARRAY_SIZE(pages) << PAGE_SHIFT)); + for (addr = start; !ret && addr < end; addr += PAGE_SIZE) { + struct folio *folio; + struct page *page; - ret = make_device_exclusive_range(mm, addr, next, pages, NULL); - /* - * Do dmirror_atomic_map() iff all pages are marked for - * exclusive access to avoid accessing uninitialized - * fields of pages. - */ - if (ret == (next - addr) >> PAGE_SHIFT) - mapped = dmirror_atomic_map(addr, next, pages, dmirror); - for (i = 0; i < ret; i++) { - if (pages[i]) { - unlock_page(pages[i]); - put_page(pages[i]); - } + page = make_device_exclusive(mm, addr, NULL, &folio); + if (IS_ERR(page)) { + ret = PTR_ERR(page); + break; } - if (addr + (mapped << PAGE_SHIFT) < next) { - mmap_read_unlock(mm); - mmput(mm); - return -EBUSY; - } + ret = dmirror_atomic_map(addr, addr + PAGE_SIZE, &page, dmirror); + ret = ret == 1 ? 0 : -EBUSY; + folio_unlock(folio); + folio_put(folio); } mmap_read_unlock(mm); mmput(mm); + if (ret) + return ret; + /* Return the migrated data for verification. */ ret = dmirror_bounce_init(&bounce, start, size); if (ret) diff --git a/mm/rmap.c b/mm/rmap.c index 17fbfa61f7efb..7ccf850565d33 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2495,70 +2495,89 @@ static bool folio_make_device_exclusive(struct folio *folio, .arg = &args, }; - /* - * Restrict to anonymous folios for now to avoid potential writeback - * issues. - */ - if (!folio_test_anon(folio) || folio_test_hugetlb(folio)) - return false; - rmap_walk(folio, &rwc); return args.valid && !folio_mapcount(folio); } /** - * make_device_exclusive_range() - Mark a range for exclusive use by a device + * make_device_exclusive() - Mark a page for exclusive use by a device * @mm: mm_struct of associated target process - * @start: start of the region to mark for exclusive device access - * @end: end address of region - * @pages: returns the pages which were successfully marked for exclusive access + * @addr: the virtual address to mark for exclusive device access * @owner: passed to MMU_NOTIFY_EXCLUSIVE range notifier to allow filtering + * @foliop: folio pointer will be stored here on success. + * + * This function looks up the page mapped at the given address, grabs a + * folio reference, locks the folio and replaces the PTE with special + * device-exclusive PFN swap entry, preventing access through the process + * page tables. The function will return with the folio locked and referenced. * - * Returns: number of pages found in the range by GUP. A page is marked for - * exclusive access only if the page pointer is non-NULL. + * On fault, the device-exclusive entries are replaced with the original PTE + * under folio lock, after calling MMU notifiers. * - * This function finds ptes mapping page(s) to the given address range, locks - * them and replaces mappings with special swap entries preventing userspace CPU - * access. On fault these entries are replaced with the original mapping after - * calling MMU notifiers. + * Only anonymous non-hugetlb folios are supported and the VMA must have + * write permissions such that we can fault in the anonymous page writable + * in order to mark it exclusive. The caller must hold the mmap_lock in read + * mode. * * A driver using this to program access from a device must use a mmu notifier * critical section to hold a device specific lock during programming. Once - * programming is complete it should drop the page lock and reference after + * programming is complete it should drop the folio lock and reference after * which point CPU access to the page will revoke the exclusive access. + * + * Notes: + * #. This function always operates on individual PTEs mapping individual + * pages. PMD-sized THPs are first remapped to be mapped by PTEs before + * the conversion happens on a single PTE corresponding to @addr. + * #. While concurrent access through the process page tables is prevented, + * concurrent access through other page references (e.g., earlier GUP + * invocation) is not handled and not supported. + * #. device-exclusive entries are considered "clean" and "old" by core-mm. + * Device drivers must update the folio state when informed by MMU + * notifiers. + * + * Returns: pointer to mapped page on success, otherwise a negative error. */ -int make_device_exclusive_range(struct mm_struct *mm, unsigned long start, - unsigned long end, struct page **pages, - void *owner) +struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, + void *owner, struct folio **foliop) { - long npages = (end - start) >> PAGE_SHIFT; - long i; + struct folio *folio; + struct page *page; + long npages; + + mmap_assert_locked(mm); - npages = get_user_pages_remote(mm, start, npages, + /* + * Fault in the page writable and try to lock it; note that if the + * address would already be marked for exclusive use by a device, + * the GUP call would undo that first by triggering a fault. + */ + npages = get_user_pages_remote(mm, addr, 1, FOLL_GET | FOLL_WRITE | FOLL_SPLIT_PMD, - pages, NULL); - if (npages < 0) - return npages; - - for (i = 0; i < npages; i++, start += PAGE_SIZE) { - struct folio *folio = page_folio(pages[i]); - if (PageTail(pages[i]) || !folio_trylock(folio)) { - folio_put(folio); - pages[i] = NULL; - continue; - } + &page, NULL); + if (npages != 1) + return ERR_PTR(npages); + folio = page_folio(page); - if (!folio_make_device_exclusive(folio, mm, start, owner)) { - folio_unlock(folio); - folio_put(folio); - pages[i] = NULL; - } + if (!folio_test_anon(folio) || folio_test_hugetlb(folio)) { + folio_put(folio); + return ERR_PTR(-EOPNOTSUPP); + } + + if (!folio_trylock(folio)) { + folio_put(folio); + return ERR_PTR(-EBUSY); } - return npages; + if (!folio_make_device_exclusive(folio, mm, addr, owner)) { + folio_unlock(folio); + folio_put(folio); + return ERR_PTR(-EBUSY); + } + *foliop = folio; + return page; } -EXPORT_SYMBOL_GPL(make_device_exclusive_range); +EXPORT_SYMBOL_GPL(make_device_exclusive); #endif void __put_anon_vma(struct anon_vma *anon_vma) From patchwork Mon Feb 10 19:37:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13968584 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 651F5C021A5 for ; Mon, 10 Feb 2025 19:38:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DED6A280006; Mon, 10 Feb 2025 14:38:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D9DF9280001; Mon, 10 Feb 2025 14:38:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B7B14280006; Mon, 10 Feb 2025 14:38:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 961DD280001 for ; Mon, 10 Feb 2025 14:38:25 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5312D46854 for ; Mon, 10 Feb 2025 19:38:25 +0000 (UTC) X-FDA: 83105046570.11.4B2F1AC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf02.hostedemail.com (Postfix) with ESMTP id 2242380005 for ; Mon, 10 Feb 2025 19:38:22 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RqNmGrTJ; spf=pass (imf02.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739216303; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PTNkd4QUaFJn8BXhdNDgA0x2CIZMe/1M0YGUP/Aku4s=; b=WILgnzDr9nO6urQJPv62iBpSucGEO6VHRf1iPuw1QHHnVnHAiSXTZAV51NvMFAaC7njNKW Lu2UwtoK/UUtKKvUc9qgYqO5WU1qzmPoKo+qxBK4hdUpEP9akA3XVhdNe90SNXMyueW8YF 3LGz/BT65GJzO4uLMu8T5fJLV0AEIQ4= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RqNmGrTJ; spf=pass (imf02.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739216303; a=rsa-sha256; cv=none; b=TggKmrRxa6zxgcRcqBvgOei6eJfyPRAt6y+pMLffi3nZv8bjSg9F8MGdmX+5DW2R5wuv3U QWjST71Td2WBT2PdBCrhF6tvI/haZAWU0WGqoDgPaiLuztxq3MV/IXkh42txx+gAkMAdAl lgkes0pdmbs8HcerM0og2EdRqagpFYs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739216302; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PTNkd4QUaFJn8BXhdNDgA0x2CIZMe/1M0YGUP/Aku4s=; b=RqNmGrTJlFDUZVrqYEPbHQP3Q/eS3QQRcZuIxZ8jxbF/XrySbipuM9h9XJUJd4/peHHpy6 YdRXNdVUeDREHmqVFhR5TWckVDhbG2X7wHJPkDAXiUMnT75qwU6sY90UgQPJ9/rFTE79ie YvQOeBtSJ6aw7N3MM7FOxsiIK8JNFUc= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-118-tHempvrmMwiSQQWMz93o_Q-1; Mon, 10 Feb 2025 14:38:21 -0500 X-MC-Unique: tHempvrmMwiSQQWMz93o_Q-1 X-Mimecast-MFC-AGG-ID: tHempvrmMwiSQQWMz93o_Q Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-4393535043bso11596795e9.1 for ; Mon, 10 Feb 2025 11:38:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739216300; x=1739821100; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PTNkd4QUaFJn8BXhdNDgA0x2CIZMe/1M0YGUP/Aku4s=; b=XHzONujU2zC9qzySOq1waatetrlCj5Px8tfHZOuJ0Oex616KVS80M48KHJanxZvAGP n1E8zcfpQEGn8fwnlTfCMUJhVoUkA0eWtAbdE/0Ixg4vM9lbzrSSuirwy5gq7IEWS0YW vPlbcXB6A8oxbKj2hHL4KJD153kls4Voi8ElVRNsjKE7UCf3mxBrJF3McYRav8qViui5 VGwbFbRryvGom1tlIO63F2xxgRhfvz4JhqT+Bbj1ZMFn0v4jkdbGP/CHtKNCvKo3Hl1m Vsvl3EseqN+VZ8kQzILqbKkmWrA2WuCr/Tbe2h7+cAM2k0PrY8VOaSs3kHdOmnE8S+Ih TU+A== X-Forwarded-Encrypted: i=1; AJvYcCXSRs2ivIwJeaGcG/SHYU0gdQkRP4VYdtSZ3jRWofWlVj/kc6qvN2hQovbQBvWdBn0a51HhUDIfXw==@kvack.org X-Gm-Message-State: AOJu0YxH83UszTnnL1wGzofHb6B6+rFKG+gL/3xRUCLfZhjsQiDasjdp 8IjNFAZr2xlHNmSbecU2eNUi49igmdEPXnHr/n9iECOeCXqtkW3qExWqG7TQM9Kjww72TDpsY/W MDxAiUCHtcTsQoNlrDKPuki5J9+RUZbGth7TCMd/rdu5YlQPg X-Gm-Gg: ASbGnctJQuTutwimUxwVfCenQClArN0HnbeZzCBOoBzgZzmq3+dZUlBtyc0UBDU5xzb hFs4rBmoq6kalYCI1/ON51fQ7FtxBLAKUhDeIw+jUkndOaSQsAMcevA0TumZ2eb3SEXuPHimQuV /aMOJRK3elrIH9Xo+HNS0/fMXqS5ONysAS2tDzgSQsPHO6WsmF9/4do5wQOgnj7JKkKv3EZYgaS T84F7fM0iZY3AatfEbdAOX5sny3Ej1un9BTJanKBhwxlEdQ4QmyIqKyFkxbqSgl6oehtU1zLjbr MVW/c3zYZCA7zygSi98XdmGTz/oLBzM/8tS00s35nxphKof77iVwsSBqUhK09jZcXA== X-Received: by 2002:a05:600c:490f:b0:438:9280:61d5 with SMTP id 5b1f17b1804b1-4394ceb21e5mr4501635e9.5.1739216299917; Mon, 10 Feb 2025 11:38:19 -0800 (PST) X-Google-Smtp-Source: AGHT+IFc0kj/uTsAeIklzzlg0/rE3LVYXx2C/9ALcZ7/TUEZYAL5lOHh0P4FpVZlZXUmD9AHXTDheA== X-Received: by 2002:a05:600c:490f:b0:438:9280:61d5 with SMTP id 5b1f17b1804b1-4394ceb21e5mr4501495e9.5.1739216299521; Mon, 10 Feb 2025 11:38:19 -0800 (PST) Received: from localhost (p200300cbc734b80012c465cd348aaee6.dip0.t-ipconnect.de. [2003:cb:c734:b800:12c4:65cd:348a:aee6]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38dc2f6aeafsm11910943f8f.20.2025.02.10.11.38.17 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 10 Feb 2025 11:38:19 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, damon@lists.linux.dev, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Masami Hiramatsu , Oleg Nesterov , Peter Zijlstra , SeongJae Park , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v2 04/17] mm/rmap: implement make_device_exclusive() using folio_walk instead of rmap walk Date: Mon, 10 Feb 2025 20:37:46 +0100 Message-ID: <20250210193801.781278-5-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250210193801.781278-1-david@redhat.com> References: <20250210193801.781278-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: Zlfgl219rTp4qosfbLcnQEwE8zCPCkM9Uvu9Q3AMPfk_1739216300 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 2242380005 X-Stat-Signature: kfi39b1gkf88jhsqkh4ysnbpzgbs1t58 X-HE-Tag: 1739216302-187591 X-HE-Meta: U2FsdGVkX1/px53HLsF/thoxcXaOJk8shHEWxLm4LU7Y1uKkI9hl3cqXbrkeFVEgOuaPYukQYFYsupBEwWoyqnEtfSjPRJZqGnaGM/3yaLLwj+VbCFSJIBY0Qg6FXCrf8CDDHdaF+Zik0RgcW9+du+HJaDwUKm5kVjjtJ4Sy0rHrLvZZLU6esaK7SQMTOyaIFTU2V+cau3nZUBBAfjw2FMqmJoEbAyLd2tSaA5vzZcg0wJ9zzcMMMjObVGcLSWMTx2P/vRvANhtnUrjHLcwy/7tICC5ROmzQmezFXqrL6NcEwY6aB4Qnx37Eu9vVRcehUb2ybWgShLf/uhsHxXEXUDgopdmMl2ri8VcHeoLmXgyTitRxxArcQSYkwF0DjmbTVnA4C7FGpj8iuH2IrsHaWCHQH07i2E+MvROT2zbjrqLSaWl5JSesZ9Q9QA1t6zR5RKWD4uSk8C6EOVhwa/t7Ffmuzl91XiSYE9RmDCa5BLyFX1Z1oxEj2bpDN6WzPdTH6s8UK8qPKinyaOt5b4A659d+dADRkjwjZTY+nyTJXGjf64tUQ0HCLa2a3c/pMAJ+O5JyU/oEAeC+pdLVNWANbLkzpPIqFUZ/LLA7J992Ss7R04ugsfNkQCuwHzwjXXuVbV1MXkZC7v4S9vQkXzRTeXzzwR9b6yFKlUIw6CuRLUKnoARYR7Vi48ey2xSEHEr+MzoHPUHllBCZDG2WqS3rxdcjFen4o4LdWmbYFa926qxCLG2nM9I7L4TuxaBlro1Botyel1NRuGdknqZtQtURwzVLcTzA0G+JcYfmQAnwqFplv4nGJUbLZNJHOE19prVparIxHErAWrg4QHiJbBqQk7YOca9A+tubTTOh+vlusMkhVf1jcP0KTnKB4mglKdOx+vmMSNrFOL/nv0XBqIygr+ad97xykiAjGqSGEam3KoAv8Y7WA3KIaERjytgUUVi8at+TNzXdu7bqnFiuzBD ty2THpuu r6S6+28VbQOx7NmD3xd0DXmz8sM0IyrTcvc1VYj75/tEL5gmJAQfQsuqg9mMY+nsPgJqU9AVBDeRH9sekk8WGWoY5uXr60sw8xeew3E8tKTh7f8/eRkYQVqJnxIwTH/QTt7Q6oqaaB+mNp5bBjXVBOlAaKpZqIxlfu8yZBsJIijbSCpVbYF3dAjfNGgU7eKHJgVcJ0adNvbJjj6dp5GzCTBVCeN9elIp/yCp/W/0DlCb1fwSXuG2jzet/45xdbExsPgKW472S93Bl/0HsjJrxNkDlLEsglvjeu3Ik/ve5QbjMQ1SvgPxQN5zBqDkixDKvBV7Mk2DrM/DOHnNkNIbkYivTyhHLvpYS6Zd8emfgJkxfzxs4bsmwE3WZoVbw+gaoJw1DODIDlhTNupCUnpT6je+H6SfFB8ZXD2udbhgNuGQUvYJaVOkpARBgTHP2oZ2PFG3cEZIgmGzcy89SFD2Aem8Xyw1Klj9BX1L5Z8cTrY+pM9Ndc5mge6mZCCVe3gqox38voZhHpvL3AShPBSQP8BLAavtiOwWmSxXXnVfGS5cFyPA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We require a writable PTE and only support anonymous folio: we can only have exactly one PTE pointing at that page, which we can just lookup using a folio walk, avoiding the rmap walk and the anon VMA lock. So let's stop doing an rmap walk and perform a folio walk instead, so we can easily just modify a single PTE and avoid relying on rmap/mapcounts. We now effectively work on a single PTE instead of multiple PTEs of a large folio, allowing for conversion of individual PTEs from non-exclusive to device-exclusive -- note that the opposite direction always works on single PTEs: restore_exclusive_pte(). With this change, device-exclusive handling is fully compatible with THPs / large folios. We still require PMD-sized THPs to get PTE-mapped, and supporting PMD-mapped THP (without the PTE-remapping) is a different endeavour that might not be worth it at this point: it might even have negative side-effects [1]. This gets rid of the "folio_mapcount()" usage and let's us fix ordinary rmap walks (migration/swapout) next. Spell out that messing with the mapcount is wrong and must be fixed. [1] https://lkml.kernel.org/r/Z5tI-cOSyzdLjoe_@phenom.ffwll.local Signed-off-by: David Hildenbrand --- mm/rmap.c | 200 ++++++++++++++++++------------------------------------ 1 file changed, 67 insertions(+), 133 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 7ccf850565d33..0cd2a2d3de00d 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2375,131 +2375,6 @@ void try_to_migrate(struct folio *folio, enum ttu_flags flags) } #ifdef CONFIG_DEVICE_PRIVATE -struct make_exclusive_args { - struct mm_struct *mm; - unsigned long address; - void *owner; - bool valid; -}; - -static bool page_make_device_exclusive_one(struct folio *folio, - struct vm_area_struct *vma, unsigned long address, void *priv) -{ - struct mm_struct *mm = vma->vm_mm; - DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0); - struct make_exclusive_args *args = priv; - pte_t pteval; - struct page *subpage; - bool ret = true; - struct mmu_notifier_range range; - swp_entry_t entry; - pte_t swp_pte; - pte_t ptent; - - mmu_notifier_range_init_owner(&range, MMU_NOTIFY_EXCLUSIVE, 0, - vma->vm_mm, address, min(vma->vm_end, - address + folio_size(folio)), - args->owner); - mmu_notifier_invalidate_range_start(&range); - - while (page_vma_mapped_walk(&pvmw)) { - /* Unexpected PMD-mapped THP? */ - VM_BUG_ON_FOLIO(!pvmw.pte, folio); - - ptent = ptep_get(pvmw.pte); - if (!pte_present(ptent)) { - ret = false; - page_vma_mapped_walk_done(&pvmw); - break; - } - - subpage = folio_page(folio, - pte_pfn(ptent) - folio_pfn(folio)); - address = pvmw.address; - - /* Nuke the page table entry. */ - flush_cache_page(vma, address, pte_pfn(ptent)); - pteval = ptep_clear_flush(vma, address, pvmw.pte); - - /* Set the dirty flag on the folio now the pte is gone. */ - if (pte_dirty(pteval)) - folio_mark_dirty(folio); - - /* - * Check that our target page is still mapped at the expected - * address. - */ - if (args->mm == mm && args->address == address && - pte_write(pteval)) - args->valid = true; - - /* - * Store the pfn of the page in a special migration - * pte. do_swap_page() will wait until the migration - * pte is removed and then restart fault handling. - */ - if (pte_write(pteval)) - entry = make_writable_device_exclusive_entry( - page_to_pfn(subpage)); - else - entry = make_readable_device_exclusive_entry( - page_to_pfn(subpage)); - swp_pte = swp_entry_to_pte(entry); - if (pte_soft_dirty(pteval)) - swp_pte = pte_swp_mksoft_dirty(swp_pte); - if (pte_uffd_wp(pteval)) - swp_pte = pte_swp_mkuffd_wp(swp_pte); - - set_pte_at(mm, address, pvmw.pte, swp_pte); - - /* - * There is a reference on the page for the swap entry which has - * been removed, so shouldn't take another. - */ - folio_remove_rmap_pte(folio, subpage, vma); - } - - mmu_notifier_invalidate_range_end(&range); - - return ret; -} - -/** - * folio_make_device_exclusive - Mark the folio exclusively owned by a device. - * @folio: The folio to replace page table entries for. - * @mm: The mm_struct where the folio is expected to be mapped. - * @address: Address where the folio is expected to be mapped. - * @owner: passed to MMU_NOTIFY_EXCLUSIVE range notifier callbacks - * - * Tries to remove all the page table entries which are mapping this - * folio and replace them with special device exclusive swap entries to - * grant a device exclusive access to the folio. - * - * Context: Caller must hold the folio lock. - * Return: false if the page is still mapped, or if it could not be unmapped - * from the expected address. Otherwise returns true (success). - */ -static bool folio_make_device_exclusive(struct folio *folio, - struct mm_struct *mm, unsigned long address, void *owner) -{ - struct make_exclusive_args args = { - .mm = mm, - .address = address, - .owner = owner, - .valid = false, - }; - struct rmap_walk_control rwc = { - .rmap_one = page_make_device_exclusive_one, - .done = folio_not_mapped, - .anon_lock = folio_lock_anon_vma_read, - .arg = &args, - }; - - rmap_walk(folio, &rwc); - - return args.valid && !folio_mapcount(folio); -} - /** * make_device_exclusive() - Mark a page for exclusive use by a device * @mm: mm_struct of associated target process @@ -2541,22 +2416,31 @@ static bool folio_make_device_exclusive(struct folio *folio, struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, void *owner, struct folio **foliop) { - struct folio *folio; + struct mmu_notifier_range range; + struct folio *folio, *fw_folio; + struct vm_area_struct *vma; + struct folio_walk fw; struct page *page; - long npages; + swp_entry_t entry; + pte_t swp_pte; mmap_assert_locked(mm); + addr = PAGE_ALIGN_DOWN(addr); /* * Fault in the page writable and try to lock it; note that if the * address would already be marked for exclusive use by a device, * the GUP call would undo that first by triggering a fault. + * + * If any other device would already map this page exclusively, the + * fault will trigger a conversion to an ordinary + * (non-device-exclusive) PTE and issue a MMU_NOTIFY_EXCLUSIVE. */ - npages = get_user_pages_remote(mm, addr, 1, - FOLL_GET | FOLL_WRITE | FOLL_SPLIT_PMD, - &page, NULL); - if (npages != 1) - return ERR_PTR(npages); + page = get_user_page_vma_remote(mm, addr, + FOLL_GET | FOLL_WRITE | FOLL_SPLIT_PMD, + &vma); + if (IS_ERR(page)) + return page; folio = page_folio(page); if (!folio_test_anon(folio) || folio_test_hugetlb(folio)) { @@ -2569,11 +2453,61 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, return ERR_PTR(-EBUSY); } - if (!folio_make_device_exclusive(folio, mm, addr, owner)) { + /* + * Inform secondary MMUs that we are going to convert this PTE to + * device-exclusive, such that they unmap it now. Note that the + * caller must filter this event out to prevent livelocks. + */ + mmu_notifier_range_init_owner(&range, MMU_NOTIFY_EXCLUSIVE, 0, + mm, addr, addr + PAGE_SIZE, owner); + mmu_notifier_invalidate_range_start(&range); + + /* + * Let's do a second walk and make sure we still find the same page + * mapped writable. Note that any page of an anonymous folio can + * only be mapped writable using exactly one PTE ("exclusive"), so + * there cannot be other mappings. + */ + fw_folio = folio_walk_start(&fw, vma, addr, 0); + if (fw_folio != folio || fw.page != page || + fw.level != FW_LEVEL_PTE || !pte_write(fw.pte)) { + if (fw_folio) + folio_walk_end(&fw, vma); + mmu_notifier_invalidate_range_end(&range); folio_unlock(folio); folio_put(folio); return ERR_PTR(-EBUSY); } + + /* Nuke the page table entry so we get the uptodate dirty bit. */ + flush_cache_page(vma, addr, page_to_pfn(page)); + fw.pte = ptep_clear_flush(vma, addr, fw.ptep); + + /* Set the dirty flag on the folio now the PTE is gone. */ + if (pte_dirty(fw.pte)) + folio_mark_dirty(folio); + + /* + * Store the pfn of the page in a special device-exclusive PFN swap PTE. + * do_swap_page() will trigger the conversion back while holding the + * folio lock. + */ + entry = make_writable_device_exclusive_entry(page_to_pfn(page)); + swp_pte = swp_entry_to_pte(entry); + if (pte_soft_dirty(fw.pte)) + swp_pte = pte_swp_mksoft_dirty(swp_pte); + /* The pte is writable, uffd-wp does not apply. */ + set_pte_at(mm, addr, fw.ptep, swp_pte); + + /* + * TODO: The device-exclusive PFN swap PTE holds a folio reference but + * does not count as a mapping (mapcount), which is wrong and must be + * fixed, otherwise RMAP walks don't behave as expected. + */ + folio_remove_rmap_pte(folio, page, vma); + + folio_walk_end(&fw, vma); + mmu_notifier_invalidate_range_end(&range); *foliop = folio; return page; } From patchwork Mon Feb 10 19:37:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13968585 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A22CAC021A2 for ; Mon, 10 Feb 2025 19:38:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F085280007; Mon, 10 Feb 2025 14:38:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 253E0280001; Mon, 10 Feb 2025 14:38:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 07E51280007; Mon, 10 Feb 2025 14:38:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D7C21280001 for ; Mon, 10 Feb 2025 14:38:28 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 92D76C050C for ; Mon, 10 Feb 2025 19:38:28 +0000 (UTC) X-FDA: 83105046696.01.D206266 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 5DC5740010 for ; Mon, 10 Feb 2025 19:38:26 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AV8j6qoZ; spf=pass (imf07.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739216306; a=rsa-sha256; cv=none; b=ab45taC175xY7bbTlKge8nrqmh6NfUJzqjHS6JxOIOCkU8ADA60EuyNzbHLyiW5ImR7vwL 1tf16+ADyNv+FX8dt/bEH3E9SsNwxX+wrlt0hAZ6AIJWxqdIPQ1RnWU7FhB8cfmbpJGXwf n1254mtZWGC030yLtjqLJ5ex9J4VUa4= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AV8j6qoZ; spf=pass (imf07.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739216306; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mARFTZKeX08fN9+zJg7ejGdzGLsE2rU+aFhmUgxRNAg=; b=HB8iR2s09R7gwcDonBjrh7epXnBmg3UVCZet0JI4qq1U3BHHH/ei4hh/+mpbxLB/eidQOt KVKk/9k9DEPlZaBu1zrJOKznyKD+63wWCojFkdCFkcdAhr7+eodiMHxQsX53cTe1zofreS LpAZuXGk0cMnN6LTtxsR6EzSJzQf+9s= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739216305; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mARFTZKeX08fN9+zJg7ejGdzGLsE2rU+aFhmUgxRNAg=; b=AV8j6qoZnbT0BbYa00T4kEJo2tTqvKZvxlXh/86/z+Eh2T73iv0GLtO6G4lpRE/UMHCJuZ M7FnqEugkbXwikFGKl/dquIvghZ+BifM3o3DwQfEb3aEI/SqbizLWa74HvV6W2mKyIgfN4 Lu5D6Ab8jGNu1so8LLnZt1duxHuAltw= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-677-L1ajbMGDPdCtkSQmgwTNeQ-1; Mon, 10 Feb 2025 14:38:24 -0500 X-MC-Unique: L1ajbMGDPdCtkSQmgwTNeQ-1 X-Mimecast-MFC-AGG-ID: L1ajbMGDPdCtkSQmgwTNeQ Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-4393e873962so8859455e9.3 for ; Mon, 10 Feb 2025 11:38:24 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739216303; x=1739821103; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mARFTZKeX08fN9+zJg7ejGdzGLsE2rU+aFhmUgxRNAg=; b=T5KqVQIZgsJ043YZodtDddGfuHdVFemjZa+s9RwKLJ2m1lRssjnlzVENA+o40MulL9 yas0yiER/1POdeYOLifmatFGclsfql8qtOUfeAzg40nYcp5iiHdaEWb8i9lmnvFEZhuk 2AFTEbb7vPwp+cdYAhcWTdo8ltB+lfs0ribOSmx0Q13tm1fR+NCS3ECjkNOYfDsb4pi/ ZmAxaVFhM5KQZ0taEcmQCc9JJVy5yljjyEy9nkMDAEeDXse5xATF0y3llIimPf48RUcS BeOuJeY9iJfYea499n09O7LwKSbpNK47G5U0qJlmwuQlFwuXxpZ+oJxroiFXy+8fBUW5 krhw== X-Forwarded-Encrypted: i=1; AJvYcCX8y/0ATtdz2bXj8PXPA9KlMlunDvCyroHA9z4HZEKhS8mdJj9jtIdszwfXg5pPz/ntDg+aaT6BvA==@kvack.org X-Gm-Message-State: AOJu0Yz0pVAuZ8Bv3IlfzcTVo1IeCsu5dC2+BdRpZuiCDn4nt1L/A0Fy txRr9Bos+/7+ZQywILqNCwJqpJ/YB0BB6R/Kjy42xxltDVohyKmNxgL1vv9PLTYKvDh/ysyJTjL D0KFzF3yx9pe/FQqefBZ+qBAbxPnEfPL9Q9hjGRbvKeO26EHA X-Gm-Gg: ASbGnctja/e+2nSyYkoAoqE2CjXDf9EXZQAAaxxELIJP6fNKoUuqRNIRS6KYchaSo9Z KEXlc6GP+/PqpgDOjx+UJpm7ZVl7qI9qd7dReJaBxS0FW3ewihyu2uTCLN+oLh44wwnK72wX47J E7Mjk+oCz2wmPTsQX1yAv1n79HpeytgBVZPiA0+cR5kfQhVRE0jtoPD0c9ssZ7BNTXFnkgCngUg cJjT2QgsD5no5fnR0ri+w4QfpA/9fi8gWVXgzSKw7ollp3jp6NZe023XnjWpRPusB0MXShAClCc Yt5rUpJYAhJq6STOL6PCZVzPPhyeuGgqEtIldoExu+pRmLrRFpKkWNYunR0dfJ4hRg== X-Received: by 2002:a05:6000:18a5:b0:38d:e33d:d0db with SMTP id ffacd0b85a97d-38de33dd2b2mr2312799f8f.14.1739216303484; Mon, 10 Feb 2025 11:38:23 -0800 (PST) X-Google-Smtp-Source: AGHT+IH1Llhtf765OHh3Rdp8g77cDKhZyIe7rPLF3LamWfRFHXBXmPgwl/7uacqNHueSdZU9tH9+kA== X-Received: by 2002:a05:6000:18a5:b0:38d:e33d:d0db with SMTP id ffacd0b85a97d-38de33dd2b2mr2312758f8f.14.1739216303053; Mon, 10 Feb 2025 11:38:23 -0800 (PST) Received: from localhost (p200300cbc734b80012c465cd348aaee6.dip0.t-ipconnect.de. [2003:cb:c734:b800:12c4:65cd:348a:aee6]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38dd3fc7ee5sm7734941f8f.39.2025.02.10.11.38.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 10 Feb 2025 11:38:21 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, damon@lists.linux.dev, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Masami Hiramatsu , Oleg Nesterov , Peter Zijlstra , SeongJae Park , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v2 05/17] mm/memory: detect writability in restore_exclusive_pte() through can_change_pte_writable() Date: Mon, 10 Feb 2025 20:37:47 +0100 Message-ID: <20250210193801.781278-6-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250210193801.781278-1-david@redhat.com> References: <20250210193801.781278-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: m-Xr3VdcQKSrbQxY5TSpnSIh_ukVb1wvF0_jahzmWEo_1739216303 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Stat-Signature: r4tm4tkagr76re8cyusftbmw1mn8hwkx X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 5DC5740010 X-Rspam-User: X-HE-Tag: 1739216306-543831 X-HE-Meta: U2FsdGVkX18gTtSfHgg6zqOcv/nH5JExJuw+OQhXQLJ0IfKO3lOQiR7DOhHEwkWe7FSY0671nOuFZch30i3etYtHqM9DlRrKw/ntAGZUfI6DP+23n+qP3AjRS1vDpD0t2feXO+srGnWv+Vvwk/5ZOtwoMMvkzo35N6n/6qCbDXddaunehqcl9qqx9XgK8ONipx9T1bjVFKQF85k9csfJUKtd+VcDnSvdsDcOgOx2ACk59/SZ5zDVHKRkQMcGfssKiPCdLUbZZB0C27DR1Ab+A6pfxPqeCNARuZL6bu7Undo7YNhdz4YAdEUWDWBeG0BfLd2PjFqIg76/HiSU0V29pHJ89TqD9tXF+wIIh90x/d4MCOd/Zqbhs9y+zAZ6uWHVIhO9sFx1wEr9Uq2Ah+/6un/1Ssl878792eDn7PHSJDc+KyXtn7/S8OYgDMb7zCtUM9YvQK67mj4ehLi1hu+FgdRUBE5bDw0MeVSRXGmRV0D2LvpkKlvrZwJ7yovNlvHYwahuwSs1COwGsYjLCf1+SgDlJYWTJL7NQYiiW81dNmeQpJjzXDpq/VAt0F+dHGeF9VUfwTepn2L9gHK+kIsid9uFu5ZOEpkoZFJAGuZaoQr2vppC37T0C52Ug+MMKysaA6/8ynUlyTpYUkwyHJ1w/Qf0Cr5OzoznMYBY5IPw4n2BHN3OaM04BQQFE2J8k9Zt8vmiu7QGYV6hll/lVSlzQXl+xvVr8JS3e60cpTYqEuysbYY/VtlxD/SvkUnp3MuJ/HPawAaFMYwiiAj438BoJYLIfS8IzaUXajploPpy+6MtA4wPV0qw66j0IZrJCwDcrTQ8xf1XXxy5oV3joy19gErre86DkjF1TM747Oo2VGNXBe2EUe4PPWn259ZIQ8CIDmujN4l4hSZS0MK9GfeTe2yldZtca0xWSlgNbvXF1T0TKfwVgB9dJ/WcGulIVzFuGJXOd7pgUWnwZmrtsH5 pP9vhhFR io2FG0Et5MYZ+2+Dzag5ssThebguEfuCh4kqnSpJQHndvzIBOMTsCcTMZAtY0qVIwoTto/aLdhaALPBihP9RFbrd4vT//y9rl8Xq4qSIWHzBCaMa0a3Q3eFoYI9uPXtAvDNVVxKXfbJ+7Sx6F1JhctRY88Fy9uahsH8eQiaKGsUk6qQyg4pnez6qazFcMWVHqtlvEo1EEDKjkLuGSF990qzO0399u3SGnhDs75p/7uC2zFE96ITiGL6/7H9VZk7hlUESKLiSlA10m3t66LWg5n9fIRG+ckYTrGeSqVQMKitQDFOBKiwFjp0YbHLoUlzTaXYTbXFq3bbcPepzqY25NMHAXiW6wBEv61Z/IQK3xLhmHaHzLXOUqEYR5dR+5Y50AFkcvwirQ+zy/ubU55S8x5/aYt/jSCNbfIxmT0ysxbsW+zqvdIWjSTgaR6weik+vkZxnvJRBU8jlohvMyWrrMQEC2CZgC8KCOICPlr0gWeKYJFpcbfKotF9ep9qKH7lMJ2PXR1Yd+FyrbESfW/KZ+EcgYHw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000008, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's do it just like mprotect write-upgrade or during NUMA-hinting faults on PROT_NONE PTEs: detect if the PTE can be writable by using can_change_pte_writable(). Set the PTE only dirty if the folio is dirty: we might not necessarily have a write access, and setting the PTE writable doesn't require setting the PTE dirty. From a CPU perspective, these entries are clean. So only set the PTE dirty if the folios is dirty. With this change in place, there is no need to have separate readable and writable device-exclusive entry types, and we'll merge them next separately. Note that, during fork(), we first convert the device-exclusive entries back to ordinary PTEs, and we only ever allow conversion of writable PTEs to device-exclusive -- only mprotect can currently change them to readable-device-exclusive. Consequently, we always expect PageAnonExclusive(page)==true and can_change_pte_writable()==true, unless we are dealing with soft-dirty tracking or uffd-wp. But reusing can_change_pte_writable() for now is cleaner. Signed-off-by: David Hildenbrand --- mm/memory.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 539c0f7c6d545..ba33ba3b7ea17 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -723,18 +723,21 @@ static void restore_exclusive_pte(struct vm_area_struct *vma, struct folio *folio = page_folio(page); pte_t orig_pte; pte_t pte; - swp_entry_t entry; orig_pte = ptep_get(ptep); pte = pte_mkold(mk_pte(page, READ_ONCE(vma->vm_page_prot))); if (pte_swp_soft_dirty(orig_pte)) pte = pte_mksoft_dirty(pte); - entry = pte_to_swp_entry(orig_pte); if (pte_swp_uffd_wp(orig_pte)) pte = pte_mkuffd_wp(pte); - else if (is_writable_device_exclusive_entry(entry)) - pte = maybe_mkwrite(pte_mkdirty(pte), vma); + + if ((vma->vm_flags & VM_WRITE) && + can_change_pte_writable(vma, address, pte)) { + if (folio_test_dirty(folio)) + pte = pte_mkdirty(pte); + pte = pte_mkwrite(pte, vma); + } VM_BUG_ON_FOLIO(pte_write(pte) && (!folio_test_anon(folio) && PageAnonExclusive(page)), folio); From patchwork Mon Feb 10 19:37:48 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13968586 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6834DC021A4 for ; Mon, 10 Feb 2025 19:38:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 00C18280008; Mon, 10 Feb 2025 14:38:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ED677280001; Mon, 10 Feb 2025 14:38:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D2A07280008; Mon, 10 Feb 2025 14:38:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id AD061280001 for ; Mon, 10 Feb 2025 14:38:32 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 6C966C0508 for ; Mon, 10 Feb 2025 19:38:32 +0000 (UTC) X-FDA: 83105046864.24.31E04C3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf03.hostedemail.com (Postfix) with ESMTP id 3C98720008 for ; Mon, 10 Feb 2025 19:38:30 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=L3pjQWRh; spf=pass (imf03.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739216310; a=rsa-sha256; cv=none; b=18FfhvfZAjeg+3DroJtV9ClbEFPw9p1cOlqtjc/aLEoDkJBAkFZOWmliNdN3Q5gzmr4lac oeYOk26puCXb7oQXkmNdOsUdKT4fKl2Avs36W4xh4qUiWKCLbMwtgsm0zH5A7Wc8BbqKJ6 1dQeOIrmrjs9tL11c6R2eP9QGInYvDI= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=L3pjQWRh; spf=pass (imf03.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739216310; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fLiFHA5Aiy4k7pcGr1QPi7NyoGcLfJYJaAk0JBe4xM8=; b=1oSLKE/shAHExVo7nDYgOXcczqbjgcqsi4Sqxi7XqXXJNBsoS6s5cfTKa1XpiBCt3HLoG5 P6WNihwy+GSxSKROYkRkI/w7r3E27x4TA81+5Q8RXQ+BiT5jLlaD8Zf/biGMwgWgjyh+Yk uIRFxD+K+dWcX89HK40u5w5YN9GTL68= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739216309; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fLiFHA5Aiy4k7pcGr1QPi7NyoGcLfJYJaAk0JBe4xM8=; b=L3pjQWRhh3BsZDDLXrPjOxLWOO/HCflNIsa1vUTIVh1+ke2RU0Kiu5uE7bCacpq454GToh rVwqEIA/WSLT3GDqAZOn6xP+ojn57Ad1BOPmGdiRF+rjr0DJ1wFl4JDTkqTd1gKoqHeJuh JpCfuTB4epolwqzGum9In0oBSJRCfgg= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-447-9IEjIixsMiKAfjhWZoZGqg-1; Mon, 10 Feb 2025 14:38:28 -0500 X-MC-Unique: 9IEjIixsMiKAfjhWZoZGqg-1 X-Mimecast-MFC-AGG-ID: 9IEjIixsMiKAfjhWZoZGqg Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-4388eee7073so27031045e9.0 for ; Mon, 10 Feb 2025 11:38:28 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739216307; x=1739821107; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fLiFHA5Aiy4k7pcGr1QPi7NyoGcLfJYJaAk0JBe4xM8=; b=C4+mV7xzpkJN2/2nrtNwte71peYENDIZUwq5KwLHreGb4r4q5tV84eU5FEC4fofgwd 3zSCbyPSele0Xj09e7a0Rp6gWPk/1OcA2cJL9a3y/vYHS6bm/Nab+LcZ+sFAF0jpNPwi C9cIUJ6SGAoTSsHToVwlRsbq2pBicl1OcKRIIysYfTGx2dWZSKaAKnJY/P35I6ZlWYnY F4jOA4k/069CRkBdu26PS38bOX8y7WfadxQNCZ4ZMuVF4HbBY8fmAeWwy9o7IXwKErm5 PZh40F3HRPZFGPKj+3xENicl9uOIuK31MTyE8s+4stbClZ1GwaZ3zlbmJwVCePULPefF HO7Q== X-Forwarded-Encrypted: i=1; AJvYcCXk7lAKAy84EeiTAA2FkkpQ1QNbxsyXcv3aBzLYjH8ZPsdSKnSljvBf/7sIhYnW74GpTajxBNIhAg==@kvack.org X-Gm-Message-State: AOJu0YxyAFpYH9I2/VBWIwd7HBmQYJLhOxnDThBenoegcFdVUW/pGINQ a1moXQnnSsbA/Io8huVLyG8wAO347siu5TuxA1ojNw//81NkCX2aR7Sq/LwTrXZSit0d0mP6eoq G4OeaPeYjkYqUNGH4H3eAbRrBZf+kC9E49AFyy4rkA/0LiXoJ X-Gm-Gg: ASbGncsLUcS9EewCutn2UTuvqfsbFoG5yyNiDC8SLLxlgfj0e2ONb/Aa7KnlyQVJo0u 7710ch1mpDan2oCMVSiLzqDNpsS+kB7a2R6GtH+vOhrfY+hz2txRgOyDOI7Akoik560wnbbLXxp znAiUFS1JmjsTRhRiH/WXwfXJ2IFq58qevyEFhU1XQv0kRCX3wkzZo5BwHNj5bMWLFGLz2tHefK uqAzM/zsZ18TnAxmVmPryiMDBiIFF/ZS5glgAVmIGSd1oAsvkuvGf1hKCFWG0C97kvMdGqMgJpP Mc/FjC+bLWtnVlE+F1VK0cP2pxDerS3RnGLQ2tMKOgyAqQRO78r8U9/5pfCL2Zxa5g== X-Received: by 2002:a05:6000:2ab:b0:38d:e3fd:1a1c with SMTP id ffacd0b85a97d-38de43b38d8mr568326f8f.23.1739216307218; Mon, 10 Feb 2025 11:38:27 -0800 (PST) X-Google-Smtp-Source: AGHT+IGsdgSvAB5qD8mlN6AGjEYvBbSJNfBUob/X5TfyCrhv/WAdQoby+DEB/v6QRZcPR+eQm0f2lQ== X-Received: by 2002:a05:6000:2ab:b0:38d:e3fd:1a1c with SMTP id ffacd0b85a97d-38de43b38d8mr568308f8f.23.1739216306759; Mon, 10 Feb 2025 11:38:26 -0800 (PST) Received: from localhost (p200300cbc734b80012c465cd348aaee6.dip0.t-ipconnect.de. [2003:cb:c734:b800:12c4:65cd:348a:aee6]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38dc2f6aeafsm11911098f8f.20.2025.02.10.11.38.23 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 10 Feb 2025 11:38:25 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, damon@lists.linux.dev, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Masami Hiramatsu , Oleg Nesterov , Peter Zijlstra , SeongJae Park , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe , Simona Vetter Subject: [PATCH v2 06/17] mm: use single SWP_DEVICE_EXCLUSIVE entry type Date: Mon, 10 Feb 2025 20:37:48 +0100 Message-ID: <20250210193801.781278-7-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250210193801.781278-1-david@redhat.com> References: <20250210193801.781278-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: FI46hNkC42djEBkXMoH4Eo4eLZxDrt_dFktWTB7kPVo_1739216307 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Stat-Signature: hmbpcsnetab3hno5r7mo4w4kxumo96ma X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 3C98720008 X-Rspam-User: X-HE-Tag: 1739216310-745505 X-HE-Meta: U2FsdGVkX18j6Ayt4ksnk7IUbyzBe5N6pwsdzFja21zf9J8b1h1ypqSn41cplMN/R00Lk5NE4J33R/XQCDul2A1teaZe5Evcq3zpndyACzad5KB7O6oLMrIGuF8X8px5M9g96iaz63h0oRgW4pDlQ0pbBBA7tOfl8eqOzWMyfRHnUuYTVe67WLVsgkVw+1UAXf0CsFQMGDPzLYz6poz6phKTcDCQZ35+ACIiIy4mX6OjOHX7IGIR9G0qRxj8lWg/qum2vgXvjNh+B7ZFsEKAIvSlit+MiYzibAr5RJL21hrS3SiMDBChPSF3UrWb37Vr82IYB/Mmwk1vx0xcsICIDyjWpLULjk3m/AZMFUpYCQECiXNNtaVWaxHOqbOHxd80q47MyiNC5aPSs1lq6GRA4gHjErYEbq59d/G6NHlihRPwKt9kAZTpcG/PFkIKsVCDYK/tDuTwK/2jIyMlJirRVSldv86Yw+ijaDEhRBZi+6A5KL0wozIInojsXKwGLaKMsg1jhTQCHuJJl5fJAMt9xkk9+eRFBwmlwrgBY/ekmYqMYNVwpqyu0JBMqySwqHz00LwjcOFsW0VnSPcedOcmhZnS4l8WaJCgndpgiDU5tB4r79Bsv4rhsOSC1Y8WQn5xC8FdDZc8xL9bO650NSV3bkLPnOWe7pV7LCQdafJIHW25ao32bidHvv4Z23/FnTCjkmT5qd3vQWRw4BQzJVO2rqFQSV8+6n+KdWODRgskqUdmjm6tO5lLe+pBlw796zvEjoYBiFiJNxr6aeGQ6LsQx8XU0bo3iOxsNPIzoHUtj8dzxBmJnK6Ci4yw630nh2mLEan4BS4ZBpXB5iVfdljME8pBaZsA6t6qYEqcfNSXpfnxsCEBUBawdQ0+WsOEmCdNzvJg6GyVrrhcpH5b9fiK4K737x1ErKJlQ2hZAdVHs5jzLTc2GJlITn68KGsu3kwu8HEul8lcCEGOh8Ufc8p D5YR9nHe lMpPV8gq4brqSqbjoadJz2b/TKPAXpUOwN6KfCXbcwKsBhhhPuRYbZCgqT3hPWzpQKkLzXf91Nu4sawiIMHCaK1kGRWF02PI6DS+PKOmN0yUzo12XBgKQdb4Qe+26ZmRFgTGkVzbRufkBAtvaUcHgAwW8EQyGRA+uEyIpm8M1XTi95yTDAb0vWHm7ziwY4KvM3InFVhjBRgvz0leDf6MP3DmX0zYhZAAi4oKlD9hzJ5UU7N7sT/IsILwhu77eMML1dUcwEVIFJba/9A+RxrlzYhpgd1mhe4bARYe/BjKRqXLWyvk1GLBr7JZRJwFp6oKyjHWsgwfD2593s5tqYGSOMMbFpmDgUybVvfVdavYusj+fTG4cXHQO0/zuqW4mk2wXWLWUWnarrrxaGO5j7HcG88+ZPLHYDJ52Qbup7EoGGGq0ec3mjh9goGZn+EJBIDEItuMpHYP9ARqY8NX+MC4XHZvolwFqIdEo53zTTx/XIRYOs38WBt3sulybN25lbN3KzlD6UzjJ37MkA4pTT6LxbsoDmcoXR/3ix8FdF2o/ob5fNf8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There is no need for the distinction anymore; let's merge the readable and writable device-exclusive entries into a single device-exclusive entry type. Acked-by: Simona Vetter Reviewed-by: Alistair Popple Signed-off-by: David Hildenbrand --- include/linux/swap.h | 7 +++---- include/linux/swapops.h | 27 ++++----------------------- mm/mprotect.c | 8 -------- mm/page_table_check.c | 5 ++--- mm/rmap.c | 2 +- 5 files changed, 10 insertions(+), 39 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index b13b72645db33..26b1d8cc5b0e7 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -74,14 +74,13 @@ static inline int current_is_kswapd(void) * to a special SWP_DEVICE_{READ|WRITE} entry. * * When a page is mapped by the device for exclusive access we set the CPU page - * table entries to special SWP_DEVICE_EXCLUSIVE_* entries. + * table entries to a special SWP_DEVICE_EXCLUSIVE entry. */ #ifdef CONFIG_DEVICE_PRIVATE -#define SWP_DEVICE_NUM 4 +#define SWP_DEVICE_NUM 3 #define SWP_DEVICE_WRITE (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM) #define SWP_DEVICE_READ (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+1) -#define SWP_DEVICE_EXCLUSIVE_WRITE (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+2) -#define SWP_DEVICE_EXCLUSIVE_READ (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+3) +#define SWP_DEVICE_EXCLUSIVE (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+2) #else #define SWP_DEVICE_NUM 0 #endif diff --git a/include/linux/swapops.h b/include/linux/swapops.h index 96f26e29fefed..64ea151a7ae39 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -186,26 +186,16 @@ static inline bool is_writable_device_private_entry(swp_entry_t entry) return unlikely(swp_type(entry) == SWP_DEVICE_WRITE); } -static inline swp_entry_t make_readable_device_exclusive_entry(pgoff_t offset) +static inline swp_entry_t make_device_exclusive_entry(pgoff_t offset) { - return swp_entry(SWP_DEVICE_EXCLUSIVE_READ, offset); -} - -static inline swp_entry_t make_writable_device_exclusive_entry(pgoff_t offset) -{ - return swp_entry(SWP_DEVICE_EXCLUSIVE_WRITE, offset); + return swp_entry(SWP_DEVICE_EXCLUSIVE, offset); } static inline bool is_device_exclusive_entry(swp_entry_t entry) { - return swp_type(entry) == SWP_DEVICE_EXCLUSIVE_READ || - swp_type(entry) == SWP_DEVICE_EXCLUSIVE_WRITE; + return swp_type(entry) == SWP_DEVICE_EXCLUSIVE; } -static inline bool is_writable_device_exclusive_entry(swp_entry_t entry) -{ - return unlikely(swp_type(entry) == SWP_DEVICE_EXCLUSIVE_WRITE); -} #else /* CONFIG_DEVICE_PRIVATE */ static inline swp_entry_t make_readable_device_private_entry(pgoff_t offset) { @@ -227,12 +217,7 @@ static inline bool is_writable_device_private_entry(swp_entry_t entry) return false; } -static inline swp_entry_t make_readable_device_exclusive_entry(pgoff_t offset) -{ - return swp_entry(0, 0); -} - -static inline swp_entry_t make_writable_device_exclusive_entry(pgoff_t offset) +static inline swp_entry_t make_device_exclusive_entry(pgoff_t offset) { return swp_entry(0, 0); } @@ -242,10 +227,6 @@ static inline bool is_device_exclusive_entry(swp_entry_t entry) return false; } -static inline bool is_writable_device_exclusive_entry(swp_entry_t entry) -{ - return false; -} #endif /* CONFIG_DEVICE_PRIVATE */ #ifdef CONFIG_MIGRATION diff --git a/mm/mprotect.c b/mm/mprotect.c index 516b1d847e2cd..9cb6ab7c40480 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -225,14 +225,6 @@ static long change_pte_range(struct mmu_gather *tlb, newpte = swp_entry_to_pte(entry); if (pte_swp_uffd_wp(oldpte)) newpte = pte_swp_mkuffd_wp(newpte); - } else if (is_writable_device_exclusive_entry(entry)) { - entry = make_readable_device_exclusive_entry( - swp_offset(entry)); - newpte = swp_entry_to_pte(entry); - if (pte_swp_soft_dirty(oldpte)) - newpte = pte_swp_mksoft_dirty(newpte); - if (pte_swp_uffd_wp(oldpte)) - newpte = pte_swp_mkuffd_wp(newpte); } else if (is_pte_marker_entry(entry)) { /* * Ignore error swap entries unconditionally, diff --git a/mm/page_table_check.c b/mm/page_table_check.c index 509c6ef8de400..c2b3600429a0c 100644 --- a/mm/page_table_check.c +++ b/mm/page_table_check.c @@ -196,9 +196,8 @@ EXPORT_SYMBOL(__page_table_check_pud_clear); /* Whether the swap entry cached writable information */ static inline bool swap_cached_writable(swp_entry_t entry) { - return is_writable_device_exclusive_entry(entry) || - is_writable_device_private_entry(entry) || - is_writable_migration_entry(entry); + return is_writable_device_private_entry(entry) || + is_writable_migration_entry(entry); } static inline void page_table_check_pte_flags(pte_t pte) diff --git a/mm/rmap.c b/mm/rmap.c index 0cd2a2d3de00d..1129ed132af94 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2492,7 +2492,7 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, * do_swap_page() will trigger the conversion back while holding the * folio lock. */ - entry = make_writable_device_exclusive_entry(page_to_pfn(page)); + entry = make_device_exclusive_entry(page_to_pfn(page)); swp_pte = swp_entry_to_pte(entry); if (pte_soft_dirty(fw.pte)) swp_pte = pte_swp_mksoft_dirty(swp_pte); From patchwork Mon Feb 10 19:37:49 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13968587 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7B5FC0219E for ; Mon, 10 Feb 2025 19:38:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5897C280009; Mon, 10 Feb 2025 14:38:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 53949280001; Mon, 10 Feb 2025 14:38:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3642A280009; Mon, 10 Feb 2025 14:38:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 169C0280001 for ; Mon, 10 Feb 2025 14:38:37 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id DB26D1A04BF for ; Mon, 10 Feb 2025 19:38:36 +0000 (UTC) X-FDA: 83105047032.07.F56C390 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf12.hostedemail.com (Postfix) with ESMTP id A438F40005 for ; Mon, 10 Feb 2025 19:38:34 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ikpo1MCT; spf=pass (imf12.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739216314; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NPITbiV5yRtNZIanqm2v+SKlb5hdEoVu/zyhpD+AgeU=; b=dMJQIZh0VeeugPqOvlV9UjLFXrZA6LJY2AlthJ60rE4+11M63HRoZzuxTnq21tZLuC9cAH x8ud8tnM1MUgxCa4KgQzvLoDpSQvjuNcWFA4rVOYijufogkX0Gbb/BjsoW2VTJR+XjYkaA +4wcO6XklobnvVCsqk8qWVAqILv67no= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ikpo1MCT; spf=pass (imf12.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739216314; a=rsa-sha256; cv=none; b=jT4nXQv2j11QeCexM7fIE1shDmB1p7gsatf+byH5w69f0/LnZnvQQGEXcJrzpypNpCcztR XLp3hLomIO2sAyczAwW2NnSlF1KN6C1oTVF+/Hj6Plw/DI0V4ChE+xViSlMruQaEval0I/ KbXJBwTrIMenqYEjGSRG1D4bPfpWEEs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739216314; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NPITbiV5yRtNZIanqm2v+SKlb5hdEoVu/zyhpD+AgeU=; b=ikpo1MCTffr8Ho/CsjTxQinDzBJSBDm6LK1sMOgGp3IeFkuZ97IiW/rcOxRahPXs2RnVNs UB/aNkXi1DB6RTEIKPoJ69lKDqtUfbor+VUMI/31D+kaJ2UE4q8zU9NGL4O+TF+ZPf7h+I 8mrTEJmoqpQwZfgdk3D9cL5yoKTgl7c= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-665-Xy49bpHPO12omMHT1HudHA-1; Mon, 10 Feb 2025 14:38:32 -0500 X-MC-Unique: Xy49bpHPO12omMHT1HudHA-1 X-Mimecast-MFC-AGG-ID: Xy49bpHPO12omMHT1HudHA Received: by mail-wm1-f69.google.com with SMTP id 5b1f17b1804b1-4392fc6bceaso13133075e9.2 for ; Mon, 10 Feb 2025 11:38:32 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739216311; x=1739821111; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NPITbiV5yRtNZIanqm2v+SKlb5hdEoVu/zyhpD+AgeU=; b=u5LxzROyyiOEZRM4WddzTY/yMahr2VolvYdosXss+YV9ihAktRaGiCbOPEuFvssDyS aLei8GMOL/b2BzVZaXHIVlyz4BlStVfFELSaoIVpEN6KMiE8mAhO38S84887eex8uycm XxTu8I3JLigzAFvaBe0N9aIMbI+mm1fLV9JuTBpYdEq/RfNKACUrEzUtLCFuxgyeTbPq HhfT8g/+y6zt7ATNR6JXm3LQkuhy7WYFBoH7Sz/uJD2tJXlFDSl8ooIZfgdLMtjaz7sy zJ3HauslazruqeEj2soemYb7pNO2vFRifxW+5FQ10HwL1y4MCIrwDwzAjPZq9jruLh/Q IX5w== X-Forwarded-Encrypted: i=1; AJvYcCUfhjcT+mV9nxL7NewJr8xsf8RVC8umYXEAokqTe16SCAl6Pqv2vX4cdYrrPnUV2LR4p2vOZymP+Q==@kvack.org X-Gm-Message-State: AOJu0YzxJIoLr6yIG34gPtzgDOZ561kRN40VgXK6QgU2LGDWVYPehHK4 jTSk6bM2Jnwm3YVexOinB5JQ7xDlz+pwcTgsqoF+YEPc+ITyi1L3UdnsyADuXphJb/6Qjwv1Bn5 FTNRkKklhY/ajDWrtb1XJbhOQLbq84nvIAMgNOtEFKjcAH3OF X-Gm-Gg: ASbGncsocJqzPlADV1pqUX6wxiUFmtfZXpHms3wXu2HMGU5e8y93aHulYan9pvesYiC sHO+pcox6R2suQ+1uW9pWeh1WY3UR6w4S5ifAEQtxWN5Q6SINWTu0O/3ieBLp/wVNatqrSAT7pl dgRivJn25+iKftvNBVVbSC/LY1GdPrznMwl0lakejUFXUH+EBVcSnaG7uKd8i1Msy+uNvzSvEFe hnFFBLbgdbGgUPDO83sYrvBfGs8IB9Mc0xkG8/KUX3ms+C68tU71K/nrM+SHZJ0QLktxw9yqMe7 P6c8FVmvolWxsDYp6YD3CrymidfXJpIIUxqKuZXDRzy1AdnXOTWELvLrX8O8UHdO3g== X-Received: by 2002:a05:600c:500d:b0:439:3dc0:29b6 with SMTP id 5b1f17b1804b1-4393dc02be1mr57220355e9.2.1739216311481; Mon, 10 Feb 2025 11:38:31 -0800 (PST) X-Google-Smtp-Source: AGHT+IHbUQzL3Yq1p3VwDR+lxH773I/PQZNBDcgTpUgE2nEJHxULXhio/DNj6yNxLtRtBjWIFSwTIg== X-Received: by 2002:a05:600c:500d:b0:439:3dc0:29b6 with SMTP id 5b1f17b1804b1-4393dc02be1mr57220015e9.2.1739216311167; Mon, 10 Feb 2025 11:38:31 -0800 (PST) Received: from localhost (p200300cbc734b80012c465cd348aaee6.dip0.t-ipconnect.de. [2003:cb:c734:b800:12c4:65cd:348a:aee6]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-4390d933523sm192523445e9.1.2025.02.10.11.38.27 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 10 Feb 2025 11:38:29 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, damon@lists.linux.dev, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Masami Hiramatsu , Oleg Nesterov , Peter Zijlstra , SeongJae Park , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v2 07/17] mm/page_vma_mapped: device-exclusive entries are not migration entries Date: Mon, 10 Feb 2025 20:37:49 +0100 Message-ID: <20250210193801.781278-8-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250210193801.781278-1-david@redhat.com> References: <20250210193801.781278-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: rOwe8I3yhbyXZKWhdSatL4zrtvbTtI0FiKmt3zXj5dc_1739216311 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Queue-Id: A438F40005 X-Stat-Signature: ep3onmu49m9ues5o6dyzegdxeib3pikb X-Rspamd-Server: rspam03 X-HE-Tag: 1739216314-998026 X-HE-Meta: U2FsdGVkX1+4IDMUfhrqKEignI3ZCYwVxtQPRdA7gLqQ768DGSbEXVBDXIidSyCqNQ6Rdo8v7sTV2YhHk5xyqzTVJnIh3yKO+5ootP6KH1TpA2hAXabpC0gEXsdukT/NH5h1N2Qqbr+U1MLF6x+DzExwFtFOkQkSrg7rl5Au4jUcHg2j4mge7qeNGkeNa0QVbGYR0lW5i7SP3egIQz3cAuk0yfPtFFvOnWMd2e1byCnm2pXcnTVaUUoYqELRhNcNE7wuE9wsUl+GkTJwO5YGtKWixKbfWHRa2r6WvivQI9c+aQ/7nyGEWdAfMtbmiVOy8X+PlRjvnJS1406LLTHlFpWx31oo2+CcnSBRPME4jOR30rSr1egXDWQCTwVZUnNI2A8AMEDxGBksRdy/21aM201/GS+WuVRYydPwEHtywd+s80an3OPbifUuW8BhvijMSAfT7bI28zqjOHniLWKFMBZNvqg9n3o0TUYkBL0Xnhf/lvz5aEKwgWR8OMzn2STsvAS4Qe+gBQ7TwI1iwB5EIW+D9z/gRTuc/exHaJGCtn38IOMU0Yb/COEQDFfws8K5NOJLCsZUgoabXpvqIAtec6ufbnDnXypGEdQ+Bm2IsnsJaxCJffUkGgJOd9impU+FfIczWgOJaQunS167OCLFJKrbaC3w63qyUv3vbiinkFhE8EdtKTKLJUessXR/mi4AqXfm5usOn0IuVHWF6h24HK7LFT92YEa0vp8lI7n8voS60uwATG6rNfmP1vM1+vYjbAsArzVeqSMqQVFJjN1IQRmCQNTHYfegZxq5bBOirIinN7V84t9P4aA1/QzaAHxLFSTMOFG2eUH5EhRBYUs/1X95oA2s5tMM2iE2FnpFP9BiMm/sVXL4haunnByoT9gcPiqR/jis6J9b/LKsRh83OY6DWldbWt0tYF28DeSpd7hhOhpTx/YaxPWgmrvU5LKhsuv1TI8lYw9iH4VuhUA JHgI3ST5 RuAM+M7H45/NZy0LkjVLcLz+UOUDFpP2S3QwD5aHlmDcZnFtJEqn1oxDMlFuls2QNnTLA82r57oihk8tCqZAL4FGTlT7yVct+mQGclmeg1wLiKwmwBompuBjoWU/Phf7JUMOXy5lN3w1WhEsRFOcwXVmqMJ2Tv8eRzOdasNX4WcHOQfz2jbPy+wIoXJlKo1J+gLISzwUQmwxhBgecFAcPnjbiVAVlBaByfM9QR7BYi2zAvn7waszG1eaHbrQIa5GHD24Iy+rV1aqAptXAJoB3qMXyJXoyTxphDinXNNjo8/WuQ/GtesoOKH3YTL2o3Xr6cRu06yC4GnUbY+6ZbSXaSRjJHPrxblatjo/1AT/wy62OqdhN26jXzKxDfAunVv63oxyPX0EkTU4pkSD4Vev9VeLqv5OjDWlbaJ7G+xUamwtw3wKXYWrsC95ZULkd/5Vc2K0VZARQf9aPtYEY9OOBY177QPgeOQrPv1u8p0yB5xPaOYgw/FbzyLK/cKN28oz8MIIPlFg1V5cegeX89C5qIRMoAooYNAyXKCdgwEAZeRQ6WfQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000205, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: It's unclear why they would be considered migration entries; they are not. Likely we'll never really trigger that case in practice, because migration (including folio split) of a folio that has device-exclusive entries is never started, as we would detect "additional references": device-exclusive entries adjust the mapcount, but not the refcount. Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Reviewed-by: Alistair Popple Signed-off-by: David Hildenbrand --- mm/page_vma_mapped.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 81839a9e74f16..32679be22d30c 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -111,8 +111,7 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw) return false; entry = pte_to_swp_entry(ptent); - if (!is_migration_entry(entry) && - !is_device_exclusive_entry(entry)) + if (!is_migration_entry(entry)) return false; pfn = swp_offset_pfn(entry); From patchwork Mon Feb 10 19:37:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13968588 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A178C0219E for ; Mon, 10 Feb 2025 19:38:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 95940280002; Mon, 10 Feb 2025 14:38:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 90961280001; Mon, 10 Feb 2025 14:38:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 75CFA280002; Mon, 10 Feb 2025 14:38:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 55CA1280001 for ; Mon, 10 Feb 2025 14:38:41 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E414F1C5DB0 for ; Mon, 10 Feb 2025 19:38:40 +0000 (UTC) X-FDA: 83105047200.30.B3AC0D4 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf25.hostedemail.com (Postfix) with ESMTP id 9B4D7A0006 for ; Mon, 10 Feb 2025 19:38:38 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PdE02XrA; spf=pass (imf25.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739216318; a=rsa-sha256; cv=none; b=bdSGEQf8xTZjU3sd7+oaav83rJL6Fi1+Jc8QYTf1/vTfuvwN03V6UXKFIRTPdMMho5xOsV lh3Btb/DT0jghZjEEcs2359FpEljpflojfwWQJfGWddpLAnYPOKuJigqHf8mMk2pYn69Ym PiGfwVaCyk+7ixzzg+2dWgo6izonE1Q= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PdE02XrA; spf=pass (imf25.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739216318; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8sC+CKPivo9nJ2nwAVN+E4SAgEKJPEiCVQrD/mzwcqM=; b=OfazDnN16dM5b26UXw2WYPazev37jhNUVvXmpnwv3C9Kz5zZBRyQfeWyhGF6KJCxSrEKe6 et2R5L2l3HcDl9vEQ+JkbQbztB/4n6+fN0VhQ4wGby93ObLS7UQJP6ACziBFIiFuxGrKHh ga0IAjddTBLn0JFyfWc30varerpkdjk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739216318; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8sC+CKPivo9nJ2nwAVN+E4SAgEKJPEiCVQrD/mzwcqM=; b=PdE02XrAQ0Z3/Ks7U0EExqc5jHg+iXq8xaygP2zD56pBR0wV0Xp9iIJZLOARoAfeLZDwTi h8Za3ZPWLahslaLCX/0DTXTI+vBvT1koONkJyptkdL10130Sp+aoYjPXHh90X4coRYib3u +Nj62xbcvj14Jd8gYNiHMowR1L6wwAA= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-25-HKXvo7p8NNKDjMYtszkLAA-1; Mon, 10 Feb 2025 14:38:36 -0500 X-MC-Unique: HKXvo7p8NNKDjMYtszkLAA-1 X-Mimecast-MFC-AGG-ID: HKXvo7p8NNKDjMYtszkLAA Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-43933b8d9b1so12560685e9.3 for ; Mon, 10 Feb 2025 11:38:36 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739216315; x=1739821115; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8sC+CKPivo9nJ2nwAVN+E4SAgEKJPEiCVQrD/mzwcqM=; b=jC7G5W0lNeoGX3e3Gj/WLivvj9+2X2jMQJcDfJw+r3mQbis3T775j1pFvpGC3hcYsM cZPAsA0aPBGdDlCS8k6tvgMrZgVg/Jckkj0bBXhRfrQnHuqwdAzmKdrssRPp3KLy4xaU 9Apnf0FDR9J9lWFUBczyXML3NbUIc/1o8hqhaWAJymLZ3GGoOtM4RL7sleT1W39GICmt yHYETmvuxZh4mbNBrQPDT7x3lAJ5NsNQd937vIJnGWYycfferc/1FJwtbhXxrbtp474F W99EJvV2J5mWd08UCJs35Bh14zsWhDxHCzPjWfSRd3x4bxEK+qXQKRcygIt0OlwhRDCx NLJQ== X-Forwarded-Encrypted: i=1; AJvYcCVzKctic8JxBFFxR52NyfOiBYWJW244h1MCKZbsjqLko3h44l6vIOkclF84CtiQhdCTrJsV/Nk0VA==@kvack.org X-Gm-Message-State: AOJu0YwciKDU7OPLC98KCWimRIyzEdqT2+KVJwKjYmjdjcCCqJ1r4JTh +Hf5XiZMf9gG+aWHJNBXyqH8k9znEE8ZF6kUA5IktZ1u0KZxy1MV5irW4k+yhlfXGRZXe70znaQ KPycalMXuC9O1VVcl2ZQXneQBxTdhbOVEAZA3vTNMNYvdQN1U X-Gm-Gg: ASbGncvVnv2h5oiAYZEkHgnCG6xYiEYUcIT8fa/JQhC6Ue3EuF96Fz/i+J3quhA8pA3 CO2QU7SdWbOQC//9cqPWl7SvMTR1ykgyzEBbckTOwbUvipWmIarnFowA+WpWLrS6//wUkoI9esB U6t+tYGJesr65tbkFG0UKns8wkEQnqfRIrquse1xEL9K+xojyCaDbA7GoqEbeMXGLB+gSkeC7zf KamQrLZxrB/b0u3r8ZGGKmVaBz4FCvE2igmI/uX4HM+4az8h8TJna4s1DxEJwVIfuYOdUbblrXu jRvG7TXtubsd5u0eHAH1chayvVgUMDZosVbxGTSrzUgRn4QXzugWWFgUF93Hr8MwmQ== X-Received: by 2002:a05:600c:4e91:b0:439:4637:9d9 with SMTP id 5b1f17b1804b1-43946370d97mr43287575e9.12.1739216315615; Mon, 10 Feb 2025 11:38:35 -0800 (PST) X-Google-Smtp-Source: AGHT+IF6B7q2dhfsb9dj8LY+8M1CJU6DFUQTZK7m2VERaPDmly8AB+qIiZRItOcPeyw5uy4LKosd2g== X-Received: by 2002:a05:600c:4e91:b0:439:4637:9d9 with SMTP id 5b1f17b1804b1-43946370d97mr43287075e9.12.1739216315147; Mon, 10 Feb 2025 11:38:35 -0800 (PST) Received: from localhost (p200300cbc734b80012c465cd348aaee6.dip0.t-ipconnect.de. [2003:cb:c734:b800:12c4:65cd:348a:aee6]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-4390d94d802sm195260345e9.12.2025.02.10.11.38.31 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 10 Feb 2025 11:38:33 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, damon@lists.linux.dev, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Masami Hiramatsu , Oleg Nesterov , Peter Zijlstra , SeongJae Park , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v2 08/17] kernel/events/uprobes: handle device-exclusive entries correctly in __replace_page() Date: Mon, 10 Feb 2025 20:37:50 +0100 Message-ID: <20250210193801.781278-9-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250210193801.781278-1-david@redhat.com> References: <20250210193801.781278-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: _OLNrBsubkuirE3lMwF-PTefCf5NCEd5T8yByHIfMSM_1739216316 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Queue-Id: 9B4D7A0006 X-Stat-Signature: jac9w3aa7681h9tgfczg4krcczmukqb9 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1739216318-329840 X-HE-Meta: U2FsdGVkX1/iKv9+/FAppCJ86o8iSfsadZYlVBrrSw9y8+U+M9ChPDJo3rGv1vkTfS5Kw5GILhQHMauWuKPNSrafahfTKooB4fux9UzsS62N4C/HA10H34sJ5QxyI0qzu38eWHhVWrMZJP9z0BmHHyNu4I2KDaPX9Twy7wf6klzu47VU0zPPcvWzYM8uB80SW3QkZUL5hBg8NokGfMbTPh/VAMuLIUE251PLVDQe6PZfArgxEyHQt/btSL7hvpiVSlho5zQVPeEK++6Y7qQOTF2e8Nj3GzUYXViXdxoD/5uvFO2B70g/ewTdmP9kzzvJQZQDACqRUgPUKcW0JLdtwjHs2s2IxTgE8d+ymGDmfawAQ1VhuRke7D5ldmQkfKPO3Ii/pOmEWLCGFfuEDkkaG21bCr8Q0kvFavjNeyJ0dFc3S49vEUCN0DnO9GMVdbTOX+NZ8u+h5BgSSVIq8KEr2BGTE3VEsbdVCLqoMTyiK0NmJvMFmGxYFxcCvklB+uI2R0kCUFrqM3DJcF6OAqlE+z+lr69y6aWGppiiJ4/r0XbRJgyxL2pEDY1H4o3KX7CB7uqhImhcgvhC2KCn5a2htsrre4fTFq0eoqYRRBk14YyRLSwXUktko0VplCWEJHORNx6Zxo2xSt+NJfW7V3m++B4sONGTncvQPoUuxLAqLC8toWqBe2BAmpNqW0aUSrUBGVNNWUY06//mqxroZOlNl5nh8lMQ5QJChzGds+/SrQrj5YPsNHe1i0tO/YZYYv19mAkqAB4pbqjLdMU9hoA2/ub15yk6rAPYQ+MS2IgMrAzhi2HHunHbiQAHDB/4gEI8M4uMUc8YQgT3iTarBpaY4uhnYkL3mU3jVHuqWYHdMbuhXDsCv3ZmIg6SVVv4F0wQF7+/sMWsepYGPAAN0LRUqK/ehwiPdiJVpfieTitNsSSYAM/63gwvHeps0dwh0PVlkHun61G4BGofJl1JkUM qbdCDLSe N1tEfj28HolR5TL5WcKpGb6KknLFpFkh8Gb0W9cZRz4JnqFtZds4Wfdid6hl6MUDx/gU5ajBao7/c6VCv2umkqyVIsAWlGrb68Vn8lPwThRNsB0bZHmiEGxMS2nSxFEdTm5CVviIbWCWQM3zekDPcEbjMEy8u5IgAabzQOhqaNXuNkFcB+ByEW1rSrkUpGOlQlD3QGyVpWWcgePeCZUEsiw/7CEKI/XzOUHYa5CKrP9pdIuE/ywn7wOffde5+iLDYQ0FazV0pXmr6tZNaVE+CJZs+noZOnhRUdVU+09evhBJ8XXT3UtC9A+vnsdt0JQPDwE23roARUaH8Lh2LydZgTdUM1E88JcVp2Sx0IqN73RDcJTVIr958TfjhtBbpnBQuXQfT/6GuYzAygqg8X2GWAmPUJEt1kFfaSqYkAPJvXeH/aKgSNHaP7nTeGkPuTrQ6Lv8ersAgIiDsUg+iL+u0XnPrEXzpEHTZjMzUxZnEcTtaGeZH9dpVgKMljFUTRJaMN+bZg8MRv1yl2QfoHhNdOZO4Og== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000027, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Ever since commit b756a3b5e7ea ("mm: device exclusive memory access") we can return with a device-exclusive entry from page_vma_mapped_walk(). __replace_page() is not prepared for that, so teach it about these PFN swap PTEs. Note that device-private entries are so far not applicable on that path, because GUP would never have returned such folios (conversion to device-private happens by page migration, not in-place conversion of the PTE). There is a race between GUP and us locking the folio to look it up using page_vma_mapped_walk(), so this is likely a fix (unless something else could prevent that race, but it doesn't look like). pte_pfn() on something that is not a present pte could give use garbage, and we'd wrongly mess up the mapcount because it was already adjusted by calling folio_remove_rmap_pte() when making the entry device-exclusive. Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Signed-off-by: David Hildenbrand --- kernel/events/uprobes.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 2ca797cbe465f..cd6105b100325 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -173,6 +173,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, DEFINE_FOLIO_VMA_WALK(pvmw, old_folio, vma, addr, 0); int err; struct mmu_notifier_range range; + pte_t pte; mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, addr, addr + PAGE_SIZE); @@ -192,6 +193,16 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, if (!page_vma_mapped_walk(&pvmw)) goto unlock; VM_BUG_ON_PAGE(addr != pvmw.address, old_page); + pte = ptep_get(pvmw.pte); + + /* + * Handle PFN swap PTES, such as device-exclusive ones, that actually + * map pages: simply trigger GUP again to fix it up. + */ + if (unlikely(!pte_present(pte))) { + page_vma_mapped_walk_done(&pvmw); + goto unlock; + } if (new_page) { folio_get(new_folio); @@ -206,7 +217,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, inc_mm_counter(mm, MM_ANONPAGES); } - flush_cache_page(vma, addr, pte_pfn(ptep_get(pvmw.pte))); + flush_cache_page(vma, addr, pte_pfn(pte)); ptep_clear_flush(vma, addr, pvmw.pte); if (new_page) set_pte_at(mm, addr, pvmw.pte, From patchwork Mon Feb 10 19:37:51 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13968589 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1A58C021A1 for ; Mon, 10 Feb 2025 19:38:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4515828000A; Mon, 10 Feb 2025 14:38:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4000D280001; Mon, 10 Feb 2025 14:38:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 253E728000A; Mon, 10 Feb 2025 14:38:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 053B7280001 for ; Mon, 10 Feb 2025 14:38:44 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id AEF9016055F for ; Mon, 10 Feb 2025 19:38:44 +0000 (UTC) X-FDA: 83105047368.18.39D2513 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf29.hostedemail.com (Postfix) with ESMTP id 7FEA812000A for ; Mon, 10 Feb 2025 19:38:42 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WDw8D0Gg; spf=pass (imf29.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739216322; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=b7jLfJ9vH0eUU1vYEXZ3dUW5UEIvFQqjIYltl+lFPPU=; b=pyFA9buL1fJtw06h0N/OQnad7sK7SxMHwxfTcgjNktJ/swBsLaSuINjf7sRigRZR4Ro+Na OCurSTl5IVjnoGKmmAtRzFqt8MQZSPseosXS3Xf+BeImEHmZ7QMyx36fa7MrxJl9lmSFB0 Y6ehFPJBK7PyfhzGx/n0vyMgoG7ra7w= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WDw8D0Gg; spf=pass (imf29.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739216322; a=rsa-sha256; cv=none; b=Fd4yGrn+s+IyK4EcejH3Gw+zc7anXlApe6xk4SBmkt51rI2E24Xpn6lQHs3Nssj0nupYkU 4oKUkuiNvSJl6gYCxbQwXOPZEjuoiGKXr8zvcgNVbffZzLkJbI6OiaPyHLLUuntrEIac2a JrzqaKZsUYsBYI1o5ksUs5wmNn733+M= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739216321; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=b7jLfJ9vH0eUU1vYEXZ3dUW5UEIvFQqjIYltl+lFPPU=; b=WDw8D0GgGNKMZT0mBqbsJGqDgZ1bopJ6Wn8XeBgE2JfTPy/bXAIs9x7Icx+dU9j2gwqsH7 jNsRRx84/y3XfBK/gbSYVm6ut4C2//TEmmPA6N0oEjLTBJ09j5fk5qlU3sV7olL4mmVpB9 mHzpSMXwHM0FyOzUlGzTbGCVrtAXu2I= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-471-dafbIV8CNTmGrBoMKVC_Mw-1; Mon, 10 Feb 2025 14:38:39 -0500 X-MC-Unique: dafbIV8CNTmGrBoMKVC_Mw-1 X-Mimecast-MFC-AGG-ID: dafbIV8CNTmGrBoMKVC_Mw Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-4394c489babso2567015e9.1 for ; Mon, 10 Feb 2025 11:38:39 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739216318; x=1739821118; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=b7jLfJ9vH0eUU1vYEXZ3dUW5UEIvFQqjIYltl+lFPPU=; b=Is2r5g6W7fruZlQIAyq8RfeDxNllzUK0UkN1yC/XuD7MJhyPjvXdF5rgDhPh8gAUTn PVDxWxaxflmMtJsWKG3y8HR8d/mCE7gVDwU8zJQoBuhd1hhH/9NCLuIjlBzIZHgRHuS3 mvLilh7CgF16J+fTlCYWXWkRPLeDSK2CQpRlEPr9fLC8TsYzxKGdmL2T6smK2Gu7EEOb 2QSemQtDtqltCxj/Jw9IG0LEQafylTpnERL6001NxTjsdMSGvtBfSp4KdM9Wi0+EI2+x K/amt92aM6DnHBlrJKnG4BuaPgpRkEe5Dz3/cP3CJe/K1OZbG36KhS8xxmL81qTUX87A Ptbg== X-Forwarded-Encrypted: i=1; AJvYcCWUhSJSi2S7mDA/Tq9OkGUram7Rz73ezSn4g80WQdGQb1Ar/8nwFYmLRGTjuO1M7AQxxLNi33egIA==@kvack.org X-Gm-Message-State: AOJu0Yy8jDjoloTVKuRCKT3Kjb3VhlMxc+z1tMzQt5gtFo+FfQbKcRYz EdVbE1RLb6WNekEp7PBlWQi5fTrBOFCrC3FJkDojx/CNpzbJ2N5bkMgfB7s3y1R2dVGUszRjxuS wRXwkiR8r9Kh96RoW1m12221FQwiM0C8iChRoiNCphpShbNPf X-Gm-Gg: ASbGnctQdwnfLzW0iFTpKeyj/Z9JgO02mGO4rJ6QJrvNF2gOK8wwVnxielu9VncGMP0 denwKnMonDzXPJSja2FiQyPtkucXlK778gewih27LDgw3hmjD0HbLBfUJjstjgz1dXSuSy5O5Im X+AKfUwZxwxLJepGpqNF2B+E+jw0cXUVn1hxkiZXJI8SvbCgBREw1QiJQQKX4Zz3aa3DqQbf4NY y4RC4Zp6TkTH3qRnzXHABlEqNPz7EdGBnFFXktIJybWuFlEKc9IiKHmr7l2gyi2teoIUL6Kf0cd UoZmS3rWYvM2dJScH2OxwX/9+WUYyLIOoFDWvrCbcvMYeGAn3ECf/lf/rbguI0R4vA== X-Received: by 2002:a05:600c:4f90:b0:434:a7e7:a1ca with SMTP id 5b1f17b1804b1-439249b04f8mr116077435e9.20.1739216318623; Mon, 10 Feb 2025 11:38:38 -0800 (PST) X-Google-Smtp-Source: AGHT+IHdlczuWpIJN55C/ZdLw7mis7jMK/FKd714PEZJ/2b6DQtULEf1bpuI2yab2McnQRseLHdLgg== X-Received: by 2002:a05:600c:4f90:b0:434:a7e7:a1ca with SMTP id 5b1f17b1804b1-439249b04f8mr116077285e9.20.1739216318299; Mon, 10 Feb 2025 11:38:38 -0800 (PST) Received: from localhost (p200300cbc734b80012c465cd348aaee6.dip0.t-ipconnect.de. [2003:cb:c734:b800:12c4:65cd:348a:aee6]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-4391da96502sm158809495e9.1.2025.02.10.11.38.36 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 10 Feb 2025 11:38:37 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, damon@lists.linux.dev, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Masami Hiramatsu , Oleg Nesterov , Peter Zijlstra , SeongJae Park , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v2 09/17] mm/ksm: handle device-exclusive entries correctly in write_protect_page() Date: Mon, 10 Feb 2025 20:37:51 +0100 Message-ID: <20250210193801.781278-10-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250210193801.781278-1-david@redhat.com> References: <20250210193801.781278-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: WBSISAlZ0uEqVlirnYg-_HsnzhwrdqKLXKA_M0FUUAc_1739216319 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 7FEA812000A X-Stat-Signature: yrbyhjxrw7d1m6muqg91fij6yphwah85 X-Rspam-User: X-HE-Tag: 1739216322-720356 X-HE-Meta: U2FsdGVkX1/J7xSr23s17R9r+PDjGOheuO9aJHaJpVgKJqc2zEGVKBiQiQQgUIvknFl68rXZkAyEQRvvdBcZBEltJ3OxOBm8KyAd9EqJ+pSIrB2W31jdzl83mSHwu97n6K7gjkPWgjldLXS3qTc+GzyGB8laLGnm4b+phCJ3AEdKQH3dtFWHRpuK+z9+hketFwathmqnD14NOkxDBjd4MNOPIxxMcSYEZxZSoRApUeheUswuOV2wRywaTX5wsZ7V0fUEBgYpurOvAL67D+Xt2UU6dLliClBHv1QGQyzZLcp8JZ3XdNuLkf/s1gAusrORYSOPafnIpsFRvz59TpuYcstkehq8zFWN7OFyNG1wDmR1jtxBv6dBLYDzcGhCDUbEogf2JMQdJJSD/wGRlcWuZgN87AITV5cbD8VV1y4rkLQD3VxpICQCP9wi/gzQUKoXCYup0+Fbjp7c73KBpz6hCvgu/yxD8UF75FmZdaERqhoYz372NsrGj7yvMpv/8fAxQkWY2qYr8brREcRK/WS4kRnrNB+ku+Vs4zOF0Vh9Qhc3G6VuMsIkWVs5cs7GQFHPFo8a11xS7OwQXknh5Bq7OcE9ZElfh23+L2XkSFL++etPwl3j552QtqGYq53cRWRgE/hFEpXJ8Aahtp2VV55kMahLC2/NkOlVrzM8yUWcH50TZ+FxLC7BjDhZz+/V0SusgRRavGwZuy/PQBbE3IfHpWGEtXPAuCr8cEU3hRMOhtL0EaIpPHUnw+hBqZJBnEIbe82JzzndKuVVYeLxQ49wNpK5bEfkNKZKcRTq/gvZfnNBNPKU02HZIoFa0X9rTFT9AL2CyiFSvvy4WaY2QUKO0p4vb6DfdlvCI3fxFIHc2cJqyA6CEooEYHm5zT91HHF4QMLwRMoRlrfKRjENxu4qD3ZxOeLD0wYUI02aM9H6geFrL0R5OpeAHEFdXdxIzo/WZb4VB3HhOqsTKLnZrY7 GBNYGKi0 us/K3dT0hSSkvrY8wkG+U/UAQ5RuqC19EWWUrNk5ej7k2n1b5Z9xz2s3A7b8tnkxZx8epjIcbMqnI1nadZwj5jRaZFUyMffwmThlFKTztaHJ2drPlKzzRtyOKf+PSgWzRHzf/+rjix0x20svGCZpeGfLixN4GfLl5ICx2ypv4mIBLFkhhOFCB1pM7SRXrgdnV3W3EnF80M8Jq/TNlxePpTlIopUSCAP3iFeObiJ8VgdeqK/k2kx6An181rlW4PZ5ZkQWhQ5iEN62X0W753qHAT3Fum97UCyy0oCmQkZ56hsejvCdyH9pCL4WB+yxOyF88q+Ug/nxzXjpWUFoO7Xp9ypjKGph1QO9ykZNt+UQR50hdhXLH0+rFvYdMMIZnsP88PAoDZSWTX9lW9TU9rQYARpE20aVPH3EIe/ppwG3ybUBOLt1sEO54cEhoiwBGC7wk5qb0VlxlWSjKTsyFtSgWQ5qwAN+C8A9mJ5iUsWQXqXB+gSNb1peZ9mmFm52YiUSjnsFs6manQsA3VeoX3xOEDl+7ZQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000004, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Ever since commit b756a3b5e7ea ("mm: device exclusive memory access") we can return with a device-exclusive entry from page_vma_mapped_walk(). write_protect_page() is not prepared for that, so teach it about these PFN swap PTEs. Note that device-private entries are so far not applicable on that path, because GUP would never have returned such folios (conversion to device-private happens by page migration, not in-place conversion of the PTE). There is a race between performing the folio_walk (which fails on non-present PTEs) and locking the folio to look it up using page_vma_mapped_walk() again, so this is likely a fix (unless something else could prevent that race, but it doesn't look like). In the future it could be handled if ever required, for now just give up and ignore them like folio_walk would. Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Signed-off-by: David Hildenbrand --- mm/ksm.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/ksm.c b/mm/ksm.c index 8be2b144fefd6..8583fb91ef136 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1270,8 +1270,15 @@ static int write_protect_page(struct vm_area_struct *vma, struct folio *folio, if (WARN_ONCE(!pvmw.pte, "Unexpected PMD mapping?")) goto out_unlock; - anon_exclusive = PageAnonExclusive(&folio->page); entry = ptep_get(pvmw.pte); + /* + * Handle PFN swap PTEs, such as device-exclusive ones, that actually + * map pages: give up just like the next folio_walk would. + */ + if (unlikely(!pte_present(entry))) + goto out_unlock; + + anon_exclusive = PageAnonExclusive(&folio->page); if (pte_write(entry) || pte_dirty(entry) || anon_exclusive || mm_tlb_flush_pending(mm)) { swapped = folio_test_swapcache(folio); From patchwork Mon Feb 10 19:37:52 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13968590 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BB78C0219E for ; Mon, 10 Feb 2025 19:38:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9804B28000C; Mon, 10 Feb 2025 14:38:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 930BF28000B; Mon, 10 Feb 2025 14:38:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B64128000C; Mon, 10 Feb 2025 14:38:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 5996028000B for ; Mon, 10 Feb 2025 14:38:48 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 150521404A6 for ; Mon, 10 Feb 2025 19:38:48 +0000 (UTC) X-FDA: 83105047536.28.4E2919E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf15.hostedemail.com (Postfix) with ESMTP id D580BA0004 for ; Mon, 10 Feb 2025 19:38:45 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=S7+Hv9qA; spf=pass (imf15.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739216325; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LvU1uLGkpTPZRA0QaV3EggyJAUMnys9Fj6An1hzKeYY=; b=3toT9jSBq1Mddxc8F3/SYdshN83wgPCBcXUxeYV4///fUXqrCLId5uBpf9TDpfnMXqugPr 52EgKFg26eajZKgSB1Thw071Q8wMHZa3EdBLfl8rzmGna6vdWqA4QSEFotVUVzBJw5vG+y EchtkYYYbeVxR1uAFU6ENKL/hZfmZOU= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=S7+Hv9qA; spf=pass (imf15.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739216325; a=rsa-sha256; cv=none; b=LrVFQ583xtJRbJiWvfQz+SvJtoNi+2wAzJS9SklyzMV9+8wl/y9AtBvjf0FzelpLQq1ljS iukHrfffButFBx11idRXQd5nuuEVPFwJnKweL/xBp4TfvmsQi9tIqRDk7O2ddfQLkml6ci YjuO7eLVejdyVqEglmc5oWFgjAX5TjU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739216325; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LvU1uLGkpTPZRA0QaV3EggyJAUMnys9Fj6An1hzKeYY=; b=S7+Hv9qATbgD7eU6BGWJHQtjFg/XvWB5bxFfUvSqvsOhtRdcEE4IeNgiClOMF31rNHIsEG P3IclXPcm/3yAy8TpJcR7uPWWc8jYMJuSkC+S2LCDxm01ykS8d4CFcyeFxTJ9lDHvH/Ymv F1F/6HsUs6VtGKyFIiYYc8ITAqq2JEU= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-612-qAW7PO1uNM6XNHE_OFeuRA-1; Mon, 10 Feb 2025 14:38:44 -0500 X-MC-Unique: qAW7PO1uNM6XNHE_OFeuRA-1 X-Mimecast-MFC-AGG-ID: qAW7PO1uNM6XNHE_OFeuRA Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-43933b8d9b1so12562095e9.3 for ; Mon, 10 Feb 2025 11:38:43 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739216322; x=1739821122; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LvU1uLGkpTPZRA0QaV3EggyJAUMnys9Fj6An1hzKeYY=; b=gfVgyHmp2DlBsVLGf/2Nh6rv+MeVKtJcO1I42eH5teWObh+MsnlrX6ZAIol0Ilh2MZ LxInqSHuULtOwpUMtKF/021eqvaMIZENF24A2GajCnIXtV1ARtQZ8iO00zSCJtAA8rKS Bv9ea1to0ee9vQsp9h8HbCSGvUGLkO9mtXbVaV2orN9mMixUG1y26bjDlZb1B9inqWPl a4J/PQ8HMHeHmKQj49LbhAmpvJJh0uMaqO3f8bGpy2JIA21l2IXE7KRpaEkMxREocEEH jPYWp1PruvAEEuFDYAOoMQk/q8Vc0Czh63K21dRBXxb/7npOEnx6cT6IrTMipWVxX2r4 Z/SA== X-Forwarded-Encrypted: i=1; AJvYcCUrWSyYWr5KXtDzSPcYYVAezoEWqlMUIf5nZQWqJhe7rUqhTim+Jq4vCin61OCrvVOqSXCQqItcng==@kvack.org X-Gm-Message-State: AOJu0YyNBrNtPbi0SlPRYd+aCZtySOO5WUsgmtJTIvbWWUI9QjNkYRlR FeN/qe9UZ6MJYF7Wxqtv99wdZZzDJQLjzDW6FR5U9Yu2F+2UH2dmOW3oKxvh1/frzOA/6JuSaMH ZiB3dtUpypRFCeiiATsTtewAdyO9RukdhyWl8XXxPbcTPj7FK X-Gm-Gg: ASbGncsgtZ9TbjNYZNJvgFaW0MnqvUaLMquxAAOuD6FqBfx1a+AY7Qbl8/HgYTE+qxt iWRuPFx0urCPByuCOze62CFS1ffWmW7g9a1CCVRtyGafx30gDSTke7mt12Zdm8fZo1pRWpnuQj2 OgS3WhPT15fueEAQE6y813OovqtFHNuOtp16s7oiBbyFkY9cS9TMM4QTsEw0Rj+zsbiKvLXCsEr z1+cF5T0kI4evI1auy9u2vsDyCdZ/DwhBGxjZKpyDrxilXKb93fnYvSsXNhVHFqs5FMEYGAinMx IPNP2WYdXl16qRINUKnYm3EKw2EcldcPYSdqntrPU8FAbBSxddWaEQx4Id/sGj9YYg== X-Received: by 2002:a05:600c:1e0e:b0:431:5e3c:2ff0 with SMTP id 5b1f17b1804b1-439249889a8mr117196245e9.8.1739216322490; Mon, 10 Feb 2025 11:38:42 -0800 (PST) X-Google-Smtp-Source: AGHT+IGjSKW3zWL6vzVGiNBIPirrt7XpGCC7ElTlo8fIz4HKSkBODD3ovt81FL+SvLTQ6DCkqg1UrQ== X-Received: by 2002:a05:600c:1e0e:b0:431:5e3c:2ff0 with SMTP id 5b1f17b1804b1-439249889a8mr117195855e9.8.1739216321998; Mon, 10 Feb 2025 11:38:41 -0800 (PST) Received: from localhost (p200300cbc734b80012c465cd348aaee6.dip0.t-ipconnect.de. [2003:cb:c734:b800:12c4:65cd:348a:aee6]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38ddaf333c5sm5084761f8f.36.2025.02.10.11.38.39 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 10 Feb 2025 11:38:40 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, damon@lists.linux.dev, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Masami Hiramatsu , Oleg Nesterov , Peter Zijlstra , SeongJae Park , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v2 10/17] mm/rmap: handle device-exclusive entries correctly in try_to_unmap_one() Date: Mon, 10 Feb 2025 20:37:52 +0100 Message-ID: <20250210193801.781278-11-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250210193801.781278-1-david@redhat.com> References: <20250210193801.781278-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: Lodyc2mD8XroVASU4GZSuYfmWqxyL7D0oFviF0EL9U8_1739216322 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Queue-Id: D580BA0004 X-Stat-Signature: bpp3tk8o3zqw4wm7zye1bh3ymi44ja5m X-Rspamd-Server: rspam03 X-HE-Tag: 1739216325-495659 X-HE-Meta: U2FsdGVkX19NdOxPYqoduvdEtNlphgmZb1+AQJyUi0FI/tpa8jhHjkg9TITF220k0H3nivqFoTDzl3d0yxfEmByg2n+bN547geUZ4Ay9t8eeC8ICMYdmoilhdr4rYSCFbYMNFLw64uByuyDZ2yC6xNeyZ/3pj7o4+mUCA+GODJHBoKNprp6mYJx2ppvHnvE6yOqMQXCTUO8Zh4USUjqgAhTu1wOk0nn4vCOQk1C27IfVxgS78ZJyNVUM2ma5OIuQ9O1WGLKoO3f2FkEHudW8sx0o7uKA5D4l04pZ/DFpTY1TUPgS85gufCOMf0UtqS07HGm+u0dvzAKNxTf8jNZXsCXIjEA8UwhRqRZAwetTAkzYpNUSpTJkSdkB0n/CanR7vYbHBSsXHrJxpNL3XOltVWYI2VYIQ89Z7CC5MHmbLSj29JTH7nkTZkHST00z1f5Fus/rwKQW8ydxmVxolx7+197S0Cn1WhxUuHeb2Pewgvh6N9OJX/B23gK1pNNTkwe/HWN3sy78eSkflPj+nIQoY+4RVGjWdA+aQJO3xHr9DRxrD52QZwpMKa9oToUtoJbUa/stU/xLg0TfUP0YiIkM4yeBFs8MNjPwJXR8Zymrgd65G5L0e/eYFTLdUHnCAt4+uYbELTbeiPR1sZkm6Tq/iKnS6/57zyPc9Ixvr6b1NbcXlgCk5So24tX1wZ03FPkxOfOSU0gTKOS4m9M9pa25zX0yLYrFmZXrkSDeTi3mwCF3D8UQsDYExxpyuH6lC8lqz8vDGTqTEurMeg2QOJYKFB+39U8VPuTHk37rXaN6+SSzM+NHxvsQgKlLfsA5ksqFOPOG4BUW2w1PDnE4rEtG8i3sy+xtjWN4Jbib/AjQWjCooPEqqMEk2iagJpo5LG4c7T6ZGPkP9CXQi9jRQBEvc+KIFvbwpx2BXTdVoB6mySP5Xv7c9t77mTsSLJ18djQ6S1g3SEU03+bS3vVhidm 5+E2YnEx +wKHoemkHGTViSy5mYVdoOKuhKlmkue+eMhq1OlbQKC1ozeQdLzspAIReP2UmbD46j35VKeOTPHblm2Pb0CuyxQmizpWCTXaFsU7K+21AyqMKNZisEWODIEDeEWxSpwRDENp7vAj12kKi79eS4rgR7/p1HNe7MMmw0IHQhM7KorT9HK9oIA8sIbgbAS/18C3cNLuOqILpdccZAt3Eakb34HvqzR6ko0LwdSb3ppFebhUMlQN9p+LU5KVLQxM5xbgXPzHy1ItRAUu5OrVlAi0jKwegOkY/mDnc2288UFtqhH6+utPh5Hg8tYYn5N80Q9NZur/TakoByCTBi2u97Ghs+7LwJjqyMaInLIImE4v7tpYzWhwdfn3YjZxAYjKDwuksSmzUE633bVmZmHMWDgJFG9uIGXjmVzf3ZSHCCAfY55m5WNhbL0R2s626LMc3KwRha4a9/F0gc6JhRB7+pKUV0ce75uaQwa47RsGTnWkg6PcT5h61ojaulVccTgxwAiz51S6snfHjiAndcQswJbF2+XbkAg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000011, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Ever since commit b756a3b5e7ea ("mm: device exclusive memory access") we can return with a device-exclusive entry from page_vma_mapped_walk(). try_to_unmap_one() is not prepared for that, so teach it about these PFN swap PTEs. Note that device-private entries are so far not applicable on that path, as we expect ZONE_DEVICE pages so far only in migration code when it comes to the RMAP. Note that we could currently only run into this case with device-exclusive entries on THPs. We still adjust the mapcount on conversion to device-exclusive; this makes the rmap walk abort early for small folios, because we'll always have !folio_mapped() with a single device-exclusive entry. We'll adjust the mapcount logic once all page_vma_mapped_walk() users can properly handle device-exclusive entries. Further note that try_to_unmap() calls MMU notifiers and holds the folio lock, so any device-exclusive users should be properly prepared for a device-exclusive PTE to "vanish". Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Signed-off-by: David Hildenbrand --- mm/rmap.c | 52 +++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 39 insertions(+), 13 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 1129ed132af94..47142a656ae51 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1648,9 +1648,9 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, { struct mm_struct *mm = vma->vm_mm; DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0); + bool anon_exclusive, ret = true; pte_t pteval; struct page *subpage; - bool anon_exclusive, ret = true; struct mmu_notifier_range range; enum ttu_flags flags = (enum ttu_flags)(long)arg; unsigned long pfn; @@ -1722,7 +1722,18 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, /* Unexpected PMD-mapped THP? */ VM_BUG_ON_FOLIO(!pvmw.pte, folio); - pfn = pte_pfn(ptep_get(pvmw.pte)); + /* + * Handle PFN swap PTEs, such as device-exclusive ones, that + * actually map pages. + */ + pteval = ptep_get(pvmw.pte); + if (likely(pte_present(pteval))) { + pfn = pte_pfn(pteval); + } else { + pfn = swp_offset_pfn(pte_to_swp_entry(pteval)); + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); + } + subpage = folio_page(folio, pfn - folio_pfn(folio)); address = pvmw.address; anon_exclusive = folio_test_anon(folio) && @@ -1778,7 +1789,9 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, hugetlb_vma_unlock_write(vma); } pteval = huge_ptep_clear_flush(vma, address, pvmw.pte); - } else { + if (pte_dirty(pteval)) + folio_mark_dirty(folio); + } else if (likely(pte_present(pteval))) { flush_cache_page(vma, address, pfn); /* Nuke the page table entry. */ if (should_defer_flush(mm, flags)) { @@ -1796,6 +1809,10 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, } else { pteval = ptep_clear_flush(vma, address, pvmw.pte); } + if (pte_dirty(pteval)) + folio_mark_dirty(folio); + } else { + pte_clear(mm, address, pvmw.pte); } /* @@ -1805,10 +1822,6 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, */ pte_install_uffd_wp_if_needed(vma, address, pvmw.pte, pteval); - /* Set the dirty flag on the folio now the pte is gone. */ - if (pte_dirty(pteval)) - folio_mark_dirty(folio); - /* Update high watermark before we lower rss */ update_hiwater_rss(mm); @@ -1822,8 +1835,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, dec_mm_counter(mm, mm_counter(folio)); set_pte_at(mm, address, pvmw.pte, pteval); } - - } else if (pte_unused(pteval) && !userfaultfd_armed(vma)) { + } else if (likely(pte_present(pteval)) && pte_unused(pteval) && + !userfaultfd_armed(vma)) { /* * The guest indicated that the page content is of no * interest anymore. Simply discard the pte, vmscan @@ -1902,6 +1915,12 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, set_pte_at(mm, address, pvmw.pte, pteval); goto walk_abort; } + + /* + * arch_unmap_one() is expected to be a NOP on + * architectures where we could have PFN swap PTEs, + * so we'll not check/care. + */ if (arch_unmap_one(mm, vma, address, pteval) < 0) { swap_free(entry); set_pte_at(mm, address, pvmw.pte, pteval); @@ -1926,10 +1945,17 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, swp_pte = swp_entry_to_pte(entry); if (anon_exclusive) swp_pte = pte_swp_mkexclusive(swp_pte); - if (pte_soft_dirty(pteval)) - swp_pte = pte_swp_mksoft_dirty(swp_pte); - if (pte_uffd_wp(pteval)) - swp_pte = pte_swp_mkuffd_wp(swp_pte); + if (likely(pte_present(pteval))) { + if (pte_soft_dirty(pteval)) + swp_pte = pte_swp_mksoft_dirty(swp_pte); + if (pte_uffd_wp(pteval)) + swp_pte = pte_swp_mkuffd_wp(swp_pte); + } else { + if (pte_swp_soft_dirty(pteval)) + swp_pte = pte_swp_mksoft_dirty(swp_pte); + if (pte_swp_uffd_wp(pteval)) + swp_pte = pte_swp_mkuffd_wp(swp_pte); + } set_pte_at(mm, address, pvmw.pte, swp_pte); } else { /* From patchwork Mon Feb 10 19:37:53 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13968591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D15E5C021A1 for ; Mon, 10 Feb 2025 19:38:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 626A428000D; Mon, 10 Feb 2025 14:38:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5AEB328000B; Mon, 10 Feb 2025 14:38:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 400B428000D; Mon, 10 Feb 2025 14:38:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1E7BE28000B for ; Mon, 10 Feb 2025 14:38:52 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B11B01204A8 for ; Mon, 10 Feb 2025 19:38:51 +0000 (UTC) X-FDA: 83105047662.06.3371B8D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 6BA9E180012 for ; Mon, 10 Feb 2025 19:38:49 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Er3L2kI6; spf=pass (imf06.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739216329; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QxHe6mgzr5LViG+7+kvBnycnKkPt1kZTw9hZ/c9dl+o=; b=OQRX0H5eWDg0hAjG3o88k4b7E92SIEYESSVsh1Q1aAiwBRkq1+fbJTWRbbrXEKOoNBMQVS U9h2ToHew15zBbQbFpU8gV5nHglpskQ/dqPh6U75EzQcSVjM53CIaGRcZZJ8GZJ3FotPAO OWPvI9r8R1eO0HumnUnte5AXa3iXkyA= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Er3L2kI6; spf=pass (imf06.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739216329; a=rsa-sha256; cv=none; b=JpZ1aeCGXoy4i4JRmpLwVjij6Yy+FfYsXnpFAS0FpdwPOVP5exQ2kUlJnYX2RzE5Fmb7v3 TLgZZon24fPfHyJrmZpzms/I3ysh1ysBewi1mweeenr23/XN5gxuysVDmu/tswvKKpJAGG HLVn/BZHpr5AR3ZAJii0M5XNtFpRIF4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739216328; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QxHe6mgzr5LViG+7+kvBnycnKkPt1kZTw9hZ/c9dl+o=; b=Er3L2kI6LbrBA74nW9z4h8Uqh9EhvS2+IJI6ini5gmcM0qMywBFHmic2B5uHSeAOXkLx+y qSfnlYhT1mwgjQRaNvW1tOQrVWo3J0gbgcbdlLYDPsYwBlOeVUpaBwHphxjzNBaxiFDXJS CJ+hRfSt8DUZ+qvv9wqvgQ0t79pBoC0= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-107-r8Xi-7WkMWO_jjg1xquNDw-1; Mon, 10 Feb 2025 14:38:47 -0500 X-MC-Unique: r8Xi-7WkMWO_jjg1xquNDw-1 X-Mimecast-MFC-AGG-ID: r8Xi-7WkMWO_jjg1xquNDw Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-43626224274so28059405e9.0 for ; Mon, 10 Feb 2025 11:38:47 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739216326; x=1739821126; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QxHe6mgzr5LViG+7+kvBnycnKkPt1kZTw9hZ/c9dl+o=; b=MS8BYM8Af++UAxQd5jdGE8rZ5AjkNCWtN2Wmg4Z3qvkjDzJ9frquQTi9BAvrfqLZI/ 38P8t10k+Bh6VRbQMSemO0zHuTJSrE8TJ7vVpsk6jztdhbN8HUX3HiNMk2K7jK/XRi4X N2cDyaXPFnGIXynElClSo5z4e34I7v5L6B8mn0VrGBATiGHvsczywft4UVSVXGAP55qF AhXHu5qT1PqTnj+t8T0tS+qEgpVKWK47iZnJyl0y7KLRVXwyVqFsQ2jcjufvwdbXmOIv Fm7FjyEB2NzQPwfHqyITh/EzAFEQJ6ZDSTUYA6f1pkO4oyXF7vox71rftMANcgpPjQlw bCpA== X-Forwarded-Encrypted: i=1; AJvYcCWyR8ykqk7oK8Wix+LCGTyi9EF+5U49fukd2nOfHb3+6rPGtY3YzRnjHDwACM+Viyw64G/4wtRJmA==@kvack.org X-Gm-Message-State: AOJu0YzHCtG9LNlL30Y00QfJsjK0p1vj6obae/VrWU5af29n890ArQIs wl8Of2L8aoBqArmXXiO4JVzUS9lQDKATdI3bRXcEUXeNdH8uMST0INfdWQMYpcVPyBI8lWAwj8x 1xs2C6JPLl76hUcsWUMMYRP9eDBf49PaThrsLw1Qozdq5vJ2E X-Gm-Gg: ASbGncuMcqutc3vhcsNonIH1fm8bogwfbmYvbd/dmOsixMrysnh3VZR0SaFNEJoVSad 3N7hDQsoUm0F9rfnFEHdsfbtvsZg2cZ9FVnrovMxZ5OxIslCTschaA7NrbtZ4uBDBqU4sCjzdJz TkC3gMHlmhl5LBK3LLy2bS/AM1USMlnbWisWwXlOnSPXc0AEjQLnNdp2HGk0zID25NECif2NbzI 7EVOAU/k9KZ0so2xxuk0Br57afB5eS/utM6cC+C6fmsGBZZC34ipw7zP9XKwp9D3WeK/6VDsK0g 6ZV5aaWL4d3Y73+jvt8whOrUSuU179gYEkfpURynIwHF962salXWPmJ9I4oaKL4tLg== X-Received: by 2002:a05:6000:2a6:b0:38d:dc4d:3473 with SMTP id ffacd0b85a97d-38ddc4d34b0mr6018880f8f.51.1739216326372; Mon, 10 Feb 2025 11:38:46 -0800 (PST) X-Google-Smtp-Source: AGHT+IEI9wcsziW5nBYTCF/f9iYx8hWmPInDi/J4NMO1Qty0GFsip5PSFKOaGK1UrWp2GzbjhG1/jA== X-Received: by 2002:a05:6000:2a6:b0:38d:dc4d:3473 with SMTP id ffacd0b85a97d-38ddc4d34b0mr6018834f8f.51.1739216325915; Mon, 10 Feb 2025 11:38:45 -0800 (PST) Received: from localhost (p200300cbc734b80012c465cd348aaee6.dip0.t-ipconnect.de. [2003:cb:c734:b800:12c4:65cd:348a:aee6]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38dd295200asm7894656f8f.44.2025.02.10.11.38.42 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 10 Feb 2025 11:38:44 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, damon@lists.linux.dev, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Masami Hiramatsu , Oleg Nesterov , Peter Zijlstra , SeongJae Park , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v2 11/17] mm/rmap: handle device-exclusive entries correctly in try_to_migrate_one() Date: Mon, 10 Feb 2025 20:37:53 +0100 Message-ID: <20250210193801.781278-12-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250210193801.781278-1-david@redhat.com> References: <20250210193801.781278-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: lw2nkDjxMXE5x73T4g7zbtha3QrKXJyzh8u0b9GsCHU_1739216326 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 6BA9E180012 X-Stat-Signature: a3cpukuqwaifwjegj3xy5y3nfxuhuyiy X-Rspam-User: X-HE-Tag: 1739216329-293861 X-HE-Meta: U2FsdGVkX19faexw9HQ4pFHaqliHVqrT7B84bC9gXfuWHU0MB4pyR8yXrZaSXznDrcZuTdIkl9OMQO+r8N/huJOAZOmkueQi5A1ecSt7PPoYd/W1Dm5rqQOo9qhADEtCcXf3/VbpqO3cEkE/c+9vyVFvlQ5TTM8ZNicubXjFYz+gSIzW6mKs5rJRwGdTqjhhkGNeHrJWsCJ2WplKWgt/1v4nPNw3rhx5HuV98F3Rv/Ae+hbUYYTFFx1QA6WNd7kw0bL33DQ9bKLEuZ3BB98k0N/r6O/m8qbzygCsxPEQxN43hjoI1uGiRZNHqWLCY6NhcaEkb2zYnd6no9HAyJY4E9GjzND2U7DCYidt0f7hzYQHqIWJqxZ7pJWsCxXvnz+jEOg6RWUP+f1726usr861K1V5VVJ2XRg2ElxZjE6ocjbE54gG7KeysKhVlepU9O16e5BdqvhC//uKr7DuYGRUaGsyuzykH3RcU1gC0WWEi9GBtPxuW55u/MN2/42VPlxJh1gUK4RvLdVP7hOiHR03BPumvKGiAdIv/EVAPes9hAjZr+9kOjwlbwzAbZFRwnH1d5k3qgEdaZDluWl2ywDHpeX4jKgBxhmFyjZDSYmcZR7DmIYwcCKCqGkHEBh0mLPmivoRMhv12Jw9CLLMB4ZPzq0nQIDH/zSnDAQF3ZonLhLfEmWvOHUNczcyozm9m2jQ0IIj1nDiBVjCyx/hdD59ZwSXX/YXm/FKyUa+tKzK01QtQN/jEoGODojm++Bu7odn019Kp6dnjSVCK5XU2bfTMnCSp2l8Y4hxHYY+UrEz/Eqc4+fCeUGBtJofJJdYSxqKsfdic1yrj++DNU8cIrMlMuXrIDCaT2m0aVBZovANESXtKK2gM6qJzr2UjNJ7sXfTPocqpV6AL0Ek5/1hXcCKA6wszw0z0hN5fgVr2sDzP+2znWylmwjNnCf1Arz0NknWP7yq8fB3PEUyEBCQNC9 cz9b87bw aCMAVIklStdcYGKJK7OHqI/JgLOIiUxYkBl0V7DKDWqgB5NrplDmtdH1KeAKLdMyaDvpVh4Hm36uCTtC/nYFkhGDHtMCRymLlZbL/XDIMaBVyJsNd1KjhIFvy6P+3knnJylll5GgqzSF2dVPFtoza5AVm4D8uu3/MEYWh8+XeMIJz7OuZP/rmeiFviW7eiQnLbjGJuvyPE0zTZOrZoCbaIH3xKitGFgH2WyOBZ4xNxWNZYiGOI6H+6YZVjbyNhI05Rdu0Skrp983TVSFJgoNqFF/evZdjygKm7JgLuhhEURi9X0nM4hcmPRFoRuBvlkQRC+l745y08BNO6edNkLkPRPyIjse7hKiusvx9JMyROy6ZMkomcZ8dbq/Os5L9gQkktohC3WXqlBxb3tXkFPshA/EI7q+Y+uXni13Mi9mLYGB8/37CANT8bQwZb3eT7KEWcmiW8KGqF+LUdtu3PVQtIlVkV8DwXDqGDXu7LOfz038lkAA3MJdYifTyTZgKoZsShcuX0onVXaYJADfXD/G0snk62Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Ever since commit b756a3b5e7ea ("mm: device exclusive memory access") we can return with a device-exclusive entry from page_vma_mapped_walk(). try_to_migrate_one() is not prepared for that, so teach it about these PFN swap PTEs. We already handle device-private entries by specializing on the folio, so we can reshuffle that code to make it work on the PFN swap PTEs instead. Get rid of the folio_is_device_private() handling. Note that we never currently expect device-private folios with HWPoison flag set at that point, so add a warning in case that ever changes and we can figure out what the right thing to do is. Note that we could currently only run into this case with device-exclusive entries on THPs. We still adjust the mapcount on conversion to device-exclusive; this makes the rmap walk abort early for small folios, because we'll always have !folio_mapped() with a single device-exclusive entry. We'll adjust the mapcount logic once all page_vma_mapped_walk() users can properly handle device-exclusive entries. Further note that try_to_migrate() calls MMU notifiers and holds the folio lock, so any device-exclusive users should be properly prepared for a device-exclusive PTE to "vanish". Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Signed-off-by: David Hildenbrand --- mm/rmap.c | 124 ++++++++++++++++++++++-------------------------------- 1 file changed, 51 insertions(+), 73 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 47142a656ae51..7c471c3ea64c4 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2039,9 +2039,9 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, { struct mm_struct *mm = vma->vm_mm; DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0); + bool anon_exclusive, writable, ret = true; pte_t pteval; struct page *subpage; - bool anon_exclusive, ret = true; struct mmu_notifier_range range; enum ttu_flags flags = (enum ttu_flags)(long)arg; unsigned long pfn; @@ -2108,24 +2108,19 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, /* Unexpected PMD-mapped THP? */ VM_BUG_ON_FOLIO(!pvmw.pte, folio); - pfn = pte_pfn(ptep_get(pvmw.pte)); - - if (folio_is_zone_device(folio)) { - /* - * Our PTE is a non-present device exclusive entry and - * calculating the subpage as for the common case would - * result in an invalid pointer. - * - * Since only PAGE_SIZE pages can currently be - * migrated, just set it to page. This will need to be - * changed when hugepage migrations to device private - * memory are supported. - */ - VM_BUG_ON_FOLIO(folio_nr_pages(folio) > 1, folio); - subpage = &folio->page; + /* + * Handle PFN swap PTEs, such as device-exclusive ones, that + * actually map pages. + */ + pteval = ptep_get(pvmw.pte); + if (likely(pte_present(pteval))) { + pfn = pte_pfn(pteval); } else { - subpage = folio_page(folio, pfn - folio_pfn(folio)); + pfn = swp_offset_pfn(pte_to_swp_entry(pteval)); + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); } + + subpage = folio_page(folio, pfn - folio_pfn(folio)); address = pvmw.address; anon_exclusive = folio_test_anon(folio) && PageAnonExclusive(subpage); @@ -2181,7 +2176,10 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, } /* Nuke the hugetlb page table entry */ pteval = huge_ptep_clear_flush(vma, address, pvmw.pte); - } else { + if (pte_dirty(pteval)) + folio_mark_dirty(folio); + writable = pte_write(pteval); + } else if (likely(pte_present(pteval))) { flush_cache_page(vma, address, pfn); /* Nuke the page table entry. */ if (should_defer_flush(mm, flags)) { @@ -2199,54 +2197,23 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, } else { pteval = ptep_clear_flush(vma, address, pvmw.pte); } + if (pte_dirty(pteval)) + folio_mark_dirty(folio); + writable = pte_write(pteval); + } else { + pte_clear(mm, address, pvmw.pte); + writable = is_writable_device_private_entry(pte_to_swp_entry(pteval)); } - /* Set the dirty flag on the folio now the pte is gone. */ - if (pte_dirty(pteval)) - folio_mark_dirty(folio); + VM_WARN_ON_FOLIO(writable && folio_test_anon(folio) && + !anon_exclusive, folio); /* Update high watermark before we lower rss */ update_hiwater_rss(mm); - if (folio_is_device_private(folio)) { - unsigned long pfn = folio_pfn(folio); - swp_entry_t entry; - pte_t swp_pte; - - if (anon_exclusive) - WARN_ON_ONCE(folio_try_share_anon_rmap_pte(folio, - subpage)); + if (PageHWPoison(subpage)) { + VM_WARN_ON_FOLIO(folio_is_device_private(folio), folio); - /* - * Store the pfn of the page in a special migration - * pte. do_swap_page() will wait until the migration - * pte is removed and then restart fault handling. - */ - entry = pte_to_swp_entry(pteval); - if (is_writable_device_private_entry(entry)) - entry = make_writable_migration_entry(pfn); - else if (anon_exclusive) - entry = make_readable_exclusive_migration_entry(pfn); - else - entry = make_readable_migration_entry(pfn); - swp_pte = swp_entry_to_pte(entry); - - /* - * pteval maps a zone device page and is therefore - * a swap pte. - */ - if (pte_swp_soft_dirty(pteval)) - swp_pte = pte_swp_mksoft_dirty(swp_pte); - if (pte_swp_uffd_wp(pteval)) - swp_pte = pte_swp_mkuffd_wp(swp_pte); - set_pte_at(mm, pvmw.address, pvmw.pte, swp_pte); - trace_set_migration_pte(pvmw.address, pte_val(swp_pte), - folio_order(folio)); - /* - * No need to invalidate here it will synchronize on - * against the special swap migration pte. - */ - } else if (PageHWPoison(subpage)) { pteval = swp_entry_to_pte(make_hwpoison_entry(subpage)); if (folio_test_hugetlb(folio)) { hugetlb_count_sub(folio_nr_pages(folio), mm); @@ -2256,8 +2223,8 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, dec_mm_counter(mm, mm_counter(folio)); set_pte_at(mm, address, pvmw.pte, pteval); } - - } else if (pte_unused(pteval) && !userfaultfd_armed(vma)) { + } else if (likely(pte_present(pteval)) && pte_unused(pteval) && + !userfaultfd_armed(vma)) { /* * The guest indicated that the page content is of no * interest anymore. Simply discard the pte, vmscan @@ -2273,6 +2240,11 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, swp_entry_t entry; pte_t swp_pte; + /* + * arch_unmap_one() is expected to be a NOP on + * architectures where we could have PFN swap PTEs, + * so we'll not check/care. + */ if (arch_unmap_one(mm, vma, address, pteval) < 0) { if (folio_test_hugetlb(folio)) set_huge_pte_at(mm, address, pvmw.pte, @@ -2283,8 +2255,6 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, page_vma_mapped_walk_done(&pvmw); break; } - VM_BUG_ON_PAGE(pte_write(pteval) && folio_test_anon(folio) && - !anon_exclusive, subpage); /* See folio_try_share_anon_rmap_pte(): clear PTE first. */ if (folio_test_hugetlb(folio)) { @@ -2309,7 +2279,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, * pte. do_swap_page() will wait until the migration * pte is removed and then restart fault handling. */ - if (pte_write(pteval)) + if (writable) entry = make_writable_migration_entry( page_to_pfn(subpage)); else if (anon_exclusive) @@ -2318,15 +2288,23 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, else entry = make_readable_migration_entry( page_to_pfn(subpage)); - if (pte_young(pteval)) - entry = make_migration_entry_young(entry); - if (pte_dirty(pteval)) - entry = make_migration_entry_dirty(entry); - swp_pte = swp_entry_to_pte(entry); - if (pte_soft_dirty(pteval)) - swp_pte = pte_swp_mksoft_dirty(swp_pte); - if (pte_uffd_wp(pteval)) - swp_pte = pte_swp_mkuffd_wp(swp_pte); + if (likely(pte_present(pteval))) { + if (pte_young(pteval)) + entry = make_migration_entry_young(entry); + if (pte_dirty(pteval)) + entry = make_migration_entry_dirty(entry); + swp_pte = swp_entry_to_pte(entry); + if (pte_soft_dirty(pteval)) + swp_pte = pte_swp_mksoft_dirty(swp_pte); + if (pte_uffd_wp(pteval)) + swp_pte = pte_swp_mkuffd_wp(swp_pte); + } else { + swp_pte = swp_entry_to_pte(entry); + if (pte_swp_soft_dirty(pteval)) + swp_pte = pte_swp_mksoft_dirty(swp_pte); + if (pte_swp_uffd_wp(pteval)) + swp_pte = pte_swp_mkuffd_wp(swp_pte); + } if (folio_test_hugetlb(folio)) set_huge_pte_at(mm, address, pvmw.pte, swp_pte, hsz); From patchwork Mon Feb 10 19:37:54 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13968592 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99768C021A3 for ; Mon, 10 Feb 2025 19:38:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 22F4528000E; Mon, 10 Feb 2025 14:38:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1DF5528000B; Mon, 10 Feb 2025 14:38:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0316B28000E; Mon, 10 Feb 2025 14:38:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D7C7F28000B for ; Mon, 10 Feb 2025 14:38:56 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9B71D14057F for ; Mon, 10 Feb 2025 19:38:56 +0000 (UTC) X-FDA: 83105047872.17.53DCCD3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 62319180011 for ; Mon, 10 Feb 2025 19:38:54 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=exaFgc4i; spf=pass (imf06.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739216334; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LTEWxPe8B8Wum8eXVDX5hUwjNBhNNdMWJNli9ItwISM=; b=BtfPNnKgaNl7urgaAh8fVBFrFnGUXAD99xXyhTsxi4uFg7I4tJCqbsE682fXRljZujVxaV WZxrFd1ZeJ9WhZ3+PSKawSMPtRSfZKOo6V3OWdHo9bLLQaRolJ6bGUFrDuv1LEPliFzeaD d7qu2X3S/2qgwBHJSMNjkPJVAWUdyNw= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=exaFgc4i; spf=pass (imf06.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739216334; a=rsa-sha256; cv=none; b=PgkVqcEy3Mu7/6Hj0n+6xUqcB4MasJ8XfbntBXZ7UrATw7ODskNakGmkKz64Ue6UEVSptu NxplA851DXnJdPMOxCq4N4bvVCYJ56gQ7ek5aDC8dzQbya5ytIDSx1WFUlJMPBSMJSJoTI OSFrQ+B+uDF/0IUkxcPtKIq7gVZgriI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739216333; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LTEWxPe8B8Wum8eXVDX5hUwjNBhNNdMWJNli9ItwISM=; b=exaFgc4ixzGQcjp0O1eWUTQ+b33P0eABwx3ZgkLYTWrtwk3TSYWFNCooosvbW8CvLNnRrx 9ItVMarkxoF62jUwRDFhOxW4qqmH5dK39PoyJo/UpG7pkhqb7SeWxuwfb/VUB+KIkq9Q3Q DB+PjRqvC/bo061YceYShwF5cUfuzKw= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-121-Bv4l3Ig0NvOAdXwgjm107A-1; Mon, 10 Feb 2025 14:38:51 -0500 X-MC-Unique: Bv4l3Ig0NvOAdXwgjm107A-1 X-Mimecast-MFC-AGG-ID: Bv4l3Ig0NvOAdXwgjm107A Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-43933eb7f7eso13411635e9.3 for ; Mon, 10 Feb 2025 11:38:50 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739216330; x=1739821130; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LTEWxPe8B8Wum8eXVDX5hUwjNBhNNdMWJNli9ItwISM=; b=KBFrljibxA1rZTnW5l8HwsSfTsgNfhM0p9Qvw1cZaL8SAVC0sC7ooRxee7AmMyOnMv so0BQuaWcQdniU/kEX7BOs8vjzcSPFr/n5mThG01VVn5TA02rb83C/l/+d42WaBW0StQ t1xOLgN9datjbZCnNiUWdUQUrmJAibDSEBNvCg6stcokINzZ3HsWaMNF+LG7/HzAKQDd KLDEFwjWsgxp6vHKUvRXsWUy/Bdh+7hqkCWmdkqbMXqqcum2lzfdWiJQPG1b2vL+iHtY 4WiUFa7Gc069DS82WaQ4S2zzlUQMlXgNDrfBItNv1FT4E52vtZg7aMhWBmZwYrpHkbid uc+Q== X-Forwarded-Encrypted: i=1; AJvYcCUh7tkw3enzHnn+RY5mQa9mLGBGu4nPwXEAPRzMq7wC8iDwXu4zYY6YmL4FSBqI+vu+vZUhVy1YSg==@kvack.org X-Gm-Message-State: AOJu0YwSGJHpq9/9X2y0PJv9eblqNCD3YRjtjoIAaVyCsUfEUq0NXDqi b0I8s4SnN2h/1cu5rg/UM5HMVUhdKVisxWkPhuzmos22DcIBVsBcadcgCwk3n2MzWzT2JgxWPQ5 lpG8+jL8EhCBi9pAij2qSFJ/CWLLX3Yl0sHpkjnVzXBwve2kT+p/0VnYInd0= X-Gm-Gg: ASbGncuo0psEGNEibaLgLXaXk39AazEepKnWhhuDygvFGj4FKb5rTuuIigC2diQ8J2c DyhICPKNxM7ieZtTTMxXkn37yBCPQmto49T2fhZmlH1OhpI+mbTLI9x/S7A5Ku8wUu2eEsJhpsO ourqy7tcM6RIOafBcSP6JY+9nPscLpFJ2ZcJTAKUiAxhVylWHWeyB8inUQf7exbtbeBu/JwofGW +rUJdlbdCeLFtUqnHF2fgKH+QkylO1GpQuIZ8rnwnDQIrjHb5gVYX/07Yco2Uo7ub2s++F1G8Ll OCADjWem+nrP+nBfzR68Rx0Xd2oKN99V8lUo8LpxyHPz5m6Kj1NmWddCYeG/FSmTNg== X-Received: by 2002:a05:600c:4f89:b0:439:4bb0:aba0 with SMTP id 5b1f17b1804b1-4394bb0adb6mr17902635e9.8.1739216329956; Mon, 10 Feb 2025 11:38:49 -0800 (PST) X-Google-Smtp-Source: AGHT+IH0Un4mhmMUf8CyrAHOSxNb3M+hrifrlRxr2GMjHtCC/u6kqhVG2aMU260gZj/4uoW1cKICNg== X-Received: by 2002:a05:600c:4f89:b0:439:4bb0:aba0 with SMTP id 5b1f17b1804b1-4394bb0adb6mr17902495e9.8.1739216329619; Mon, 10 Feb 2025 11:38:49 -0800 (PST) Received: from localhost (p200300cbc734b80012c465cd348aaee6.dip0.t-ipconnect.de. [2003:cb:c734:b800:12c4:65cd:348a:aee6]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-43947bdc5c4sm26951255e9.23.2025.02.10.11.38.46 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 10 Feb 2025 11:38:48 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, damon@lists.linux.dev, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Masami Hiramatsu , Oleg Nesterov , Peter Zijlstra , SeongJae Park , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v2 12/17] mm/rmap: handle device-exclusive entries correctly in page_vma_mkclean_one() Date: Mon, 10 Feb 2025 20:37:54 +0100 Message-ID: <20250210193801.781278-13-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250210193801.781278-1-david@redhat.com> References: <20250210193801.781278-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: XcD2NSayrJfJMiV_MOG3cOwqnHh_WzcY-DS8m37gGwM_1739216330 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Queue-Id: 62319180011 X-Stat-Signature: 6nzy6e78fe5tsf58s7dthdzrs1eaxfob X-Rspamd-Server: rspam03 X-HE-Tag: 1739216334-765153 X-HE-Meta: U2FsdGVkX185GZQ/CBudzj9FRSFgbvoPuiHj0qrHEqHZ4W7QSLHcLL1eu/YZcufY6Se1Kd81QPWnOO6Fp6Ibb6n8UJi2EQSL/1Kuox+fcglSinoILg794QeYJbud1z6Tc49g2ROvELt5ij3v01EJikXFtVaq7zvpwMzMp3++brjB7Tea7OKnu0l4FomuLqZDS1yp/jWYGw4LgGYuuB5vNnopBA8gF3jBqWRAvXSEH12rzBB1f9eoNYCSH/qMblvbPmPoH0n8zkFQJuP0JHx2Aelu1lTr5ZKhWja3zMJghd73nB7ko6a3ydWkyMgMhhNjQ2M/CqT8plXnVD/WPG7K7G3NpYaosxHbWikoL2gD+0R4znFE92WAZxg7HAbCs7WMSEi2bnYNy8IyyO27GQH3/pC8C4zfhvP9Z+r/8FFRXcNZdVnOltlP5Kl2aPx9KaDEgSLL/IuEc+ilW43iNu7vJGOAd/iDPBBHgCZERhKGOJuZ+9ihPpoLgwBY2WNRZbPU4rHbqbaBYVBOIKJqm0INfuSxKITu+NN1c5JLdDIkIEzMiqKHD+m1Sv8Gvj8UXcnTOg97HLO6eyyuZKlgFPFSC45T9rhIvOHiNn4EWUgyCDOqzDYZaOx48n+72KnqljezQLWuboN3lCSfX5Tgfp1sRGrMTfqQYM8zXuzTZhECDGA/47Bts1WtXL39sS50F5i875KzmUkZ4oc5LcxTaDlBzlD5lGziUU7U8HbNcDbznHJM+U8o7KIcZidxGhAzOko6BZxJGct4gaT8G8BX0T4LIcUTYWLMizF8yubWuIeJm1+lH9vGrXQ9aEUMJFIsNJ6pZiBbm+jam7eAABi9MlA3jAAjnUDWTo20NYiCBkugzzdtt22S4CEtVEoyjt4Q8GoeATWtygXbl0+FqFPpnxIHgpmvXwUqZaIokYTHR4Ae+3JqnRuhCLExMzCIso0MmGfllQCWoU5kJRMl2B5xMqX Ay0hpl4X BVTlLfl7kGyRkSKSJfjdsqIxp94qoTu8iUn8nvSz1zfNXzBHTibAnUnf68QzsAvqGE072nnlBatI3XyHXuzn1FPWItR7PNYsuj64VWSe1UJa4IckanIocWG0OMHj3tmgDaRPKBk5EmNb5VSLyrv4AQSUsmYZXOINZGbqmmQT8kf5S50gYmWp1JJjsdhPYUwznoBs1d3NTdkJZj16bDaqK+gsVfIdNqYB1/FBGxAm/A7jah0ZGh99KkW9ZT4ZamDL5Gwr3tnxjK6gGPyKYaCW+b44RCeXAT5C59LAXQLdla1FswLP4ZCX4v/GO5w4stQv7UxF2NWP+OKsoolfMip+mWu0NM5irCLmuFM3BrCN73VnELEKdxShiDRQDMVRRSY6WkBv2KdvGva5qfSt59dmzM8pMb4YYUhoJTAzgZw7PZQT/cYvqiztWFsHfpfTH7ilHgEGO+VYVPmm7VepScMh6MgFDVCBmy3P/U5sTsJ5CNDEVL3gUg/aqHsA1SGrJEF7Sf4uJrfbrbFLZ2Ex798nIwKOzpg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000013, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Ever since commit b756a3b5e7ea ("mm: device exclusive memory access") we can return with a device-exclusive entry from page_vma_mapped_walk(). page_vma_mkclean_one() is not prepared for that, so teach it about these PFN swap PTEs. Note that device-private entries are so far not applicable on that path, as we expect ZONE_DEVICE pages so far only in migration code when it comes to the RMAP. Note that we could currently only run into this case with device-exclusive entries on THPs. We still adjust the mapcount on conversion to device-exclusive; this makes the rmap walk abort early for small folios, because we'll always have !folio_mapped() with a single device-exclusive entry. We'll adjust the mapcount logic once all page_vma_mapped_walk() users can properly handle device-exclusive entries. Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Signed-off-by: David Hildenbrand --- mm/rmap.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/mm/rmap.c b/mm/rmap.c index 7c471c3ea64c4..7b737f0f68fb5 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1044,6 +1044,14 @@ static int page_vma_mkclean_one(struct page_vma_mapped_walk *pvmw) pte_t *pte = pvmw->pte; pte_t entry = ptep_get(pte); + /* + * PFN swap PTEs, such as device-exclusive ones, that + * actually map pages are clean and not writable from a + * CPU perspective. The MMU notifier takes care of any + * device aspects. + */ + if (!pte_present(entry)) + continue; if (!pte_dirty(entry) && !pte_write(entry)) continue; From patchwork Mon Feb 10 19:37:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13968593 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43A11C0219E for ; Mon, 10 Feb 2025 19:39:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C419E28000F; Mon, 10 Feb 2025 14:39:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BF13128000B; Mon, 10 Feb 2025 14:39:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A6A8228000F; Mon, 10 Feb 2025 14:39:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 851F428000B for ; Mon, 10 Feb 2025 14:39:01 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0AE5680515 for ; Mon, 10 Feb 2025 19:39:01 +0000 (UTC) X-FDA: 83105048082.21.62D35F1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf29.hostedemail.com (Postfix) with ESMTP id C6852120014 for ; Mon, 10 Feb 2025 19:38:58 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KnvVN5IM; spf=pass (imf29.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739216338; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mwHnciP+OvnJG9alUSG9/jEypyVF3ZPTXQLV9dVcfyg=; b=svyfhMss+ELEIaYrnJBhLxIGStdkbLZadyNNFR+Vv3ikir10Nm2BExs9ABngxeRTFG3lBm vRGky9XQPkNb1WQM8vt4tSB1Vga2cLlgzaI4k8A65qP9f4gh/oIsHjfnnbxMHkFme0NkoF w1/NLCAl6BURobVtvdkhnOqhIaFLn/U= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KnvVN5IM; spf=pass (imf29.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739216338; a=rsa-sha256; cv=none; b=1eL7ScwFzpR/3ropSueW8oEj59bIa6DUenYEIZKbksS+/vtLu6JIGYUpOqGJZbSrt8L3ss KD9/gxOe+x8ZDlPfN5jl48ae36RKfiP8UaO9qpF0W6g+KceDmOInVXh+tm+RFP0Z9116Di aDoNYn2eL1u6y4q74i+6o2yRyRhLfUY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739216338; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mwHnciP+OvnJG9alUSG9/jEypyVF3ZPTXQLV9dVcfyg=; b=KnvVN5IMxtkcDDowFZqh8kQ42q8NrSJE/w9c6lx64bRs/ftMieqW5Fku5dXR6/AXHzliy9 g8GFOHvit0FYelXwJteVtcIbv8VG8gbgbzJMAmyu9Slv+QzYhyRU55gGdwfpvPdDO1MhfU zksoefqjEMs0O5gRtjGL4O9R67iDvvY= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-121-0dl4y5PCMW6riv9Z_zUIrg-1; Mon, 10 Feb 2025 14:38:55 -0500 X-MC-Unique: 0dl4y5PCMW6riv9Z_zUIrg-1 X-Mimecast-MFC-AGG-ID: 0dl4y5PCMW6riv9Z_zUIrg Received: by mail-wr1-f69.google.com with SMTP id ffacd0b85a97d-38dc6aad9f8so1835252f8f.1 for ; Mon, 10 Feb 2025 11:38:54 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739216334; x=1739821134; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mwHnciP+OvnJG9alUSG9/jEypyVF3ZPTXQLV9dVcfyg=; b=bI/tYL7b4dqXQGigdMgdTi5Gjnnbs6LsNmX8X1zXFd1gkgQAHJjX8mnmi6k+wGqs+i dF1Bf4aOKE5niBjFmWegkDyI/VY+RsdgtVXP/Yfeicg6+gN1YnHh/eGxnlMivcK0BrML vw+uAFK8wTvfbl8sn/WZiIE8UHLy9M4RZ2TG9359Oli0toF5lLkVSbvOEkm4MTAuzFRU 7rEWM8nvyQ3hmXlEw4uSAfhY5AfW7DMsNRdjOXp05H/dcziYKkTXeuHIjTliKi6YQa/U TBFo61z5sDElVkH+kQ7pZjUyNk2vd2wShM+iWC3KXjjcL1wrxXOOyn3MB/IJbCm0peq+ cJrA== X-Forwarded-Encrypted: i=1; AJvYcCUBINRx2GNLqJzbfkOSkfV24KrFhtJuurwyt78Xjy1h66IOAZ97bnR7+a2lwEZ3fFY1+OMHhWM1Xw==@kvack.org X-Gm-Message-State: AOJu0Yz+VPMojPS2PjDVuZekOSaTUV4VjAfPuyRi5o4Fn4YpMeGiDw8b 590Ot3IUkYt2CBHLbQqaf5H/WSdDy9nJlOOLilTMEx+RzoHpVSnROitEBNtdiO7gphz/kGqt9Aj vMmCUjGKMpJ4BCAzHQg9t5+lkJGG8mSxuiDtopA9c7yBM4jYv X-Gm-Gg: ASbGncssroXTkpBIjhVbuUVjVRUcgPzfHCN9AsP/uEXXNJ6951kyM1AGOOWL5jd1Zqk Un1m20kohW68aMVP0CQeG86+BtaxJP08g6ijnBg7iHLIytHggkaVpzay2gVG1LwBd9bp710m6jt /YwZP0neHglOX3fb375XbLEU3rtYokF9lfIvEAk8gco3I3slIjyxidVrJkoSjUYBfEX2qEQo4cy vq7CEbkMuEgx/M+mZ5qA6cTPrsUS7ith1OMSJm7clCM/po5OEP0VrhZ+ljRIzGYNfWuLRzbyI6R licSAwMPzOMYrLY/tWC/gWu1ciUULs1LxcWtJOqO6vbahFeSnURVhKOqRHfN1jkbPA== X-Received: by 2002:a5d:5f42:0:b0:38d:df15:2770 with SMTP id ffacd0b85a97d-38de432d90fmr568617f8f.0.1739216333904; Mon, 10 Feb 2025 11:38:53 -0800 (PST) X-Google-Smtp-Source: AGHT+IEQ8jI3Pr0PhgIrX5XFi88nBpwoaMc1TLFqexlyUt/iDFsCN9k6Hzlrlovdfu0/+3mYproMTQ== X-Received: by 2002:a5d:5f42:0:b0:38d:df15:2770 with SMTP id ffacd0b85a97d-38de432d90fmr568579f8f.0.1739216333460; Mon, 10 Feb 2025 11:38:53 -0800 (PST) Received: from localhost (p200300cbc734b80012c465cd348aaee6.dip0.t-ipconnect.de. [2003:cb:c734:b800:12c4:65cd:348a:aee6]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-4390db11200sm187831345e9.38.2025.02.10.11.38.50 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 10 Feb 2025 11:38:52 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, damon@lists.linux.dev, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Masami Hiramatsu , Oleg Nesterov , Peter Zijlstra , SeongJae Park , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v2 13/17] mm/page_idle: handle device-exclusive entries correctly in page_idle_clear_pte_refs_one() Date: Mon, 10 Feb 2025 20:37:55 +0100 Message-ID: <20250210193801.781278-14-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250210193801.781278-1-david@redhat.com> References: <20250210193801.781278-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: Twe-AR8wHI6pEjPbTI6f1WrzteQlg5tCkqYxMqiSW6s_1739216334 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Queue-Id: C6852120014 X-Stat-Signature: zes1n6auf7prwf1nu4pny3dbife5dh8i X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1739216338-697116 X-HE-Meta: U2FsdGVkX18RYD4XQh7wBVgd4NIk6kYpOuG8Pt15kCE3bItG+763bizcbINOMhdW3AR09GEd+TldkJWWmy3YXIfKaw2j3qjzJYr2Udci4bOXIE7Qeyvdz5xDnkzZqy7VmYOdAkvMh2Rel8VopuPjks9J4IvYaM/NAV/gUcuLAwa/vrwlciBK47eTdkrnQEdklDkJUz3WJEdsiTGokf4G462kpdg1Yj0gKkhwSB0iRMYZ4baLrvD4ur7/qbivCTN71nAKufNp3TiHJVkXPFynF+ak7zKoUHOYtyBayKB6/9locdePbMU/FMUX1v+jpvXlG2tOSw9S6phF5ZLX/d8ItnDfQR+rbubOTdceIRDfwNIObLYVkxKb/FVoONqPWM39KsnkD2uYTCL5lGpYDAqYvrk+GFepfEDOK2rN9qWzalnN6zNaM8s1a2uxe02DBhJLvTlC2qzSc2qGCDHQMuOa9rsYwLVv5NLOuDrWUdGo2ausCCrXq3TL6lJW9XC0gHivlpGgKnUskJW+TawhI00se9BUmOpnaXUHiC9qDXysrfSJ+fXqH2Y8gLDcwT9lgkdz4TU3qLlMLHC2gXfIdIpQgUVfuhRPriJCsfgF0r4kvFk2JG7hIORsxtJYITFeJhKhRLP4P8/gOkxLEfczsgIElRGloajV4erD2S7Api1ekkDZi34/H7UA9C89toEPP0bXktI8J0D0u/1MfAFv1exn1Wd3DLZZX1LmFzA9gwpgbQB//on2Crw7QkRREzpVfbHoIQwiVBHCQl8IaYiALBcx+Nz+UDtuHuw52apsF+OZhsqX721SgV+gnLuJ3MNISIm4W0rnAtErP+z0Oa3rmxOd9zj5FNFL8AVaQmU1yk+05HHQE4XRhscMdJEJXHLzhnCHeAgUSNrVMQWGnV2cvj6P4l3hkG9heBgMF/PnL9+ihOCtbRCcSO1Uq1mYmxJgG9WKI6wEfVI6sFnz9qoQaL2 VF1PQJy7 QMV8KMM6mFTl+8k7yztUjqBHltFQmXCAo0VbhnebLufkewKYrsVF7lQ602RqYxigZjpEO/LX9BP2KY5GEpbnHU+0G4jHVLxtu2dTCLM9MiDkbe22MiExY8IjD8TJbJblqoGDTk5dVuif1doxyTQqxkbHEGfZjEcdFRVCOh7/mjAAtxzFR8O1Zje2O3cQTULsfu5bdfU3enUkIBZmySOEIseMMN0Q8bER615vdDG6+/Dd7JI7mV7g3WYC1xgtikrXcxJcyKozcVtF4oMJ748oQ7h1igL9eqjxMfKOplg1UMI/F6aVLsyAvhy6LIvAe3nFSeO2ZCrvGpSmLXwKGKSfo1Ld1qeZ040Xw92It1xuqTEROK78cmcz8nU7jmrOiJeoDex1UQgyr/ATDVjFvAm4YmtOeyvAjW+B+EBqNPc2ZdHDhF88ygxxkHS2m9dkz9Ck2dqfQQ2P3j91mnhNJSxizcOWVkvRbwIO8ERZasV82Sex0WL0HNMxc2Zk90/sg1mDlJgxdg3DhyO42AVm5n6Rr33idUg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000008, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Ever since commit b756a3b5e7ea ("mm: device exclusive memory access") we can return with a device-exclusive entry from page_vma_mapped_walk(). page_idle_clear_pte_refs_one() is not prepared for that, so let's teach it what to do with these PFN swap PTEs. Note that device-private entries are so far not applicable on that path, as page_idle_get_folio() filters out non-lru folios. Should we just skip PFN swap PTEs completely? Possible, but it seems straight forward to just handle them correctly. Note that we could currently only run into this case with device-exclusive entries on THPs. We still adjust the mapcount on conversion to device-exclusive; this makes the rmap walk abort early for small folios, because we'll always have !folio_mapped() with a single device-exclusive entry. We'll adjust the mapcount logic once all page_vma_mapped_walk() users can properly handle device-exclusive entries. Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Signed-off-by: David Hildenbrand Reviewed-by: SeongJae Park --- mm/page_idle.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/mm/page_idle.c b/mm/page_idle.c index 947c7c7a37289..408aaf29a3ea6 100644 --- a/mm/page_idle.c +++ b/mm/page_idle.c @@ -62,9 +62,14 @@ static bool page_idle_clear_pte_refs_one(struct folio *folio, /* * For PTE-mapped THP, one sub page is referenced, * the whole THP is referenced. + * + * PFN swap PTEs, such as device-exclusive ones, that + * actually map pages are "old" from a CPU perspective. + * The MMU notifier takes care of any device aspects. */ - if (ptep_clear_young_notify(vma, addr, pvmw.pte)) - referenced = true; + if (likely(pte_present(ptep_get(pvmw.pte)))) + referenced |= ptep_test_and_clear_young(vma, addr, pvmw.pte); + referenced |= mmu_notifier_clear_young(vma->vm_mm, addr, addr + PAGE_SIZE); } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { if (pmdp_clear_young_notify(vma, addr, pvmw.pmd)) referenced = true; From patchwork Mon Feb 10 19:37:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13968594 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F1C5C021A1 for ; Mon, 10 Feb 2025 19:39:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F0368280010; Mon, 10 Feb 2025 14:39:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EB21528000B; Mon, 10 Feb 2025 14:39:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CDC3C280010; Mon, 10 Feb 2025 14:39:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AA51D28000B for ; Mon, 10 Feb 2025 14:39:03 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 5EE3C160548 for ; Mon, 10 Feb 2025 19:39:03 +0000 (UTC) X-FDA: 83105048166.12.87CE574 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 2AAC320004 for ; Mon, 10 Feb 2025 19:39:01 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HENOEqYW; spf=pass (imf13.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739216341; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oE9Ja/aC61CHup4qeT96Odiagg40Q7TPEFXRUvHwXRo=; b=BJR2giGfO7ycaa7cUcz00akHacTZN6QRSqzztQsopnR/bdMfORApubCfPwyOC3/FBqpiO0 AUXyqmtegXHRkaPWrsGh0JNFQEAAFVxW0cF8nk4TDWT51kaIBtvxmNueaTSrD568JbuW93 O11BAVlY+heEIvIqwbmp7DpQIHbhwxY= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HENOEqYW; spf=pass (imf13.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739216341; a=rsa-sha256; cv=none; b=4LjYKgzoWlPqx/DhIq573Hjw+z0tlDBZSRWezMNNzCvb3YbhNsoc0TnTW7S1CLvgopHcnJ km/aYF4pgDcwtk7Hd2uquUZyP5MnqGJ77Gsq2tikuJGMfPe4fzmwU404A1foAYFnNmJKXS 1SY6NK8rJPBgpeMHnD5/c0oyN6G67WA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739216340; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oE9Ja/aC61CHup4qeT96Odiagg40Q7TPEFXRUvHwXRo=; b=HENOEqYW23FVkl2qDgOt1R9Yb2YQsPWKLOM7RE3Lz48FdX4WK6IjmMjQ3pPwnpSTyXYDuf w7jBnPIHOaeceoQaNQSfuxT7dsFlYfyy3jzEx0zmDHeAk5kH6M+hYBFq0RurpIBcTjw23V EHg3DsgrXoTX+ZQ2e/tGel8oyacktDM= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-470-6bh0ebsuP2S2vgngL1_Gnw-1; Mon, 10 Feb 2025 14:38:59 -0500 X-MC-Unique: 6bh0ebsuP2S2vgngL1_Gnw-1 X-Mimecast-MFC-AGG-ID: 6bh0ebsuP2S2vgngL1_Gnw Received: by mail-wr1-f72.google.com with SMTP id ffacd0b85a97d-38dc6aad9f8so1835276f8f.1 for ; Mon, 10 Feb 2025 11:38:58 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739216338; x=1739821138; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oE9Ja/aC61CHup4qeT96Odiagg40Q7TPEFXRUvHwXRo=; b=JrDq+vWWKO+Rlas7m8p8aIhPaCrth69uduHca8vUdphtX3waDeEPQEUunQAAunkEnC s9VvdBfc2Y2QMEbJkd1LEPzdRZoY1cJudzO4dU0nozMh67GpYyLD+GGyw4MElf1G12kp 6FCQZbEq6P7P46AkFMFNXORErgeMbYcWnUC67yZTG0CxlcSwMIIB/itnutUE2T+sZtDH Uh6SZOSr6JUDvG0Ke7kUuKE7EtcvR1T7QmoAyTHYAJRzL6amnz7XHAs5fxhaP/tRpKUh /ovVOeV/HQClwNVKuz1Snu3ULiD+a2uSxEIjOHYGJ/eykISs4LDCFE6xa0lOm2JCV3L/ EjQA== X-Forwarded-Encrypted: i=1; AJvYcCUlbqF79eg8HHyZilEeb1X9X1sQK3p7Z5n2rKjktZimSrV3Dn7x8/3hPJ+aALbGK+/4+jDBRWHFKw==@kvack.org X-Gm-Message-State: AOJu0YxD0ridq+21Esqy88hpHKiO3tp8Omop7qS+E0sXurNX2f6q2khc FPAVS5LSGnDVZ0+E4MSzfjMGkO7657KlV1fKAcdKzEzAPPFd8efF1kJKBmKRMFOuIgBx6Rkknaw pyFUjgFEFWcnnzhmHXXHku+PXS+ghjR/LcyYtuz8rqLngsbgy X-Gm-Gg: ASbGncse6IejTFImLYENr2RLidwPr+m0zrKbj9A2PqenvAKSE4/w6rzvJZvQiD0g3oF mkWzoZ1Id63HSgHjuwo3gctEsnqoTFYIB+WQf3Zsgjctjjam2pRzlqNiNrhl3fxp4pKrUrO/LKk 6ww4mbDC3+190CmZ/2qtCT6RuDpzbkRvyDDQP6JrtpQ264nYK9oq1bHmtYHvHh+tbPsqD+zQ6dQ sYJP9tWDBK3QQrG728lvGkaZj4eRx6irL+3UnTlKzVgQXGfenzpx0jEpztvcNmZ7+0YBuAi76Rd HEn/L2LfO4sLUU+CaMV8ZqOPUTy9uXEKK6s3qdBIVVn9HahYJrEbIAI5bq8sUV1WdA== X-Received: by 2002:a05:6000:1887:b0:38b:f4e6:21aa with SMTP id ffacd0b85a97d-38de439b7e5mr512550f8f.5.1739216337897; Mon, 10 Feb 2025 11:38:57 -0800 (PST) X-Google-Smtp-Source: AGHT+IF/uzgNmJSm7r4vj5m1RC+0Hk4d4M8qsuHzJI2p/6gaMOev6b/snbfbww+U3ctbsFbws8/HTA== X-Received: by 2002:a05:6000:1887:b0:38b:f4e6:21aa with SMTP id ffacd0b85a97d-38de439b7e5mr512516f8f.5.1739216337518; Mon, 10 Feb 2025 11:38:57 -0800 (PST) Received: from localhost (p200300cbc734b80012c465cd348aaee6.dip0.t-ipconnect.de. [2003:cb:c734:b800:12c4:65cd:348a:aee6]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38dcc9bd251sm9816921f8f.9.2025.02.10.11.38.54 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 10 Feb 2025 11:38:56 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, damon@lists.linux.dev, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Masami Hiramatsu , Oleg Nesterov , Peter Zijlstra , SeongJae Park , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v2 14/17] mm/damon: handle device-exclusive entries correctly in damon_folio_young_one() Date: Mon, 10 Feb 2025 20:37:56 +0100 Message-ID: <20250210193801.781278-15-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250210193801.781278-1-david@redhat.com> References: <20250210193801.781278-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: c6YDcDGUdajFGxZNzRy0UWfG55N5xSixbDj1bk4rhrc_1739216338 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 2AAC320004 X-Stat-Signature: ceonqk3yysq4ue6wbccptdq9jds5keru X-HE-Tag: 1739216341-855387 X-HE-Meta: U2FsdGVkX18L+ORTJCjPI2uTr4YRZ+0dfnn0tyE5c+g7PhTyaGFKqSidEVPy2OO4RlM9MQo6ytXNktbNRBSpoXFX08ijK1GjzYFhwBd2H7etC0pUeC8/q1LagfV4Wx5PQGb09bd39Mj5QjuXK7SPTO/CSGzI3JCqMW1ffX3KuYEbMQ3kn/iV/oORPhqFMp/T8CCbvENwSv1ajfEbaT9Hun0w2sht8JAuqmQVuMnKBhd2NbqOosNLvxsDE3rWpfRLJH/srCK/M53UtTU2GzrgdNGhG0UZnMrzO7b9v+/ZILUjPizG/6MX40ksNjfCPM2PHPUoXPlynsa5DPwmcju9eph49AMoJkrCuu3z4d6Rcrgxv0/2pe/Nw9XlZqPGTRZFhpH/OixUJMz3UaTykYq7s2KcMBO1ZZ+5TPiO5Q6M0sKPftrbuBNsuAoWNOyjvNYxicpdivdVOMP4WOlirDQvl4+QYuIBZLF0IjYdqst85zrWwHnELKfEW5kAKc/eEONKg47Zgf3rOdtswFQwnrlLgKiLZRXlP4IjLof0XbaDKFGt4xB4HeR7aWaQhCVWvYYftlZi85zL+UC9KlvQb5fqQ0R33UBDtJqHuJs8swsnWUvZMYC359Nx+GWRmn++O2sQhinx73FRpSBU0MXYT3HxlrziGRQlI7Jie71iHqnj3Q9CaKqn0CiQ7MB8Mr3Lqa0UmQk01i2qy5JffbBouL+5qHK4Jp5VsaSQyLYU9RaOkAYhAAsOZMoDWso9mvsWiFgNfcYbYrV9cFGp4+dZGmFV5ZskQ/O3tM8zgis3/352wWvqhXvWacTedm6phjf1e7w1r0Tojm/WxVjGiZ75CGrnQR1SiIUKN+l5GB7BzsAxZH2J1KBnvYIv5DZKlJ8AUMO1mF4+5Ut/iuPpIXBvE3oigwRR9J7F37AK6lTe0vIZPg38AtHaqaeqnSevQPZoyB9mq8bPh+SfMTUmm1Q+x6A nO04OLmR +v87jUIGtOJkjBg4a5u4FFxm0xC8eGapEPeL2TcUYf5v476LHu03eE8I1EVCv57uMoP811QF53LIdf0EFXlzUFegDO2S95ZuLKrQR+LGbSaNUvwXEMfrvVyF22jyXeZ2D8p9sSi1zcfZBb0c4LHGX5WoWctgwGSrIQ1HX5WhgMc4jB4i89aDAl20Oy85OQOV7pNmVxf0MWQRmlxOiDQHBlkO7J/DOT/1lfK7TL4ouQlH73VDlXLkNPGiTWqgV0LY9hbMMEoW6MJuC9nmTY/JM26GqgFigbBfvzZIaj7APCxZSVTyU8pG0xphYUbo5CLsWxn0qJTE/lEkl5bbOtzk2e0PP5VSxegBfcn746e9HGfSe6zSGHmwJpq5JyryztuO+8jKOyRHd0Wp6+l6NlWCd7neDVyUsQWQOkRkrX8rFf5dbjO/Y1YSguicSJjFCTy5ULZzGPzIXm54zk2IHdgiWc8FThu63PY8pOyIkGOwaT1fD2ub24NYDbOWzPlsoAQHu1IzU4574VDsVTMLq0e3lwX4cWg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.002423, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Ever since commit b756a3b5e7ea ("mm: device exclusive memory access") we can return with a device-exclusive entry from page_vma_mapped_walk(). damon_folio_young_one() is not prepared for that, so teach it about these PFN swap PTEs. Note that device-private entries are so far not applicable on that path, as we expect ZONE_DEVICE pages so far only in migration code when it comes to the RMAP. The impact is rather small: we'd be calling pte_young() on a non-present PTE, which is not really defined to have semantic. Note that we could currently only run into this case with device-exclusive entries on THPs. We still adjust the mapcount on conversion to device-exclusive; this makes the rmap walk abort early for small folios, because we'll always have !folio_mapped() with a single device-exclusive entry. We'll adjust the mapcount logic once all page_vma_mapped_walk() users can properly handle device-exclusive entries. Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Signed-off-by: David Hildenbrand Reviewed-by: SeongJae Park --- mm/damon/paddr.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c index 0f9ae14f884dd..10d75f9ceeafb 100644 --- a/mm/damon/paddr.c +++ b/mm/damon/paddr.c @@ -92,12 +92,20 @@ static bool damon_folio_young_one(struct folio *folio, { bool *accessed = arg; DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, addr, 0); + pte_t pte; *accessed = false; while (page_vma_mapped_walk(&pvmw)) { addr = pvmw.address; if (pvmw.pte) { - *accessed = pte_young(ptep_get(pvmw.pte)) || + pte = ptep_get(pvmw.pte); + + /* + * PFN swap PTEs, such as device-exclusive ones, that + * actually map pages are "old" from a CPU perspective. + * The MMU notifier takes care of any device aspects. + */ + *accessed = (pte_present(pte) && pte_young(pte)) || !folio_test_idle(folio) || mmu_notifier_test_young(vma->vm_mm, addr); } else { From patchwork Mon Feb 10 19:37:57 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13968597 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52736C02198 for ; Mon, 10 Feb 2025 19:44:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B809E6B007B; Mon, 10 Feb 2025 14:44:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B096C6B0099; Mon, 10 Feb 2025 14:44:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 95CAF6B009A; Mon, 10 Feb 2025 14:44:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 749976B007B for ; Mon, 10 Feb 2025 14:44:08 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3380CAF607 for ; Mon, 10 Feb 2025 19:44:08 +0000 (UTC) X-FDA: 83105060976.02.DE9156A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf14.hostedemail.com (Postfix) with ESMTP id E7906100004 for ; Mon, 10 Feb 2025 19:44:05 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CGyoavkB; spf=pass (imf14.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739216646; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oNpxFEEEyxoH6lwF3K+DQE9+aA/s7jzY02gjXCRiBPg=; b=RmABoBtm9Aa93NT+CjhS8I59K58G7rujbaKWYK/smqWdvvkP3vHZnLZtUEhnGxpefijLGC GQOUhAS5yS1wkKgkzMEgOhoDpWj7BU9uQz4sgN2SMmNQfVPO1LG3rg9pI5gIv5srowzgk8 Z9fxajCDog+ZyRnzuYNlysbvnpTRU1A= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CGyoavkB; spf=pass (imf14.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739216646; a=rsa-sha256; cv=none; b=IwX+9ey5w5uNDkgdMNuK0zZaxnVGe/fzFrIR+Tow/xCrpeeQjxgP9Mk6d35Yi134Oeosf/ PHcLgDcKKYy+uDAhkes23/wA5JocA1GJrYCYEfbktKtKXELSEk5heIHmbSmsMsmLz3iXoI hgrqhnu1pqkUJopGoVOeJMDxV8hGSmU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739216645; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oNpxFEEEyxoH6lwF3K+DQE9+aA/s7jzY02gjXCRiBPg=; b=CGyoavkBVyC50MF8rOsLJc8xDqmIhI2/+zL/fr08U4M+Sjz3q6SsjaxLEerEz7DhCM6MhD w306AYtVj9oawzhhCfrjgGhhPoGnXn0kRLPvCohggr/Nz/ahttmsnHu/4OjEut5WicZCdR O+1CrdfGLKCgaNk6/mwWqtccMIqzRpM= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-588-wW5wHEQ8MzavGke3laYYLg-1; Mon, 10 Feb 2025 14:44:04 -0500 X-MC-Unique: wW5wHEQ8MzavGke3laYYLg-1 X-Mimecast-MFC-AGG-ID: wW5wHEQ8MzavGke3laYYLg Received: by mail-wr1-f72.google.com with SMTP id ffacd0b85a97d-38dd692b6d9so895041f8f.3 for ; Mon, 10 Feb 2025 11:44:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739216342; x=1739821142; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oNpxFEEEyxoH6lwF3K+DQE9+aA/s7jzY02gjXCRiBPg=; b=oNNz2nCJL1Ax9lt83PFtdT5MSwl/SThjoKIa+tnLtgF97XV0rXaSv0Lg10KKaTRYmK wwI1Ew52pjyYwqvxjCRBfQ03rPcZtAENY3sB/uEpxH5yXa8nKvbQgH9KM0qOnk3L2ZNg uSjqMR2r++LogHMnm18A/HdWMso6XnMO45fk777uhmSSb+sk8JCrMaNBMtJrlYSvdvai aLAwckJTMGAamJw2zeQlDWUlVPbUzf36Lp+cbahtkc5bLYC5EUHllWWSuFW3fuZtIyVF 0jwb9FGu0HO/Cg+2i888OThoqL9F3mrIdTtqwj709qE8N1GY25aCuVPZC6d6AQvt3dZv 5zsQ== X-Forwarded-Encrypted: i=1; AJvYcCU6rGhpfXfHXww3JcPU37wByBg0lSlPiBGO3zYtAWYFMXE1eC8lBNfydFLuvkPk5EgPme+FCdkuvg==@kvack.org X-Gm-Message-State: AOJu0YzCIS0bA7h3mlVOG1GSYAtEuuDu9woGPLABxjqLsYnJ9OMzcnC9 +WalZTxBdh8HV2A05232d7ZyTZN/Od8476/wP3rxUajHIEA149rGEHFMNd9ntbothHCwD4d/UPg d5u27geHMhYeiT7ViYxAZkfAasq73BbqgJML3k+r9m+Lie+BC X-Gm-Gg: ASbGncvhkcdxTsp1YN6VFqQeP3ejlMRXJfg/aB2coEO5xcwRgd0NFe5sg/oKeZV8OwS Ln72RTR1mXv/a/jEoNQpEMYS24/DrckFuB/DEkhOmyWZhbo0aG5O/aSfe5lUEwYwHId7oNUfeod r0DDCDpVhaCP6INLfBGEj09M7Be24YbDg3nBjKtTEWmpWh+lZgnU8CySMjJlP+tl8zrqUtQSjFp 5ziP9tDwfrFQktKKmx0DQlLJy+8MKrxtJVGlpj8O+J8rCcJ8EPJcwIxMhVTEA3Ln0rpd3b6zM5i mLw6V/9GazGKCBNADp1XoBZcJC2coDxydmojV3kNpks7U+TRbAmPt0b2K60vemuObQ== X-Received: by 2002:a05:6000:1813:b0:38a:418e:21c7 with SMTP id ffacd0b85a97d-38dc935246fmr8277035f8f.53.1739216342067; Mon, 10 Feb 2025 11:39:02 -0800 (PST) X-Google-Smtp-Source: AGHT+IHTiiTPkwAyfuLx0qL+LO6PapdaXuVNjUwBeGg/Z0ah/0RffyIckymQJ3LkKa1NUzxInLX6yQ== X-Received: by 2002:a05:6000:1813:b0:38a:418e:21c7 with SMTP id ffacd0b85a97d-38dc935246fmr8276996f8f.53.1739216341643; Mon, 10 Feb 2025 11:39:01 -0800 (PST) Received: from localhost (p200300cbc734b80012c465cd348aaee6.dip0.t-ipconnect.de. [2003:cb:c734:b800:12c4:65cd:348a:aee6]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38dc4d00645sm11916376f8f.66.2025.02.10.11.38.58 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 10 Feb 2025 11:39:00 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, damon@lists.linux.dev, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Masami Hiramatsu , Oleg Nesterov , Peter Zijlstra , SeongJae Park , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v2 15/17] mm/damon: handle device-exclusive entries correctly in damon_folio_mkold_one() Date: Mon, 10 Feb 2025 20:37:57 +0100 Message-ID: <20250210193801.781278-16-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250210193801.781278-1-david@redhat.com> References: <20250210193801.781278-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 811Ez73u3823iX3-BWneyz3TLBxMTC-53NYH8Km-_bk_1739216643 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Queue-Id: E7906100004 X-Stat-Signature: 5qs9rhb7fskj5y51mtjq3oduwpiupgsq X-Rspamd-Server: rspam03 X-HE-Tag: 1739216645-606623 X-HE-Meta: U2FsdGVkX19inY7SGYyFlE+7kpIhAz/qwg0dsLZktxvUMaQUOaEgoWRmDp0TzExcZdX3JGPuqTJeQ+T+CJKzugSVST8Hva48+7/eKmCQlE9SFH40EO72wJ+QobzkJA/VcV+utdHMZXGMo4bX9s5CA4ym4/dn+ZpW3898BqXCa8Qmd1f1eXRDb9NJeSGIIpEor9IDA7KYS1iODfGUwk5k4msT1aXsf+avxr1AmbBsUR8AdU5Wr993e1j77R4me12Ms3Ki4QqFT1IqhPw8R0VcFMHQ0gAnQTpt0GWh/7pXtPvzUqaEMJ72UNttFYp+iEYxUF7lFyP44ckpiVNSoK84XAdSbtQbO0ri2953hjycaUqnIpZDN715+ocjQzoC0jesdA7VWQqm7j1+JUYNShr1ZmQDIYta2pnQ3dhvp+P7H3WaamobIW1Izs4iGkWc33Y9u3bv4nAb647/n8U1td2r+mWBSpGMqQUefLO/yTwckiu4dlgQE6l3VRUSL49/6BzgT4haEYZTgATIbQwSvn9aqpkSU1GZRXiSU8GUakb3i2gaugxrFltIADU16R4rMlqJEEk+I0qgdAaFTbjf0PvjPfS6Di9yHzNaBQ1XSujiHFX6S+e7gEC8rRo5qkAFHCqlR9mzaPPcUEiWQU2Z7lY7/mQ6IbQTAmgKSWEwK9e0Y70FX/6rsGAMfbxgzvCN2NvVPJV8leo9Ybd9RzCTMR6TWCqp+v4iu/pwK0DtdteqMaCXt1qUNnkfy25tpRV5AcYH8UidaLu4MafjT0dbvJl/WlZBkUUamWYrh8+QOL7oukb3k8pd8A+S1S9znwI/PkAdwUYCw4g0MlP3xOGCuUe2tqny4PNMwTN+FZF3ZTHb/4s6v73PR4zAZZWz1KzBycqoSFEqbGjJeeptck0vUs2ZXZ5xG+BK/MRHeeFTCycz7InJpbtWl5BQRbjCRXsVmM0NYxge9GLX2N0tJhtknLd UdL/dLqx JcuU9ILe/uM7n5C3vd8jJGanl6W8CGR5DHD1aotPyu5hjvgDmJBS6v0M2j5XSj91efTLEOf28N50kT40a6VYZaO1xaaor+D4Vr/TKFwS5Zwsdyma0GJWlUO/OvhgPV+kbDUYHnBluhYksFk6znO6tMh24Do4Qz8MINg0W34USbycpbp/jBzSs447yONbFYpAQ55TGiaBmIOws82LfVjrWVNsNNKdhQpSPtcmIJRpFvUsL83BM81YAnd5pHyYW5zx+zmaGZahL2yenlrHk2G+lWm8NMNF8X40WHA7QpzGr7HREXKp3p1LhLW1N4KGlMJgfpWHFA8pFsA+CCW3Vmh5i+n1X/o3mFXieQTJzmTOkgoCArN3ozMtdjoSJH8MEtrm6BGPeJIL55FZhAdDetM6S9UnEexWhZBpOUoXKGw+2nUPEJp+wxFEPorfqaTgvYxQORriYiTbTJntJiDHi9hHyIiHqacYUZAFh3lqxKRFETgna9nV0SAomInxvjXSNDKkrQhLY3+JQuFXilNrCAumXClQ7dp3P93Zg8HZkXxb+F+Fu25gG1Hey41NzGokK2NWKetV2 X-Bogosity: Ham, tests=bogofilter, spamicity=0.001960, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Ever since commit b756a3b5e7ea ("mm: device exclusive memory access") we can return with a device-exclusive entry from page_vma_mapped_walk(). damon_folio_mkold_one() is not prepared for that and calls damon_ptep_mkold() with PFN swap PTEs. Teach damon_ptep_mkold() to deal with these PFN swap PTEs. Note that device-private entries are so far not applicable on that path, as damon_get_folio() filters out non-lru folios. Should we just skip PFN swap PTEs completely? Possible, but it seems straight forward to just handle it correctly. Note that we could currently only run into this case with device-exclusive entries on THPs. We still adjust the mapcount on conversion to device-exclusive; this makes the rmap walk abort early for small folios, because we'll always have !folio_mapped() with a single device-exclusive entry. We'll adjust the mapcount logic once all page_vma_mapped_walk() users can properly handle device-exclusive entries. Signed-off-by: David Hildenbrand Reviewed-by: SeongJae Park --- mm/damon/ops-common.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c index d25d99cb5f2bb..86a50e8fbc806 100644 --- a/mm/damon/ops-common.c +++ b/mm/damon/ops-common.c @@ -9,6 +9,8 @@ #include #include #include +#include +#include #include "ops-common.h" @@ -39,12 +41,29 @@ struct folio *damon_get_folio(unsigned long pfn) void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addr) { - struct folio *folio = damon_get_folio(pte_pfn(ptep_get(pte))); + pte_t pteval = ptep_get(pte); + struct folio *folio; + bool young = false; + unsigned long pfn; + + if (likely(pte_present(pteval))) + pfn = pte_pfn(pteval); + else + pfn = swp_offset_pfn(pte_to_swp_entry(pteval)); + folio = damon_get_folio(pfn); if (!folio) return; - if (ptep_clear_young_notify(vma, addr, pte)) + /* + * PFN swap PTEs, such as device-exclusive ones, that actually map pages + * are "old" from a CPU perspective. The MMU notifier takes care of any + * device aspects. + */ + if (likely(pte_present(pteval))) + young |= ptep_test_and_clear_young(vma, addr, pte); + young |= mmu_notifier_clear_young(vma->vm_mm, addr, addr + PAGE_SIZE); + if (young) folio_set_young(folio); folio_set_idle(folio); From patchwork Mon Feb 10 19:37:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13968595 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0E48C0219E for ; Mon, 10 Feb 2025 19:39:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 40315280011; Mon, 10 Feb 2025 14:39:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3B2B828000B; Mon, 10 Feb 2025 14:39:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 22D13280011; Mon, 10 Feb 2025 14:39:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 02EF228000B for ; Mon, 10 Feb 2025 14:39:11 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id A80B0120557 for ; Mon, 10 Feb 2025 19:39:11 +0000 (UTC) X-FDA: 83105048502.25.F2A2555 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 6272840006 for ; Mon, 10 Feb 2025 19:39:09 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QK+hDwHu; spf=pass (imf07.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739216349; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=e5kPSWbMiVGXXHUJW6xa869lKm+zAgfCoOcPS4ARFgc=; b=CcdV/Y4ZiECbH3ErZGlcbRYG3ni6uflqq3vgmvvRIwhuB/YDrH5B2Sxy0C2ep7IJBtAKst ZjWHB8rkmQENkLkyHnWrSXaN05TAYzTtH7jiQe8tT7sqLpyhWbUHWNZdFZE0o8RaLyTURJ YCEi0um5x3GZJyxWq4nEpG+mr5ysmLU= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QK+hDwHu; spf=pass (imf07.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739216349; a=rsa-sha256; cv=none; b=qm4htvR+IX3fbDpy/x7Vhzm90RUt1x5jX42FJEiTRlWpAWSQD1oFG79ttv3qTW8OaLqze1 jwQFPALOAhU46vZJiVCjGbqdYtE0MK5vitg5UMddcBgnH3QGXAX7o03/K1RUx8quzql3ZU OQhBaGqDE89B9lUEmwBZKN1/9VBga/8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739216348; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=e5kPSWbMiVGXXHUJW6xa869lKm+zAgfCoOcPS4ARFgc=; b=QK+hDwHufu5+xDOo8no0EHLZEQGh5hKXHLjZKTQmUzmmYZC6kQn4zyQVnSbOqYUt80HSjf 6UVCqFyHyAgyyHOxfPUkDzqecgota90VdoabH0v21eJ6WTtcgqXGaYMKSOZ7T6JBFxaaQw MCPwAbpmVModELR3naTef5e0NxKyiZg= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-600-smxyEOdYOCWvnmeUguMBEw-1; Mon, 10 Feb 2025 14:39:07 -0500 X-MC-Unique: smxyEOdYOCWvnmeUguMBEw-1 X-Mimecast-MFC-AGG-ID: smxyEOdYOCWvnmeUguMBEw Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-43933a8ff58so13555135e9.2 for ; Mon, 10 Feb 2025 11:39:06 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739216346; x=1739821146; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=e5kPSWbMiVGXXHUJW6xa869lKm+zAgfCoOcPS4ARFgc=; b=l2tjLK8AUGwdIeslJgaphLfeyamL4Si+A0cZQ6fSqG+Qx+nw05+mtw02lIX1TEzU7I qB4pdnp51DP6L44oHuXluP7tcG8tX75wkN+LEA+GovQHOeP5hcRfNznqLn3NjjFKb1Mk NhrAkAM86uTNHGFW7hNjHlBQa0oerc0aNWVUEQgTFfGepvYAeEXSyCXyZZlaJZdcoWIQ 9/cjIXprecqZXB3QXG/lWanzcK6cj5jYXYsuQeRy5mXlGUK3TP6t8kfPPMq6cMX4qI0k seECyB8gWz7Xbc8n6Vg3W3vgZGTdZYdc4WKQ6ScCqShijJ9u3mKKqxKbtcD68NFmeORD YXxg== X-Forwarded-Encrypted: i=1; AJvYcCW6RY3Db8RfhA8N6Ln5+AFGyW7XqeQhq0iLT84oMrYosEImPB0jSYr2wrAT4MvOQqtdYp61dZau0Q==@kvack.org X-Gm-Message-State: AOJu0YxkfO3Z7les/IngF+8wBOMY7cWm/Me6+CnehVc5fhyAYv2R54hy Y3croEU3rtDYLhpxxYMwLHrWifw06kb8HpY6M0TdgKyX2VJcfHf+Ql7b4pQE0tO+JQfnNMwkY83 oU5847C1fPSLslHPmiQlDlEmqhGE0lUGRwESnPdf8Ka4dn0jE X-Gm-Gg: ASbGncvo+Mo1aKOuA6+GGLAI6XZMi0UXJ8qMvjWmzgpJcISYk+lDVqnZSBF3Kpvfx/X 9vjngRM69tG8RlhH+a9rbyG8TeZIKRr1csCgu8gsUUVw2aEJu2ScPqPfTZWOdyITILgdbEO+aeL QO8b3fIct6glPJXuDrfhd2c4m/WTSiv6dEmvMa+mcuZISaQWO85uR5DgBxt4ZYTQfsf3oMIN96p +M2oUjPTU81CSJtQO9C6UbxUrpzXevZZpHICCKq2vZZqw+ZFo3037BaUo6gMgNBuJKEENf34zof xJ8DYawMeD+RdeBKDJLmn9ywjwoAoJ4504LL/RdZSFWx3Nq/YPojAfEBFoc199+nZg== X-Received: by 2002:a05:600c:1913:b0:434:faa9:5266 with SMTP id 5b1f17b1804b1-43924991f73mr122648835e9.13.1739216345778; Mon, 10 Feb 2025 11:39:05 -0800 (PST) X-Google-Smtp-Source: AGHT+IE0N2Q/3/pgsBEDUurQcQIsnMqE2C8UdYDG2bl1ZHVr+k7cZL7qtDol6R4B+8CggSnILo8JzA== X-Received: by 2002:a05:600c:1913:b0:434:faa9:5266 with SMTP id 5b1f17b1804b1-43924991f73mr122648595e9.13.1739216345384; Mon, 10 Feb 2025 11:39:05 -0800 (PST) Received: from localhost (p200300cbc734b80012c465cd348aaee6.dip0.t-ipconnect.de. [2003:cb:c734:b800:12c4:65cd:348a:aee6]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38dd9c48173sm5308677f8f.37.2025.02.10.11.39.02 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 10 Feb 2025 11:39:04 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, damon@lists.linux.dev, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Masami Hiramatsu , Oleg Nesterov , Peter Zijlstra , SeongJae Park , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v2 16/17] mm/rmap: keep mapcount untouched for device-exclusive entries Date: Mon, 10 Feb 2025 20:37:58 +0100 Message-ID: <20250210193801.781278-17-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250210193801.781278-1-david@redhat.com> References: <20250210193801.781278-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 9g1XwR8W-gkal9wuh8oMV_UQ5-TckhwKZqMnxGQSg-U_1739216346 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Queue-Id: 6272840006 X-Stat-Signature: 35n7bftz816zhr4onwyyapdpezsnp9gf X-Rspamd-Server: rspam03 X-HE-Tag: 1739216349-250 X-HE-Meta: U2FsdGVkX19IY9TG6FGQ2Dnz/nMccJsM6//1+WT1B+1m++mvkjoNd0SgbMY14V4JGyviGj3XUddLpEoL1SYjA6iuQ6IuZZkthrAIm+90KqUiOuEsARkh+5tq4LG0QL69AFltUV4i5N7o28l26ZfP+Nq+uuxk4PiuyufNW5NPbK/MbI6rQLCd+nGGylyfkiXqzieMyqtqNlT+3azU/p6cnjBh4OH9dZ7RIglDj7fVGFHcJ6r8qE+qOMk/gV/0IqGxJEulguaivR+yHqBaz6WJXZV1T6E1pS5PyjPvAdvxnkJkIdllBy94gwk8kA4Ff+OrO6aeSccva0cvtX4w+wp82v8p/5WqLD7EN3ocg5WDPyEfKrkFIWoh+JAiraCn6riuJuu3eDzrvOp6l/lJ4n8pxirZ9rV4Vqxpj0eZfB5aRIfD8FHfIqugLAHXZnMWmWv6her5R5+mzSHazVw8lgrOEcxUDCkUE5fLo7ofrBr3xhxBCd4mSG+JCDdPEvM68209zrk/ekgbHaBbIyr1TltPzZ1uHkSF0fdn9s8xvVfTNXTX3Wr2FnKdbWpd5su8dHmY7HoUmAR7zmYqdOPDxAd5MoKtEdnVk3x3jw22DqwpYRbZ3inYUyjGNHTRd+z1yuTkhViK8DGtqfovF1z5EsbcRye2PlpFbYkCmVrknGu1HUiq5TXVaYdvR0yag4TVuRYCkPg+WxPDN/btN8M0OYzoPi3A1f8BEjVe2adl2DMWY6JewgmDc0OUpME7ccxjJdf+pVgLWEC8y+rER0X/o5FhGOv5X1RYbi1rHU+zBGwn+6atq0yc5DQhsIvWDhGR0/c8UXRwjrRncLT0hWeS033To9xBtGXnMQBYVP9VK67CWL4uCTOeU3YJf4hSlSYslDHY/6QslJn9A8tMto4zRvD8IMDNmiET+dU0TjYZgWiDPEN5igoNzq5+QsFa7qjhXm2l1G0Qp7YyQXKQNp4yQmP t2Y7BjGa JmAcGJs7w+08MmPvqcjnahT7o1EjqyjuC8/RWorqybapWOCWtNoZABp0+9aSDpCD2elWuH49sb99YyOceJ8sjYN+PCTSuuyb0pBXcfc8vJSCfmpStZUqrMzQ7vdaKqirEIMODsDf/QoDrfz9nBjwTX7alx0Tl2STwbC9qd5e+mNZ/RUMCvPqNifmJWKelERoORGZg7um6yf9yNdYOlJonecRdHK/b26YMtOurrV6sh760K14ryLr0W7KPm8u6K7Z6BZ66sZFr3X+4Y+gLTnyhOxM4TsOwLaZKQSKSgkYPF4jNfjb5FD63WIntfZL0LwwBBJsdLheBzor0uMsAS8HTtiolPtebaBF12C3ROSGGOj+U+QyF85OHMuuOqMhZjq/hzyPuRy/bSnujTIhCJJCvwg83O2CzWkAqzGhVZC5Gb+etEA1zODWMJ2I3KGAhR1QmXwVj624AmnhUFVI2L867pp6VWKwrYMHrWHPp X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Now that conversion to device-exclusive does no longer perform an rmap walk and all page_vma_mapped_walk() users were taught to properly handle device-exclusive entries, let's treat device-exclusive entries just as if they would be present, similar to how we handle device-private entries already. This fixes swapout/migration/split/hwpoison of folios with device-exclusive entries. We only had to take care of page_vma_mapped_walk() users, because these traditionally assume pte_present(). Other page table walkers already have to handle !pte_present(), and some of them might simply skip them (e.g., MADV_PAGEOUT) if they are not specialized on them. This change doesn't modify the latter. Note that while folios with device-exclusive PTEs can now get migrated, khugepaged will not collapse a THP if there is device-exclusive PTE. Doing so might also not be desired if the device frequently performs atomics to the same page. Similarly, KSM will never merge order-0 folios that are device-exclusive. Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Signed-off-by: David Hildenbrand --- mm/memory.c | 17 +---------------- mm/rmap.c | 7 ------- 2 files changed, 1 insertion(+), 23 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index ba33ba3b7ea17..e9f54065b117f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -741,20 +741,6 @@ static void restore_exclusive_pte(struct vm_area_struct *vma, VM_BUG_ON_FOLIO(pte_write(pte) && (!folio_test_anon(folio) && PageAnonExclusive(page)), folio); - - /* - * No need to take a page reference as one was already - * created when the swap entry was made. - */ - if (folio_test_anon(folio)) - folio_add_anon_rmap_pte(folio, page, vma, address, RMAP_NONE); - else - /* - * Currently device exclusive access only supports anonymous - * memory so the entry shouldn't point to a filebacked page. - */ - WARN_ON_ONCE(1); - set_pte_at(vma->vm_mm, address, ptep, pte); /* @@ -1626,8 +1612,7 @@ static inline int zap_nonpresent_ptes(struct mmu_gather *tlb, */ WARN_ON_ONCE(!vma_is_anonymous(vma)); rss[mm_counter(folio)]--; - if (is_device_private_entry(entry)) - folio_remove_rmap_pte(folio, page, vma); + folio_remove_rmap_pte(folio, page, vma); folio_put(folio); } else if (!non_swap_entry(entry)) { /* Genuine swap entries, hence a private anon pages */ diff --git a/mm/rmap.c b/mm/rmap.c index 7b737f0f68fb5..e2a543f639ce3 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2511,13 +2511,6 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, /* The pte is writable, uffd-wp does not apply. */ set_pte_at(mm, addr, fw.ptep, swp_pte); - /* - * TODO: The device-exclusive PFN swap PTE holds a folio reference but - * does not count as a mapping (mapcount), which is wrong and must be - * fixed, otherwise RMAP walks don't behave as expected. - */ - folio_remove_rmap_pte(folio, page, vma); - folio_walk_end(&fw, vma); mmu_notifier_invalidate_range_end(&range); *foliop = folio; From patchwork Mon Feb 10 19:37:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13968596 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3A8DC0219E for ; Mon, 10 Feb 2025 19:39:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 787F4280012; Mon, 10 Feb 2025 14:39:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 70EC528000B; Mon, 10 Feb 2025 14:39:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 56170280012; Mon, 10 Feb 2025 14:39:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 352E128000B for ; Mon, 10 Feb 2025 14:39:18 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E45F1C022E for ; Mon, 10 Feb 2025 19:39:17 +0000 (UTC) X-FDA: 83105048754.19.84C7BA8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id AF18F8000C for ; Mon, 10 Feb 2025 19:39:15 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=iMPn+tH9; spf=pass (imf30.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739216355; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gxIo4/a9ZeK4nnisZvQU6AU2gxtpzJDN86Rrv62j7Tk=; b=KCoFeVI3vXr0N3KsztcsF6czndgje1NhX9h9RbPmVdmNy4gADPxMrUQys6+Y/w5dlWcpuJ Yrn47W4kk6Kx//KzwLQUtmDdeEPDugXgIdI6a7+70yFYhMJrd//YSCV9MtFPSoBTf7Au72 uVHHgTMJsmc5kngmvhXQimJ0NoPlvW4= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=iMPn+tH9; spf=pass (imf30.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739216355; a=rsa-sha256; cv=none; b=6HX9VoN9QXLdmoT/C/6w+/eVyWsciQpbAycHvM70wgtr7d3AMZXV5kuhhJccJVxNX1213t mHu3iyT0yzkFxUPqHcP4Rprak6HHNItiwMzvaq/JUilw+vedyQcbQ0FJTqBCQbKZlMLhD3 USXsNRxAdx78NWGy0A6z2LePUs5suLU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739216354; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gxIo4/a9ZeK4nnisZvQU6AU2gxtpzJDN86Rrv62j7Tk=; b=iMPn+tH9G3FTpgC/rPymOg1on9OC9SCUJvtz1LitdmiybXPFY/ai+T5CDMa/sYBG8dFaDu 2L0RBt4HbbZnprfGeKJISs1ykaWSH39JG+BkZpmTGliQZq12T/aYyZlIT/0u2Pr/F9Xzh4 YaPD/ITZuj4fwHdwpDhN/jiHWcDgxXo= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-146-RJ6-qt6TN3KICgCpVX9-Bg-1; Mon, 10 Feb 2025 14:39:10 -0500 X-MC-Unique: RJ6-qt6TN3KICgCpVX9-Bg-1 X-Mimecast-MFC-AGG-ID: RJ6-qt6TN3KICgCpVX9-Bg Received: by mail-wr1-f71.google.com with SMTP id ffacd0b85a97d-38ddba9814bso733058f8f.3 for ; Mon, 10 Feb 2025 11:39:10 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739216349; x=1739821149; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gxIo4/a9ZeK4nnisZvQU6AU2gxtpzJDN86Rrv62j7Tk=; b=XTqm7L95fA8/IydO4ngsH2j/HqLKtsz2LANZMuzx6ezDzDPNskX0rJmWH3Yu2/LPor HDR6jktsQpYSSijo3vpvLYzkQWSOLCjXj+SFdVy/R7VM1tCFM5tsL8HoUP09DVk83UFx mdC5b1L8jLxUtgqrKMRx2FoyJ0LASz/iOM+ovbIevvkb7SAlqNDpcqp0WAqxtLcSTmSm RyQ3X5Z4qc/0dkwYxSABnyJpGiHlzVLFEvlYP0O2To20WO5CCb9oCpBJyH5DcVVkmfy3 5WcWBTlavgEycSXnvAL5cILo+nVKmtcvaLoT9XPWysBHDkSiggGOgyal2BJ503MBWu0Y KTEg== X-Forwarded-Encrypted: i=1; AJvYcCWEHs7foBFTcdylvg8GU6IRp5m+OMhQgnFgJghKhv+3z8MlJDrK8xcmQMWXJX9ytZI/EpS7C2MyCg==@kvack.org X-Gm-Message-State: AOJu0Yx0juHz/420t93cm/Md8aGUw1YTTbMDW5QHZgDd/5TkuWOwA8UA f7u69LyhQF2dX4Bzjvq8su1Vx1IrPZL9zGeIcaemG+p4drgrd5ZSzTpiNUwdPKo3wazCQHKYzys NdfpAqndMwP+ptEAxU0v7bGqCgMnlgb/mCXZXi8piarDmrx0R X-Gm-Gg: ASbGncsZRVY5PdawPTMXaxBOcu7DB2XO+MOZKOPKdye9aHFWbOCzJpf3rlc3uZL0OHE 5eM5wLMkI2YbwJ7Ii+W+2dDf3TvxHt20zZGmF9dbPuEd4ARfeZUwQQ/Ih5g7xxoLPVVoubpmbTH JP6Meyx2tIFj4wQ1u7rWGahwVz6827t5ZMKwMh0gml8dgBKvHKu7h566GiFLbyc3K2jdtXOHPgY CLFYkvLg3exLJOgqotutgp2LbfImESsNoGRLnpWdC17MO+kMGesWgqQxHdSJ7h6lk+tHg4SFhFf 2EQUUv7TJdpWUtBQ4Qd2EHxXItJLODsRMmtNOY2FZN8PJvTI9BdUCYfodSfcn3xGXQ== X-Received: by 2002:a05:6000:1448:b0:38d:a879:4778 with SMTP id ffacd0b85a97d-38dc9343f89mr13325598f8f.33.1739216349517; Mon, 10 Feb 2025 11:39:09 -0800 (PST) X-Google-Smtp-Source: AGHT+IFOm3YA+KmUhyhlND1SLmjpOJWvwKHSp1MsMH3YdrdBRb69cVHv165Q1M/eFn8bTRexW9yPlw== X-Received: by 2002:a05:6000:1448:b0:38d:a879:4778 with SMTP id ffacd0b85a97d-38dc9343f89mr13325571f8f.33.1739216349113; Mon, 10 Feb 2025 11:39:09 -0800 (PST) Received: from localhost (p200300cbc734b80012c465cd348aaee6.dip0.t-ipconnect.de. [2003:cb:c734:b800:12c4:65cd:348a:aee6]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38dca0b4237sm10326047f8f.85.2025.02.10.11.39.06 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 10 Feb 2025 11:39:07 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, damon@lists.linux.dev, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Masami Hiramatsu , Oleg Nesterov , Peter Zijlstra , SeongJae Park , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v2 17/17] mm/rmap: avoid -EBUSY from make_device_exclusive() Date: Mon, 10 Feb 2025 20:37:59 +0100 Message-ID: <20250210193801.781278-18-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250210193801.781278-1-david@redhat.com> References: <20250210193801.781278-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: X4rVh1e1Abl2iF_6t-0-jIN9A1u7YB8XMNw0pat18X0_1739216349 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: AF18F8000C X-Stat-Signature: swkt6rro8fwpnd19gfgmdg5hp7gaehiq X-Rspam-User: X-HE-Tag: 1739216355-220490 X-HE-Meta: U2FsdGVkX1+83fNlLF6Og6+edMljqIqW0qfus+O9bx0wXUDE5RzX0WM3pR4ZEr/ysODph6e2W8GAlzK5wEzpOFyFlL7PX/7ff21Zce/hp1XjaFjyN+JqiPp+1N54XBKoHzm0e6dl35QcStj6tyKxvt8uwCiFHsDEdTLX8DloUvUTX54QWz0u+EfFzm0HaJYYzvjolmERDlms+5vs5K34fy5XL/2M5bHP6d2NdV+VSKoUBsoYnEjdyKqpJi2unNsvTRnZUDehlee38bz5vehsll83GOsJLdL6aDNE+WETqJPLCHPSWPSHvytBDi1uAk07TM94POzBPnLi/u3zN72KfSUiefaf9G4z7id2rh86mo9mWA5+q+p1qSJi/t/42kVDmtY60gzuwL8cUwk9O3w/oVa4SjjWPkI+yvxXAWGt13aFKFWyxIn1S1zdPYIvZkqNqQ7b6HG4BpMRKJZpJybn1/BZnTFM7RZUX6Ot04zLwGYdo6sbS4dlEKu6JJSgvjTAk9nB9wK92bfmiu/x25HN4u0JhUj8gEfURAzZSsl5cMj9s6eZE2bPjU5jvFWPIMHF21ywcT/rZKYYOy+H4HmTMMmfXmWBePtVirXH1Pc/yqZ1Z1s5I04SsyUPTwvuX4nEc0FmecHM6y8Onn9hZONGXyZIEMfqLFtABc4SY8i0l3Z1che2Ev8fkIQchPDqfcX1qqL3dKVk8G9wqjW1lnlxB7k2AETqL8xP58hk9o3sWBfm5GTIj8YDOCKF888svnJUnBriyenp72oLtRLIbX/msOu8g9LEfDr/ojZ/22w/ZQeI6pW0hJ6QLBdtgytNobSe64YnyoHGY1vkgdaZyZZx9sodOnJMqv4l+Vlx8O3CjZrfrKXH8wjbrdjORHeTB9cFprZyF4kKnkh98ZisASsBPGbeflfHkFBTXNKbtZxKgouBhZ5NTWOaYBNs9APN6vj7VVKBbFMBoOPmijAU2qr hmKb7hAQ Vyy+pF1rUhjZ9SRpGN4upcrhbm/VOMiFNQhnOZ6ycfiE0eh41zoEFEnfgt/mdM6/VIPAUMHVex+KUPCfw/9Q2zbbS0JyASndJjlU1n0u0W5hFbjzce2NTih7vPsgYsrMyw9I924Eea6ExrOQZRLs9k7u21Zjz6iufXRQd8q2wsovIqqY30GU3EMe+ZyKXOPH3f+aIIgvXhl31rJnso+fQZiU1dcZ3vOejqLAq8IT/TRAtk0A9fuoTQeOBar4Ic+UrABuW6QhPrD35WnrbuO6th2RGkElr2no7njawbxidSJO29BrRqINOWfYmRu5M+r8EPv28tiZhdUYyArfVI+GOq6UVbjWecGPyE25/73ireeuMuuJUigGUl7tHvpMWjHf5B0sOTI7mWzKRcUDFR01bHHrwZFaqkcdmyoKL3pwvCeZFCJtNlsDabDhtoAgSqa9x46Z6Lj4G+sJB5bpPEGKkHVOQFVl6OpBy73pIVYv4wdnRJGWsybJ/7yVSxVkK2bwaj8nnUitlWZpdl2+GMQE1LsQWqw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Failing to obtain the folio lock, for example because the folio is concurrently getting migrated or swapped out, can easily make the callers fail: for example, the hmm selftest can sometimes be observed to fail because of this. Instead of forcing the caller to retry, let's simply retry in this to-be-expected case. Similarly, avoid spurious failures simply because we raced with someone (e.g., swapout) modifying the page table such that our folio_walk fails. Simply unconditionally lock the folio, and retry GUP if our folio_walk fails. Note that the folio_walk repeatedly failing is not something we expect. Note that we might want to avoid grabbing the folio lock at some point; for now, keep that as is and only unconditionally lock the folio. With this change, the hmm selftests don't fail simply because the folio is already locked. While this fixes the selftests in some cases, it's likely not something that deserves a "Fixes:". Signed-off-by: David Hildenbrand --- mm/rmap.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index e2a543f639ce3..0f760b93fc0a2 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2435,6 +2435,7 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, struct page *page; swp_entry_t entry; pte_t swp_pte; + int ret; mmap_assert_locked(mm); addr = PAGE_ALIGN_DOWN(addr); @@ -2448,6 +2449,7 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, * fault will trigger a conversion to an ordinary * (non-device-exclusive) PTE and issue a MMU_NOTIFY_EXCLUSIVE. */ +retry: page = get_user_page_vma_remote(mm, addr, FOLL_GET | FOLL_WRITE | FOLL_SPLIT_PMD, &vma); @@ -2460,9 +2462,10 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, return ERR_PTR(-EOPNOTSUPP); } - if (!folio_trylock(folio)) { + ret = folio_lock_killable(folio); + if (ret) { folio_put(folio); - return ERR_PTR(-EBUSY); + return ERR_PTR(ret); } /* @@ -2488,7 +2491,7 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, mmu_notifier_invalidate_range_end(&range); folio_unlock(folio); folio_put(folio); - return ERR_PTR(-EBUSY); + goto retry; } /* Nuke the page table entry so we get the uptodate dirty bit. */