From patchwork Wed Jan 29 11:53:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13953675 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8285C02195 for ; Wed, 29 Jan 2025 11:54:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 23E90280007; Wed, 29 Jan 2025 06:54:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1EEB0280011; Wed, 29 Jan 2025 06:54:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 067E2280007; Wed, 29 Jan 2025 06:54:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D22DB280011 for ; Wed, 29 Jan 2025 06:54:22 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 6F8DFC0C4C for ; Wed, 29 Jan 2025 11:54:22 +0000 (UTC) X-FDA: 83060331564.29.353AC7E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf03.hostedemail.com (Postfix) with ESMTP id 3611720007 for ; Wed, 29 Jan 2025 11:54:19 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QRxQBC4G; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf03.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738151660; a=rsa-sha256; cv=none; b=2h5EzPVPiQQA+f1Ezt/4XsNdzVyKeoj7FXIGOjkEPEv4ZeEgAesx+ZJS1WV2h9ko+ghrgj pG4+wH1EQaH8QDzBpbBmQX+hIDtbupV/9dDDhA2AynaCGAnJWXZ4Ph6Iqdy6tHO97/Eygt P3a0S9o2Fi+p/ZXcox/g57KPrUYHES0= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QRxQBC4G; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf03.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738151660; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hZQr+sWP4JJuuRk9zHd8B8Ps+AYS2S2IVxxQCmOGAGE=; b=8EM3LZMbNGhzt5XrWK8dIm4YsoGB/frt/Yq+hAO9/YU8mW9JGDtIBuOBM+IF3PPEdkOVXk GkvUFkwGqK9AkyKVuuw5pOLN4wx7Ng4Pqyksb0wJgHzo+wrjUsijlN1fC7ujNXSxwXmR+b yuvkB1LtY6HKANnrDknF3ZEjC2dOg6A= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1738151659; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hZQr+sWP4JJuuRk9zHd8B8Ps+AYS2S2IVxxQCmOGAGE=; b=QRxQBC4Gvv3LeLJDN6XffFnhQq2gUXBzy10P6j5xBouUcKzkpq76oeXt5JEQO6iK7fgO4K 9oxmmUp2pOojiSw7uDlqOQ8JD6Mw2dNvl4FFcF8g7eTnEubx71Q0ebOGtFYOg88N3iFV5C r7QkZD01Sm/gocvkvaONHW5ujvzIc9A= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-103-xW0QirazOAapg1lfP6WyYw-1; Wed, 29 Jan 2025 06:54:18 -0500 X-MC-Unique: xW0QirazOAapg1lfP6WyYw-1 X-Mimecast-MFC-AGG-ID: xW0QirazOAapg1lfP6WyYw Received: by mail-wm1-f69.google.com with SMTP id 5b1f17b1804b1-4361ecebc5bso34369965e9.1 for ; Wed, 29 Jan 2025 03:54:18 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738151657; x=1738756457; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hZQr+sWP4JJuuRk9zHd8B8Ps+AYS2S2IVxxQCmOGAGE=; b=az1MKrPLppf0Xcnut94rFbBpi0Rh1h3d2VDc35ug0A9ywd+427aKyuQ3OV31gq1SDL ijAcOpab3ZwkaD3J/XjIbWAgq6C9l6HfRbl4rSzMgGcmPZ8416oMUNb6LTN8MDVhp5GZ zBqYvKViQkhoVGHvR6664nsoH4v21BobycvJnUlevOC42CoDaKscYqoLb5yMN3fBOj6m IaPnyAbwqC2A8Cbs1aE5f9XROoJTou2J+kOCl5mRexWmGWri7wUyNt7UM3ubNVvgOtgz 2aK+A4dbAhIKWcKngZoO1P/pAUJFhKoLkUkKptU6aHvSuk6vBy/3w1a0lRXDgi+vjH5k cunA== X-Forwarded-Encrypted: i=1; AJvYcCUYCcur2mZ1Chxnf01GW7Xq76AHOK9ZFt4VRxgIBzBj9+6/e5yJEv+3bUtaIUOQPReOpQCG26rOVA==@kvack.org X-Gm-Message-State: AOJu0Yxw38uBS/iqDuPLwYpXxUqRCd6wsOCrpd4/JtwYKyJ22DUPoFwf nwFJ1BQM+jJjnsMtX+U+5QgnXkf7M1CCoFi1lwYmlSvxpHD2Y9xxqKNBXVq5e3ECV7XrNLqeWn/ Jgn5NNyGP8xaSZ7n++E4VL7y8oLQO9oKjiEH2sdRqLKcB+gGH X-Gm-Gg: ASbGncvZbzWsmy60gK5P7t3qm+PlAmLwyhlo5F5m96QSonyMuJsfNGGoQBGN22kzWB4 kOxYMsP8mxl3DVzVc42PkQInoDQNYX3l4X1GFFhqJ5EESBzq13D5tNCpoOtUqDMbwgGta8nWDma Hnw0NjJv9dyhwQCNjfoPBJQAmSZbG3jUUSl0QflYe6u8xXLCxJTAi4R7aSccf60sCCBcxx1x4JD blGp9YAa+SnLM5LM4YQosbVsM+JgHkQHau0kFl62Su6iAfyYaARrtjIDnlK/+HPAU4CX16k6auu zMwKXBhUpsHs5DHGacaEWHV+lXofQFScHSRdLDl+/4vSVuw9n7gI3fxM3ptMkwAzTg== X-Received: by 2002:a05:600c:cce:b0:434:ff9d:a370 with SMTP id 5b1f17b1804b1-438dc3366f8mr24658505e9.0.1738151657110; Wed, 29 Jan 2025 03:54:17 -0800 (PST) X-Google-Smtp-Source: AGHT+IEIkSHUg8u1CZ6d8yOIa8EuB96MLk/b/TKJcDJCOFvAxggFPdf4ivg/UC1RFEcPGInF41YvMw== X-Received: by 2002:a05:600c:cce:b0:434:ff9d:a370 with SMTP id 5b1f17b1804b1-438dc3366f8mr24658175e9.0.1738151656750; Wed, 29 Jan 2025 03:54:16 -0800 (PST) Received: from localhost (p200300cbc7053b0064b867195794bf13.dip0.t-ipconnect.de. [2003:cb:c705:3b00:64b8:6719:5794:bf13]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38c2a1bad92sm16868229f8f.61.2025.01.29.03.54.14 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 03:54:15 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe , stable@vger.kernel.org Subject: [PATCH v1 01/12] mm/gup: reject FOLL_SPLIT_PMD with hugetlb VMAs Date: Wed, 29 Jan 2025 12:53:59 +0100 Message-ID: <20250129115411.2077152-2-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250129115411.2077152-1-david@redhat.com> References: <20250129115411.2077152-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: oqYT4wQ4eMfuEo6lbuQQ0cv6l6ICndDOc5uaC9bE7sM_1738151657 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3611720007 X-Stat-Signature: h1jbghz55rxi37m3o5d38d79m68t3tak X-Rspam-User: X-HE-Tag: 1738151659-473004 X-HE-Meta: U2FsdGVkX19VtHTsMnkStaPjQ+1ndqyy9OPZnoJX6fRNtwWz8XZqmnV+MFHxBr/sdeP/THGnyoaMm1cS/WEiMo/j5S4QKI4vNf3Up5kowy7wlBkny1FVVrEo6s/qJAU1hjNg3v8mlE4OHdVuSDlEMMjWeCI/Y/fu8xJe2HyASP2Wg7Z0AaKZa2glsP1Pr3ZcE1b2vix6f+XlhJ4BTLHTo3+B13/46QJeOebgwbn+Q4tQMBIOq1E0aViW5w4UdhoFz6ztvL+Rf1wgMaqDFn3KQOP5lSycMAjDj7BDSOH6zeZVP3x9zP0nlL87GxiV/kMLP56saWg/9zCKJ3p2Iz9W2POHjFY9aZ92iZ5xrMn5RBSb9kprJgh0jPkAPA4P+KqTKQNr3zbBkTBawguY0nRfL/Qu9uhixWWUY0WENws+LyBuuT1nyGr+nEIvv/EKicGIVHeTJzmnmpkVg3QK1o6RWxA+dUgHyYhiu5QLHtadnq9zk2vLYY2w5iFLudLHGcHl/BoYatvEvbSWdSKb/IDE74Kj0EcTexNSMAclkKyxKjiiVsUVFlUHmD+EmN7Rw5EVC6UrW8N7eiDhfahRe9JL2y5H5EDMd67kw7/lkuzOvV4FGE2O1Wm/hZY01VZaFC+Vgsvj/vFbqAProjNjDZx8Ikkt01GuC2EAl8to54C2c5h9oh28tFPYAAA8Q0BDeZpUK75gFzGgNGAg9wpJpJMx5LrdTTGWi8gHfOgcEgRjJfyUg4WUgs9VSHA7NAUCmOVkdTHSHcTDV02P5LC6qgz4vr4JrcvuTfSR/Ed42WYAG2LT4L/PUbJMyqQzHOL+MB1RfZ2HjQA+BA626L7dqEfJT569h9U+o4atR9dSoKpdFDE0tBQ6RZuD+KicIMmjWtkGwQvvvs/501OOgHd4xDDyNML4vMUuV1yWUmR7s5KoXAesvyOjx+2F+FUdPJBJFLPRpc1sTaQgIpPw5DjNpWs 3wRleS2P PqQCel1eUzie8tOH0osnEDnKMyKsg3apR5E6dDi0dXpzi1SqN+UQXne4Q/65QvZPvbCBg5so2l3rIGNcQsWewYN/qCXsTr6ay2CXLyc5Rm9wjT2HLNqIpob4dTY+HlqgGI/5ubcl1MypRbJQOR5hvG0aGHB81vZGJC/v2rW8cUU8mHXtLZWiN3EWIalXrrmnoTqj5iA0N6RCKpxzimRG4L349i5e9DHNOPThFHXkTejkUodmsojnkHsogNzSXB7wny1eavyV3n/ebi7FyZ7995boKvI9oEuyPynnMpeDyTFH/Fe2GDesfsWgZIkfeCwTYl4hTHTZogUOCdhfjKmElCEB7fvIFjLjhvLKngY2wWRNRMtFpTtvR5c12DDa9eKEPr0YlFwBhIzzO276bG5Pk9PhDN+GizD0T/gSMmc9oRTG+Q4akmv5uMfIHIcOP/6i0LWWDzIuCYGJ825nWnx/gjPIKXojPhPYdStetCIyX53E/vczwU3RBIg8un0+MyTuCAojsyrS+twx6RQlISd9ysEBfvj0PJUqC0rxxzoswVj82k9c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We only have two FOLL_SPLIT_PMD users. While uprobe refuses hugetlb early, make_device_exclusive_range() can end up getting called on hugetlb VMAs. Right now, this means that with a PMD-sized hugetlb page, we can end up calling split_huge_pmd(), because pmd_trans_huge() also succeeds with hugetlb PMDs. For example, using a modified hmm-test selftest one can trigger: [ 207.017134][T14945] ------------[ cut here ]------------ [ 207.018614][T14945] kernel BUG at mm/page_table_check.c:87! [ 207.019716][T14945] Oops: invalid opcode: 0000 [#1] PREEMPT SMP KASAN NOPTI [ 207.021072][T14945] CPU: 3 UID: 0 PID: ... [ 207.023036][T14945] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-2.fc40 04/01/2014 [ 207.024834][T14945] RIP: 0010:page_table_check_clear.part.0+0x488/0x510 [ 207.026128][T14945] Code: ... [ 207.029965][T14945] RSP: 0018:ffffc9000cb8f348 EFLAGS: 00010293 [ 207.031139][T14945] RAX: 0000000000000000 RBX: 00000000ffffffff RCX: ffffffff8249a0cd [ 207.032649][T14945] RDX: ffff88811e883c80 RSI: ffffffff8249a357 RDI: ffff88811e883c80 [ 207.034183][T14945] RBP: ffff888105c0a050 R08: 0000000000000005 R09: 0000000000000000 [ 207.035688][T14945] R10: 00000000ffffffff R11: 0000000000000003 R12: 0000000000000001 [ 207.037203][T14945] R13: 0000000000000200 R14: 0000000000000001 R15: dffffc0000000000 [ 207.038711][T14945] FS: 00007f2783275740(0000) GS:ffff8881f4980000(0000) knlGS:0000000000000000 [ 207.040407][T14945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 207.041660][T14945] CR2: 00007f2782c00000 CR3: 0000000132356000 CR4: 0000000000750ef0 [ 207.043196][T14945] PKRU: 55555554 [ 207.043880][T14945] Call Trace: [ 207.044506][T14945] [ 207.045086][T14945] ? __die+0x51/0x92 [ 207.045864][T14945] ? die+0x29/0x50 [ 207.046596][T14945] ? do_trap+0x250/0x320 [ 207.047430][T14945] ? do_error_trap+0xe7/0x220 [ 207.048346][T14945] ? page_table_check_clear.part.0+0x488/0x510 [ 207.049535][T14945] ? handle_invalid_op+0x34/0x40 [ 207.050494][T14945] ? page_table_check_clear.part.0+0x488/0x510 [ 207.051681][T14945] ? exc_invalid_op+0x2e/0x50 [ 207.052589][T14945] ? asm_exc_invalid_op+0x1a/0x20 [ 207.053596][T14945] ? page_table_check_clear.part.0+0x1fd/0x510 [ 207.054790][T14945] ? page_table_check_clear.part.0+0x487/0x510 [ 207.055993][T14945] ? page_table_check_clear.part.0+0x488/0x510 [ 207.057195][T14945] ? page_table_check_clear.part.0+0x487/0x510 [ 207.058384][T14945] __page_table_check_pmd_clear+0x34b/0x5a0 [ 207.059524][T14945] ? __pfx___page_table_check_pmd_clear+0x10/0x10 [ 207.060775][T14945] ? __pfx___mutex_unlock_slowpath+0x10/0x10 [ 207.061940][T14945] ? __pfx___lock_acquire+0x10/0x10 [ 207.062967][T14945] pmdp_huge_clear_flush+0x279/0x360 [ 207.064024][T14945] split_huge_pmd_locked+0x82b/0x3750 ... Before commit 9cb28da54643 ("mm/gup: handle hugetlb in the generic follow_page_mask code"), we would have ignored the flag; instead, let's simply refuse the combination completely in check_vma_flags(): the caller is likely not prepared to handle any hugetlb folios. We'll teach make_device_exclusive_range() separately to ignore any hugetlb folios as a future-proof safety net. Fixes: 9cb28da54643 ("mm/gup: handle hugetlb in the generic follow_page_mask code") Cc: Signed-off-by: David Hildenbrand Reviewed-by: John Hubbard Reviewed-by: Alistair Popple --- mm/gup.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index 3883b307780e..61e751baf862 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1283,6 +1283,9 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags) if ((gup_flags & FOLL_LONGTERM) && vma_is_fsdax(vma)) return -EOPNOTSUPP; + if ((gup_flags & FOLL_SPLIT_PMD) && is_vm_hugetlb_page(vma)) + return -EOPNOTSUPP; + if (vma_is_secretmem(vma)) return -EFAULT; From patchwork Wed Jan 29 11:54:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13953685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CDE52C0218D for ; Wed, 29 Jan 2025 11:55:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 76F73280041; Wed, 29 Jan 2025 06:55:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 72076280025; Wed, 29 Jan 2025 06:55:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 573D0280041; Wed, 29 Jan 2025 06:55:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3714C280025 for ; Wed, 29 Jan 2025 06:55:07 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id EB655120CEE for ; Wed, 29 Jan 2025 11:54:27 +0000 (UTC) X-FDA: 83060331774.04.D945E94 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf11.hostedemail.com (Postfix) with ESMTP id AD70440010 for ; Wed, 29 Jan 2025 11:54:25 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hMADFfoo; spf=pass (imf11.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738151665; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=herznK9N3+37fO8/yUnjjPI+kU1qrhYMSQrHDNQrbI8=; b=dkkU07lvKNTtXoa8hUA7wDkYBSJLFweeQhmObSipEpoxF8Tlhj53E5zCT6MZFyzdy3YpKm 6yGuCcOWF56XVtgYhKSRwSNqV50hmK87woC0skpKW0XhXtNQKTjUDom29/lcVo6vM7pCO1 VnqIEcfcAnlP8T++g7aSStQAWUNKmDc= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hMADFfoo; spf=pass (imf11.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738151665; a=rsa-sha256; cv=none; b=S5ZRFulnz5mSAKxTAey9se6aFVAcJpvyBeaFtQ2gDl7clKXrhEOVvqWFjlv6qoVrwODxZN Yd+AZtS6IGjrxRtk2umO5Jlvl0k0eHmcFUXEBHLmSaI476oTMA0VQ9/F15NT8parNAb8Qo Z4cYdbUOyEJRcGqG5ziHZUY4sLvvsFM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1738151665; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=herznK9N3+37fO8/yUnjjPI+kU1qrhYMSQrHDNQrbI8=; b=hMADFfoopB18sNNmEW1bLa3glL3c9ZuvalEuwSIxmuL/58tRa8jO36GXrMC645H5eYSe6z Hr4v7esVJU/t4Le5oxZjz9YMiZ67rzzsG9fxPj5HMs+Dozq+aAzgge8bmtDLnOzhnVvMDl yrW7HujKj6aPVcOYPAMTdqv8ZmPi/kw= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-691-an8ZvwIvNGyhl4jjfmhnvA-1; Wed, 29 Jan 2025 06:54:21 -0500 X-MC-Unique: an8ZvwIvNGyhl4jjfmhnvA-1 X-Mimecast-MFC-AGG-ID: an8ZvwIvNGyhl4jjfmhnvA Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-4361efc9dc6so36447515e9.3 for ; Wed, 29 Jan 2025 03:54:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738151661; x=1738756461; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=herznK9N3+37fO8/yUnjjPI+kU1qrhYMSQrHDNQrbI8=; b=uuIbHpx1oQeasKwaNr2VHgl6EtPm/eHKkXrcssXAgst6GviYrBFM6kL4wdwuRj3HhG aq7TtvG/cyfxbaGxM4U4180cCtY7SMI2VnOkOyhnkjNxNh1RpjAzieJrIrVtvDwtyOMq dQonwVW+PLr5UQTpCRcLOSZDkCzsM4OCJ9mLye/FGLOjfkxzCfwUwCaov4dYme2h4Gf1 /NPeltM820/e1vzxFEovaXtkSZWplo586jw0kwR/XwhAaM4GCSPeygbMVUEdDi5GMKoE eS0u3wuIBWcZPZn1WRsk7K8sJ5E6W2oU1z7S9qRzYNbj6YLIb72RV6wDj1Swt0wmJhI8 WLpA== X-Forwarded-Encrypted: i=1; AJvYcCVoKslQe1eF/jt/SgYXaYhOk/Mrr0r8XXqe6xXFS9uFTKg+IJQlBsGpifxC5EP1td8l8EKP1lpy7g==@kvack.org X-Gm-Message-State: AOJu0YzS7npS/+211bKVRp2eNusIWjhxfjs9428QpJKDt7irZpD+TPhZ xjYPNu0pebFA2L5niLnlBaL6vR6J+s7dFuqW1gnye0iUXTSyJ5JLebLJM6OjUm0WBimbzK3Ic+/ P0oeih1xSGJ+dH4Xn/KFT4J9eoGGp9AySfiJsqYtAFbxSWisO X-Gm-Gg: ASbGncs/twSee6uHxZryKJnYEHAES30l9jElBpV0DQ5WJG47s5aIrVawRyUSHYSFtjD 40aJC+WeXquvp58wCWQnSjwFxcEijjefgGMguAwXS1LPTjDMHF8P/wruVGDaHbvOrq4jRFzNupj dQPoyRJXIY8wZymuEj6+pTrEfTdDnP0e/m8mp6Z1MX0he4KdCj8a32Oplv+ueQqbZ3m5CYfVHYG eYP7MuMYvbGb+3vbhQyK/GkNHMRdHSmIU9uDGylgXTRHnG3TuZUu563+ja+HLqqfGLP0IzXpwFK C9PC0EI3OzsmMfT1trY7WfG11raQOcYIZyNubAvujSKIEOjZlduyjHzg6FGF9pejvw== X-Received: by 2002:a05:600c:5486:b0:433:c76d:d57e with SMTP id 5b1f17b1804b1-438dc3a40d3mr26190565e9.5.1738151660695; Wed, 29 Jan 2025 03:54:20 -0800 (PST) X-Google-Smtp-Source: AGHT+IEopxoArzHie066mjWjUvNKTSc/02V79PTjUrPj2QcjbwdMpR1m12X2+qvEI4pAjYXanIEeWA== X-Received: by 2002:a05:600c:5486:b0:433:c76d:d57e with SMTP id 5b1f17b1804b1-438dc3a40d3mr26190115e9.5.1738151660040; Wed, 29 Jan 2025 03:54:20 -0800 (PST) Received: from localhost (p200300cbc7053b0064b867195794bf13.dip0.t-ipconnect.de. [2003:cb:c705:3b00:64b8:6719:5794:bf13]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-438dcbbc52dsm21427725e9.0.2025.01.29.03.54.17 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 03:54:18 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe , stable@vger.kernel.org Subject: [PATCH v1 02/12] mm/rmap: reject hugetlb folios in folio_make_device_exclusive() Date: Wed, 29 Jan 2025 12:54:00 +0100 Message-ID: <20250129115411.2077152-3-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250129115411.2077152-1-david@redhat.com> References: <20250129115411.2077152-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 7MAYCi4wGh2XClFBW5T9MD-DK7ioUM0CijBh_0GEOIQ_1738151661 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: AD70440010 X-Stat-Signature: 5pip9bzahun6i9syq4s3b4hy8smp63kb X-HE-Tag: 1738151665-187769 X-HE-Meta: U2FsdGVkX18RH5KyShFCp/7nO1nFhtGFRZQKc1HzK3Ux8CgviMO7o35QOGE3t4ZGByuZKS5d6p+hGsAR54oqXafN8ujpRLr9TvvRTQ/DHOLzlaybO04cdhR/4HVBfPwXukvLDlecV2CFnzVXUI4GLYjgWVXkU030VKfyViP1Rl6Z0xPddkDpWSOxVaU/WiVBFzzpWS7BWolpQf8qPlOmeNYLlwZPr5O4RFJfn9ZJqZc0eQJEd5U7dJNhoxE3boEqxdZLIoxruNgXpuwSWMgSy7hIe9ifvtXh4ZBcbaJtnCJcKFlh/IlSeGz0+meSxzuo364CSmLycC9CorStbxLzNgENJQSJL3oZHsJuNN8iNRNaCB+X79Mch72thEAFxW948RU4ast2pWzmss2EUnRSyZaIv6ESxusI2CwvYZRB2+gAnzMta6ckcTDKjTQZncCraXJgtkCKHNEdDI2woZWfTVld4GXushAMTPDrdatlVVwPe9fpQGNGk91z0fXYNoeJv0xAmcJ9GSuvwlJgCsxZ2IftSRMSVFtlhDQPmjnGnCkZ/2Iohu2wdxd4Pp5zFqBbSD3ZJ2CE0XDOA2fHcooV9viRF4i5D2GppeNiNwBCmB9ZNQzwULVUXjFrZMyXdLBMDMIi1LZQtMqoIvVbSVXE0l+hTApz0g3s/9YSjn9NKCmNQyZL79nMhNEjfq9b6A/m58ZZgYwOpRoE2kSDx+6HV9h2jA3FTiM2wY4zR1b0jHzc2bDlhRtLfb9/9kRqMgftz0dBFqRPp/yI4gIG8PWUBXmQTa/ZiwIqRqBkEvRSUEdnsrz4Du0UuKoKpY0e/OKAzUmiBaH3skyynadm0kHXmlRQb5DwD30JQlWIGvAJA0MIH7HN4k9h9H9EIBG/7m3C3mNqZNpvv+Ry+ZoSOnrCDHohwhdvt5orAV10JwpumFyGAL2u84EdOQBRvgG86BcrjU0NfjMz5npb1km/Z6e rBn9BP7b CLRLorX53I6dZscTx4kn//GN8GJieMYI2Wmn+wr8CBCj/CNGjRf2Nc/N0Ud3e7mjGvWjQ/VVd8jc/fnuzoYOKmZlr4FeeeA6bv0y4M3XBY6284uyXlgiSYRTsjPFZm4elXsmjM62BDi5HmOHjDeKmWrkaQJ5DRqKvPxJ1i89StK4fGmu8viaN0p6MXBMdj5xP18EZ2G7Ix7AbbGC76wvckUBWhXsKzyu04jS7jteZ6es1ymPHN5ALxqhuRQ2stMHcroo/fRuTpF4VmQWyeIw9nPIdwOAPFLxHMlBcQdNMyy3EYC5ByHn7vVwuzezHrqXET7fP+PfGOXZGjlXLt0Vj4Oz9MTprTrNxFdfFd9cLlZSslURgb6ZmvQMfPXhlVYy2YgOsygVL0X/eRO4jI8P2aekFf0BMwsHRXDZOhTRhAEMbW4byS/U4rcrdqWKtHh/Ds6PosWc/sXZsk1qOesZYbDjiAWxVhPFQ1mocGDcYB+Ef19jxFNVpzu04k4SZRFqf6C5ZSPGnnW6I9e5Jmr/ncLAMUbH7eVWwqw/pwa2w7ud0KTo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.009333, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Even though FOLL_SPLIT_PMD on hugetlb now always fails with -EOPNOTSUPP, let's add a safety net in case FOLL_SPLIT_PMD usage would ever be reworked. In particular, before commit 9cb28da54643 ("mm/gup: handle hugetlb in the generic follow_page_mask code"), GUP(FOLL_SPLIT_PMD) would just have returned a page. In particular, hugetlb folios that are not PMD-sized would never have been prone to FOLL_SPLIT_PMD. hugetlb folios can be anonymous, and page_make_device_exclusive_one() is not really prepared for handling them at all. So let's spell that out. Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Cc: Signed-off-by: David Hildenbrand Reviewed-by: Alistair Popple --- mm/rmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/rmap.c b/mm/rmap.c index c6c4d4ea29a7..17fbfa61f7ef 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2499,7 +2499,7 @@ static bool folio_make_device_exclusive(struct folio *folio, * Restrict to anonymous folios for now to avoid potential writeback * issues. */ - if (!folio_test_anon(folio)) + if (!folio_test_anon(folio) || folio_test_hugetlb(folio)) return false; rmap_walk(folio, &rwc); From patchwork Wed Jan 29 11:54:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13953682 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F3AAC0218D for ; Wed, 29 Jan 2025 11:55:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 17E2328002E; Wed, 29 Jan 2025 06:55:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 12C82280025; Wed, 29 Jan 2025 06:55:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE77028002E; Wed, 29 Jan 2025 06:55:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C753D280025 for ; Wed, 29 Jan 2025 06:55:01 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D4DF7120DA6 for ; Wed, 29 Jan 2025 11:54:29 +0000 (UTC) X-FDA: 83060331858.04.1D4961F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf01.hostedemail.com (Postfix) with ESMTP id 8BC094000F for ; Wed, 29 Jan 2025 11:54:27 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=N1nw3rxa; spf=pass (imf01.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738151667; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FbpRkzZA4TilYwHsov5kO/PS96AeznJfw9pJaA8wpMg=; b=BBczGH5Sq+SJ5HGBBcruTCeyziaF3JNRZU8uMyPSmnekcFv4iOP2zmiqMeHwlOhblXYUyo 3XNuEZ7HxFHgi50nY2AlWbq+4yOQGlucz1QVOBuCQAQf0zMh2JfnwlyAWsvMcmU8rl9fif HVcvUJoj/+jO02Wdb90hdOQ6frO21w4= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=N1nw3rxa; spf=pass (imf01.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738151667; a=rsa-sha256; cv=none; b=CNRy5SaD11CAGQFE8Oinmen75ZKQNcx1vnaUzpWOm3PNcgLlGBeOrGxLOvnU0DJIJow6+/ cqPtnc/pDQOX7C/JfYc0EW7CSevL8yVNNQyGb8BsG8FuF4q3oIwvFScS2B6TvMkDE4ctS/ 9ByZmTOKF3r1yzTmuth1gCTb+YpM4VM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1738151666; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FbpRkzZA4TilYwHsov5kO/PS96AeznJfw9pJaA8wpMg=; b=N1nw3rxaCjaMMKo5m7OvbxQYP1a+kPUcfXjWPQguOW9x+z6acMCIdpuCUDGFiMnF3DGQsJ CPlQmBO7pRLrIobPOAkbWVxfgXhn+62N5OU0GCgt2Spf7FeiPGA6mHVOlSEljcY5zt9D0n BqTXXfoYGnJoY1saCfLlCzptlmjBzoo= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-115-HZZAvdDPMMme004xLJow8g-1; Wed, 29 Jan 2025 06:54:25 -0500 X-MC-Unique: HZZAvdDPMMme004xLJow8g-1 X-Mimecast-MFC-AGG-ID: HZZAvdDPMMme004xLJow8g Received: by mail-wr1-f69.google.com with SMTP id ffacd0b85a97d-38bee9ae3b7so4420936f8f.1 for ; Wed, 29 Jan 2025 03:54:25 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738151664; x=1738756464; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FbpRkzZA4TilYwHsov5kO/PS96AeznJfw9pJaA8wpMg=; b=CmdIEGf0Abqupw3jFSJmDeHVxHkwtpqm90i3Q4hHqSil+auEK37W27ZfJ7ZcUnuUIV EuZ9VoxsoY2YhVk5Wl06fvcu4QJgYdGOh4M1gXwtcYYdSNMX5ik5zueVXiP51vokIjy7 V/zvCbp7Nkxd/DCtslfmz2w2pKsR1PCJfS49joTTFVoy5ONWsXBCpBOgrRMDiXt7y0l9 kHh4GzBJzRu5v3NLUlAZotSNfnug1tXiMF0SY9mxheK4rzNt5rgWdAturxFIHTMPe6at N+aiC7yAAyDQpCG/nxx6RfqArfhZdLt2bPCHoXbuGcBN+zjleXahKqXJRPvwtRIU8z4N Sraw== X-Forwarded-Encrypted: i=1; AJvYcCXH83hvTIczZXouNKXQVcYr9t05OsNnZaO7k+9GXnHC6o0IQbmzk0LbQHedNLsrZCmGZub2O/jTLg==@kvack.org X-Gm-Message-State: AOJu0YyIzd29WnzEAOoaJo5YBpOagf9jm69oVQKUUCftrL6BGoZsoNIb Wc+NHdr8ZzNhHPaw3gpoOOeKoO3sQQqxdNe7DoFTUIixuzk9awxDqeL0dGrgsWSoXajfitMdz8f YDgjVE75F8SFPSjD+P8NNEXAuo4bhwvMTdeIQMxA5krhoqmNu X-Gm-Gg: ASbGncsIxiG1GB6B+CKDUJeo5LASiY19WrF+YZLzX15dXu7Io/A+jj/z4E/IuicEsvE WmIK7Un93aduu3rqY//9qoGzVejnXPcY/DYfo+bBHCihCu/F+hSxy7en2UWXUt201/f1+g/CzBR KTqkkgquWVB8yYTv/OCHN9UNXrhjkUsgip352du7hdkElzAsVIc+WtxQ7LLjRIsleHwN6mxT95C gcBxADpQc8lcnk83FQhwCjG3pXK8u16poB3n4xilum6r8AeHXJ3zqlaoxRqr859pLdAZJ5r3uGl HSwOrsSrh7r5CpWE2bxY17w9bVczIZu40152EcAiMDdnHQcBbv4ka5+h8GkTWsqyDg== X-Received: by 2002:a5d:64c3:0:b0:385:e5d8:2bea with SMTP id ffacd0b85a97d-38c519460aemr2346430f8f.20.1738151663853; Wed, 29 Jan 2025 03:54:23 -0800 (PST) X-Google-Smtp-Source: AGHT+IECXzKNlO/KxhDgtZSrbOQmu7i91GFToVYgJKpoIoUvo8DlvKFDTSa6/lsKfWQw3fetIFbvYQ== X-Received: by 2002:a5d:64c3:0:b0:385:e5d8:2bea with SMTP id ffacd0b85a97d-38c519460aemr2346391f8f.20.1738151663351; Wed, 29 Jan 2025 03:54:23 -0800 (PST) Received: from localhost (p200300cbc7053b0064b867195794bf13.dip0.t-ipconnect.de. [2003:cb:c705:3b00:64b8:6719:5794:bf13]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-438dcc27130sm20111455e9.16.2025.01.29.03.54.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 03:54:22 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v1 03/12] mm/rmap: convert make_device_exclusive_range() to make_device_exclusive() Date: Wed, 29 Jan 2025 12:54:01 +0100 Message-ID: <20250129115411.2077152-4-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250129115411.2077152-1-david@redhat.com> References: <20250129115411.2077152-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: yRj55y7_mbWhT0SZN8GVgeHk5fQ5TXeiQl8dYSc00g0_1738151664 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 8BC094000F X-Stat-Signature: 15zuh8n41z7n9ga6aky8r8t7x9998xhi X-Rspam-User: X-HE-Tag: 1738151667-977916 X-HE-Meta: U2FsdGVkX1/5iY4jYl8+bMERy9msFjsft1zxr5sGgYHhonjIZHJYc8LAoTCyzJRbiVhsNW0Pd0jgtl25xJRYLFoWeY0vxooLamPoRMzW6T8w5s6WkOueLrdaEPKSY7jaLMS+R3naoJTGoD6lxgZ3kjZUb5DOoTa50131a5w/sW+hoc3wd7errYab30kvJexczwGKqdfqvTZPJdXMo17P6/ja1p+QF9luIKKMoM76d0f65qjCK0Tdz6b14tWTqozvJxPjgJpMkwdJsSdqsLCTheWrVDpIJQyP/4QZLXFe6h38YBKgG2eT+oTaYpVaE1KKhfKlSILWBLQb+e3xDP44l9ZuuyLWdrW/V5yqe707GhMTOKHTZ76Cw9DbPBQVR85wt7VkobyJwKemTEpPt6s+UeKSRsBD47RUOgLXssqHZ96K6+b6qsBrW7rHSXS1DMGf8Z4hEbifzDmex7tkaC3HSwGgwuM8BL5ZG6Q2NIGzPQOrvY9cUe+IhTx8ZvBgYLslrwQ8YbCmtMFPixxAWHdByp8/PDupf3g4Lrwr7AM2afp5KNUr0UqAYeFmuTW9qXKaWvB3FqnVjzo3V40JO9W0hb5GDAp9RSRLPYnAZfTqimHRO2OFyKMwur9K0ltzv90OBQDECEJhlo0BW3GzA576h3ZmVr6q+UYiL5P4Ajp29maYkG0P1bfsHBpW9pVmQpVZ1V6EGvZXx9u9NlGd7yEKHhozVR8HnTyEisFyFoLUJct8Z/o/eWztY4GeIhRbHpym5wskR9MnW9/kVYOM291LVf6aPrJfPSIDzDVP1JJeNNCfEYHBoyYYKTLVqyT3J6L502CdvCEvnZINLz22HQp6SoOo+DJfXIxTmw6ojo46B4MWGd3nV/Bb8AQflRGA7k9NhjAWXwEIxdxtPFiIQcYl+MvYoCgBz/5hYVTcFihIKSUQPnecQdg5RhdQnkNbf9Hk0OwJAO3VuLC1dXVJ5RD xoyUDbUi WSTzEMh0IZ0QuSUvpb75UOc4Cud6mJHGBzbHRVvsavyw9f1zElLeUtaPt1kIftkvE2zzz4mrZdYsrmJdtrqQboVO9Zo891OlaiAfjTWj+pHY9iIQQUVyZ+694Vdq9wQEetvIgKQ8gFaZo9ya3SKa46jLeGC1yzNK6yIQ0WNrjFb18ZdjoFQbpllCSpiGjpPoKD3owFi5M+3l82SldT/bEdJn3w9Z8qrgbgPkT0glVkR1OLl2LMMN5h3pj83sB0h2tn7kyOaMomXjZPS1ZiOzP0FLagmqxV8CSw7gpN4MpCFJpPMj8rcHrNkoBvxb5jwXAllJhSG47pT4ue+77Jd13tiXHVtXe6cQ1ImEav3+MI0qRS3Xcv38BUioD5gd2mHbshZMIDr4tKrwm0kiH3MgSjeQ9nJPciYmLo386AeG6c2yeI7pnbF/u/xQLw0XrU8eL7ru4OFAH2tM8AvA+i8hm7od7HwMcLXqIYaUKqKKapH1HqdVtOlCu51UAh+UvS6GGY3idg9fsCFstWdiu9nOBWoji5CumZjMUItxNOk76q0zT4l1y2eIps1IdJA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The single "real" user in the tree of make_device_exclusive_range() always requests making only a single address exclusive. The current implementation is hard to fix for properly supporting anonymous THP / large folios and for avoiding messing with rmap walks in weird ways. So let's always process a single address/page and return folio + page to minimize page -> folio lookups. This is a preparation for further changes. Reject any non-anonymous or hugetlb folios early, directly after GUP. Signed-off-by: David Hildenbrand Reviewed-by: Alistair Popple Acked-by: Simona Vetter --- Documentation/mm/hmm.rst | 2 +- Documentation/translations/zh_CN/mm/hmm.rst | 2 +- drivers/gpu/drm/nouveau/nouveau_svm.c | 5 +- include/linux/mmu_notifier.h | 2 +- include/linux/rmap.h | 5 +- lib/test_hmm.c | 45 +++++------ mm/rmap.c | 90 +++++++++++---------- 7 files changed, 75 insertions(+), 76 deletions(-) diff --git a/Documentation/mm/hmm.rst b/Documentation/mm/hmm.rst index f6d53c37a2ca..7d61b7a8b65b 100644 --- a/Documentation/mm/hmm.rst +++ b/Documentation/mm/hmm.rst @@ -400,7 +400,7 @@ Exclusive access memory Some devices have features such as atomic PTE bits that can be used to implement atomic access to system memory. To support atomic operations to a shared virtual memory page such a device needs access to that page which is exclusive of any -userspace access from the CPU. The ``make_device_exclusive_range()`` function +userspace access from the CPU. The ``make_device_exclusive()`` function can be used to make a memory range inaccessible from userspace. This replaces all mappings for pages in the given range with special swap diff --git a/Documentation/translations/zh_CN/mm/hmm.rst b/Documentation/translations/zh_CN/mm/hmm.rst index 0669f947d0bc..22c210f4e94f 100644 --- a/Documentation/translations/zh_CN/mm/hmm.rst +++ b/Documentation/translations/zh_CN/mm/hmm.rst @@ -326,7 +326,7 @@ devm_memunmap_pages() 和 devm_release_mem_region() 当资源可以绑定到 ``s 一些设备具有诸如原子PTE位的功能，可以用来实现对系统内存的原子访问。为了支持对一 个共享的虚拟内存页的原子操作，这样的设备需要对该页的访问是排他的，而不是来自CPU -的任何用户空间访问。 ``make_device_exclusive_range()`` 函数可以用来使一 +的任何用户空间访问。 ``make_device_exclusive()`` 函数可以用来使一 个内存范围不能从用户空间访问。 这将用特殊的交换条目替换给定范围内的所有页的映射。任何试图访问交换条目的行为都会 diff --git a/drivers/gpu/drm/nouveau/nouveau_svm.c b/drivers/gpu/drm/nouveau/nouveau_svm.c index b4da82ddbb6b..39e3740980bb 100644 --- a/drivers/gpu/drm/nouveau/nouveau_svm.c +++ b/drivers/gpu/drm/nouveau/nouveau_svm.c @@ -609,10 +609,9 @@ static int nouveau_atomic_range_fault(struct nouveau_svmm *svmm, notifier_seq = mmu_interval_read_begin(¬ifier->notifier); mmap_read_lock(mm); - ret = make_device_exclusive_range(mm, start, start + PAGE_SIZE, - &page, drm->dev); + page = make_device_exclusive(mm, start, drm->dev, &folio); mmap_read_unlock(mm); - if (ret <= 0 || !page) { + if (IS_ERR(page)) { ret = -EINVAL; goto out; } diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index e2dd57ca368b..d4e714661826 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -46,7 +46,7 @@ struct mmu_interval_notifier; * @MMU_NOTIFY_EXCLUSIVE: to signal a device driver that the device will no * longer have exclusive access to the page. When sent during creation of an * exclusive range the owner will be initialised to the value provided by the - * caller of make_device_exclusive_range(), otherwise the owner will be NULL. + * caller of make_device_exclusive(), otherwise the owner will be NULL. */ enum mmu_notifier_event { MMU_NOTIFY_UNMAP = 0, diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 683a04088f3f..86425d42c1a9 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -663,9 +663,8 @@ int folio_referenced(struct folio *, int is_locked, void try_to_migrate(struct folio *folio, enum ttu_flags flags); void try_to_unmap(struct folio *, enum ttu_flags flags); -int make_device_exclusive_range(struct mm_struct *mm, unsigned long start, - unsigned long end, struct page **pages, - void *arg); +struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, + void *owner, struct folio **foliop); /* Avoid racy checks */ #define PVMW_SYNC (1 << 0) diff --git a/lib/test_hmm.c b/lib/test_hmm.c index 056f2e411d7b..9e1b07a227a3 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -780,10 +780,8 @@ static int dmirror_exclusive(struct dmirror *dmirror, unsigned long start, end, addr; unsigned long size = cmd->npages << PAGE_SHIFT; struct mm_struct *mm = dmirror->notifier.mm; - struct page *pages[64]; struct dmirror_bounce bounce; - unsigned long next; - int ret; + int ret = 0; start = cmd->addr; end = start + size; @@ -795,36 +793,31 @@ static int dmirror_exclusive(struct dmirror *dmirror, return -EINVAL; mmap_read_lock(mm); - for (addr = start; addr < end; addr = next) { - unsigned long mapped = 0; - int i; - - next = min(end, addr + (ARRAY_SIZE(pages) << PAGE_SHIFT)); + for (addr = start; addr < end; addr += PAGE_SIZE) { + struct folio *folio; + struct page *page; - ret = make_device_exclusive_range(mm, addr, next, pages, NULL); - /* - * Do dmirror_atomic_map() iff all pages are marked for - * exclusive access to avoid accessing uninitialized - * fields of pages. - */ - if (ret == (next - addr) >> PAGE_SHIFT) - mapped = dmirror_atomic_map(addr, next, pages, dmirror); - for (i = 0; i < ret; i++) { - if (pages[i]) { - unlock_page(pages[i]); - put_page(pages[i]); - } + page = make_device_exclusive(mm, addr, &folio, NULL); + if (IS_ERR(page)) { + ret = PTR_ERR(page); + break; } - if (addr + (mapped << PAGE_SHIFT) < next) { - mmap_read_unlock(mm); - mmput(mm); - return -EBUSY; - } + ret = dmirror_atomic_map(addr, addr + PAGE_SIZE, &page, dmirror); + if (!ret) + ret = -EBUSY; + folio_unlock(folio); + folio_put(folio); + + if (ret) + break; } mmap_read_unlock(mm); mmput(mm); + if (ret) + return -EBUSY; + /* Return the migrated data for verification. */ ret = dmirror_bounce_init(&bounce, start, size); if (ret) diff --git a/mm/rmap.c b/mm/rmap.c index 17fbfa61f7ef..676df4fba5b0 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2495,70 +2495,78 @@ static bool folio_make_device_exclusive(struct folio *folio, .arg = &args, }; - /* - * Restrict to anonymous folios for now to avoid potential writeback - * issues. - */ - if (!folio_test_anon(folio) || folio_test_hugetlb(folio)) - return false; - rmap_walk(folio, &rwc); return args.valid && !folio_mapcount(folio); } /** - * make_device_exclusive_range() - Mark a range for exclusive use by a device + * make_device_exclusive() - Mark an address for exclusive use by a device * @mm: mm_struct of associated target process - * @start: start of the region to mark for exclusive device access - * @end: end address of region - * @pages: returns the pages which were successfully marked for exclusive access + * @addr: the virtual address to mark for exclusive device access * @owner: passed to MMU_NOTIFY_EXCLUSIVE range notifier to allow filtering + * @foliop: folio pointer will be stored here on success. + * + * This function looks up the page mapped at the given address, grabs a + * folio reference, locks the folio and replaces the PTE with special + * device-exclusive non-swap entry, preventing userspace CPU access. The + * function will return with the folio locked and referenced. * - * Returns: number of pages found in the range by GUP. A page is marked for - * exclusive access only if the page pointer is non-NULL. + * On fault these special device-exclusive entries are replaced with the + * original PTE under folio lock, after calling MMU notifiers. * - * This function finds ptes mapping page(s) to the given address range, locks - * them and replaces mappings with special swap entries preventing userspace CPU - * access. On fault these entries are replaced with the original mapping after - * calling MMU notifiers. + * Only anonymous non-hugetlb folios are supported and the VMA must have + * write permissions such that we can fault in the anonymous page writable + * in order to mark it exclusive. The caller must hold the mmap_lock in read + * mode. * * A driver using this to program access from a device must use a mmu notifier * critical section to hold a device specific lock during programming. Once * programming is complete it should drop the page lock and reference after * which point CPU access to the page will revoke the exclusive access. + * + * Returns: pointer to mapped page on success, otherwise a negative error. */ -int make_device_exclusive_range(struct mm_struct *mm, unsigned long start, - unsigned long end, struct page **pages, - void *owner) +struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, + void *owner, struct folio **foliop) { - long npages = (end - start) >> PAGE_SHIFT; - long i; + struct folio *folio; + struct page *page; + long npages; + + mmap_assert_locked(mm); - npages = get_user_pages_remote(mm, start, npages, + /* + * Fault in the page writable and try to lock it; note that if the + * address would already be marked for exclusive use by the device, + * the GUP call would undo that first by triggering a fault. + */ + npages = get_user_pages_remote(mm, addr, 1, FOLL_GET | FOLL_WRITE | FOLL_SPLIT_PMD, - pages, NULL); - if (npages < 0) - return npages; - - for (i = 0; i < npages; i++, start += PAGE_SIZE) { - struct folio *folio = page_folio(pages[i]); - if (PageTail(pages[i]) || !folio_trylock(folio)) { - folio_put(folio); - pages[i] = NULL; - continue; - } + &page, NULL); + if (npages != 1) + return ERR_PTR(npages); + folio = page_folio(page); - if (!folio_make_device_exclusive(folio, mm, start, owner)) { - folio_unlock(folio); - folio_put(folio); - pages[i] = NULL; - } + if (!folio_test_anon(folio) || folio_test_hugetlb(folio)) { + folio_put(folio); + return ERR_PTR(-EOPNOTSUPP); + } + + if (!folio_trylock(folio)) { + folio_put(folio); + return ERR_PTR(-EBUSY); } - return npages; + if (!folio_make_device_exclusive(folio, mm, addr, owner)) { + folio_unlock(folio); + folio_put(folio); + return ERR_PTR(-EBUSY); + } + *foliop = folio; + return page; } -EXPORT_SYMBOL_GPL(make_device_exclusive_range); +EXPORT_SYMBOL_GPL(make_device_exclusive); #endif void __put_anon_vma(struct anon_vma *anon_vma) From patchwork Wed Jan 29 11:54:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13953677 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C729C02193 for ; Wed, 29 Jan 2025 11:54:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA46D280013; Wed, 29 Jan 2025 06:54:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 99104280011; Wed, 29 Jan 2025 06:54:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 80AE4280013; Wed, 29 Jan 2025 06:54:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 456FC280011 for ; Wed, 29 Jan 2025 06:54:43 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id E6A5BB0D45 for ; Wed, 29 Jan 2025 11:54:33 +0000 (UTC) X-FDA: 83060332068.17.BE98CDC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf26.hostedemail.com (Postfix) with ESMTP id AC685140010 for ; Wed, 29 Jan 2025 11:54:31 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Lza5kFiS; spf=pass (imf26.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738151671; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ttFA6KXnCKLshNnRCUvMdxdw/LdWUHfs73bnazzKqOM=; b=6+DxoxZMNWcPaxccQvFykn6hrFuxrgiKls5xyYeITiiEFgY1MPwZ7TmokSFISh796YRgUD gUDkWuoywdYnC/gI9N35RWP9BiDgWr2fbnQpBFB68IY7OgRlDMS0VyIdLPCkziM9j/WNyJ ymcqecH1UxsFCzpQ7yFJEuVSGxzt+3I= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Lza5kFiS; spf=pass (imf26.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738151671; a=rsa-sha256; cv=none; b=nRzE8pkBa8q3VGzi5bvhlDv86qbGNWPxWBTH0E5bIXmrGQP15h/bAAgWNSAZMPC2lB4CBS dcuK1VthQzchnqZujZjgbHMeHDnAKCtEW+LyRNMnfN9ZEaPB4c3hHOjOPta7kIgUbsw74r HtBPSvvoodHHz3oImpUUgB6IPdE0Ykg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1738151671; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ttFA6KXnCKLshNnRCUvMdxdw/LdWUHfs73bnazzKqOM=; b=Lza5kFiSqD7O0mgFvtQ743XqjYkLhOfSKrEbDIvge/0WrugxO7GSvGplkR5tHe8lXfsIHW KizPrAAjbpQyGs5IGx7ZVuXjo6FoXYxAsAvmBJztodJbyMwHvXiAt2lQMIh9pBgRAOmnDR bEqlICEY8IEy3atUy4M3RShXysgYAxA= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-513-BXH1-cXIMiOcJjdNQy79sw-1; Wed, 29 Jan 2025 06:54:27 -0500 X-MC-Unique: BXH1-cXIMiOcJjdNQy79sw-1 X-Mimecast-MFC-AGG-ID: BXH1-cXIMiOcJjdNQy79sw Received: by mail-wr1-f69.google.com with SMTP id ffacd0b85a97d-385e00ebb16so2352545f8f.3 for ; Wed, 29 Jan 2025 03:54:27 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738151666; x=1738756466; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ttFA6KXnCKLshNnRCUvMdxdw/LdWUHfs73bnazzKqOM=; b=XPuuSsW1sXfnZKMLMLtMrTq+kQEkpd4U+jct0MVMIeCNHX3GzOQh+DPMFoTYqw6nkk izHZXcp7759sm1rLobiPuR53OA8XTIPZ8nnjRprNTTg/LX5utrpsvuHIX9Of1pqiuOiQ 1FYwtlt8e7r4r4TrVTnybFOSbNjJhZGx+fl0CULpzLqz45Tz85aSRWXZNV7Zt11aIP6k ojZ39ClJ43e3YtQ4+692HssprjivWfgbJyuNfL3WI+kX31xFS1pDlA5PDlTLd/tsls1Q eo3Uo11v71YRbFhdJX/uWLXFWRKo4GlV9lDn9HjTaBJB4HJEVdx3wTO+ZSusPkeESU83 60Ag== X-Forwarded-Encrypted: i=1; AJvYcCVeqGWK+/ft15l46qt5FkRML7tH2lxh3Y9gTdQtkBlYk+3EPpT+SUyU8jdT6pryt2/oJztT6XKZpQ==@kvack.org X-Gm-Message-State: AOJu0YxfANRwfaQUPo2EDmFoFa+kil32QIZo8pu8Jr2Iu5rRkhAUtdZI aKYHiFKjffgT3Vsm/QwkWZpswSSGnYS8L5D3Q2JOfgjQig+mL906q2sJHoek90jdxIEoOvYyUix XO7RUCih20OP+i5pMd5AqcXa8Q44zQ3dAR5B3LaRtbka9AaI7 X-Gm-Gg: ASbGncsqKDjmMWdyU1MoWCISS3BYZvriV+StvOoKIwSeImLKs/FKze+r9F+Bzaf/78N peZ1OVkg5RMDaazQKqLAy4UbYuKORNyByEG21Vmxn9JJT97NSY+1K5+Oe4SHV74yR1z2inHToIO MGwmpbeY5v41blEqiVOHwu5em6hPMZPtFo8k4WrMJuBIJjn6/q+WFwetJ9rAYtqa66BsZ0sQgKJ OOpNj3SgYI6CYucY/xY11Ug+mmpIKdG5Ay7OPzMgWbRots989PJWPEwY78UROLKiGnXpObpORh+ SgANo/S4wx1Vqgt7isK7rYfaK/qDDGitvx6ECzZskYhcT3rJKId1/deK8CxHiUh+Pw== X-Received: by 2002:a05:6000:11c9:b0:38c:1270:f961 with SMTP id ffacd0b85a97d-38c520b7c7fmr2019778f8f.46.1738151666562; Wed, 29 Jan 2025 03:54:26 -0800 (PST) X-Google-Smtp-Source: AGHT+IExNZpiTenD6CJ57erMOTnO5s4zsyHNUaMa/bCtN9QyTRZVkdeAROPCHCw+IAs2gC6Ff7tvTA== X-Received: by 2002:a05:6000:11c9:b0:38c:1270:f961 with SMTP id ffacd0b85a97d-38c520b7c7fmr2019743f8f.46.1738151666064; Wed, 29 Jan 2025 03:54:26 -0800 (PST) Received: from localhost (p200300cbc7053b0064b867195794bf13.dip0.t-ipconnect.de. [2003:cb:c705:3b00:64b8:6719:5794:bf13]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38c2a1bb0besm17079668f8f.79.2025.01.29.03.54.24 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 03:54:25 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v1 04/12] mm/rmap: implement make_device_exclusive() using folio_walk instead of rmap walk Date: Wed, 29 Jan 2025 12:54:02 +0100 Message-ID: <20250129115411.2077152-5-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250129115411.2077152-1-david@redhat.com> References: <20250129115411.2077152-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: vFigDQ4gqTJwy_IJV9xerbMguAR_TgiqkbA_K4gzNTg_1738151667 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Queue-Id: AC685140010 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: octqkiafd5hjr9ebyw35tns7o1f9ru5u X-HE-Tag: 1738151671-774246 X-HE-Meta: U2FsdGVkX1/Tyl9a/hbkqDdw/PRbIY7BLfH1ctR1PAAcBWqlL0T5S5TEB656KiLCGEfbVsa2DcVbWk1X/0zLQyYHHTUdWSZyNnkJuaG7sgaZm1ybg2G78WEa+ZdRxbCybNtG5ZahkzLBQ21CXdmMUCszhF2ro8HVgWZtg/GnILLSPu1I7bSIvdOnSCMQjL+6IaUiKfjYq4t4oQnb42x+bJ3M+WgbMvqzdeznErO5wXmebJhRtoPP8+KclVRMnPxU6FseklDa8DF9KWArE2jlxVye58GaVQ26NslzWilP+dPuIQyFbhe5IInSMGIQ53V4c2vibTvai3XmrRLB1dvI+JsW7yW9Xz0ZSWIp5HD/tbTp834uyRHb16vG21uWbW9+4LB7Edr0JId9dtn7Y+613N8X+lxDXouhXJZpC05neGU82tGNXg6HE9hNuOKpoifzpnAzlK+HtP3Wx812Mp/c6lHiYkqp+dtBcE26l7cT1kaq8zDy/T10GFm+IN/miy5qRDjUeAdHXdDTX2MO4sbSRMFV1LxKpefhkCEZhBrP5SCepUES2sXs5rKYeiyBn/KjJZ8cCpMp4rhx1Nd0AMIdTxqmqErXcp+jJMn0LK5mX0e4Wf8uiJpyxNWKQXoPxZsJpk3ZOAO37bzWQwo4aiIWYEExiD0TdSC0NmiLu3Qr+flUjimubbB6avAXehpUMnJ+sn5grldN3vnj3ryJsoNjIxeIeweKXLhciQgXy8Zc+PD/qggTZcE7s30gS2W0N4C249+02BiTaJ3oFRr1zZCBtU3ljxMp9yYyWUVwA3Q3Syp+9lDgOIJrC2J2SxSF3ULTYOcazPiJOhj7Mb8r5mAxiyv6qJVT5+Nj0v0VHMG5uTLbHWCq4fBUEVkYS/EGY5RUVgNPZIGduNMh6hSH17n8LOjkmbrrqiz3WsD7t7cIa52Ta1uhSWgVPUoChKdISUgLCystEfgVYeLLv+R5WpI f6yNLp+u NQwh5BV4kBdMhVjB9V8mFDTtNgl4lWZOGOPDqcAKu5Z7jll2lhgnyUMMRHRazf13gbkKg7vAw0RoDSb0bdVb0yVQRLNqhn3NsCRu8CJ7Jvbg9EmusX5XqbyWcwND5qImzwvhHRjwmFccqtHJKFN+H0qJqSTUoao8iLGbstTYI4Ud5DqLYweEPPdZBYnZCCunXydjuPs0l1g7fnoMcwRgc2WFPGNahs2baPj7nCkS3jz37ArEJ9nBigC8XX0zu8iX4Qc+n4LUp/N1yHsNxftO0lTPOJhKIE9u2YLHzM3w3aHGWy2Bc6/gyzelhWooLvQHAr3+cXEMEr41s/TSi9lQPsdln69cx97Bdr8FIwffB7qjWGDRvmcUH7KQAvaY9QtCjTqJbX0a8xEamWv4RHTX6g9GlfH3x7FVCGqkyyo4nFT497ky5gbiqlyn8IFKGKREtVo1jjPJMRUdvnad+7ZVtmg8AoXwP3zAiG4l1pEnhkPIiwajzHWwiCzrI4RLo5UzA3RTJeCCJ00XM5wpTlaf4UCEeSg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We require a writable PTE and only support anonymous folio: we can only have exactly one PTE pointing at that page, which we can just lookup using a folio walk, avoiding the rmap walk and the anon VMA lock. So let's stop doing an rmap walk and perform a folio walk instead, so we can easily just modify a single PTE and avoid relying on rmap/mapcounts. We now effectively work on a single PTE instead of multiple PTEs of a large folio, allowing for conversion of individual PTEs from non-exclusive to device-exclusive -- note that the other way always worked on single PTEs. We can drop the MMU_NOTIFY_EXCLUSIVE MMU notifier call and document why that is not required: GUP will already take care of the MMU_NOTIFY_EXCLUSIVE call if required (there is already a device-exclusive entry) when not finding a present PTE and having to trigger a fault and ending up in remove_device_exclusive_entry(). Note that the PTE is always writable, and we can always create a writable-device-exclusive entry. With this change, device-exclusive is fully compatible with THPs / large folios. We still require PMD-sized THPs to get PTE-mapped, and supporting PMD-mapped THP (without the PTE-remapping) is a different endeavour that might not be worth it at this point. This gets rid of the "folio_mapcount()" usage and let's us fix ordinary rmap walks (migration/swapout) next. Spell out that messing with the mapcount is wrong and must be fixed. Signed-off-by: David Hildenbrand --- mm/rmap.c | 188 ++++++++++++++++-------------------------------------- 1 file changed, 55 insertions(+), 133 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 676df4fba5b0..49ffac6d27f8 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2375,131 +2375,6 @@ void try_to_migrate(struct folio *folio, enum ttu_flags flags) } #ifdef CONFIG_DEVICE_PRIVATE -struct make_exclusive_args { - struct mm_struct *mm; - unsigned long address; - void *owner; - bool valid; -}; - -static bool page_make_device_exclusive_one(struct folio *folio, - struct vm_area_struct *vma, unsigned long address, void *priv) -{ - struct mm_struct *mm = vma->vm_mm; - DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0); - struct make_exclusive_args *args = priv; - pte_t pteval; - struct page *subpage; - bool ret = true; - struct mmu_notifier_range range; - swp_entry_t entry; - pte_t swp_pte; - pte_t ptent; - - mmu_notifier_range_init_owner(&range, MMU_NOTIFY_EXCLUSIVE, 0, - vma->vm_mm, address, min(vma->vm_end, - address + folio_size(folio)), - args->owner); - mmu_notifier_invalidate_range_start(&range); - - while (page_vma_mapped_walk(&pvmw)) { - /* Unexpected PMD-mapped THP? */ - VM_BUG_ON_FOLIO(!pvmw.pte, folio); - - ptent = ptep_get(pvmw.pte); - if (!pte_present(ptent)) { - ret = false; - page_vma_mapped_walk_done(&pvmw); - break; - } - - subpage = folio_page(folio, - pte_pfn(ptent) - folio_pfn(folio)); - address = pvmw.address; - - /* Nuke the page table entry. */ - flush_cache_page(vma, address, pte_pfn(ptent)); - pteval = ptep_clear_flush(vma, address, pvmw.pte); - - /* Set the dirty flag on the folio now the pte is gone. */ - if (pte_dirty(pteval)) - folio_mark_dirty(folio); - - /* - * Check that our target page is still mapped at the expected - * address. - */ - if (args->mm == mm && args->address == address && - pte_write(pteval)) - args->valid = true; - - /* - * Store the pfn of the page in a special migration - * pte. do_swap_page() will wait until the migration - * pte is removed and then restart fault handling. - */ - if (pte_write(pteval)) - entry = make_writable_device_exclusive_entry( - page_to_pfn(subpage)); - else - entry = make_readable_device_exclusive_entry( - page_to_pfn(subpage)); - swp_pte = swp_entry_to_pte(entry); - if (pte_soft_dirty(pteval)) - swp_pte = pte_swp_mksoft_dirty(swp_pte); - if (pte_uffd_wp(pteval)) - swp_pte = pte_swp_mkuffd_wp(swp_pte); - - set_pte_at(mm, address, pvmw.pte, swp_pte); - - /* - * There is a reference on the page for the swap entry which has - * been removed, so shouldn't take another. - */ - folio_remove_rmap_pte(folio, subpage, vma); - } - - mmu_notifier_invalidate_range_end(&range); - - return ret; -} - -/** - * folio_make_device_exclusive - Mark the folio exclusively owned by a device. - * @folio: The folio to replace page table entries for. - * @mm: The mm_struct where the folio is expected to be mapped. - * @address: Address where the folio is expected to be mapped. - * @owner: passed to MMU_NOTIFY_EXCLUSIVE range notifier callbacks - * - * Tries to remove all the page table entries which are mapping this - * folio and replace them with special device exclusive swap entries to - * grant a device exclusive access to the folio. - * - * Context: Caller must hold the folio lock. - * Return: false if the page is still mapped, or if it could not be unmapped - * from the expected address. Otherwise returns true (success). - */ -static bool folio_make_device_exclusive(struct folio *folio, - struct mm_struct *mm, unsigned long address, void *owner) -{ - struct make_exclusive_args args = { - .mm = mm, - .address = address, - .owner = owner, - .valid = false, - }; - struct rmap_walk_control rwc = { - .rmap_one = page_make_device_exclusive_one, - .done = folio_not_mapped, - .anon_lock = folio_lock_anon_vma_read, - .arg = &args, - }; - - rmap_walk(folio, &rwc); - - return args.valid && !folio_mapcount(folio); -} - /** * make_device_exclusive() - Mark an address for exclusive use by a device * @mm: mm_struct of associated target process @@ -2530,9 +2405,12 @@ static bool folio_make_device_exclusive(struct folio *folio, struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, void *owner, struct folio **foliop) { - struct folio *folio; + struct folio *folio, *fw_folio; + struct vm_area_struct *vma; + struct folio_walk fw; struct page *page; - long npages; + swp_entry_t entry; + pte_t swp_pte; mmap_assert_locked(mm); @@ -2540,12 +2418,16 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, * Fault in the page writable and try to lock it; note that if the * address would already be marked for exclusive use by the device, * the GUP call would undo that first by triggering a fault. + * + * If any other device would already map this page exclusively, the + * fault will trigger a conversion to an ordinary + * (non-device-exclusive) PTE and issue a MMU_NOTIFY_EXCLUSIVE. */ - npages = get_user_pages_remote(mm, addr, 1, - FOLL_GET | FOLL_WRITE | FOLL_SPLIT_PMD, - &page, NULL); - if (npages != 1) - return ERR_PTR(npages); + page = get_user_page_vma_remote(mm, addr, + FOLL_GET | FOLL_WRITE | FOLL_SPLIT_PMD, + &vma); + if (IS_ERR(page)) + return page; folio = page_folio(page); if (!folio_test_anon(folio) || folio_test_hugetlb(folio)) { @@ -2558,11 +2440,51 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, return ERR_PTR(-EBUSY); } - if (!folio_make_device_exclusive(folio, mm, addr, owner)) { + /* + * Let's do a second walk and make sure we still find the same page + * mapped writable. If we don't find what we expect, we will trigger + * GUP again to fix it up. Note that a page of an anonymous folio can + * only be mapped writable using exactly one page table mapping + * ("exclusive"), so there cannot be other mappings. + */ + fw_folio = folio_walk_start(&fw, vma, addr, 0); + if (fw_folio != folio || fw.page != page || + fw.level != FW_LEVEL_PTE || !pte_write(fw.pte)) { + if (fw_folio) + folio_walk_end(&fw, vma); folio_unlock(folio); folio_put(folio); return ERR_PTR(-EBUSY); } + + /* Nuke the page table entry so we get the uptodate dirty bit. */ + flush_cache_page(vma, addr, page_to_pfn(page)); + fw.pte = ptep_clear_flush(vma, addr, fw.ptep); + + /* Set the dirty flag on the folio now the pte is gone. */ + if (pte_dirty(fw.pte)) + folio_mark_dirty(folio); + + /* + * Store the pfn of the page in a special device-exclusive non-swap pte. + * do_swap_page() will trigger the conversion back while holding the + * folio lock. + */ + entry = make_writable_device_exclusive_entry(page_to_pfn(page)); + swp_pte = swp_entry_to_pte(entry); + if (pte_soft_dirty(fw.pte)) + swp_pte = pte_swp_mksoft_dirty(swp_pte); + /* The pte is writable, uffd-wp does not apply. */ + set_pte_at(mm, addr, fw.ptep, swp_pte); + + /* + * TODO: The device-exclusive non-swap PTE holds a folio reference but + * does not count as a mapping (mapcount), which is wrong and must be + * fixed, otherwise RMAP walks don't behave as expected. + */ + folio_remove_rmap_pte(folio, page, vma); + + folio_walk_end(&fw, vma); *foliop = folio; return page; } From patchwork Wed Jan 29 11:54:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13953679 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77894C02193 for ; Wed, 29 Jan 2025 11:54:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 838BA280016; Wed, 29 Jan 2025 06:54:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7BAB0280017; Wed, 29 Jan 2025 06:54:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5F261280016; Wed, 29 Jan 2025 06:54:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 3992A280017 for ; Wed, 29 Jan 2025 06:54:46 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 7E6521A0C51 for ; Wed, 29 Jan 2025 11:54:33 +0000 (UTC) X-FDA: 83060332026.02.C06FC83 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 38B2640007 for ; Wed, 29 Jan 2025 11:54:31 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=M0txsj2m; spf=pass (imf12.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738151671; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/5TTv0Iu+aL+vrfHcl/JjDhmk1gN271wttPyRyQHADk=; b=JxoZOX/GfaAdubQXYpmMDkpSO2qt7gT2tHt88vLhOm+rNM6SiH/UDzyqYRDuBGNtfDNcsH G1f+q2Ao+X6/wCrfNsFrT5ZDY2WtI83idFSZF/oAMGh7JdS0i5MfHxjygbAM4RkFHypt7m HdRkCuIh0Xx06YQh8OzQyu2ufIE7TO8= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=M0txsj2m; spf=pass (imf12.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738151671; a=rsa-sha256; cv=none; b=SYITbftiFIFNd9rX4jyaFDdh9Cpt9jtz6v4iDE80i+8AhXvBUSX/iVQ6eMSQ29Yusqpqf2 Tdz8SWA/CztiIeHiv5Lnhh/DO9gzbwD99wVumwqJUjblK+eICNmCovLXatagHxPJ9nMf1B 38qgz6hi3+hfXiEx4tXWpt/baDy+gdc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1738151670; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/5TTv0Iu+aL+vrfHcl/JjDhmk1gN271wttPyRyQHADk=; b=M0txsj2mnKZA2JnGX+StEY7X9CFVO6LK7mzDxUKl8pRpwGU9xs8hUuQ8NAO0Cr8CpLGZRI QHQJqQgxSBWG2sOicDFIaHdMhTzbB25q5tSrAbYvC+CQguBpx5qlcdqm6KjcYCYftAtxXu 7kE8AgBNEr8ehKssuQaBdjt3xxksBps= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-549-5w24BLl9N6SAHXFOOunXfA-1; Wed, 29 Jan 2025 06:54:29 -0500 X-MC-Unique: 5w24BLl9N6SAHXFOOunXfA-1 X-Mimecast-MFC-AGG-ID: 5w24BLl9N6SAHXFOOunXfA Received: by mail-wr1-f70.google.com with SMTP id ffacd0b85a97d-38a2140a400so5195812f8f.0 for ; Wed, 29 Jan 2025 03:54:29 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738151668; x=1738756468; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/5TTv0Iu+aL+vrfHcl/JjDhmk1gN271wttPyRyQHADk=; b=cwTanqLl5kN7PeR0jPhvyYv3ZG0Vg9a71e3SxF02yzfSD2TMhJxbyJ9M+6XbAbZL3m c5GJbILBK6pdDALyC6comz60udgYH6WSSyj9PuTpxCnMK1Gbx8ZO4mJYAvra2keKuiG5 bbH1xFj+RyHBvV/7BAeo8tsCnVvucAlEuLc3wFofIdQptoFjjlRJ8tkVxi1qqXxc9wa1 tnsRYU3FUpbmbhf6tZoTOOJv4/qpyOXnxNfOE7hAFXfSOe8ZONsRxzo9YM9dhCg2FBIX Qf3h5lOJWpvwmJzotl6AhGWDNlwth9wAuuZ1O8mNpI9g7gLccPEckLQUSZs1Rh6t4uHJ r2Gw== X-Forwarded-Encrypted: i=1; AJvYcCU3TRq0/3wvG+GT19Z/fetHB1NzWuEBc9sXdU6NW/LVuFp8xzI/yrL8tZPHIoDAu5+a2stnm2zbyA==@kvack.org X-Gm-Message-State: AOJu0YyOsS2rP+vCjKgCF36VMQ7sQVQvxDNdg/JEikNLyG8bHS1AnoFF j6sq3osdndGNWR3HJxRDILqjFcFMKP1CyWGDUfKVdHL67QxNJ6FQSzMkWoVCO6d6LoufsAck+b3 4XwOw+KYo7pLUmm+3h+FyTbdN18iT1FyVfFUnvXmrRSW24m/n X-Gm-Gg: ASbGncuPlAxCtfyM1hhdrBwQfPEIsVJsq9qTCOtATAhpRfLcku07itYigMgUF2cRT2J utjyL+VZcC+5XKCK3UoW7Pnednh8no69GG0gMhC7O1NVGBHeg/inmK5Lp41Xszjn73BbFNuMh/T JSlKc8hhwkxA6ewBNKfpwtzTsBs91h0ynsAKnGXFZhDEbJIM4aFmf+M4EDbEqNBfR6aiibXrqpk 85Ts/4tWHDG+5rnrQqhfyWknN1DW0aFEvBrCPDQXpDiakawVIers8tHEHg35LQ2Jvv6j9T1VpsT nj8bJbPsxpkfJBZTStOjifMkHAz9Ttk729jc3W9lVsGZDj5K0dwLoF89qVGHgAv61g== X-Received: by 2002:a05:6000:1fac:b0:386:3329:6a04 with SMTP id ffacd0b85a97d-38c51e8de63mr2549369f8f.39.1738151668337; Wed, 29 Jan 2025 03:54:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IH3OskkNotGDGa9hKk/bhinDJ94hVo+TUAo4zN3qwEB+ZEXq9Nn9aY72BgysSoJ/r2gZh7hQQ== X-Received: by 2002:a05:6000:1fac:b0:386:3329:6a04 with SMTP id ffacd0b85a97d-38c51e8de63mr2549347f8f.39.1738151667985; Wed, 29 Jan 2025 03:54:27 -0800 (PST) Received: from localhost (p200300cbc7053b0064b867195794bf13.dip0.t-ipconnect.de. [2003:cb:c705:3b00:64b8:6719:5794:bf13]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38c2a188fe8sm17066981f8f.56.2025.01.29.03.54.26 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 03:54:27 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v1 05/12] mm/memory: detect writability in restore_exclusive_pte() through can_change_pte_writable() Date: Wed, 29 Jan 2025 12:54:03 +0100 Message-ID: <20250129115411.2077152-6-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250129115411.2077152-1-david@redhat.com> References: <20250129115411.2077152-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: GqJtNwJtgYB5ooHkjTThEEviCb-Vyv-jHgG11pzC_GE_1738151668 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 38B2640007 X-Stat-Signature: zkwzf6qgogssdfpefb48gwugiehzt1xp X-Rspam-User: X-HE-Tag: 1738151671-915446 X-HE-Meta: U2FsdGVkX1/1el0MaqAPn+fhL45sQ+q809U/n11xydmDjVC+aWdtFVNDFtX8gfwEjW9Pmr0emGyCPBzUiUCvZsYMxht1PfDouwj7/X61B63yaredhRAV78uA95Jv4Mc4Hra2nroanEvYtLuFSgQnUVjsyMd+zJLUJlbnURD4ANaAijjhOnMnGrPj9B92zWSkNNNmqgYxYHTiR71Y+fEKie1T2OseiCJiG8+/dSUoQ5a0uNhEDNrsINXcvLPH/VoJMyi7TIBrxpLdNuEcTtSgxTF9MkwdzMPM5ebfSYlbRufPspOXESBmwYFfo2bwNNSlW8eLOMQUv/RK0cZNZAuWkkSGWTcDyYG7l/+iEj+KzVnDfHHmQ92q6bAAMddUemlel+9dOTkI1GaFYT4pHQRPTc3W058bmEnuz35LCsDI4GYoBExmTeQtocghOywoLEXjwLGBnHw8+qqJcMDRIrBGr30+QSGw4kwgOr7LQHzIGZgTp7QKPOZT92xMtv01Y6xcUsWcuAbXgiMKlza11TH0Qo+cBCb2paWP4pxPpmMmx4qq8djWvpAwrgRMSAAknUJRy4YUrYbOrtSM2DbFT80PzKyRVsnmTgHtdab2PG2nrAIqUqkXFwI7Mqx1AxYy/Lugv1Y/jA2ZyPl9SpUdOiCrWjNQMigC3FAscSUG0BmjCvpv7mlVMbKaDjdZMpde/zm6Kouk9ileaXQs2SeIydgtKr8orl16Fwi1Ul8zyTzZJQfEtfJNfST43P0WGR80D8Pc+RzANb64krGD+FrHHM+wzj+G4EGkWNHX1mKtJv9NwNIJfoTI3NUU5xE/WYhFClz+FYgrwZetkk8rWlNAUs9FXZ6AJUy8JV2RNROjltnfAQWIpo51gTbVKo6Cjp/qKGRvCfwqcZrVFF82V8cjuq0yOALS3sT+gIrI9AUpGRBEaIo3aMwxUrVeOpKcX+sMcpDO9fN5MXBNDPKDDv11+K2 XLCVf7Mo 8H3U8LTmnYDzfag9JYWhxLO69tSLu0XDq8Yt0xoPH40U3ko1FTVsqAN1gRG69u+LJJ5ifNDrlyvD7lxfeXaqnpm4GIGKGIe0NZQf9l7CQJ0sEjBRqnw14ubAjv3MXgtBSUf1nDAyNt1jANF9jJRwzidtqqE/RL94O8TDVtjvt0jDHehUDtGWrAQJJwByZ5fTSfDqna+wFsM2Knw4Yzviu7CUV57TT82AiSSNwiAHmLZJqCw+N/+Hcz/aBbL5jgPsOW6YU/rNyzdsoMDLe07fVS6Qs6CdYErSFYVelWKcZCnOLwTtEnFiHQxpRo7TE4Qg+jYbvgFEroGYTsdi53ytax0Qkxd/Lq2L8RHHJUnjgAzIxUq0L7QIgYZ2N7ah/hp71DJlS2bJZYgIMwR/6BwZop0CwVcdbxXPfHVAdPTfFYDVKBtXszltgcmQJ+SY9kjrTEUwHNlvuCLw7t2D59I7DiKIH+yRSAbyHUhTU1q86SNnLUJjIv1EIZiQ7lAslK9p5s7A6xly7e59l8/sUX8i7KtZAnw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's do it just like mprotect write-upgrade or during NUMA-hinting faults on PROT_NONE PTEs: detect if the PTE can be writable by using can_change_pte_writable(). Set the PTE only dirty if the folio is dirty: we might not necessarily have a write access, and setting the PTE writable doesn't require setting the PTE dirty. With this change in place, there is no need to have separate readable and writable device-exclusive entry types, and we'll merge them next separately. Note that, during fork(), we first convert the device-exclusive entries back to ordinary PTEs, and we only ever allow conversion of writable PTEs to device-exclusive -- only mprotect can currently change them to readable-device-exclusive. Consequently, we always expect PageAnonExclusive(page)==true and can_change_pte_writable()==true, unless we are dealing with soft-dirty tracking or uffd-wp. But reusing can_change_pte_writable() for now is cleaner. Signed-off-by: David Hildenbrand --- mm/memory.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 03efeeef895a..db38d6ae4e74 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -725,18 +725,21 @@ static void restore_exclusive_pte(struct vm_area_struct *vma, struct folio *folio = page_folio(page); pte_t orig_pte; pte_t pte; - swp_entry_t entry; orig_pte = ptep_get(ptep); pte = pte_mkold(mk_pte(page, READ_ONCE(vma->vm_page_prot))); if (pte_swp_soft_dirty(orig_pte)) pte = pte_mksoft_dirty(pte); - entry = pte_to_swp_entry(orig_pte); if (pte_swp_uffd_wp(orig_pte)) pte = pte_mkuffd_wp(pte); - else if (is_writable_device_exclusive_entry(entry)) - pte = maybe_mkwrite(pte_mkdirty(pte), vma); + + if ((vma->vm_flags & VM_WRITE) && + can_change_pte_writable(vma, address, pte)) { + if (folio_test_dirty(folio)) + pte = pte_mkdirty(pte); + pte = pte_mkwrite(pte, vma); + } VM_BUG_ON_FOLIO(pte_write(pte) && (!folio_test_anon(folio) && PageAnonExclusive(page)), folio); From patchwork Wed Jan 29 11:54:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13953678 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8A24C02190 for ; Wed, 29 Jan 2025 11:54:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 26BE7280015; Wed, 29 Jan 2025 06:54:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F439280016; Wed, 29 Jan 2025 06:54:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F0EB5280015; Wed, 29 Jan 2025 06:54:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id CB1AE280011 for ; Wed, 29 Jan 2025 06:54:45 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id F22161A0C6C for ; Wed, 29 Jan 2025 11:54:36 +0000 (UTC) X-FDA: 83060332152.20.968791F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf13.hostedemail.com (Postfix) with ESMTP id C27B220004 for ; Wed, 29 Jan 2025 11:54:34 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LKUWO0PY; spf=pass (imf13.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738151674; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bwRdkJe7S5eVWCBWfyg7xh/8u+L2oGZrd41rkCVbduE=; b=K/SQFngVNcaFZFdpie/zDU8uFDLgBky8JwDkh0PfSml70U2/lyrBv6zZjI37/Rx4aKHeBA Tk0U4dWME4lIBk705KDvLPtS1xYU4vBO7cw3qoRT113bZ/4whOrbQ9al1cuiBQq/UkXRQs 67k3e7KkuVtsYt1vJCwX0LYrV7NU0Vc= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LKUWO0PY; spf=pass (imf13.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738151674; a=rsa-sha256; cv=none; b=XJsi7VvUyUUA4HuQKzIfWlgaiKFMT2N67eHqYQ5tYsXUGWh/jTrnwSkOTa+GUgG3A/v5GD r8Ezu+WWbgJtaKgbmPmq4BENq9bdKUdgRBpphdJdeIR8b+2unanM/6SmZRE35wGnMzjnZP mvuEW43IegW60Unjcsw1q6nTovVtLeI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1738151674; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bwRdkJe7S5eVWCBWfyg7xh/8u+L2oGZrd41rkCVbduE=; b=LKUWO0PYGzc9dUy1UErMnP8OYzjxyiNMxuix/cKwbPyIYHzTWSCZCK/Ib22baSZ9WegrhN ldpZLa/hVFB859ju/s4ficWmRE3y6m/T5i8guJeslw1W7X2ECh7t6EfQIgmSCK2Y2tpH3O MtgNE6Qmg5ZnLmmDXl36+UQx08cRwOI= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-501-3XGaISYtNHOiwu_mg_4GPA-1; Wed, 29 Jan 2025 06:54:32 -0500 X-MC-Unique: 3XGaISYtNHOiwu_mg_4GPA-1 X-Mimecast-MFC-AGG-ID: 3XGaISYtNHOiwu_mg_4GPA Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-43651b1ba8aso46674425e9.1 for ; Wed, 29 Jan 2025 03:54:32 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738151672; x=1738756472; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bwRdkJe7S5eVWCBWfyg7xh/8u+L2oGZrd41rkCVbduE=; b=vKItgidUhEzil73THPnLCmXNbg4G4GecLIF3D37P33xT+FwPgfLmtdHrmv9qAVYs2y A45EVuXMAiUFEeFvjxsTN1EAEf6j3a2vff6xhHvFG3/rGUA7+CX0A2suffc9l81a1OxH vdvVQQn4V1KDdGvnB0zonjjxQAFNPIYLmmA3DwvcgtEgYSULz42jQk7IJQScNsgTjqz6 yYFN/7j51wPzNWxQMqYH4W1bVQlLYaiiGYT3T47Zk2bqdMznVEqp3yhaMy9QSC8KHbuk IFAOtsd+dIt7wjfwAcJai8LHLrCMx+MMf+bCG4CdLTHDJtwbBg4Hr10oHyAPSHk06XtB GQqg== X-Forwarded-Encrypted: i=1; AJvYcCXjr2s8tz2Bonky+W/DLK66WYbJU7vHP/AF7a2Hxw5mxDZ8obpaMNjCMVP6vXKi7eBYZPpci9I1qQ==@kvack.org X-Gm-Message-State: AOJu0Yy+4pkrAy7F4nNuNCnZCU64jQz4jtnky+q4CjjDmE6pBxICU6ql fHxj0ca9e2RfyT5RvFA1x7xVV3HQxZ/x6kJMUQ2+8GJ0vZ9txFEWU6MY4T3UkXYKcXhVSmKJaEU ZytNJyfvLIOZO+oZdEGyyx94KCLmU4cFIoiTU6ZdsZY7VA04G X-Gm-Gg: ASbGncsEn2Aj1Xtr9SENGsoIlTdmaNZhh1Pn28t2ufY+3c/yq7d/btb22kJXlbM2a61 e13n9DfaLx2jeem5KAfOLjTAuROW3rl6zUdUJ52p49hXrp1//w28UlE6sDxKbAP6rVoaStOF3lE 0yNdUbWEV0QRWYLLAWbdFfQHVJQVlsgA4rvV2iE8e6QaN5geHao74yiy9Piscm7o2xUJt1JDgDA 3cdzaq+TlBB0eMYMKcwFfxh6N2upC+aH1OZiwlMRgt3o0kpBmbPQuwU7Sq5rTHUkzxa9++557AV UiicUe21NYcpd8PI0bZws51tswibjO7yTOeMZT3nf8aRuJG0vfMGN5yobY17vNME0w== X-Received: by 2002:a05:600c:1c1a:b0:434:f9c4:a850 with SMTP id 5b1f17b1804b1-438dc3c4623mr25664845e9.10.1738151671649; Wed, 29 Jan 2025 03:54:31 -0800 (PST) X-Google-Smtp-Source: AGHT+IEhuPkSFqdGYqW92VDVvy4lKssWj+By02GdvgYFn3sg13RZ9e45UT5b96yUwTD+A2GCuT4qDg== X-Received: by 2002:a05:600c:1c1a:b0:434:f9c4:a850 with SMTP id 5b1f17b1804b1-438dc3c4623mr25664525e9.10.1738151671268; Wed, 29 Jan 2025 03:54:31 -0800 (PST) Received: from localhost (p200300cbc7053b0064b867195794bf13.dip0.t-ipconnect.de. [2003:cb:c705:3b00:64b8:6719:5794:bf13]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-438dcc24e73sm20192895e9.16.2025.01.29.03.54.28 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 03:54:30 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v1 06/12] mm: use single SWP_DEVICE_EXCLUSIVE entry type Date: Wed, 29 Jan 2025 12:54:04 +0100 Message-ID: <20250129115411.2077152-7-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250129115411.2077152-1-david@redhat.com> References: <20250129115411.2077152-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: HFEw9AJyuhojI5F5QYZl-dixnryWrDMPAiFZfUGD2ms_1738151672 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: C27B220004 X-Stat-Signature: wo59ksm79mnxhwei59y8kxkeaquep3yh X-HE-Tag: 1738151674-76876 X-HE-Meta: U2FsdGVkX198GFHpjackiEiF4CLXh/UkvRftPslTj3KbOSXACseDKPg6YuyaNRtZ5IDplq5VdFP9nnYBrH9x/kfimmqd5PuJspFISrnLsSSN6EN1pqfsM+RYu6HZfQxYF9gTvm3wcZWIEbZixMMIwi4YVrdWhL+e5DOYiI2g3tzNxNA4IYDMHtHu1lCw3FSXO5xRoU0Ma657m+bbDrLD+V4ghTYNRaVRS5DgppehcEzuRXLN/V0bZT8L4kHxR7XatJEHPOicHbQzfxgXNh5ZQdtU6kN/lAxHQrXEPFP6/vkpVe6P1M77vSmbqr7VAIpAkfO1u5q4cnW2Ex/VqQKCCh5vuX9DN4zXbp20vXMHepu2x+ccpEHiACepMgpuWDEau71G5+H/jeivSVDyh2J0VO8HGE9kZnIZE1hu1UlfvdK8D17vLws4Z6hRz5QUZvpQC5tJnnIm/r53UadILWGva92whOhTcad1Cwy/XguNa9TcgSb2Vm6ep4qR+PsTrAWNcYW5CPk1MJjZLVNt13OVBTvzjbl0LdY2zwUHn9WUTp2IO8ItoQdeQoTHWbRZxZLpWo2bwQyPazQ4FaId+aQo+zBLfotA86LhOD40g9Bqx7x31UjWDcHD/0kEfDYIrXNbRpkl48XA+B994trSdrhX1xFrfv3tFKC5P/JTgWkJ9sx3npoUbs5MN2pVtc8R9/pI79G87r47r8NBqj1DDq2XHKm/WreU8pUQnb/TjqmtwNyf2z2/Nl5ggTqbJJKY9PQxmSW8QHaJa6MVr8BoFGmCCnrQ0X02ktWUQYbDPqnKD08Y8lSzhuThA21JrUyFmXojLmI0JqWALXx2HtZoLrbME4VS1NPZuBWK3ufX6IK99X5ALgNXw6KD2y0MbK59F23YL9Cl1uF2KJbklt9sP+hc48E0nCPl2OQGXJ9WT88XrHVo7mk4E2pBnSnzPD20ku9vZlZOftoJ4mfczxnpFqY V8CVcn0G 30t+pRMhU7JbbAZiixsBRhiI9WiMuvJ/YIesjSFRTUiETUW6PyU3bgx6lUYHqgtrphAd5DwnRfE9zo5oreM571jZ63mLIeTrd0CIwwjvm8hXF/clZ7JXjPzn0IvHaUvyxAZ+pOKJtdkuCsH1S4dXycCZ7evwE7i36Mpo97rAyXG2Di2nTzyuqzOoL2UDPwYDaTS0qs3vIhzJouP2juqvcmNT49an8JaM2mEXPI20ciycHClchJiI8BU7kwttQO3U8CWMhkfK+6bcaLrdbYHEuFmAOofVpUD6+pVCvABEx/zo9EvBtHdvJ5XGirr0s1RavSSyOcgoAzAVwvYevom4aTSSyfCqTaQ9LtTheEyjenhx1hpsQChmeTwatsxBTgyCvch6mewNVNsCwAuAW0PxIHRek7fDxMLFTgpdSPiaLSHMNhTAvnG+dkSWAkyk7kr0KH96JSxivX4zJ0mGqR0lE02BUKn4E+S9UyxOG6rFWo3TdQ0+4+PFeAi115Vm47q6kfHWT/yk08WNh68wjRpAdQLDADA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There is no need for the distinction anymore; let's merge the readable and writable device-exclusive entries into a single device-exclusive entry type. Signed-off-by: David Hildenbrand Acked-by: Simona Vetter Reviewed-by: Alistair Popple --- include/linux/swap.h | 7 +++---- include/linux/swapops.h | 27 ++++----------------------- mm/mprotect.c | 8 -------- mm/page_table_check.c | 5 ++--- mm/rmap.c | 2 +- 5 files changed, 10 insertions(+), 39 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 91b30701274e..9a48e79a0a52 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -74,14 +74,13 @@ static inline int current_is_kswapd(void) * to a special SWP_DEVICE_{READ|WRITE} entry. * * When a page is mapped by the device for exclusive access we set the CPU page - * table entries to special SWP_DEVICE_EXCLUSIVE_* entries. + * table entries to a special SWP_DEVICE_EXCLUSIVE entry. */ #ifdef CONFIG_DEVICE_PRIVATE -#define SWP_DEVICE_NUM 4 +#define SWP_DEVICE_NUM 3 #define SWP_DEVICE_WRITE (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM) #define SWP_DEVICE_READ (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+1) -#define SWP_DEVICE_EXCLUSIVE_WRITE (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+2) -#define SWP_DEVICE_EXCLUSIVE_READ (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+3) +#define SWP_DEVICE_EXCLUSIVE (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+2) #else #define SWP_DEVICE_NUM 0 #endif diff --git a/include/linux/swapops.h b/include/linux/swapops.h index 96f26e29fefe..64ea151a7ae3 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -186,26 +186,16 @@ static inline bool is_writable_device_private_entry(swp_entry_t entry) return unlikely(swp_type(entry) == SWP_DEVICE_WRITE); } -static inline swp_entry_t make_readable_device_exclusive_entry(pgoff_t offset) +static inline swp_entry_t make_device_exclusive_entry(pgoff_t offset) { - return swp_entry(SWP_DEVICE_EXCLUSIVE_READ, offset); -} - -static inline swp_entry_t make_writable_device_exclusive_entry(pgoff_t offset) -{ - return swp_entry(SWP_DEVICE_EXCLUSIVE_WRITE, offset); + return swp_entry(SWP_DEVICE_EXCLUSIVE, offset); } static inline bool is_device_exclusive_entry(swp_entry_t entry) { - return swp_type(entry) == SWP_DEVICE_EXCLUSIVE_READ || - swp_type(entry) == SWP_DEVICE_EXCLUSIVE_WRITE; + return swp_type(entry) == SWP_DEVICE_EXCLUSIVE; } -static inline bool is_writable_device_exclusive_entry(swp_entry_t entry) -{ - return unlikely(swp_type(entry) == SWP_DEVICE_EXCLUSIVE_WRITE); -} #else /* CONFIG_DEVICE_PRIVATE */ static inline swp_entry_t make_readable_device_private_entry(pgoff_t offset) { @@ -227,12 +217,7 @@ static inline bool is_writable_device_private_entry(swp_entry_t entry) return false; } -static inline swp_entry_t make_readable_device_exclusive_entry(pgoff_t offset) -{ - return swp_entry(0, 0); -} - -static inline swp_entry_t make_writable_device_exclusive_entry(pgoff_t offset) +static inline swp_entry_t make_device_exclusive_entry(pgoff_t offset) { return swp_entry(0, 0); } @@ -242,10 +227,6 @@ static inline bool is_device_exclusive_entry(swp_entry_t entry) return false; } -static inline bool is_writable_device_exclusive_entry(swp_entry_t entry) -{ - return false; -} #endif /* CONFIG_DEVICE_PRIVATE */ #ifdef CONFIG_MIGRATION diff --git a/mm/mprotect.c b/mm/mprotect.c index 516b1d847e2c..9cb6ab7c4048 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -225,14 +225,6 @@ static long change_pte_range(struct mmu_gather *tlb, newpte = swp_entry_to_pte(entry); if (pte_swp_uffd_wp(oldpte)) newpte = pte_swp_mkuffd_wp(newpte); - } else if (is_writable_device_exclusive_entry(entry)) { - entry = make_readable_device_exclusive_entry( - swp_offset(entry)); - newpte = swp_entry_to_pte(entry); - if (pte_swp_soft_dirty(oldpte)) - newpte = pte_swp_mksoft_dirty(newpte); - if (pte_swp_uffd_wp(oldpte)) - newpte = pte_swp_mkuffd_wp(newpte); } else if (is_pte_marker_entry(entry)) { /* * Ignore error swap entries unconditionally, diff --git a/mm/page_table_check.c b/mm/page_table_check.c index 509c6ef8de40..c2b3600429a0 100644 --- a/mm/page_table_check.c +++ b/mm/page_table_check.c @@ -196,9 +196,8 @@ EXPORT_SYMBOL(__page_table_check_pud_clear); /* Whether the swap entry cached writable information */ static inline bool swap_cached_writable(swp_entry_t entry) { - return is_writable_device_exclusive_entry(entry) || - is_writable_device_private_entry(entry) || - is_writable_migration_entry(entry); + return is_writable_device_private_entry(entry) || + is_writable_migration_entry(entry); } static inline void page_table_check_pte_flags(pte_t pte) diff --git a/mm/rmap.c b/mm/rmap.c index 49ffac6d27f8..65d9bbea16d0 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2470,7 +2470,7 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, * do_swap_page() will trigger the conversion back while holding the * folio lock. */ - entry = make_writable_device_exclusive_entry(page_to_pfn(page)); + entry = make_device_exclusive_entry(page_to_pfn(page)); swp_pte = swp_entry_to_pte(entry); if (pte_soft_dirty(fw.pte)) swp_pte = pte_swp_mksoft_dirty(swp_pte); From patchwork Wed Jan 29 11:54:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13953683 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22F0EC02190 for ; Wed, 29 Jan 2025 11:55:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9938E280035; Wed, 29 Jan 2025 06:55:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 94371280025; Wed, 29 Jan 2025 06:55:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 60FC3280035; Wed, 29 Jan 2025 06:55:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 30583280025 for ; Wed, 29 Jan 2025 06:55:04 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B8065C0C9C for ; Wed, 29 Jan 2025 11:54:39 +0000 (UTC) X-FDA: 83060332278.08.4C32FE5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 7DF5A180007 for ; Wed, 29 Jan 2025 11:54:37 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CIPZkaeh; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf06.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738151677; a=rsa-sha256; cv=none; b=wSFimgJgu4fEfZKId8178k2nq0AOacDFfvAR7lcwZUgZRKW1wrZ/9jbfkpOCeoOTzmfU0l oUtueNAQlxvljccRHcOVpyr4eMN7OBdAIf6y98a8ANf6bOYtXOYWf5WLFyrkvuDX67iOVG Leuw8paOKmVJK3ZD51uDQMNsq/qMFi4= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CIPZkaeh; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf06.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738151677; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SXa0V5qonctJf7EFSiCmTQng5dHGkyNA7f/zw6SOSBk=; b=cdJg64M6GZll1i5os4br/SxYG3kuLVsl+VMT3vvVW+oI/sAE8Ggb4xvxXhKUgiSAatt+Gt noyBx0mx6xlXlqOwtH3Y4NfYP8jISS6FtOwtX1pP782sjsh2YdwtDAnrbCcHSWaPCaMLEn DjGUK44bEZOAvHC9sdZ8mDySRzOHUBc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1738151676; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SXa0V5qonctJf7EFSiCmTQng5dHGkyNA7f/zw6SOSBk=; b=CIPZkaehG2lV94NEzq/CCGbNhSua8eIcjJCDRGNpUTOWgLUqUOz1YXYpDgZTa/yxauvRr7 8+xR39MKTl1962QncOXOD9iqyHxgilsXa5OP7Jd2/Qab/MbaDVzH7Vmr8bHXBtW3W+cdn/ ktQSUgbqYIhOg8DidA6T/r2bs9OhKh4= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-37-jIv0dDi4O7iUMcCCIXR6bA-1; Wed, 29 Jan 2025 06:54:35 -0500 X-MC-Unique: jIv0dDi4O7iUMcCCIXR6bA-1 X-Mimecast-MFC-AGG-ID: jIv0dDi4O7iUMcCCIXR6bA Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-43646b453bcso34325835e9.3 for ; Wed, 29 Jan 2025 03:54:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738151674; x=1738756474; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SXa0V5qonctJf7EFSiCmTQng5dHGkyNA7f/zw6SOSBk=; b=nh6BRz1hmTRPmprs4FGwjmIjjzaeEIMV4KaDBo/NPW4t0IvNiJzfKBNPC3AyaHjJcf IDuA2ZRMQKHNv+mAXCx7eSC4pt+Fhom0vPLMOW20g8nvgXgOJDV5wJcWvx3t67pvkNuA on9bRP0uLbu071HJuMyc6lXRHWJt/hKsodxFLkL65pREaQJOYeqy7M17yGdVV0F3uQws V2ceaoSYbiUrJabd2jf4Qi52suZUZN2qbNqb2oj5tTEBRYoMK64XvuvJJ516vhk0KgJA Tgbl2ZmQDehXfSzyjjiv3p2j27PF9NEOfoYnAp1FHvvMGYKSyYlkaxllakv+xMyBDvKV e8iQ== X-Forwarded-Encrypted: i=1; AJvYcCXsPnM+dQ7mww+mEfcdQfZfuHYqhgjMd1dXKyu7+SrdLo7hUrK/h0XPVYYlx/LrOj5UtzzVESIRHw==@kvack.org X-Gm-Message-State: AOJu0Yz96ffiIO7nrU6zBZ/j81V7kE6O77D9Nqa0HTCd43L5R3CVsRFl l1LGMyTE/IaXVLbW23FJNUI9kb4eVGxkJo47atlkgdkdkOzkNgT89s6veOhG1rukE4WJnZW7qqN 5HIckcTC7iW1YF8kf0O7RNaukHDFOCN701QTB3FVE+rls3pP/nWuu7LwYiZ8= X-Gm-Gg: ASbGncsMGmHTPTkm6bfcbN+Ork0PKUQVcndPIZtZ2nj94Q6omGDmIbw0uyOJ/bG8dGo qJVCsT+VcQz4EcU68/tF912uBmvR58b0aXKt8IjkPN3uyH9pLGqwH65S3QeXjCORwTGvIwEpVTG Hy04MK21DC/EO3vpaoCCB2AvDw/Nx41KZZoW6XhrvpBp/iFCI1Vm0Lki4D4X7xLaK4vELEZA3pj uD7p/dEHI+ttHqkvziX+D1eME7knfc+WxbzugTpMIhwet6nACGbtvc32fZzVXOn0G0zLE2BVUhl fXD9fi2uG52Aw2WZxf4l0kq0Vc7Jnva3DlzuOnwDd6oEHTtuY40M7qqfnFz4RnLB7Q== X-Received: by 2002:a5d:61cd:0:b0:385:f7ef:a57f with SMTP id ffacd0b85a97d-38c519744d5mr2113628f8f.27.1738151674160; Wed, 29 Jan 2025 03:54:34 -0800 (PST) X-Google-Smtp-Source: AGHT+IGYVCfHEVHB5xz9x9tgAenDwJY8DuXR9neg40xsYB0gvwDn7frPxCs2ZGd0+jvQzBFPN5gg9A== X-Received: by 2002:a5d:61cd:0:b0:385:f7ef:a57f with SMTP id ffacd0b85a97d-38c519744d5mr2113590f8f.27.1738151673837; Wed, 29 Jan 2025 03:54:33 -0800 (PST) Received: from localhost (p200300cbc7053b0064b867195794bf13.dip0.t-ipconnect.de. [2003:cb:c705:3b00:64b8:6719:5794:bf13]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38c2a1bb040sm16943248f8f.67.2025.01.29.03.54.31 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 03:54:32 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v1 07/12] mm/page_vma_mapped: device-private entries are not migration entries Date: Wed, 29 Jan 2025 12:54:05 +0100 Message-ID: <20250129115411.2077152-8-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250129115411.2077152-1-david@redhat.com> References: <20250129115411.2077152-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: acZ3F-Y15ChdFSQJexe-arbGIlIXZ3NJ3OUDpPdYrUk_1738151674 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 7DF5A180007 X-Stat-Signature: 9ra594yicadwfisx1bg357eac66dzyuc X-Rspam-User: X-HE-Tag: 1738151677-302545 X-HE-Meta: U2FsdGVkX1+9XZltnTYuQcqtCrjNt2WZDklYV9Chk6erj8GfB/o/wtuDp5zLzkeKZSOwbS/UVUtSWJpMd+9x1Ybmxi8TK1QMyxReTf0MfuKrojHNFA6pyTT38aKIO3TDEMF0L7AcSlcxq8zceMJtAZ+/JtGpnI2Z65PBezWnuX1ngboXwzQMhIiMO11NNcIvUe4pluVt1zH9hbfL+E2Bmpk7sA3FPr2xWrhOr07m5NJmoqbHGzMIF/erKNU58ejpPCyAXXJQ4lLnqzaPjlUwdIDjpTv/gjFd2ETBuxfA5Yuqi9fetmP6qODwgvWa7SW/jggdDcPbNFrsxaLP1kQBNnYyH4+90iriuDFHrbYI1kzqQDTVnF6cpTrygQZ4hAltG0aOGH2j7TweSTgXQi2gF2eYgYpB3DumiMxmqMqOFzIwB6PWVXpjbnMQB/MJGqHKdk0dPt7mggEz1Ia/Pi8W6wR940+mronu4KjyGeu6Cb1tJ239h1CAXqBOP66874Snv6GSbuxjMCu1ghNBIxeohK3ywGFeKC+S0jFgr7bZ5uDAGK371TmB7eKapJI6QZTEF3RsPdKIcEWunKFShtQ/IEw6ESAncar44ufL02s/rcPAi0J/uRBhx/NYclSOq74DKbZnxoZMl22rl4np09PDrr6gcSUc/BIDhDL9J/ytRqqX4ckWFTXCWyK5IuU1cHiNB/D0RUU2KXEsDMgnKgMYjtS9gJ2dXPo+m91chtOJY6OY2BJiPPkH3pMbJ7u8tCNcFU37C92d2TPOOMl2f5xYJurfKTfiGprU1vjGmC7GmK4LXHkNuxLNM8TXZpJoiO2KVJU+9otDHrLUdZl/qQbkaqTiQsscF7dGDS9HHRcIijiTJf5VgXVLT6P7NznNq2wL/xaJx5sJdYuPE38pvm6ZaE93ZXRovQLnx/KSjTw3/7ER9dyPF8sVcIVrc+QotZ6j35QvZ6vE7MeSFLNcGbB KydzERko RS64QE+wrGR+z4PVtbkPV8SIscxoL85a8/KDvxpwkjTrnLJ3XqTvsmskZjaL0hj0AV4+WE+NPdFyKGM8atRR7lIZZqQUUdM1bIZfFoFzSe84R95lCbNTCEgrVaZY3nP399LqLjg08mPdCh40/8M5atQSmlzSWkPRs/s1A4SiHgmy8hti7Roc/1+7UoHD0efbOvrZ9YmciuVvH1h8/QRV3WBjL+N9bZRpk+T5FlairwnXQfllOZy3NKs3wJwW0hjSXx1RYUTc7D3gZhjvjYXDFe59dcAn4QCUf4uj5EkNnx6Jc2rPoIeJnRlOdajoqSagBivdT84pA0bUAqbNBUm1SPNMc5Fx8rumbuF7XVBzUmGsKmvcoxe2ud49AbYwkjb3WW7si3FXJpEyyJ4O1pbFVXiqVPyhGVpp45RRKZ7U82952vx+AgR8N5GfyhQBoGLIL7nZkVpQdqL5tPbiv8RnZjHI/8m59FTKvFjgig3TGMco657w9D3hjKiiF8gysNZqyifW/XOvwYqueO8CVLO3Yjj0fBA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000008, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: It's unclear why they would be considered migration entries; they are not. Likely we'll never really trigger that case in practice, because migration (including folio split) of a folio that has device-private entries is never started, as we would detect "additional references": device-private entries adjust the mapcount, but not the refcount. Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Signed-off-by: David Hildenbrand Reviewed-by: Alistair Popple --- mm/page_vma_mapped.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 81839a9e74f1..32679be22d30 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -111,8 +111,7 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw) return false; entry = pte_to_swp_entry(ptent); - if (!is_migration_entry(entry) && - !is_device_exclusive_entry(entry)) + if (!is_migration_entry(entry)) return false; pfn = swp_offset_pfn(entry); From patchwork Wed Jan 29 11:54:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13953676 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31C17C0218D for ; Wed, 29 Jan 2025 11:54:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B66B6280012; Wed, 29 Jan 2025 06:54:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B1691280011; Wed, 29 Jan 2025 06:54:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B717280012; Wed, 29 Jan 2025 06:54:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7E165280011 for ; Wed, 29 Jan 2025 06:54:42 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 33E39160C37 for ; Wed, 29 Jan 2025 11:54:42 +0000 (UTC) X-FDA: 83060332404.04.1ADCA6E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf03.hostedemail.com (Postfix) with ESMTP id D96602000A for ; Wed, 29 Jan 2025 11:54:39 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AN7tnBJH; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf03.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738151680; a=rsa-sha256; cv=none; b=xPzbjeuTKcYVoxURVp4H4vw0h1p9/rbwSNLQSyFf7YfqpMMsVTCQNIWLEYlPjqUK4mDc39 W2bHufh/8NTOhsyTU4qTQHcF1Y1gUXbTpYXeWziiBUj0gAJK9NUN3cAWkZRzUahGu7TeTn OlEnNhY/c7yPGZqjKzCvkUgvtjRWM0I= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AN7tnBJH; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf03.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738151679; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PlRguNVTZsTlNEYBD/J+r00pj+4jYSoE9sS4zp2ee/k=; b=kjVF6zUXn58OrfwvBnE1cEmr8XQbxk6yoicsIYKpdIUZ1txgtlvjB8S7zUeAlLLxSW7xjR +605DjsgMZ5xCZKJk8ZxL5ugovLxtymMvgiPjL1jO3BdkEgucHrhFasy+xT32Niesmn+7/ tapzj4HQAmlE+0pGYNbrabChgfdSGIM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1738151679; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PlRguNVTZsTlNEYBD/J+r00pj+4jYSoE9sS4zp2ee/k=; b=AN7tnBJHlzIYtwcIwmI99ZTK8RhuzM0inlF6SliAPadv8NH+T3pAfKWc5l5sA+2xK2hZ4n /oX+MYsSt4SJYBXS2IpDhVLX2MyjAHqCavSjU5Ec2Dd21prSgFc2lUwqBXn6S8Ow75575I kDnJIIdSTP7GtKnROhauZUPpv6kMgjk= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-648-JdKW3BTuMwWnU1Wk2ZAzdA-1; Wed, 29 Jan 2025 06:54:38 -0500 X-MC-Unique: JdKW3BTuMwWnU1Wk2ZAzdA-1 X-Mimecast-MFC-AGG-ID: JdKW3BTuMwWnU1Wk2ZAzdA Received: by mail-wr1-f70.google.com with SMTP id ffacd0b85a97d-38a684a096eso3012842f8f.2 for ; Wed, 29 Jan 2025 03:54:37 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738151677; x=1738756477; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PlRguNVTZsTlNEYBD/J+r00pj+4jYSoE9sS4zp2ee/k=; b=oSpeUnAgqJp+EQJWLArGSo4unRgA/vXD7lkQZhKx0vWFevAFoKgqYwJI6mbsQp0Bay +MV8XFHfvx3Yo2whXJjx7Vi1MxaY02gZAZzrmv62YLHa2zcjyIp449EZrsUH7n5Xsi6H HJ8NsfyOJt2c0DizuaFdVxQzJ3iyGweqVCIC45XEyv8HsZgvbBLHAoCmVswlf6poVlMd P6/M+BVZ0ZNd8yj6Y3wYKI1EZg8BLkqnH2IjEeAVRoCpIiqONqoGDWaSsJG3+KvxvHta R0Qej1JTO+Ocw/ESaAIx3yR5+92nRtklqDclzdYkioNjiImKgIpTmKe9KwzZxvern+zy w8Aw== X-Forwarded-Encrypted: i=1; AJvYcCUnqAf0TQTp/hA5lMRDOeLKMOBjMHyICwhbWp20ML6ORqJYJmDSmu8WKwi6s+cBLu6FukV2E6XEJw==@kvack.org X-Gm-Message-State: AOJu0Yw64fpaqpEu1V8t67M23qRyZrkrTDsIjAwfa0S0p8B8rioNNLTG m+T0wCA1GhFu7nBgszKPmb7fO/OskxRd0CfjgWqYPpg1NMZXD4+ZOsvnGDhvqnwqM4btfYfIFAE O9KDy4ORzfTJn4FfMexX3QikF7AXeixPij6ZglRLOydiBKx0fK5q8Zn4q2IM= X-Gm-Gg: ASbGncursdN1E2rfKT9tFGodyNiUBukg0w8cZD8rRB/c4+y4EnD4jOprZPpP66oDeZp qCZ1VJApHSxWhMYl0WSfIAnTFLZi4PhXlkVABn3VYHe/Qtyz/fHLQR2ZlqnHnqpkxCk7guVUT8Y GcYuCGxIXDpnipxamN6oWQWD2k+3nJ9PX09ekrW/f2QFV+MIxfvPUC0sESuRhP9TxV5MtKTRIkx VYhbi2VPQXRlRRLt8KjZ2hWUqAZH7OMW+G3DNVwNBIoD8TCD5WAd1kdom2lPtjcDpRRa65JIxYE jbKNbRglTJlEFQveu9dnc01yCANnLBS6KltWfxNdvfmew8Sv1lBTupEVF7k5BYj+UA== X-Received: by 2002:a5d:50c2:0:b0:38b:d7d2:12f2 with SMTP id ffacd0b85a97d-38c520bf925mr1637948f8f.54.1738151676876; Wed, 29 Jan 2025 03:54:36 -0800 (PST) X-Google-Smtp-Source: AGHT+IE238aJbXQykMLxOmkA/GlxflTBLrEd6FkSw7fKwOzieBzddAgHKUTAMn1ykR+OZNV3l083wA== X-Received: by 2002:a5d:50c2:0:b0:38b:d7d2:12f2 with SMTP id ffacd0b85a97d-38c520bf925mr1637927f8f.54.1738151676497; Wed, 29 Jan 2025 03:54:36 -0800 (PST) Received: from localhost (p200300cbc7053b0064b867195794bf13.dip0.t-ipconnect.de. [2003:cb:c705:3b00:64b8:6719:5794:bf13]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38c2a17d7a7sm17234978f8f.32.2025.01.29.03.54.34 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 03:54:36 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v1 08/12] mm/rmap: handle device-exclusive entries correctly in try_to_unmap_one() Date: Wed, 29 Jan 2025 12:54:06 +0100 Message-ID: <20250129115411.2077152-9-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250129115411.2077152-1-david@redhat.com> References: <20250129115411.2077152-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: IG3ghe6dvRyrX8Uj1Y3J1yJH_Vw3-S8TkHAqtd62lTo_1738151677 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Queue-Id: D96602000A X-Rspamd-Server: rspam10 X-Stat-Signature: icnjeppwi54txi1ca6ogjjt38mbbkoye X-HE-Tag: 1738151679-238868 X-HE-Meta: U2FsdGVkX187EWXyvApljRuxGlDSgxji0rgqt+gHhiHo4ByBsFrhHgfRcH+7qUWJUQ0UMrUtC5WYns+tsyKRK2/V/7OZr0r2jsukh/gRHNLh+BljhSttdFGQOcbrszYJm/Uohl1aiDbCrAQKqOUVtbc+6sqjWiWP5836NVoYhGw1IxChtpmzRULaBAbPt2WnUZG0nzPqTZcezB4ONuVOlVwTh6SnZfw7Yy8wyg951h+ChsZGH8olA9YwYSb7L7OeQr7sKlKa96PzYsl3DisCiQ5eS+iUmw8IQSbl+ZlaNyBFlHsqNQcs30J8hVWeCe9r2I7l8fl3mWU3kF7I7TbCVuxzBCvoXJQALBkDqC+O81v43zPKWsZKz6vmIQE2YHDdWK6d9BmZN55Orb5opJ6LxLj05KW96HCmbk9b6UvF4YKRlY5atp1dTcoXu6isF+qLqFc8LLKgKuj7Wnu2GhzwUldPPKjx9HBoDlC5VsypRh4GYmClkR/ZSqHFULMI23j7DTGp2XDcn8A5pT+FwQHHOEBy2Dqsnq6qxspAgHPX82zIy9plY2GutA1BpmsHbxDqdpU8cL7EmFcYyNqjDloGFA6c2g8fhFytZbpwoPKjUhA63jeSWZMqCQVPQiJlu0mqD3DRpGJ9Vo2J6IT4sbSD68b8flzJZeyjljn+DShN3M4sj/H6KkCH5mkaD5g82hiSDR9aY1RcRLc6k1b1pQAyr0ApRpY5h41vDeYVDb9Khm+Z14J802u6YimWZfzUWY3AMtctIHH5zCW2lRIPkbTBOZjbGV4h0+PaFuUs/vZCj7A6DhwfLd9/zCt5NVMeJnva0ZZAtr4TW1zkWrn4gBNxm/fWGaYJ6F+m2VQK/l/wz87w49A1B0mtxJmG9kpS7hpr2I/+tk9s09bcqQq1PxuVXXnz3bTS20hDxbmee7iXYxmPHUY5gIEScPO8C/0GBSaHMOET0HGO1aMyMfcNJH6 DB5QCgwW agV6AHdRdnKEyWqa/p4GqJ43/SlKd2Z8+TANAXZOVUF/A/2bQN05aDtESWt86OSGH1//hv9EwgCZja35Iqw6VieQbg/Ui79c8xmgP16YhA1qx2x30BK15YQVyI2xf3BIjK1zxYE0lUiswh2iE8iy4Ox5UuwpIxmEOjol7PjlTOiNEAEPo07reBVJTiRYyJ6Dd76e6/VVilQcG3zgGSTKxzBX+mtvBaaz08kNL12u4RmziwqLSX91SV/7fv9KuHcwRKFUsR0fGXET1jltpf+pYjDnIdqvwpWzSdnGWwZOl+YjEfqeOihOGbHv6LBlRHyUaj8hPiiMH2sb36ofufqW/tN+lIO167D6F439xu1qNe8zFYAOGMXNxb96IhxO4mLUk9r4gBmDAIzFRdO7yRImBeoyXyZ4BzjDex72DCRZ+pVnJPHD5lCUCbhOjgaOeLdX9nurVeZTxRvPsELFZJLDXroQUqX4oi6Rzul9pJ9imfy8EWO9YVD8spDs9/STQ9kKfypvtb6nT4WaSpcD7yaXkbGdN8w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Ever since commit b756a3b5e7ea ("mm: device exclusive memory access") we can return with a device-exclusive entry from page_vma_mapped_walk(). try_to_unmap_one() is not prepared for that, so teach it about these non-present nonswap PTEs. Before that, could we also have triggered this case with device-private entries? Unlikely. Note that we could currently only run into this case with device-exclusive entries on THPs. For order-0 folios, we still adjust the mapcount on conversion to device-exclusive, making the rmap walk abort early (folio_mapcount() == 0 and breaking swapout). We'll fix that next, now that try_to_unmap_one() can handle it. Further note that try_to_unmap() calls MMU notifiers and holds the folio lock, so any device-exclusive users should be properly prepared for this device-exclusive PTE to "vanish". Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Signed-off-by: David Hildenbrand --- mm/rmap.c | 53 ++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 40 insertions(+), 13 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 65d9bbea16d0..12900f367a2a 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1648,9 +1648,9 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, { struct mm_struct *mm = vma->vm_mm; DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0); + bool anon_exclusive, ret = true; pte_t pteval; struct page *subpage; - bool anon_exclusive, ret = true; struct mmu_notifier_range range; enum ttu_flags flags = (enum ttu_flags)(long)arg; unsigned long pfn; @@ -1722,7 +1722,19 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, /* Unexpected PMD-mapped THP? */ VM_BUG_ON_FOLIO(!pvmw.pte, folio); - pfn = pte_pfn(ptep_get(pvmw.pte)); + /* + * We can end up here with selected non-swap entries that + * actually map pages similar to PROT_NONE; see + * page_vma_mapped_walk()->check_pte(). + */ + pteval = ptep_get(pvmw.pte); + if (likely(pte_present(pteval))) { + pfn = pte_pfn(pteval); + } else { + pfn = swp_offset_pfn(pte_to_swp_entry(pteval)); + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); + } + subpage = folio_page(folio, pfn - folio_pfn(folio)); address = pvmw.address; anon_exclusive = folio_test_anon(folio) && @@ -1778,7 +1790,9 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, hugetlb_vma_unlock_write(vma); } pteval = huge_ptep_clear_flush(vma, address, pvmw.pte); - } else { + if (pte_dirty(pteval)) + folio_mark_dirty(folio); + } else if (likely(pte_present(pteval))) { flush_cache_page(vma, address, pfn); /* Nuke the page table entry. */ if (should_defer_flush(mm, flags)) { @@ -1796,6 +1810,10 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, } else { pteval = ptep_clear_flush(vma, address, pvmw.pte); } + if (pte_dirty(pteval)) + folio_mark_dirty(folio); + } else { + pte_clear(mm, address, pvmw.pte); } /* @@ -1805,10 +1823,6 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, */ pte_install_uffd_wp_if_needed(vma, address, pvmw.pte, pteval); - /* Set the dirty flag on the folio now the pte is gone. */ - if (pte_dirty(pteval)) - folio_mark_dirty(folio); - /* Update high watermark before we lower rss */ update_hiwater_rss(mm); @@ -1822,8 +1836,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, dec_mm_counter(mm, mm_counter(folio)); set_pte_at(mm, address, pvmw.pte, pteval); } - - } else if (pte_unused(pteval) && !userfaultfd_armed(vma)) { + } else if (likely(pte_present(pteval)) && pte_unused(pteval) && + !userfaultfd_armed(vma)) { /* * The guest indicated that the page content is of no * interest anymore. Simply discard the pte, vmscan @@ -1902,6 +1916,12 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, set_pte_at(mm, address, pvmw.pte, pteval); goto walk_abort; } + + /* + * arch_unmap_one() is expected to be a NOP on + * architectures where we could have non-swp entries + * here, so we'll not check/care. + */ if (arch_unmap_one(mm, vma, address, pteval) < 0) { swap_free(entry); set_pte_at(mm, address, pvmw.pte, pteval); @@ -1926,10 +1946,17 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, swp_pte = swp_entry_to_pte(entry); if (anon_exclusive) swp_pte = pte_swp_mkexclusive(swp_pte); - if (pte_soft_dirty(pteval)) - swp_pte = pte_swp_mksoft_dirty(swp_pte); - if (pte_uffd_wp(pteval)) - swp_pte = pte_swp_mkuffd_wp(swp_pte); + if (likely(pte_present(pteval))) { + if (pte_soft_dirty(pteval)) + swp_pte = pte_swp_mksoft_dirty(swp_pte); + if (pte_uffd_wp(pteval)) + swp_pte = pte_swp_mkuffd_wp(swp_pte); + } else { + if (pte_swp_soft_dirty(pteval)) + swp_pte = pte_swp_mksoft_dirty(swp_pte); + if (pte_swp_uffd_wp(pteval)) + swp_pte = pte_swp_mkuffd_wp(swp_pte); + } set_pte_at(mm, address, pvmw.pte, swp_pte); } else { /* From patchwork Wed Jan 29 11:54:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13953680 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8DBDC0218D for ; Wed, 29 Jan 2025 11:54:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E76BE280017; Wed, 29 Jan 2025 06:54:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E01BF280018; Wed, 29 Jan 2025 06:54:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C28D6280017; Wed, 29 Jan 2025 06:54:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 9DF12280018 for ; Wed, 29 Jan 2025 06:54:46 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id AD8541A0BE7 for ; Wed, 29 Jan 2025 11:54:44 +0000 (UTC) X-FDA: 83060332488.10.A30F7C6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 6970A140004 for ; Wed, 29 Jan 2025 11:54:42 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fdzn0TjX; spf=pass (imf23.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738151682; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dBI2mWOVOmwqsdVlja7jNAEkV5b5lmKCa22lnGaqVYs=; b=ZZa772ON37NkdNfWiFMwh51uwJESOb1exYbfhYdAKgPEugtpqIO8jsPyF/SyuGhnF14t0h rICnlojOa+ZxhkTxqfcjeGJP4sUC2b6iFjZPzJoYvvbx+r6E/b/fabckyu3wTmUZ2AGMYb OPeBMYL+Fd1wOpgpJu+OttFYHp0Al5k= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738151682; a=rsa-sha256; cv=none; b=1WfVOJK6KPI4MFZQXwIoUoTJfdsKdrsxLWxn++43xGvHuueb59zbGMR3HVQ5TSaSFcth7b Fnp2jRH0wk11rUDNHpdMB3DBchx165WmMHT2wQ7h1ojvRroYZTWtHwxxaGT4nW5dsNOCWf PgCTs0IUUYs33tB8MzZO3ilo04Bfa6s= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fdzn0TjX; spf=pass (imf23.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1738151681; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dBI2mWOVOmwqsdVlja7jNAEkV5b5lmKCa22lnGaqVYs=; b=fdzn0TjXMN2bKaGhrTThvk5lBfdREaaanDT9xDIGz2rFrQJkdEjDrbSu4GR4nsbD0p5qAT aimdZp1o1gnscr11A8Dxew1pUVm/i/GYc3fTrGPGcutinfCX7ulhCOeE1Husuv5FRR8E0j P0asp1NcFM4i01+ph+vT5F5PAxAPnPw= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-412-yHuk-rRkOeSOAz4m74G3QQ-1; Wed, 29 Jan 2025 06:54:40 -0500 X-MC-Unique: yHuk-rRkOeSOAz4m74G3QQ-1 X-Mimecast-MFC-AGG-ID: yHuk-rRkOeSOAz4m74G3QQ Received: by mail-wr1-f70.google.com with SMTP id ffacd0b85a97d-38bf4913659so5053250f8f.1 for ; Wed, 29 Jan 2025 03:54:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738151679; x=1738756479; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dBI2mWOVOmwqsdVlja7jNAEkV5b5lmKCa22lnGaqVYs=; b=w2efGcJGeB5F/o+qXxbY/4wtwy63xuMg/XWd46gi0FY/rxHHZvK8sCz/yb2+HijRV6 YrCMcQ16F1prupGCUdJrDpS0yJh+UleLEkkDkJa6VGXNpbOpdTKhwOR2rwXbNeLOBICi ZAb4G8PswckwguEwH0Eu3OZBCjuJhxLesdY3yoNrik2on6wJBVQhcng8lq3eQbUJseFA ECbFiDSkZY1dMhSnTvZ4exIPMJ+FJZvU/gcSfW8/i8oghFAq2hkA4B8wjqsWDH3pLs8O J6dleDCao+KHFmqz1of0iwS823iFBBatYfEZ71QNgChEZQeE2Fq0oTcsy2eSr49WRpHl roaw== X-Forwarded-Encrypted: i=1; AJvYcCWvf2NriXJH3656SMOOkBLPZuAyk84bILyWalyhpWe9qAVnNp3oijArBmVJHZHhn5JTXAaC1kbpVg==@kvack.org X-Gm-Message-State: AOJu0YzkDE4C+a20zy0c7N3HqlPFblHpdER8/3Xh5hscixQBHoYferHr pvioQyKNmrprG8/AqwezYtUiuUbPH4ep6uIk3dT+YU1ChUpu7GcmUHkJ3BNe75foigfDJSkXnUZ AkAygNw7UymcfjN6hyUFksdwsq9aDlGOSAFVOJgrcylNvCg6q X-Gm-Gg: ASbGncvJIqfqIfTj0/XBoAdHKhCEGcl8agtOmZNBuytY5mbxqBygFrC2kB/ihqUo+ym NRU9vO/rbi7gVCWFyrC2A4LctZPAjkUjDW00FXZiXj8C5tqhU9qwYEzuzqObIXpyar6LvvGvPQX QdJRcRywd9PP3b/ThzhazgT5mL8OkBJOLzO89uUNy1bHogJhbjsFwcHU52VBeonhKsV4Qa/0YuS pUKhD+uK3IkUk2rp2kZdwjVjR7DzMZ5Kj8szXkAOPINy+rtpwrEYnXjsBEfbs7gQlJBQ36e+8S2 yWtGh1K98/oPZAn8hnKh36F+8G1WzFTzrMd6cTJ3oxULGwhnV1B8d7sPdiMAB3wFdw== X-Received: by 2002:a5d:47c9:0:b0:38c:3eab:2e17 with SMTP id ffacd0b85a97d-38c5194dae9mr2038635f8f.2.1738151679503; Wed, 29 Jan 2025 03:54:39 -0800 (PST) X-Google-Smtp-Source: AGHT+IEuFMqyDWaa/JVieaul0deYFiTz0a9oHzheZm+vFN4IrpXYFm3MUhtZq6TOTjv8DbS0i6Ouqg== X-Received: by 2002:a5d:47c9:0:b0:38c:3eab:2e17 with SMTP id ffacd0b85a97d-38c5194dae9mr2038593f8f.2.1738151679034; Wed, 29 Jan 2025 03:54:39 -0800 (PST) Received: from localhost (p200300cbc7053b0064b867195794bf13.dip0.t-ipconnect.de. [2003:cb:c705:3b00:64b8:6719:5794:bf13]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38c2a1764d3sm17086479f8f.19.2025.01.29.03.54.37 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 03:54:38 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v1 09/12] mm/rmap: handle device-exclusive entries correctly in try_to_migrate_one() Date: Wed, 29 Jan 2025 12:54:07 +0100 Message-ID: <20250129115411.2077152-10-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250129115411.2077152-1-david@redhat.com> References: <20250129115411.2077152-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: kLdNtL7dRGQ9lOqCfPieft0NJn6-Xjve2ic_CYYXeaI_1738151680 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Queue-Id: 6970A140004 X-Stat-Signature: zo6h343qorkd48jr8bdxnkftrhz6bezf X-Rspam-User: X-Rspamd-Server: rspam12 X-HE-Tag: 1738151682-629642 X-HE-Meta: U2FsdGVkX1/PJKwI3M8UhgWAUrYszSw4lBvizHinl7OgrmJqHlPt9tDjWt0tORUEGh/phgrSospaQKAcN0OU4fjR/jl6JH5bkEJbwkBvJ9uTMVsenY5HeLtUsGn7mHSC+x+Uw5mPqBGMihbGV7h5ApcnvQHhqJZhWiYKwMMYSFXEXiFSxWqvS8rA9TYT/Tg/U3kNAvpaUcpyYXuXKhihuI27OQ+XCzoYZ7mVtNqIeP7bREp4bRhwT8ijZafgHRo341rLvaHYDeKSsNYPNhfyp1Am9p05P2TlBrQBDhaTfO6DnhI0zMguGAaGHpOtXAFcpvaVwjUX5+VwQHcJjR/J73yyF9ksGDvdUmMZjcsJNPPfFDuhPZV6Qq0+9iwahxryk+NATT9S385qw9x0AsBryWJ3127IBTdPQh71PTEt018dk3FWmtsaLMWjn8RUnlzlCr8zPGA2efqMGkG7m0XXogf3VJUsSgySvmqrBbTMIrLaFHUB4GIyVD8xxoOhtouPebCF4FFOH1vtP65BwLvBDaJD/4pDWNwT/Ry0kYuBZgenxxsC5RorU4VWkT0zF2KQVeRTDZJMyXGBcKIyP1CjS9r59u5SzKolrR9fVIav6tej46AgVYn4to8kTZ9deUcdHH5/4DEQmQpvRcBGChnCiNX9susVCajl6a2eA5xdCdlHsPfQ3CqSsw9hLFZtGVZsanU5lznFvaqRvzhnH95prc1zrPIKrpzJmpO7UFkrmm3VDiJHTDP8PS+hJzyYwDLvk4vO5K15fEghxfxlNOmlwH2ngqjOibNmQ9SUFj1Yx3doAPhg6o0AAdFTFEhrm2cygBB2YuOF+y74Fd/rPFeS7fRCYkrkl84JZn/8PIgH6mZXUoxGIlJSJqgfXpudpo24ZA7bceaKXgVwLuD3+B5ieej9Chgj7h9SpJAZrSgIwRin+Q1d1UkcH4E2g9S4qU/IkpavqCMUhlzpQl8/K3L 1NGxnruN wa2Swx4v+92Ei1azGV+M3ObEwRxtoxJvunGQHgyRbMZnWquZG3wrlHDeXVM3IxQzlO+rY84oNyG2gJHrFko2JxKbKjp/za807yVr9EauhyWA1u+DkdkZ2fWZblTSNxM06JRyQq29J30bDQHOU9Yld2apsGCG0G0pawxX+OOJY3B7bZIu+I/yQkhVPHuIoXPD5eucCoszmZdm1DkkLZNb9JmLW54oQuR84VNViFsdQ/he76X+ktUiwCNNA5cWBMEnbLg/D4daUvp4lrBkZEk4g15r6HiO7o08NWZwbNAYn6v7oYQRo0X1yJRRbMVmltsObVZB9r3Uf/BxzB4Ajc5qfHoqiXQCvJQMgf4mf/1EYBM7Sz1Jey/4Ujk6P69q2yc98VLz8wUvVYa5BrvV3Gp/kiiTvPY3BfrCxtTE5bi1hJZqqfGy9wqHa1Dt3akNRqRn12rNoov45y5TOFkUO3Kvges5ZU0ULlD6Vu9o/g1GUTnIYvWReWHrc0upCz8fXMcJzNa/meWy9l/2u41cZJiuGnbiSjw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Ever since commit b756a3b5e7ea ("mm: device exclusive memory access") we can return with a device-exclusive entry from page_vma_mapped_walk(). try_to_migrate_one() is not prepared for that, so teach it about these non-present nonswap PTEs. We already handle device-private entries by specializing on the folio, so we can reshuffle that code to make it work on the non-present nonswap PTEs instead. Get rid of most folio_is_device_private() handling, except when handling HWPoison. It's unclear what the right thing to do here is. Note that we could currently only run into this case with device-exclusive entries on THPs; but as we have a refcount vs. mapcount inbalance, folio splitting etc. will just bail out early and not even try migrating. For order-0 folios, we still adjust the mapcount on conversion to device-exclusive, making the rmap walk abort early (folio_mapcount() == 0 and breaking swapout). We'll fix that next, now that try_to_migrate_one() can handle it. Further note that try_to_migrate() calls MMU notifiers and holds the folio lock, so any device-exclusive users should be properly prepared for this device-exclusive PTE to "vanish". Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Signed-off-by: David Hildenbrand --- mm/rmap.c | 125 ++++++++++++++++++++++-------------------------------- 1 file changed, 51 insertions(+), 74 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 12900f367a2a..903a78e60781 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2040,9 +2040,9 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, { struct mm_struct *mm = vma->vm_mm; DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0); + bool anon_exclusive, writable, ret = true; pte_t pteval; struct page *subpage; - bool anon_exclusive, ret = true; struct mmu_notifier_range range; enum ttu_flags flags = (enum ttu_flags)(long)arg; unsigned long pfn; @@ -2109,24 +2109,20 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, /* Unexpected PMD-mapped THP? */ VM_BUG_ON_FOLIO(!pvmw.pte, folio); - pfn = pte_pfn(ptep_get(pvmw.pte)); - - if (folio_is_zone_device(folio)) { - /* - * Our PTE is a non-present device exclusive entry and - * calculating the subpage as for the common case would - * result in an invalid pointer. - * - * Since only PAGE_SIZE pages can currently be - * migrated, just set it to page. This will need to be - * changed when hugepage migrations to device private - * memory are supported. - */ - VM_BUG_ON_FOLIO(folio_nr_pages(folio) > 1, folio); - subpage = &folio->page; + /* + * We can end up here with selected non-swap entries that + * actually map pages similar to PROT_NONE; see + * page_vma_mapped_walk()->check_pte(). + */ + pteval = ptep_get(pvmw.pte); + if (likely(pte_present(pteval))) { + pfn = pte_pfn(pteval); } else { - subpage = folio_page(folio, pfn - folio_pfn(folio)); + pfn = swp_offset_pfn(pte_to_swp_entry(pteval)); + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); } + + subpage = folio_page(folio, pfn - folio_pfn(folio)); address = pvmw.address; anon_exclusive = folio_test_anon(folio) && PageAnonExclusive(subpage); @@ -2182,7 +2178,10 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, } /* Nuke the hugetlb page table entry */ pteval = huge_ptep_clear_flush(vma, address, pvmw.pte); - } else { + if (pte_dirty(pteval)) + folio_mark_dirty(folio); + writable = pte_write(pteval); + } else if (likely(pte_present(pteval))) { flush_cache_page(vma, address, pfn); /* Nuke the page table entry. */ if (should_defer_flush(mm, flags)) { @@ -2200,54 +2199,21 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, } else { pteval = ptep_clear_flush(vma, address, pvmw.pte); } + if (pte_dirty(pteval)) + folio_mark_dirty(folio); + writable = pte_write(pteval); + } else { + pte_clear(mm, address, pvmw.pte); + writable = is_writable_device_private_entry(pte_to_swp_entry(pteval)); } - /* Set the dirty flag on the folio now the pte is gone. */ - if (pte_dirty(pteval)) - folio_mark_dirty(folio); + VM_WARN_ON_FOLIO(writable && folio_test_anon(folio) && + !anon_exclusive, folio); /* Update high watermark before we lower rss */ update_hiwater_rss(mm); - if (folio_is_device_private(folio)) { - unsigned long pfn = folio_pfn(folio); - swp_entry_t entry; - pte_t swp_pte; - - if (anon_exclusive) - WARN_ON_ONCE(folio_try_share_anon_rmap_pte(folio, - subpage)); - - /* - * Store the pfn of the page in a special migration - * pte. do_swap_page() will wait until the migration - * pte is removed and then restart fault handling. - */ - entry = pte_to_swp_entry(pteval); - if (is_writable_device_private_entry(entry)) - entry = make_writable_migration_entry(pfn); - else if (anon_exclusive) - entry = make_readable_exclusive_migration_entry(pfn); - else - entry = make_readable_migration_entry(pfn); - swp_pte = swp_entry_to_pte(entry); - - /* - * pteval maps a zone device page and is therefore - * a swap pte. - */ - if (pte_swp_soft_dirty(pteval)) - swp_pte = pte_swp_mksoft_dirty(swp_pte); - if (pte_swp_uffd_wp(pteval)) - swp_pte = pte_swp_mkuffd_wp(swp_pte); - set_pte_at(mm, pvmw.address, pvmw.pte, swp_pte); - trace_set_migration_pte(pvmw.address, pte_val(swp_pte), - folio_order(folio)); - /* - * No need to invalidate here it will synchronize on - * against the special swap migration pte. - */ - } else if (PageHWPoison(subpage)) { + if (PageHWPoison(subpage) && !folio_is_device_private(folio)) { pteval = swp_entry_to_pte(make_hwpoison_entry(subpage)); if (folio_test_hugetlb(folio)) { hugetlb_count_sub(folio_nr_pages(folio), mm); @@ -2257,8 +2223,8 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, dec_mm_counter(mm, mm_counter(folio)); set_pte_at(mm, address, pvmw.pte, pteval); } - - } else if (pte_unused(pteval) && !userfaultfd_armed(vma)) { + } else if (likely(pte_present(pteval)) && pte_unused(pteval) && + !userfaultfd_armed(vma)) { /* * The guest indicated that the page content is of no * interest anymore. Simply discard the pte, vmscan @@ -2274,6 +2240,11 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, swp_entry_t entry; pte_t swp_pte; + /* + * arch_unmap_one() is expected to be a NOP on + * architectures where we could have non-swp entries + * here. + */ if (arch_unmap_one(mm, vma, address, pteval) < 0) { if (folio_test_hugetlb(folio)) set_huge_pte_at(mm, address, pvmw.pte, @@ -2284,8 +2255,6 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, page_vma_mapped_walk_done(&pvmw); break; } - VM_BUG_ON_PAGE(pte_write(pteval) && folio_test_anon(folio) && - !anon_exclusive, subpage); /* See folio_try_share_anon_rmap_pte(): clear PTE first. */ if (folio_test_hugetlb(folio)) { @@ -2310,7 +2279,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, * pte. do_swap_page() will wait until the migration * pte is removed and then restart fault handling. */ - if (pte_write(pteval)) + if (writable) entry = make_writable_migration_entry( page_to_pfn(subpage)); else if (anon_exclusive) @@ -2319,15 +2288,23 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, else entry = make_readable_migration_entry( page_to_pfn(subpage)); - if (pte_young(pteval)) - entry = make_migration_entry_young(entry); - if (pte_dirty(pteval)) - entry = make_migration_entry_dirty(entry); - swp_pte = swp_entry_to_pte(entry); - if (pte_soft_dirty(pteval)) - swp_pte = pte_swp_mksoft_dirty(swp_pte); - if (pte_uffd_wp(pteval)) - swp_pte = pte_swp_mkuffd_wp(swp_pte); + if (likely(pte_present(pteval))) { + if (pte_young(pteval)) + entry = make_migration_entry_young(entry); + if (pte_dirty(pteval)) + entry = make_migration_entry_dirty(entry); + swp_pte = swp_entry_to_pte(entry); + if (pte_soft_dirty(pteval)) + swp_pte = pte_swp_mksoft_dirty(swp_pte); + if (pte_uffd_wp(pteval)) + swp_pte = pte_swp_mkuffd_wp(swp_pte); + } else { + swp_pte = swp_entry_to_pte(entry); + if (pte_swp_soft_dirty(pteval)) + swp_pte = pte_swp_mksoft_dirty(swp_pte); + if (pte_swp_uffd_wp(pteval)) + swp_pte = pte_swp_mkuffd_wp(swp_pte); + } if (folio_test_hugetlb(folio)) set_huge_pte_at(mm, address, pvmw.pte, swp_pte, hsz); From patchwork Wed Jan 29 11:54:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13953686 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6219C0218D for ; Wed, 29 Jan 2025 11:55:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 526C9280042; Wed, 29 Jan 2025 06:55:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4AF34280025; Wed, 29 Jan 2025 06:55:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 301D8280042; Wed, 29 Jan 2025 06:55:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0DC01280025 for ; Wed, 29 Jan 2025 06:55:14 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 38FEB1C8001 for ; Wed, 29 Jan 2025 11:54:47 +0000 (UTC) X-FDA: 83060332656.24.AF62EED Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 03AD8100010 for ; Wed, 29 Jan 2025 11:54:44 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bO9At6Bx; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf14.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738151685; a=rsa-sha256; cv=none; b=IByV9B4UVUaLVVPI+MRsK38lzq0tghEAdgInyugJabLnBhKqX6hoXltnodAMdWNgZCHBd1 cbuwBt1EFJfBo+R8kgM9qVwGKm9A7dS2BIa9LipQoG6YtYTTgtRFoGQJLS61Jd0H1HoG/A DDOqKzoO4nAPPByRqeUGAA041wubAkI= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bO9At6Bx; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf14.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738151685; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mNM6Zw+nv5CKX0FXtzXMADzAxRUJZ/HteqC2IEBjnh8=; b=eldfgR1MXlWk9rV0JAm8uilIgH2i89yFChQjQ2TKbAoSGScW6SP+30ocbq3zeWMKWKCwJL jkBz8u84/dGSNQDPEUZeaQXS3OfCOiHxhyJTSZ38NvMPg6OmHL21HWz7sdVRuIZDHIOYvj 0YIoZurzEqvJ+T863lbTzchtPuAm0+k= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1738151684; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mNM6Zw+nv5CKX0FXtzXMADzAxRUJZ/HteqC2IEBjnh8=; b=bO9At6Bx6yVnB4LQRYoi4poyPOA/wIgJ9YksvQlBEBeNrLVxxicFX+myPSRXZCP9JG4VDh P2C8DTiOIoNG0WknXdumRcdaUAjzQql6eMxUEqKPon7wr02deEEPDZbSJFUcoPx9vgJZn7 1NdiVQSiUcKOXzeBHxAxVnvnoGNoRCA= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-605-hCJQFP-COUeColKpYXrf-Q-1; Wed, 29 Jan 2025 06:54:43 -0500 X-MC-Unique: hCJQFP-COUeColKpYXrf-Q-1 X-Mimecast-MFC-AGG-ID: hCJQFP-COUeColKpYXrf-Q Received: by mail-wr1-f70.google.com with SMTP id ffacd0b85a97d-3862c67763dso2402813f8f.3 for ; Wed, 29 Jan 2025 03:54:43 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738151682; x=1738756482; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mNM6Zw+nv5CKX0FXtzXMADzAxRUJZ/HteqC2IEBjnh8=; b=DoFSg7nmhFwM747Mah1Nc5otWFLJOEeSXU0yhdnEt9UXhGJOqg2W8fRmJ4n6zefd/l 9AYWNcWxWSuGwJC5VFAYJJhk+lNcI6xrHM/8itBBC7iNRdn8oXyzmQZNqodiWnb7aEvJ yn5LoL8WQLxU8bj74rxPFc1qLOFebRSH1sZwSo25lblF21cXZ1oVuO5TCs7OlInBmHwP VbA6gl/yIwDs3V2LM7nxQBIpCOmX6sRELW2SjwDy23f9jVRizbj7ZQjfXqItcbBP8muM RAmXjhgIreNcHeZU6uN4oFUdDWXJ2wgJAU540xZi5S3IAbXdlXQLB22rj3W0fpnXLoNQ ekrg== X-Forwarded-Encrypted: i=1; AJvYcCWAC3OQpRL1Lnv44ZCoa/ogn+Jh0RO4jhgzMHW8yxzIM/aTEHBg2eR8dmk87hV8d/lLD/eKcJdUTg==@kvack.org X-Gm-Message-State: AOJu0Yw6jBc6WfFkm5DQIVA5ymKYzohHm2J3ExMBLOqEvA+76Lj28bk+ JhH2IfTB7T410OHmKssGoXX+8sQjYro6xl9ZFaO5CI5m6WS7W+aafaqyTGdM+AHdQ8aErxPGiC9 bUflSvJst5xygiHdzQcFz0pqiHSCKzbde7cnJk1BL9Rq8iD/V5lSdtZODEzA= X-Gm-Gg: ASbGncsBnmVlNtY1lDCf1FQTGGWPvowyCuRWu3SlhlntUiybyxAx2/vmRiHtFvt0tVI PH0fXZQd8O7D4D97p8PoxTBUESJdzJRuCXMcFABG9wpsJKoL5TKQBMatdiHfVNXbfmAgDY2r48y GfFP95A16bFkTm3iN2PqUhH0oegyHx3YLpZpzEuanRApMezCQ5nSUGBwL1KEQU7maM80poECrSk yVo15k/d1hEnWrxu4iVHYlePHU0JpUPP3qcd/A79digrwVtGr3QyZuBrQHIFON5m4D9ieI2bXyL Iwl+4dlmMA4XM4OhpGNAe+/gwoWmMPozBCngolryAxCJmEGyVDyc7XPv8C3cjZ0Oww== X-Received: by 2002:a05:6000:4013:b0:385:f631:612 with SMTP id ffacd0b85a97d-38c5195f2e5mr2414977f8f.17.1738151682118; Wed, 29 Jan 2025 03:54:42 -0800 (PST) X-Google-Smtp-Source: AGHT+IEfVb9Gb5/TQQdvQH0d8Y86brLumLHRijGlRqbPqwAUyE2LkrZ873cAyk0+z1AQr8yjahg13A== X-Received: by 2002:a05:6000:4013:b0:385:f631:612 with SMTP id ffacd0b85a97d-38c5195f2e5mr2414952f8f.17.1738151681703; Wed, 29 Jan 2025 03:54:41 -0800 (PST) Received: from localhost (p200300cbc7053b0064b867195794bf13.dip0.t-ipconnect.de. [2003:cb:c705:3b00:64b8:6719:5794:bf13]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38c2a1c4212sm16316119f8f.87.2025.01.29.03.54.39 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 03:54:41 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v1 10/12] mm/rmap: handle device-exclusive entries correctly in folio_referenced_one() Date: Wed, 29 Jan 2025 12:54:08 +0100 Message-ID: <20250129115411.2077152-11-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250129115411.2077152-1-david@redhat.com> References: <20250129115411.2077152-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: t0H4xmmHlZPwn4C1NiYcsH4Jksym3QcQ38I2qqa0mZs_1738151682 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 03AD8100010 X-Stat-Signature: gapbikinxe9gwurf1do7mmdkz759gfop X-Rspam-User: X-HE-Tag: 1738151684-924841 X-HE-Meta: U2FsdGVkX18vkyuDE6jIDP2skSZCXBPH9OlCO5iM+8gZ88cVCfwF/jbQ++MW5f5Nn+0kcDNtF6+PaOAukBpukP0Aw1kIEmidiX71B/8zzLLNXZ3+uoogyZRBR3E1xYA3E1HuHVRUN9OENKSejicCEyAGR0dywpCoGmIOzK9fqCzjK8hw8pRSiiFJ4ULQTtTWXKYbgfGbWY/v3sXAb2ttvZYuwFvH23xQD39pV5hxuYg6qAbpx1gBnqCJZABhg7bs8ZA032Pc/JX7d2t5VBY25j2+V/vlQ0tgRt9UXxz20IwVt1OXmaM+DW6IXbCORNkMIJsOYBQ2Rmu0EeyBu+mcG81pNWv/cPv6N5P1zySiMamyPmHHT2quhK7kudsVu9bmJZ6Q3aHGdgYHCbUVBvObIy3L6P5dXdFGkLXgs9NBCkigj5LY+qEFoaVSrdZ1TP+ft4Wo3um5H7pz4mTyjJMfHFlvTOGlEIMSBQ+E3VKyFUxQDgqXCuirJyKlkS853WCV4FBBWuFsXUFCgtdARZQeYRT3jaKxY1CRtJjkF8P8+vOukchjAp/FNHyqG/ZxkuEuVxXnWrx+qttvPBUcXlnHlUGYHBE6zpQb0UA1IK4q5CK8ul3KyrA2fGuf0HBZMSVUChn2j4y6ZixclWluxBGRlwoojJsH4kNQ70g13cFHfG4RIYWbgPUzp5sv7cGF18eG5OCbqnnBOhVAlC/3CV6793oyCs8KtARDA8AdUWuGePL7ldneykXc4dgEAjakuXjBOFiI08vHZBtGKHYo6ZGb5O8Nyz4Yk554SLLTDsxAGWB+3bpRex2YbrJhoxa7sICLzooAgurO+TiJWUiPHrYVKUPbmrkg9jJ+Pw1pKQualje1D2YsYz9OtsKY8Pr3KksI7ZMzoIYVmfxiM4eNASQDiuS7A8XheCHSfjtMbHb5QGFuEvPV/0g9o0cJjIFlBnT+nVivVitr+S39g1bWoNT bVQ50XmL glatrNWGWLq71yIo+WXt5RTU1bb7e+YpWDl49rF5TK8eFAuvUItUTrTux/pRhnlhfPIOH7vlKT7AJH4J19tqSkWlsa3WPgNUo9p1InFfymC2O+WL8h4W+lJUaao/WiTnoARbT2f7QEFBZ3tjZZRvWhu9OXNyBnbwlAsAbbSffdQ5fA/KrSb0EJc4rYqAOD0iZ8qdpcqF+WlYGOqqt/wVO6sep9lruU9keHzM7H1L1JgAN/qMf0a8K4GJ18MS1bFVszLlCKvngGCVgHDrQrXrfNBmXcJ7pihMbZnJQ7D0yK1LJiH/6G+uv7zeE022Y5+eW4Gc1g4oRuFfbyXiR/8KzjbsbfjJ3EaqzVrF9AAgRebsBpfogSkQSUrJasuiL1n9UKBtzGyqJnSdKfXYWFjKiP6/CoTLuF5K2RwWzpvhX30AjAjZFqmncwbhwmrkM72KN6plAUxfJkDV0ckCn51sVz5OxRlf+njkw6pdFOMLpzeh1pNfY5YJgpFKGwQo5GF3XN77EhAZ/3SygxQIe6vOfEATb1A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000003, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Ever since commit b756a3b5e7ea ("mm: device exclusive memory access") we can return with a device-exclusive entry from page_vma_mapped_walk(). folio_referenced_one() is not prepared for that, so teach it about these non-present nonswap PTEs. We'll likely never hit that path with device-private entries, but we could with device-exclusive ones. It's not really clear what to do: the device could be accessing this PTE, but we don't have that information in the PTE. Likely MMU notifiers should be taking care of that, and we can just assume "not referenced by the CPU". Note that we could currently only run into this case with device-exclusive entries on THPs. For order-0 folios, we still adjust the mapcount on conversion to device-exclusive, making the rmap walk abort early (folio_mapcount() == 0). We'll fix that next, now that folio_referenced_one() can handle it. Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Signed-off-by: David Hildenbrand --- mm/rmap.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 903a78e60781..77b063e9aec4 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -899,8 +899,14 @@ static bool folio_referenced_one(struct folio *folio, if (lru_gen_look_around(&pvmw)) referenced++; } else if (pvmw.pte) { - if (ptep_clear_flush_young_notify(vma, address, - pvmw.pte)) + /* + * We can end up here with selected non-swap entries + * that actually map pages similar to PROT_NONE; see + * page_vma_mapped_walk()->check_pte(). From a CPU + * perspective, these PTEs are old. + */ + if (pte_present(ptep_get(pvmw.pte)) && + ptep_clear_flush_young_notify(vma, address, pvmw.pte)) referenced++; } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { if (pmdp_clear_flush_young_notify(vma, address, From patchwork Wed Jan 29 11:54:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13953681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EDAA5C0218D for ; Wed, 29 Jan 2025 11:54:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 508D728001A; Wed, 29 Jan 2025 06:54:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4665A280018; Wed, 29 Jan 2025 06:54:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 28C2528001A; Wed, 29 Jan 2025 06:54:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id F0661280018 for ; Wed, 29 Jan 2025 06:54:52 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id ABFD944F19 for ; Wed, 29 Jan 2025 11:54:52 +0000 (UTC) X-FDA: 83060332824.13.69AA71F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf17.hostedemail.com (Postfix) with ESMTP id 729E640007 for ; Wed, 29 Jan 2025 11:54:50 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VWeyw6W7; spf=pass (imf17.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738151690; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1V8bFv4UpnBG412679q6Wat0SbxTmvFr7JjVIwnpoX4=; b=pFfjNOvw6jG6ymnBTZrXjawJG7EBoru43029BN3iP7V8tXeJ6jQKFqLpYkffQE9i5GDc5J UEO3CW7TsjzUVxWNoRAPtrRJ0NvQpc12xI6jpumyv/LIcw/a9NZNGcGcOQjqux7T8bH0BA np//gPpBjPOvaw0pHKoPU20DfY0RsaM= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VWeyw6W7; spf=pass (imf17.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738151690; a=rsa-sha256; cv=none; b=V8hWqTwruU8WCEmuq0LsUpnX6OBzpo9P3uWPJwdfpTRLiZT+tOFjiPJnEG+8/Ax4Nq00fW jXvgP9SxEy9p5BRKREXHgx7aXMm3tiekMxTzTj8AY7MOC6aWOw/dhBgMuuZwT9uCMT6qNW NJQ9YDuLnTq97d1IORDnqeZT0mjGgMo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1738151689; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1V8bFv4UpnBG412679q6Wat0SbxTmvFr7JjVIwnpoX4=; b=VWeyw6W74m6rOCvyXrdJZ8gyf/OBwXXcoG/uLji8TZ1iORNrpMp+JKTrPCCYNV+E+hVyq9 S4K0t1+uke/o2Y7o4SiCxHoJH43hrW5xPy0KjFZyS6szPJI088cqK7i1mi1NoHcILHDg/3 A4ONrmZil5wi5g17/aN3bR0ZBvEwF9M= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-122-hdoAmLa6N5Kiij3paPKA0A-1; Wed, 29 Jan 2025 06:54:46 -0500 X-MC-Unique: hdoAmLa6N5Kiij3paPKA0A-1 X-Mimecast-MFC-AGG-ID: hdoAmLa6N5Kiij3paPKA0A Received: by mail-wr1-f69.google.com with SMTP id ffacd0b85a97d-38c24ac3706so5219057f8f.0 for ; Wed, 29 Jan 2025 03:54:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738151685; x=1738756485; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1V8bFv4UpnBG412679q6Wat0SbxTmvFr7JjVIwnpoX4=; b=v9lEzswVhxV8C5k2GdZ0+L7gxpcr2Q4ZSzmZNQxOT8rN3h+7vfzaWNtEhHt9i49Nru bf5Dq8xMWcnjwwFMUXthYhi5YAmWd3tF+ajaHAS07TY8NX4D8XT0f7HdNiF6H7Sq0oKb oUtnpAUrp8clRTjVPxqEo/jgh8NMjc73jEQCMWtmy7oMQO1iyG5/geaZFkgA4LEl27IX XMXaFdm9UkKTfzPRccLOxjE8S60aLsasYRYTDu3laV53nxHd8gK9OXHALs4xwOfaY550 C/3DN7HEoa9cYjZ4GznmyYmC09yIAunp2tFMAvjnQsA102aDF64qASEpdHBu90rqdJqv 6hyg== X-Forwarded-Encrypted: i=1; AJvYcCXfcYlmL/dHkdbua58EbkTfGyXyv8biAKHfn4P0keaLyvLfHQjzDQ6370gSLlJ9aClhYoWgO9kHxg==@kvack.org X-Gm-Message-State: AOJu0YxbCvV39Mi6RgFSW5RLnrHndgPhErt6htD3kRTw5NhLI7bEseWW 4Y3dEYJaAVE4RhX096xTQmqkJCaBz/IJ41MkoMA48Mm81NwKfbdopFrUO1kpFHxyLNZVPNCQ7T0 qkj2XCUiHJ4d8a88UGUbBn0JtnzpayKd7U/6oV2IXSyrQa9ob X-Gm-Gg: ASbGncuZYAEgK58Rll2ZkzStPEPxdZta3bAqST1cYIl2I1LyAH7YxA7IAXpKzNEsYez PisHC+ZYswlXvTXlkUzHOTgKW/QmOY0HgFJH6JzhPEjVRgA3Nm1GZDmVIfGCoZFASWh2pvVEGEn Fqk39GMaWYWmlpnABvDH5sg4VIve7QGDR+aswZMp3h7jWnDYgP5eYZffl0AoUvmRhTQbDM7FxY6 C9lh6VH4ZpePRIx6DjelPfdgsn+MlET0fqybU8VWxaJgeVd9lQPcQ/Rcj2JWArgWUvF/dqwz7bY JeeeoQK1rQa8DDtxWj0RlY7prXRUKMPcfMqHTpo15QvEkD00cwr52ls9hsZ+lSe/dw== X-Received: by 2002:a05:6000:1f88:b0:385:e176:4420 with SMTP id ffacd0b85a97d-38c5194da70mr2305422f8f.10.1738151685456; Wed, 29 Jan 2025 03:54:45 -0800 (PST) X-Google-Smtp-Source: AGHT+IHUbIgz0SPBde6y+6m4GBK9jxdSxFJzhj9pSmwm9jjlsojNxFlCvTBxohQUGPQAzMIjP8ET+w== X-Received: by 2002:a05:6000:1f88:b0:385:e176:4420 with SMTP id ffacd0b85a97d-38c5194da70mr2305401f8f.10.1738151685052; Wed, 29 Jan 2025 03:54:45 -0800 (PST) Received: from localhost (p200300cbc7053b0064b867195794bf13.dip0.t-ipconnect.de. [2003:cb:c705:3b00:64b8:6719:5794:bf13]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-438dcc2ef08sm20681625e9.22.2025.01.29.03.54.42 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 03:54:43 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v1 11/12] mm/rmap: handle device-exclusive entries correctly in page_vma_mkclean_one() Date: Wed, 29 Jan 2025 12:54:09 +0100 Message-ID: <20250129115411.2077152-12-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250129115411.2077152-1-david@redhat.com> References: <20250129115411.2077152-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: V_hpEI_1Arew1g4GELaLatR64rYXz-UIn_zuvFEKVAA_1738151685 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Queue-Id: 729E640007 X-Stat-Signature: 55n4ax9wcz3ie8rkygxfbcrt5qpf9nrz X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1738151690-240985 X-HE-Meta: U2FsdGVkX18JSZrOMH0pYMIPjk/zY6kTDxD8bLeMwkJSuK04Y3mn/ub1gpBQX0pyYXekg8Qr9+z2xsl7nfo/FlneVH/ZWxphZagsMm5imlmHPFzYvohUTg/lFmGsREeCoT7DLyRTetfuC5Jup0AedP8puXXDtPCBIN3ccoODZ9aEzljth4sAdx89JHci+qwaqUnBnd3q7Ixx51OWuhPGChIBdPIcOAA7zFQrjkuL2vN5IN9DPrsgMJTeURNF97tPokfk/blm9IRIuqXe7QCTaFOtnnXFaa/tv0XfcTFNzH7OXgEch9LksGQRSjsopyR28aUwJi1627pJnZouANrVn8FYQe8e49OnJe7gCKrh/7VIGJtU0jPcCEReVxFG7W6RrrKOgYdpYQ3yMHlWa4UuDyZ5OxxVy5Us8OfYYg8KJm+1UkcUeXvtuUfsX3jIvDH37IuBri19CJV3GkjimUwJ4SgOlAqEXmhyNDt2vDm5dvzoXGiVmdEiDnF+V+mb2JFuW/e+IiZqExWfZFqu9PAtDB94vlUf8xErW8RNOvooKhChaQUP2Oks9N+uKrkU6f3iSUkqm0IUF67WL86JjHIBt7OmIDkfIb+cFOL8j7KNy3zAL+BRl3T/HeMZYvtVsGhHjbY/BaBzyf0oQ2bbdMjO+9zy3Sgzus9Zo+MreC8ZKDR1iXcp3f0GTzfvtX6hr/bFBtovr6mkbXR1/sag/i9oT1Sg78fEwmLqTCo2gnFVB6HrzZGpY2UsthHLbjuWbaDzSnFaLxAbdGvKaSPMLccFes7f9R0HOMpfaVGrOICkzljo63NWap1SEACipH/V0PK4q+DMMTZ7WrwyNhl2jlGW52pDdJ8BZNngQsgV6l5FdhHDjXptWZR9MqGcwC1fa4MSj33TCqjmW3LhedlwV/7XVyOcqtFOMjfIC9MeGwuLH4obEhKGc5Asfzd1bSU6krKicRjX18umBcLF/rVuSnp L65+i62C ypZjXsQlXa+zfHTRTRM0beBaJlRJqVfFqkfraZle8ZWdL43yuYWjzKiI9oXPIowK8+074yIR5yviYxLsYTGriTRsCo3lSaPmQShXQrQqYFncYO/ZE9Bf6C8QG8tcYpehOznf7e0WnRp4Nea7IXaA3CegvaZtPwji80otV0TT+pqi+M51evUzQebVnTTkE8g/pEHeMwNdb0nQeXLHWGsNA++H4Q+hExUa/ozqvW8IImvsJORxMQCfskdQ20WhS7898Tr+LjagrGroTrIYds8GHY9hZV4IvAGgnPzQXr8FiNMsYzavi0chZ7NYDogfGEUAvRPbdWEdS7EiBwFJi3/d2ggYmcPU2zRGT0sLceDH537iaoG8GCz/SmdvD3AS9yhCTw90WIyLLcmTtjRQwouHRWbscQllJEycM2Uu08oYjRD6Gli1nI3mUO9UygGF7HCWr1Pw7LFWvhWvSWhGcExVyfBR5t80QcPAnr2rdGwcRV1UFX0Uo7IBjRPR9ChyR6Ue6ke0xy6snEKlqHIqnMF1f+oJj1A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000006, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Ever since commit b756a3b5e7ea ("mm: device exclusive memory access") we can return with a device-exclusive entry from page_vma_mapped_walk(). page_vma_mkclean_one() is not prepared for that, so teach it about these non-present nonswap PTEs. We'll likely never hit that path with device-private entries, but we could with device-exclusive ones. It's not really clear what to do: the device could be accessing this PTE, but we don't have that information in the PTE. Likely MMU notifiers should be taking care of that, and we can just assume "not writable and not dirty from CPU perspective". Note that we could currently only run into this case with device-exclusive entries on THPs. We still adjust the mapcount on conversion to device-exclusive, making the rmap walk abort early (folio_mapcount() == 0) for order-0 folios. We'll fix that next, now that page_vma_mkclean_one() can handle it. Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Signed-off-by: David Hildenbrand --- mm/rmap.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/mm/rmap.c b/mm/rmap.c index 77b063e9aec4..9e2002d97d6f 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1050,6 +1050,14 @@ static int page_vma_mkclean_one(struct page_vma_mapped_walk *pvmw) pte_t *pte = pvmw->pte; pte_t entry = ptep_get(pte); + /* + * We can end up here with selected non-swap entries + * that actually map pages similar to PROT_NONE; see + * page_vma_mapped_walk()->check_pte(). From a CPU + * perspective, these PTEs are clean and not writable. + */ + if (!pte_present(entry)) + continue; if (!pte_dirty(entry) && !pte_write(entry)) continue; From patchwork Wed Jan 29 11:54:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13953684 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1C18C0218D for ; Wed, 29 Jan 2025 11:55:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 80CA7280038; Wed, 29 Jan 2025 06:55:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7468B280025; Wed, 29 Jan 2025 06:55:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 54D4D280038; Wed, 29 Jan 2025 06:55:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2B52E280025 for ; Wed, 29 Jan 2025 06:55:05 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 13279120E1D for ; Wed, 29 Jan 2025 11:54:54 +0000 (UTC) X-FDA: 83060332950.26.4C9F521 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf12.hostedemail.com (Postfix) with ESMTP id CBF4740003 for ; Wed, 29 Jan 2025 11:54:51 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hIpLnsqD; spf=pass (imf12.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738151691; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8FdUGdC2w/N5TeOcJ3tL/h1ogRWBFVvHy4t3b3m6isA=; b=HAN60+dKEZ4gSo5i/f8S97GjbGjR4+7FGw+re5s7W5oHFLFRc2Br7zfOIp76fa8OZ1cxdg 2pCSFaWszNRu3OnybZnwvbKyugQubWSDCpJOyZZ/rRIowPMD4x+7WNoAEa8KfLpUdJthKK 6bcyXOBwKypPSLV5yhXzBgbLbUeBwhA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738151691; a=rsa-sha256; cv=none; b=j5V4Z8O1jWlHsjjJzP8U/C/SP0LzPekwhduYeDnzXkqzilpB3Kagrzg28cKQcelHlDANTe H8B/Mtzrh0AELl+ih6kMkYF6gZAJeIb+J0qiRjQiA8XFPUbKme39ie1TIYNepVRvsAAIvS cNVk2vn0EAv6RZIrQ7+WvIUfHIJM5bI= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hIpLnsqD; spf=pass (imf12.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1738151691; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8FdUGdC2w/N5TeOcJ3tL/h1ogRWBFVvHy4t3b3m6isA=; b=hIpLnsqD5RbDU8gw0WGmPPyfiIak/MxLjmueg0GlNudyinRIKdeyXdoeBzP8T/S5FY8jRP bsk7YaIEj60jfF+hUeWx/lHDtFZjuc030VsLFF4s3IzrLB28o660Wgey2nxtDQrMJEcZYN SbDZWpLXGxYPz/D491Io2+TH7dvuSfc= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-223-Lfa4pMu7NtSFPqz3yHVJRQ-1; Wed, 29 Jan 2025 06:54:49 -0500 X-MC-Unique: Lfa4pMu7NtSFPqz3yHVJRQ-1 X-Mimecast-MFC-AGG-ID: Lfa4pMu7NtSFPqz3yHVJRQ Received: by mail-wr1-f72.google.com with SMTP id ffacd0b85a97d-385e49efd59so2588086f8f.0 for ; Wed, 29 Jan 2025 03:54:49 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738151689; x=1738756489; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8FdUGdC2w/N5TeOcJ3tL/h1ogRWBFVvHy4t3b3m6isA=; b=AlcLz6mnSfQUXwu2vwzZ/lobpFC8jnvr4cDFrSPVXYEQD5XWm35dMTLUa7LjkGfvJS A1cJFmDR2T8z6IIwkSvziK5MfXW4DJHMXVdE+rJBXWEdQIH4aEae3SfpEE8AfDLrGk5N tB0LmLK8wAEUxfVKEw5jvQHKkDQw0tOUlJpUPO9h5JlaidyXk2yHRz6OupD7E6mtxDBc ZjjKBeQ20rUU1g4b8xj1nPIIoWEMC4/n3YYn3N0cwSCfPSjwPl8s6UDlzT06S8YJb3O+ 51Q24WSq/ZQ665K3ine5Wt8YkAGajciet2LX7SalMco5HZVRnxvsysofnpQm/x0S5FzT IA6A== X-Forwarded-Encrypted: i=1; AJvYcCXKRGj3Cx5FCwBGUHYUCnrmfOly2iXE+IZYGh6QsKWh1lfzcbogAYmvEd6bA6M0mtOVupAdLaGJ6g==@kvack.org X-Gm-Message-State: AOJu0YwzHGfrRFb1aufyw5m0GEmc/AesyGhXhywkIRx1pdQQttfy7gzf 5jZP4Ty8XAYyrL62nzi2Mx2pkNDuCI8iSBkotprwraqzE18iXHao6iFKuAZvKlKKFWYT1bB0mwU FuD4kvSrLNG+IbvyaHMgLkfo8SYi0jWvOhxaIg4Q5GcbBvo5L X-Gm-Gg: ASbGncvLBgT4KcqTh9o6Wrbf0lBeWNNbb+FLTVyuy76Wlb9dBccaixSWE4afp70MYmp 2wYCfQYyKn4sN0LpxOqUBuNt+KDAawri4ietqkDpg7Kucm194VsStgfM2sJsa6m9mHtMOt+CLjy ElixsH0frRHfZzZhLbTPyTRPWEWQfTc1V7uhg8Df1NIN8a0UYIUdtLUuNn6OYWWXH99QFx3r7lu jDoZZWWgE3mWKINOX7lFYcuk9zWh3LwGwMeAqDceVeR2oFttdnWiXoTIcSQodiZ3Vk0KdiCZNI7 e83CxBbiOSoMOJZhzY1rFTxmKQjLKoUcV6qj8yimBKrpl4syPAqT7GrfcXVK/kywzA== X-Received: by 2002:a5d:5384:0:b0:38a:8ed1:c5c7 with SMTP id ffacd0b85a97d-38c520bdb45mr1846672f8f.46.1738151688842; Wed, 29 Jan 2025 03:54:48 -0800 (PST) X-Google-Smtp-Source: AGHT+IEQObdLJ4S/sxdMO/dY9vI6fd5p4fQOJgmwgX26yitgXbCNTkIYH7aPDZgCkfY36quLJ2o2xA== X-Received: by 2002:a5d:5384:0:b0:38a:8ed1:c5c7 with SMTP id ffacd0b85a97d-38c520bdb45mr1846636f8f.46.1738151688502; Wed, 29 Jan 2025 03:54:48 -0800 (PST) Received: from localhost (p200300cbc7053b0064b867195794bf13.dip0.t-ipconnect.de. [2003:cb:c705:3b00:64b8:6719:5794:bf13]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38c2a1c4006sm17020800f8f.94.2025.01.29.03.54.45 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 03:54:47 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v1 12/12] mm/rmap: keep mapcount untouched for device-exclusive entries Date: Wed, 29 Jan 2025 12:54:10 +0100 Message-ID: <20250129115411.2077152-13-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250129115411.2077152-1-david@redhat.com> References: <20250129115411.2077152-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: uxGnIROl4AK4DDe-vmAKVzRiRJgCiG4HSuJ4hRuYcns_1738151689 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Queue-Id: CBF4740003 X-Stat-Signature: wfkshtktkirh16zk8k98rma7r7afyu3s X-Rspam-User: X-Rspamd-Server: rspam12 X-HE-Tag: 1738151691-874674 X-HE-Meta: U2FsdGVkX1/VucadSj5WqUfRHR/J225n9r3Hzw6bsiJwY9prKCMGCHoTY2xqpvETAgDAszrrnLLxy8PLjsgh3tJ2QXO7hBS+49dl9KfVfVe1TsvnZuHwZyo+dV1vpjxs/6kz+LT1cdTVsctTMGyvaJXHhHr2CFGo8iwiufxyl+oFYCekOPn2qMw+cZzGe0B22//y1aoNoLkerTcaQ6RhPHJNe3znfDeRhqDeK65QxG1O7DEYeREHtc4ZwHqbKC+U2aI4YktMIGHftjwymEzdyVNDmUYXqkoBBu3IqdACSQxd+rA3uwEq6V8Xmg2qiVvE4YUKTeHR9OtvbshYMYfzbVVC/rNawMdGzsnoLTqRJZeQTnEIcD05qy2AwWEcT8MdhuTPvGqjDspi2NLZfgP1U1EMj91mDcZnkG+96OP+SZCjlankjaiZ2GwrBeOWU7MhfEHEtZUBXSn+GzD0mk/JGPrw00IaRMkBwMlVOGNGlGSDBXYRdamzMWzlV99qKolSdvR0+ti8uO1F4xOJsqCK+bsuabbkW7DmnDqIxOM9iQ5DOsOakGkEUZnJYjgfCSMVITfCqs5s6IDjXgsR7DbwGYZARkTZ8iVtEJsMXyeJdXDd9zoMOfnVvO8jsZrEB/P4za79u+dVqVwqPXc6KNws/NhJEnom11Ttc8y0NTDRvdHFetA+Jba50ckAmNk3XvGTfPoDRfx5eTELIeyuIcMo5idN/bpeRydUzjhrcx0MqZXYaXsBxysIay53RW66mI+kepBJBmesEFAv1u60prxb91MiuZ9enjhWc9g+0JiVEj4ym3SPMs8ji3M+RkvZDNskDRuHxzlAF+51CssSetaX07FxKWA/6z0yeVKSizuVLUUzMxXFgn3uD+mAZrfDHrDhjF5m47uYWJ5+dOMhLV3sLMIwL9gG71hKiI7L608zyGQlgqtDHGrlLVrL2HaqcEpcGBQwmmE8bgr3Eo3JcQr jhrrEZCe GB5KPEOxaAm3dNNVt2+aADCecpxguaEwhYP/RdKDSzXxetx+KoYBa3UBIifW0PBnfZniyMpWksQLlTFQ3WrFp8YYqllGGOtH4g05ph0V1eedH9UVTEUsocJtfYLqDUGzRrOEpyjks9UId9voB+DfVWFi1FyD1g2mqMokc4PLTPxMDbEFSFmC26uvgacDXNTjqlYJhpxeIu1hqF/0FrPo9O+9ArvFnmkIiMXkpgtPSfXElSig/uV0dXUCusD2a7s0Xu4sMWgmYgoJrdaw1eY7n+1R/AHBFRbXZHicj++iDYAW4x5YD5v1Qj6/4wBb++tXZTyv4EP6EK7V9w42nHr9ogXq45ajsC0S4s4NDcjldHHoSvNrbnprhiXfXL15w9DiphFTmHFCllc4KycTC1L4rHq2b1hYpsz7GBvMKA96hhQ3F8iZH72iOmOApMkw3Qj17NrU41+oK5aBEe9NpEnodfbeaM2Qo/3QXRP0g X-Bogosity: Ham, tests=bogofilter, spamicity=0.000012, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Now that conversion to device-exclusive does no longer perform an rmap walk and the main page_vma_mapped_walk() users were taught to properly handle nonswap entries, let's treat device-exclusive entries just as if they would be present, similar to how we handle device-private entries already. This fixes swapout/migration of folios with device-exclusive entries. Likely there are still some page_vma_mapped_walk() callers that are not fully prepared for these entries, and where we simply want to refuse !pte_present() entries. They have to be fixed independently; the ones in mm/rmap.c are prepared. Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Signed-off-by: David Hildenbrand --- mm/memory.c | 17 +---------------- mm/rmap.c | 7 ------- 2 files changed, 1 insertion(+), 23 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index db38d6ae4e74..cd689cd8a7c8 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -743,20 +743,6 @@ static void restore_exclusive_pte(struct vm_area_struct *vma, VM_BUG_ON_FOLIO(pte_write(pte) && (!folio_test_anon(folio) && PageAnonExclusive(page)), folio); - - /* - * No need to take a page reference as one was already - * created when the swap entry was made. - */ - if (folio_test_anon(folio)) - folio_add_anon_rmap_pte(folio, page, vma, address, RMAP_NONE); - else - /* - * Currently device exclusive access only supports anonymous - * memory so the entry shouldn't point to a filebacked page. - */ - WARN_ON_ONCE(1); - set_pte_at(vma->vm_mm, address, ptep, pte); /* @@ -1628,8 +1614,7 @@ static inline int zap_nonpresent_ptes(struct mmu_gather *tlb, */ WARN_ON_ONCE(!vma_is_anonymous(vma)); rss[mm_counter(folio)]--; - if (is_device_private_entry(entry)) - folio_remove_rmap_pte(folio, page, vma); + folio_remove_rmap_pte(folio, page, vma); folio_put(folio); } else if (!non_swap_entry(entry)) { /* Genuine swap entries, hence a private anon pages */ diff --git a/mm/rmap.c b/mm/rmap.c index 9e2002d97d6f..4acc9f6d743a 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2495,13 +2495,6 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, /* The pte is writable, uffd-wp does not apply. */ set_pte_at(mm, addr, fw.ptep, swp_pte); - /* - * TODO: The device-exclusive non-swap PTE holds a folio reference but - * does not count as a mapping (mapcount), which is wrong and must be - * fixed, otherwise RMAP walks don't behave as expected. - */ - folio_remove_rmap_pte(folio, page, vma); - folio_walk_end(&fw, vma); *foliop = folio; return page;