From patchwork Mon Apr 17 19:53:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13214488 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C983C77B76 for ; Mon, 17 Apr 2023 19:53:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 155ED6B007B; Mon, 17 Apr 2023 15:53:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 106B78E0002; Mon, 17 Apr 2023 15:53:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE9498E0001; Mon, 17 Apr 2023 15:53:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id DD5E96B007B for ; Mon, 17 Apr 2023 15:53:28 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id B09CF1A060E for ; Mon, 17 Apr 2023 19:53:28 +0000 (UTC) X-FDA: 80691932496.11.7351EB3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 87F671C0007 for ; Mon, 17 Apr 2023 19:53:26 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=c297cepQ; spf=pass (imf21.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681761206; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4kH7JNUmkrK1TIcnPTAjFCorjkRaA4Lvx3pRbnAdGF0=; b=ThM8Jy0LAaDM4K3UmyKPeNc7X82rafzd8eFJPBBiyMrTxZkTsonGSwi9ZTHEW5krYSU4vz HB+FLxxxkAD/jUzsAbJpzJpkp+WK51uVjwmBCdYrU4MBk/WK10uf34VIqcOM8xKdRfXvv9 4dG040yvy9fh6pvdPhKO52AWZfjOCwM= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=c297cepQ; spf=pass (imf21.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681761206; a=rsa-sha256; cv=none; b=lLP089XlLaAAeITQV8EqWbDchVct1xxAmQd7oqUhSMt7+W6/zERB5FtPnGaqZyfILSxEpp 7TwQXgd/NgTPhrkKcF/eNEKpdaJe6GB9U5TVmFQ21SGzGr8JeREI1VjYp3/SpA6IyYe/l8 iHF78fluULzb3pgj4Aytc7S40tJu1B4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681761205; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4kH7JNUmkrK1TIcnPTAjFCorjkRaA4Lvx3pRbnAdGF0=; b=c297cepQFvarQ5pqK9s4YFmCcQdkWGT24LACG5guK2QVmkeEON52kVB9DzL1e/WK9bWeyL mLeWEpuWUGCSn57eTBVaYV6rrgyo01CHH52w7Yqd89XZwcenyWqhAs6h6vgbVV5u0bDqz6 CQBw4zrsEQt0ldLpxMLP3eXOCDGKkTQ= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-646-K9zGQh5mOe2nqAkhKfK69g-1; Mon, 17 Apr 2023 15:53:22 -0400 X-MC-Unique: K9zGQh5mOe2nqAkhKfK69g-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-3ef3116d1dcso1936961cf.1 for ; Mon, 17 Apr 2023 12:53:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681761202; x=1684353202; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4kH7JNUmkrK1TIcnPTAjFCorjkRaA4Lvx3pRbnAdGF0=; b=CoueRYAMUG82sZOFMgW1zN4Az4hOZlEnc6/NktHozN6+vZAtxJhRz81tmE43pTTEU2 E/cqsmiBQvmVipTJUuCRzeHKUoOmlVO/62YQfrsRde5DqMQKRSWFY2L4YWsRuVma8I55 ymE0v3gNHIb+Pn6ZcvMUkwhtXB8VVmuMXS+eT1hHPAeAvc0rEEsgIgayXVbs4CLtXRiI 7ObxJkxOIS/KS8Ii+R2OgjpY1atFwCiNVV82Wuj4ai0NQzloyHA7fg/4aZIS1yWVcxmE ZGZGReSs6THQqyI2b2MgCBHjD73IujLFrtnxrA/jfy1ewYVmXUjwc8qVTi05MP2RZlo1 TPdw== X-Gm-Message-State: AAQBX9eIkNNLZkEI/LsiLt8xLqo037NwfUuVNeQ8FxmJ6nGFxXWlHgVv 2GiDU83QSq+O0muf8BoHjeagqaoA7hAyl1oxIrklAW11uShLiiSSOuEaW8SsqV1lN3v1LpttLb8 rGOKzYupYrMo= X-Received: by 2002:a05:622a:19a4:b0:3e8:e986:b20b with SMTP id u36-20020a05622a19a400b003e8e986b20bmr17654467qtc.6.1681761201983; Mon, 17 Apr 2023 12:53:21 -0700 (PDT) X-Google-Smtp-Source: AKy350ZImFWbK1bH7N466hKMggFlbQGVKGvl3FHNzHOn9K4F9JpZRZ/yhdls5xIATBP5zpu3FcayoQ== X-Received: by 2002:a05:622a:19a4:b0:3e8:e986:b20b with SMTP id u36-20020a05622a19a400b003e8e986b20bmr17654449qtc.6.1681761201723; Mon, 17 Apr 2023 12:53:21 -0700 (PDT) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-40-70-52-229-124.dsl.bell.ca. [70.52.229.124]) by smtp.gmail.com with ESMTPSA id r17-20020ac87ef1000000b003edfb5d7637sm1731278qtc.73.2023.04.17.12.53.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Apr 2023 12:53:21 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , Andrea Arcangeli , =?utf-8?q?Mika_Penttil=C3=A4?= , Andrew Morton , peterx@redhat.com, Axel Rasmussen , Nadav Amit , David Hildenbrand , linux-stable Subject: [PATCH v2 1/6] mm/hugetlb: Fix uffd-wp during fork() Date: Mon, 17 Apr 2023 15:53:12 -0400 Message-Id: <20230417195317.898696-2-peterx@redhat.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230417195317.898696-1-peterx@redhat.com> References: <20230417195317.898696-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 87F671C0007 X-Stat-Signature: 3sg5x6ihrr6nxc11a7szog1rm94itzhk X-HE-Tag: 1681761206-546014 X-HE-Meta: U2FsdGVkX1+xDJpitoZTmvaPj9Brmxa22wpPdZ4ouG9W2RTy+n33DBTRSatC6fVtSmPHksuU3LI1S1lYcnRttcdGUKQmrK43Qq1DvrZFLvZajkn0xF9HAqwgnrZvlspXhRKV6HEDOtVKAjJGHg4OStW1lXSacnPmk8QjToeQsPbvs30zcCwC4HaZ2fQnO9MTBurLMM/oSwVtpRkK6cZej7HUG4vlxCnMtaLwV6DMbYi0S5BkUUj4r/nquYH1JXkdFKxY7IeN50bwHSbnPZ/AJBThbCE4WJU++ujs1f70zOl6dHUmiMCVXLazJq3vLJtpk6CV20TOO0QgeqG8tsyKJiwPEgFqPh6cB5Wful0O/nM3+RLkvlC2i3TG4pxaxY89eEssZyFKc8hCULPznqwCF21TiUWf/Owc+N5WI65oj8C0k/uuMn7sX4jYbC8W3IhBOZ8qAMb0bJ5j/EViVVtpVonnvn8AeaSmD4OS3Jby041WR/IxOMSCy8tEBkM3MglWF+bRqPMKxcPOnRfxEpDHtlWprUpqnUxYlOkiALbcx8yhIgJ/3KOkLhQU6OXY1Eb9CIYQqXqc+1Y28DTpt4uXSCc5pN/GZ5gkLsmsWr6jZMUdzb6tgguz1zwOFUC+chMGbR3dRTtjL2ySN0z4Ze7O71eyZtKgH7ru/u8LXUVjuwv572uhBF4l3Cv72dNs8wyC/XwglThnxjyEEk8peplhFACZoDT34QDVXu/y9rpHqWirMTfQpwqv1z38gTNCGamAYda773aHysp0fTbWX7Yh9LGmSY2BhooRDwimqhQBQot893SnFR+zAKnEhemcBPO6lERWkmXEfEOyn+ja7Fanz9D0HSm5aYUasoVZLjOiudgi/KcYUNsP/8hbtO53lh/EOunZUup4oVxsSJflLPge78MDXj0iA5jQH4daH9nlG8M+CY50T8E2tT4fPD9LLcnmbEO3N7G0VL8k7/3LAOK Fts40O4D ZXFtcnK9yc+VIad+Pa6ClxYUXPsXfP5Av+zJcjVj/Zkd4BGe6M1WvrMyFMcOWZPmc0i5g9rlj8rIQ54prvuer++swsIYGkTk5Re/SGybxtfCIoakuIbKibpqF+KhUr2NVSlB8xc1ZTMmWWeC3CPiljEz42UBCqGe1dxg02E1i0WHvoaxxpN5j6iX/Ckn0t/I2D5eexM0ToNZYHqfyBXkC6pZfAdCuUgiNNk7mIeC4XOF/9XV/Cj3eX1lq6JRUPiPoYGD61RLA9xcH3o0hJZgA42FfO4PhfpHtk2fUDcn1CTWMJk9Ms5iZAJlvaWMSwoAEfXyeED/vjVU1tICzhAeW07XzczhANZA5txtR4PDFXvGldOyLbsk5u2I6U62f1dT3TY0IwmvzxvKw4KvIehPTK4F9cDY2C2ZL1X45X/8ADlEr28/I+oYbnKGAfNa1JqbTNRB3JDJSoypdJ54KdcKvlaGcFxIO4ikyI/ssYmiHf1eeVIuslrinMNNTt9Tmk3jGbwkp7S9cGherLkfuQOGgupy0RQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There're a bunch of things that were wrong: - Reading uffd-wp bit from a swap entry should use pte_swp_uffd_wp() rather than huge_pte_uffd_wp(). - When copying over a pte, we should drop uffd-wp bit when !EVENT_FORK (aka, when !userfaultfd_wp(dst_vma)). - When doing early CoW for private hugetlb (e.g. when the parent page was pinned), uffd-wp bit should be properly carried over if necessary. No bug reported probably because most people do not even care about these corner cases, but they are still bugs and can be exposed by the recent unit tests introduced, so fix all of them in one shot. Cc: linux-stable Fixes: bc70fbf269fd ("mm/hugetlb: handle uffd-wp during fork()") Reviewed-by: David Hildenbrand Signed-off-by: Peter Xu --- mm/hugetlb.c | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index f16b25b1a6b9..0213efaf31be 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4953,11 +4953,15 @@ static bool is_hugetlb_entry_hwpoisoned(pte_t pte) static void hugetlb_install_folio(struct vm_area_struct *vma, pte_t *ptep, unsigned long addr, - struct folio *new_folio) + struct folio *new_folio, pte_t old) { + pte_t newpte = make_huge_pte(vma, &new_folio->page, 1); + __folio_mark_uptodate(new_folio); hugepage_add_new_anon_rmap(new_folio, vma, addr); - set_huge_pte_at(vma->vm_mm, addr, ptep, make_huge_pte(vma, &new_folio->page, 1)); + if (userfaultfd_wp(vma) && huge_pte_uffd_wp(old)) + newpte = huge_pte_mkuffd_wp(newpte); + set_huge_pte_at(vma->vm_mm, addr, ptep, newpte); hugetlb_count_add(pages_per_huge_page(hstate_vma(vma)), vma->vm_mm); folio_set_hugetlb_migratable(new_folio); } @@ -5032,14 +5036,12 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, */ ; } else if (unlikely(is_hugetlb_entry_hwpoisoned(entry))) { - bool uffd_wp = huge_pte_uffd_wp(entry); - - if (!userfaultfd_wp(dst_vma) && uffd_wp) + if (!userfaultfd_wp(dst_vma)) entry = huge_pte_clear_uffd_wp(entry); set_huge_pte_at(dst, addr, dst_pte, entry); } else if (unlikely(is_hugetlb_entry_migration(entry))) { swp_entry_t swp_entry = pte_to_swp_entry(entry); - bool uffd_wp = huge_pte_uffd_wp(entry); + bool uffd_wp = pte_swp_uffd_wp(entry); if (!is_readable_migration_entry(swp_entry) && cow) { /* @@ -5050,10 +5052,10 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, swp_offset(swp_entry)); entry = swp_entry_to_pte(swp_entry); if (userfaultfd_wp(src_vma) && uffd_wp) - entry = huge_pte_mkuffd_wp(entry); + entry = pte_swp_mkuffd_wp(entry); set_huge_pte_at(src, addr, src_pte, entry); } - if (!userfaultfd_wp(dst_vma) && uffd_wp) + if (!userfaultfd_wp(dst_vma)) entry = huge_pte_clear_uffd_wp(entry); set_huge_pte_at(dst, addr, dst_pte, entry); } else if (unlikely(is_pte_marker(entry))) { @@ -5114,7 +5116,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, /* huge_ptep of dst_pte won't change as in child */ goto again; } - hugetlb_install_folio(dst_vma, dst_pte, addr, new_folio); + hugetlb_install_folio(dst_vma, dst_pte, addr, + new_folio, src_pte_old); spin_unlock(src_ptl); spin_unlock(dst_ptl); continue; @@ -5132,6 +5135,9 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, entry = huge_pte_wrprotect(entry); } + if (!userfaultfd_wp(dst_vma)) + entry = huge_pte_clear_uffd_wp(entry); + set_huge_pte_at(dst, addr, dst_pte, entry); hugetlb_count_add(npages, dst); }