From patchwork Thu Apr 13 23:11:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13210718 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1AB46C77B6E for ; Thu, 13 Apr 2023 23:11:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CD8C96B0075; Thu, 13 Apr 2023 19:11:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C86EE900004; Thu, 13 Apr 2023 19:11:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B00156B007B; Thu, 13 Apr 2023 19:11:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A1F5A6B0075 for ; Thu, 13 Apr 2023 19:11:29 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4C90B80492 for ; Thu, 13 Apr 2023 23:11:29 +0000 (UTC) X-FDA: 80677916298.25.B12DD19 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf08.hostedemail.com (Postfix) with ESMTP id CEBEA160003 for ; Thu, 13 Apr 2023 23:11:26 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bCwt7HAh; spf=pass (imf08.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681427487; a=rsa-sha256; cv=none; b=iZ67PTWYzwg1Bel+88FpIDTMbhVLoUblslDn4n/QfPkFd28HqgedFhma/1duEI13HOfM5a vv3chWjXe6AUbYgDH+0L+guWYeIGTD86CK3YRge/v411JvVTcKeMOpUZ9Uniiezt4eKsS0 H862AAgPySsfEvcDyq+HS0VdaeBDifY= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bCwt7HAh; spf=pass (imf08.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681427487; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yQk3ShNsJBmwBAeq1XAgk+wt78LOM0H1LyrpvOkMJhI=; b=v6nnu475SjLnImhlYaoQZVJmhZ+Ue7naTG3sajVfF0thikDn3pojaXoa3Wg+OyQ1I1woM7 anv5PbPeHNcwlBzK15MDTm1/klP2dripqvFHCF+ur6hhuIFg6JFbmxqziBJeCG7ViXXbhQ e1QVuFfDYO/AwqKS74A8r3h8wEFQbb4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681427486; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yQk3ShNsJBmwBAeq1XAgk+wt78LOM0H1LyrpvOkMJhI=; b=bCwt7HAhsgdXxOZXgvT5oI1qo1zYjKg+4Q30CAS6Q8L7f1RCrQpNcUn0/alWqJztZKjgLQ ph4Jm9lYbtyw9sWw+uQwnuX83Psn+9a+5kzvKd7evcfNyZ4y2P1VLn3dEeb3vEgC+/HdZ5 fmSdFdAhINOGrE6ZqntSO3gqDX92cTw= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-3-Mey82hocNwi9AmpvwO4GVw-1; Thu, 13 Apr 2023 19:11:25 -0400 X-MC-Unique: Mey82hocNwi9AmpvwO4GVw-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-3e947d81d7dso3391291cf.1 for ; Thu, 13 Apr 2023 16:11:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681427484; x=1684019484; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yQk3ShNsJBmwBAeq1XAgk+wt78LOM0H1LyrpvOkMJhI=; b=ft2rtEhmkhxKR9uQpgsLDfM1v7GKzfHtODT9ic44VaYWZ7lJn2agS3ZF02l/mnCUiX 29/gmApaigOr2Fb5z6BP46p7Gmt9gU87YnBJm0d39r8zjwtFLhSfvvLgTLh1QKdjyMqg fIGB7m5tJ+g6czgl5DtrabCa8ZmgqpQgPkTj+EU2ISwpCipAhFCyV9k4I5H4qCJ9XlsZ cKF/rytJePQvsR3dNzOQ/fiJO8F8hLkEX7GxAgg6/gADLV6AwXq1peJjq/zDprsr7+Zn xlj4sABspKMxOmslv56jHEzspoGHZUSF7wV1Xkb/lgPKkZWOTZFsOpVhrG9N8YOtZ8tK Uksg== X-Gm-Message-State: AAQBX9f2ybymKWbLnNrTVkYvkiXfbo0xL5gPgdwgiAecO90dogSJlZHZ ZmcHu2tJaveaYoK1yH4FAOmsaOYAAyt7T5CdLlRmzdTppZsAMFROuKgTImg9AYRpQpvdmtwo4z3 6Z4U8kBB+JFM= X-Received: by 2002:a05:622a:19a1:b0:3e3:98cf:68ae with SMTP id u33-20020a05622a19a100b003e398cf68aemr1101140qtc.2.1681427484546; Thu, 13 Apr 2023 16:11:24 -0700 (PDT) X-Google-Smtp-Source: AKy350bljVojz9f9gM3R2vNG1p3GyeG+24EcdnppJPHidpEMFzohOSepYYLSiIIjIIlL/AyTNhMu9Q== X-Received: by 2002:a05:622a:19a1:b0:3e3:98cf:68ae with SMTP id u33-20020a05622a19a100b003e398cf68aemr1101122qtc.2.1681427484321; Thu, 13 Apr 2023 16:11:24 -0700 (PDT) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-40-70-52-229-124.dsl.bell.ca. [70.52.229.124]) by smtp.gmail.com with ESMTPSA id g3-20020ac81243000000b003ea1b97acfasm612446qtj.49.2023.04.13.16.11.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Apr 2023 16:11:23 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Axel Rasmussen , Andrew Morton , David Hildenbrand , peterx@redhat.com, Mike Kravetz , Nadav Amit , Andrea Arcangeli , linux-stable Subject: [PATCH 1/6] mm/hugetlb: Fix uffd-wp during fork() Date: Thu, 13 Apr 2023 19:11:15 -0400 Message-Id: <20230413231120.544685-2-peterx@redhat.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230413231120.544685-1-peterx@redhat.com> References: <20230413231120.544685-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Queue-Id: CEBEA160003 X-Rspamd-Server: rspam01 X-Stat-Signature: 79qoakygjofe691nmkwxdu7jgnngfnrz X-HE-Tag: 1681427486-773595 X-HE-Meta: U2FsdGVkX1+447wIMil3bLOrCr8SbxzLQEoZPUIAPCZ+4ECczvwoSgBXzVF5+IKvIJUuvIHYvlp7700wX+3fkT4u9nZDghhlgIrHi0wE3eW3iqfbTU7dUyo6t6GYNRL+bzoM6F5/3Lh++gzN/vxm2ZfrCMqMZWn0XIvpTGFSX09bfv5EwKsB0xcmGwcC6SPMbAwTu9YKuW9wH2s3i+2dh0l07eT8qxP0lKsz4f2Ay56AnituAeQ21ICZRDd6tUcjsGv9HMHzPsIBMLe3nuWrt80ggfn/gRGXzQQf4hi7smQxpt74HwIzLDPyWkR23xJXuFMV3edXqgGy8+Lsthxz3CbzbO5QluLmN/A4cwkGlDWTvNIZwmvZe+plGoXULre5MIllKEhX6In3152Cb1eqpEuzZJ6XwjmFTOZQNjWvuePbdx04D9oDr+HuoSScYU+qwGjEhOS+/3O/e3QMEjXsmbo8knWmn/J8GN62ywREaRW+JMmZOqVWhz96UDx7IYBKtkK+MNDTkEZ9WQeJn39InhLeo9nV05v7zQn7A0WSBRO2oPdTo0SLMGWWCnRd9jjtocnpukFy7R/SefZN7YkzJ0TOLIhj8XsOpVaih2y1VZAVc+7ByHNcepiZm8PCkUHlTthYQQfAatJiXmCTdAPFxdvI9L7df10jZXXvUsNkRRcnUCukqv95H7Za+VaG24BJtwgUstQ2WDVmXHK2iNKR5IID/0nJZk56vL1oSnrut/5PLTScWRDm/3ad/UefKfE2KgfJsIJBDVdnzR9NplleRpMTb/wvdjGD5r73z9b2YMA5VVWgYLYNv4neWTSexAZSq10xnrtYJ6qF96pFtY+3uLgPgbRTKuVfnFSCzjRZYX8Gksp0+ICLmnDu1lYZ3QjPoVB8npIags6y7NlCWq8Oil/JE8QX1ZYejUKYg9aRZE6pYvO6VINv1Gd+LelJwSXAZOQut+64a9Fi5OQ1X+O Q6lKfy1O Abj5EGG3uUtECTxQ5eYlH6oHHHKm+B2IWd2vkqqYCtr9pXmNL3XnxMBYNvxbBRUOuVOO9LyA3S5GmoQJ2Hyo/FoH9OCXLCSw4uLzm70HphhHWofkbmHHCXP59PEe5evYbIMxFSFwwkR1ApE99LOfE+uqUAx7ynftNoU/WdWocpl2IhqTfrYgF3WYIYLZPZvNyi8aUKXqRn4uNeHVXtXF0JZTupDT9iySvHfBBF6Q5cQOViDqAHoZTbReCwGEj1JYuQYssfAnYblEdNKi/wS+xs0VG4HBcm+kJpFadWHaR/9DQYrCwZp7bEgqOsDuQyJKz1DMGAmI8emq0xWSV+TgSPntc62tsZvltuoRcXWKB/o0J7fzcCWz33O3cQejiOuB5dwUmZF+8cCXPKUa+/nEa7HpxsqPyrg35NCkHHVj6vQjXZ66bIKeXsRyi/FwN9MXgxihl2jtybhxpWI5Z3bvMhbikfbBAaG976PNWh2y+Ru6qP4ttJJ2FATMJjIX274//t/fHkOW63xTfc8VpR7SC6hyOvA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There're a bunch of things that were wrong: - Reading uffd-wp bit from a swap entry should use pte_swp_uffd_wp() rather than huge_pte_uffd_wp(). - When copying over a pte, we should drop uffd-wp bit when !EVENT_FORK (aka, when !userfaultfd_wp(dst_vma)). - When doing early CoW for private hugetlb (e.g. when the parent page was pinned), uffd-wp bit should be properly carried over if necessary. No bug reported probably because most people do not even care about these corner cases, but they are still bugs and can be exposed by the recent unit tests introduced, so fix all of them in one shot. Cc: linux-stable Fixes: bc70fbf269fd ("mm/hugetlb: handle uffd-wp during fork()") Signed-off-by: Peter Xu Reviewed-by: David Hildenbrand --- mm/hugetlb.c | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index f16b25b1a6b9..7320e64aacc6 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4953,11 +4953,15 @@ static bool is_hugetlb_entry_hwpoisoned(pte_t pte) static void hugetlb_install_folio(struct vm_area_struct *vma, pte_t *ptep, unsigned long addr, - struct folio *new_folio) + struct folio *new_folio, pte_t old) { + pte_t newpte = make_huge_pte(vma, &new_folio->page, 1); + __folio_mark_uptodate(new_folio); hugepage_add_new_anon_rmap(new_folio, vma, addr); - set_huge_pte_at(vma->vm_mm, addr, ptep, make_huge_pte(vma, &new_folio->page, 1)); + if (userfaultfd_wp(vma) && huge_pte_uffd_wp(old)) + newpte = huge_pte_mkuffd_wp(newpte); + set_huge_pte_at(vma->vm_mm, addr, ptep, newpte); hugetlb_count_add(pages_per_huge_page(hstate_vma(vma)), vma->vm_mm); folio_set_hugetlb_migratable(new_folio); } @@ -5032,14 +5036,11 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, */ ; } else if (unlikely(is_hugetlb_entry_hwpoisoned(entry))) { - bool uffd_wp = huge_pte_uffd_wp(entry); - - if (!userfaultfd_wp(dst_vma) && uffd_wp) + if (!userfaultfd_wp(dst_vma)) entry = huge_pte_clear_uffd_wp(entry); set_huge_pte_at(dst, addr, dst_pte, entry); } else if (unlikely(is_hugetlb_entry_migration(entry))) { swp_entry_t swp_entry = pte_to_swp_entry(entry); - bool uffd_wp = huge_pte_uffd_wp(entry); if (!is_readable_migration_entry(swp_entry) && cow) { /* @@ -5049,11 +5050,12 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, swp_entry = make_readable_migration_entry( swp_offset(swp_entry)); entry = swp_entry_to_pte(swp_entry); - if (userfaultfd_wp(src_vma) && uffd_wp) - entry = huge_pte_mkuffd_wp(entry); + if (userfaultfd_wp(src_vma) && + pte_swp_uffd_wp(entry)) + entry = pte_swp_mkuffd_wp(entry); set_huge_pte_at(src, addr, src_pte, entry); } - if (!userfaultfd_wp(dst_vma) && uffd_wp) + if (!userfaultfd_wp(dst_vma)) entry = huge_pte_clear_uffd_wp(entry); set_huge_pte_at(dst, addr, dst_pte, entry); } else if (unlikely(is_pte_marker(entry))) { @@ -5114,7 +5116,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, /* huge_ptep of dst_pte won't change as in child */ goto again; } - hugetlb_install_folio(dst_vma, dst_pte, addr, new_folio); + hugetlb_install_folio(dst_vma, dst_pte, addr, + new_folio, src_pte_old); spin_unlock(src_ptl); spin_unlock(dst_ptl); continue; @@ -5132,6 +5135,9 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, entry = huge_pte_wrprotect(entry); } + if (!userfaultfd_wp(dst_vma)) + entry = huge_pte_clear_uffd_wp(entry); + set_huge_pte_at(dst, addr, dst_pte, entry); hugetlb_count_add(npages, dst); }