From patchwork Wed Apr 28 22:50:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12230195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F347C433B4 for ; Wed, 28 Apr 2021 22:51:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2DFA66143C for ; Wed, 28 Apr 2021 22:51:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2DFA66143C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C32406B0070; Wed, 28 Apr 2021 18:51:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C09A36B0071; Wed, 28 Apr 2021 18:51:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A81956B0072; Wed, 28 Apr 2021 18:51:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0140.hostedemail.com [216.40.44.140]) by kanga.kvack.org (Postfix) with ESMTP id 8DD256B0070 for ; Wed, 28 Apr 2021 18:51:20 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 403ED180AD820 for ; Wed, 28 Apr 2021 22:51:20 +0000 (UTC) X-FDA: 78083273520.09.295B344 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf14.hostedemail.com (Postfix) with ESMTP id BF0A2C0001FA for ; Wed, 28 Apr 2021 22:51:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619650279; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ikZqRIOJ+24TbcCJgwh1xPSL1L/jSOOAtcHmb6Z6UD4=; b=G7YjTuoyfI5GQOMquCgnhJ+VUuwB4gEJiCf5AgNnMRedcYi757GsrBGkaZiHyeMHD42JQy BKRvYyT2YywMEOk7sohuD2YL+SboW1vLJD3KM+Q8Fmrg6X7naWQ1DUxhrWW676LgXuaCgk JvjPh/pnE4GjBXpB7tIcY8Mj2PawRRU= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-410-RAFlfduUPr6X-P4QeAn8nA-1; Wed, 28 Apr 2021 18:51:16 -0400 X-MC-Unique: RAFlfduUPr6X-P4QeAn8nA-1 Received: by mail-qk1-f200.google.com with SMTP id h22-20020a05620a13f6b02902e3e9aad4bdso22326944qkl.14 for ; Wed, 28 Apr 2021 15:51:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ikZqRIOJ+24TbcCJgwh1xPSL1L/jSOOAtcHmb6Z6UD4=; b=f4JtYSb6AhcioxEXbpP6Hwex/AJ12ZP6AFg/rIFcl7AUbVfarCLMl+9j6vh7JFItDG zVJWDcQQZG7Mha7HsBLlcJLMudo5t7SkTqSEumqSv+JNhi91O7cyiiF10HNdmvz0Shyz zdLxUB9MuKpM3cNT2sncqalpCf68dyDMoKe9OH4u28N5JFATwGEXOQffczpOcrdMgIah JWc+FKdGUWIp72X45KSaRwiSOWvaqKLO/ccxgve1wtGJDUbo/mpTu3Lxoy5GsDz2zKDz qMVzLVwXSF0p2GYCm7fyzqUubN6gEY0LasrlhukssDkUy2MAUHzHuGwxh/Ldygwb04lW OmUw== X-Gm-Message-State: AOAM532BFyf4JyhALPltkIFEt1j6yqmeS+ak4y2zia2Q6iTBCxtjxj5Q 9XV3K67nCMwKfNkLQ6+7N+eq9zPE0ex6pwCy6XDi3pqQX1n5Tm9J11V0W8qkPGwJOqvbdAle5vS K/9SXZEkxtWwSncblAxnPQW9ff4fgqG4FpRWGIozehjlQDVLU2o2IUtpqZ48B X-Received: by 2002:a05:622a:589:: with SMTP id c9mr28868887qtb.363.1619650275096; Wed, 28 Apr 2021 15:51:15 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyrX3v3lU9B1pVnyP5Nb9hlKNKEN2NRRr8EhEd2xq4l8HH5y/69IZS/YQAW1T0WiwuZgL+rNg== X-Received: by 2002:a05:622a:589:: with SMTP id c9mr28868855qtb.363.1619650274762; Wed, 28 Apr 2021 15:51:14 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-77-184-145-104-227.dsl.bell.ca. [184.145.104.227]) by smtp.gmail.com with ESMTPSA id 191sm897459qkk.31.2021.04.28.15.51.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Apr 2021 15:51:14 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Mike Kravetz , peterx@redhat.com, Mike Rapoport , Axel Rasmussen , Andrea Arcangeli , Hugh Dickins , "Kirill A . Shutemov" , Andrew Morton , Jerome Glisse Subject: [PATCH 1/6] mm/thp: Simplify copying of huge zero page pmd when fork Date: Wed, 28 Apr 2021 18:50:25 -0400 Message-Id: <20210428225030.9708-2-peterx@redhat.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210428225030.9708-1-peterx@redhat.com> References: <20210428225030.9708-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: ng6ro3krk1mg7hsmyc77emyoco5ou6jk X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: BF0A2C0001FA Received-SPF: none (redhat.com>: No applicable sender policy available) receiver=imf14; identity=mailfrom; envelope-from=""; helo=us-smtp-delivery-124.mimecast.com; client-ip=170.10.133.124 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619650263-826352 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Huge zero page is handled in a special path in copy_huge_pmd(), however it should share most codes with a normal thp page. Trying to share more code with it by removing the special path. The only leftover so far is the huge zero page refcounting (mm_get_huge_zero_page()), because that's separately done with a global counter. This prepares for a future patch to modify the huge pmd to be installed, so that we don't need to duplicate it explicitly into huge zero page case too. Cc: Kirill A. Shutemov Signed-off-by: Peter Xu --- mm/huge_memory.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 98456017744d6..22bf2d0fff79b 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1076,17 +1076,13 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, * a page table. */ if (is_huge_zero_pmd(pmd)) { - struct page *zero_page; /* * get_huge_zero_page() will never allocate a new page here, * since we already have a zero page to copy. It just takes a * reference. */ - zero_page = mm_get_huge_zero_page(dst_mm); - set_huge_zero_page(pgtable, dst_mm, vma, addr, dst_pmd, - zero_page); - ret = 0; - goto out_unlock; + mm_get_huge_zero_page(dst_mm); + goto out_zero_page; } src_page = pmd_page(pmd); @@ -1110,6 +1106,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, get_page(src_page); page_dup_rmap(src_page, true); add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); +out_zero_page: mm_inc_nr_ptes(dst_mm); pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); From patchwork Wed Apr 28 22:50:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12230197 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF5BBC43460 for ; Wed, 28 Apr 2021 22:51:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 554256143C for ; Wed, 28 Apr 2021 22:51:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 554256143C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8FF0E6B0071; Wed, 28 Apr 2021 18:51:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DA3B6B0072; Wed, 28 Apr 2021 18:51:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6DC766B0073; Wed, 28 Apr 2021 18:51:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0234.hostedemail.com [216.40.44.234]) by kanga.kvack.org (Postfix) with ESMTP id 3BD756B0071 for ; Wed, 28 Apr 2021 18:51:21 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E4A70181AEF21 for ; Wed, 28 Apr 2021 22:51:20 +0000 (UTC) X-FDA: 78083273520.01.8B36A0A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf07.hostedemail.com (Postfix) with ESMTP id CA2B4A0009C5 for ; Wed, 28 Apr 2021 22:51:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619650279; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OG28X1p0LN5qmWCjh+VmHMXYqZhJcXnpmOsWEKY5ICY=; b=C+g0sPuPUKOb1XkL5goYl1EiC+MAkHxsxvfG+JYQDjPKYQNk5lv1BHsna3l+EJSNaeR6A3 RtnpvetPDRw96W+n3Sx0CU5ZOOAeIEemaUcnCjdSmxpxXQWWIe8hnhxJ+YD3JOmY912eQG sLHxMLSbgu8vnh9fVYVSLyPr3C+k2MM= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-177-v-Mo3GDiMuGqng6EpG-yiw-1; Wed, 28 Apr 2021 18:51:18 -0400 X-MC-Unique: v-Mo3GDiMuGqng6EpG-yiw-1 Received: by mail-qv1-f72.google.com with SMTP id x15-20020a0ce0cf0000b029019cb3e75c62so29307973qvk.15 for ; Wed, 28 Apr 2021 15:51:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OG28X1p0LN5qmWCjh+VmHMXYqZhJcXnpmOsWEKY5ICY=; b=EPY0dxFNmpzwHUKvb+dx5ddRR9RpvNddDAUHRAG+o2C/f58hSlTGZ9QRPS+6E5Coeg YU3jp1pCJjZfW3RM4QD6GA0BajNjZ+SaLFvsTojlC8EU03n6waIEg0HjvrRmVTXJX5Gs IsT9sKcYPTxjxjV95Zt0AcfT/MVlIZRokDxtJ1iNnG8weXl0luNPLIAU2mT03xT2ldqP 2nabDTVvCxSr2lhk9XP687e0gbNSS1nFaIgwelp9UJkj0Jf3XbJFB0HddKnxkwOAJxDo wnr20Np6VfMaIvkrqrKB/B6ZycsOQp3R+1mHWlwSOr7zDe73WLjWLtdTys5uD+3abt22 fqUA== X-Gm-Message-State: AOAM531+HIA/DJanQTQnKvG6FVMTDEKFWivaUmFymykivCRrGfL9ecji SddN5Ph3zU0DtLgpOGaO83uDXBOdtHjc6ca5FGV90Bi16S3kYixQOukzW6jRIz8OmPTDFDOeLbQ KZ9fwyUeCroTzbHs2qtr1cFOhoiZIPL2CqvqF/SIk1zr6IogmlmjPNElVg/kV X-Received: by 2002:a37:8ec4:: with SMTP id q187mr31300675qkd.381.1619650276894; Wed, 28 Apr 2021 15:51:16 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyi99fErJAPLQ1t4NVfvD0N51OiO720CW6bMsxTi+DZzEyVg/1oxclKoH8kWI9KuoB0CouZNQ== X-Received: by 2002:a37:8ec4:: with SMTP id q187mr31300646qkd.381.1619650276529; Wed, 28 Apr 2021 15:51:16 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-77-184-145-104-227.dsl.bell.ca. [184.145.104.227]) by smtp.gmail.com with ESMTPSA id 191sm897459qkk.31.2021.04.28.15.51.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Apr 2021 15:51:16 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Mike Kravetz , peterx@redhat.com, Mike Rapoport , Axel Rasmussen , Andrea Arcangeli , Hugh Dickins , "Kirill A . Shutemov" , Andrew Morton , Jerome Glisse Subject: [PATCH 2/6] mm/userfaultfd: Fix uffd-wp special cases for fork() Date: Wed, 28 Apr 2021 18:50:26 -0400 Message-Id: <20210428225030.9708-3-peterx@redhat.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210428225030.9708-1-peterx@redhat.com> References: <20210428225030.9708-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: angneho6axbgue6ei3u98kaj3934rmik X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: CA2B4A0009C5 Received-SPF: none (redhat.com>: No applicable sender policy available) receiver=imf07; identity=mailfrom; envelope-from=""; helo=us-smtp-delivery-124.mimecast.com; client-ip=216.205.24.124 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619650279-78047 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We tried to do something similar in b569a1760782 ("userfaultfd: wp: drop _PAGE_UFFD_WP properly when fork") previously, but it's not doing it all right.. A few fixes around the code path: 1. We were referencing VM_UFFD_WP vm_flags on the _old_ vma rather than the new vma. That's overlooked in b569a1760782, so it won't work as expected. Thanks to the recent rework on fork code (7a4830c380f3a8b3), we can easily get the new vma now, so switch the checks to that. 2. Dropping the uffd-wp bit in copy_huge_pmd() could be wrong if the huge pmd is a migration huge pmd. When it happens, instead of using pmd_uffd_wp(), we should use pmd_swp_uffd_wp(). The fix is simply to handle them separately. 3. Forget to carry over uffd-wp bit for a write migration huge pmd entry. This also happens in copy_huge_pmd(), where we converted a write huge migration entry into a read one. 4. In copy_nonpresent_pte(), drop uffd-wp if necessary for swap ptes. 5. In copy_present_page() when COW is enforced when fork(), we also need to pass over the uffd-wp bit if VM_UFFD_WP is armed on the new vma, and when the pte to be copied has uffd-wp bit set. Remove the comment in copy_present_pte() about this. It won't help a huge lot to only comment there, but comment everywhere would be an overkill. Let's assume the commit messages would help. Cc: Jerome Glisse Cc: Mike Rapoport Fixes: b569a1760782f3da03ff718d61f74163dea599ff Signed-off-by: Peter Xu --- include/linux/huge_mm.h | 2 +- mm/huge_memory.c | 23 ++++++++++------------- mm/memory.c | 25 +++++++++++++------------ 3 files changed, 24 insertions(+), 26 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 9626fda5efcea..60dad7c88d72b 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -10,7 +10,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf); int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr, - struct vm_area_struct *vma); + struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma); void huge_pmd_set_accessed(struct vm_fault *vmf, pmd_t orig_pmd); int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, pud_t *dst_pud, pud_t *src_pud, unsigned long addr, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 22bf2d0fff79b..20a4569895254 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1014,7 +1014,7 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr, - struct vm_area_struct *vma) + struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) { spinlock_t *dst_ptl, *src_ptl; struct page *src_page; @@ -1023,7 +1023,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, int ret = -ENOMEM; /* Skip if can be re-fill on fault */ - if (!vma_is_anonymous(vma)) + if (!vma_is_anonymous(dst_vma)) return 0; pgtable = pte_alloc_one(dst_mm); @@ -1037,14 +1037,6 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, ret = -EAGAIN; pmd = *src_pmd; - /* - * Make sure the _PAGE_UFFD_WP bit is cleared if the new VMA - * does not have the VM_UFFD_WP, which means that the uffd - * fork event is not enabled. - */ - if (!(vma->vm_flags & VM_UFFD_WP)) - pmd = pmd_clear_uffd_wp(pmd); - #ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION if (unlikely(is_swap_pmd(pmd))) { swp_entry_t entry = pmd_to_swp_entry(pmd); @@ -1055,11 +1047,15 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmd = swp_entry_to_pmd(entry); if (pmd_swp_soft_dirty(*src_pmd)) pmd = pmd_swp_mksoft_dirty(pmd); + if (pmd_swp_uffd_wp(*src_pmd)) + pmd = pmd_swp_mkuffd_wp(pmd); set_pmd_at(src_mm, addr, src_pmd, pmd); } add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); mm_inc_nr_ptes(dst_mm); pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); + if (!userfaultfd_wp(dst_vma)) + pmd = pmd_swp_clear_uffd_wp(pmd); set_pmd_at(dst_mm, addr, dst_pmd, pmd); ret = 0; goto out_unlock; @@ -1095,11 +1091,11 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, * best effort that the pinned pages won't be replaced by another * random page during the coming copy-on-write. */ - if (unlikely(page_needs_cow_for_dma(vma, src_page))) { + if (unlikely(page_needs_cow_for_dma(src_vma, src_page))) { pte_free(dst_mm, pgtable); spin_unlock(src_ptl); spin_unlock(dst_ptl); - __split_huge_pmd(vma, src_pmd, addr, false, NULL); + __split_huge_pmd(src_vma, src_pmd, addr, false, NULL); return -EAGAIN; } @@ -1109,8 +1105,9 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, out_zero_page: mm_inc_nr_ptes(dst_mm); pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); - pmdp_set_wrprotect(src_mm, addr, src_pmd); + if (!userfaultfd_wp(dst_vma)) + pmd = pmd_clear_uffd_wp(pmd); pmd = pmd_mkold(pmd_wrprotect(pmd)); set_pmd_at(dst_mm, addr, dst_pmd, pmd); diff --git a/mm/memory.c b/mm/memory.c index 045daf58608f7..a17a53a7dade6 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -708,10 +708,10 @@ struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr, static unsigned long copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, - pte_t *dst_pte, pte_t *src_pte, struct vm_area_struct *vma, - unsigned long addr, int *rss) + pte_t *dst_pte, pte_t *src_pte, struct vm_area_struct *dst_vma, + struct vm_area_struct *src_vma, unsigned long addr, int *rss) { - unsigned long vm_flags = vma->vm_flags; + unsigned long vm_flags = dst_vma->vm_flags; pte_t pte = *src_pte; struct page *page; swp_entry_t entry = pte_to_swp_entry(pte); @@ -780,6 +780,8 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, set_pte_at(src_mm, addr, src_pte, pte); } } + if (!userfaultfd_wp(dst_vma)) + pte = pte_swp_clear_uffd_wp(pte); set_pte_at(dst_mm, addr, dst_pte, pte); return 0; } @@ -845,6 +847,9 @@ copy_present_page(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma /* All done, just insert the new page copy in the child */ pte = mk_pte(new_page, dst_vma->vm_page_prot); pte = maybe_mkwrite(pte_mkdirty(pte), dst_vma); + if (userfaultfd_pte_wp(dst_vma, *src_pte)) + /* Uffd-wp needs to be delivered to dest pte as well */ + pte = pte_wrprotect(pte_mkuffd_wp(pte)); set_pte_at(dst_vma->vm_mm, addr, dst_pte, pte); return 0; } @@ -894,12 +899,7 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, pte = pte_mkclean(pte); pte = pte_mkold(pte); - /* - * Make sure the _PAGE_UFFD_WP bit is cleared if the new VMA - * does not have the VM_UFFD_WP, which means that the uffd - * fork event is not enabled. - */ - if (!(vm_flags & VM_UFFD_WP)) + if (!userfaultfd_wp(dst_vma)) pte = pte_clear_uffd_wp(pte); set_pte_at(dst_vma->vm_mm, addr, dst_pte, pte); @@ -974,7 +974,8 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, if (unlikely(!pte_present(*src_pte))) { entry.val = copy_nonpresent_pte(dst_mm, src_mm, dst_pte, src_pte, - src_vma, addr, rss); + dst_vma, src_vma, + addr, rss); if (entry.val) break; progress += 8; @@ -1051,8 +1052,8 @@ copy_pmd_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, || pmd_devmap(*src_pmd)) { int err; VM_BUG_ON_VMA(next-addr != HPAGE_PMD_SIZE, src_vma); - err = copy_huge_pmd(dst_mm, src_mm, - dst_pmd, src_pmd, addr, src_vma); + err = copy_huge_pmd(dst_mm, src_mm, dst_pmd, src_pmd, + addr, dst_vma, src_vma); if (err == -ENOMEM) return -ENOMEM; if (!err) From patchwork Wed Apr 28 22:50:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12230199 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A83F0C433B4 for ; Wed, 28 Apr 2021 22:51:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4A8856143F for ; Wed, 28 Apr 2021 22:51:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4A8856143F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 83AD96B0072; Wed, 28 Apr 2021 18:51:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8171C6B0073; Wed, 28 Apr 2021 18:51:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5A26B6B0074; Wed, 28 Apr 2021 18:51:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3BBCB6B0072 for ; Wed, 28 Apr 2021 18:51:22 -0400 (EDT) Received: from smtpin39.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 0388A2C96 for ; Wed, 28 Apr 2021 22:51:22 +0000 (UTC) X-FDA: 78083273604.39.374565D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf10.hostedemail.com (Postfix) with ESMTP id E98ED40002C7 for ; Wed, 28 Apr 2021 22:51:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619650281; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qqKszEE/+R68ngYo71RnOXLGHoNHIK32QNLimL1RNSc=; b=IVJjk8NZi4QI+UO/1FOeHjxzw2ejbb0ZIc/r/fiv0OKlJ+WRJMEzXA+D53SluBhtlIAQgD nKvEYgj6REyIfhTXLha2f0RikE9LQlKXvD76N+a1UTjfqFDhl9R1+FB36T1lC20URP2Vab Ao4b3LzUFT1LuvoaoR7JgDEBNRXPxsY= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-577-BBYZNn3rNjy8isENv0tApA-1; Wed, 28 Apr 2021 18:51:19 -0400 X-MC-Unique: BBYZNn3rNjy8isENv0tApA-1 Received: by mail-qt1-f200.google.com with SMTP id 10-20020ac8594a0000b02901b9f6ae286fso18193457qtz.23 for ; Wed, 28 Apr 2021 15:51:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qqKszEE/+R68ngYo71RnOXLGHoNHIK32QNLimL1RNSc=; b=monSuAwd9feBxQp/xjeagI1MrLdRcYnQlrW7/5+hXFEQQhg5kIVgd7blIiHO50gDHN O7uI4m7jYfn13s0pC1VFt+O5W8wqjpzER8kyEb3jnn0C7mqyT1r0XXESVKxCaOZvHyLO 6ZgHyOnL94MfNkAY1/2IXkN3xFLMCHfUy2094kRRT30VVzWx2PabTUbB9evV4ryUvuEp MVSs9EH5qBuKIBLNbXpLdLEqeez3Npt5eNgZ4ppB5giVgx5DtQXZ1/NLVJ2P7SyhLTuI CmvS8Grny+lcjJqI/rmut9mGrcqehqS4M4DTxybrM3baTXDW75x1P12fxNJ2qwW2AsFq zoTA== X-Gm-Message-State: AOAM532KduxVfbb+qABoqrSl2s4LW1Pqn8z9wR8PVJ5gFlZmOlr5wffj FN0JrKGcWlBztzilY4CPTFfNoO1L+vQ67yLypPTZtzPHig7Y32Cg8XqUEYWXBa3iB2zqXAaJfr9 MSh5o1ciWC9Xw+TQhRoJ+Bh4xtSLcqRL+6KWl+7WJUb15Q2lmfnV7lNk0RQXn X-Received: by 2002:a05:622a:94:: with SMTP id o20mr29753504qtw.158.1619650278450; Wed, 28 Apr 2021 15:51:18 -0700 (PDT) X-Google-Smtp-Source: ABdhPJziX7HpqcsgsbqQA9N4s/mqNOzeFKuNqQnRXLynrFsn+2UQuu62PHd5dx6d2x3VzOCu4IwPXQ== X-Received: by 2002:a05:622a:94:: with SMTP id o20mr29753476qtw.158.1619650278187; Wed, 28 Apr 2021 15:51:18 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-77-184-145-104-227.dsl.bell.ca. [184.145.104.227]) by smtp.gmail.com with ESMTPSA id 191sm897459qkk.31.2021.04.28.15.51.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Apr 2021 15:51:17 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Mike Kravetz , peterx@redhat.com, Mike Rapoport , Axel Rasmussen , Andrea Arcangeli , Hugh Dickins , "Kirill A . Shutemov" , Andrew Morton , Jerome Glisse Subject: [PATCH 3/6] mm/userfaultfd: Fix a few thp pmd missing uffd-wp bit Date: Wed, 28 Apr 2021 18:50:27 -0400 Message-Id: <20210428225030.9708-4-peterx@redhat.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210428225030.9708-1-peterx@redhat.com> References: <20210428225030.9708-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: E98ED40002C7 X-Stat-Signature: 8634adsnffghowx4rc8pj4ku3bxuansw X-Rspamd-Server: rspam02 Received-SPF: none (redhat.com>: No applicable sender policy available) receiver=imf10; identity=mailfrom; envelope-from=""; helo=us-smtp-delivery-124.mimecast.com; client-ip=216.205.24.124 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619650270-358158 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: These include: 1. When remove migration pmd entry, should keep the uffd-wp bit from swap pte. Note that we need to do this after setting write bit just in case we need to remove it. 2. When change huge pmd and convert write -> read migration entry, persist the same uffd-wp bit. 3. When convert pmd to swap entry, should drop the uffd-wp bit always. Signed-off-by: Peter Xu --- include/linux/swapops.h | 2 ++ mm/huge_memory.c | 4 ++++ 2 files changed, 6 insertions(+) diff --git a/include/linux/swapops.h b/include/linux/swapops.h index d9b7c9132c2f6..7dd57303bb0c3 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -258,6 +258,8 @@ static inline swp_entry_t pmd_to_swp_entry(pmd_t pmd) if (pmd_swp_soft_dirty(pmd)) pmd = pmd_swp_clear_soft_dirty(pmd); + if (pmd_swp_uffd_wp(pmd)) + pmd = pmd_swp_clear_uffd_wp(pmd); arch_entry = __pmd_to_swp_entry(pmd); return swp_entry(__swp_type(arch_entry), __swp_offset(arch_entry)); } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 20a4569895254..e6423fe11a8f2 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1823,6 +1823,8 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, newpmd = swp_entry_to_pmd(entry); if (pmd_swp_soft_dirty(*pmd)) newpmd = pmd_swp_mksoft_dirty(newpmd); + if (pmd_swp_uffd_wp(*pmd)) + newpmd = pmd_swp_mkuffd_wp(newpmd); set_pmd_at(mm, addr, pmd, newpmd); } goto unlock; @@ -3234,6 +3236,8 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) pmde = pmd_mksoft_dirty(pmde); if (is_write_migration_entry(entry)) pmde = maybe_pmd_mkwrite(pmde, vma); + if (pmd_swp_uffd_wp(*pvmw->pmd)) + pmde = pmd_wrprotect(pmd_mkuffd_wp(pmde)); flush_cache_range(vma, mmun_start, mmun_start + HPAGE_PMD_SIZE); if (PageAnon(new)) From patchwork Wed Apr 28 22:50:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12230201 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47E60C43462 for ; Wed, 28 Apr 2021 22:51:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C83FE6143F for ; Wed, 28 Apr 2021 22:51:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C83FE6143F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4456F6B0073; Wed, 28 Apr 2021 18:51:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3CC736B0074; Wed, 28 Apr 2021 18:51:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 26F656B0075; Wed, 28 Apr 2021 18:51:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0094.hostedemail.com [216.40.44.94]) by kanga.kvack.org (Postfix) with ESMTP id 0B8FA6B0073 for ; Wed, 28 Apr 2021 18:51:24 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id C505B3644 for ; Wed, 28 Apr 2021 22:51:23 +0000 (UTC) X-FDA: 78083273646.02.ECE30B4 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf29.hostedemail.com (Postfix) with ESMTP id 5200913A for ; Wed, 28 Apr 2021 22:51:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619650283; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aG6Es1lhsdq1W+SfyrFpv9ZTWYnqP0BhVj1uRKdeKnk=; b=gCyzTK+dD/1LNR8sei1HPglFjmQJoNWqEVZtbKDQPPDafyioyvGKZmGkDNYzd32I7K/rRI ZtYkJ1l9WroTF2W14RuVFg7IHM2aOseglFj+NOvwj72RzmJqzOPD1zqCcIuK2dyGj7BdwK STyzUaUK5YXh7/GVz2KuFfw4ywbQDXQ= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-156-xZTpu5RgOlS42vklXaUZ4g-1; Wed, 28 Apr 2021 18:51:21 -0400 X-MC-Unique: xZTpu5RgOlS42vklXaUZ4g-1 Received: by mail-qv1-f70.google.com with SMTP id l19-20020a0ce5130000b02901b6795e3304so9717472qvm.2 for ; Wed, 28 Apr 2021 15:51:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aG6Es1lhsdq1W+SfyrFpv9ZTWYnqP0BhVj1uRKdeKnk=; b=ZfZA9uQBwcno0vIrSHP4xtA3XxxdLNYnuCBArjf3Cv2E54V4sNVPHv7vlf1n6e6+cI Ups+1qf0detuqJWbFAtlCOj4nc2lCr5LUF0oiqMkWMRaYOqZ2sSH2C0N59bC70TQ4AB6 MRKDGY15K4AYHcw3qp3Bj3RyCpZbIC0U7CxZ3BaQ2VVMSryG9eKXUS8hOqLH8x26hZJw O2VwOByt2Q71LNk6HF1QGVhlSJBAD1cO/eU7ze8tSs/72ztzeRLuO/rypMNVo2uB8NK+ tgNry7woMCL5TvpB9S39S2O9aDFKxXAiHHtCmNzgAwK0A0Y56WeXJmcSyOjUIuw8eR1u YefA== X-Gm-Message-State: AOAM5308fT2RyTvi2Bsf/uon7kIIo9egmJtQm7IXGXC+fOm3Cdr0Befm HGWvEbRslWMnhFA4ORy2hhl8bNsmkTdJTvNCqlmKZR8FYx1Jppw8UuzM+n+OW+WX5aLE936wEMz QXu2ddHjZedN4NTn3+047InBoE2B3OiIoQwq3oqwtedNUEfJN3Vl/vq9f1vky X-Received: by 2002:a0c:9ae9:: with SMTP id k41mr24204958qvf.40.1619650280409; Wed, 28 Apr 2021 15:51:20 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzi0KqzO23V+VDmbdXTP1koTP2GA72Z0kaCSBKAzMlcmEcNe5JkFaEp7X/r2ioxQbKVKn1aRQ== X-Received: by 2002:a0c:9ae9:: with SMTP id k41mr24204929qvf.40.1619650280118; Wed, 28 Apr 2021 15:51:20 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-77-184-145-104-227.dsl.bell.ca. [184.145.104.227]) by smtp.gmail.com with ESMTPSA id 191sm897459qkk.31.2021.04.28.15.51.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Apr 2021 15:51:19 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Mike Kravetz , peterx@redhat.com, Mike Rapoport , Axel Rasmussen , Andrea Arcangeli , Hugh Dickins , "Kirill A . Shutemov" , Andrew Morton , Jerome Glisse Subject: [PATCH 4/6] mm/userfaultfd: Fail uffd-wp registeration if not supported Date: Wed, 28 Apr 2021 18:50:28 -0400 Message-Id: <20210428225030.9708-5-peterx@redhat.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210428225030.9708-1-peterx@redhat.com> References: <20210428225030.9708-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 5200913A X-Stat-Signature: pjjsu89z8fsq14hnrwjukeexrbmhawth Received-SPF: none (redhat.com>: No applicable sender policy available) receiver=imf29; identity=mailfrom; envelope-from=""; helo=us-smtp-delivery-124.mimecast.com; client-ip=170.10.133.124 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619650279-342838 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We should fail uffd-wp registration immediately if the arch does not even have CONFIG_HAVE_ARCH_USERFAULTFD_WP defined. That'll block also relevant ioctls on e.g. UFFDIO_WRITEPROTECT because that'll check against VM_UFFD_WP, which can only be applied with a success registration. Remove the WP feature bit too for those archs when handling UFFDIO_API ioctl. Signed-off-by: Peter Xu --- fs/userfaultfd.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 14f92285d04f8..5dd78238cc156 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -1304,8 +1304,12 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, vm_flags = 0; if (uffdio_register.mode & UFFDIO_REGISTER_MODE_MISSING) vm_flags |= VM_UFFD_MISSING; - if (uffdio_register.mode & UFFDIO_REGISTER_MODE_WP) + if (uffdio_register.mode & UFFDIO_REGISTER_MODE_WP) { +#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_WP + goto out; +#endif vm_flags |= VM_UFFD_WP; + } if (uffdio_register.mode & UFFDIO_REGISTER_MODE_MINOR) { #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR goto out; @@ -1942,6 +1946,9 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, uffdio_api.features = UFFD_API_FEATURES; #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR uffdio_api.features &= ~UFFD_FEATURE_MINOR_HUGETLBFS; +#endif +#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_WP + uffdio_api.features &= ~UFFD_FEATURE_PAGEFAULT_FLAG_WP; #endif uffdio_api.ioctls = UFFD_API_IOCTLS; ret = -EFAULT; From patchwork Wed Apr 28 22:50:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12230203 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E574CC433ED for ; Wed, 28 Apr 2021 22:51:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8939A61440 for ; Wed, 28 Apr 2021 22:51:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8939A61440 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 248946B0074; Wed, 28 Apr 2021 18:51:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F96E6B0075; Wed, 28 Apr 2021 18:51:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F1AD66B0078; Wed, 28 Apr 2021 18:51:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0145.hostedemail.com [216.40.44.145]) by kanga.kvack.org (Postfix) with ESMTP id D1ABD6B0074 for ; Wed, 28 Apr 2021 18:51:25 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 9694A180AD820 for ; Wed, 28 Apr 2021 22:51:25 +0000 (UTC) X-FDA: 78083273730.28.6710AFD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf25.hostedemail.com (Postfix) with ESMTP id 2A9F56000100 for ; Wed, 28 Apr 2021 22:51:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619650284; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mX5tB3J49H84VgNhuzdVmDuPOZ+GeLNjOa4r3iSgeg4=; b=JZ9Zr8IIyMdCkkSkknYDYSu8TQW2wfFv3zJV1s49mrdm54mqlFoR90ibUyOgXVQ4SNkdKr ab24FIUkwNAICHJpPhtV0RXSRD0x2xz2nKpyrYYELti17Zn8EYYp4zr4/+EH3KDVwB9Jw9 xmRktC1a0AGefvpuN9tdL5JYoYaskjM= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-89-ZiosHp4bMHyuEEXg9tcE_g-1; Wed, 28 Apr 2021 18:51:23 -0400 X-MC-Unique: ZiosHp4bMHyuEEXg9tcE_g-1 Received: by mail-qt1-f199.google.com with SMTP id w10-20020ac86b0a0000b02901ba74ac38c9so14368091qts.22 for ; Wed, 28 Apr 2021 15:51:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mX5tB3J49H84VgNhuzdVmDuPOZ+GeLNjOa4r3iSgeg4=; b=JaslLwioctNYkwxvHuoB+1JRKlbZesXhz/5z2HEW5kc5un+aEF4BlG8sGwgEOHENWr 9PEBtWq5cmaBbL55H+DCOJ+wAvJJo3JC6wWlSf+v+yeqRzPaZaeFtUYfP7HastS+xlRR R4QtxpJ/ppYSpL6i++z0ThPJLWkClaF0A/zM6y2Jmje6M2cD7ImJuXWENwRb8B0Ba8Io 5lWgdk4g4hBcOGhUev/eCNx6M3ppWgWkO7gnqCu8pqH2sfDr6l3vRAD3DyK81WYuEYLz qXKQ/3v8d+Dh4/w927eFgGM8MOTZUXD/EsXpYzDVem+fcfILrvEt+Li+jp76j7aiPa0c 5mqQ== X-Gm-Message-State: AOAM530p+4/n+Tt3OCuF5f1aEvzXu8l9WabHY/zA8Ns1DlrEdVVgXEzx eUg2DxWSbNkOxpI/uWJciPNrUGWX4hVgas+xeUV1YONrisJhswK+KT8SfzqVkOP2Vh//b1lqLZT rAmwBv7jTgzsFpNrvaqLI+VuN0U2VeWtNmOJXiueKOsaDkJp6jmQGmG3cyVGk X-Received: by 2002:a0c:fb4e:: with SMTP id b14mr31673249qvq.28.1619650282180; Wed, 28 Apr 2021 15:51:22 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz4HCkJm81ksUeMlHST386Jw+apuC8iK/I7PR3SG4syb6rigMTQEREtHbkFfHxe1EdAh2hbFg== X-Received: by 2002:a0c:fb4e:: with SMTP id b14mr31673214qvq.28.1619650281917; Wed, 28 Apr 2021 15:51:21 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-77-184-145-104-227.dsl.bell.ca. [184.145.104.227]) by smtp.gmail.com with ESMTPSA id 191sm897459qkk.31.2021.04.28.15.51.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Apr 2021 15:51:21 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Mike Kravetz , peterx@redhat.com, Mike Rapoport , Axel Rasmussen , Andrea Arcangeli , Hugh Dickins , "Kirill A . Shutemov" , Andrew Morton , Jerome Glisse Subject: [PATCH 5/6] mm/pagemap: Export uffd-wp protection information Date: Wed, 28 Apr 2021 18:50:29 -0400 Message-Id: <20210428225030.9708-6-peterx@redhat.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210428225030.9708-1-peterx@redhat.com> References: <20210428225030.9708-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 2A9F56000100 X-Stat-Signature: gpkhmg68hhuxugaaqhpx3oxcsyb3pfdr Received-SPF: none (redhat.com>: No applicable sender policy available) receiver=imf25; identity=mailfrom; envelope-from=""; helo=us-smtp-delivery-124.mimecast.com; client-ip=216.205.24.124 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619650279-234215 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Export the PTE/PMD status of uffd-wp to pagemap too. Signed-off-by: Peter Xu --- Documentation/admin-guide/mm/pagemap.rst | 2 ++ fs/proc/task_mmu.c | 9 +++++++++ 2 files changed, 11 insertions(+) diff --git a/Documentation/admin-guide/mm/pagemap.rst b/Documentation/admin-guide/mm/pagemap.rst index 340a5aee9b802..fb578fbbb76ca 100644 --- a/Documentation/admin-guide/mm/pagemap.rst +++ b/Documentation/admin-guide/mm/pagemap.rst @@ -21,6 +21,8 @@ There are four components to pagemap: * Bit 55 pte is soft-dirty (see :ref:`Documentation/admin-guide/mm/soft-dirty.rst `) * Bit 56 page exclusively mapped (since 4.2) + * Bit 57 pte is uffd-wp write-protected (since 5.13) (see + :ref:`Documentation/admin-guide/mm/userfaultfd.rst `) * Bits 57-60 zero * Bit 61 page is file-page or shared-anon (since 3.5) * Bit 62 page swapped diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index fc9784544b241..ce3b5f33b44c4 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1302,6 +1302,7 @@ struct pagemapread { #define PM_PFRAME_MASK GENMASK_ULL(PM_PFRAME_BITS - 1, 0) #define PM_SOFT_DIRTY BIT_ULL(55) #define PM_MMAP_EXCLUSIVE BIT_ULL(56) +#define PM_UFFD_WP BIT_ULL(57) #define PM_FILE BIT_ULL(61) #define PM_SWAP BIT_ULL(62) #define PM_PRESENT BIT_ULL(63) @@ -1375,10 +1376,14 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm, page = vm_normal_page(vma, addr, pte); if (pte_soft_dirty(pte)) flags |= PM_SOFT_DIRTY; + if (pte_uffd_wp(pte)) + flags |= PM_UFFD_WP; } else if (is_swap_pte(pte)) { swp_entry_t entry; if (pte_swp_soft_dirty(pte)) flags |= PM_SOFT_DIRTY; + if (pte_swp_uffd_wp(pte)) + flags |= PM_UFFD_WP; entry = pte_to_swp_entry(pte); if (pm->show_pfn) frame = swp_type(entry) | @@ -1426,6 +1431,8 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, flags |= PM_PRESENT; if (pmd_soft_dirty(pmd)) flags |= PM_SOFT_DIRTY; + if (pmd_uffd_wp(pmd)) + flags |= PM_UFFD_WP; if (pm->show_pfn) frame = pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); @@ -1444,6 +1451,8 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, flags |= PM_SWAP; if (pmd_swp_soft_dirty(pmd)) flags |= PM_SOFT_DIRTY; + if (pmd_swp_uffd_wp(pmd)) + flags |= PM_UFFD_WP; VM_BUG_ON(!is_pmd_migration_entry(pmd)); page = migration_entry_to_page(entry); } From patchwork Wed Apr 28 22:50:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12230205 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4529C433B4 for ; Wed, 28 Apr 2021 22:51:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4A6B56143F for ; Wed, 28 Apr 2021 22:51:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4A6B56143F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 703276B0075; Wed, 28 Apr 2021 18:51:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6699C6B0078; Wed, 28 Apr 2021 18:51:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E0836B007B; Wed, 28 Apr 2021 18:51:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0108.hostedemail.com [216.40.44.108]) by kanga.kvack.org (Postfix) with ESMTP id 2F7B86B0075 for ; Wed, 28 Apr 2021 18:51:28 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id D80A2181AEF21 for ; Wed, 28 Apr 2021 22:51:27 +0000 (UTC) X-FDA: 78083273814.22.CCCCF4B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf23.hostedemail.com (Postfix) with ESMTP id C94AEA0003A2 for ; Wed, 28 Apr 2021 22:51:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619650286; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pvGdUvHXhr1/W39+miDgJTu6lUz1WoLUPPok+oPerLQ=; b=fu7Ou/sKElY/6Ph+TvyHB/e+j1UHzJkjRgf/Pjgl8fnwMmwBxf9ciHtrSGxWJkKo4hH8P6 UR7ZcJYID6HyMG4SdS3hcalFtJqVdRu5byVBYsdJ6woPQEAbFrQvO3GOU4NvujUlLACG8p Jb0qBoO8aBpZn9zkb3MuufOGcEkE02w= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-175-Dbp017nBMAmOxqN74yUaPA-1; Wed, 28 Apr 2021 18:51:25 -0400 X-MC-Unique: Dbp017nBMAmOxqN74yUaPA-1 Received: by mail-qv1-f69.google.com with SMTP id s13-20020a0cdc0d0000b02901bbc03198caso6814607qvk.22 for ; Wed, 28 Apr 2021 15:51:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pvGdUvHXhr1/W39+miDgJTu6lUz1WoLUPPok+oPerLQ=; b=lYvhJa/FQ3DkBrmvpLmWdwk4iUbrxUKVwS9tBkXGr1cF2jv3fdMhvV+dJuFIE8w8m1 SasddcuDWevmBleYgegLoS2ktbW8agOHbXijJh0UvSO7rpaD1jWzR/iblynGXY0glhrb ZyFmqgasDspoXYs7v7f0XxJJ93SKHgKy3n6sBAdME2KbqFJHdFWdt5+jKhWURz0lgevG mqh8jNagFzLYgIFjCc8+lptY1mdQ8WO9pxFjV5nQIvSFY/Bwz+4pGPsqaVpNyEVP6njr V2Lkn8poPug0bCDfqnCCj++zqaNP6eXCp/kkDq+/tPNe9FU/eWFPBm0M9y4NKpvQb9yr EqLg== X-Gm-Message-State: AOAM531gOw6XXPwJDKLm3/IoqAlx5DNyX0Mbh6Q+5b7jat+59ROL54by +rnwCKl5/2NLndjDwiBMwfp/oaGBVo2C3M/elmzWNT0ThhJVIH3qbyQcMoBnDVMh0Qye/yldSWP Nf4QVY+7hMCjXx17eCSCR2KdY0BaB8rE5T2O9k/JaGDBr6dVikee1m7b436je X-Received: by 2002:a05:620a:74b:: with SMTP id i11mr22108327qki.445.1619650284158; Wed, 28 Apr 2021 15:51:24 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwwEgCtCU64i2kK9aQbDrUJjCGfyCMRzR2OuJxyRzNOzWF+EO1/lGcK/SKkGiPz+jvxPbchFA== X-Received: by 2002:a05:620a:74b:: with SMTP id i11mr22108292qki.445.1619650283835; Wed, 28 Apr 2021 15:51:23 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-77-184-145-104-227.dsl.bell.ca. [184.145.104.227]) by smtp.gmail.com with ESMTPSA id 191sm897459qkk.31.2021.04.28.15.51.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Apr 2021 15:51:23 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Mike Kravetz , peterx@redhat.com, Mike Rapoport , Axel Rasmussen , Andrea Arcangeli , Hugh Dickins , "Kirill A . Shutemov" , Andrew Morton , Jerome Glisse Subject: [PATCH 6/6] userfaultfd/selftests: Add pagemap uffd-wp test Date: Wed, 28 Apr 2021 18:50:30 -0400 Message-Id: <20210428225030.9708-7-peterx@redhat.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210428225030.9708-1-peterx@redhat.com> References: <20210428225030.9708-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: C94AEA0003A2 X-Stat-Signature: rqbt4hsizk7rzfemx4ny5zsrpi6nxw9h Received-SPF: none (redhat.com>: No applicable sender policy available) receiver=imf23; identity=mailfrom; envelope-from=""; helo=us-smtp-delivery-124.mimecast.com; client-ip=216.205.24.124 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619650282-894602 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add one anonymous specific test to start using pagemap. With pagemap support, we can directly read the uffd-wp bit from pgtable without triggering any fault, so it's easier to do sanity checks in unit tests. Meanwhile this test also leverages the newly introduced MADV_PAGEOUT madvise function to test swap ptes with uffd-wp bit set, and across fork()s. Signed-off-by: Peter Xu --- tools/testing/selftests/vm/userfaultfd.c | 154 +++++++++++++++++++++++ 1 file changed, 154 insertions(+) diff --git a/tools/testing/selftests/vm/userfaultfd.c b/tools/testing/selftests/vm/userfaultfd.c index 6339aeaeeff8b..93eae095b61e6 100644 --- a/tools/testing/selftests/vm/userfaultfd.c +++ b/tools/testing/selftests/vm/userfaultfd.c @@ -1170,6 +1170,144 @@ static int userfaultfd_minor_test(void) return stats.missing_faults != 0 || stats.minor_faults != nr_pages; } +#define BIT_ULL(nr) (1ULL << (nr)) +#define PM_SOFT_DIRTY BIT_ULL(55) +#define PM_MMAP_EXCLUSIVE BIT_ULL(56) +#define PM_UFFD_WP BIT_ULL(57) +#define PM_FILE BIT_ULL(61) +#define PM_SWAP BIT_ULL(62) +#define PM_PRESENT BIT_ULL(63) + +static int pagemap_open(void) +{ + int fd = open("/proc/self/pagemap", O_RDONLY); + + if (fd < 0) + err("open pagemap"); + + return fd; +} + +static uint64_t pagemap_read_vaddr(int fd, void *vaddr) +{ + uint64_t value; + int ret; + + ret = pread(fd, &value, sizeof(uint64_t), + ((uint64_t)vaddr >> 12) * sizeof(uint64_t)); + if (ret != sizeof(uint64_t)) + err("pread() on pagemap failed"); + + return value; +} + +/* This macro let __LINE__ works in err() */ +#define pagemap_check_wp(value, wp) do { \ + if (!!(value & PM_UFFD_WP) != wp) \ + err("pagemap uffd-wp bit error: 0x%"PRIx64, value); \ + } while (0) + +static int pagemap_test_fork(bool present) +{ + pid_t child = fork(); + uint64_t value; + int fd, result; + + if (!child) { + /* Open the pagemap fd of the child itself */ + fd = pagemap_open(); + value = pagemap_read_vaddr(fd, area_dst); + /* + * After fork() uffd-wp bit should be gone as long as we're + * without UFFD_FEATURE_EVENT_FORK + */ + pagemap_check_wp(value, false); + /* Succeed */ + exit(0); + } + waitpid(child, &result, 0); + return result; +} + +static void userfaultfd_pagemap_test(unsigned int test_pgsize) +{ + struct uffdio_register uffdio_register; + int pagemap_fd; + uint64_t value; + + /* Pagemap tests uffd-wp only */ + if (!test_uffdio_wp) + return; + + /* Not enough memory to test this page size */ + if (test_pgsize > nr_pages * page_size) + return; + + printf("testing uffd-wp with pagemap (pgsize=%u): ", test_pgsize); + /* Flush so it doesn't flush twice in parent/child later */ + fflush(stdout); + + uffd_test_ops->release_pages(area_dst); + + if (test_pgsize > page_size) { + /* This is a thp test */ + if (madvise(area_dst, nr_pages * page_size, MADV_HUGEPAGE)) + err("madvise(MADV_HUGEPAGE) failed"); + } else if (test_pgsize == page_size) { + /* This is normal page test; force no thp */ + if (madvise(area_dst, nr_pages * page_size, MADV_NOHUGEPAGE)) + err("madvise(MADV_NOHUGEPAGE) failed"); + } + + if (userfaultfd_open(0)) + err("userfaultfd_open"); + + uffdio_register.range.start = (unsigned long) area_dst; + uffdio_register.range.len = nr_pages * page_size; + uffdio_register.mode = UFFDIO_REGISTER_MODE_WP; + if (ioctl(uffd, UFFDIO_REGISTER, &uffdio_register)) + err("register failed"); + + pagemap_fd = pagemap_open(); + + /* Touch the page */ + *area_dst = 1; + wp_range(uffd, (uint64_t)area_dst, test_pgsize, true); + value = pagemap_read_vaddr(pagemap_fd, area_dst); + pagemap_check_wp(value, true); + /* Make sure uffd-wp bit dropped when fork */ + if (pagemap_test_fork(true)) + err("Detected stall uffd-wp bit in child"); + + /* Exclusive required or PAGEOUT won't work */ + if (!(value & PM_MMAP_EXCLUSIVE)) + err("multiple mapping detected: 0x%"PRIx64, value); + + if (madvise(area_dst, test_pgsize, MADV_PAGEOUT)) + err("madvise(MADV_PAGEOUT) failed"); + + /* Uffd-wp should persist even swapped out */ + value = pagemap_read_vaddr(pagemap_fd, area_dst); + pagemap_check_wp(value, true); + /* Make sure uffd-wp bit dropped when fork */ + if (pagemap_test_fork(false)) + err("Detected stall uffd-wp bit in child"); + + /* Unprotect; this tests swap pte modifications */ + wp_range(uffd, (uint64_t)area_dst, page_size, false); + value = pagemap_read_vaddr(pagemap_fd, area_dst); + pagemap_check_wp(value, false); + + /* Fault in the page from disk */ + *area_dst = 2; + value = pagemap_read_vaddr(pagemap_fd, area_dst); + pagemap_check_wp(value, false); + + close(pagemap_fd); + close(uffd); + printf("done\n"); +} + static int userfaultfd_stress(void) { void *area; @@ -1341,6 +1479,22 @@ static int userfaultfd_stress(void) } close(uffd); + + if (test_type == TEST_ANON) { + /* + * shmem/hugetlb won't be able to run since they have different + * behavior on fork() (file-backed memory normally drops ptes + * directly when fork), meanwhile the pagemap test will verify + * pgtable entry of fork()ed child. + */ + userfaultfd_pagemap_test(page_size); + /* + * Hard-code for x86_64 for now for 2M THP, as x86_64 is + * currently the only one that supports uffd-wp + */ + userfaultfd_pagemap_test(page_size * 512); + } + return userfaultfd_zeropage_test() || userfaultfd_sig_test() || userfaultfd_events_test() || userfaultfd_minor_test(); }