From patchwork Thu May 27 20:19:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285251 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B541AC4708B for ; Thu, 27 May 2021 20:19:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F2019613EB for ; Thu, 27 May 2021 20:19:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F2019613EB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 528706B006E; Thu, 27 May 2021 16:19:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4FD2B6B0070; Thu, 27 May 2021 16:19:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 375CD6B0071; Thu, 27 May 2021 16:19:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0070.hostedemail.com [216.40.44.70]) by kanga.kvack.org (Postfix) with ESMTP id F08976B0070 for ; Thu, 27 May 2021 16:19:37 -0400 (EDT) Received: from smtpin37.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 955E68249980 for ; Thu, 27 May 2021 20:19:37 +0000 (UTC) X-FDA: 78188126394.37.158B81F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 49A97C005A3E for ; Thu, 27 May 2021 20:19:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622146776; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hhEVhq4PGqE11D2YgrHebPE630Tm9aX+8YBhc8WEcaU=; b=QoL0zb2rMTGuCQMTLYv/wm9YFpCHaeu4CRBXBzi9F7GE1sPF5d1fx/IZEtOOFeOwMie6OS PAg14k0c5GpK1cullAUhE+epQhc+yyTCPBAY88iEfxffKIo3FBrJXQ2clfBuqUoDL57MHX TxnvUaRZXZH4bis1LcxgnqPJ0APOYRo= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-18-SccHNAOgP2WXd6JRleZvlw-1; Thu, 27 May 2021 16:19:34 -0400 X-MC-Unique: SccHNAOgP2WXd6JRleZvlw-1 Received: by mail-qv1-f69.google.com with SMTP id i14-20020a0cf10e0000b02901eeced6480dso1039203qvl.4 for ; Thu, 27 May 2021 13:19:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hhEVhq4PGqE11D2YgrHebPE630Tm9aX+8YBhc8WEcaU=; b=VxRskClfXif9RW14VJ69eTvcmpLVk8BL4YJ8AwmSHdSh1IJBPE1UvSGRyiTllr5Y/x fnE1BVZIUpQ+ZOtgFm/wH0Mts2A9HiVK0SYoVv9+YAjtwwAwPl6z1ImE79dhT4MAcwkq XxYtVrtPxPYeVEjQ8kU0gDCNZYDyaU/ZXFx1Ljmy7+qCwFrd1m62VhNXVt6izWJBihU0 hkGnUPgkD856xjrlNA+zsvM5dkqgoa2ijLWgWtSFT0eZITNGiUSvUyvamn23H/etOC27 Zo2k8PgwfHvWim5nv3+zmhqbQDQaH6Vk0VXCSj2LyFOVi/oLQ1wASzW4AQryxSr+WA0e SnOQ== X-Gm-Message-State: AOAM53339y6SkWutsfIcVCh6NR5iTOJvr6SZqA2IIxHkPk2PjNCw3bRX pgyi/CJhkLL2aRsJvU5JrZDTshDy//9uMe+Zi5elcTELZAo2ZH0vsWoezFSVo2eTL00zkVkBU6Y G/WP4fWpwFNY= X-Received: by 2002:a05:620a:56a:: with SMTP id p10mr329560qkp.238.1622146774398; Thu, 27 May 2021 13:19:34 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwhQTpN08TmWtg7+wKKOD9e8MtoWozMDaw4Mi24l38BYVLrAve1MtC/JueAUjGIQC5CxsYQeQ== X-Received: by 2002:a05:620a:56a:: with SMTP id p10mr329530qkp.238.1622146774179; Thu, 27 May 2021 13:19:34 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id u14sm2089536qkp.80.2021.05.27.13.19.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:19:33 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Axel Rasmussen , "Kirill A . Shutemov" , Hugh Dickins , Andrew Morton , Miaohe Lin , Mike Rapoport , Jerome Glisse , Andrea Arcangeli , Nadav Amit , Mike Kravetz , peterx@redhat.com, Jason Gunthorpe , Matthew Wilcox Subject: [PATCH v3 01/27] mm/shmem: Unconditionally set pte dirty in mfill_atomic_install_pte Date: Thu, 27 May 2021 16:19:01 -0400 Message-Id: <20210527201927.29586-2-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QoL0zb2r; spf=none (imf06.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 49A97C005A3E X-Stat-Signature: q8m84j7fufkcmwx8msqjkfwfqgpdg8jn X-HE-Tag: 1622146769-535047 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It was conditionally done previously, as there's one shmem special case that we use SetPageDirty() instead. However that's not necessary and it should be easier and cleaner to do it unconditionally in mfill_atomic_install_pte(). The most recent discussion about this is here, where Hugh explained the history of SetPageDirty() and why it's possible that it's not required at all: https://lore.kernel.org/lkml/alpine.LSU.2.11.2104121657050.1097@eggly.anvils/ Currently mfill_atomic_install_pte() has three callers: 1. shmem_mfill_atomic_pte 2. mcopy_atomic_pte 3. mcontinue_atomic_pte After the change: case (1) should have its SetPageDirty replaced by the dirty bit on pte (so we unify them together, finally), case (2) should have no functional change at all as it has page_in_cache==false, case (3) may add a dirty bit to the pte. However since case (3) is UFFDIO_CONTINUE for shmem, it's merely 100% sure the page is dirty after all, so should not make a real difference either. This should make it much easier to follow on which case will set dirty for uffd, as we'll simply set it all now for all uffd related ioctls. Meanwhile, no special handling of SetPageDirty() if there's no need. Cc: Hugh Dickins Cc: Axel Rasmussen Cc: Andrea Arcangeli Signed-off-by: Peter Xu --- mm/shmem.c | 1 - mm/userfaultfd.c | 3 +-- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index ccce139d4e5c..4085a5cf4a13 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2449,7 +2449,6 @@ int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, shmem_recalc_inode(inode); spin_unlock_irq(&info->lock); - SetPageDirty(page); unlock_page(page); return 0; out_delete_from_cache: diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 9ce5a3793ad4..462fa6e25e03 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -69,10 +69,9 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, pgoff_t offset, max_off; _dst_pte = mk_pte(page, dst_vma->vm_page_prot); + _dst_pte = pte_mkdirty(_dst_pte); if (page_in_cache && !vm_shared) writable = false; - if (writable || !page_in_cache) - _dst_pte = pte_mkdirty(_dst_pte); if (writable) { if (wp_copy) _dst_pte = pte_mkuffd_wp(_dst_pte); From patchwork Thu May 27 20:19:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285255 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F3BEC4708A for ; Thu, 27 May 2021 20:19:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DA75D6054E for ; Thu, 27 May 2021 20:19:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DA75D6054E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 749FE6B0070; Thu, 27 May 2021 16:19:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7225D6B0071; Thu, 27 May 2021 16:19:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 54BFC6B0072; Thu, 27 May 2021 16:19:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0067.hostedemail.com [216.40.44.67]) by kanga.kvack.org (Postfix) with ESMTP id 1C01D6B0070 for ; Thu, 27 May 2021 16:19:43 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id AC88B181AEF2A for ; Thu, 27 May 2021 20:19:42 +0000 (UTC) X-FDA: 78188126604.05.FE598E1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 40BE52000ADE for ; Thu, 27 May 2021 20:19:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622146781; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f9xr4bMl+zJeAQFvcMC4T+nTSZSbNTKgkWVTyL0bma8=; b=KRIpc8Pj2m3MFxsAUPf7cjc2XxMNgZ2SLCHi9Nxp7uOTDkGPGDBR2qGQ/mEhvYoZPh9C8i qJ7iHDgRZTBJjkuvJ7Nk/bYNYs9Wdkrhv78UTo331iKsLZjuuCVGxbZU/f+Vti6r4rm+Ch Q+GHACU+ymRAoqSj+42yHZNZbsrzv8Y= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-575-YvOfOQI5PWKrMUZCZS0Tug-1; Thu, 27 May 2021 16:19:38 -0400 X-MC-Unique: YvOfOQI5PWKrMUZCZS0Tug-1 Received: by mail-qk1-f200.google.com with SMTP id x12-20020ae9f80c0000b02903a6a80ade45so1294920qkh.22 for ; Thu, 27 May 2021 13:19:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=f9xr4bMl+zJeAQFvcMC4T+nTSZSbNTKgkWVTyL0bma8=; b=cGPw9vTF6spymFiwoKXp0NwO+p+Na3kaWkNE5aQcZmMDXsWHnVPS7uzm8/GUDAVwN1 VvwPtxvFs3LEG62YCCdrt15cTN+aa92UjgDIPvqtBBeppkWCTgK5LQ/9p3xLkz2UcBbj KK/jQ1pia5qU+lTtcljU4ALcsG2ODgjE+r5GGs7g1Fvy5nIOEZbjSyGdqV/YMRQkIgRL iUn6mctKnX2GXW8cZy3yQrxJ9+mwEGQmF65mVNE93o8yBoXyS/s9wnjk99FJlNKHRb9p kJM3+gXkekHOVtlIxL3LEtyq/xqErKGP4Tvfwcf9bHOOoS2LLLM0sAA++g5IbMQ3EQG9 mEPA== X-Gm-Message-State: AOAM5302m69RbafeFjofDErwO4XO2/OU2MHJJlT0St/itEI+B5/qj9vv V7ZAt1SiJ6dv6wtK4n4ZZoWQkYZmgBb5K6V01Sp4JeVXFxB+DJIc18WorkgMCB1LmIC5RZ9JP3x WDVMAJ5p7Vcw= X-Received: by 2002:a05:6214:934:: with SMTP id dk20mr276632qvb.26.1622146777633; Thu, 27 May 2021 13:19:37 -0700 (PDT) X-Google-Smtp-Source: ABdhPJx93jjVodboMgC/3iEgSR5YS6lBTH8CpTnxClpAjjIJe0S75YVhvKxzrlxBwhIfF9hncVRBmA== X-Received: by 2002:a05:6214:934:: with SMTP id dk20mr276611qvb.26.1622146777383; Thu, 27 May 2021 13:19:37 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id u14sm2089536qkp.80.2021.05.27.13.19.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:19:36 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Axel Rasmussen , "Kirill A . Shutemov" , Hugh Dickins , Andrew Morton , Miaohe Lin , Mike Rapoport , Jerome Glisse , Andrea Arcangeli , Nadav Amit , Mike Kravetz , peterx@redhat.com, Jason Gunthorpe , Matthew Wilcox Subject: [PATCH v3 02/27] shmem/userfaultfd: Take care of UFFDIO_COPY_MODE_WP Date: Thu, 27 May 2021 16:19:02 -0400 Message-Id: <20210527201927.29586-3-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 40BE52000ADE Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KRIpc8Pj; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf11.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com X-Rspamd-Server: rspam03 X-Stat-Signature: yyt88gqmipz5jp68hre97o8hbyc14juh X-HE-Tag: 1622146772-442731 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Firstly, pass wp_copy into shmem_mfill_atomic_pte() through the stack. Then apply the UFFD_WP bit properly when the UFFDIO_COPY on shmem is with UFFDIO_COPY_MODE_WP, then wp_copy lands mfill_atomic_install_pte() which is newly introduced very recently. We need to make sure shmem_mfill_atomic_pte() will always set the dirty bit in pte even if UFFDIO_COPY_MODE_WP is set. After the rework of minor fault series on shmem we need to slightly touch up the logic there, since uffd-wp needs to be applied even if writable==false previously (e.g., for shmem private mapping). Note: we must do pte_wrprotect() if !writable in mfill_atomic_install_pte(), as mk_pte() could return a writable pte (e.g., when VM_SHARED on a shmem file). Signed-off-by: Peter Xu --- include/linux/shmem_fs.h | 4 ++-- mm/shmem.c | 4 ++-- mm/userfaultfd.c | 23 +++++++++++++++-------- 3 files changed, 19 insertions(+), 12 deletions(-) diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h index a69ea4d97fdd..0b6a8e871036 100644 --- a/include/linux/shmem_fs.h +++ b/include/linux/shmem_fs.h @@ -128,11 +128,11 @@ extern int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, struct vm_area_struct *dst_vma, unsigned long dst_addr, unsigned long src_addr, - bool zeropage, + bool zeropage, bool wp_copy, struct page **pagep); #else /* !CONFIG_SHMEM */ #define shmem_mfill_atomic_pte(dst_mm, dst_pmd, dst_vma, dst_addr, \ - src_addr, zeropage, pagep) ({ BUG(); 0; }) + src_addr, zeropage, wp_copy, pagep) ({ BUG(); 0; }) #endif /* CONFIG_SHMEM */ #endif /* CONFIG_USERFAULTFD */ diff --git a/mm/shmem.c b/mm/shmem.c index 4085a5cf4a13..5b648cbae37a 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2368,7 +2368,7 @@ int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, struct vm_area_struct *dst_vma, unsigned long dst_addr, unsigned long src_addr, - bool zeropage, + bool zeropage, bool wp_copy, struct page **pagep) { struct inode *inode = file_inode(dst_vma->vm_file); @@ -2439,7 +2439,7 @@ int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, goto out_release; ret = mfill_atomic_install_pte(dst_mm, dst_pmd, dst_vma, dst_addr, - page, true, false); + page, true, wp_copy); if (ret) goto out_delete_from_cache; diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 462fa6e25e03..3636f5be6390 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -70,14 +70,22 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, _dst_pte = mk_pte(page, dst_vma->vm_page_prot); _dst_pte = pte_mkdirty(_dst_pte); - if (page_in_cache && !vm_shared) + /* Don't write if uffd-wp wr-protected */ + if (wp_copy) { + _dst_pte = pte_mkuffd_wp(_dst_pte); writable = false; - if (writable) { - if (wp_copy) - _dst_pte = pte_mkuffd_wp(_dst_pte); - else - _dst_pte = pte_mkwrite(_dst_pte); } + /* Don't write if page cache privately mapped */ + if (page_in_cache && !vm_shared) + writable = false; + if (writable) + _dst_pte = pte_mkwrite(_dst_pte); + else + /* + * We need this to make sure write bit removed; as mk_pte() + * could return a pte with write bit set. + */ + _dst_pte = pte_wrprotect(_dst_pte); dst_pte = pte_offset_map_lock(dst_mm, dst_pmd, dst_addr, &ptl); @@ -515,11 +523,10 @@ static __always_inline ssize_t mfill_atomic_pte(struct mm_struct *dst_mm, err = mfill_zeropage_pte(dst_mm, dst_pmd, dst_vma, dst_addr); } else { - VM_WARN_ON_ONCE(wp_copy); err = shmem_mfill_atomic_pte(dst_mm, dst_pmd, dst_vma, dst_addr, src_addr, mode != MCOPY_ATOMIC_NORMAL, - page); + wp_copy, page); } return err; From patchwork Thu May 27 20:19:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285257 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6346EC4707F for ; Thu, 27 May 2021 20:19:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0D3E36054E for ; Thu, 27 May 2021 20:19:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0D3E36054E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A07456B0071; Thu, 27 May 2021 16:19:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B8126B0072; Thu, 27 May 2021 16:19:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8590D8D0001; Thu, 27 May 2021 16:19:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0034.hostedemail.com [216.40.44.34]) by kanga.kvack.org (Postfix) with ESMTP id 4E2A46B0071 for ; Thu, 27 May 2021 16:19:45 -0400 (EDT) Received: from smtpin38.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id ECE89BBC5 for ; Thu, 27 May 2021 20:19:44 +0000 (UTC) X-FDA: 78188126688.38.4AC7B63 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 54872E005F2C for ; Thu, 27 May 2021 20:19:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622146784; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wtpikodp79ycSP9XpJQBScIRhvTfN0ZcMiLJ1ThP9JQ=; b=dy4i3t47swMFHKoto79JZRoQVk/5nTd4ofnX4tRGbZ72u7geDJzWgSrmhBYMIOO0OaqZ1U WlIvZUkPS2F2dFp21u32nbNjUl+rrJLr8n3ZTF5NoAxKhet8CkxP3Mi+Y5jeiuNvfYwKKe Q/2v4apCns4FKd3lxDPjCIA12vajYrc= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-272-kcQa0sgDOCqt98qeH0vc7A-1; Thu, 27 May 2021 16:19:40 -0400 X-MC-Unique: kcQa0sgDOCqt98qeH0vc7A-1 Received: by mail-qk1-f199.google.com with SMTP id s4-20020a3790040000b02902fa7aa987e8so1319187qkd.14 for ; Thu, 27 May 2021 13:19:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wtpikodp79ycSP9XpJQBScIRhvTfN0ZcMiLJ1ThP9JQ=; b=JL4uO+7EiH/kLrLk59OTxBAvXylFFwI3lEK6V1SepKWn7pCN/zQ46QUcbbzKFwaOtN rkElEl+owXs+xc+W/y0PFWUMH9S75+p1mibGe0/ZYP3sOx61lAAzIKSvi99M2U8+LulM j/XkaWmSi6UXB/QhgG+wYe71Z4PqtHkeOUvSOdfnZAarRA+xETfaaPuqc40YDWkZq2zp nvU2yOQl/jPNa/rda7fbSA00d9xBmMecTDJAh7jIDOhfGRgW3650KSnnayf5+4Rk99D7 ahE4TzaAFy1+4RBaon1MwGreTa2k2P6Wg55dNnXNYoO7+KCqElzkcj/7o2ZJd8xDjfeA UUGA== X-Gm-Message-State: AOAM533/LlZShriL55e6yalMlTWCKwx+gY3+81xjWufqq+Qkk9CIMBDH rpD6a3+DjFsUDEQYE8Gb3mtAmJAEXxExh0yg938lGBaFfAeO3WzpZjSl0Vj9b4lx+eJPBXX7Uvd qqwtXorTPa4c= X-Received: by 2002:a37:9b84:: with SMTP id d126mr286656qke.209.1622146779775; Thu, 27 May 2021 13:19:39 -0700 (PDT) X-Google-Smtp-Source: ABdhPJy4BrGFtNQT6/UXbcwV9vau/8TD3QkZQ1uE6aEORoTDdjOwC1wOoNc2OKlNxy0x4hF1f7DlsQ== X-Received: by 2002:a37:9b84:: with SMTP id d126mr286628qke.209.1622146779507; Thu, 27 May 2021 13:19:39 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id u14sm2089536qkp.80.2021.05.27.13.19.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:19:38 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Axel Rasmussen , "Kirill A . Shutemov" , Hugh Dickins , Andrew Morton , Miaohe Lin , Mike Rapoport , Jerome Glisse , Andrea Arcangeli , Nadav Amit , Mike Kravetz , peterx@redhat.com, Jason Gunthorpe , Matthew Wilcox Subject: [PATCH v3 03/27] mm: Clear vmf->pte after pte_unmap_same() returns Date: Thu, 27 May 2021 16:19:03 -0400 Message-Id: <20210527201927.29586-4-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dy4i3t47; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf13.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 54872E005F2C X-Stat-Signature: pcoey5s7xpmnk7zkkcei6o7wkenb59wd X-HE-Tag: 1622146776-638458 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: pte_unmap_same() will always unmap the pte pointer. After the unmap, vmf->pte will not be valid any more. We should clear it. It was safe only because no one is accessing vmf->pte after pte_unmap_same() returns, since the only caller of pte_unmap_same() (so far) is do_swap_page(), where vmf->pte will in most cases be overwritten very soon. pte_unmap_same() will be used in other places in follow up patches, so that vmf->pte will not always be re-written. This patch enables us to call functions like finish_fault() because that'll conditionally unmap the pte by checking vmf->pte first. Or, alloc_set_pte() will make sure to allocate a new pte even after calling pte_unmap_same(). Since we'll need to modify vmf->pte, directly pass in vmf into pte_unmap_same() and then we can also avoid the long parameter list. Reviewed-by: Miaohe Lin Signed-off-by: Peter Xu --- mm/memory.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 2b7ffcbca175..0ccaae2647c0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2710,19 +2710,20 @@ EXPORT_SYMBOL_GPL(apply_to_existing_page_range); * proceeding (but do_wp_page is only called after already making such a check; * and do_anonymous_page can safely check later on). */ -static inline int pte_unmap_same(struct mm_struct *mm, pmd_t *pmd, - pte_t *page_table, pte_t orig_pte) +static inline int pte_unmap_same(struct vm_fault *vmf) { int same = 1; #if defined(CONFIG_SMP) || defined(CONFIG_PREEMPTION) if (sizeof(pte_t) > sizeof(unsigned long)) { - spinlock_t *ptl = pte_lockptr(mm, pmd); + spinlock_t *ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); spin_lock(ptl); - same = pte_same(*page_table, orig_pte); + same = pte_same(*vmf->pte, vmf->orig_pte); spin_unlock(ptl); } #endif - pte_unmap(page_table); + pte_unmap(vmf->pte); + /* After unmap of pte, the pointer is invalid now - clear it. */ + vmf->pte = NULL; return same; } @@ -3441,7 +3442,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vm_fault_t ret = 0; void *shadow = NULL; - if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, vmf->orig_pte)) + if (!pte_unmap_same(vmf)) goto out; entry = pte_to_swp_entry(vmf->orig_pte); From patchwork Thu May 27 20:19:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285259 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D052DC4707F for ; Thu, 27 May 2021 20:19:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 777E66054E for ; Thu, 27 May 2021 20:19:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 777E66054E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7F9156B0072; Thu, 27 May 2021 16:19:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 782F48D0001; Thu, 27 May 2021 16:19:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1FF636B0074; Thu, 27 May 2021 16:19:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0026.hostedemail.com [216.40.44.26]) by kanga.kvack.org (Postfix) with ESMTP id CF6FE6B0072 for ; Thu, 27 May 2021 16:19:49 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 6E068AF81 for ; Thu, 27 May 2021 20:19:49 +0000 (UTC) X-FDA: 78188126898.26.0642A64 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf24.hostedemail.com (Postfix) with ESMTP id 24577A0001F4 for ; Thu, 27 May 2021 20:19:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622146788; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fvvxzxRyoFFa1CRYUKJFbNEb/H/I+lhHBy9dt3/F7KA=; b=FIs1nLRfXi2NmCYjNDxWAvDo0mSZ9yszEfR7Re8oGJrySzUzA9CRAErojabgxqhG05D/Ev IJLJ51zsBRKtePdUNhZpN/5ga79fxriP10mCoeFqjHnSqhadj2Z7YnHmZTK6Hz9WMGxrsI UBZlng04L8pWSo0QNQaOAZ9gudkyagY= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-100-9OoDThRPPOOo4jr1iWrhTw-1; Thu, 27 May 2021 16:19:43 -0400 X-MC-Unique: 9OoDThRPPOOo4jr1iWrhTw-1 Received: by mail-qk1-f200.google.com with SMTP id l6-20020a3770060000b02902fa5329f2b4so1309971qkc.18 for ; Thu, 27 May 2021 13:19:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fvvxzxRyoFFa1CRYUKJFbNEb/H/I+lhHBy9dt3/F7KA=; b=M9Ha1QQ/bQCTyTMVSRuOCagfeucTMD2+fP8+0j20oYd0UQO0IUYKH2jANmeC3RYyZp k48P0gmH1l956wCn3jZkrJlIV+iZwKcEdyUGPm29AZrr3a9SWUfRdDNnTaydgvH6G8eV L+JUwsXtTuCjHTWzMz2sU5cg4s3CH/MnBccR/RWBadVJF77d+bV3R7fJGBRxu5n6hyK9 x5FUG6Ew67Uey7+QxKVdUGwK2qFNwaXrGC7haTjxQ7NUYnlInzUkutWz1XVBEatZJXLh GhNA7ajXZU2nD800CIvf/ptwoROK54604E2Dt4J/6jTvKcAD6FZzShU8lGKPlLGe1kb0 vOHg== X-Gm-Message-State: AOAM533/CL6DMXyYTQMvggCCt/6Ed92ChlXavW5Drlg67yFxBEm7WafW zhukO95xhUD6IdY9U49Ec0EE6rDGFqx0KnrAyW9/T1/izP3i8DqtTXl4mSTHbnxWr1J3m5qgg3a X5+/npRkvnUo= X-Received: by 2002:ad4:4baf:: with SMTP id i15mr312396qvw.61.1622146781820; Thu, 27 May 2021 13:19:41 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw9mMbN6hPTbdFTATmfDi5zfbVFhTvV13COXxAbZrbayo1xXYeoyJUoyvd4oXVYnWHsq6J0Gg== X-Received: by 2002:ad4:4baf:: with SMTP id i15mr312376qvw.61.1622146781565; Thu, 27 May 2021 13:19:41 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id u14sm2089536qkp.80.2021.05.27.13.19.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:19:40 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Axel Rasmussen , "Kirill A . Shutemov" , Hugh Dickins , Andrew Morton , Miaohe Lin , Mike Rapoport , Jerome Glisse , Andrea Arcangeli , Nadav Amit , Mike Kravetz , peterx@redhat.com, Jason Gunthorpe , Matthew Wilcox Subject: [PATCH v3 04/27] mm/userfaultfd: Introduce special pte for unmapped file-backed mem Date: Thu, 27 May 2021 16:19:04 -0400 Message-Id: <20210527201927.29586-5-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FIs1nLRf; spf=none (imf24.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 24577A0001F4 X-Stat-Signature: 7uqxxoh5wsrzub5pmpfnt5uiem7u47q7 X-HE-Tag: 1622146781-479760 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch introduces a very special swap-like pte for file-backed memories. Currently it's only defined for x86_64 only, but as long as any arch that can properly define the UFFD_WP_SWP_PTE_SPECIAL value as requested, it should conceptually work too. We will use this special pte to arm the ptes that got either unmapped or swapped out for a file-backed region that was previously wr-protected. This special pte could trigger a page fault just like swap entries, and as long as the page fault will satisfy pte_none()==false && pte_present()==false. Then we can revive the special pte into a normal pte backed by the page cache. This idea is greatly inspired by Hugh and Andrea in the discussion, which is referenced in the links below. The other idea (from Hugh) is that we use swp_type==1 and swp_offset=0 as the special pte. The current solution (as pointed out by Andrea) is slightly preferred in that we don't even need swp_entry_t knowledge at all in trapping these accesses. Meanwhile, we also reuse _PAGE_SWP_UFFD_WP from the anonymous swp entries. This patch only introduces the special pte and its operators. It's not yet applied to have any functional difference. Link: https://lore.kernel.org/lkml/20201126222359.8120-1-peterx@redhat.com/ Link: https://lore.kernel.org/lkml/20201130230603.46187-1-peterx@redhat.com/ Suggested-by: Andrea Arcangeli Suggested-by: Hugh Dickins Signed-off-by: Peter Xu --- arch/x86/include/asm/pgtable.h | 28 ++++++++++++++++++++++++++++ include/asm-generic/pgtable_uffd.h | 3 +++ include/linux/userfaultfd_k.h | 21 +++++++++++++++++++++ 3 files changed, 52 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index b1099f2d9800..9781ba2da049 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1329,6 +1329,34 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd) #endif #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP + +/* + * This is a very special swap-like pte that marks this pte as "wr-protected" + * by userfaultfd-wp. It should only exist for file-backed memory where the + * page (previously got wr-protected) has been unmapped or swapped out. + * + * For anonymous memories, the userfaultfd-wp _PAGE_SWP_UFFD_WP bit is kept + * along with a real swp entry instead. + * + * Let's make some rules for this special pte: + * + * (1) pte_none()==false, so that it'll not trigger a missing page fault. + * + * (2) pte_present()==false, so that it's recognized as swap (is_swap_pte). + * + * (3) pte_swp_uffd_wp()==true, so it can be tested just like a swap pte that + * contains a valid swap entry, so that we can check a swap pte always + * using "is_swap_pte() && pte_swp_uffd_wp()" without caring about whether + * there's one swap entry inside of the pte. + * + * (4) It should not be a valid swap pte anywhere, so that when we see this pte + * we know it does not contain a swap entry. + * + * For x86, the simplest special pte which satisfies all of above should be the + * pte with only _PAGE_SWP_UFFD_WP bit set (where swp_type==swp_offset==0). + */ +#define UFFD_WP_SWP_PTE_SPECIAL __pte(_PAGE_SWP_UFFD_WP) + static inline pte_t pte_swp_mkuffd_wp(pte_t pte) { return pte_set_flags(pte, _PAGE_SWP_UFFD_WP); diff --git a/include/asm-generic/pgtable_uffd.h b/include/asm-generic/pgtable_uffd.h index 828966d4c281..95e9811ce9d1 100644 --- a/include/asm-generic/pgtable_uffd.h +++ b/include/asm-generic/pgtable_uffd.h @@ -2,6 +2,9 @@ #define _ASM_GENERIC_PGTABLE_UFFD_H #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_WP + +#define UFFD_WP_SWP_PTE_SPECIAL __pte(0) + static __always_inline int pte_uffd_wp(pte_t pte) { return 0; diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h index 331d2ccf0bcc..93f932b53a71 100644 --- a/include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -145,6 +145,17 @@ extern int userfaultfd_unmap_prep(struct vm_area_struct *vma, extern void userfaultfd_unmap_complete(struct mm_struct *mm, struct list_head *uf); +static inline pte_t pte_swp_mkuffd_wp_special(struct vm_area_struct *vma) +{ + WARN_ON_ONCE(vma_is_anonymous(vma)); + return UFFD_WP_SWP_PTE_SPECIAL; +} + +static inline bool pte_swp_uffd_wp_special(pte_t pte) +{ + return pte_same(pte, UFFD_WP_SWP_PTE_SPECIAL); +} + #else /* CONFIG_USERFAULTFD */ /* mm helpers */ @@ -234,6 +245,16 @@ static inline void userfaultfd_unmap_complete(struct mm_struct *mm, { } +static inline pte_t pte_swp_mkuffd_wp_special(struct vm_area_struct *vma) +{ + return __pte(0); +} + +static inline bool pte_swp_uffd_wp_special(pte_t pte) +{ + return false; +} + #endif /* CONFIG_USERFAULTFD */ #endif /* _LINUX_USERFAULTFD_K_H */ From patchwork Thu May 27 20:21:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285261 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A532C47089 for ; Thu, 27 May 2021 20:21:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BA17D6054E for ; Thu, 27 May 2021 20:21:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BA17D6054E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 524916B006C; Thu, 27 May 2021 16:21:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4FBD66B006E; Thu, 27 May 2021 16:21:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C34D6B0070; Thu, 27 May 2021 16:21:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0229.hostedemail.com [216.40.44.229]) by kanga.kvack.org (Postfix) with ESMTP id 00C716B006C for ; Thu, 27 May 2021 16:21:25 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 7AC5C181AF5C7 for ; Thu, 27 May 2021 20:21:25 +0000 (UTC) X-FDA: 78188130930.14.060BAC3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 010FC2BD8 for ; Thu, 27 May 2021 20:21:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622146884; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rsRTgtg4IeZXx632vvcSV9nqjRspSTIPhkPklzMJhFg=; b=DP32hyIQCRzPoW66oXlVZPXgt7tn1hQgQYaRDIJk+9BGEIYl/NrHh+WGYG8nUdiMIn1VZG WRbDsgy3kpFnQo+0SNdcA/GhLplE8I7h22cpGwm5gGXOSw9LERvARZKaG2Pd5goid9uPme q2H6jJ/Yq62Ry7f6kNEO/GpvEZdLjSM= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-346-BR7OMno7NpeqIYNSwycpmg-1; Thu, 27 May 2021 16:21:23 -0400 X-MC-Unique: BR7OMno7NpeqIYNSwycpmg-1 Received: by mail-qk1-f200.google.com with SMTP id o14-20020a05620a130eb02902ea53a6ef80so1345847qkj.6 for ; Thu, 27 May 2021 13:21:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rsRTgtg4IeZXx632vvcSV9nqjRspSTIPhkPklzMJhFg=; b=IIoAuTS53AAEqRqhlcUVVmUj4fxkkpPXWgHBQUNCjsJsOhG4+syKzf7Gz3wmGYYmvi 6gyO/4t93d2eUsTg0PmA975yJRcL4+MjJAGT9kev1bnXtKk4O5wawK/cZ3YkYkbbSbzY 2vgL5Etx+CyxEL8OBJE8JGt/xeb+lf8o3sCpiMAeHrAMcdmcSZoueBuzArVHMJNlf7No JVY3CcFRe/CHCvuXSDiu60vUtGiL5XDxjCErN98vojEFm6LgvP58JxPgbKnADPpD6tbH Qr7N8yHQJLDrek+r2h3ItWRkQNKusn1+Ck4dz4bI9e1N3tnqzDdVrvDxwsgH4tyg8enS 23Dg== X-Gm-Message-State: AOAM533cdnxAl0erdPeNpOlWY7JphYCrJSUTTDsvh2kxVg61y1iO3umC 4dh70kk+UeHQXXIHiHjon+oDKM3dmiEErd06hXs7xRrjwfcY8blI9JVBnthHB27Jsro9U7IWMKK 19Q1aX7vp7pj9RUwVGsre9Xru7/jW2bbGitVhOlagR1fjWZcfY4R1KjCLp1ut X-Received: by 2002:ac8:7e95:: with SMTP id w21mr366850qtj.76.1622146882096; Thu, 27 May 2021 13:21:22 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzsu99SqphxaV5fS6RYnOwpaKyYPzTZnLgRLgGScTmj89eNlKWiyekef21U2Dc89wGhI0XgCQ== X-Received: by 2002:ac8:7e95:: with SMTP id w21mr366795qtj.76.1622146881599; Thu, 27 May 2021 13:21:21 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id g18sm2062201qke.37.2021.05.27.13.21.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:21:21 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrea Arcangeli , "Kirill A . Shutemov" , Axel Rasmussen , Nadav Amit , Hugh Dickins , Jerome Glisse , Jason Gunthorpe , peterx@redhat.com, Andrew Morton , Miaohe Lin , Mike Rapoport , Matthew Wilcox , Mike Kravetz Subject: [PATCH v3 05/27] mm/swap: Introduce the idea of special swap ptes Date: Thu, 27 May 2021 16:21:17 -0400 Message-Id: <20210527202117.30689-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DP32hyIQ; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf12.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 010FC2BD8 X-Stat-Signature: x89b73k3mqk4f8xdmb4zm91gu8cs1yw9 X-HE-Tag: 1622146873-207826 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We used to have special swap entries, like migration entries, hw-poison entries, device private entries, etc. Those "special swap entries" reside in the range that they need to be at least swap entries first, and their types are decided by swp_type(entry). This patch introduces another idea called "special swap ptes". It's very easy to get confused against "special swap entries", but a speical swap pte should never contain a swap entry at all. It means, it's illegal to call pte_to_swp_entry() upon a special swap pte. Make the uffd-wp special pte to be the first special swap pte. Before this patch, is_swap_pte()==true means one of the below: (a.1) The pte has a normal swap entry (non_swap_entry()==false). For example, when an anonymous page got swapped out. (a.2) The pte has a special swap entry (non_swap_entry()==true). For example, a migration entry, a hw-poison entry, etc. After this patch, is_swap_pte()==true means one of the below, where case (b) is added: (a) The pte contains a swap entry. (a.1) The pte has a normal swap entry (non_swap_entry()==false). For example, when an anonymous page got swapped out. (a.2) The pte has a special swap entry (non_swap_entry()==true). For example, a migration entry, a hw-poison entry, etc. (b) The pte does not contain a swap entry at all (so it cannot be passed into pte_to_swp_entry()). For example, uffd-wp special swap pte. Teach the whole mm core about this new idea. It's done by introducing another helper called pte_has_swap_entry() which stands for case (a.1) and (a.2). Before this patch, it will be the same as is_swap_pte() because there's no special swap pte yet. Now for most of the previous use of is_swap_entry() in mm core, we'll need to use the new helper pte_has_swap_entry() instead, to make sure we won't try to parse a swap entry from a swap special pte (which does not contain a swap entry at all!). We either handle the swap special pte, or it'll naturally use the default "else" paths. Warn properly (e.g., in do_swap_page()) when we see a special swap pte - we should never call do_swap_page() upon those ptes, but just to bail out early if it happens. Signed-off-by: Peter Xu --- arch/arm64/kernel/mte.c | 2 +- fs/proc/task_mmu.c | 14 ++++++++------ include/linux/swapops.h | 39 ++++++++++++++++++++++++++++++++++++++- mm/gup.c | 2 +- mm/hmm.c | 2 +- mm/khugepaged.c | 11 ++++++++++- mm/madvise.c | 4 ++-- mm/memcontrol.c | 2 +- mm/memory.c | 7 +++++++ mm/migrate.c | 4 ++-- mm/mincore.c | 2 +- mm/mprotect.c | 2 +- mm/mremap.c | 2 +- mm/page_vma_mapped.c | 6 +++--- mm/swapfile.c | 2 +- 15 files changed, 78 insertions(+), 23 deletions(-) diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index 125a10e413e9..a6fd3fb3eacb 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -36,7 +36,7 @@ static void mte_sync_page_tags(struct page *page, pte_t *ptep, bool check_swap) { pte_t old_pte = READ_ONCE(*ptep); - if (check_swap && is_swap_pte(old_pte)) { + if (check_swap && pte_has_swap_entry(old_pte)) { swp_entry_t entry = pte_to_swp_entry(old_pte); if (!non_swap_entry(entry) && mte_restore_tags(entry, page)) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index eb97468dfe4c..9c5af77b5290 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -498,7 +498,7 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr, if (pte_present(*pte)) { page = vm_normal_page(vma, addr, *pte); - } else if (is_swap_pte(*pte)) { + } else if (pte_has_swap_entry(*pte)) { swp_entry_t swpent = pte_to_swp_entry(*pte); if (!non_swap_entry(swpent)) { @@ -516,8 +516,10 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr, } } else if (is_pfn_swap_entry(swpent)) page = pfn_swap_entry_to_page(swpent); - } else if (unlikely(IS_ENABLED(CONFIG_SHMEM) && mss->check_shmem_swap - && pte_none(*pte))) { + } else if (unlikely(IS_ENABLED(CONFIG_SHMEM) && + mss->check_shmem_swap && + /* Here swap special pte is the same as none pte */ + (pte_none(*pte) || is_swap_special_pte(*pte)))) { page = xa_load(&vma->vm_file->f_mapping->i_pages, linear_page_index(vma, addr)); if (xa_is_value(page)) @@ -689,7 +691,7 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask, if (pte_present(*pte)) { page = vm_normal_page(vma, addr, *pte); - } else if (is_swap_pte(*pte)) { + } else if (pte_has_swap_entry(*pte)) { swp_entry_t swpent = pte_to_swp_entry(*pte); if (is_pfn_swap_entry(swpent)) @@ -1071,7 +1073,7 @@ static inline void clear_soft_dirty(struct vm_area_struct *vma, ptent = pte_wrprotect(old_pte); ptent = pte_clear_soft_dirty(ptent); ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent); - } else if (is_swap_pte(ptent)) { + } else if (pte_has_swap_entry(ptent)) { ptent = pte_swp_clear_soft_dirty(ptent); set_pte_at(vma->vm_mm, addr, pte, ptent); } @@ -1374,7 +1376,7 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm, flags |= PM_SOFT_DIRTY; if (pte_uffd_wp(pte)) flags |= PM_UFFD_WP; - } else if (is_swap_pte(pte)) { + } else if (pte_has_swap_entry(pte)) { swp_entry_t entry; if (pte_swp_soft_dirty(pte)) flags |= PM_SOFT_DIRTY; diff --git a/include/linux/swapops.h b/include/linux/swapops.h index af3d2661e41e..4a316c015fe0 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -5,6 +5,7 @@ #include #include #include +#include #ifdef CONFIG_MMU @@ -52,12 +53,48 @@ static inline pgoff_t swp_offset(swp_entry_t entry) return entry.val & SWP_OFFSET_MASK; } -/* check whether a pte points to a swap entry */ +/* + * is_swap_pte() returns true for three cases: + * + * (a) The pte contains a swap entry. + * + * (a.1) The pte has a normal swap entry (non_swap_entry()==false). For + * example, when an anonymous page got swapped out. + * + * (a.2) The pte has a special swap entry (non_swap_entry()==true). For + * example, a migration entry, a hw-poison entry, etc. + * + * (b) The pte does not contain a swap entry at all (so it cannot be passed + * into pte_to_swp_entry()). For example, uffd-wp special swap pte. + */ static inline int is_swap_pte(pte_t pte) { return !pte_none(pte) && !pte_present(pte); } +/* + * A swap-like special pte should only be used as special marker to trigger a + * page fault. We should treat them similarly as pte_none() in most cases, + * except that it may contain some special information that can persist within + * the pte. Currently the only special swap pte is UFFD_WP_SWP_PTE_SPECIAL. + * + * Note: we should never call pte_to_swp_entry() upon a special swap pte, + * Because a swap special pte does not contain a swap entry! + */ +static inline bool is_swap_special_pte(pte_t pte) +{ + return pte_swp_uffd_wp_special(pte); +} + +/* + * Returns true if the pte contains a swap entry. This includes not only the + * normal swp entry case, but also for migration entries, etc. + */ +static inline bool pte_has_swap_entry(pte_t pte) +{ + return is_swap_pte(pte) && !is_swap_special_pte(pte); +} + /* * Convert the arch-dependent pte representation of a swp_entry_t into an * arch-independent swp_entry_t. diff --git a/mm/gup.c b/mm/gup.c index 29a0c7d87024..e03590c9c68e 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -485,7 +485,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, */ if (likely(!(flags & FOLL_MIGRATION))) goto no_page; - if (pte_none(pte)) + if (!pte_has_swap_entry(pte)) goto no_page; entry = pte_to_swp_entry(pte); if (!is_migration_entry(entry)) diff --git a/mm/hmm.c b/mm/hmm.c index fad6be2bf072..aba1bf2c6742 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -239,7 +239,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, pte_t pte = *ptep; uint64_t pfn_req_flags = *hmm_pfn; - if (pte_none(pte)) { + if (pte_none(pte) || is_swap_special_pte(pte)) { required_fault = hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0); if (required_fault) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index b0412be08fa2..7376a9b5bfc9 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1018,7 +1018,7 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm, vmf.pte = pte_offset_map(pmd, address); vmf.orig_pte = *vmf.pte; - if (!is_swap_pte(vmf.orig_pte)) { + if (!pte_has_swap_entry(vmf.orig_pte)) { pte_unmap(vmf.pte); continue; } @@ -1245,6 +1245,15 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, _pte++, _address += PAGE_SIZE) { pte_t pteval = *_pte; if (is_swap_pte(pteval)) { + if (is_swap_special_pte(pteval)) { + /* + * Reuse SCAN_PTE_UFFD_WP. If there will be + * new users of is_swap_special_pte(), we'd + * better introduce a new result type. + */ + result = SCAN_PTE_UFFD_WP; + goto out_unmap; + } if (++unmapped <= khugepaged_max_ptes_swap) { /* * Always be strict with uffd-wp diff --git a/mm/madvise.c b/mm/madvise.c index 012129fbfaf8..ebde36d685ad 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -204,7 +204,7 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start, pte = *(orig_pte + ((index - start) / PAGE_SIZE)); pte_unmap_unlock(orig_pte, ptl); - if (pte_present(pte) || pte_none(pte)) + if (!pte_has_swap_entry(pte)) continue; entry = pte_to_swp_entry(pte); if (unlikely(non_swap_entry(entry))) @@ -596,7 +596,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, for (; addr != end; pte++, addr += PAGE_SIZE) { ptent = *pte; - if (pte_none(ptent)) + if (pte_none(ptent) || is_swap_special_pte(ptent)) continue; /* * If the pte has swp_entry, just clear page table to diff --git a/mm/memcontrol.c b/mm/memcontrol.c index cb864f87b01d..f684f6cf6fce 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5719,7 +5719,7 @@ static enum mc_target_type get_mctgt_type(struct vm_area_struct *vma, if (pte_present(ptent)) page = mc_handle_present_pte(vma, addr, ptent); - else if (is_swap_pte(ptent)) + else if (pte_has_swap_entry(ptent)) page = mc_handle_swap_pte(vma, ptent, &ent); else if (pte_none(ptent)) page = mc_handle_file_pte(vma, addr, ptent, &ent); diff --git a/mm/memory.c b/mm/memory.c index 0ccaae2647c0..2b24af4616df 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3445,6 +3445,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (!pte_unmap_same(vmf)) goto out; + /* + * We should never call do_swap_page upon a swap special pte; just be + * safe to bail out if it happens. + */ + if (WARN_ON_ONCE(is_swap_special_pte(vmf->orig_pte))) + goto out; + entry = pte_to_swp_entry(vmf->orig_pte); if (unlikely(non_swap_entry(entry))) { if (is_migration_entry(entry)) { diff --git a/mm/migrate.c b/mm/migrate.c index 91ee6f0941b4..2468c5d00f30 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -294,7 +294,7 @@ void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, spin_lock(ptl); pte = *ptep; - if (!is_swap_pte(pte)) + if (!pte_has_swap_entry(pte)) goto out; entry = pte_to_swp_entry(pte); @@ -2248,7 +2248,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, pte = *ptep; - if (pte_none(pte)) { + if (pte_none(pte) || is_swap_special_pte(pte)) { if (vma_is_anonymous(vma)) { mpfn = MIGRATE_PFN_MIGRATE; migrate->cpages++; diff --git a/mm/mincore.c b/mm/mincore.c index 9122676b54d6..5728c3e6473f 100644 --- a/mm/mincore.c +++ b/mm/mincore.c @@ -121,7 +121,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, for (; addr != end; ptep++, addr += PAGE_SIZE) { pte_t pte = *ptep; - if (pte_none(pte)) + if (pte_none(pte) || is_swap_special_pte(pte)) __mincore_unmapped_range(addr, addr + PAGE_SIZE, vma, vec); else if (pte_present(pte)) diff --git a/mm/mprotect.c b/mm/mprotect.c index 883e2cc85cad..4b743394afbe 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -139,7 +139,7 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, } ptep_modify_prot_commit(vma, addr, pte, oldpte, ptent); pages++; - } else if (is_swap_pte(oldpte)) { + } else if (pte_has_swap_entry(oldpte)) { swp_entry_t entry = pte_to_swp_entry(oldpte); pte_t newpte; diff --git a/mm/mremap.c b/mm/mremap.c index b7523589f218..64cd6581e05a 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -125,7 +125,7 @@ static pte_t move_soft_dirty_pte(pte_t pte) #ifdef CONFIG_MEM_SOFT_DIRTY if (pte_present(pte)) pte = pte_mksoft_dirty(pte); - else if (is_swap_pte(pte)) + else if (pte_has_swap_entry(pte)) pte = pte_swp_mksoft_dirty(pte); #endif return pte; diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index f535bcb4950c..c2f9bcee2273 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -36,7 +36,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw) * For more details on device private memory see HMM * (include/linux/hmm.h or mm/hmm.c). */ - if (is_swap_pte(*pvmw->pte)) { + if (pte_has_swap_entry(*pvmw->pte)) { swp_entry_t entry; /* Handle un-addressable ZONE_DEVICE memory */ @@ -90,7 +90,7 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw) if (pvmw->flags & PVMW_MIGRATION) { swp_entry_t entry; - if (!is_swap_pte(*pvmw->pte)) + if (!pte_has_swap_entry(*pvmw->pte)) return false; entry = pte_to_swp_entry(*pvmw->pte); @@ -99,7 +99,7 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw) return false; pfn = swp_offset(entry); - } else if (is_swap_pte(*pvmw->pte)) { + } else if (pte_has_swap_entry(*pvmw->pte)) { swp_entry_t entry; /* Handle un-addressable ZONE_DEVICE memory */ diff --git a/mm/swapfile.c b/mm/swapfile.c index cbb4c0795284..2401b2a90443 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1972,7 +1972,7 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, si = swap_info[type]; pte = pte_offset_map(pmd, addr); do { - if (!is_swap_pte(*pte)) + if (!pte_has_swap_entry(*pte)) continue; entry = pte_to_swp_entry(*pte); From patchwork Thu May 27 20:21:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285263 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46A34C47089 for ; Thu, 27 May 2021 20:21:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DC297613EB for ; Thu, 27 May 2021 20:21:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DC297613EB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 794C36B006E; Thu, 27 May 2021 16:21:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 76BDE6B0070; Thu, 27 May 2021 16:21:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C0026B0071; Thu, 27 May 2021 16:21:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0219.hostedemail.com [216.40.44.219]) by kanga.kvack.org (Postfix) with ESMTP id 2A1176B006E for ; Thu, 27 May 2021 16:21:31 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id B58F1181AF5C7 for ; Thu, 27 May 2021 20:21:30 +0000 (UTC) X-FDA: 78188131140.15.8F59433 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf10.hostedemail.com (Postfix) with ESMTP id A606E4142C67 for ; Thu, 27 May 2021 20:21:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622146890; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QYVv4WAkRU7fRACps66Ocj6/XAOeYepv/px19tfG5cs=; b=cDcpw4OWFT7Xqlu9Wxxj/n4CTvHTVEFBUUFXmZ07gSyDIbmZ3DZqz+aw5Ft1W3L889MSLH g9/IAd7hDIZzzoVPEbm3jGoK9rp6Tecl6/zw1nzl2sFG6wZYi3l3MdAi0/sjwT3pmY6xEl rwono5vqdjFgOkWIgyJHFECeooYutoY= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-414-M6jnJ_CENxmq_F9-63a02A-1; Thu, 27 May 2021 16:21:27 -0400 X-MC-Unique: M6jnJ_CENxmq_F9-63a02A-1 Received: by mail-qv1-f71.google.com with SMTP id f17-20020a0cf3d10000b02901eda24e6b92so1051826qvm.1 for ; Thu, 27 May 2021 13:21:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QYVv4WAkRU7fRACps66Ocj6/XAOeYepv/px19tfG5cs=; b=GjZIXWUOFVoOOkXT8hII0T+il5ks1K0R972VQjPqoShdIBS84c0I4PgmKt1LVzARcA ZnLdCNDkki2FF38cp2uli5lTn1Cgv8dGOlsZy0MKF7bqimpnU0draHZNb+fDiUep9Y7q r367tGrGhv9Sldt3aSTnNofDgagmQgF04PqiEFOdcHzoi+e3RF7vzY8Q5RDzMZT4HJSz tX7S67NaykBwEGpiy1bNjItpQu75prIcfu4iKhR8bIWYp9en5xVPBo6Y21AeLPB19iNM mOk+aj5syFV8YD5O1Wxks71RALcyu3SMftBTtpPa1PQF2DsZfdVDy/ZxuLXwabUcVY7Z +wIA== X-Gm-Message-State: AOAM531/1u7HAZmQFNeWTT0tl6eSXT4YYdDZBdqVSVgHVLYeJxukv+RC AUBSqo/cDCQmeiKknp8fIshCrqq53le1MK1JzVkDhzvJQT+lF+k44NFkDgS8wLN1j0NbCPQAx0z y5p2R9HskaphXYQyvk2Cwj4vy3yMsjEgWSn1aW408m6snjGcyti0AY8d0Aje8 X-Received: by 2002:ac8:594a:: with SMTP id 10mr301484qtz.293.1622146886420; Thu, 27 May 2021 13:21:26 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw7lqcECT8KPsmIK0r+r6dBmL5bSGmCLQ2KpX8k9jXbC/1IEeudtC3hjPjvAXnHx6Ajngv1fA== X-Received: by 2002:ac8:594a:: with SMTP id 10mr301440qtz.293.1622146885908; Thu, 27 May 2021 13:21:25 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id i14sm2222461qkn.99.2021.05.27.13.21.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:21:25 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrea Arcangeli , "Kirill A . Shutemov" , Axel Rasmussen , Nadav Amit , Hugh Dickins , Jerome Glisse , Jason Gunthorpe , peterx@redhat.com, Andrew Morton , Miaohe Lin , Mike Rapoport , Matthew Wilcox , Mike Kravetz Subject: [PATCH v3 06/27] shmem/userfaultfd: Handle uffd-wp special pte in page fault handler Date: Thu, 27 May 2021 16:21:22 -0400 Message-Id: <20210527202122.30739-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cDcpw4OW; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf10.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: A606E4142C67 X-Stat-Signature: m5o8zujkady9f19smdbfk53eqjfhwheu X-HE-Tag: 1622146882-334586 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: File-backed memories are prone to unmap/swap so the ptes are always unstable. This could lead to userfaultfd-wp information got lost when unmapped or swapped out on such types of memory, for example, shmem. To keep such an information persistent, we will start to use the newly introduced swap-like special ptes to replace a null pte when those ptes were removed. Prepare this by handling such a special pte first before it is applied in the general page fault handler. The handling of this special pte page fault is similar to missing fault, but it should happen after the pte missing logic since the special pte is designed to be a swap-like pte. Meanwhile it should be handled before do_swap_page() so that the swap core logic won't be confused to see such an illegal swap pte. This is a slow path of uffd-wp handling, because unmap of wr-protected shmem ptes should be rare. So far it should only trigger in two conditions: (1) When trying to punch holes in shmem_fallocate(), there will be a pre-unmap optimization before evicting the page. That will create unmapped shmem ptes with wr-protected pages covered. (2) Swapping out of shmem pages Because of this, the page fault handling is simplifed too by not sending the wr-protect message in the 1st page fault, instead the page will be installed read-only, so the message will be generated until the next write, which will trigger the do_wp_page() path of general uffd-wp handling. Disable fault-around for all uffd-wp registered ranges for extra safety, and clean the code up a bit after we introduced MINOR fault. Signed-off-by: Peter Xu --- include/linux/userfaultfd_k.h | 12 +++++ mm/memory.c | 88 +++++++++++++++++++++++++++++++---- 2 files changed, 90 insertions(+), 10 deletions(-) diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h index 93f932b53a71..ca3f794d07e9 100644 --- a/include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -94,6 +94,18 @@ static inline bool uffd_disable_huge_pmd_share(struct vm_area_struct *vma) return vma->vm_flags & (VM_UFFD_WP | VM_UFFD_MINOR); } +/* + * Don't do fault around for either WP or MINOR registered uffd range. For + * MINOR registered range, fault around will be a total disaster and ptes can + * be installed without notifications; for WP it should mostly be fine as long + * as the fault around checks for pte_none() before the installation, however + * to be super safe we just forbid it. + */ +static inline bool uffd_disable_fault_around(struct vm_area_struct *vma) +{ + return vma->vm_flags & (VM_UFFD_WP | VM_UFFD_MINOR); +} + static inline bool userfaultfd_missing(struct vm_area_struct *vma) { return vma->vm_flags & VM_UFFD_MISSING; diff --git a/mm/memory.c b/mm/memory.c index 2b24af4616df..45a2f71e447a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3917,6 +3917,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) void do_set_pte(struct vm_fault *vmf, struct page *page, unsigned long addr) { struct vm_area_struct *vma = vmf->vma; + bool uffd_wp = pte_swp_uffd_wp_special(vmf->orig_pte); bool write = vmf->flags & FAULT_FLAG_WRITE; bool prefault = vmf->address != addr; pte_t entry; @@ -3929,6 +3930,8 @@ void do_set_pte(struct vm_fault *vmf, struct page *page, unsigned long addr) if (write) entry = maybe_mkwrite(pte_mkdirty(entry), vma); + if (unlikely(uffd_wp)) + entry = pte_mkuffd_wp(pte_wrprotect(entry)); /* copy-on-write page */ if (write && !(vma->vm_flags & VM_SHARED)) { inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); @@ -3996,8 +3999,12 @@ vm_fault_t finish_fault(struct vm_fault *vmf) vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); ret = 0; - /* Re-check under ptl */ - if (likely(pte_none(*vmf->pte))) + + /* + * Re-check under ptl. Note: this will cover both none pte and + * uffd-wp-special swap pte + */ + if (likely(pte_same(*vmf->pte, vmf->orig_pte))) do_set_pte(vmf, page, vmf->address); else ret = VM_FAULT_NOPAGE; @@ -4101,9 +4108,21 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf) return vmf->vma->vm_ops->map_pages(vmf, start_pgoff, end_pgoff); } +/* Return true if we should do read fault-around, false otherwise */ +static inline bool should_fault_around(struct vm_fault *vmf) +{ + /* No ->map_pages? No way to fault around... */ + if (!vmf->vma->vm_ops->map_pages) + return false; + + if (uffd_disable_fault_around(vmf->vma)) + return false; + + return fault_around_bytes >> PAGE_SHIFT > 1; +} + static vm_fault_t do_read_fault(struct vm_fault *vmf) { - struct vm_area_struct *vma = vmf->vma; vm_fault_t ret = 0; /* @@ -4111,12 +4130,10 @@ static vm_fault_t do_read_fault(struct vm_fault *vmf) * if page by the offset is not ready to be mapped (cold cache or * something). */ - if (vma->vm_ops->map_pages && fault_around_bytes >> PAGE_SHIFT > 1) { - if (likely(!userfaultfd_minor(vmf->vma))) { - ret = do_fault_around(vmf); - if (ret) - return ret; - } + if (should_fault_around(vmf)) { + ret = do_fault_around(vmf); + if (ret) + return ret; } ret = __do_fault(vmf); @@ -4435,6 +4452,57 @@ static vm_fault_t wp_huge_pud(struct vm_fault *vmf, pud_t orig_pud) return VM_FAULT_FALLBACK; } +static vm_fault_t uffd_wp_clear_special(struct vm_fault *vmf) +{ + vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); + /* + * Be careful so that we will only recover a special uffd-wp pte into a + * none pte. Otherwise it means the pte could have changed, so retry. + */ + if (pte_swp_uffd_wp_special(*vmf->pte)) + pte_clear(vmf->vma->vm_mm, vmf->address, vmf->pte); + pte_unmap_unlock(vmf->pte, vmf->ptl); + return 0; +} + +/* + * This is actually a page-missing access, but with uffd-wp special pte + * installed. It means this pte was wr-protected before being unmapped. + */ +static vm_fault_t uffd_wp_handle_special(struct vm_fault *vmf) +{ + /* Careful! vmf->pte unmapped after return */ + if (!pte_unmap_same(vmf)) + return 0; + + /* + * Just in case there're leftover special ptes even after the region + * got unregistered - we can simply clear them. + */ + if (unlikely(!userfaultfd_wp(vmf->vma) || vma_is_anonymous(vmf->vma))) + return uffd_wp_clear_special(vmf); + + /* + * Here we share most code with do_fault(), in which we can identify + * whether this is "none pte fault" or "uffd-wp-special fault" by + * checking the vmf->orig_pte. + */ + return do_fault(vmf); +} + +static vm_fault_t do_swap_pte(struct vm_fault *vmf) +{ + /* + * We need to handle special swap ptes before handling ptes that + * contain swap entries, always. + */ + if (unlikely(pte_swp_uffd_wp_special(vmf->orig_pte))) + return uffd_wp_handle_special(vmf); + + return do_swap_page(vmf); +} + /* * These routines also need to handle stuff like marking pages dirty * and/or accessed for architectures that don't do it in hardware (most @@ -4509,7 +4577,7 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) } if (!pte_present(vmf->orig_pte)) - return do_swap_page(vmf); + return do_swap_pte(vmf); if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma)) return do_numa_page(vmf); From patchwork Thu May 27 20:21:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285265 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66351C4707F for ; Thu, 27 May 2021 20:21:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1468261360 for ; Thu, 27 May 2021 20:21:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1468261360 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9CB566B0070; Thu, 27 May 2021 16:21:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 975906B0071; Thu, 27 May 2021 16:21:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 445078D0001; Thu, 27 May 2021 16:21:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0224.hostedemail.com [216.40.44.224]) by kanga.kvack.org (Postfix) with ESMTP id F22356B0070 for ; Thu, 27 May 2021 16:21:35 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 96547BA12 for ; Thu, 27 May 2021 20:21:35 +0000 (UTC) X-FDA: 78188131350.20.B3B0D96 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf25.hostedemail.com (Postfix) with ESMTP id C1C67600027A for ; Thu, 27 May 2021 20:21:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622146894; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h5FCImkTLc8C+iW7llWT4B7oKe4vYYc/EBx9K8XBnj4=; b=CLY7ckGXT/WOpqSbRMeiALAPLaLEO3i+PDqGUTeBbr+Qz+LgJweRqFbYSIOWuDWk4wJLOu Tu46ueenjqwelZyWoc88mcYbCQB+AvaDQECuXLFKEmFeeU9IsuPCI5wtCOtnaZNDzo1gUn H4YNw5xSwms54f6VVF3xmXlRyELK9lg= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-260-orlctl-vMISrL73hH6XfkQ-1; Thu, 27 May 2021 16:21:31 -0400 X-MC-Unique: orlctl-vMISrL73hH6XfkQ-1 Received: by mail-qk1-f197.google.com with SMTP id z12-20020a05620a08ccb02902ea1e4a963dso1323658qkz.13 for ; Thu, 27 May 2021 13:21:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=h5FCImkTLc8C+iW7llWT4B7oKe4vYYc/EBx9K8XBnj4=; b=T6oBu1SjFJ8/y/XYkcZmzws13O99mBnJrmMVq3g5KLAdTDi/o5fQTOS90r4dPUGqTL EI73V4dL+p/JvhyZfN+IXVeZ+pqWgQtz1FBY/V2eVAVMS6HeP5YT1LEpPuKTnWDrA+4X pCRHyfmwr28eCi1xcIXnniu02lWvmlHEGNOakrNJlO2FfPnoiMStcyEOxMz9xpoQQave olJHK+ZGyX7uktpsFZMeycevVv/TWCIiR95JOL/M/tk6LWS4v1juT4kY/VeZeWQPImFC NhiFLiZAQEf9xdz7aEdJV/UAVu42JzIjDZvkf4g8VqlZAxl4OK0OQPlshMCZ5/AiO4cs Dm9g== X-Gm-Message-State: AOAM5327waTGiuYSWYdw+D9DJkio39AqIs/Ck+ThtTSEMdlabqK+y0VL bB5VI+CKvdKotE8BQpquDfWhkAn3MeHOra0fyNaCIqF6DF+eMuc1S1t82q1riaI0JUGDbvoo8f+ rCn/OAu3wnOev9MsDOQoZyTm0VJ6z4RkWKFvDIsE+85N5zHqOQKp3212t+YXF X-Received: by 2002:ac8:75d3:: with SMTP id z19mr362923qtq.44.1622146890439; Thu, 27 May 2021 13:21:30 -0700 (PDT) X-Google-Smtp-Source: ABdhPJypIsqd4DUNG1ZqS0UQQmBuBbvTh9zi1M5X23Oonzevxam33oO8kSwmkp38m1qPCCeZl1p/ig== X-Received: by 2002:ac8:75d3:: with SMTP id z19mr362893qtq.44.1622146890139; Thu, 27 May 2021 13:21:30 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id 42sm2070151qtf.37.2021.05.27.13.21.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:21:29 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrea Arcangeli , "Kirill A . Shutemov" , Axel Rasmussen , Nadav Amit , Hugh Dickins , Jerome Glisse , Jason Gunthorpe , peterx@redhat.com, Andrew Morton , Miaohe Lin , Mike Rapoport , Matthew Wilcox , Mike Kravetz Subject: [PATCH v3 07/27] mm: Drop first_index/last_index in zap_details Date: Thu, 27 May 2021 16:21:26 -0400 Message-Id: <20210527202126.30790-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CLY7ckGX; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf25.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com X-Stat-Signature: nhgz5xkka7fa65nbj1m5nzbo1gage145 X-Rspamd-Queue-Id: C1C67600027A X-Rspamd-Server: rspam02 X-HE-Tag: 1622146886-185022 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The first_index/last_index parameters in zap_details are actually only used in unmap_mapping_range_tree(). At the meantime, this function is only called by unmap_mapping_pages() once. Instead of passing these two variables through the whole stack of page zapping code, remove them from zap_details and let them simply be parameters of unmap_mapping_range_tree(), which is inlined. Signed-off-by: Peter Xu Reviewed-by: Alistair Popple --- include/linux/mm.h | 2 -- mm/memory.c | 20 ++++++++++---------- 2 files changed, 10 insertions(+), 12 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index ef9ea6dfefff..db155be8e66c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1726,8 +1726,6 @@ extern void user_shm_unlock(size_t, struct user_struct *); */ struct zap_details { struct address_space *check_mapping; /* Check page->mapping if set */ - pgoff_t first_index; /* Lowest page->index to unmap */ - pgoff_t last_index; /* Highest page->index to unmap */ }; struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr, diff --git a/mm/memory.c b/mm/memory.c index 45a2f71e447a..27cf8a6375c6 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3305,20 +3305,20 @@ static void unmap_mapping_range_vma(struct vm_area_struct *vma, } static inline void unmap_mapping_range_tree(struct rb_root_cached *root, + pgoff_t first_index, + pgoff_t last_index, struct zap_details *details) { struct vm_area_struct *vma; pgoff_t vba, vea, zba, zea; - vma_interval_tree_foreach(vma, root, - details->first_index, details->last_index) { - + vma_interval_tree_foreach(vma, root, first_index, last_index) { vba = vma->vm_pgoff; vea = vba + vma_pages(vma) - 1; - zba = details->first_index; + zba = first_index; if (zba < vba) zba = vba; - zea = details->last_index; + zea = last_index; if (zea > vea) zea = vea; @@ -3344,17 +3344,17 @@ static inline void unmap_mapping_range_tree(struct rb_root_cached *root, void unmap_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t nr, bool even_cows) { + pgoff_t first_index = start, last_index = start + nr - 1; struct zap_details details = { }; details.check_mapping = even_cows ? NULL : mapping; - details.first_index = start; - details.last_index = start + nr - 1; - if (details.last_index < details.first_index) - details.last_index = ULONG_MAX; + if (last_index < first_index) + last_index = ULONG_MAX; i_mmap_lock_write(mapping); if (unlikely(!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root))) - unmap_mapping_range_tree(&mapping->i_mmap, &details); + unmap_mapping_range_tree(&mapping->i_mmap, first_index, + last_index, &details); i_mmap_unlock_write(mapping); } From patchwork Thu May 27 20:21:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285267 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7985FC4707F for ; Thu, 27 May 2021 20:21:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0A81A61360 for ; Thu, 27 May 2021 20:21:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0A81A61360 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A06E36B0071; Thu, 27 May 2021 16:21:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 991DD8D0001; Thu, 27 May 2021 16:21:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 809006B0075; Thu, 27 May 2021 16:21:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0170.hostedemail.com [216.40.44.170]) by kanga.kvack.org (Postfix) with ESMTP id 44CC76B0071 for ; Thu, 27 May 2021 16:21:39 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 31DED6135 for ; Thu, 27 May 2021 20:21:38 +0000 (UTC) X-FDA: 78188131476.06.A13228D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 839BC2BD3 for ; Thu, 27 May 2021 20:21:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622146897; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=y2JdT0mH38cplAf1wx/5ANh95qkzxlwLzfafgEJNeGw=; b=DMebqFGEC17S6jPu+2ycn9EfJFyacnZJMniVVdr46yvel+v4NrsxTVzmCAQlNz7G5hVaDe 4jixUR2353/HGRMgEGR+3xuo8aGl8Px1Bb3o7/hAQSIpjLWNJYcZ88TsarZkj0WtoyCq6+ AD6KGIwPr5I0NwQqhayzZk2ApsRN010= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-134-OLtizAWBPh6szoST-TM1LA-1; Thu, 27 May 2021 16:21:36 -0400 X-MC-Unique: OLtizAWBPh6szoST-TM1LA-1 Received: by mail-qt1-f199.google.com with SMTP id b17-20020ac854110000b02901f279c73d75so886680qtq.2 for ; Thu, 27 May 2021 13:21:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=y2JdT0mH38cplAf1wx/5ANh95qkzxlwLzfafgEJNeGw=; b=ZdO7IQk5uYMlXAxG2i9TUcYwl7N7Ts5qAThDxSQb6l9vWHfAFvcoiLIEKPlAHrw69G QAV+xyG1uzXu2GgPcVoPb/U+K1LaEK65GEzgtk93WIIJLqSLEKiEHFaEYuLMp+D2pkUY H+FFyZ0H3djGLwOxTCz64RS5dJ+lHJ6q91rU0IOhQSGZ8kvifqTT1xBrRdzx26oT61KC Ut/nTgFvOhdcpSNHaFtRmNib94yYOz5lWf3Wtp65A7deGn2PwMf0cx3hmh00VxeZbzqI VR2ozujUtkVkF9Nf31rGY2I+678CVqug386On71hDUBs4fOkwqaD5eFWqJd5vPl90++8 2EIA== X-Gm-Message-State: AOAM530M9uHnkW3952+VbYP8av6yuDG0FwPZURLmpmAU4WnaN3bamKNx gM03OG5x+yryS56Yn9ZxjW+R3BSu89Qt3aa+SJBPYaX9h3mitqQ23D4YOykjnvu9wT/7heNJUZr ZfnkhkZ1BpNaRKzwirB0qGfXgmJWTUOUuUkw/IxHq8lsZUGaOQOdRu6anVVMk X-Received: by 2002:ac8:5e0e:: with SMTP id h14mr331685qtx.117.1622146895047; Thu, 27 May 2021 13:21:35 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxoIpkNIQzO6Y2dcRvaoAmnmF0fwzYQWpYm5GB3jVmlkja21NwVMuJciLokF/MtY1ToQcNROA== X-Received: by 2002:ac8:5e0e:: with SMTP id h14mr331647qtx.117.1622146894684; Thu, 27 May 2021 13:21:34 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id x9sm2001277qto.33.2021.05.27.13.21.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:21:34 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrea Arcangeli , "Kirill A . Shutemov" , Axel Rasmussen , Nadav Amit , Hugh Dickins , Jerome Glisse , Jason Gunthorpe , peterx@redhat.com, Andrew Morton , Miaohe Lin , Mike Rapoport , Matthew Wilcox , Mike Kravetz Subject: [PATCH v3 08/27] mm: Introduce zap_details.zap_flags Date: Thu, 27 May 2021 16:21:30 -0400 Message-Id: <20210527202130.30840-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DMebqFGE; spf=none (imf04.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 839BC2BD3 X-Stat-Signature: 9ozpxf9d5unfypnar6gt346hrq15jteg X-HE-Tag: 1622146893-804636 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Instead of trying to introduce one variable for every new zap_details fields, let's introduce a flag so that it can start to encode true/false informations. Let's start to use this flag first to clean up the only check_mapping variable. Firstly, the name "check_mapping" implies this is a "boolean", but actually it stores the mapping inside, just in a way that it won't be set if we don't want to check the mapping. To make things clearer, introduce the 1st zap flag ZAP_FLAG_CHECK_MAPPING, so that we only check against the mapping if this bit set. At the same time, we can rename check_mapping into zap_mapping and set it always. Since at it, introduce another helper zap_check_mapping_skip() and use it in zap_pte_range() properly. Some old comments have been removed in zap_pte_range() because they're duplicated, and since now we're with ZAP_FLAG_CHECK_MAPPING flag, it'll be very easy to grep this information by simply grepping the flag. It'll also make life easier when we want to e.g. pass in zap_flags into the callers like unmap_mapping_pages() (instead of adding new booleans besides the even_cows parameter). Signed-off-by: Peter Xu --- include/linux/mm.h | 19 ++++++++++++++++++- mm/memory.c | 31 ++++++++----------------------- 2 files changed, 26 insertions(+), 24 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index db155be8e66c..52d3ef2ed753 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1721,13 +1721,30 @@ static inline bool can_do_mlock(void) { return false; } extern int user_shm_lock(size_t, struct user_struct *); extern void user_shm_unlock(size_t, struct user_struct *); +/* Whether to check page->mapping when zapping */ +#define ZAP_FLAG_CHECK_MAPPING BIT(0) + /* * Parameter block passed down to zap_pte_range in exceptional cases. */ struct zap_details { - struct address_space *check_mapping; /* Check page->mapping if set */ + struct address_space *zap_mapping; + unsigned long zap_flags; }; +/* Return true if skip zapping this page, false otherwise */ +static inline bool +zap_check_mapping_skip(struct zap_details *details, struct page *page) +{ + if (!details || !page) + return false; + + if (!(details->zap_flags & ZAP_FLAG_CHECK_MAPPING)) + return false; + + return details->zap_mapping != page_rmapping(page); +} + struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr, pte_t pte); struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr, diff --git a/mm/memory.c b/mm/memory.c index 27cf8a6375c6..c9dc4e9e05b5 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1330,16 +1330,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, struct page *page; page = vm_normal_page(vma, addr, ptent); - if (unlikely(details) && page) { - /* - * unmap_shared_mapping_pages() wants to - * invalidate cache without truncating: - * unmap shared but keep private pages. - */ - if (details->check_mapping && - details->check_mapping != page_rmapping(page)) - continue; - } + if (unlikely(zap_check_mapping_skip(details, page))) + continue; ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); tlb_remove_tlb_entry(tlb, pte, addr); @@ -1372,17 +1364,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, is_device_exclusive_entry(entry)) { struct page *page = pfn_swap_entry_to_page(entry); - if (unlikely(details && details->check_mapping)) { - /* - * unmap_shared_mapping_pages() wants to - * invalidate cache without truncating: - * unmap shared but keep private pages. - */ - if (details->check_mapping != - page_rmapping(page)) - continue; - } - + if (unlikely(zap_check_mapping_skip(details, page))) + continue; pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); rss[mm_counter(page)]--; @@ -3345,9 +3328,11 @@ void unmap_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t nr, bool even_cows) { pgoff_t first_index = start, last_index = start + nr - 1; - struct zap_details details = { }; + struct zap_details details = { .zap_mapping = mapping }; + + if (!even_cows) + details.zap_flags |= ZAP_FLAG_CHECK_MAPPING; - details.check_mapping = even_cows ? NULL : mapping; if (last_index < first_index) last_index = ULONG_MAX; From patchwork Thu May 27 20:21:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285269 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9CC3C4707F for ; Thu, 27 May 2021 20:21:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7FC5661360 for ; Thu, 27 May 2021 20:21:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7FC5661360 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1A2F06B0074; Thu, 27 May 2021 16:21:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1790D6B0075; Thu, 27 May 2021 16:21:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 019B96B0078; Thu, 27 May 2021 16:21:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0196.hostedemail.com [216.40.44.196]) by kanga.kvack.org (Postfix) with ESMTP id BF2006B0074 for ; Thu, 27 May 2021 16:21:42 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 5E22D181AF5C7 for ; Thu, 27 May 2021 20:21:42 +0000 (UTC) X-FDA: 78188131644.03.B7AFBBE Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id 0EC03E005F0D for ; Thu, 27 May 2021 20:21:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622146901; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tgZ4fuz9b2G4xHW7EB/zzIIXBN4xHF3mPH57XGKakgk=; b=UK1SKT3ejs9E3Bzk00tjoj1EH0StJ1hD+e/klfrInLSYskdgrpGB08C/HIBYPYAi7yrqOL 14z3cPil8YS56XJakTJQ8VWjS8/7svhGcmktopOnYRUq/0krxO7DnwBzZHJlsrQ8vBl5ga HKhJVrhCNdouhDtV1LFeVhifI9F2fyU= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-146-Yh_kq4W4N-SOTLFT9DDBZg-1; Thu, 27 May 2021 16:21:40 -0400 X-MC-Unique: Yh_kq4W4N-SOTLFT9DDBZg-1 Received: by mail-qk1-f200.google.com with SMTP id e8-20020a05620a2088b02903a5edeec4d6so1323064qka.11 for ; Thu, 27 May 2021 13:21:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tgZ4fuz9b2G4xHW7EB/zzIIXBN4xHF3mPH57XGKakgk=; b=pCdbexw6kOUfLVHKdjBAlhlo5icpdTWieMiN/p960vn0l8jDiYy0MKOYTLrCXL0Tuk FiBGRAguP6kVV0fwTH8vCoK7StjHFfbFIWOva1OsQEPPI2m5D/aMF7oHf3RyX81Heuka 8KyQ0i8e/4uyfnNkkE4PjMMEwNNfLe8ye8aeuOY03rbtA2uOtUjk1RL2bjSZeLhQvfl3 RnjYXuszYaOBfFkQkdwUoz9dbYwZAyNvRl4bnFQG3SDxEjiSZ7EzW2DjjxlOYBa1QRaB BtnPrnIsyM1HQm9EwycEW4RwGvtbZm9MuUZIRdxD2xrRmHSjBw0Kcb/wlBnqB4oxWRJe KVPg== X-Gm-Message-State: AOAM533cNNbNb+5Sl7vGrGOSrMoeXmSvDuz1Namf2/WbDar/hxhxt40u wYxd6cRhmGFQrR62BMFEhyEnJUk5gJNOJgP2j3h649BGaWbW4tLqc/eQmOD63lDl+qaFURzepRu +EwqilTP8w9SPYcPpBOrCOdtmdsZSGKtnXIpxmhBfLs7aC1d0fDceIaDenOEk X-Received: by 2002:a37:a45:: with SMTP id 66mr355758qkk.138.1622146899140; Thu, 27 May 2021 13:21:39 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyWc8f/UGUmyATH0FVxTT8Qc+Z0pWOiLKfvX0NL7phglOJRhpW7nCQYZyntRo/3ZjxrKn8Cbg== X-Received: by 2002:a37:a45:: with SMTP id 66mr355721qkk.138.1622146898781; Thu, 27 May 2021 13:21:38 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id a68sm2038889qkd.51.2021.05.27.13.21.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:21:38 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrea Arcangeli , "Kirill A . Shutemov" , Axel Rasmussen , Nadav Amit , Hugh Dickins , Jerome Glisse , Jason Gunthorpe , peterx@redhat.com, Andrew Morton , Miaohe Lin , Mike Rapoport , Matthew Wilcox , Mike Kravetz Subject: [PATCH v3 09/27] mm: Introduce ZAP_FLAG_SKIP_SWAP Date: Thu, 27 May 2021 16:21:35 -0400 Message-Id: <20210527202135.30890-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 0EC03E005F0D Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UK1SKT3e; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf30.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com X-Rspamd-Server: rspam03 X-Stat-Signature: woy16b3qeytp1gferbqpt1omma91hqkn X-HE-Tag: 1622146895-951187 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Firstly, the comment in zap_pte_range() is misleading because it checks against details rather than check_mappings, so it's against what the code did. Meanwhile, it's confusing too on not explaining why passing in the details pointer would mean to skip all swap entries. New user of zap_details could very possibly miss this fact if they don't read deep until zap_pte_range() because there's no comment at zap_details talking about it at all, so swap entries could be errornously skipped without being noticed. This partly reverts 3e8715fdc03e ("mm: drop zap_details::check_swap_entries"), but introduce ZAP_FLAG_SKIP_SWAP flag, which means the opposite of previous "details" parameter: the caller should explicitly set this to skip swap entries, otherwise swap entries will always be considered (which is still the major case here). Cc: Kirill A. Shutemov Signed-off-by: Peter Xu Reviewed-by: Alistair Popple --- include/linux/mm.h | 12 ++++++++++++ mm/memory.c | 8 +++++--- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 52d3ef2ed753..1adf313a01fe 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1723,6 +1723,8 @@ extern void user_shm_unlock(size_t, struct user_struct *); /* Whether to check page->mapping when zapping */ #define ZAP_FLAG_CHECK_MAPPING BIT(0) +/* Whether to skip zapping swap entries */ +#define ZAP_FLAG_SKIP_SWAP BIT(1) /* * Parameter block passed down to zap_pte_range in exceptional cases. @@ -1745,6 +1747,16 @@ zap_check_mapping_skip(struct zap_details *details, struct page *page) return details->zap_mapping != page_rmapping(page); } +/* Return true if skip swap entries, false otherwise */ +static inline bool +zap_skip_swap(struct zap_details *details) +{ + if (!details) + return false; + + return details->zap_flags & ZAP_FLAG_SKIP_SWAP; +} + struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr, pte_t pte); struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr, diff --git a/mm/memory.c b/mm/memory.c index c9dc4e9e05b5..8a3751be87ba 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1376,8 +1376,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; } - /* If details->check_mapping, we leave swap entries. */ - if (unlikely(details)) + if (unlikely(zap_skip_swap(details))) continue; if (!non_swap_entry(entry)) @@ -3328,7 +3327,10 @@ void unmap_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t nr, bool even_cows) { pgoff_t first_index = start, last_index = start + nr - 1; - struct zap_details details = { .zap_mapping = mapping }; + struct zap_details details = { + .zap_mapping = mapping, + .zap_flags = ZAP_FLAG_SKIP_SWAP, + }; if (!even_cows) details.zap_flags |= ZAP_FLAG_CHECK_MAPPING; From patchwork Thu May 27 20:21:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285271 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FF07C4708B for ; Thu, 27 May 2021 20:21:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1748C60C3F for ; Thu, 27 May 2021 20:21:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1748C60C3F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AC9336B0075; Thu, 27 May 2021 16:21:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A78966B0078; Thu, 27 May 2021 16:21:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8A47C6B007B; Thu, 27 May 2021 16:21:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 52A736B0075 for ; Thu, 27 May 2021 16:21:50 -0400 (EDT) Received: from smtpin36.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id E40C0C5A4 for ; Thu, 27 May 2021 20:21:49 +0000 (UTC) X-FDA: 78188131938.36.4E67E01 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 8651DA0003A3 for ; Thu, 27 May 2021 20:21:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622146909; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sXc/mxSeZGULf6gugl776EzWZkGQ1iCBUhKtJkC//uw=; b=L0aeuRejlhEPEkB0/J0pAk+fXSP3r523Ei9mmaj07l3Y8/e1x8I08Io10nLQNB9E0hVJCo zNvQhXFKQI1GH+N1BPuLLvc8NHSpJHBnDvR+QQX5PSKYtY+wDv0j3BXmlbHKG4KJBORFvC MkBBbnmzRTjNaWd3BZnDmIoJN3T7Bt4= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-425-o5BC9kdoNQSluffpVc5s7w-1; Thu, 27 May 2021 16:21:47 -0400 X-MC-Unique: o5BC9kdoNQSluffpVc5s7w-1 Received: by mail-qv1-f69.google.com with SMTP id r11-20020a0cb28b0000b02901c87a178503so979415qve.22 for ; Thu, 27 May 2021 13:21:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sXc/mxSeZGULf6gugl776EzWZkGQ1iCBUhKtJkC//uw=; b=o1JEmvkdVcdWGk1bwHJgIxEZw3W5CEg2VeHbXW8dEapVK+vcj7MO9XQrR8Y+6XK4tp /nsmKkEA94lCYdIAFksy/dM+7oiLr28t1B4jifD0pXezcd3W8ZRucUagdNDIq/s+RoVi wDR5/kUhZvAGlgxY/k+cnqXwwidClQrvpjjbJ/Vk6UAy1UalajBAZDsygvMsHLxBGs3O CTEE152blk9QLv3lOwVLIY9jon15G9R81qX71av5+ax3wBjm449aopEwTDotF7UcMQMD zft2KgMOQTWnuMg1Uj6NYLaP0K7Dq+oZWgl3bOOcxpSeiwzbAEcCBVbHF8x0g/zPPrxD NCVA== X-Gm-Message-State: AOAM530A5HyeuTzS31qp4D4uQ94YFVx59j7yeaIqMffXrW24vM1V5Cd/ oduKT/v9XptSBsPiMbKVpTVdxlo9eKs1jQfngPmfg4rxNHTHTlpOBUpIF9AYFDT98T4b+lqjmXD 2+6SwrZU9Wy90wDGdKIO53RH0pbs6FwpwV2mCLmsMbcWo93cUFdDJKxkixn5F X-Received: by 2002:a05:620a:1036:: with SMTP id a22mr358553qkk.186.1622146903346; Thu, 27 May 2021 13:21:43 -0700 (PDT) X-Google-Smtp-Source: ABdhPJy5Oz7COp0aUtdezIVVE5UQeBKcbJN8nq1U8uOtI97y0IFIL9UJHzPhkuIkBKyAiw7/snh96w== X-Received: by 2002:a05:620a:1036:: with SMTP id a22mr358500qkk.186.1622146902914; Thu, 27 May 2021 13:21:42 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id e19sm2042910qtr.45.2021.05.27.13.21.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:21:42 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrea Arcangeli , "Kirill A . Shutemov" , Axel Rasmussen , Nadav Amit , Hugh Dickins , Jerome Glisse , Jason Gunthorpe , peterx@redhat.com, Andrew Morton , Miaohe Lin , Mike Rapoport , Matthew Wilcox , Mike Kravetz Subject: [PATCH v3 10/27] mm: Pass zap_flags into unmap_mapping_pages() Date: Thu, 27 May 2021 16:21:39 -0400 Message-Id: <20210527202139.30942-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 8651DA0003A3 Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=L0aeuRej; spf=none (imf23.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam04 X-Stat-Signature: 93ugfnzq8due6471pguouo1zkxtm1iwj X-HE-Tag: 1622146902-287233 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Give unmap_mapping_pages() more power by allowing to specify a zap flag so that it can pass in more information than "whether we'd also like to zap cow pages". With the new flag, we can remove the even_cow parameter because even_cow==false equals to zap_flags==ZAP_FLAG_CHECK_MAPPING, while even_cow==true means a none zap flag to pass in (though in most cases we have had even_cow==false). No functional change intended. Signed-off-by: Peter Xu --- fs/dax.c | 10 ++++++---- include/linux/mm.h | 4 ++-- mm/khugepaged.c | 3 ++- mm/memory.c | 15 ++++++++------- mm/truncate.c | 11 +++++++---- 5 files changed, 25 insertions(+), 18 deletions(-) diff --git a/fs/dax.c b/fs/dax.c index 62352cbcf0f4..09d482c1595b 100644 --- a/fs/dax.c +++ b/fs/dax.c @@ -528,7 +528,7 @@ static void *grab_mapping_entry(struct xa_state *xas, xas_unlock_irq(xas); unmap_mapping_pages(mapping, xas->xa_index & ~PG_PMD_COLOUR, - PG_PMD_NR, false); + PG_PMD_NR, ZAP_FLAG_CHECK_MAPPING); xas_reset(xas); xas_lock_irq(xas); } @@ -623,7 +623,8 @@ struct page *dax_layout_busy_page_range(struct address_space *mapping, * guaranteed to either see new references or prevent new * references from being established. */ - unmap_mapping_pages(mapping, start_idx, end_idx - start_idx + 1, 0); + unmap_mapping_pages(mapping, start_idx, end_idx - start_idx + 1, + ZAP_FLAG_CHECK_MAPPING); xas_lock_irq(&xas); xas_for_each(&xas, entry, end_idx) { @@ -754,9 +755,10 @@ static void *dax_insert_entry(struct xa_state *xas, /* we are replacing a zero page with block mapping */ if (dax_is_pmd_entry(entry)) unmap_mapping_pages(mapping, index & ~PG_PMD_COLOUR, - PG_PMD_NR, false); + PG_PMD_NR, ZAP_FLAG_CHECK_MAPPING); else /* pte entry */ - unmap_mapping_pages(mapping, index, 1, false); + unmap_mapping_pages(mapping, index, 1, + ZAP_FLAG_CHECK_MAPPING); } xas_reset(xas); diff --git a/include/linux/mm.h b/include/linux/mm.h index 1adf313a01fe..b1fb2826e29c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1803,7 +1803,7 @@ extern int fixup_user_fault(struct mm_struct *mm, unsigned long address, unsigned int fault_flags, bool *unlocked); void unmap_mapping_pages(struct address_space *mapping, - pgoff_t start, pgoff_t nr, bool even_cows); + pgoff_t start, pgoff_t nr, unsigned long zap_flags); void unmap_mapping_range(struct address_space *mapping, loff_t const holebegin, loff_t const holelen, int even_cows); #else @@ -1823,7 +1823,7 @@ static inline int fixup_user_fault(struct mm_struct *mm, unsigned long address, return -EFAULT; } static inline void unmap_mapping_pages(struct address_space *mapping, - pgoff_t start, pgoff_t nr, bool even_cows) { } + pgoff_t start, pgoff_t nr, unsigned long zap_flags) { } static inline void unmap_mapping_range(struct address_space *mapping, loff_t const holebegin, loff_t const holelen, int even_cows) { } #endif diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 7376a9b5bfc9..9e89a032e2fd 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1830,7 +1830,8 @@ static void collapse_file(struct mm_struct *mm, } if (page_mapped(page)) - unmap_mapping_pages(mapping, index, 1, false); + unmap_mapping_pages(mapping, index, 1, + ZAP_FLAG_CHECK_MAPPING); xas_lock_irq(&xas); xas_set(&xas, index); diff --git a/mm/memory.c b/mm/memory.c index 8a3751be87ba..319552efc782 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3316,7 +3316,10 @@ static inline void unmap_mapping_range_tree(struct rb_root_cached *root, * @mapping: The address space containing pages to be unmapped. * @start: Index of first page to be unmapped. * @nr: Number of pages to be unmapped. 0 to unmap to end of file. - * @even_cows: Whether to unmap even private COWed pages. + * @zap_flags: Zap flags for the process. E.g., when ZAP_FLAG_CHECK_MAPPING is + * passed into it, we will only zap the pages that are in the same mapping + * specified in the @mapping parameter; otherwise we will not check mapping, + * IOW cow pages will be zapped too. * * Unmap the pages in this address space from any userspace process which * has them mmaped. Generally, you want to remove COWed pages as well when @@ -3324,17 +3327,14 @@ static inline void unmap_mapping_range_tree(struct rb_root_cached *root, * cache. */ void unmap_mapping_pages(struct address_space *mapping, pgoff_t start, - pgoff_t nr, bool even_cows) + pgoff_t nr, unsigned long zap_flags) { pgoff_t first_index = start, last_index = start + nr - 1; struct zap_details details = { .zap_mapping = mapping, - .zap_flags = ZAP_FLAG_SKIP_SWAP, + .zap_flags = zap_flags | ZAP_FLAG_SKIP_SWAP, }; - if (!even_cows) - details.zap_flags |= ZAP_FLAG_CHECK_MAPPING; - if (last_index < first_index) last_index = ULONG_MAX; @@ -3376,7 +3376,8 @@ void unmap_mapping_range(struct address_space *mapping, hlen = ULONG_MAX - hba + 1; } - unmap_mapping_pages(mapping, hba, hlen, even_cows); + unmap_mapping_pages(mapping, hba, hlen, even_cows ? + 0 : ZAP_FLAG_CHECK_MAPPING); } EXPORT_SYMBOL(unmap_mapping_range); diff --git a/mm/truncate.c b/mm/truncate.c index 57a618c4a0d6..85cd84486589 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -172,7 +172,8 @@ truncate_cleanup_page(struct address_space *mapping, struct page *page) { if (page_mapped(page)) { unsigned int nr = thp_nr_pages(page); - unmap_mapping_pages(mapping, page->index, nr, false); + unmap_mapping_pages(mapping, page->index, nr, + ZAP_FLAG_CHECK_MAPPING); } if (page_has_private(page)) @@ -652,14 +653,15 @@ int invalidate_inode_pages2_range(struct address_space *mapping, * Zap the rest of the file in one hit. */ unmap_mapping_pages(mapping, index, - (1 + end - index), false); + (1 + end - index), + ZAP_FLAG_CHECK_MAPPING); did_range_unmap = 1; } else { /* * Just zap this page */ unmap_mapping_pages(mapping, index, - 1, false); + 1, ZAP_FLAG_CHECK_MAPPING); } } BUG_ON(page_mapped(page)); @@ -685,7 +687,8 @@ int invalidate_inode_pages2_range(struct address_space *mapping, * get remapped later. */ if (dax_mapping(mapping)) { - unmap_mapping_pages(mapping, start, end - start + 1, false); + unmap_mapping_pages(mapping, start, end - start + 1, + ZAP_FLAG_CHECK_MAPPING); } out: cleancache_invalidate_inode(mapping); From patchwork Thu May 27 20:22:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285273 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44159C47089 for ; Thu, 27 May 2021 20:22:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DB87961360 for ; Thu, 27 May 2021 20:22:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DB87961360 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7461D6B0078; Thu, 27 May 2021 16:22:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 71CC26B007B; Thu, 27 May 2021 16:22:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 597388D0001; Thu, 27 May 2021 16:22:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0165.hostedemail.com [216.40.44.165]) by kanga.kvack.org (Postfix) with ESMTP id 276C16B0078 for ; Thu, 27 May 2021 16:22:25 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id BF8BE181AF5C7 for ; Thu, 27 May 2021 20:22:24 +0000 (UTC) X-FDA: 78188133408.27.5E2E9C0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 2AD368019347 for ; Thu, 27 May 2021 20:22:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622146944; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OeYSriSsh+QfEkLdbAN1Lkm2SKHr26WYg3zZzwgiYd4=; b=ZQZPy08Mn2GV8TeeIWTPRR4WrIY0QmShGg3TZNMsYT1U6PTgTTAz27L6eO4thzHpBTpqiG kZkCP3+YlujY8N37BQVbA6MCyLcphxnhqyrwhgVbgAQFPcaiYf4cAROPP0tHtiRlHXDWeX 1GwwA910d30J+9EI7kwfY/vUmcib7r4= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-70-5Co5exXeMJactT6WMOG7Fw-1; Thu, 27 May 2021 16:22:22 -0400 X-MC-Unique: 5Co5exXeMJactT6WMOG7Fw-1 Received: by mail-qv1-f69.google.com with SMTP id n12-20020a0c8c0c0000b02901edb8963d4dso999357qvb.18 for ; Thu, 27 May 2021 13:22:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OeYSriSsh+QfEkLdbAN1Lkm2SKHr26WYg3zZzwgiYd4=; b=Qc/2c/Shi9NZYn9ajx+/bxhAOzNU5csAm673vBxNQkt+723RSXUk43/EoNhshIkGNj YlMVIt0OsfryYDzxLF+8dCJRRVl1Ept7l+dsOqprRGnK1gR+FcKvZfJkc6FtYEVIP4wW FvMRs7TR5HNCDAgtb7NYZ3DXyTeNrNKftcrjjnDH0BF5monEmoissR6d+RJr4JPUmvz2 B1a8wHV+HV1k6/K+/Y/THGMX9lfglPKkAuxb9z4EbaW/8y4SgzO0aegvBDhGdhx5qII8 myXvXgEyb14JyZQCmgLAGQ17SibYf/gRKHfnpkp5KTRZUL5CkbL882SvKC97nKhBy5t0 2brw== X-Gm-Message-State: AOAM530KNZULfIj6GAV+uYTxMbrEf4lTT9AHE3NJZ9GlSPkttpJl52bW 3AEV0vKDVD+7K5DeksUF5OjWS9NMq/f9Y8BZJPNuzTl7VQyJdaGyB07gWHj8o+UNNIxDXYdvkH7 C263XZ+l7TCU= X-Received: by 2002:a05:622a:1302:: with SMTP id v2mr319782qtk.249.1622146938634; Thu, 27 May 2021 13:22:18 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyqyJv0mMx2ONGss7dcFYYY0GabLMFAgVcEuoifYAbtVwjqSVSBKEX6G9LpssgOY4fgVj+ezg== X-Received: by 2002:a05:622a:1302:: with SMTP id v2mr319758qtk.249.1622146938218; Thu, 27 May 2021 13:22:18 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id k13sm2023591qke.46.2021.05.27.13.22.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:22:17 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: peterx@redhat.com, Mike Kravetz , "Kirill A . Shutemov" , Jason Gunthorpe , Hugh Dickins , Matthew Wilcox , Andrew Morton , Miaohe Lin , Jerome Glisse , Nadav Amit , Axel Rasmussen , Andrea Arcangeli , Mike Rapoport Subject: [PATCH v3 11/27] shmem/userfaultfd: Persist uffd-wp bit across zapping for file-backed Date: Thu, 27 May 2021 16:22:14 -0400 Message-Id: <20210527202214.31319-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZQZPy08M; spf=none (imf27.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 2AD368019347 X-Stat-Signature: 6xzcumno7zqykau3ykonem986cf4uezb X-HE-Tag: 1622146936-747328 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: File-backed memory is prone to being unmapped at any time. It means all information in the pte will be dropped, including the uffd-wp flag. Since the uffd-wp info cannot be stored in page cache or swap cache, persist this wr-protect information by installing the special uffd-wp marker pte when we're going to unmap a uffd wr-protected pte. When the pte is accessed again, we will know it's previously wr-protected by recognizing the special pte. Meanwhile add a new flag ZAP_FLAG_DROP_FILE_UFFD_WP when we don't want to persist such an information. For example, when destroying the whole vma, or punching a hole in a shmem file. For the latter, we can only drop the uffd-wp bit when holding the page lock. It means the unmap_mapping_range() in shmem_fallocate() still reuqires to zap without ZAP_FLAG_DROP_FILE_UFFD_WP because that's still racy with the page faults. Signed-off-by: Peter Xu --- include/linux/mm.h | 11 ++++++++++ include/linux/mm_inline.h | 43 +++++++++++++++++++++++++++++++++++++++ mm/memory.c | 42 +++++++++++++++++++++++++++++++++++++- mm/rmap.c | 8 ++++++++ mm/truncate.c | 8 +++++++- 5 files changed, 110 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index b1fb2826e29c..5989fc7ed00d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1725,6 +1725,8 @@ extern void user_shm_unlock(size_t, struct user_struct *); #define ZAP_FLAG_CHECK_MAPPING BIT(0) /* Whether to skip zapping swap entries */ #define ZAP_FLAG_SKIP_SWAP BIT(1) +/* Whether to completely drop uffd-wp entries for file-backed memory */ +#define ZAP_FLAG_DROP_FILE_UFFD_WP BIT(2) /* * Parameter block passed down to zap_pte_range in exceptional cases. @@ -1757,6 +1759,15 @@ zap_skip_swap(struct zap_details *details) return details->zap_flags & ZAP_FLAG_SKIP_SWAP; } +static inline bool +zap_drop_file_uffd_wp(struct zap_details *details) +{ + if (!details) + return false; + + return details->zap_flags & ZAP_FLAG_DROP_FILE_UFFD_WP; +} + struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr, pte_t pte); struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr, diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 355ea1ee32bd..c29a6ef3a642 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -4,6 +4,8 @@ #include #include +#include +#include /** * page_is_file_lru - should the page be on a file LRU or anon LRU? @@ -104,4 +106,45 @@ static __always_inline void del_page_from_lru_list(struct page *page, update_lru_size(lruvec, page_lru(page), page_zonenum(page), -thp_nr_pages(page)); } + +/* + * If this pte is wr-protected by uffd-wp in any form, arm the special pte to + * replace a none pte. NOTE! This should only be called when *pte is already + * cleared so we will never accidentally replace something valuable. Meanwhile + * none pte also means we are not demoting the pte so if tlb flushed then we + * don't need to do it again; otherwise if tlb flush is postponed then it's + * even better. + * + * Must be called with pgtable lock held. + */ +static inline void +pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr, + pte_t *pte, pte_t pteval) +{ +#ifdef CONFIG_USERFAULTFD + bool arm_uffd_pte = false; + + /* The current status of the pte should be "cleared" before calling */ + WARN_ON_ONCE(!pte_none(*pte)); + + if (vma_is_anonymous(vma)) + return; + + /* A uffd-wp wr-protected normal pte */ + if (unlikely(pte_present(pteval) && pte_uffd_wp(pteval))) + arm_uffd_pte = true; + + /* + * A uffd-wp wr-protected swap pte. Note: this should even work for + * pte_swp_uffd_wp_special() too. + */ + if (unlikely(is_swap_pte(pteval) && pte_swp_uffd_wp(pteval))) + arm_uffd_pte = true; + + if (unlikely(arm_uffd_pte)) + set_pte_at(vma->vm_mm, addr, pte, + pte_swp_mkuffd_wp_special(vma)); +#endif +} + #endif diff --git a/mm/memory.c b/mm/memory.c index 319552efc782..3453b8ae5f4f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -73,6 +73,7 @@ #include #include #include +#include #include @@ -1298,6 +1299,21 @@ copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) return ret; } +/* + * This function makes sure that we'll replace the none pte with an uffd-wp + * swap special pte marker when necessary. Must be with the pgtable lock held. + */ +static inline void +zap_install_uffd_wp_if_needed(struct vm_area_struct *vma, + unsigned long addr, pte_t *pte, + struct zap_details *details, pte_t pteval) +{ + if (zap_drop_file_uffd_wp(details)) + return; + + pte_install_uffd_wp_if_needed(vma, addr, pte, pteval); +} + static unsigned long zap_pte_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, @@ -1335,6 +1351,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); tlb_remove_tlb_entry(tlb, pte, addr); + zap_install_uffd_wp_if_needed(vma, addr, pte, details, + ptent); if (unlikely(!page)) continue; @@ -1359,6 +1377,22 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; } + /* + * If this is a special uffd-wp marker pte... Drop it only if + * enforced to do so. + */ + if (unlikely(is_swap_special_pte(ptent))) { + WARN_ON_ONCE(!pte_swp_uffd_wp_special(ptent)); + /* + * If this is a common unmap of ptes, keep this as is. + * Drop it only if this is a whole-vma destruction. + */ + if (zap_drop_file_uffd_wp(details)) + ptep_get_and_clear_full(mm, addr, pte, + tlb->fullmm); + continue; + } + entry = pte_to_swp_entry(ptent); if (is_device_private_entry(entry) || is_device_exclusive_entry(entry)) { @@ -1373,6 +1407,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, page_remove_rmap(page, false); put_page(page); + zap_install_uffd_wp_if_needed(vma, addr, pte, details, + ptent); continue; } @@ -1390,6 +1426,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, if (unlikely(!free_swap_and_cache(entry))) print_bad_pte(vma, addr, ptent, NULL); pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); + zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); } while (pte++, addr += PAGE_SIZE, addr != end); add_mm_rss_vec(mm, rss); @@ -1589,12 +1626,15 @@ void unmap_vmas(struct mmu_gather *tlb, unsigned long end_addr) { struct mmu_notifier_range range; + struct zap_details details = { + .zap_flags = ZAP_FLAG_DROP_FILE_UFFD_WP, + }; mmu_notifier_range_init(&range, MMU_NOTIFY_UNMAP, 0, vma, vma->vm_mm, start_addr, end_addr); mmu_notifier_invalidate_range_start(&range); for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next) - unmap_single_vma(tlb, vma, start_addr, end_addr, NULL); + unmap_single_vma(tlb, vma, start_addr, end_addr, &details); mmu_notifier_invalidate_range_end(&range); } diff --git a/mm/rmap.c b/mm/rmap.c index 0419c9a1a280..a94d9aed9d95 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -72,6 +72,7 @@ #include #include #include +#include #include @@ -1509,6 +1510,13 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, pteval = ptep_clear_flush(vma, address, pvmw.pte); } + /* + * Now the pte is cleared. If this is uffd-wp armed pte, we + * may want to replace a none pte with a marker pte if it's + * file-backed, so we don't lose the tracking information. + */ + pte_install_uffd_wp_if_needed(vma, address, pvmw.pte, pteval); + /* Move the dirty bit to the page. Now the pte is gone. */ if (pte_dirty(pteval)) set_page_dirty(page); diff --git a/mm/truncate.c b/mm/truncate.c index 85cd84486589..62f9c488b986 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -173,7 +173,13 @@ truncate_cleanup_page(struct address_space *mapping, struct page *page) if (page_mapped(page)) { unsigned int nr = thp_nr_pages(page); unmap_mapping_pages(mapping, page->index, nr, - ZAP_FLAG_CHECK_MAPPING); + ZAP_FLAG_CHECK_MAPPING | + /* + * Now it's safe to drop uffd-wp because + * we're with page lock, and the page is + * being truncated. + */ + ZAP_FLAG_DROP_FILE_UFFD_WP); } if (page_has_private(page)) From patchwork Thu May 27 20:22:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285275 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37987C4707F for ; Thu, 27 May 2021 20:22:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CD338613CC for ; Thu, 27 May 2021 20:22:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CD338613CC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 682C16B006C; Thu, 27 May 2021 16:22:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 658ED6B006E; Thu, 27 May 2021 16:22:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4FA0E6B0070; Thu, 27 May 2021 16:22:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0030.hostedemail.com [216.40.44.30]) by kanga.kvack.org (Postfix) with ESMTP id 1D8646B006C for ; Thu, 27 May 2021 16:22:54 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id B2DD7181AF5C7 for ; Thu, 27 May 2021 20:22:53 +0000 (UTC) X-FDA: 78188134626.18.17031DC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 85CF6600027A for ; Thu, 27 May 2021 20:22:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622146973; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8MXCY9EtHu8l00OeGEX9KSWIeSxJZ91wjW9WQpZLBho=; b=JNJb190BzckPJojNXouHff0883NwMCeoZ4IdxWt9SGSuKZt2Tkal5CXDUMTFKAPs4uGI4Y bkZn7M/gFfn5gEsmWq4rELB4X+6EBzn/6LxObwodL2IKynjiVcazIdKSvFGxRuqvS/GvvX FIH4/DlbmNHPGAyieMrnphjMJpzF19M= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-308-7wOOhiiVPQe_3e8eKCz2sQ-1; Thu, 27 May 2021 16:22:51 -0400 X-MC-Unique: 7wOOhiiVPQe_3e8eKCz2sQ-1 Received: by mail-qv1-f72.google.com with SMTP id j14-20020a0cf30e0000b02902142ba51ef2so1021360qvl.10 for ; Thu, 27 May 2021 13:22:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8MXCY9EtHu8l00OeGEX9KSWIeSxJZ91wjW9WQpZLBho=; b=nrRaxrb17UOBViHDTGTfmUz3AOTIgn+rNMb5nfmsOgndgU2yln/VtcIuG264axBh+o zfyApDK6M3RAPsCoR2LBYan7HF/ccpjUDpKm+OcGkx4dgE9bObGBthcR1n2XaehzbzOG RX/c3p1zzzVhtmVhlvPBQraOR0QDu4KV5Vtd2NP0uA09gQptIro/mKe769V8SGVO4glb KeEbs7OFfDMNLEitEQB7FXxdW5ogcL+s28tfIagHs89YIt5QXcVWdwtioBxmrGlBBMqc i5QxJc6czF9E9SlNsWxU6pgHff+HPPj+pF8KVyy/oonsgWzLeAeFJy+mx2bhH+a6RYnP xo+Q== X-Gm-Message-State: AOAM533WaxoWwIqb9nrDk6tECcd7f7wNV1g/IBazxycSZaSfY87KhbID Y+5Qzq/LajSGxjXEc47fXDTO313+TjrkXTjfZmK/voptlkOlSSGtspS9moTrYtX5CBLXp25B8pt +kLcvKZS07qc= X-Received: by 2002:a05:620a:787:: with SMTP id 7mr285034qka.397.1622146970779; Thu, 27 May 2021 13:22:50 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxRUikWEOLkY4CRYqDWC3dJNWhf0mol6ag7uMDBXVUXW2OhGQF4dTOGhSEyiU91Ub93XNgb1A== X-Received: by 2002:a05:620a:787:: with SMTP id 7mr284996qka.397.1622146970458; Thu, 27 May 2021 13:22:50 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id n19sm1977380qtp.65.2021.05.27.13.22.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:22:49 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Rapoport , peterx@redhat.com, Andrew Morton , Mike Kravetz , Jerome Glisse , Miaohe Lin , Nadav Amit , Hugh Dickins , Matthew Wilcox , Jason Gunthorpe , "Kirill A . Shutemov" , Andrea Arcangeli , Axel Rasmussen Subject: [PATCH v3 12/27] shmem/userfaultfd: Allow wr-protect none pte for file-backed mem Date: Thu, 27 May 2021 16:22:47 -0400 Message-Id: <20210527202247.31613-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JNJb190B; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf09.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 85CF6600027A X-Stat-Signature: iuxg3jghpsneaf9e9tbejic3os1thnnf X-HE-Tag: 1622146968-17280 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: File-backed memory differs from anonymous memory in that even if the pte is missing, the data could still resides either in the file or in page/swap cache. So when wr-protect a pte, we need to consider none ptes too. We do that by installing the uffd-wp special swap pte as a marker. So when there's a future write to the pte, the fault handler will go the special path to first fault-in the page as read-only, then report to userfaultfd server with the wr-protect message. On the other hand, when unprotecting a page, it's also possible that the pte got unmapped but replaced by the special uffd-wp marker. Then we'll need to be able to recover from a uffd-wp special swap pte into a none pte, so that the next access to the page will fault in correctly as usual when trigger the fault handler next time, rather than sending a uffd-wp message. Special care needs to be taken throughout the change_protection_range() process. Since now we allow user to wr-protect a none pte, we need to be able to pre-populate the page table entries if we see !anonymous && MM_CP_UFFD_WP requests, otherwise change_protection_range() will always skip when the pgtable entry does not exist. Note that this patch only covers the small pages (pte level) but not covering any of the transparent huge pages yet. But this will be a base for thps too. Signed-off-by: Peter Xu --- mm/mprotect.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/mm/mprotect.c b/mm/mprotect.c index 4b743394afbe..8ec85b276975 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -29,6 +29,7 @@ #include #include #include +#include #include #include #include @@ -186,6 +187,32 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, set_pte_at(vma->vm_mm, addr, pte, newpte); pages++; } + } else if (unlikely(is_swap_special_pte(oldpte))) { + if (uffd_wp_resolve && !vma_is_anonymous(vma) && + pte_swp_uffd_wp_special(oldpte)) { + /* + * This is uffd-wp special pte and we'd like to + * unprotect it. What we need to do is simply + * recover the pte into a none pte; the next + * page fault will fault in the page. + */ + pte_clear(vma->vm_mm, addr, pte); + pages++; + } + } else { + /* It must be an none page, or what else?.. */ + WARN_ON_ONCE(!pte_none(oldpte)); + if (unlikely(uffd_wp && !vma_is_anonymous(vma))) { + /* + * For file-backed mem, we need to be able to + * wr-protect even for a none pte! Because + * even if the pte is null, the page/swap cache + * could exist. + */ + set_pte_at(vma->vm_mm, addr, pte, + pte_swp_mkuffd_wp_special(vma)); + pages++; + } } } while (pte++, addr += PAGE_SIZE, addr != end); arch_leave_lazy_mmu_mode(); @@ -219,6 +246,25 @@ static inline int pmd_none_or_clear_bad_unless_trans_huge(pmd_t *pmd) return 0; } +/* + * File-backed vma allows uffd wr-protect upon none ptes, because even if pte + * is missing, page/swap cache could exist. When that happens, the wr-protect + * information will be stored in the page table entries with the marker (e.g., + * PTE_SWP_UFFD_WP_SPECIAL). Prepare for that by always populating the page + * tables to pte level, so that we'll install the markers in change_pte_range() + * where necessary. + * + * Note that we only need to do this in pmd level, because if pmd does not + * exist, it means the whole range covered by the pmd entry (of a pud) does not + * contain any valid data but all zeros. Then nothing to wr-protect. + */ +#define change_protection_prepare(vma, pmd, addr, cp_flags) \ + do { \ + if (unlikely((cp_flags & MM_CP_UFFD_WP) && pmd_none(*pmd) && \ + !vma_is_anonymous(vma))) \ + WARN_ON_ONCE(pte_alloc(vma->vm_mm, pmd)); \ + } while (0) + static inline unsigned long change_pmd_range(struct vm_area_struct *vma, pud_t *pud, unsigned long addr, unsigned long end, pgprot_t newprot, unsigned long cp_flags) @@ -237,6 +283,8 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma, next = pmd_addr_end(addr, end); + change_protection_prepare(vma, pmd, addr, cp_flags); + /* * Automatic NUMA balancing walks the tables with mmap_lock * held for read. It's possible a parallel update to occur From patchwork Thu May 27 20:22:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285277 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72592C4708A for ; Thu, 27 May 2021 20:23:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1BF8A613CC for ; Thu, 27 May 2021 20:23:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1BF8A613CC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AEF676B006E; Thu, 27 May 2021 16:23:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A9F946B0070; Thu, 27 May 2021 16:23:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 940A76B0071; Thu, 27 May 2021 16:23:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0211.hostedemail.com [216.40.44.211]) by kanga.kvack.org (Postfix) with ESMTP id 612426B006E for ; Thu, 27 May 2021 16:23:00 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 0376818041626 for ; Thu, 27 May 2021 20:23:00 +0000 (UTC) X-FDA: 78188134920.06.71CA0DC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 6B9A8E00203C for ; Thu, 27 May 2021 20:22:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622146979; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ltWxXgdsERQEx3E6ySA6em4pDoRaSyrvcOj0w7z20l8=; b=LQjs54OQdgoOoAQklNH+D7mRBEbpIXiHy4BNJliU818u4V9iD0KhtFStppVM+Hcxli+FI/ QkmiqHlGIjhjc3bO96dAf6jI8SdBJPGTEXxUw5z75yh+avv89z3s6B/ZPqMP+rldTcaAQr NJzotgZvrdW83ZHgBgksdvaR1U9xpHo= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-204-ll8Ylpu_NqCXv5NPD7WYTg-1; Thu, 27 May 2021 16:22:55 -0400 X-MC-Unique: ll8Ylpu_NqCXv5NPD7WYTg-1 Received: by mail-qk1-f198.google.com with SMTP id z2-20020a3765020000b02903a5f51b1c74so1341283qkb.7 for ; Thu, 27 May 2021 13:22:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ltWxXgdsERQEx3E6ySA6em4pDoRaSyrvcOj0w7z20l8=; b=jiX6l2Roz4Qfhwi5WiNQMd428g1aDvU/CoVmHzUWUmJWKVCmW+lnEIth3Wm/XDHYRS bj73JfR054kFnmfNosZMfJyLzcqgoTmD0ZxIyDKZFanAJwQ3ykCGkcKWoPUt91lNNb7G dNICIK2U/wYg6A5BrIovlKLIPcFWIHdRqTJVyZPtBkphMqy3GbHUXFWo3gU+94bIc9gR iWLvCjuGRbFVEgwgV2Kw+rQPDreUyPZ8mZCjkmaYiUT6ZcusS6h2QHJyW1VlL9ss7VWU e7dFVClfnc0XJ3Z4l03wBkS7nWk/PICDINukws2bcmBSpU0kCaRh7A+dEFWzekNWuI/j 9j3A== X-Gm-Message-State: AOAM531k2YbsdEANrmomIcARdr3PG5/Aae36stlf5QwEud/fjy61XgVd DXXBELXdhikY49lMVR8TjN7Rhf8cXhbfn9IsrhKwyMp/CSgU+qGEY71yGNTQb4f88d6srP17+FM cj2iep2F1WO4= X-Received: by 2002:ac8:5784:: with SMTP id v4mr352659qta.264.1622146974727; Thu, 27 May 2021 13:22:54 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw13s6ljQoCm32IG4W01f9UJM4WrdJHKw4FnJpF0acfwSUTrTtR0tUbI/5AXylz2fDUX/g5XA== X-Received: by 2002:ac8:5784:: with SMTP id v4mr352632qta.264.1622146974465; Thu, 27 May 2021 13:22:54 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id j62sm2075122qkf.125.2021.05.27.13.22.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:22:53 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Rapoport , peterx@redhat.com, Andrew Morton , Mike Kravetz , Jerome Glisse , Miaohe Lin , Nadav Amit , Hugh Dickins , Matthew Wilcox , Jason Gunthorpe , "Kirill A . Shutemov" , Andrea Arcangeli , Axel Rasmussen Subject: [PATCH v3 13/27] shmem/userfaultfd: Allows file-back mem to be uffd wr-protected on thps Date: Thu, 27 May 2021 16:22:50 -0400 Message-Id: <20210527202250.31663-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LQjs54OQ; spf=none (imf13.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 6B9A8E00203C X-Stat-Signature: hh8shbda4z5dtokgf4cha9ffz1acoeeu X-HE-Tag: 1622146971-233653 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We don't have "huge" version of PTE_SWP_UFFD_WP_SPECIAL, instead when necessary we split the thp if the huge page is uffd wr-protected previously. However split the thp is not enough, because file-backed thp is handled totally differently comparing to anonymous thps - rather than doing a real split, the thp pmd will simply got dropped in __split_huge_pmd_locked(). That is definitely not enough if e.g. when there is a thp covers range [0, 2M) but we want to wr-protect small page resides in [4K, 8K) range, because after __split_huge_pmd() returns, there will be a none pmd. Here we leverage the previously introduced change_protection_prepare() macro so that we'll populate the pmd with a pgtable page. Then change_pte_range() will do all the rest for us, e.g., install the uffd-wp swap special pte marker at any pte that we'd like to wr-protect, under the protection of pgtable lock. Signed-off-by: Peter Xu --- mm/mprotect.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/mm/mprotect.c b/mm/mprotect.c index 8ec85b276975..3fcb87b59696 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -306,8 +306,16 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma, } if (is_swap_pmd(*pmd) || pmd_trans_huge(*pmd) || pmd_devmap(*pmd)) { - if (next - addr != HPAGE_PMD_SIZE) { + if (next - addr != HPAGE_PMD_SIZE || + /* Uffd wr-protecting a file-backed memory range */ + unlikely(!vma_is_anonymous(vma) && + (cp_flags & MM_CP_UFFD_WP))) { __split_huge_pmd(vma, pmd, addr, false, NULL); + /* + * For file-backed, the pmd could have been + * gone; still provide a pte pgtable if needed. + */ + change_protection_prepare(vma, pmd, addr, cp_flags); } else { int nr_ptes = change_huge_pmd(vma, pmd, addr, newprot, cp_flags); From patchwork Thu May 27 20:22:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285281 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB1FBC47089 for ; Thu, 27 May 2021 20:23:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5635D61360 for ; Thu, 27 May 2021 20:23:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5635D61360 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 85A176B0071; Thu, 27 May 2021 16:23:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 82F516B0074; Thu, 27 May 2021 16:23:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 60E8E6B007B; Thu, 27 May 2021 16:23:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0146.hostedemail.com [216.40.44.146]) by kanga.kvack.org (Postfix) with ESMTP id 198C16B0071 for ; Thu, 27 May 2021 16:23:05 -0400 (EDT) Received: from smtpin37.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 9A17FBEF5 for ; Thu, 27 May 2021 20:23:04 +0000 (UTC) X-FDA: 78188135088.37.2038DAA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 42C90801934B for ; Thu, 27 May 2021 20:22:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622146983; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CPzjkZSYUZ+9ChSaHHQDBWB6stQ1ZKRzO1smrJVYP5w=; b=YfL+yozfiU2duUgY78T78eqkI87KOavNT4sE1s8eaztCpl5YiD/C+d0gafKbbolpofxpVQ tzU5ZODfOXRkCmI16aJqjxGldeSR//aQKuos00frtc0EzqmQV+x7sKJnjyDL+HDg84fua/ hYi33rRK32D8xYGnneM3Izx+Syp1ORA= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-69-UCf0oPJ-NP6tI8KJlV5Jmg-1; Thu, 27 May 2021 16:23:00 -0400 X-MC-Unique: UCf0oPJ-NP6tI8KJlV5Jmg-1 Received: by mail-qt1-f197.google.com with SMTP id d7-20020ac811870000b02901e65f85117bso848533qtj.18 for ; Thu, 27 May 2021 13:23:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CPzjkZSYUZ+9ChSaHHQDBWB6stQ1ZKRzO1smrJVYP5w=; b=fjp/k5XwZc8kauJrd403pBlIHpXl6tCmEaNJL9mGuicRx5RmxqoheIveCqUi67LKT7 MijsqHtFQ3tAjcxnKq8DRQt3s+ncRLTl5uLl1oqukMIUDsPFuHN+BcPTvHEEz5g7UUx9 v3nQGMjyxiCylxXdEb9LGQdW+XbV174Ovul3VrLN97mb69ccSe9bCFhmwKs4Xefgrati 9mtyPZSG7GeElSyxp/TnslCH20iPMezJ84elkgRhJft5e/iAEesuY+SNWoO2iYVh7hKT Ib8ZhnOpcCrA5MbGii6D/HnO24D6tBfkMRZb94MpHGRIzuXzfW0/pVrOjGx4L1unsH7p C+Uw== X-Gm-Message-State: AOAM530BHvliFqmEpc/NSANGyG9WnAXELE2Ce6DnPHegLDaartjN+6NM c5imOG3oU/he8CrG8Dw5wVvMS2OcAAmVRYkUqSzZYz1u+IHoM19/h5TId763Bek8HdD+IC3Gg+k 90uWlAeWH0HA= X-Received: by 2002:ac8:758a:: with SMTP id s10mr335554qtq.31.1622146978657; Thu, 27 May 2021 13:22:58 -0700 (PDT) X-Google-Smtp-Source: ABdhPJx8kaYOKObAD+x3/GT0ewDBGBt60RVnJQo86yzj8GxZwEzdHtJ0I2hFBzBcRwaJqMd8+fukTg== X-Received: by 2002:ac8:758a:: with SMTP id s10mr335533qtq.31.1622146978391; Thu, 27 May 2021 13:22:58 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id e12sm1355961qtj.48.2021.05.27.13.22.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:22:57 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Rapoport , peterx@redhat.com, Andrew Morton , Mike Kravetz , Jerome Glisse , Miaohe Lin , Nadav Amit , Hugh Dickins , Matthew Wilcox , Jason Gunthorpe , "Kirill A . Shutemov" , Andrea Arcangeli , Axel Rasmussen Subject: [PATCH v3 14/27] shmem/userfaultfd: Handle the left-overed special swap ptes Date: Thu, 27 May 2021 16:22:55 -0400 Message-Id: <20210527202255.31716-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 42C90801934B Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YfL+yozf; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf08.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com X-Rspamd-Server: rspam03 X-Stat-Signature: t7b9jsfbc8ye3wg6mcfzyotmf9wbkwtg X-HE-Tag: 1622146976-833534 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Note that the special uffd-wp swap pte can be left over even if the page under the pte got evicted. Normally when evict a page, we will unmap the ptes by walking through the reverse mapping. However we never tracked such information for the special swap ptes because they're not real mappings but just markers. So we need to take care of that when we see a marker but when it's actually meaningless (the page behind it got evicted). We have already taken care of that in e.g. alloc_set_pte() where we'll treat the special swap pte as pte_none() when necessary. However we need to also teach userfaultfd itself on either UFFDIO_COPY or handling page faults, so that everything will still work as expected. Signed-off-by: Peter Xu --- fs/userfaultfd.c | 15 +++++++++++++++ mm/userfaultfd.c | 13 ++++++++++++- 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 19ebae443ade..15031d6f1f17 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -329,6 +329,21 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx, */ if (pte_none(*pte)) ret = true; + /* + * We also treat the swap special uffd-wp pte as the pte_none() here. + * This should in most cases be a missing event, as we never handle + * wr-protect upon a special uffd-wp swap pte - it should first be + * converted into a normal read request before handling wp. It just + * means the page/swap cache that backing this pte is gone, so this + * special pte is leftover. + * + * We can't simply replace it with a none pte because we're not with + * the pgtable lock here. Instead of taking it and clearing the pte, + * the easy way is to let UFFDIO_COPY understand this pte too when + * trying to install a new page onto it. + */ + if (pte_swp_uffd_wp_special(*pte)) + ret = true; if (!pte_write(*pte) && (reason & VM_UFFD_WP)) ret = true; pte_unmap(pte); diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 3636f5be6390..147e86095070 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -100,7 +100,18 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, } ret = -EEXIST; - if (!pte_none(*dst_pte)) + /* + * Besides the none pte, we also allow UFFDIO_COPY to install a pte + * onto the uffd-wp swap special pte, because that pte should be the + * same as a pte_none() just in that it contains wr-protect information + * (which could only be dropped when unmap the memory). + * + * It's safe to drop that marker because we know this is part of a + * MISSING fault, and the caller is very clear about this page missing + * rather than wr-protected. Then we're sure the wr-protect bit is + * just a leftover so it's useless already and is the same as none pte. + */ + if (!pte_none(*dst_pte) && !pte_swp_uffd_wp_special(*dst_pte)) goto out_unlock; if (page_in_cache) From patchwork Thu May 27 20:22:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285283 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 112D8C4707F for ; Thu, 27 May 2021 20:23:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AF4F9613CC for ; Thu, 27 May 2021 20:23:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AF4F9613CC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4FB2E6B0074; Thu, 27 May 2021 16:23:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4AC1A6B007B; Thu, 27 May 2021 16:23:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 34BFD6B007D; Thu, 27 May 2021 16:23:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0183.hostedemail.com [216.40.44.183]) by kanga.kvack.org (Postfix) with ESMTP id 042C66B0074 for ; Thu, 27 May 2021 16:23:08 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 96E84181AF5C7 for ; Thu, 27 May 2021 20:23:08 +0000 (UTC) X-FDA: 78188135256.17.EC0ACCC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 13A66C007765 for ; Thu, 27 May 2021 20:22:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622146988; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VwFrdF3JNaZBFM+3FwQbg8GtNhiuS7dByxCQ/uoQV90=; b=LP30Y1p3UiEY5u+T+NceJ+hh0cJ022DLI0nOq2UzlLPGR9WKgs6vR5Yrzk9ytPViL2O74T mM0bcoYdWDVWyUcQbEiae723PgD5YOC+yfp8bT6nMwvhdf5b0D0fVXO5F+x1cTiBD7aNOs wmpIA5nxQVAnQccoTeHNRf+IrfpmNXI= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-219-AtUg7qgRNhOu-g-YRIigsQ-1; Thu, 27 May 2021 16:23:04 -0400 X-MC-Unique: AtUg7qgRNhOu-g-YRIigsQ-1 Received: by mail-qk1-f198.google.com with SMTP id a76-20020ae9e84f0000b02903a69ae4796aso1326495qkg.12 for ; Thu, 27 May 2021 13:23:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VwFrdF3JNaZBFM+3FwQbg8GtNhiuS7dByxCQ/uoQV90=; b=SBIuW5yCHyjNwuBmeKQH9lRTHWdFG9IawgzaMHrVjRDQn4r0ALTx7npJz73/DWoNj+ G6mHyLFQzrysrTv+A6SR4BGbHZAGKAUFDC+I27MyiS63OrojDXFsbaRjjOzXVO+HRljd Bas44fX33OxjQa3ctED3fLmlrtwNi+Irsm0i2xM0ih8IPPxAkJvs+lfWF7MTYs0pnNhf jqZwo/86lt7M2TmjZV6Yb1PhifLWy4Sc1W1rKQIzHWwXGYVxr0PsKwv8X9E5aJw1wMKK 20mBuu/LfWYBiusqCBomWEJFTIOJ+NzP2pIPpgRdabz22cYU3iiLcD7CIDy67ATRZfh2 fnuw== X-Gm-Message-State: AOAM5311+J0U7mgVT4gwqv0RTzrZ5cYyw9e8MSfetpFyEZsNiAgdy3L9 MOAaOhr/5t0Gz20SAcBjv564NJBsRMU8sIs4k0K+/st8Bi0vnexSf9ARKAv9veXD+eHhZlsY+dx WxGCzFOvzijI= X-Received: by 2002:a37:aa4c:: with SMTP id t73mr330202qke.79.1622146983322; Thu, 27 May 2021 13:23:03 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwzsk606Pw4SsfXcguARe8KrXTgefcf4445STrLT/TYxczzrKEpV25kmsn40HUYN5w/RJZYAQ== X-Received: by 2002:a37:aa4c:: with SMTP id t73mr330178qke.79.1622146983093; Thu, 27 May 2021 13:23:03 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id g85sm2024356qke.123.2021.05.27.13.23.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:23:02 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Rapoport , peterx@redhat.com, Andrew Morton , Mike Kravetz , Jerome Glisse , Miaohe Lin , Nadav Amit , Hugh Dickins , Matthew Wilcox , Jason Gunthorpe , "Kirill A . Shutemov" , Andrea Arcangeli , Axel Rasmussen Subject: [PATCH v3 15/27] shmem/userfaultfd: Pass over uffd-wp special swap pte when fork() Date: Thu, 27 May 2021 16:22:59 -0400 Message-Id: <20210527202259.31767-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LP30Y1p3; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf06.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 13A66C007765 X-Stat-Signature: axr1f9cgeq8gytkcxtbch8r8tr19iucb X-HE-Tag: 1622146979-550741 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It should be handled similarly like other uffd-wp wr-protected ptes: we should pass it over when the dst_vma has VM_UFFD_WP armed, otherwise drop it. Signed-off-by: Peter Xu --- mm/memory.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 3453b8ae5f4f..8372b212993a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -777,8 +777,21 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, unsigned long vm_flags = dst_vma->vm_flags; pte_t pte = *src_pte; struct page *page; - swp_entry_t entry = pte_to_swp_entry(pte); + swp_entry_t entry; + + if (unlikely(is_swap_special_pte(pte))) { + /* + * uffd-wp special swap pte is the only possibility for now. + * If dst vma is registered with uffd-wp, copy it over. + * Otherwise, ignore this pte as if it's a none pte would work. + */ + WARN_ON_ONCE(!pte_swp_uffd_wp_special(pte)); + if (userfaultfd_wp(dst_vma)) + set_pte_at(dst_mm, addr, dst_pte, pte); + return 0; + } + entry = pte_to_swp_entry(pte); if (likely(!non_swap_entry(entry))) { if (swap_duplicate(entry) < 0) return -EAGAIN; From patchwork Thu May 27 20:23:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285285 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCBC9C47089 for ; Thu, 27 May 2021 20:23:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 615D0613D4 for ; Thu, 27 May 2021 20:23:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 615D0613D4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 018DB6B007B; Thu, 27 May 2021 16:23:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EE4AE6B007D; Thu, 27 May 2021 16:23:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D856A6B007E; Thu, 27 May 2021 16:23:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0060.hostedemail.com [216.40.44.60]) by kanga.kvack.org (Postfix) with ESMTP id A543D6B007B for ; Thu, 27 May 2021 16:23:13 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 3F33F18016699 for ; Thu, 27 May 2021 20:23:13 +0000 (UTC) X-FDA: 78188135466.27.CD48D7E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf05.hostedemail.com (Postfix) with ESMTP id 4F582E005F2A for ; Thu, 27 May 2021 20:23:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622146992; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RoKsIz4q4xSpRNm9hs0BrG9ToefGgP2GtNCOUn+H6ys=; b=Xsn8lpx82oVKpgTDr/uHOxvcMwslY9jnP/WUSQkXVPPcKR24+Wgs2aYpEDxX+M4WyaFLAF nCknWO6G+2NGI2O6UWD54t4GzEA96xlUOvA1K0lBBzmecm74UfPJ5S0y0TxwhR5utynNV3 mISv55U/8Qv+3K0Oj6pLW7IxYBDOoqI= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-302-H68u661VNzSUMJP2AIqa7g-1; Thu, 27 May 2021 16:23:07 -0400 X-MC-Unique: H68u661VNzSUMJP2AIqa7g-1 Received: by mail-qk1-f199.google.com with SMTP id e8-20020a05620a2088b02903a5edeec4d6so1325826qka.11 for ; Thu, 27 May 2021 13:23:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RoKsIz4q4xSpRNm9hs0BrG9ToefGgP2GtNCOUn+H6ys=; b=CanmYdEwwXLtjddZo6JvFjxXpgWBbo50M7KzwkTMnI/wJIPcnWRpIiaPzTyMnZ5sME fYVxRUG5D2yF6IXp05qLbP9LQd8QYXsb0DNNC+VcQ+JQ65RlbscrBJcqqsmOuON0hrdE RUAhYNmH66gFPmzTt6hrYAoYOl25TKbAMqGhsOFo+FBF3dwS7Ss9aMDl+fCdPABUbp+G dT8ee+dUGe4dciorwnT4Xz+jKC5Dabfh3GJ4NJoJP8vxAyQNhJfUCT8JxIWg4EkK0Ffu sViwXbBGpMHctnszFelnfhM2i3Qn+NaE3/JVyzYnKX2dhVPznKbBMjivgosdPYpTr88p oztw== X-Gm-Message-State: AOAM531fnNk7lkSR7DYI5EAWxaARo6i5hzT66KKH0NuhSY0d+NQeV2pd 5eGzNfQq4fMv7n7pcgWZCdbhslX4oKZeFVbhz6bTkfCYGw+GXWwv3BM05CEtsP+wvNOFzjt9ogu kR2ujTML8xis= X-Received: by 2002:a05:620a:1654:: with SMTP id c20mr356009qko.282.1622146987393; Thu, 27 May 2021 13:23:07 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxPBMzsCNUZPXEhKlaRCORWI9CkBI4AzSpTEO2+Zm/kawOiakFSPgTl6iyiNKbrAzVRR5rK5g== X-Received: by 2002:a05:620a:1654:: with SMTP id c20mr355999qko.282.1622146987200; Thu, 27 May 2021 13:23:07 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id y20sm2076116qkm.5.2021.05.27.13.23.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:23:06 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Rapoport , peterx@redhat.com, Andrew Morton , Mike Kravetz , Jerome Glisse , Miaohe Lin , Nadav Amit , Hugh Dickins , Matthew Wilcox , Jason Gunthorpe , "Kirill A . Shutemov" , Andrea Arcangeli , Axel Rasmussen Subject: [PATCH v3 16/27] mm/hugetlb: Drop __unmap_hugepage_range definition from hugetlb.h Date: Thu, 27 May 2021 16:23:03 -0400 Message-Id: <20210527202303.31817-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 4F582E005F2A Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Xsn8lpx8; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf05.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com X-Rspamd-Server: rspam03 X-Stat-Signature: 333r7tzibseygswqd8qk58reiqfs9kyr X-HE-Tag: 1622146982-38152 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Drop it in the header since it's only used in hugetlb.c. Suggested-by: Mike Kravetz Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index d944aa0202f0..7ef2b8c2ff41 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -143,9 +143,6 @@ void __unmap_hugepage_range_final(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, unsigned long end, struct page *ref_page); -void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, - unsigned long start, unsigned long end, - struct page *ref_page); void hugetlb_report_meminfo(struct seq_file *); int hugetlb_report_node_meminfo(char *buf, int len, int nid); void hugetlb_show_meminfo(void); @@ -381,13 +378,6 @@ static inline void __unmap_hugepage_range_final(struct mmu_gather *tlb, BUG(); } -static inline void __unmap_hugepage_range(struct mmu_gather *tlb, - struct vm_area_struct *vma, unsigned long start, - unsigned long end, struct page *ref_page) -{ - BUG(); -} - static inline vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, unsigned int flags) From patchwork Thu May 27 20:23:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285287 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75FB0C4708B for ; Thu, 27 May 2021 20:23:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 22250613CC for ; Thu, 27 May 2021 20:23:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 22250613CC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5AE496B007D; Thu, 27 May 2021 16:23:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 52E036B007E; Thu, 27 May 2021 16:23:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CFE7C6B0080; Thu, 27 May 2021 16:23:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0001.hostedemail.com [216.40.44.1]) by kanga.kvack.org (Postfix) with ESMTP id 842E86B007D for ; Thu, 27 May 2021 16:23:14 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 2B63B1801C1A2 for ; Thu, 27 May 2021 20:23:14 +0000 (UTC) X-FDA: 78188135508.18.4486878 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf23.hostedemail.com (Postfix) with ESMTP id C5B03A0003AC for ; Thu, 27 May 2021 20:23:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622146993; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SAmjiglASlkj6C13Cutnvz5k0d0w8IXdZ7Qyh6+DwYY=; b=DcmN0TE1J7W3flR/ry4yJlzUasUgyMIkZM+kRyPNI3BWwEtlimeTOx+bsu2cdAlbmkghDu VVS3AlU3sraiyFYy9U2fMzjnFt8RJa8cUScXMiRCXSNNcCoO9SCmGJgDpqqb3TewnpvPb2 90K0DECXalgV6eYMocCuirsXWT4Vt/A= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-502-wmSOMiChNYCadnaFle_C4A-1; Thu, 27 May 2021 16:23:11 -0400 X-MC-Unique: wmSOMiChNYCadnaFle_C4A-1 Received: by mail-qt1-f199.google.com with SMTP id g21-20020ac870d50000b0290230013e97e6so841284qtp.21 for ; Thu, 27 May 2021 13:23:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SAmjiglASlkj6C13Cutnvz5k0d0w8IXdZ7Qyh6+DwYY=; b=kSmFc/2OthMNFYunVCq2GtaFfF9mZ0ixT55/dnSfONoFZoDQAzdwOsmpB8dhqT28GI ggL4eWWo/RuM1d0RtQIQpApGL483w7U2QxcO5qiJ9wrNjl4jtX07VumMcDYYxZe64G7r LuHA0CljggrwJ/yBN3QTBwYDYQ+yN8ytAul+vEGUM9HQKiR4wVgEpmRaTcNvPVMIftEn fNW1FNlbFoKlEPxmBJz3hg+ImC/Gbv2eHwW9YjQYWsZh25iPTu1HzInIxXOcrjmFgf5U EpLxnAw9DbqLyUsB+MN5fdYE7DFyHvCh0bpBQAbOMprM5u92ae6T+cTpYUzsVyocPH3U 6+Qw== X-Gm-Message-State: AOAM532GLSEGa1i8/CNnegjc57DJJjC7l4faZwm/uTYa+FAQyr6yqF7A Nib+locIgF0KJUeYhtcYhzp6yQ6r51bbUG3/WsxVWiXCSphpwe061D9ybrqkaZExTOC6GyXl9+E /yXgzeizlERk= X-Received: by 2002:ad4:5227:: with SMTP id r7mr340808qvq.41.1622146991377; Thu, 27 May 2021 13:23:11 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxtCJbSTqUjM7suwiR8IhoI+l1zb1zSSJrik2m04Tqem1imNMBVQvJJonjARh3dJDvw2YkfBA== X-Received: by 2002:ad4:5227:: with SMTP id r7mr340783qvq.41.1622146991171; Thu, 27 May 2021 13:23:11 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id c7sm2044392qkc.114.2021.05.27.13.23.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:23:10 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Rapoport , peterx@redhat.com, Andrew Morton , Mike Kravetz , Jerome Glisse , Miaohe Lin , Nadav Amit , Hugh Dickins , Matthew Wilcox , Jason Gunthorpe , "Kirill A . Shutemov" , Andrea Arcangeli , Axel Rasmussen Subject: [PATCH v3 17/27] mm/hugetlb: Introduce huge pte version of uffd-wp helpers Date: Thu, 27 May 2021 16:23:07 -0400 Message-Id: <20210527202307.31882-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DcmN0TE1; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf23.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com X-Stat-Signature: 9pwm8y6p7fc8zs8icsutd8pwnfs7mmzc X-Rspamd-Queue-Id: C5B03A0003AC X-Rspamd-Server: rspam02 X-HE-Tag: 1622146986-744250 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: They will be used in the follow up patches to either check/set/clear uffd-wp bit of a huge pte. So far it reuses all the small pte helpers. Archs can overwrite these versions when necessary (with __HAVE_ARCH_HUGE_PTE_UFFD_WP* macros) in the future. Signed-off-by: Peter Xu --- include/asm-generic/hugetlb.h | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/include/asm-generic/hugetlb.h b/include/asm-generic/hugetlb.h index 8e1e6244a89d..c45b9deb41ff 100644 --- a/include/asm-generic/hugetlb.h +++ b/include/asm-generic/hugetlb.h @@ -32,6 +32,21 @@ static inline pte_t huge_pte_modify(pte_t pte, pgprot_t newprot) return pte_modify(pte, newprot); } +static inline pte_t huge_pte_mkuffd_wp(pte_t pte) +{ + return pte_mkuffd_wp(pte); +} + +static inline pte_t huge_pte_clear_uffd_wp(pte_t pte) +{ + return pte_clear_uffd_wp(pte); +} + +static inline int huge_pte_uffd_wp(pte_t pte) +{ + return pte_uffd_wp(pte); +} + #ifndef __HAVE_ARCH_HUGE_PTE_CLEAR static inline void huge_pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long sz) From patchwork Thu May 27 20:23:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285289 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADAD5C4708A for ; Thu, 27 May 2021 20:23:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5D852613CC for ; Thu, 27 May 2021 20:23:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5D852613CC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 019DB6B007E; Thu, 27 May 2021 16:23:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F32356B0080; Thu, 27 May 2021 16:23:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DFA678D0001; Thu, 27 May 2021 16:23:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0070.hostedemail.com [216.40.44.70]) by kanga.kvack.org (Postfix) with ESMTP id AE71E6B007E for ; Thu, 27 May 2021 16:23:19 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 59758B7A0 for ; Thu, 27 May 2021 20:23:19 +0000 (UTC) X-FDA: 78188135718.12.B574267 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 47196C007748 for ; Thu, 27 May 2021 20:23:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622146998; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=C2AIp+Ymbm/rO+WssUczToBWrAYbIWlMSeka5+35yLo=; b=J2oaq7rtW0j5FOppWVh1SUCaQOhLDa9ldKx+wwl+kPAYrdYetTjbZCLgXHSCaNRlX6ySnw J/ciIxPgd94VVX5t5C+u4s2Np4xd+eEsIb1CXylDQcY5S9pjPNEvdfOKgLTl5vCaXFMAJ/ a1R+fODBqlagSq3C4eb/iODjay2HL1Y= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-228-mj00NJvUPJS1mu0V_pQwpg-1; Thu, 27 May 2021 16:23:16 -0400 X-MC-Unique: mj00NJvUPJS1mu0V_pQwpg-1 Received: by mail-qv1-f70.google.com with SMTP id d3-20020a0cdb030000b029021313001eeeso1008981qvk.15 for ; Thu, 27 May 2021 13:23:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=C2AIp+Ymbm/rO+WssUczToBWrAYbIWlMSeka5+35yLo=; b=sANyqjXAyYtMSH+PR7bP0c3BEAF0uV9IVKTEaQFMpc7q+J1LuOpBm2FE1c7KwKyUH6 zR0WEX7s+PPsgm4cuMyDyuf5TIJOUSf/EFCIgLw6xIB4Xl3uzjN6KtQeJAvVZMhOHDHC I5c0PcaL3x0cXUtU6HNnehRJqL+IkRcvHzaYKFfi5zwC9RM+D8mpwcXRzn3aG5H/KZWR 4lNWAHcyyj+ksDIlkJSWRphFlw4KE/JtINJ1VVpeehoJMRHuqAI8CYGtqTAslhMrOPpR HZP7YbWmDTnfqKzgmh34auqOOG9SNCrQ19+0NT69WSEfPp3lQPMU3SuWMi8DZ/ev4fqj IWiA== X-Gm-Message-State: AOAM530zlfPwuJUCigvzhPsttdeuKBReIyd1Q3zrSQlK4b3uxqim9ixQ UZQNA6eDwVPTXD7b65LDQiFocXlNzYrvORZeXrPjGNmA/eIqDLUjuHDlSneqquQXUDICIu2W1ez 71jZ+P5x5BsM= X-Received: by 2002:ac8:6c4d:: with SMTP id z13mr362289qtu.70.1622146995373; Thu, 27 May 2021 13:23:15 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxxoISek+Gk6rvKj0UcX69Zu2O92OlUW8NlblGfi2ZCDVyoeOnKbowkA5n1gTlEZ3HdJTxJVg== X-Received: by 2002:ac8:6c4d:: with SMTP id z13mr362267qtu.70.1622146995147; Thu, 27 May 2021 13:23:15 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id m15sm2005366qtn.47.2021.05.27.13.23.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:23:14 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Rapoport , peterx@redhat.com, Andrew Morton , Mike Kravetz , Jerome Glisse , Miaohe Lin , Nadav Amit , Hugh Dickins , Matthew Wilcox , Jason Gunthorpe , "Kirill A . Shutemov" , Andrea Arcangeli , Axel Rasmussen Subject: [PATCH v3 18/27] hugetlb/userfaultfd: Hook page faults for uffd write protection Date: Thu, 27 May 2021 16:23:11 -0400 Message-Id: <20210527202311.31937-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=J2oaq7rt; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf14.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com X-Stat-Signature: yr6fo9ugwqy7h9ozp1qd9dfbm6uq3za7 X-Rspamd-Queue-Id: 47196C007748 X-Rspamd-Server: rspam02 X-HE-Tag: 1622146990-551517 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hook up hugetlbfs_fault() with the capability to handle userfaultfd-wp faults. We do this slightly earlier than hugetlb_cow() so that we can avoid taking some extra locks that we definitely don't need. Reviewed-by: Mike Kravetz Signed-off-by: Peter Xu --- mm/hugetlb.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 69a4b551c157..4cbbffd50080 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4980,6 +4980,25 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, if (unlikely(!pte_same(entry, huge_ptep_get(ptep)))) goto out_ptl; + /* Handle userfault-wp first, before trying to lock more pages */ + if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(ptep)) && + (flags & FAULT_FLAG_WRITE) && !huge_pte_write(entry)) { + struct vm_fault vmf = { + .vma = vma, + .address = haddr, + .flags = flags, + }; + + spin_unlock(ptl); + if (pagecache_page) { + unlock_page(pagecache_page); + put_page(pagecache_page); + } + mutex_unlock(&hugetlb_fault_mutex_table[hash]); + i_mmap_unlock_read(mapping); + return handle_userfault(&vmf, VM_UFFD_WP); + } + /* * hugetlb_cow() requires page locks of pte_page(entry) and * pagecache_page, so here we need take the former one From patchwork Thu May 27 20:23:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285291 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 244FCC4708E for ; Thu, 27 May 2021 20:23:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BACF361360 for ; Thu, 27 May 2021 20:23:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BACF361360 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5BF8A6B0080; Thu, 27 May 2021 16:23:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5951D6B0081; Thu, 27 May 2021 16:23:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 436CF6B0082; Thu, 27 May 2021 16:23:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0019.hostedemail.com [216.40.44.19]) by kanga.kvack.org (Postfix) with ESMTP id 13F5E6B0080 for ; Thu, 27 May 2021 16:23:26 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id AD72A180286F3 for ; Thu, 27 May 2021 20:23:25 +0000 (UTC) X-FDA: 78188135970.18.596BD3D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf24.hostedemail.com (Postfix) with ESMTP id 5BE9BA0003AC for ; Thu, 27 May 2021 20:23:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622147004; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1GDjCOxfDPOEsRYzqON2M4AMdWZ8835TBDlBiZOtoT4=; b=irdkWhTjixc8IgnTXARliK4YD6yvjWyIpbb39uqjCBhpGOaapR7ANqqw38BFH5kRQGWK5g DH30JbubFxb2wtuqYnxLeihCxl+dMN7nHzaYYjKVABT3WOcBRgf1Pqt7ZhRmq7L1NwD60o 1syhNN6J9szjW+Rv2ui3DR0GL3my1qk= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-599-TI0fSb8-M7OzH3miqq8VWw-1; Thu, 27 May 2021 16:23:23 -0400 X-MC-Unique: TI0fSb8-M7OzH3miqq8VWw-1 Received: by mail-qk1-f200.google.com with SMTP id k125-20020a3788830000b02903a65618d46cso1366219qkd.3 for ; Thu, 27 May 2021 13:23:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1GDjCOxfDPOEsRYzqON2M4AMdWZ8835TBDlBiZOtoT4=; b=WSQJTzcapByCbVfgvpOj5QzO+prTX+flqI6ENzd6QOJdNFjn7J9RcVxbJe+IH9M/pn FcjfGDuXlK52+YSPvmJ0wFWjt2WTfDOd0B0rPABJTFcoBpNWTbVBw8DerujiM0eZQVNl Lgia/VX8toVxCO6w82KUo9ehFf8/cs4899Zlefe83wIBkGV2hY+ffiyEjrrAVHRKhJMz HbtMEOahJHUuAgjbyA1N6dezrJl1+dwxgK+yI38VhTS9CYxmtMA3tQPfiQV2ZNTft5Zp nu9ZZCrXLGQHKVBKv4n4OsGv0BWPpFe/A0M4yvoJ3OWqTtV1WL3RbbzHMpCNAIWEkhsI LNXw== X-Gm-Message-State: AOAM532eiFjmSJUwVki0A2ctH0CT1+NhcrsGUrJOjU8WWsMZVMcqCq0K 4O6u5QurUJu5Wf+kDcfYG/XiZyAj5XS7K8twhBB9EUHOLN4ap4I27KDEHOVdxClnAcCaEaWxHws ZFEfL3WFAMfU= X-Received: by 2002:a05:620a:248f:: with SMTP id i15mr310902qkn.239.1622146999689; Thu, 27 May 2021 13:23:19 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzt71gbe8C1zJT9R17fnlIm87KTerMisk37d4KUupzC0L7JGl/wu6GvNWmVhJL7cHOazzx7Tw== X-Received: by 2002:a05:620a:248f:: with SMTP id i15mr310872qkn.239.1622146999420; Thu, 27 May 2021 13:23:19 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id x11sm2057642qkh.129.2021.05.27.13.23.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:23:18 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Rapoport , peterx@redhat.com, Andrew Morton , Mike Kravetz , Jerome Glisse , Miaohe Lin , Nadav Amit , Hugh Dickins , Matthew Wilcox , Jason Gunthorpe , "Kirill A . Shutemov" , Andrea Arcangeli , Axel Rasmussen Subject: [PATCH v3 19/27] hugetlb/userfaultfd: Take care of UFFDIO_COPY_MODE_WP Date: Thu, 27 May 2021 16:23:15 -0400 Message-Id: <20210527202315.31993-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 5BE9BA0003AC Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=irdkWhTj; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf24.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com X-Rspamd-Server: rspam03 X-Stat-Signature: da19dp9i1ydrwbrrzankgud6crm3g6ud X-HE-Tag: 1622146997-327289 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Firstly, pass the wp_copy variable into hugetlb_mcopy_atomic_pte() thoughout the stack. Then, apply the UFFD_WP bit if UFFDIO_COPY_MODE_WP is with UFFDIO_COPY. Introduce huge_pte_mkuffd_wp() for it. Hugetlb pages are only managed by hugetlbfs, so we're safe even without setting dirty bit in the huge pte if the page is installed as read-only. However we'd better still keep the dirty bit set for a read-only UFFDIO_COPY pte (when UFFDIO_COPY_MODE_WP bit is set), not only to match what we do with shmem, but also because the page does contain dirty data that the kernel just copied from the userspace. Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 6 ++++-- mm/hugetlb.c | 22 +++++++++++++++++----- mm/userfaultfd.c | 12 ++++++++---- 3 files changed, 29 insertions(+), 11 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 7ef2b8c2ff41..d238a69bcbb3 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -155,7 +155,8 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, pte_t *dst_pte, unsigned long dst_addr, unsigned long src_addr, enum mcopy_atomic_mode mode, - struct page **pagep); + struct page **pagep, + bool wp_copy); #endif /* CONFIG_USERFAULTFD */ bool hugetlb_reserve_pages(struct inode *inode, long from, long to, struct vm_area_struct *vma, @@ -337,7 +338,8 @@ static inline int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, unsigned long dst_addr, unsigned long src_addr, enum mcopy_atomic_mode mode, - struct page **pagep) + struct page **pagep, + bool wp_copy) { BUG(); return 0; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4cbbffd50080..9bdcc208f5d9 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5062,7 +5062,8 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, unsigned long dst_addr, unsigned long src_addr, enum mcopy_atomic_mode mode, - struct page **pagep) + struct page **pagep, + bool wp_copy) { bool is_continue = (mode == MCOPY_ATOMIC_CONTINUE); struct hstate *h = hstate_vma(dst_vma); @@ -5203,17 +5204,28 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, hugepage_add_new_anon_rmap(page, dst_vma, dst_addr); } - /* For CONTINUE on a non-shared VMA, don't set VM_WRITE for CoW. */ - if (is_continue && !vm_shared) + /* + * For either: (1) CONTINUE on a non-shared VMA, or (2) UFFDIO_COPY + * with wp flag set, don't set pte write bit. + */ + if (wp_copy || (is_continue && !vm_shared)) writable = 0; else writable = dst_vma->vm_flags & VM_WRITE; _dst_pte = make_huge_pte(dst_vma, page, writable); - if (writable) - _dst_pte = huge_pte_mkdirty(_dst_pte); + /* + * Always mark UFFDIO_COPY page dirty; note that this may not be + * extremely important for hugetlbfs for now since swapping is not + * supported, but we should still be clear in that this page cannot be + * thrown away at will, even if write bit not set. + */ + _dst_pte = huge_pte_mkdirty(_dst_pte); _dst_pte = pte_mkyoung(_dst_pte); + if (wp_copy) + _dst_pte = huge_pte_mkuffd_wp(_dst_pte); + set_huge_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte); (void)huge_ptep_set_access_flags(dst_vma, dst_addr, dst_pte, _dst_pte, diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 147e86095070..424d0adc3f80 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -297,7 +297,8 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, unsigned long dst_start, unsigned long src_start, unsigned long len, - enum mcopy_atomic_mode mode) + enum mcopy_atomic_mode mode, + bool wp_copy) { int vm_alloc_shared = dst_vma->vm_flags & VM_SHARED; int vm_shared = dst_vma->vm_flags & VM_SHARED; @@ -394,7 +395,8 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, } err = hugetlb_mcopy_atomic_pte(dst_mm, dst_pte, dst_vma, - dst_addr, src_addr, mode, &page); + dst_addr, src_addr, mode, &page, + wp_copy); mutex_unlock(&hugetlb_fault_mutex_table[hash]); i_mmap_unlock_read(mapping); @@ -496,7 +498,8 @@ extern ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, unsigned long dst_start, unsigned long src_start, unsigned long len, - enum mcopy_atomic_mode mode); + enum mcopy_atomic_mode mode, + bool wp_copy); #endif /* CONFIG_HUGETLB_PAGE */ static __always_inline ssize_t mfill_atomic_pte(struct mm_struct *dst_mm, @@ -616,7 +619,8 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm, */ if (is_vm_hugetlb_page(dst_vma)) return __mcopy_atomic_hugetlb(dst_mm, dst_vma, dst_start, - src_start, len, mcopy_mode); + src_start, len, mcopy_mode, + wp_copy); if (!vma_is_anonymous(dst_vma) && !vma_is_shmem(dst_vma)) goto out_unlock; From patchwork Thu May 27 20:23:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285293 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A48BFC4707F for ; Thu, 27 May 2021 20:23:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 465CB613CC for ; Thu, 27 May 2021 20:23:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 465CB613CC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DAE5D6B0075; Thu, 27 May 2021 16:23:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D86E36B0081; Thu, 27 May 2021 16:23:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD7AF6B0082; Thu, 27 May 2021 16:23:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0081.hostedemail.com [216.40.44.81]) by kanga.kvack.org (Postfix) with ESMTP id 8A49A6B0081 for ; Thu, 27 May 2021 16:23:30 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 19698181AF5CA for ; Thu, 27 May 2021 20:23:30 +0000 (UTC) X-FDA: 78188136180.07.FD88CC2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 2423CE005F2A for ; Thu, 27 May 2021 20:23:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622147009; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WbZt0eoEWLox3U/frFubDdAmo5YNm3Rwmy7KTYyiQYM=; b=Ab5r7yGfLjsI3nuqjJGO6eRRgJGeQGVSIpnaB2Kc1xjPkhXIzjCurKDzg6SkaGzR9TWUKM ZcoPA+P4bvZDzleUI76xyWzJfFKKkWhOgcNUT9oz2a6dFAlhfUjXFG2bq+avhqqY0tyRLI qdizSD0ZAyaUFcEz3l1GqEz9MXuNKtw= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-163-rWw2ytkyOu-AjYshqr0N5w-1; Thu, 27 May 2021 16:23:25 -0400 X-MC-Unique: rWw2ytkyOu-AjYshqr0N5w-1 Received: by mail-qv1-f70.google.com with SMTP id fi6-20020a0562141a46b02901f064172b74so1049514qvb.3 for ; Thu, 27 May 2021 13:23:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WbZt0eoEWLox3U/frFubDdAmo5YNm3Rwmy7KTYyiQYM=; b=Q496BME/waUmHDqLJSUUuNnfwoiG7JvTol0Qqr+UDB5R4cSaAfxC9HZAyH0Xhc+rei wXSsJi74No+ur8QPkFchAri6n8hQU9qNiOFUw5EyFVQ6bJK/UsItCa7y622/E972VfM3 49/5tBSRGs5U5ODLl03vxAYnY+5jlYMG2KrWGM8TmEdivHKQNUF/pp2dwjVLSFjq7izu KcECXtSHzspRstW/D+1h2GdCa0QkbrHQaKszTnRAzyB0HtlDOwsS7A36ZwIzZ5EvH4bN 7CgTPhPxCwM70Mki+fs3YxxPGc5qAVVVAHXtLU0DUMLXfbehgHGysB1v3VkDXJ9Fcga7 IQPQ== X-Gm-Message-State: AOAM532ZyKK9wuQjsEOwf6xHTOZLK+L2fGocqHN6WHp8bU9b9XezCcCv 0EGj0XUNYVBBkNKyx1+oOS2Xu1SZVtRh4SSpvGFmpgVXbZGhzE/OLPotl6opA319zh12oXqHZa/ +MC9TRNOf+3U= X-Received: by 2002:a05:620a:8d6:: with SMTP id z22mr289787qkz.407.1622147004607; Thu, 27 May 2021 13:23:24 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz7EDf1VO1anNf/T5xNjyyn6/xaNm6K5kKhKB1cqb49ErCH+/+zpWA+AqfvzIh476npMMVMZA== X-Received: by 2002:a05:620a:8d6:: with SMTP id z22mr289770qkz.407.1622147004349; Thu, 27 May 2021 13:23:24 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id t25sm2011860qkt.62.2021.05.27.13.23.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:23:23 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Rapoport , peterx@redhat.com, Andrew Morton , Mike Kravetz , Jerome Glisse , Miaohe Lin , Nadav Amit , Hugh Dickins , Matthew Wilcox , Jason Gunthorpe , "Kirill A . Shutemov" , Andrea Arcangeli , Axel Rasmussen Subject: [PATCH v3 20/27] hugetlb/userfaultfd: Handle UFFDIO_WRITEPROTECT Date: Thu, 27 May 2021 16:23:20 -0400 Message-Id: <20210527202320.32048-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Ab5r7yGf; spf=none (imf21.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 2423CE005F2A X-Stat-Signature: zd8twup4kj5o748be6pz8h8sqg8muyjk X-HE-Tag: 1622146999-655364 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This starts from passing cp_flags into hugetlb_change_protection() so hugetlb will be able to handle MM_CP_UFFD_WP[_RESOLVE] requests. huge_pte_clear_uffd_wp() is introduced to handle the case where the UFFDIO_WRITEPROTECT is requested upon migrating huge page entries. Reviewed-by: Mike Kravetz Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 6 ++++-- mm/hugetlb.c | 13 ++++++++++++- mm/mprotect.c | 3 ++- mm/userfaultfd.c | 8 ++++++++ 4 files changed, 26 insertions(+), 4 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index d238a69bcbb3..3e4c5c64d867 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -204,7 +204,8 @@ struct page *follow_huge_pgd(struct mm_struct *mm, unsigned long address, int pmd_huge(pmd_t pmd); int pud_huge(pud_t pud); unsigned long hugetlb_change_protection(struct vm_area_struct *vma, - unsigned long address, unsigned long end, pgprot_t newprot); + unsigned long address, unsigned long end, pgprot_t newprot, + unsigned long cp_flags); bool is_hugetlb_entry_migration(pte_t pte); void hugetlb_unshare_all_pmds(struct vm_area_struct *vma); @@ -368,7 +369,8 @@ static inline void move_hugetlb_state(struct page *oldpage, static inline unsigned long hugetlb_change_protection( struct vm_area_struct *vma, unsigned long address, - unsigned long end, pgprot_t newprot) + unsigned long end, pgprot_t newprot, + unsigned long cp_flags) { return 0; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 9bdcc208f5d9..b101c3af3ab5 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5444,7 +5444,8 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, } unsigned long hugetlb_change_protection(struct vm_area_struct *vma, - unsigned long address, unsigned long end, pgprot_t newprot) + unsigned long address, unsigned long end, + pgprot_t newprot, unsigned long cp_flags) { struct mm_struct *mm = vma->vm_mm; unsigned long start = address; @@ -5454,6 +5455,8 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long pages = 0; bool shared_pmd = false; struct mmu_notifier_range range; + bool uffd_wp = cp_flags & MM_CP_UFFD_WP; + bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE; /* * In the case of shared PMDs, the area to flush could be beyond @@ -5495,6 +5498,10 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, entry = make_readable_migration_entry( swp_offset(entry)); newpte = swp_entry_to_pte(entry); + if (uffd_wp) + newpte = pte_swp_mkuffd_wp(newpte); + else if (uffd_wp_resolve) + newpte = pte_swp_clear_uffd_wp(newpte); set_huge_swap_pte_at(mm, address, ptep, newpte, huge_page_size(h)); pages++; @@ -5509,6 +5516,10 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, old_pte = huge_ptep_modify_prot_start(vma, address, ptep); pte = pte_mkhuge(huge_pte_modify(old_pte, newprot)); pte = arch_make_huge_pte(pte, shift, vma->vm_flags); + if (uffd_wp) + pte = huge_pte_mkuffd_wp(huge_pte_wrprotect(pte)); + else if (uffd_wp_resolve) + pte = huge_pte_clear_uffd_wp(pte); huge_ptep_modify_prot_commit(vma, address, ptep, old_pte, pte); pages++; } diff --git a/mm/mprotect.c b/mm/mprotect.c index 3fcb87b59696..96f4df023439 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -426,7 +426,8 @@ unsigned long change_protection(struct vm_area_struct *vma, unsigned long start, BUG_ON((cp_flags & MM_CP_UFFD_WP_ALL) == MM_CP_UFFD_WP_ALL); if (is_vm_hugetlb_page(vma)) - pages = hugetlb_change_protection(vma, start, end, newprot); + pages = hugetlb_change_protection(vma, start, end, newprot, + cp_flags); else pages = change_protection_range(vma, start, end, newprot, cp_flags); diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 424d0adc3f80..82c235f555b8 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -743,6 +743,7 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, unsigned long len, bool enable_wp, bool *mmap_changing) { struct vm_area_struct *dst_vma; + unsigned long page_mask; pgprot_t newprot; int err; @@ -779,6 +780,13 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, if (!vma_is_anonymous(dst_vma)) goto out_unlock; + if (is_vm_hugetlb_page(dst_vma)) { + err = -EINVAL; + page_mask = vma_kernel_pagesize(dst_vma) - 1; + if ((start & page_mask) || (len & page_mask)) + goto out_unlock; + } + if (enable_wp) newprot = vm_get_page_prot(dst_vma->vm_flags & ~(VM_WRITE)); else From patchwork Thu May 27 20:23:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285295 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0439AC4708B for ; Thu, 27 May 2021 20:23:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A7574613CC for ; Thu, 27 May 2021 20:23:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A7574613CC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 158846B0081; Thu, 27 May 2021 16:23:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 12B2B6B0082; Thu, 27 May 2021 16:23:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E25BD8D0001; Thu, 27 May 2021 16:23:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0129.hostedemail.com [216.40.44.129]) by kanga.kvack.org (Postfix) with ESMTP id A15E66B0081 for ; Thu, 27 May 2021 16:23:31 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 32BF18249980 for ; Thu, 27 May 2021 20:23:31 +0000 (UTC) X-FDA: 78188136222.12.4E8D18E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf12.hostedemail.com (Postfix) with ESMTP id AB13D2BC9 for ; Thu, 27 May 2021 20:23:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622147010; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Mr9CxfY0aflFHSlW/5vIhFmyytmHYg/MM4gViCf8fs4=; b=ULC89pmy+HlQKmDoZQZiEKGbEUpOgdkkfT0vrLnk7cExEGQ3CI2hRvULwxCZGX7VlXm+Vh iSYaBTmLXI7/bexZ7wk/G/dI81yVgZdXHcl8dGOi4/2StROQJZwuDkrYUHTaTizhXiQB56 Nc5tmV6wLJkO9TuqEbENHEUIqj12M5M= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-497-AbnRgSekNRuCi5khWDSPlQ-1; Thu, 27 May 2021 16:23:28 -0400 X-MC-Unique: AbnRgSekNRuCi5khWDSPlQ-1 Received: by mail-qk1-f199.google.com with SMTP id b203-20020a3767d40000b02903a6207bfef5so1376121qkc.1 for ; Thu, 27 May 2021 13:23:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Mr9CxfY0aflFHSlW/5vIhFmyytmHYg/MM4gViCf8fs4=; b=WbVIoNA9UBwD+VXimC9J00Gj6yvCs66u4EU0NkLhgU3ydlToQfMxYId4otPYEYp5YJ 795msfzTBa+jMEVd2u04JqVklRQpWLjw3jimd9LPXeHRx4LuEqc0ZbR2iczHt0R0P56X 36W12xHUjNUeOSZw1aO1l/rP4l9fNqzf4KAvoWkDysSN9e61j7fq8Uoo7wRLN0bJYVr9 lO/J5VPGXhArKwblePu1KLKs3vwm3s56p0/723HFNS0Srnw1Nyt56XzqENE7Mq+/L9la +Y9fF247Fpp1jcN6GmotObQIyttqoT0s22AdoYF8hSubPqnlTN/OzcZXZ8+GVEg6b3GY eSDw== X-Gm-Message-State: AOAM530SfNQ/cwKT3wNzcRJ5QnAWByvJnh6PimYuKzNLYE3OVR/tZEEY PU4K1QwJjapT44dPRq755FgyH2hkGb1jnR1cDPNM6OLrmDJgDcoey0NXDfYWclW4e14NXgdMLB1 W5wHDxRlMl+M= X-Received: by 2002:ac8:4a98:: with SMTP id l24mr352238qtq.378.1622147008355; Thu, 27 May 2021 13:23:28 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxRAIFI/HJUQmPZpJYQ5Br+Zpl1lpkfA2/NL21zRaxxTI5PdiJKqF+10vpJO8ROFiXz1aAxrA== X-Received: by 2002:ac8:4a98:: with SMTP id l24mr352203qtq.378.1622147008104; Thu, 27 May 2021 13:23:28 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id l197sm2069286qke.121.2021.05.27.13.23.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:23:27 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Rapoport , peterx@redhat.com, Andrew Morton , Mike Kravetz , Jerome Glisse , Miaohe Lin , Nadav Amit , Hugh Dickins , Matthew Wilcox , Jason Gunthorpe , "Kirill A . Shutemov" , Andrea Arcangeli , Axel Rasmussen Subject: [PATCH v3 21/27] mm/hugetlb: Introduce huge version of special swap pte helpers Date: Thu, 27 May 2021 16:23:24 -0400 Message-Id: <20210527202324.32104-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ULC89pmy; spf=none (imf12.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: AB13D2BC9 X-Stat-Signature: nrh5ggojieftudu8xm6zazfoq6eebxkm X-HE-Tag: 1622146999-226985 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is to let hugetlbfs be prepared to also recognize swap special ptes just like uffd-wp special swap ptes. Reviewed-by: Mike Kravetz Signed-off-by: Peter Xu --- mm/hugetlb.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index b101c3af3ab5..c64dfd0a9883 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -133,6 +133,26 @@ static inline bool subpool_is_free(struct hugepage_subpool *spool) return true; } +/* + * These are sister versions of is_swap_pte() and pte_has_swap_entry(). We + * need standalone ones because huge_pte_none() is handled differently from + * pte_none(). For more information, please refer to comments above + * is_swap_pte() and pte_has_swap_entry(). + * + * Here we directly reuse the pte level of swap special ptes, for example, the + * pte_swp_uffd_wp_special(). It just stands for a huge page rather than a + * small page for hugetlbfs pages. + */ +static inline bool is_huge_swap_pte(pte_t pte) +{ + return !huge_pte_none(pte) && !pte_present(pte); +} + +static inline bool huge_pte_has_swap_entry(pte_t pte) +{ + return is_huge_swap_pte(pte) && !is_swap_special_pte(pte); +} + static inline void unlock_or_release_subpool(struct hugepage_subpool *spool, unsigned long irq_flags) { @@ -4061,7 +4081,7 @@ bool is_hugetlb_entry_migration(pte_t pte) { swp_entry_t swp; - if (huge_pte_none(pte) || pte_present(pte)) + if (!huge_pte_has_swap_entry(pte)) return false; swp = pte_to_swp_entry(pte); if (is_migration_entry(swp)) @@ -4074,7 +4094,7 @@ static bool is_hugetlb_entry_hwpoisoned(pte_t pte) { swp_entry_t swp; - if (huge_pte_none(pte) || pte_present(pte)) + if (!huge_pte_has_swap_entry(pte)) return false; swp = pte_to_swp_entry(pte); if (is_hwpoison_entry(swp)) From patchwork Thu May 27 20:23:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285297 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C34C1C4708F for ; Thu, 27 May 2021 20:23:39 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 71C2B613E2 for ; Thu, 27 May 2021 20:23:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 71C2B613E2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 956C06B007E; Thu, 27 May 2021 16:23:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8FBF16B0083; Thu, 27 May 2021 16:23:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 383076B0085; Thu, 27 May 2021 16:23:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0064.hostedemail.com [216.40.44.64]) by kanga.kvack.org (Postfix) with ESMTP id E9DF06B007E for ; Thu, 27 May 2021 16:23:37 -0400 (EDT) Received: from smtpin40.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 87CC58249980 for ; Thu, 27 May 2021 20:23:37 +0000 (UTC) X-FDA: 78188136474.40.6B1B242 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 75293C00774F for ; Thu, 27 May 2021 20:23:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622147016; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oBNk7u3ivtT+37hOCTF6A87HkSdxXSKsCgL3KlLLZ5E=; b=NrjOfxutEfS/cS38IWBQiOqz0C4sJGsJoHXO/JI6dbjkll2mdu7PWt1+1CMtAr8glMGzuj QYTVXcbtNaF4SVzfjq+SGny2H8ku/Jm2uTNeQpGDzZHkLm6Hd41jBFKdgkk57E7SpBpEFo HdPLDETSirb4ncmNMfDWG3o6mg8jpdM= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-411-r-m7kmY5Oq-_DLKgOiRbkg-1; Thu, 27 May 2021 16:23:33 -0400 X-MC-Unique: r-m7kmY5Oq-_DLKgOiRbkg-1 Received: by mail-qv1-f72.google.com with SMTP id j14-20020a0cf30e0000b02902142ba51ef2so1022610qvl.10 for ; Thu, 27 May 2021 13:23:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=oBNk7u3ivtT+37hOCTF6A87HkSdxXSKsCgL3KlLLZ5E=; b=AP4NqQZb5yWIIrywexMKaSCBQbsEkRj2i4ndgQV3ds9uLFR/K60eemIcYD/q/5LVCQ 3bMeaUspeq0hAPakYjkMCefi8IemzvaKtb+3s+IP2MLEVUCfHRobRVn7kXBFhvh5G3Eo akbKGmO0KDIdT+lQIedTKqYClAyahIfAbsTeKxbt88RsgZ6JJ/9e5AqxvAAllhzGIzVl Rio7RbxVyOaHOI9dW+DWF9bi6gziDOKNHhkT0jcSpRUnrVaQYUkv/asYaeQv8Nm+TT8l zOgON5fWNUWioe6htWlwKRuL1j9uBbGQaCIm7O4GkFu7sOOLEMhTYZ+cNQqHfEi3FI6S hDyw== X-Gm-Message-State: AOAM530pFBVyvWdPIRc6jy5LuaBg/7+hpaeo2+xbbJgYEWgxW6bkdmo3 GBYkmsbHuzyyzWJJF/kwQE7w2as3309LTisY0//uy8j8vO5N+x/V092tJc4TZ7FTtJBTOWmilhk FTq1G6Zus2C8= X-Received: by 2002:ac8:4d19:: with SMTP id w25mr323901qtv.175.1622147012418; Thu, 27 May 2021 13:23:32 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwzFops8/UrKEf5HfwY3iEbhr5r+2t8bpk0m5+pQMv9Qf3pzrX5D7bqd/iDokt9cv0Uh9qJDA== X-Received: by 2002:ac8:4d19:: with SMTP id w25mr323885qtv.175.1622147012142; Thu, 27 May 2021 13:23:32 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id t17sm1981378qto.92.2021.05.27.13.23.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:23:31 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Rapoport , peterx@redhat.com, Andrew Morton , Mike Kravetz , Jerome Glisse , Miaohe Lin , Nadav Amit , Hugh Dickins , Matthew Wilcox , Jason Gunthorpe , "Kirill A . Shutemov" , Andrea Arcangeli , Axel Rasmussen Subject: [PATCH v3 22/27] hugetlb/userfaultfd: Handle uffd-wp special pte in hugetlb pf handler Date: Thu, 27 May 2021 16:23:28 -0400 Message-Id: <20210527202328.32154-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 75293C00774F Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NrjOfxut; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf14.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com X-Rspamd-Server: rspam03 X-Stat-Signature: d71twztsfbmasx3hcuaxgw5pwkjyom4a X-HE-Tag: 1622147008-425252 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Teach the hugetlb page fault code to understand uffd-wp special pte. For example, when seeing such a pte we need to convert any write fault into a read one (which is fake - we'll retry the write later if so). Meanwhile, for handle_userfault() we'll need to make sure we must wait for the special swap pte too just like a none pte. Note that we also need to teach UFFDIO_COPY about this special pte across the code path so that we can safely install a new page at this special pte as long as we know it's a stall entry. Signed-off-by: Peter Xu --- fs/userfaultfd.c | 5 ++++- mm/hugetlb.c | 26 ++++++++++++++++++++------ mm/userfaultfd.c | 5 ++++- 3 files changed, 28 insertions(+), 8 deletions(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 15031d6f1f17..6ef7b56760bf 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -245,8 +245,11 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx, /* * Lockless access: we're in a wait_event so it's ok if it * changes under us. + * + * Regarding uffd-wp special case, please refer to comments in + * userfaultfd_must_wait(). */ - if (huge_pte_none(pte)) + if (huge_pte_none(pte) || pte_swp_uffd_wp_special(pte)) ret = true; if (!huge_pte_write(pte) && (reason & VM_UFFD_WP)) ret = true; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c64dfd0a9883..a17d894312c0 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4707,7 +4707,8 @@ static inline vm_fault_t hugetlb_handle_userfault(struct vm_area_struct *vma, static vm_fault_t hugetlb_no_page(struct mm_struct *mm, struct vm_area_struct *vma, struct address_space *mapping, pgoff_t idx, - unsigned long address, pte_t *ptep, unsigned int flags) + unsigned long address, pte_t *ptep, + pte_t old_pte, unsigned int flags) { struct hstate *h = hstate_vma(vma); vm_fault_t ret = VM_FAULT_SIGBUS; @@ -4831,7 +4832,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, ptl = huge_pte_lock(h, mm, ptep); ret = 0; - if (!huge_pte_none(huge_ptep_get(ptep))) + if (!pte_same(huge_ptep_get(ptep), old_pte)) goto backout; if (anon_rmap) { @@ -4841,6 +4842,12 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, page_dup_rmap(page, true); new_pte = make_huge_pte(vma, page, ((vma->vm_flags & VM_WRITE) && (vma->vm_flags & VM_SHARED))); + /* + * If this pte was previously wr-protected, keep it wr-protected even + * if populated. + */ + if (unlikely(pte_swp_uffd_wp_special(old_pte))) + new_pte = huge_pte_wrprotect(huge_pte_mkuffd_wp(new_pte)); set_huge_pte_at(mm, haddr, ptep, new_pte); hugetlb_count_add(pages_per_huge_page(h), mm); @@ -4956,8 +4963,13 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, mutex_lock(&hugetlb_fault_mutex_table[hash]); entry = huge_ptep_get(ptep); - if (huge_pte_none(entry)) { - ret = hugetlb_no_page(mm, vma, mapping, idx, address, ptep, flags); + /* + * uffd-wp-special should be handled merely the same as pte none + * because it's basically a none pte with a special marker + */ + if (huge_pte_none(entry) || pte_swp_uffd_wp_special(entry)) { + ret = hugetlb_no_page(mm, vma, mapping, idx, address, ptep, + entry, flags); goto out_mutex; } @@ -5091,7 +5103,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, pgoff_t idx = vma_hugecache_offset(h, dst_vma, dst_addr); unsigned long size; int vm_shared = dst_vma->vm_flags & VM_SHARED; - pte_t _dst_pte; + pte_t _dst_pte, cur_pte; spinlock_t *ptl; int ret = -ENOMEM; struct page *page; @@ -5213,8 +5225,10 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, if (idx >= size) goto out_release_unlock; + cur_pte = huge_ptep_get(dst_pte); ret = -EEXIST; - if (!huge_pte_none(huge_ptep_get(dst_pte))) + /* Please refer to shmem_mfill_atomic_pte() for uffd-wp special case */ + if (!huge_pte_none(cur_pte) && !pte_swp_uffd_wp_special(cur_pte)) goto out_release_unlock; if (vm_shared) { diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 82c235f555b8..af79f3d3a001 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -364,6 +364,8 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, } while (src_addr < src_start + len) { + pte_t pteval; + BUG_ON(dst_addr >= dst_start + len); /* @@ -386,8 +388,9 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, goto out_unlock; } + pteval = huge_ptep_get(dst_pte); if (mode != MCOPY_ATOMIC_CONTINUE && - !huge_pte_none(huge_ptep_get(dst_pte))) { + !huge_pte_none(pteval) && !pte_swp_uffd_wp_special(pteval)) { err = -EEXIST; mutex_unlock(&hugetlb_fault_mutex_table[hash]); i_mmap_unlock_read(mapping); From patchwork Thu May 27 20:23:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F95FC4708D for ; Thu, 27 May 2021 20:23:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0CB23613D4 for ; Thu, 27 May 2021 20:23:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0CB23613D4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 64AF26B0085; Thu, 27 May 2021 16:23:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 606FD6B0087; Thu, 27 May 2021 16:23:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0498A6B0088; Thu, 27 May 2021 16:23:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0098.hostedemail.com [216.40.44.98]) by kanga.kvack.org (Postfix) with ESMTP id AE2BF6B0085 for ; Thu, 27 May 2021 16:23:40 -0400 (EDT) Received: from smtpin31.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 5021218017376 for ; Thu, 27 May 2021 20:23:40 +0000 (UTC) X-FDA: 78188136600.31.1B0F7DF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 0E8E5C00775A for ; Thu, 27 May 2021 20:23:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622147019; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ibDzUgI+Vk9OcaOLv+5pNYdmMu7OVveyljm4h1/HCNE=; b=GOlZz/GtdNM/GGCwsRSH6iLgzJ5M5vZO5aC6yaSHBE6hu8W510uWtyniKgN8p8+IIX+k5h wGez7T6KOgBYc/kl5px7nzfJ9GhhM+giHUsHcy9Rl9TGQqNt3WoQ6gYmjlIhBH+zpqWn8f aLtDq6H/NxuW803KBx2PW2GiU9xtpy4= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-281-tgNMt0XHOZyKwY4WELWkyQ-1; Thu, 27 May 2021 16:23:37 -0400 X-MC-Unique: tgNMt0XHOZyKwY4WELWkyQ-1 Received: by mail-qv1-f72.google.com with SMTP id w14-20020a056214012eb02901f3a4388530so997882qvs.17 for ; Thu, 27 May 2021 13:23:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ibDzUgI+Vk9OcaOLv+5pNYdmMu7OVveyljm4h1/HCNE=; b=A0I4h3uskvzO4/NNnwiGMcxUDGx5Q5oNDA19NtX33KlgVhSWRSezaeAhUyldB8rKsd totcRtmkRWwx+P++RoeKhPFbR2seW0MKC+TBrz26oMeTcT1n0PIbdI4oOI7o7RN1/soA 6ktQO89H2RYyvSWVC0cBgXiVCX11d/jDmO+OFCcU6Z4+wrc4jPDAV7aZ1tS7CPU6RPGp 4V2TtjzyNWWFTFUJmhJYq4BPRyowSgeKZulEFAFAXjHz9DdT9Vi5TaOOUosbVnqQpxP7 aayHEd1Qw+HX2d//aZzZH7/cdQW6S7ioqTiZfLWfbU3d8+qnO30OTD/obSyQmUvwmMKb L7xA== X-Gm-Message-State: AOAM532Ct5TuF5+0s7CNDCNx+zGuhpYB0Y4tDAG7vVLzofdqeeSub7e1 P8sJ0HsoP1hz4y9i0z6I7lfKwsCTu2xEBJnQYzWJhbRVu+0X0EJuZztlrDX/c+D1YRbARA3nC0D fVrrKKOIyIds= X-Received: by 2002:ac8:4319:: with SMTP id z25mr368489qtm.262.1622147016869; Thu, 27 May 2021 13:23:36 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzRfnnHZwDBhN/ZaRlD0zP/AETW62hdvxMzoXmXTSL4dORe6PtNpJSYMnRHeAUI2t6h/U4u8Q== X-Received: by 2002:ac8:4319:: with SMTP id z25mr368460qtm.262.1622147016558; Thu, 27 May 2021 13:23:36 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id a27sm2057437qtn.97.2021.05.27.13.23.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:23:35 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Rapoport , peterx@redhat.com, Andrew Morton , Mike Kravetz , Jerome Glisse , Miaohe Lin , Nadav Amit , Hugh Dickins , Matthew Wilcox , Jason Gunthorpe , "Kirill A . Shutemov" , Andrea Arcangeli , Axel Rasmussen Subject: [PATCH v3 23/27] hugetlb/userfaultfd: Allow wr-protect none ptes Date: Thu, 27 May 2021 16:23:32 -0400 Message-Id: <20210527202332.32205-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="GOlZz/Gt"; spf=none (imf06.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 0E8E5C00775A X-Stat-Signature: eo6ta3st8sfm1h9mra76yqqw6mtqttsu X-HE-Tag: 1622147011-343226 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Teach hugetlbfs code to wr-protect none ptes just in case the page cache existed for that pte. Meanwhile we also need to be able to recognize a uffd-wp marker pte and remove it for uffd_wp_resolve. Since at it, introduce a variable "psize" to replace all references to the huge page size fetcher. Reviewed-by: Mike Kravetz Signed-off-by: Peter Xu --- mm/hugetlb.c | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a17d894312c0..c4dd0c531bb5 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5486,7 +5486,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, pte_t *ptep; pte_t pte; struct hstate *h = hstate_vma(vma); - unsigned long pages = 0; + unsigned long pages = 0, psize = huge_page_size(h); bool shared_pmd = false; struct mmu_notifier_range range; bool uffd_wp = cp_flags & MM_CP_UFFD_WP; @@ -5506,13 +5506,19 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, mmu_notifier_invalidate_range_start(&range); i_mmap_lock_write(vma->vm_file->f_mapping); - for (; address < end; address += huge_page_size(h)) { + for (; address < end; address += psize) { spinlock_t *ptl; - ptep = huge_pte_offset(mm, address, huge_page_size(h)); + ptep = huge_pte_offset(mm, address, psize); if (!ptep) continue; ptl = huge_pte_lock(h, mm, ptep); if (huge_pmd_unshare(mm, vma, &address, ptep)) { + /* + * When uffd-wp is enabled on the vma, unshare + * shouldn't happen at all. Warn about it if it + * happened due to some reason. + */ + WARN_ON_ONCE(uffd_wp || uffd_wp_resolve); pages++; spin_unlock(ptl); shared_pmd = true; @@ -5537,12 +5543,21 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, else if (uffd_wp_resolve) newpte = pte_swp_clear_uffd_wp(newpte); set_huge_swap_pte_at(mm, address, ptep, - newpte, huge_page_size(h)); + newpte, psize); pages++; } spin_unlock(ptl); continue; } + if (unlikely(is_swap_special_pte(pte))) { + WARN_ON_ONCE(!pte_swp_uffd_wp_special(pte)); + /* + * This is changing a non-present pte into a none pte, + * no need for huge_ptep_modify_prot_start/commit(). + */ + if (uffd_wp_resolve) + huge_pte_clear(mm, address, ptep, psize); + } if (!huge_pte_none(pte)) { pte_t old_pte; unsigned int shift = huge_page_shift(hstate_vma(vma)); @@ -5556,6 +5571,12 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, pte = huge_pte_clear_uffd_wp(pte); huge_ptep_modify_prot_commit(vma, address, ptep, old_pte, pte); pages++; + } else { + /* None pte */ + if (unlikely(uffd_wp)) + /* Safe to modify directly (none->non-present). */ + set_huge_pte_at(mm, address, ptep, + pte_swp_mkuffd_wp_special(vma)); } spin_unlock(ptl); } From patchwork Thu May 27 20:23:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285303 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BCBEC47090 for ; Thu, 27 May 2021 20:23:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 06B8161360 for ; Thu, 27 May 2021 20:23:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 06B8161360 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A03856B0088; Thu, 27 May 2021 16:23:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B4B06B0089; Thu, 27 May 2021 16:23:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 82C906B008A; Thu, 27 May 2021 16:23:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0247.hostedemail.com [216.40.44.247]) by kanga.kvack.org (Postfix) with ESMTP id 4C7B86B0088 for ; Thu, 27 May 2021 16:23:44 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id E010F18047612 for ; Thu, 27 May 2021 20:23:43 +0000 (UTC) X-FDA: 78188136726.15.35F1980 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 907A3C007749 for ; Thu, 27 May 2021 20:23:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622147023; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CB/ZZ041zggBCCBv7NLw8xpkW7+qlOpR2v+yvM7l654=; b=HT5nM+tM9tdQfYBWwbash98RM4M4f4ZuJrGrfS3sO4Nea/dhNHpL2E+oee0Ip741lUW2xl L9mzxM3pNgE1U1QgdaD8J5+4yCouPihWD1V4jOax+GycMegGaPanfgqxBMbhNIJ8t0dADA zc+Qlxmy2mJCQ8q8juS0ziU1hsI16cA= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-209-xoeF608cNWSeM7ymu4yOVg-1; Thu, 27 May 2021 16:23:41 -0400 X-MC-Unique: xoeF608cNWSeM7ymu4yOVg-1 Received: by mail-qv1-f71.google.com with SMTP id k12-20020a0cfd6c0000b029020df9543019so1010782qvs.14 for ; Thu, 27 May 2021 13:23:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CB/ZZ041zggBCCBv7NLw8xpkW7+qlOpR2v+yvM7l654=; b=Pgp15B1rLOUfxcfu2gU0O/qGSjN9rZZeVd0pJixwJDkbe3F/YSTIsZux2XABlBqSl0 CHVI4dB7OzBr1RO5bB64zMQHVmR0oGWEewA5iUEsWCP9JGcDXX9FsAmuaIJMuen4n4Zp 4fNtuPCfqJ1fOyXVnVFHL60RCQbatUzoXgh0AuT9m6AAdqzCcriTCKRdKnWprKxGdXmG /G4y10kwLNtkFIhmJkCaTFb2MsqP8+uakokzkHB2U20NHpxytfnWtTYIkPueQ4SnGhlf E7/bDntMcuinyPLnvSLDbm3fCZoMkiWIed/fB6wUgKuMpkC7r04CGGQs8x7dfhonngEn fHQA== X-Gm-Message-State: AOAM532/yH8QlUDN27arlt8TGnDZjHbg6RIuFzp22YQePQdN4Ol8MZSB JUtoSBEw/mxZ9TuWwh0O5YCLjqd8Lwet+AfFh4qHSViirIQiZMw+GDIRtSZ/E13Fiox3JIM7vT1 GI/jVxWn4FKg= X-Received: by 2002:ac8:5a8f:: with SMTP id c15mr354817qtc.162.1622147020665; Thu, 27 May 2021 13:23:40 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwPUX4VOylyMwlokkX/25yMH5dG/jiMR6d/tM2BGaCOW3lGyhn9A9qZLMyhTc0YE/SXfOxZLQ== X-Received: by 2002:ac8:5a8f:: with SMTP id c15mr354789qtc.162.1622147020352; Thu, 27 May 2021 13:23:40 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id j1sm1973305qtn.83.2021.05.27.13.23.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:23:39 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Rapoport , peterx@redhat.com, Andrew Morton , Mike Kravetz , Jerome Glisse , Miaohe Lin , Nadav Amit , Hugh Dickins , Matthew Wilcox , Jason Gunthorpe , "Kirill A . Shutemov" , Andrea Arcangeli , Axel Rasmussen Subject: [PATCH v3 24/27] hugetlb/userfaultfd: Only drop uffd-wp special pte if required Date: Thu, 27 May 2021 16:23:37 -0400 Message-Id: <20210527202337.32256-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HT5nM+tM; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf06.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 907A3C007749 X-Stat-Signature: dkq5w8bkqioaa3iy14zz5j4an4hmcxhb X-HE-Tag: 1622147015-510882 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As with shmem uffd-wp special ptes, only drop the uffd-wp special swap pte if unmapping an entire vma or synchronized such that faults can not race with the unmap operation. This requires passing zap_flags all the way to the lowest level hugetlb unmap routine: __unmap_hugepage_range. In general, unmap calls originated in hugetlbfs code will pass the ZAP_FLAG_DROP_FILE_UFFD_WP flag as synchronization is in place to prevent faults. The exception is hole punch which will first unmap without any synchronization. Later when hole punch actually removes the page from the file, it will check to see if there was a subsequent fault and if so take the hugetlb fault mutex while unmapping again. This second unmap will pass in ZAP_FLAG_DROP_FILE_UFFD_WP. The core justification of "whether to apply ZAP_FLAG_DROP_FILE_UFFD_WP flag when unmap a hugetlb range" is (IMHO): we should never reach a state when a page fault could errornously fault in a page-cache page that was wr-protected to be writable, even in an extremely short period. That could happen if e.g. we pass ZAP_FLAG_DROP_FILE_UFFD_WP in hugetlbfs_punch_hole() when calling hugetlb_vmdelete_list(), because if a page fault triggers after that call and before the remove_inode_hugepages() right after it, the page cache can be mapped writable again in the small window, which can cause data corruption. Reviewed-by: Mike Kravetz Signed-off-by: Peter Xu --- fs/hugetlbfs/inode.c | 15 +++++++++------ include/linux/hugetlb.h | 8 +++++--- mm/hugetlb.c | 27 +++++++++++++++++++++------ mm/memory.c | 5 ++++- 4 files changed, 39 insertions(+), 16 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 55efd3dd04f6..b917fb4c670e 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -404,7 +404,8 @@ static void remove_huge_page(struct page *page) } static void -hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end) +hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end, + unsigned long zap_flags) { struct vm_area_struct *vma; @@ -437,7 +438,7 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end) } unmap_hugepage_range(vma, vma->vm_start + v_offset, v_end, - NULL); + NULL, zap_flags); } } @@ -515,7 +516,8 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart, mutex_lock(&hugetlb_fault_mutex_table[hash]); hugetlb_vmdelete_list(&mapping->i_mmap, index * pages_per_huge_page(h), - (index + 1) * pages_per_huge_page(h)); + (index + 1) * pages_per_huge_page(h), + ZAP_FLAG_DROP_FILE_UFFD_WP); i_mmap_unlock_write(mapping); } @@ -581,7 +583,8 @@ static void hugetlb_vmtruncate(struct inode *inode, loff_t offset) i_mmap_lock_write(mapping); i_size_write(inode, offset); if (!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root)) - hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0); + hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0, + ZAP_FLAG_DROP_FILE_UFFD_WP); i_mmap_unlock_write(mapping); remove_inode_hugepages(inode, offset, LLONG_MAX); } @@ -614,8 +617,8 @@ static long hugetlbfs_punch_hole(struct inode *inode, loff_t offset, loff_t len) i_mmap_lock_write(mapping); if (!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root)) hugetlb_vmdelete_list(&mapping->i_mmap, - hole_start >> PAGE_SHIFT, - hole_end >> PAGE_SHIFT); + hole_start >> PAGE_SHIFT, + hole_end >> PAGE_SHIFT, 0); i_mmap_unlock_write(mapping); remove_inode_hugepages(inode, hole_start, hole_end); inode_unlock(inode); diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 3e4c5c64d867..d3e8b3b38ded 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -138,11 +138,12 @@ long follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *, unsigned long *, unsigned long *, long, unsigned int, int *); void unmap_hugepage_range(struct vm_area_struct *, - unsigned long, unsigned long, struct page *); + unsigned long, unsigned long, struct page *, + unsigned long); void __unmap_hugepage_range_final(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, unsigned long end, - struct page *ref_page); + struct page *ref_page, unsigned long zap_flags); void hugetlb_report_meminfo(struct seq_file *); int hugetlb_report_node_meminfo(char *buf, int len, int nid); void hugetlb_show_meminfo(void); @@ -377,7 +378,8 @@ static inline unsigned long hugetlb_change_protection( static inline void __unmap_hugepage_range_final(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, - unsigned long end, struct page *ref_page) + unsigned long end, struct page *ref_page, + unsigned long zap_flags) { BUG(); } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c4dd0c531bb5..78675158911c 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4274,7 +4274,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, unsigned long end, - struct page *ref_page) + struct page *ref_page, unsigned long zap_flags) { struct mm_struct *mm = vma->vm_mm; unsigned long address; @@ -4326,6 +4326,19 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, continue; } + if (unlikely(is_swap_special_pte(pte))) { + WARN_ON_ONCE(!pte_swp_uffd_wp_special(pte)); + /* + * Only drop the special swap uffd-wp pte if + * e.g. unmapping a vma or punching a hole (with proper + * lock held so that concurrent page fault won't happen). + */ + if (zap_flags & ZAP_FLAG_DROP_FILE_UFFD_WP) + huge_pte_clear(mm, address, ptep, sz); + spin_unlock(ptl); + continue; + } + /* * Migrating hugepage or HWPoisoned hugepage is already * unmapped and its refcount is dropped, so just clear pte here. @@ -4377,9 +4390,10 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, void __unmap_hugepage_range_final(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, - unsigned long end, struct page *ref_page) + unsigned long end, struct page *ref_page, + unsigned long zap_flags) { - __unmap_hugepage_range(tlb, vma, start, end, ref_page); + __unmap_hugepage_range(tlb, vma, start, end, ref_page, zap_flags); /* * Clear this flag so that x86's huge_pmd_share page_table_shareable @@ -4395,12 +4409,13 @@ void __unmap_hugepage_range_final(struct mmu_gather *tlb, } void unmap_hugepage_range(struct vm_area_struct *vma, unsigned long start, - unsigned long end, struct page *ref_page) + unsigned long end, struct page *ref_page, + unsigned long zap_flags) { struct mmu_gather tlb; tlb_gather_mmu(&tlb, vma->vm_mm); - __unmap_hugepage_range(&tlb, vma, start, end, ref_page); + __unmap_hugepage_range(&tlb, vma, start, end, ref_page, zap_flags); tlb_finish_mmu(&tlb); } @@ -4455,7 +4470,7 @@ static void unmap_ref_private(struct mm_struct *mm, struct vm_area_struct *vma, */ if (!is_vma_resv_set(iter_vma, HPAGE_RESV_OWNER)) unmap_hugepage_range(iter_vma, address, - address + huge_page_size(h), page); + address + huge_page_size(h), page, 0); } i_mmap_unlock_write(mapping); } diff --git a/mm/memory.c b/mm/memory.c index 8372b212993a..4427f48e446d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1607,8 +1607,11 @@ static void unmap_single_vma(struct mmu_gather *tlb, * safe to do nothing in this case. */ if (vma->vm_file) { + unsigned long zap_flags = details ? + details->zap_flags : 0; i_mmap_lock_write(vma->vm_file->f_mapping); - __unmap_hugepage_range_final(tlb, vma, start, end, NULL); + __unmap_hugepage_range_final(tlb, vma, start, end, + NULL, zap_flags); i_mmap_unlock_write(vma->vm_file->f_mapping); } } else From patchwork Thu May 27 20:23:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285307 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3506C47092 for ; Thu, 27 May 2021 20:23:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 93993613CC for ; Thu, 27 May 2021 20:23:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 93993613CC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 34DEF6B0075; Thu, 27 May 2021 16:23:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 323AE6B008A; Thu, 27 May 2021 16:23:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 19D4D6B008C; Thu, 27 May 2021 16:23:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0178.hostedemail.com [216.40.44.178]) by kanga.kvack.org (Postfix) with ESMTP id DF76C6B0075 for ; Thu, 27 May 2021 16:23:47 -0400 (EDT) Received: from smtpin31.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 88F23BBEB for ; Thu, 27 May 2021 20:23:47 +0000 (UTC) X-FDA: 78188136894.31.539D49C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 3243AA0003B1 for ; Thu, 27 May 2021 20:23:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622147026; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d+qr2gkGCf8UGCRyFBArhsHy6tP5+E2Jys4dwvbHGGg=; b=TBPKnwAuLTflpfUgu8K6NcXMn0d+Z7soPJnA21G8nOgEGcObbIIYwKArKhJV+Vu8S+Zezq /TIHjWql10wBe0+1DhNmycpWlsjMjcDVaKaY37OkARZ6XduDNywiKAsAZKBlQan4TBIteI 0/TnHWcu80ZDNb4/NsUKqWy1Q3AAob8= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-206-_cJHQhcGMZuJpMIeJESsBA-1; Thu, 27 May 2021 16:23:45 -0400 X-MC-Unique: _cJHQhcGMZuJpMIeJESsBA-1 Received: by mail-qt1-f199.google.com with SMTP id g21-20020ac870d50000b0290230013e97e6so842082qtp.21 for ; Thu, 27 May 2021 13:23:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=d+qr2gkGCf8UGCRyFBArhsHy6tP5+E2Jys4dwvbHGGg=; b=n6DCAhxFApKoaBm4Kh2SiveRr56vPCyQDvQGKKTQfr7bxC6t4esgxcpi2kFhmIjIHm z+FiX5D5eOFK2AmaGa+9AJd6DdNnEfyLb/qPBilKVLAktcHSiXGC77QbPmcZl4Lb71gs F3S0wmkLcjmX+fmRsr3hBgNcN7gbBlIhE8FZPs8Uex5fInPN0ItEgBP6n+aYgsO+GCJw MVmZwzTH8yGK9YopxBTYEQE/82GAuuKSzLjnlS5+vBR4hmC3eJ1kfwlWJeJNBMjza8ga 2Hjrof0RgZ9oV+noySg0Q3CK1GMKVpGZIyz11LhBRaWJUwdalQ+B5rctQrnOd5tH80NV Ct7w== X-Gm-Message-State: AOAM530vIO2W7jcJ6aJPbp5pvJhDGxKQh+MxBrBpWYUyNxEEhAU3emhg ggE98zImIiRVp7BA09f1HplgZkJsN64S2mlSHW65MNqvD14iRnBPv8LkD3CjzUHGeAhzRoh/xy5 4G7y9TiYHMDg= X-Received: by 2002:a05:620a:751:: with SMTP id i17mr322481qki.87.1622147024828; Thu, 27 May 2021 13:23:44 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxZVwpfNFKbYL8so0SPHDLq2pNae/ct+JgV8mbINE0J29tIc1QVdSRi+fw1P99ktJh8UsVEWg== X-Received: by 2002:a05:620a:751:: with SMTP id i17mr322450qki.87.1622147024615; Thu, 27 May 2021 13:23:44 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id 2sm2237729qtr.64.2021.05.27.13.23.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:23:44 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Rapoport , peterx@redhat.com, Andrew Morton , Mike Kravetz , Jerome Glisse , Miaohe Lin , Nadav Amit , Hugh Dickins , Matthew Wilcox , Jason Gunthorpe , "Kirill A . Shutemov" , Andrea Arcangeli , Axel Rasmussen Subject: [PATCH v3 25/27] mm/pagemap: Recognize uffd-wp bit for shmem/hugetlbfs Date: Thu, 27 May 2021 16:23:40 -0400 Message-Id: <20210527202340.32306-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TBPKnwAu; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf23.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com X-Stat-Signature: 5hcz455yaczm4stqoad1ajfobkqjw1mh X-Rspamd-Queue-Id: 3243AA0003B1 X-Rspamd-Server: rspam02 X-HE-Tag: 1622147020-364503 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This requires the pagemap code to be able to recognize the newly introduced swap special pte for uffd-wp, meanwhile the general case for hugetlb that we recently start to support. It should make pagemap uffd-wp support complete. Signed-off-by: Peter Xu --- fs/proc/task_mmu.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 9c5af77b5290..988e29fa1f00 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1389,6 +1389,8 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm, flags |= PM_SWAP; if (is_pfn_swap_entry(entry)) page = pfn_swap_entry_to_page(entry); + } else if (pte_swp_uffd_wp_special(pte)) { + flags |= PM_UFFD_WP; } if (page && !PageAnon(page)) @@ -1522,10 +1524,15 @@ static int pagemap_hugetlb_range(pte_t *ptep, unsigned long hmask, if (page_mapcount(page) == 1) flags |= PM_MMAP_EXCLUSIVE; + if (huge_pte_uffd_wp(pte)) + flags |= PM_UFFD_WP; + flags |= PM_PRESENT; if (pm->show_pfn) frame = pte_pfn(pte) + ((addr & ~hmask) >> PAGE_SHIFT); + } else if (pte_swp_uffd_wp_special(pte)) { + flags |= PM_UFFD_WP; } for (; addr != end; addr += PAGE_SIZE) { From patchwork Thu May 27 20:23:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3403C4708F for ; Thu, 27 May 2021 20:23:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9B17D613CC for ; Thu, 27 May 2021 20:23:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9B17D613CC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id ADAF56B008C; Thu, 27 May 2021 16:23:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A82A26B0092; Thu, 27 May 2021 16:23:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 52FA16B0093; Thu, 27 May 2021 16:23:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 121336B008C for ; Thu, 27 May 2021 16:23:52 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id A622F8249980 for ; Thu, 27 May 2021 20:23:51 +0000 (UTC) X-FDA: 78188137062.24.4049ECD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf14.hostedemail.com (Postfix) with ESMTP id A9449C005A0C for ; Thu, 27 May 2021 20:23:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622147031; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ngMPi0v8szsDymkLXEE1yzQNi7Qk4LQoecvqvdNIG9I=; b=hHbbqik175LQAgePk54NoidkSJ1sngAKJwbo5R+xW3iOMYTwP7D6oQQXoNs2WY0spdN94T 245YbUOpvxpCRp3XpqYFRG2ioNLTqQgOo1u0lepNG4YTXod6ny8pPomZL+SHSOfdNKCA4R 7W1g2HHzdY11kQqfTY4Fih17pD2iCYg= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-120-AHipoVJMO5C5KvaPGXmmJQ-1; Thu, 27 May 2021 16:23:49 -0400 X-MC-Unique: AHipoVJMO5C5KvaPGXmmJQ-1 Received: by mail-qt1-f200.google.com with SMTP id q3-20020a05622a0303b02902390ac8c906so868034qtw.11 for ; Thu, 27 May 2021 13:23:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ngMPi0v8szsDymkLXEE1yzQNi7Qk4LQoecvqvdNIG9I=; b=beL3c6iBDDuZsUHwyD9pKfYvvU/iJ5vLuuptPy+G0YQ5XMV8+lqOsb5FkOZb8nDdap gr39THjIbN71GjIbT0ecJnvnV3clTznrvy4Fxs314vEY6g4b88YX3wMtDw0rHQPYmR9f xoFmlIz4jFrAHIVMXt+3uehvsXWr0/RVJQAiWuKl1+pNJBj1SADUjXV02PdCBrCpFDvi F3Qcin6TViVJWb2gVxlABl7oyDNu/yeQ8rWeAUlzUYh3GpiFtjBgFDGGfFzugYAuLChH FWGrpSvbD5jM6KPoDLMMcoWODKkx9DdLZBqtiOKIWJs6f7g1KdAftSOPrlyzwfwSRJWD EIAw== X-Gm-Message-State: AOAM532zaHonZPeDUXg3tpETVwlDSxj6fg/Jt08aFxp0M5vs35iTiKZ0 lPuCggcCvjJo1ovZizKuo9LgLcz16ESnYC42ZUxM5uBbU2dNhdzKdT1kzz+d3D5l5GDrvb/8gN4 Vw5aaolegNAE= X-Received: by 2002:ad4:4e91:: with SMTP id dy17mr330673qvb.55.1622147028610; Thu, 27 May 2021 13:23:48 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyxERXmrLVdBH2ktYPjs4f1N04FyCnVjMWl8pzE8ly931umyxfVaq9veVsqvYipuRhEbaHuyA== X-Received: by 2002:ad4:4e91:: with SMTP id dy17mr330647qvb.55.1622147028362; Thu, 27 May 2021 13:23:48 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id q7sm2094142qki.17.2021.05.27.13.23.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:23:47 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Rapoport , peterx@redhat.com, Andrew Morton , Mike Kravetz , Jerome Glisse , Miaohe Lin , Nadav Amit , Hugh Dickins , Matthew Wilcox , Jason Gunthorpe , "Kirill A . Shutemov" , Andrea Arcangeli , Axel Rasmussen Subject: [PATCH v3 26/27] mm/userfaultfd: Enable write protection for shmem & hugetlbfs Date: Thu, 27 May 2021 16:23:45 -0400 Message-Id: <20210527202345.32356-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hHbbqik1; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf14.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: A9449C005A0C X-Stat-Signature: mbkiwpbyynnrg3e4xeiu987p7gp6ds8g X-HE-Tag: 1622147022-393801 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We've had all the necessary changes ready for both shmem and hugetlbfs. Turn on all the shmem/hugetlbfs switches for userfaultfd-wp. We can expand UFFD_API_RANGE_IOCTLS_BASIC with _UFFDIO_WRITEPROTECT too because all existing types now support write protection mode. Since vma_can_userfault() will be used elsewhere, move into userfaultfd_k.h. Signed-off-by: Peter Xu --- fs/userfaultfd.c | 18 ------------------ include/linux/userfaultfd_k.h | 12 ++++++++++++ include/uapi/linux/userfaultfd.h | 10 ++++++++-- mm/userfaultfd.c | 9 +++------ 4 files changed, 23 insertions(+), 26 deletions(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 6ef7b56760bf..140bab07935e 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -1275,24 +1275,6 @@ static __always_inline int validate_range(struct mm_struct *mm, return 0; } -static inline bool vma_can_userfault(struct vm_area_struct *vma, - unsigned long vm_flags) -{ - /* FIXME: add WP support to hugetlbfs and shmem */ - if (vm_flags & VM_UFFD_WP) { - if (is_vm_hugetlb_page(vma) || vma_is_shmem(vma)) - return false; - } - - if (vm_flags & VM_UFFD_MINOR) { - if (!(is_vm_hugetlb_page(vma) || vma_is_shmem(vma))) - return false; - } - - return vma_is_anonymous(vma) || is_vm_hugetlb_page(vma) || - vma_is_shmem(vma); -} - static int userfaultfd_register(struct userfaultfd_ctx *ctx, unsigned long arg) { diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h index ca3f794d07e9..489fb375e66c 100644 --- a/include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -16,6 +16,7 @@ #include #include #include +#include /* The set of all possible UFFD-related VM flags. */ #define __VM_UFFD_FLAGS (VM_UFFD_MISSING | VM_UFFD_WP | VM_UFFD_MINOR) @@ -138,6 +139,17 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma) return vma->vm_flags & __VM_UFFD_FLAGS; } +static inline bool vma_can_userfault(struct vm_area_struct *vma, + unsigned long vm_flags) +{ + if (vm_flags & VM_UFFD_MINOR) + return is_vm_hugetlb_page(vma) || vma_is_shmem(vma); + + return vma_is_anonymous(vma) || is_vm_hugetlb_page(vma) || + vma_is_shmem(vma); +} + + extern int dup_userfaultfd(struct vm_area_struct *, struct list_head *); extern void dup_userfaultfd_complete(struct list_head *); diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h index 159a74e9564f..09b7ec69c97d 100644 --- a/include/uapi/linux/userfaultfd.h +++ b/include/uapi/linux/userfaultfd.h @@ -32,7 +32,8 @@ UFFD_FEATURE_SIGBUS | \ UFFD_FEATURE_THREAD_ID | \ UFFD_FEATURE_MINOR_HUGETLBFS | \ - UFFD_FEATURE_MINOR_SHMEM) + UFFD_FEATURE_MINOR_SHMEM | \ + UFFD_FEATURE_WP_HUGETLBFS_SHMEM) #define UFFD_API_IOCTLS \ ((__u64)1 << _UFFDIO_REGISTER | \ (__u64)1 << _UFFDIO_UNREGISTER | \ @@ -46,7 +47,8 @@ #define UFFD_API_RANGE_IOCTLS_BASIC \ ((__u64)1 << _UFFDIO_WAKE | \ (__u64)1 << _UFFDIO_COPY | \ - (__u64)1 << _UFFDIO_CONTINUE) + (__u64)1 << _UFFDIO_CONTINUE | \ + (__u64)1 << _UFFDIO_WRITEPROTECT) /* * Valid ioctl command number range with this API is from 0x00 to @@ -189,6 +191,9 @@ struct uffdio_api { * * UFFD_FEATURE_MINOR_SHMEM indicates the same support as * UFFD_FEATURE_MINOR_HUGETLBFS, but for shmem-backed pages instead. + * + * UFFD_FEATURE_WP_HUGETLBFS_SHMEM indicates that userfaultfd + * write-protection mode is supported on both shmem and hugetlbfs. */ #define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0) #define UFFD_FEATURE_EVENT_FORK (1<<1) @@ -201,6 +206,7 @@ struct uffdio_api { #define UFFD_FEATURE_THREAD_ID (1<<8) #define UFFD_FEATURE_MINOR_HUGETLBFS (1<<9) #define UFFD_FEATURE_MINOR_SHMEM (1<<10) +#define UFFD_FEATURE_WP_HUGETLBFS_SHMEM (1<<11) __u64 features; __u64 ioctls; diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index af79f3d3a001..7ff9176149e0 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -772,15 +772,12 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, err = -ENOENT; dst_vma = find_dst_vma(dst_mm, start, len); - /* - * Make sure the vma is not shared, that the dst range is - * both valid and fully within a single existing vma. - */ - if (!dst_vma || (dst_vma->vm_flags & VM_SHARED)) + + if (!dst_vma) goto out_unlock; if (!userfaultfd_wp(dst_vma)) goto out_unlock; - if (!vma_is_anonymous(dst_vma)) + if (!vma_can_userfault(dst_vma, dst_vma->vm_flags)) goto out_unlock; if (is_vm_hugetlb_page(dst_vma)) { From patchwork Thu May 27 20:23:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12285305 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEF0EC47089 for ; Thu, 27 May 2021 20:23:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9411361360 for ; Thu, 27 May 2021 20:23:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9411361360 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2D8686B006C; Thu, 27 May 2021 16:23:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2AEAE6B0093; Thu, 27 May 2021 16:23:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 129516B0095; Thu, 27 May 2021 16:23:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0228.hostedemail.com [216.40.44.228]) by kanga.kvack.org (Postfix) with ESMTP id D86CE6B006C for ; Thu, 27 May 2021 16:23:58 -0400 (EDT) Received: from smtpin39.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 7C5471803D420 for ; Thu, 27 May 2021 20:23:58 +0000 (UTC) X-FDA: 78188137356.39.6D1488B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf02.hostedemail.com (Postfix) with ESMTP id 9FDB94142C51 for ; Thu, 27 May 2021 20:23:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622147037; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Uq4QT2ZrcVHt/byJv1ujYeldpYEbg+Q7RaFfi3UPyeE=; b=K9DbR+DrQxDgcJd6+X33FseF6NeKGreCjA0f4HVk+ulpYFgCCQGG6UioC7L8P9zZWk88SV i0uVoijRabQvNhH1iLWJgxZzx4lbce1NBq5H3kUtZh6mgTSWMhWvotCZibL6xH3ABFCclk vNOekWNoLhkCLeE8L4jfF1LjmGOvEpw= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-123-Y12DBMgWNza1g9CcPv5-8w-1; Thu, 27 May 2021 16:23:53 -0400 X-MC-Unique: Y12DBMgWNza1g9CcPv5-8w-1 Received: by mail-qt1-f197.google.com with SMTP id h12-20020ac8776c0000b02901f1228fdb1bso876243qtu.6 for ; Thu, 27 May 2021 13:23:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Uq4QT2ZrcVHt/byJv1ujYeldpYEbg+Q7RaFfi3UPyeE=; b=ETsA8dF/iUL+DMsCibSVwN1HN1xSVhCbNhGrO70pw5sKRbtjTe1GZFFCKIvz1PvNRy fyBrPJoOoHwv0SoisExDcJw2ovLT2ifbduVfMuX1exuIEqmR4zV6cfF6elE3NK710Aa2 vBYHShSRMLM9ZfcSo/Wa+Z74Bk26928S7Qi9tiaKV0iQW9ykaxTL+SygC/nYoOvelCGf qkoX26KehSNPU0Hwjrbam9DgMK91OzbiPD7oYbQOzBlbnlhl3of5a9XcB6yxLlMVK1VI XvUKdND9s8BbXzrrVYlgEVVnGoEtQZ71SyfN6tMhw/4HtbWoizx9y2bOAoGs8soBcUab Y8sg== X-Gm-Message-State: AOAM533/qGGgECG3qSp1bTvmS/kyWmGS5Al4UkUYROTXQfmABDSuYuPg Pya3FzU8SFsxxmcWTcmkylnIGZxnBZwpjBusnzcJhwnO3v2AG5ba2O4OfFtGDkZiN87EcFnV5GN svdzkjDEsMwc= X-Received: by 2002:ac8:5484:: with SMTP id h4mr360837qtq.338.1622147032999; Thu, 27 May 2021 13:23:52 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz3zi5PoIwHGhlPFnbYVnDOeLkXM0Dn9r4qegg3oaBy1B1xbjvo3TJ2sVyd9Gu8IcG8naBLlA== X-Received: by 2002:ac8:5484:: with SMTP id h4mr360813qtq.338.1622147032703; Thu, 27 May 2021 13:23:52 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id w4sm1981299qki.57.2021.05.27.13.23.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:23:52 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Rapoport , peterx@redhat.com, Andrew Morton , Mike Kravetz , Jerome Glisse , Miaohe Lin , Nadav Amit , Hugh Dickins , Matthew Wilcox , Jason Gunthorpe , "Kirill A . Shutemov" , Andrea Arcangeli , Axel Rasmussen Subject: [PATCH v3 27/27] userfaultfd/selftests: Enable uffd-wp for shmem/hugetlbfs Date: Thu, 27 May 2021 16:23:48 -0400 Message-Id: <20210527202348.32406-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527201927.29586-1-peterx@redhat.com> References: <20210527201927.29586-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 9FDB94142C51 Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=K9DbR+Dr; spf=none (imf02.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam04 X-Stat-Signature: hs5kzt5endyqn4hheofoa3qci9gfzcq7 X-HE-Tag: 1622147033-195623 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: After we added support for shmem and hugetlbfs, we can turn uffd-wp test on always now. Define HUGETLB_EXPECTED_IOCTLS to avoid using UFFD_API_RANGE_IOCTLS_BASIC, because UFFD_API_RANGE_IOCTLS_BASIC is normally a superset of capabilities, while the test may not satisfy them all. E.g., when hugetlb registered without minor mode, then we need to explicitly remove _UFFDIO_CONTINUE. Same thing to uffd-wp, as we'll need to explicitly remove _UFFDIO_WRITEPROTECT if not registered with uffd-wp. For the long term, we may consider dropping UFFD_API_* macros completely from uapi/linux/userfaultfd.h header files, because it may cause kernel header update to easily break userspace. Signed-off-by: Peter Xu --- tools/testing/selftests/vm/userfaultfd.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/vm/userfaultfd.c b/tools/testing/selftests/vm/userfaultfd.c index e363bdaff59d..015f2df8ece4 100644 --- a/tools/testing/selftests/vm/userfaultfd.c +++ b/tools/testing/selftests/vm/userfaultfd.c @@ -80,7 +80,7 @@ static int test_type; static volatile bool test_uffdio_copy_eexist = true; static volatile bool test_uffdio_zeropage_eexist = true; /* Whether to test uffd write-protection */ -static bool test_uffdio_wp = false; +static bool test_uffdio_wp = true; /* Whether to test uffd minor faults */ static bool test_uffdio_minor = false; @@ -320,6 +320,9 @@ struct uffd_test_ops { (1 << _UFFDIO_ZEROPAGE) | \ (1 << _UFFDIO_WRITEPROTECT)) +#define HUGETLB_EXPECTED_IOCTLS ((1 << _UFFDIO_WAKE) | \ + (1 << _UFFDIO_COPY)) + static struct uffd_test_ops anon_uffd_test_ops = { .expected_ioctls = ANON_EXPECTED_IOCTLS, .allocate_area = anon_allocate_area, @@ -335,7 +338,7 @@ static struct uffd_test_ops shmem_uffd_test_ops = { }; static struct uffd_test_ops hugetlb_uffd_test_ops = { - .expected_ioctls = UFFD_API_RANGE_IOCTLS_BASIC & ~(1 << _UFFDIO_CONTINUE), + .expected_ioctls = HUGETLB_EXPECTED_IOCTLS, .allocate_area = hugetlb_allocate_area, .release_pages = hugetlb_release_pages, .alias_mapping = hugetlb_alias_mapping, @@ -1580,8 +1583,6 @@ static void set_test_type(const char *type) if (!strcmp(type, "anon")) { test_type = TEST_ANON; uffd_test_ops = &anon_uffd_test_ops; - /* Only enable write-protect test for anonymous test */ - test_uffdio_wp = true; } else if (!strcmp(type, "hugetlb")) { test_type = TEST_HUGETLB; uffd_test_ops = &hugetlb_uffd_test_ops;