From patchwork Thu Apr 8 23:43:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Axel Rasmussen X-Patchwork-Id: 12192625 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34B6FC433B4 for ; Thu, 8 Apr 2021 23:43:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 14FDC61108 for ; Thu, 8 Apr 2021 23:43:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233014AbhDHXnt (ORCPT ); Thu, 8 Apr 2021 19:43:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60576 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232978AbhDHXnq (ORCPT ); Thu, 8 Apr 2021 19:43:46 -0400 Received: from mail-qv1-xf49.google.com (mail-qv1-xf49.google.com [IPv6:2607:f8b0:4864:20::f49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 454C0C061761 for ; Thu, 8 Apr 2021 16:43:34 -0700 (PDT) Received: by mail-qv1-xf49.google.com with SMTP id n15so2099997qvo.0 for ; Thu, 08 Apr 2021 16:43:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=vPRUOYIolP6Zcf38F8T1KJCfFhHdJ3+UjhE2Kl0rYHk=; b=quRxZdtCDPR6OhkaUBqKezAnyT6r8AW9F0E7tfBCP1bYDJm9ipupmhQTfWf9kyG8xt m+SRz2SscoDX7EKEaGoKp6HuzxgAH9yyaAZnP1bqfMzue9WosB9yW8iN1l8qLendN2u1 SajVcyh9h5+huUD1MAQQtB5RQml42Fd9SPwoJOUPopR/TD0Rb1xJKaaw7UHHj8+VQOHo V90Oya1oHmZ/yMZm3KOejvvgtvJeOqsPh/uMFmWHV6u4s66e2Oj6tsTt+PntFWwuwGbK U1vk1/unTU4bvIaSYQBJUJ1gBFp0dRJM3bnsdhNxGOPJs+aWVKzxhR86usPdXp74qQBT OU7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=vPRUOYIolP6Zcf38F8T1KJCfFhHdJ3+UjhE2Kl0rYHk=; b=GkQzZa7lzfhRXulBbElDgkvg4SDWFpfFTae/Q9VPWWgOmRSKY97LFi8UFFUgCU5odG i9SB+7V/JIcw8pIPovB+i2P1zbQ/0nDj718xhQP1vTs6JH7k1RGfGl0pxS1q8qCa5V4C wQMuioeGIp88pXl2rLfJ7d0RfaCSqO8T5MKboCColyVVTsObHpPIsHuUoxdPnyY/k3dZ fXJxNH/z2WC53ygbqcyLj6tt93tlMdhQJo5RTTL/Km73NoCHjeFgGmuJMwAg7ptadp5+ 9vXFMSdk/RLSMeH8FwiBCbvZW9iB5VQNBgtW5TUoeaQCYIlNBaPnFblUPUd8dArAXgx2 BHMA== X-Gm-Message-State: AOAM53058xTimx/+BDPD4C+z7Yb1kzdn3TKetcy/oG9bgCIxB2TNnMNC 4wC+PMdo3i0t92jhs2ReawC0b+iNB5PzEINAZNCK X-Google-Smtp-Source: ABdhPJwULITQJfoF7WCDXR8cK9vKqqgb+uM2S+dG1ktXyeGLc7FDRrwd7Mn2cbdmOUuWSvSi21YeCNDU0IeWdmpFA6Zw X-Received: from ajr0.svl.corp.google.com ([2620:15c:2cd:203:3d06:d00f:a626:675a]) (user=axelrasmussen job=sendgmr) by 2002:a05:6214:905:: with SMTP id dj5mr11624979qvb.34.1617925413457; Thu, 08 Apr 2021 16:43:33 -0700 (PDT) Date: Thu, 8 Apr 2021 16:43:19 -0700 In-Reply-To: <20210408234327.624367-1-axelrasmussen@google.com> Message-Id: <20210408234327.624367-2-axelrasmussen@google.com> Mime-Version: 1.0 References: <20210408234327.624367-1-axelrasmussen@google.com> X-Mailer: git-send-email 2.31.1.295.g9ea45b61b8-goog Subject: [PATCH 1/9] userfaultfd/hugetlbfs: avoid including userfaultfd_k.h in hugetlb.h From: Axel Rasmussen To: Alexander Viro , Andrea Arcangeli , Andrew Morton , Daniel Colascione , Hugh Dickins , Jerome Glisse , Joe Perches , Lokesh Gidra , Mike Kravetz , Mike Rapoport , Peter Xu , Shaohua Li , Shuah Khan , Stephen Rothwell , Wang Qing Cc: linux-api@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mm@kvack.org, Axel Rasmussen , Brian Geffon , "Dr . David Alan Gilbert" , Mina Almasry , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Minimizing header file inclusion is desirable. In this case, we can do so just by forward declaring the enumeration our signature relies upon. Signed-off-by: Axel Rasmussen Reviewed-by: Peter Xu --- include/linux/hugetlb.h | 5 ++++- mm/hugetlb.c | 1 + 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 1d3246b31a41..dfb749eaf348 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -11,7 +11,6 @@ #include #include #include -#include struct ctl_table; struct user_struct; @@ -136,6 +135,8 @@ unsigned long hugetlb_total_pages(void); vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, unsigned int flags); #ifdef CONFIG_USERFAULTFD +enum mcopy_atomic_mode; + int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, pte_t *dst_pte, struct vm_area_struct *dst_vma, unsigned long dst_addr, @@ -315,6 +316,8 @@ static inline void hugetlb_free_pgd_range(struct mmu_gather *tlb, } #ifdef CONFIG_USERFAULTFD +enum mcopy_atomic_mode; + static inline int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, pte_t *dst_pte, struct vm_area_struct *dst_vma, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 9973dec4976c..3b93bbf8c80f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -40,6 +40,7 @@ #include #include #include +#include #include "internal.h" int hugetlb_max_hstate __read_mostly; From patchwork Thu Apr 8 23:43:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Axel Rasmussen X-Patchwork-Id: 12192629 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E47FC43611 for ; Thu, 8 Apr 2021 23:43:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2662961159 for ; Thu, 8 Apr 2021 23:43:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233034AbhDHXnu (ORCPT ); Thu, 8 Apr 2021 19:43:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60590 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233006AbhDHXnr (ORCPT ); Thu, 8 Apr 2021 19:43:47 -0400 Received: from mail-qk1-x749.google.com (mail-qk1-x749.google.com [IPv6:2607:f8b0:4864:20::749]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ECE3CC061760 for ; Thu, 8 Apr 2021 16:43:35 -0700 (PDT) Received: by mail-qk1-x749.google.com with SMTP id h21so2310599qkl.12 for ; Thu, 08 Apr 2021 16:43:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Yxcc+FyoEk6BMiPROPZkCG4vnvX4iWyE7l55TnQaADU=; b=PF9dJ9fOgxA1OLmNnZEX8/eI4dVSJy+BZ5oFMx5i/4/Hy3sy0/nDMlyNp2WzHW+Uo/ s0Nf7ZJzbGIwI4xyx49U0nsJcsL/U0b65KZJpIW2RimMVUGzwkPG05XtAR9jXYLhOnmr HQOpQF1KvUgJEPIQrxgaPEQfNPDoEU94KwNl4zzxzgD9mXkia0VCkZk+C91VlnHqZOCX GElhOPnAK3wYb1lUAKGFRysWsr660QdeDbqz/mtqypIxon9swvuA6Uq5Ax0TdOJv8oF0 y6b2e83Zwb99r7PR593uxBirE9C7IiEjmOovu6D4UQTvJejXWJHd22N+SImscLPfwasV 7CTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Yxcc+FyoEk6BMiPROPZkCG4vnvX4iWyE7l55TnQaADU=; b=mDpOongO+hurZ0KPH3fAjK0Ilrp3gNetyG/K3IgVv98+C1WkrjjULRXy/LtKOko1gy zOm1SyWJhBttGMi45HJ9TrbRsje/lfk6IJuT6DEdE88zNe8VvQs0LjgUUWTk2OJVeiTy LaC77PQos29MR+4kY6qcNDzepLLW1gS5l4TfmYvvokq7SUiZ1kCDKt2XAENOY+nxki+C Mq7Q576jKXR5uASMgFTwBAnI+kVxQMi9UXgNmLSIWuw5tvMJM/l8/1DKfx5LzDiYCOjf qLSEOlk6c9MXvi+vhqKS0mpIT1/GPLkXKt62UBccsVhGnbAHUiEThaQKXZUshqdioong sBiA== X-Gm-Message-State: AOAM532hVOFmQ+GGqFG5BrlGLCNpiE0C7qpjsMKkuM4epm9JBU7IG2JD 4ouEa/3v6TiT/+oU77z4LLneKGNM5lIO7qquNO/v X-Google-Smtp-Source: ABdhPJyFGHrzfpgKJFY2ZDF3sxxiWHVpvayvEO6k2+WV5N7qfuAXsqVa4yAzv+2HgxqKnuW5Ks2f5/wVg2WnaEBuFLnd X-Received: from ajr0.svl.corp.google.com ([2620:15c:2cd:203:3d06:d00f:a626:675a]) (user=axelrasmussen job=sendgmr) by 2002:a0c:b348:: with SMTP id a8mr11191833qvf.7.1617925415096; Thu, 08 Apr 2021 16:43:35 -0700 (PDT) Date: Thu, 8 Apr 2021 16:43:20 -0700 In-Reply-To: <20210408234327.624367-1-axelrasmussen@google.com> Message-Id: <20210408234327.624367-3-axelrasmussen@google.com> Mime-Version: 1.0 References: <20210408234327.624367-1-axelrasmussen@google.com> X-Mailer: git-send-email 2.31.1.295.g9ea45b61b8-goog Subject: [PATCH 2/9] userfaultfd/shmem: combine shmem_{mcopy_atomic,mfill_zeropage}_pte From: Axel Rasmussen To: Alexander Viro , Andrea Arcangeli , Andrew Morton , Daniel Colascione , Hugh Dickins , Jerome Glisse , Joe Perches , Lokesh Gidra , Mike Kravetz , Mike Rapoport , Peter Xu , Shaohua Li , Shuah Khan , Stephen Rothwell , Wang Qing Cc: linux-api@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mm@kvack.org, Axel Rasmussen , Brian Geffon , "Dr . David Alan Gilbert" , Mina Almasry , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Previously, we did a dance where we had one calling path in userfaultfd.c (mfill_atomic_pte), but then we split it into two in shmem_fs.h (shmem_{mcopy_atomic,mfill_zeropage}_pte), and then rejoined into a single shared function in shmem.c (shmem_mfill_atomic_pte). This is all a bit overly complex. Just call the single combined shmem function directly, allowing us to clean up various branches, boilerplate, etc. While we're touching this function, two other small cleanup changes: - offset is equivalent to pgoff, so we can get rid of offset entirely. - Split two VM_BUG_ON cases into two statements. This means the line number reported when the BUG is hit specifies exactly which condition was true. Signed-off-by: Axel Rasmussen Reviewed-by: Peter Xu --- include/linux/shmem_fs.h | 15 +++++------- mm/shmem.c | 52 +++++++++++++--------------------------- mm/userfaultfd.c | 10 +++----- 3 files changed, 25 insertions(+), 52 deletions(-) diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h index d82b6f396588..919e36671fe6 100644 --- a/include/linux/shmem_fs.h +++ b/include/linux/shmem_fs.h @@ -122,21 +122,18 @@ static inline bool shmem_file(struct file *file) extern bool shmem_charge(struct inode *inode, long pages); extern void shmem_uncharge(struct inode *inode, long pages); +#ifdef CONFIG_USERFAULTFD #ifdef CONFIG_SHMEM extern int shmem_mcopy_atomic_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, struct vm_area_struct *dst_vma, unsigned long dst_addr, unsigned long src_addr, + bool zeropage, struct page **pagep); -extern int shmem_mfill_zeropage_pte(struct mm_struct *dst_mm, - pmd_t *dst_pmd, - struct vm_area_struct *dst_vma, - unsigned long dst_addr); -#else +#else /* !CONFIG_SHMEM */ #define shmem_mcopy_atomic_pte(dst_mm, dst_pte, dst_vma, dst_addr, \ - src_addr, pagep) ({ BUG(); 0; }) -#define shmem_mfill_zeropage_pte(dst_mm, dst_pmd, dst_vma, \ - dst_addr) ({ BUG(); 0; }) -#endif + src_addr, zeropage, pagep) ({ BUG(); 0; }) +#endif /* CONFIG_SHMEM */ +#endif /* CONFIG_USERFAULTFD */ #endif diff --git a/mm/shmem.c b/mm/shmem.c index b2db4ed0fbc7..c21f20cc4204 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2354,13 +2354,14 @@ static struct inode *shmem_get_inode(struct super_block *sb, const struct inode return inode; } -static int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, - pmd_t *dst_pmd, - struct vm_area_struct *dst_vma, - unsigned long dst_addr, - unsigned long src_addr, - bool zeropage, - struct page **pagep) +#ifdef CONFIG_USERFAULTFD +int shmem_mcopy_atomic_pte(struct mm_struct *dst_mm, + pmd_t *dst_pmd, + struct vm_area_struct *dst_vma, + unsigned long dst_addr, + unsigned long src_addr, + bool zeropage, + struct page **pagep) { struct inode *inode = file_inode(dst_vma->vm_file); struct shmem_inode_info *info = SHMEM_I(inode); @@ -2372,7 +2373,7 @@ static int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, struct page *page; pte_t _dst_pte, *dst_pte; int ret; - pgoff_t offset, max_off; + pgoff_t max_off; ret = -ENOMEM; if (!shmem_inode_acct_block(inode, 1)) @@ -2383,7 +2384,7 @@ static int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, if (!page) goto out_unacct_blocks; - if (!zeropage) { /* mcopy_atomic */ + if (!zeropage) { /* COPY */ page_kaddr = kmap_atomic(page); ret = copy_from_user(page_kaddr, (const void __user *)src_addr, @@ -2397,7 +2398,7 @@ static int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, /* don't free the page */ return -ENOENT; } - } else { /* mfill_zeropage_atomic */ + } else { /* ZEROPAGE */ clear_highpage(page); } } else { @@ -2405,15 +2406,15 @@ static int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, *pagep = NULL; } - VM_BUG_ON(PageLocked(page) || PageSwapBacked(page)); + VM_BUG_ON(PageLocked(page)); + VM_BUG_ON(PageSwapBacked(page)); __SetPageLocked(page); __SetPageSwapBacked(page); __SetPageUptodate(page); ret = -EFAULT; - offset = linear_page_index(dst_vma, dst_addr); max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); - if (unlikely(offset >= max_off)) + if (unlikely(pgoff >= max_off)) goto out_release; ret = shmem_add_to_page_cache(page, mapping, pgoff, NULL, @@ -2439,7 +2440,7 @@ static int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, ret = -EFAULT; max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); - if (unlikely(offset >= max_off)) + if (unlikely(pgoff >= max_off)) goto out_release_unlock; ret = -EEXIST; @@ -2476,28 +2477,7 @@ static int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, shmem_inode_unacct_blocks(inode, 1); goto out; } - -int shmem_mcopy_atomic_pte(struct mm_struct *dst_mm, - pmd_t *dst_pmd, - struct vm_area_struct *dst_vma, - unsigned long dst_addr, - unsigned long src_addr, - struct page **pagep) -{ - return shmem_mfill_atomic_pte(dst_mm, dst_pmd, dst_vma, - dst_addr, src_addr, false, pagep); -} - -int shmem_mfill_zeropage_pte(struct mm_struct *dst_mm, - pmd_t *dst_pmd, - struct vm_area_struct *dst_vma, - unsigned long dst_addr) -{ - struct page *page = NULL; - - return shmem_mfill_atomic_pte(dst_mm, dst_pmd, dst_vma, - dst_addr, 0, true, &page); -} +#endif /* CONFIG_USERFAULTFD */ #ifdef CONFIG_TMPFS static const struct inode_operations shmem_symlink_inode_operations; diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index ce6cb4760d2c..60ae22207761 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -440,13 +440,9 @@ static __always_inline ssize_t mfill_atomic_pte(struct mm_struct *dst_mm, dst_vma, dst_addr); } else { VM_WARN_ON_ONCE(wp_copy); - if (!zeropage) - err = shmem_mcopy_atomic_pte(dst_mm, dst_pmd, - dst_vma, dst_addr, - src_addr, page); - else - err = shmem_mfill_zeropage_pte(dst_mm, dst_pmd, - dst_vma, dst_addr); + err = shmem_mcopy_atomic_pte(dst_mm, dst_pmd, dst_vma, + dst_addr, src_addr, zeropage, + page); } return err; From patchwork Thu Apr 8 23:43:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Axel Rasmussen X-Patchwork-Id: 12192627 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B04DC43461 for ; Thu, 8 Apr 2021 23:43:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2E53A61157 for ; Thu, 8 Apr 2021 23:43:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233044AbhDHXnv (ORCPT ); Thu, 8 Apr 2021 19:43:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60608 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233026AbhDHXnt (ORCPT ); Thu, 8 Apr 2021 19:43:49 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A51E3C061761 for ; Thu, 8 Apr 2021 16:43:37 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id g7so3558273ybm.13 for ; Thu, 08 Apr 2021 16:43:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=iJrBKic6BWLSUzVU0etk70MUxAfaDA5mu3HMcg/WB2s=; b=ic+SugPMZft+1nzSi4Yef4EU2Ccy8KcK3OzqNulnPL8gkFYK92kKgxPDtoqnaitbq1 sL7Zz0Wg8X4Ux3qEN51dsekfArXaBoa37tMzdIs2ORXk/KM83dsMJCHF5DKpsgK7IwWw xfsl2OgAD7uWHV9JWdMHHUHIBH6IukevDzfj43G5TijUVbesqk5OxyCfGUW10QaxvhJo ++gG0U2pixyy2dbD+TzlnaTkoLx0uBsZ6hgOnsReJTVElIz8zKjURLKfzkQZwgLCZA7F I2E21NbyhbJi91WQqlLXpecYru25fOB4bOFna/WqtTzB8DaRNpR1PGZPXT7hLvXdGizB pkWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=iJrBKic6BWLSUzVU0etk70MUxAfaDA5mu3HMcg/WB2s=; b=a2lTr/guvvqTgGndaypmFSxyp6oXr1MLNbDu3sJIQXYSFYeXGD4HllG6FysYTauk7I uEAhM1usARYpAAe7nE3Uy9eaj7e5z9Fy1Z5HRQpa5TyuWnSlYfoZvMY/cLbB+iR3J1H/ 9dhEKoT+kyu9y8JmLqRhg+NsQZDzqvZIx+K/F1o20x7gsh5OALMMoXMhL2adQO6LS7wN 4VSnvHKuxGHGu6sehFAJ3vpznIvC3R3793d9UVo0xP2IzPGy4QPahxw3VQIDZKF4BKF3 9GQA9H1mjsU6gfPTsAw+5B/3VyWjhoTYpLPSJqlfxqmDlalQortVybAJSADi9X8IkfWD oOvw== X-Gm-Message-State: AOAM532kkI2d2lZDPGnrTLHy7QZR2n59iJDD7wZdOtH41AUFdqqBOwL+ atGdw5qIQWlGSsi+APdhQtBLWj+9zk77SZAQPZPQ X-Google-Smtp-Source: ABdhPJxpp4lDpLiYYEnpqNp54PjbYV/IE4p232ydPzXfrLRRoYXvEvVTNo4kRm16OanUNhbB9X3yYs7J9+1UroUEsKtN X-Received: from ajr0.svl.corp.google.com ([2620:15c:2cd:203:3d06:d00f:a626:675a]) (user=axelrasmussen job=sendgmr) by 2002:a25:6e87:: with SMTP id j129mr15135621ybc.215.1617925416939; Thu, 08 Apr 2021 16:43:36 -0700 (PDT) Date: Thu, 8 Apr 2021 16:43:21 -0700 In-Reply-To: <20210408234327.624367-1-axelrasmussen@google.com> Message-Id: <20210408234327.624367-4-axelrasmussen@google.com> Mime-Version: 1.0 References: <20210408234327.624367-1-axelrasmussen@google.com> X-Mailer: git-send-email 2.31.1.295.g9ea45b61b8-goog Subject: [PATCH 3/9] userfaultfd/shmem: support minor fault registration for shmem From: Axel Rasmussen To: Alexander Viro , Andrea Arcangeli , Andrew Morton , Daniel Colascione , Hugh Dickins , Jerome Glisse , Joe Perches , Lokesh Gidra , Mike Kravetz , Mike Rapoport , Peter Xu , Shaohua Li , Shuah Khan , Stephen Rothwell , Wang Qing Cc: linux-api@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mm@kvack.org, Axel Rasmussen , Brian Geffon , "Dr . David Alan Gilbert" , Mina Almasry , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This patch allows shmem-backed VMAs to be registered for minor faults. Minor faults are appropriately relayed to userspace in the fault path, for VMAs with the relevant flag. This commit doesn't hook up the UFFDIO_CONTINUE ioctl for shmem-backed minor faults, though, so userspace doesn't yet have a way to resolve such faults. Signed-off-by: Axel Rasmussen --- fs/userfaultfd.c | 6 +++--- include/uapi/linux/userfaultfd.h | 7 ++++++- mm/memory.c | 8 +++++--- mm/shmem.c | 20 ++++++++++++++++---- 4 files changed, 30 insertions(+), 11 deletions(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 14f92285d04f..9f3b8684cf3c 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -1267,8 +1267,7 @@ static inline bool vma_can_userfault(struct vm_area_struct *vma, } if (vm_flags & VM_UFFD_MINOR) { - /* FIXME: Add minor fault interception for shmem. */ - if (!is_vm_hugetlb_page(vma)) + if (!(is_vm_hugetlb_page(vma) || vma_is_shmem(vma))) return false; } @@ -1941,7 +1940,8 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, /* report all available features and ioctls to userland */ uffdio_api.features = UFFD_API_FEATURES; #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR - uffdio_api.features &= ~UFFD_FEATURE_MINOR_HUGETLBFS; + uffdio_api.features &= + ~(UFFD_FEATURE_MINOR_HUGETLBFS | UFFD_FEATURE_MINOR_SHMEM); #endif uffdio_api.ioctls = UFFD_API_IOCTLS; ret = -EFAULT; diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h index bafbeb1a2624..159a74e9564f 100644 --- a/include/uapi/linux/userfaultfd.h +++ b/include/uapi/linux/userfaultfd.h @@ -31,7 +31,8 @@ UFFD_FEATURE_MISSING_SHMEM | \ UFFD_FEATURE_SIGBUS | \ UFFD_FEATURE_THREAD_ID | \ - UFFD_FEATURE_MINOR_HUGETLBFS) + UFFD_FEATURE_MINOR_HUGETLBFS | \ + UFFD_FEATURE_MINOR_SHMEM) #define UFFD_API_IOCTLS \ ((__u64)1 << _UFFDIO_REGISTER | \ (__u64)1 << _UFFDIO_UNREGISTER | \ @@ -185,6 +186,9 @@ struct uffdio_api { * UFFD_FEATURE_MINOR_HUGETLBFS indicates that minor faults * can be intercepted (via REGISTER_MODE_MINOR) for * hugetlbfs-backed pages. + * + * UFFD_FEATURE_MINOR_SHMEM indicates the same support as + * UFFD_FEATURE_MINOR_HUGETLBFS, but for shmem-backed pages instead. */ #define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0) #define UFFD_FEATURE_EVENT_FORK (1<<1) @@ -196,6 +200,7 @@ struct uffdio_api { #define UFFD_FEATURE_SIGBUS (1<<7) #define UFFD_FEATURE_THREAD_ID (1<<8) #define UFFD_FEATURE_MINOR_HUGETLBFS (1<<9) +#define UFFD_FEATURE_MINOR_SHMEM (1<<10) __u64 features; __u64 ioctls; diff --git a/mm/memory.c b/mm/memory.c index c8e357627318..a1e5ff55027e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3929,9 +3929,11 @@ static vm_fault_t do_read_fault(struct vm_fault *vmf) * something). */ if (vma->vm_ops->map_pages && fault_around_bytes >> PAGE_SHIFT > 1) { - ret = do_fault_around(vmf); - if (ret) - return ret; + if (likely(!userfaultfd_minor(vmf->vma))) { + ret = do_fault_around(vmf); + if (ret) + return ret; + } } ret = __do_fault(vmf); diff --git a/mm/shmem.c b/mm/shmem.c index c21f20cc4204..99c54b165c16 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1785,7 +1785,7 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index, * vm. If we swap it in we mark it dirty since we also free the swap * entry since a page cannot live in both the swap and page cache. * - * vmf and fault_type are only supplied by shmem_fault: + * vma, vmf, and fault_type are only supplied by shmem_fault: * otherwise they are NULL. */ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, @@ -1802,6 +1802,7 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, pgoff_t hindex = index; gfp_t huge_gfp; int error; + bool swapped; int once = 0; int alloced = 0; @@ -1820,16 +1821,27 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, page = pagecache_get_page(mapping, index, FGP_ENTRY | FGP_HEAD | FGP_LOCK, 0); - if (xa_is_value(page)) { + swapped = xa_is_value(page); + if (swapped) { error = shmem_swapin_page(inode, index, &page, sgp, gfp, vma, fault_type); if (error == -EEXIST) goto repeat; - *pagep = page; - return error; + if (error) + return error; } + if (page && vma && userfaultfd_minor(vma)) { + unlock_page(page); + put_page(page); + *fault_type = handle_userfault(vmf, VM_UFFD_MINOR); + return 0; + } + + if (swapped) + return 0; + if (page) hindex = page->index; if (page && sgp == SGP_WRITE) From patchwork Thu Apr 8 23:43:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Axel Rasmussen X-Patchwork-Id: 12192631 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 758ECC43462 for ; Thu, 8 Apr 2021 23:43:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5672561028 for ; Thu, 8 Apr 2021 23:43:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233077AbhDHXn4 (ORCPT ); Thu, 8 Apr 2021 19:43:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60626 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233042AbhDHXnv (ORCPT ); Thu, 8 Apr 2021 19:43:51 -0400 Received: from mail-qv1-xf4a.google.com (mail-qv1-xf4a.google.com [IPv6:2607:f8b0:4864:20::f4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7C6FBC061763 for ; Thu, 8 Apr 2021 16:43:39 -0700 (PDT) Received: by mail-qv1-xf4a.google.com with SMTP id n15so2100071qvo.0 for ; Thu, 08 Apr 2021 16:43:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=b1l2Yiaa4e1GScvqO3tIqjfF3IL448iCcHjGxtB3S5U=; b=TnAwE/T+b2WxJDB34xqNZqms0Uq5Jfv4/M2aUflRzWby1PbjsEIWIE4JjVy60EWtfB Nh2zGATLALwuepjx5KMSUQvvtDWeOkW+nophCrFShYHjTashfLngUwzObzi9ow+vpY4B ofxVY75OaJBumpmoo0b4G370K8+Nqclr3GeQFsz04Ds58tOe6bDTH26UZhxlwcwsmZcq hEE8JfMw2SFnhrU0R6gaSJR1I2QmY1loupyMEw1UgbkkaaY0XypYsGYRnK/TnvIw/poy YMF/fISzHNa3O1sE+OhwMneUxM/EIYBrXEgTARqPjmqIUVQ+R+UlwREhSmq41fbfteHe jZHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=b1l2Yiaa4e1GScvqO3tIqjfF3IL448iCcHjGxtB3S5U=; b=aTe2/55ADtwejTyqHlTeSMlvg7RGmZRM6FUej6aZEFiIEMkAF5gvDS0KIzfjNbcez6 bUD+2cs72GPI4lWLJQBlOCUQXtGbJ26iO9Bv2mIpqX73wMcfibFvYoFvWwpoVBjiWLt1 9hF1JN7sFPezfvXgeDi3Ai2/YFRdAP345/C9C12NHZoeQlvmWTceZAGtlSzy1tDRXqtG rpX/CmIBJef14Hquo2Tqj2BPJmvyzH/Yn7Rox266RT4ZWHHZ8Z4finYI7SrmVSVnGRT0 A3e7hQ3GQ+9H623RNolHaZLv4FdeY1Lt3f1abvloTWL4CcKh4GE0PMz0a3PXhosMgNTG guYA== X-Gm-Message-State: AOAM533ReBaMaf+g3p9tir6J7ZCvjoan9WMBSg3GvMod1OfThJwy/Ubo 9KJTb/RGD5iuVF1jLIoFgKgxy4yHAqzPo0dUvTXX X-Google-Smtp-Source: ABdhPJz0UAF8Im/K3ITirnNbKBmhZ4xFaQkyg32dZUvA2tJpSDdDYSz60qhtlPK3HEKOk5Ga4+gbetkNxuSOVSbijbZn X-Received: from ajr0.svl.corp.google.com ([2620:15c:2cd:203:3d06:d00f:a626:675a]) (user=axelrasmussen job=sendgmr) by 2002:ad4:5572:: with SMTP id w18mr11739166qvy.0.1617925418617; Thu, 08 Apr 2021 16:43:38 -0700 (PDT) Date: Thu, 8 Apr 2021 16:43:22 -0700 In-Reply-To: <20210408234327.624367-1-axelrasmussen@google.com> Message-Id: <20210408234327.624367-5-axelrasmussen@google.com> Mime-Version: 1.0 References: <20210408234327.624367-1-axelrasmussen@google.com> X-Mailer: git-send-email 2.31.1.295.g9ea45b61b8-goog Subject: [PATCH 4/9] userfaultfd/shmem: support UFFDIO_CONTINUE for shmem From: Axel Rasmussen To: Alexander Viro , Andrea Arcangeli , Andrew Morton , Daniel Colascione , Hugh Dickins , Jerome Glisse , Joe Perches , Lokesh Gidra , Mike Kravetz , Mike Rapoport , Peter Xu , Shaohua Li , Shuah Khan , Stephen Rothwell , Wang Qing Cc: linux-api@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mm@kvack.org, Axel Rasmussen , Brian Geffon , "Dr . David Alan Gilbert" , Mina Almasry , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org With this change, userspace can resolve a minor fault within a shmem-backed area with a UFFDIO_CONTINUE ioctl. The semantics for this match those for hugetlbfs - we look up the existing page in the page cache, and install PTEs for it. This commit introduces a new helper: mcopy_atomic_install_ptes. Why handle UFFDIO_CONTINUE for shmem in mm/userfaultfd.c, instead of in shmem.c? The existing userfault implementation only relies on shmem.c for VM_SHARED VMAs. However, minor fault handling / CONTINUE work just fine for !VM_SHARED VMAs as well. We'd prefer to handle CONTINUE for shmem in one place, regardless of shared/private (to reduce code duplication). Why add a new mcopy_atomic_install_ptes helper? A problem we have with continue is that shmem_mcopy_atomic_pte() and mcopy_atomic_pte() are *close* to what we want, but not exactly. We do want to setup the PTEs in a CONTINUE operation, but we don't want to e.g. allocate a new page, charge it (e.g. to the shmem inode), manipulate various flags, etc. Also we have the problem stated above: shmem_mcopy_atomic_pte() and mcopy_atomic_pte() both handle one-half of the problem (shared / private) continue cares about. So, introduce mcontinue_atomic_pte(), to handle all of the shmem continue cases. Introduce the helper so it doesn't duplicate code with mcopy_atomic_pte(). In a future commit, shmem_mcopy_atomic_pte() will also be modified to use this new helper. However, since this is a bigger refactor, it seems most clear to do it as a separate change. Signed-off-by: Axel Rasmussen --- mm/userfaultfd.c | 176 +++++++++++++++++++++++++++++++++++------------ 1 file changed, 131 insertions(+), 45 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 60ae22207761..a539fe18b9a7 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -48,6 +48,87 @@ struct vm_area_struct *find_dst_vma(struct mm_struct *dst_mm, return dst_vma; } +/* + * Install PTEs, to map dst_addr (within dst_vma) to page. + * + * This function handles MCOPY_ATOMIC_CONTINUE (which is always file-backed), + * whether or not dst_vma is VM_SHARED. It also handles the more general + * MCOPY_ATOMIC_NORMAL case, when dst_vma is *not* VM_SHARED (it may be file + * backed, or not). + * + * Note that MCOPY_ATOMIC_NORMAL for a VM_SHARED dst_vma is handled by + * shmem_mcopy_atomic_pte instead. + */ +static int mcopy_atomic_install_ptes(struct mm_struct *dst_mm, pmd_t *dst_pmd, + struct vm_area_struct *dst_vma, + unsigned long dst_addr, struct page *page, + bool newly_allocated, bool wp_copy) +{ + int ret; + pte_t _dst_pte, *dst_pte; + int writable; + bool vm_shared = dst_vma->vm_flags & VM_SHARED; + spinlock_t *ptl; + struct inode *inode; + pgoff_t offset, max_off; + + _dst_pte = mk_pte(page, dst_vma->vm_page_prot); + writable = dst_vma->vm_flags & VM_WRITE; + /* For private, non-anon we need CoW (don't write to page cache!) */ + if (!vma_is_anonymous(dst_vma) && !vm_shared) + writable = 0; + + if (writable || vma_is_anonymous(dst_vma)) + _dst_pte = pte_mkdirty(_dst_pte); + if (writable) { + if (wp_copy) + _dst_pte = pte_mkuffd_wp(_dst_pte); + else + _dst_pte = pte_mkwrite(_dst_pte); + } else if (vm_shared) { + /* + * Since we didn't pte_mkdirty(), mark the page dirty or it + * could be freed from under us. We could do this + * unconditionally, but doing it only if !writable is faster. + */ + set_page_dirty(page); + } + + dst_pte = pte_offset_map_lock(dst_mm, dst_pmd, dst_addr, &ptl); + + if (vma_is_shmem(dst_vma)) { + /* The shmem MAP_PRIVATE case requires checking the i_size */ + inode = dst_vma->vm_file->f_inode; + offset = linear_page_index(dst_vma, dst_addr); + max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); + ret = -EFAULT; + if (unlikely(offset >= max_off)) + goto out_unlock; + } + + ret = -EEXIST; + if (!pte_none(*dst_pte)) + goto out_unlock; + + inc_mm_counter(dst_mm, mm_counter(page)); + if (vma_is_shmem(dst_vma)) + page_add_file_rmap(page, false); + else + page_add_new_anon_rmap(page, dst_vma, dst_addr, false); + + if (newly_allocated) + lru_cache_add_inactive_or_unevictable(page, dst_vma); + + set_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte); + + /* No need to invalidate - it was non-present before */ + update_mmu_cache(dst_vma, dst_addr, dst_pte); + ret = 0; +out_unlock: + pte_unmap_unlock(dst_pte, ptl); + return ret; +} + static int mcopy_atomic_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, struct vm_area_struct *dst_vma, @@ -56,13 +137,9 @@ static int mcopy_atomic_pte(struct mm_struct *dst_mm, struct page **pagep, bool wp_copy) { - pte_t _dst_pte, *dst_pte; - spinlock_t *ptl; void *page_kaddr; int ret; struct page *page; - pgoff_t offset, max_off; - struct inode *inode; if (!*pagep) { ret = -ENOMEM; @@ -99,43 +176,12 @@ static int mcopy_atomic_pte(struct mm_struct *dst_mm, if (mem_cgroup_charge(page, dst_mm, GFP_KERNEL)) goto out_release; - _dst_pte = pte_mkdirty(mk_pte(page, dst_vma->vm_page_prot)); - if (dst_vma->vm_flags & VM_WRITE) { - if (wp_copy) - _dst_pte = pte_mkuffd_wp(_dst_pte); - else - _dst_pte = pte_mkwrite(_dst_pte); - } - - dst_pte = pte_offset_map_lock(dst_mm, dst_pmd, dst_addr, &ptl); - if (dst_vma->vm_file) { - /* the shmem MAP_PRIVATE case requires checking the i_size */ - inode = dst_vma->vm_file->f_inode; - offset = linear_page_index(dst_vma, dst_addr); - max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); - ret = -EFAULT; - if (unlikely(offset >= max_off)) - goto out_release_uncharge_unlock; - } - ret = -EEXIST; - if (!pte_none(*dst_pte)) - goto out_release_uncharge_unlock; - - inc_mm_counter(dst_mm, MM_ANONPAGES); - page_add_new_anon_rmap(page, dst_vma, dst_addr, false); - lru_cache_add_inactive_or_unevictable(page, dst_vma); - - set_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte); - - /* No need to invalidate - it was non-present before */ - update_mmu_cache(dst_vma, dst_addr, dst_pte); - - pte_unmap_unlock(dst_pte, ptl); - ret = 0; + ret = mcopy_atomic_install_ptes(dst_mm, dst_pmd, dst_vma, dst_addr, + page, true, wp_copy); + if (ret) + goto out_release; out: return ret; -out_release_uncharge_unlock: - pte_unmap_unlock(dst_pte, ptl); out_release: put_page(page); goto out; @@ -176,6 +222,41 @@ static int mfill_zeropage_pte(struct mm_struct *dst_mm, return ret; } +/* Handles UFFDIO_CONTINUE for all shmem VMAs (shared or private). */ +static int mcontinue_atomic_pte(struct mm_struct *dst_mm, + pmd_t *dst_pmd, + struct vm_area_struct *dst_vma, + unsigned long dst_addr, + bool wp_copy) +{ + struct inode *inode = file_inode(dst_vma->vm_file); + pgoff_t pgoff = linear_page_index(dst_vma, dst_addr); + struct page *page; + int ret; + + ret = shmem_getpage(inode, pgoff, &page, SGP_READ); + if (ret) + goto out; + if (!page) { + ret = -EFAULT; + goto out; + } + + ret = mcopy_atomic_install_ptes(dst_mm, dst_pmd, dst_vma, dst_addr, + page, false, wp_copy); + if (ret) + goto out_release; + + unlock_page(page); + ret = 0; +out: + return ret; +out_release: + unlock_page(page); + put_page(page); + goto out; +} + static pmd_t *mm_alloc_pmd(struct mm_struct *mm, unsigned long address) { pgd_t *pgd; @@ -415,11 +496,16 @@ static __always_inline ssize_t mfill_atomic_pte(struct mm_struct *dst_mm, unsigned long dst_addr, unsigned long src_addr, struct page **page, - bool zeropage, + enum mcopy_atomic_mode mode, bool wp_copy) { ssize_t err; + if (mode == MCOPY_ATOMIC_CONTINUE) { + return mcontinue_atomic_pte(dst_mm, dst_pmd, dst_vma, dst_addr, + wp_copy); + } + /* * The normal page fault path for a shmem will invoke the * fault, fill the hole in the file and COW it right away. The @@ -431,7 +517,7 @@ static __always_inline ssize_t mfill_atomic_pte(struct mm_struct *dst_mm, * and not in the radix tree. */ if (!(dst_vma->vm_flags & VM_SHARED)) { - if (!zeropage) + if (mode == MCOPY_ATOMIC_NORMAL) err = mcopy_atomic_pte(dst_mm, dst_pmd, dst_vma, dst_addr, src_addr, page, wp_copy); @@ -441,7 +527,8 @@ static __always_inline ssize_t mfill_atomic_pte(struct mm_struct *dst_mm, } else { VM_WARN_ON_ONCE(wp_copy); err = shmem_mcopy_atomic_pte(dst_mm, dst_pmd, dst_vma, - dst_addr, src_addr, zeropage, + dst_addr, src_addr, + mode != MCOPY_ATOMIC_NORMAL, page); } @@ -463,7 +550,6 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm, long copied; struct page *page; bool wp_copy; - bool zeropage = (mcopy_mode == MCOPY_ATOMIC_ZEROPAGE); /* * Sanitize the command parameters: @@ -526,7 +612,7 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm, if (!vma_is_anonymous(dst_vma) && !vma_is_shmem(dst_vma)) goto out_unlock; - if (mcopy_mode == MCOPY_ATOMIC_CONTINUE) + if (!vma_is_shmem(dst_vma) && mcopy_mode == MCOPY_ATOMIC_CONTINUE) goto out_unlock; /* @@ -574,7 +660,7 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm, BUG_ON(pmd_trans_huge(*dst_pmd)); err = mfill_atomic_pte(dst_mm, dst_pmd, dst_vma, dst_addr, - src_addr, &page, zeropage, wp_copy); + src_addr, &page, mcopy_mode, wp_copy); cond_resched(); if (unlikely(err == -ENOENT)) { From patchwork Thu Apr 8 23:43:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Axel Rasmussen X-Patchwork-Id: 12192633 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75027C433ED for ; Thu, 8 Apr 2021 23:43:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 536B161151 for ; Thu, 8 Apr 2021 23:43:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233061AbhDHXn5 (ORCPT ); Thu, 8 Apr 2021 19:43:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233058AbhDHXnz (ORCPT ); Thu, 8 Apr 2021 19:43:55 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 15617C0613D7 for ; Thu, 8 Apr 2021 16:43:41 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id w7so3594906ybq.4 for ; Thu, 08 Apr 2021 16:43:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=5FNNG5JC3AqKoqOHTe9pY2QoEFbztQ33Z0RnEe1qU+U=; b=MVh7wJOhgi98eYTGizGpz0fXWcVL+XnU2Bk/6XennU2H8x3R8rGMIne2hUKtCj7Utt eSnfsyszk2SbhoLSlZrWVVaFCCE7eaBnQ5J6l6ckDfRSG9fS/h8IJOb2SOBzCo+3dzXO OI5agAhrI80ggOIT0sOcuShvr7PEtN1+4RgvSo8SRj1RNTLRThXzJHJy50V71YkNGiD9 f7ZtDNsM7B929F/Nlvn4FMe3RNuU30g+glCgh228xH0CsSWOBiGHHZU6aC8fvpXBG+A4 YDSpjEpnYdXYiLnrlVIbzlruOiCQaHz9PQyrr0dhO27DBYS4e6MULLi4YA+jYY3tlHTb Wg/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=5FNNG5JC3AqKoqOHTe9pY2QoEFbztQ33Z0RnEe1qU+U=; b=fQFznXoMw5yco614Ffvoqns7rNvgBLmCa3z12ZCAOUSJJOlAkGeFZaKGSs8u1Iowxs 8sxcXBDS2mbrQ4pV0yeJxwvvPQbzY8MZG4x/bmkkP6ET1NC1kjpVwFF0d6LWhS/0p/Ug mhO6/RPuI3X49p6tsNd7nRG2Too+iNI4zgBg5YGM2C3+BIRWA4bpHRX5AefyIVzb9naM OaBN43xU2nBYNnNnq9uTDZL0+kw/IsrPyQXDgGBC+RfPD8eKRaa2HC1yaqM4otfX/jTv t0s5lk9BLCp1f13G9hfHoj13b49yR+Lpmn3JID0rIxXbGezXbrVo4+ckuJ5Jvt4G0wb3 2b2w== X-Gm-Message-State: AOAM530DhHQzNasgktIA6IHvy1hKsu47k6g7dswOI6BkVkU/tEyAVSLT CtZ8MVukhBaB/6eZwiGN0IwLpph95lPwODlJMxD6 X-Google-Smtp-Source: ABdhPJyHO2FVkr1CboGpSzUwiSg/3NdbiftgLdqxWDrb4ETSP7JCT+1vBZ/v/JbVEtqQcUZRebwBFwQE6Qi+5WAlsHtN X-Received: from ajr0.svl.corp.google.com ([2620:15c:2cd:203:3d06:d00f:a626:675a]) (user=axelrasmussen job=sendgmr) by 2002:a25:ad89:: with SMTP id z9mr15823267ybi.267.1617925420343; Thu, 08 Apr 2021 16:43:40 -0700 (PDT) Date: Thu, 8 Apr 2021 16:43:23 -0700 In-Reply-To: <20210408234327.624367-1-axelrasmussen@google.com> Message-Id: <20210408234327.624367-6-axelrasmussen@google.com> Mime-Version: 1.0 References: <20210408234327.624367-1-axelrasmussen@google.com> X-Mailer: git-send-email 2.31.1.295.g9ea45b61b8-goog Subject: [PATCH 5/9] userfaultfd/selftests: use memfd_create for shmem test type From: Axel Rasmussen To: Alexander Viro , Andrea Arcangeli , Andrew Morton , Daniel Colascione , Hugh Dickins , Jerome Glisse , Joe Perches , Lokesh Gidra , Mike Kravetz , Mike Rapoport , Peter Xu , Shaohua Li , Shuah Khan , Stephen Rothwell , Wang Qing Cc: linux-api@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mm@kvack.org, Axel Rasmussen , Brian Geffon , "Dr . David Alan Gilbert" , Mina Almasry , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This is a preparatory commit. In the future, we want to be able to setup alias mappings for area_src and area_dst in the shmem test, like we do in the hugetlb_shared test. With a VMA obtained via mmap(MAP_ANONYMOUS | MAP_SHARED), it isn't clear how to do this. So, mmap() with an fd, so we can create alias mappings. Use memfd_create instead of actually passing in a tmpfs path like hugetlb does, since it's more convenient / simpler to run, and works just as well. Future commits will: 1. Setup the alias mappings. 2. Extend our tests to actually take advantage of this, to test new userfaultfd behavior being introduced in this series. Also, a small fix in the area we're changing: when the hugetlb setup fails in main(), pass in the right argv[] so we actually print out the hugetlb file path. Signed-off-by: Axel Rasmussen --- tools/testing/selftests/vm/userfaultfd.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/vm/userfaultfd.c b/tools/testing/selftests/vm/userfaultfd.c index 310fc617c383..b0af88b258d7 100644 --- a/tools/testing/selftests/vm/userfaultfd.c +++ b/tools/testing/selftests/vm/userfaultfd.c @@ -85,6 +85,7 @@ static bool test_uffdio_wp = false; static bool test_uffdio_minor = false; static bool map_shared; +static int shm_fd; static int huge_fd; static char *huge_fd_off0; static unsigned long long *count_verify; @@ -278,10 +279,13 @@ static void shmem_release_pages(char *rel_area) static void shmem_allocate_area(void **alloc_area) { + unsigned long offset = + alloc_area == (void **)&area_src ? 0 : nr_pages * page_size; + *alloc_area = mmap(NULL, nr_pages * page_size, PROT_READ | PROT_WRITE, - MAP_ANONYMOUS | MAP_SHARED, -1, 0); + MAP_SHARED, shm_fd, offset); if (*alloc_area == MAP_FAILED) - err("shared memory mmap failed"); + err("mmap of memfd failed"); } struct uffd_test_ops { @@ -1446,6 +1450,16 @@ int main(int argc, char **argv) err("Open of %s failed", argv[4]); if (ftruncate(huge_fd, 0)) err("ftruncate %s to size 0 failed", argv[4]); + } else if (test_type == TEST_SHMEM) { + shm_fd = memfd_create(argv[0], 0); + if (shm_fd < 0) + err("memfd_create"); + if (ftruncate(shm_fd, nr_pages * page_size * 2)) + err("ftruncate"); + if (fallocate(shm_fd, + FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, + nr_pages * page_size * 2)) + err("fallocate"); } printf("nr_pages: %lu, nr_pages_per_cpu: %lu\n", nr_pages, nr_pages_per_cpu); From patchwork Thu Apr 8 23:43:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Axel Rasmussen X-Patchwork-Id: 12192635 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2061FC43611 for ; Thu, 8 Apr 2021 23:43:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E86E961151 for ; Thu, 8 Apr 2021 23:43:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233108AbhDHXoA (ORCPT ); Thu, 8 Apr 2021 19:44:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60660 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233019AbhDHXnz (ORCPT ); Thu, 8 Apr 2021 19:43:55 -0400 Received: from mail-qk1-x74a.google.com (mail-qk1-x74a.google.com [IPv6:2607:f8b0:4864:20::74a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D5252C0613DD for ; Thu, 8 Apr 2021 16:43:42 -0700 (PDT) Received: by mail-qk1-x74a.google.com with SMTP id h19so2318085qkk.4 for ; Thu, 08 Apr 2021 16:43:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=y+JvwE75GgC5IEnxBEL5JZdrd4moWn3oGAIMC9oxZzA=; b=b/Wf0aWC0cG4uYtiJ9VGNFi97zqNT9Ig+q0z8hJw5s/xOlRJa684gjh0V8qqomB2hT WZjORDQCX/pDQ55Cwv1jH16lar+FcyZ2vaPUk0rh2HvTC7b8YB5MmaF//T3tfeb+x2td JMnCPZB4qJdEjs7H2ZkHdPqpk7dnjRmmNhaw+6Ew6iWTJj8i+SIvUDgPZOmKroyePa89 1NgNFkaKLMIY1jrnTGDVjJe8QDstJmISDtvNakCMHgwf6XY8hHdhFknhKyzHFyDdmLUS ypL/ntHQIoQFWGz93HcNIfHYekI5Lx0IpVSxBIjd97QoazXIugagpeCJW95g1Zqq9kuA ew7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=y+JvwE75GgC5IEnxBEL5JZdrd4moWn3oGAIMC9oxZzA=; b=CQENcj/pYAJzi0xJsHf1UUl+SO6q4GPhMiXlu59XbQM895vgyjTgkjoHWlNhsrYiH2 yDQZFQ8mrBVJ4TMTMZTvFK/erM/03QCzywi0fi/4zRrSkEYM0KRE+Z/a9IRTP4QKkOKu KQ32nd4H2c+hNXTk4+TfEh6lexvZXWbXDBAu7xxhDnOe7movIBRiMfm7IZ5b6nWefbJv CtO3CwktlQh/lZsPj4lgVHi294ywCerX5/C5EGxAJnPB69GFUw9ljFhgTafuQs25f2Ez 70Da3A2+SyosBtkmxzjqOy+XdknZY7/JC8hU1ovDxtvJhbxw3iZsghGCnSK0snfSUObD Zx5w== X-Gm-Message-State: AOAM530usODbMrjrvSDUWk3pASGQ3P3EVUuDCzHq1ZeSarqz3IFNa6rL A2H/a9YQMRLoNvPsbzqaHE4OzLsF8Lve3DDj5WAC X-Google-Smtp-Source: ABdhPJxtFO9U8LecU3todxiWF7PRcHx+2Dtw3lc++CgfpcE8LU8WsLyqSI3lld0UepwP7rjyD+Pf92LrC1DzWayxE4DM X-Received: from ajr0.svl.corp.google.com ([2620:15c:2cd:203:3d06:d00f:a626:675a]) (user=axelrasmussen job=sendgmr) by 2002:a05:6214:7e4:: with SMTP id bp4mr11257020qvb.5.1617925422072; Thu, 08 Apr 2021 16:43:42 -0700 (PDT) Date: Thu, 8 Apr 2021 16:43:24 -0700 In-Reply-To: <20210408234327.624367-1-axelrasmussen@google.com> Message-Id: <20210408234327.624367-7-axelrasmussen@google.com> Mime-Version: 1.0 References: <20210408234327.624367-1-axelrasmussen@google.com> X-Mailer: git-send-email 2.31.1.295.g9ea45b61b8-goog Subject: [PATCH 6/9] userfaultfd/selftests: create alias mappings in the shmem test From: Axel Rasmussen To: Alexander Viro , Andrea Arcangeli , Andrew Morton , Daniel Colascione , Hugh Dickins , Jerome Glisse , Joe Perches , Lokesh Gidra , Mike Kravetz , Mike Rapoport , Peter Xu , Shaohua Li , Shuah Khan , Stephen Rothwell , Wang Qing Cc: linux-api@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mm@kvack.org, Axel Rasmussen , Brian Geffon , "Dr . David Alan Gilbert" , Mina Almasry , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Previously, we just allocated two shm areas: area_src and area_dst. With this commit, change this so we also allocate area_src_alias, and area_dst_alias. area_*_alias and area_* (respectively) point to the same underlying physical pages, but are different VMAs. In a future commit in this series, we'll leverage this setup to exercise minor fault handling support for shmem, just like we do in the hugetlb_shared test. Signed-off-by: Axel Rasmussen --- tools/testing/selftests/vm/userfaultfd.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/vm/userfaultfd.c b/tools/testing/selftests/vm/userfaultfd.c index b0af88b258d7..4b49b2cf9819 100644 --- a/tools/testing/selftests/vm/userfaultfd.c +++ b/tools/testing/selftests/vm/userfaultfd.c @@ -279,13 +279,29 @@ static void shmem_release_pages(char *rel_area) static void shmem_allocate_area(void **alloc_area) { - unsigned long offset = - alloc_area == (void **)&area_src ? 0 : nr_pages * page_size; + void *area_alias = NULL; + bool is_src = alloc_area == (void **)&area_src; + unsigned long offset = is_src ? 0 : nr_pages * page_size; *alloc_area = mmap(NULL, nr_pages * page_size, PROT_READ | PROT_WRITE, MAP_SHARED, shm_fd, offset); if (*alloc_area == MAP_FAILED) err("mmap of memfd failed"); + + area_alias = mmap(NULL, nr_pages * page_size, PROT_READ | PROT_WRITE, + MAP_SHARED, shm_fd, offset); + if (area_alias == MAP_FAILED) + err("mmap of memfd alias failed"); + + if (is_src) + area_src_alias = area_alias; + else + area_dst_alias = area_alias; +} + +static void shmem_alias_mapping(__u64 *start, size_t len, unsigned long offset) +{ + *start = (unsigned long)area_dst_alias + offset; } struct uffd_test_ops { @@ -315,7 +331,7 @@ static struct uffd_test_ops shmem_uffd_test_ops = { .expected_ioctls = SHMEM_EXPECTED_IOCTLS, .allocate_area = shmem_allocate_area, .release_pages = shmem_release_pages, - .alias_mapping = noop_alias_mapping, + .alias_mapping = shmem_alias_mapping, }; static struct uffd_test_ops hugetlb_uffd_test_ops = { From patchwork Thu Apr 8 23:43:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Axel Rasmussen X-Patchwork-Id: 12192637 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DAAA2C43616 for ; Thu, 8 Apr 2021 23:43:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B52AF61151 for ; Thu, 8 Apr 2021 23:43:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233112AbhDHXoB (ORCPT ); Thu, 8 Apr 2021 19:44:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60642 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233043AbhDHXn4 (ORCPT ); Thu, 8 Apr 2021 19:43:56 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 81110C061760 for ; Thu, 8 Apr 2021 16:43:44 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id y13so3532879ybk.20 for ; Thu, 08 Apr 2021 16:43:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=k822bRDmtTrziFIcDdTViBhFB3AGLYRYm2w1YLs2VOg=; b=f4YMCWG4vKdVJmmBUAtXNIwSMYRV3ij0I4fC+cvpCp9uf6F11PZFl41XLH9pf3d0QP 3k9gr0FVzKIbK2aT2PEybMZT+cE6mxavym4u6jBDsjVyecO1VZKOn2E00+nL603MQts7 9TvHWrf9UnJLUkbhxxdRjlQzQwbA4Ntwiq4bGUgRHKmrK04smKF58bQNCP+GinAuyRY8 ijRkQjaCEIquZi8K5IpZcQjUH4VbmTbbP3xpzY1EkvAd99RTSSLYMYWaVUhahN5cLaye LFbuszRDnYEXg4AzIR74gSQojrU1VOwwdAqsE9bXFUkoUDMMLb9CME7sdtdE/1XnifpI WzEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=k822bRDmtTrziFIcDdTViBhFB3AGLYRYm2w1YLs2VOg=; b=f+qSKpJHZOXfduw3I9eVGzi2jn+4KDI6j4fNo1q2Zh0dL3gg4US5HBMbs1KAhPZfMo GIC/wNkLEsRQn42R719LUolc1grkqzwVKxeKm/7QwmPDiAM1RfY6iRF0YJmzxTT6SI9M pES1UKlWGKQLmxg1pNIdW1zJ2le6ZDpxXFCpdsF1o9m8zETUgKn1YQIX2M9Z1YtShZMB sdF8xnw4fCVIP3WmoIP6zcPHsQgEZNhmM4zJ65D/847kKd5s9nR4ELk5f9zBRP1Y6g5I 2IQwnjILEHUd78JD6HiWZcjJKPC/xMXajwP2HRQBtkb5LdXwQwsrFvCxnMao5MzPrw62 uDog== X-Gm-Message-State: AOAM531Etrep8g2eWI1g0BGXCH14j09lrWvsGlJUEhnTgGCcW08Qu5eh /7pQtyVr9NaiiRdn1zE1ElOJ1wqStndEQP3ynK3r X-Google-Smtp-Source: ABdhPJye+kHIsbkEGD8DVxtB2/PLogEU4neb7Zlc4c9yBH3sz33SCcoh3TaGKUxFZuLR6pRwrWvarcuMIkcjKH787diS X-Received: from ajr0.svl.corp.google.com ([2620:15c:2cd:203:3d06:d00f:a626:675a]) (user=axelrasmussen job=sendgmr) by 2002:a5b:802:: with SMTP id x2mr10296897ybp.28.1617925423799; Thu, 08 Apr 2021 16:43:43 -0700 (PDT) Date: Thu, 8 Apr 2021 16:43:25 -0700 In-Reply-To: <20210408234327.624367-1-axelrasmussen@google.com> Message-Id: <20210408234327.624367-8-axelrasmussen@google.com> Mime-Version: 1.0 References: <20210408234327.624367-1-axelrasmussen@google.com> X-Mailer: git-send-email 2.31.1.295.g9ea45b61b8-goog Subject: [PATCH 7/9] userfaultfd/selftests: reinitialize test context in each test From: Axel Rasmussen To: Alexander Viro , Andrea Arcangeli , Andrew Morton , Daniel Colascione , Hugh Dickins , Jerome Glisse , Joe Perches , Lokesh Gidra , Mike Kravetz , Mike Rapoport , Peter Xu , Shaohua Li , Shuah Khan , Stephen Rothwell , Wang Qing Cc: linux-api@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mm@kvack.org, Axel Rasmussen , Brian Geffon , "Dr . David Alan Gilbert" , Mina Almasry , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Currently, the context (fds, mmap-ed areas, etc.) are global. Each test mutates this state in some way, in some cases really "clobbering it" (e.g., the events test mremap-ing area_dst over the top of area_src, or the minor faults tests overwriting the count_verify values in the test areas). We run the tests in a particular order, each test is careful to make the right assumptions about its starting state, etc. But, this is fragile. It's better for a test's success or failure to not depend on what some other prior test case did to the global state. To that end, clear and reinitialize the test context at the start of each test case, so whatever prior test cases did doesn't affect future tests. This is particularly relevant to this series because the events test's mremap of area_dst screws up assumptions the minor fault test was relying on. This wasn't a problem for hugetlb, as we don't mremap in that case. Signed-off-by: Axel Rasmussen --- tools/testing/selftests/vm/userfaultfd.c | 221 +++++++++++++---------- 1 file changed, 129 insertions(+), 92 deletions(-) diff --git a/tools/testing/selftests/vm/userfaultfd.c b/tools/testing/selftests/vm/userfaultfd.c index 4b49b2cf9819..9b032cfdc262 100644 --- a/tools/testing/selftests/vm/userfaultfd.c +++ b/tools/testing/selftests/vm/userfaultfd.c @@ -89,7 +89,8 @@ static int shm_fd; static int huge_fd; static char *huge_fd_off0; static unsigned long long *count_verify; -static int uffd, uffd_flags, finished, *pipefd; +static int uffd = -1; +static int uffd_flags, finished, *pipefd; static char *area_src, *area_src_alias, *area_dst, *area_dst_alias; static char *zeropage; pthread_attr_t attr; @@ -343,6 +344,124 @@ static struct uffd_test_ops hugetlb_uffd_test_ops = { static struct uffd_test_ops *uffd_test_ops; +static int userfaultfd_open(uint64_t *features) +{ + struct uffdio_api uffdio_api; + + uffd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK); + if (uffd < 0) + err("userfaultfd syscall not available in this kernel"); + uffd_flags = fcntl(uffd, F_GETFD, NULL); + + uffdio_api.api = UFFD_API; + uffdio_api.features = *features; + if (ioctl(uffd, UFFDIO_API, &uffdio_api)) + err("UFFDIO_API failed.\nPlease make sure to " + "run with either root or ptrace capability."); + if (uffdio_api.api != UFFD_API) + err("UFFDIO_API error: %" PRIu64, (uint64_t)uffdio_api.api); + + *features = uffdio_api.features; + return 0; +} + +static int uffd_test_ctx_init_ext(uint64_t *features) +{ + unsigned long nr, cpu; + + uffd_test_ops->allocate_area((void **)&area_src); + if (!area_src) + return 1; + uffd_test_ops->allocate_area((void **)&area_dst); + if (!area_dst) + return 1; + + if (uffd_test_ops->release_pages(area_src)) + return 1; + + if (uffd_test_ops->release_pages(area_dst)) + return 1; + + if (userfaultfd_open(features)) + return 1; + + count_verify = malloc(nr_pages * sizeof(unsigned long long)); + if (!count_verify) + err("count_verify"); + + for (nr = 0; nr < nr_pages; nr++) { + *area_mutex(area_src, nr) = + (pthread_mutex_t)PTHREAD_MUTEX_INITIALIZER; + count_verify[nr] = *area_count(area_src, nr) = 1; + /* + * In the transition between 255 to 256, powerpc will + * read out of order in my_bcmp and see both bytes as + * zero, so leave a placeholder below always non-zero + * after the count, to avoid my_bcmp to trigger false + * positives. + */ + *(area_count(area_src, nr) + 1) = 1; + } + + pipefd = malloc(sizeof(int) * nr_cpus * 2); + if (!pipefd) + err("pipefd"); + for (cpu = 0; cpu < nr_cpus; cpu++) + if (pipe2(&pipefd[cpu * 2], O_CLOEXEC | O_NONBLOCK)) + err("pipe"); + + return 0; +} + +static inline int uffd_test_ctx_init(uint64_t features) +{ + return uffd_test_ctx_init_ext(&features); +} + +static inline int munmap_area(void **area) +{ + if (*area) + if (munmap(*area, nr_pages * page_size)) + err("munmap"); + + *area = NULL; + return 0; +} + +static int uffd_test_ctx_clear(void) +{ + int ret = 0; + size_t i; + + if (pipefd) { + for (i = 0; i < nr_cpus * 2; ++i) { + if (close(pipefd[i])) + err("close pipefd"); + } + free(pipefd); + pipefd = NULL; + } + + if (count_verify) { + free(count_verify); + count_verify = NULL; + } + + if (uffd != -1) { + if (close(uffd)) + err("close uffd"); + uffd = -1; + } + + huge_fd_off0 = NULL; + ret |= munmap_area((void **)&area_src); + ret |= munmap_area((void **)&area_src_alias); + ret |= munmap_area((void **)&area_dst); + ret |= munmap_area((void **)&area_dst_alias); + + return ret; +} + static int my_bcmp(char *str1, char *str2, size_t n) { unsigned long i; @@ -727,40 +846,6 @@ static int stress(struct uffd_stats *uffd_stats) return 0; } -static int userfaultfd_open_ext(uint64_t *features) -{ - struct uffdio_api uffdio_api; - - uffd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY); - if (uffd < 0) { - fprintf(stderr, - "userfaultfd syscall not available in this kernel\n"); - return 1; - } - uffd_flags = fcntl(uffd, F_GETFD, NULL); - - uffdio_api.api = UFFD_API; - uffdio_api.features = *features; - if (ioctl(uffd, UFFDIO_API, &uffdio_api)) { - fprintf(stderr, "UFFDIO_API failed.\nPlease make sure to " - "run with either root or ptrace capability.\n"); - return 1; - } - if (uffdio_api.api != UFFD_API) { - fprintf(stderr, "UFFDIO_API error: %" PRIu64 "\n", - (uint64_t)uffdio_api.api); - return 1; - } - - *features = uffdio_api.features; - return 0; -} - -static int userfaultfd_open(uint64_t features) -{ - return userfaultfd_open_ext(&features); -} - sigjmp_buf jbuf, *sigbuf; static void sighndl(int sig, siginfo_t *siginfo, void *ptr) @@ -869,6 +954,8 @@ static int faulting_process(int signal_test) MREMAP_MAYMOVE | MREMAP_FIXED, area_src); if (area_dst == MAP_FAILED) err("mremap"); + /* Reset area_src since we just clobbered it */ + area_src = NULL; for (; nr < nr_pages; nr++) { count = *area_count(area_dst, nr); @@ -962,11 +1049,9 @@ static int userfaultfd_zeropage_test(void) printf("testing UFFDIO_ZEROPAGE: "); fflush(stdout); - if (uffd_test_ops->release_pages(area_dst)) + if (uffd_test_ctx_clear() || uffd_test_ctx_init(0)) return 1; - if (userfaultfd_open(0)) - return 1; uffdio_register.range.start = (unsigned long) area_dst; uffdio_register.range.len = nr_pages * page_size; uffdio_register.mode = UFFDIO_REGISTER_MODE_MISSING; @@ -983,7 +1068,6 @@ static int userfaultfd_zeropage_test(void) if (my_bcmp(area_dst, zeropage, page_size)) err("zeropage is not zero"); - close(uffd); printf("done.\n"); return 0; } @@ -1001,13 +1085,11 @@ static int userfaultfd_events_test(void) printf("testing events (fork, remap, remove): "); fflush(stdout); - if (uffd_test_ops->release_pages(area_dst)) - return 1; - features = UFFD_FEATURE_EVENT_FORK | UFFD_FEATURE_EVENT_REMAP | UFFD_FEATURE_EVENT_REMOVE; - if (userfaultfd_open(features)) + if (uffd_test_ctx_clear() || uffd_test_ctx_init(features)) return 1; + fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); uffdio_register.range.start = (unsigned long) area_dst; @@ -1040,8 +1122,6 @@ static int userfaultfd_events_test(void) if (pthread_join(uffd_mon, NULL)) return 1; - close(uffd); - uffd_stats_report(&stats, 1); return stats.missing_faults != nr_pages; @@ -1061,12 +1141,10 @@ static int userfaultfd_sig_test(void) printf("testing signal delivery: "); fflush(stdout); - if (uffd_test_ops->release_pages(area_dst)) - return 1; - features = UFFD_FEATURE_EVENT_FORK|UFFD_FEATURE_SIGBUS; - if (userfaultfd_open(features)) + if (uffd_test_ctx_clear() || uffd_test_ctx_init(features)) return 1; + fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); uffdio_register.range.start = (unsigned long) area_dst; @@ -1128,10 +1206,7 @@ static int userfaultfd_minor_test(void) printf("testing minor faults: "); fflush(stdout); - if (uffd_test_ops->release_pages(area_dst)) - return 1; - - if (userfaultfd_open_ext(&features)) + if (uffd_test_ctx_clear() || uffd_test_ctx_init_ext(&features)) return 1; /* If kernel reports the feature isn't supported, skip the test. */ if (!(features & UFFD_FEATURE_MINOR_HUGETLBFS)) { @@ -1186,8 +1261,6 @@ static int userfaultfd_minor_test(void) if (pthread_join(uffd_mon, NULL)) return 1; - close(uffd); - uffd_stats_report(&stats, 1); return stats.missing_faults != 0 || stats.minor_faults != nr_pages; @@ -1199,44 +1272,11 @@ static int userfaultfd_stress(void) char *tmp_area; unsigned long nr; struct uffdio_register uffdio_register; - unsigned long cpu; struct uffd_stats uffd_stats[nr_cpus]; - uffd_test_ops->allocate_area((void **)&area_src); - if (!area_src) - return 1; - uffd_test_ops->allocate_area((void **)&area_dst); - if (!area_dst) - return 1; - - if (userfaultfd_open(0)) + if (uffd_test_ctx_init(0)) return 1; - count_verify = malloc(nr_pages * sizeof(unsigned long long)); - if (!count_verify) - err("count_verify"); - - for (nr = 0; nr < nr_pages; nr++) { - *area_mutex(area_src, nr) = (pthread_mutex_t) - PTHREAD_MUTEX_INITIALIZER; - count_verify[nr] = *area_count(area_src, nr) = 1; - /* - * In the transition between 255 to 256, powerpc will - * read out of order in my_bcmp and see both bytes as - * zero, so leave a placeholder below always non-zero - * after the count, to avoid my_bcmp to trigger false - * positives. - */ - *(area_count(area_src, nr) + 1) = 1; - } - - pipefd = malloc(sizeof(int) * nr_cpus * 2); - if (!pipefd) - err("pipefd"); - for (cpu = 0; cpu < nr_cpus; cpu++) - if (pipe2(&pipefd[cpu*2], O_CLOEXEC | O_NONBLOCK)) - err("pipe"); - if (posix_memalign(&area, page_size, page_size)) err("out of memory"); zeropage = area; @@ -1356,9 +1396,6 @@ static int userfaultfd_stress(void) uffd_stats_report(uffd_stats, nr_cpus); } - if (close(uffd)) - err("close uffd"); - return userfaultfd_zeropage_test() || userfaultfd_sig_test() || userfaultfd_events_test() || userfaultfd_minor_test(); } From patchwork Thu Apr 8 23:43:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Axel Rasmussen X-Patchwork-Id: 12192639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E66BEC433ED for ; Thu, 8 Apr 2021 23:43:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CA38E61159 for ; Thu, 8 Apr 2021 23:43:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233087AbhDHXoJ (ORCPT ); Thu, 8 Apr 2021 19:44:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60682 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233090AbhDHXn7 (ORCPT ); Thu, 8 Apr 2021 19:43:59 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5E050C061765 for ; Thu, 8 Apr 2021 16:43:46 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id y13so3532915ybk.20 for ; Thu, 08 Apr 2021 16:43:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=WSNoP8c3GVSSHFh1g2WvnJRaFbPgXevhOBLl09IZVIU=; b=GTSRqeAtpIM5Fb79iJUqC7M8Tk0OMkIKgRA91v9QUk8XPSho24dNwSGb+x2suMVH7g ZUyFpBL/CCGtfiYGoUiEVuGEjdCqDoTMRa8rLnvH5LCdhI4fyiCDRIAZdiVwEGmaLs4l oHnXarP2qV+AwK4CixPK5vJXrCZVDWiog4QDfK8PDfikecEOaLU9aAaKeYilLpftkQm0 HMWI958gN/YKEwmMHf6YOAZVpR4R9u1icRa5OMpLrjjI+4/9uuof+lT+Q5hVJAiV2Eks US5fO+lAPes914Qy6GCFTlcPyAKC035xNlr9iS7vnabPVnkD9ejloGRXEb3zHFILuVFu S8qQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=WSNoP8c3GVSSHFh1g2WvnJRaFbPgXevhOBLl09IZVIU=; b=flpg25r9XsGHAJ629M45aC/AazJ2VoX+3TICX9tTkxuxh2bChFEzDfvv++yAd6nLER hXlBIfARF8e5ZHZRVwLhADF3hWz//0BeYCgYDdXTxStN3bbQpNzhtzIMTb4LjQb3TW4Y +4R5Gfy7foIgl0k8TVI3H8teatKOZDBqtuBh5Aq+Tzg7FXJTEnmI90ATyTuJft3TWXLi zRS8r8CuuZc04D9OH9ZkiQpNHF5Og/X9GmyJWx5muMphpGgr75OB29BwXoT6xyiGPSRH xQ/eA//lB3aGBfRCGNYYBFNJv+kqkCjSgk7Qd49liOvGM83eaY0lDnHYIBmEUzRkPfn4 qISA== X-Gm-Message-State: AOAM530lRjEDg6NS2++Hk0vVdEImiwGXpx9pPuEazm5lJVsH0YIwJg9D +gHy7IADQNFI6ybQX64U9c84HXusnsSEKljv6i1A X-Google-Smtp-Source: ABdhPJwvO+MZlsHbO+qMgNHWdl/g5ZOBZCUrI2P4iJAvMCv9bDmGUW1aJu5V83dw1rhmvOWNIncWTDrBaDb2rLqiRPTk X-Received: from ajr0.svl.corp.google.com ([2620:15c:2cd:203:3d06:d00f:a626:675a]) (user=axelrasmussen job=sendgmr) by 2002:a25:3304:: with SMTP id z4mr16256601ybz.330.1617925425618; Thu, 08 Apr 2021 16:43:45 -0700 (PDT) Date: Thu, 8 Apr 2021 16:43:26 -0700 In-Reply-To: <20210408234327.624367-1-axelrasmussen@google.com> Message-Id: <20210408234327.624367-9-axelrasmussen@google.com> Mime-Version: 1.0 References: <20210408234327.624367-1-axelrasmussen@google.com> X-Mailer: git-send-email 2.31.1.295.g9ea45b61b8-goog Subject: [PATCH 8/9] userfaultfd/selftests: exercise minor fault handling shmem support From: Axel Rasmussen To: Alexander Viro , Andrea Arcangeli , Andrew Morton , Daniel Colascione , Hugh Dickins , Jerome Glisse , Joe Perches , Lokesh Gidra , Mike Kravetz , Mike Rapoport , Peter Xu , Shaohua Li , Shuah Khan , Stephen Rothwell , Wang Qing Cc: linux-api@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mm@kvack.org, Axel Rasmussen , Brian Geffon , "Dr . David Alan Gilbert" , Mina Almasry , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Enable test_uffdio_minor for test_type == TEST_SHMEM, and modify the test slightly to pass in / check for the right feature flags. Signed-off-by: Axel Rasmussen --- tools/testing/selftests/vm/userfaultfd.c | 29 ++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/vm/userfaultfd.c b/tools/testing/selftests/vm/userfaultfd.c index 9b032cfdc262..640d0a2d107d 100644 --- a/tools/testing/selftests/vm/userfaultfd.c +++ b/tools/testing/selftests/vm/userfaultfd.c @@ -488,6 +488,7 @@ static void wp_range(int ufd, __u64 start, __u64 len, bool wp) static void continue_range(int ufd, __u64 start, __u64 len) { struct uffdio_continue req; + int ret; req.range.start = start; req.range.len = len; @@ -496,6 +497,17 @@ static void continue_range(int ufd, __u64 start, __u64 len) if (ioctl(ufd, UFFDIO_CONTINUE, &req)) err("UFFDIO_CONTINUE failed for address 0x%" PRIx64, (uint64_t)start); + + /* + * Error handling within the kernel for continue is subtly different + * from copy or zeropage, so it may be a source of bugs. Trigger an + * error (-EEXIST) on purpose, to verify doing so doesn't cause a BUG. + */ + req.mapped = 0; + ret = ioctl(ufd, UFFDIO_CONTINUE, &req); + if (ret >= 0 || req.mapped != -EEXIST) + err("failed to exercise UFFDIO_CONTINUE error handling, ret=%d, mapped=%" PRId64, + ret, req.mapped); } static void *locking_thread(void *arg) @@ -1198,7 +1210,7 @@ static int userfaultfd_minor_test(void) void *expected_page; char c; struct uffd_stats stats = { 0 }; - uint64_t features = UFFD_FEATURE_MINOR_HUGETLBFS; + uint64_t req_features, features_out; if (!test_uffdio_minor) return 0; @@ -1206,10 +1218,18 @@ static int userfaultfd_minor_test(void) printf("testing minor faults: "); fflush(stdout); - if (uffd_test_ctx_clear() || uffd_test_ctx_init_ext(&features)) + if (test_type == TEST_HUGETLB) + req_features = UFFD_FEATURE_MINOR_HUGETLBFS; + else if (test_type == TEST_SHMEM) + req_features = UFFD_FEATURE_MINOR_SHMEM; + else + return 1; + + features_out = req_features; + if (uffd_test_ctx_clear() || uffd_test_ctx_init_ext(&features_out)) return 1; - /* If kernel reports the feature isn't supported, skip the test. */ - if (!(features & UFFD_FEATURE_MINOR_HUGETLBFS)) { + /* If kernel reports required features aren't supported, skip test. */ + if ((features_out & req_features) != req_features) { printf("skipping test due to lack of feature support\n"); fflush(stdout); return 0; @@ -1444,6 +1464,7 @@ static void set_test_type(const char *type) map_shared = true; test_type = TEST_SHMEM; uffd_test_ops = &shmem_uffd_test_ops; + test_uffdio_minor = true; } else { err("Unknown test type: %s", type); } From patchwork Thu Apr 8 23:43:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Axel Rasmussen X-Patchwork-Id: 12192641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4931C43460 for ; Thu, 8 Apr 2021 23:44:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B33BC61175 for ; Thu, 8 Apr 2021 23:44:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233150AbhDHXoQ (ORCPT ); Thu, 8 Apr 2021 19:44:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60660 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233006AbhDHXoA (ORCPT ); Thu, 8 Apr 2021 19:44:00 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4BCE9C0613DA for ; Thu, 8 Apr 2021 16:43:48 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 10so3524451ybu.18 for ; Thu, 08 Apr 2021 16:43:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=SA+DqFU+tgMShZZq9qaZAZYTg8gGJxsmK6MPTeE2ry8=; b=uKP3BtZibz2XAPKw4vvNyzwrxvh/ZYYDs9RxYJl2+CpRzB2Z5KFuRqca9wKV9UKTYa UqAJwFGddPUBBBbAxB08uVP/Z+1Kx9WTvOs6zi4Q0Y48GFxPz46Pywb/cXM1zwRYuziU 5AfjajljiCrur5A/ggK6zMwzsmHt4CpBD4/kxC0eTFVkxrMUzI3XP/NzU011N5oh5l8E mEZoflUq/RpYdw6cGlqAAj+0oS4q8rhNcPPEG59ModK2vJ5jEC3dzmv2DaYivMTEtYP2 O0Qz0Jz8dkwc7Jo2IIXs0AyRKjNLh/dHuakZHeaFgxVcK8Oa6UNTqyOYWP2RqfXmVT6Q oAhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=SA+DqFU+tgMShZZq9qaZAZYTg8gGJxsmK6MPTeE2ry8=; b=GIkLSRCguazgPX2nwpdjIrxaGXEtaMau5A1jO8M0/qbHpaycsptkoRapaN03LNbNat pt0Hbh+ZLP4VPDYj3wx9cqLFM1XIDtsvNnCRCRQXc3eyuLwnxPvHl/KDaqwjsT7lnVkt xD33YDqcEiknWfppXcHvk9pClAMyiGpGdB0wPc103aaCr6ylpTTg3SaKICmUuxd4ZO1d O25KAmwBHWmCgzZAy5jxPpGSmXrYO6YrfB3VJQ3tFUgry1eRd9957PcL9KeGhQk6GG7F bEvE4POtbr/aGX+pZUSpPkY7WInwu4Xe/Nhcqc1jQzCG1yo6C0F+VKZqfC3UCj2iBSnA k2Ww== X-Gm-Message-State: AOAM532kZNITMvTPCm/GOUPolYJwZs0FcXhpIVQftfq7EWsRNPYjOaNB xnvou3Nr4Ywqy+dYkuekQ4WRuumjUdf7VyOjVOut X-Google-Smtp-Source: ABdhPJxuajF5zhvsVdOW0fkBia17zTyP9ZyuXT52doUy8oHVIE8g4t2RV/j5GXkZwQczAVAFQfTPCu3qCD1utqi0DImA X-Received: from ajr0.svl.corp.google.com ([2620:15c:2cd:203:3d06:d00f:a626:675a]) (user=axelrasmussen job=sendgmr) by 2002:a25:a265:: with SMTP id b92mr14490367ybi.486.1617925427524; Thu, 08 Apr 2021 16:43:47 -0700 (PDT) Date: Thu, 8 Apr 2021 16:43:27 -0700 In-Reply-To: <20210408234327.624367-1-axelrasmussen@google.com> Message-Id: <20210408234327.624367-10-axelrasmussen@google.com> Mime-Version: 1.0 References: <20210408234327.624367-1-axelrasmussen@google.com> X-Mailer: git-send-email 2.31.1.295.g9ea45b61b8-goog Subject: [PATCH 9/9] userfaultfd/shmem: modify shmem_mcopy_atomic_pte to use install_ptes From: Axel Rasmussen To: Alexander Viro , Andrea Arcangeli , Andrew Morton , Daniel Colascione , Hugh Dickins , Jerome Glisse , Joe Perches , Lokesh Gidra , Mike Kravetz , Mike Rapoport , Peter Xu , Shaohua Li , Shuah Khan , Stephen Rothwell , Wang Qing Cc: linux-api@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mm@kvack.org, Axel Rasmussen , Brian Geffon , "Dr . David Alan Gilbert" , Mina Almasry , Oliver Upton Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org In a previous commit, we added the mcopy_atomic_install_ptes() helper. This helper does the job of setting up PTEs for an existing page, to map it into a given VMA. It deals with both the anon and shmem cases, as well as the shared and private cases. In other words, shmem_mcopy_atomic_pte() duplicates a case it already handles. So, expose it, and let shmem_mcopy_atomic_pte() use it directly, to reduce code duplication. This requires that we refactor shmem_mcopy_atomic-pte() a bit: Instead of doing accounting (shmem_recalc_inode() et al) part-way through the PTE setup, do it beforehand. This frees up mcopy_atomic_install_ptes() from having to care about this accounting, but it does mean we need to clean it up if we get a failure afterwards (shmem_uncharge()). We can *almost* use shmem_charge() to do this, reducing code duplication. But, it does `inode->i_mapping->nrpages++`, which would double-count since shmem_add_to_page_cache() also does this. Signed-off-by: Axel Rasmussen --- include/linux/userfaultfd_k.h | 5 ++++ mm/shmem.c | 52 +++++++---------------------------- mm/userfaultfd.c | 25 ++++++++--------- 3 files changed, 27 insertions(+), 55 deletions(-) diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h index 794d1538b8ba..3e20bfa9ef80 100644 --- a/include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -53,6 +53,11 @@ enum mcopy_atomic_mode { MCOPY_ATOMIC_CONTINUE, }; +extern int mcopy_atomic_install_ptes(struct mm_struct *dst_mm, pmd_t *dst_pmd, + struct vm_area_struct *dst_vma, + unsigned long dst_addr, struct page *page, + bool newly_allocated, bool wp_copy); + extern ssize_t mcopy_atomic(struct mm_struct *dst_mm, unsigned long dst_start, unsigned long src_start, unsigned long len, bool *mmap_changing, __u64 mode); diff --git a/mm/shmem.c b/mm/shmem.c index 99c54b165c16..5d4b82e9bcb2 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2380,10 +2380,8 @@ int shmem_mcopy_atomic_pte(struct mm_struct *dst_mm, struct address_space *mapping = inode->i_mapping; gfp_t gfp = mapping_gfp_mask(mapping); pgoff_t pgoff = linear_page_index(dst_vma, dst_addr); - spinlock_t *ptl; void *page_kaddr; struct page *page; - pte_t _dst_pte, *dst_pte; int ret; pgoff_t max_off; @@ -2393,8 +2391,10 @@ int shmem_mcopy_atomic_pte(struct mm_struct *dst_mm, if (!*pagep) { page = shmem_alloc_page(gfp, info, pgoff); - if (!page) - goto out_unacct_blocks; + if (!page) { + shmem_inode_unacct_blocks(inode, 1); + goto out; + } if (!zeropage) { /* COPY */ page_kaddr = kmap_atomic(page); @@ -2434,59 +2434,27 @@ int shmem_mcopy_atomic_pte(struct mm_struct *dst_mm, if (ret) goto out_release; - _dst_pte = mk_pte(page, dst_vma->vm_page_prot); - if (dst_vma->vm_flags & VM_WRITE) - _dst_pte = pte_mkwrite(pte_mkdirty(_dst_pte)); - else { - /* - * We don't set the pte dirty if the vma has no - * VM_WRITE permission, so mark the page dirty or it - * could be freed from under us. We could do it - * unconditionally before unlock_page(), but doing it - * only if VM_WRITE is not set is faster. - */ - set_page_dirty(page); - } - - dst_pte = pte_offset_map_lock(dst_mm, dst_pmd, dst_addr, &ptl); - - ret = -EFAULT; - max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); - if (unlikely(pgoff >= max_off)) - goto out_release_unlock; - - ret = -EEXIST; - if (!pte_none(*dst_pte)) - goto out_release_unlock; - - lru_cache_add(page); - spin_lock_irq(&info->lock); info->alloced++; inode->i_blocks += BLOCKS_PER_PAGE; shmem_recalc_inode(inode); spin_unlock_irq(&info->lock); - inc_mm_counter(dst_mm, mm_counter_file(page)); - page_add_file_rmap(page, false); - set_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte); + ret = mcopy_atomic_install_ptes(dst_mm, dst_pmd, dst_vma, dst_addr, + page, true, false); + if (ret) + goto out_release_uncharge; - /* No need to invalidate - it was non-present before */ - update_mmu_cache(dst_vma, dst_addr, dst_pte); - pte_unmap_unlock(dst_pte, ptl); unlock_page(page); ret = 0; out: return ret; -out_release_unlock: - pte_unmap_unlock(dst_pte, ptl); - ClearPageDirty(page); +out_release_uncharge: delete_from_page_cache(page); + shmem_uncharge(inode, 1); out_release: unlock_page(page); put_page(page); -out_unacct_blocks: - shmem_inode_unacct_blocks(inode, 1); goto out; } #endif /* CONFIG_USERFAULTFD */ diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index a539fe18b9a7..8fc597782219 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -51,18 +51,13 @@ struct vm_area_struct *find_dst_vma(struct mm_struct *dst_mm, /* * Install PTEs, to map dst_addr (within dst_vma) to page. * - * This function handles MCOPY_ATOMIC_CONTINUE (which is always file-backed), - * whether or not dst_vma is VM_SHARED. It also handles the more general - * MCOPY_ATOMIC_NORMAL case, when dst_vma is *not* VM_SHARED (it may be file - * backed, or not). - * - * Note that MCOPY_ATOMIC_NORMAL for a VM_SHARED dst_vma is handled by - * shmem_mcopy_atomic_pte instead. + * This function handles both MCOPY_ATOMIC_NORMAL and _CONTINUE for both shmem + * and anon, and for both shared and private VMAs. */ -static int mcopy_atomic_install_ptes(struct mm_struct *dst_mm, pmd_t *dst_pmd, - struct vm_area_struct *dst_vma, - unsigned long dst_addr, struct page *page, - bool newly_allocated, bool wp_copy) +int mcopy_atomic_install_ptes(struct mm_struct *dst_mm, pmd_t *dst_pmd, + struct vm_area_struct *dst_vma, + unsigned long dst_addr, struct page *page, + bool newly_allocated, bool wp_copy) { int ret; pte_t _dst_pte, *dst_pte; @@ -116,8 +111,12 @@ static int mcopy_atomic_install_ptes(struct mm_struct *dst_mm, pmd_t *dst_pmd, else page_add_new_anon_rmap(page, dst_vma, dst_addr, false); - if (newly_allocated) - lru_cache_add_inactive_or_unevictable(page, dst_vma); + if (newly_allocated) { + if (vma_is_shmem(dst_vma) && vm_shared) + lru_cache_add(page); + else + lru_cache_add_inactive_or_unevictable(page, dst_vma); + } set_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte);