From patchwork Mon Apr 5 17:19:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Axel Rasmussen X-Patchwork-Id: 12183383 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19581C433B4 for ; Mon, 5 Apr 2021 17:19:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 96C4F613B3 for ; Mon, 5 Apr 2021 17:19:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 96C4F613B3 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 27EB16B0070; Mon, 5 Apr 2021 13:19:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 208D26B0073; Mon, 5 Apr 2021 13:19:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 033236B0075; Mon, 5 Apr 2021 13:19:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D2A3A6B0070 for ; Mon, 5 Apr 2021 13:19:22 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 82309824999B for ; Mon, 5 Apr 2021 17:19:22 +0000 (UTC) X-FDA: 77998974564.08.096D21B Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf14.hostedemail.com (Postfix) with ESMTP id 27B80C0007CF for ; Mon, 5 Apr 2021 17:19:18 +0000 (UTC) Received: by mail-pg1-f202.google.com with SMTP id 83so9463317pgh.4 for ; Mon, 05 Apr 2021 10:19:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=smRHtKh0sofAH08YA8piFhOVhZ9jZVfIv3hRd5nMU2Y=; b=IYxiRs3GRQsuxWSmfi3pdhs/xvvWvCPlYnOnkGhyLlGnSMFahlFGnRFxaiag4Bsurh 8R5TfyXkCm8wG87E0i8Z7F7SxF6oZ+w/LkYPzc94tPb5+VuL4a+GVLwVIkPv6yMIkW0w jTm0z75F/iZljMJxygUVk2taZums0r9HWVZfkqjmel/wW0ji55I0lMnODqJTxa/Ho+dP 7N/AVc/Esp5KqmMdjJ4QbNHZ6iBfql5iZi88aIf4KsXoVR2HHoJ5oB+fW+qdowRF7bqv y8gp3pkgKXgyeofArhYedgYpEotOYmO1cgDoIvziijichp87kFcN0cmIwZVry+4xnd/f VYMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=smRHtKh0sofAH08YA8piFhOVhZ9jZVfIv3hRd5nMU2Y=; b=qhQgPf6/9BqclZ5/5rgTZppBqeEj+jHiAWCx1EKKWnXY+mcjrgG3MSzYZ8I6NXocyN nygPN7RDl224SXcJaeZx55RzLhbQiBQAe01BYDDvwsBqiXJv+sc6CZNofEJLUGE4Qk8o VUr/hGM12kyrTD5tL+IPkjgRn2eTX/MoPmw/6iYYYaDTg83RoMNpAoWty/oHm457vAKy lOSjG+Q1oHDP/nLAkschA5jdtEHIjMdcUr+SLft4hylaFD0RIwF1pD/IAOVJElUbYvfy mgC+g1loq0L4ODqDavtVS0pW6xR5u75s410THbtReXxmdd67nG160sew1vWemEZ3ba4Y NXtA== X-Gm-Message-State: AOAM533bQliuxSCf6G6G2gbX+8mrMlRMuBZ3ktHYRoTYNrM85g0FDNed faLN4rGuuXS9+17wCZa/VDucHKQe1o0IJQD9ZYZH X-Google-Smtp-Source: ABdhPJxtr07M3HN+M9pbN3alal/MseWQUtDZfU1bch2xeMf/X5Y2ip1A5sSV1jo4A5t0ih8l4nIwWbnNgDnnRM+NWt0r X-Received: from ajr0.svl.corp.google.com ([2620:15c:2cd:203:b518:9786:f15a:398b]) (user=axelrasmussen job=sendgmr) by 2002:a17:90a:4d81:: with SMTP id m1mr142662pjh.143.1617643160576; Mon, 05 Apr 2021 10:19:20 -0700 (PDT) Date: Mon, 5 Apr 2021 10:19:17 -0700 Message-Id: <20210405171917.2423068-1-axelrasmussen@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.31.0.208.g409f899ff0-goog Subject: [PATCH v5] userfaultfd/shmem: fix MCOPY_ATOMIC_CONTINUE behavior From: Axel Rasmussen To: Alexander Viro , Andrea Arcangeli , Andrew Morton , Hugh Dickins , Jerome Glisse , Joe Perches , Lokesh Gidra , Mike Rapoport , Peter Xu , Shaohua Li , Shuah Khan , Stephen Rothwell , Wang Qing Cc: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, Axel Rasmussen , Brian Geffon , Cannon Matthews , "Dr . David Alan Gilbert" , David Rientjes , Michel Lespinasse , Mina Almasry , Oliver Upton X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 27B80C0007CF X-Stat-Signature: dgyabpk8b4gc5yeuxmkzp9rmi47utcm7 Received-SPF: none (flex--axelrasmussen.bounces.google.com>: No applicable sender policy available) receiver=imf14; identity=mailfrom; envelope-from="<3mEZrYA0KCNs7UBIO7PJRPPBKDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--axelrasmussen.bounces.google.com>"; helo=mail-pg1-f202.google.com; client-ip=209.85.215.202 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617643158-811540 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Previously, the continue implementation in shmem_mcopy_atomic_pte was incorrect for two main reasons: - It didn't correctly skip some sections of code which make sense for newly allocated pages, but absolutely don't make sense for pre-existing page cache pages. - Because shmem_mcopy_continue_pte is called only if VM_SHARED is detected in mm/userfaultd.c, we were incorrectly not supporting the case where a tmpfs file had been mmap()-ed with MAP_PRIVATE. So, this patch does the following: In mm/userfaultfd.c, break the logic to install PTEs out of mcopy_atomic_pte, into a separate helper function. In mfill_atomic_pte, for the MCOPY_ATOMIC_CONTINUE case, simply look up the existing page in the page cache, and then use the PTE installation helper to setup the mapping. This addresses the two issues noted above. The previous code's bugs manifested clearly in the error handling path. So, to detect this kind of issue in the future, modify the selftest to exercise the error handling path as well. Note that this patch is based on linux-next/akpm; the "fixes" line refers to a SHA1 in that branch. Changes since v4: - Added back the userfaultfd.c selftest changes from v3; this file was mistakenly reverted in v4. Changes since v3: - Significantly refactored the patch. Continue handling now happens in mm/userfaultfd.c, via a PTE installation helper. Most of the mm/shmem.c changes from the patch being fixed [1] are reverted. Changes since v2: - Drop the ClearPageDirty() entirely, instead of trying to remember the old value. - Modify both pgoff / max_off checks to use pgoff. It's equivalent to offset, but offset wasn't initialized until the first check (which we're skipping). - Keep the second pgoff / max_off check in the continue case. Changes since v1: - Refactor to skip ahead with goto, instead of adding several more "if (!is_continue)". - Fix unconditional ClearPageDirty(). - Don't pte_mkwrite() when is_continue && !VM_SHARED. [1] https://lore.kernel.org/patchwork/patch/1392464/ Fixes: 00da60b9d0a0 ("userfaultfd: support minor fault handling for shmem") Signed-off-by: Axel Rasmussen --- mm/shmem.c | 56 +++---- mm/userfaultfd.c | 183 ++++++++++++++++------- tools/testing/selftests/vm/userfaultfd.c | 12 ++ 3 files changed, 168 insertions(+), 83 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 5cfd2fb6e52b..9d9a9f254f33 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2366,7 +2366,6 @@ int shmem_mcopy_atomic_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, unsigned long dst_addr, unsigned long src_addr, enum mcopy_atomic_mode mode, struct page **pagep) { - bool is_continue = (mode == MCOPY_ATOMIC_CONTINUE); struct inode *inode = file_inode(dst_vma->vm_file); struct shmem_inode_info *info = SHMEM_I(inode); struct address_space *mapping = inode->i_mapping; @@ -2377,18 +2376,17 @@ int shmem_mcopy_atomic_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, struct page *page; pte_t _dst_pte, *dst_pte; int ret; - pgoff_t offset, max_off; + pgoff_t max_off; + + /* Handled by mcontinue_atomic_pte instead. */ + if (WARN_ON_ONCE(mode == MCOPY_ATOMIC_CONTINUE)) + return -EINVAL; ret = -ENOMEM; if (!shmem_inode_acct_block(inode, 1)) goto out; - if (is_continue) { - ret = -EFAULT; - page = find_lock_page(mapping, pgoff); - if (!page) - goto out_unacct_blocks; - } else if (!*pagep) { + if (!*pagep) { page = shmem_alloc_page(gfp, info, pgoff); if (!page) goto out_unacct_blocks; @@ -2415,27 +2413,21 @@ int shmem_mcopy_atomic_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, *pagep = NULL; } - if (!is_continue) { - VM_BUG_ON(PageSwapBacked(page)); - VM_BUG_ON(PageLocked(page)); - __SetPageLocked(page); - __SetPageSwapBacked(page); - __SetPageUptodate(page); - } + VM_BUG_ON(PageSwapBacked(page)); + VM_BUG_ON(PageLocked(page)); + __SetPageLocked(page); + __SetPageSwapBacked(page); + __SetPageUptodate(page); ret = -EFAULT; - offset = linear_page_index(dst_vma, dst_addr); max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); - if (unlikely(offset >= max_off)) + if (unlikely(pgoff >= max_off)) goto out_release; - /* If page wasn't already in the page cache, add it. */ - if (!is_continue) { - ret = shmem_add_to_page_cache(page, mapping, pgoff, NULL, - gfp & GFP_RECLAIM_MASK, dst_mm); - if (ret) - goto out_release; - } + ret = shmem_add_to_page_cache(page, mapping, pgoff, NULL, + gfp & GFP_RECLAIM_MASK, dst_mm); + if (ret) + goto out_release; _dst_pte = mk_pte(page, dst_vma->vm_page_prot); if (dst_vma->vm_flags & VM_WRITE) @@ -2455,22 +2447,20 @@ int shmem_mcopy_atomic_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, ret = -EFAULT; max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); - if (unlikely(offset >= max_off)) + if (unlikely(pgoff >= max_off)) goto out_release_unlock; ret = -EEXIST; if (!pte_none(*dst_pte)) goto out_release_unlock; - if (!is_continue) { - lru_cache_add(page); + lru_cache_add(page); - spin_lock_irq(&info->lock); - info->alloced++; - inode->i_blocks += BLOCKS_PER_PAGE; - shmem_recalc_inode(inode); - spin_unlock_irq(&info->lock); - } + spin_lock_irq(&info->lock); + info->alloced++; + inode->i_blocks += BLOCKS_PER_PAGE; + shmem_recalc_inode(inode); + spin_unlock_irq(&info->lock); inc_mm_counter(dst_mm, mm_counter_file(page)); page_add_file_rmap(page, false); diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index cbb7c8d79a4d..286d0657fbe2 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -48,21 +48,103 @@ struct vm_area_struct *find_dst_vma(struct mm_struct *dst_mm, return dst_vma; } +/* + * Install PTEs, to map dst_addr (within dst_vma) to page. + * + * This function handles MCOPY_ATOMIC_CONTINUE (which is always file-backed), + * whether or not dst_vma is VM_SHARED. It also handles the more general + * MCOPY_ATOMIC_NORMAL case, when dst_vma is *not* VM_SHARED (it may be file + * backed, or not). + * + * Note that MCOPY_ATOMIC_NORMAL for a VM_SHARED dst_vma is handled by + * shmem_mcopy_atomic_pte instead. + */ +static int mcopy_atomic_install_ptes(struct mm_struct *dst_mm, pmd_t *dst_pmd, + struct vm_area_struct *dst_vma, + unsigned long dst_addr, struct page *page, + enum mcopy_atomic_mode mode, bool wp_copy) +{ + int ret; + pte_t _dst_pte, *dst_pte; + bool is_continue = mode == MCOPY_ATOMIC_CONTINUE; + int writable; + bool vm_shared = dst_vma->vm_flags & VM_SHARED; + bool is_file_backed = dst_vma->vm_file; + spinlock_t *ptl; + struct inode *inode; + pgoff_t offset, max_off; + + _dst_pte = mk_pte(page, dst_vma->vm_page_prot); + writable = dst_vma->vm_flags & VM_WRITE; + /* For CONTINUE on a non-shared VMA, don't pte_mkwrite for CoW. */ + if (is_continue && !vm_shared) + writable = 0; + + if (writable) { + _dst_pte = pte_mkdirty(_dst_pte); + if (wp_copy) + _dst_pte = pte_mkuffd_wp(_dst_pte); + else + _dst_pte = pte_mkwrite(_dst_pte); + } else if (vm_shared) { + /* + * Since we didn't pte_mkdirty(), mark the page dirty or it + * could be freed from under us. We could do this + * unconditionally, but doing it only if !writable is faster. + */ + set_page_dirty(page); + } + + dst_pte = pte_offset_map_lock(dst_mm, dst_pmd, dst_addr, &ptl); + + if (is_file_backed) { + /* The shmem MAP_PRIVATE case requires checking the i_size */ + inode = dst_vma->vm_file->f_inode; + offset = linear_page_index(dst_vma, dst_addr); + max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); + ret = -EFAULT; + if (unlikely(offset >= max_off)) + goto out_unlock; + } + + ret = -EEXIST; + if (!pte_none(*dst_pte)) + goto out_unlock; + + inc_mm_counter(dst_mm, mm_counter(page)); + if (is_file_backed) + page_add_file_rmap(page, false); + else + page_add_new_anon_rmap(page, dst_vma, dst_addr, false); + + if (!is_continue) + lru_cache_add_inactive_or_unevictable(page, dst_vma); + + set_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte); + + /* No need to invalidate - it was non-present before */ + update_mmu_cache(dst_vma, dst_addr, dst_pte); + pte_unmap_unlock(dst_pte, ptl); + ret = 0; +out: + return ret; +out_unlock: + pte_unmap_unlock(dst_pte, ptl); + goto out; +} + static int mcopy_atomic_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, struct vm_area_struct *dst_vma, unsigned long dst_addr, unsigned long src_addr, struct page **pagep, + enum mcopy_atomic_mode mode, bool wp_copy) { - pte_t _dst_pte, *dst_pte; - spinlock_t *ptl; void *page_kaddr; int ret; struct page *page; - pgoff_t offset, max_off; - struct inode *inode; if (!*pagep) { ret = -ENOMEM; @@ -99,43 +181,12 @@ static int mcopy_atomic_pte(struct mm_struct *dst_mm, if (mem_cgroup_charge(page, dst_mm, GFP_KERNEL)) goto out_release; - _dst_pte = pte_mkdirty(mk_pte(page, dst_vma->vm_page_prot)); - if (dst_vma->vm_flags & VM_WRITE) { - if (wp_copy) - _dst_pte = pte_mkuffd_wp(_dst_pte); - else - _dst_pte = pte_mkwrite(_dst_pte); - } - - dst_pte = pte_offset_map_lock(dst_mm, dst_pmd, dst_addr, &ptl); - if (dst_vma->vm_file) { - /* the shmem MAP_PRIVATE case requires checking the i_size */ - inode = dst_vma->vm_file->f_inode; - offset = linear_page_index(dst_vma, dst_addr); - max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); - ret = -EFAULT; - if (unlikely(offset >= max_off)) - goto out_release_uncharge_unlock; - } - ret = -EEXIST; - if (!pte_none(*dst_pte)) - goto out_release_uncharge_unlock; - - inc_mm_counter(dst_mm, MM_ANONPAGES); - page_add_new_anon_rmap(page, dst_vma, dst_addr, false); - lru_cache_add_inactive_or_unevictable(page, dst_vma); - - set_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte); - - /* No need to invalidate - it was non-present before */ - update_mmu_cache(dst_vma, dst_addr, dst_pte); - - pte_unmap_unlock(dst_pte, ptl); - ret = 0; + ret = mcopy_atomic_install_ptes(dst_mm, dst_pmd, dst_vma, dst_addr, + page, mode, wp_copy); + if (ret) + goto out_release; out: return ret; -out_release_uncharge_unlock: - pte_unmap_unlock(dst_pte, ptl); out_release: put_page(page); goto out; @@ -176,6 +227,38 @@ static int mfill_zeropage_pte(struct mm_struct *dst_mm, return ret; } +static int mcontinue_atomic_pte(struct mm_struct *dst_mm, + pmd_t *dst_pmd, + struct vm_area_struct *dst_vma, + unsigned long dst_addr, + bool wp_copy) +{ + struct inode *inode = file_inode(dst_vma->vm_file); + struct address_space *mapping = inode->i_mapping; + pgoff_t pgoff = linear_page_index(dst_vma, dst_addr); + struct page *page; + int ret; + + ret = -EFAULT; + page = find_lock_page(mapping, pgoff); + if (!page) + goto out; + + ret = mcopy_atomic_install_ptes(dst_mm, dst_pmd, dst_vma, dst_addr, + page, MCOPY_ATOMIC_CONTINUE, wp_copy); + if (ret) + goto out_release; + + unlock_page(page); + ret = 0; +out: + return ret; +out_release: + unlock_page(page); + put_page(page); + goto out; +} + static pmd_t *mm_alloc_pmd(struct mm_struct *mm, unsigned long address) { pgd_t *pgd; @@ -418,7 +501,13 @@ static __always_inline ssize_t mfill_atomic_pte(struct mm_struct *dst_mm, enum mcopy_atomic_mode mode, bool wp_copy) { - ssize_t err; + ssize_t err = 0; + + if (mode == MCOPY_ATOMIC_CONTINUE) { + err = mcontinue_atomic_pte(dst_mm, dst_pmd, dst_vma, dst_addr, + wp_copy); + goto out; + } /* * The normal page fault path for a shmem will invoke the @@ -431,26 +520,20 @@ static __always_inline ssize_t mfill_atomic_pte(struct mm_struct *dst_mm, * and not in the radix tree. */ if (!(dst_vma->vm_flags & VM_SHARED)) { - switch (mode) { - case MCOPY_ATOMIC_NORMAL: + if (mode == MCOPY_ATOMIC_NORMAL) err = mcopy_atomic_pte(dst_mm, dst_pmd, dst_vma, dst_addr, src_addr, page, - wp_copy); - break; - case MCOPY_ATOMIC_ZEROPAGE: + mode, wp_copy); + else if (mode == MCOPY_ATOMIC_ZEROPAGE) err = mfill_zeropage_pte(dst_mm, dst_pmd, dst_vma, dst_addr); - break; - case MCOPY_ATOMIC_CONTINUE: - err = -EINVAL; - break; - } } else { VM_WARN_ON_ONCE(wp_copy); err = shmem_mcopy_atomic_pte(dst_mm, dst_pmd, dst_vma, dst_addr, src_addr, mode, page); } +out: return err; } diff --git a/tools/testing/selftests/vm/userfaultfd.c b/tools/testing/selftests/vm/userfaultfd.c index f6c86b036d0f..d8541a59dae5 100644 --- a/tools/testing/selftests/vm/userfaultfd.c +++ b/tools/testing/selftests/vm/userfaultfd.c @@ -485,6 +485,7 @@ static void wp_range(int ufd, __u64 start, __u64 len, bool wp) static void continue_range(int ufd, __u64 start, __u64 len) { struct uffdio_continue req; + int ret; req.range.start = start; req.range.len = len; @@ -493,6 +494,17 @@ static void continue_range(int ufd, __u64 start, __u64 len) if (ioctl(ufd, UFFDIO_CONTINUE, &req)) err("UFFDIO_CONTINUE failed for address 0x%" PRIx64, (uint64_t)start); + + /* + * Error handling within the kernel for continue is subtly different + * from copy or zeropage, so it may be a source of bugs. Trigger an + * error (-EEXIST) on purpose, to verify doing so doesn't cause a BUG. + */ + req.mapped = 0; + ret = ioctl(ufd, UFFDIO_CONTINUE, &req); + if (ret >= 0 || req.mapped != -EEXIST) + err("failed to exercise UFFDIO_CONTINUE error handling, ret=%d, mapped=%" PRId64, + ret, req.mapped); } static void *locking_thread(void *arg)