From patchwork Mon Dec 3 20:08:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 10710495 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A347E13BF for ; Mon, 3 Dec 2018 20:09:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 95FF02B2E0 for ; Mon, 3 Dec 2018 20:09:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8A4382B334; Mon, 3 Dec 2018 20:09:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 883D52B2E0 for ; Mon, 3 Dec 2018 20:09:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 67C6F6B6ADE; Mon, 3 Dec 2018 15:09:13 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 565A06B6ADB; Mon, 3 Dec 2018 15:09:13 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E2836B6ADD; Mon, 3 Dec 2018 15:09:13 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by kanga.kvack.org (Postfix) with ESMTP id 122546B6ADB for ; Mon, 3 Dec 2018 15:09:13 -0500 (EST) Received: by mail-qk1-f199.google.com with SMTP id 92so14286932qkx.19 for ; Mon, 03 Dec 2018 12:09:13 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=xU5EuFc9W8EoxN71ybMSu+c4vkKUSMPVgkDvAU5zpnQ=; b=XyEvKJdJRz19FoHUZBJnFJBTwVL6resXaUMMvHqJA5AXxilYHWOBBR4tB9OImOaqmK dcmLAfgaTsZj1QEgnW7TLw0hJnmQA7P64FzPWsTvHIAoqT9Qg43vqggxJ40v00JqI8fc i/BTNK6ghAS764gzdPeHoYDk8NDc+UuwSYm3jRxbVlL7dw+nkn76mNMf0RhP6zsyPuUJ J0ewiKwFuSQtRuB2t+EHf/zRlwCNyrZUMYDuvf2ZprhHLsXs8RrT1nB4m2rK4D6PpLKx H8GLMfLx7gzJ1ROkins+uf3KDWLfEoBeAz42cVQrcnt5b0o0cYbJN4s6A9Qx0ovfrorV Mv3A== X-Gm-Message-State: AA+aEWbFcwi54nsO7FKmHcngisd7BDj4zWsCORJbGUDabtsxa7axAsXI zduzN7OFRtGZ5SrpH7gpmgrjiB+wm7uWr7zzvTMTsaIKEHOZoiZSE2k3cfDeY3migokvhEnsew8 /rzIh1HSIQXCW94cqnrnUdn6MdX264BJUolW1uCpetEzoGqUQOjJ1uH1eRsBkQjYKMA== X-Received: by 2002:ac8:16d8:: with SMTP id y24mr17121892qtk.253.1543867752757; Mon, 03 Dec 2018 12:09:12 -0800 (PST) X-Google-Smtp-Source: AFSGD/W6TuK0XaRdnKhPfJXQJmEcHG0iNyDY9z0rkoeJA/xbMWiGfQgneTpWquW8tQkdwHswhr9g X-Received: by 2002:ac8:16d8:: with SMTP id y24mr17121809qtk.253.1543867751469; Mon, 03 Dec 2018 12:09:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1543867751; cv=none; d=google.com; s=arc-20160816; b=AYmzX9EZF3bkVvBm66c67u0YUbYztK9d29d2d+JUv6fCft2UFBNrKh/ZEP9mTvf5CZ fj3ns+OTl1RQ08yrAz3m1FNUllwX6fnNLQebp2R2j8g8NtbB7WlB5JdbcQTu+6jpKxEk MsLyDInMHOStg/qhdVothyYZRwT5E87Ctzeso9CIY82w6jNlN0mCZxkvQbJlbzmNciO+ czePudUf++Qhaf+3q+t8ICQnwxhi9c6/uOPThcjfnt4lYW7sG4Jnvr4yU2XtKYtm3a7t DI7//cIobyj0kglVFaR1UrkemVpf6QjJ1s5KcQELXjXkyPUCJkaZl/MO3wUo5VVYkBl2 CQjQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=xU5EuFc9W8EoxN71ybMSu+c4vkKUSMPVgkDvAU5zpnQ=; b=rTQ3kA2AaCq+CEfeMDexLsQEt2ytK7jcynncvX95pnjzZeNpjpN4QMqhtJu6pUICUg egAjkMC20vwcv+yBzZ9jqUegAd1utzXa/CD7n5KqV1FPM+YUghmT1A9Hmp5YudkBjQZ4 FLr3FTDzx7LZoWnnNMa2y1rcm/l6RNUe4mDSzkR/dNi2/OpQPw79+MgH2h+B8JJ0rP9a NswdQsUa/XPiJatyTgjavvSWs1eRXePT1Z18MGlv0JAZk8GooNmztByIL+2DTESaGPKt K2DrTGIiRP5BzwvbhyzXYgIqrpNjfZrVJx42rQflTmrRyVgxK1pXvYPfR1y0f4RFSVU5 d37A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=Hu8s8sNz; spf=pass (google.com: domain of mike.kravetz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2120.oracle.com (userp2120.oracle.com. [156.151.31.85]) by mx.google.com with ESMTPS id v11si2539352qvj.128.2018.12.03.12.09.10 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 03 Dec 2018 12:09:11 -0800 (PST) Received-SPF: pass (google.com: domain of mike.kravetz@oracle.com designates 156.151.31.85 as permitted sender) client-ip=156.151.31.85; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=Hu8s8sNz; spf=pass (google.com: domain of mike.kravetz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wB3Jwcpq078831; Mon, 3 Dec 2018 20:09:01 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=xU5EuFc9W8EoxN71ybMSu+c4vkKUSMPVgkDvAU5zpnQ=; b=Hu8s8sNzmWVGpEocIeYhZ8dcf0nftkRjDSiwoC+tex5HrsCHAC5g5uVVYUt7FraKYBKI m8JR38r8cqQAOoKUtbehpoghmpM0iiNkMdrJCeuEB3E/jY4OstGAgYpChdDJgW+g/6Am azZdAw/HmIxL4x4c/aR89zx+j2OsH8/n5JYI0fwGTJYCb5fF1RvbLK9STimBS4IPqh3L fYOOlj+Z1Jh8wU9YyU0wZwFE8R7B6fusV8rIM6dV6h9/E/L8wwzyzEi+6u4XvKDh3HcK /fG6vUPXzgq15iaFuHTNpisfPHFdgYeI4BX07DbRXdwb2I0xZ0tCRYtaiXUftJU+cVen 9A== Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp2120.oracle.com with ESMTP id 2p3jxr8gjx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Dec 2018 20:09:01 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wB3K901Z030134 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 3 Dec 2018 20:09:00 GMT Received: from abhmp0012.oracle.com (abhmp0012.oracle.com [141.146.116.18]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wB3K8xWu028562; Mon, 3 Dec 2018 20:08:59 GMT Received: from monkey.oracle.com (/50.38.38.67) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 03 Dec 2018 12:08:59 -0800 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Michal Hocko , Hugh Dickins , Naoya Horiguchi , "Aneesh Kumar K . V" , Andrea Arcangeli , "Kirill A . Shutemov" , Davidlohr Bueso , Prakash Sangappa , Andrew Morton , Mike Kravetz , stable@vger.kernel.org Subject: [PATCH 1/3] hugetlbfs: use i_mmap_rwsem for more pmd sharing synchronization Date: Mon, 3 Dec 2018 12:08:48 -0800 Message-Id: <20181203200850.6460-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181203200850.6460-1-mike.kravetz@oracle.com> References: <20181203200850.6460-1-mike.kravetz@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9096 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812030183 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP While looking at BUGs associated with invalid huge page map counts, it was discovered and observed that a huge pte pointer could become 'invalid' and point to another task's page table. Consider the following: A task takes a page fault on a shared hugetlbfs file and calls huge_pte_alloc to get a ptep. Suppose the returned ptep points to a shared pmd. Now, another task truncates the hugetlbfs file. As part of truncation, it unmaps everyone who has the file mapped. If the range being truncated is covered by a shared pmd, huge_pmd_unshare will be called. For all but the last user of the shared pmd, huge_pmd_unshare will clear the pud pointing to the pmd. If the task in the middle of the page fault is not the last user, the ptep returned by huge_pte_alloc now points to another task's page table or worse. This leads to bad things such as incorrect page map/reference counts or invalid memory references. To fix, expand the use of i_mmap_rwsem as follows: - i_mmap_rwsem is held in read mode whenever huge_pmd_share is called. huge_pmd_share is only called via huge_pte_alloc, so callers of huge_pte_alloc take i_mmap_rwsem before calling. In addition, callers of huge_pte_alloc continue to hold the semaphore until finished with the ptep. - i_mmap_rwsem is held in write mode whenever huge_pmd_unshare is called. Cc: Fixes: 39dde65c9940 ("shared page table for hugetlb page") Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 70 ++++++++++++++++++++++++++++++++++----------- mm/memory-failure.c | 14 ++++++++- mm/migrate.c | 13 ++++++++- mm/rmap.c | 3 ++ mm/userfaultfd.c | 11 +++++-- 5 files changed, 91 insertions(+), 20 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 1931a3d9b282..362601b69c56 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3239,6 +3239,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, int cow; struct hstate *h = hstate_vma(vma); unsigned long sz = huge_page_size(h); + struct address_space *mapping = vma->vm_file->f_mapping; unsigned long mmun_start; /* For mmu_notifiers */ unsigned long mmun_end; /* For mmu_notifiers */ int ret = 0; @@ -3252,11 +3253,23 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, for (addr = vma->vm_start; addr < vma->vm_end; addr += sz) { spinlock_t *src_ptl, *dst_ptl; + src_pte = huge_pte_offset(src, addr, sz); if (!src_pte) continue; + + /* + * i_mmap_rwsem must be held to call huge_pte_alloc. + * Continue to hold until finished with dst_pte, otherwise + * it could go away if part of a shared pmd. + * + * Technically, i_mmap_rwsem is only needed in the non-cow + * case as cow mappings are not shared. + */ + i_mmap_lock_read(mapping); dst_pte = huge_pte_alloc(dst, addr, sz); if (!dst_pte) { + i_mmap_unlock_read(mapping); ret = -ENOMEM; break; } @@ -3271,8 +3284,10 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, * after taking the lock below. */ dst_entry = huge_ptep_get(dst_pte); - if ((dst_pte == src_pte) || !huge_pte_none(dst_entry)) + if ((dst_pte == src_pte) || !huge_pte_none(dst_entry)) { + i_mmap_unlock_read(mapping); continue; + } dst_ptl = huge_pte_lock(h, dst, dst_pte); src_ptl = huge_pte_lockptr(h, src, src_pte); @@ -3321,6 +3336,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, } spin_unlock(src_ptl); spin_unlock(dst_ptl); + + i_mmap_unlock_read(mapping); } if (cow) @@ -3772,14 +3789,18 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, }; /* - * hugetlb_fault_mutex must be dropped before - * handling userfault. Reacquire after handling - * fault to make calling code simpler. + * hugetlb_fault_mutex and i_mmap_rwsem must be + * dropped before handling userfault. Reacquire + * after handling fault to make calling code simpler. */ hash = hugetlb_fault_mutex_hash(h, mm, vma, mapping, idx, haddr); mutex_unlock(&hugetlb_fault_mutex_table[hash]); + i_mmap_unlock_read(mapping); + ret = handle_userfault(&vmf, VM_UFFD_MISSING); + + i_mmap_lock_read(mapping); mutex_lock(&hugetlb_fault_mutex_table[hash]); goto out; } @@ -3927,6 +3948,11 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, ptep = huge_pte_offset(mm, haddr, huge_page_size(h)); if (ptep) { + /* + * Since we hold no locks, ptep could be stale. That is + * OK as we are only making decisions based on content and + * not actually modifying content here. + */ entry = huge_ptep_get(ptep); if (unlikely(is_hugetlb_entry_migration(entry))) { migration_entry_wait_huge(vma, mm, ptep); @@ -3934,20 +3960,31 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, } else if (unlikely(is_hugetlb_entry_hwpoisoned(entry))) return VM_FAULT_HWPOISON_LARGE | VM_FAULT_SET_HINDEX(hstate_index(h)); - } else { - ptep = huge_pte_alloc(mm, haddr, huge_page_size(h)); - if (!ptep) - return VM_FAULT_OOM; } + /* + * Acquire i_mmap_rwsem before calling huge_pte_alloc and hold + * until finished with ptep. This prevents huge_pmd_unshare from + * being called elsewhere and making the ptep no longer valid. + * + * ptep could have already be assigned via huge_pte_offset. That + * is OK, as huge_pte_alloc will return the same value unless + * something changed. + */ mapping = vma->vm_file->f_mapping; - idx = vma_hugecache_offset(h, vma, haddr); + i_mmap_lock_read(mapping); + ptep = huge_pte_alloc(mm, haddr, huge_page_size(h)); + if (!ptep) { + i_mmap_unlock_read(mapping); + return VM_FAULT_OOM; + } /* * Serialize hugepage allocation and instantiation, so that we don't * get spurious allocation failures if two CPUs race to instantiate * the same page in the page cache. */ + idx = vma_hugecache_offset(h, vma, haddr); hash = hugetlb_fault_mutex_hash(h, mm, vma, mapping, idx, haddr); mutex_lock(&hugetlb_fault_mutex_table[hash]); @@ -4035,6 +4072,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, } out_mutex: mutex_unlock(&hugetlb_fault_mutex_table[hash]); + i_mmap_unlock_read(mapping); /* * Generally it's safe to hold refcount during waiting page lock. But * here we just wait to defer the next page fault to avoid busy loop and @@ -4639,10 +4677,12 @@ void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma, * Search for a shareable pmd page for hugetlb. In any case calls pmd_alloc() * and returns the corresponding pte. While this is not necessary for the * !shared pmd case because we can allocate the pmd later as well, it makes the - * code much cleaner. pmd allocation is essential for the shared case because - * pud has to be populated inside the same i_mmap_rwsem section - otherwise - * racing tasks could either miss the sharing (see huge_pte_offset) or select a - * bad pmd for sharing. + * code much cleaner. + * + * This routine must be called with i_mmap_rwsem held in at least read mode. + * For hugetlbfs, this prevents removal of any page table entries associated + * with the address space. This is important as we are setting up sharing + * based on existing page table entries (mappings). */ pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud) { @@ -4659,7 +4699,6 @@ pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud) if (!vma_shareable(vma, addr)) return (pte_t *)pmd_alloc(mm, pud, addr); - i_mmap_lock_write(mapping); vma_interval_tree_foreach(svma, &mapping->i_mmap, idx, idx) { if (svma == vma) continue; @@ -4689,7 +4728,6 @@ pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud) spin_unlock(ptl); out: pte = (pte_t *)pmd_alloc(mm, pud, addr); - i_mmap_unlock_write(mapping); return pte; } @@ -4700,7 +4738,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud) * indicated by page_count > 1, unmap is achieved by clearing pud and * decrementing the ref count. If count == 1, the pte page is not shared. * - * called with page table lock held. + * Called with page table lock held and i_mmap_rwsem held in write mode. * * returns: 1 successfully unmapped a shared pte page * 0 the underlying pte page is not shared, or it is the last user diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 0cd3de3550f0..b992d1295578 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -1028,7 +1028,19 @@ static bool hwpoison_user_mappings(struct page *p, unsigned long pfn, if (kill) collect_procs(hpage, &tokill, flags & MF_ACTION_REQUIRED); - unmap_success = try_to_unmap(hpage, ttu); + if (!PageHuge(hpage)) { + unmap_success = try_to_unmap(hpage, ttu); + } else { + /* + * For hugetlb pages, try_to_unmap could potentially call + * huge_pmd_unshare. Because of this, take semaphore in + * write mode here and set TTU_RMAP_LOCKED to indicate we + * have taken the lock at this higer level. + */ + i_mmap_lock_write(mapping); + unmap_success = try_to_unmap(hpage, ttu|TTU_RMAP_LOCKED); + i_mmap_unlock_write(mapping); + } if (!unmap_success) pr_err("Memory failure: %#lx: failed to unmap page (mapcount=%d)\n", pfn, page_mapcount(hpage)); diff --git a/mm/migrate.c b/mm/migrate.c index 84381b55b2bd..725edaef238a 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1307,8 +1307,19 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, goto put_anon; if (page_mapped(hpage)) { + struct address_space *mapping = page_mapping(hpage); + + /* + * try_to_unmap could potentially call huge_pmd_unshare. + * Because of this, take semaphore in write mode here and + * set TTU_RMAP_LOCKED to let lower levels know we have + * taken the lock. + */ + i_mmap_lock_write(mapping); try_to_unmap(hpage, - TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS); + TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS| + TTU_RMAP_LOCKED); + i_mmap_unlock_write(mapping); page_was_mapped = 1; } diff --git a/mm/rmap.c b/mm/rmap.c index 1e79fac3186b..aed241f69fcf 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1374,6 +1374,9 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, /* * If sharing is possible, start and end will be adjusted * accordingly. + * + * If called for a huge page, caller must hold i_mmap_rwsem + * in write mode as it is possible to call huge_pmd_unshare. */ adjust_range_if_pmd_sharing_possible(vma, &start, &end); } diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 5029f241908f..7cf4d8f7494b 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -244,10 +244,14 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, VM_BUG_ON(dst_addr & ~huge_page_mask(h)); /* - * Serialize via hugetlb_fault_mutex + * Serialize via i_mmap_rwsem and hugetlb_fault_mutex. + * i_mmap_rwsem ensures the dst_pte remains valid even + * in the case of shared pmds. fault mutex prevents + * races with other faulting threads. */ - idx = linear_page_index(dst_vma, dst_addr); mapping = dst_vma->vm_file->f_mapping; + i_mmap_lock_read(mapping); + idx = linear_page_index(dst_vma, dst_addr); hash = hugetlb_fault_mutex_hash(h, dst_mm, dst_vma, mapping, idx, dst_addr); mutex_lock(&hugetlb_fault_mutex_table[hash]); @@ -256,6 +260,7 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, dst_pte = huge_pte_alloc(dst_mm, dst_addr, huge_page_size(h)); if (!dst_pte) { mutex_unlock(&hugetlb_fault_mutex_table[hash]); + i_mmap_unlock_read(mapping); goto out_unlock; } @@ -263,6 +268,7 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, dst_pteval = huge_ptep_get(dst_pte); if (!huge_pte_none(dst_pteval)) { mutex_unlock(&hugetlb_fault_mutex_table[hash]); + i_mmap_unlock_read(mapping); goto out_unlock; } @@ -270,6 +276,7 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, dst_addr, src_addr, &page); mutex_unlock(&hugetlb_fault_mutex_table[hash]); + i_mmap_unlock_read(mapping); vm_alloc_shared = vm_shared; cond_resched(); From patchwork Mon Dec 3 20:08:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 10710499 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3377115A6 for ; Mon, 3 Dec 2018 20:09:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 255162B2E0 for ; Mon, 3 Dec 2018 20:09:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 19B9E2B32D; Mon, 3 Dec 2018 20:09:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9D6342B2E0 for ; Mon, 3 Dec 2018 20:09:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 25B826B6ADA; Mon, 3 Dec 2018 15:09:16 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 235DC6B6ADC; Mon, 3 Dec 2018 15:09:16 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0B1566B6ADD; Mon, 3 Dec 2018 15:09:16 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f198.google.com (mail-yb1-f198.google.com [209.85.219.198]) by kanga.kvack.org (Postfix) with ESMTP id CCD0F6B6ADA for ; Mon, 3 Dec 2018 15:09:15 -0500 (EST) Received: by mail-yb1-f198.google.com with SMTP id i15-v6so8794771ybp.7 for ; Mon, 03 Dec 2018 12:09:15 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=0t52fLgngXHEH1UGn44+Uw2HE3xbOfQ4NJc5Y8fZs1w=; b=Jx/+04xJpxHKStAup71qYJxpamiTh/NJpcXZ0Q2spZc6sgw7KEo1j3txs02P8FEz7M 7U5QIt8xkIcc5I/d0N+k2+VK9BiV+7WlY8WE7+pWpFvhpMsxAWPgmumFvAgV7NMqqOiU R9dCOcS1zyV1WALcqj08L8gGJfp3OQwvJHqfEf3c1TBtGwCNnDFoxRVkVCiWWHpUXLKT 5u4kR5DBFVSwXiTpJtixrcQa1zb3DjdwPOsKnxTP9uXr2QpLXP9dKZeQiqkX3hlk2biX +qSUF8NLpfK2dGgcw+yo8yICapDXCiLW1RyNfZCCT9FP+7ejL4qW+Z4rCvBkJxuUkZEf 0LWQ== X-Gm-Message-State: AA+aEWaTTwYrFaHvpvO4GOxpfxe+21O6ePNoBT1sjGNtv3/BOhB/GInn b4EBdhNjcrY7wd3NhaOO3NTtGz1+J7labScM7QL+xhR6aSdxG+/weTbyoX3qsRBGgCfEQobL4/M 9jNfkGLAElGsfQqJ/ohu1uIOMd+xNfGW7YTkOZUWSeeRxdg/BaLopmbiAyAodAeUOJg== X-Received: by 2002:a25:3fc5:: with SMTP id m188-v6mr17182740yba.359.1543867755521; Mon, 03 Dec 2018 12:09:15 -0800 (PST) X-Google-Smtp-Source: AFSGD/VZ/vm3jBU9cMBkc8QoHFYbgm67P5fGvp/ELCyIWeazPKztTbWQXs7j8uU2Zj8sqLPRY9j4 X-Received: by 2002:a25:3fc5:: with SMTP id m188-v6mr17182685yba.359.1543867754792; Mon, 03 Dec 2018 12:09:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1543867754; cv=none; d=google.com; s=arc-20160816; b=bCdw9nuQr9S1+mHUGqRMDTydqe5sADzQ0Sig75XsrHLvkXI0ihKG78ysLUZ5yFIGmO iPtkyiZjwYw6gZJGUXVrx85BVSm8Te6K/CoslVjmUHq3waaTP0Qa8ti0Xb1J5UNsla5q vTy4BT+bt3D7Vn2ACHFFyBMLy/uTb6ySTzgMaSGAVfO/+vsAZy94Vd6eUN9UR16XbBfL Kz5JzNZLb6fUfrS1MW+nMyQ0mKJQaq/hX27DJf9TRYrrMLa6G6eopDzcxIR0JXIw6vJG BWLoMcaq477vKLExXYbknpdJkg+64eDMxuusQgqA0eUd7cUvJtpO4vvDZjkNrxXeEhq4 YlHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0t52fLgngXHEH1UGn44+Uw2HE3xbOfQ4NJc5Y8fZs1w=; b=JN0JW3N7jcfr8I1p5FmgFLyCZRgr2++rjTaGCT+NnLOIfVXOsO2Or5wCnW84XODJpJ credRPLaYNLNyXYUjwWxXj7Wtw2grJ4hf7l2BvhSk23MNpmETmrj8LqgHPLPAkKQhTYC aTeWc6R/ADD9BviNznMBXNGii0aBqC6rgLCbu9i8/jD6JsdH2h/nLYdGsjDxIQX8sQ/7 jzn0n4UAo3jq7GsNq0B9aVRrklQaixZXUTvz7+JwxYvilcRiUhce2hfZLV78knrovMAr QtyCP7cruzEGi2L60bHVjyxv20H27gusimzZPMmySiBTYRhZwAYjXcQduyn0Mz3yn8u+ GjfQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=BRqp1y5Z; spf=pass (google.com: domain of mike.kravetz@oracle.com designates 141.146.126.78 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2120.oracle.com (aserp2120.oracle.com. [141.146.126.78]) by mx.google.com with ESMTPS id c26-v6si8289868ybe.442.2018.12.03.12.09.14 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 03 Dec 2018 12:09:14 -0800 (PST) Received-SPF: pass (google.com: domain of mike.kravetz@oracle.com designates 141.146.126.78 as permitted sender) client-ip=141.146.126.78; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=BRqp1y5Z; spf=pass (google.com: domain of mike.kravetz@oracle.com designates 141.146.126.78 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wB3JwaRr086200; Mon, 3 Dec 2018 20:09:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=0t52fLgngXHEH1UGn44+Uw2HE3xbOfQ4NJc5Y8fZs1w=; b=BRqp1y5ZfntM59ay9BfDE/kWnTDhTBqYvoosUr7//8dE9WKWqHmYgIiajfJOFo54VCSt iVOiU4WLSSZ5Uyl8wJcql+ebxSO5atKnVKgOUKtwnIGfHI4h7GNQN74uClNF3196BUol GVOTueNl6HVPPZ7aVOIc3E+pdjyvg+pSoXP8Q1rcalcCMfzPg8EBlMr6Dj7Rzt3tv8v+ j6GPjivj0pS6qiFlPYy+2nYc3Wm3s8LoaRMmgN7/YL64+hQVDhvNMG8laHvh5h7b/xDE cxcLRJmTqxyewMoutlpgnQGb/FyMjmbARu7S73GoFTMV81+UdzQ50WEbINkBtK+ksW9m rg== Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by aserp2120.oracle.com with ESMTP id 2p3j8q8kqn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Dec 2018 20:09:07 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wB3K91Kp024324 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 3 Dec 2018 20:09:02 GMT Received: from abhmp0012.oracle.com (abhmp0012.oracle.com [141.146.116.18]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wB3K91ph006920; Mon, 3 Dec 2018 20:09:01 GMT Received: from monkey.oracle.com (/50.38.38.67) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 03 Dec 2018 12:09:01 -0800 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Michal Hocko , Hugh Dickins , Naoya Horiguchi , "Aneesh Kumar K . V" , Andrea Arcangeli , "Kirill A . Shutemov" , Davidlohr Bueso , Prakash Sangappa , Andrew Morton , Mike Kravetz , stable@vger.kernel.org Subject: [PATCH 2/3] hugetlbfs: Use i_mmap_rwsem to fix page fault/truncate race Date: Mon, 3 Dec 2018 12:08:49 -0800 Message-Id: <20181203200850.6460-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181203200850.6460-1-mike.kravetz@oracle.com> References: <20181203200850.6460-1-mike.kravetz@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9096 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812030183 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP hugetlbfs page faults can race with truncate and hole punch operations. Current code in the page fault path attempts to handle this by 'backing out' operations if we encounter the race. One obvious omission in the current code is removing a page newly added to the page cache. This is pretty straight forward to address, but there is a more subtle and difficult issue of backing out hugetlb reservations. To handle this correctly, the 'reservation state' before page allocation needs to be noted so that it can be properly backed out. There are four distinct possibilities for reservation state: shared/reserved, shared/no-resv, private/reserved and private/no-resv. Backing out a reservation may require memory allocation which could fail so that needs to be taken into account as well. Instead of writing the required complicated code for this rare occurrence, just eliminate the race. i_mmap_rwsem is now held in read mode for the duration of page fault processing. Hold i_mmap_rwsem longer in truncation and hold punch code to cover the call to remove_inode_hugepages. Cc: Fixes: ebed4bfc8da8 ("hugetlb: fix absurd HugePages_Rsvd") Signed-off-by: Mike Kravetz Signed-off-by: Mike Kravetz --- fs/hugetlbfs/inode.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 32920a10100e..3244147fc42b 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -505,8 +505,8 @@ static int hugetlb_vmtruncate(struct inode *inode, loff_t offset) i_mmap_lock_write(mapping); if (!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root)) hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0); - i_mmap_unlock_write(mapping); remove_inode_hugepages(inode, offset, LLONG_MAX); + i_mmap_unlock_write(mapping); return 0; } @@ -540,8 +540,8 @@ static long hugetlbfs_punch_hole(struct inode *inode, loff_t offset, loff_t len) hugetlb_vmdelete_list(&mapping->i_mmap, hole_start >> PAGE_SHIFT, hole_end >> PAGE_SHIFT); - i_mmap_unlock_write(mapping); remove_inode_hugepages(inode, hole_start, hole_end); + i_mmap_unlock_write(mapping); inode_unlock(inode); } From patchwork Mon Dec 3 20:08:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 10710497 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C56AA15A6 for ; Mon, 3 Dec 2018 20:09:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B70752B2E0 for ; Mon, 3 Dec 2018 20:09:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AABC82B32D; Mon, 3 Dec 2018 20:09:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0ECE82B2E0 for ; Mon, 3 Dec 2018 20:09:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8671E6B6ADB; Mon, 3 Dec 2018 15:09:13 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 678316B6ADA; Mon, 3 Dec 2018 15:09:13 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 517E36B6ADC; Mon, 3 Dec 2018 15:09:13 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id E9F366B6ADA for ; Mon, 3 Dec 2018 15:09:12 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id c34so4734921edb.8 for ; Mon, 03 Dec 2018 12:09:12 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=vxLAF8bT0VIgTD2UGzdu92swisUy1mIrbjZx+y8Wxck=; b=CGG+OHNLWmK45ywrtx32fm7ob01iYbwJffwthKd8JAum+u1KVjiEOXJD92PFxE1qQb kz5MxHLGcNX/R9yM/bEswR+vaB+i6hWo22y6wps+if158+QrihYOy42QirkPo/v1P5Ye yJPmYGwL66moNCx225RiuGsqYwLiz3nUIW26TL8RRG1SHKF+Rkx2rw0HQsxkXffiMYyL PUpKAh6XXsZnxOSNuTaGuYirE5UPuRJOmnArlWl9dy9RIMLqoKBUrIsLFbnBhike1gCU 8K4tflShWsefA1YUCxZh0+fk7MkmvD27beeVDtmMv0ZVRKpNOBgG8v3ufTpxIQckPNsR yeLw== X-Gm-Message-State: AA+aEWa5ok7xvqaRElmRwLg2Y6MhpWjTFERA1C6vk70aqMd+fT4JNC6V 8n7q90dTVI5XK/5cpOEJCtY7u4GvV49+LL1AOFXcqN8u9F6/bZfBPVmfgEAoCV/KaKn/cU1EIhr NsV0wM6Ba7bpwVoONOHVhFZNwjWjyswDPMJU2KcTs6WQHN/fgGD02DdTOEUMldnhdZQ== X-Received: by 2002:a50:9784:: with SMTP id e4mr15643210edb.165.1543867752394; Mon, 03 Dec 2018 12:09:12 -0800 (PST) X-Google-Smtp-Source: AFSGD/XhPj7ti7VNHUsNERe0XVg9edYZt4qmN0D7HJiiokaLDmmAl/GO3wto6HJaFavdU2gMrXLD X-Received: by 2002:a50:9784:: with SMTP id e4mr15643172edb.165.1543867751324; Mon, 03 Dec 2018 12:09:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1543867751; cv=none; d=google.com; s=arc-20160816; b=nOe8cpsc8LTt8mrrObMGb25ZYt3tHlSWDz9jWf6Av4PCu+3KFWGy88BFrDpT5n9GTP Te3GtnIxipS3eruZXSXfOCzK3Ya+PHhOYAJisOEmeH3+q5dPAY2XL+iY2WaLSZCi9Rmt Y6p5e5rntQSyXOLBkRdO93s6Cj+KxCmcIsXLkq0YSxwXRY/HdKMa75iydUEoFdyqYJVc LCne0/aAOBIJwH/dPmuH0NPKCff/V/UXrg5iZehW9E8pgwLl0HPL0JjI9+t4FKeizcbf 0llAa5lps1kpwjLh8k5g5Pmp/uwi0Jw1sVSSP1sXa2cyF+L7i+DMlFAmG7134OSmAKZl kQgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vxLAF8bT0VIgTD2UGzdu92swisUy1mIrbjZx+y8Wxck=; b=KUR5qwL/kX7TK1+icxkQ+1O7LoRaxOwmYBQc2+CTCWNhRDiqbVg1hq6/0yihygO1H4 6ugmw9LXxUcGQXeVXDOp0t8sNG/CMRDMOqfIq2XqKoB5plCnkIgdLrRcLp3dfGHQtQLG +NITvp8kFvmAxBSFDBYqW2QXW41/PSfaECqVuoxFi4ZoIuewLeEr1NkQdDY/gS+0nagw opv+bZ8JXnEIL9qSgz8Zn+m92zvsXC2Y19isV1HRD5XeKNiZ5SG0Etut6pWVzT74kEEN EVVvnGkKYHkX8AE2zEQ9Z/uIhAI5hnJpvVmn3gfAxiMM9LzYvYfkh05saPT3vl6bIDAX fIrg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=hegYeheN; spf=pass (google.com: domain of mike.kravetz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2120.oracle.com (userp2120.oracle.com. [156.151.31.85]) by mx.google.com with ESMTPS id e10-v6si3138979eji.18.2018.12.03.12.09.10 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 03 Dec 2018 12:09:11 -0800 (PST) Received-SPF: pass (google.com: domain of mike.kravetz@oracle.com designates 156.151.31.85 as permitted sender) client-ip=156.151.31.85; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=hegYeheN; spf=pass (google.com: domain of mike.kravetz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wB3JwZoj078799; Mon, 3 Dec 2018 20:09:04 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=vxLAF8bT0VIgTD2UGzdu92swisUy1mIrbjZx+y8Wxck=; b=hegYeheNo5PYGbcsOmPdUBgHYv5ujLGFmWxnqWCcG7cMhXP8j4RUtS3+V7pY69CWOzCF AiFW/GOsCSNQYWgrKPUyhvnWNBRYO0BKVDl3L1Hx8EeoPlieI/noPQ9GFbCreHVQI3E/ 9x+Y9fM7dpjCry5zm8owPH84kqFuWq7KcugLSVpqyS5WrxtgGcKrHHTfRs4x9+tEsJ49 MEs5G5v9VfMTHgNlG9Dpl8e9k6L450TZ7FoyfswHzt7JQBDRYNls011eVLJskd6Ow4IQ qfjy2OfjkRpYNwfIWfzkX4gBpm/glU6Wn1l+ycpfTbUrQ2dtwtyWr9YfWMcYNAK/odDY IQ== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2120.oracle.com with ESMTP id 2p3jxr8gk0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Dec 2018 20:09:04 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wB3K93xM018213 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 3 Dec 2018 20:09:03 GMT Received: from abhmp0012.oracle.com (abhmp0012.oracle.com [141.146.116.18]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wB3K93PW017194; Mon, 3 Dec 2018 20:09:03 GMT Received: from monkey.oracle.com (/50.38.38.67) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 03 Dec 2018 12:09:02 -0800 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Michal Hocko , Hugh Dickins , Naoya Horiguchi , "Aneesh Kumar K . V" , Andrea Arcangeli , "Kirill A . Shutemov" , Davidlohr Bueso , Prakash Sangappa , Andrew Morton , Mike Kravetz , stable@vger.kernel.org Subject: [PATCH 3/3] hugetlbfs: remove unnecessary code after i_mmap_rwsem synchronization Date: Mon, 3 Dec 2018 12:08:50 -0800 Message-Id: <20181203200850.6460-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181203200850.6460-1-mike.kravetz@oracle.com> References: <20181203200850.6460-1-mike.kravetz@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9096 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=890 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812030183 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP After expanding i_mmap_rwsem use for better shared pmd and page fault/ truncation synchronization, remove code that is no longer necessary. Cc: Fixes: ebed4bfc8da8 ("hugetlb: fix absurd HugePages_Rsvd") Signed-off-by: Mike Kravetz --- fs/hugetlbfs/inode.c | 46 +++++++++++++++----------------------------- mm/hugetlb.c | 21 ++++++++++---------- 2 files changed, 25 insertions(+), 42 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 3244147fc42b..a9c00c6ef80d 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -383,17 +383,16 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end) * truncation is indicated by end of range being LLONG_MAX * In this case, we first scan the range and release found pages. * After releasing pages, hugetlb_unreserve_pages cleans up region/reserv - * maps and global counts. Page faults can not race with truncation - * in this routine. hugetlb_no_page() prevents page faults in the - * truncated range. It checks i_size before allocation, and again after - * with the page table lock for the page held. The same lock must be - * acquired to unmap a page. + * maps and global counts. * hole punch is indicated if end is not LLONG_MAX * In the hole punch case we scan the range and release found pages. * Only when releasing a page is the associated region/reserv map * deleted. The region/reserv map for ranges without associated - * pages are not modified. Page faults can race with hole punch. - * This is indicated if we find a mapped page. + * pages are not modified. + * + * Callers of this routine must hold the i_mmap_rwsem in write mode to prevent + * races with page faults. + * * Note: If the passed end of range value is beyond the end of file, but * not LLONG_MAX this routine still performs a hole punch operation. */ @@ -423,32 +422,14 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart, for (i = 0; i < pagevec_count(&pvec); ++i) { struct page *page = pvec.pages[i]; - u32 hash; index = page->index; - hash = hugetlb_fault_mutex_hash(h, current->mm, - &pseudo_vma, - mapping, index, 0); - mutex_lock(&hugetlb_fault_mutex_table[hash]); - /* - * If page is mapped, it was faulted in after being - * unmapped in caller. Unmap (again) now after taking - * the fault mutex. The mutex will prevent faults - * until we finish removing the page. - * - * This race can only happen in the hole punch case. - * Getting here in a truncate operation is a bug. + * A mapped page is impossible as callers should unmap + * all references before calling. And, i_mmap_rwsem + * prevents the creation of additional mappings. */ - if (unlikely(page_mapped(page))) { - BUG_ON(truncate_op); - - i_mmap_lock_write(mapping); - hugetlb_vmdelete_list(&mapping->i_mmap, - index * pages_per_huge_page(h), - (index + 1) * pages_per_huge_page(h)); - i_mmap_unlock_write(mapping); - } + VM_BUG_ON(page_mapped(page)); lock_page(page); /* @@ -470,7 +451,6 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart, } unlock_page(page); - mutex_unlock(&hugetlb_fault_mutex_table[hash]); } huge_pagevec_release(&pvec); cond_resched(); @@ -624,7 +604,11 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, /* addr is the offset within the file (zero based) */ addr = index * hpage_size; - /* mutex taken here, fault path and hole punch */ + /* + * fault mutex taken here, protects against fault path + * and hole punch. inode_lock previously taken protects + * against truncation. + */ hash = hugetlb_fault_mutex_hash(h, mm, &pseudo_vma, mapping, index, addr); mutex_lock(&hugetlb_fault_mutex_table[hash]); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 362601b69c56..89e1a253a40b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3760,16 +3760,16 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, } /* - * Use page lock to guard against racing truncation - * before we get page_table_lock. + * We can not race with truncation due to holding i_mmap_rwsem. + * Check once here for faults beyond end of file. */ + size = i_size_read(mapping->host) >> huge_page_shift(h); + if (idx >= size) + goto out; + retry: page = find_lock_page(mapping, idx); if (!page) { - size = i_size_read(mapping->host) >> huge_page_shift(h); - if (idx >= size) - goto out; - /* * Check for page in userfault range */ @@ -3859,9 +3859,6 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, } ptl = huge_pte_lock(h, mm, ptep); - size = i_size_read(mapping->host) >> huge_page_shift(h); - if (idx >= size) - goto backout; ret = 0; if (!huge_pte_none(huge_ptep_get(ptep))) @@ -3964,8 +3961,10 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, /* * Acquire i_mmap_rwsem before calling huge_pte_alloc and hold - * until finished with ptep. This prevents huge_pmd_unshare from - * being called elsewhere and making the ptep no longer valid. + * until finished with ptep. This serves two purposes: + * 1) It prevents huge_pmd_unshare from being called elsewhere + * and making the ptep no longer valid. + * 2) It synchronizes us with file truncation. * * ptep could have already be assigned via huge_pte_offset. That * is OK, as huge_pte_alloc will return the same value unless