From patchwork Fri Jul 22 12:19:31 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kara X-Patchwork-Id: 9243441 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 817E1602F0 for ; Fri, 22 Jul 2016 12:20:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 722A327FA3 for ; Fri, 22 Jul 2016 12:20:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 63DF627FA7; Fri, 22 Jul 2016 12:20:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D11EE27FA5 for ; Fri, 22 Jul 2016 12:20:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753964AbcGVMT5 (ORCPT ); Fri, 22 Jul 2016 08:19:57 -0400 Received: from mx2.suse.de ([195.135.220.15]:49621 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753880AbcGVMTz (ORCPT ); Fri, 22 Jul 2016 08:19:55 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 17E10AD74; Fri, 22 Jul 2016 12:19:51 +0000 (UTC) Received: by quack2.suse.cz (Postfix, from userid 1000) id A18381E0F1E; Fri, 22 Jul 2016 14:19:47 +0200 (CEST) From: Jan Kara To: linux-mm@kvack.org Cc: linux-fsdevel@vger.kernel.org, linux-nvdimm@lists.01.org, Dan Williams , Ross Zwisler , Jan Kara Subject: [PATCH 05/15] mm: Factor out functionality to finish page faults Date: Fri, 22 Jul 2016 14:19:31 +0200 Message-Id: <1469189981-19000-6-git-send-email-jack@suse.cz> X-Mailer: git-send-email 2.6.6 In-Reply-To: <1469189981-19000-1-git-send-email-jack@suse.cz> References: <1469189981-19000-1-git-send-email-jack@suse.cz> Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Introduce function finish_fault() which handles locking of page tables and insertion of PTE after page for the page fault is prepared. This will be somewhat easier to use from page fault handlers than current do_set_pte() which is unnecessarily low-level for most uses. Signed-off-by: Jan Kara --- include/linux/mm.h | 1 + mm/memory.c | 67 ++++++++++++++++++++++++++++++++++++++---------------- 2 files changed, 48 insertions(+), 20 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 2442f972bdc8..21226cc2b1cd 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -606,6 +606,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma) void do_set_pte(struct vm_area_struct *vma, unsigned long address, struct page *page, pte_t *pte, bool write, bool anon); +int finish_fault(struct vm_area_struct *vma, struct vm_fault *vmf); #endif /* diff --git a/mm/memory.c b/mm/memory.c index aef88d634072..b785f823caa4 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2890,6 +2890,49 @@ void do_set_pte(struct vm_area_struct *vma, unsigned long address, update_mmu_cache(vma, address, pte); } +/** + * finish_fault - finish page fault once we have prepared the page to fault + * + * @vma: virtual memory area + * @vmf: structure describing the fault + * + * This function handles all that is needed to finish a page fault once the + * page to fault in is prepared. It handles locking of PTEs, inserts PTE for + * given page, adds reverse page mapping, handles memcg charges and LRU + * addition. The function returns 0 on success, error in case page could not + * be inserted into page tables. + * + * The function expects the page to be locked. + */ +int finish_fault(struct vm_area_struct *vma, struct vm_fault *vmf) +{ + unsigned long address = (unsigned long)vmf->virtual_address; + struct page *page = vmf->page; + bool anon = false; + spinlock_t *ptl; + pte_t *pte; + + if (vmf->cow_page) { + page = vmf->cow_page; + anon = true; + } + + pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, address, &ptl); + if (unlikely(!pte_same(*pte, vmf->orig_pte))) { + pte_unmap_unlock(pte, ptl); + return -EBUSY; + } + do_set_pte(vma, address, page, pte, vmf->flags & FAULT_FLAG_WRITE, + anon); + if (anon) { + mem_cgroup_commit_charge(page, vmf->memcg, false, false); + lru_cache_add_active_or_unevictable(page, vma); + } + pte_unmap_unlock(pte, ptl); + + return 0; +} + static unsigned long fault_around_bytes __read_mostly = rounddown_pow_of_two(65536); @@ -3022,15 +3065,13 @@ static int do_read_fault(struct mm_struct *mm, struct vm_area_struct *vma, if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY))) return ret; - pte = pte_offset_map_lock(mm, vmf->pmd, address, &ptl); - if (unlikely(!pte_same(*pte, vmf->orig_pte))) { - pte_unmap_unlock(pte, ptl); + if (unlikely(finish_fault(vma, vmf) < 0)) { unlock_page(vmf->page); put_page(vmf->page); return ret; } - do_set_pte(vma, address, vmf->page, pte, false, false); unlock_page(vmf->page); + return ret; unlock_out: pte_unmap_unlock(pte, ptl); return ret; @@ -3041,8 +3082,6 @@ static int do_cow_fault(struct mm_struct *mm, struct vm_area_struct *vma, { struct page *new_page; struct mem_cgroup *memcg; - spinlock_t *ptl; - pte_t *pte; int ret; unsigned long address = (unsigned long)vmf->virtual_address; @@ -3070,9 +3109,7 @@ static int do_cow_fault(struct mm_struct *mm, struct vm_area_struct *vma, copy_user_highpage(new_page, vmf->page, address, vma); __SetPageUptodate(new_page); - pte = pte_offset_map_lock(mm, vmf->pmd, address, &ptl); - if (unlikely(!pte_same(*pte, vmf->orig_pte))) { - pte_unmap_unlock(pte, ptl); + if (unlikely(finish_fault(vma, vmf) < 0)) { if (!(ret & VM_FAULT_DAX_LOCKED)) { unlock_page(vmf->page); put_page(vmf->page); @@ -3082,10 +3119,6 @@ static int do_cow_fault(struct mm_struct *mm, struct vm_area_struct *vma, } goto uncharge_out; } - do_set_pte(vma, address, new_page, pte, true, true); - mem_cgroup_commit_charge(new_page, memcg, false, false); - lru_cache_add_active_or_unevictable(new_page, vma); - pte_unmap_unlock(pte, ptl); if (!(ret & VM_FAULT_DAX_LOCKED)) { unlock_page(vmf->page); put_page(vmf->page); @@ -3104,8 +3137,6 @@ static int do_shared_fault(struct mm_struct *mm, struct vm_area_struct *vma, { struct address_space *mapping; unsigned long address = (unsigned long)vmf->virtual_address; - spinlock_t *ptl; - pte_t *pte; int dirtied = 0; int ret, tmp; @@ -3128,15 +3159,11 @@ static int do_shared_fault(struct mm_struct *mm, struct vm_area_struct *vma, } } - pte = pte_offset_map_lock(mm, vmf->pmd, address, &ptl); - if (unlikely(!pte_same(*pte, vmf->orig_pte))) { - pte_unmap_unlock(pte, ptl); + if (unlikely(finish_fault(vma, vmf) < 0)) { unlock_page(vmf->page); put_page(vmf->page); return ret; } - do_set_pte(vma, address, vmf->page, pte, true, false); - pte_unmap_unlock(pte, ptl); if (set_page_dirty(vmf->page)) dirtied = 1;