From patchwork Wed Apr 7 01:44:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186465 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E96EC433ED for ; Wed, 7 Apr 2021 01:45:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CF42A61165 for ; Wed, 7 Apr 2021 01:45:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CF42A61165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EB8158E0005; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 58E418E0013; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 711056B0093; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0243.hostedemail.com [216.40.44.243]) by kanga.kvack.org (Postfix) with ESMTP id 8B8AC8E0001 for ; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 4E592181AEF3E for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) X-FDA: 78003877896.29.8C87004 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf02.hostedemail.com (Postfix) with ESMTP id F2A9840002C3 for ; Wed, 7 Apr 2021 01:44:59 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=XkD4f0ULJymIOIeAqVxcBhuCK1FNqNRxb6Ygkc1ApCs=; b=T+2gjG7c+P1JGIgyykqnKaQiD2Wtt2L+tO+pRnvCt2JO1FKjXP26ZCM6hHIjsZg0F91Pv IUbkovCOPxRyi8dAA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=XkD4f0ULJymIOIeAqVxcBhuCK1FNqNRxb6Ygkc1ApCs=; b=RTBwrQVEuGgTfXMpMfbrh/D5mOUGsMVm80V0u7rXaSdXDkPKYPpLKgLTiCRWVRop96qn6 0H69mSmJu7OkmjqPwNX7g+7lzVmBTBOnkqtihAaT+U/O2gxSb293SkAW7EIfEHbmGHMYZWk pOscXZjW7Z0CSgRkvL7L15AFZtvdGsJOlu2QZxqeLxSHccd/OsdnNeVdL56bmw0Rd1+uEX3 VukmTQUvytixFC1AIzrmU1IDhnsHFdOkwUHs5+5/3GbtMI8p+2GamYUAQkZO2nK072EtqZp iyjY45FT2zBP25QHQ+U8WOHV240vWPDjZQGCB3bXmbBMsDK2fiPRHyEjp86w== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 35EA7160531; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 26A2B19F31D; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 28/37] mm: implement speculative handling in filemap_map_pages() Date: Tue, 6 Apr 2021 18:44:53 -0700 Message-Id: <20210407014502.24091-29-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: F2A9840002C3 X-Stat-Signature: 5bbku3x36usu9kx8gjwksuyxbexpcgzo Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf02; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759899-375029 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In the speculative case, we know the page table already exists, and it must be locked with pte_map_lock(). In the case where no page is found for the given address, return VM_FAULT_RETRY which will abort the fault before we get into the vm_ops->fault() callback. This is fine because if filemap_map_pages does not find the page in page cache, vm_ops->fault() will not either. Initialize addr and last_pgoff to correspond to the pte at the original fault address (which was mapped with pte_map_lock()), rather than the pte at start_pgoff. The choice of initial values doesn't matter as they will all be adjusted together before use, so they just need to be consistent with each other, and using the original fault address and pte allows us to reuse pte_map_lock() without any changes to it. Signed-off-by: Michel Lespinasse --- mm/filemap.c | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 6e8505fe5df9..d496771749e6 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3136,25 +3136,31 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, struct vm_area_struct *vma = vmf->vma; struct file *file = vma->vm_file; struct address_space *mapping = file->f_mapping; - pgoff_t last_pgoff = start_pgoff; + pgoff_t last_pgoff; unsigned long addr; XA_STATE(xas, &mapping->i_pages, start_pgoff); struct page *head, *page; unsigned int mmap_miss = READ_ONCE(file->f_ra.mmap_miss); - vm_fault_t ret = 0; + vm_fault_t ret = (vmf->flags & FAULT_FLAG_SPECULATIVE) ? + VM_FAULT_RETRY : 0; - rcu_read_lock(); + /* filemap_map_pages() is called within an rcu read lock already. */ head = first_map_page(mapping, &xas, end_pgoff); if (!head) - goto out; + return ret; - if (filemap_map_pmd(vmf, head)) { - ret = VM_FAULT_NOPAGE; - goto out; + if (!(vmf->flags & FAULT_FLAG_SPECULATIVE) && + filemap_map_pmd(vmf, head)) + return VM_FAULT_NOPAGE; + + if (!pte_map_lock(vmf)) { + unlock_page(head); + put_page(head); + return VM_FAULT_RETRY; } + addr = vmf->address; + last_pgoff = vmf->pgoff; - addr = vma->vm_start + ((start_pgoff - vma->vm_pgoff) << PAGE_SHIFT); - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl); do { page = find_subpage(head, xas.xa_index); if (PageHWPoison(page)) @@ -3184,8 +3190,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, put_page(head); } while ((head = next_map_page(mapping, &xas, end_pgoff)) != NULL); pte_unmap_unlock(vmf->pte, vmf->ptl); -out: - rcu_read_unlock(); + vmf->pte = NULL; WRITE_ONCE(file->f_ra.mmap_miss, mmap_miss); return ret; }