From patchwork Sun Dec 1 01:57:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11268441 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D1C9317E0 for ; Sun, 1 Dec 2019 01:57:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 95574205ED for ; Sun, 1 Dec 2019 01:57:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="HjD6xP8A" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 95574205ED Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 61DE46B0367; Sat, 30 Nov 2019 20:57:08 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5CE2A6B0369; Sat, 30 Nov 2019 20:57:08 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 50BDB6B036B; Sat, 30 Nov 2019 20:57:08 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0191.hostedemail.com [216.40.44.191]) by kanga.kvack.org (Postfix) with ESMTP id 385036B0367 for ; Sat, 30 Nov 2019 20:57:08 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id E3723180AD817 for ; Sun, 1 Dec 2019 01:57:07 +0000 (UTC) X-FDA: 76214909694.05.scene21_3e3abac5c0610 X-Spam-Summary: 10,1,0,eafa94c0bceb896c,d41d8cd98f00b204,akpm@linux-foundation.org,:akpm@linux-foundation.org:hzhongzhang@tencent.com:knightzhang@tencent.com::mike.kravetz@oracle.com:mm-commits@vger.kernel.org:tonnylu@tencent.com:torvalds@linux-foundation.org:willy@infradead.org,RULES_HIT:41:355:379:588:800:960:967:973:988:989:1260:1263:1345:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:2393:2525:2559:2563:2682:2685:2859:2892:2895:2898:2902:2924:2926:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3870:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4605:5007:6261:6630:6653:7514:7576:9025:9121:9545:10004:10913:11026:11233:11658:11914:12043:12048:12295:12296:12297:12438:12517:12519:12555:12679:12783:12986:13225:13229:13255:13846:13870:14181:14721:14849:21080:21324:21433:21451:21627:21819:21939:30054:30056:30064:30070:30075,0,RBL:error,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,S PF:fp,MS X-HE-Tag: scene21_3e3abac5c0610 X-Filterd-Recvd-Size: 3532 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf42.hostedemail.com (Postfix) with ESMTP for ; Sun, 1 Dec 2019 01:57:07 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 82493215E5; Sun, 1 Dec 2019 01:57:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1575165426; bh=7+wvV3gZq5vFyvYUTM8y6xf9bmqVI1iWgdN4mIOvVH0=; h=Date:From:To:Subject:From; b=HjD6xP8AfTV8V/eyM/XH0vQ/MuZA/QH00B8yBnBMMowMsVtiAGmirJMQJ4zywcTmF CehhpEjTLVpdO3mfztEho7oXeD1HbzkQWEsZeA9ykxiX5G9atVHjEaFF7yiLpm0jSJ 7oshZL35rZtY+iMFeSgbLOBKQ73n/owYC1C1VGPQ= Date: Sat, 30 Nov 2019 17:57:06 -0800 From: akpm@linux-foundation.org To: akpm@linux-foundation.org, hzhongzhang@tencent.com, knightzhang@tencent.com, linux-mm@kvack.org, mike.kravetz@oracle.com, mm-commits@vger.kernel.org, tonnylu@tencent.com, torvalds@linux-foundation.org, willy@infradead.org Subject: [patch 132/158] mm/hugetlb: avoid looping to the same hugepage if !pages and !vmas Message-ID: <20191201015706.RdZ5u8gLH%akpm@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zhigang Lu Subject: mm/hugetlb: avoid looping to the same hugepage if !pages and !vmas When mmapping an existing hugetlbfs file with MAP_POPULATE, we find it is very time consuming. For example, mmapping a 128GB file takes about 50 milliseconds. Sampling with perfevent shows it spends 99% time in the same_page loop in follow_hugetlb_page(). samples: 205 of event 'cycles', Event count (approx.): 136686374 - 99.04% test_mmap_huget [kernel.kallsyms] [k] follow_hugetlb_page follow_hugetlb_page __get_user_pages __mlock_vma_pages_range __mm_populate vm_mmap_pgoff sys_mmap_pgoff sys_mmap system_call_fastpath __mmap64 follow_hugetlb_page() is called with pages=NULL and vmas=NULL, so for each hugepage, we run into the same_page loop for pages_per_huge_page() times, but doing nothing. With this change, it takes less then 1 millisecond to mmap a 128GB file in hugetlbfs. Link: http://lkml.kernel.org/r/1567581712-5992-1-git-send-email-totty.lu@gmail.com Signed-off-by: Zhigang Lu Reviewed-by: Haozhong Zhang Reviewed-by: Zongming Zhang Reviewed-by: Mike Kravetz Acked-by: Matthew Wilcox Signed-off-by: Andrew Morton --- mm/hugetlb.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) --- a/mm/hugetlb.c~mm-hugetlb-avoid-looping-to-the-same-hugepage-if-pages-and-vmas +++ a/mm/hugetlb.c @@ -4338,6 +4338,21 @@ long follow_hugetlb_page(struct mm_struc break; } } + + /* + * If subpage information not requested, update counters + * and skip the same_page loop below. + */ + if (!pages && !vmas && !pfn_offset && + (vaddr + huge_page_size(h) < vma->vm_end) && + (remainder >= pages_per_huge_page(h))) { + vaddr += huge_page_size(h); + remainder -= pages_per_huge_page(h); + i += pages_per_huge_page(h); + spin_unlock(ptl); + continue; + } + same_page: if (pages) { pages[i] = mem_map_offset(page, pfn_offset);