From patchwork Thu Aug 9 23:36:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 10562139 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 461F01057 for ; Thu, 9 Aug 2018 23:36:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 34048289DD for ; Thu, 9 Aug 2018 23:36:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 288862BB3A; Thu, 9 Aug 2018 23:36:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5B0D7289DD for ; Thu, 9 Aug 2018 23:36:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4AC7B6B000E; Thu, 9 Aug 2018 19:36:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 459636B0010; Thu, 9 Aug 2018 19:36:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 284666B0266; Thu, 9 Aug 2018 19:36:55 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id D3C436B000E for ; Thu, 9 Aug 2018 19:36:54 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id c8-v6so4240634pfn.2 for ; Thu, 09 Aug 2018 16:36:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=gAHnU0r/L5IJLhkHPCSm+EWrpcn9NZt5T126pSoAJ00=; b=YFmYkJE8QjpITUk5viWuRas+sS4u8eYNgjHVZ318K2Ne9o6H7p3i4cnldJy2o+9NvI 7cFfwmOapNghjFxNH0seSe2A5WxOQHmjwspIgK+hqmqOHCdrzH9Ck/h8Kjd7QtHAEOe4 GBd5M2QJoqyKPDJe1uQqLnl8kzX2gWlcKgTzn2EO5K+9JNWkI2Wd6PfugpTDAmNx3SK7 Fi+EKEqMlM6Dd8F58UOHY/H+iHDaQgsVOWegkXgDcSMHNRiFlBnzHqnes13y+vh3QP5v FnicVS0OTw+bxn6ao2q/R3KoME6TQexNaY/BbiLMeemG5zNcQAs7R+ugYo5EPLOeMnUe oyyA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.36 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com X-Gm-Message-State: AOUpUlH7q/rOGj1cwCWTdSEXt/4iLvGRN445CfMeg9SEkoELjY6tYCiW Bqlk/JddXB6ZH3K1DqN+imW62t9SpCCyEfuBuG8QS+VSidELwCIhWGVAVC3w5OB/tVbfSlm1QjL AzdZ6+or3uQFPkE6ixdwtcnZz3a9uGON6NW/w+Ce57tqoeajotChwyc0PddESWMb+mw== X-Received: by 2002:a17:902:158b:: with SMTP id m11-v6mr3856332pla.102.1533857814506; Thu, 09 Aug 2018 16:36:54 -0700 (PDT) X-Google-Smtp-Source: AA+uWPyB0tfoaPynxddwIFBkyjK10YdEtYPPzVMShY/HnhOpXa+IuzGU/KQ0Jv4UUUV746T74g04 X-Received: by 2002:a17:902:158b:: with SMTP id m11-v6mr3856298pla.102.1533857813475; Thu, 09 Aug 2018 16:36:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533857813; cv=none; d=google.com; s=arc-20160816; b=Wh9HE5M5C3FwZ42+gAMic4+8GmQgqlSDVdaY67kIBIXuBw6YvSPWeFwjLCxSe0AJkp A+Sjqe4/Tm1+0r4tov4uXhAxm7ehCcjMY9AzI+Uza0SU7SbanTM/5bSmNRcgPyjeXokO EzJRAYL2zGKXRNnlSOoBkEkW/S3N/2qpBA7BrmoK922r0DfUlLwjUGIjPhPiaM+HRXOs MQqthcsVN52AkpsdPmc8PtxoZ5RRqvSLI6CXiQJ2SEfKUGX7G+1spe+RtYc/BR4fp/HW Q4L1EopgJPSwgfzxyeOfIeE/BLcV1VzxJH50tP1ATO1fiki8z3L5DOZlkNyzKtr+tQyi B2RA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=gAHnU0r/L5IJLhkHPCSm+EWrpcn9NZt5T126pSoAJ00=; b=pI3SWaN9WSzWaf2we4VcGDvJXnMIslCjyFZ4I3KprlshRCnY3L1d9mjIPh7O23NE1k sRIFkzjzQtwr3jzkimyv3KH6Cj94wD2gt4wkwPk+AEWWWgblFgaZT1zVkePE+fqpM1Z4 QYRGLfdbJqA/3IjaBBSRURDBFXLenhBVoKjhi33mCUkQUrXd2xtVrphoQugfzZJODPix JY/+ADq99RIl0RJ0Wl2pF1S0xkl9nCTEwYbBVr0Y54Xld11Is7LonJynYryBP0O+V5N5 8h9AV/LKL7X71vLKAoCuhEzEWnHl24mSyoOQLC+tmrD0ziYIlJz2F3QzzCHJQWv9B1NO 0F2Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.36 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out4436.biz.mail.alibaba.com (out4436.biz.mail.alibaba.com. [47.88.44.36]) by mx.google.com with ESMTPS id p11-v6si7431802pgh.274.2018.08.09.16.36.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 09 Aug 2018 16:36:53 -0700 (PDT) Received-SPF: pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.36 as permitted sender) client-ip=47.88.44.36; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.36 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R101e4;CH=green;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04400;MF=yang.shi@linux.alibaba.com;NM=1;PH=DS;RN=15;SR=0;TI=SMTPD_---0T6MWc.W_1533857777; Received: from e19h19392.et15sqa.tbsite.net(mailfrom:yang.shi@linux.alibaba.com fp:SMTPD_---0T6MWc.W_1533857777) by smtp.aliyun-inc.com(127.0.0.1); Fri, 10 Aug 2018 07:36:26 +0800 From: Yang Shi To: mhocko@kernel.org, willy@infradead.org, ldufour@linux.vnet.ibm.com, kirill@shutemov.name, vbabka@suse.cz, akpm@linux-foundation.org, peterz@infradead.org, mingo@redhat.com, acme@kernel.org, alexander.shishkin@linux.intel.com, jolsa@redhat.com, namhyung@kernel.org Cc: yang.shi@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC v7 PATCH 2/4] mm: mmap: zap pages with read mmap_sem in munmap Date: Fri, 10 Aug 2018 07:36:01 +0800 Message-Id: <1533857763-43527-3-git-send-email-yang.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1533857763-43527-1-git-send-email-yang.shi@linux.alibaba.com> References: <1533857763-43527-1-git-send-email-yang.shi@linux.alibaba.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When running some mmap/munmap scalability tests with large memory (i.e. > 300GB), the below hung task issue may happen occasionally. INFO: task ps:14018 blocked for more than 120 seconds. Tainted: G E 4.9.79-009.ali3000.alios7.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. ps D 0 14018 1 0x00000004 ffff885582f84000 ffff885e8682f000 ffff880972943000 ffff885ebf499bc0 ffff8828ee120000 ffffc900349bfca8 ffffffff817154d0 0000000000000040 00ffffff812f872a ffff885ebf499bc0 024000d000948300 ffff880972943000 Call Trace: [] ? __schedule+0x250/0x730 [] schedule+0x36/0x80 [] rwsem_down_read_failed+0xf0/0x150 [] call_rwsem_down_read_failed+0x18/0x30 [] down_read+0x20/0x40 [] proc_pid_cmdline_read+0xd9/0x4e0 [] ? do_filp_open+0xa5/0x100 [] __vfs_read+0x37/0x150 [] ? security_file_permission+0x9b/0xc0 [] vfs_read+0x96/0x130 [] SyS_read+0x55/0xc0 [] entry_SYSCALL_64_fastpath+0x1a/0xc5 It is because munmap holds mmap_sem exclusively from very beginning to all the way down to the end, and doesn't release it in the middle. When unmapping large mapping, it may take long time (take ~18 seconds to unmap 320GB mapping with every single page mapped on an idle machine). Zapping pages is the most time consuming part, according to the suggestion from Michal Hocko [1], zapping pages can be done with holding read mmap_sem, like what MADV_DONTNEED does. Then re-acquire write mmap_sem to cleanup vmas. But, some part may need write mmap_sem, for example, vma splitting. So, the design is as follows: acquire write mmap_sem lookup vmas (find and split vmas) deal with special mappings detach vmas downgrade_write zap pages free page tables release mmap_sem The vm events with read mmap_sem may come in during page zapping, but since vmas have been detached before, they, i.e. page fault, gup, etc, will not be able to find valid vma, then just return SIGSEGV or -EFAULT as expected. If the vma has VM_HUGETLB | VM_PFNMAP or uprobe, they are considered as special mappings. For the safer and bisectable sake, they will be handled by falling back to regular do_munmap() with exclusive mmap_sem held in a separate patch. Since it may be not safe to update vm_flags with read mmap_sem, although it sounds safe for this specific case hence vmas have been detached. With the "detach vmas first" approach we don't have to re-acquire mmap_sem again to clean up vmas to avoid race window which might get the address space changed since downgrade_write() doesn't release the lock to lead regression, which simply downgrades to read lock. And, since the lock acquire/release cost is managed to the minimum and almost as same as before, the optimization could be extended to any size of mapping without incurring significant penalty to small mappings. For the time being, just do this in munmap syscall path. Other vm_munmap() or do_munmap() call sites (i.e mmap, mremap, etc) remain intact due to some implementation difficulties since they acquire write mmap_sem from very beginning and hold it until the end, do_munmap() might be called in the middle. But, the optimized do_munmap would like to be called without mmap_sem held so that we can do the optimization. So, if we want to do the similar optimization for mmap/mremap path, I'm afraid we would have to redesign them. mremap might be called on very large area depending on the usecases, the optimization to it will be considered in the future. With the patches, exclusive mmap_sem hold time when munmap a 80GB address space on a machine with 32 cores of E5-2680 @ 2.70GHz dropped to us level from second. munmap_test-15002 [008] 594.380138: funcgraph_entry: | vm_munmap_zap_rlock() { munmap_test-15002 [008] 594.380146: funcgraph_entry: !2485684 us | unmap_region(); munmap_test-15002 [008] 596.865836: funcgraph_exit: !2485692 us | } Here the excution time of unmap_region() is used to evaluate the time of holding read mmap_sem, then the remaining time is used with holding exclusive lock. [1] https://lwn.net/Articles/753269/ Suggested-by: Michal Hocko Suggested-by: Kirill A. Shutemov Cc: Matthew Wilcox Cc: Laurent Dufour Cc: Andrew Morton Signed-off-by: Yang Shi --- mm/mmap.c | 81 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 79 insertions(+), 2 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 2a6898b..2234d5a 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2766,6 +2766,73 @@ static inline void munlock_vmas(struct vm_area_struct *vma, } } +/* + * Zap pages with read mmap_sem held + * + * uf is the list for userfaultfd + */ +static int do_munmap_zap_rlock(struct mm_struct *mm, unsigned long start, + size_t len, struct list_head *uf) +{ + unsigned long end; + struct vm_area_struct *start_vma, *prev, *vma; + int ret = 0; + + if (!addr_ok(start, len)) + return -EINVAL; + + len = PAGE_ALIGN(len); + + end = start + len; + + /* + * Need write mmap_sem to split vmas and detach vmas + * splitting vma up-front to save PITA to clean if it is failed + */ + if (down_write_killable(&mm->mmap_sem)) + return -EINTR; + + start_vma = munmap_lookup_vma(mm, start, end); + if (!start_vma) + goto out; + if (IS_ERR(start_vma)) { + ret = PTR_ERR(start_vma); + goto out; + } + + prev = start_vma->vm_prev; + + if (unlikely(uf)) { + ret = userfaultfd_unmap_prep(start_vma, start, end, uf); + if (ret) + goto out; + } + + /* Handle mlocked vmas */ + if (mm->locked_vm) { + vma = start_vma; + munlock_vmas(vma, end); + } + + /* Detach vmas from rbtree */ + detach_vmas_to_be_unmapped(mm, start_vma, prev, end); + + downgrade_write(&mm->mmap_sem); + + /* Zap mappings with read mmap_sem */ + unmap_region(mm, start_vma, prev, start, end); + + arch_unmap(mm, start_vma, start, end); + remove_vma_list(mm, start_vma); + up_read(&mm->mmap_sem); + + return 0; + +out: + up_write(&mm->mmap_sem); + return ret; +} + /* Munmap is split into 2 main parts -- this part which finds * what needs doing, and the areas themselves, which do the * work. This now handles partial unmappings. @@ -2829,6 +2896,17 @@ int do_munmap(struct mm_struct *mm, unsigned long start, size_t len, return 0; } +static int vm_munmap_zap_rlock(unsigned long start, size_t len) +{ + int ret; + struct mm_struct *mm = current->mm; + LIST_HEAD(uf); + + ret = do_munmap_zap_rlock(mm, start, len, &uf); + userfaultfd_unmap_complete(mm, &uf); + return ret; +} + int vm_munmap(unsigned long start, size_t len) { int ret; @@ -2848,10 +2926,9 @@ int vm_munmap(unsigned long start, size_t len) SYSCALL_DEFINE2(munmap, unsigned long, addr, size_t, len) { profile_munmap(addr); - return vm_munmap(addr, len); + return vm_munmap_zap_rlock(addr, len); } - /* * Emulation of deprecated remap_file_pages() syscall. */