From patchwork Thu Jul 26 18:10:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 10546293 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 35DE8180E for ; Thu, 26 Jul 2018 18:11:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 28F8B2BBF0 for ; Thu, 26 Jul 2018 18:11:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 247662BBF3; Thu, 26 Jul 2018 18:11:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 564DD2BC12 for ; Thu, 26 Jul 2018 18:11:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5A65D6B000E; Thu, 26 Jul 2018 14:11:15 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 52D8B6B0010; Thu, 26 Jul 2018 14:11:15 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3CE3E6B0266; Thu, 26 Jul 2018 14:11:15 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl0-f72.google.com (mail-pl0-f72.google.com [209.85.160.72]) by kanga.kvack.org (Postfix) with ESMTP id EDB126B000E for ; Thu, 26 Jul 2018 14:11:14 -0400 (EDT) Received: by mail-pl0-f72.google.com with SMTP id m15-v6so1721743pls.23 for ; Thu, 26 Jul 2018 11:11:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:mime-version:content-transfer-encoding; bh=mLZDgUtEU59YX9Fpgdagh+ZyqzU9cm7Ru48OlV6yIds=; b=XGG9PGo1TWY66AHcuJo2WfL3+Glsc296G0IhBhafOK3ZIxQ3Fd+/iv0fvyMOaN0Pc9 GI1K5M7R69KlwhUK2paok/QbK+c2GO6rqtZabpgiwHTEyWy9ooS2PTJrF6zDKcJgcYf4 ZZmcZQhx69lnqT+XlAeiy6UMjJEFwVXWykEVaIu8uEGoJII4C/LTuIz/huV3yNCoHwY3 q+U3hI8U0Ow+iRi4oV2dpV0B7nw0FJ/U/LdD6VRcLc1ZUHzvS7EMNa/qQqiAftDMjIAW yr39RtVubBl7EM46kMFd55sPj4Io0tcv9B/MsMU1CPcWbpXjVtCk0nY0UHBZVV8rzVry IzuQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.36 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com X-Gm-Message-State: AOUpUlHWeWXR7zriRkmEi1gD3kXiCd9NhuQWMYEJpKsB20fSeMGH1xlp QpukFUcZRuJoAOxYwbSsyuozLvpOkGAHRmqUXru4awg2sUIig3NREptBVZa2Aeu9u4oeYhXuC9f GYQCTc9Tol0Huqppx3K/dU62Bei93IVeviNyo5Zks4IBXtI6Otmeno4/LfOU9Ml6A0Q== X-Received: by 2002:a62:198e:: with SMTP id 136-v6mr3140214pfz.103.1532628674627; Thu, 26 Jul 2018 11:11:14 -0700 (PDT) X-Google-Smtp-Source: AAOMgpfqYDRauYRdPWOu6qxgL0vNAH2gUPqvCtQginAi3O+PxMfYgFVrR+D4z+B9+b2iOjx+q6Vm X-Received: by 2002:a62:198e:: with SMTP id 136-v6mr3140167pfz.103.1532628673661; Thu, 26 Jul 2018 11:11:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1532628673; cv=none; d=google.com; s=arc-20160816; b=e3qYNHmMg0S53cxiaBX7J1BSnk5GgnthZCj5IMSpT0n+fouEx2mwEHd5DKPinVTeZ5 NpK5wQG8tq9gi9f9TJ4Wi8PcK7U5QDH6jczUFJUU1mXaH9ULWPoFTcSBKjj8XTxd+FEP f6MabK665LZRKMTQ4HcCP3dOZUy3blBDl4f+62nMRz+bYpNoXsYkLg4Pvs08O86UCPvj uUeLRmrrtLrn46yUQaY5/7ke7lLrVHpH6Lr+fXginJKZAeSa6x0QU0IM8kMujwTW3bUq +0fDPjR8PGwhqXKV8Hup3/KoHv+dJ2snucr1HZ5Kr/YKEa//qkn5guqS+TqplLLuVKkI E2CQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:arc-authentication-results; bh=mLZDgUtEU59YX9Fpgdagh+ZyqzU9cm7Ru48OlV6yIds=; b=S6BknWxTci/fn3wZhB6KAgMl2yIkVCb33k8QrbYXuPPMw5xVhtmMFcGwX5cRXKAoM6 xceVOKT8cpFl707blLmSNXBE9e/QC5qSsTpycAWYA/CtCLIxy8OFlG2RWNOsuPP/aYMG rJx03ZW27vd4aFRnOh7/5XqPlqFq/p5NpKt9rbRlJ0VVH8YHuuG/yR+erxtnhv//o2Oq +YF/fvOurDAF9TqMX/Cw7X8R4A207ht/hhiSutOoVWepg62IbjiRafT4CDoKBhB9+DOE DXfo7Gkt6DWD2eHGEuJfHb4eE4cYEbUZ11eXD6+qFtudniAHGhhZAvU9AMI5PXXH/34+ RVYQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.36 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out4436.biz.mail.alibaba.com (out4436.biz.mail.alibaba.com. [47.88.44.36]) by mx.google.com with ESMTPS id 194-v6si2157026pfz.101.2018.07.26.11.11.12 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 26 Jul 2018 11:11:13 -0700 (PDT) Received-SPF: pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.36 as permitted sender) client-ip=47.88.44.36; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.36 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R121e4;CH=green;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e01422;MF=yang.shi@linux.alibaba.com;NM=1;PH=DS;RN=8;SR=0;TI=SMTPD_---0T5ObfW8_1532628628; Received: from e19h19392.et15sqa.tbsite.net(mailfrom:yang.shi@linux.alibaba.com fp:SMTPD_---0T5ObfW8_1532628628) by smtp.aliyun-inc.com(127.0.0.1); Fri, 27 Jul 2018 02:10:35 +0800 From: Yang Shi To: mhocko@kernel.org, willy@infradead.org, ldufour@linux.vnet.ibm.com, kirill@shutemov.name, akpm@linux-foundation.org Cc: yang.shi@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC v6 PATCH 0/2] mm: zap pages with read mmap_sem in munmap for large mapping Date: Fri, 27 Jul 2018 02:10:12 +0800 Message-Id: <1532628614-111702-1-git-send-email-yang.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Background: Recently, when we ran some vm scalability tests on machines with large memory, we ran into a couple of mmap_sem scalability issues when unmapping large memory space, please refer to https://lkml.org/lkml/2017/12/14/733 and https://lkml.org/lkml/2018/2/20/576. History: Then akpm suggested to unmap large mapping section by section and drop mmap_sem at a time to mitigate it (see https://lkml.org/lkml/2018/3/6/784). V1 patch series was submitted to the mailing list per Andrew's suggestion (see https://lkml.org/lkml/2018/3/20/786). Then I received a lot great feedback and suggestions. Then this topic was discussed on LSFMM summit 2018. In the summit, Michal Hocko suggested (also in the v1 patches review) to try "two phases" approach. Zapping pages with read mmap_sem, then doing via cleanup with write mmap_sem (for discussion detail, see https://lwn.net/Articles/753269/) Approach: Zapping pages is the most time consuming part, according to the suggestion from Michal Hocko [1], zapping pages can be done with holding read mmap_sem, like what MADV_DONTNEED does. Then re-acquire write mmap_sem to cleanup vmas. But, we can't call MADV_DONTNEED directly, since there are two major drawbacks: * The unexpected state from PF if it wins the race in the middle of munmap. It may return zero page, instead of the content or SIGSEGV. * Can’t handle VM_LOCKED | VM_HUGETLB | VM_PFNMAP and uprobe mappings, which is a showstopper from akpm But, some part may need write mmap_sem, for example, vma splitting. So, the design is as follows: acquire write mmap_sem lookup vmas (find and split vmas) detach vmas deal with special mappings downgrade_write zap pages free page tables release mmap_sem The vm events with read mmap_sem may come in during page zapping, but since vmas have been detached before, they, i.e. page fault, gup, etc, will not be able to find valid vma, then just return SIGSEGV or -EFAULT as expected. If the vma has VM_LOCKED | VM_HUGETLB | VM_PFNMAP or uprobe, they are considered as special mappings. They will be dealt with before zapping pages with write mmap_sem held. Basically, just update vm_flags. And, since they are also manipulated by unmap_single_vma() which is called by unmap_vma() with read mmap_sem held in this case, to prevent from updating vm_flags in read critical section, a new parameter, called "skip_flags" is added to unmap_region(), unmap_vmas() and unmap_single_vma(). If it is true, then just skip unmap those special mappings. Currently, the only place which pass true to this parameter is us. With this approach we don't have to re-acquire mmap_sem again to clean up vmas to avoid race window which might get the address space changed. And, since the lock acquire/release cost is managed to the minimum and almost as same as before, the optimization could be extended to any size of mapping without incuring significant penalty to small mappings. For the time being, just do this in munmap syscall path. Other vm_munmap() or do_munmap() call sites (i.e mmap, mremap, etc) remain intact for stability reason. Changelog: v5 -> v6: * Fixed the comments from Kirill and Laurent * Added Laurent's reviewed-by to patch 1/2. Thanks. v4 -> v5: * Detach vmas before zapping pages so that we don't have to use VM_DEAD to mark a being unmapping vma since they have been detached from rbtree when zapping pages. Per Kirill * Eliminate VM_DEAD stuff * With this change we don't have to re-acquire write mmap_sem to do cleanup. So, we could eliminate a potential race window * Eliminate PUD_SIZE check, and extend this optimization to all size v3 -> v4: * Extend check_stable_address_space to check VM_DEAD as Michal suggested * Deal with vm_flags update of VM_LOCKED | VM_HUGETLB | VM_PFNMAP and uprobe mappings with exclusive lock held. The actual unmapping is still done with read mmap_sem to solve akpm's concern * Clean up vmas with calling do_munmap to prevent from race condition by not carrying vmas as Kirill suggested * Extracted more common code * Solved some code cleanup comments from akpm * Dropped uprobe and arch specific code, now all the changes are mm only * Still keep PUD_SIZE threshold, if everyone thinks it is better to extend to all sizes or smaller size, will remove it * Make this optimization 64 bit only explicitly per akpm's suggestion v2 -> v3: * Refactor do_munmap code to extract the common part per Peter's sugestion * Introduced VM_DEAD flag per Michal's suggestion. Just handled VM_DEAD in x86's page fault handler for the time being. Other architectures will be covered once the patch series is reviewed * Now lookup vma (find and split) and set VM_DEAD flag with write mmap_sem, then zap mapping with read mmap_sem, then clean up pgtables and vmas with write mmap_sem per Peter's suggestion v1 -> v2: * Re-implemented the code per the discussion on LSFMM summit Regression and performance data: Did the below regression test with setting thresh to 4K manually in the code: * Full LTP * Trinity (munmap/all vm syscalls) * Stress-ng: mmap/mmapfork/mmapfixed/mmapaddr/mmapmany/vm * mm-tests: kernbench, phpbench, sysbench-mariadb, will-it-scale * vm-scalability With the patches, exclusive mmap_sem hold time when munmap a 80GB address space on a machine with 32 cores of E5-2680 @ 2.70GHz dropped to us level from second. munmap_test-15002 [008] 594.380138: funcgraph_entry: | vm_munmap_zap_rlock() { munmap_test-15002 [008] 594.380146: funcgraph_entry: !2485684 us | unmap_region(); munmap_test-15002 [008] 596.865836: funcgraph_exit: !2485692 us | } Here the excution time of unmap_region() is used to evaluate the time of holding read mmap_sem, then the remaining time is used with holding exclusive lock. Yang Shi (2): mm: refactor do_munmap() to extract the common part mm: mmap: zap pages with read mmap_sem in munmap include/linux/mm.h | 2 +- mm/memory.c | 41 ++++++++++---- mm/mmap.c | 219 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------------- 3 files changed, 205 insertions(+), 57 deletions(-)