From patchwork Thu Oct 10 13:58:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Yang X-Patchwork-Id: 11183651 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C36D21575 for ; Thu, 10 Oct 2019 13:59:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 99B5C21835 for ; Thu, 10 Oct 2019 13:59:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 99B5C21835 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AA9428E0003; Thu, 10 Oct 2019 09:59:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A5A2B6B0005; Thu, 10 Oct 2019 09:59:23 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 949228E0003; Thu, 10 Oct 2019 09:59:23 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0198.hostedemail.com [216.40.44.198]) by kanga.kvack.org (Postfix) with ESMTP id 724556B0003 for ; Thu, 10 Oct 2019 09:59:23 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with SMTP id EDF4C55FAF for ; Thu, 10 Oct 2019 13:59:22 +0000 (UTC) X-FDA: 76028032164.02.lip53_8c08200272119 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,richardw.yang@linux.intel.com,:akpm@linux-foundation.org:kirill.shutemov@linux.intel.com:jglisse@redhat.com:mike.kravetz@oracle.com:riel@surriel.com:khlebnikov@yandex-team.ru:cai@lca.pw:shakeelb@google.com::linux-kernel@vger.kernel.org:richardw.yang@linux.intel.com,RULES_HIT:30054:30070:30079,0,RBL:192.55.52.115:@linux.intel.com:.lbl8.mailshell.net-62.18.0.100 64.95.201.95,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: lip53_8c08200272119 X-Filterd-Recvd-Size: 5225 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by imf11.hostedemail.com (Postfix) with ESMTP for ; Thu, 10 Oct 2019 13:59:22 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 10 Oct 2019 06:59:20 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.67,280,1566889200"; d="scan'208";a="277785443" Received: from richard.sh.intel.com (HELO localhost) ([10.239.159.54]) by orsmga001.jf.intel.com with ESMTP; 10 Oct 2019 06:59:17 -0700 From: Wei Yang To: akpm@linux-foundation.org, kirill.shutemov@linux.intel.com, jglisse@redhat.com, mike.kravetz@oracle.com, riel@surriel.com, khlebnikov@yandex-team.ru, cai@lca.pw, shakeelb@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Wei Yang Subject: [Patch v2 1/2] mm/rmap.c: don't reuse anon_vma if we just want a copy Date: Thu, 10 Oct 2019 21:58:24 +0800 Message-Id: <20191010135825.28153-1-richardw.yang@linux.intel.com> X-Mailer: git-send-email 2.17.1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Before commit 7a3ef208e662 ("mm: prevent endless growth of anon_vma hierarchy"), anon_vma_clone() doesn't change dst->anon_vma. While after this commit, anon_vma_clone() will try to reuse an exist one on forking. But this commit go a little bit further for the case not forking. anon_vma_clone() is called from __vma_split(), __split_vma(), copy_vma() and anon_vma_fork(). For the first three places, the purpose here is get a copy of src and we don't expect to touch dst->anon_vma even it is NULL. While after that commit, it is possible to reuse an anon_vma when dst->anon_vma is NULL. This is not we intend to have. This patch stop reuse anon_vma for non-fork cases. Fix commit 7a3ef208e662 ("mm: prevent endless growth of anon_vma hierarchy") Signed-off-by: Wei Yang --- include/linux/rmap.h | 3 ++- mm/mmap.c | 6 +++--- mm/rmap.c | 7 ++++--- 3 files changed, 9 insertions(+), 7 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 988d176472df..963e6ab09b9b 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -142,7 +142,8 @@ static inline void anon_vma_unlock_read(struct anon_vma *anon_vma) void anon_vma_init(void); /* create anon_vma_cachep */ int __anon_vma_prepare(struct vm_area_struct *); void unlink_anon_vmas(struct vm_area_struct *); -int anon_vma_clone(struct vm_area_struct *, struct vm_area_struct *); +int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src, + bool reuse); int anon_vma_fork(struct vm_area_struct *, struct vm_area_struct *); static inline int anon_vma_prepare(struct vm_area_struct *vma) diff --git a/mm/mmap.c b/mm/mmap.c index 93f221785956..21e94f8ac4c7 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -791,7 +791,7 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, int error; importer->anon_vma = exporter->anon_vma; - error = anon_vma_clone(importer, exporter); + error = anon_vma_clone(importer, exporter, false); if (error) return error; } @@ -2666,7 +2666,7 @@ int __split_vma(struct mm_struct *mm, struct vm_area_struct *vma, if (err) goto out_free_vma; - err = anon_vma_clone(new, vma); + err = anon_vma_clone(new, vma, false); if (err) goto out_free_mpol; @@ -3247,7 +3247,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, new_vma->vm_pgoff = pgoff; if (vma_dup_policy(vma, new_vma)) goto out_free_vma; - if (anon_vma_clone(new_vma, vma)) + if (anon_vma_clone(new_vma, vma, false)) goto out_free_mempol; if (new_vma->vm_file) get_file(new_vma->vm_file); diff --git a/mm/rmap.c b/mm/rmap.c index d9a23bb773bf..f729e4013613 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -258,7 +258,8 @@ static inline void unlock_anon_vma_root(struct anon_vma *root) * good chance of avoiding scanning the whole hierarchy when it searches where * page is mapped. */ -int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src) +int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src, + bool reuse) { struct anon_vma_chain *avc, *pavc; struct anon_vma *root = NULL; @@ -286,7 +287,7 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src) * will always reuse it. Root anon_vma is never reused: * it has self-parent reference and at least one child. */ - if (!dst->anon_vma && anon_vma != src->anon_vma && + if (reuse && !dst->anon_vma && anon_vma != src->anon_vma && anon_vma->degree < 2) dst->anon_vma = anon_vma; } @@ -329,7 +330,7 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma) * First, attach the new VMA to the parent VMA's anon_vmas, * so rmap can find non-COWed pages in child processes. */ - error = anon_vma_clone(vma, pvma); + error = anon_vma_clone(vma, pvma, true); if (error) return error; From patchwork Thu Oct 10 13:58:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Yang X-Patchwork-Id: 11183653 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BC9081575 for ; Thu, 10 Oct 2019 13:59:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 93D92208C3 for ; Thu, 10 Oct 2019 13:59:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 93D92208C3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5EF216B0003; Thu, 10 Oct 2019 09:59:26 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 578718E0005; Thu, 10 Oct 2019 09:59:26 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4403B6B0006; Thu, 10 Oct 2019 09:59:26 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0163.hostedemail.com [216.40.44.163]) by kanga.kvack.org (Postfix) with ESMTP id 2598E6B0003 for ; Thu, 10 Oct 2019 09:59:26 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id C261A180AD806 for ; Thu, 10 Oct 2019 13:59:25 +0000 (UTC) X-FDA: 76028032290.05.bit23_8c714f9090b55 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,richardw.yang@linux.intel.com,:akpm@linux-foundation.org:kirill.shutemov@linux.intel.com:jglisse@redhat.com:mike.kravetz@oracle.com:riel@surriel.com:khlebnikov@yandex-team.ru:cai@lca.pw:shakeelb@google.com::linux-kernel@vger.kernel.org:richardw.yang@linux.intel.com,RULES_HIT:30051:30054:30070:30090,0,RBL:192.55.52.43:@linux.intel.com:.lbl8.mailshell.net-62.18.0.100 64.95.201.95,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:29,LUA_SUMMARY:none X-HE-Tag: bit23_8c714f9090b55 X-Filterd-Recvd-Size: 4163 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by imf18.hostedemail.com (Postfix) with ESMTP for ; Thu, 10 Oct 2019 13:59:24 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 10 Oct 2019 06:59:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.67,280,1566889200"; d="scan'208";a="194014801" Received: from richard.sh.intel.com (HELO localhost) ([10.239.159.54]) by fmsmga007.fm.intel.com with ESMTP; 10 Oct 2019 06:59:21 -0700 From: Wei Yang To: akpm@linux-foundation.org, kirill.shutemov@linux.intel.com, jglisse@redhat.com, mike.kravetz@oracle.com, riel@surriel.com, khlebnikov@yandex-team.ru, cai@lca.pw, shakeelb@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Wei Yang Subject: [Patch v2 2/2] mm/rmap.c: reuse mergeable anon_vma as parent when fork Date: Thu, 10 Oct 2019 21:58:25 +0800 Message-Id: <20191010135825.28153-2-richardw.yang@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20191010135825.28153-1-richardw.yang@linux.intel.com> References: <20191010135825.28153-1-richardw.yang@linux.intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In function __anon_vma_prepare(), we will try to find anon_vma if it is possible to reuse it. While on fork, the logic is different. Since commit 5beb49305251 ("mm: change anon_vma linking to fix multi-process server scalability issue"), function anon_vma_clone() tries to allocate new anon_vma for child process. But the logic here will allocate a new anon_vma for each vma, even in parent this vma is mergeable and share the same anon_vma with its sibling. This may do better for scalability issue, while it is not necessary to do so especially after interval tree is used. Commit 7a3ef208e662 ("mm: prevent endless growth of anon_vma hierarchy") tries to reuse some anon_vma by counting child anon_vma and attached vmas. While for those mergeable anon_vmas, we can just reuse it and not necessary to go through the logic. After this change, kernel build test reduces 20% anon_vma allocation. Do the same kernel build test, it shows run time in sys reduced 8.1%. Origin real 2m50.467s user 17m52.002s sys 1m51.953s real 2m48.662s user 17m55.464s sys 1m50.553s real 2m51.143s user 17m59.687s sys 1m53.600s Patched real 2m45.478s user 17m37.069s sys 1m42.671s real 2m46.420s user 17m45.970s sys 1m43.175s real 2m47.404s user 17m51.531s sys 1m43.005s Signed-off-by: Wei Yang --- mm/rmap.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/mm/rmap.c b/mm/rmap.c index f729e4013613..4314b9be70f6 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -250,7 +250,7 @@ static inline void unlock_anon_vma_root(struct anon_vma *root) * Attach the anon_vmas from src to dst. * Returns 0 on success, -ENOMEM on failure. * - * If dst->anon_vma is NULL this function tries to find and reuse existing + * If reuse is true, this function tries to find and reuse existing * anon_vma which has no vmas and only one child anon_vma. This prevents * degradation of anon_vma hierarchy to endless linear chain in case of * constantly forking task. On the other hand, an anon_vma with more than one @@ -263,6 +263,18 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src, { struct anon_vma_chain *avc, *pavc; struct anon_vma *root = NULL; + struct vm_area_struct *prev = dst->vm_prev, *pprev = src->vm_prev; + + /* + * If parent share anon_vma with its vm_prev, keep this sharing in in + * child. + * + * 1. Parent has vm_prev, which implies we have vm_prev. + * 2. Parent and its vm_prev have the same anon_vma. + */ + if (reuse && pprev && pprev->anon_vma == src->anon_vma) + dst->anon_vma = prev->anon_vma; + list_for_each_entry_reverse(pavc, &src->anon_vma_chain, same_vma) { struct anon_vma *anon_vma;