From patchwork Wed May 29 21:20:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 10967807 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 570CB1575 for ; Wed, 29 May 2019 21:36:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4657B288BB for ; Wed, 29 May 2019 21:36:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3A16C288D9; Wed, 29 May 2019 21:36:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 80149288BB for ; Wed, 29 May 2019 21:36:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CCE1F6B026A; Wed, 29 May 2019 17:36:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CA8436B026D; Wed, 29 May 2019 17:36:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD12F6B0274; Wed, 29 May 2019 17:36:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 5432F6B026D for ; Wed, 29 May 2019 17:36:54 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id r191so786966pgr.23 for ; Wed, 29 May 2019 14:36:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=e4fE3iJAPMCO/Qbuvxgy9RjCcmbzJ+JxIMLpqr9tekI=; b=B8eLg95d1zofvcCONdptKO0klnQMhVFY2WSsBZRxI9pLvpvGQkc8X4fWO5mYJofVBY vF9K1NTaCr49D4TBW76iIfDka25UyQgohk86DqMsw5Ro6RN0DHZIt1K5FAqwH0s5Fe8e 3lMfm1ZnAmbeuWqnxsVzwjy0aygkWpDiC4nIo3FO7UQFI8l+s+X1/Uc5nJfJgA+vIM1z FChovOGiXor3N9OzGEKiAfjtIAvH20fk7zXnvK5uxFT2EvaHj4rEkKOmBlGvOxZWsYI9 zFD0qYfEnyZ4+fIeU48R1LUQjpSjTxzEj/mFSdrPF8cmExCUS+z0+I3fCkH/R2Qh9Mr7 GP2w== X-Gm-Message-State: APjAAAU4+Z302ivVZ522qmM4V7nkNyCl0BS0bvHsbq5Tj/H8HQH0pBwO uvhr52emkn2Wyn9iJ/gIeV4uIh6lo3HA7h7+YBBgPBtyPRwxEC5Qx4/lyJRYUbpCwUGkqCBDqXu VUm0p4ciudjieWpMRnh8xpk5PLgVcVMN2B1umEi5SCnzg0SjrWMERMgL+wfYrNVF79Q== X-Received: by 2002:a63:ee0b:: with SMTP id e11mr93726pgi.453.1559165813900; Wed, 29 May 2019 14:36:53 -0700 (PDT) X-Google-Smtp-Source: APXvYqy5HJq6HynN9xSxkO/u3kYbTd8P0d0bAp4h8+ONUM/HPit4Y+QJNo/3khDmKQg+pqKnanDc X-Received: by 2002:a63:ee0b:: with SMTP id e11mr93675pgi.453.1559165812699; Wed, 29 May 2019 14:36:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559165812; cv=none; d=google.com; s=arc-20160816; b=pZe++amySsExI5Ik3E95J+EHvEdaN9krgMBMMYe+/vD/gxu286AP1ILjCRAbatPoj2 9NnxLejQMmgXpfRyn+09zRnW9jN4xqMeW23NzBu+PSJzQyxbLRB4Mvr4HUr15oQ9JELj 1vJKnh2NXC+P5v0n7QPA3nUxb87FHCFGJrDoMpFihRdeeAKgupAkoFQPWQzCHvEbDpfd mlS0dXghW0XqIGf1h2k2Z6a9XL/gNDobI1QfBdx1ntnAsuAo4gC4BE6zk0PTROKcc1uX Mgyup7g6lyjvH61j2918DdPPnS4SSzovv8hrwsfZzItOVT2/Ah8o2g3jFYo9COaET/3r DUBQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=e4fE3iJAPMCO/Qbuvxgy9RjCcmbzJ+JxIMLpqr9tekI=; b=bt9eVLEPwJW9QnL8C5Tww3MT3VRpVRdTcYrEYZqdvpb/remCqdd3y7vV5wLohLuGoC 2ew2KDwvgbCkqkp2ii+h4XrxscuHFyYjG273QY1t+bXQsNhGedD1ItEPeJK6TpYAoglk LAg05BnqvYg8yutOy1PF28CyLvwsSyHRAzemnoceewHhjiXuCSsGyOpZL86XVbdySTfS kpojUQ/Br6a+Fx2CRomLYh1gWdxaQaAZaDFBc0H8Zd3Rin58xPbwZFQItXJNqXVwSzNo PSS6jUrXFLgHCkxFsq7JLfCN28XybfrPPK6NNWJlO2Wd0VDxn7fL2Z+6dw6cnkoImf67 01Ew== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=h0HkURYE; spf=pass (google.com: domain of prvs=105246f206=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=105246f206=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id q17si981771pgl.27.2019.05.29.14.36.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 29 May 2019 14:36:52 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=105246f206=songliubraving@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=h0HkURYE; spf=pass (google.com: domain of prvs=105246f206=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=105246f206=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044008.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x4TLTHvl011980 for ; Wed, 29 May 2019 14:36:50 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=e4fE3iJAPMCO/Qbuvxgy9RjCcmbzJ+JxIMLpqr9tekI=; b=h0HkURYEjg9fLq+Wub9w8QuMyIbE+9Tj+Oqdm/UqK7ZWyiyevkDoiDhjnAiyxq3fmWjb CmnSzy8S7tTvI9S7+mGGz+SQD1UwS4UVUYfXqpEZo/bWpJfacN1KdQUipB0xsjLGZR8z 4BaEmDl5lLMvC7QiE8fzH4U6+/q3lzwXcAY= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2ssv0esfw1-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 29 May 2019 14:36:50 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::125) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Wed, 29 May 2019 14:36:49 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id C392762E1F88; Wed, 29 May 2019 14:21:11 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH uprobe, thp 1/4] mm, thp: allow preallocate pgtable for split_huge_pmd_address() Date: Wed, 29 May 2019 14:20:46 -0700 Message-ID: <20190529212049.2413886-2-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190529212049.2413886-1-songliubraving@fb.com> References: <20190529212049.2413886-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-05-29_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=971 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905290134 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently, __split_huge_pmd_locked() uses page fault to handle file backed THP. This is required because splitting pmd requires allocating a new pgtable. This patch allows the caller of __split_huge_pmd_locked() and split_huge_pmd_address() to preallocate the pgtable, so that refault is not required. This is useful when the caller of split_huge_pmd_address() would like to use small pages before refault. Signed-off-by: Song Liu --- include/linux/huge_mm.h | 5 +++-- mm/huge_memory.c | 33 +++++++++++++++++++++++---------- mm/rmap.c | 2 +- 3 files changed, 27 insertions(+), 13 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 7cd5c150c21d..2d8a40fd06e4 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -161,7 +161,7 @@ void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long address, - bool freeze, struct page *page); + bool freeze, struct page *page, pgtable_t prealloc_pgtable); void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, unsigned long address); @@ -299,7 +299,8 @@ static inline void deferred_split_huge_page(struct page *page) {} static inline void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, unsigned long address, bool freeze, struct page *page) {} static inline void split_huge_pmd_address(struct vm_area_struct *vma, - unsigned long address, bool freeze, struct page *page) {} + unsigned long address, bool freeze, struct page *page, + pgtable_t prealloc_pgtable) {} #define split_huge_pud(__vma, __pmd, __address) \ do { } while (0) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9f8bce9a6b32..dcb0e30213af 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2118,7 +2118,7 @@ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma, } static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, - unsigned long haddr, bool freeze) + unsigned long haddr, bool freeze, pgtable_t prealloc_pgtable) { struct mm_struct *mm = vma->vm_mm; struct page *page; @@ -2133,10 +2133,15 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, VM_BUG_ON_VMA(vma->vm_end < haddr + HPAGE_PMD_SIZE, vma); VM_BUG_ON(!is_pmd_migration_entry(*pmd) && !pmd_trans_huge(*pmd) && !pmd_devmap(*pmd)); + /* only file backed vma need preallocate pgtable*/ + VM_BUG_ON(vma_is_anonymous(vma) && prealloc_pgtable); count_vm_event(THP_SPLIT_PMD); - if (!vma_is_anonymous(vma)) { + if (prealloc_pgtable) { + pgtable_trans_huge_deposit(mm, pmd, prealloc_pgtable); + mm_inc_nr_pmds(mm); + } else if (!vma_is_anonymous(vma)) { _pmd = pmdp_huge_clear_flush_notify(vma, haddr, pmd); /* * We are going to unmap this huge page. So @@ -2277,8 +2282,9 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, } } -void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, - unsigned long address, bool freeze, struct page *page) +static void ____split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, + unsigned long address, bool freeze, struct page *page, + pgtable_t prealloc_pgtable) { spinlock_t *ptl; struct mmu_notifier_range range; @@ -2303,7 +2309,8 @@ void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, clear_page_mlock(page); } else if (!(pmd_devmap(*pmd) || is_pmd_migration_entry(*pmd))) goto out; - __split_huge_pmd_locked(vma, pmd, range.start, freeze); + __split_huge_pmd_locked(vma, pmd, range.start, freeze, + prealloc_pgtable); out: spin_unlock(ptl); /* @@ -2322,8 +2329,14 @@ void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, mmu_notifier_invalidate_range_only_end(&range); } +void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, + unsigned long address, bool freeze, struct page *page) +{ + ____split_huge_pmd(vma, pmd, address, freeze, page, NULL); +} + void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long address, - bool freeze, struct page *page) + bool freeze, struct page *page, pgtable_t prealloc_pgtable) { pgd_t *pgd; p4d_t *p4d; @@ -2344,7 +2357,7 @@ void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long address, pmd = pmd_offset(pud, address); - __split_huge_pmd(vma, pmd, address, freeze, page); + ____split_huge_pmd(vma, pmd, address, freeze, page, prealloc_pgtable); } void vma_adjust_trans_huge(struct vm_area_struct *vma, @@ -2360,7 +2373,7 @@ void vma_adjust_trans_huge(struct vm_area_struct *vma, if (start & ~HPAGE_PMD_MASK && (start & HPAGE_PMD_MASK) >= vma->vm_start && (start & HPAGE_PMD_MASK) + HPAGE_PMD_SIZE <= vma->vm_end) - split_huge_pmd_address(vma, start, false, NULL); + split_huge_pmd_address(vma, start, false, NULL, NULL); /* * If the new end address isn't hpage aligned and it could @@ -2370,7 +2383,7 @@ void vma_adjust_trans_huge(struct vm_area_struct *vma, if (end & ~HPAGE_PMD_MASK && (end & HPAGE_PMD_MASK) >= vma->vm_start && (end & HPAGE_PMD_MASK) + HPAGE_PMD_SIZE <= vma->vm_end) - split_huge_pmd_address(vma, end, false, NULL); + split_huge_pmd_address(vma, end, false, NULL, NULL); /* * If we're also updating the vma->vm_next->vm_start, if the new @@ -2384,7 +2397,7 @@ void vma_adjust_trans_huge(struct vm_area_struct *vma, if (nstart & ~HPAGE_PMD_MASK && (nstart & HPAGE_PMD_MASK) >= next->vm_start && (nstart & HPAGE_PMD_MASK) + HPAGE_PMD_SIZE <= next->vm_end) - split_huge_pmd_address(next, nstart, false, NULL); + split_huge_pmd_address(next, nstart, false, NULL, NULL); } } diff --git a/mm/rmap.c b/mm/rmap.c index e5dfe2ae6b0d..6970d732507c 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1361,7 +1361,7 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, if (flags & TTU_SPLIT_HUGE_PMD) { split_huge_pmd_address(vma, address, - flags & TTU_SPLIT_FREEZE, page); + flags & TTU_SPLIT_FREEZE, page, NULL); } /* From patchwork Wed May 29 21:20:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 10967811 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AFC5113AD for ; Wed, 29 May 2019 21:37:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A0E18288BB for ; Wed, 29 May 2019 21:37:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 928C7288D9; Wed, 29 May 2019 21:37:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0EA0F288BB for ; Wed, 29 May 2019 21:37:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 282246B026E; Wed, 29 May 2019 17:36:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 10DEE6B0272; Wed, 29 May 2019 17:36:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E0DF16B0274; Wed, 29 May 2019 17:36:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 84B806B026E for ; Wed, 29 May 2019 17:36:54 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id i8so2854993pfo.21 for ; Wed, 29 May 2019 14:36:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=vH2HloBkxs0E6y5mQ4CjNY7e4RAkgwr31RBvR6O3DQc=; b=cTasGexP+GlezuLmbCDCqlsCpBcQs3YUqdgdJTIJggvgzTWayYlSiIwTNtsR+BnxjF 5uDdiiPhedY7x2cwKnnAXPMs/23aU/tCaKVoce9WpZGeQVm9cTBGrYWqr+Q0fujUUYjS s/+q0TBVYTk2bW5bmN1O0i6TbKM7pduXXaXhl7T12sbsd4XPwKnCLAV5la0btlpV4GKh s0O953c85yB+j+bU7b7D05vo05h+pS0zet0wpCzaJZWuDjJTkuKHEbCiGLFK0CQXzTfA cqXAbwR0npTQLekeOIVqj+aBZkgZM3cpfkPtR3zMZpree9Xeq1V5bZuKSTij9H6ksOxM OJuA== X-Gm-Message-State: APjAAAWg7o/VPReIv5XtErHLeqXoQFsSa7ox364toXhK/SI7VV7MHuty b2cWZ4tJ7U5ETOpHJTkI5SFtZJhgND2xfc9JWqbR94nlqF1snhvfNi+Vfp7pnaXG4Z6nJ+rMhmz OmS1z4p4bur+ki2fTQyQv01R43IFTJMHAQJbIDrXtYpvHnw2//Km4O92AicW0bMn+4A== X-Received: by 2002:a17:902:ab98:: with SMTP id f24mr25035plr.223.1559165814041; Wed, 29 May 2019 14:36:54 -0700 (PDT) X-Google-Smtp-Source: APXvYqxuuHkRRTDwkP0+suyjTsMzBKFH7LISuUJ9CnWZjw89Tuo7YiJgHrtnrYSTDlDLKx7pHe3e X-Received: by 2002:a17:902:ab98:: with SMTP id f24mr24987plr.223.1559165813033; Wed, 29 May 2019 14:36:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559165813; cv=none; d=google.com; s=arc-20160816; b=Do6y24gSrtET62eMRSeYq6nsL8/5TyY9z75FgHCFR41f8Ijsg6n43IQA8JH1s2RGnl K5eGgh0L5n+I5wg4obsyi7QnzrY75QhrG2JY3Wsr/InWXID5W2P8faKi8n/M4Axh6Ajf M1zrZx1HFwRZDnwQnR13oJ/FgFwiHEIZQbBAPqCi8RYtYRbWMDaHOWOK+l8IBKQdSaB7 Hk8XF/bREzJEIXXWky27LQeeFcgB+rnRi/hPodxVielLUSbLeCPflwrf0GCu2jHf/fD9 41MzVI1OLrJlst3zhvgvPtFAybItoRc1hgL0jDNpzSYGzAWfn/VF/QCAm5F5whUqlQRo La4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=vH2HloBkxs0E6y5mQ4CjNY7e4RAkgwr31RBvR6O3DQc=; b=QGJg8USfC/x4CvkxRcnv3Vu17mr7LDxRGhNj8QbpCZ1r/FB2K+pqzeYQHpZUnSOdGz HRCtz5KbtmENfrY1GRVWEm47Pv7RIIE4VHMxVDZw1G+HjImgL07fX0PP9om69sLYFwu/ NPFe/t3Ku6OgVG2XtFcDEdbFOe7DupLqKUvQQABYKcz7YuO/kRkDrpQ9IR+H0qcr6Iax qhDeB8J3MMCnY4cyimcCLiZfq+I2I9wa8Qv1nsEDROatPrCExE74clNPw43tJerT8fuH t6us2H7awG0BBLi11qfd0DpWSOB8JYC3jpavaC3eC1RzPxpIIlKq3mOgZeltgO5By86Q iSAw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b="leQ/opXV"; spf=pass (google.com: domain of prvs=105246f206=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=105246f206=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id p7si903989pgh.497.2019.05.29.14.36.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 29 May 2019 14:36:53 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=105246f206=songliubraving@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b="leQ/opXV"; spf=pass (google.com: domain of prvs=105246f206=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=105246f206=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x4TLXFaE029780 for ; Wed, 29 May 2019 14:36:52 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=vH2HloBkxs0E6y5mQ4CjNY7e4RAkgwr31RBvR6O3DQc=; b=leQ/opXVYpqVUDrxSdj4Ktb74UjnKrLbSU4Q57tkH+Tr8ZlEQE8x7tcWa2DqRRneIB6v W5K/0bzvmab0GSW6VbzFxHczPg7EevoFPI2JeUZc7zgyigmd+qtUkTo1UYgWgJxtsI20 Kdm2ZOn+1azQzr3ZPEXJLQNhUgLy5Q/vLFo= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2ssqq9jb6r-12 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 29 May 2019 14:36:52 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::129) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Wed, 29 May 2019 14:36:48 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id E135662E2076; Wed, 29 May 2019 14:21:16 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH uprobe, thp 2/4] uprobe: use original page when all uprobes are removed Date: Wed, 29 May 2019 14:20:47 -0700 Message-ID: <20190529212049.2413886-3-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190529212049.2413886-1-songliubraving@fb.com> References: <20190529212049.2413886-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-05-29_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905290134 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently, uprobe swaps the target page with a anonymous page in both install_breakpoint() and remove_breakpoint(). When all uprobes on a page are removed, the given mm is still using an anonymous page (not the original page). This patch allows uprobe to use original page when possible (all uprobes on the page are already removed). Signed-off-by: Song Liu --- kernel/events/uprobes.c | 43 ++++++++++++++++++++++++++++++++--------- 1 file changed, 34 insertions(+), 9 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 78f61bfc6b79..ba49da99d2a2 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -160,16 +160,19 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, int err; struct mmu_notifier_range range; struct mem_cgroup *memcg; + bool orig = new_page->mapping != NULL; /* new_page == orig_page */ mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, addr, addr + PAGE_SIZE); VM_BUG_ON_PAGE(PageTransHuge(old_page), old_page); - err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, &memcg, - false); - if (err) - return err; + if (!orig) { + err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, + &memcg, false); + if (err) + return err; + } /* For try_to_free_swap() and munlock_vma_page() below */ lock_page(old_page); @@ -177,15 +180,22 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, mmu_notifier_invalidate_range_start(&range); err = -EAGAIN; if (!page_vma_mapped_walk(&pvmw)) { - mem_cgroup_cancel_charge(new_page, memcg, false); + if (!orig) + mem_cgroup_cancel_charge(new_page, memcg, false); goto unlock; } VM_BUG_ON_PAGE(addr != pvmw.address, old_page); get_page(new_page); - page_add_new_anon_rmap(new_page, vma, addr, false); - mem_cgroup_commit_charge(new_page, memcg, false, false); - lru_cache_add_active_or_unevictable(new_page, vma); + if (orig) { + page_add_file_rmap(new_page, false); + inc_mm_counter(mm, mm_counter_file(new_page)); + dec_mm_counter(mm, MM_ANONPAGES); + } else { + page_add_new_anon_rmap(new_page, vma, addr, false); + mem_cgroup_commit_charge(new_page, memcg, false, false); + lru_cache_add_active_or_unevictable(new_page, vma); + } if (!PageAnon(old_page)) { dec_mm_counter(mm, mm_counter_file(old_page)); @@ -461,9 +471,10 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, unsigned long vaddr, uprobe_opcode_t opcode) { struct uprobe *uprobe; - struct page *old_page, *new_page; + struct page *old_page, *new_page, *orig_page = NULL; struct vm_area_struct *vma; int ret, is_register, ref_ctr_updated = 0; + pgoff_t index; is_register = is_swbp_insn(&opcode); uprobe = container_of(auprobe, struct uprobe, arch); @@ -501,6 +512,20 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, copy_highpage(new_page, old_page); copy_to_page(new_page, vaddr, &opcode, UPROBE_SWBP_INSN_SIZE); + index = vaddr_to_offset(vma, vaddr & PAGE_MASK) >> PAGE_SHIFT; + orig_page = find_get_page(vma->vm_file->f_inode->i_mapping, index); + if (orig_page) { + if (memcmp(page_address(orig_page), + page_address(new_page), PAGE_SIZE) == 0) { + /* if new_page matches orig_page, use orig_page */ + put_page(new_page); + new_page = orig_page; + } else { + put_page(orig_page); + orig_page = NULL; + } + } + ret = __replace_page(vma, vaddr, old_page, new_page); put_page(new_page); put_old: From patchwork Wed May 29 21:20:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 10967809 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0BB9F13AD for ; Wed, 29 May 2019 21:37:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F069D288BB for ; Wed, 29 May 2019 21:37:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E2BD8288D9; Wed, 29 May 2019 21:37:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 37F95288BB for ; Wed, 29 May 2019 21:37:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A2BB6B026D; Wed, 29 May 2019 17:36:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EEF596B0273; Wed, 29 May 2019 17:36:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C3CC36B0272; Wed, 29 May 2019 17:36:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 7A0DE6B026A for ; Wed, 29 May 2019 17:36:54 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id f1so2902184pfb.0 for ; Wed, 29 May 2019 14:36:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=qndMC0xqMluPTNov9/3iwaOnMII97myVrvnzznwxlUw=; b=ZANSFr9ZbjsnlcmC+cLdjhGQLth06hxWNScIqfDa9v2lMqqBGfHIR+eeI8vd0clXYl ysmkRUPngBcPt7HOVzrOp+1+tuhJYyGhKD2SQwAJdWgWCNKp54OzLZh8HPhidCNV4WTd cXHMUNLyaD6VnBb4mliM1MCGQSBgVN1EU9JGg18Bd1/zwll04tdm1Z823b/Wn+DrdMUh QDBBr5OEsVKUHBMndfB3UKw6Og/xO5fZhp2PcA84NK8s3GePmtj01yuUy1ya7KPyixmy JqnQaAeo/gsCKfLA9bMCR4qveJf9kH9qIswCNaLPYfgPEi/mAs59kU3vn9WebZo1djwU r3Jw== X-Gm-Message-State: APjAAAX7ki4TFsspf03otl3/i5c9Qx/eOqZtggmdZNtMJIE8Gs5FE1bf Ee3ZTP+7hBBzyM01/cFQNhN2S8Yu5oTPN6Hj4xkuBircTHsDlCRi4W240EUijdVGgiSd2eLoSrk aj9eyuDpni9b83zJrYpkhAsY5zHv4vXduwmu5DAI80d/1RidQAZF6RcuzrBfjLOA3AQ== X-Received: by 2002:a17:90a:9dca:: with SMTP id x10mr103429pjv.105.1559165813947; Wed, 29 May 2019 14:36:53 -0700 (PDT) X-Google-Smtp-Source: APXvYqxV1I+Ej+jSpQKU/0gctSjl2ZFb0xmeOjFhUP6i+XOPKoyRuMJfMkja4K38CnvTQ7rTDuxM X-Received: by 2002:a17:90a:9dca:: with SMTP id x10mr103373pjv.105.1559165812697; Wed, 29 May 2019 14:36:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559165812; cv=none; d=google.com; s=arc-20160816; b=OwvHQLSV/IeJRJmUopLOUVLnzbC7ZGtIaJUwXUlXSktWHn9D4hxqEJYYU9Wz58t3LI yaz3hSA152h6NFku3DLYLCQsVNHqVcyTp0nLyisB/gmJWi+YVIt2+wxCB+zrDFCKsrwE /1ZJmadhZ6FCBQ+YgBy6opfrVm87wmk42W85t5X0uW1O3WfagNC5K3JYPd1fAC6S3+7J 15flbD3OoMCWV6LY8XVKOZwNhezF7amH292x8Y1J79sZ8TCy+xkhhduypYOGc+7tHaqW 6vhdyxuSyKjMXfy4e4Ca7Fzs0L5Y/NUahOWShGXiPCmEVBqyzaEDU+3OK+qnAGznHJzU pATA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=qndMC0xqMluPTNov9/3iwaOnMII97myVrvnzznwxlUw=; b=i7hJidAAvBb56i9dzfSxJMxBkufujDSR8K5r89MzsSnMCYgE0bYR2oNlbEeGBaws4h 30EXDK99s60vh5KmXf/rNwzaPgkuJg1aaTto1+N1BZRUjiOLvx8cAH98sLG0cGcIZc2V V34ZIu41yDaHZPFlNNdv6HjF9fnAygvRJq4dpzF8n0n+jiOfA5S4BstcMGmQK6pBfzB/ 6qXU9jjYim6AKJAZGEjFnEh6dJjxJaWh/S3sbN+hDYUNKp7FVY4+EHa93GvzE0XgT7zf a+nsJgOVi0qGd5BsWla4E+F1TZgR5SwcrOUb1eJqoEoxI6Gp+K+3itAk75q6AR/flquL jgQA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=LTkQ1wQE; spf=pass (google.com: domain of prvs=105246f206=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=105246f206=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id x21si691542pjn.61.2019.05.29.14.36.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 29 May 2019 14:36:52 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=105246f206=songliubraving@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=LTkQ1wQE; spf=pass (google.com: domain of prvs=105246f206=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=105246f206=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x4TLXFaA029780 for ; Wed, 29 May 2019 14:36:51 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=qndMC0xqMluPTNov9/3iwaOnMII97myVrvnzznwxlUw=; b=LTkQ1wQEoMYl6maX4Qm4eWZZSKF3T2qxbtzfn/2yYogdb+4V3LFaHgo8wiAq6NGxBkJd aTe758DWrooYpZU6e6V+eiqY1EGkBCpvv9Z69AZbPJDMWWk8QIZwKrbAS3IHE9TcYJkb ZWVuxyitfP5YHkU3ABSScb9jAn2gSEVFV0E= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2ssqq9jb6r-8 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 29 May 2019 14:36:50 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::129) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Wed, 29 May 2019 14:36:48 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 47ACB62E215C; Wed, 29 May 2019 14:21:22 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH uprobe, thp 3/4] uprobe: support huge page by only splitting the pmd Date: Wed, 29 May 2019 14:20:48 -0700 Message-ID: <20190529212049.2413886-4-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190529212049.2413886-1-songliubraving@fb.com> References: <20190529212049.2413886-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-05-29_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905290134 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Instead of splitting the compound page with FOLL_SPLIT, this patch allows uprobe to only split pmd for huge pages. A helper function mm_address_trans_huge(mm, address) was introduced to test whether the address in mm is pointing to THP. Signed-off-by: Song Liu --- include/linux/huge_mm.h | 8 ++++++++ kernel/events/uprobes.c | 38 ++++++++++++++++++++++++++++++++------ mm/huge_memory.c | 24 ++++++++++++++++++++++++ 3 files changed, 64 insertions(+), 6 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 2d8a40fd06e4..4832d6580969 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -163,6 +163,8 @@ void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long address, bool freeze, struct page *page, pgtable_t prealloc_pgtable); +bool mm_address_trans_huge(struct mm_struct *mm, unsigned long address); + void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, unsigned long address); @@ -302,6 +304,12 @@ static inline void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long address, bool freeze, struct page *page, pgtable_t prealloc_pgtable) {} +static inline bool mm_address_trans_huge(struct mm_struct *mm, + unsigned long address) +{ + return false; +} + #define split_huge_pud(__vma, __pmd, __address) \ do { } while (0) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index ba49da99d2a2..56eeccc2f7a2 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -26,6 +26,7 @@ #include #include #include +#include #include @@ -153,7 +154,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, { struct mm_struct *mm = vma->vm_mm; struct page_vma_mapped_walk pvmw = { - .page = old_page, + .page = compound_head(old_page), .vma = vma, .address = addr, }; @@ -165,8 +166,6 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, addr, addr + PAGE_SIZE); - VM_BUG_ON_PAGE(PageTransHuge(old_page), old_page); - if (!orig) { err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, &memcg, false); @@ -188,7 +187,8 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, get_page(new_page); if (orig) { - page_add_file_rmap(new_page, false); + page_add_file_rmap(compound_head(new_page), + PageTransHuge(compound_head(new_page))); inc_mm_counter(mm, mm_counter_file(new_page)); dec_mm_counter(mm, MM_ANONPAGES); } else { @@ -207,7 +207,8 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, set_pte_at_notify(mm, addr, pvmw.pte, mk_pte(new_page, vma->vm_page_prot)); - page_remove_rmap(old_page, false); + page_remove_rmap(compound_head(old_page), + PageTransHuge(compound_head(old_page))); if (!page_mapped(old_page)) try_to_free_swap(old_page); page_vma_mapped_walk_done(&pvmw); @@ -475,17 +476,42 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, struct vm_area_struct *vma; int ret, is_register, ref_ctr_updated = 0; pgoff_t index; + pgtable_t prealloc_pgtable = NULL; + unsigned long foll_flags = FOLL_FORCE; is_register = is_swbp_insn(&opcode); uprobe = container_of(auprobe, struct uprobe, arch); + /* do not FOLL_SPLIT yet */ + ret = get_user_pages_remote(NULL, mm, vaddr, 1, + foll_flags, &old_page, &vma, NULL); + + if (ret <= 0) + return ret; + + if (mm_address_trans_huge(mm, vaddr)) { + prealloc_pgtable = pte_alloc_one(mm); + if (likely(prealloc_pgtable)) { + split_huge_pmd_address(vma, vaddr, false, NULL, + prealloc_pgtable); + goto verify; + } else { + /* fallback to FOLL_SPLIT */ + foll_flags |= FOLL_SPLIT; + put_page(old_page); + } + } else { + goto verify; + } + retry: /* Read the page with vaddr into memory */ ret = get_user_pages_remote(NULL, mm, vaddr, 1, - FOLL_FORCE | FOLL_SPLIT, &old_page, &vma, NULL); + foll_flags, &old_page, &vma, NULL); if (ret <= 0) return ret; +verify: ret = verify_opcode(old_page, vaddr, &opcode); if (ret <= 0) goto put_old; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index dcb0e30213af..4714871353c0 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2360,6 +2360,30 @@ void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long address, ____split_huge_pmd(vma, pmd, address, freeze, page, prealloc_pgtable); } +bool mm_address_trans_huge(struct mm_struct *mm, unsigned long address) +{ + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + + pgd = pgd_offset(mm, address); + if (!pgd_present(*pgd)) + return false; + + p4d = p4d_offset(pgd, address); + if (!p4d_present(*p4d)) + return false; + + pud = pud_offset(p4d, address); + if (!pud_present(*pud)) + return false; + + pmd = pmd_offset(pud, address); + + return pmd_trans_huge(*pmd); +} + void vma_adjust_trans_huge(struct vm_area_struct *vma, unsigned long start, unsigned long end, From patchwork Wed May 29 21:20:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 10967805 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B5EDC1575 for ; Wed, 29 May 2019 21:36:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9BFC1288BB for ; Wed, 29 May 2019 21:36:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8E07F288D9; Wed, 29 May 2019 21:36:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D59CB288BB for ; Wed, 29 May 2019 21:36:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8FD446B026F; Wed, 29 May 2019 17:36:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8AD8F6B0273; Wed, 29 May 2019 17:36:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7003C6B026F; Wed, 29 May 2019 17:36:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 365A86B026A for ; Wed, 29 May 2019 17:36:54 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id 93so2423674plf.14 for ; Wed, 29 May 2019 14:36:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=UArkez1wLqfjMeDv/pROrH5C6HuDZSMjR0N7lqYER04=; b=ZxlnCkfuwXWcJ0+TCk5SUjSkXGgBQsYrCx3rVP6BsLL+FYAeEMCEem+gM25/rLnQ27 OGJTPM8a0KqefIGk7p7z6RENRdyneXJthoFK8C7Xo+Kxcra+82cCs55nlYbgFmII3iXW hfvaKvyotPzY4xEKAyVnITD81yXhaPAfCfT92D7bzRFFkf3i2/cG7h45BRQ1UG4YWcxX 3htljo04J88/2SRRIzuOVjctrJ3lKsGSrMiEfWZ5PEwFxcD6Bt22pfI/DFI91PChRjBt 8YXwLsujoTcg0ysjrnWv0Jl6CnMcp7fOUi0SjygnpqpM0niDYD0x61X4+RZ7JQTKvxkr t3Ww== X-Gm-Message-State: APjAAAWRxl6TcER+d9WBelydwx+5ONsHXLpfaIuY8nkXHXCTAVb1cmGH EBsnPMtszqpKj6HORJAQSbNgSdZ96MPH0eOiPCCLxF+VLE90HrV1pkrea7Z9/WHXRERaw06ZehH yg2uVb1ATns0qqgvh/nFRXuDiF/f18eCrK/cmqfdlPzaVaBaWWqlfKTR3Vk/znFFzsQ== X-Received: by 2002:aa7:87c3:: with SMTP id i3mr150551940pfo.85.1559165813731; Wed, 29 May 2019 14:36:53 -0700 (PDT) X-Google-Smtp-Source: APXvYqwsQPJq6uYZ0V7AHC+cFez2ImjFAQUWvfgnTY9QG9C9BWejKMGqpOhMC4TpAJN6HhFk3ZeQ X-Received: by 2002:aa7:87c3:: with SMTP id i3mr150551882pfo.85.1559165812698; Wed, 29 May 2019 14:36:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559165812; cv=none; d=google.com; s=arc-20160816; b=yC7KyQa0HqhA2FdO81Uw3RnEpZMeinfdxwQlReSlOQ3CsimLpeAFqd43G7V+PF6xxq gb6GzXTdAMTrZ3biL9hhVhLWxl6/TIz8VO03mRFcRf1Svzz+yADg8MBtKeYF7gNwPi9l rzEGeYzqtO5nVc0NB6tQVhcHtxrNSfH0TtaQcn8sSl00mke8yuSn6XgDXtZqcFK+YNGW X1jHEI+o6rCXG1KaS4SKQtRwIo4XZOOHUgL0Pw1lV1clV9BML1N4jvGY7FrHAjhe1I5M 4u2Kh3VGFFetboRoJx4pNDq2RPbrbYRL5i7uq6jvvaaja8G+4zvesCjNsZj2sz6xcuzF hOvw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=UArkez1wLqfjMeDv/pROrH5C6HuDZSMjR0N7lqYER04=; b=q7MJuqNMYpjvcnte8b/N5n8Eszikb3o0xCaakc8h7N+jE0aY/t+PkuQMY0hbM+lZQ3 bxFJPA9r0FLotlzzhwpaR2D8FvxBYHnt5hLwHQq7El3M5KVPTZTW5gR2w7jrTJbhKeYt vmRlTS79bVtGAj3yQFFECCE6EWJkMVVkDSvnjTNNNtDXESx9TeLHQbKHDd6WpLOvbuAB eSsCAXV4aaYZM3WgCOkemJ95Hg3lOISJNdDaWc6JAOV5S9ciy85pjzSr3rRvR4c0M7xv U0mvl6HLxPDI7fef4hS0CqVHCk8VXrqUmf7M18PtfcdD8pCi+iHSUowsp+s/Qq1OO6kD cJlQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=i1ckGPPp; spf=pass (google.com: domain of prvs=105246f206=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=105246f206=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id m13si908313pgj.577.2019.05.29.14.36.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 29 May 2019 14:36:52 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=105246f206=songliubraving@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=i1ckGPPp; spf=pass (google.com: domain of prvs=105246f206=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=105246f206=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x4TLXFa9029780 for ; Wed, 29 May 2019 14:36:50 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=UArkez1wLqfjMeDv/pROrH5C6HuDZSMjR0N7lqYER04=; b=i1ckGPPp7A/6a0YxZ05oFISrNStf/pYAXhCeWxn8FmdL5JNi1JGPfKAllMordz6uf/DV EktRl4V9y59BoXDexVHPEMovn5SLL9w5p/zqSFlL24v3+A8lE8YzOXGA5rOBY/hjs40q JW4PcnlMn5jMrSCpFZquBhAUs/CD5Y+C1b4= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2ssqq9jb6r-7 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 29 May 2019 14:36:50 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::129) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Wed, 29 May 2019 14:36:48 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 7984C62E2174; Wed, 29 May 2019 14:21:27 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH uprobe, thp 4/4] uprobe: collapse THP pmd after removing all uprobes Date: Wed, 29 May 2019 14:20:49 -0700 Message-ID: <20190529212049.2413886-5-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190529212049.2413886-1-songliubraving@fb.com> References: <20190529212049.2413886-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-05-29_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1011 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=725 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905290134 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP After all uprobes are removed from the huge page (with PTE pgtable), it is possible to collapse the pmd and benefit from THP again. This patch does the collapse. An issue on earlier version was discovered by kbuild test robot. Reported-by: kbuild test robot Signed-off-by: Song Liu --- include/linux/huge_mm.h | 9 ++++++++ kernel/events/uprobes.c | 3 +++ mm/huge_memory.c | 47 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 59 insertions(+) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 4832d6580969..61f6d574d9b4 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -252,6 +252,10 @@ static inline bool thp_migration_supported(void) return IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION); } +extern inline void try_collapse_huge_pmd(struct mm_struct *mm, + struct vm_area_struct *vma, + unsigned long vaddr); + #else /* CONFIG_TRANSPARENT_HUGEPAGE */ #define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; }) #define HPAGE_PMD_MASK ({ BUILD_BUG(); 0; }) @@ -377,6 +381,11 @@ static inline bool thp_migration_supported(void) { return false; } + +static inline void try_collapse_huge_pmd(struct mm_struct *mm, + struct vm_area_struct *vma, + unsigned long vaddr) {} + #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif /* _LINUX_HUGE_MM_H */ diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 56eeccc2f7a2..422617bdd5ff 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -564,6 +564,9 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, if (ret && is_register && ref_ctr_updated) update_ref_ctr(uprobe, mm, -1); + if (!ret && orig_page && PageTransCompound(orig_page)) + try_collapse_huge_pmd(mm, vma, vaddr); + return ret; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 4714871353c0..e2edec3ffd43 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2923,6 +2923,53 @@ static struct shrinker deferred_split_shrinker = { .flags = SHRINKER_NUMA_AWARE, }; +/** + * This function only checks whether all PTEs in this PMD point to + * continuous pages, the caller should make sure at least of these PTEs + * points to a huge page, e.g. PageTransCompound(one_page) != 0. + */ +void try_collapse_huge_pmd(struct mm_struct *mm, + struct vm_area_struct *vma, + unsigned long vaddr) +{ + struct mmu_notifier_range range; + unsigned long addr; + pmd_t *pmd, _pmd; + spinlock_t *ptl; + long long head; + int i; + + pmd = mm_find_pmd(mm, vaddr); + if (!pmd) + return; + + addr = vaddr & HPAGE_PMD_MASK; + head = pte_val(*pte_offset_map(pmd, addr)); + ptl = pmd_lock(mm, pmd); + for (i = 0; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { + pte_t *pte = pte_offset_map(pmd, addr); + + if (pte_val(*pte) != head + i * PAGE_SIZE) { + spin_unlock(ptl); + return; + } + } + + addr = vaddr & HPAGE_PMD_MASK; + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, NULL, mm, + addr, addr + HPAGE_PMD_SIZE); + mmu_notifier_invalidate_range_start(&range); + + _pmd = pmdp_collapse_flush(vma, addr, pmd); + spin_unlock(ptl); + mmu_notifier_invalidate_range_end(&range); + mm_dec_nr_ptes(mm); + pte_free(mm, pmd_pgtable(_pmd)); + add_mm_counter(mm, + shmem_file(vma->vm_file) ? MM_SHMEMPAGES : MM_FILEPAGES, + -HPAGE_PMD_NR); +} + #ifdef CONFIG_DEBUG_FS static int split_huge_pages_set(void *data, u64 val) {