From patchwork Sat Jun 22 00:01:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 11010965 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E104B76 for ; Sat, 22 Jun 2019 00:01:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D084228BB1 for ; Sat, 22 Jun 2019 00:01:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C1F8C28BB4; Sat, 22 Jun 2019 00:01:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4A73928BB1 for ; Sat, 22 Jun 2019 00:01:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 512248E0003; Fri, 21 Jun 2019 20:01:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4C3748E0001; Fri, 21 Jun 2019 20:01:23 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 363A28E0003; Fri, 21 Jun 2019 20:01:23 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id F2ECE8E0001 for ; Fri, 21 Jun 2019 20:01:22 -0400 (EDT) Received: by mail-pl1-f200.google.com with SMTP id s22so4481939plp.5 for ; Fri, 21 Jun 2019 17:01:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=pQnGGve/Vnh2HyxOCOdI1CzGbYYB1SvJlXif4X9k+UU=; b=ZF2iUOsSBpCocdF4ex+m/5Ueve+O5Y3bfgdMBrl0Ys230bxQgFuUT/cNpGt6P8XRGZ lSr+vESyBRWBMFKl9kqTWo+/naykY4AJFrOjhqPAgQNSUocVQlN+qdGGWSTiRedgR+oh 9gmODP7WjyWBmI5xkwnWmE2Y3NLbi0kv02WLSzDO12/d4gK7HpPycvdvL4hTBJVOBTaC 3tDvdrIfVkqsXfMVkhnUWcT3B8Ce8ka0ifA3qNBjEkWjdqRvBrQsYkpedJC4KoyZl1U2 bK9UfOXlYjCQtHfUC8uG2aM60vv+EoeNm0xEXSMiiw6ok6eP+4awzUvSgL0sEEX19t+f y/yg== X-Gm-Message-State: APjAAAUvMHb9Zon8BKm72F5LFL3iZlHFJ0ICiA14Eo9qVThqePOba6To iyGqH+GL39SHEXNWPhs3FqIxFj8hKm58+OpgRh9pnp7chDwJrWdPAhhoFOGOkZhDblqapqrAyW4 djj18OvrLtRMSjcNAqRZV8NmLZ48kGtzhIWu/Ysp/fsuNmLT4tpYq8WskiEQKDySsmw== X-Received: by 2002:a63:6c4a:: with SMTP id h71mr20731378pgc.331.1561161682544; Fri, 21 Jun 2019 17:01:22 -0700 (PDT) X-Google-Smtp-Source: APXvYqzoSUGHq+xTN1eU+mPBIkOnPezNWkQ85HR2gyH9d0JdIyax08levNiLDngbNDBIvM8feuxL X-Received: by 2002:a63:6c4a:: with SMTP id h71mr20731321pgc.331.1561161681836; Fri, 21 Jun 2019 17:01:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561161681; cv=none; d=google.com; s=arc-20160816; b=CiZszBAkhVJOkWoMYMrAY1dZAKYyxMR1z2GfwXirf97JT/wqK0ZfhAw0vaDryQP+Sc H4zcpR5bmZj0l3gQ71EUJf6ELBj9hQ47ODkKX9yEHFynSFjoB6OznOoyfcayPQsfYGw2 zWqLDGDpdV7e5bE7ns8fkFivFZCdIJwAwvV1KGbmgVS/2Bj3lb2uhiXX5WO1hKLp/wci 8GzooJtd//s7pPmjGuWKV+H70ecZuxi7Nep7GUv+HRdjD5TATnCBVApQjLX9kCSY3oeU kpftLs2oBkYUz3C1sMco6OdZGEG0xX96nQ7VLMYCGuc+xDKIPr2kIspmhujZ8u+5yelF sl1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=pQnGGve/Vnh2HyxOCOdI1CzGbYYB1SvJlXif4X9k+UU=; b=00YTvQxPwfgk+ikZgsX0NeAN166Sq9XkDh6Z4pVsAjh3SjiL7oK9M3er6qxhblAJKu qVH27BAYFGrUHI7hbj6iY2Y74PAjWR86WToWyIVtZZPAKxtMSwJBkg51Rr3ytn23dDDh Xd/nPtyD/pnxx//8cELcvRum7jBSHg6p8bD1NPh/Kw/jxWuqsNBtVs2Sveqz+iPy+Pim 3usuD82AwEdOTN7hb3iuVTVub6go25YA7vOchxXiEldLpXtGqBH5/78+Td58JVghaHni 4/pLw4X2TDnNvSgg5lIvV5ZSOqx6mbh7Wnw/R3rLhsRt8TkOY2Zb8DmWYaZ5HeNX56VB yftA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=L6sdhAib; spf=pass (google.com: domain of prvs=1076a8f7d5=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=1076a8f7d5=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id i4si3955746pfa.218.2019.06.21.17.01.21 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Jun 2019 17:01:21 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=1076a8f7d5=songliubraving@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=L6sdhAib; spf=pass (google.com: domain of prvs=1076a8f7d5=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=1076a8f7d5=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5LNt0FM005691 for ; Fri, 21 Jun 2019 17:01:21 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=pQnGGve/Vnh2HyxOCOdI1CzGbYYB1SvJlXif4X9k+UU=; b=L6sdhAibl2kDYaNERgXyACpf60GF1GnZcsIOni0a2TSENSqJxq8vjZD+EXtc/OZjVfNW kRzPQ4H7FywUWdxJDaSdsX5SkfuKWwOy8xWd3AvnW/0oAKi5ObXe4/unjMF1LHIqu1GD vuF5rZ4br5mqBhWvsyeuVWKrt5nKKagT5tk= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2t90mjj0ya-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 21 Jun 2019 17:01:21 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Fri, 21 Jun 2019 17:01:18 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 173AC62E2D56; Fri, 21 Jun 2019 17:01:18 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v5 1/5] mm: move memcmp_pages() and pages_identical() Date: Fri, 21 Jun 2019 17:01:05 -0700 Message-ID: <20190622000109.914695-2-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190622000109.914695-1-songliubraving@fb.com> References: <20190622000109.914695-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-21_16:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=948 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906210182 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch moves memcmp_pages() to mm/util.c and pages_identical() to mm.h, so that we can use them in other files. Signed-off-by: Song Liu --- include/linux/mm.h | 7 +++++++ mm/ksm.c | 18 ------------------ mm/util.c | 13 +++++++++++++ 3 files changed, 20 insertions(+), 18 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index dd0b5f4e1e45..0ab8c7d84cd0 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2891,5 +2891,12 @@ void __init setup_nr_node_ids(void); static inline void setup_nr_node_ids(void) {} #endif +extern int memcmp_pages(struct page *page1, struct page *page2); + +static inline int pages_identical(struct page *page1, struct page *page2) +{ + return !memcmp_pages(page1, page2); +} + #endif /* __KERNEL__ */ #endif /* _LINUX_MM_H */ diff --git a/mm/ksm.c b/mm/ksm.c index 81c20ed57bf6..6f153f976c4c 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1030,24 +1030,6 @@ static u32 calc_checksum(struct page *page) return checksum; } -static int memcmp_pages(struct page *page1, struct page *page2) -{ - char *addr1, *addr2; - int ret; - - addr1 = kmap_atomic(page1); - addr2 = kmap_atomic(page2); - ret = memcmp(addr1, addr2, PAGE_SIZE); - kunmap_atomic(addr2); - kunmap_atomic(addr1); - return ret; -} - -static inline int pages_identical(struct page *page1, struct page *page2) -{ - return !memcmp_pages(page1, page2); -} - static int write_protect_page(struct vm_area_struct *vma, struct page *page, pte_t *orig_pte) { diff --git a/mm/util.c b/mm/util.c index 9834c4ab7d8e..750e586d50bc 100644 --- a/mm/util.c +++ b/mm/util.c @@ -755,3 +755,16 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen) out: return res; } + +int memcmp_pages(struct page *page1, struct page *page2) +{ + char *addr1, *addr2; + int ret; + + addr1 = kmap_atomic(page1); + addr2 = kmap_atomic(page2); + ret = memcmp(addr1, addr2, PAGE_SIZE); + kunmap_atomic(addr2); + kunmap_atomic(addr1); + return ret; +} From patchwork Sat Jun 22 00:01:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 11010973 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1E71F924 for ; Sat, 22 Jun 2019 00:01:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 10A2428BB1 for ; Sat, 22 Jun 2019 00:01:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0465028BB4; Sat, 22 Jun 2019 00:01:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7BE7928BB1 for ; Sat, 22 Jun 2019 00:01:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 698DC8E0008; Fri, 21 Jun 2019 20:01:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 649778E0001; Fri, 21 Jun 2019 20:01:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 539BE8E0008; Fri, 21 Jun 2019 20:01:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 1D8B38E0001 for ; Fri, 21 Jun 2019 20:01:43 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id s195so4985155pgs.13 for ; Fri, 21 Jun 2019 17:01:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=qKr0k6k4xZHZmykxLb47XrFO0p8RI9jBEYkCbTeGgSI=; b=sfj3uhY4FAgfB7NxkAoXwtNea8pZ1dxoL9euDEx5grDsfjYvjTGxuJukhQCuPuiVH0 /zWMPHNP6pvyC5oZDxF2iHKI5LqXatXdkJ7ZSkCeZxIgr6pJEi2pH8iD4f2sapQ1YKbu sPvjSMGgdFPDqxy2Oec41qGjCuVhLWsxAadZP65AmXfbsfXQfLIjL2i8NHQHcZCte1iV AoUCKChTYQ7yZpeh3BGpI2D8dqULcoCgFZ0Flj0s0PyoVyjLvc/PPj92/iV5tF4mRdaV 6Kn53VLls2axgYrEO5QLEf2rybPMd02V8EXZ7XWWOKTNZp3S3twHbk59lIigLPke4Nrs SP1A== X-Gm-Message-State: APjAAAU8MFc1MvCX4xmzqsdPPA8MOzGLoxoyOtwfWjA3MOwFqsHfX5Rs mEyo4acIq936XV99LHNoFtAU7/35oFlUhNYeeFtA1Tgtgxw2r4Ae4oDpmB674cr2p6bFRHD0uKl xzNNFytgZ66Av5gtjq9kNoV4fl/Qd7DH/sPKh6eukCNkny95DqYOEtwhYzcU9BVFCHQ== X-Received: by 2002:a17:90a:a00d:: with SMTP id q13mr9802592pjp.80.1561161702730; Fri, 21 Jun 2019 17:01:42 -0700 (PDT) X-Google-Smtp-Source: APXvYqxDuyGb2qE8vxWyIfRUfa2Bs6G/aEUj/o0D+Hf/lSMxUUdodrqW+NC8sRMYHVQyz2zvf0S7 X-Received: by 2002:a17:90a:a00d:: with SMTP id q13mr9802539pjp.80.1561161702094; Fri, 21 Jun 2019 17:01:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561161702; cv=none; d=google.com; s=arc-20160816; b=n3tncKq78wCLTMM4gRlyFsypRO2aK+SQG4Q1IyuEOrn4YqQER9uduNod0To00/DgjN z4gQfxO7flOAU11SqXI2nxmG+41AqfomliKzgn1OoB13QT3B1gXXOm/9EnOY6hoi+hLR eIlGSp37fV7RNG34zEGs0RHmc+tMFFfQEXSvV6j9DEbfxEBe9oShk79WId+jrc3G7T2m TRLJYVmfy7GquA8rhv2LNors9xXLk7G6mhyO9oFc2v7SdKOld+Ky4uJkPkW0MDMEUtkz uCX52JdseRWENcPom2ydluefYnGvSgHC7P8B3sve9AaroyISRhpFI+cmNVGTGmRrC+yh VE6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=qKr0k6k4xZHZmykxLb47XrFO0p8RI9jBEYkCbTeGgSI=; b=FK9oD/P5QrVnZm7voxN6NI/VS1sB/6mOOuYimOns/XWIgJZ8ElbLfgCU3Xbc46U9tr kgqNBNOtCx6wRk67JpVNO92b7XooHpNnco/raJ0JsjE9kJj/j68EAt1LejsjvGVOYika Aoq77/ePV0bIsm/0N8G2BIue5/NuB79mMn5KHVs5vS8n8ERl9/4UbraBpMztkjZL1u4m f/oYyrmsSFnNAKVnp+c4jDkfqHKG44G6AMY9qPJCHTNBQYp/dJSapAM9Q09lakDT5fFB 2wr9AfmQ1seZuXP7sH1dkNQddE3mfJmjGcOGOfICEU0BF3oAgPJPXTY7j8nBry4IYay6 CLIg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=EpfTC14a; spf=pass (google.com: domain of prvs=1076a8f7d5=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=1076a8f7d5=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id h32si4180894pld.402.2019.06.21.17.01.41 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Jun 2019 17:01:42 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=1076a8f7d5=songliubraving@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=EpfTC14a; spf=pass (google.com: domain of prvs=1076a8f7d5=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=1076a8f7d5=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5LNr924018757 for ; Fri, 21 Jun 2019 17:01:41 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=qKr0k6k4xZHZmykxLb47XrFO0p8RI9jBEYkCbTeGgSI=; b=EpfTC14aDequsN24ugMVjz7cSVSmNk/9t1KWhsJJOj4AW/9APHs+9tK0sHr1h0KeLCRc hLhyS9GSaaJ+LXrNaoDykrEtFxZ2mhqh7WVPiilxuY2wsF3UJwVtlJLWhfochv5C9bof UtZsoJbO05yxTdznP+nAq7KAONnGflnalAo= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2t90est208-9 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Fri, 21 Jun 2019 17:01:41 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::130) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Fri, 21 Jun 2019 17:01:22 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id CAA2862E2D56; Fri, 21 Jun 2019 17:01:20 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v5 2/5] uprobe: use original page when all uprobes are removed Date: Fri, 21 Jun 2019 17:01:06 -0700 Message-ID: <20190622000109.914695-3-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190622000109.914695-1-songliubraving@fb.com> References: <20190622000109.914695-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-21_16:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=996 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906210182 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently, uprobe swaps the target page with a anonymous page in both install_breakpoint() and remove_breakpoint(). When all uprobes on a page are removed, the given mm is still using an anonymous page (not the original page). This patch allows uprobe to use original page when possible (all uprobes on the page are already removed). Signed-off-by: Song Liu --- kernel/events/uprobes.c | 45 +++++++++++++++++++++++++++++++++-------- 1 file changed, 37 insertions(+), 8 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 78f61bfc6b79..f7c61a1ef720 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -160,16 +160,19 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, int err; struct mmu_notifier_range range; struct mem_cgroup *memcg; + bool orig = new_page->mapping != NULL; /* new_page == orig_page */ mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, addr, addr + PAGE_SIZE); VM_BUG_ON_PAGE(PageTransHuge(old_page), old_page); - err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, &memcg, - false); - if (err) - return err; + if (!orig) { + err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, + &memcg, false); + if (err) + return err; + } /* For try_to_free_swap() and munlock_vma_page() below */ lock_page(old_page); @@ -177,15 +180,24 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, mmu_notifier_invalidate_range_start(&range); err = -EAGAIN; if (!page_vma_mapped_walk(&pvmw)) { - mem_cgroup_cancel_charge(new_page, memcg, false); + if (!orig) + mem_cgroup_cancel_charge(new_page, memcg, false); goto unlock; } VM_BUG_ON_PAGE(addr != pvmw.address, old_page); get_page(new_page); - page_add_new_anon_rmap(new_page, vma, addr, false); - mem_cgroup_commit_charge(new_page, memcg, false, false); - lru_cache_add_active_or_unevictable(new_page, vma); + if (orig) { + lock_page(new_page); /* for page_add_file_rmap() */ + page_add_file_rmap(new_page, false); + unlock_page(new_page); + inc_mm_counter(mm, mm_counter_file(new_page)); + dec_mm_counter(mm, MM_ANONPAGES); + } else { + page_add_new_anon_rmap(new_page, vma, addr, false); + mem_cgroup_commit_charge(new_page, memcg, false, false); + lru_cache_add_active_or_unevictable(new_page, vma); + } if (!PageAnon(old_page)) { dec_mm_counter(mm, mm_counter_file(old_page)); @@ -501,6 +513,23 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, copy_highpage(new_page, old_page); copy_to_page(new_page, vaddr, &opcode, UPROBE_SWBP_INSN_SIZE); + if (!is_register) { + struct page *orig_page; + pgoff_t index; + + index = vaddr_to_offset(vma, vaddr & PAGE_MASK) >> PAGE_SHIFT; + orig_page = find_get_page(vma->vm_file->f_inode->i_mapping, + index); + + if (orig_page) { + if (pages_identical(new_page, orig_page)) { + put_page(new_page); + new_page = orig_page; + } else + put_page(orig_page); + } + } + ret = __replace_page(vma, vaddr, old_page, new_page); put_page(new_page); put_old: From patchwork Sat Jun 22 00:01:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 11010967 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DF16776 for ; Sat, 22 Jun 2019 00:01:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D1AF428BB1 for ; Sat, 22 Jun 2019 00:01:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C50B628BB6; Sat, 22 Jun 2019 00:01:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4AB0928BB1 for ; Sat, 22 Jun 2019 00:01:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E88358E0005; Fri, 21 Jun 2019 20:01:26 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E39898E0001; Fri, 21 Jun 2019 20:01:26 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D01AA8E0005; Fri, 21 Jun 2019 20:01:26 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f198.google.com (mail-yb1-f198.google.com [209.85.219.198]) by kanga.kvack.org (Postfix) with ESMTP id ABD2B8E0001 for ; Fri, 21 Jun 2019 20:01:26 -0400 (EDT) Received: by mail-yb1-f198.google.com with SMTP id e7so7363266ybk.22 for ; Fri, 21 Jun 2019 17:01:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=VaTYZS3FdWj202Z1cIYQJggrJMEpd/wr+92gO5GUCcA=; b=mWZev+096aNOGQTEMwUCU9F2CmtCLB1M3vXQO1EcvSApnf/hzr7BB834XoVgUyA/fb iWvv/PC2u3OG6P88EFbykbp3DZBr6jhQo6t6hzwn6k9+3c/bhgMtPwwkL+nIJbIO2+XE /1+d3Fe3gdqMIjQkh726j+rhM0MkkYngVWp51xSYY907qlN35m+O4BFpRyelC6/D6o1a z81DA34smyiO9pFvqzwvp4BWUhxLl6YzhkBvxa4F50e2Hqt1DpkwaMTrRLkrktbfc/wo A7UDt7qvghnAKGeDpVgdKoQ6PW9E6TSSd7bWD6sJ5e9NTZBmVU5SVtIIQHRKwd1WYL7j bBCA== X-Gm-Message-State: APjAAAXQR2A7Ld6ToZefPeWd1QrAwf4vZYe77xRkw/du9AKROK+8OSnQ HhLRxIDB6UHN/TzzNgC/lBROkoHaEUf18gfqaf25MXh0OZeuOQoNKrHTe0/AOvObo7McHRIuamM Vm4YZNSATgW42tBbtjOMpaBG+0Ov1uHqKHyvASz3KnNzl63WbHu2K+Z6L5h2H+wcwuw== X-Received: by 2002:a05:6902:52f:: with SMTP id y15mr20024122ybs.328.1561161686432; Fri, 21 Jun 2019 17:01:26 -0700 (PDT) X-Google-Smtp-Source: APXvYqzX2BDwizvYAuuR7s6wvMmZXh7BSLv0Xgylomaq53opidgLQKMH/eL75mR7+6N929U8LlGH X-Received: by 2002:a05:6902:52f:: with SMTP id y15mr20024084ybs.328.1561161685780; Fri, 21 Jun 2019 17:01:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561161685; cv=none; d=google.com; s=arc-20160816; b=WWieuQpe4XYiwHyD5CE+Q86AMXyrUgg1XtVCln4HfaxqnfHs5rwLZuFDQ3upS30GTG u3FZgnNhKz3+mg2/4ubrPbRhkLDUA2Cmr9u1joscW0djchhHsOq6lpjeuVmEiiwsZ+Kf 5362dlYpLlyQyIay+y6rNfyFMq4QSDWIQ3/JBBXo8ZqSGLfDUwW/gtFaTshd345HmvSS l17vvhTtogmlLQAWrDEA15PhLC3LphnDlp0byEeKLNhPuwrgvnmaxfWKtjyrm6aQ8dG8 5CnVaWwKo/sS6nFW+iBtHNgu1FTO0sPNejzlDnlKVjOFAyZcMJ7LVqYf2olwJOHb3Pnn T+CQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=VaTYZS3FdWj202Z1cIYQJggrJMEpd/wr+92gO5GUCcA=; b=NqW5SEBDTCfxV3SOm/luJ8F0wzKY8H6lt0+HUf5Zd6b1hiNH0+aORL1ev64aIughL1 4ieTC+/vvYrCJY+1AFPUUxQN17TBavuRXqK8F9Z+gemJAMbpOFqieAk31qrAwDUnnOMd IBMaB+w1z/ggUDer6GzbjpV1Alqp8uV+kFrts2MK+pjXMHVeMfP8lVXjnIluQynTXbyT Oj6xWfCv7DemR+Kbo/gyuif9Q2nXB5d0QWuabbYrCd4LnbZhyaVe8JLtTefB5sK+nzZw mlERb666iEVQJQdmWxGyMkss2mVy9PXYitQCG6r2AaBWnaLmdttDo/rFlxf5Sjhclpho YBcQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=QXr0LedM; spf=pass (google.com: domain of prvs=1076a8f7d5=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1076a8f7d5=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id i192si1425646ybg.26.2019.06.21.17.01.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Jun 2019 17:01:25 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=1076a8f7d5=songliubraving@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=QXr0LedM; spf=pass (google.com: domain of prvs=1076a8f7d5=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1076a8f7d5=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5LNsAEk006695 for ; Fri, 21 Jun 2019 17:01:25 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=VaTYZS3FdWj202Z1cIYQJggrJMEpd/wr+92gO5GUCcA=; b=QXr0LedM/Vc4aA9QWlq8TFaaf6K7yB1JiO8LZC1T4cAWNojl+OGoSbtmgounz/i1/Fhx A/rshCVAOA43bxGsJrPKEfp+lec9VQdtWAoeP5fJuBJKKxVIB9WDIX6WXXRABbIQZuh6 j6nQZFOEExiNyRXTCyYdBVX9kWDN7ZgnPXg= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2t94ucrwj6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 21 Jun 2019 17:01:25 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Fri, 21 Jun 2019 17:01:24 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 6FAAD62E2D56; Fri, 21 Jun 2019 17:01:23 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v5 3/5] mm, thp: introduce FOLL_SPLIT_PMD Date: Fri, 21 Jun 2019 17:01:07 -0700 Message-ID: <20190622000109.914695-4-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190622000109.914695-1-songliubraving@fb.com> References: <20190622000109.914695-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-21_16:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=979 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906210182 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patches introduces a new foll_flag: FOLL_SPLIT_PMD. As the name says FOLL_SPLIT_PMD splits huge pmd for given mm_struct, the underlining huge page stays as-is. FOLL_SPLIT_PMD is useful for cases where we need to use regular pages, but would switch back to huge page and huge pmd on. One of such example is uprobe. The following patches use FOLL_SPLIT_PMD in uprobe. Signed-off-by: Song Liu --- include/linux/mm.h | 1 + mm/gup.c | 8 ++++++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0ab8c7d84cd0..e605acc4fc81 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2642,6 +2642,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address, #define FOLL_COW 0x4000 /* internal GUP flag */ #define FOLL_ANON 0x8000 /* don't do file mappings */ #define FOLL_LONGTERM 0x10000 /* mapping lifetime is indefinite: see below */ +#define FOLL_SPLIT_PMD 0x20000 /* split huge pmd before returning */ /* * NOTE on FOLL_LONGTERM: diff --git a/mm/gup.c b/mm/gup.c index ddde097cf9e4..41f2a1fcc6f0 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -398,7 +398,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, spin_unlock(ptl); return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } - if (flags & FOLL_SPLIT) { + if (flags & (FOLL_SPLIT | FOLL_SPLIT_PMD)) { int ret; page = pmd_page(*pmd); if (is_huge_zero_page(page)) { @@ -407,7 +407,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, split_huge_pmd(vma, pmd, address); if (pmd_trans_unstable(pmd)) ret = -EBUSY; - } else { + } else if (flags & FOLL_SPLIT) { if (unlikely(!try_get_page(page))) { spin_unlock(ptl); return ERR_PTR(-ENOMEM); @@ -419,6 +419,10 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, put_page(page); if (pmd_none(*pmd)) return no_page_table(vma, flags); + } else { /* flags & FOLL_SPLIT_PMD */ + spin_unlock(ptl); + split_huge_pmd(vma, pmd, address); + ret = pte_alloc(mm, pmd); } return ret ? ERR_PTR(ret) : From patchwork Sat Jun 22 00:01:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 11010971 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AD8B8924 for ; Sat, 22 Jun 2019 00:01:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9F76428BB1 for ; Sat, 22 Jun 2019 00:01:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 92FDF28BB4; Sat, 22 Jun 2019 00:01:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 12D2128BB1 for ; Sat, 22 Jun 2019 00:01:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3B28A8E0007; Fri, 21 Jun 2019 20:01:34 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 365C48E0001; Fri, 21 Jun 2019 20:01:34 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1DD048E0007; Fri, 21 Jun 2019 20:01:34 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id DEF028E0001 for ; Fri, 21 Jun 2019 20:01:33 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id d3so4989143pgc.9 for ; Fri, 21 Jun 2019 17:01:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=IZysbkBOWuoojpHfDDSz6omhgc5cHCXvaawoTaDfhK0=; b=sJABK1zpDg7Ft2uXH4vAnIUzHhjbvzrUfLcW5nvbJCGPrSef3Hx+S+MAx8fTe6mdaG 2rKGhzV4LzPjYYm7I3FZ14e/oRcUO4NcFx4tm2QWlsSsysZPFX1O1U50oan9qnrB8scH VFcL8QkY78TS5Sdg6fzfOwsb9gbwQhO1KseM5lu6vX1ivKeLfBTGsnYvIejp0Aw5/1tA TY5FawOA84RmUCh2anAWcWLXbkl0R76/M82XWjj4mQjxk2DW7f4EtcHPNzBU/XV9ksLL IM++VRMoKtNDU1fihL1HkduzbhdvlT7x5RQruEP4g7QpV1QgS+BSS6wgrF/eQIRO6SLi Jq5Q== X-Gm-Message-State: APjAAAXI9QPHcSLJ4+S9AKAnZF75PQSde44Lg0P3G8edU8whEDXvNDDq /Kv7SkCTAVztcHyGv3xkNDh7QGsVz8NCD2gIEn4qUohAAOPXsWkqr2OJLOgIkLebrqv1Ymyabj/ fZ7iN3X6XBQo4pb8iKe9L3YOX7TSj6UnKyGD8liwCKXLfJicyRR2zglE++FNW//EXVw== X-Received: by 2002:a17:902:70c3:: with SMTP id l3mr14010604plt.248.1561161693586; Fri, 21 Jun 2019 17:01:33 -0700 (PDT) X-Google-Smtp-Source: APXvYqyQ6m82j9bWX/vHqZxCG5VBOc/zXTT3WGcW4XotgiYtgY/bn1JJVOWOyeXBoTzuQv+lmPhS X-Received: by 2002:a17:902:70c3:: with SMTP id l3mr14010548plt.248.1561161693033; Fri, 21 Jun 2019 17:01:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561161693; cv=none; d=google.com; s=arc-20160816; b=RobfB206U0FEmk2JDij29b/rkKeaFSVCncQ3o9ahrnFHfEe4d8uNMqLi8GfF8S04c6 0YoOmXjGvwcEX8BvNFZCy3OGaQfvFA57dp1gZs7V8a6UJMP6nPcQgSz4ZD/MBAN2u3Xp Loz9Mu37uVWwsX963WVnwnbbgVGz+TjIUnIaUfLv66Rd1YTcpAzl5PhG91WjkaQVVjAQ CBDsJG6dJLQ0WpKyu7Ugr90jXTa3Ti3sC+CGXQRWGf1aCNQFu3ELOsqkZzQZLGYGNEed LWvKAUs8XzFlBcdSFgNJUOIjfP0B/0imjiHXoZHE22ek4WcojMmEHGs+CM9X3wfe1fGj jMpQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=IZysbkBOWuoojpHfDDSz6omhgc5cHCXvaawoTaDfhK0=; b=U7378p/jxE8mPZJ3HdfAPaUng/xB3JDX1aCG7d705jH6CC+70rqZxYU1o5yYgcP6YL b7lZTM4PKTjJx4NQ2RzjfeyuyVOGeYGr+TihNLfVjruYdSj0uFjH8jBajTrT52QqMX/s AkLkp0zEj7sYmT9LeV5PjgIXJlm002/o+I8XcShC3ZVYPlylUg4aylVsyoDtgAoqQ9eO IUhFeKP+0kJcN1UwO1eXOCKJyw3GimzXXn4TmhUqZWScEgsteE47rPsutUcKq3ahNVRh AKKM6MlFLHEROIL/tGeG5Lv3y2ka8c//kIjqJq6PIDtYibPJEDh+H13gXnvaicbJMcdE Q6Sw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=lRKaScX8; spf=pass (google.com: domain of prvs=1076a8f7d5=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=1076a8f7d5=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id m63si4024752pld.385.2019.06.21.17.01.32 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Jun 2019 17:01:33 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=1076a8f7d5=songliubraving@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=lRKaScX8; spf=pass (google.com: domain of prvs=1076a8f7d5=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=1076a8f7d5=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5LNseqQ018820 for ; Fri, 21 Jun 2019 17:01:32 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=IZysbkBOWuoojpHfDDSz6omhgc5cHCXvaawoTaDfhK0=; b=lRKaScX8kRdJ9jxl9lJwhSZxaoYO8XtSXrxKFfReMH3oL3f6tFB7wD3OirK8h+dC2qdU sGJovoq9+SNK3fdYYFXO+UV7qXhcL81lNR/RiJzqO61dy+J6TD18GKmTJHh3GP45quCE 0A8AtF92OBqf4NtUDbLsHc1dV5/zvzKhbKE= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2t91rg1q9d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Fri, 21 Jun 2019 17:01:32 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::127) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Fri, 21 Jun 2019 17:01:31 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id A354762E2D56; Fri, 21 Jun 2019 17:01:25 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v5 4/5] uprobe: use FOLL_SPLIT_PMD instead of FOLL_SPLIT Date: Fri, 21 Jun 2019 17:01:08 -0700 Message-ID: <20190622000109.914695-5-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190622000109.914695-1-songliubraving@fb.com> References: <20190622000109.914695-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-21_16:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=728 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906210182 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patches uses newly added FOLL_SPLIT_PMD in uprobe. This enables easy regroup of huge pmd after the uprobe is disabled (in next patch). Signed-off-by: Song Liu --- kernel/events/uprobes.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index f7c61a1ef720..a20d7b43a056 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -153,7 +153,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, { struct mm_struct *mm = vma->vm_mm; struct page_vma_mapped_walk pvmw = { - .page = old_page, + .page = compound_head(old_page), .vma = vma, .address = addr, }; @@ -165,8 +165,6 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, addr, addr + PAGE_SIZE); - VM_BUG_ON_PAGE(PageTransHuge(old_page), old_page); - if (!orig) { err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, &memcg, false); @@ -483,7 +481,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, retry: /* Read the page with vaddr into memory */ ret = get_user_pages_remote(NULL, mm, vaddr, 1, - FOLL_FORCE | FOLL_SPLIT, &old_page, &vma, NULL); + FOLL_FORCE | FOLL_SPLIT_PMD, &old_page, &vma, NULL); if (ret <= 0) return ret; From patchwork Sat Jun 22 00:01:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 11010969 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D947B76 for ; Sat, 22 Jun 2019 00:01:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CA6A628BB1 for ; Sat, 22 Jun 2019 00:01:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BE00F28BB4; Sat, 22 Jun 2019 00:01:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2A30628BB1 for ; Sat, 22 Jun 2019 00:01:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DED0D8E0006; Fri, 21 Jun 2019 20:01:31 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D9D1E8E0001; Fri, 21 Jun 2019 20:01:31 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C64F48E0006; Fri, 21 Jun 2019 20:01:31 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f69.google.com (mail-yw1-f69.google.com [209.85.161.69]) by kanga.kvack.org (Postfix) with ESMTP id A7E808E0001 for ; Fri, 21 Jun 2019 20:01:31 -0400 (EDT) Received: by mail-yw1-f69.google.com with SMTP id p68so8113870ywp.2 for ; Fri, 21 Jun 2019 17:01:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=yfkmPBW/NLQwl/dH7qhTlDffK3PtcU3kE6J70hPIVKM=; b=NYB1F7f9mtRYu1OVPimKMqYP9Vwa9MbG0WelFgVUM9v3MhX4Rf291+rDEUN3fq9qw2 GPuNkQrzT1R6XsXKzobGrkI6nUwGTvIYY3L+KyuI3gkl8RYreKegCQVs8Djoed7Aj5aC b8Ucb8NRSo3jzxIhI6KdTz0N81+MNsoqjGJ0kER2ic1f+FGppXP2e3jWu8HEVvT+XQan wcctKKM+ZLZx/2q8kEta40LJwNiFXvqxtUwoQJfEn14y2AVblc5gKva3aPzWLGehie4M 3ZqAjvM8SA8GsGJl6JmY+2Syt26Lw7NRyZMWtxbgG+LwztX/1/AZqgDbYsGGhycQF4qa kmNA== X-Gm-Message-State: APjAAAU+nD1+42n1KU8T7L3NFpHTlpVWTWL4iqP0WJssOwxtN1qEfVKv ex9yewekixtIdajkTgrGIzNVgKV8ID0SJ6Tf5n3FCEoemZ79ZTOmXKgLKfiWHNRfLV9qJsHPXqR VJ353oxyZAT3xCUjdhjoaywqhw9mlQq36hgF59bYfs5QC3L488MRum2scQnhnz8+EIQ== X-Received: by 2002:a81:48c:: with SMTP id 134mr42887725ywe.387.1561161691444; Fri, 21 Jun 2019 17:01:31 -0700 (PDT) X-Google-Smtp-Source: APXvYqxQx3HM6EMS/cgWsGXqu89UdD/GTc8sOchdylmAfstbybee4kGKidD+I1UULOuvpfII0LfM X-Received: by 2002:a81:48c:: with SMTP id 134mr42887680ywe.387.1561161690781; Fri, 21 Jun 2019 17:01:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561161690; cv=none; d=google.com; s=arc-20160816; b=Bsn5oRpaqFm3Dj0iNE0k2i6Ha0cPfccWpjlNMb2iu3QdjoMBFzByRdd3DiY/9ChXX1 eSH3ucJa7VUgFkxRgXqmGeT0Pcs6NX/avdTeiYxzya7c2ySvCSOi/+AO670MxDgzgXaY BPC4Ir4S4c1Szl/4Q1qSXmJvF4AkdLZUhWzi7A+fAX8m5VIiSQzG2ZXEwALt0oD0VFJD 9+xdf8r9jXePAfDRwSkEZA9XRldbZrT1fRW6zleGCf+ATvJZNxA6GJZ/f0iXBw/z4jCx 2xGOosBQrDK1DfQFFG2xRzEQZ0ylxjCJXO9+f9AzTOC5v5QsqNmbILkW329zhmBgmzeX 14wg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=yfkmPBW/NLQwl/dH7qhTlDffK3PtcU3kE6J70hPIVKM=; b=IIs2UhgRUMHKnibJ5IsuszjhDTNXzJfXKDHm+o9SxUkzZVCvw18yCcJisvzeQQtE35 PwPGZRVzg94OY3q4A2HFARWyGbKce9sLEKGJAEMR33wXBTNBsJ565W51MNbRJdC+FuD4 byNOQVENhKRiXuV5AL+sN7j6xLj9S5bQpmh9CS+ek/fb8iRUXdOZYWsBgE0mogRxXqoo c+TcXLXXLc+ienf17y0LFJzYhUnhne4H+N67Q0S/7ytu8TtX+TaXSCB0wCJOzg5QymaT H6YV9JVeiVrk2ShF1XHru4CFqsPBK3JBaRA+wRnGUF01b9pTG6HYX8OuFRz3eiw57RgI pQIg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=pbhEZGJc; spf=pass (google.com: domain of prvs=1076a8f7d5=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1076a8f7d5=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id x188si1515106ywe.453.2019.06.21.17.01.30 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Jun 2019 17:01:30 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=1076a8f7d5=songliubraving@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=pbhEZGJc; spf=pass (google.com: domain of prvs=1076a8f7d5=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1076a8f7d5=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5LNqbfH020377 for ; Fri, 21 Jun 2019 17:01:30 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=yfkmPBW/NLQwl/dH7qhTlDffK3PtcU3kE6J70hPIVKM=; b=pbhEZGJcMv8lBkUoI4bsmZ/nWCPwPp7qiQSGQUHSpGiO0sbZK6HW+bQRcz8PaOwIbuma c2V2g9C5Ga2iRcxq0pZD9YUdgxhpEqNbVUU49XIzGAeJyYvp9E3uAOuwXTYe/aT40yqX OnHQnie6uRslri3Y1apHpD0kPMrSwD//OFc= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2t8uemtyw9-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 21 Jun 2019 17:01:30 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::c) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Fri, 21 Jun 2019 17:01:29 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 18E7862E2D56; Fri, 21 Jun 2019 17:01:28 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v5 5/5] uprobe: collapse THP pmd after removing all uprobes Date: Fri, 21 Jun 2019 17:01:09 -0700 Message-ID: <20190622000109.914695-6-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190622000109.914695-1-songliubraving@fb.com> References: <20190622000109.914695-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-21_16:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=790 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906210182 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP After all uprobes are removed from the huge page (with PTE pgtable), it is possible to collapse the pmd and benefit from THP again. This patch does the collapse. An issue on earlier version was discovered by kbuild test robot. Reported-by: kbuild test robot Signed-off-by: Song Liu --- include/linux/huge_mm.h | 7 +++++ kernel/events/uprobes.c | 5 ++- mm/huge_memory.c | 69 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 80 insertions(+), 1 deletion(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 7cd5c150c21d..30669e9a9340 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -250,6 +250,9 @@ static inline bool thp_migration_supported(void) return IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION); } +extern void try_collapse_huge_pmd(struct vm_area_struct *vma, + struct page *page); + #else /* CONFIG_TRANSPARENT_HUGEPAGE */ #define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; }) #define HPAGE_PMD_MASK ({ BUILD_BUG(); 0; }) @@ -368,6 +371,10 @@ static inline bool thp_migration_supported(void) { return false; } + +static inline void try_collapse_huge_pmd(struct vm_area_struct *vma, + struct page *page) {} + #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif /* _LINUX_HUGE_MM_H */ diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index a20d7b43a056..9bec602bf79e 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -474,6 +474,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, struct page *old_page, *new_page; struct vm_area_struct *vma; int ret, is_register, ref_ctr_updated = 0; + struct page *orig_page = NULL; is_register = is_swbp_insn(&opcode); uprobe = container_of(auprobe, struct uprobe, arch); @@ -512,7 +513,6 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, copy_to_page(new_page, vaddr, &opcode, UPROBE_SWBP_INSN_SIZE); if (!is_register) { - struct page *orig_page; pgoff_t index; index = vaddr_to_offset(vma, vaddr & PAGE_MASK) >> PAGE_SHIFT; @@ -540,6 +540,9 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, if (ret && is_register && ref_ctr_updated) update_ref_ctr(uprobe, mm, -1); + if (!ret && orig_page && PageTransCompound(orig_page)) + try_collapse_huge_pmd(vma, orig_page); + return ret; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9f8bce9a6b32..cc8464650b72 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2886,6 +2886,75 @@ static struct shrinker deferred_split_shrinker = { .flags = SHRINKER_NUMA_AWARE, }; +/** + * try_collapse_huge_pmd - try collapse pmd for a pte mapped huge page + * @vma: vma containing the huge page + * @page: any sub page of the huge page + */ +void try_collapse_huge_pmd(struct vm_area_struct *vma, + struct page *page) +{ + struct page *hpage = compound_head(page); + struct mm_struct *mm = vma->vm_mm; + struct mmu_notifier_range range; + unsigned long haddr; + unsigned long addr; + pmd_t *pmd, _pmd; + spinlock_t *ptl; + int i, count = 0; + + VM_BUG_ON_PAGE(!PageCompound(page), page); + + haddr = page_address_in_vma(hpage, vma); + pmd = mm_find_pmd(mm, haddr); + if (!pmd) + return; + + lock_page(hpage); + ptl = pmd_lock(mm, pmd); + + /* step 1: check all mapped PTEs */ + for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { + pte_t *pte = pte_offset_map(pmd, addr); + + if (pte_none(*pte)) + continue; + if (hpage + i != vm_normal_page(vma, addr, *pte)) { + spin_unlock(ptl); + unlock_page(hpage); + return; + } + count++; + } + + /* step 2: adjust rmap */ + for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { + pte_t *pte = pte_offset_map(pmd, addr); + struct page *p; + + if (pte_none(*pte)) + continue; + p = vm_normal_page(vma, addr, *pte); + page_remove_rmap(p, false); + } + + /* step 3: flip page table */ + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, NULL, mm, + haddr, haddr + HPAGE_PMD_SIZE); + mmu_notifier_invalidate_range_start(&range); + + _pmd = pmdp_collapse_flush(vma, haddr, pmd); + spin_unlock(ptl); + mmu_notifier_invalidate_range_end(&range); + + /* step 4: free pgtable, set refcount, mm_counters, etc. */ + page_ref_sub(page, count); + unlock_page(hpage); + mm_dec_nr_ptes(mm); + pte_free(mm, pmd_pgtable(_pmd)); + add_mm_counter(mm, mm_counter_file(page), -count); +} + #ifdef CONFIG_DEBUG_FS static int split_huge_pages_set(void *data, u64 val) {