From patchwork Tue Jun 4 16:51:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 10975779 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8A2103E8C for ; Tue, 4 Jun 2019 16:51:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 77A662870E for ; Tue, 4 Jun 2019 16:51:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6BAE428710; Tue, 4 Jun 2019 16:51:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8FCD1286DB for ; Tue, 4 Jun 2019 16:51:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A00216B0271; Tue, 4 Jun 2019 12:51:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9AFF76B0272; Tue, 4 Jun 2019 12:51:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 89FFB6B0273; Tue, 4 Jun 2019 12:51:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id 5457A6B0271 for ; Tue, 4 Jun 2019 12:51:49 -0400 (EDT) Received: by mail-pl1-f197.google.com with SMTP id b24so5744649plz.20 for ; Tue, 04 Jun 2019 09:51:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=czGyebjwq6q9H91dvt642bzkXDyJo2km23TAp5AUCEw=; b=kIGPIqrhRftx/kiys0JlVjh6yXVZlfKW40g1BzeXiTLP6l1icXSsPyxjPRz9Llo8go GQhwxve4Gqudmsp+0iaCh4PA4JJBJPkT3zaMxubA77B/BWfysK8kPGZEz5YyEvPjn4/+ 9s13pS0eCvioz+t6bsLpE8943LteZwQkFXyB0nCxqdjIj01QPh0RIGOBdb4EOmyIDz/N 8aI0N4NOdIRA11hhU38qZDu+pHcEJLoj4miRDhl4S1oFzctHzHPf5mBTHSkpNAlbrE28 v7ZFKH7T3hZJGqNBiWN2auer6k9N0FHtz6LA5/1hFHvNx/e9ITAiDM7kH+TdrZ5AJT0u 7r6Q== X-Gm-Message-State: APjAAAV5UDfgQSI7pfabF62L3MnwDKRlU+wInKDF/L0ckh5wME0m+GGs Hjg3QaXW3Q62RC+zTUsE9iH2Cb8M0zF6VQIqoRW1ckpqN0mCPZ6XF6Asz4T2OC9xIeyVPt6YgUq WlMQFs6N4CAxo9neJ4/Ne7LEEbsBGFW/P9RdLrNEVMViPuxYaj0wb+yh3VztoSRZ0Ig== X-Received: by 2002:a62:2e46:: with SMTP id u67mr39752514pfu.206.1559667108847; Tue, 04 Jun 2019 09:51:48 -0700 (PDT) X-Google-Smtp-Source: APXvYqyn7kj3jK9vSY/netUuqIBeZPd8F6Tm0RxmBKQcOBj6ITbmhHMEoyd8ustK/11oAwfxMNJ3 X-Received: by 2002:a62:2e46:: with SMTP id u67mr39752465pfu.206.1559667108056; Tue, 04 Jun 2019 09:51:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559667108; cv=none; d=google.com; s=arc-20160816; b=KmHyS0Zhpc9HhCWljE+TBngI+RqwFd8OBVmgr894sEpex4n7kxOmZqr5drjsIk4obq qXEMlISAShw8NoeFuu+aI7KyIoghCO1D5rvRoU5Bfs7rlNYXOXtRzJgzsyRTpM/xEeTT 29eTxJfh8+0a8t63Ud30YrwRtVx80UgeEtxnb+QftoqgERV6/JwSNpWTbL2GlceqhSUr YB4HSoHiTdA7ALelI0eVKlpNG48+qOitc1n6j24ARYe/W+WrMou1jF3/9vqS/yLwEXaT CEIEsbxZImKHIA9hWkZ9DBEutoZQURuFIme70uHeRw5zvvC9l/hymsz4URIBT4NBAbI4 Fhyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=czGyebjwq6q9H91dvt642bzkXDyJo2km23TAp5AUCEw=; b=MHQb0WABYZZF2mcbMlIe/elZk3Ov9riQUTRTxD0NG9zVbC1BkEFZqgmpaOregZj7Hl uz3KXRmwPIWTsdyvfoaGSsdyZyi0La116HSURkXbr8F3ur9Bn//3Iws8vjJgXDwgaBU4 Lvx1cJpGDFlMOLosb2uuHeY5nRKrDOryj21u3TFsSDoQb0qTyDbhbNKvPDY2jki8b/WY 3PhKU1CdKgUYlcOewP33vw6Shc8kFdfnDKapPSTtV2M3pG0vTdMVzCu6Dl/TqB2Q9eJB 5GbQIo//sVkx2EJ25aJAywmVZV8GXBRJ7mUoQtSwejNVZeos+tgpgfpuu0nxO7eTtF3s hBaA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=a9iZtmop; spf=pass (google.com: domain of prvs=1058d0e874=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=1058d0e874=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id h1si18769228plr.116.2019.06.04.09.51.47 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Jun 2019 09:51:48 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=1058d0e874=songliubraving@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=a9iZtmop; spf=pass (google.com: domain of prvs=1058d0e874=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=1058d0e874=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109333.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x54Gb3QU011365 for ; Tue, 4 Jun 2019 09:51:47 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=czGyebjwq6q9H91dvt642bzkXDyJo2km23TAp5AUCEw=; b=a9iZtmopl4uqheVbwFpn4pF0Qr+XnQZnYd2PUYmWPNhzu0lDAppIVjySlV/uavoqwpd+ U0FrM/2BkeXNzvMXIv/vF4RrYssMJVHrUDtMPHYl+BqeLgOHOElVF+pPvqoOVdXrF48L NC6l5PQN3d8/BRekSLh8XzTm9OG6BDl1KY4= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2swr7a9061-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Tue, 04 Jun 2019 09:51:47 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::130) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Tue, 4 Jun 2019 09:51:46 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id BC20762E1EE3; Tue, 4 Jun 2019 09:51:44 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH uprobe, thp v2 1/5] mm: move memcmp_pages() and pages_identical() Date: Tue, 4 Jun 2019 09:51:34 -0700 Message-ID: <20190604165138.1520916-2-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190604165138.1520916-1-songliubraving@fb.com> References: <20190604165138.1520916-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-04_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=938 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906040106 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch moves memcmp_pages() to mm/util.c and pages_identical() to mm.h, so that we can use them in other files. Signed-off-by: Song Liu --- include/linux/mm.h | 7 +++++++ mm/ksm.c | 18 ------------------ mm/util.c | 13 +++++++++++++ 3 files changed, 20 insertions(+), 18 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0f57b5dfb331..1bdaf1872492 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2881,5 +2881,12 @@ void __init setup_nr_node_ids(void); static inline void setup_nr_node_ids(void) {} #endif +extern int memcmp_pages(struct page *page1, struct page *page2); + +static inline int pages_identical(struct page *page1, struct page *page2) +{ + return !memcmp_pages(page1, page2); +} + #endif /* __KERNEL__ */ #endif /* _LINUX_MM_H */ diff --git a/mm/ksm.c b/mm/ksm.c index 81c20ed57bf6..6f153f976c4c 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1030,24 +1030,6 @@ static u32 calc_checksum(struct page *page) return checksum; } -static int memcmp_pages(struct page *page1, struct page *page2) -{ - char *addr1, *addr2; - int ret; - - addr1 = kmap_atomic(page1); - addr2 = kmap_atomic(page2); - ret = memcmp(addr1, addr2, PAGE_SIZE); - kunmap_atomic(addr2); - kunmap_atomic(addr1); - return ret; -} - -static inline int pages_identical(struct page *page1, struct page *page2) -{ - return !memcmp_pages(page1, page2); -} - static int write_protect_page(struct vm_area_struct *vma, struct page *page, pte_t *orig_pte) { diff --git a/mm/util.c b/mm/util.c index c2fb8fd807df..c122718de550 100644 --- a/mm/util.c +++ b/mm/util.c @@ -801,3 +801,16 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen) out: return res; } + +int memcmp_pages(struct page *page1, struct page *page2) +{ + char *addr1, *addr2; + int ret; + + addr1 = kmap_atomic(page1); + addr2 = kmap_atomic(page2); + ret = memcmp(addr1, addr2, PAGE_SIZE); + kunmap_atomic(addr2); + kunmap_atomic(addr1); + return ret; +} From patchwork Tue Jun 4 16:51:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 10975781 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AF93D2D49 for ; Tue, 4 Jun 2019 16:51:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9D800286DB for ; Tue, 4 Jun 2019 16:51:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 912882870E; Tue, 4 Jun 2019 16:51:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0499D2870C for ; Tue, 4 Jun 2019 16:51:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5B99E6B0272; Tue, 4 Jun 2019 12:51:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 56A2A6B0273; Tue, 4 Jun 2019 12:51:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 435D96B0274; Tue, 4 Jun 2019 12:51:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f70.google.com (mail-yw1-f70.google.com [209.85.161.70]) by kanga.kvack.org (Postfix) with ESMTP id 21E796B0272 for ; Tue, 4 Jun 2019 12:51:51 -0400 (EDT) Received: by mail-yw1-f70.google.com with SMTP id k10so20195176ywb.18 for ; Tue, 04 Jun 2019 09:51:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=e4naQYLOgSB4W5gzyf2D9ya4eydHyxTdM1krXnD1/N4=; b=C0TGezvbC3mdRlRH66UCoXDtJUrMQ08ZKBbUV1Gi/3m6CYn1awgKvpjiI0ikwmTcTF fJDTBqWfnQ00oPTjsrgD4gDuGya0QtNH15ahfQg5Z3pPJpkqKE7vK/8Egj3XNnpDfF0E DLW1wzGo/w+Zw1+iiaCtklsp3HRE0OTAXjH7JbvdZr0H7r3rVTwqhdokg9ww77rn7fR/ oKEeAL6V7yarnJV+g2vIQ/nRKOV71tqXiMOa7L4uOJQs31LSIULdauavVK0+M/JmdpMw s5WCiVU4PbU2zkP9sdtsZm+K2o0YlC+8crB1X7Ox7Mg8gBnPhKcfJH6NPDeMVg6XcxUO 7NWA== X-Gm-Message-State: APjAAAUers7OhyA4zcvmEgaQRoHnU9Y2c8qK0VBFILIAW8uvPhP0p8EJ UGCEfEm0bMAtPawUZZXM7t3I1aRUQpsBC6PCIHcK7bmHqIjuEVrpUOqQ4k3nvYfwSC2xZiEcnle Ye1hzNX1c5IaPi7SpsDgnS/UPKeeZXnFKSSa9FYsLTLt8gWjTeTpKM+Dr882Kjzpn1g== X-Received: by 2002:a05:6902:523:: with SMTP id y3mr14526958ybs.494.1559667110827; Tue, 04 Jun 2019 09:51:50 -0700 (PDT) X-Google-Smtp-Source: APXvYqwJ6WxDZ2jKG37/PhMHiZG0TegjZDazGdyzQ1OIKFgIMghQJq+ujaYH3HnDRnS0Dr2nnPmZ X-Received: by 2002:a05:6902:523:: with SMTP id y3mr14526941ybs.494.1559667110262; Tue, 04 Jun 2019 09:51:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559667110; cv=none; d=google.com; s=arc-20160816; b=eUV+cp7Vm1rLgaWrgV17D5mLfoDFGFRfuFUSMnSBTzbD3gSDbTtqc2u3cz0J7IbyH9 vV3dcRTe7RySnuE/97aAHJmx5HDCnVpymtxZVCG93ax0EJIcflIj5hzyjZeTQkTFWNI7 ZsvK5SW29z6v4VlbXgU86Ynlzvb+RbuQGrNemnmZ2+OGILYeH9531QWXSpNcP11Wkf62 0mAyZl08ux/T1MHwBxPVRWbiuk+U5DIj+AiZQidtYCCMl46Dbu5LokdMcQp1yX/LI/6V EPNtdt+R1wuL6NJMybuM2pae4DhVyM9e04UVVcdUOvmApPLhFenuOzIDR2LlfdMwlpnS MSZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=e4naQYLOgSB4W5gzyf2D9ya4eydHyxTdM1krXnD1/N4=; b=o+LKe1J5RgvAc8KdMV9+1LD91suQDvTv5eD9fAzKelqO7SMgiAUckAC0EruBidT2py lFl7HZZcp10oQxCKlNMITIWpus7gI+8jVto4QquvAqC8lbyxVkk3vYFYgqmzUJmdZINL aYhPlr9WdpStP3yJh5HJL8tLSvQyDMBpSOYaQOc5OFEJXkAJ0PECWRjkDAS9SfwhEQLr h4Qq8f7ChysonppUUFsT5V8lQP1uyJV61SI0JcpiRtSh/3Kpfe3GX0zK88nYzIKjDsVF kGo4+UthO5o8T4IZZ5N/zSoiKEY5vMeKlO4CDaCD4hFwAT1yvI5uQdGe08pjLWRbBp0L MI9Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=Z04b3DzW; spf=pass (google.com: domain of prvs=1058d0e874=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1058d0e874=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id n6si5124176ybc.103.2019.06.04.09.51.50 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Jun 2019 09:51:50 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=1058d0e874=songliubraving@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=Z04b3DzW; spf=pass (google.com: domain of prvs=1058d0e874=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1058d0e874=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.16.0.27/8.16.0.27) with SMTP id x54GY8b3010365 for ; Tue, 4 Jun 2019 09:51:50 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=e4naQYLOgSB4W5gzyf2D9ya4eydHyxTdM1krXnD1/N4=; b=Z04b3DzWU4kO2rrzPtBVxVYW2ApReohxpsqDTZsr29EGlceJxjTx7Vc+eoXxo/oYcOd2 17ZShpiE05PHl3mV+NVbk2380mTGhxoTNPm/5qXW/Wgg/uOJ6Ei2DMTmC9yYsFuVfshi BFajJji+x6CqhiPJHnudKzXS+F6qJgAP91A= Received: from maileast.thefacebook.com ([163.114.130.16]) by m0089730.ppops.net with ESMTP id 2swun487rh-7 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 04 Jun 2019 09:51:49 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 4 Jun 2019 09:51:48 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id E527B62E1EE3; Tue, 4 Jun 2019 09:51:46 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH uprobe, thp v2 2/5] uprobe: use original page when all uprobes are removed Date: Tue, 4 Jun 2019 09:51:35 -0700 Message-ID: <20190604165138.1520916-3-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190604165138.1520916-1-songliubraving@fb.com> References: <20190604165138.1520916-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-04_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906040106 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently, uprobe swaps the target page with a anonymous page in both install_breakpoint() and remove_breakpoint(). When all uprobes on a page are removed, the given mm is still using an anonymous page (not the original page). This patch allows uprobe to use original page when possible (all uprobes on the page are already removed). Signed-off-by: Song Liu --- kernel/events/uprobes.c | 42 ++++++++++++++++++++++++++++++++--------- 1 file changed, 33 insertions(+), 9 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 78f61bfc6b79..3fca7c55d370 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -160,16 +160,19 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, int err; struct mmu_notifier_range range; struct mem_cgroup *memcg; + bool orig = new_page->mapping != NULL; /* new_page == orig_page */ mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, addr, addr + PAGE_SIZE); VM_BUG_ON_PAGE(PageTransHuge(old_page), old_page); - err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, &memcg, - false); - if (err) - return err; + if (!orig) { + err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, + &memcg, false); + if (err) + return err; + } /* For try_to_free_swap() and munlock_vma_page() below */ lock_page(old_page); @@ -177,15 +180,22 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, mmu_notifier_invalidate_range_start(&range); err = -EAGAIN; if (!page_vma_mapped_walk(&pvmw)) { - mem_cgroup_cancel_charge(new_page, memcg, false); + if (!orig) + mem_cgroup_cancel_charge(new_page, memcg, false); goto unlock; } VM_BUG_ON_PAGE(addr != pvmw.address, old_page); get_page(new_page); - page_add_new_anon_rmap(new_page, vma, addr, false); - mem_cgroup_commit_charge(new_page, memcg, false, false); - lru_cache_add_active_or_unevictable(new_page, vma); + if (orig) { + page_add_file_rmap(new_page, false); + inc_mm_counter(mm, mm_counter_file(new_page)); + dec_mm_counter(mm, MM_ANONPAGES); + } else { + page_add_new_anon_rmap(new_page, vma, addr, false); + mem_cgroup_commit_charge(new_page, memcg, false, false); + lru_cache_add_active_or_unevictable(new_page, vma); + } if (!PageAnon(old_page)) { dec_mm_counter(mm, mm_counter_file(old_page)); @@ -461,9 +471,10 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, unsigned long vaddr, uprobe_opcode_t opcode) { struct uprobe *uprobe; - struct page *old_page, *new_page; + struct page *old_page, *new_page, *orig_page = NULL; struct vm_area_struct *vma; int ret, is_register, ref_ctr_updated = 0; + pgoff_t index; is_register = is_swbp_insn(&opcode); uprobe = container_of(auprobe, struct uprobe, arch); @@ -501,6 +512,19 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, copy_highpage(new_page, old_page); copy_to_page(new_page, vaddr, &opcode, UPROBE_SWBP_INSN_SIZE); + index = vaddr_to_offset(vma, vaddr & PAGE_MASK) >> PAGE_SHIFT; + orig_page = find_get_page(vma->vm_file->f_inode->i_mapping, index); + if (orig_page) { + if (pages_identical(new_page, orig_page)) { + /* if new_page matches orig_page, use orig_page */ + put_page(new_page); + new_page = orig_page; + } else { + put_page(orig_page); + orig_page = NULL; + } + } + ret = __replace_page(vma, vaddr, old_page, new_page); put_page(new_page); put_old: From patchwork Tue Jun 4 16:51:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 10975783 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7E53014C0 for ; Tue, 4 Jun 2019 16:51:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6C719286DB for ; Tue, 4 Jun 2019 16:51:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 60E2C2870E; Tue, 4 Jun 2019 16:51:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DD698286DB for ; Tue, 4 Jun 2019 16:51:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 62EF46B0273; Tue, 4 Jun 2019 12:51:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5B7F26B0274; Tue, 4 Jun 2019 12:51:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 458F56B0276; Tue, 4 Jun 2019 12:51:55 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f71.google.com (mail-yw1-f71.google.com [209.85.161.71]) by kanga.kvack.org (Postfix) with ESMTP id 285066B0273 for ; Tue, 4 Jun 2019 12:51:55 -0400 (EDT) Received: by mail-yw1-f71.google.com with SMTP id k134so19724367ywe.7 for ; Tue, 04 Jun 2019 09:51:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=MR6fp4k0GSjen+IjWvEdAVeV4aJcgnyY2DTlkJp1kX0=; b=sWHWk+pvH1uw6UVZfM3tG4ln1v46DILX1DW2XEE1BWopC3TExR22dzvq3sHAF2QfvN bepScKzesArO/PdvYniewF9CBwL+3tUpmaaUDwEbn+YWS7nwzDUxzICjQZD4TzMDGpzT 1dXMfPrJiTPLt6dbmpu4cInvXVuxT+h4+eohw608CHkUVetEFPI/9itjvqEIKESa4le2 dUaB3y7MFaFoGtasX0bxu1NvjZPUp/VxWtvXnyMpWP2BxCC204WVy76o71dWKTwQ6JKF 1DvAG73tnBFHEcVFq5bD2iykmh9sq3/MRsYAQNKu+QZFSHcdbxQTTezJeweu0V1xcWJ/ JbXA== X-Gm-Message-State: APjAAAVwWUz02H6dUwyI3k5/hXbqJnZ8bi0bOelVkLQYWifGTdSyHsIk 10bFtkhGQaq1JiNMvuUsRVpltPDxqHsEWbIkU2l1IVzNMC6sy0FcBnn3mCYo2EJc4rURwLkEoEC 6jDnQs19G6ELEcOKQh8QxCbyPQ2/YHUTmcxUWQqyrCps9jO6NaUm0SlBY9c2vpQpCRw== X-Received: by 2002:a25:af89:: with SMTP id g9mr15916020ybh.483.1559667114886; Tue, 04 Jun 2019 09:51:54 -0700 (PDT) X-Google-Smtp-Source: APXvYqwVMXXp3DUlHInt2hzyhVEKO1TCJ2v5+LBzk53MDFbg4BdneJDvkHL5/D5DgS65n2okWxTI X-Received: by 2002:a25:af89:: with SMTP id g9mr15915999ybh.483.1559667114295; Tue, 04 Jun 2019 09:51:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559667114; cv=none; d=google.com; s=arc-20160816; b=oq5c4LQslzFTltSe8xHlc3n2DFbi+JL6IrgFHcE+s74MlI8eyQESAYEUDowzskGJHE fFgjpcrYWS1wJR02qcgwEsmM+Pye7aoIgyOSsiG8LNPnNmRPVawd6alaCGW3P5bX9gYk lh60bYa2HPScwlcqmdk8T+376mJs7iSG/4V84ScVUYaFAtQFLte4hOD5ExwY7QRbplcA dibSY8ggVjUZ+hrCYUfGf9fyF/oT0Kno6n6innIHGwOluhdAd5w7g9/uJcwdH30q8ugh mIri929RvyImKPUNQXvYbRldhlApJD/frP3UAZzrM6lxypyIhkQU7XK3XbX17VwHLHpM Nrcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=MR6fp4k0GSjen+IjWvEdAVeV4aJcgnyY2DTlkJp1kX0=; b=rX5HYTC6rKRhBDllMKDa2D9JM0xMZFNMklCTYBn5vGEbNeWOWruA4fRzxzi9mnm3Yv yiX8ciaV/0Ga2FTon1thkTP9rvG3t8mi3toV4XxPrvNfgjvNTyPRkJK5llAB11ggS7be H3xXQFQ5QgQu3nlYOOCQA1y/WtZP4pR8aq7iaY9g1f9Ni7MYGfernuUKwyaTVbn49nSI 7tQxq+g2CbhVpUN+uzaDLunQhYD5/KLADaRix86t936YlOh1yB1gCJ/uwbWN+nN+sMAd BEodvGyfOTUg0+Adi35oZ9swzyuxwF4VgBtoxT/QUAn4zpY0Q5nD9naShF8TCOuSV2AL wVtw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=oQFRwoGG; spf=pass (google.com: domain of prvs=1058d0e874=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1058d0e874=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id x83si1156538ywa.193.2019.06.04.09.51.54 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Jun 2019 09:51:54 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=1058d0e874=songliubraving@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=oQFRwoGG; spf=pass (google.com: domain of prvs=1058d0e874=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1058d0e874=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x54Gd82U023788 for ; Tue, 4 Jun 2019 09:51:53 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=MR6fp4k0GSjen+IjWvEdAVeV4aJcgnyY2DTlkJp1kX0=; b=oQFRwoGG5Aj4+OVviID7vM+knNq4x1P90EH6YS1PoN1K9BuhMEwB63zAQuwYWGL21OKU X/8XNBDCMyFfD16zbPVw8iOkAnaTnnH2HWFhVYIC8iRG6YevuOlSN9+cZAWCYHlQA7kp V9lKC1P4J2NDJsD5oDYk+KEDQkgY4eNZMtQ= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2swr7ms081-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 04 Jun 2019 09:51:52 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::d) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 4 Jun 2019 09:51:52 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id D993D62E1EE3; Tue, 4 Jun 2019 09:51:48 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH uprobe, thp v2 3/5] mm, thp: introduce FOLL_SPLIT_PMD Date: Tue, 4 Jun 2019 09:51:36 -0700 Message-ID: <20190604165138.1520916-4-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190604165138.1520916-1-songliubraving@fb.com> References: <20190604165138.1520916-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-04_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906040106 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patches introduces a new foll_flag: FOLL_SPLIT_PMD. As the name says FOLL_SPLIT_PMD splits huge pmd for given mm_struct, the underlining huge page stays as-is. FOLL_SPLIT_PMD is useful for cases where we need to use regular pages, but would switch back to huge page and huge pmd on. One of such example is uprobe. The following patches use FOLL_SPLIT_PMD in uprobe. Signed-off-by: Song Liu --- include/linux/mm.h | 1 + mm/gup.c | 15 ++++++++++++--- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 1bdaf1872492..8b5f4a9aea0b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2633,6 +2633,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address, #define FOLL_COW 0x4000 /* internal GUP flag */ #define FOLL_ANON 0x8000 /* don't do file mappings */ #define FOLL_LONGTERM 0x10000 /* mapping lifetime is indefinite: see below */ +#define FOLL_SPLIT_PMD 0x20000 /* split huge pmd before returning */ /* * NOTE on FOLL_LONGTERM: diff --git a/mm/gup.c b/mm/gup.c index 63ac50e48072..bdc350d95d99 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -398,7 +398,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, spin_unlock(ptl); return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } - if (flags & FOLL_SPLIT) { + if (flags & (FOLL_SPLIT | FOLL_SPLIT_PMD)) { int ret; page = pmd_page(*pmd); if (is_huge_zero_page(page)) { @@ -407,7 +407,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, split_huge_pmd(vma, pmd, address); if (pmd_trans_unstable(pmd)) ret = -EBUSY; - } else { + } else if (flags & FOLL_SPLIT) { if (unlikely(!try_get_page(page))) { spin_unlock(ptl); return ERR_PTR(-ENOMEM); @@ -419,8 +419,17 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, put_page(page); if (pmd_none(*pmd)) return no_page_table(vma, flags); - } + } else { /* flags & FOLL_SPLIT_PMD */ + pte_t *pte; + spin_unlock(ptl); + split_huge_pmd(vma, pmd, address); + pte = get_locked_pte(mm, address, &ptl); + if (!pte) + return no_page_table(vma, flags); + spin_unlock(ptl); + ret = 0; + } return ret ? ERR_PTR(ret) : follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } From patchwork Tue Jun 4 16:51:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 10975785 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4168114C0 for ; Tue, 4 Jun 2019 16:52:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2FAC4286C4 for ; Tue, 4 Jun 2019 16:52:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 208222870E; Tue, 4 Jun 2019 16:52:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 98972286DB for ; Tue, 4 Jun 2019 16:51:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 175D66B0274; Tue, 4 Jun 2019 12:51:56 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0B0B46B0276; Tue, 4 Jun 2019 12:51:56 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D594F6B0277; Tue, 4 Jun 2019 12:51:55 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id A21E76B0274 for ; Tue, 4 Jun 2019 12:51:55 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id o184so5961981pfg.1 for ; Tue, 04 Jun 2019 09:51:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=uQVFU2CvZFbm2ziFu1mR1VOcrYWG8oH6RytDeroq8Og=; b=RDiOHl+uUOIsZCmqWByfkwhTRivif7l1hK5cfqONoku3lJ0Hd+klLRVpGHBOAkQsRj Kboac4S1xM8lz27yRvbyeK1zeIDycQB8RgxoE/MJwyxde4rz8h/zG++n34kpEQ71w+Ki kN/e4p5WP4PjpqrFoweH51A+Yg5cnVZsciEUsVaozSOanrY3BUy6UoJCA5CMmz68ZOCl EGdvWmtSkfF/4dOLmNQqbdhc8wR1+m02mbE3SFq91l99qtiPzhLF+PVxaMB7S+uOCe/r QtL2PXBFpAQiPVw2dmBa300v1HqL0zwngMRCGNWpMkSDdcGNA+07fmG2cMBv+JX4u3AS s6pg== X-Gm-Message-State: APjAAAU8CtbUKB0htqqymQP4hx89+68fjj02XVWU84DH5IfTMz/oEHtw l+EgCHQFP48mJN+rUAovsGjlTo0IbGEjtSbxtqhhZDDX4KC3mFpd+9/wNW23Ronm/PFaxq7f0sJ PThf7cDlyLeUdbKGoUZxgJd28kKoSRmv0HfwV2FsIsMHvR83QVUPN+jawShl7fvJKJg== X-Received: by 2002:a63:d658:: with SMTP id d24mr36857689pgj.191.1559667115074; Tue, 04 Jun 2019 09:51:55 -0700 (PDT) X-Google-Smtp-Source: APXvYqx/hFJpOJrHQM2ULgwI+HcStP74jErEgDQK5Uk+S7arhBxBt03hE5gmPLkTCLtITJ7IAHC2 X-Received: by 2002:a63:d658:: with SMTP id d24mr36857647pgj.191.1559667114329; Tue, 04 Jun 2019 09:51:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559667114; cv=none; d=google.com; s=arc-20160816; b=SRw98bwk4uJ4Rv1hlFaXMHLVQ8yz7EMDDqI+kIHHaI15C5MXMUoy7oXs/M7BTtG5pk 3Kbcad1+Cm7iEjQVorO45vdMP/cWl7L2WVRKL24QO0vfqrjqqeO4hqyfjLxE5i/1NnbT U09F6c728Yny2l5xJQl0dLCouL30WPnb24aq+SSzauFtX6RxpQpp+sXB5i1OuwT7uCV+ waCDBAuv5P9oUNOm00WXb8CDs4cKyk5P6n2JFSD2woTlGAZo7bCoWJxoJ0R7NILJbUNs e43ziR6rdvMgizUlpddECnx8EZ78zbj+XNSQvJ2YJQea5pqxjZVAx2/hOJ1tFGZ33uJs 4W4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=uQVFU2CvZFbm2ziFu1mR1VOcrYWG8oH6RytDeroq8Og=; b=pkeMx16BUS4dNMjU6a5zrdt2as/kVeFYk31vOpGjAQLbcucAJ9mKZKo2UX2WSi0ZMm EoHQi/c87vb+EzI6wx1YBe3rZtciOXWHrriH37je/jlF+4JAk5rXL3MtXF85umY8l+uW 3sIE50/6QqhBU0dBlewUXrJF9Dk51oZLLzaCDPYBmcT9dAGO7SJKD4kWd0Y4yqaeeKe8 wyW//O8MUo0NAvqts0P/BYHVg35acMOAaC4J8dzWup9g8sJ1qxiBgi7mBkcyEBVe+uIZ dl4DutHtN+okFwJ9hQg0ppYNKoS1HXr5+JeymnQm9L5I9L2h223+IIVFT02eA/E+OSrX D7RA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=hlet+lin; spf=pass (google.com: domain of prvs=1058d0e874=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=1058d0e874=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id e7si15178034pjj.21.2019.06.04.09.51.54 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Jun 2019 09:51:54 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=1058d0e874=songliubraving@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=hlet+lin; spf=pass (google.com: domain of prvs=1058d0e874=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=1058d0e874=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x54Gbc5g009600 for ; Tue, 4 Jun 2019 09:51:53 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=uQVFU2CvZFbm2ziFu1mR1VOcrYWG8oH6RytDeroq8Og=; b=hlet+lin/ilJqoGqNCD/y1sisAALXvDGG82vfA10F54gO8l2cJj5kbmk44HkgL1oXn5j vioRETuIMB+nIfHuKt8jyob8cjdaYHI2fUHPPShTbzCFJY7I5vLlLqr7e18Ki5W+E0k6 8D9S3JVDkBfi8P7xi/RUPIkugGWQx8HpkI8= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2swe1ftj56-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Tue, 04 Jun 2019 09:51:53 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::125) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Tue, 4 Jun 2019 09:51:52 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 8ECF162E1EE3; Tue, 4 Jun 2019 09:51:50 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH uprobe, thp v2 4/5] uprobe: use FOLL_SPLIT_PMD instead of FOLL_SPLIT Date: Tue, 4 Jun 2019 09:51:37 -0700 Message-ID: <20190604165138.1520916-5-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190604165138.1520916-1-songliubraving@fb.com> References: <20190604165138.1520916-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-04_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=811 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906040106 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patches uses newly added FOLL_SPLIT_PMD in uprobe. This enables easy regroup of huge pmd after the uprobe is disabled (in next patch). Signed-off-by: Song Liu --- kernel/events/uprobes.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 3fca7c55d370..88a8e1624bfa 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -153,7 +153,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, { struct mm_struct *mm = vma->vm_mm; struct page_vma_mapped_walk pvmw = { - .page = old_page, + .page = compound_head(old_page), .vma = vma, .address = addr, }; @@ -165,8 +165,6 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, addr, addr + PAGE_SIZE); - VM_BUG_ON_PAGE(PageTransHuge(old_page), old_page); - if (!orig) { err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, &memcg, false); @@ -188,7 +186,9 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, get_page(new_page); if (orig) { + lock_page(new_page); /* for page_add_file_rmap() */ page_add_file_rmap(new_page, false); + unlock_page(new_page); inc_mm_counter(mm, mm_counter_file(new_page)); dec_mm_counter(mm, MM_ANONPAGES); } else { @@ -482,7 +482,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, retry: /* Read the page with vaddr into memory */ ret = get_user_pages_remote(NULL, mm, vaddr, 1, - FOLL_FORCE | FOLL_SPLIT, &old_page, &vma, NULL); + FOLL_FORCE | FOLL_SPLIT_PMD, &old_page, &vma, NULL); if (ret <= 0) return ret; From patchwork Tue Jun 4 16:51:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 10975787 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6A40E14C0 for ; Tue, 4 Jun 2019 16:52:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 56BA4286C4 for ; Tue, 4 Jun 2019 16:52:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4A68C2870C; Tue, 4 Jun 2019 16:52:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AB340286C4 for ; Tue, 4 Jun 2019 16:52:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EB9A26B0276; Tue, 4 Jun 2019 12:51:58 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E6D976B0277; Tue, 4 Jun 2019 12:51:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC0B26B0278; Tue, 4 Jun 2019 12:51:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f69.google.com (mail-yw1-f69.google.com [209.85.161.69]) by kanga.kvack.org (Postfix) with ESMTP id A34376B0276 for ; Tue, 4 Jun 2019 12:51:58 -0400 (EDT) Received: by mail-yw1-f69.google.com with SMTP id t8so9788414ywf.19 for ; Tue, 04 Jun 2019 09:51:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=Dvy8H8Me5IhbWeZE5T8MjN8uMJLZ8NXUKaabsq8Cic4=; b=VvL5Aar5GCKQLGSLD4v2i5rkziwTurwGs476lTjfaWyrXPLOYEwe+0D9Y1/0pCJVqZ AUPigZf2/XhyvrZB6Z/oP1LbLvsTW9uq5Nkmmw+1MSRlJhkExTe8gLMxrk+F10frB9l1 bahkq6go3p/riOVIqeln5QakRjMJOq4ua8oZ8QmH/jxSJXECxV70Cch0BSX+I1twR1vs iGcyIxeSrd98KSGJ5yOmc0YN/mR3SposcZ0AsqSxqpJFr1RtFySJmwIEjFdSu6ijD2Q/ 8NbSCV/AL46z3PJ5qyHvtrT5vxFpI20pQlVXhkFyu5tg7h5p9Dc0TlLKHG+WFOPIE9XK W08w== X-Gm-Message-State: APjAAAUeS+0fwL/keRwM0O/u8ycsRE5wBiKanUj0rxafw+L7YJ6CySGk vqcPlwpUSXBug4aYta5cOhoVqYqlqooWP3FTvTdvpGeOoDQcecMYrIJSTKKSi2BUxZVk9CHJiGi mEAn+zFMyDw/0JetdgrLzW6HQg2Qrwb/5gBu9abnLErd8FMo5n9qgCvdal1cXMdh9oA== X-Received: by 2002:a25:4b84:: with SMTP id y126mr15084886yba.136.1559667118420; Tue, 04 Jun 2019 09:51:58 -0700 (PDT) X-Google-Smtp-Source: APXvYqyHkaxF9cbBnx6umfsM60/iuN3LPi6mIJLfuwaNhjYrvxpPouM95DRsPnxzt3D3j1dMzTbQ X-Received: by 2002:a25:4b84:: with SMTP id y126mr15084859yba.136.1559667117833; Tue, 04 Jun 2019 09:51:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559667117; cv=none; d=google.com; s=arc-20160816; b=cmMeOxwXvXZ8mG3a8lC5K1BWjBvRwu6G9RdHYdEbv7D+Kga9gYRSykqJ8PNn8pjdmg 8IDnte2C44A08fMqwxCS5b+BJjKnc4QBiqZ4GqzllGQ1PY48Y8M3Sl6n5nxW0JTzTepZ C+n9f4dmO09kKNuW9IPH/+JvUPMkfqz0FQ/aQVyyUpsz/+fessz3eze9GP+A1nDwA1Hq +y2tBviYiu09fjPSlbeJnHRHh0f8Tjy3OrJKT8Lz33ldZgTLa/9fcA5jnS57n9b4bcd0 JoBCY/r1F4YzycTrZ/3e7gV07sy12/3rsfVsSdKWY4Ta5UnXayqA45SRCDLpHPtSbUai WjDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=Dvy8H8Me5IhbWeZE5T8MjN8uMJLZ8NXUKaabsq8Cic4=; b=E+FpFMWJgUemZgg2Qr0qKCmV14Te1mYptKpDywX3Jxxb9ucjnXse+3VfpeY1SpR/3Q WsYUeC9K7YvWF+bqSU331Kisj/ZxoITW9WYT1DGwahJrVLOaUXqmlB4pNTLLkNTeVHg2 fm2gqlzJsdWpsqFcXy1y+nOoNJ6AOH/L5S4gSaA0NxKYifvNWpnvKB0typym7B/VvpS4 EYkhw2bPIBawwi/tf9H7bq57eWGeDyD3nOnlOvQKixiHBW7pJAD27RFarrUJnByG3P53 OiIxgDJyJ/yHvk+1zD2b5Z0aVA3xBoq+VGGHdNeF2IViEL/3Dtfdo0HvAIn47Xfl9T9/ CK4w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=Ou6wWs+h; spf=pass (google.com: domain of prvs=1058d0e874=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1058d0e874=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id r128si6311372ywf.303.2019.06.04.09.51.57 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Jun 2019 09:51:57 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=1058d0e874=songliubraving@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=Ou6wWs+h; spf=pass (google.com: domain of prvs=1058d0e874=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1058d0e874=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.16.0.27/8.16.0.27) with SMTP id x54GeO0B008987 for ; Tue, 4 Jun 2019 09:51:56 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=Dvy8H8Me5IhbWeZE5T8MjN8uMJLZ8NXUKaabsq8Cic4=; b=Ou6wWs+hOMKUtCOADrukHWccLV9czCMuqvnZ0cwB2fbFH+0j8l3XbwMmI/Hu3dEvrpqb 6iAFbuh7jDr8sgCJoqPWHDpT4g1TIKlzfq4OxXR5aWmc81f2N1iukMO5ZWt3cKd8o/VC 2e3Z1xTRKTbucIT8RiMMyTJPUlW/0A7R/gQ= Received: from maileast.thefacebook.com ([163.114.130.16]) by m0001303.ppops.net with ESMTP id 2sw8mybnsc-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 04 Jun 2019 09:51:56 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::f) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 4 Jun 2019 09:51:55 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 8A48A62E1EE3; Tue, 4 Jun 2019 09:51:52 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH uprobe, thp v2 5/5] uprobe: collapse THP pmd after removing all uprobes Date: Tue, 4 Jun 2019 09:51:38 -0700 Message-ID: <20190604165138.1520916-6-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190604165138.1520916-1-songliubraving@fb.com> References: <20190604165138.1520916-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-04_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=783 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906040106 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP After all uprobes are removed from the huge page (with PTE pgtable), it is possible to collapse the pmd and benefit from THP again. This patch does the collapse. An issue on earlier version was discovered by kbuild test robot. Reported-by: kbuild test robot Signed-off-by: Song Liu --- include/linux/huge_mm.h | 7 +++++ kernel/events/uprobes.c | 3 ++ mm/huge_memory.c | 70 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 80 insertions(+) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 7cd5c150c21d..b969022dc922 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -250,6 +250,9 @@ static inline bool thp_migration_supported(void) return IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION); } +extern inline void try_collapse_huge_pmd(struct vm_area_struct *vma, + struct page *page); + #else /* CONFIG_TRANSPARENT_HUGEPAGE */ #define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; }) #define HPAGE_PMD_MASK ({ BUILD_BUG(); 0; }) @@ -368,6 +371,10 @@ static inline bool thp_migration_supported(void) { return false; } + +static inline void try_collapse_huge_pmd(struct vm_area_struct *vma, + struct page *page) {} + #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif /* _LINUX_HUGE_MM_H */ diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 88a8e1624bfa..0c8e2358dbf5 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -537,6 +537,9 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, if (ret && is_register && ref_ctr_updated) update_ref_ctr(uprobe, mm, -1); + if (!ret && orig_page && PageTransCompound(orig_page)) + try_collapse_huge_pmd(vma, orig_page); + return ret; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9f8bce9a6b32..03855a480fd2 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2886,6 +2886,76 @@ static struct shrinker deferred_split_shrinker = { .flags = SHRINKER_NUMA_AWARE, }; +/** + * try_collapse_huge_pmd - try collapse pmd for a pte mapped huge page + * @vma: vma containing the huge page + * @page: any sub page of the huge page + */ +void try_collapse_huge_pmd(struct vm_area_struct *vma, + struct page *page) +{ + struct page *hpage = compound_head(page); + struct mm_struct *mm = vma->vm_mm; + struct mmu_notifier_range range; + unsigned long haddr; + unsigned long addr; + pmd_t *pmd, _pmd; + int i, count = 0; + spinlock_t *ptl; + + VM_BUG_ON_PAGE(!PageCompound(page), page); + + haddr = page_address_in_vma(hpage, vma); + pmd = mm_find_pmd(mm, haddr); + if (!pmd) + return; + + ptl = pmd_lock(mm, pmd); + + /* step 1: check all mapped PTEs */ + for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { + pte_t *pte = pte_offset_map(pmd, addr); + + if (pte_none(*pte)) + continue; + if (hpage + i != vm_normal_page(vma, addr, *pte)) { + spin_unlock(ptl); + return; + } + count++; + } + + /* step 2: adjust rmap and refcount */ + for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { + pte_t *pte = pte_offset_map(pmd, addr); + struct page *p; + + if (pte_none(*pte)) + continue; + p = vm_normal_page(vma, addr, *pte); + lock_page(p); + page_remove_rmap(p, false); + unlock_page(p); + put_page(p); + } + + /* step 3: flip page table */ + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, NULL, mm, + haddr, haddr + HPAGE_PMD_SIZE); + mmu_notifier_invalidate_range_start(&range); + + _pmd = pmdp_collapse_flush(vma, haddr, pmd); + spin_unlock(ptl); + mmu_notifier_invalidate_range_end(&range); + + /* step 4: free pgtable, clean up counters, etc. */ + mm_dec_nr_ptes(mm); + pte_free(mm, pmd_pgtable(_pmd)); + add_mm_counter(mm, + shmem_file(vma->vm_file) ? MM_SHMEMPAGES : MM_FILEPAGES, + -count); +} + #ifdef CONFIG_DEBUG_FS static int split_huge_pages_set(void *data, u64 val) {