From patchwork Thu Jun 13 17:57:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 10993375 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6C48414BB for ; Thu, 13 Jun 2019 17:58:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5E80E22701 for ; Thu, 13 Jun 2019 17:58:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 52FC826223; Thu, 13 Jun 2019 17:58:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D2C4322701 for ; Thu, 13 Jun 2019 17:58:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A230B8E0005; Thu, 13 Jun 2019 13:58:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9D3748E0004; Thu, 13 Jun 2019 13:58:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8763A8E0005; Thu, 13 Jun 2019 13:58:02 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 4746A8E0002 for ; Thu, 13 Jun 2019 13:58:02 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id a125so14984846pfa.13 for ; Thu, 13 Jun 2019 10:58:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=pQnGGve/Vnh2HyxOCOdI1CzGbYYB1SvJlXif4X9k+UU=; b=biS5ln6UsTWYjUH9a0CFRxUsco8IlGSSXr/xI1FxGqoJRkyX0LnEt+mIqgabYJO7Jb GhnuVXW2s6xAyDwhyI6Dh6LdRWbekctkqxBPHRQbEYKIYnMwyn1Nr+kY9h8O1P0AxGos pSoG6HLN4z2KzRnf+p8EUhI55kRFKiSFusjAtm+So4VQJiHyCw0uj/R4zEh9dr/iiPb/ HzX6jc6Z/JC0fLDYMXYVa2CyQOZqYuv9Kf4khDxOELeZMLSu+wkV/hNjTdFrKCEnrkzF nHJn8kGVIgyQLjI5dU7roOzJ6w3O+QRNz0PNGElSrcaZXQ4wu5UUXBI76CM5dddPPhQ2 HzIg== X-Gm-Message-State: APjAAAWqSoyWBU6wSTbeKwP+goP73T8SWhfsWs+UtOOxvXJPwQIn0nao gabSANez7yRJmIVA+lRFSWYWr7g3Eam3stXXnaXwvpIFevp6PinN6qNf+a6Xh5Nle9fCUvhNaWu mYgU2C6TYNkeTPxty39m+4so5RjnvH34jlACTBzM0bBdeidz0/GZzURKXCB5SdT7NvQ== X-Received: by 2002:a17:90a:2706:: with SMTP id o6mr6985191pje.62.1560448681952; Thu, 13 Jun 2019 10:58:01 -0700 (PDT) X-Google-Smtp-Source: APXvYqxADinQx+crDzPW+u5Su15Uh0vlrTupeTMgDpUKNSGqG+qlY/jWoDubuJckvdHyprpyIoZb X-Received: by 2002:a17:90a:2706:: with SMTP id o6mr6985140pje.62.1560448681271; Thu, 13 Jun 2019 10:58:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560448681; cv=none; d=google.com; s=arc-20160816; b=hu3XpJUSCnKd1iCGNDrpYKyuRhJ3pzNMaX2QUyyXriCBOqLUnZI0tVdrxA2fD80ANE Ts4dDdKujhzPi5Po4HZ0/cL/KQRMSM3B4FKPHtWzqWq+XK+dH/WrkigBtRi33FGtJ9C/ zHCC7omwKLzw8/RUx4+b7cayjYVNKlogQLtVsWQj4AsJWOjafjOnYgCzpKSwrXkdfZkI JFr5vSfF+J0mCDoBvpmioMgGrfwiIIB/oI8DXY+M/r3Y8RfZnk3luuWv1i7McpvzmLKg vio2jxeM4fm+FdE5B2TA8ce9B0vAv/aIyn/w0DXW29Gfpxi0beQYnQDwH9oDC0BxQCeE rp3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=pQnGGve/Vnh2HyxOCOdI1CzGbYYB1SvJlXif4X9k+UU=; b=PH7QLzJ2a/vpix/PJcm3Fv+963T49ERe0ukUhBr9jWlPWBY5aQh+InLV6a3wUq7yHS S8A6TuHsYbA1pjGE/sTX974wgDbpug2/gorHuzpLBEb28/sn8oFCa2Gz9KhOWWiCjXhi hrFgaNPNO6AIiEDhVGnObEplHdwLfRPT1HNpSLlzs5yRfC3NMDbgIX+5YA24sr1UqSRq jAVO9XyDY6y1HgTxZAterYeBVurtWycyHIwocy1CVQeVq/SXnnF8C+UKKxS26DevBD4w FY/u+NocdywjsH+owUrZ3pigBpBWVBKqrhF7Q9wUrAjdnalkHqNileUyA4V0St+0ULHN MK0w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b="ag/wAUTh"; spf=pass (google.com: domain of prvs=1067aa1dbb=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=1067aa1dbb=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id q28si323832pgb.375.2019.06.13.10.58.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Jun 2019 10:58:01 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=1067aa1dbb=songliubraving@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b="ag/wAUTh"; spf=pass (google.com: domain of prvs=1067aa1dbb=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=1067aa1dbb=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044008.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5DHpv8g018958 for ; Thu, 13 Jun 2019 10:58:00 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=pQnGGve/Vnh2HyxOCOdI1CzGbYYB1SvJlXif4X9k+UU=; b=ag/wAUThFEH/A8UzXs1W+1ceFNTM0hmanYYh0NhLUM+E/J5iTbmPpOW00HmObI+fabof ajBQe3ilhE4r5z+0H0m27BKxF4eZNyYl7Ao4bUm7HG+iOs+dPduYAwItUPsFs25atQ0h tJg6ASlqckIDTAAQ+j+smnTl4HR3IAR7oXw= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2t3ru7gn0a-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 13 Jun 2019 10:58:00 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::f) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 13 Jun 2019 10:57:58 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 0DC4A62E1C18; Thu, 13 Jun 2019 10:57:57 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v4 1/5] mm: move memcmp_pages() and pages_identical() Date: Thu, 13 Jun 2019 10:57:43 -0700 Message-ID: <20190613175747.1964753-2-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190613175747.1964753-1-songliubraving@fb.com> References: <20190613175747.1964753-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-13_12:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=940 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906130131 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch moves memcmp_pages() to mm/util.c and pages_identical() to mm.h, so that we can use them in other files. Signed-off-by: Song Liu --- include/linux/mm.h | 7 +++++++ mm/ksm.c | 18 ------------------ mm/util.c | 13 +++++++++++++ 3 files changed, 20 insertions(+), 18 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index dd0b5f4e1e45..0ab8c7d84cd0 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2891,5 +2891,12 @@ void __init setup_nr_node_ids(void); static inline void setup_nr_node_ids(void) {} #endif +extern int memcmp_pages(struct page *page1, struct page *page2); + +static inline int pages_identical(struct page *page1, struct page *page2) +{ + return !memcmp_pages(page1, page2); +} + #endif /* __KERNEL__ */ #endif /* _LINUX_MM_H */ diff --git a/mm/ksm.c b/mm/ksm.c index 81c20ed57bf6..6f153f976c4c 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1030,24 +1030,6 @@ static u32 calc_checksum(struct page *page) return checksum; } -static int memcmp_pages(struct page *page1, struct page *page2) -{ - char *addr1, *addr2; - int ret; - - addr1 = kmap_atomic(page1); - addr2 = kmap_atomic(page2); - ret = memcmp(addr1, addr2, PAGE_SIZE); - kunmap_atomic(addr2); - kunmap_atomic(addr1); - return ret; -} - -static inline int pages_identical(struct page *page1, struct page *page2) -{ - return !memcmp_pages(page1, page2); -} - static int write_protect_page(struct vm_area_struct *vma, struct page *page, pte_t *orig_pte) { diff --git a/mm/util.c b/mm/util.c index 9834c4ab7d8e..750e586d50bc 100644 --- a/mm/util.c +++ b/mm/util.c @@ -755,3 +755,16 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen) out: return res; } + +int memcmp_pages(struct page *page1, struct page *page2) +{ + char *addr1, *addr2; + int ret; + + addr1 = kmap_atomic(page1); + addr2 = kmap_atomic(page2); + ret = memcmp(addr1, addr2, PAGE_SIZE); + kunmap_atomic(addr2); + kunmap_atomic(addr1); + return ret; +} From patchwork Thu Jun 13 17:57:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 10993377 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8990C924 for ; Thu, 13 Jun 2019 17:58:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 79D2722701 for ; Thu, 13 Jun 2019 17:58:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6E23B26223; Thu, 13 Jun 2019 17:58:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 486CC22701 for ; Thu, 13 Jun 2019 17:58:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C78A88E0002; Thu, 13 Jun 2019 13:58:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id ABE0E8E0006; Thu, 13 Jun 2019 13:58:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C16F8E0002; Thu, 13 Jun 2019 13:58:02 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f72.google.com (mail-yw1-f72.google.com [209.85.161.72]) by kanga.kvack.org (Postfix) with ESMTP id 5A7C08E0004 for ; Thu, 13 Jun 2019 13:58:02 -0400 (EDT) Received: by mail-yw1-f72.google.com with SMTP id p18so6113997ywe.17 for ; Thu, 13 Jun 2019 10:58:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=qKr0k6k4xZHZmykxLb47XrFO0p8RI9jBEYkCbTeGgSI=; b=eideOYUhc5+MxBIN1l8yAXO1dgs/6YkiGhHT9/2NtChDOLVYDefdK91Ho7dWhTN7wx uVuueKp2LLolQ+pa56lR5X3KkRohScXoEwOfBEExvdCZGzn0BcmpdHZ+GyyWOKw/U0lI K+ONYt2R0kcQkE0yM9lJKyXugx7l+4vXKe3pTeVyrrixOMED0KbHICqu8+4nIwWX2Y5Y /y0/6TW/qAb47rxoYjki/U9kDSYWydHXQAj3JQyQ4woT2yhvhe1EvYAjU2ofNc8FY+fM 6ZuTlwWSmVuCxwcR3kcuiHthwxxJjbckqHFII1CS+hk49MHTLjXZSpfUEeNXtztVf3/1 t2Cg== X-Gm-Message-State: APjAAAVT9Z660i68NaVYAumCYaXp0A3dh9LGyNFPw6KkoGWkrYSf0tky 0O2RG6emWxbbB+qBf/7LQbBbwI4a8hp0Z/NJS6UulvlGsg2ZMSh4yiO1L4Q09QVs6MDFsaFNu36 d59VTqyhuZn+e5MpCPSi8SWwVqzDRixmESRokd8JumkGd1LdBl76RNWVlhyXuK7Ev9g== X-Received: by 2002:a81:84cb:: with SMTP id u194mr2708974ywf.297.1560448682101; Thu, 13 Jun 2019 10:58:02 -0700 (PDT) X-Google-Smtp-Source: APXvYqyJ8cpVcA+TuEFWEvR0U78HJLOjq1v2vBuH1gcmBe9Xxm1SS1UskSbwIUfYjn8bttX0A/jd X-Received: by 2002:a81:84cb:: with SMTP id u194mr2708951ywf.297.1560448681595; Thu, 13 Jun 2019 10:58:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560448681; cv=none; d=google.com; s=arc-20160816; b=PQMu5xys93nxhWRucMTz171VhYP28qWgQ3v79pzLQdzdGexkupLQw+j1XO3nAq0kM/ 1fFP65jV5CtbJBH7dCebJCyrmrXGTEmFP+/mNVs223fhYJ1MyfRdHDHwFCy53jRPCkYZ YsXyhLznEyl+CKAOOaBZyIPvSp05Lgy3F/mfwpr+whBRlamgt88EPDL6V1k72eGX5ErY TUriejVajUV6m+XdawPLph3tkVTJ7Olec99jhJhaWxzJ53iRcBV3/AKXhL6Xj0DtKTVo 5J2gRIajJPWdFDBUU4VzSp45/5IIg2pBsHeUQ8R5G8PX6wPqhBv6akELYT0vRxM6a50X gDkA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=qKr0k6k4xZHZmykxLb47XrFO0p8RI9jBEYkCbTeGgSI=; b=deDO5HQvITi3NZz5k4p/77amyR8MjBssbcXAjwwiTTtmBjvwR94TZN15jwlr5/etMq UHbhyxEmO9OWyhRRgfj8Rz79iHZILic8e4oTO4jJNTu6cfUaOy/tUpGevYJwzM0H/ifd MoYx9aJD3Qpt8qPZARLwwbc4QcJ+RKIvL1Fof6hOQIH8qZjz8xU9O1dQZca47+YBFEl7 zkCJnIjZ+ZRBKINuZ6GHDz9NI5L2fiZ1Mogt9wVcRZhiDdosDW3MBnZXQull4rfw+woM HrKf3oORQaerQvKEex1hYBAUOAfhWpZPFK8s0K6dlMcCFN9jvFBcVOU9bsV3sfn5xxAD BWJg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=LRAuwhpR; spf=pass (google.com: domain of prvs=1067aa1dbb=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1067aa1dbb=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id 201si218264yws.178.2019.06.13.10.58.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Jun 2019 10:58:01 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=1067aa1dbb=songliubraving@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=LRAuwhpR; spf=pass (google.com: domain of prvs=1067aa1dbb=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1067aa1dbb=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5DHvvTC009155 for ; Thu, 13 Jun 2019 10:58:01 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=qKr0k6k4xZHZmykxLb47XrFO0p8RI9jBEYkCbTeGgSI=; b=LRAuwhpRLdF4NIyM7KSUny260aIUcNCUFEtuZn1db7kJn38FgQfHqB/k4yoVQXMs4bws i0VfK22s2wa8aj+9YtUT/oMagjWVVP1gj2OZWjGJ6w4y+SvUDOTPUwXZOpj7y2emxVTl 5wZElCSmAv2Kza/lGl6l+uTFfhlekmhwLgY= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2t3pr5h370-13 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Thu, 13 Jun 2019 10:58:01 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::126) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Thu, 13 Jun 2019 10:58:00 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id A55C062E1C18; Thu, 13 Jun 2019 10:57:58 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v4 2/5] uprobe: use original page when all uprobes are removed Date: Thu, 13 Jun 2019 10:57:44 -0700 Message-ID: <20190613175747.1964753-3-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190613175747.1964753-1-songliubraving@fb.com> References: <20190613175747.1964753-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-13_12:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=987 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906130132 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently, uprobe swaps the target page with a anonymous page in both install_breakpoint() and remove_breakpoint(). When all uprobes on a page are removed, the given mm is still using an anonymous page (not the original page). This patch allows uprobe to use original page when possible (all uprobes on the page are already removed). Signed-off-by: Song Liu --- kernel/events/uprobes.c | 45 +++++++++++++++++++++++++++++++++-------- 1 file changed, 37 insertions(+), 8 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 78f61bfc6b79..f7c61a1ef720 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -160,16 +160,19 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, int err; struct mmu_notifier_range range; struct mem_cgroup *memcg; + bool orig = new_page->mapping != NULL; /* new_page == orig_page */ mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, addr, addr + PAGE_SIZE); VM_BUG_ON_PAGE(PageTransHuge(old_page), old_page); - err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, &memcg, - false); - if (err) - return err; + if (!orig) { + err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, + &memcg, false); + if (err) + return err; + } /* For try_to_free_swap() and munlock_vma_page() below */ lock_page(old_page); @@ -177,15 +180,24 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, mmu_notifier_invalidate_range_start(&range); err = -EAGAIN; if (!page_vma_mapped_walk(&pvmw)) { - mem_cgroup_cancel_charge(new_page, memcg, false); + if (!orig) + mem_cgroup_cancel_charge(new_page, memcg, false); goto unlock; } VM_BUG_ON_PAGE(addr != pvmw.address, old_page); get_page(new_page); - page_add_new_anon_rmap(new_page, vma, addr, false); - mem_cgroup_commit_charge(new_page, memcg, false, false); - lru_cache_add_active_or_unevictable(new_page, vma); + if (orig) { + lock_page(new_page); /* for page_add_file_rmap() */ + page_add_file_rmap(new_page, false); + unlock_page(new_page); + inc_mm_counter(mm, mm_counter_file(new_page)); + dec_mm_counter(mm, MM_ANONPAGES); + } else { + page_add_new_anon_rmap(new_page, vma, addr, false); + mem_cgroup_commit_charge(new_page, memcg, false, false); + lru_cache_add_active_or_unevictable(new_page, vma); + } if (!PageAnon(old_page)) { dec_mm_counter(mm, mm_counter_file(old_page)); @@ -501,6 +513,23 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, copy_highpage(new_page, old_page); copy_to_page(new_page, vaddr, &opcode, UPROBE_SWBP_INSN_SIZE); + if (!is_register) { + struct page *orig_page; + pgoff_t index; + + index = vaddr_to_offset(vma, vaddr & PAGE_MASK) >> PAGE_SHIFT; + orig_page = find_get_page(vma->vm_file->f_inode->i_mapping, + index); + + if (orig_page) { + if (pages_identical(new_page, orig_page)) { + put_page(new_page); + new_page = orig_page; + } else + put_page(orig_page); + } + } + ret = __replace_page(vma, vaddr, old_page, new_page); put_page(new_page); put_old: From patchwork Thu Jun 13 17:57:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 10993379 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 58D9714BB for ; Thu, 13 Jun 2019 17:58:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4B6E522701 for ; Thu, 13 Jun 2019 17:58:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3FC6326223; Thu, 13 Jun 2019 17:58:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C1D5622701 for ; Thu, 13 Jun 2019 17:58:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B36078E0006; Thu, 13 Jun 2019 13:58:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AEA9E8E0004; Thu, 13 Jun 2019 13:58:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8EB908E0006; Thu, 13 Jun 2019 13:58:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 54DCC8E0004 for ; Thu, 13 Jun 2019 13:58:04 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id k36so9859552pgl.7 for ; Thu, 13 Jun 2019 10:58:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=9+PXCMixt9lRw9/JwM+cwAKSDJ/N3K3PePNnztPNT8U=; b=hnzbu+tu6CdizrHgfLRTw/PNpZBpWYRZGbfonOjL2wcw6jgvnHyp3ORIdj3nKoHffO ThBpuw3t63wcHjc9fgbJKIP9kVVO7CnDq03lwUdkrVbL9wFDl1t9mly2m+gAMmUaweSY lg2i4ihZPpaqNkHhzFu0nXL7aEAokbVEmIZ8CJpgdp3WNK85UroTNw5UV3u2DqSZV9Wu H1vlfKuiR3cEnZ+nE7cTydkYi/N1hSXCZhRT0UdyN1p6ehUH5BH+bCxDZV7EZ9LyoqUW BXYZ7qPZoYYcBZ3Zvif2XEwjCiyJl4BM9oIu0i7VxXinMvMU06gjFlKQjKlAKkVvywmW H0Kg== X-Gm-Message-State: APjAAAUzBlvNNgj8+PYv/HjLGqo5EmDFpDwSkyIkRyYO+wcFwJXj9If+ 3wgxWtD5iv6dRySvct0iPYJyu5y7S3ONfB0iiPxw+C1xhH6Lvur5c8El8hcfeEcglTFQXBOScpn BclqtkoZQwYvvKOPy2b3GeiP7QEHWgeFM3AAuHbV4z5/80D7la/W52i2rh1GL1NeORQ== X-Received: by 2002:a65:64d6:: with SMTP id t22mr28944915pgv.406.1560448683921; Thu, 13 Jun 2019 10:58:03 -0700 (PDT) X-Google-Smtp-Source: APXvYqzN8R0CA3c2BDaqMkpxYOv1WelQQGHaEG9BCpg2gA6Ul7o8XZJlcLY3R0eaDEop04wh9Ghp X-Received: by 2002:a65:64d6:: with SMTP id t22mr28944863pgv.406.1560448683194; Thu, 13 Jun 2019 10:58:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560448683; cv=none; d=google.com; s=arc-20160816; b=GlzJ9Fs9R46W0ibNeP4jRUrBj4hsyzcLH9TPQl1/qWULMB4OVsb9OErh5SWk4HWQmT XUdpsXRPApbnX7CqOJSKFlhoxjKxAeNic7VQQJCVVazbmp4YoMAL8QBPm0zmb+QKU3H8 L5WeTiaOqj5sdIiTkg9JmYhO+FtFUt/v6BDWJ+8AVuiPxlVEwd4I5+7YK0QQMmhzmfIZ CotnNjyfc1XtJmC33lIq7GPKTyaVriAQd5PHsYCxGXmjUhSuU0qE/g7am0eG4irHV/ge /sozLMzqHsvikYy7NRPVjw6XxPSMvAQr8mBZvDq75w8NaqI8nziQhouLhd0LsXKXRmoi vPSQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=9+PXCMixt9lRw9/JwM+cwAKSDJ/N3K3PePNnztPNT8U=; b=uGXVIEZOxQ/Ywkzb8GWoQc9JQM1+djtPSNhdUbJH+rYGp/y/TOVEgt/MNFOdEhOB16 KD9fl78q7mBRSYTXAWRv66wBHiCxB+4jvNJD/fEOUYFK+Z9H5q1kkUvMtN3pLwgUEByv fV1DGGM8I/qYb9hxEehYPv2d44cDRak04GwID6abADaaHJGttKhbOof/lfvkoSUS7Bq0 Ly6D3mWdmm6+/7QproUid7wDdHpg0NTri5D1EFuGtWTTBUKgZUZm7OxWayjTPYxs/L0o QiRIybEa5/kg+noG57Gu7E+ENqsaSdJxqwyOwHM0XDEuQuRcZv/ngqgsU5k/l1CPgVIO 54iw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=b7Nfvr06; spf=pass (google.com: domain of prvs=1067aa1dbb=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=1067aa1dbb=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id h9si275046pjk.47.2019.06.13.10.58.03 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Jun 2019 10:58:03 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=1067aa1dbb=songliubraving@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=b7Nfvr06; spf=pass (google.com: domain of prvs=1067aa1dbb=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=1067aa1dbb=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109333.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5DHpumX012700 for ; Thu, 13 Jun 2019 10:58:02 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=9+PXCMixt9lRw9/JwM+cwAKSDJ/N3K3PePNnztPNT8U=; b=b7Nfvr067DMelu6Yc/rrrMnDWuH5y+Aue0tApSYAe8vFf/SYmb+jNvsNtKoIes2OWueC biR2uwNKLh0SsWD5C24++c0gPkVuHGGHnjy45aRTTvzb1s6iWJtMLHX+1NJ91ajBLB/C OAXZxpq7Dx2ietQ0atTW6+4HiT/FvNDyTDk= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2t3mtdsgec-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 13 Jun 2019 10:58:02 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::e) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 13 Jun 2019 10:58:00 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 51B5662E1C18; Thu, 13 Jun 2019 10:58:00 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v4 3/5] mm, thp: introduce FOLL_SPLIT_PMD Date: Thu, 13 Jun 2019 10:57:45 -0700 Message-ID: <20190613175747.1964753-4-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190613175747.1964753-1-songliubraving@fb.com> References: <20190613175747.1964753-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-13_12:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=973 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906130131 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patches introduces a new foll_flag: FOLL_SPLIT_PMD. As the name says FOLL_SPLIT_PMD splits huge pmd for given mm_struct, the underlining huge page stays as-is. FOLL_SPLIT_PMD is useful for cases where we need to use regular pages, but would switch back to huge page and huge pmd on. One of such example is uprobe. The following patches use FOLL_SPLIT_PMD in uprobe. Signed-off-by: Song Liu --- include/linux/mm.h | 1 + mm/gup.c | 9 +++++++-- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0ab8c7d84cd0..e605acc4fc81 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2642,6 +2642,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address, #define FOLL_COW 0x4000 /* internal GUP flag */ #define FOLL_ANON 0x8000 /* don't do file mappings */ #define FOLL_LONGTERM 0x10000 /* mapping lifetime is indefinite: see below */ +#define FOLL_SPLIT_PMD 0x20000 /* split huge pmd before returning */ /* * NOTE on FOLL_LONGTERM: diff --git a/mm/gup.c b/mm/gup.c index ddde097cf9e4..0cd3ce599f41 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -398,7 +398,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, spin_unlock(ptl); return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } - if (flags & FOLL_SPLIT) { + if (flags & (FOLL_SPLIT | FOLL_SPLIT_PMD)) { int ret; page = pmd_page(*pmd); if (is_huge_zero_page(page)) { @@ -407,7 +407,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, split_huge_pmd(vma, pmd, address); if (pmd_trans_unstable(pmd)) ret = -EBUSY; - } else { + } else if (flags & FOLL_SPLIT) { if (unlikely(!try_get_page(page))) { spin_unlock(ptl); return ERR_PTR(-ENOMEM); @@ -419,6 +419,11 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, put_page(page); if (pmd_none(*pmd)) return no_page_table(vma, flags); + } else { /* flags & FOLL_SPLIT_PMD */ + spin_unlock(ptl); + ret = 0; + split_huge_pmd(vma, pmd, address); + pte_alloc(mm, pmd); } return ret ? ERR_PTR(ret) : From patchwork Thu Jun 13 17:57:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 10993383 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4F000924 for ; Thu, 13 Jun 2019 17:58:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3DC1E22701 for ; Thu, 13 Jun 2019 17:58:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2D17726223; Thu, 13 Jun 2019 17:58:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B5FF222701 for ; Thu, 13 Jun 2019 17:58:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 40E6C8E0004; Thu, 13 Jun 2019 13:58:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 321608E0009; Thu, 13 Jun 2019 13:58:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 03A408E0007; Thu, 13 Jun 2019 13:58:06 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id AC1258E0004 for ; Thu, 13 Jun 2019 13:58:06 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id f25so14960295pfk.14 for ; Thu, 13 Jun 2019 10:58:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=IZysbkBOWuoojpHfDDSz6omhgc5cHCXvaawoTaDfhK0=; b=K1KEBZIxufjXuMIz4uKAgBe015eGsoeqZgBeu+CYQRtlcRcWy6CsNHRLAVQuCRFir4 JGjl65kyaefGwqIoeTSzYfLC0uTmm841MDw+x2M8oFHG8jzWTrfjJz9fiprEsgyVp9Dm q0zLFV92T0eezhyupT4BltHLaOE0V4GnZIl4sJcPpbpBenDvxgt1KVvB4Za0OY6FcDM+ j+pLTr2DUnbkdqEkDJQkJp6+JS9a8tExk5RxClLhagDhlb8xsJe3eBsqbOtuNVGlV2HT +mMJTi0myq6jPGm8suqH5xxIJ5Q++dkNvlj1T9CVurVCvDZjC6gSrTJ1MKia1oQGwgAw n/dg== X-Gm-Message-State: APjAAAVW0Zry6cfUS+QR+CoM4K3Tunkg1pZJhGQ+ShUmKAdeTxqVqyOS +r2vVbW+ZD1DtmLne9DolGygwv2cn7I9Acf3jpHW9yI/I4rRF/ssszjxkYO7Os1VMiBUixhYCNR 2Vh7//bWqH7S5kaZH01t4IwYzmCJ12ug3kLJTAlAXBJ5s8g0/i9jCHqKF5TEm1v2nSQ== X-Received: by 2002:a17:902:44f:: with SMTP id 73mr31173681ple.192.1560448686364; Thu, 13 Jun 2019 10:58:06 -0700 (PDT) X-Google-Smtp-Source: APXvYqzE8NnHKXXT4xclP5RIOEEK9XOeYitlTWZgphC+ei5ZrOZRU52Tw28Rpt24w0wKlEY5p4gT X-Received: by 2002:a17:902:44f:: with SMTP id 73mr31173634ple.192.1560448685756; Thu, 13 Jun 2019 10:58:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560448685; cv=none; d=google.com; s=arc-20160816; b=VHlGDkuwIV0Yl7Dois409zCzQhKLQRfHbZ/+c0VIKZWVMIG6zH2lDgcncXu3qchh16 T7lHlTU69W083yOgCqv2UMzoc5dYL8XRjbPLQ4hTrfmHucbX8+Tp4Fm578MK7nFHk+/0 BLT8EahVLWDh/f8iN/kO22psCTlaWo32xBK4iIgWKKKELde2n5hA+d7oBY2EEwRAJkLT q+tqch78iF1EhO5F8Nlh6EsLIBwQj6DPHCHZedEmmCBWbveZR3mbddF4+IHdyi07/CeF vQfoMHJsv/LProWt8fZZUqSPoLsfVjLDzciUzkqdjlNasChY+UAa4a+fPHXA7Ip4FoBp 4QfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=IZysbkBOWuoojpHfDDSz6omhgc5cHCXvaawoTaDfhK0=; b=zwbkhQNEIKGrzKAx8X88zUpve/ZMjEipZNaWsL6q2y6iPd7gCo7kYR8+0uh/4Una34 4jIADkBmJgMYgih04SKEM3wNuDl64HqJEW5SBEYVl2bilY06WCrpPsG7q3bzDOreP/n2 CMhRhsMSi3wev9nkmYkZ3lYvccQPNbqBptJZWpv2WhiP5g/z0TL8UQ/bZheaah7MhbNT uMzgShkSt7e1VMKZ1ZBBb3tFo3/b/rfh+z/vwAY3+oyXcUZLczgUZ2MLPjfID9z9p5Rh M+gukLJ18mlrnllV2XvprnRTrXrvcmOxWVtxsOioWFLr8G/+4zxmaPTFLi7qQGp16Gpt hWNA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=Wt1FUbG5; spf=pass (google.com: domain of prvs=1067aa1dbb=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=1067aa1dbb=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id d21si199458pfn.113.2019.06.13.10.58.05 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Jun 2019 10:58:05 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=1067aa1dbb=songliubraving@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=Wt1FUbG5; spf=pass (google.com: domain of prvs=1067aa1dbb=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=1067aa1dbb=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5DHvkL5025828 for ; Thu, 13 Jun 2019 10:58:05 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=IZysbkBOWuoojpHfDDSz6omhgc5cHCXvaawoTaDfhK0=; b=Wt1FUbG5zYZH20fMhnUz0i8kVLVYdntEjTXnRGetNks+vBCiC1z0LcTNTFA2sEK7lvvt H2ykAxthK3KG9X5EQzeJ/GYOfbf9C01EyjwczI8Xy6RVbTT4oEGZajqxktRjj1vN+K0U nJ+S1yGus6Kj/9MNS6P9sQzCYl+JT+9iqj8= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2t3h8dt21s-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 13 Jun 2019 10:58:05 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::e) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 13 Jun 2019 10:58:03 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 31CE662E1C18; Thu, 13 Jun 2019 10:58:02 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v4 4/5] uprobe: use FOLL_SPLIT_PMD instead of FOLL_SPLIT Date: Thu, 13 Jun 2019 10:57:46 -0700 Message-ID: <20190613175747.1964753-5-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190613175747.1964753-1-songliubraving@fb.com> References: <20190613175747.1964753-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-13_12:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=727 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906130132 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patches uses newly added FOLL_SPLIT_PMD in uprobe. This enables easy regroup of huge pmd after the uprobe is disabled (in next patch). Signed-off-by: Song Liu --- kernel/events/uprobes.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index f7c61a1ef720..a20d7b43a056 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -153,7 +153,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, { struct mm_struct *mm = vma->vm_mm; struct page_vma_mapped_walk pvmw = { - .page = old_page, + .page = compound_head(old_page), .vma = vma, .address = addr, }; @@ -165,8 +165,6 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, addr, addr + PAGE_SIZE); - VM_BUG_ON_PAGE(PageTransHuge(old_page), old_page); - if (!orig) { err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, &memcg, false); @@ -483,7 +481,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, retry: /* Read the page with vaddr into memory */ ret = get_user_pages_remote(NULL, mm, vaddr, 1, - FOLL_FORCE | FOLL_SPLIT, &old_page, &vma, NULL); + FOLL_FORCE | FOLL_SPLIT_PMD, &old_page, &vma, NULL); if (ret <= 0) return ret; From patchwork Thu Jun 13 17:57:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 10993381 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5323914BB for ; Thu, 13 Jun 2019 17:58:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 43E1322701 for ; Thu, 13 Jun 2019 17:58:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 382FD26223; Thu, 13 Jun 2019 17:58:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9CBCC22701 for ; Thu, 13 Jun 2019 17:58:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 126E78E0008; Thu, 13 Jun 2019 13:58:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 083BE8E0004; Thu, 13 Jun 2019 13:58:06 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E433E8E0008; Thu, 13 Jun 2019 13:58:06 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f199.google.com (mail-yb1-f199.google.com [209.85.219.199]) by kanga.kvack.org (Postfix) with ESMTP id C015E8E0007 for ; Thu, 13 Jun 2019 13:58:06 -0400 (EDT) Received: by mail-yb1-f199.google.com with SMTP id d6so4114ybj.16 for ; Thu, 13 Jun 2019 10:58:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=yfkmPBW/NLQwl/dH7qhTlDffK3PtcU3kE6J70hPIVKM=; b=m+AntQYIpxpLKIQlAJ9h/dbywQZDa26WKa5uDOXUK7+aoGYZcSEbYOSGwQRFvKRsRN C1Ib52TbgG4crWZ5Db+rF4Sc4dbnetZkzGqJ4+MMACxNGXbhXBtiEVm3FYg4Ek9KCD8b vnjGM7JzBLszrWz6ZMTuWZwvyTIlbVRNL8QcfAqfnxiCEX6RY/JqcuMX57yMOiyAFrsm A6RhRVFlf/8k9ujJLyntnjtlX2yRbYu5UfYvKpSUXYBe23zwPWZRw06X7nyTmk12dQZt Rq+PYr2QyHh39MVzObqACeLg9Ena05CHOH9wx5sPy9OgHyxuMyDJ5mJTw9EVnyAEFi6m pG9w== X-Gm-Message-State: APjAAAWm/GOqJCUp2XgHQfEod2VZFXeyEaSbkhpeIaySgmUxNf+1ANUi +mECkfdYqjv51i5mLzoyI4p2AXdqTPFm380hVB1XZUqnaNIJautQBdGvRgEVP/b/AZuB2EK6VFx u8I+KgZxhYl+AkjnKUzNUDgrUtAVyboZ5WR29YuFmLnv28nJDAlEa8xySuvR89MuNIw== X-Received: by 2002:a81:5b82:: with SMTP id p124mr39878564ywb.63.1560448686474; Thu, 13 Jun 2019 10:58:06 -0700 (PDT) X-Google-Smtp-Source: APXvYqykxSPrfeQ/PaWsJNXdUoIWT1+VKveqbOkLT4sv0wJkTGXpnJ/mHCBMQu+hKAmEUNK0Pm9F X-Received: by 2002:a81:5b82:: with SMTP id p124mr39878537ywb.63.1560448685912; Thu, 13 Jun 2019 10:58:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560448685; cv=none; d=google.com; s=arc-20160816; b=MF32QBhgt6U4Y34qme2c3pkklRJ/Vr0CbjLSE6WjGQFaE3aGMG0gM8XdDdPqkiM5he LhSmvgVQcrDdeKw98ykijjYLNwW+m67Al/IBHaGOp+Oya62vG1S8qI55OXkibF4+tWru VH2xjTvcGznDfyyDxAp+etOXOzwIy9A0qT8DvCnAraeACbR8JLheOOPMdOz++YxZUi73 PgeXcd8Gf7b87bl69onaVAvS6p/c7MH1YxttSOiNJFBrPcOpFq8kP4lfg71Tphi3lW8m hWs6b92A7oLXTOJSCSoehv7CvwqtpaY80JZD2uXMhV/144CKqAzIjQhyr+dePUPyN3oS soog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=yfkmPBW/NLQwl/dH7qhTlDffK3PtcU3kE6J70hPIVKM=; b=gOTeDISaXB1pZXiw1vB09IMAIEuIjfqriUlC55eIhOwvDNXVvWo0+g+ClWa0GP/D1V AVCNuBzKN0eCrB3ssBohOjD2kyp+gZbJTmbVPTPfHvU1u333Ukih3t+yonCKwwBiMUcf gyKhawD2z9foI+rbP1feGtaYLIUsqsRparIcYhRi7zD6+cX/1Ua5+Zqk+ed69jjNDhcO ZAdSIVF3hAVVseP8DlsEVOGlW3sfEUBo2IFL7eXPA/r7fzrsagoiMafnvwEE/yAPNsdo YqjUH6nLJ7P63wx+BDNxlfQt4nuxkA7Znl0kJbbOdsK8lAvonJl3FcHUaa/vy1uAuJmO wmQw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=fHejNoar; spf=pass (google.com: domain of prvs=1067aa1dbb=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1067aa1dbb=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id p129si155907ybb.106.2019.06.13.10.58.05 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Jun 2019 10:58:05 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=1067aa1dbb=songliubraving@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=fHejNoar; spf=pass (google.com: domain of prvs=1067aa1dbb=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1067aa1dbb=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0001255.ppops.net [127.0.0.1]) by mx0b-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5DHsATo007002 for ; Thu, 13 Jun 2019 10:58:05 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=yfkmPBW/NLQwl/dH7qhTlDffK3PtcU3kE6J70hPIVKM=; b=fHejNoarTn64Gikvo4FFamfYIdvs6ajKJcLdK5q3xw4Zm9wkYGjy/mB33kFOSoet5jyS zfxzG9TOdJcoE6zHPTMmT/Rx0+eOwH6Bl50WHROZOxOqiRezkmv4wTyffg2ofOEJHJtP 4DuVKuZFFCC6AVvyP4pYtKQOLb2RRqtSGcQ= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0b-00082601.pphosted.com with ESMTP id 2t3qmj0w0v-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 13 Jun 2019 10:58:05 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 13 Jun 2019 10:58:04 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id BD41E62E1C18; Thu, 13 Jun 2019 10:58:03 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v4 5/5] uprobe: collapse THP pmd after removing all uprobes Date: Thu, 13 Jun 2019 10:57:47 -0700 Message-ID: <20190613175747.1964753-6-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190613175747.1964753-1-songliubraving@fb.com> References: <20190613175747.1964753-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-13_12:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=788 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906130131 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP After all uprobes are removed from the huge page (with PTE pgtable), it is possible to collapse the pmd and benefit from THP again. This patch does the collapse. An issue on earlier version was discovered by kbuild test robot. Reported-by: kbuild test robot Signed-off-by: Song Liu --- include/linux/huge_mm.h | 7 +++++ kernel/events/uprobes.c | 5 ++- mm/huge_memory.c | 69 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 80 insertions(+), 1 deletion(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 7cd5c150c21d..30669e9a9340 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -250,6 +250,9 @@ static inline bool thp_migration_supported(void) return IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION); } +extern void try_collapse_huge_pmd(struct vm_area_struct *vma, + struct page *page); + #else /* CONFIG_TRANSPARENT_HUGEPAGE */ #define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; }) #define HPAGE_PMD_MASK ({ BUILD_BUG(); 0; }) @@ -368,6 +371,10 @@ static inline bool thp_migration_supported(void) { return false; } + +static inline void try_collapse_huge_pmd(struct vm_area_struct *vma, + struct page *page) {} + #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif /* _LINUX_HUGE_MM_H */ diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index a20d7b43a056..9bec602bf79e 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -474,6 +474,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, struct page *old_page, *new_page; struct vm_area_struct *vma; int ret, is_register, ref_ctr_updated = 0; + struct page *orig_page = NULL; is_register = is_swbp_insn(&opcode); uprobe = container_of(auprobe, struct uprobe, arch); @@ -512,7 +513,6 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, copy_to_page(new_page, vaddr, &opcode, UPROBE_SWBP_INSN_SIZE); if (!is_register) { - struct page *orig_page; pgoff_t index; index = vaddr_to_offset(vma, vaddr & PAGE_MASK) >> PAGE_SHIFT; @@ -540,6 +540,9 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, if (ret && is_register && ref_ctr_updated) update_ref_ctr(uprobe, mm, -1); + if (!ret && orig_page && PageTransCompound(orig_page)) + try_collapse_huge_pmd(vma, orig_page); + return ret; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9f8bce9a6b32..cc8464650b72 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2886,6 +2886,75 @@ static struct shrinker deferred_split_shrinker = { .flags = SHRINKER_NUMA_AWARE, }; +/** + * try_collapse_huge_pmd - try collapse pmd for a pte mapped huge page + * @vma: vma containing the huge page + * @page: any sub page of the huge page + */ +void try_collapse_huge_pmd(struct vm_area_struct *vma, + struct page *page) +{ + struct page *hpage = compound_head(page); + struct mm_struct *mm = vma->vm_mm; + struct mmu_notifier_range range; + unsigned long haddr; + unsigned long addr; + pmd_t *pmd, _pmd; + spinlock_t *ptl; + int i, count = 0; + + VM_BUG_ON_PAGE(!PageCompound(page), page); + + haddr = page_address_in_vma(hpage, vma); + pmd = mm_find_pmd(mm, haddr); + if (!pmd) + return; + + lock_page(hpage); + ptl = pmd_lock(mm, pmd); + + /* step 1: check all mapped PTEs */ + for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { + pte_t *pte = pte_offset_map(pmd, addr); + + if (pte_none(*pte)) + continue; + if (hpage + i != vm_normal_page(vma, addr, *pte)) { + spin_unlock(ptl); + unlock_page(hpage); + return; + } + count++; + } + + /* step 2: adjust rmap */ + for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { + pte_t *pte = pte_offset_map(pmd, addr); + struct page *p; + + if (pte_none(*pte)) + continue; + p = vm_normal_page(vma, addr, *pte); + page_remove_rmap(p, false); + } + + /* step 3: flip page table */ + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, NULL, mm, + haddr, haddr + HPAGE_PMD_SIZE); + mmu_notifier_invalidate_range_start(&range); + + _pmd = pmdp_collapse_flush(vma, haddr, pmd); + spin_unlock(ptl); + mmu_notifier_invalidate_range_end(&range); + + /* step 4: free pgtable, set refcount, mm_counters, etc. */ + page_ref_sub(page, count); + unlock_page(hpage); + mm_dec_nr_ptes(mm); + pte_free(mm, pmd_pgtable(_pmd)); + add_mm_counter(mm, mm_counter_file(page), -count); +} + #ifdef CONFIG_DEBUG_FS static int split_huge_pages_set(void *data, u64 val) {