From patchwork Wed Jun 12 22:03:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 10990909 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 07B8414E5 for ; Wed, 12 Jun 2019 22:06:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E7EF828A6A for ; Wed, 12 Jun 2019 22:06:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DBF8228AA2; Wed, 12 Jun 2019 22:06:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6094428A6A for ; Wed, 12 Jun 2019 22:06:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 760176B0269; Wed, 12 Jun 2019 18:06:25 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 710D66B026A; Wed, 12 Jun 2019 18:06:25 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 53CE36B026B; Wed, 12 Jun 2019 18:06:25 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f199.google.com (mail-yb1-f199.google.com [209.85.219.199]) by kanga.kvack.org (Postfix) with ESMTP id 2EC646B0269 for ; Wed, 12 Jun 2019 18:06:25 -0400 (EDT) Received: by mail-yb1-f199.google.com with SMTP id z124so16713240ybz.15 for ; Wed, 12 Jun 2019 15:06:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=pQnGGve/Vnh2HyxOCOdI1CzGbYYB1SvJlXif4X9k+UU=; b=feg0WBsqPfOAXgaXs/KYz5TwWeivQAfrRSvplYqhfrtxtz5F6lmfnG4sysgpXYI6bM thbyidXN/LIbxIkG6wse6kaUOaKkvJHzqEkmFKXlaabY5MBhhrEWpRhNGLWt2h5b3l7M zGIMSuwRmtJU8qv7AC4U9CBzP2cBerajMptWdwLbn8C07DXL8/ig0tyP6HJjukcD/VGY TQ1OvksO42Ok5MYbj9VSnF04OTLYXRfxEtF8l0qWqz9eAQQdLMX4+mF86WJ3pUxo6yL/ QP7CdIuRdDGfHIKCw/QgSpeHCIzDG4M3IcT0QNohykb5hNsUU5qjDuqyh5j7kD9k+1hd 4lug== X-Gm-Message-State: APjAAAX45ts/Fz7Z0h5S9OWYEeHI4z60QdUI9/282HeFGPPbb1M5hWrs QOJwrRSXlrZ/bPNAuaUGeFTLzPhiuW00ZSLgmY6K1QERZ1ZwZLisB5+9y+UpurJuykwH8c795CT 0+OQzrg3K0j6jcy6gRhItS+2TMpqUrZ3z2BM9FK8YVTwR3/wAFxDVjjSJnTEKdmXsyg== X-Received: by 2002:a25:ef0d:: with SMTP id g13mr2674660ybd.425.1560377184934; Wed, 12 Jun 2019 15:06:24 -0700 (PDT) X-Google-Smtp-Source: APXvYqyvlDoF3KbcG1xH8SaeR5xxjOorrO76ti0VovoWta08+UOFVafKCVP4n5+YVXRNj/pisLHV X-Received: by 2002:a25:ef0d:: with SMTP id g13mr2674628ybd.425.1560377184268; Wed, 12 Jun 2019 15:06:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560377184; cv=none; d=google.com; s=arc-20160816; b=u4mXOgiSmhBWXF7dAg/9PBTscjTQhOpz7Ytb8/v2EtTDrVHAMOUWGPnslJsK56OIty v0yMQMseHPmI0xCMHdT/d5ft7XPxsRn/CZ1jgvGAap4wlO1j+BBggT4oshMCEYo3VX+m jY7L3+s0YPF/xHUlD9OhQriN9yWakGsG+fcxiv5UL9qVScbE9HvxLZLu/YdVSyJxUy55 ewKJi5Htds2LOXoN44VlO2O49csC/pulrE61atTHGEClKErcOGq7Jy7D/j/7lc+dFm9V mWYoqDgi18lmGkarXLD5COY/zVD+ZQ+qWIFgjvwFfVJeZ+X9J9QWVCxVnvXXy4bfLTVV 5nfg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=pQnGGve/Vnh2HyxOCOdI1CzGbYYB1SvJlXif4X9k+UU=; b=fNtMW155CKHtYUB04WI5J1vottETzTksuLEl1tUFWmEqxBjFMqId0RSQlXdM+ZN8hS nheuq0pDBGYmBPusN7+lqLEfYheoI4Pvcg+mNO4Ar0ot+u+bASVAB/aHDthn21WtnLeZ A34uLHnkUmGltgO8vO9ipF8W5OGgwIzhjFuCH1P11Z6VaFtPyeJ50NanDxv933x3V0IW USpnRFsFWrPqHBX4TskSIoo5hqyoDHVd16+olWcM5dZ718wHf6yiaA6Ui0B3R7isl5fC qBzXMTKHfJXIXp/dV8ZVyiNAuGsIvTIcUxBb45ec5Hlh896BWNLnVGc72icUB8VvAKej zxow== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=JWAuNS78; spf=pass (google.com: domain of prvs=106696cf5d=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=106696cf5d=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id z199si287754ywa.83.2019.06.12.15.06.24 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 12 Jun 2019 15:06:24 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=106696cf5d=songliubraving@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=JWAuNS78; spf=pass (google.com: domain of prvs=106696cf5d=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=106696cf5d=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5CM2iuL014732 for ; Wed, 12 Jun 2019 15:06:24 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=pQnGGve/Vnh2HyxOCOdI1CzGbYYB1SvJlXif4X9k+UU=; b=JWAuNS789fcNPxnofRvQvtkoJwvLIrNhL2jvavikaTKn60o5iKcg8PEGbEhX69QfO0Q0 M3KtRcs/HlxJn+Tk34Pq9dKYLOTfwpNSbBk+SU0CPWIfSI6Yc18Z1bO8XLNAGOq2cEMn H4LeEqV95Y/j+iGEYsT58hInBnNXoDdiZ+g= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2t37b0gj76-10 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 12 Jun 2019 15:06:23 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::c) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 12 Jun 2019 15:06:15 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id EA96962E2CA8; Wed, 12 Jun 2019 15:03:26 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v3 1/5] mm: move memcmp_pages() and pages_identical() Date: Wed, 12 Jun 2019 15:03:15 -0700 Message-ID: <20190612220320.2223898-2-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190612220320.2223898-1-songliubraving@fb.com> References: <20190612220320.2223898-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-12_13:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=940 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906120153 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch moves memcmp_pages() to mm/util.c and pages_identical() to mm.h, so that we can use them in other files. Signed-off-by: Song Liu --- include/linux/mm.h | 7 +++++++ mm/ksm.c | 18 ------------------ mm/util.c | 13 +++++++++++++ 3 files changed, 20 insertions(+), 18 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index dd0b5f4e1e45..0ab8c7d84cd0 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2891,5 +2891,12 @@ void __init setup_nr_node_ids(void); static inline void setup_nr_node_ids(void) {} #endif +extern int memcmp_pages(struct page *page1, struct page *page2); + +static inline int pages_identical(struct page *page1, struct page *page2) +{ + return !memcmp_pages(page1, page2); +} + #endif /* __KERNEL__ */ #endif /* _LINUX_MM_H */ diff --git a/mm/ksm.c b/mm/ksm.c index 81c20ed57bf6..6f153f976c4c 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1030,24 +1030,6 @@ static u32 calc_checksum(struct page *page) return checksum; } -static int memcmp_pages(struct page *page1, struct page *page2) -{ - char *addr1, *addr2; - int ret; - - addr1 = kmap_atomic(page1); - addr2 = kmap_atomic(page2); - ret = memcmp(addr1, addr2, PAGE_SIZE); - kunmap_atomic(addr2); - kunmap_atomic(addr1); - return ret; -} - -static inline int pages_identical(struct page *page1, struct page *page2) -{ - return !memcmp_pages(page1, page2); -} - static int write_protect_page(struct vm_area_struct *vma, struct page *page, pte_t *orig_pte) { diff --git a/mm/util.c b/mm/util.c index 9834c4ab7d8e..750e586d50bc 100644 --- a/mm/util.c +++ b/mm/util.c @@ -755,3 +755,16 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen) out: return res; } + +int memcmp_pages(struct page *page1, struct page *page2) +{ + char *addr1, *addr2; + int ret; + + addr1 = kmap_atomic(page1); + addr2 = kmap_atomic(page2); + ret = memcmp(addr1, addr2, PAGE_SIZE); + kunmap_atomic(addr2); + kunmap_atomic(addr1); + return ret; +} From patchwork Wed Jun 12 22:03:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 10990913 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3973314E5 for ; Wed, 12 Jun 2019 22:06:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2307828A8E for ; Wed, 12 Jun 2019 22:06:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1380428A6A; Wed, 12 Jun 2019 22:06:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8353528A6A for ; Wed, 12 Jun 2019 22:06:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 785716B026B; Wed, 12 Jun 2019 18:06:39 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 70EF66B026C; Wed, 12 Jun 2019 18:06:39 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5D6E46B026D; Wed, 12 Jun 2019 18:06:39 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 2685C6B026B for ; Wed, 12 Jun 2019 18:06:39 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id k36so7802635pgl.7 for ; Wed, 12 Jun 2019 15:06:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=qKr0k6k4xZHZmykxLb47XrFO0p8RI9jBEYkCbTeGgSI=; b=DMbliIZ2wc+egBwvLU7K+7oi5ZezlZeacDoJ3ygFbosK/Bd5B0cyLaaToF68/o4XrT 2/BXih1gri/D8SJ4FdD90uQ8z+5KC74HrLLspqYxt8ZJ4LP/YR/HX/E5fa/UE+nkU0Ba 00HrBVZy5BaN3fiS7eGPKLvbltm9Nl2uVMJgI0n1SORGLtfUc3eqiXppjA0+NN6xk53N 8b6zKS1dLaU4kpD4BTDs6Mk4kwLUNs2J4msOZ1Hun4oUTq9Fys2mz6XU3xQBNp5tKZ2m YFzrb+sbyuW9xsom3IJVmrepoKjFZWhlZwxbot7xR7IG/NJ46S5H4AIt238fAJZMntD1 i9kQ== X-Gm-Message-State: APjAAAVFz14n3PeCErOsY56Op71d3wKAaavbkQZzSetS1dmGHxL+xco/ 8IIQNSoekydC2eRSyX1wH5rkR3AHVlaUFciSLGZ0Af568W5tpVJ53klRlHnEGGAxZLWr9mO9eVo ytkLEHJopCJLFdI83QTbkhh5F0PRICF76Koei8igbMTrUKPGB/Untg8prA4xdU07QUQ== X-Received: by 2002:a17:902:2ae8:: with SMTP id j95mr46055196plb.276.1560377198678; Wed, 12 Jun 2019 15:06:38 -0700 (PDT) X-Google-Smtp-Source: APXvYqxmhYzMYqYU94+HbGEvMNpUHZXRPXtesZT0DTUe/Tlx/qzUc6Ng5xSbhb31/VtHCMNAztGW X-Received: by 2002:a17:902:2ae8:: with SMTP id j95mr46055158plb.276.1560377197919; Wed, 12 Jun 2019 15:06:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560377197; cv=none; d=google.com; s=arc-20160816; b=Lhi05Qtvy0p4nGRK6Vu2xjMgNIi4w3Mu4Cqgm6vj4ryosImGptEn2uzNUh0ZmmnINE 0cdbpzL4V72RVabCKrU336RuiVRdJIOd+iXHy6VsgGfwhgIahCfP58pB85xUuGHgLW+B dZmzZGlfNQLFqs8XYVm/AQ1gkB6tjPVGrRdO0lmj9L84o/g9HRy9w9H2pusTjxVOy99Q ZgUuEH3+RArtbTUxSTwujO04Erq1qR1QdFDkOM14sYucDLpT/S1SC4FDCFq8Gx+xuosq gCrH9nrkznmXfedFvnkilCMAKPjRbT8nq5qLHk3g0gZO+TIKavouRnr66sX7mZWFaa+b cMeQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=qKr0k6k4xZHZmykxLb47XrFO0p8RI9jBEYkCbTeGgSI=; b=HsNm1bCErmZRI7wPvGBC788Est5ROmcQGgSZcaqV9r12aPxgQsi8gB2jYt6L/OuorN eMAoPkoLPIz3TJTTP1+0lsSD/qhydV8JMjbK5BbBErwPZ2NsG0bkC5iBZK5r8ml++lfW xvDk/5mj9dc/wpiZJfaTOEPpuhBqj9eDDGB+K12jHVJOuAuifLLAxnh8BuaQMVM4nBNO H2ub5XMJy09S1pib47GbP9BtglB2s6Vu91epz93w9ePUbcFHPvBv9fwvxdIXbsEdncfk 5u7bd4xHIpO3lvwzXFqBcBU6RyNzusmumzTWkI/SP6wVOpwClk5gnPQIVKLwtPIXD/j3 jy6w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=nIbNa+YY; spf=pass (google.com: domain of prvs=106696cf5d=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=106696cf5d=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id s1si853396pgs.62.2019.06.12.15.06.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 12 Jun 2019 15:06:37 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=106696cf5d=songliubraving@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=nIbNa+YY; spf=pass (google.com: domain of prvs=106696cf5d=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=106696cf5d=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5CM3E9j000799 for ; Wed, 12 Jun 2019 15:06:37 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=qKr0k6k4xZHZmykxLb47XrFO0p8RI9jBEYkCbTeGgSI=; b=nIbNa+YYFSAy1ydAmHbF7duS/BDB/NT5auxXJafV3HyQhLjCJy1OblKjxSDOHQDNXn91 aD4nuqRR0JromRfkZJ3rD4uwnCO5EEuqBwiRvhz6C1x/mE/cSlXl0Yq4wL4t33pHfsgR gUotB+3tufuOhPDnR0SnXAV0IjoClztmHhM= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2t34g8s94u-8 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 12 Jun 2019 15:06:37 -0700 Received: from prn-hub02.TheFacebook.com (2620:10d:c081:35::126) by prn-hub06.TheFacebook.com (2620:10d:c081:35::130) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.1.1713.5; Wed, 12 Jun 2019 15:06:23 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::126) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Wed, 12 Jun 2019 15:06:17 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id CFA9262E2CC4; Wed, 12 Jun 2019 15:03:35 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v3 2/5] uprobe: use original page when all uprobes are removed Date: Wed, 12 Jun 2019 15:03:16 -0700 Message-ID: <20190612220320.2223898-3-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190612220320.2223898-1-songliubraving@fb.com> References: <20190612220320.2223898-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-12_13:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=974 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906120153 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently, uprobe swaps the target page with a anonymous page in both install_breakpoint() and remove_breakpoint(). When all uprobes on a page are removed, the given mm is still using an anonymous page (not the original page). This patch allows uprobe to use original page when possible (all uprobes on the page are already removed). Signed-off-by: Song Liu --- kernel/events/uprobes.c | 45 +++++++++++++++++++++++++++++++++-------- 1 file changed, 37 insertions(+), 8 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 78f61bfc6b79..f7c61a1ef720 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -160,16 +160,19 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, int err; struct mmu_notifier_range range; struct mem_cgroup *memcg; + bool orig = new_page->mapping != NULL; /* new_page == orig_page */ mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, addr, addr + PAGE_SIZE); VM_BUG_ON_PAGE(PageTransHuge(old_page), old_page); - err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, &memcg, - false); - if (err) - return err; + if (!orig) { + err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, + &memcg, false); + if (err) + return err; + } /* For try_to_free_swap() and munlock_vma_page() below */ lock_page(old_page); @@ -177,15 +180,24 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, mmu_notifier_invalidate_range_start(&range); err = -EAGAIN; if (!page_vma_mapped_walk(&pvmw)) { - mem_cgroup_cancel_charge(new_page, memcg, false); + if (!orig) + mem_cgroup_cancel_charge(new_page, memcg, false); goto unlock; } VM_BUG_ON_PAGE(addr != pvmw.address, old_page); get_page(new_page); - page_add_new_anon_rmap(new_page, vma, addr, false); - mem_cgroup_commit_charge(new_page, memcg, false, false); - lru_cache_add_active_or_unevictable(new_page, vma); + if (orig) { + lock_page(new_page); /* for page_add_file_rmap() */ + page_add_file_rmap(new_page, false); + unlock_page(new_page); + inc_mm_counter(mm, mm_counter_file(new_page)); + dec_mm_counter(mm, MM_ANONPAGES); + } else { + page_add_new_anon_rmap(new_page, vma, addr, false); + mem_cgroup_commit_charge(new_page, memcg, false, false); + lru_cache_add_active_or_unevictable(new_page, vma); + } if (!PageAnon(old_page)) { dec_mm_counter(mm, mm_counter_file(old_page)); @@ -501,6 +513,23 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, copy_highpage(new_page, old_page); copy_to_page(new_page, vaddr, &opcode, UPROBE_SWBP_INSN_SIZE); + if (!is_register) { + struct page *orig_page; + pgoff_t index; + + index = vaddr_to_offset(vma, vaddr & PAGE_MASK) >> PAGE_SHIFT; + orig_page = find_get_page(vma->vm_file->f_inode->i_mapping, + index); + + if (orig_page) { + if (pages_identical(new_page, orig_page)) { + put_page(new_page); + new_page = orig_page; + } else + put_page(orig_page); + } + } + ret = __replace_page(vma, vaddr, old_page, new_page); put_page(new_page); put_old: From patchwork Wed Jun 12 22:03:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 10990911 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 79FD513AF for ; Wed, 12 Jun 2019 22:06:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6809E28A6A for ; Wed, 12 Jun 2019 22:06:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5BA3F28AA2; Wed, 12 Jun 2019 22:06:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D5ECD28A6A for ; Wed, 12 Jun 2019 22:06:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 367386B026A; Wed, 12 Jun 2019 18:06:28 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 27E2E6B026B; Wed, 12 Jun 2019 18:06:28 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D2196B026C; Wed, 12 Jun 2019 18:06:28 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f69.google.com (mail-yw1-f69.google.com [209.85.161.69]) by kanga.kvack.org (Postfix) with ESMTP id DC8886B026A for ; Wed, 12 Jun 2019 18:06:27 -0400 (EDT) Received: by mail-yw1-f69.google.com with SMTP id b75so18905180ywh.8 for ; Wed, 12 Jun 2019 15:06:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=kq4OSN4555gAGkuBdC1pCdofeLuUMnKNDEdg6BpuyMo=; b=kzzcEjo8S5h/lXJnpu7BaffayCCIleyjivja1eD3kryqwifj4UFaJaQJ9wM+JtN+FS d/wS2ZlTmuDaljXUYpcsxhx96IrPW6tqm4qCHLAQDnnkXisTAI4nVShXohG9gC8xTot9 UK2ahi0WPBH5xInWNMysk92oRpyGxYarCQADZVVyqNqvGyu9188/15In8AJRnvGHn97A v4GqMW0njyS/zB2SI97w9invtY7uI7GMi2wQj2RjjAgmLJdZhASeMs946SzBa+HVmajA oybZlKndeKi4D+SCd15GYki9bnAaZWSnz06q0HOp/T+/uM/i3AKAplCuB1iE8HRS/LHf AFFw== X-Gm-Message-State: APjAAAXO7+eqDk/rvadEhUwz9E1Tuuip3wZOL9nzOBrJs5XzkWQsGtXv ZBGF2A0bwn46l4yfQf6vK0wrpEXHPaWgPTSJ0dHEA5WGfHrmFvxGkpk5VDtN3VmAhKOw+habSB7 8U//Y+/rCqPm8E2W9uBjfYITa8nVFPSSRG9rWKW2D+ls25mZP5F0z4v2Xqfpuk/enKg== X-Received: by 2002:a25:1ed6:: with SMTP id e205mr41147803ybe.467.1560377187648; Wed, 12 Jun 2019 15:06:27 -0700 (PDT) X-Google-Smtp-Source: APXvYqyPkUVAcaLZDtmJBZmpfVtb8/GXMmPtZkSJTnseSSkHel2jdLrJtCLQB72J0NrNQNN5wAYU X-Received: by 2002:a25:1ed6:: with SMTP id e205mr41147773ybe.467.1560377187011; Wed, 12 Jun 2019 15:06:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560377187; cv=none; d=google.com; s=arc-20160816; b=0oFS6SJe8lNr2Pl4vFWlmh44+ib/0FqXW+40tydeStvfHn+URXgVccTowWMIWqKn85 tcl8bwqJxkszxnuWk9jKZSo1LHt0zrzQJW0tymS+K8lecJznMkiaM2zxeQxS9q7xtgFY Px5Oav7/PhtlfnbxNZByudI9xKLnn21fb2DJKP4RQYowtcTteCt+jwLSUh7BjSLp/f7i B++pFhsn5j6IJq+kXqCsjD4UQXONLebJxrdsXJVHJB7VlIh1gxOOD5qB069S4mwtXKZg OrDDrq2XMEQk+b41rtA2csFVggymGmw50iOrRKrDNcXk06hp6BiayuVQvgNBQhVMyOAx ue3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=kq4OSN4555gAGkuBdC1pCdofeLuUMnKNDEdg6BpuyMo=; b=aMA2pTT743GflljnqMlgx6wlG9Veh3SWeIlrFQHEdbd92nqcV+IeUMeMwMMDBB8EA/ ec6ACrVAXXDgdTx0sjglWYB3R7FJeljsMQezffDdCl20YzOOH2EbRxmlBORv84Vu1BjH 4X4RgQH/k7d+RYEe3dNIiZdjMhB9XdBWWxSjV+mEVdndXGkGSlD2ur43YTFBO2bKgJ02 3Gb8wCPbyanwaENBo6XWRPFcBUtONH3JjTgaMQsVi4cEKdjLVrSHLYPVzQcOo3sDbWf8 qvyzARs9iEvm6Kd0ACcOfV7/RQV07VGbBO15PAYFd23oGVBNgc5WU3mBBpQVYuuP2r85 z+Tg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=QsXbYSHh; spf=pass (google.com: domain of prvs=106696cf5d=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=106696cf5d=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id j3si272721ywj.185.2019.06.12.15.06.26 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 12 Jun 2019 15:06:26 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=106696cf5d=songliubraving@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=QsXbYSHh; spf=pass (google.com: domain of prvs=106696cf5d=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=106696cf5d=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5CM2iuU014732 for ; Wed, 12 Jun 2019 15:06:26 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=kq4OSN4555gAGkuBdC1pCdofeLuUMnKNDEdg6BpuyMo=; b=QsXbYSHh7Y5nBOQGU5AcXSPjzNbulwnUiVKVn4jp8cyv/Sun086cx62N7Oj2+KZockvL Iu3MBR/K0+kNPbUTu56QRZL6PRnM0hC2pE5hbR9H48TdrYnMeLVN21e0V3wGkfpAX+b5 TeEzKzk714BjKog4pauweFFXys5ukhy9A8o= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2t37b0gj76-19 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 12 Jun 2019 15:06:26 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::c) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 12 Jun 2019 15:06:17 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 2694F62E2CCC; Wed, 12 Jun 2019 15:03:39 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v3 3/5] mm, thp: introduce FOLL_SPLIT_PMD Date: Wed, 12 Jun 2019 15:03:17 -0700 Message-ID: <20190612220320.2223898-4-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190612220320.2223898-1-songliubraving@fb.com> References: <20190612220320.2223898-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-12_13:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906120153 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patches introduces a new foll_flag: FOLL_SPLIT_PMD. As the name says FOLL_SPLIT_PMD splits huge pmd for given mm_struct, the underlining huge page stays as-is. FOLL_SPLIT_PMD is useful for cases where we need to use regular pages, but would switch back to huge page and huge pmd on. One of such example is uprobe. The following patches use FOLL_SPLIT_PMD in uprobe. Signed-off-by: Song Liu --- include/linux/mm.h | 1 + mm/gup.c | 38 +++++++++++++++++++++++++++++++++++--- 2 files changed, 36 insertions(+), 3 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0ab8c7d84cd0..e605acc4fc81 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2642,6 +2642,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address, #define FOLL_COW 0x4000 /* internal GUP flag */ #define FOLL_ANON 0x8000 /* don't do file mappings */ #define FOLL_LONGTERM 0x10000 /* mapping lifetime is indefinite: see below */ +#define FOLL_SPLIT_PMD 0x20000 /* split huge pmd before returning */ /* * NOTE on FOLL_LONGTERM: diff --git a/mm/gup.c b/mm/gup.c index ddde097cf9e4..3d05bddb56c9 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -398,7 +398,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, spin_unlock(ptl); return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } - if (flags & FOLL_SPLIT) { + if (flags & (FOLL_SPLIT | FOLL_SPLIT_PMD)) { int ret; page = pmd_page(*pmd); if (is_huge_zero_page(page)) { @@ -407,7 +407,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, split_huge_pmd(vma, pmd, address); if (pmd_trans_unstable(pmd)) ret = -EBUSY; - } else { + } else if (flags & FOLL_SPLIT) { if (unlikely(!try_get_page(page))) { spin_unlock(ptl); return ERR_PTR(-ENOMEM); @@ -419,8 +419,40 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, put_page(page); if (pmd_none(*pmd)) return no_page_table(vma, flags); - } + } else { /* flags & FOLL_SPLIT_PMD */ + unsigned long addr; + pgprot_t prot; + pte_t *pte; + int i; + + spin_unlock(ptl); + split_huge_pmd(vma, pmd, address); + lock_page(page); + pte = get_locked_pte(mm, address, &ptl); + if (!pte) { + unlock_page(page); + return no_page_table(vma, flags); + } + /* get refcount for every small page */ + page_ref_add(page, HPAGE_PMD_NR); + + prot = READ_ONCE(vma->vm_page_prot); + for (i = 0, addr = address & PMD_MASK; + i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { + struct page *p = page + i; + + pte = pte_offset_map(pmd, addr); + VM_BUG_ON(!pte_none(*pte)); + set_pte_at(mm, addr, pte, mk_pte(p, prot)); + page_add_file_rmap(p, false); + } + + spin_unlock(ptl); + unlock_page(page); + add_mm_counter(mm, mm_counter_file(page), HPAGE_PMD_NR); + ret = 0; + } return ret ? ERR_PTR(ret) : follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } From patchwork Wed Jun 12 22:03:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 10990907 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8C77F14E5 for ; Wed, 12 Jun 2019 22:06:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7921928A6A for ; Wed, 12 Jun 2019 22:06:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6C3F328AA2; Wed, 12 Jun 2019 22:06:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EB14F28A6A for ; Wed, 12 Jun 2019 22:06:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6279A6B0266; Wed, 12 Jun 2019 18:06:22 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5629F6B0269; Wed, 12 Jun 2019 18:06:22 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 403176B026A; Wed, 12 Jun 2019 18:06:22 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f71.google.com (mail-yw1-f71.google.com [209.85.161.71]) by kanga.kvack.org (Postfix) with ESMTP id 1B9E16B0266 for ; Wed, 12 Jun 2019 18:06:22 -0400 (EDT) Received: by mail-yw1-f71.google.com with SMTP id b75so18904805ywh.8 for ; Wed, 12 Jun 2019 15:06:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=IZysbkBOWuoojpHfDDSz6omhgc5cHCXvaawoTaDfhK0=; b=j3IiouoecYuKasjj30d7TRh34nTnGSuY0nz/gVXZFkDA/sAv5iztFou17znxB5khSV Wz3Ad4oZ2oSHagJZPzXi73XmD+vUogI1oI3/4avZVrCcMxPM3a8cMaAcZB/ipXSIBChH qNsWeCoPt2toKzfTpGEaC/k39QwDVXPbzxX4ExrZQlETtQ4Clg9aXoYhZKIA/MReuUVf uuZDUGaRAz1wnFQRf008rmiTSwoXvHBGmVGDIG2q4fuRWmRQkRYW0XXKM1CBX2U9Zg0h oNHia25cmXeAKPdYfjYrgCDcaF4wS9FDfHwNQNG1EEu7r2Qvgg7J5Dv79JZxaWvEDx+x 2ajg== X-Gm-Message-State: APjAAAW+WHPZnjrgduMo5IW6JQyDbCpxHmll+K0jPP9Y8W5f5hvk1Ej5 XoIwYT3mSwUMzc71SxXA2UDU0fPKGCavFYxbUdAvkA16KptaWY+0udkCW60GhBQxCabgdrOGkfR DOlXpAwEjG3t9w+SymJVswQ7kG3rn0zpCR2Xo/kXV7V7PICet/oT2koAFJEcVlNOsTA== X-Received: by 2002:a81:5ec2:: with SMTP id s185mr6351834ywb.35.1560377181785; Wed, 12 Jun 2019 15:06:21 -0700 (PDT) X-Google-Smtp-Source: APXvYqzXjcSOJq1B6X/rYc4R9iQ4maNOU4sVodaxb/Fk73GijxWpVXfAWDBGxeABzbMyJNZ9VHGT X-Received: by 2002:a81:5ec2:: with SMTP id s185mr6351801ywb.35.1560377181275; Wed, 12 Jun 2019 15:06:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560377181; cv=none; d=google.com; s=arc-20160816; b=MfsOBuQbq0ZzIus4iNpuSDPN833ZEPh8Ya2CZie06AP1CT5uQNAXB8cO0Uvp29D6zi 370XSyVv2jy6FrZTMpOVUNKE40QCOdyIS0AKTMckAP0cuC+egdoUFkyoGYNKYPLO20nM VKYKcXLQg19Ho2VQaQmSbwtSSG0aBoq2DpEtEcy8GDiGJs89QEVkf90i9+/BTPDnge/G IxDIss/qqhK6AO53+SZa0n9czdFfWy955hADgkTkXtWwvZOj8TXTbeLQrCxP8eOqhvqp tYlvuaHmu9HeZ8/j+UpWyUBXSIWoNv3+P2Vn2V9MkSw9LU+AUIAzDdxZvUo63L81r+0z In0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=IZysbkBOWuoojpHfDDSz6omhgc5cHCXvaawoTaDfhK0=; b=VEQEUF/h5+vxYb6tGRI5858If89DgtZS278xJzHRlqEuMT34lPM00edK9Yvmaom9CI iQzGxgrr3YTgAVta3j3RK1VNZes6nqZ2ZrBymQrmu1UOQLjV8ysA8MXBT2RXjhliE7EZ QMgjsgC/YlPKi7pokDuD4s6YYm4w2KJlAB36Exzi2CpnWMSqQXEQAY5SiYHS0YGt5mGU SoR9OjC/MhNwXq8EjKFds/Onfxhj6UGTNUVhTrIETzwk+f/Sg9Gcxr7gbOCTZDxNMWKP vsFHzlYZL5ZvjEqvPSE/WEjDxv2fE/i06Ho4Bkojj2tc7iNU6esV6WAr93ZiU76nMfi2 Mppg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=DUVDHpee; spf=pass (google.com: domain of prvs=106696cf5d=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=106696cf5d=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id z138si256755ywa.342.2019.06.12.15.06.21 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 12 Jun 2019 15:06:21 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=106696cf5d=songliubraving@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=DUVDHpee; spf=pass (google.com: domain of prvs=106696cf5d=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=106696cf5d=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5CLxCWZ008898 for ; Wed, 12 Jun 2019 15:06:21 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=IZysbkBOWuoojpHfDDSz6omhgc5cHCXvaawoTaDfhK0=; b=DUVDHpeehWi10Lab//cDJwDOb/LI4bsHpLt5aH50Rgna7p8/rCnjYziNeuis0xU0Wxws jfRURV1mTQbo2ldDDAYkYwWv7EwKRnaLoj2tPFSsi1yqOxUQeMk+3cyfGWoYULT+FU/k Czvqi3Ku+4e2072gRPXrVbJCSLzjTDFnxS0= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2t3338hk6n-10 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 12 Jun 2019 15:06:20 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::d) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 12 Jun 2019 15:06:14 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 04C3C62E2CFA; Wed, 12 Jun 2019 15:03:41 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v3 4/5] uprobe: use FOLL_SPLIT_PMD instead of FOLL_SPLIT Date: Wed, 12 Jun 2019 15:03:18 -0700 Message-ID: <20190612220320.2223898-5-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190612220320.2223898-1-songliubraving@fb.com> References: <20190612220320.2223898-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-12_13:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=727 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906120153 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patches uses newly added FOLL_SPLIT_PMD in uprobe. This enables easy regroup of huge pmd after the uprobe is disabled (in next patch). Signed-off-by: Song Liu --- kernel/events/uprobes.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index f7c61a1ef720..a20d7b43a056 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -153,7 +153,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, { struct mm_struct *mm = vma->vm_mm; struct page_vma_mapped_walk pvmw = { - .page = old_page, + .page = compound_head(old_page), .vma = vma, .address = addr, }; @@ -165,8 +165,6 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, addr, addr + PAGE_SIZE); - VM_BUG_ON_PAGE(PageTransHuge(old_page), old_page); - if (!orig) { err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, &memcg, false); @@ -483,7 +481,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, retry: /* Read the page with vaddr into memory */ ret = get_user_pages_remote(NULL, mm, vaddr, 1, - FOLL_FORCE | FOLL_SPLIT, &old_page, &vma, NULL); + FOLL_FORCE | FOLL_SPLIT_PMD, &old_page, &vma, NULL); if (ret <= 0) return ret; From patchwork Wed Jun 12 22:03:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 10990901 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B1F7713AF for ; Wed, 12 Jun 2019 22:06:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9D51F28A6A for ; Wed, 12 Jun 2019 22:06:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8F07D28AA2; Wed, 12 Jun 2019 22:06:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C629A28A6A for ; Wed, 12 Jun 2019 22:06:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DC7E26B000D; Wed, 12 Jun 2019 18:06:18 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D77A36B000E; Wed, 12 Jun 2019 18:06:18 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C18D66B0010; Wed, 12 Jun 2019 18:06:18 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f197.google.com (mail-yb1-f197.google.com [209.85.219.197]) by kanga.kvack.org (Postfix) with ESMTP id 99F186B000D for ; Wed, 12 Jun 2019 18:06:18 -0400 (EDT) Received: by mail-yb1-f197.google.com with SMTP id r142so16760554ybc.0 for ; Wed, 12 Jun 2019 15:06:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=x9wxrExlRXk0Gxj35W1xu8WnApoUM0ZZw68741IhNJg=; b=SAImKBxwA4BsvTwGi+NtyXCjCZR3ffvs1L2wrB9mnC92SvWQEaBAwuKXSvbMQvayon P1Ee/zEVPp2c/n5JO+HU7uOKbkiy+EsToLkYd1JZOqhSop+bEw/h2uwEY+P+uIdbjCl6 j9dhWpdb1El/jbA/h1DJNNdqH/1ee8aLL5E7E5/eTQUhtoOzZLZJr5e5hmqKWVpBJjrt kkKTnNDkqxOEbKcYR+51Xte7/3+7Z7BYmsbL184wEcCuRPSlCWyeMpRaq/mOq/rfrApb qw774LVZEhEVjCcdisoQjKeWFvIPyuh8L0T93WD5Cy1vSNpvxr1M5ZAuE3XyHvHEQflo 4YtQ== X-Gm-Message-State: APjAAAUcCYCmiHOlOOvUamXgsykAHBzfCsrehz+HaUFHlCBTFfxwxEWA GQLZHcHeEb8coPdWywPmjWzx++0idpEqyWsDyQ2cgTlhUC26NGbxCApprDtC9X5xWsZENhfsyCd 1Xtxh4fdc0eO+exLiE5icbuvVvrtJQAi5TG4Rh9cvgjEjutD2xzB1MFvRSqtQHZiPRw== X-Received: by 2002:a25:cf44:: with SMTP id f65mr41198629ybg.66.1560377178293; Wed, 12 Jun 2019 15:06:18 -0700 (PDT) X-Google-Smtp-Source: APXvYqyIYR0g1tTr4WhfZfbdEGV5H7tn8KNjs9e2OLDrUBRBSTXzhp8M6gd5vmkD/t/4laGsm9zy X-Received: by 2002:a25:cf44:: with SMTP id f65mr41198603ybg.66.1560377177578; Wed, 12 Jun 2019 15:06:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560377177; cv=none; d=google.com; s=arc-20160816; b=LlvZdE3lyD2dyU0HcLbjMcqQAbjSW8q3MNnIyIZmLyaZ4pc5bDz0HQapTpH+KEGNo5 XbVfe+zcB+YUhcBlx7ukOcipZlgoDfZFzauiReGD4xXNCPNf9vhOR3whUWbtcJfSt2X8 ciiWjByHZ9bbn+znaUBad+2g6Z0eoKVSip1q4w8tIkpmAWDL9FYyJgD3idaB3iiplDS1 esvjcpdxC0jxBMugekIU76GRj/zabobuZVM+/mTixrom8WJuXIpGFpmlT3X5bWl4xNMs yq5bVsw15B6fNMli97PAUIcqLyQyolJXtj35pRCyxXpjw2bKqu+LbMEJVULw+Zj/GJv7 6S2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=x9wxrExlRXk0Gxj35W1xu8WnApoUM0ZZw68741IhNJg=; b=XM856bIzkwfkjzQsvnRkvGF7oMY2/0fqtLIXczhu855wsaQF2W/TcnVFgnPnbeqtYf ZhEP1YdQziRgVvUP3C4Uz4mkaTZ82UhKC5trjOS4XdTq+IROHmayLtxO4knb1cN9u/W9 XK2orhSygAjWivZZYtJOZ6dQaTgIvwXjrmrLpwE9olaZ5goVci9gwXYH0+rwKAj5B85+ DZFRIMkDpytK/H14cPamkLbkxsP446gB2o53RL08LhPs3SEAlVN1NSdT9doXEC90iUQw EnjtLaHeXT7oJQHdVZTq/fBo53SpQRIEVzoZpmp02zEst6XtdIxj3ifn3n5/QFed28PA 5qpg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=o6PPi6iG; spf=pass (google.com: domain of prvs=106696cf5d=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=106696cf5d=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id i84si354505ybi.110.2019.06.12.15.06.17 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 12 Jun 2019 15:06:17 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=106696cf5d=songliubraving@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=o6PPi6iG; spf=pass (google.com: domain of prvs=106696cf5d=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=106696cf5d=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5CLxCWT008898 for ; Wed, 12 Jun 2019 15:06:17 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=x9wxrExlRXk0Gxj35W1xu8WnApoUM0ZZw68741IhNJg=; b=o6PPi6iGRpKRqjhqgZG3PiGt6y32v1mSFaSxH2y3WY+iflMDtaSJMqE7EoV4Nzdb9ND0 tYWwi4GF14OooU+Wwf6eHMSoqnqy48FtSSDMalXGJNjVZWnFmQxtVf3RJpR/eI0n4ogc drIEllVUq9LJiqWboIGfaBjGkIQErxJFFQU= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2t3338hk6n-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 12 Jun 2019 15:06:16 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::d) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 12 Jun 2019 15:06:14 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 32B5462E2D12; Wed, 12 Jun 2019 15:03:44 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v3 5/5] uprobe: collapse THP pmd after removing all uprobes Date: Wed, 12 Jun 2019 15:03:19 -0700 Message-ID: <20190612220320.2223898-6-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190612220320.2223898-1-songliubraving@fb.com> References: <20190612220320.2223898-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-12_13:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=784 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906120153 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP After all uprobes are removed from the huge page (with PTE pgtable), it is possible to collapse the pmd and benefit from THP again. This patch does the collapse. An issue on earlier version was discovered by kbuild test robot. Reported-by: kbuild test robot Signed-off-by: Song Liu --- include/linux/huge_mm.h | 7 +++++ kernel/events/uprobes.c | 5 +++- mm/huge_memory.c | 64 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 75 insertions(+), 1 deletion(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 7cd5c150c21d..30669e9a9340 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -250,6 +250,9 @@ static inline bool thp_migration_supported(void) return IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION); } +extern void try_collapse_huge_pmd(struct vm_area_struct *vma, + struct page *page); + #else /* CONFIG_TRANSPARENT_HUGEPAGE */ #define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; }) #define HPAGE_PMD_MASK ({ BUILD_BUG(); 0; }) @@ -368,6 +371,10 @@ static inline bool thp_migration_supported(void) { return false; } + +static inline void try_collapse_huge_pmd(struct vm_area_struct *vma, + struct page *page) {} + #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif /* _LINUX_HUGE_MM_H */ diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index a20d7b43a056..9bec602bf79e 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -474,6 +474,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, struct page *old_page, *new_page; struct vm_area_struct *vma; int ret, is_register, ref_ctr_updated = 0; + struct page *orig_page = NULL; is_register = is_swbp_insn(&opcode); uprobe = container_of(auprobe, struct uprobe, arch); @@ -512,7 +513,6 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, copy_to_page(new_page, vaddr, &opcode, UPROBE_SWBP_INSN_SIZE); if (!is_register) { - struct page *orig_page; pgoff_t index; index = vaddr_to_offset(vma, vaddr & PAGE_MASK) >> PAGE_SHIFT; @@ -540,6 +540,9 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, if (ret && is_register && ref_ctr_updated) update_ref_ctr(uprobe, mm, -1); + if (!ret && orig_page && PageTransCompound(orig_page)) + try_collapse_huge_pmd(vma, orig_page); + return ret; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9f8bce9a6b32..48e951550988 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2886,6 +2886,70 @@ static struct shrinker deferred_split_shrinker = { .flags = SHRINKER_NUMA_AWARE, }; +/** + * try_collapse_huge_pmd - try collapse pmd for a pte mapped huge page + * @vma: vma containing the huge page + * @page: any sub page of the huge page + */ +void try_collapse_huge_pmd(struct vm_area_struct *vma, + struct page *page) +{ + struct page *hpage = compound_head(page); + struct mm_struct *mm = vma->vm_mm; + struct mmu_notifier_range range; + unsigned long haddr; + unsigned long addr; + pmd_t *pmd, _pmd; + spinlock_t *ptl; + int i; + + VM_BUG_ON_PAGE(!PageCompound(page), page); + + haddr = page_address_in_vma(hpage, vma); + pmd = mm_find_pmd(mm, haddr); + if (!pmd) + return; + + lock_page(hpage); + ptl = pmd_lock(mm, pmd); + + /* step 1: check all mapped PTEs */ + for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { + pte_t *pte = pte_offset_map(pmd, addr); + + if (hpage + i != vm_normal_page(vma, addr, *pte)) { + spin_unlock(ptl); + unlock_page(hpage); + return; + } + } + + /* step 2: adjust rmap */ + for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { + pte_t *pte = pte_offset_map(pmd, addr); + struct page *p; + + p = vm_normal_page(vma, addr, *pte); + page_remove_rmap(p, false); + } + + /* step 3: flip page table */ + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, NULL, mm, + haddr, haddr + HPAGE_PMD_SIZE); + mmu_notifier_invalidate_range_start(&range); + + _pmd = pmdp_collapse_flush(vma, haddr, pmd); + spin_unlock(ptl); + mmu_notifier_invalidate_range_end(&range); + + /* step 4: free pgtable, set refcount, mm_counters, etc. */ + page_ref_sub(page, HPAGE_PMD_NR); + unlock_page(hpage); + mm_dec_nr_ptes(mm); + pte_free(mm, pmd_pgtable(_pmd)); + add_mm_counter(mm, mm_counter_file(page), -HPAGE_PMD_NR); +} + #ifdef CONFIG_DEBUG_FS static int split_huge_pages_set(void *data, u64 val) { From patchwork Wed Jun 12 22:03:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 10990905 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4237F13AF for ; Wed, 12 Jun 2019 22:06:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2EAB028A6A for ; Wed, 12 Jun 2019 22:06:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 22E3828AA2; Wed, 12 Jun 2019 22:06:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9986228A6A for ; Wed, 12 Jun 2019 22:06:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 899436B0010; Wed, 12 Jun 2019 18:06:21 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7FD476B0266; Wed, 12 Jun 2019 18:06:21 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6788D6B0269; Wed, 12 Jun 2019 18:06:21 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f199.google.com (mail-yb1-f199.google.com [209.85.219.199]) by kanga.kvack.org (Postfix) with ESMTP id 3C9A06B0010 for ; Wed, 12 Jun 2019 18:06:21 -0400 (EDT) Received: by mail-yb1-f199.google.com with SMTP id u3so10338651ybu.7 for ; Wed, 12 Jun 2019 15:06:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=kq4OSN4555gAGkuBdC1pCdofeLuUMnKNDEdg6BpuyMo=; b=Lmy0LG7fVJepai1HRiDF2c0IFFri8D8eFtedp0pknocNEIysnBbaZZalLkuOR2zizB PmTX9To677R3b8pev7hahCss5JWWkhtfe8G9SDqxFJRtR1jf8C1vfWXfuUPNTOilJ1tx 3o2Ddo206g6U3tBKhRmlUR0X3flhi4tLKAsEDjlThRYXjgg4pQLxhNg+EpX44Wj04Waf 6yKJkCgqhqYCf+9AaokQwmroyvKA32oNL+1/BE8q1XeDLKRrMi8jwbwGM+zaUL7gdMpE ltG1zYcWfNNkTGztwuQmXHwv8Wg/WanAPkPCcDRQOc3XEiC4fqCOJCWS3eUCTOjLflNa LfEg== X-Gm-Message-State: APjAAAWB7D81JL97q8+EbkfhaPUsnUJAXA1fqKAC6OJMWOQnmiVXr7uH ymtvzG0rWpN0FsTVMIgqJVLXWSDWECslyXIpOX2EBznSGdO4QsfW/XmoNgyNWrr/OudEJdNxLX5 xO5z1zdSbb9fqBtNqClpe7hAK8dPCWPaJrb10DYIwfZ4RjrNVB+iv0fG5Tdt4MpjYng== X-Received: by 2002:a25:d789:: with SMTP id o131mr24823324ybg.20.1560377180958; Wed, 12 Jun 2019 15:06:20 -0700 (PDT) X-Google-Smtp-Source: APXvYqypDep0T3M3VvEpVrp0+k6/FIEg4MLa5PAZiMiUU2aASTmH/uNCYNH+zCVrPcNe2wdUT/Jg X-Received: by 2002:a25:d789:: with SMTP id o131mr24823291ybg.20.1560377180357; Wed, 12 Jun 2019 15:06:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560377180; cv=none; d=google.com; s=arc-20160816; b=cPh6eVrwp/IOnvhpVp0JmfdVCnrGFpaRdm++pZbaAQOrSvHhf8kJP4PFs+8S3NYBJt r12PIu2uS+0/Nrf8SaWXYVr98mvmt93HPYbDk9jxFqKyhfWdPbPOUqRaVSwKt4d3WpVN N5Tm2/cEE4PiPdhCxkLVpYDIZW1iXTMm44TIFjgogx5RkF6CeMHBKRgThvYi2cu7fpz9 g/L59hPWhyGH7OBffQDqEnnrvTpGuN9rsuz3AEf1aBo7taL2UaNTDjQx8bpfIHktktkb HDf/G5jiiYnpBtJuyMVRfDl/fs1WXuTseHlutK20TTUMqR7o0A461oXVSZOosRMcNEDv 4snw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=kq4OSN4555gAGkuBdC1pCdofeLuUMnKNDEdg6BpuyMo=; b=yWJzry7v14+eDjT2p7vnH93rgBB24VQ4k1ZKOmCvmQRXn0pi0S3inijvn71Z4c6uO2 VDsPKaWYEEgmr4nTM7kjq8iypzPi0W+u25iIG3s6fdK0U6qcHPuJjB5ofQZYaaO1TYPA dKs/qKVtUVqWPDoUOD3P4WaSS3HBcSSrgNDkUUiyQD122GSY1RDU8a3TlS/mxXgpV48w h27YqTeyypxqEdXMG4ENCaa6+DijdOsOBgNqscCPJpcuwkAU6neFRvyv1symFvKZNAtb 9nLWNv2JH4umheU90otG84cKAF+kFmeY7Tuu6krB2Ip+8p0SrEht1XKA5ji4KE1YzmAl l9TQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=MYkh3U+J; spf=pass (google.com: domain of prvs=106696cf5d=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=106696cf5d=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id l195si281642ywc.3.2019.06.12.15.06.20 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 12 Jun 2019 15:06:20 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=106696cf5d=songliubraving@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=MYkh3U+J; spf=pass (google.com: domain of prvs=106696cf5d=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=106696cf5d=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.16.0.27/8.16.0.27) with SMTP id x5CM3M4A016862 for ; Wed, 12 Jun 2019 15:06:20 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=kq4OSN4555gAGkuBdC1pCdofeLuUMnKNDEdg6BpuyMo=; b=MYkh3U+Ju7SqSxVPQrNXSAF+zvZ9PcRsQQDXJE24KvAVT9tY7UMNyR+CJi3WNtay3Uqh A2Gd0CQrwjeMl8FMIvWIQa+WwTyvwpbAQzh2ItVZvgwSI/YPjM7YuEJJTYMZMxLfcSli WR6tRr0HbvftJ4if31pP0NfFpizKrmvQquU= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by m0089730.ppops.net with ESMTP id 2t353y1470-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 12 Jun 2019 15:06:20 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::125) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Wed, 12 Jun 2019 15:06:15 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id B9B4B62E2D1F; Wed, 12 Jun 2019 15:03:45 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v3 6/6] mm, thp: introduce FOLL_SPLIT_PMD Date: Wed, 12 Jun 2019 15:03:20 -0700 Message-ID: <20190612220320.2223898-7-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190612220320.2223898-1-songliubraving@fb.com> References: <20190612220320.2223898-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-12_13:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906120153 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patches introduces a new foll_flag: FOLL_SPLIT_PMD. As the name says FOLL_SPLIT_PMD splits huge pmd for given mm_struct, the underlining huge page stays as-is. FOLL_SPLIT_PMD is useful for cases where we need to use regular pages, but would switch back to huge page and huge pmd on. One of such example is uprobe. The following patches use FOLL_SPLIT_PMD in uprobe. Signed-off-by: Song Liu --- include/linux/mm.h | 1 + mm/gup.c | 38 +++++++++++++++++++++++++++++++++++--- 2 files changed, 36 insertions(+), 3 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0ab8c7d84cd0..e605acc4fc81 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2642,6 +2642,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address, #define FOLL_COW 0x4000 /* internal GUP flag */ #define FOLL_ANON 0x8000 /* don't do file mappings */ #define FOLL_LONGTERM 0x10000 /* mapping lifetime is indefinite: see below */ +#define FOLL_SPLIT_PMD 0x20000 /* split huge pmd before returning */ /* * NOTE on FOLL_LONGTERM: diff --git a/mm/gup.c b/mm/gup.c index ddde097cf9e4..3d05bddb56c9 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -398,7 +398,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, spin_unlock(ptl); return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } - if (flags & FOLL_SPLIT) { + if (flags & (FOLL_SPLIT | FOLL_SPLIT_PMD)) { int ret; page = pmd_page(*pmd); if (is_huge_zero_page(page)) { @@ -407,7 +407,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, split_huge_pmd(vma, pmd, address); if (pmd_trans_unstable(pmd)) ret = -EBUSY; - } else { + } else if (flags & FOLL_SPLIT) { if (unlikely(!try_get_page(page))) { spin_unlock(ptl); return ERR_PTR(-ENOMEM); @@ -419,8 +419,40 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, put_page(page); if (pmd_none(*pmd)) return no_page_table(vma, flags); - } + } else { /* flags & FOLL_SPLIT_PMD */ + unsigned long addr; + pgprot_t prot; + pte_t *pte; + int i; + + spin_unlock(ptl); + split_huge_pmd(vma, pmd, address); + lock_page(page); + pte = get_locked_pte(mm, address, &ptl); + if (!pte) { + unlock_page(page); + return no_page_table(vma, flags); + } + /* get refcount for every small page */ + page_ref_add(page, HPAGE_PMD_NR); + + prot = READ_ONCE(vma->vm_page_prot); + for (i = 0, addr = address & PMD_MASK; + i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { + struct page *p = page + i; + + pte = pte_offset_map(pmd, addr); + VM_BUG_ON(!pte_none(*pte)); + set_pte_at(mm, addr, pte, mk_pte(p, prot)); + page_add_file_rmap(p, false); + } + + spin_unlock(ptl); + unlock_page(page); + add_mm_counter(mm, mm_counter_file(page), HPAGE_PMD_NR); + ret = 0; + } return ret ? ERR_PTR(ret) : follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); }