From patchwork Wed Aug 7 23:37:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 11082899 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 38FA814DB for ; Wed, 7 Aug 2019 23:37:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 20D1A28AAE for ; Wed, 7 Aug 2019 23:37:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1494F28AB0; Wed, 7 Aug 2019 23:37:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8B72B28AAE for ; Wed, 7 Aug 2019 23:37:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 70DD46B0003; Wed, 7 Aug 2019 19:37:37 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6BE466B0006; Wed, 7 Aug 2019 19:37:37 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5ADF46B0007; Wed, 7 Aug 2019 19:37:37 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f199.google.com (mail-yb1-f199.google.com [209.85.219.199]) by kanga.kvack.org (Postfix) with ESMTP id 3B10B6B0003 for ; Wed, 7 Aug 2019 19:37:37 -0400 (EDT) Received: by mail-yb1-f199.google.com with SMTP id z7so14098914ybp.9 for ; Wed, 07 Aug 2019 16:37:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=LjmLHqeEBIcPM1EzxLLHUMjC3IQ3Iyup262DaJYHDoM=; b=ReKtS/hr1IVqDbEI01Oqwys2MmH3zUEm5pAp3pa1jyHiZCYFqq0Km0qYrXj265q2LD B6jWBo26WEqexWRN2evmpz9VwRHlNIsv00hfyXRYLIYNu36h9rRKX9PAsrB4yE8KqbB3 9q1346gvCxPoSOaONPo1VcgneupoO831CZ3OQdnGjO0zf8CQr6vsJ5pYMpoW1NgnxgZd R0MUDS/JBtxC9hwkThmO72+O+VBgyIJ+XZqtCdeV14aLo969WxZQ6DeV3FZOR/qCFGWs j8sVjWNCFw9Kov6wAiSg6K468eYFUZLVGqtuTxHRvcSiBmSspQvfV5TPWxiM4CWC8Gf6 tdnQ== X-Gm-Message-State: APjAAAUHCEe2igusuTXwasJsXfHXKAn4n75fQPzO72E6GN6fzJqvz/73 pODtFD7mHWe02+2UjHzJiuRKnsCA/3ASmt7pDjYShRwSC25y/SNzCnaUWdhsEzf7nNqQodc0cCD lz6ZpHpODnx8GQOV4aLEAmrU7PUkC681YEKSinApeS6ODpr7168q+0MIeOtMpLIjx8w== X-Received: by 2002:a25:818d:: with SMTP id p13mr3686282ybk.322.1565221056975; Wed, 07 Aug 2019 16:37:36 -0700 (PDT) X-Google-Smtp-Source: APXvYqzO9NoEy5hi/xfSn3EET5jLIHHxnlKMBcYYXrfadI2mpq+MDMQ7kZCC8KKWH4LHxRRYgxi9 X-Received: by 2002:a25:818d:: with SMTP id p13mr3686269ybk.322.1565221056417; Wed, 07 Aug 2019 16:37:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1565221056; cv=none; d=google.com; s=arc-20160816; b=gbjOPa/1Rk3zwE7o5cg6NiIDXnwL9Xg2CQN7EbwEaJ/AkwXJJGU3RfWUzW3BJ2deMG uOMKq/kyY72ewubzWmcfP3JWDlJMOI5h2o1Lsh+IuWaw9SqVBjzhQQR03ViAl1mjxMNz CT1IGdEfLbbNV7yfgBvaZgnOOGx7fJQZvmIMiGhwK9UrMA4+aFhjaooBkYvZmoKouqz2 Q3hZbyWMOHkrqQDYf5/29tHLqfpbc+zR/TfZo0S+J0D0zecNPFw/Rnnh5xwcrAI4mO9e kx4bWyDUXPD7WQ3oDCP5ukTM2XWpSEYvzovWOKKcCyfkyaQ1MjfNh7CG5zKesQ+NqEtb SG6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=LjmLHqeEBIcPM1EzxLLHUMjC3IQ3Iyup262DaJYHDoM=; b=NWxC4eM1MyDhSXbJ0Mlq0YxZzYs3NQie6BvI22SF6rj1/H7+C8Wj4FvkbxIpLK5vp+ GKIbs0SwmGimQrYAGg2xEaE/IZWe35TZNqt1E5Ih+2F7Fqnpp0qfvBuN0kDsCK1YMFwI bN/CNRCZu+vd9E4gnU8rUVw82Qttgt23Matev06pTBb7WJ3Hrjt4Hyv3i7ZiWrrvBbs2 wtYIFDtNcgkjN2UkgwOCwvkMdcbQVZAn5j7yztYKR/LRh+1mBherEhQANeYhXfiMF7s3 J+JJjUjxxf8QC6gdxLENVB4kI0+F5GfQTlNxIz8psiC6b5hf7emf4jHPDxmr2qO+0SJ8 OlPQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=Gv5UZLvI; spf=pass (google.com: domain of prvs=31225916b7=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=31225916b7=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id y4si30415567ybp.210.2019.08.07.16.37.36 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 07 Aug 2019 16:37:36 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=31225916b7=songliubraving@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=Gv5UZLvI; spf=pass (google.com: domain of prvs=31225916b7=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=31225916b7=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x77NXfmb023061 for ; Wed, 7 Aug 2019 16:37:36 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=LjmLHqeEBIcPM1EzxLLHUMjC3IQ3Iyup262DaJYHDoM=; b=Gv5UZLvIQpk3H5o9jYYsuAFPeT+5OaaAH9R4uN6i5nQ3Gskfy9CClU5zuv5DP2v/byKQ 4v5SRjemSgck2QGK6iKRQWBviTZExsycZqi7esWLI27m2y6lE2oCe7Sg9/t57O0TNa59 0xG5IKVQ8t0sxSRPn9CNiAZeuedzqIT2L8w= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2u87ue83xw-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 07 Aug 2019 16:37:36 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 7 Aug 2019 16:37:35 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id F006D62E2D9E; Wed, 7 Aug 2019 16:37:34 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , , CC: , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v12 1/6] mm: move memcmp_pages() and pages_identical() Date: Wed, 7 Aug 2019 16:37:24 -0700 Message-ID: <20190807233729.3899352-2-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190807233729.3899352-1-songliubraving@fb.com> References: <20190807233729.3899352-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-08-07_07:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=973 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1906280000 definitions=main-1908070208 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch moves memcmp_pages() to mm/util.c and pages_identical() to mm.h, so that we can use them in other files. Acked-by: Kirill A. Shutemov Signed-off-by: Song Liu --- include/linux/mm.h | 7 +++++++ mm/ksm.c | 18 ------------------ mm/util.c | 13 +++++++++++++ 3 files changed, 20 insertions(+), 18 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0334ca97c584..f189176dabed 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2891,5 +2891,12 @@ void __init setup_nr_node_ids(void); static inline void setup_nr_node_ids(void) {} #endif +extern int memcmp_pages(struct page *page1, struct page *page2); + +static inline int pages_identical(struct page *page1, struct page *page2) +{ + return !memcmp_pages(page1, page2); +} + #endif /* __KERNEL__ */ #endif /* _LINUX_MM_H */ diff --git a/mm/ksm.c b/mm/ksm.c index 3dc4346411e4..dbee2eb4dd05 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1029,24 +1029,6 @@ static u32 calc_checksum(struct page *page) return checksum; } -static int memcmp_pages(struct page *page1, struct page *page2) -{ - char *addr1, *addr2; - int ret; - - addr1 = kmap_atomic(page1); - addr2 = kmap_atomic(page2); - ret = memcmp(addr1, addr2, PAGE_SIZE); - kunmap_atomic(addr2); - kunmap_atomic(addr1); - return ret; -} - -static inline int pages_identical(struct page *page1, struct page *page2) -{ - return !memcmp_pages(page1, page2); -} - static int write_protect_page(struct vm_area_struct *vma, struct page *page, pte_t *orig_pte) { diff --git a/mm/util.c b/mm/util.c index e6351a80f248..0d5e2f425612 100644 --- a/mm/util.c +++ b/mm/util.c @@ -783,3 +783,16 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen) out: return res; } + +int memcmp_pages(struct page *page1, struct page *page2) +{ + char *addr1, *addr2; + int ret; + + addr1 = kmap_atomic(page1); + addr2 = kmap_atomic(page2); + ret = memcmp(addr1, addr2, PAGE_SIZE); + kunmap_atomic(addr2); + kunmap_atomic(addr1); + return ret; +} From patchwork Wed Aug 7 23:37:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 11082903 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 02314112C for ; Wed, 7 Aug 2019 23:37:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E738128AAE for ; Wed, 7 Aug 2019 23:37:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DB36028AB0; Wed, 7 Aug 2019 23:37:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4A07C28AAE for ; Wed, 7 Aug 2019 23:37:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 69B9D6B0008; Wed, 7 Aug 2019 19:37:40 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 64D666B000A; Wed, 7 Aug 2019 19:37:40 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5164A6B000C; Wed, 7 Aug 2019 19:37:40 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 19A326B0008 for ; Wed, 7 Aug 2019 19:37:40 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id b18so56538191pgg.8 for ; Wed, 07 Aug 2019 16:37:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=i6y9YuM25PmZvfyze3uwWL3WudRVKnCnrUVAiWkqwhw=; b=qhHgXBo+OrnyhHMeUAVfFmxKdTwm0BUrQvLoh5iP4HD6UX0kLqkkKjfFLbxHcAd+79 qntvPrblB9/lj7rpTumw0L7oPiEYvWrGWzwQSIrQyL4c8JLEHptQTi0SvgNWvTHcMjR0 BQKBojIUhQC26zNwv8ALa844xfCiu5ifRbjUNaWgmoLhFO5IIY4l1LUVpgFDAkUM6LfW W3F2fHz5WXmlX7rwT4XIi04MSrjLcPWSFPn/YnhvLUhwJS16Xt06xO8AHL0JwcnC8JZH 9QRLRk535L40ryPW0qndIcHgQcAjVSMG09WzjbNek2YRPUp0inQC1syawbB2gROgRKuo FUDw== X-Gm-Message-State: APjAAAU9bdGxOslnMaPA3Wgap4XpVeOlamgaK5YUBvhiMkk0XFB8VqYi ITJm5M2R3caIQR/Cdk4fXCSfURw23WZs4fpU6CSvB3A3/xAEkRuofBd0DBMeIDIYmd1KTN70Apy t73/ZfdUX3UxMkABlJ34wku51SGuz1wuuMwQk/7/z6vdSAjrAp/UoymLXoS2JNEAplg== X-Received: by 2002:a17:90a:5884:: with SMTP id j4mr952747pji.142.1565221059711; Wed, 07 Aug 2019 16:37:39 -0700 (PDT) X-Google-Smtp-Source: APXvYqxGGlHo5lJAJXYDtOpTyf70Uhy2I14WcupaaAk7zMuBp2anrgriLhosw8y/lNQkhdgBwRUe X-Received: by 2002:a17:90a:5884:: with SMTP id j4mr952692pji.142.1565221058791; Wed, 07 Aug 2019 16:37:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1565221058; cv=none; d=google.com; s=arc-20160816; b=U3FNiXgSwiXDfWOt1xvOk93317MQhLT34qOolAw9RZteXwrnJvB/cdMP2blpkOaKaT x2wdR186hxWLblEUHroz+GwjsPnVzOL8/NXHdI3pHZ49Ek4ViCX5tJAZcy27kag7xyGV DGhuYNP0ZH8C4PoTw4K10DYaR07hbpTrO/M3S8Uvn6Ih4fYog0Y+IW4uXlZISlRLES5V P/UskXZDNbf5AVe4lPDEy1I/6igAicQtTO8VQocbnUABnCrYXbJjn7tTM3mbtBVznMwp 7G8hIY/VBt6Ur62F2IBNpc2+Q1zCwa3Y5bsSWTyxr1H8YdqXbP3PtiTqLrhWuiXp21IC tXCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=i6y9YuM25PmZvfyze3uwWL3WudRVKnCnrUVAiWkqwhw=; b=pvP/LQm9cnKRJpTpV95sgQboNWQOCIY53YwhIMplJp8bwvXmJ7KmEdGVCaMGFCvw7m XrNO/OiDTjIcy3F9FTGPtWvc0NPTvOr3eCK52lsKfGxhFpe0jwCwvUe+cQp2yd1nAkj7 QOHMwfwRmXcx1iebW396nJ1CU3idU+QPs4y8riHzO/fVXTXo7LFxXjzqqW1Z+p8qb0pR 1g+Al4o/3Dq2ypsLPbVb9HPDgoVyGi3OS6xXs8Gy0+yUGIcQPutOc3wcfrdmSr2PkxQK Y4sEExRWZtc3XQf5POhZYZ1wt3fU8sfH1a3IgZ2KUos0mw+lIv77rM/bGTadrJsw61V8 llhA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=bpp63PKW; spf=pass (google.com: domain of prvs=31225916b7=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=31225916b7=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id g24si51361346pfi.119.2019.08.07.16.37.38 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 07 Aug 2019 16:37:38 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=31225916b7=songliubraving@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=bpp63PKW; spf=pass (google.com: domain of prvs=31225916b7=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=31225916b7=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x77NZ1TG031673 for ; Wed, 7 Aug 2019 16:37:38 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=i6y9YuM25PmZvfyze3uwWL3WudRVKnCnrUVAiWkqwhw=; b=bpp63PKWsG2+WIjAGLPTp20V2vkMPg1W0rHosWw7cWWyRbR7K2CIjvInurtiSWuPSOBz UYflakVwnggiwLhplauukSIm7Cc1Ar5vmU1wdsc5M2KSJWHsj+++YlBq1Mq3dE7Puwpr +yG3PKVTReooHvg4Q5S18V19vVsNzsAntHI= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2u87ufg4d8-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 07 Aug 2019 16:37:38 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 7 Aug 2019 16:37:37 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 1AA3862E2D9E; Wed, 7 Aug 2019 16:37:37 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , , CC: , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v12 2/6] uprobe: use original page when all uprobes are removed Date: Wed, 7 Aug 2019 16:37:25 -0700 Message-ID: <20190807233729.3899352-3-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190807233729.3899352-1-songliubraving@fb.com> References: <20190807233729.3899352-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-08-07_07:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=949 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1906280000 definitions=main-1908070208 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently, uprobe swaps the target page with a anonymous page in both install_breakpoint() and remove_breakpoint(). When all uprobes on a page are removed, the given mm is still using an anonymous page (not the original page). This patch allows uprobe to use original page when possible (all uprobes on the page are already removed, and the original page is in page cache and uptodate). As suggested by Oleg, we unmap the old_page and let the original page fault in. Suggested-by: Oleg Nesterov Signed-off-by: Song Liu --- kernel/events/uprobes.c | 66 +++++++++++++++++++++++++++++++---------- 1 file changed, 51 insertions(+), 15 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 84fa00497c49..648f47553bff 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -143,10 +143,12 @@ static loff_t vaddr_to_offset(struct vm_area_struct *vma, unsigned long vaddr) * * @vma: vma that holds the pte pointing to page * @addr: address the old @page is mapped at - * @page: the cowed page we are replacing by kpage - * @kpage: the modified page we replace page by + * @old_page: the page we are replacing by new_page + * @new_page: the modified page we replace page by * - * Returns 0 on success, -EFAULT on failure. + * If @new_page is NULL, only unmap @old_page. + * + * Returns 0 on success, negative error code otherwise. */ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, struct page *old_page, struct page *new_page) @@ -166,10 +168,12 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, VM_BUG_ON_PAGE(PageTransHuge(old_page), old_page); - err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, &memcg, - false); - if (err) - return err; + if (new_page) { + err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, + &memcg, false); + if (err) + return err; + } /* For try_to_free_swap() and munlock_vma_page() below */ lock_page(old_page); @@ -177,15 +181,20 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, mmu_notifier_invalidate_range_start(&range); err = -EAGAIN; if (!page_vma_mapped_walk(&pvmw)) { - mem_cgroup_cancel_charge(new_page, memcg, false); + if (new_page) + mem_cgroup_cancel_charge(new_page, memcg, false); goto unlock; } VM_BUG_ON_PAGE(addr != pvmw.address, old_page); - get_page(new_page); - page_add_new_anon_rmap(new_page, vma, addr, false); - mem_cgroup_commit_charge(new_page, memcg, false, false); - lru_cache_add_active_or_unevictable(new_page, vma); + if (new_page) { + get_page(new_page); + page_add_new_anon_rmap(new_page, vma, addr, false); + mem_cgroup_commit_charge(new_page, memcg, false, false); + lru_cache_add_active_or_unevictable(new_page, vma); + } else + /* no new page, just dec_mm_counter for old_page */ + dec_mm_counter(mm, MM_ANONPAGES); if (!PageAnon(old_page)) { dec_mm_counter(mm, mm_counter_file(old_page)); @@ -194,8 +203,9 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, flush_cache_page(vma, addr, pte_pfn(*pvmw.pte)); ptep_clear_flush_notify(vma, addr, pvmw.pte); - set_pte_at_notify(mm, addr, pvmw.pte, - mk_pte(new_page, vma->vm_page_prot)); + if (new_page) + set_pte_at_notify(mm, addr, pvmw.pte, + mk_pte(new_page, vma->vm_page_prot)); page_remove_rmap(old_page, false); if (!page_mapped(old_page)) @@ -488,6 +498,10 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, ref_ctr_updated = 1; } + ret = 0; + if (!is_register && !PageAnon(old_page)) + goto put_old; + ret = anon_vma_prepare(vma); if (ret) goto put_old; @@ -501,8 +515,30 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, copy_highpage(new_page, old_page); copy_to_page(new_page, vaddr, &opcode, UPROBE_SWBP_INSN_SIZE); + if (!is_register) { + struct page *orig_page; + pgoff_t index; + + VM_BUG_ON_PAGE(!PageAnon(old_page), old_page); + + index = vaddr_to_offset(vma, vaddr & PAGE_MASK) >> PAGE_SHIFT; + orig_page = find_get_page(vma->vm_file->f_inode->i_mapping, + index); + + if (orig_page) { + if (PageUptodate(orig_page) && + pages_identical(new_page, orig_page)) { + /* let go new_page */ + put_page(new_page); + new_page = NULL; + } + put_page(orig_page); + } + } + ret = __replace_page(vma, vaddr, old_page, new_page); - put_page(new_page); + if (new_page) + put_page(new_page); put_old: put_page(old_page); From patchwork Wed Aug 7 23:37:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 11082905 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 86679112C for ; Wed, 7 Aug 2019 23:37:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 77AA828AAE for ; Wed, 7 Aug 2019 23:37:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6BF5B28AB0; Wed, 7 Aug 2019 23:37:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E81BD28AAE for ; Wed, 7 Aug 2019 23:37:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7D5206B000A; Wed, 7 Aug 2019 19:37:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 70E906B000C; Wed, 7 Aug 2019 19:37:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5D8956B000D; Wed, 7 Aug 2019 19:37:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 19C4C6B000A for ; Wed, 7 Aug 2019 19:37:43 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id 191so57754476pfy.20 for ; Wed, 07 Aug 2019 16:37:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=QCXgQ1j+DyNDDAhgp7fIcjUO5uiv3v51QF0EDrfCLxw=; b=VvAQiqqG0Eou9HhkJHGe8VIjXOXrO2B3M/Dl+C2azT+Ai7hIPraWwcBm5AdemB5fI4 k0EM9aecwRA2ozEiMXb/xyQpvNBNi/9WfWlRCwl/cp0Q0VfKe4AoP9VBulAvWS+3gqGL svR7b9ttFBHdh5Xd83uaXxw/FtxeJ9NO9iQgn09Od5DjsWeFO52UO4JGUL7CiIn0IzYu w0KJg6vuTSNl6bkdAF3teRXDIxKoek0znKvYSunynwZZsuZDXsQMHSBN35sfbePJqp9f 2QtszUoJB155uxwJODnqx73Wxged7uHGqfvcufN4MNAGw+i+B5UWlpDvnosQEqty6YFN RCCA== X-Gm-Message-State: APjAAAWmuV9k1SWRPx5gUflbMVes4tFquBmVkOe1Jy74VhojfP4ChhVO 5rRm0FhBwphZC8JnanXFGGlYg+Jbr/LcU7vCWCiKqQQNTic7Lna5IvQ17n9jcmEWRJ49Ou+2PHo Ak4pcyColw+LfwPsTod68Zew4EdE4HS5qIArj7ZspYijvh0vfH6zbWIrHHOhQGHcJfw== X-Received: by 2002:a65:6497:: with SMTP id e23mr9558226pgv.89.1565221062594; Wed, 07 Aug 2019 16:37:42 -0700 (PDT) X-Google-Smtp-Source: APXvYqzWjXpC9IyHn/Yw2iklPXGVtqc6yBDUb77cpKjbSsxVt33eOtqdyQYefXgYp5xjiXuLYjvj X-Received: by 2002:a65:6497:: with SMTP id e23mr9558184pgv.89.1565221061762; Wed, 07 Aug 2019 16:37:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1565221061; cv=none; d=google.com; s=arc-20160816; b=IRflJ+q+sG34tIAwwa9R/ErUMgptzBd7GFLcSMcnzZfjip4rYrDlZKpYLQ0DFtMKzt wCcEpQU40QFC1qBRY9s8l2bHiHS05byeHL4IrnKPSFKCHv7YNgDHUXJewmhqBAjVViK8 X5/4+iunVVSB61iDuYcRPAP3CNBpfHV+1+ozECy96Ena5c7dy2NnYsTJg6W9GzVwYbf0 xd2I+z78P3z6xRc3Csu9QZxLQ1P/+j2whpYs0nHjN+c3H9Jon2dwtiSyEaOWzR1Qtqsp 4gkVsKgFxQXkGuzEmBc1opds/Q7QsyJkwkoq0Hda5OkjAlsHDsM3G/x/mYneXaJzWKE8 jNSQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=QCXgQ1j+DyNDDAhgp7fIcjUO5uiv3v51QF0EDrfCLxw=; b=NeIJNfKzEaCGncI0fBWAwI1C4us9PRq3wKp2U0iGdLZieRVU0puyCXhRbN16FlPk8y I3xPbWJH+a1nPmQr889i8SurY7m2VWj+/OuX+BBgzI1XF5KTTI225OMT5crJx8H78M/y lHPU/KTy65KrKDvNQPKulvoaA189KPv8iVlm+Dd/qsLfMKN6sd+kNpe4kLzxU2+VZTTh bL+FOw9QL7MsYx0NqfVojAjUD51snQaXwduv0iYGDeZ0LMlvMg41rOJqX5XESYXQd81T HQ46EfTicZkb60wBGwqQRxtEig1MbuEM7AiSebwmfUyzU87qsW0pjxcEzPOB5bCtV4T6 dpJA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b="J+m5f/t4"; spf=pass (google.com: domain of prvs=31225916b7=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=31225916b7=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id s13si12415808pgn.123.2019.08.07.16.37.41 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 07 Aug 2019 16:37:41 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=31225916b7=songliubraving@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b="J+m5f/t4"; spf=pass (google.com: domain of prvs=31225916b7=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=31225916b7=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044008.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x77NYIRm003950 for ; Wed, 7 Aug 2019 16:37:41 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=QCXgQ1j+DyNDDAhgp7fIcjUO5uiv3v51QF0EDrfCLxw=; b=J+m5f/t4QpaD1Wrw+w57zoiiogbQqFA+DYikA89WMmTFzK3vpitmrsmCQkdquggXEV1+ +JyroykUIoO1yXXYS9WXD3BLSKUXgk59E+l+WL4OrDcX+ghbjJj2jR8UidZUdeIsXFw2 lz8Z4vl5Kb0fUbgxwxOp5dQNqBY5R8fBTso= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2u87ueg4sc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 07 Aug 2019 16:37:41 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::c) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 7 Aug 2019 16:37:39 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 314A762E2D9E; Wed, 7 Aug 2019 16:37:39 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , , CC: , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v12 3/6] mm, thp: introduce FOLL_SPLIT_PMD Date: Wed, 7 Aug 2019 16:37:26 -0700 Message-ID: <20190807233729.3899352-4-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190807233729.3899352-1-songliubraving@fb.com> References: <20190807233729.3899352-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-08-07_07:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1906280000 definitions=main-1908070208 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patches introduces a new foll_flag: FOLL_SPLIT_PMD. As the name says FOLL_SPLIT_PMD splits huge pmd for given mm_struct, the underlining huge page stays as-is. FOLL_SPLIT_PMD is useful for cases where we need to use regular pages, but would switch back to huge page and huge pmd on. One of such example is uprobe. The following patches use FOLL_SPLIT_PMD in uprobe. Cc: Oleg Nesterov Acked-by: Kirill A. Shutemov Signed-off-by: Song Liu --- include/linux/mm.h | 1 + mm/gup.c | 8 ++++++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index f189176dabed..74db879711eb 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2614,6 +2614,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address, #define FOLL_COW 0x4000 /* internal GUP flag */ #define FOLL_ANON 0x8000 /* don't do file mappings */ #define FOLL_LONGTERM 0x10000 /* mapping lifetime is indefinite: see below */ +#define FOLL_SPLIT_PMD 0x20000 /* split huge pmd before returning */ /* * NOTE on FOLL_LONGTERM: diff --git a/mm/gup.c b/mm/gup.c index 98f13ab37bac..c20afe800b3f 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -399,7 +399,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, spin_unlock(ptl); return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } - if (flags & FOLL_SPLIT) { + if (flags & (FOLL_SPLIT | FOLL_SPLIT_PMD)) { int ret; page = pmd_page(*pmd); if (is_huge_zero_page(page)) { @@ -408,7 +408,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, split_huge_pmd(vma, pmd, address); if (pmd_trans_unstable(pmd)) ret = -EBUSY; - } else { + } else if (flags & FOLL_SPLIT) { if (unlikely(!try_get_page(page))) { spin_unlock(ptl); return ERR_PTR(-ENOMEM); @@ -420,6 +420,10 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, put_page(page); if (pmd_none(*pmd)) return no_page_table(vma, flags); + } else { /* flags & FOLL_SPLIT_PMD */ + spin_unlock(ptl); + split_huge_pmd(vma, pmd, address); + ret = pte_alloc(mm, pmd) ? -ENOMEM : 0; } return ret ? ERR_PTR(ret) : From patchwork Wed Aug 7 23:37:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 11082907 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 244D714DB for ; Wed, 7 Aug 2019 23:37:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 15C5328AAE for ; Wed, 7 Aug 2019 23:37:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0A51A28AB0; Wed, 7 Aug 2019 23:37:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 75CE028AAE for ; Wed, 7 Aug 2019 23:37:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F1E16B000C; Wed, 7 Aug 2019 19:37:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 253D06B000D; Wed, 7 Aug 2019 19:37:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F6CC6B000E; Wed, 7 Aug 2019 19:37:45 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id CD6A36B000C for ; Wed, 7 Aug 2019 19:37:44 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id e20so57808631pfd.3 for ; Wed, 07 Aug 2019 16:37:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=+zWjU0C6LD6HpRrP4wreVSMF6ug2Y3cK0xwApLG5ZA4=; b=OklF3yXLacnk4QIX+75J2C4uAXtxYcEwrYM/KukEC7POu6Nf7K0R/+GVSH2s7BRDGG U3ZLgzfVU5SLAEypqNZfVDUzYFGCPtyvlSevpvub8tO8m4YU5KDKfIK4DjiTCk1UBpMI 09BvNsfA7w9qnb7X1tBYay5eWF82rws8BMhfyG2/zPog7ALaO9NArOoEyRRni+OJRanR XXbjgFdKHwRMbeXho8pFLYZ4JTYdmqmm7oYLAYBj80CuCMKIyuREIJZAPv8fvwQyiH77 0N43U5MYesmMSqPpqt5o2UZwEevOU2AcOBdFOh/SGOzkzA+H5z/x/pRBd9L2vODckXLz jtgg== X-Gm-Message-State: APjAAAUeKO6k20Dyj7lN05lj8W03AvWzM78W0xg8f/4EkLnpk2KMW9Ma cyzqqLEdFFcbZoVZrGtWSAQUegI7nE6IipHoexcPAny6jBxUIjMDKL9dK0KlmliZyPQdMl6bHFT TVjkSyHF7FtqcBrDWbZxUcMd7EH7AsOw4KCd/iddml69D12USc1Q24p5FzQVm4SCLfg== X-Received: by 2002:a17:902:8ec7:: with SMTP id x7mr10475884plo.224.1565221064516; Wed, 07 Aug 2019 16:37:44 -0700 (PDT) X-Google-Smtp-Source: APXvYqxj4ZcOPNzlMWG3DPWPA9MZpquDyNLAYxG8UcvaNMoVDWRRBRHgwPcH420K6cPSizxrDpLO X-Received: by 2002:a17:902:8ec7:: with SMTP id x7mr10475857plo.224.1565221063880; Wed, 07 Aug 2019 16:37:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1565221063; cv=none; d=google.com; s=arc-20160816; b=rN0Z9DjDUIAktIxDK76n9VrbB72dHUA3pV0ODkGky4z7oDW7MfRqbt8vjBaJgSh/2C H/0Dodzaxn7bGcBKnD4duPQYf1fA71PJqvENMSa8JKtGEjpguM3FcFdB/XUcUNZCXxM6 v0Wzv1SpSbGRKHwW03C3ClU2xYVtnyYPo9X7f3nHLLsHqvn+RDtEM0xz7kDhe6ANq4TV QCkS7uJcRWmH3rfi6QS3EKL2PDtQZ71+/MkqoJzPHQ3lqwpcXG720h6QaDoARLm2PC6w /8szDtWYOccA9h/dKm9VJ6tAbx0E3pR4vJjZaZKwDhwVuTr3I14GwPXichsUurKYoiu6 bTOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=+zWjU0C6LD6HpRrP4wreVSMF6ug2Y3cK0xwApLG5ZA4=; b=iWpZaJA2VY2FUdCjRZ9ZW1yp/yoypshkBnVd7BnvNzU5K/hweeSmyiee3QiLG0KgDP qe1YQXl4rcSVwKslBytinUN6uhpLGEtQAH89rpZUou6iEXlPCvE48BMdRgzgMmZ2n3zh QPPfb+Y7cMjKn2k1NK4q0FgOSQHBtXrbdqueq1d8vqHMcgBTbrlqOdmFPHwn0UmgxsgN HyTIITJZitDr04cggTxQVRY9knW8IHi1FcunfB88kZCFXFNE0DqBynRpqtdjmSTJXVco scSpau69O0NjlDtt3K2Fhv/29ZVAZYp1SPRpuxqItH1IWE0k2ySb6RyUD1fNO9lwqHPU mTPQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=fBsqb63h; spf=pass (google.com: domain of prvs=31225916b7=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=31225916b7=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id f59si51365781plf.220.2019.08.07.16.37.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 07 Aug 2019 16:37:43 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=31225916b7=songliubraving@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=fBsqb63h; spf=pass (google.com: domain of prvs=31225916b7=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=31225916b7=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x77NbOgh012893 for ; Wed, 7 Aug 2019 16:37:43 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=+zWjU0C6LD6HpRrP4wreVSMF6ug2Y3cK0xwApLG5ZA4=; b=fBsqb63hUdkjfm3MVSkoshMICce62OK0B0pWD24A0iTxFYJJFooPgx4TMwhbAYMZuikx KPOvmn3+GSl0HOhokd8YHWYB0F5QFPR0VDY6WjjBR2cuANN0w484i9z+8tpXEE9Wyq9X OeDCnBjge64sKhBPNskFtf+TpJ59YZMzWz4= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2u87ueg4jw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 07 Aug 2019 16:37:43 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::f) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 7 Aug 2019 16:37:42 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 42B5562E2D9E; Wed, 7 Aug 2019 16:37:41 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , , CC: , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v12 4/6] uprobe: use FOLL_SPLIT_PMD instead of FOLL_SPLIT Date: Wed, 7 Aug 2019 16:37:27 -0700 Message-ID: <20190807233729.3899352-5-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190807233729.3899352-1-songliubraving@fb.com> References: <20190807233729.3899352-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-08-07_07:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=809 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1906280000 definitions=main-1908070209 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch uses newly added FOLL_SPLIT_PMD in uprobe. This preserves the huge page when the uprobe is enabled. When the uprobe is disabled, newer instances of the same application could still benefit from huge page. For the next step, we will enable khugepaged to regroup the pmd, so that existing instances of the application could also benefit from huge page after the uprobe is disabled. Acked-by: Kirill A. Shutemov Reviewed-by: Srikar Dronamraju Signed-off-by: Song Liu --- kernel/events/uprobes.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 648f47553bff..27b596f14463 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -155,7 +155,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, { struct mm_struct *mm = vma->vm_mm; struct page_vma_mapped_walk pvmw = { - .page = old_page, + .page = compound_head(old_page), .vma = vma, .address = addr, }; @@ -166,8 +166,6 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, addr, addr + PAGE_SIZE); - VM_BUG_ON_PAGE(PageTransHuge(old_page), old_page); - if (new_page) { err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, &memcg, false); @@ -481,7 +479,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, retry: /* Read the page with vaddr into memory */ ret = get_user_pages_remote(NULL, mm, vaddr, 1, - FOLL_FORCE | FOLL_SPLIT, &old_page, &vma, NULL); + FOLL_FORCE | FOLL_SPLIT_PMD, &old_page, &vma, NULL); if (ret <= 0) return ret; From patchwork Wed Aug 7 23:37:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 11082909 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D7AB814DB for ; Wed, 7 Aug 2019 23:37:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C7E6D28AAF for ; Wed, 7 Aug 2019 23:37:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BCA2428AE0; Wed, 7 Aug 2019 23:37:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F0F6E28AAF for ; Wed, 7 Aug 2019 23:37:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6EFA16B000D; Wed, 7 Aug 2019 19:37:46 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 625936B000E; Wed, 7 Aug 2019 19:37:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5141E6B0010; Wed, 7 Aug 2019 19:37:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f198.google.com (mail-yb1-f198.google.com [209.85.219.198]) by kanga.kvack.org (Postfix) with ESMTP id 2E9DB6B000D for ; Wed, 7 Aug 2019 19:37:46 -0400 (EDT) Received: by mail-yb1-f198.google.com with SMTP id z7so14099140ybp.9 for ; Wed, 07 Aug 2019 16:37:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=xgodh64ee0jKQ8kN2eeV7Hq8JQNKQmsQpdhVwpwYA5c=; b=YAiwBRaUqcmZofVGyqw4CrdmAaRfqJyVmKvf5wgTtVDYJVS/y6CWsIlC1a2GP/bo47 MBWi/M1kd9TDPNGotE6voyRxZAC8A93RNEsx6CbeyLY1Re/oADWmqeAZxN80OdJ6g66j 69g3X+jx1jEfF/thx7Pm8aw/5EzoAdqbqmi7VYR7PNlkCdR+ocMgcfMNjv9pFpoYtiaL xRYQecm23ZkyUJBb/vOQKVPmIJaZPb1fNHbkFSE/o9dzdc4XBCC7T1DClaTVYRtiAeqw msjsmVhG1tJ+BSroJnR6IS1/PBOlVqG546D85CNZ0LzrDdQJsszlsQSTpxOl2SPFLvwt /O9g== X-Gm-Message-State: APjAAAVJgbGlgIaONASwRzNxYH+0+VjWY2xwIAvXENOZKyURmJa0M5T5 o7KCHmjNiphe8pQcH2kmkgn7gjn1j7NcPpA37baT7FzFEeQ/I3DIquV1Plg4zELVqwjNhwRB7m9 NZ50s7tBRRwUeTyTsHof324UApZrgzlAOroHiaw0Jo+zGmfqIauRKg/2VDfSXnXZXVg== X-Received: by 2002:a81:11d5:: with SMTP id 204mr8110639ywr.506.1565221065897; Wed, 07 Aug 2019 16:37:45 -0700 (PDT) X-Google-Smtp-Source: APXvYqwXPZXoMtZHN8bKcSX+qU/eOaWgnEkaFgIVMvHOhEYP5wB3+Th1M7xj889ZSe0QrFhIoTEu X-Received: by 2002:a81:11d5:: with SMTP id 204mr8110610ywr.506.1565221065073; Wed, 07 Aug 2019 16:37:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1565221065; cv=none; d=google.com; s=arc-20160816; b=teCUYjacf1cmRoUinkEfZ+Fsh1G+p4A95q936dYl2ZOsXCZrGc6q3CIHO1tbvWGOjQ YZSKAB72+emF3aIFQ8w4029pF6UDjSxgruVt2pd5Wb9i2ptiOlrvJOSt09LsTwviShLu OfF+rLDMbsxMvdozMNX6t3cYDg3xncMVmsgutaIfCdYY4iXTzGho1TOYqG8VjN2TATNe Jzfbe8Nyi9js4Xd6Aj/TNaaNE/eDToywmW+e2zHc3Bn+IE9xczHGQTrr3lxTyMJDY80h O/Hj7YujXLTuJ584SFNEDW6BoOLrtM7pMDGpLwuSiQAJPe4YGOxYSuv4IdAy1kfWu5El zQ7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=xgodh64ee0jKQ8kN2eeV7Hq8JQNKQmsQpdhVwpwYA5c=; b=Kl3HV2+w05aqrNHTz56PRr6cWkzLBY0hOxcXs/pfpxx87piVFJZve3SkA9DO+k8+dz AproWFN9CoG20uNO2sYVO7LBTEP2wABaJkdgL2LeyWgN5K2kvxKXJqaYtJcgw7z8/yo/ xdSCFEYQkDuWmbq5hbAsKD0BRXhF+/ZZomg5zm8nW0HZ/K6mxKQtezsbJoNf4Jpkfst0 BQab5isTsqxtlaw1vCfJbFnTvwGUUJfUf3JR390DOG109BZ+Q0q8pdvUkcpblHFD/d2P 63SNrNu2mvn+9YEGaga0e8scW7QOT4SUnvW3Q92F6zgff22lVEjU8irzkMk29ssLvuAW 1uMQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=gShq6dJB; spf=pass (google.com: domain of prvs=31225916b7=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=31225916b7=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id i15si18441839ybp.494.2019.08.07.16.37.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 07 Aug 2019 16:37:45 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=31225916b7=songliubraving@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=gShq6dJB; spf=pass (google.com: domain of prvs=31225916b7=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=31225916b7=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x77NXS5f019157 for ; Wed, 7 Aug 2019 16:37:44 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=xgodh64ee0jKQ8kN2eeV7Hq8JQNKQmsQpdhVwpwYA5c=; b=gShq6dJB1WdvRg48owb4j8iwga0ViYUYGZlDS/YKVM2L8da1sRrP51dgM9Kveo1Oq4TM QV70SFhVEKcxjhZbTasOZnkX57jkVtlcpaRzuA6FNNWZbW5PteVMCcA67NoYb4ny0h2I VVrGdLo7Qb0+DPgPL1M6tX2YJfOnGny4viY= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2u87ueg40e-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 07 Aug 2019 16:37:44 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::d) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 7 Aug 2019 16:37:44 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 5CC8F62E2D9E; Wed, 7 Aug 2019 16:37:43 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , , CC: , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v12 5/6] khugepaged: enable collapse pmd for pte-mapped THP Date: Wed, 7 Aug 2019 16:37:28 -0700 Message-ID: <20190807233729.3899352-6-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190807233729.3899352-1-songliubraving@fb.com> References: <20190807233729.3899352-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-08-07_07:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=471 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1906280000 definitions=main-1908070208 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP khugepaged needs exclusive mmap_sem to access page table. When it fails to lock mmap_sem, the page will fault in as pte-mapped THP. As the page is already a THP, khugepaged will not handle this pmd again. This patch enables the khugepaged to retry collapse the page table. struct mm_slot (in khugepaged.c) is extended with an array, containing addresses of pte-mapped THPs. We use array here for simplicity. We can easily replace it with more advanced data structures when needed. In khugepaged_scan_mm_slot(), if the mm contains pte-mapped THP, we try to collapse the page table. Since collapse may happen at an later time, some pages may already fault in. collapse_pte_mapped_thp() is added to properly handle these pages. collapse_pte_mapped_thp() also double checks whether all ptes in this pmd are mapping to the same THP. This is necessary because some subpage of the THP may be replaced, for example by uprobe. In such cases, it is not possible to collapse the pmd. Acked-by: Kirill A. Shutemov Cc: Oleg Nesterov Signed-off-by: Song Liu Acked-by: Kirill A. Shutemov Signed-off-by: Song Liu Acked-by: Kirill A. Shutemov Signed-off-by: Song Liu Signed-off-by: Kirill A. Shutemov --- include/linux/khugepaged.h | 12 ++++ mm/khugepaged.c | 140 ++++++++++++++++++++++++++++++++++++- 2 files changed, 151 insertions(+), 1 deletion(-) diff --git a/include/linux/khugepaged.h b/include/linux/khugepaged.h index 082d1d2a5216..bc45ea1efbf7 100644 --- a/include/linux/khugepaged.h +++ b/include/linux/khugepaged.h @@ -15,6 +15,14 @@ extern int __khugepaged_enter(struct mm_struct *mm); extern void __khugepaged_exit(struct mm_struct *mm); extern int khugepaged_enter_vma_merge(struct vm_area_struct *vma, unsigned long vm_flags); +#ifdef CONFIG_SHMEM +extern void collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr); +#else +static inline void collapse_pte_mapped_thp(struct mm_struct *mm, + unsigned long addr) +{ +} +#endif #define khugepaged_enabled() \ (transparent_hugepage_flags & \ @@ -73,6 +81,10 @@ static inline int khugepaged_enter_vma_merge(struct vm_area_struct *vma, { return 0; } +static inline void collapse_pte_mapped_thp(struct mm_struct *mm, + unsigned long addr) +{ +} #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif /* _LINUX_KHUGEPAGED_H */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 40c25ddf29e4..208ea1ce204a 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -77,6 +77,8 @@ static __read_mostly DEFINE_HASHTABLE(mm_slots_hash, MM_SLOTS_HASH_BITS); static struct kmem_cache *mm_slot_cache __read_mostly; +#define MAX_PTE_MAPPED_THP 8 + /** * struct mm_slot - hash lookup from mm to mm_slot * @hash: hash collision list @@ -87,6 +89,10 @@ struct mm_slot { struct hlist_node hash; struct list_head mm_node; struct mm_struct *mm; + + /* pte-mapped THP in this mm */ + int nr_pte_mapped_thp; + unsigned long pte_mapped_thp[MAX_PTE_MAPPED_THP]; }; /** @@ -1254,6 +1260,131 @@ static void collect_mm_slot(struct mm_slot *mm_slot) } #if defined(CONFIG_SHMEM) && defined(CONFIG_TRANSPARENT_HUGE_PAGECACHE) +/* + * Notify khugepaged that given addr of the mm is pte-mapped THP. Then + * khugepaged should try to collapse the page table. + */ +static int khugepaged_add_pte_mapped_thp(struct mm_struct *mm, + unsigned long addr) +{ + struct mm_slot *mm_slot; + + VM_BUG_ON(addr & ~HPAGE_PMD_MASK); + + spin_lock(&khugepaged_mm_lock); + mm_slot = get_mm_slot(mm); + if (likely(mm_slot && mm_slot->nr_pte_mapped_thp < MAX_PTE_MAPPED_THP)) + mm_slot->pte_mapped_thp[mm_slot->nr_pte_mapped_thp++] = addr; + spin_unlock(&khugepaged_mm_lock); + return 0; +} + +/** + * Try to collapse a pte-mapped THP for mm at address haddr. + * + * This function checks whether all the PTEs in the PMD are pointing to the + * right THP. If so, retract the page table so the THP can refault in with + * as pmd-mapped. + */ +void collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr) +{ + unsigned long haddr = addr & HPAGE_PMD_MASK; + struct vm_area_struct *vma = find_vma(mm, haddr); + struct page *hpage = NULL; + pmd_t *pmd, _pmd; + spinlock_t *ptl; + int count = 0; + int i; + + if (!vma || !vma->vm_file || + vma->vm_start > haddr || vma->vm_end < haddr + HPAGE_PMD_SIZE) + return; + + /* + * This vm_flags may not have VM_HUGEPAGE if the page was not + * collapsed by this mm. But we can still collapse if the page is + * the valid THP. Add extra VM_HUGEPAGE so hugepage_vma_check() + * will not fail the vma for missing VM_HUGEPAGE + */ + if (!hugepage_vma_check(vma, vma->vm_flags | VM_HUGEPAGE)) + return; + + pmd = mm_find_pmd(mm, haddr); + if (!pmd) + return; + + /* step 1: check all mapped PTEs are to the right huge page */ + for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { + pte_t *pte = pte_offset_map(pmd, addr); + struct page *page; + + if (pte_none(*pte)) + continue; + + page = vm_normal_page(vma, addr, *pte); + + if (!page || !PageCompound(page)) + return; + + if (!hpage) { + hpage = compound_head(page); + if (hpage->mapping != vma->vm_file->f_mapping) + return; + } + + if (hpage + i != page) + return; + count++; + } + + /* step 2: adjust rmap */ + for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { + pte_t *pte = pte_offset_map(pmd, addr); + struct page *page; + + if (pte_none(*pte)) + continue; + page = vm_normal_page(vma, addr, *pte); + page_remove_rmap(page, false); + } + + /* step 3: set proper refcount and mm_counters. */ + if (hpage) { + page_ref_sub(hpage, count); + add_mm_counter(vma->vm_mm, mm_counter_file(hpage), -count); + } + + /* step 4: collapse pmd */ + ptl = pmd_lock(vma->vm_mm, pmd); + _pmd = pmdp_collapse_flush(vma, addr, pmd); + spin_unlock(ptl); + mm_dec_nr_ptes(mm); + pte_free(mm, pmd_pgtable(_pmd)); +} + +static int khugepaged_collapse_pte_mapped_thps(struct mm_slot *mm_slot) +{ + struct mm_struct *mm = mm_slot->mm; + int i; + + if (likely(mm_slot->nr_pte_mapped_thp == 0)) + return 0; + + if (!down_write_trylock(&mm->mmap_sem)) + return -EBUSY; + + if (unlikely(khugepaged_test_exit(mm))) + goto out; + + for (i = 0; i < mm_slot->nr_pte_mapped_thp; i++) + collapse_pte_mapped_thp(mm, mm_slot->pte_mapped_thp[i]); + +out: + mm_slot->nr_pte_mapped_thp = 0; + up_write(&mm->mmap_sem); + return 0; +} + static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) { struct vm_area_struct *vma; @@ -1287,7 +1418,8 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) up_write(&vma->vm_mm->mmap_sem); mm_dec_nr_ptes(vma->vm_mm); pte_free(vma->vm_mm, pmd_pgtable(_pmd)); - } + } else + khugepaged_add_pte_mapped_thp(vma->vm_mm, addr); } i_mmap_unlock_write(mapping); } @@ -1709,6 +1841,11 @@ static void khugepaged_scan_file(struct mm_struct *mm, { BUILD_BUG(); } + +static int khugepaged_collapse_pte_mapped_thps(struct mm_slot *mm_slot) +{ + return 0; +} #endif static unsigned int khugepaged_scan_mm_slot(unsigned int pages, @@ -1733,6 +1870,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, khugepaged_scan.mm_slot = mm_slot; } spin_unlock(&khugepaged_mm_lock); + khugepaged_collapse_pte_mapped_thps(mm_slot); mm = mm_slot->mm; /* From patchwork Wed Aug 7 23:37:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 11082911 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 080C7112C for ; Wed, 7 Aug 2019 23:37:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EFED828AAF for ; Wed, 7 Aug 2019 23:37:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E486028AD6; Wed, 7 Aug 2019 23:37:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 70A6D28AAF for ; Wed, 7 Aug 2019 23:37:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 784166B000E; Wed, 7 Aug 2019 19:37:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 739856B0010; Wed, 7 Aug 2019 19:37:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 587D06B0266; Wed, 7 Aug 2019 19:37:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f199.google.com (mail-yb1-f199.google.com [209.85.219.199]) by kanga.kvack.org (Postfix) with ESMTP id 395956B000E for ; Wed, 7 Aug 2019 19:37:49 -0400 (EDT) Received: by mail-yb1-f199.google.com with SMTP id r206so5243449ybc.6 for ; Wed, 07 Aug 2019 16:37:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=NLo+kUCCETQ4ngAnRD6GoEVLB9ZFwANLd+99Aj/UcOA=; b=den/rCogEjvyLNHE3Uv6s+dv2Bfi/sMOpSlwEb5CgT/MmiZJoSuW0IIoO4LH0o2Q4s TTBMkX/YU3RcHYEWHGBgPLnhHqrCD1EgsCdnpBjy8eDoFVoOUje4yGhjKN9DlkFojIRi HOAeFZZrxvBSJIRTHVoL214MaM9Q54rt39rgkoK0YfQRZLVqcksP0WuzKN9lJyqTIYkt 3A0NzUd43aufbWTrIwNd9cSWJgDnfL3CMEuWOnz11sq6Rfh0Lr5mEQpohpAxNCUzPoDp aDiVR3QIB3EVkMNlbNJ9o32byJN7WvyPTamA4acvUaECEdnswjjLjZ+4tm4rN8OoHNkc jbcg== X-Gm-Message-State: APjAAAUH2w8TekRyP4heDuQKYVoWKTFtuyjAcjRINwmBHV0XtsSaB7XA d13ze3J7c8y8Lyk/zFy1/qrgmzSqkYU4SEmkzhm8Gwh6OaMIk0t6tH18TRcdg2UD/uNDjVL0xG5 1mZVVuI3IRQcX0Ii8mGm/KuI1pryK7B5AAChKmNgELW+UM8eQ6Ylz4D9EOSp6RUGxAQ== X-Received: by 2002:a81:1188:: with SMTP id 130mr994299ywr.132.1565221068848; Wed, 07 Aug 2019 16:37:48 -0700 (PDT) X-Google-Smtp-Source: APXvYqxSWDzX8LgKSI8LyeT39eXTPpwIYOEpy4UY0dHiq6twcuIg4U0ltR+XEcf5O/87suehoSoA X-Received: by 2002:a81:1188:: with SMTP id 130mr994282ywr.132.1565221068324; Wed, 07 Aug 2019 16:37:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1565221068; cv=none; d=google.com; s=arc-20160816; b=dQyTLxkvvo3bZ88gaH1nRs2i+3XcRmbdneyrGqw5ki966zjl0uR08N0+HhW9KSyjlN NcHl/jhdpkyA3EuKPDoXQsRoDGvozfxkrrtj/9yKarhlcomSJ10kGxSeGkg7/cOUtdfX FIrsA7iM/Vp5BFk7cmU4qwf7zdM1OWMsk9Qote/aNz0/t8x6ToANzPTaFypol/u7kmy4 vFd5CAoFfsNZ7ZcXCM+DrcKq8lCW6yzVdio6tUnFJu/8f5E1RPoY74uhFQK5f5Jw9IJV 7Ax3uPvZmH9L6Q8vO8xl2HJ1jl7ptw+p/XRC6kiKjAzpVj9YRfiJTgKvbsA2Sp+vXKo6 K9Mw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=NLo+kUCCETQ4ngAnRD6GoEVLB9ZFwANLd+99Aj/UcOA=; b=X1PK70+lW9vBX6hIBODb/zK4zvv3f1dwA38UH1q5PA6nWw9yhu1oVQqj2rVCIOxsBC ZuX63szhr5erf941GzkSqErtUget0q8lYzIAodSdS1ZYMpv5LBH8RtRlBl9sZebiXC5Q SP7PYn7IlvdJFpIMyv541acIuuzI8DLqYxD/2SS8L9AgAaqNh0hP/EhvQsBsbHHYtigj BRb43J0OAYV1fER7qG1+GmhVgpTiklpaYmnCvrk05aJh9mMus+MBPaxVG4Y0AQLmYr9F rwYI+Ohd0br274uhUi3oYcvPuTRQhY3xCO5eiRzGOQCXdLwQO7TrXxIjtdE6VTUgjrtn 9Cew== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=AiQRTnJE; spf=pass (google.com: domain of prvs=31225916b7=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=31225916b7=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id d67si29963435ywf.407.2019.08.07.16.37.48 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 07 Aug 2019 16:37:48 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=31225916b7=songliubraving@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=AiQRTnJE; spf=pass (google.com: domain of prvs=31225916b7=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=31225916b7=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.16.0.27/8.16.0.27) with SMTP id x77NXnwX003599 for ; Wed, 7 Aug 2019 16:37:48 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=NLo+kUCCETQ4ngAnRD6GoEVLB9ZFwANLd+99Aj/UcOA=; b=AiQRTnJEbjw7LPcoxeULFKcunsOEME5Pa3F/2vM5Z9o8A3yN9qTXLKIB/3rnvNWQ8nZu sOMS9zfVMpLXOYDv9k4kRFAlFmBMPneHXCSFPkWPqqVTKwhMRfVADDQPtWwgXI+FDvIZ 8gqnyjsfz/2Qj4ewiFp+jTqSEDDmofuEZ70= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by m0089730.ppops.net with ESMTP id 2u87uf8414-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 07 Aug 2019 16:37:48 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::130) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Wed, 7 Aug 2019 16:37:46 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 8712062E2D9E; Wed, 7 Aug 2019 16:37:45 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , , CC: , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v12 6/6] uprobe: collapse THP pmd after removing all uprobes Date: Wed, 7 Aug 2019 16:37:29 -0700 Message-ID: <20190807233729.3899352-7-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190807233729.3899352-1-songliubraving@fb.com> References: <20190807233729.3899352-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-08-07_07:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=424 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1906280000 definitions=main-1908070208 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP After all uprobes are removed from the huge page (with PTE pgtable), it is possible to collapse the pmd and benefit from THP again. This patch does the collapse by calling collapse_pte_mapped_thp(). Acked-by: Kirill A. Shutemov Reported-by: kbuild test robot Signed-off-by: Song Liu --- kernel/events/uprobes.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 27b596f14463..94d38a39d72e 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -26,6 +26,7 @@ #include #include #include +#include #include @@ -472,6 +473,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, struct page *old_page, *new_page; struct vm_area_struct *vma; int ret, is_register, ref_ctr_updated = 0; + bool orig_page_huge = false; is_register = is_swbp_insn(&opcode); uprobe = container_of(auprobe, struct uprobe, arch); @@ -529,6 +531,9 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, /* let go new_page */ put_page(new_page); new_page = NULL; + + if (PageCompound(orig_page)) + orig_page_huge = true; } put_page(orig_page); } @@ -547,6 +552,10 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, if (ret && is_register && ref_ctr_updated) update_ref_ctr(uprobe, mm, -1); + /* try collapse pmd for compound page */ + if (!ret && orig_page_huge) + collapse_pte_mapped_thp(mm, vaddr); + return ret; }