From patchwork Sun Jun 23 05:48:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 11011583 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2F3F014B6 for ; Sun, 23 Jun 2019 05:48:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1ECAC28B2A for ; Sun, 23 Jun 2019 05:48:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1270928B2C; Sun, 23 Jun 2019 05:48:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9429E28B2A for ; Sun, 23 Jun 2019 05:48:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5AAE76B0008; Sun, 23 Jun 2019 01:48:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 46CB48E0007; Sun, 23 Jun 2019 01:48:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 30CDB8E0006; Sun, 23 Jun 2019 01:48:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id E7B9E6B0008 for ; Sun, 23 Jun 2019 01:48:47 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id 14so6870005pgo.14 for ; Sat, 22 Jun 2019 22:48:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=pQnGGve/Vnh2HyxOCOdI1CzGbYYB1SvJlXif4X9k+UU=; b=qtRN1mmH79ABi2FyJBgqg5NeeFv8sb36n30i4zAYEaDr3528/vt06RvXecRrea+Czn BRJ3i8Es/xO00Dmz0EP0nrXGMyCNocb9E7lu1P3h6uCtYXi2uROTLdhbreNNsKMX1PTW NzniD3EwP4EwlqnwY5fnDX4E+GH34QWH7sNs3PKi+iEcbcKqo0YnS+MVu0KhHO0UIdaG qQLZxi+2iHmW+fXebX8tiT087SB52Uf4F/ueO8gr3pREb8yFHkBiw+ErYPFR+S9TaPyl 6pzBWFp8S3bi6t1B6YfPcB/aBk4IRVEoGXQwbiVCu0Tb9wPMJ2JBNcGp4Yvkge8neTCQ iVbQ== X-Gm-Message-State: APjAAAWTszeUoWH+YG3PsjJmt3tb+rYWCv9LG0wiVvR3jFGDyf9Hiiyj dNbUzKzY7TRArsXuG6p+llTdpa+7xqPjDScxQ+NbUv+A2/fW8ewZ0F2kkeUX6yw39zrY46EyCAK qvG1IdGwmrJ/TJl641S68ezuSo6s2ot5Ardr5yNfE3a7Zdsk11VzeawfJWaEqAY7FWQ== X-Received: by 2002:a17:90a:e397:: with SMTP id b23mr16950861pjz.117.1561268927570; Sat, 22 Jun 2019 22:48:47 -0700 (PDT) X-Google-Smtp-Source: APXvYqyn32nZcEB7LvN4ab9Ud0nrcUO00tauphRGMPJRUouCb8OmWZVCC9sGFzGfXJdY1Y+LhehH X-Received: by 2002:a17:90a:e397:: with SMTP id b23mr16950821pjz.117.1561268926885; Sat, 22 Jun 2019 22:48:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561268926; cv=none; d=google.com; s=arc-20160816; b=hxuyzN8P5lbFweApAx5RQeYvDffOIrRHN5oE3d4U6MI8PRq1pES6fdgoFRpzgi+wiq EWlE7BY7poGd5tGTBMBipGRJ6JwrBiqKoVGokn/Ez8ajkt5BX5HXXrQQnMas5I7dcFWd uwWQPxNIWHwjpyZjaGiOdGsRNnGYFC3kocDqlTxznNcRXoZTXDTMer7FH7EjmNRj2eYQ jYRaMh1V9PrNzDEx/ghk4CBJ7w54Q7tDjCKJytqbEGP3Fn7IAT+qTMXTvh5gsCXamGxf KD/akt0FGuOwjCs0oTVyo9XGElT7hLHPH6h0+RIHGXih+Rc8I9DjPmDL/y0aFsuhb2+4 hzgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=pQnGGve/Vnh2HyxOCOdI1CzGbYYB1SvJlXif4X9k+UU=; b=YeDcWWDee4zi6WEfN7osEfeSLMvbvvR1BmvQnazv7r+VThzyX8pHJXaUywi7qdOU1x ckw59UPzPevhxWX27jsf9IibzAfElQpSfklAcALFxtNkm1SLdSpqv9X/u1xQHkK/fG/g MbX53xaCqK5e1cGjzWLiyzq0MK+Gp8WQMidIe8t0La0Z8PqkZFcnx5rFuJCQHbktO/fU SwH9u5+2TuWREkFjUEEWwRSIMbXP9KyT59kYNHQ3iNg/ykIlk6tqsArkHtMCTtg9GUr4 55o5gqz8QwkOhmt91+UDNpoQFU0vVewJyf3g5C2DVQ+5DFxWLxKskFXCNR1YmGxYegOb ENuA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=ShkJ0gVd; spf=pass (google.com: domain of prvs=1077171f80=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=1077171f80=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id e21si6974817pgh.571.2019.06.22.22.48.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 22:48:46 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=1077171f80=songliubraving@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=ShkJ0gVd; spf=pass (google.com: domain of prvs=1077171f80=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=1077171f80=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5N5j2fI008151 for ; Sat, 22 Jun 2019 22:48:46 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=pQnGGve/Vnh2HyxOCOdI1CzGbYYB1SvJlXif4X9k+UU=; b=ShkJ0gVdzE4CAaYKqcPhWAwugyakkRFNJ5zY9jje2zQZ2BWgEtwAANNbymtMaifRBTV6 LrLlUA+hWOc8UUIQ5a+T5WRnDRgNHuyLN5FuE2d5CsexniYeGzPNaQyNalYJbQ/Fcq20 qMIWDslH9lg3i+DpmWYbnRpovxOiQjNNRl4= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2t9fn2agbk-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Sat, 22 Jun 2019 22:48:46 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::c) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Sat, 22 Jun 2019 22:48:45 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id E4FD862E2CFB; Sat, 22 Jun 2019 22:48:44 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v6 1/6] mm: move memcmp_pages() and pages_identical() Date: Sat, 22 Jun 2019 22:48:24 -0700 Message-ID: <20190623054829.4018117-2-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190623054829.4018117-1-songliubraving@fb.com> References: <20190623054829.4018117-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-23_04:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=948 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906230050 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch moves memcmp_pages() to mm/util.c and pages_identical() to mm.h, so that we can use them in other files. Signed-off-by: Song Liu --- include/linux/mm.h | 7 +++++++ mm/ksm.c | 18 ------------------ mm/util.c | 13 +++++++++++++ 3 files changed, 20 insertions(+), 18 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index dd0b5f4e1e45..0ab8c7d84cd0 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2891,5 +2891,12 @@ void __init setup_nr_node_ids(void); static inline void setup_nr_node_ids(void) {} #endif +extern int memcmp_pages(struct page *page1, struct page *page2); + +static inline int pages_identical(struct page *page1, struct page *page2) +{ + return !memcmp_pages(page1, page2); +} + #endif /* __KERNEL__ */ #endif /* _LINUX_MM_H */ diff --git a/mm/ksm.c b/mm/ksm.c index 81c20ed57bf6..6f153f976c4c 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1030,24 +1030,6 @@ static u32 calc_checksum(struct page *page) return checksum; } -static int memcmp_pages(struct page *page1, struct page *page2) -{ - char *addr1, *addr2; - int ret; - - addr1 = kmap_atomic(page1); - addr2 = kmap_atomic(page2); - ret = memcmp(addr1, addr2, PAGE_SIZE); - kunmap_atomic(addr2); - kunmap_atomic(addr1); - return ret; -} - -static inline int pages_identical(struct page *page1, struct page *page2) -{ - return !memcmp_pages(page1, page2); -} - static int write_protect_page(struct vm_area_struct *vma, struct page *page, pte_t *orig_pte) { diff --git a/mm/util.c b/mm/util.c index 9834c4ab7d8e..750e586d50bc 100644 --- a/mm/util.c +++ b/mm/util.c @@ -755,3 +755,16 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen) out: return res; } + +int memcmp_pages(struct page *page1, struct page *page2) +{ + char *addr1, *addr2; + int ret; + + addr1 = kmap_atomic(page1); + addr2 = kmap_atomic(page2); + ret = memcmp(addr1, addr2, PAGE_SIZE); + kunmap_atomic(addr2); + kunmap_atomic(addr1); + return ret; +} From patchwork Sun Jun 23 05:48:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 11011585 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CC07D1398 for ; Sun, 23 Jun 2019 05:48:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BBC7628B2A for ; Sun, 23 Jun 2019 05:48:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AFCAC28B2C; Sun, 23 Jun 2019 05:48:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 745E728B2A for ; Sun, 23 Jun 2019 05:48:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A5F366B000A; Sun, 23 Jun 2019 01:48:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A10568E0007; Sun, 23 Jun 2019 01:48:50 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8897E8E0006; Sun, 23 Jun 2019 01:48:50 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f72.google.com (mail-yw1-f72.google.com [209.85.161.72]) by kanga.kvack.org (Postfix) with ESMTP id 62C276B000A for ; Sun, 23 Jun 2019 01:48:50 -0400 (EDT) Received: by mail-yw1-f72.google.com with SMTP id y205so10985692ywy.19 for ; Sat, 22 Jun 2019 22:48:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=qKr0k6k4xZHZmykxLb47XrFO0p8RI9jBEYkCbTeGgSI=; b=AtQj8VLyTrqRYrPUwfju4QOAPsUo2N5QiDA9TqjFheknvrhpcp/Liygp5UYhuFvf+1 4bO2BNd0CbaBCsdAXFAr01OcOi532MMvHo7Y9ujzVSwXWWD2lQ2hzLbIP6/QTPSIpw86 b6XIq5TXxki1hBVtQygHUAZY6MaifWKyPZ5jOa5TerfbISsSZ0RUzJLGsxPj6AmOoMUp XUlSAxnCN6W3M+tTvTn5CqHXqbxYnpuVYA0ipkVSZG39mpAWeohLSvMzNFeHWQCScI/x aSI9xzwEu59Zza6InST8ESw6jJ53uhOPAQ4sDKYXUuKU4cGYuCDb9bsfW3BtXpZpPbEG HSyQ== X-Gm-Message-State: APjAAAXPpSMHketC/lGMpS2ilyuSna0xO3SiLgW/J9fWjrw/AvuJ2epC Dd1fpj4mU2jF0VrlKjkW35lOgYbK3r89hID2XMq7OG9chWWqQV2TZDIPbOeUngbtIECAvxvB3qi rJV8Fi/r+R81gOOyJMgTRwMvdhglgfKuwLXXLu+TpEuOVU82QMneq1N+YURaK8dSBCA== X-Received: by 2002:a81:3903:: with SMTP id g3mr79560339ywa.304.1561268930136; Sat, 22 Jun 2019 22:48:50 -0700 (PDT) X-Google-Smtp-Source: APXvYqzuYeWpFoga+qJL5XgDNrNvEPOBRxWOn1NdBVvbxhzX8Xk/OXuKstISjRfioD929GXsO6jk X-Received: by 2002:a81:3903:: with SMTP id g3mr79560334ywa.304.1561268929622; Sat, 22 Jun 2019 22:48:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561268929; cv=none; d=google.com; s=arc-20160816; b=jm9c4F8cQvPq6Bfe/2+B049Ue3Ix/cqfeoIwGdb6kbHGomFQmM9hiKnpBAywBzbZ9b DNVvsF3UkpmcvyS4Trir5200QKQVpW0eKuZKx+d55lnctKqJ2+1wx2bA9FjqXsVrDGrP sz0zSup+mEihGOCvdatlaQgA0OiBgKLUSAR3t1zeGT801g9PE6fKlB2Ke6SJwVPoY/sM 9RxhOxLft+oZVJryDCNspM8OX0k5DQ8qqTXlEX7Sm9x3tkDro5RVk6xkzybFNeFuB4KH 81t8bxZqko2GyOTENv/bc3DN4lwRCMrsu0u52Zj0CEJvLvhuNAxAGJMIXkvhtkZVlcop CsjA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=qKr0k6k4xZHZmykxLb47XrFO0p8RI9jBEYkCbTeGgSI=; b=vtSEnBKJkqsEElXhVXPO0jdOQyNNJwvXyg8fDY0DnL+BnDzugPanbrhc2YgqzIIj6M R4Sjy920CAb3f5TVFD76HVRJaKMWR9swA0x37YRShodvJb5ItLom44lpaXltCnL1g/zc XRdZ4MzHHL0yo01cMVFhr8HCkzdIGZondGZ/8okPUf042uS6+5lU875ZwDtC2LHHod5r Fhuxrjp6P/ZmMXQ+ZfH/QnnHxDKu3RrzRsyDhafBiDI/7+syOeKp1pfPaHfsxorKlq3s ZXCMbDd1a1TAAUCUhgz+wDKffuFQSKQWbAMltD9oZZ3Umtiq711MjPZ9l85s+rvelpZa CBFg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b="Dt/12qf8"; spf=pass (google.com: domain of prvs=1077171f80=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1077171f80=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id e1si150681yba.281.2019.06.22.22.48.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 22:48:49 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=1077171f80=songliubraving@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b="Dt/12qf8"; spf=pass (google.com: domain of prvs=1077171f80=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1077171f80=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5N5mlt0006687 for ; Sat, 22 Jun 2019 22:48:49 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=qKr0k6k4xZHZmykxLb47XrFO0p8RI9jBEYkCbTeGgSI=; b=Dt/12qf8MBN6wdcG5X0gb23qTJOL1XuW1pi+Digq5HYIcaVku4TAWJ1tMk2ODFLK916b WKBtkxy9Yy77GXKfnDRsXgCMvqjVdAxdOC2ti12rz7uv1AfgswRH4WrHQ+/NXd1u2nO4 uYo2KFAIwHwYPDABcb1zO+3b2TIjg3+LKu4= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2t9j2ca3jp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Sat, 22 Jun 2019 22:48:49 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::d) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Sat, 22 Jun 2019 22:48:48 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 3C08062E2CFB; Sat, 22 Jun 2019 22:48:47 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v6 2/6] uprobe: use original page when all uprobes are removed Date: Sat, 22 Jun 2019 22:48:25 -0700 Message-ID: <20190623054829.4018117-3-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190623054829.4018117-1-songliubraving@fb.com> References: <20190623054829.4018117-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-23_04:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=990 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906230051 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently, uprobe swaps the target page with a anonymous page in both install_breakpoint() and remove_breakpoint(). When all uprobes on a page are removed, the given mm is still using an anonymous page (not the original page). This patch allows uprobe to use original page when possible (all uprobes on the page are already removed). Signed-off-by: Song Liu --- kernel/events/uprobes.c | 45 +++++++++++++++++++++++++++++++++-------- 1 file changed, 37 insertions(+), 8 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 78f61bfc6b79..f7c61a1ef720 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -160,16 +160,19 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, int err; struct mmu_notifier_range range; struct mem_cgroup *memcg; + bool orig = new_page->mapping != NULL; /* new_page == orig_page */ mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, addr, addr + PAGE_SIZE); VM_BUG_ON_PAGE(PageTransHuge(old_page), old_page); - err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, &memcg, - false); - if (err) - return err; + if (!orig) { + err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, + &memcg, false); + if (err) + return err; + } /* For try_to_free_swap() and munlock_vma_page() below */ lock_page(old_page); @@ -177,15 +180,24 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, mmu_notifier_invalidate_range_start(&range); err = -EAGAIN; if (!page_vma_mapped_walk(&pvmw)) { - mem_cgroup_cancel_charge(new_page, memcg, false); + if (!orig) + mem_cgroup_cancel_charge(new_page, memcg, false); goto unlock; } VM_BUG_ON_PAGE(addr != pvmw.address, old_page); get_page(new_page); - page_add_new_anon_rmap(new_page, vma, addr, false); - mem_cgroup_commit_charge(new_page, memcg, false, false); - lru_cache_add_active_or_unevictable(new_page, vma); + if (orig) { + lock_page(new_page); /* for page_add_file_rmap() */ + page_add_file_rmap(new_page, false); + unlock_page(new_page); + inc_mm_counter(mm, mm_counter_file(new_page)); + dec_mm_counter(mm, MM_ANONPAGES); + } else { + page_add_new_anon_rmap(new_page, vma, addr, false); + mem_cgroup_commit_charge(new_page, memcg, false, false); + lru_cache_add_active_or_unevictable(new_page, vma); + } if (!PageAnon(old_page)) { dec_mm_counter(mm, mm_counter_file(old_page)); @@ -501,6 +513,23 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, copy_highpage(new_page, old_page); copy_to_page(new_page, vaddr, &opcode, UPROBE_SWBP_INSN_SIZE); + if (!is_register) { + struct page *orig_page; + pgoff_t index; + + index = vaddr_to_offset(vma, vaddr & PAGE_MASK) >> PAGE_SHIFT; + orig_page = find_get_page(vma->vm_file->f_inode->i_mapping, + index); + + if (orig_page) { + if (pages_identical(new_page, orig_page)) { + put_page(new_page); + new_page = orig_page; + } else + put_page(orig_page); + } + } + ret = __replace_page(vma, vaddr, old_page, new_page); put_page(new_page); put_old: From patchwork Sun Jun 23 05:48:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 11011587 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9359C1398 for ; Sun, 23 Jun 2019 05:48:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8263828B2A for ; Sun, 23 Jun 2019 05:48:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 76B4F28B2C; Sun, 23 Jun 2019 05:48:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3D30D28B2A for ; Sun, 23 Jun 2019 05:48:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 46F906B000C; Sun, 23 Jun 2019 01:48:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3AA3F8E0007; Sun, 23 Jun 2019 01:48:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 246808E0006; Sun, 23 Jun 2019 01:48:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f71.google.com (mail-yw1-f71.google.com [209.85.161.71]) by kanga.kvack.org (Postfix) with ESMTP id F349D6B000C for ; Sun, 23 Jun 2019 01:48:53 -0400 (EDT) Received: by mail-yw1-f71.google.com with SMTP id f11so11084119ywc.4 for ; Sat, 22 Jun 2019 22:48:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=VaTYZS3FdWj202Z1cIYQJggrJMEpd/wr+92gO5GUCcA=; b=SrSsmYuVfqtHXhDqA7bnXrlI1hwCpBL/DkD4xyXCsC8FXyYN4ctKctMo0YgPz+8ELq IPN08S6vlVmRsBCl55mHxTVGm3OA4x3an5Fx6DCj392Vhfw5yxU2nchWu1Vq1SGMLqwl OQWU9x9F71fBkDhoTcpcFkh+ipbIOG23HldxMPNyQbxBjYZTSU3OWPXxAnFxi7a8E0wc KMJQxdLvuKuRJBZFMF41LCTeubr5JnME6WuBFFyQMkHqGcIIt+258WjyVsn8DkHEzmib C7eHyIdo1EHsmofYUNwUJiEpEcxg0Da5KNFFIPnMzZC0zAGrb5PCH7SLCbCCmP/HtHvw llfQ== X-Gm-Message-State: APjAAAUtaoJaEyUjiBiHwY0VUulvE4VI/Jjm48PeUdfW+AP0ERBRnkAe QPoWdwVfwfxS4iA1cNhHwcDGgcUqnAQjN04nz9ONG3PB3HrtuR6bPmvIpIyLSa8haQHyQxyqHzT Jl4rdZpcOZo6ENs6bGqnjWerUxEqNNnQ3ruqwZftu14q0RYboXhYOp7ZCReWLi2QGTQ== X-Received: by 2002:a81:7815:: with SMTP id t21mr79384569ywc.222.1561268933735; Sat, 22 Jun 2019 22:48:53 -0700 (PDT) X-Google-Smtp-Source: APXvYqyMx0+hN0cCHLmvGqw44cHAwAl6dy8tCEeiI7/+xB+lNrXK9rmx20Dd4L+nRIZFdy3vJuPY X-Received: by 2002:a81:7815:: with SMTP id t21mr79384555ywc.222.1561268933094; Sat, 22 Jun 2019 22:48:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561268933; cv=none; d=google.com; s=arc-20160816; b=EnrLnMUiUJv/c55wOf85mglsycbXFiZpr0i6SWmyHVWfRgTs5MXuSoKLDgoPzo9gQY iiS+j20zuKWLFei21JLQyJTort05SCAcnZOgBpEWaVcy3sZlrSeGkQVJ/1I1S3/PA+ii wYo29bOTHUVXjgCXEPJClq8G+vkg1aiaDuqm3mdK4DAYlAB2o89hsujXiZxH9Ht+zabF yovVrJ6P/aQ9W9gVaZQrZSjU42uwDPrfTQso92xBpR+aUMetDCtufpa287HX+kLkb+m6 aTaboBib2g9ddRX4huCJvJ4VQYvK3UEtEtJHcIsfJNLJBIHrSGQDP94oBlmPMqM4hPmn HkKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=VaTYZS3FdWj202Z1cIYQJggrJMEpd/wr+92gO5GUCcA=; b=aDjCjuQyD9cauZ2WWgoGGUKmo/Kv75AEEFbryIDLJgV/ZRKaU+6o+l2w46jSgadwiZ Hnd6NmPePKMYqWRM8Qgq9dYCJw5OUlU5ceZdJdeFkOB0B3jVM3/giEeMcHl8Xu5FwRj3 H/Y7yiw8q1yrraGKiTFJ3MJONwtpVt4wh5CUJyWCMXaQRZLowCiBWCyiEpwmtoiqThKQ Dnwnxe/6f2o7twZOv34FLRxmUBXzvXCtl4krxmKAqopit5rn0VwMNsTQgpU03bZSLbKz y0el/hi1uGechkQNVpcJx/bHGUln8aGztCwr+JUMFP48EBYlyrwYZmQH63IVmJ9qdua+ qUbw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=R51XPFQ2; spf=pass (google.com: domain of prvs=1077171f80=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1077171f80=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id g9si1490346ywc.199.2019.06.22.22.48.53 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 22:48:53 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=1077171f80=songliubraving@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=R51XPFQ2; spf=pass (google.com: domain of prvs=1077171f80=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1077171f80=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5N5mlt4006687 for ; Sat, 22 Jun 2019 22:48:52 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=VaTYZS3FdWj202Z1cIYQJggrJMEpd/wr+92gO5GUCcA=; b=R51XPFQ2x7NrBKNEtYxeij9Ig8VOtiudDe77XRijTVwui2ISAx5usXUxkWd/sMwCRqp6 32BInFccXA9rl6yxYakxPBV5Ws2p9/HxpYbbnui7lNTYWC3PcEv/YH7NffZVOpnGntZs 33ejP5P3GkpkXvZogeJesLNoJDiLG1fxb34= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2t9j2ca3jy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Sat, 22 Jun 2019 22:48:52 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::e) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Sat, 22 Jun 2019 22:48:52 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 2465562E2CFB; Sat, 22 Jun 2019 22:48:50 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v6 3/6] mm, thp: introduce FOLL_SPLIT_PMD Date: Sat, 22 Jun 2019 22:48:26 -0700 Message-ID: <20190623054829.4018117-4-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190623054829.4018117-1-songliubraving@fb.com> References: <20190623054829.4018117-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-23_04:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=980 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906230051 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patches introduces a new foll_flag: FOLL_SPLIT_PMD. As the name says FOLL_SPLIT_PMD splits huge pmd for given mm_struct, the underlining huge page stays as-is. FOLL_SPLIT_PMD is useful for cases where we need to use regular pages, but would switch back to huge page and huge pmd on. One of such example is uprobe. The following patches use FOLL_SPLIT_PMD in uprobe. Signed-off-by: Song Liu --- include/linux/mm.h | 1 + mm/gup.c | 8 ++++++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0ab8c7d84cd0..e605acc4fc81 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2642,6 +2642,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address, #define FOLL_COW 0x4000 /* internal GUP flag */ #define FOLL_ANON 0x8000 /* don't do file mappings */ #define FOLL_LONGTERM 0x10000 /* mapping lifetime is indefinite: see below */ +#define FOLL_SPLIT_PMD 0x20000 /* split huge pmd before returning */ /* * NOTE on FOLL_LONGTERM: diff --git a/mm/gup.c b/mm/gup.c index ddde097cf9e4..41f2a1fcc6f0 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -398,7 +398,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, spin_unlock(ptl); return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } - if (flags & FOLL_SPLIT) { + if (flags & (FOLL_SPLIT | FOLL_SPLIT_PMD)) { int ret; page = pmd_page(*pmd); if (is_huge_zero_page(page)) { @@ -407,7 +407,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, split_huge_pmd(vma, pmd, address); if (pmd_trans_unstable(pmd)) ret = -EBUSY; - } else { + } else if (flags & FOLL_SPLIT) { if (unlikely(!try_get_page(page))) { spin_unlock(ptl); return ERR_PTR(-ENOMEM); @@ -419,6 +419,10 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, put_page(page); if (pmd_none(*pmd)) return no_page_table(vma, flags); + } else { /* flags & FOLL_SPLIT_PMD */ + spin_unlock(ptl); + split_huge_pmd(vma, pmd, address); + ret = pte_alloc(mm, pmd); } return ret ? ERR_PTR(ret) : From patchwork Sun Jun 23 05:48:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 11011589 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A587A14B6 for ; Sun, 23 Jun 2019 05:49:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 94D1028B2A for ; Sun, 23 Jun 2019 05:49:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8948728B2C; Sun, 23 Jun 2019 05:49:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C703D28B2A for ; Sun, 23 Jun 2019 05:48:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DBB336B000D; Sun, 23 Jun 2019 01:48:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D2A408E0006; Sun, 23 Jun 2019 01:48:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B96B48E0006; Sun, 23 Jun 2019 01:48:55 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f72.google.com (mail-yw1-f72.google.com [209.85.161.72]) by kanga.kvack.org (Postfix) with ESMTP id 97B9C6B000D for ; Sun, 23 Jun 2019 01:48:55 -0400 (EDT) Received: by mail-yw1-f72.google.com with SMTP id j144so11042177ywa.15 for ; Sat, 22 Jun 2019 22:48:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=IZysbkBOWuoojpHfDDSz6omhgc5cHCXvaawoTaDfhK0=; b=mWaK7+Tws2dKkTI8FVVeQO88BGeP22tJYxWtGJecPd3Hkc0SOPWO9pW1o7Pnmt5zLj 3sypflM8HcSDkh5Yt0vWobvjP8wPYF2UnmPhhZI/qDw0ymp1YSiY6lhpiGqXnr9XiFtp K6bVl1pR3Vyr59eeN1B//cNvDjQlckbYrgEyRoIxvZlM0ACVfTvdIqIT+X6AEFk7JUfL GDDeGe5fzf1+Fq6B2hqJ2i9lz4qlyumPg5cfRanerL4INszXerz3gtDGx0cVoS53A4J5 dcxrAU/3b+L23tBmTwsKizBvAg1zP0dFtLuSVEqsjggZPz4+eHXNql+67GSd8NzKaiXW rG+w== X-Gm-Message-State: APjAAAU8B6zO7zJnL1vSLQtvy7brW/reSZfYFBirHCzZtoNpdaJEc6bo mM4rvxLjEXvO/N9DmPvxuzejLf9S3I4XUDtw0BQXeqwHDcFrV6rRl0FErUKl9eAEuZdwDNcpyEK qx7kbC+Ivea8CwU1GFcmlcW6Ur0cd45eyWfVZ05pBi4+HYxd/yawTEyOf3wPNGK2qsQ== X-Received: by 2002:a81:e0f:: with SMTP id 15mr69676366ywo.288.1561268935386; Sat, 22 Jun 2019 22:48:55 -0700 (PDT) X-Google-Smtp-Source: APXvYqztsxavSlT5dZUinl5yoCIJu/BH2V0dSb376AWEJpYKxd+7KVvQN60k9kNkwkfYfX2P4T+N X-Received: by 2002:a81:e0f:: with SMTP id 15mr69676358ywo.288.1561268934861; Sat, 22 Jun 2019 22:48:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561268934; cv=none; d=google.com; s=arc-20160816; b=fvc6mBNv3L1NEDw/6ykLVjbIBRuyEenOG88r79fzjbSiwgCfCNAaFX3NRkfr591Cvd 8IGirLgnWJkInpOZtbcKhsNp5rF1oV1Q7q/Fuh1klRK0qz1IXhjGdvfV6Hu51xVtw/7p dJ8REY4qBDR0Q3Hac9cadRvcflIfV+C6M4DL+FbaB4DeMJTYYYse/QwFQU4bw2fWuBq4 G70FQU0QE6kMbVA9hEQRYi4gNfcUx2IaiLniJTftL7664MN7587UDvjTSPTnIl3xS/m2 p+rIX53bWr3Z1m5gQcqvdu5nF/M/1K5ujdk6VMh91TSlJhGKY6Ln+xlAKRQmXMupnHca YfvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=IZysbkBOWuoojpHfDDSz6omhgc5cHCXvaawoTaDfhK0=; b=mXq03vqV2zNVqfPM+vDtMsqAjRkt6/ld6DfrJl/lYq3LpuHKluPy8V255sq9/ycUrV yzaYkPZTwig5J/9jTrqmBPCgoiyiXlqmnk5BtYyzLbMLXQKGMzq3YfkL6jpaeQa1RKsP hESw+CvSX/bgZGkEt4xrSCSCQG2KAmEu/Kvntas9LAcs2/udorhWRsz/jLzB09EfivNQ oosOEhPlppTStGMIYQqz4kHfPQVysxGsp7trYle9CbiHTnNibPQEO0XR50a/PCagcMmE ka0c6XBSpsTSz6r1PxH8lPd0I4laZTkvE4KLmzhqagOMw/7H3DfxGvNqcPC7oRY3qHg5 LyEA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=fuYxqoDm; spf=pass (google.com: domain of prvs=1077171f80=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1077171f80=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id o205si2656539ywo.208.2019.06.22.22.48.54 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 22:48:54 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=1077171f80=songliubraving@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=fuYxqoDm; spf=pass (google.com: domain of prvs=1077171f80=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1077171f80=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5N5mrFK021094 for ; Sat, 22 Jun 2019 22:48:54 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=IZysbkBOWuoojpHfDDSz6omhgc5cHCXvaawoTaDfhK0=; b=fuYxqoDml0m23kIXQduWqGuENzNdgDgevHC2MiuhD3dp+y2xjbc4brzUharJJS5buJqD 0mcyGuX5KyeVD9xo69HXJOItTlyqBApEtG32g9MSLGZQQ1uWoU0FgRaNpJt2TQ78Zixr YzkAqi31hMPZSI1dD0rI5W5/MOaWgwBunnk= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2t9hubt441-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Sat, 22 Jun 2019 22:48:54 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::128) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Sat, 22 Jun 2019 22:48:53 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 34E9D62E2CFB; Sat, 22 Jun 2019 22:48:52 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v6 4/6] uprobe: use FOLL_SPLIT_PMD instead of FOLL_SPLIT Date: Sat, 22 Jun 2019 22:48:27 -0700 Message-ID: <20190623054829.4018117-5-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190623054829.4018117-1-songliubraving@fb.com> References: <20190623054829.4018117-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-23_04:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=715 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906230051 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patches uses newly added FOLL_SPLIT_PMD in uprobe. This enables easy regroup of huge pmd after the uprobe is disabled (in next patch). Signed-off-by: Song Liu --- kernel/events/uprobes.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index f7c61a1ef720..a20d7b43a056 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -153,7 +153,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, { struct mm_struct *mm = vma->vm_mm; struct page_vma_mapped_walk pvmw = { - .page = old_page, + .page = compound_head(old_page), .vma = vma, .address = addr, }; @@ -165,8 +165,6 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, addr, addr + PAGE_SIZE); - VM_BUG_ON_PAGE(PageTransHuge(old_page), old_page); - if (!orig) { err = mem_cgroup_try_charge(new_page, vma->vm_mm, GFP_KERNEL, &memcg, false); @@ -483,7 +481,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, retry: /* Read the page with vaddr into memory */ ret = get_user_pages_remote(NULL, mm, vaddr, 1, - FOLL_FORCE | FOLL_SPLIT, &old_page, &vma, NULL); + FOLL_FORCE | FOLL_SPLIT_PMD, &old_page, &vma, NULL); if (ret <= 0) return ret; From patchwork Sun Jun 23 05:48:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 11011591 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F2F0A14B6 for ; Sun, 23 Jun 2019 05:49:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E219928B2A for ; Sun, 23 Jun 2019 05:49:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D665F28B2C; Sun, 23 Jun 2019 05:49:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3265C28B2A for ; Sun, 23 Jun 2019 05:49:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E09AF6B0010; Sun, 23 Jun 2019 01:49:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DDF5F8E0008; Sun, 23 Jun 2019 01:49:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C59498E0007; Sun, 23 Jun 2019 01:49:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 8781C6B000E for ; Sun, 23 Jun 2019 01:49:01 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id x18so7166891pfj.4 for ; Sat, 22 Jun 2019 22:49:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=CFhlysmGz57clyaJ1HAINnv1wDIl1pPPAi1K6HzPDZ0=; b=FrrhyVbmAQB54sfa42L8ChHYyZiHAb4J0U9PZ5eWCa3e6HNtSr8AQhtJLyRTgwWacL rGCUSRVrZtIRaE1uv98rSYoF+yq9NVm1Cw4xMbpVRTF9ushQytSCF1AtpVgwV3sq0Gaa ekfRriZvguuTHnu97t26DEm6AHHnRu0mKEFTqO1rlJWcR6O+uATfEceBq8SKoB4BR/WX lxwHDFsH9dTKsL0WQ6wEjf4fhcWImujiPdDc7vNb8nvmnEjYFQR2zU6ARlJF12lPVF9y FYXZECBk/Gbd4OsoKmWIrisx8qUSjDaLJbMGkWb19tB3ODBP0SqXb+SdIm9mi63+JZWJ +cfw== X-Gm-Message-State: APjAAAUvwAwsU+BX09qTxFuJSQSeEkM+GSQKl3hlZh14Im0wuTuDGUvG 4jGaWeZcIDoTltKcbeOsZosGXW3TWHcEUth/PHFbypBbxkunoRJ07gauc1MPP/o7tD+A59YeWEx FGvsGzxF/0i3BMa1SlzHlqUSSQbZ9mNTNqOXYTxE6PXM/ws+2cJXo3qVZu4QwhxmDEA== X-Received: by 2002:a63:e001:: with SMTP id e1mr26855903pgh.306.1561268941054; Sat, 22 Jun 2019 22:49:01 -0700 (PDT) X-Google-Smtp-Source: APXvYqyl7nD4f2lKWp5EOdvA3oK+caxOxkpKxDBQY9/innhQALe8DF5u2j1agDjIikjHl7fSwNU8 X-Received: by 2002:a63:e001:: with SMTP id e1mr26855871pgh.306.1561268940159; Sat, 22 Jun 2019 22:49:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561268940; cv=none; d=google.com; s=arc-20160816; b=om39U38oBT/h+6wwG0DGLRjdpgMIpgchx1sir6wOm4SEWydrPDWC4WgJ4c0Yx7iOUy YjeKMhRgOBR+nAsnvnquMpKKTVXstd7Jmqb0xYhXVdYxe9XiBOy3rB9wKsI/J2U5UAg+ KOmQXk83uhgeU+DI6zU5lB9NpDGDn24QJTYb9PNGWg1YQvUq6AUlU0R20TP4ajuoeN8F 5OkH7A6wZamx3152+/KkOAP/2PSPkB9M9pEDGey//UnrAISbG7p6I7H+Tdi0CQsPtC5W QoAkSfgQ7voShmOF0M3d6MfNXzGp/wxq+VEGLuruGFZre10P8QCA85WwdJuoiJZ/uAHF UGsw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=CFhlysmGz57clyaJ1HAINnv1wDIl1pPPAi1K6HzPDZ0=; b=Igh1yFZkFcOZ6VmdnG+1a6zOehvBDpapCYUVx8F99U7t5R5ahV2ZpwR7VUQml3KHco EnaNeqYOe7YOg3oGd7Aqq8GDpC78plnYDiTiGoGXz1rTbKVqFGGSwFtJ2i/4/iDcUFDR EOSqAIWs9fvaxhzz9rawBej3w7Yue5viyiZXIoGIOTiWE4qssZaLeOB2oSbCtaByw+lL YB1PjNmu20ZWLthorxW903sr+wN2ZfnJpMOs2dpcRKb3yHBlXxd6LAL9qMgEvWAFjx0V HGSi6lhRIqZZHARoSO/IXa0nlD2a3XTTv+gayc8v1OblH6F7oYoeGfVxEZg8+QPcMzau s8Gg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=gafcg6C8; spf=pass (google.com: domain of prvs=1077171f80=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=1077171f80=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id a12si6974683pgq.208.2019.06.22.22.48.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 22:49:00 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=1077171f80=songliubraving@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=gafcg6C8; spf=pass (google.com: domain of prvs=1077171f80=songliubraving@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=1077171f80=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109333.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5N5iprh015299 for ; Sat, 22 Jun 2019 22:48:59 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=CFhlysmGz57clyaJ1HAINnv1wDIl1pPPAi1K6HzPDZ0=; b=gafcg6C8zP7ZBtoTG180sNk0r9ODkCNup9DgAtwrNhAafglZwxjmkMOIdDOdX4f6rBcZ gXDmKHBh5OmHh/aNzF0JLzF4/1Eugd5wpDQqo2J1e+rU4MAfN6QMKAOXDfphSvXS8hsx jn2PaPJumGspkXlSYEA8tLPi8R03XIGzCe8= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2t9gc0jd4a-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Sat, 22 Jun 2019 22:48:59 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::125) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Sat, 22 Jun 2019 22:48:58 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id A436162E2CFB; Sat, 22 Jun 2019 22:48:55 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v6 5/6] khugepaged: enable collapse pmd for pte-mapped THP Date: Sat, 22 Jun 2019 22:48:28 -0700 Message-ID: <20190623054829.4018117-6-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190623054829.4018117-1-songliubraving@fb.com> References: <20190623054829.4018117-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-23_04:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=347 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906230050 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP khugepaged needs exclusive mmap_sem to access page table. When it fails to lock mmap_sem, the page will fault in as pte-mapped THP. As the page is already a THP, khugepaged will not handle this pmd again. This patch enables the khugepaged to retry retract_page_tables(). A new flag AS_COLLAPSE_PMD is introduced to show the address_space may contain pte-mapped THPs. When khugepaged fails to trylock the mmap_sem, it sets AS_COLLAPSE_PMD. Then, at a later time, khugepaged will retry compound pages in this address_space. Since collapse may happen at an later time, some pages may already fault in. To handle these pages properly, it is necessary to prepare the pmd before collapsing. prepare_pmd_for_collapse() is introduced to prepare the pmd by removing rmap, adjusting refcount and mm_counter. prepare_pmd_for_collapse() also double checks whether all ptes in this pmd are mapping to the same THP. This is necessary because some subpage of the THP may be replaced, for example by uprobe. In such cases, it is not possible to collapse the pmd, so we fall back. Signed-off-by: Song Liu Acked-by: Kirill A. Shutemov --- include/linux/pagemap.h | 1 + mm/khugepaged.c | 69 +++++++++++++++++++++++++++++++++++------ 2 files changed, 60 insertions(+), 10 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 9ec3544baee2..eac881de2a46 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -29,6 +29,7 @@ enum mapping_flags { AS_EXITING = 4, /* final truncate in progress */ /* writeback related tags are not used */ AS_NO_WRITEBACK_TAGS = 5, + AS_COLLAPSE_PMD = 6, /* try collapse pmd for THP */ }; /** diff --git a/mm/khugepaged.c b/mm/khugepaged.c index a4f90a1b06f5..9b980327fd9b 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1254,7 +1254,47 @@ static void collect_mm_slot(struct mm_slot *mm_slot) } #if defined(CONFIG_SHMEM) && defined(CONFIG_TRANSPARENT_HUGE_PAGECACHE) -static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) + +/* return whether the pmd is ready for collapse */ +bool prepare_pmd_for_collapse(struct vm_area_struct *vma, pgoff_t pgoff, + struct page *hpage, pmd_t *pmd) +{ + unsigned long haddr = page_address_in_vma(hpage, vma); + unsigned long addr; + int i, count = 0; + + /* step 1: check all mapped PTEs are to this huge page */ + for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { + pte_t *pte = pte_offset_map(pmd, addr); + + if (pte_none(*pte)) + continue; + + if (hpage + i != vm_normal_page(vma, addr, *pte)) + return false; + count++; + } + + /* step 2: adjust rmap */ + for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { + pte_t *pte = pte_offset_map(pmd, addr); + struct page *page; + + if (pte_none(*pte)) + continue; + page = vm_normal_page(vma, addr, *pte); + page_remove_rmap(page, false); + } + + /* step 3: set proper refcount and mm_counters. */ + page_ref_sub(hpage, count); + add_mm_counter(vma->vm_mm, mm_counter_file(hpage), -count); + return true; +} + +extern pid_t sysctl_dump_pt_pid; +static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff, + struct page *hpage) { struct vm_area_struct *vma; unsigned long addr; @@ -1273,21 +1313,21 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) pmd = mm_find_pmd(vma->vm_mm, addr); if (!pmd) continue; - /* - * We need exclusive mmap_sem to retract page table. - * If trylock fails we would end up with pte-mapped THP after - * re-fault. Not ideal, but it's more important to not disturb - * the system too much. - */ if (down_write_trylock(&vma->vm_mm->mmap_sem)) { spinlock_t *ptl = pmd_lock(vma->vm_mm, pmd); - /* assume page table is clear */ + + if (!prepare_pmd_for_collapse(vma, pgoff, hpage, pmd)) { + spin_unlock(ptl); + up_write(&vma->vm_mm->mmap_sem); + continue; + } _pmd = pmdp_collapse_flush(vma, addr, pmd); spin_unlock(ptl); up_write(&vma->vm_mm->mmap_sem); mm_dec_nr_ptes(vma->vm_mm); pte_free(vma->vm_mm, pmd_pgtable(_pmd)); - } + } else + set_bit(AS_COLLAPSE_PMD, &mapping->flags); } i_mmap_unlock_write(mapping); } @@ -1561,7 +1601,7 @@ static void collapse_file(struct mm_struct *mm, /* * Remove pte page tables, so we can re-fault the page as huge. */ - retract_page_tables(mapping, start); + retract_page_tables(mapping, start, new_page); *hpage = NULL; khugepaged_pages_collapsed++; @@ -1622,6 +1662,7 @@ static void khugepaged_scan_file(struct mm_struct *mm, int present, swap; int node = NUMA_NO_NODE; int result = SCAN_SUCCEED; + bool collapse_pmd = false; present = 0; swap = 0; @@ -1640,6 +1681,14 @@ static void khugepaged_scan_file(struct mm_struct *mm, } if (PageTransCompound(page)) { + if (collapse_pmd || + test_and_clear_bit(AS_COLLAPSE_PMD, + &mapping->flags)) { + collapse_pmd = true; + retract_page_tables(mapping, start, + compound_head(page)); + continue; + } result = SCAN_PAGE_COMPOUND; break; } From patchwork Sun Jun 23 05:48:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 11011593 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C8E7F1398 for ; Sun, 23 Jun 2019 05:49:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B6E4728B2A for ; Sun, 23 Jun 2019 05:49:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AA1CE28B2C; Sun, 23 Jun 2019 05:49:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 235FC28B2A for ; Sun, 23 Jun 2019 05:49:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1FBFC6B000E; Sun, 23 Jun 2019 01:49:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 188608E0006; Sun, 23 Jun 2019 01:49:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E05F66B000E; Sun, 23 Jun 2019 01:49:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f72.google.com (mail-yw1-f72.google.com [209.85.161.72]) by kanga.kvack.org (Postfix) with ESMTP id B625B8E0006 for ; Sun, 23 Jun 2019 01:49:01 -0400 (EDT) Received: by mail-yw1-f72.google.com with SMTP id o135so10961270ywo.16 for ; Sat, 22 Jun 2019 22:49:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=nrhKkEIFZQZyLO47iU67e7PRaxciftm8IRTz/uqOqR0=; b=oD3C26mM+dYc5kZ0lGw7R49432/aY3FfXOzPtaoMpY421CCBbwUetQU5rs9FzqkLcb GQZPE096C5v+dz/md9Eu+42cQ95lYjJnotjmdAwjNgAvG793ttil2KzILSXzLOxYCJgf 7QDimKWD3/t9QfOQvRUQhuCd9O4KDLGS5rrjrC7jzKgXpA/Uy5unJDGtHqz7sJLM6/rh JDs7niKtYY1ae19nrab+dqurhbhMNUvpmRCsURAmmRjsyquVLQVGF4yFS7wMU/7K3vde tICQEdQ2gCyfuegirwzoFI1lNaQMKUAgJew2NhN2oBB+9YkiPQThQ1cmbNb8tviJbDu1 OcYg== X-Gm-Message-State: APjAAAX3IVNv6GpDaYEVhvB4Ey04mAtAVCqlPWSCxjvgRHMZb7b5lE6c Plt6HqD7Bbj9l4IgYHMVBHllgWYZlwgS5Vd9TPHshvGfp4oc/oe73f4MYGTMeUDOCgI1PGG7cLZ DAwZGXAJthloFUmuPQ0Q6wq1Yi5OHhwxYavBECrSniC4rCDVk3+TCVu3GXffXQn0y2w== X-Received: by 2002:a25:cc0a:: with SMTP id l10mr15659734ybf.433.1561268941489; Sat, 22 Jun 2019 22:49:01 -0700 (PDT) X-Google-Smtp-Source: APXvYqyBgdrAib88ZSNTQQyj8cf+AyEZQzM8Uk6+fmkOQZM97pwBMDSPXSXPhTtGyBiq7veondc5 X-Received: by 2002:a25:cc0a:: with SMTP id l10mr15659728ybf.433.1561268940978; Sat, 22 Jun 2019 22:49:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561268940; cv=none; d=google.com; s=arc-20160816; b=ak3FtZNeBZWImN10tFra29dEnukYddWQo+gPwAwauepO9Vupd/oNJhlhycXm4VrB6F VWRGVfVj3paWL7kGFmk0J/P4n9IF+VhH3VroLrOvRGx+Plmebx8KYKyvPTOd0KBPHRGd 4JJDP0P94zJD2Gews/1w6uzDBJ5WmaRXib9TLwFXpAb1WntQ3PmWgVyE+TTwEO+6mAEp DEtkeLHY6WZSgjlvkgJYjnSs6ENKiZzQwmZ0O/heNiedEkVpYQxSii3Fax/jo6/NtUO3 vctNMrx4Otmgk6p8y2h+kymuQEi7IzkRvD0WOdwMJHmQi1LabOtNo0G04pvMxG1sdutY mWqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=nrhKkEIFZQZyLO47iU67e7PRaxciftm8IRTz/uqOqR0=; b=HSFhRovPssOaElOvRdWTocNwPX0crRGJ3jOAhnLvXoyC2M3iH5SIe8YCO0jyhjG4E5 RqH234usbN0amepse08j8lHLz1kVtDQYYyZLB/WydLGg7IH24TbLwnveycZP6R8XgBMN v/2PClVqQxtq2Z6rgcfDMpta3SS8txDSNrukebaAne0wqEDml3iqyr3sYkMjB7vnEWj/ eB+/EMn1bfb+GLfReCjAO/+PXVSzpydm4lHQQ2tmaFL/KOlDd05JI4M0ifEgTzZXY6vc q5kMb+Zueqat8ugKRmKjQ//hHgTQyxmuYDoiZtQg8eH9BaP1DElBR0JLEARmwqLKeikq ZXkw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=n1Mcb0xG; spf=pass (google.com: domain of prvs=1077171f80=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1077171f80=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id l189si2667685ywf.14.2019.06.22.22.49.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Jun 2019 22:49:00 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=1077171f80=songliubraving@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=n1Mcb0xG; spf=pass (google.com: domain of prvs=1077171f80=songliubraving@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=1077171f80=songliubraving@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5N5jBqZ023070 for ; Sat, 22 Jun 2019 22:49:00 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=nrhKkEIFZQZyLO47iU67e7PRaxciftm8IRTz/uqOqR0=; b=n1Mcb0xGlFicl1b6shGpkYWp0dyD67YcBa33e1T8uSrJxtYb4UfwBYdtfsPWoV7EkMhp 5uVeE8tweF3P7RCVVxSK3WJq5YpcBEOxpchg/g8wF8itEhVIC/ddefSvFxVw/VIvlCBt W3QMPnENFjD20JsVCgt2dOeCFPCeAyljreE= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2t9j58a3fa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Sat, 22 Jun 2019 22:49:00 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Sat, 22 Jun 2019 22:48:59 -0700 Received: by devbig006.ftw2.facebook.com (Postfix, from userid 4523) id 4567E62E2CFB; Sat, 22 Jun 2019 22:48:58 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Song Liu Smtp-Origin-Hostname: devbig006.ftw2.facebook.com To: , CC: , , , , , , , Song Liu Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v6 6/6] uprobe: collapse THP pmd after removing all uprobes Date: Sat, 22 Jun 2019 22:48:29 -0700 Message-ID: <20190623054829.4018117-7-songliubraving@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190623054829.4018117-1-songliubraving@fb.com> References: <20190623054829.4018117-1-songliubraving@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-23_04:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=724 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906230050 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP After all uprobes are removed from the huge page (with PTE pgtable), it is possible to collapse the pmd and benefit from THP again. This patch does the collapse by setting AS_COLLAPSE_PMD. khugepage would retrace the page table. A check for vma->anon_vma is removed from retract_page_tables(). The check was initially marked as "probably overkill". The code works well without the check. An issue on earlier version was discovered by kbuild test robot. Reported-by: kbuild test robot Signed-off-by: Song Liu --- kernel/events/uprobes.c | 6 +++++- mm/khugepaged.c | 3 --- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index a20d7b43a056..418382259f61 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -474,6 +474,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, struct page *old_page, *new_page; struct vm_area_struct *vma; int ret, is_register, ref_ctr_updated = 0; + struct page *orig_page = NULL; is_register = is_swbp_insn(&opcode); uprobe = container_of(auprobe, struct uprobe, arch); @@ -512,7 +513,6 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, copy_to_page(new_page, vaddr, &opcode, UPROBE_SWBP_INSN_SIZE); if (!is_register) { - struct page *orig_page; pgoff_t index; index = vaddr_to_offset(vma, vaddr & PAGE_MASK) >> PAGE_SHIFT; @@ -540,6 +540,10 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, if (ret && is_register && ref_ctr_updated) update_ref_ctr(uprobe, mm, -1); + if (!ret && orig_page && PageTransCompound(orig_page)) + set_bit(AS_COLLAPSE_PMD, + &compound_head(orig_page)->mapping->flags); + return ret; } diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 9b980327fd9b..2e277a2d731f 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1302,9 +1302,6 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff, i_mmap_lock_write(mapping); vma_interval_tree_foreach(vma, &mapping->i_mmap, pgoff, pgoff) { - /* probably overkill */ - if (vma->anon_vma) - continue; addr = vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT); if (addr & ~HPAGE_PMD_MASK) continue;