From patchwork Mon Apr 29 07:24:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13646412 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A509EC04FFE for ; Mon, 29 Apr 2024 07:24:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 22EA56B008A; Mon, 29 Apr 2024 03:24:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1DD986B008C; Mon, 29 Apr 2024 03:24:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A6C36B0092; Mon, 29 Apr 2024 03:24:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DC4886B008A for ; Mon, 29 Apr 2024 03:24:36 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 9E814A25A6 for ; Mon, 29 Apr 2024 07:24:36 +0000 (UTC) X-FDA: 82061731752.28.5C8D6D8 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by imf15.hostedemail.com (Postfix) with ESMTP id 474DEA0004 for ; Mon, 29 Apr 2024 07:24:33 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=none; spf=pass (imf15.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714375475; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dPwSMqfyeRfZEvN7u+MuvciL8ss1iBEKKPOI5/l7uv4=; b=7DPl0FWF/U4c9ambzeNsSthVvtgt5NGuiiTpdRP+Ypjuf61cYj03fcIfIcnG3ejiT2OsX/ jXAF2I+/1UGqGdH4U+e5wkb57WWchxe5YYl41ygPXdZbvQUYcqEprkDTH6ERPxTcHcBoqg 7fMEUmJxtAaB65Uv0Yw6ml++62TYcR0= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=none; spf=pass (imf15.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714375475; a=rsa-sha256; cv=none; b=WtPDkl9KBTVyweJP+f59xBqE9Yxd+x7kH01NunRez5LhVmn3g96mYg0eHxhnySL/S937Qj xn56pVcyXGwVayQqKMEHVJbE/4OfJizEHG/VsWk058pXidpccaNxLze/q5n/d2IhGVt5JQ 9Xm7j/4qOsennwVgdBB8ctdAE8GY1Rs= Received: from mail.maildlp.com (unknown [172.19.163.174]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4VSZWN5QPRzvPrW; Mon, 29 Apr 2024 15:21:20 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id CBA76140429; Mon, 29 Apr 2024 15:24:28 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Mon, 29 Apr 2024 15:24:28 +0800 From: Kefeng Wang To: Andrew Morton CC: "Matthew Wilcox (Oracle)" , , , Kefeng Wang Subject: [PATCH rfc 1/4] mm: memory: add prepare_range_pte_entry() Date: Mon, 29 Apr 2024 15:24:14 +0800 Message-ID: <20240429072417.2146732-2-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240429072417.2146732-1-wangkefeng.wang@huawei.com> References: <20240429072417.2146732-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To dggpemm100001.china.huawei.com (7.185.36.93) X-Rspamd-Queue-Id: 474DEA0004 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: cj8c1ea1kprsorjgqstptb5ooecngq5p X-HE-Tag: 1714375473-449214 X-HE-Meta: U2FsdGVkX19Tq3XTup8Pn14rwclKnH4H3Dva/T6LPudE2c/H55BsF2fyQv4SDdCjHQhtF95aEeqrolmT2S/Gj22mpWnQ2wyRh4Zy6hOorKdMobsayM7vF9CMs0cTTPRW+gK5rz5xehr7rld5IsqBtLrptzFa+Zd4fWl1X+PhK9DB4OjpJKL9GhTfDOsVdG7SOks+1VIwoGbIrz1BkCyLBYUsni+s3UJgbU3GD/tp39dqTZlFke+Hp5nJksgxl5RLGYD0zwO0F2WQypRK7JKU13nYqUVNBXzrpe4ZSKij8d8s8sFlP5zGtPLHpzWzrmAw6wCbeaVcmCRdWlvbwj4Zi51gO/GuGc42oVyfy8QYQc+X6XBhs9qnkd6JE0mmk9K8aCZJlDLi7QuNOlc39UIe2rNLOvx5zqaZiNPdxFDhN2tYNVS1r9wafemw+VRZwdugehzvwXH9yFs3i4AWDhNaw0P+viZl22ckYlnu66nhEbwev6KSaOdkAjsIYqWzAyoleQ9nxq2Cx52KXGYvvE7wymKfp0JDkdBMkOsLXIlSmTIbYA7Uuu5GMYFWbglBdXbCRbzM7gowVDJ1bBq2uAmWsovlev00AdlXVL/KHXs14Ix0WLJTNiNvZnz0sdKKKTAMftOCEpNuZNQParkfRyvXh2E50sIUDTFZH9ko+MyLGiaa5IdoYxrIs9tr/dFXuWzoNpPhV9NM2PCc4WySNm6+eWmHiXIl3F/+BoGMBPMo6rP9HCdAFMBlczqdOywCTzrOPSH8IIQgEoO7ZGIEI94BZP1rYmmqathMXQiWjPdkdbyYRo/j7OvcGF7lCVk/GpM+gP7tZE1qQs9a12xQw6tmM7A3RlE3XYcaKItmzpStNFz0a8puVT88q9d7TRUV23cZ13IwYyw21WAqBmkrvF8nfSzb6/eLgUmyhl34goV3GIKsZikNXe/ugBH2o9JCKLRGrcny4ch3CE4yIxomAEl uUt4n6TV HFXg5YRvo8omiYdzj48x7/Ek3BthbSLhgVG7exw9gbiRRIyzhuNge1Qf5+WcNB8CVXZxNuZJpVt18IfeIU7hFGmzF8qGONKzwOxxZWZI4jK4aTHRj9miRGmgbHrSdSDrSVqK7zvCXhIMrZ2Htd4jFLZ01DFQmaCgSyEXWiy4ZZPi5otj+Z90lYREiqiTyT9xHhkiT X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is prepare for a separate filemap_set_pte_range(), add a prepare_range_pte_entry(), no functional changes. Signed-off-by: Kefeng Wang --- include/linux/mm.h | 2 ++ mm/memory.c | 33 ++++++++++++++++++++++----------- 2 files changed, 24 insertions(+), 11 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 9849dfda44d4..bcbeb8a4cd43 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1372,6 +1372,8 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma) } vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page); +pte_t prepare_range_pte_entry(struct vm_fault *vmf, bool write, struct folio *folio, + struct page *page, unsigned int nr, unsigned long addr); void set_pte_range(struct vm_fault *vmf, struct folio *folio, struct page *page, unsigned int nr, unsigned long addr); diff --git a/mm/memory.c b/mm/memory.c index 6647685fd3c4..ccbeb58fa136 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4652,19 +4652,11 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) } #endif -/** - * set_pte_range - Set a range of PTEs to point to pages in a folio. - * @vmf: Fault decription. - * @folio: The folio that contains @page. - * @page: The first page to create a PTE for. - * @nr: The number of PTEs to create. - * @addr: The first address to create a PTE for. - */ -void set_pte_range(struct vm_fault *vmf, struct folio *folio, - struct page *page, unsigned int nr, unsigned long addr) +pte_t prepare_range_pte_entry(struct vm_fault *vmf, bool write, + struct folio *folio, struct page *page, + unsigned int nr, unsigned long addr) { struct vm_area_struct *vma = vmf->vma; - bool write = vmf->flags & FAULT_FLAG_WRITE; bool prefault = in_range(vmf->address, addr, nr * PAGE_SIZE); pte_t entry; @@ -4680,6 +4672,25 @@ void set_pte_range(struct vm_fault *vmf, struct folio *folio, entry = maybe_mkwrite(pte_mkdirty(entry), vma); if (unlikely(vmf_orig_pte_uffd_wp(vmf))) entry = pte_mkuffd_wp(entry); + + return entry; +} + +/** + * set_pte_range - Set a range of PTEs to point to pages in a folio. + * @vmf: Fault description. + * @folio: The folio that contains @page. + * @page: The first page to create a PTE for. + * @nr: The number of PTEs to create. + * @addr: The first address to create a PTE for. + */ +void set_pte_range(struct vm_fault *vmf, struct folio *folio, + struct page *page, unsigned int nr, unsigned long addr) +{ + struct vm_area_struct *vma = vmf->vma; + bool write = vmf->flags & FAULT_FLAG_WRITE; + pte_t entry = prepare_range_pte_entry(vmf, write, folio, page, nr, addr); + /* copy-on-write page */ if (write && !(vma->vm_flags & VM_SHARED)) { VM_BUG_ON_FOLIO(nr != 1, folio); From patchwork Mon Apr 29 07:24:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13646408 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17017C4345F for ; Mon, 29 Apr 2024 07:24:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 594F26B0085; Mon, 29 Apr 2024 03:24:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 513096B0087; Mon, 29 Apr 2024 03:24:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 364516B0089; Mon, 29 Apr 2024 03:24:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 194566B0085 for ; Mon, 29 Apr 2024 03:24:36 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 6EFE8140D46 for ; Mon, 29 Apr 2024 07:24:35 +0000 (UTC) X-FDA: 82061731710.17.3C9469D Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by imf02.hostedemail.com (Postfix) with ESMTP id E43AE8000B for ; Mon, 29 Apr 2024 07:24:32 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714375473; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eMrGXSIKBSiih200eyCVfCV6cTNkIK6OZo1ibcy6MLc=; b=vbOp4avKa1p1Vua3im8n9iHv6X1OK4dEXUYHhuHPllFc/OY5jSQ14CyA9hTna7YRyxwOug aFwXuz3jaa1H9QL8dxG3WoaE5E95MSnED+NUNrebdYj3bbyjxIIixB8om4bmpZywCV9i/o Pbktn5UmmTMxI9yo5tAZcnFRaZ/V7N4= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714375473; a=rsa-sha256; cv=none; b=ltp4jZ28CY/70EvntV81vw7q0yp4CzNOKLpe8iWtmeWQ1ft6VBHh3GtfUwgkRRBEFaB1C7 ViCL30ucPWZNNdMh/pROIsaYh6ieRyWHc1vdJdlEeeg+dGfE24NlAyZiXTk2lxi/GUpLkE Aq8O8a6rWJkRbDloju1nR5OalUYzW+U= Received: from mail.maildlp.com (unknown [172.19.163.174]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4VSZWP0vXzzvQsT; Mon, 29 Apr 2024 15:21:21 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id 316A7140444; Mon, 29 Apr 2024 15:24:29 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Mon, 29 Apr 2024 15:24:28 +0800 From: Kefeng Wang To: Andrew Morton CC: "Matthew Wilcox (Oracle)" , , , Kefeng Wang Subject: [PATCH rfc 2/4] mm: filemap: add filemap_set_pte_range() Date: Mon, 29 Apr 2024 15:24:15 +0800 Message-ID: <20240429072417.2146732-3-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240429072417.2146732-1-wangkefeng.wang@huawei.com> References: <20240429072417.2146732-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To dggpemm100001.china.huawei.com (7.185.36.93) X-Rspam-User: X-Rspamd-Queue-Id: E43AE8000B X-Rspamd-Server: rspam06 X-Stat-Signature: u1mruquitcu7np1hgepe7pneiwwkjrsz X-HE-Tag: 1714375472-279454 X-HE-Meta: U2FsdGVkX1/u2MX4cfWwY6tNeN2ho7XAhGtlzbSEMG5wgHseKQLic9VzVMpTP5wm9B5TmRMQRe1ZUFRUBXzHW4RzJ6TvoOzsr8FEhNy2UE9A/Oq55CtCX/d+YDUJRKZEymuZx0v06ZP/srNUZMv7X+UCXe04X7oJvm8MB+mDHTqUYY5OrCbANXphbLSouhTJnHiE7cXWwWToslahFIkC057eUxBRZHs0hGO0GgtERmXY6C5DhGZ7Mg26rOpagod0E9sFEz9OtQHhnZbvkoFIgNUkrB6KyhQ2+ziaW8rEXv8xrxEe3OG28epb0inAAqxZGrjXVs94kqpnixcsNeKTrJTlMn2zkRiLKLOCPm1nIFOwIow0uae4QrgwRuRaRiCPnXENPlqbc7fLT7ngJXHMlqTRnYHcfha+s/NYzzSvGZq5m8ELNQYCtHclzen8LCjziw74pcb4mkgRzczkCV5xzxqlHMvli8rL1l/kj37wQgU7enhP0Gob65dWCJm4JWqxyTOm6cTbeO4UuUhQT5BKkWpTaNCGpiLGvM0LIC5jNu+lznkNAnSn8fVOwCdRk6A5gMuWi5E56PCZeA/lAhlkqwgwxlIA9lt3rRWyQnM89XTFmGj7aKNloBAzUuyNGWemJiswbDSUZWEuE9+rwFicCWDnn13Q5TQbkASD1s2UTnP3qqu00W9iDo+BZQqxoK6UFjSbx+gXz74bzGKpngRD8i3w2m8IAjqMitONuwLFm/aSPPaDqjy72SLxD5UzrmsnF0TinHoRE85UcSjv9/htnCETYhGWO9darEOSNFpzfK8bKLvbh3r4u0zpwvtGJ426feV9qXWXIC3lRC0KVFogoonmTB1HjRW9w579iX1oTgVPJQKuNmXefMH7KMxbsM/TSX4EMSEIkW9Rwb6NGvxGz1xfRdBats8wBUlyiU3/t4I4wHEpLQLH+c0VF1BoHIoAelvL0nWsc/eOc48pLsf 94MjDz/K yZFUYPvVmTp8AN7kN6zda5dygLHrT8ZELOq6cOW6X9EKF9pkkjKqsw7p1R7BpFJoERMzSYqWiwNwdDYhA9X8eCFuNrcpV2Gbj55dxc6VCFprpa8yYMKLzAyoz8axOLwdaXWBW6c1fNE4MdgilfWSz5CzT5781onFyssorpI6WFebp2PlZ++N0KQ0XzZjnVvTz6O2a X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Adding filemap_set_pte_range() independent of set_pte_range() to unify the rss and folio reference update for small folio and large folio, which also is prepare for the upcoming lruvec stat batch updating. Signed-off-by: Kefeng Wang --- mm/filemap.c | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index ec273b00ce5f..7019692daddd 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3499,6 +3499,25 @@ static struct folio *next_uptodate_folio(struct xa_state *xas, return NULL; } +static void filemap_set_pte_range(struct vm_fault *vmf, struct folio *folio, + struct page *page, unsigned int nr, unsigned long addr, + unsigned long *rss) +{ + struct vm_area_struct *vma = vmf->vma; + pte_t entry; + + entry = prepare_range_pte_entry(vmf, false, folio, page, nr, addr); + + folio_add_file_rmap_ptes(folio, page, nr, vma); + set_ptes(vma->vm_mm, addr, vmf->pte, entry, nr); + + /* no need to invalidate: a not-present page won't be cached */ + update_mmu_cache_range(vmf, vma, addr, vmf->pte, nr); + + *rss += nr; + folio_ref_add(folio, nr); +} + /* * Map page range [start_page, start_page + nr_pages) of folio. * start_page is gotten from start by folio_page(folio, start) @@ -3539,9 +3558,7 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, continue; skip: if (count) { - set_pte_range(vmf, folio, page, count, addr); - *rss += count; - folio_ref_add(folio, count); + filemap_set_pte_range(vmf, folio, page, count, addr, rss); if (in_range(vmf->address, addr, count * PAGE_SIZE)) ret = VM_FAULT_NOPAGE; } @@ -3554,9 +3571,7 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, } while (--nr_pages > 0); if (count) { - set_pte_range(vmf, folio, page, count, addr); - *rss += count; - folio_ref_add(folio, count); + filemap_set_pte_range(vmf, folio, page, count, addr, rss); if (in_range(vmf->address, addr, count * PAGE_SIZE)) ret = VM_FAULT_NOPAGE; } @@ -3591,9 +3606,7 @@ static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf, if (vmf->address == addr) ret = VM_FAULT_NOPAGE; - set_pte_range(vmf, folio, page, 1, addr); - (*rss)++; - folio_ref_inc(folio); + filemap_set_pte_range(vmf, folio, page, 1, addr, rss); return ret; } From patchwork Mon Apr 29 07:24:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13646409 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43C88C19F53 for ; Mon, 29 Apr 2024 07:24:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 80F756B0087; Mon, 29 Apr 2024 03:24:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 75E3C6B0092; Mon, 29 Apr 2024 03:24:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4C6646B008C; Mon, 29 Apr 2024 03:24:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 26CEF6B0087 for ; Mon, 29 Apr 2024 03:24:36 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id BDCA2418E6 for ; Mon, 29 Apr 2024 07:24:35 +0000 (UTC) X-FDA: 82061731710.21.07EA869 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by imf18.hostedemail.com (Postfix) with ESMTP id 522FA1C000F for ; Mon, 29 Apr 2024 07:24:32 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf18.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714375474; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ziwqm9Ta6Oac20CoDTsmUctDiFdn1EuMPpT2nbUkDZg=; b=3WznfWG2CYZTpkFcag4tPBUtqOb/KA9bg8ItLGi/E1DRZdRZ0hiK3UQ/wRuqe1qZaCO6dn sA/rt/dJu6K+OUfIjLl2NllEuVl5oAvJgaA+0A8sakh1PqI4yAXxDQpeC2UKuWlYUssCQG mTtuKbGX8fQcYplV5eboQ79kcC287Q4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714375474; a=rsa-sha256; cv=none; b=X89fPLCk8nzo4PLwOgjyQqKUrduaiFLdO8cf7jRG8dxs19MoR/reE/mah0vE1/6fcxsoGL gCTrqyQwKACYY4JuJ+LO+acRfoe2mM6JcU6ranCJEp6HohlOfDq2GxCAA/d8Yz8xggVO7n U4IVIfuWWfmfM8TMJ3SJh45tFJSugEM= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf18.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com Received: from mail.maildlp.com (unknown [172.19.163.252]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4VSZWP3rhQzvQsr; Mon, 29 Apr 2024 15:21:21 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id 93C4918007D; Mon, 29 Apr 2024 15:24:29 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Mon, 29 Apr 2024 15:24:29 +0800 From: Kefeng Wang To: Andrew Morton CC: "Matthew Wilcox (Oracle)" , , , Kefeng Wang Subject: [PATCH rfc 3/4] mm: filemap: move __lruvec_stat_mod_folio() out of filemap_set_pte_range() Date: Mon, 29 Apr 2024 15:24:16 +0800 Message-ID: <20240429072417.2146732-4-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240429072417.2146732-1-wangkefeng.wang@huawei.com> References: <20240429072417.2146732-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To dggpemm100001.china.huawei.com (7.185.36.93) X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 522FA1C000F X-Stat-Signature: hie7ymoptkso1tepw1fsf3kyk87xiaxf X-Rspam-User: X-HE-Tag: 1714375472-399581 X-HE-Meta: U2FsdGVkX1+CzAWw89AZZDWuaFwXxb+Pmfzlc2rTGc0HEtItRd8XGz/fR7FyrjTB38KwItYT61kziJQNwZBSM9RwCucaUm3TGNK6ch/9cqL1s+MsZ3lDEA3JuS2pXTgUURLazIzCSr/cRpWfbHaZYXSkv54WBtGcG8SiSL+4YehFJ+IiXPBXrgDCkeanZKt9fQFdbpt9I86eNWvxS10G5PTotRJapEWBs3njq/s3k346WjSOAe2QDVwHRVNnXgoDtCAzO3XkIxM05exVO85zYDM7kDAAaFmdOTtj9hL7hugG4c9jZS1Z3tUafrV0mo/+m8HQXu+2A17ElnFD2Wo8OUSH8ZMJso2ph/ORWRd5HOZjWHUdkBFStqLMtsmfbhB4TZbBhoBL0bPqrynhyPt76iMWf2ZmX/sTDyqotARhQhFDw6WL29o7wdUSbgbnlLbABQUlJ9r72l1sgh8+2u1oJjaULihDemf8VU61E2nWJaQnRoidXqgISzSZKBW/t46uBXVviB3FFsohxDBs4QfbnzYC843vDnRa1DtMgRd+1r3iPKEAie3Vrqi6ZxbzdX9GQHymE502tXV4uvN9QC29/f4TyIEgrH7Hopk7XI0qsW25GqVJxb6ccmdTWZcr9V2GPFrLhJGinKr8sy7y1bECv56wVB7JQ0K6Ym5fSEr9RTV47e4/7bnGwLYM923nfQSmL7FFfIJCKEkXamPe+HjoyoFL8mvSeXu7nvq/u3EIXZR9sJBsZexWm5uKZQ4WnosMIfubhtM1BLS8hK4iy+eBTN88rOXhgwfDWqPXCUqG+RSOfI0OQB+B2bPkp9YaoK8tbxxVJ53deY1pSZFTl1xgrzCaFGDnE1Xr7icZZJ0zulcTJ35iALnmQXc5GKGKwhLk9CpIrN+7n2m5H6DnXYSFlSWA+1AX7XzPufIJlep0pbkktT8OLWe/DLC2lE8AKmUJ5gZLIqBDQM2MwF+ABvS LDUCnKuY Yh4INJc0OyPYJjf+fi6DDxyupR1cat2imhcouTkpOiYNsmPD4XhZqM5HvJuLZRdNWOLFFN08+O8Xv30M6qr7IkC/6/lGZ6Drx1qoJEHohU23sLYshpiOYKYolcwUEVzgvb/ej3G8dFD2juq+fwybbaeYjf5TbD+zy62TUd3kzgmriWs8hfT9KH0jDdKIeHaHdSc3E X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Adding __folio_add_file_rmap_ptes() which don't update lruvec stat, it is used in filemap_set_pte_range(), with it, lruvec stat updating is moved into the caller, no functional changes. Signed-off-by: Kefeng Wang --- include/linux/rmap.h | 2 ++ mm/filemap.c | 27 ++++++++++++++++++--------- mm/rmap.c | 16 ++++++++++++++++ 3 files changed, 36 insertions(+), 9 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 7229b9baf20d..43014ddd06f9 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -242,6 +242,8 @@ void folio_add_anon_rmap_pmd(struct folio *, struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); +int __folio_add_file_rmap_ptes(struct folio *, struct page *, int nr_pages, + struct vm_area_struct *); void folio_add_file_rmap_ptes(struct folio *, struct page *, int nr_pages, struct vm_area_struct *); #define folio_add_file_rmap_pte(folio, page, vma) \ diff --git a/mm/filemap.c b/mm/filemap.c index 7019692daddd..3966b6616d02 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3501,14 +3501,15 @@ static struct folio *next_uptodate_folio(struct xa_state *xas, static void filemap_set_pte_range(struct vm_fault *vmf, struct folio *folio, struct page *page, unsigned int nr, unsigned long addr, - unsigned long *rss) + unsigned long *rss, int *nr_mapped) { struct vm_area_struct *vma = vmf->vma; pte_t entry; entry = prepare_range_pte_entry(vmf, false, folio, page, nr, addr); - folio_add_file_rmap_ptes(folio, page, nr, vma); + *nr_mapped += __folio_add_file_rmap_ptes(folio, page, nr, vma); + set_ptes(vma->vm_mm, addr, vmf->pte, entry, nr); /* no need to invalidate: a not-present page won't be cached */ @@ -3525,7 +3526,8 @@ static void filemap_set_pte_range(struct vm_fault *vmf, struct folio *folio, static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, struct folio *folio, unsigned long start, unsigned long addr, unsigned int nr_pages, - unsigned long *rss, unsigned int *mmap_miss) + unsigned long *rss, int *nr_mapped, + unsigned int *mmap_miss) { vm_fault_t ret = 0; struct page *page = folio_page(folio, start); @@ -3558,7 +3560,8 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, continue; skip: if (count) { - filemap_set_pte_range(vmf, folio, page, count, addr, rss); + filemap_set_pte_range(vmf, folio, page, count, addr, + rss, nr_mapped); if (in_range(vmf->address, addr, count * PAGE_SIZE)) ret = VM_FAULT_NOPAGE; } @@ -3571,7 +3574,8 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, } while (--nr_pages > 0); if (count) { - filemap_set_pte_range(vmf, folio, page, count, addr, rss); + filemap_set_pte_range(vmf, folio, page, count, addr, rss, + nr_mapped); if (in_range(vmf->address, addr, count * PAGE_SIZE)) ret = VM_FAULT_NOPAGE; } @@ -3583,7 +3587,7 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf, struct folio *folio, unsigned long addr, - unsigned long *rss, unsigned int *mmap_miss) + unsigned long *rss, int *nr_mapped, unsigned int *mmap_miss) { vm_fault_t ret = 0; struct page *page = &folio->page; @@ -3606,7 +3610,7 @@ static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf, if (vmf->address == addr) ret = VM_FAULT_NOPAGE; - filemap_set_pte_range(vmf, folio, page, 1, addr, rss); + filemap_set_pte_range(vmf, folio, page, 1, addr, rss, nr_mapped); return ret; } @@ -3646,6 +3650,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, folio_type = mm_counter_file(folio); do { unsigned long end; + int nr_mapped = 0; addr += (xas.xa_index - last_pgoff) << PAGE_SHIFT; vmf->pte += xas.xa_index - last_pgoff; @@ -3655,11 +3660,15 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, if (!folio_test_large(folio)) ret |= filemap_map_order0_folio(vmf, - folio, addr, &rss, &mmap_miss); + folio, addr, &rss, &nr_mapped, + &mmap_miss); else ret |= filemap_map_folio_range(vmf, folio, xas.xa_index - folio->index, addr, - nr_pages, &rss, &mmap_miss); + nr_pages, &rss, &nr_mapped, + &mmap_miss); + + __lruvec_stat_mod_folio(folio, NR_FILE_MAPPED, nr_mapped); folio_unlock(folio); folio_put(folio); diff --git a/mm/rmap.c b/mm/rmap.c index 2608c40dffad..55face4024f2 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1452,6 +1452,22 @@ static __always_inline void __folio_add_file_rmap(struct folio *folio, mlock_vma_folio(folio, vma); } +int __folio_add_file_rmap_ptes(struct folio *folio, struct page *page, + int nr_pages, struct vm_area_struct *vma) +{ + int nr, nr_pmdmapped = 0; + + VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); + + nr = __folio_add_rmap(folio, page, nr_pages, RMAP_LEVEL_PTE, + &nr_pmdmapped); + + /* See comments in folio_add_anon_rmap_*() */ + if (!folio_test_large(folio)) + mlock_vma_folio(folio, vma); + + return nr; +} /** * folio_add_file_rmap_ptes - add PTE mappings to a page range of a folio * @folio: The folio to add the mappings to From patchwork Mon Apr 29 07:24:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13646411 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53676C4345F for ; Mon, 29 Apr 2024 07:24:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D43046B0088; Mon, 29 Apr 2024 03:24:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BA1E26B0092; Mon, 29 Apr 2024 03:24:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A62A6B0088; Mon, 29 Apr 2024 03:24:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 5416D6B0089 for ; Mon, 29 Apr 2024 03:24:36 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id F40C0A29E5 for ; Mon, 29 Apr 2024 07:24:35 +0000 (UTC) X-FDA: 82061731710.08.B2EF83F Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by imf03.hostedemail.com (Postfix) with ESMTP id 8808D20020 for ; Mon, 29 Apr 2024 07:24:33 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714375474; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P83nek/83zfmOjfgmemVghMYO55C7SajunYhkBe1QLM=; b=QERIn0i9gZ+59zYlSSMptgQUyyRL4hIPbNNsVYteRqWKZlVCYp1ECv7TkCl9OmpG2pnP6+ K9x2XxRTxpR1Gh/OHo+Oy/CHetcxmGLvw2ygL1rERR9x4nApxA5CEXufkQlTxuJMod/wam ElIFpcKPQqHm6N45I5uSlivKHH5pklw= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714375474; a=rsa-sha256; cv=none; b=5xP4Vf/Jmb6xSpmaxBBFefs7DBWRPDQoHJvnC0xgBbQ/AH+RZ+4KxJfA6gqCwuIroUxrvT yUbf0zwb0BsjphkLmAjbO3PrWGBV+wpSLvDoMjms2grjVlFvdM5q4EKQnL+1QudF5P9cf+ 1hq3V6FfjqgSb2juRWgXlTdULoKfOdQ= Received: from mail.maildlp.com (unknown [172.19.88.194]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4VSZYl0X6Pzcb0R; Mon, 29 Apr 2024 15:23:23 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id E866E14035F; Mon, 29 Apr 2024 15:24:29 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Mon, 29 Apr 2024 15:24:29 +0800 From: Kefeng Wang To: Andrew Morton CC: "Matthew Wilcox (Oracle)" , , , Kefeng Wang Subject: [PATCH rfc 4/4] mm: filemap: try to batch lruvec stat updating Date: Mon, 29 Apr 2024 15:24:17 +0800 Message-ID: <20240429072417.2146732-5-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240429072417.2146732-1-wangkefeng.wang@huawei.com> References: <20240429072417.2146732-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To dggpemm100001.china.huawei.com (7.185.36.93) X-Rspam-User: X-Rspamd-Queue-Id: 8808D20020 X-Rspamd-Server: rspam06 X-Stat-Signature: ojzord19o1bq9nkit7jbot1rkjejkgcm X-HE-Tag: 1714375473-9421 X-HE-Meta: U2FsdGVkX198GpLNAp4qswTWBpa4GGczhfmWqwJjvWoi4pACa0qpWKgEA6MFHbE9JJ+S68ukXLJNI+UsvxyczQVvOZECCik9ism3UjH0vP2QCWGDJ8OJ67GhdaDEOiC6f4wkKAUwIQcNXe6MwtRkjUnak597obh+1vEwyk/lfeTZTE5aIacIHXzYSwA07Ry9ZtwYwoN+xJaFX2VkUc6Ocnf0s7qmH9wFwiohi3ZPQyYQfhQoHbV+kK8Loc/JrVgJI34hT1aaFgTkyAhUykeIudj0aFzpBJ5Hm7y6lPGEoIlH18RxYidjDOGpMZZ09NoVQSvjFLxNXe+02b2tlDg58ldEGAvSO+MuKuGfT4NfytzeWbaqgoK2sj0PAF1sIGfJxT+xow4wAoBZn28ymnZyGg5dYEy5pWeH8oEigR6p/epbdONV8mj8HaGsxHeWXtaXqBlK5IJO9gXIRxM/37yoYMIrslFp+X8ZxHTfBI4Z06wX073Bqdrn0vzKnd0aNGigdGklUau0/EBEB64QogxqCxJnwa1P7sNcUG/ElPotpPQpGuKA5F/GOA6so6mbS0d4TSTozfI/T/7veOoYJyY8H9lVN24F4e7Ebg8ofv8RkxDic3zHGg/vTry9QwcOQDrhm3Uxz3MVvu//kwLvdvOI0uFjpqVlRJJhUHNulsrtcA9y+PnMFRaDQU9oTTN//3OP9G2VSMnV3EgxUmfNFOFFqaJtJgm1kwvtQTESkkIPnkt/BNwGwMXqqugBp7PWuh8CoAy1Uu+7YmoiKsw7xP2k2yI/Ag8GzhbNscBwD/ZVI4M4kdb+l+bTpSu3MU1Er6i2pvNdcbniNdpVyku3Uea9lm0kX1uckA/zh49z7WSdClRG783m645nbVb757Rv5RKmQTLlYnykuV/TWDsmUJ9Jx5OvjdpbcpeT07yg0tOcNu1Z7pDfNjEpsrkqRPCAOsP6yIw49QSOLE10xZndOJE pQcV8KGp ElS3mwwskGcqh+XcmYY/l7tzkRHsSslRHuFG4dKSSHgh+iiblhLCYW5y2aBJoH2ff4uokg3jVD3BBXKm68IWFt+yi6AQewBSKrfGExJoBIRVwqVDpvxHF29AJ7G69am2I2ew+ZThPBj/DZrc9dojQFLhVbA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The filemap_map_pages() tries to map few pages(eg, 16 pages), but the lruvec stat updating is called on each mapping, since the updating is time-consuming, especially with memcg, so try to batch it when the memcg and pgdat are same during the mapping, if luckily, we could save most of time of lruvec stat updating, the lat_pagefault shows 3~4% improvement. Signed-off-by: Kefeng Wang --- mm/filemap.c | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 3966b6616d02..b27281707098 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3615,6 +3615,20 @@ static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf, return ret; } +static void filemap_lruvec_stat_update(struct mem_cgroup *memcg, + pg_data_t *pgdat, int nr) +{ + struct lruvec *lruvec; + + if (!memcg) { + __mod_node_page_state(pgdat, NR_FILE_MAPPED, nr); + return; + } + + lruvec = mem_cgroup_lruvec(memcg, pgdat); + __mod_lruvec_state(lruvec, NR_FILE_MAPPED, nr); +} + vm_fault_t filemap_map_pages(struct vm_fault *vmf, pgoff_t start_pgoff, pgoff_t end_pgoff) { @@ -3628,6 +3642,9 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, vm_fault_t ret = 0; unsigned long rss = 0; unsigned int nr_pages = 0, mmap_miss = 0, mmap_miss_saved, folio_type; + struct mem_cgroup *memcg, *memcg_cur; + pg_data_t *pgdat, *pgdat_cur; + int nr_mapped = 0; rcu_read_lock(); folio = next_uptodate_folio(&xas, mapping, end_pgoff); @@ -3648,9 +3665,20 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, } folio_type = mm_counter_file(folio); + memcg = folio_memcg(folio); + pgdat = folio_pgdat(folio); do { unsigned long end; - int nr_mapped = 0; + + memcg_cur = folio_memcg(folio); + pgdat_cur = folio_pgdat(folio); + + if (unlikely(memcg != memcg_cur || pgdat != pgdat_cur)) { + filemap_lruvec_stat_update(memcg, pgdat, nr_mapped); + nr_mapped = 0; + memcg = memcg_cur; + pgdat = pgdat_cur; + } addr += (xas.xa_index - last_pgoff) << PAGE_SHIFT; vmf->pte += xas.xa_index - last_pgoff; @@ -3668,11 +3696,10 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, nr_pages, &rss, &nr_mapped, &mmap_miss); - __lruvec_stat_mod_folio(folio, NR_FILE_MAPPED, nr_mapped); - folio_unlock(folio); folio_put(folio); } while ((folio = next_uptodate_folio(&xas, mapping, end_pgoff)) != NULL); + filemap_lruvec_stat_update(memcg, pgdat, nr_mapped); add_mm_counter(vma->vm_mm, folio_type, rss); pte_unmap_unlock(vmf->pte, vmf->ptl); out: