From patchwork Mon Apr 29 07:24:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13646411 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53676C4345F for ; Mon, 29 Apr 2024 07:24:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D43046B0088; Mon, 29 Apr 2024 03:24:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BA1E26B0092; Mon, 29 Apr 2024 03:24:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A62A6B0088; Mon, 29 Apr 2024 03:24:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 5416D6B0089 for ; Mon, 29 Apr 2024 03:24:36 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id F40C0A29E5 for ; Mon, 29 Apr 2024 07:24:35 +0000 (UTC) X-FDA: 82061731710.08.B2EF83F Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by imf03.hostedemail.com (Postfix) with ESMTP id 8808D20020 for ; Mon, 29 Apr 2024 07:24:33 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714375474; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P83nek/83zfmOjfgmemVghMYO55C7SajunYhkBe1QLM=; b=QERIn0i9gZ+59zYlSSMptgQUyyRL4hIPbNNsVYteRqWKZlVCYp1ECv7TkCl9OmpG2pnP6+ K9x2XxRTxpR1Gh/OHo+Oy/CHetcxmGLvw2ygL1rERR9x4nApxA5CEXufkQlTxuJMod/wam ElIFpcKPQqHm6N45I5uSlivKHH5pklw= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714375474; a=rsa-sha256; cv=none; b=5xP4Vf/Jmb6xSpmaxBBFefs7DBWRPDQoHJvnC0xgBbQ/AH+RZ+4KxJfA6gqCwuIroUxrvT yUbf0zwb0BsjphkLmAjbO3PrWGBV+wpSLvDoMjms2grjVlFvdM5q4EKQnL+1QudF5P9cf+ 1hq3V6FfjqgSb2juRWgXlTdULoKfOdQ= Received: from mail.maildlp.com (unknown [172.19.88.194]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4VSZYl0X6Pzcb0R; Mon, 29 Apr 2024 15:23:23 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id E866E14035F; Mon, 29 Apr 2024 15:24:29 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Mon, 29 Apr 2024 15:24:29 +0800 From: Kefeng Wang To: Andrew Morton CC: "Matthew Wilcox (Oracle)" , , , Kefeng Wang Subject: [PATCH rfc 4/4] mm: filemap: try to batch lruvec stat updating Date: Mon, 29 Apr 2024 15:24:17 +0800 Message-ID: <20240429072417.2146732-5-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240429072417.2146732-1-wangkefeng.wang@huawei.com> References: <20240429072417.2146732-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To dggpemm100001.china.huawei.com (7.185.36.93) X-Rspam-User: X-Rspamd-Queue-Id: 8808D20020 X-Rspamd-Server: rspam06 X-Stat-Signature: ojzord19o1bq9nkit7jbot1rkjejkgcm X-HE-Tag: 1714375473-9421 X-HE-Meta: U2FsdGVkX198GpLNAp4qswTWBpa4GGczhfmWqwJjvWoi4pACa0qpWKgEA6MFHbE9JJ+S68ukXLJNI+UsvxyczQVvOZECCik9ism3UjH0vP2QCWGDJ8OJ67GhdaDEOiC6f4wkKAUwIQcNXe6MwtRkjUnak597obh+1vEwyk/lfeTZTE5aIacIHXzYSwA07Ry9ZtwYwoN+xJaFX2VkUc6Ocnf0s7qmH9wFwiohi3ZPQyYQfhQoHbV+kK8Loc/JrVgJI34hT1aaFgTkyAhUykeIudj0aFzpBJ5Hm7y6lPGEoIlH18RxYidjDOGpMZZ09NoVQSvjFLxNXe+02b2tlDg58ldEGAvSO+MuKuGfT4NfytzeWbaqgoK2sj0PAF1sIGfJxT+xow4wAoBZn28ymnZyGg5dYEy5pWeH8oEigR6p/epbdONV8mj8HaGsxHeWXtaXqBlK5IJO9gXIRxM/37yoYMIrslFp+X8ZxHTfBI4Z06wX073Bqdrn0vzKnd0aNGigdGklUau0/EBEB64QogxqCxJnwa1P7sNcUG/ElPotpPQpGuKA5F/GOA6so6mbS0d4TSTozfI/T/7veOoYJyY8H9lVN24F4e7Ebg8ofv8RkxDic3zHGg/vTry9QwcOQDrhm3Uxz3MVvu//kwLvdvOI0uFjpqVlRJJhUHNulsrtcA9y+PnMFRaDQU9oTTN//3OP9G2VSMnV3EgxUmfNFOFFqaJtJgm1kwvtQTESkkIPnkt/BNwGwMXqqugBp7PWuh8CoAy1Uu+7YmoiKsw7xP2k2yI/Ag8GzhbNscBwD/ZVI4M4kdb+l+bTpSu3MU1Er6i2pvNdcbniNdpVyku3Uea9lm0kX1uckA/zh49z7WSdClRG783m645nbVb757Rv5RKmQTLlYnykuV/TWDsmUJ9Jx5OvjdpbcpeT07yg0tOcNu1Z7pDfNjEpsrkqRPCAOsP6yIw49QSOLE10xZndOJE pQcV8KGp ElS3mwwskGcqh+XcmYY/l7tzkRHsSslRHuFG4dKSSHgh+iiblhLCYW5y2aBJoH2ff4uokg3jVD3BBXKm68IWFt+yi6AQewBSKrfGExJoBIRVwqVDpvxHF29AJ7G69am2I2ew+ZThPBj/DZrc9dojQFLhVbA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The filemap_map_pages() tries to map few pages(eg, 16 pages), but the lruvec stat updating is called on each mapping, since the updating is time-consuming, especially with memcg, so try to batch it when the memcg and pgdat are same during the mapping, if luckily, we could save most of time of lruvec stat updating, the lat_pagefault shows 3~4% improvement. Signed-off-by: Kefeng Wang --- mm/filemap.c | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 3966b6616d02..b27281707098 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3615,6 +3615,20 @@ static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf, return ret; } +static void filemap_lruvec_stat_update(struct mem_cgroup *memcg, + pg_data_t *pgdat, int nr) +{ + struct lruvec *lruvec; + + if (!memcg) { + __mod_node_page_state(pgdat, NR_FILE_MAPPED, nr); + return; + } + + lruvec = mem_cgroup_lruvec(memcg, pgdat); + __mod_lruvec_state(lruvec, NR_FILE_MAPPED, nr); +} + vm_fault_t filemap_map_pages(struct vm_fault *vmf, pgoff_t start_pgoff, pgoff_t end_pgoff) { @@ -3628,6 +3642,9 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, vm_fault_t ret = 0; unsigned long rss = 0; unsigned int nr_pages = 0, mmap_miss = 0, mmap_miss_saved, folio_type; + struct mem_cgroup *memcg, *memcg_cur; + pg_data_t *pgdat, *pgdat_cur; + int nr_mapped = 0; rcu_read_lock(); folio = next_uptodate_folio(&xas, mapping, end_pgoff); @@ -3648,9 +3665,20 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, } folio_type = mm_counter_file(folio); + memcg = folio_memcg(folio); + pgdat = folio_pgdat(folio); do { unsigned long end; - int nr_mapped = 0; + + memcg_cur = folio_memcg(folio); + pgdat_cur = folio_pgdat(folio); + + if (unlikely(memcg != memcg_cur || pgdat != pgdat_cur)) { + filemap_lruvec_stat_update(memcg, pgdat, nr_mapped); + nr_mapped = 0; + memcg = memcg_cur; + pgdat = pgdat_cur; + } addr += (xas.xa_index - last_pgoff) << PAGE_SHIFT; vmf->pte += xas.xa_index - last_pgoff; @@ -3668,11 +3696,10 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, nr_pages, &rss, &nr_mapped, &mmap_miss); - __lruvec_stat_mod_folio(folio, NR_FILE_MAPPED, nr_mapped); - folio_unlock(folio); folio_put(folio); } while ((folio = next_uptodate_folio(&xas, mapping, end_pgoff)) != NULL); + filemap_lruvec_stat_update(memcg, pgdat, nr_mapped); add_mm_counter(vma->vm_mm, folio_type, rss); pte_unmap_unlock(vmf->pte, vmf->ptl); out: