From patchwork Fri Apr 12 02:57:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13626704 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B3F4C4345F for ; Fri, 12 Apr 2024 02:30:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ED6846B009A; Thu, 11 Apr 2024 22:30:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D36D86B009E; Thu, 11 Apr 2024 22:30:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9169A6B009A; Thu, 11 Apr 2024 22:30:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 743EA6B009B for ; Thu, 11 Apr 2024 22:30:04 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 365B3A1802 for ; Fri, 12 Apr 2024 02:30:04 +0000 (UTC) X-FDA: 81999299928.23.2D5B6C1 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by imf03.hostedemail.com (Postfix) with ESMTP id 5481D20003 for ; Fri, 12 Apr 2024 02:30:00 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712889002; a=rsa-sha256; cv=none; b=i8WNvy/HLL7sCSZTzRPN2nU4ikZ6kZ7mQYvQGlUyNiVedrm/TWrwsOw6IvAg1Vi0AqVyGI duvVJLWsbp6fmFR/sSaU+CtxrVgJJBnoPCV1sWEC+jyJDwOLjEFWZR5D244zzLMyOCpfdr tgeGzeZs5KNNlb0jmneQl3bT2aLcxmI= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712889002; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aEFWBT5ZydaCUZYDh0NjrgQAWYDm5eJtG9qKSnoZ+lI=; b=NCJR7U4makquDNSzfnRFhywBbS5BNG1u2YssSvwtJS5BaP7iamHg+6SRIjZM9606Q7UV73 r71px+dyVd87fqYTW3RuBcsXCzcyyDtDTo/axqlbtJ92MEy7smiFmA1G0sXfT5Kq5YBYg9 gyj/cqDs70UM4vTH3ptU2Kw3dWs67fY= Received: from mail.maildlp.com (unknown [172.19.88.194]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4VG0nc1L0vzwRwS; Fri, 12 Apr 2024 10:27:00 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id 77832140427; Fri, 12 Apr 2024 10:29:57 +0800 (CST) Received: from localhost.localdomain.localdomain (10.175.113.25) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Fri, 12 Apr 2024 10:29:57 +0800 From: Kefeng Wang To: Andrew Morton CC: "Matthew Wilcox (Oracle)" , , , Kefeng Wang Subject: [PATCH 2/2] mm: filemap: batch mm counter updating in filemap_map_pages() Date: Fri, 12 Apr 2024 10:57:04 +0800 Message-ID: <20240412025704.53245-3-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20240412025704.53245-1-wangkefeng.wang@huawei.com> References: <20240412025704.53245-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.113.25] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To dggpemm100001.china.huawei.com (7.185.36.93) X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 5481D20003 X-Stat-Signature: qcu3cyaczmr81bp4psf87n581xf44w43 X-HE-Tag: 1712889000-890156 X-HE-Meta: U2FsdGVkX1/SCx0LY75BVC1gceFUI2m7NRRk1SYT7VwSUcn1j39syW+DgYyBKQCLgnxkFocovAFkvazc9D48yTCvmYbKRCYjp3Wave4TIPfYozjtU2kSPTIB3vzvKdMBoNFa1cOy8KLm4nvgmuiaxIkPGcqMUAFYLspShEBWnRuKfX9jUNSNOjwPuHKLh0hPZQhvc4Vgs+PQ9M88QQvAkktghA8e28BrtzxFr1PCqDDNoxntvg5lnnXvXTH1znlG7skWw9/cHfASCRNXgxQBG1obp5bgqfvXNCRm4rzH53WZ06b/y9cujnG5+gKoctIoiZ+q73R2msqndKz4oI3tIjsCI7SzUp5GmCA4NjTIC/A/loX48Cdsd3Pl5hh7TEVkE0lj6f6i8fobeNtbAjMUsx3HGShmT3bkz5wzML/QAgNtl+fYWNksvUFEH1J0juTo3d/EDbL8K6LLNUEYYpa9g9AtFs/7dOdkZXTBrg2uDxbbokznJY+ipDiHpl0byEaLMoFpx+0LNrXd250jLaL04/8ERgLb7wqel7H40aMy8iuMZ8McRAeKRuh9g7OyVbRSmqdvPV5c24cxh3ajzBdI202JeNeKTiHorBbjwiRveQOf+KRllx7ilqzb4ZGParZ3fKC6y6gTQoGiiPRVC0ixA9UkvuSy4IqWPnDj1q9ZcQ11XzRBIvhB4snDhldHgPHFxNoKjHh2EGmZlheZNZeBPKVjVi1CuP/DCLDy4uhT6MHD/lVVkmmBVP4ypaYqM3ixMFkq1k1vdVICG0W2jHs0nqLn24AM5eczkaYz4WFHJ5TnmEguyrpUreArurEg1v4tMd6M8v0v+0Sh9zcZwD+2tmnSb/GOc0yxNZPyhuteBNW/vXQS/tQrSB9h5jcod7qUTNTWGczfkq/00Iuy0MbqssPIyYiqNf9aDgURP3Aoe3F708gWW4kJ0xGys2e53S7kXfHBpDrZ66LdsLjj7kj JQ2FSma4 fZ2iYlw18XbdKkAwGefHkZQccjn7FaG6BNDsh8BnWeuyaT2qtjmXVi5JveC6jV+29evPB9fyL+S4WMChoAmtivkYeJIQrHaUMsNCfP4VysRqMTn8sD25hp07L2DW1+6leXuLh1slBz3yOM+ARK0omQjIWx51uF7MCo10tmfveCRrKGHmoM6M30iLi71WmkTi1K127 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Like copy_pte_range()/zap_pte_range(), make mm counter batch updating in filemap_map_pages(), the 'lat_pagefault -P 1 file' test from lmbench shows 12% improvement, and the percpu_counter_add_batch() is gone from perf flame graph. Signed-off-by: Kefeng Wang --- mm/filemap.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 04b813f0146c..c8d41ab5034b 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3506,7 +3506,7 @@ static struct folio *next_uptodate_folio(struct xa_state *xas, static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, struct folio *folio, unsigned long start, unsigned long addr, unsigned int nr_pages, - unsigned int *mmap_miss) + unsigned long *rss, unsigned int *mmap_miss) { vm_fault_t ret = 0; struct page *page = folio_page(folio, start); @@ -3540,8 +3540,7 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, skip: if (count) { set_pte_range(vmf, folio, page, count, addr); - add_mm_counter(vmf->vma->vm_mm, mm_counter_file(folio), - count); + *rss += count; folio_ref_add(folio, count); if (in_range(vmf->address, addr, count * PAGE_SIZE)) ret = VM_FAULT_NOPAGE; @@ -3556,7 +3555,7 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, if (count) { set_pte_range(vmf, folio, page, count, addr); - add_mm_counter(vmf->vma->vm_mm, mm_counter_file(folio), count); + *rss += count; folio_ref_add(folio, count); if (in_range(vmf->address, addr, count * PAGE_SIZE)) ret = VM_FAULT_NOPAGE; @@ -3569,7 +3568,7 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf, struct folio *folio, unsigned long addr, - unsigned int *mmap_miss) + unsigned long *rss, unsigned int *mmap_miss) { vm_fault_t ret = 0; struct page *page = &folio->page; @@ -3593,7 +3592,7 @@ static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf, ret = VM_FAULT_NOPAGE; set_pte_range(vmf, folio, page, 1, addr); - add_mm_counter(vmf->vma->vm_mm, mm_counter_file(folio), 1); + (*rss)++; folio_ref_inc(folio); return ret; @@ -3610,6 +3609,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, XA_STATE(xas, &mapping->i_pages, start_pgoff); struct folio *folio; vm_fault_t ret = 0; + unsigned long rss = 0; unsigned int nr_pages = 0, mmap_miss = 0, mmap_miss_saved; rcu_read_lock(); @@ -3640,15 +3640,17 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, if (!folio_test_large(folio)) ret |= filemap_map_order0_folio(vmf, - folio, addr, &mmap_miss); + folio, addr, &rss, &mmap_miss); else ret |= filemap_map_folio_range(vmf, folio, xas.xa_index - folio->index, addr, - nr_pages, &mmap_miss); + nr_pages, &rss, &mmap_miss); folio_unlock(folio); folio_put(folio); } while ((folio = next_uptodate_folio(&xas, mapping, end_pgoff)) != NULL); + + add_mm_counter(vma->vm_mm, mm_counter_file(folio), rss); pte_unmap_unlock(vmf->pte, vmf->ptl); out: rcu_read_unlock();