From patchwork Thu Apr 11 13:09:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13625904 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3E2ECD1299 for ; Thu, 11 Apr 2024 12:42:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3EFCE6B00A0; Thu, 11 Apr 2024 08:42:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 34FC76B00A1; Thu, 11 Apr 2024 08:42:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F1E96B00A2; Thu, 11 Apr 2024 08:42:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E62766B00A0 for ; Thu, 11 Apr 2024 08:42:46 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2823AA0965 for ; Thu, 11 Apr 2024 12:42:45 +0000 (UTC) X-FDA: 81997215090.10.BCDFA1F Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by imf18.hostedemail.com (Postfix) with ESMTP id BC0C01C0011 for ; Thu, 11 Apr 2024 12:42:42 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=none; spf=pass (imf18.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712839363; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jfwYyyZMwKC5C0L+TJvHhVQzAIU+60/qHaSiCHSyDDk=; b=p7c+oCykkXeHGdNDHj6zh8Qh+Vw1d4VWu+XgQYgCzvNCb+1SG6V2ckR/ZUN0hlE78Vmh/2 nWpf3/56cvJm0iyjFa6+zP5dyN8kC6dp/YqQYePqSVcm8QCRyJyl1YPQqv6sDp1KEe5tIg RVJB1hQG6AFNrgDOw8Tac0C4LqlzMOY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712839363; a=rsa-sha256; cv=none; b=6aK0eOuyDtWKup8KN1EXeIIPU2q7Y0nxm0wlh/876oCwIbMh5Y1pdHwXKsn+lbCRZrZi3u u73B8AWxNPvNW7wztB7BV144y9Uh3ATiZZzbW8BSBYx0XkMjhzLjDCSznvxqYRT9jfR688 4zTcHB+wWowc8Ir7W/jLFhX8J8R3Yh4= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=none; spf=pass (imf18.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com Received: from mail.maildlp.com (unknown [172.19.163.174]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4VFfTK1GXtzYd80; Thu, 11 Apr 2024 20:41:41 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id CFB3F140156; Thu, 11 Apr 2024 20:42:39 +0800 (CST) Received: from localhost.localdomain.localdomain (10.175.113.25) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Thu, 11 Apr 2024 20:42:39 +0800 From: Kefeng Wang To: Andrew Morton CC: "Matthew Wilcox (Oracle)" , , , Kefeng Wang Subject: [PATCH 2/2] mm: filemap: batch mm counter updating in filemap_map_pages() Date: Thu, 11 Apr 2024 21:09:50 +0800 Message-ID: <20240411130950.73512-3-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20240411130950.73512-1-wangkefeng.wang@huawei.com> References: <20240411130950.73512-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.113.25] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To dggpemm100001.china.huawei.com (7.185.36.93) X-Stat-Signature: u6ggsqpmexuukfhu5hewatxeyjtdp6p4 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: BC0C01C0011 X-Rspam-User: X-HE-Tag: 1712839362-722622 X-HE-Meta: U2FsdGVkX1/mKGotzRK0nabvow/m6Hxl5U/nbpuBZQ1YwlcqhoGjlpgVHQrm7sAKegZ1pkRvRX2DXcYQl6udB4X0LLUvFec677FhCyt7vatQwH+FeyA9DbX0sVmx6CMmgRAr/sGvAXHVDo2B/AyuPY1VsewGG4qkAyLAtVjcJvxj021TrDo6XxPK+9BUMu9RS37H644Lw242UDuYBy2e9pUtGWYq1XdXjDnh2giXuJhN4WLXVP5sNJNidwgWjfEDN68yB/T8oNjQML1qCZLg/nYE9cGmXfDqfKxZVFg37ULbEMAYdFGQkHtZVZdJXSDh7LOx8XfKEeJwbAGFMfX0SaPsVWwwydh72GLash/s+rIL6qktY8alulCgsRlatBHtEjS6/qaQy+K5KI//OQ31tZWnvOPKvHxwdyySEdS7Z6OmWl8zWORVYdBSBvy5wiyQRJQKfMA4JXuupWx6/oD4EvbS5kXMAcC9wo46t37VtD655oYyxRwjyu1S1dKdnZaKP9Kr90iCd283Bfmkm0aDuyZPE2xhVBKdku669aSxrUF6GdDNbvJ+u6FS4wCxTgSyjepbvrh9jTWo97xY3CP3UlgQeEpKPGlANHx5KjmZHhnVDDjMmTEpmhbhKFbdZQB99XjYt0pBZoHkZPj+CIMy85BAKIhRkWDH2SraedOAfcFsPxm+KyOyLmOB3hYkB67XASwUyySMZDVPaimj0T0Lrezb/Wzy+DijdvrDyVynR3RJTiCcfS+rcc6rJwzbCfbuAMjyuFA3evXVQzhvHXzMQ8K3sqR+QVguSoIIiuM34I49Z2VWrka+gugqTeanVhyxaeIA3BEFDwQTj+Vv3W8UnekCSQXHPAu2V/SOFCBFqNzjqtS0JPUlwz/wUZuWN8QsPPbkr9tu0r4WUhJ8VVlY86fq7RtRjAGepk6TuT5r3k/BV9u2QQJLq0/j7GiqLmJNFT9A1WNpzQuOlDAhlyJ YDHLO1GI JIdD0MNWVWUd5OQRaGc/ieWp2C/CfyazAhgsEdwfx9aJ0hJKimjkJfJRTye4DmKf/aHmiFErbjGloIdWvJ3sKkQy1P6lLBEfOVcIs27rDrefYtyPGFyaZOXRp5ep0rUzw0LYIXxAScjqJJ+kRHXQjz6u25Mi8Mau8RuYkW8JE3kA3z/QkTdre4mMkRCISBg3zyitS X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Like copy_pte_range()/zap_pte_range(), make mm counter batch updating in filemap_map_pages(), the 'lat_pagefault -P 1 file' test from lmbench shows 12% improve, and the percpu_counter_add_batch() is gone from perf flame graph. Signed-off-by: Kefeng Wang --- include/linux/mm.h | 14 ++++++++++++++ mm/filemap.c | 19 +++++++++++-------- mm/memory.c | 14 -------------- 3 files changed, 25 insertions(+), 22 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 6ad440ac3706..c7dffd358088 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2655,6 +2655,20 @@ static inline int mm_counter(struct folio *folio) return mm_counter_file(folio); } +static inline void init_rss_vec(int *rss) +{ + memset(rss, 0, sizeof(int) * NR_MM_COUNTERS); +} + +static inline void add_mm_rss_vec(struct mm_struct *mm, int *rss) +{ + int i; + + for (i = 0; i < NR_MM_COUNTERS; i++) + if (rss[i]) + add_mm_counter(mm, i, rss[i]); +} + static inline unsigned long get_mm_rss(struct mm_struct *mm) { return get_mm_counter(mm, MM_FILEPAGES) + diff --git a/mm/filemap.c b/mm/filemap.c index 2274e590bab4..d8b23e976a43 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3506,7 +3506,7 @@ static struct folio *next_uptodate_folio(struct xa_state *xas, static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, struct folio *folio, unsigned long start, unsigned long addr, unsigned int nr_pages, - unsigned int *mmap_miss) + int *rss, unsigned int *mmap_miss) { vm_fault_t ret = 0; struct page *page = folio_page(folio, start); @@ -3541,7 +3541,7 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, skip: if (count) { type = set_pte_range(vmf, folio, page, count, addr); - add_mm_counter(vmf->vma->vm_mm, type, count); + rss[type] += count; folio_ref_add(folio, count); if (in_range(vmf->address, addr, count * PAGE_SIZE)) ret = VM_FAULT_NOPAGE; @@ -3556,7 +3556,7 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, if (count) { type = set_pte_range(vmf, folio, page, count, addr); - add_mm_counter(vmf->vma->vm_mm, type, count); + rss[type] += count; folio_ref_add(folio, count); if (in_range(vmf->address, addr, count * PAGE_SIZE)) ret = VM_FAULT_NOPAGE; @@ -3569,7 +3569,7 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf, struct folio *folio, unsigned long addr, - unsigned int *mmap_miss) + int *rss, unsigned int *mmap_miss) { vm_fault_t ret = 0; struct page *page = &folio->page; @@ -3592,8 +3592,7 @@ static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf, if (vmf->address == addr) ret = VM_FAULT_NOPAGE; - add_mm_counter(vmf->vma->vm_mm, - set_pte_range(vmf, folio, page, 1, addr), 1); + rss[set_pte_range(vmf, folio, page, 1, addr)]++; folio_ref_inc(folio); return ret; @@ -3610,6 +3609,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, XA_STATE(xas, &mapping->i_pages, start_pgoff); struct folio *folio; vm_fault_t ret = 0; + int rss[NR_MM_COUNTERS]; unsigned int nr_pages = 0, mmap_miss = 0, mmap_miss_saved; rcu_read_lock(); @@ -3629,6 +3629,8 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, folio_put(folio); goto out; } + + init_rss_vec(rss); do { unsigned long end; @@ -3640,15 +3642,16 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, if (!folio_test_large(folio)) ret |= filemap_map_order0_folio(vmf, - folio, addr, &mmap_miss); + folio, addr, rss, &mmap_miss); else ret |= filemap_map_folio_range(vmf, folio, xas.xa_index - folio->index, addr, - nr_pages, &mmap_miss); + nr_pages, rss, &mmap_miss); folio_unlock(folio); folio_put(folio); } while ((folio = next_uptodate_folio(&xas, mapping, end_pgoff)) != NULL); + add_mm_rss_vec(vma->vm_mm, rss); pte_unmap_unlock(vmf->pte, vmf->ptl); out: rcu_read_unlock(); diff --git a/mm/memory.c b/mm/memory.c index 485ffec9d4c7..149208da1652 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -465,20 +465,6 @@ int __pte_alloc_kernel(pmd_t *pmd) return 0; } -static inline void init_rss_vec(int *rss) -{ - memset(rss, 0, sizeof(int) * NR_MM_COUNTERS); -} - -static inline void add_mm_rss_vec(struct mm_struct *mm, int *rss) -{ - int i; - - for (i = 0; i < NR_MM_COUNTERS; i++) - if (rss[i]) - add_mm_counter(mm, i, rss[i]); -} - /* * This function is called to print an error when a bad pte * is found. For example, we might have a PFN-mapped pte in