From patchwork Fri Apr 12 02:57:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13626703 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5ADA2C001CC for ; Fri, 12 Apr 2024 02:30:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C093D6B009C; Thu, 11 Apr 2024 22:30:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B11F46B009B; Thu, 11 Apr 2024 22:30:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7BFE56B009C; Thu, 11 Apr 2024 22:30:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 50E4D6B009A for ; Thu, 11 Apr 2024 22:30:04 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E195C1C0F9C for ; Fri, 12 Apr 2024 02:30:03 +0000 (UTC) X-FDA: 81999299886.03.F9B6F41 Received: from szxga04-in.huawei.com (szxga04-in.huawei.com [45.249.212.190]) by imf05.hostedemail.com (Postfix) with ESMTP id C92C3100007 for ; Fri, 12 Apr 2024 02:30:00 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=none; spf=pass (imf05.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.190 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712889002; a=rsa-sha256; cv=none; b=RY2F2w6PdomrQRiEPfcGJzuS+yfXKyLfy/uTTOmd3rl8FHhC/svohUDVRP4ddDTWTTkKMN WjZeZ+YXMuxrmb2JLILy6wArxNBC23QPbpcKkhD6lfv2dhF1iKt+nxmhtPTs/pg3+dhodI AJIj+hia5hrv1uiU3PD8mcTQJpmR7n8= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=none; spf=pass (imf05.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.190 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712889002; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9KbkjvmuALWONra0TJwyRfZFDCz2BiUAn7df+x0JPP4=; b=xJDjSorF0gNIRxVxF9z5LunUCFIE1p8bn8wS7mZ5gt6VQZ/2f3oCPjd0PMKHctO200Sjie EWQQveRpP7vbv5ehhiMH25Xy7e28IhS7LvDGxNilZVoRbUau2epYOzZdWLatkfh5AR1Woi nidxaVdqtU9L8fApUbRYKAVoZoCPNN8= Received: from mail.maildlp.com (unknown [172.19.163.44]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4VG0pM4JXZz1ynHs; Fri, 12 Apr 2024 10:27:39 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id 19E5B140410; Fri, 12 Apr 2024 10:29:57 +0800 (CST) Received: from localhost.localdomain.localdomain (10.175.113.25) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Fri, 12 Apr 2024 10:29:56 +0800 From: Kefeng Wang To: Andrew Morton CC: "Matthew Wilcox (Oracle)" , , , Kefeng Wang Subject: [PATCH 1/2] mm: move mm counter updating out of set_pte_range() Date: Fri, 12 Apr 2024 10:57:03 +0800 Message-ID: <20240412025704.53245-2-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20240412025704.53245-1-wangkefeng.wang@huawei.com> References: <20240412025704.53245-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.113.25] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To dggpemm100001.china.huawei.com (7.185.36.93) X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: C92C3100007 X-Stat-Signature: 4fppbhm9rt8dwn5zq9576awofk6f8kh4 X-HE-Tag: 1712889000-406820 X-HE-Meta: U2FsdGVkX1/YoqW++ugU7tUHOo4T+Yt9Xh1yJ8wecMhhfBf/BvEppnY6q6/zP/REjM2jUr5CxteWtI75R4HhvRK10UzGEYAUcxA9Rp71E513DezR7MkM0GddiZL8xSSueUFin0VzozkfN9JV3ttCK7W6k0zGuonYg9q/QN08eLvVbZyr3UBeRx0H4WcLJKHJ7g+vsAqONE3KI6kHSy8u3SZZaGqQu6HXKg8Z5Dxr3cBliHDzDb9C/wfPQ5qXeoK9JGfmM1Kt5RLx35KKqklpCYtkQ7EAUQgTWdQcqK088Uj0HWeJjVlQZ1VMfgZO55Tw1okqCZ3t6BbvnOFiZsMWMmOwdDNiXNbLiYleH1iYG2YWf0NAcXXdu4o1S04wd+Bbyf4YZf3C71RUK2RTjDGBtVu6SrIR0jJPMCrlhQ0doFtV3O8sZ4Q3TlHMvg/mwtMHTPA68t8hL8qORiY40/tHYDUve8z2F+DQ9dSFLmE4VXXBOkZYT/qavyvkwjpDMo3c4JsWaYCx1PMINdlm0GkUDE3F/4Y3Sx6vedIYYd79qcYgOYL2IuhenvmVqhYhC8K9t+iJ1ab4vlpYr6SKRdQ2+WnDDmrBD3CisQ0zbTB4mzBmyuoa7dMtu5kBeXpSIJG7FM/Zz6mEFcqRLa5WLUi3bBbzokeNYVUmLia2yoJpYva8nFYsi4G2MPAJuJqyDfDkpKaBLAexTVfZGTRhjMvAbTSwERd2LiHmST6zjoEDnjI8lTTnV/MX+H0HugUnLBEUwmKB1KeTc1ge2Es+ACdFpsvhQBAW9S6/SwsbZoARKiirts/dRd5lbCVFxxapxCPO75lg29sEYkXj0vhcwKqGR4DfCJAA6MoMS96RTMJeyckzxeRSl4oUnIVC1IfqxIryu7/ZaiGD2P8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In order to support batch mm counter updating in filemap_map_pages(), move mm counter updating out of set_pte_range(), the folios are file from filemap, and distinguish folios type by vmf->flags and vma->vm_flags from another caller finish_fault(). Signed-off-by: Kefeng Wang --- mm/filemap.c | 4 ++++ mm/memory.c | 8 +++++--- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 92e2d43e4c9d..04b813f0146c 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3540,6 +3540,8 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, skip: if (count) { set_pte_range(vmf, folio, page, count, addr); + add_mm_counter(vmf->vma->vm_mm, mm_counter_file(folio), + count); folio_ref_add(folio, count); if (in_range(vmf->address, addr, count * PAGE_SIZE)) ret = VM_FAULT_NOPAGE; @@ -3554,6 +3556,7 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, if (count) { set_pte_range(vmf, folio, page, count, addr); + add_mm_counter(vmf->vma->vm_mm, mm_counter_file(folio), count); folio_ref_add(folio, count); if (in_range(vmf->address, addr, count * PAGE_SIZE)) ret = VM_FAULT_NOPAGE; @@ -3590,6 +3593,7 @@ static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf, ret = VM_FAULT_NOPAGE; set_pte_range(vmf, folio, page, 1, addr); + add_mm_counter(vmf->vma->vm_mm, mm_counter_file(folio), 1); folio_ref_inc(folio); return ret; diff --git a/mm/memory.c b/mm/memory.c index 78422d1c7381..69bc63a5d6c8 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4685,12 +4685,10 @@ void set_pte_range(struct vm_fault *vmf, struct folio *folio, entry = pte_mkuffd_wp(entry); /* copy-on-write page */ if (write && !(vma->vm_flags & VM_SHARED)) { - add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr); VM_BUG_ON_FOLIO(nr != 1, folio); folio_add_new_anon_rmap(folio, vma, addr); folio_add_lru_vma(folio, vma); } else { - add_mm_counter(vma->vm_mm, mm_counter_file(folio), nr); folio_add_file_rmap_ptes(folio, page, nr, vma); } set_ptes(vma->vm_mm, addr, vmf->pte, entry, nr); @@ -4727,9 +4725,11 @@ vm_fault_t finish_fault(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; struct page *page; vm_fault_t ret; + int is_cow = (vmf->flags & FAULT_FLAG_WRITE) && + !(vma->vm_flags & VM_SHARED); /* Did we COW the page? */ - if ((vmf->flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) + if (is_cow) page = vmf->cow_page; else page = vmf->page; @@ -4765,8 +4765,10 @@ vm_fault_t finish_fault(struct vm_fault *vmf) /* Re-check under ptl */ if (likely(!vmf_pte_changed(vmf))) { struct folio *folio = page_folio(page); + int type = is_cow ? MM_ANONPAGES : mm_counter_file(folio); set_pte_range(vmf, folio, page, 1, vmf->address); + add_mm_counter(vma->vm_mm, type, 1); ret = 0; } else { update_mmu_tlb(vma, vmf->address, vmf->pte); From patchwork Fri Apr 12 02:57:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13626704 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B3F4C4345F for ; Fri, 12 Apr 2024 02:30:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ED6846B009A; Thu, 11 Apr 2024 22:30:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D36D86B009E; Thu, 11 Apr 2024 22:30:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9169A6B009A; Thu, 11 Apr 2024 22:30:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 743EA6B009B for ; Thu, 11 Apr 2024 22:30:04 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 365B3A1802 for ; Fri, 12 Apr 2024 02:30:04 +0000 (UTC) X-FDA: 81999299928.23.2D5B6C1 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by imf03.hostedemail.com (Postfix) with ESMTP id 5481D20003 for ; Fri, 12 Apr 2024 02:30:00 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712889002; a=rsa-sha256; cv=none; b=i8WNvy/HLL7sCSZTzRPN2nU4ikZ6kZ7mQYvQGlUyNiVedrm/TWrwsOw6IvAg1Vi0AqVyGI duvVJLWsbp6fmFR/sSaU+CtxrVgJJBnoPCV1sWEC+jyJDwOLjEFWZR5D244zzLMyOCpfdr tgeGzeZs5KNNlb0jmneQl3bT2aLcxmI= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712889002; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aEFWBT5ZydaCUZYDh0NjrgQAWYDm5eJtG9qKSnoZ+lI=; b=NCJR7U4makquDNSzfnRFhywBbS5BNG1u2YssSvwtJS5BaP7iamHg+6SRIjZM9606Q7UV73 r71px+dyVd87fqYTW3RuBcsXCzcyyDtDTo/axqlbtJ92MEy7smiFmA1G0sXfT5Kq5YBYg9 gyj/cqDs70UM4vTH3ptU2Kw3dWs67fY= Received: from mail.maildlp.com (unknown [172.19.88.194]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4VG0nc1L0vzwRwS; Fri, 12 Apr 2024 10:27:00 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id 77832140427; Fri, 12 Apr 2024 10:29:57 +0800 (CST) Received: from localhost.localdomain.localdomain (10.175.113.25) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Fri, 12 Apr 2024 10:29:57 +0800 From: Kefeng Wang To: Andrew Morton CC: "Matthew Wilcox (Oracle)" , , , Kefeng Wang Subject: [PATCH 2/2] mm: filemap: batch mm counter updating in filemap_map_pages() Date: Fri, 12 Apr 2024 10:57:04 +0800 Message-ID: <20240412025704.53245-3-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20240412025704.53245-1-wangkefeng.wang@huawei.com> References: <20240412025704.53245-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.113.25] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To dggpemm100001.china.huawei.com (7.185.36.93) X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 5481D20003 X-Stat-Signature: qcu3cyaczmr81bp4psf87n581xf44w43 X-HE-Tag: 1712889000-890156 X-HE-Meta: U2FsdGVkX1/SCx0LY75BVC1gceFUI2m7NRRk1SYT7VwSUcn1j39syW+DgYyBKQCLgnxkFocovAFkvazc9D48yTCvmYbKRCYjp3Wave4TIPfYozjtU2kSPTIB3vzvKdMBoNFa1cOy8KLm4nvgmuiaxIkPGcqMUAFYLspShEBWnRuKfX9jUNSNOjwPuHKLh0hPZQhvc4Vgs+PQ9M88QQvAkktghA8e28BrtzxFr1PCqDDNoxntvg5lnnXvXTH1znlG7skWw9/cHfASCRNXgxQBG1obp5bgqfvXNCRm4rzH53WZ06b/y9cujnG5+gKoctIoiZ+q73R2msqndKz4oI3tIjsCI7SzUp5GmCA4NjTIC/A/loX48Cdsd3Pl5hh7TEVkE0lj6f6i8fobeNtbAjMUsx3HGShmT3bkz5wzML/QAgNtl+fYWNksvUFEH1J0juTo3d/EDbL8K6LLNUEYYpa9g9AtFs/7dOdkZXTBrg2uDxbbokznJY+ipDiHpl0byEaLMoFpx+0LNrXd250jLaL04/8ERgLb7wqel7H40aMy8iuMZ8McRAeKRuh9g7OyVbRSmqdvPV5c24cxh3ajzBdI202JeNeKTiHorBbjwiRveQOf+KRllx7ilqzb4ZGParZ3fKC6y6gTQoGiiPRVC0ixA9UkvuSy4IqWPnDj1q9ZcQ11XzRBIvhB4snDhldHgPHFxNoKjHh2EGmZlheZNZeBPKVjVi1CuP/DCLDy4uhT6MHD/lVVkmmBVP4ypaYqM3ixMFkq1k1vdVICG0W2jHs0nqLn24AM5eczkaYz4WFHJ5TnmEguyrpUreArurEg1v4tMd6M8v0v+0Sh9zcZwD+2tmnSb/GOc0yxNZPyhuteBNW/vXQS/tQrSB9h5jcod7qUTNTWGczfkq/00Iuy0MbqssPIyYiqNf9aDgURP3Aoe3F708gWW4kJ0xGys2e53S7kXfHBpDrZ66LdsLjj7kj JQ2FSma4 fZ2iYlw18XbdKkAwGefHkZQccjn7FaG6BNDsh8BnWeuyaT2qtjmXVi5JveC6jV+29evPB9fyL+S4WMChoAmtivkYeJIQrHaUMsNCfP4VysRqMTn8sD25hp07L2DW1+6leXuLh1slBz3yOM+ARK0omQjIWx51uF7MCo10tmfveCRrKGHmoM6M30iLi71WmkTi1K127 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Like copy_pte_range()/zap_pte_range(), make mm counter batch updating in filemap_map_pages(), the 'lat_pagefault -P 1 file' test from lmbench shows 12% improvement, and the percpu_counter_add_batch() is gone from perf flame graph. Signed-off-by: Kefeng Wang --- mm/filemap.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 04b813f0146c..c8d41ab5034b 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3506,7 +3506,7 @@ static struct folio *next_uptodate_folio(struct xa_state *xas, static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, struct folio *folio, unsigned long start, unsigned long addr, unsigned int nr_pages, - unsigned int *mmap_miss) + unsigned long *rss, unsigned int *mmap_miss) { vm_fault_t ret = 0; struct page *page = folio_page(folio, start); @@ -3540,8 +3540,7 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, skip: if (count) { set_pte_range(vmf, folio, page, count, addr); - add_mm_counter(vmf->vma->vm_mm, mm_counter_file(folio), - count); + *rss += count; folio_ref_add(folio, count); if (in_range(vmf->address, addr, count * PAGE_SIZE)) ret = VM_FAULT_NOPAGE; @@ -3556,7 +3555,7 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, if (count) { set_pte_range(vmf, folio, page, count, addr); - add_mm_counter(vmf->vma->vm_mm, mm_counter_file(folio), count); + *rss += count; folio_ref_add(folio, count); if (in_range(vmf->address, addr, count * PAGE_SIZE)) ret = VM_FAULT_NOPAGE; @@ -3569,7 +3568,7 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf, struct folio *folio, unsigned long addr, - unsigned int *mmap_miss) + unsigned long *rss, unsigned int *mmap_miss) { vm_fault_t ret = 0; struct page *page = &folio->page; @@ -3593,7 +3592,7 @@ static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf, ret = VM_FAULT_NOPAGE; set_pte_range(vmf, folio, page, 1, addr); - add_mm_counter(vmf->vma->vm_mm, mm_counter_file(folio), 1); + (*rss)++; folio_ref_inc(folio); return ret; @@ -3610,6 +3609,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, XA_STATE(xas, &mapping->i_pages, start_pgoff); struct folio *folio; vm_fault_t ret = 0; + unsigned long rss = 0; unsigned int nr_pages = 0, mmap_miss = 0, mmap_miss_saved; rcu_read_lock(); @@ -3640,15 +3640,17 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, if (!folio_test_large(folio)) ret |= filemap_map_order0_folio(vmf, - folio, addr, &mmap_miss); + folio, addr, &rss, &mmap_miss); else ret |= filemap_map_folio_range(vmf, folio, xas.xa_index - folio->index, addr, - nr_pages, &mmap_miss); + nr_pages, &rss, &mmap_miss); folio_unlock(folio); folio_put(folio); } while ((folio = next_uptodate_folio(&xas, mapping, end_pgoff)) != NULL); + + add_mm_counter(vma->vm_mm, mm_counter_file(folio), rss); pte_unmap_unlock(vmf->pte, vmf->ptl); out: rcu_read_unlock();