From patchwork Mon Feb 6 14:06:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13129998 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E922EC05027 for ; Mon, 6 Feb 2023 14:08:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 56E766B0078; Mon, 6 Feb 2023 09:08:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 51E4B6B007B; Mon, 6 Feb 2023 09:08:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 320586B007D; Mon, 6 Feb 2023 09:08:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1121E6B0078 for ; Mon, 6 Feb 2023 09:08:40 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8B38FA8F83 for ; Mon, 6 Feb 2023 14:08:39 +0000 (UTC) X-FDA: 80437047558.25.83A3D58 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by imf22.hostedemail.com (Postfix) with ESMTP id 59141C0022 for ; Mon, 6 Feb 2023 14:08:37 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=YAxb32ob; spf=pass (imf22.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675692517; a=rsa-sha256; cv=none; b=GkreBIRpCWEiswPZ1VnTDhJ35h9wqBc764JVKmTjpgbIJEZNmmbIvG0dW45Ly67sI7oiA1 g1OG/9/fPHaKX529fpXOW5YRd7GwIR3i4whJXvemNPF/ddUL+RhG+tCKX/TZ+41rYNwSxv hhxdBPnFI4/wRlbCXRZ66ZU4O0CMJss= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=YAxb32ob; spf=pass (imf22.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675692517; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FnDsIyq2O3ERrsvZTiqg/uX15dZjf3zoty8Xi+r86uY=; b=z9gRTotiwzJNhFZwMeAvuP8Rkn6z8owAsopdejpPXQoxPAwsHLaTm4T2dXDC7wq2AmyDhq d8EUl6UMBy44gZ5sCBs2gJTGyKIs0RZ4NE6zjHXhomhY+M6sLGaGtQeCtVoQyJ0w1ROgz1 TP3vw3VLAu1UrbH7SBXWEq8Ag8Ip1fU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675692517; x=1707228517; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=U1Eiy12DbqHzX/ZOo41S8PKwsnJTE57YtWWPprWmJGw=; b=YAxb32obP/u0RZ83HFGmbu93K2cfi2Fi4OxKWeCYALjEg4c7KHjkLLjD ZOlY9KaDdkIjuemrFFRWMrbKPseleW98iP+cXjWdVlj4k7JbO2xvAfzY6 u/+81Tn3R6s0F0CcVgzeEDtOV4Reo1vgjmGZP8a97BoZuR/G4pbfGRyWi gyMNRP3lrcBxr0huXrkPfzV1HD3wqSAoWUxnhmksoeHF9ywjWUipt62v+ ONabIPKqYtIEv3pmESRKWl5llZfIE9cC+MCw3Uaw9sGz+Me1lIb/BM2j8 jqFrcGEBvCVwcDSrctxhFVTnuvRyxzGUDWyNHi6BinkASa37rcI5zSf+H w==; X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="312864312" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="312864312" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Feb 2023 06:04:57 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="911937897" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="911937897" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by fmsmga006.fm.intel.com with ESMTP; 06 Feb 2023 06:04:55 -0800 From: Yin Fengwei To: willy@infradead.org, david@redhat.com, linux-mm@kvack.org Cc: dave.hansen@intel.com, tim.c.chen@intel.com, ying.huang@intel.com, fengwei.yin@intel.com Subject: [RFC PATCH v4 4/4] filemap: batched update mm counter,rmap when map file folio Date: Mon, 6 Feb 2023 22:06:39 +0800 Message-Id: <20230206140639.538867-5-fengwei.yin@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230206140639.538867-1-fengwei.yin@intel.com> References: <20230206140639.538867-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 59141C0022 X-Rspamd-Server: rspam01 X-Stat-Signature: i5xabk6r5e7meihjw4mp1o9uncpiqx9q X-HE-Tag: 1675692517-101089 X-HE-Meta: U2FsdGVkX1/CKED5oslysRdMEB4knFHNIqVaMEp9Kmgl+7N3y0rVcHDmVjiabtA0zGCH1VWqNkIJKwR/aIUxmUNo8W8zPesVkMavVMXcG74BFNdHJw/7/Bz0+g8JHMEIxLYxBfhQTaR3H9ucNhDcvJel4qcLHWtg+0Etde3K5UkuauxwamBscp9l/soomLIHl00YOTI/MWdGzSxXS1Id0oHeOgXkGn5cEJrMq40qCfueS3v5FEI07Hvz1HJiUoAA99MLTF7gcLNXF9Pf35F5adG70or6wMzA1WTJg9ID5shC2ij0KtWr1Z8EjXoQbpM9EeUPYE8DOAvzL1TGUqOPp6PbSPF+XPaX8N+58vLfFYRdNIlMHVWF68ips6pk5AWQTyuEKjbGizXNhXqxLCXp+3FQ2G1zPu3MBYtPWxVJ2AAl0PZ5uvz+ko9OsVrh81RF28Y4HTnWJkPUKLR2UXyX/rYQVNRrxbJ72O7C5mIGbm1Ymg8sFeV8OdQZhF3YQ3ZZYrDqw16Q4vqr3T1QWNse/AmRooRGPA0dHYmouSx+veps2ZRRjjWp3Myg5b/SfJ4gYOjVvmXGhsZema7TlLielwPWoMv1ij+d6JXbQMQQDpGUUIiJHVqj/wthyOVizz+jPHji6u5fLD8QBsyoPB2cSXt4SeepsbAL4kUpremHb4hIdr+JdfciieZzQjQh/fanfBhYXvOYt2SY4t6RR5F4x5Ktg6ED4LGOreCdQV6yduA6r1OwgOgRY6lsjiSabwhwnEwkRvp32bO0otV4aNJyiMusLrvnO79/OOfWca2NKJ7FIYuFSuqcBwMEcEjtc55CWY28hngbwTKaftm1UZ88HFCIUO8FxUSXxSWR+Fyf/zGB5BkwwZwCRXlGDfsJ8svOT0T/BTnSaEPMr30oaY4FGiTqlpFHgvnAoBwncuv9rpqWoorv0ER+zRJMh+ML7jK0E+zgbXHV+t4Zat8bfxT b7ymlMAY hlc5Fb22nvCxpIf/XjNtvkAW0RS4O6dHmEcOhWO8IoRjBuf8pwCpC67hbYdHDIUK2/Z2NDx9KVe82UDYb1S5g6DsBjMEvApBIBizetXKE6R5UYv8HySe0J3aMjUrVt3lIVtPtS0VtucpBcsDFzIgKOWFQXQtM6xjpMA4FHD66JtRlqIs0TNYGmgqrC94Gu7fXM6MzPwnVceYDNBI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use do_set_pte_range() in filemap_map_folio_range(). Which batched updates mm counter, rmap for large folio. With a will-it-scale.page_fault3 like app (change file write fault testing to read fault testing. Trying to upstream it to will-it-scale at [1]) got 15% performance gain on a 48C/96T Cascade Lake test box with 96 processes running against xfs. Perf data collected before/after the change: 18.73%--page_add_file_rmap | --11.60%--__mod_lruvec_page_state | |--7.40%--__mod_memcg_lruvec_state | | | --5.58%--cgroup_rstat_updated | --2.53%--__mod_lruvec_state | --1.48%--__mod_node_page_state 9.93%--page_add_file_rmap_range | --2.67%--__mod_lruvec_page_state | |--1.95%--__mod_memcg_lruvec_state | | | --1.57%--cgroup_rstat_updated | --0.61%--__mod_lruvec_state | --0.54%--__mod_node_page_state The running time of __mode_lruvec_page_state() is reduced about 9%. [1]: https://github.com/antonblanchard/will-it-scale/pull/37 Signed-off-by: Yin Fengwei --- mm/filemap.c | 35 +++++++++++++++++++++++++---------- 1 file changed, 25 insertions(+), 10 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 6f110b9e5d27..4452361e8858 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3354,11 +3354,12 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, struct file *file = vma->vm_file; struct page *page = folio_page(folio, start); unsigned int mmap_miss = READ_ONCE(file->f_ra.mmap_miss); - unsigned int ref_count = 0, count = 0; + unsigned int mapped = 0; + pte_t *pte = vmf->pte; do { if (PageHWPoison(page)) - continue; + goto map; if (mmap_miss > 0) mmap_miss--; @@ -3368,20 +3369,34 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, * handled in the specific fault path, and it'll prohibit the * fault-around logic. */ - if (!pte_none(*vmf->pte)) - continue; + if (!pte_none(pte[mapped])) + goto map; if (vmf->address == addr) ret = VM_FAULT_NOPAGE; - ref_count++; - do_set_pte(vmf, page, addr); - } while (vmf->pte++, page++, addr += PAGE_SIZE, ++count < nr_pages); + mapped++; + continue; - /* Restore the vmf->pte */ - vmf->pte -= nr_pages; +map: + if (mapped) { + do_set_pte_range(vmf, folio, addr, pte, start, mapped); + folio_ref_add(folio, mapped); + } + + /* advance 1 to jump over the HWPoison or !pte_none entry */ + mapped++; + start += mapped; + pte += mapped; + addr += mapped * PAGE_SIZE; + mapped = 0; + } while (page++, --nr_pages > 0); + + if (mapped) { + do_set_pte_range(vmf, folio, addr, pte, start, mapped); + folio_ref_add(folio, mapped); + } - folio_ref_add(folio, ref_count); WRITE_ONCE(file->f_ra.mmap_miss, mmap_miss); return ret;