From patchwork Mon Feb 13 12:34:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13138381 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4445AC636D4 for ; Mon, 13 Feb 2023 12:35:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 995EE6B0071; Mon, 13 Feb 2023 07:35:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 920286B0073; Mon, 13 Feb 2023 07:35:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 798316B0078; Mon, 13 Feb 2023 07:35:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 636E26B0071 for ; Mon, 13 Feb 2023 07:35:20 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 2A0581A09BA for ; Mon, 13 Feb 2023 12:35:20 +0000 (UTC) X-FDA: 80462214000.26.C5202EC Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf22.hostedemail.com (Postfix) with ESMTP id 5B3A9C0010 for ; Mon, 13 Feb 2023 12:35:17 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=PhKzYIXs; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676291718; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=gl6ZJipqbUpjiTIc2NbxGJTR8KTedsO1Wrk3saxK4lk=; b=whBu7yeAPQP9ohz75pRzNT9qLStKdj6hfBMCoxtzzv8gUep59kVSL/0oUzbizpc4LFlfvP ZN56soE/yq+ps53jdQfrFVavRlzYq1XBFairExS/CqhoXyP1wZHwIuJR9ZA9deIrl85sE1 hWY+0wBO/pJSALa3PsraA8IVNUIojmE= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=PhKzYIXs; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676291718; a=rsa-sha256; cv=none; b=A3cGDYOE1IMgKT6KVUsXF4NycZUgPX98PxWcIxoPB0uoQVK17dLTKE3ea9AWlpd8GEeQHW kecA1/gCsprqJdQi8u6DjJz/Y6WUrBpjxpdVnglbQXpNCe3aj1Y/e518L0xxlonRwlUSoV nWgWbogrtZgN9p2Ngub1KWip5h2j0+8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1676291717; x=1707827717; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=8kM/5jxfZlI/Ucp9ptNmm7hMWs8FA/UuOHWcst0gECs=; b=PhKzYIXsg3tCPR4hU896XW3SbGmjPgUVJdgo/6DLDGgK6IlxvGj58jxS mVGZDC7BNdo1gMjB7LGeUiUqxnPbsEIoRGcib4y9/40+QUx3Pc4Ga9L8N +6R3JZR2hD/d3CPU3XENQoePEd3vvwcUASLbW61z37k5atxS+nf5sYtzF ceWbPD8s/Uq4t5q+hR2JbR3kVL2J/Tk7uxZ5+URX40SMo+Bu+kMc164zD OibE54TcICJUTHpYFy+ySDpAY3KGtGxSfOoBotpTxAy1SVBwLdiDYCRRm 8dcZCJzFiGg0LDztcDbO/QGB3coOm3Phjz3XtKMXx7FeIuucm/CiAgKvm Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="310513140" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="310513140" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:35:15 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="646366578" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="646366578" Received: from changxin-mobl2.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.255.28.171]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:35:11 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Huang, Ying" , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , Xin Hao , Minchan Kim , Mike Kravetz , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH -v5 0/9] migrate_pages(): batch TLB flushing Date: Mon, 13 Feb 2023 20:34:35 +0800 Message-Id: <20230213123444.155149-1-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: px9nux15xj4xm93a4e53rprgest19mzr X-Rspamd-Queue-Id: 5B3A9C0010 X-HE-Tag: 1676291717-776392 X-HE-Meta: U2FsdGVkX1/w3dZVP8iRZKj9OG0HK4C5p4o3ckrrU/e4eu+fEjgVplkyo44ratCUqD1fn3E5aY/Oh0fXr13sNNF92nkZ/5QqPSRKNsiESXP3SLfXa+uImpa9dlblyJlGru9/wHCKAIPe4ZsanVLrrrEp9Yh4ypDwxYvQJLXnhNyXAr8LcpqZpJMLi6CDWM3m58ALwvEvLGlBIQ7Yt3eVdMuBD4yZdXgpKG04/6VGdfadFMzE6fOyi4CGXyguM9lHlDXfLwKYwZLZz6Oadf1V5unIMFuZrYTyGBodPQfjdcPzS8McKgDniyE/MBmBG9EKZKts0PblVIjhig6ZBp+WFfXvn8i+AAWkut3SVxa7MdhHmy2UA433UHCWpsAcpFUOJtMlRuv3YKz23nTiFiUFoQaBMA9aCjbGkup+ohYJVKGp7EBJsjtTkANNfBDRK+vsPe8zb9aJwunegNCZh1n2eVGYKHUYTDaviQk3F92Negpo41aUaql34X6tWfd5o1nmuHz7s18Ocdg8qod+/AgQ3sOlxTK3Szy8kmjS3cpMpSiciySe1xY4AO44ckihOKsMw08iYgbCOOssiH0AiAoHf1e0+u7+AJLl4Ov6OghpG1+bnpYyE5aJuHwLCoZmleLjltiewTVzXq7X4zSNennaJpZ/f0WBAfL1lLF3uQVuS8sf1h9qEZHO5kZmrFAsmZlRhg7mBuyHH/MBvuC8iiHnwtOllwqm5I7glo/Wz88fiKxF0UBygQSCihsDY6YptG7zAU+8l+b3MdyAe7c4CEgKpz/oT90YFSv9de+xzxh3CGNuiItx9ihB+ymKOOUsJJiWFVNaOSoeU8OvuVS30skPo2YHMjGXrbb+GNZXJxN/BOsAaq9sxLxxDgzmUK1KaYNPCevUejQGCs7Uxla15ltPJ8FOSU4veThVJJmWkLSllwBe7ZbvrlAn9FazLntLFS64BNJSJOGR8xlbr46w5tV kkH+56oi 99DhmCv05foO468P33zS1XQTFEYaEW4lSy8QoXO5XjaOkzkLZrvuTMMDR/bD3tAfFKmdGLLgrxpLUPensSUyXblw6tlwoVLJR6HaJPX9lsMiNg7owL3YFQtaVq3xD8hHXazqWV2sjzi3FeBVkOFmuvqWSuchjsK4Kt/e1ZCZhM8B8TY0q9XvZyM52s0VoEbFJW0JL7rC9in161r0xYgBRgwgt9UVkOZT/pQTcxRQj50qRh2REK7WgJ8VGaDYanzwXvv0qWp2TBsE2moeKSLIyD7ihYXJlJbFQyVGM X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Huang, Ying" Now, migrate_pages() migrate folios one by one, like the fake code as follows, for each folio unmap flush TLB copy restore map If multiple folios are passed to migrate_pages(), there are opportunities to batch the TLB flushing and copying. That is, we can change the code to something as follows, for each folio unmap for each folio flush TLB for each folio copy for each folio restore map The total number of TLB flushing IPI can be reduced considerably. And we may use some hardware accelerator such as DSA to accelerate the folio copying. So in this patch, we refactor the migrate_pages() implementation and implement the TLB flushing batching. Base on this, hardware accelerated folio copying can be implemented. If too many folios are passed to migrate_pages(), in the naive batched implementation, we may unmap too many folios at the same time. The possibility for a task to wait for the migrated folios to be mapped again increases. So the latency may be hurt. To deal with this issue, the max number of folios be unmapped in batch is restricted to no more than HPAGE_PMD_NR in the unit of page. That is, the influence is at the same level of THP migration. We use the following test to measure the performance impact of the patchset, On a 2-socket Intel server, - Run pmbench memory accessing benchmark - Run `migratepages` to migrate pages of pmbench between node 0 and node 1 back and forth. With the patch, the TLB flushing IPI reduces 99.1% during the test and the number of pages migrated successfully per second increases 291.7%. Xin Hao helped to test the patchset on an ARM64 server with 128 cores, 2 NUMA nodes. Test results show that the page migration performance increases up to 78%. This patchset is based on mm-unstable 2023-02-10. Changes: v5: - Added Xin's test results on ARM 64. Thanks Xin! - Fixed many comments. Thanks Zi and Xin! - Collected reviewed-by and tested-by. v4: - Fixed another bug about non-LRU folio migration. Thanks Hyeonggon! v3: - Rebased on v6.2-rc4 - Fixed a bug about non-LRU folio migration. Thanks Mike! - Fixed some comments. Thanks Baolin! - Collected reviewed-by. v2: - Rebased on v6.2-rc3 - Fixed type force cast warning. Thanks Kees! - Added more comments and cleaned up the code. Thanks Andrew, Zi, Alistair, Dan! - Collected reviewed-by. from rfc to v1: - Rebased on v6.2-rc1 - Fix the deadlock issue caused by locking multiple pages synchronously per Alistair's comments. Thanks! - Fix the autonumabench panic per Rao's comments and fix. Thanks! - Other minor fixes per comments. Thanks! Best Regards, Huang, Ying Signed-off-by: Hugh Dickins Signed-off-by: "Huang, Ying" Reported-by: "Xu, Pengfei" Signed-off-by: "Huang, Ying" Reported-by: Hugh Dickins Reported-by: "Xu, Pengfei"