From patchwork Tue Jan 10 07:53:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13094775 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE993C61DB3 for ; Tue, 10 Jan 2023 07:53:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 00C908E0002; Tue, 10 Jan 2023 02:53:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EFF868E0001; Tue, 10 Jan 2023 02:53:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D7A978E0002; Tue, 10 Jan 2023 02:53:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C8C1E8E0001 for ; Tue, 10 Jan 2023 02:53:39 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 96BB9C066E for ; Tue, 10 Jan 2023 07:53:39 +0000 (UTC) X-FDA: 80338124958.10.D084B6E Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by imf09.hostedemail.com (Postfix) with ESMTP id 2BC1E140003 for ; Tue, 10 Jan 2023 07:53:36 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=jHzDyis2; spf=pass (imf09.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673337218; a=rsa-sha256; cv=none; b=Z2M+LFQgYaFKFekDWzo8UO6bZ+v/mmMxzzIYC/FLiREmjiXpbplBB5xK7dUSU0XbAdzaU5 lLZQS/ruPPOMv43PN+DO4d/EHsTRwqmuJyYYraXEHkViRWcIucv3AX4myiZzFcqBr0JVaF +C4cQzGrTLGB1ZlPKiMKxpidTGGAjIY= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=jHzDyis2; spf=pass (imf09.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673337218; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=zKBRAP9if/eXxj0CHWRM9PANXjdY8f0KOwczVPufQwY=; b=fsJOBfH+Kd2YjJLwsgV/nrWdPrYEGnJ69lHaqBq+4vS2OjBgLL20esTlxLWruzmTSv4LJg bD29niHNnFj8lHrT97EGUYPDRKP5nSWoNILfXaPalXCCun8V/FXZd0SI7RkaiA8f0bc1TN RIsTlrvY5TaAEWYq6IB1wSXhxso/5/o= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673337217; x=1704873217; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=TCPzdo8u3EXgieM9CNMUNU67y5hV/sEIFaUkmnCEXgQ=; b=jHzDyis2p+gavXS4JiSNXZZp0ubu25vrJqcfyVG8eBhsvneVsOjIZaWM UHk5Cl0GuVfftCK/tKRKLk01Y/U39n7NDrYuuar5KtCs6FpWnsKTSXx6B gkEpQ7gxi1bB4qpfEsZzu4AJ0MrD6h30lvGSAouT2KsAgP/ldT5eC1Cuk CoL6vwrrwISOY5OKwfwpyWs1YmYCmre5vejtoiC4zS3gnLARVZWuwPnNN ruqnI0UAzdlm4HDH48mFFMIo4kL9wFMUCQSd50wYERp5pTZY2YtQPEE4V VXFkLLdgm8CDEmfxEkFveiIvsh8LXn2nUXrtgyFyO6YmeOdBan+OdVoRz Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="303449255" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="303449255" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:53:35 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="902287140" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="902287140" Received: from juxinli-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.254.214.35]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:53:32 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Huang, Ying" , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin Subject: [PATCH -v2 0/9] migrate_pages(): batch TLB flushing Date: Tue, 10 Jan 2023 15:53:18 +0800 Message-Id: <20230110075327.590514-1-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 2BC1E140003 X-Rspamd-Server: rspam01 X-Stat-Signature: 9bjirc5nakeu18gbiyxc1qpcy8m49sei X-HE-Tag: 1673337216-129667 X-HE-Meta: U2FsdGVkX19i44pQzkj4L/d+W55hEwQCg8Fg+oARr3opI+7f7p+tGxIqrq1Hj9OMcreGZOImD/kL4DNQ1F/oyKdKpLHVt0fTY/LPISax+3xUTn0Su6As4iUiOlGqRXXrizuqU6TiXKSzFIOG3cnq8OhJdrrbDkS7sWz0x7IdZ0J0iQym6cYVq8f1N+ruS/GhJGM5ryf9WNirWOj5ysey0x9uNe+xSo18LRcE9VWUrDo/NEWlSKySCwaZ7BvAi9Sbofi9waSPzQ7v6+rR63TjRYV2ItpAqFaRMWq7NZh68x4UmX3sYlDnpHAxLtPEhAStWjce/6MUFPIUBSld3qIgKTo8zdCsrsIJf0eUChWva8Bo7wS3qgtL88v/vTfE/Li75sSiyb0pVBth4Qs79DZNkznHHpMHZ1DRoRUN62QjyvKQ/KTlo89kJvGlE8hBZa4U3dJQmrLeKcN55dTUCqwxxfMsG+eGlXaNwS/JOsKiajgPeC3gSWbcZtS233kbZu6R7Kebr9FdiayN+yBwTwS69hdFMdsaZOy3/sQZgqlDSVGnJDm3T3tSrYYD4vpGb7tc08KbyxPwxKixSUV1NdDfns3hV5SjJe8WtNLYJ55ISArlDiW8vwmQOog7TgBSwTMo0hDbFTGuwymK247Jz4wOVn7Cl8S7NoxewxlEK1cuZXEyMcMC1FsNSjeB+ii183kbfvcq2XciVFmmag8tainLqi19bMYD9//m8b8XZeEXEvTbdp0eOIUI4HxwlaKbFov6gSHoioMeGqDrmQOz4HljEOiJNNGOLjcrwzj9U1EAqagoYhNrj5hFjr5eRY7yeS/ex+xMoznh/P1Mqp3gZWFzg0Zun+S7i4R7BDchxD4okKSXWuKbAZ2VSSNkr40Tv2IV5QXtR+1HUcrRmJQNSSvn0SXjcvh5gxUjyPxlZBCQeEYkzCD46/GS5VwUqGZ1w+Bf X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Huang, Ying" Now, migrate_pages() migrate folios one by one, like the fake code as follows, for each folio unmap flush TLB copy restore map If multiple folios are passed to migrate_pages(), there are opportunities to batch the TLB flushing and copying. That is, we can change the code to something as follows, for each folio unmap for each folio flush TLB for each folio copy for each folio restore map The total number of TLB flushing IPI can be reduced considerably. And we may use some hardware accelerator such as DSA to accelerate the folio copying. So in this patch, we refactor the migrate_pages() implementation and implement the TLB flushing batching. Base on this, hardware accelerated folio copying can be implemented. If too many folios are passed to migrate_pages(), in the naive batched implementation, we may unmap too many folios at the same time. The possibility for a task to wait for the migrated folios to be mapped again increases. So the latency may be hurt. To deal with this issue, the max number of folios be unmapped in batch is restricted to no more than HPAGE_PMD_NR in the unit of page. That is, the influence is at the same level of THP migration. We use the following test to measure the performance impact of the patchset, On a 2-socket Intel server, - Run pmbench memory accessing benchmark - Run `migratepages` to migrate pages of pmbench between node 0 and node 1 back and forth. With the patch, the TLB flushing IPI reduces 99.1% during the test and the number of pages migrated successfully per second increases 291.7%. This patchset is based on mm-unstable. Changes: v2: - Rebased on v6.2-rc3 - Fixed type force cast warning. Thanks Kees! - Added more comments and cleaned up the code. Thanks Andrew, Zi, Alistair, Dan! - Collected reviewed-by. from rfc to v1: - Rebased on v6.2-rc1 - Fix the deadlock issue caused by locking multiple pages synchronously per Alistair's comments. Thanks! - Fix the autonumabench panic per Rao's comments and fix. Thanks! - Other minor fixes per comments. Thanks! Best Regards, Huang, Ying