From patchwork Mon Jul 10 08:39:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yicong Yang X-Patchwork-Id: 13306471 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1368BEB64DC for ; Mon, 10 Jul 2023 08:41:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9E8156B0074; Mon, 10 Jul 2023 04:41:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9980C6B0075; Mon, 10 Jul 2023 04:41:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8867B6B0078; Mon, 10 Jul 2023 04:41:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7AD426B0074 for ; Mon, 10 Jul 2023 04:41:18 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 36C4B402D0 for ; Mon, 10 Jul 2023 08:41:18 +0000 (UTC) X-FDA: 80995057836.21.B39C115 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by imf08.hostedemail.com (Postfix) with ESMTP id D7AA916000E for ; Mon, 10 Jul 2023 08:41:14 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=none; spf=pass (imf08.hostedemail.com: domain of yangyicong@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=yangyicong@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688978475; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references; bh=OtLo42V/WRm8qAnt4eAPI7Es5QjyXnMoIJ6TF771Ub0=; b=JK0QENqR7SaJRrSsL/4Xiv5HvXOjDlegmwKvAh3EC5BCkGdG1q7PNj6nxWsSvzk4HB/1Rk APsY56wc53yIotjsxFFPBckyFDIwoM0GlNvwrJQhyiPVBwEmA0Y5ENUyKq3L3BUwNJC7l6 E9KzTaw7hzWLYbfLfZUBnv0PJ6RTtos= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688978475; a=rsa-sha256; cv=none; b=49mFIjGX5wL+2MV9mwijSsBuO8EYUFL/ncrOAZCSA4Y1Ae47eOWJn+3EWQkM6CwoLpNHeK OLv1XhxceneSmjAWpwZvuMmbwPzSTPchmxMNVqWtOm7FSZDF2SstYy5pmgRbQGB1Kd0foh LYx6ZMjnUF7CEmywiKcC9njTB/IOsPU= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; spf=pass (imf08.hostedemail.com: domain of yangyicong@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=yangyicong@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com Received: from canpemm500009.china.huawei.com (unknown [172.30.72.56]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4QzyBX1LZqzqVdF; Mon, 10 Jul 2023 16:40:36 +0800 (CST) Received: from localhost.localdomain (10.50.163.32) by canpemm500009.china.huawei.com (7.192.105.203) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Mon, 10 Jul 2023 16:41:08 +0800 From: Yicong Yang To: , , , , , , , , , CC: , , , , , , , , , , , , , , , , , Barry Song <21cnbao@gmail.com>, , , , Subject: [PATCH v10 0/4] arm64: support batched/deferred tlb shootdown during page reclamation/migration Date: Mon, 10 Jul 2023 16:39:10 +0800 Message-ID: <20230710083914.18336-1-yangyicong@huawei.com> X-Mailer: git-send-email 2.31.0 MIME-Version: 1.0 X-Originating-IP: [10.50.163.32] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To canpemm500009.china.huawei.com (7.192.105.203) X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: D7AA916000E X-Rspam-User: X-Stat-Signature: oneg7f9s6f4d9qysgfoi1t8qxiyonzwx X-Rspamd-Server: rspam03 X-HE-Tag: 1688978474-379891 X-HE-Meta: U2FsdGVkX19VPKXPBdRT2EXfNPkzXiLH+pdCrTG05hbzxBLePljDo7wPeyNSPUdvfoOnJzYrGwlm0cwTIYJs6p10ZkavQJmU9bypBjPFfGVQqHdr8RdZ/rPNFxF2yJP89nqJAjGIOxEVdicLfoWmmY3ERGD58sRElNY0jf3u1bZPsRrLTjswfOeLnqYYj4gIJE2AL0ALGtzseu6ltnGb7tVp+f1ei4oR+61dOFhAVZtU9/ntlzxBdIaBzD+LpTY7tqAQICS3fOUN1R6xza4AhVwLb5f7PZxrk+p8/BjmTwP4OyS6FnVFxGdMQ3rb9Ya3Saq7jCjGWqswQs/y0wSdtnnfOQuM7qDPR+P4nFYqSsk2YK4mho/H4iv/AiFOAyxjwBgvhSHIZPCFwMSG3vGloVUubI+JVJ1E+KSoRzmr5EnR3gOc1I52rw0lu+7YXB+WgGG/eZZyvx9Li+9+vnTt9Ej4a0KpIfwsiaJY3ydJeqsWKnbhqImT3H2a5OMV5432A/G6P7OQUQ/HIXiP51bvf3031IHcLKZwfw4YBM0UdXBqGKr0dW02u+uv+sOn6HLK1waoCjOxHtsGg0D2NLhXFHUST3o6DwNtfHnPv8XqATXK3/+CPE6zFyFc0oiwp1KiyQf8YWUCkHKBt3/4jNsgAEXt5AjLv5Z1PlKygiRNtkmDuTrO12vZAyYx2fH8fCKuMBBaF9JBktSmGZ00emLfpDnLmjULAmq/zct7mwWotSVH0mhMa5L0DAGqjyV5vQitTG1xF7UcsQch6R4U1Q1snFeA2CJyFL0LdGr67jsXLmxT+IvZMp1VSPDetlM9Hc+y0R3tBsh9D3rzInL2fEub7P6Hobg8AehozsMjTfhoDLPoLh2BI5YLONvzvluijpQYjh8ZQJQq4NJFcy1P/mzGCPqwPsosfiqyX8hSKgX47MPB+xiWeLdMiKQOFAH+HlebH5Rwfs2Yl1n0os3KHP4 vwEgz72c rNriRsmS+05rcC+XJ6oWMIBFwv/nF374IayonNBoIuazbTOC4BI3AQgEPWXYCXQwor4ZH32XL9b+MboYCY4PQdSGXUDnC1I8vKSlgMKehpWd/cl+fW+Jt0AWoGnpjOf13CPtQJX+pk3IsZuF/Nzl3gZWE1XPoXRLAZn7wRuoN7KiwR41Z/sOvo2iipug7Q1MsuPBM9511EeoZbAvK8P/YqJeTL4JQRvFB5tJai/SptoMDK6O8rhdhFwaz7SRAH9PQIYFYXkWscvPYH0+XNG92hqfIesjCtPjJzwVp X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yicong Yang Though ARM64 has the hardware to do tlb shootdown, the hardware broadcasting is not free. A simplest micro benchmark shows even on snapdragon 888 with only 8 cores, the overhead for ptep_clear_flush is huge even for paging out one page mapped by only one process: 5.36% a.out [kernel.kallsyms] [k] ptep_clear_flush While pages are mapped by multiple processes or HW has more CPUs, the cost should become even higher due to the bad scalability of tlb shootdown. The same benchmark can result in 16.99% CPU consumption on ARM64 server with around 100 cores according to the test on patch 4/4. This patchset leverages the existing BATCHED_UNMAP_TLB_FLUSH by 1. only send tlbi instructions in the first stage - arch_tlbbatch_add_mm() 2. wait for the completion of tlbi by dsb while doing tlbbatch sync in arch_tlbbatch_flush() Testing on snapdragon shows the overhead of ptep_clear_flush is removed by the patchset. The micro benchmark becomes 5% faster even for one page mapped by single process on snapdragon 888. Since BATCHED_UNMAP_TLB_FLUSH is implemented only on x86, the patchset does some renaming/extension for the current implementation first (Patch 1-3), then add the support on arm64 (Patch 4). -v10: 1. Enable BATCHED_UNMAP_TLB_FLUSH regardless of CPU numbers, per Catalin. 2. Split the renaming/extension works in a separate PATCH 2, per Catalin. Since it's split from PATCH 2/2 in v9, so inherit the tags. 3. Add arch_flush_tlb_batched_pending() to allow arch-specific implementation, per Catalin. Since it's some kind of an optimization on arm64 so a separate Patch 3/4. Link: https://lore.kernel.org/linux-mm/20230518065934.12877-1-yangyicong@huawei.com/ -v9: 1. Using a runtime tunable to control batched TLB flush, per Catalin in v7. Sorry for missing this on v8. Link: https://lore.kernel.org/all/20230329035512.57392-1-yangyicong@huawei.com/ -v8: 1. Rebase on 6.3-rc4 2. Tested the optimization on page migration and mentioned it in the commit 3. Thanks the review from Anshuman. Link: https://lore.kernel.org/linux-mm/20221117082648.47526-1-yangyicong@huawei.com/ -v7: 1. rename arch_tlbbatch_add_mm() to arch_tlbbatch_add_pending() as suggested, since it takes an extra address for arm64, per Nadav and Anshuman. Also mentioned in the commit. 2. add tags from Xin Hao, thanks. Link: https://lore.kernel.org/lkml/20221115031425.44640-1-yangyicong@huawei.com/ -v6: 1. comment we don't defer TLB flush on platforms affected by ARM64_WORKAROUND_REPEAT_TLBI 2. use cpus_have_const_cap() instead of this_cpu_has_cap() 3. add tags from Punit, Thanks. 4. default enable the feature when cpus >= 8 rather than > 8, since the original improvement is observed on snapdragon 888 with 8 cores. Link: https://lore.kernel.org/lkml/20221028081255.19157-1-yangyicong@huawei.com/ -v5: 1. Make ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH depends on EXPERT for this stage on arm64. 2. Make a threshold of CPU numbers for enabling batched TLP flush on arm64 Link: https://lore.kernel.org/linux-arm-kernel/20220921084302.43631-1-yangyicong@huawei.com/T/ -v4: 1. Add tags from Kefeng and Anshuman, Thanks. 2. Limit the TLB batch/defer on systems with >4 CPUs, per Anshuman 3. Merge previous Patch 1,2-3 into one, per Anshuman Link: https://lore.kernel.org/linux-mm/20220822082120.8347-1-yangyicong@huawei.com/ -v3: 1. Declare arch's tlbbatch defer support by arch_tlbbatch_should_defer() instead of ARCH_HAS_MM_CPUMASK, per Barry and Kefeng 2. Add Tested-by from Xin Hao Link: https://lore.kernel.org/linux-mm/20220711034615.482895-1-21cnbao@gmail.com/ -v2: 1. Collected Yicong's test result on kunpeng920 ARM64 server; 2. Removed the redundant vma parameter in arch_tlbbatch_add_mm() according to the comments of Peter Zijlstra and Dave Hansen 3. Added ARCH_HAS_MM_CPUMASK rather than checking if mm_cpumask is empty according to the comments of Nadav Amit Thanks, Peter, Dave and Nadav for your testing or reviewing , and comments. -v1: https://lore.kernel.org/lkml/20220707125242.425242-1-21cnbao@gmail.com/ Anshuman Khandual (1): mm/tlbbatch: Introduce arch_tlbbatch_should_defer() Barry Song (2): mm/tlbbatch: Rename and extend some functions arm64: support batched/deferred tlb shootdown during page reclamation/migration Yicong Yang (1): mm/tlbbatch: Introduce arch_flush_tlb_batched_pending() .../features/vm/TLB/arch-support.txt | 2 +- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/tlbbatch.h | 12 +++++ arch/arm64/include/asm/tlbflush.h | 48 +++++++++++++++++-- arch/x86/include/asm/tlbflush.h | 22 ++++++++- include/linux/mm_types_task.h | 4 +- mm/rmap.c | 23 ++++----- 7 files changed, 90 insertions(+), 22 deletions(-) create mode 100644 arch/arm64/include/asm/tlbbatch.h