From patchwork Wed Mar 29 03:55:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yicong Yang X-Patchwork-Id: 13191857 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2832C74A5B for ; Wed, 29 Mar 2023 03:56:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0332B900003; Tue, 28 Mar 2023 23:56:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ED725900002; Tue, 28 Mar 2023 23:56:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D52CC900003; Tue, 28 Mar 2023 23:56:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BE706900002 for ; Tue, 28 Mar 2023 23:56:07 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 6DA8A1407D3 for ; Wed, 29 Mar 2023 03:56:07 +0000 (UTC) X-FDA: 80620572774.12.1B826F6 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by imf02.hostedemail.com (Postfix) with ESMTP id D93CD8000F for ; Wed, 29 Mar 2023 03:56:03 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf02.hostedemail.com: domain of yangyicong@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=yangyicong@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680062164; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=y6PNL/nnQ2fEoGSpeZ5hjfhJRJ824jVzVZt73z/7IGA=; b=XAUr6GLQnllcG+CpjUErLiOcm9bg3AjkTe6Y0pnUiJvogt1vlgQTNb3cUfakykLPwizTnQ uTMM145UDuxllTTJU3rahLY8kX+UpblKP3wyRtGoiJd4p64eGERbbmlEd3BO8+tI3qGmLP SZOHxbS3KCdwp021pXPqlwDrFqlw5Ls= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf02.hostedemail.com: domain of yangyicong@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=yangyicong@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680062164; a=rsa-sha256; cv=none; b=Ce7cPzJNbgAPAJYZMOlm78IBQereiI/I9TPIX9PPzATO+EJ4RDbqV85SAIyW+cm1XODxAR xN5vEJovDxKU6cDwbi7Z4GlSXxeeylj5bq1w5k8oqRPdf03Th2rKE04TaPU3/N7269Ex0b gV9dG+HMznZipdrRK/IUJHtvUv1OYJs= Received: from canpemm500009.china.huawei.com (unknown [172.30.72.54]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4PmXgv0ZWnzgZXD; Wed, 29 Mar 2023 11:52:43 +0800 (CST) Received: from localhost.localdomain (10.50.163.32) by canpemm500009.china.huawei.com (7.192.105.203) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21; Wed, 29 Mar 2023 11:55:58 +0800 From: Yicong Yang To: , , , , , , , CC: , , , , , , , , , , , , , , , , , Barry Song <21cnbao@gmail.com>, , , , , Barry Song , Nadav Amit , Mel Gorman Subject: [PATCH v8 2/2] arm64: support batched/deferred tlb shootdown during page reclamation Date: Wed, 29 Mar 2023 11:55:12 +0800 Message-ID: <20230329035512.57392-3-yangyicong@huawei.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20230329035512.57392-1-yangyicong@huawei.com> References: <20230329035512.57392-1-yangyicong@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.50.163.32] X-ClientProxiedBy: dggems701-chm.china.huawei.com (10.3.19.178) To canpemm500009.china.huawei.com (7.192.105.203) X-CFilter-Loop: Reflected X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D93CD8000F X-Stat-Signature: 1tzukzr18ts1f9xjy1ef1r1o3cce4wbq X-HE-Tag: 1680062163-861096 X-HE-Meta: U2FsdGVkX1/mjmhnu3VaYYW8Uu7Arsw217X8y1Q7Ghrdrd0lGgFOH+NfUKR9XRYzeTpvm4NxlWkZav27o6mB7Mf9+WrHfGvv2KvtkTeIkGD4XX7gC0220KAPvqACQvoZtKAgDdyrWS8ByXtgWGXQ3hwnI5woltKQeMLn8O2Ej0XvoFXS+EnqIWj0Nk5Lo2EOppEiGVq5whS4wKsa9pOJoSJ6Gdf9SzVfhbF6QQiMCvbDGun756Q8chEcuW/hdJptSZh+Q/S13m2xHBy34IaJBqttq3G+lYcE6s1z/jk6pOGgTMPLGahwDZjXqzAnaWzS/G0LrFSEQVvAM2deEDwhm1x9JaqD674eUmrAqo8CkfQb9xUuBZb1DV/mvOtd7Y3NzwJ+o82CAYSd6SSfCbG7bNwPF8xugmlpdHPCsqlH121XBBudPyl10fO/U4uCSVJPU8VDtM0BmJfp56ihjXrVOWexMruO6/QEmo/mPKE3UyhXhJMkykYK5DLVaK+0W5Ax6rMEBDEHWGfT2NyDEZ2slclu6E31ixtOQ4osho+O8eGlzmzCWtOC/S4XJvbVfG5/kp431Rom4HPZjwJIupK5sbdZzgJrVbt3HY+n03yyN1ibBFGRQ1Rb7t10NG1+fze9w4UoE6ZqOEAcqYxK0RdqvoN1akHMm5Z4QprPM3sEhpdb9jrGvL9LeWUPN2RB3dUqFd11ctxW5pAxg2u8OjhbUfBbtOiDu03/Wt28l8Qb0S1PoM4cASlkbOm5yD39dao/cTb5gpmm5tZ/jLowaQSjcW0B5q7QNR+yJPrCEITAdNN380nxEoUBt7ZCu4Apg/oZLe5v0KjjVTbgfNBcfoDp1D/EFkmRdkGeM/FPFCenXXgGBJL4vgpvR9oris/CG3B4LZ0v+a0uEUeb6P/WaE5qJvCdr9AFrTASr/jH69X39DkmpaK6IDVitnkMA2FXNyBKqLTxfgo2xafQ7ugjpjB ZSYqBhMv znlZYlXcEpTzWvylykquRvHMPXBpNpXz8ooWZLGrrC3wv0cxwBWLN26CVpSRsQQSFy4EFSOWjUU+iOMK0ojHe1vTNRLriV4Msfnq/NfifRpl1i3q9uLT0Yu6dZsZN0Y5Kt3FQdzP5Wnn8U+c4qn9MAauKFOEAxd1SWNNJyAInvYnuhmzZoKZPi6Agbu0NaS5e3EL63eHIOWG5CmeYs0oixyq5T4sH98cyzrSVsZPEZs+4woZcqxcCYLjSMxux72EPjUzQz+1k/WtpwrvPRHKDqHLGMOdUAj4RME2JpwKV2v95GnnfSsqw7+QRSHay5I2ohaR4WIGx/3XrxMJxs4Wk7NUWwE42fbCBU7he2XzEBgmk0C1d0TZMubxOXk34G7EaeDJsEkPwO0WP5exGdCSxBdTaFHMs8y3hzqFo9Nf03Sewjg91Yo6kJEi7ty4h2p4f+lVRA6Ip/JVYizHH1okP8EcF2bhFRVHkGJVpbMKB07VIGcY/zQzZ84ZqktXtZaaFTKYb X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Barry Song on x86, batched and deferred tlb shootdown has lead to 90% performance increase on tlb shootdown. on arm64, HW can do tlb shootdown without software IPI. But sync tlbi is still quite expensive. Even running a simplest program which requires swapout can prove this is true, #include #include #include #include int main() { #define SIZE (1 * 1024 * 1024) volatile unsigned char *p = mmap(NULL, SIZE, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0); memset(p, 0x88, SIZE); for (int k = 0; k < 10000; k++) { /* swap in */ for (int i = 0; i < SIZE; i += 4096) { (void)p[i]; } /* swap out */ madvise(p, SIZE, MADV_PAGEOUT); } } Perf result on snapdragon 888 with 8 cores by using zRAM as the swap block device. ~ # perf record taskset -c 4 ./a.out [ perf record: Woken up 10 times to write data ] [ perf record: Captured and wrote 2.297 MB perf.data (60084 samples) ] ~ # perf report # To display the perf.data header info, please use --header/--header-only options. # To display the perf.data header info, please use --header/--header-only options. # # # Total Lost Samples: 0 # # Samples: 60K of event 'cycles' # Event count (approx.): 35706225414 # # Overhead Command Shared Object Symbol # ........ ....... ................. ............................................................................. # 21.07% a.out [kernel.kallsyms] [k] _raw_spin_unlock_irq 8.23% a.out [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore 6.67% a.out [kernel.kallsyms] [k] filemap_map_pages 6.16% a.out [kernel.kallsyms] [k] __zram_bvec_write 5.36% a.out [kernel.kallsyms] [k] ptep_clear_flush 3.71% a.out [kernel.kallsyms] [k] _raw_spin_lock 3.49% a.out [kernel.kallsyms] [k] memset64 1.63% a.out [kernel.kallsyms] [k] clear_page 1.42% a.out [kernel.kallsyms] [k] _raw_spin_unlock 1.26% a.out [kernel.kallsyms] [k] mod_zone_state.llvm.8525150236079521930 1.23% a.out [kernel.kallsyms] [k] xas_load 1.15% a.out [kernel.kallsyms] [k] zram_slot_lock ptep_clear_flush() takes 5.36% CPU in the micro-benchmark swapping in/out a page mapped by only one process. If the page is mapped by multiple processes, typically, like more than 100 on a phone, the overhead would be much higher as we have to run tlb flush 100 times for one single page. Plus, tlb flush overhead will increase with the number of CPU cores due to the bad scalability of tlb shootdown in HW, so those ARM64 servers should expect much higher overhead. Further perf annonate shows 95% cpu time of ptep_clear_flush is actually used by the final dsb() to wait for the completion of tlb flush. This provides us a very good chance to leverage the existing batched tlb in kernel. The minimum modification is that we only send async tlbi in the first stage and we send dsb while we have to sync in the second stage. With the above simplest micro benchmark, collapsed time to finish the program decreases around 5%. Typical collapsed time w/o patch: ~ # time taskset -c 4 ./a.out 0.21user 14.34system 0:14.69elapsed w/ patch: ~ # time taskset -c 4 ./a.out 0.22user 13.45system 0:13.80elapsed Also, Yicong Yang added the following observation. Tested with benchmark in the commit on Kunpeng920 arm64 server, observed an improvement around 12.5% with command `time ./swap_bench`. w/o w/ real 0m13.460s 0m11.771s user 0m0.248s 0m0.279s sys 0m12.039s 0m11.458s Originally it's noticed a 16.99% overhead of ptep_clear_flush() which has been eliminated by this patch: [root@localhost yang]# perf record -- ./swap_bench && perf report [...] 16.99% swap_bench [kernel.kallsyms] [k] ptep_clear_flush It is tested on 4,8,128 CPU platforms and shows to be beneficial on large systems but may not have improvement on small systems like on a 4 CPU platform. So make ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH depends on CONFIG_EXPERT for this stage and make this disabled on systems with less than 8 CPUs. User can modify this threshold according to their own platforms by CONFIG_NR_CPUS_FOR_BATCHED_TLB. Also this patch improve the performance of page migration. Using pmbench and tries to migrate the pages of pmbench between node 0 and node 1 for 20 times, this patch decrease the time used more than 50% and saved the time used by ptep_clear_flush(). This patch extends arch_tlbbatch_add_mm() to take an address of the target page to support the feature on arm64. Also rename it to arch_tlbbatch_add_pending() to better match its function since we don't need to handle the mm on arm64 and add_mm is not proper. add_pending will make sense to both as on x86 we're pending the TLB flush operations while on arm64 we're pending the synchronize operations. Cc: Anshuman Khandual Cc: Jonathan Corbet Cc: Nadav Amit Cc: Mel Gorman Tested-by: Yicong Yang Tested-by: Xin Hao Tested-by: Punit Agrawal Signed-off-by: Barry Song Signed-off-by: Yicong Yang Reviewed-by: Kefeng Wang Reviewed-by: Xin Hao Reviewed-by: Anshuman Khandual --- .../features/vm/TLB/arch-support.txt | 2 +- arch/arm64/Kconfig | 6 +++ arch/arm64/include/asm/tlbbatch.h | 12 +++++ arch/arm64/include/asm/tlbflush.h | 52 ++++++++++++++++++- arch/x86/include/asm/tlbflush.h | 5 +- include/linux/mm_types_task.h | 4 +- mm/rmap.c | 12 +++-- 7 files changed, 81 insertions(+), 12 deletions(-) create mode 100644 arch/arm64/include/asm/tlbbatch.h diff --git a/Documentation/features/vm/TLB/arch-support.txt b/Documentation/features/vm/TLB/arch-support.txt index 7f049c251a79..76208db88f3b 100644 --- a/Documentation/features/vm/TLB/arch-support.txt +++ b/Documentation/features/vm/TLB/arch-support.txt @@ -9,7 +9,7 @@ | alpha: | TODO | | arc: | TODO | | arm: | TODO | - | arm64: | N/A | + | arm64: | ok | | csky: | TODO | | hexagon: | TODO | | ia64: | TODO | diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 1023e896d46b..93b5f5f989a1 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -95,6 +95,7 @@ config ARM64 select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK + select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH if EXPERT select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT @@ -275,6 +276,11 @@ config ARM64_CONT_PMD_SHIFT default 5 if ARM64_16K_PAGES default 4 +config ARM64_NR_CPUS_FOR_BATCHED_TLB + int "Threshold to enable batched TLB flush" + default 8 + depends on ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH + config ARCH_MMAP_RND_BITS_MIN default 14 if ARM64_64K_PAGES default 16 if ARM64_16K_PAGES diff --git a/arch/arm64/include/asm/tlbbatch.h b/arch/arm64/include/asm/tlbbatch.h new file mode 100644 index 000000000000..fedb0b87b8db --- /dev/null +++ b/arch/arm64/include/asm/tlbbatch.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ARCH_ARM64_TLBBATCH_H +#define _ARCH_ARM64_TLBBATCH_H + +struct arch_tlbflush_unmap_batch { + /* + * For arm64, HW can do tlb shootdown, so we don't + * need to record cpumask for sending IPI + */ +}; + +#endif /* _ARCH_ARM64_TLBBATCH_H */ diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 412a3b9a3c25..41a763cf8c1b 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -254,17 +254,23 @@ static inline void flush_tlb_mm(struct mm_struct *mm) dsb(ish); } -static inline void flush_tlb_page_nosync(struct vm_area_struct *vma, +static inline void __flush_tlb_page_nosync(struct mm_struct *mm, unsigned long uaddr) { unsigned long addr; dsb(ishst); - addr = __TLBI_VADDR(uaddr, ASID(vma->vm_mm)); + addr = __TLBI_VADDR(uaddr, ASID(mm)); __tlbi(vale1is, addr); __tlbi_user(vale1is, addr); } +static inline void flush_tlb_page_nosync(struct vm_area_struct *vma, + unsigned long uaddr) +{ + return __flush_tlb_page_nosync(vma->vm_mm, uaddr); +} + static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long uaddr) { @@ -272,6 +278,48 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, dsb(ish); } +#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH + +static inline bool arch_tlbbatch_should_defer(struct mm_struct *mm) +{ + /* + * TLB batched flush is proved to be beneficial for systems with large + * number of CPUs, especially system with more than 8 CPUs. TLB shutdown + * is cheap on small systems which may not need this feature. So use + * a threshold for enabling this to avoid potential side effects on + * these platforms. + */ + if (num_online_cpus() < CONFIG_ARM64_NR_CPUS_FOR_BATCHED_TLB) + return false; + + /* + * TLB flush deferral is not required on systems, which are affected with + * ARM64_WORKAROUND_REPEAT_TLBI, as __tlbi()/__tlbi_user() implementation + * will have two consecutive TLBI instructions with a dsb(ish) in between + * defeating the purpose (i.e save overall 'dsb ish' cost). + */ +#ifdef CONFIG_ARM64_WORKAROUND_REPEAT_TLBI + if (unlikely(cpus_have_const_cap(ARM64_WORKAROUND_REPEAT_TLBI))) + return false; +#endif + + return true; +} + +static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, + struct mm_struct *mm, + unsigned long uaddr) +{ + __flush_tlb_page_nosync(mm, uaddr); +} + +static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) +{ + dsb(ish); +} + +#endif /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ + /* * This is meant to avoid soft lock-ups on large TLB flushing ranges and not * necessarily a performance improvement. diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 8a497d902c16..15cada9635c1 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -263,8 +263,9 @@ static inline u64 inc_mm_tlb_gen(struct mm_struct *mm) return atomic64_inc_return(&mm->context.tlb_gen); } -static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, - struct mm_struct *mm) +static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, + struct mm_struct *mm, + unsigned long uaddr) { inc_mm_tlb_gen(mm); cpumask_or(&batch->cpumask, &batch->cpumask, mm_cpumask(mm)); diff --git a/include/linux/mm_types_task.h b/include/linux/mm_types_task.h index 5414b5c6a103..aa44fff8bb9d 100644 --- a/include/linux/mm_types_task.h +++ b/include/linux/mm_types_task.h @@ -52,8 +52,8 @@ struct tlbflush_unmap_batch { #ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH /* * The arch code makes the following promise: generic code can modify a - * PTE, then call arch_tlbbatch_add_mm() (which internally provides all - * needed barriers), then call arch_tlbbatch_flush(), and the entries + * PTE, then call arch_tlbbatch_add_pending() (which internally provides + * all needed barriers), then call arch_tlbbatch_flush(), and the entries * will be flushed on all CPUs by the time that arch_tlbbatch_flush() * returns. */ diff --git a/mm/rmap.c b/mm/rmap.c index 38ccb700748c..a4e2c16a1a72 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -641,12 +641,13 @@ void try_to_unmap_flush_dirty(void) #define TLB_FLUSH_BATCH_PENDING_LARGE \ (TLB_FLUSH_BATCH_PENDING_MASK / 2) -static void set_tlb_ubc_flush_pending(struct mm_struct *mm, bool writable) +static void set_tlb_ubc_flush_pending(struct mm_struct *mm, bool writable, + unsigned long uaddr) { struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; int batch, nbatch; - arch_tlbbatch_add_mm(&tlb_ubc->arch, mm); + arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, uaddr); tlb_ubc->flush_required = true; /* @@ -724,7 +725,8 @@ void flush_tlb_batched_pending(struct mm_struct *mm) } } #else -static void set_tlb_ubc_flush_pending(struct mm_struct *mm, bool writable) +static void set_tlb_ubc_flush_pending(struct mm_struct *mm, bool writable, + unsigned long uaddr) { } @@ -1575,7 +1577,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, */ pteval = ptep_get_and_clear(mm, address, pvmw.pte); - set_tlb_ubc_flush_pending(mm, pte_dirty(pteval)); + set_tlb_ubc_flush_pending(mm, pte_dirty(pteval), address); } else { pteval = ptep_clear_flush(vma, address, pvmw.pte); } @@ -1956,7 +1958,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, */ pteval = ptep_get_and_clear(mm, address, pvmw.pte); - set_tlb_ubc_flush_pending(mm, pte_dirty(pteval)); + set_tlb_ubc_flush_pending(mm, pte_dirty(pteval), address); } else { pteval = ptep_clear_flush(vma, address, pvmw.pte); }