From patchwork Wed May 24 09:32:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 13253674 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0E58DC77B7A for ; Wed, 24 May 2023 09:32:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:Subject:Cc :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=2QzOBmyYeH/NDPiFAubF01sIgAWhTGpPJjBXMYKV3Qc=; b=3Ro0tuDkvqKezl +e1pw8iZkm1POJhAcqLWUCJuAb18abG7FxRnCJXEYzjfwiiOSTzA6JqV5Q0gEu70CZ4mUbdI3PyyR g+UQ6fk8tbTR2efwKhaF9rsP9n9Fn3nQtKbtxFqIFimDYSbHcR9x3MzKFbTTf/YvzmHcm0J1fUKJ5 LD2qg3RFMSksNQbd+9U7z1G/82wI1Z+7ODjjXxauf+36raezZt3OmoHyvPMCoaUAb0KIudAKZZTA1 FqZYcuuWeC4sPAN+FqcVkcpoh/SgJj/5zBSYKlmKsvSmH5bF3y+lKUR7XLId4Dk0Ag52Z3zf/wMhw x+pJIVi00rbcK/AZwSEw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q1kqz-00CyMi-0O; Wed, 24 May 2023 09:32:21 +0000 Received: from galois.linutronix.de ([2a0a:51c0:0:12e:550::1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q1kqw-00CyKj-10 for linux-arm-kernel@lists.infradead.org; Wed, 24 May 2023 09:32:19 +0000 From: Thomas Gleixner DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684920727; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type; bh=sOow+Z36HKhb8qHgH60zSkOdPPu0mlfd3xITXOcfKrk=; b=Fx/vUV95xyP8uirL+f6kxqVgm5AFscOUUmwjBZnEL+Dx9Pb37ZXxDi6zXzy5VnC9BmiWfj QgMgL6f5OV23x64PKZSsNdez1fG6cus8HRHWEMmGSGy2LrEfx80gs5k9EKZQeQAzvVCm0l ord9xUxw5wnAM8Lx13pPmZNPCcTRueg9PA6P0LCSPEgBtgGqw9TBNERM1Y2C5G2ahUkNNd VQyUKpTSSp2/KvhAYE9PGrXmBtLRbvWBFLuwfrgUNPOwbfPxragezE6yBXkqJjG03PMeOV 0rMImaQ2KStUnd+EFy3v+9tjzj6Zb/2bkORzBqEW63luvcx7pWaD255X85jlFw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684920727; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type; bh=sOow+Z36HKhb8qHgH60zSkOdPPu0mlfd3xITXOcfKrk=; b=RYWGxHrdmmT90IpirSHRmbCYUTtOPziBEvH1GNKkdm3Z0NyNyHj5dSHcl+GfhxrF4qbPPr z3Wq1RwdEkVwBPDA== To: linux-arm-kernel@lists.infradead.org Cc: Russell King , John Ogness , Arnd Bergmann Subject: [PATCH] ARM: tlb: Prevent flushing insane large ranges one by one Date: Wed, 24 May 2023 11:32:07 +0200 Message-ID: <87pm6qm5wo.ffs@tglx> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230524_023218_511659_1B62AE83 X-CRM114-Status: GOOD ( 15.45 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org vmalloc uses lazy TLB flushes for unmapped ranges to avoid excessive TLB flushing on every unmap. The lazy flushing coalesces unmapped ranges and invokes flush_tlb_kernel_range() with the combined range. The coalescing can result in ranges which spawn the full vmalloc address range. In the case of flushing an executable mapping in the module address space this range is extended to also flush the direct map alias. flush_tlb_kernel_range() then walks insane large ranges, the worst case observed was ~1.5GB. The range is flushed page by page, which takes several milliseconds to complete in the worst case and obviously affects all processes in the system. In the worst case observed this causes the runtime of a realtime task on an isolated CPU to be almost doubled over the normal worst case, which makes it miss the deadline. Cure this by sanity checking the range against a threshold and fall back to tlb_flush_all() when the range is too large. The default threshold is 32 pages, but for CPUs with CP15 this is evaluated at boot time via read_cpuid(CPUID_TLBTYPE) and set to the half of the TLB size. The vmalloc range coalescing could be improved to provide a list or array of ranges to flush, which allows to avoid overbroad flushing, but that's a major surgery and does not solve the problem of actual justified large range flushes which can happen due to the lazy flush mechanics in vmalloc. The lazy flush results in batching which is biased towards large range flushes by design. Fixes: db64fe02258f ("mm: rewrite vmap layer") Reported-by: John Ogness Debugged-by: John Ogness Signed-off-by: Thomas Gleixner Tested-by: John Ogness Link: https://lore.kernel.org/all/87a5y5a6kj.ffs@tglx --- arch/arm/include/asm/cputype.h | 5 +++++ arch/arm/include/asm/tlbflush.h | 2 ++ arch/arm/kernel/setup.c | 10 ++++++++++ arch/arm/kernel/smp_tlb.c | 4 ++++ 4 files changed, 21 insertions(+) --- a/arch/arm/include/asm/cputype.h +++ b/arch/arm/include/asm/cputype.h @@ -196,6 +196,11 @@ static inline unsigned int __attribute_c return read_cpuid(CPUID_MPUIR); } +static inline unsigned int __attribute_const__ read_cpuid_tlbsize(void) +{ + return 64 << ((read_cpuid(CPUID_TLBTYPE) >> 1) & 0x03); +} + #elif defined(CONFIG_CPU_V7M) static inline unsigned int __attribute_const__ read_cpuid_id(void) --- a/arch/arm/include/asm/tlbflush.h +++ b/arch/arm/include/asm/tlbflush.h @@ -210,6 +210,8 @@ struct cpu_tlb_fns { unsigned long tlb_flags; }; +extern unsigned int tlb_flush_all_threshold; + /* * Select the calling method */ --- a/arch/arm/kernel/setup.c +++ b/arch/arm/kernel/setup.c @@ -90,6 +90,8 @@ EXPORT_SYMBOL(__machine_arch_type); unsigned int cacheid __read_mostly; EXPORT_SYMBOL(cacheid); +unsigned int tlb_flush_all_threshold __ro_after_init = 32; + unsigned int __atags_pointer __initdata; unsigned int system_rev; @@ -356,6 +358,13 @@ static void __init cacheid_init(void) cache_is_vipt_nonaliasing() ? "VIPT nonaliasing" : "unknown"); } +static void __init tlbsize_init(void) +{ +#ifdef CONFIG_CPU_CP15 + tlb_flush_all_threshold = read_cpuid_tlbsize() / 2; +#endif +} + /* * These functions re-use the assembly code in head.S, which * already provide the required functionality. @@ -747,6 +756,7 @@ static void __init setup_processor(void) elf_hwcap_fixup(); cacheid_init(); + tlbsize_init(); cpu_init(); } --- a/arch/arm/kernel/smp_tlb.c +++ b/arch/arm/kernel/smp_tlb.c @@ -234,6 +234,10 @@ void flush_tlb_range(struct vm_area_stru void flush_tlb_kernel_range(unsigned long start, unsigned long end) { + if ((end - start) > (tlb_flush_all_threshold << PAGE_SHIFT)) { + flush_tlb_all(); + return; + } if (tlb_ops_need_broadcast()) { struct tlb_args ta; ta.ta_start = start;