From patchwork Fri Sep 20 03:55:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13808117 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A34DCCF58C5 for ; Fri, 20 Sep 2024 03:56:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Message-ID:Date:Subject:CC:To:From: Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender :Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=+RLxUcpsQ18JOHpFvTsUFSbtI2/Q4YLARIbtvjWC4yw=; b=fRPviGeVJO8emO3x/H7JaziWVo SUwJM2A+GLKvM18wCm1Aceei0feQwN/YnoQ++167mifLfQnqAv2QyPBxkFH6ifmp16MyggwHWDDjZ dLkXhlfzLT0gKTWIM4TaiJHHiAEpMRwrfzKrfc2UGrbYxkcjFQDERW+17MeTw+9gLRJek1O1XBt54 ydN6kZOhuYvuF95QqzjyvgqxcF2+6q/puY2odLMGr7HNrdfZgB5mudR9BzxLpmWOuYQwbOUz2381c v5GORKqO7YFliqFM8kXG7LEdFS0ZofuU422wrqYM2BKO5+jhjvNLuqxPY28134ZEGHWcrGGMmirn0 npyJJ1Jw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1srUl9-0000000BOVi-3P1r; Fri, 20 Sep 2024 03:56:43 +0000 Received: from szxga05-in.huawei.com ([45.249.212.191]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1srUk2-0000000BORq-0pui for linux-arm-kernel@lists.infradead.org; Fri, 20 Sep 2024 03:55:36 +0000 Received: from mail.maildlp.com (unknown [172.19.163.17]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4X8z6V1TLBz2QTtq; Fri, 20 Sep 2024 11:54:42 +0800 (CST) Received: from dggpemf100008.china.huawei.com (unknown [7.185.36.138]) by mail.maildlp.com (Postfix) with ESMTPS id 908F81A0190; Fri, 20 Sep 2024 11:55:22 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemf100008.china.huawei.com (7.185.36.138) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Fri, 20 Sep 2024 11:55:22 +0800 From: Kefeng Wang To: Catalin Marinas , Will Deacon CC: Ryan Roberts , , Kefeng Wang , Yicong Yang Subject: [PATCH v2] arm64: optimize flush tlb kernel range Date: Fri, 20 Sep 2024 11:55:06 +0800 Message-ID: <20240920035506.346316-1-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To dggpemf100008.china.huawei.com (7.185.36.138) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240919_205534_882717_F6FF40D8 X-CRM114-Status: GOOD ( 10.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently the kernel TLBs is flushed page by page if the target VA range is less than MAX_DVM_OPS * PAGE_SIZE, otherwise we'll brutally issue a TLBI ALL. But we could optimize it when CPU supports TLB range operations, convert to use __flush_tlb_range_op() like other tlb range flush to improve performance. Co-developed-by: Yicong Yang Signed-off-by: Yicong Yang Signed-off-by: Kefeng Wang --- v2: - address Catalin's comments and use __flush_tlb_range_op() directly arch/arm64/include/asm/tlbflush.h | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 95fbc8c05607..42f0ec14fb2c 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -492,19 +492,29 @@ static inline void flush_tlb_range(struct vm_area_struct *vma, static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end) { - unsigned long addr; + const unsigned long stride = PAGE_SIZE; + unsigned long pages; + + start = round_down(start, stride); + end = round_up(end, stride); + pages = (end - start) >> PAGE_SHIFT; - if ((end - start) > (MAX_DVM_OPS * PAGE_SIZE)) { + /* + * When not uses TLB range ops, we can handle up to + * (MAX_DVM_OPS - 1) pages; + * When uses TLB range ops, we can handle up to + * MAX_TLBI_RANGE_PAGES pages. + */ + if ((!system_supports_tlb_range() && + (end - start) >= (MAX_DVM_OPS * stride)) || + pages > MAX_TLBI_RANGE_PAGES) { flush_tlb_all(); return; } - start = __TLBI_VADDR(start, 0); - end = __TLBI_VADDR(end, 0); - dsb(ishst); - for (addr = start; addr < end; addr += 1 << (PAGE_SHIFT - 12)) - __tlbi(vaale1is, addr); + __flush_tlb_range_op(vaale1is, start, pages, stride, 0, + TLBI_TTL_UNKNOWN, false, lpa2_is_enabled()); dsb(ish); isb(); }