From patchwork Sat Dec 3 06:46:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anup Patel X-Patchwork-Id: 13063437 X-Patchwork-Delegate: palmer@dabbelt.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EBED6C4332F for ; Sat, 3 Dec 2022 06:47:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=5/z4ZfdRO3B1SFqqjJprhm/vuX8WnnmaSd1rDdIMEZs=; b=jjbIpUaTVTNF2H kIxQRiliXep9i+w7r4XzPnjSUP4n+gJdA0GVtIVUV/fpRwvrkARUXOIvuHk1tiZX2eGkdXEKeVZbg XCiGULLI4PHqsTCVjrax1rVpDE7HyP1G9k+1ceHDPtwHn46ejTOkhKtqQsYWLns7FAQTyYtM1uumn +em9DYLYQ3NLzLnDaEyzgEFM0ZdTuQkOsyLlIdHBAdehWYWX/a6vEWGbolVjufMpEGqTsFN5eLXqX 5ssm+2nT+W05Imv/bSRMDiE2NS38az4LwsD77QWKxe94++/mPfR8gX83H7oxF+LuF9YmGQqRqcR7h cXL7iSCm95bVFBiJ4zIg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1p1MJ5-002dv1-4y; Sat, 03 Dec 2022 06:47:27 +0000 Received: from mail-pg1-x52a.google.com ([2607:f8b0:4864:20::52a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1p1MJ1-002dpz-1K for linux-riscv@lists.infradead.org; Sat, 03 Dec 2022 06:47:24 +0000 Received: by mail-pg1-x52a.google.com with SMTP id 6so6130295pgm.6 for ; Fri, 02 Dec 2022 22:47:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ventanamicro.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PXYoHLXQCavco5NNh0sXuOKKVYR0blfrb+/AP/qt1J8=; b=SRWO1OIPm7D9r5xdKjxp4xaECGXNSkdwStSz7PxxiJv4HkPwAgC9enbPktE2nZfVaS FJ4cXx3XVG3XBnKbigJUdvitSNRYo5L5I3eUCtUJIgfTp08sbQTixF1PXo8b7D7PEdLb iH784Sx+3URzIDr4BkLiPAANGDipXV/4FgE0ST5P7cTV4L7FQ7AyCogSsZqm5QTt5Zd8 wH5XRe02PWbkHp1m8dR42LGXNYTaZO/Z8/ssxRGtMGukZJ+hMuuswbOGK/a3Bu60srED m+KW2sQ3Y2NozzTmpUA2UKGbvkviHok4vDvh7VJyJzBvu16Qy2STc2DB75IsQMsJNJSC EKNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PXYoHLXQCavco5NNh0sXuOKKVYR0blfrb+/AP/qt1J8=; b=Peu0XytHyYkzkHMASc7zLZWCCOq21FjlF/CKTnTj5PslJ1NOFZh2p8K7dC1/Nw533T jVbQ3mTcp66yn9NfKorrrpfhZdPNuyi5vhj7SQWx7MaeoKZdczWCNRFIgrR8P9NdrWp6 7NYGaLmeRlZ8+sz3JLuiJZ9ZJkgwKZ7PcmVIy8wIkmivzQeioKmE3OTTIHek+I36H0hq GqmZ3FivLMVGVqRR2TcMD9pZyGLjHjqHI6FVoLJ3dPtuysCYAMaaFjmGAkZ4Tzdo90Uj AWgOMnnE4QAqIGGa+8fQPeMtlG9CLEcqL2FpM2wv949EmNwCXAIgYrvwA93OszSbPp1S Df2A== X-Gm-Message-State: ANoB5pk0bac47q5x9VIGqf1NRS/HAi28QRWIkWI8/gqNNfXsgIy9yEdi YdT+OXkbgQ/BqDX0h4wWf3GrSg== X-Google-Smtp-Source: AA0mqf6w60bkdL+GArK2RcsmZfct++Y+KVRtsmVscRai6QBVbF8LDWExSXNuXk2SPGv79T41xUdhNw== X-Received: by 2002:a63:1920:0:b0:478:7e03:1ffe with SMTP id z32-20020a631920000000b004787e031ffemr7834501pgl.209.1670050040099; Fri, 02 Dec 2022 22:47:20 -0800 (PST) Received: from anup-ubuntu-vm.localdomain ([103.97.165.210]) by smtp.gmail.com with ESMTPSA id t17-20020aa79471000000b00575c8242849sm6102091pfq.69.2022.12.02.22.47.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Dec 2022 22:47:19 -0800 (PST) From: Anup Patel To: Palmer Dabbelt , Paul Walmsley , Thomas Gleixner , Marc Zyngier , Daniel Lezcano Cc: Hector Martin , Sven Peter , Alyssa Rosenzweig , Atish Patra , Alistair Francis , Anup Patel , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, asahi@lists.linux.dev, Anup Patel , Atish Patra Subject: [PATCH v15 6/9] RISC-V: Use IPIs for remote TLB flush when possible Date: Sat, 3 Dec 2022 12:16:26 +0530 Message-Id: <20221203064629.1601299-7-apatel@ventanamicro.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221203064629.1601299-1-apatel@ventanamicro.com> References: <20221203064629.1601299-1-apatel@ventanamicro.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221202_224723_101096_1410296C X-CRM114-Status: GOOD ( 12.61 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org If we have specialized interrupt controller (such as AIA IMSIC) which allows supervisor mode to directly inject IPIs without any assistance from M-mode or HS-mode then using such specialized interrupt controller, we can do remote TLB flushes directly from supervisor mode instead of using the SBI RFENCE calls. This patch extends remote TLB flush functions to use supervisor mode IPIs whenever direct supervisor mode IPIs.are supported by interrupt controller. Signed-off-by: Anup Patel Reviewed-by: Atish Patra --- arch/riscv/mm/tlbflush.c | 93 +++++++++++++++++++++++++++++++++------- 1 file changed, 78 insertions(+), 15 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 37ed760d007c..27a7db8eb2c4 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -23,14 +23,62 @@ static inline void local_flush_tlb_page_asid(unsigned long addr, : "memory"); } +static inline void local_flush_tlb_range(unsigned long start, + unsigned long size, unsigned long stride) +{ + if (size <= stride) + local_flush_tlb_page(start); + else + local_flush_tlb_all(); +} + +static inline void local_flush_tlb_range_asid(unsigned long start, + unsigned long size, unsigned long stride, unsigned long asid) +{ + if (size <= stride) + local_flush_tlb_page_asid(start, asid); + else + local_flush_tlb_all_asid(asid); +} + +static void __ipi_flush_tlb_all(void *info) +{ + local_flush_tlb_all(); +} + void flush_tlb_all(void) { - sbi_remote_sfence_vma(NULL, 0, -1); + if (riscv_use_ipi_for_rfence()) + on_each_cpu(__ipi_flush_tlb_all, NULL, 1); + else + sbi_remote_sfence_vma(NULL, 0, -1); +} + +struct flush_tlb_range_data { + unsigned long asid; + unsigned long start; + unsigned long size; + unsigned long stride; +}; + +static void __ipi_flush_tlb_range_asid(void *info) +{ + struct flush_tlb_range_data *d = info; + + local_flush_tlb_range_asid(d->start, d->size, d->stride, d->asid); +} + +static void __ipi_flush_tlb_range(void *info) +{ + struct flush_tlb_range_data *d = info; + + local_flush_tlb_range(d->start, d->size, d->stride); } -static void __sbi_tlb_flush_range(struct mm_struct *mm, unsigned long start, - unsigned long size, unsigned long stride) +static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, + unsigned long size, unsigned long stride) { + struct flush_tlb_range_data ftd; struct cpumask *cmask = mm_cpumask(mm); unsigned int cpuid; bool broadcast; @@ -45,19 +93,34 @@ static void __sbi_tlb_flush_range(struct mm_struct *mm, unsigned long start, unsigned long asid = atomic_long_read(&mm->context.id); if (broadcast) { - sbi_remote_sfence_vma_asid(cmask, start, size, asid); - } else if (size <= stride) { - local_flush_tlb_page_asid(start, asid); + if (riscv_use_ipi_for_rfence()) { + ftd.asid = asid; + ftd.start = start; + ftd.size = size; + ftd.stride = stride; + on_each_cpu_mask(cmask, + __ipi_flush_tlb_range_asid, + &ftd, 1); + } else + sbi_remote_sfence_vma_asid(cmask, + start, size, asid); } else { - local_flush_tlb_all_asid(asid); + local_flush_tlb_range_asid(start, size, stride, asid); } } else { if (broadcast) { - sbi_remote_sfence_vma(cmask, start, size); - } else if (size <= stride) { - local_flush_tlb_page(start); + if (riscv_use_ipi_for_rfence()) { + ftd.asid = 0; + ftd.start = start; + ftd.size = size; + ftd.stride = stride; + on_each_cpu_mask(cmask, + __ipi_flush_tlb_range, + &ftd, 1); + } else + sbi_remote_sfence_vma(cmask, start, size); } else { - local_flush_tlb_all(); + local_flush_tlb_range(start, size, stride); } } @@ -66,23 +129,23 @@ static void __sbi_tlb_flush_range(struct mm_struct *mm, unsigned long start, void flush_tlb_mm(struct mm_struct *mm) { - __sbi_tlb_flush_range(mm, 0, -1, PAGE_SIZE); + __flush_tlb_range(mm, 0, -1, PAGE_SIZE); } void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) { - __sbi_tlb_flush_range(vma->vm_mm, addr, PAGE_SIZE, PAGE_SIZE); + __flush_tlb_range(vma->vm_mm, addr, PAGE_SIZE, PAGE_SIZE); } void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - __sbi_tlb_flush_range(vma->vm_mm, start, end - start, PAGE_SIZE); + __flush_tlb_range(vma->vm_mm, start, end - start, PAGE_SIZE); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - __sbi_tlb_flush_range(vma->vm_mm, start, end - start, PMD_SIZE); + __flush_tlb_range(vma->vm_mm, start, end - start, PMD_SIZE); } #endif