From patchwork Tue Jan 25 05:42:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anup Patel X-Patchwork-Id: 12723302 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E89E9C433FE for ; Tue, 25 Jan 2022 05:44:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=aZbzUQluEkmkWWuumIUTd9d9OPlgIc84l5JYMDhh4K8=; b=wYf3iQHWidPC2D UgXPmbxNmFNl9V62VormoqWC740IkQ8dtCZh30Le+9XNLN1/nf8kO5xLUWETURhYhZLiSi9sIWaaD XRURbYUiQPwvCoLWipMsBedND0e0qvq6zPNftoZ6e6maG9Bado6R6rlv2qqvDlZRT8Qq5e79HDKAL qLoVo1JtbaZdkoMb0c6PDnm+GEEZNU2BdlSFBp1dKfNE2lj+satoaiELEclYUIRRqDzoo2495MSYU 8yO8SiNzdWHC3iOGR9VC4zmle314BPNyfd/D3tXfdva6khW1DwMq3oNSZLJiVH9QkqQ+rUKuzHqC7 EIS9tNtldOoTX5CyfndQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nCEct-006Vw3-Eo; Tue, 25 Jan 2022 05:44:19 +0000 Received: from mail-pg1-x534.google.com ([2607:f8b0:4864:20::534]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nCEcq-006VuB-RV for linux-riscv@lists.infradead.org; Tue, 25 Jan 2022 05:44:18 +0000 Received: by mail-pg1-x534.google.com with SMTP id z131so6386559pgz.12 for ; Mon, 24 Jan 2022 21:44:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ventanamicro.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ANfiyTNuLH3K34lM7cYA66sM88G94rOT/TdSDWy1dfM=; b=Z2l3OEwPTmcht1x90krU+HvQuTaWfAECasawdJmUmykWoDMEy1GSasrIoFi1UnDWAE Ye/rxw3S4BZl3JeodXOFKlJu+zb5l9yujbNh3LwcmQSTgv3VjTSPVSJObmo56q3kqw88 j5KeMhQUACWMLkNNo9fWDtrFHIcV/x9YjCeYPzku04KhdiAUyGwTlZufjBsdTwwJtwyd xEBaIyIwCsYpXQnJbsBfAjrlnKeEJM7pK/aw1YRN9bw64S+iLpoXAbkddKabkQ1qMPIo CdCcXVL9uOifEcjynI8Jbu53AjS6U+rSaVI3Us/xaiFGIkflxxwbdlJ8AblslAD5rQ09 QjHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ANfiyTNuLH3K34lM7cYA66sM88G94rOT/TdSDWy1dfM=; b=3YsSa3NPX72oyiWGuPqVcrhcu7oM2IVpE0EJBPIRJuXgf71ztZIxT+ho4T6PS1AZhx iD1lZmn05HPPW9lIpIF97uShLVVPm3MzQXJ+ckhwZexaJjqkuDsvuNmi+iuAW48Xw/Rd clVhu7jlU4yOynKPnQMQLF4uhVM+rEbxmzQ1L2J9qorV8Z+RPe6YSIDAo+5/3xzjrp/w cQCSGGewoqM4lNHGRJoN1plQzhG4hlm5KKpAVb0x+iBlMO31aOl/Trr0dbeBN3PwhKms ziIVUAkEIqDwa5KRPvMfSl76WK/4iM6w7lz3YpYAzjDHeW0Fs9CBg8FVvBCTFgQRbVBj K4tA== X-Gm-Message-State: AOAM533whKrvnU3Z8qMxHtZ/anPBPGR1GTIMrP+bq6hLP1gWwJ3SCMxI JuTZC2bdY6+9J1knl6bIthamng== X-Google-Smtp-Source: ABdhPJy0Zt7AAbcR7aurgb/+aFMWVQO2AH2G40/mq0eF7GIvSo6FCzs8tY79M7CoMlpekOisT6wp7w== X-Received: by 2002:a65:578b:: with SMTP id b11mr6140482pgr.318.1643089455716; Mon, 24 Jan 2022 21:44:15 -0800 (PST) Received: from localhost.localdomain ([122.179.14.218]) by smtp.gmail.com with ESMTPSA id c6sm19524508pfl.200.2022.01.24.21.44.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Jan 2022 21:44:15 -0800 (PST) From: Anup Patel To: Palmer Dabbelt , Paul Walmsley , Thomas Gleixner , Marc Zyngier , Daniel Lezcano , Rob Herring Cc: Atish Patra , Alistair Francis , Anup Patel , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Anup Patel Subject: [PATCH 5/6] RISC-V: Use IPIs for remote TLB flush when possible Date: Tue, 25 Jan 2022 11:12:16 +0530 Message-Id: <20220125054217.383482-6-apatel@ventanamicro.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220125054217.383482-1-apatel@ventanamicro.com> References: <20220125054217.383482-1-apatel@ventanamicro.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220124_214416_919095_D2AEBF44 X-CRM114-Status: GOOD ( 14.89 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org If IPI calls are injected using SBI IPI calls then remote TLB flush using SBI RFENCE calls is much faster because using IPIs for remote TLB flush would still endup as SBI IPI calls with extra processing on kernel side. It is now possible to have specialized hardware (such as RISC-V AIA and RISC-V ACLINT) which allows S-mode software to directly inject IPIs without any assistance from M-mode runtime firmware. This patch extends remote TLB flush functions to use IPIs whenever underlying IPI operations are suitable for remote FENCEs. Signed-off-by: Anup Patel --- arch/riscv/mm/tlbflush.c | 93 +++++++++++++++++++++++++++++++++------- 1 file changed, 78 insertions(+), 15 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 37ed760d007c..27a7db8eb2c4 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -23,14 +23,62 @@ static inline void local_flush_tlb_page_asid(unsigned long addr, : "memory"); } +static inline void local_flush_tlb_range(unsigned long start, + unsigned long size, unsigned long stride) +{ + if (size <= stride) + local_flush_tlb_page(start); + else + local_flush_tlb_all(); +} + +static inline void local_flush_tlb_range_asid(unsigned long start, + unsigned long size, unsigned long stride, unsigned long asid) +{ + if (size <= stride) + local_flush_tlb_page_asid(start, asid); + else + local_flush_tlb_all_asid(asid); +} + +static void __ipi_flush_tlb_all(void *info) +{ + local_flush_tlb_all(); +} + void flush_tlb_all(void) { - sbi_remote_sfence_vma(NULL, 0, -1); + if (riscv_use_ipi_for_rfence()) + on_each_cpu(__ipi_flush_tlb_all, NULL, 1); + else + sbi_remote_sfence_vma(NULL, 0, -1); +} + +struct flush_tlb_range_data { + unsigned long asid; + unsigned long start; + unsigned long size; + unsigned long stride; +}; + +static void __ipi_flush_tlb_range_asid(void *info) +{ + struct flush_tlb_range_data *d = info; + + local_flush_tlb_range_asid(d->start, d->size, d->stride, d->asid); +} + +static void __ipi_flush_tlb_range(void *info) +{ + struct flush_tlb_range_data *d = info; + + local_flush_tlb_range(d->start, d->size, d->stride); } -static void __sbi_tlb_flush_range(struct mm_struct *mm, unsigned long start, - unsigned long size, unsigned long stride) +static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, + unsigned long size, unsigned long stride) { + struct flush_tlb_range_data ftd; struct cpumask *cmask = mm_cpumask(mm); unsigned int cpuid; bool broadcast; @@ -45,19 +93,34 @@ static void __sbi_tlb_flush_range(struct mm_struct *mm, unsigned long start, unsigned long asid = atomic_long_read(&mm->context.id); if (broadcast) { - sbi_remote_sfence_vma_asid(cmask, start, size, asid); - } else if (size <= stride) { - local_flush_tlb_page_asid(start, asid); + if (riscv_use_ipi_for_rfence()) { + ftd.asid = asid; + ftd.start = start; + ftd.size = size; + ftd.stride = stride; + on_each_cpu_mask(cmask, + __ipi_flush_tlb_range_asid, + &ftd, 1); + } else + sbi_remote_sfence_vma_asid(cmask, + start, size, asid); } else { - local_flush_tlb_all_asid(asid); + local_flush_tlb_range_asid(start, size, stride, asid); } } else { if (broadcast) { - sbi_remote_sfence_vma(cmask, start, size); - } else if (size <= stride) { - local_flush_tlb_page(start); + if (riscv_use_ipi_for_rfence()) { + ftd.asid = 0; + ftd.start = start; + ftd.size = size; + ftd.stride = stride; + on_each_cpu_mask(cmask, + __ipi_flush_tlb_range, + &ftd, 1); + } else + sbi_remote_sfence_vma(cmask, start, size); } else { - local_flush_tlb_all(); + local_flush_tlb_range(start, size, stride); } } @@ -66,23 +129,23 @@ static void __sbi_tlb_flush_range(struct mm_struct *mm, unsigned long start, void flush_tlb_mm(struct mm_struct *mm) { - __sbi_tlb_flush_range(mm, 0, -1, PAGE_SIZE); + __flush_tlb_range(mm, 0, -1, PAGE_SIZE); } void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) { - __sbi_tlb_flush_range(vma->vm_mm, addr, PAGE_SIZE, PAGE_SIZE); + __flush_tlb_range(vma->vm_mm, addr, PAGE_SIZE, PAGE_SIZE); } void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - __sbi_tlb_flush_range(vma->vm_mm, start, end - start, PAGE_SIZE); + __flush_tlb_range(vma->vm_mm, start, end - start, PAGE_SIZE); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - __sbi_tlb_flush_range(vma->vm_mm, start, end - start, PMD_SIZE); + __flush_tlb_range(vma->vm_mm, start, end - start, PMD_SIZE); } #endif