From patchwork Mon Apr 18 10:53:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anup Patel X-Patchwork-Id: 12816478 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 13E5EC433FE for ; Mon, 18 Apr 2022 10:54:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=faR1DsfutCC/02isRDBNHtO6De8cT0N9OG/GXjsUAs8=; b=UYHQEuBI1a0ZVI /8Pnft0bo01ChIpHsUwIxkUPxAiY8BtmbDlUafvd8d5bymFIyl7vXx1TOoCesAl07/Z58gEH2EotD sBgrPdGmTqLb+IecCz/a02pgCZFy5N/zVYR7/n7Yqjgywaeq+jVdywX0NM8olY+vq4VTni6AhG0WI zMYPWW94wBFzfTG07y1VmFok1/UNAM6U5NZMu+nuKDuZ3Q4JuZ7EO0+HM3UqDjKdxqPUBzuQ/Q6q3 9ff7ObIJIH9lkn9hEj99E2WrSxcwtBkLDZBb5j08LoAPKJV1Ah+ryNDWDTA2MyhwxZAUSV/iktO3C ll69sBJUUweu4WLxQfFQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ngP1L-00GNaH-63; Mon, 18 Apr 2022 10:54:15 +0000 Received: from mail-pg1-x536.google.com ([2607:f8b0:4864:20::536]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ngP1H-00GNXd-KN for linux-riscv@lists.infradead.org; Mon, 18 Apr 2022 10:54:13 +0000 Received: by mail-pg1-x536.google.com with SMTP id t13so18225708pgn.8 for ; Mon, 18 Apr 2022 03:54:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ventanamicro.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gulBS1a3GDwTjV3gPNI1Ld+RqIL2YPtNwx+Ha/YsDb0=; b=FGwNqyWxfBjnnV7o8d4tmOubhRr8eZzYJsSS9Xvd01EjT/gwp14zLw0t0G8ms0v1K6 QBErSvGTW9wFbPLmWRRYXW9qA1ahUrIeDTxusAnNBDY/mKpPLzJEGfA3iaP/UsU0+sY5 Gb9nAv7P7f1D9vcY2pqQKu5V07QGVbaFmGJCYoSQBXkvgjIpMsDKbbyBDOqX5ns9CbG4 bVRcQy/KleIK1o/wa/1f3Ylh2SCauA5IwJa9FN4QOzhtlBGWzGLg0zFObb+0b6NQmpF6 4Ohm6C1rE1bHGqsP1zyT8zXQqItrDZlvvAK5whV5y/3s+SRPnOORqkQDxXtSsfNoBYRy LGAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gulBS1a3GDwTjV3gPNI1Ld+RqIL2YPtNwx+Ha/YsDb0=; b=cvY95DXidsm+exfV4L3Obypd35lY45gGWLXIbyXo0NV2E4BZn9XuP9yYy56Rigx1gV cBgrveGCt0iq5hknVLCr99l8a+mIZCqpmCALdmm+v+YuUF5pPkEz/CSvGVUNxVL0Vpf4 1AD28tyVAu8xD5R4DWstEzRYrXIF3RI9meFXz9ueGFcbReiHrI0hBj0JhsgBsEB1A2HH TaJGFCHO2GJ5CbwORvxXWeeg4E9L5yd6MRU0Yy7N8dl+z9guF/0k03UU07kxppKDDQpn 6DpBnM2hk/ndoq7ZZta9Q5heQhcH7hN+KFWdIUZp9VGFHUveiIzPbDBBp9wACyrSsdP5 f/Fw== X-Gm-Message-State: AOAM532Wg1aE3zAJV60iiiBz7dBRIjv3CtDr7RTc4hsTUEmi86zg17sJ 6IKFEEsWrCMRN5Jfxgy6tiY1YpLI0RNhyA== X-Google-Smtp-Source: ABdhPJxuX8fre1oRyzLWxGkstaaivQvs3xfUcxaT5Sb+gdlrCQXN7FheZ6kBWZyqeP28yMDxGWhumA== X-Received: by 2002:a63:2d06:0:b0:39c:f643:8720 with SMTP id t6-20020a632d06000000b0039cf6438720mr9573100pgt.371.1650279251099; Mon, 18 Apr 2022 03:54:11 -0700 (PDT) Received: from localhost.localdomain ([122.172.241.223]) by smtp.gmail.com with ESMTPSA id j13-20020a056a00130d00b004f1025a4361sm12986278pfu.202.2022.04.18.03.54.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Apr 2022 03:54:10 -0700 (PDT) From: Anup Patel To: Palmer Dabbelt , Paul Walmsley , Thomas Gleixner , Marc Zyngier , Daniel Lezcano Cc: Atish Patra , Alistair Francis , Anup Patel , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Anup Patel Subject: [PATCH v6 6/7] RISC-V: Use IPIs for remote TLB flush when possible Date: Mon, 18 Apr 2022 16:23:04 +0530 Message-Id: <20220418105305.1196665-7-apatel@ventanamicro.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220418105305.1196665-1-apatel@ventanamicro.com> References: <20220418105305.1196665-1-apatel@ventanamicro.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220418_035411_710800_39FFD17A X-CRM114-Status: GOOD ( 13.53 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org If we have specialized interrupt controller (such as AIA IMSIC) which allows supervisor mode to directly inject IPIs without any assistance from M-mode or HS-mode then using such specialized interrupt controller, we can do remote TLB flushes directly from supervisor mode instead of using the SBI RFENCE calls. This patch extends remote TLB flush functions to use supervisor mode IPIs whenever direct supervisor mode IPIs.are supported by interrupt controller. Signed-off-by: Anup Patel --- arch/riscv/mm/tlbflush.c | 93 +++++++++++++++++++++++++++++++++------- 1 file changed, 78 insertions(+), 15 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 37ed760d007c..27a7db8eb2c4 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -23,14 +23,62 @@ static inline void local_flush_tlb_page_asid(unsigned long addr, : "memory"); } +static inline void local_flush_tlb_range(unsigned long start, + unsigned long size, unsigned long stride) +{ + if (size <= stride) + local_flush_tlb_page(start); + else + local_flush_tlb_all(); +} + +static inline void local_flush_tlb_range_asid(unsigned long start, + unsigned long size, unsigned long stride, unsigned long asid) +{ + if (size <= stride) + local_flush_tlb_page_asid(start, asid); + else + local_flush_tlb_all_asid(asid); +} + +static void __ipi_flush_tlb_all(void *info) +{ + local_flush_tlb_all(); +} + void flush_tlb_all(void) { - sbi_remote_sfence_vma(NULL, 0, -1); + if (riscv_use_ipi_for_rfence()) + on_each_cpu(__ipi_flush_tlb_all, NULL, 1); + else + sbi_remote_sfence_vma(NULL, 0, -1); +} + +struct flush_tlb_range_data { + unsigned long asid; + unsigned long start; + unsigned long size; + unsigned long stride; +}; + +static void __ipi_flush_tlb_range_asid(void *info) +{ + struct flush_tlb_range_data *d = info; + + local_flush_tlb_range_asid(d->start, d->size, d->stride, d->asid); +} + +static void __ipi_flush_tlb_range(void *info) +{ + struct flush_tlb_range_data *d = info; + + local_flush_tlb_range(d->start, d->size, d->stride); } -static void __sbi_tlb_flush_range(struct mm_struct *mm, unsigned long start, - unsigned long size, unsigned long stride) +static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, + unsigned long size, unsigned long stride) { + struct flush_tlb_range_data ftd; struct cpumask *cmask = mm_cpumask(mm); unsigned int cpuid; bool broadcast; @@ -45,19 +93,34 @@ static void __sbi_tlb_flush_range(struct mm_struct *mm, unsigned long start, unsigned long asid = atomic_long_read(&mm->context.id); if (broadcast) { - sbi_remote_sfence_vma_asid(cmask, start, size, asid); - } else if (size <= stride) { - local_flush_tlb_page_asid(start, asid); + if (riscv_use_ipi_for_rfence()) { + ftd.asid = asid; + ftd.start = start; + ftd.size = size; + ftd.stride = stride; + on_each_cpu_mask(cmask, + __ipi_flush_tlb_range_asid, + &ftd, 1); + } else + sbi_remote_sfence_vma_asid(cmask, + start, size, asid); } else { - local_flush_tlb_all_asid(asid); + local_flush_tlb_range_asid(start, size, stride, asid); } } else { if (broadcast) { - sbi_remote_sfence_vma(cmask, start, size); - } else if (size <= stride) { - local_flush_tlb_page(start); + if (riscv_use_ipi_for_rfence()) { + ftd.asid = 0; + ftd.start = start; + ftd.size = size; + ftd.stride = stride; + on_each_cpu_mask(cmask, + __ipi_flush_tlb_range, + &ftd, 1); + } else + sbi_remote_sfence_vma(cmask, start, size); } else { - local_flush_tlb_all(); + local_flush_tlb_range(start, size, stride); } } @@ -66,23 +129,23 @@ static void __sbi_tlb_flush_range(struct mm_struct *mm, unsigned long start, void flush_tlb_mm(struct mm_struct *mm) { - __sbi_tlb_flush_range(mm, 0, -1, PAGE_SIZE); + __flush_tlb_range(mm, 0, -1, PAGE_SIZE); } void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) { - __sbi_tlb_flush_range(vma->vm_mm, addr, PAGE_SIZE, PAGE_SIZE); + __flush_tlb_range(vma->vm_mm, addr, PAGE_SIZE, PAGE_SIZE); } void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - __sbi_tlb_flush_range(vma->vm_mm, start, end - start, PAGE_SIZE); + __flush_tlb_range(vma->vm_mm, start, end - start, PAGE_SIZE); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - __sbi_tlb_flush_range(vma->vm_mm, start, end - start, PMD_SIZE); + __flush_tlb_range(vma->vm_mm, start, end - start, PMD_SIZE); } #endif