From patchwork Tue Jan 2 22:00:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13509548 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3DEEFC47079 for ; Tue, 2 Jan 2024 22:02:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=5OKBqdISZt7wenfkAjRwFURyJvfFiLP2kwzSD/Qv1jU=; b=gWtAVw6oPM+c42 00o6u2k9F/5YR2vzFIAvxo3kXEowA9UajGmQU1g9YSJnJtKGn0JghGgYpqChlMwCfQZtOJS6pYRiS etZ8dubQyZv1LGE6C6A+TwbbGxC0TIgdfj32uE/ThFBL8CMbEAobla8dP/baaLX+TUicU1uyHA2VF YVh40NTF5fDzk82msvSWjm9SLy23LXJ7vowEY/JKKgSStKisQm7IxlR4rWvp//3EGsHH3p/g5UsgH rMIrhrVrWiScLatTjvzQERoRQ6rbJSi41vlpzl/DmFj3EuiqYf++AI2sAVQhbkf2nCPmjvfpYQV9O np6oLLFo/xiHTjQ77ptw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpV-0098Rr-1d; Tue, 02 Jan 2024 22:01:45 +0000 Received: from mail-pj1-x1034.google.com ([2607:f8b0:4864:20::1034]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpQ-0098Nc-1v for linux-riscv@lists.infradead.org; Tue, 02 Jan 2024 22:01:43 +0000 Received: by mail-pj1-x1034.google.com with SMTP id 98e67ed59e1d1-28c7e30c83fso3031379a91.1 for ; Tue, 02 Jan 2024 14:01:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232898; x=1704837698; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Sx1CrYCYqcy0I3lbTucfKtzktUZzOavzb19InQrqEBM=; b=ZVGrLTYktJ1QuRtCnWvt+Mkrq07m/WfcVieYa22AI0F/w0fUNnPaPOqP9w2CkEGfrU 8fINTw4ePn2Pmjlv+E2K+J+1MdvlfaaflfCQ4LFZWb3E/IRoPNi5OtdwYtBxF6S2kM/D BZV2Geqdual0itFg6FoIPHf3zP/iS2GZiOgVq4chQL0VrlXWuwWakmR4+V+7y5cNKbH8 2gn1PpCWcnmDw8tFUHnbcK0OkdBmeIm79rj9vMTYHoc57DQnkwPuiwm22XRw92xU2tOr UFSwS8yEQoV5wdhqbx+GPPXO71rNUUuqzsG6cQZ8gwoU6IeaqIitzzcJjyMJf+qTpg8C T5Qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232898; x=1704837698; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Sx1CrYCYqcy0I3lbTucfKtzktUZzOavzb19InQrqEBM=; b=f/6u5Ohxm86Pge1ghOTvs3jmYbAPh2bOkLIk/ROMwK6Hcu8COj+PDIlsT1InyxvhNS LeFAP+gs02qgzbGttgnaoqaM0G7ZnUbOpU/d9eajhZw3NW4+Vsr9eOLJofb0xE6QKyTH 8PsD/+CH2DDG3BCMs6PSBZW+K69GiArnjGH6H4zkV9+wPX8qDjcebFfUBd83SE+JPN3u ZMQdoj2rESEZqnBE/qCwxezLemr4bZ5G5V3L60dUvsB6l32nT5PvFFIwQUawIjMeUznG W7kFXAtsxEw1TTwz7VLe1HXwszPR10lk+wEgH/7McXynz6QHDH1x4NE5zK+Zjq20i+45 wnmw== X-Gm-Message-State: AOJu0Yx3xf7CdWdalpcj+gqbPfhMg8+BuqDeTV9jEhZC304AMziZG1DO dqARDNOxr/xOhI1KF5txLClufTexEzqIqQ== X-Google-Smtp-Source: AGHT+IHSNOWtd16BNBpr13f63JilpUsY7HXImYHsivjSF6nbDu6T9MOgsg44oEC59WRv4BX5AKGF0w== X-Received: by 2002:a17:90a:728e:b0:28c:a9d0:33ff with SMTP id e14-20020a17090a728e00b0028ca9d033ffmr2151486pjg.62.1704232897945; Tue, 02 Jan 2024 14:01:37 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:37 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 01/12] riscv: Flush the instruction cache during SMP bringup Date: Tue, 2 Jan 2024 14:00:38 -0800 Message-ID: <20240102220134.3229156-2-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240102_140140_652454_B6F93B23 X-CRM114-Status: GOOD ( 11.12 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Instruction cache flush IPIs are sent only to CPUs in cpu_online_mask, so they will not target a CPU until it calls set_cpu_online() earlier in smp_callin(). As a result, if instruction memory is modified between the CPU coming out of reset and that point, then its instruction cache may contain stale data. Therefore, the instruction cache must be flushed after the set_cpu_online() synchronization point. Fixes: 08f051eda33b ("RISC-V: Flush I$ when making a dirty page executable") Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- Changes in v4: - New patch for v4 arch/riscv/kernel/smpboot.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c index d162bf339beb..48af5bd3ec30 100644 --- a/arch/riscv/kernel/smpboot.c +++ b/arch/riscv/kernel/smpboot.c @@ -26,7 +26,7 @@ #include #include -#include +#include #include #include #include @@ -257,9 +257,10 @@ asmlinkage __visible void smp_callin(void) riscv_user_isa_enable(); /* - * Remote TLB flushes are ignored while the CPU is offline, so emit - * a local TLB flush right now just in case. + * Remote cache and TLB flushes are ignored while the CPU is offline, + * so flush them both right now just in case. */ + local_flush_icache_all(); local_flush_tlb_all(); complete(&cpu_running); /* From patchwork Tue Jan 2 22:00:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13509544 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A86EAC4707C for ; Tue, 2 Jan 2024 22:02:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=g9t1ZojKuJWB2/JchThIUKFPeMCYhPcBMBbEjSWDHj8=; b=WQg2YZgkGdnQZa LRaLMTgCUspL5MzzXG9lKC/kEBDa8BpyZKoYajfJa5WLcfdmbUAuNg7ZtyqWOoVLMLeP5uO0GEGOR ehU9uRtc+fXndz8CRA6rTYlVS6+CQN/LRd0oODtILP6rcVSPcIK+/bM26QEc9iyqzxifaFZBWD3jQ MY9gAKbyZJWWGPz3Oy7y4krCJnJaVDHd1mDfmzlnhf8LyhtpZ/jShMke2d0aQLrdAes+y3tfO+wm9 F+l461QZOHnwkIMBlMNP60SeC1y3IJiiHiE78/wrD5fICPeH5KSXpjmaBloptG7uWofNATrqmn3IF vr+IGFjMBdwue+IQqaEg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpX-0098TU-1x; Tue, 02 Jan 2024 22:01:47 +0000 Received: from mail-pj1-x1033.google.com ([2607:f8b0:4864:20::1033]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpQ-0098Np-1v for linux-riscv@lists.infradead.org; Tue, 02 Jan 2024 22:01:44 +0000 Received: by mail-pj1-x1033.google.com with SMTP id 98e67ed59e1d1-28bd623c631so8055285a91.3 for ; Tue, 02 Jan 2024 14:01:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232899; x=1704837699; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pynuwdwrgTDQTs9EZPEkGjuJgTxII1R0VKB5FEKmh8A=; b=Ozk14naO8drEsQbcifTk8r8w81m+11gmsrPRNb18qYqaQcOyB8TaKunokujqAruE+b LEiml3W0ROiPQJGnnVzEHH4Shb8A1fwmXOyU0H326vkX63yKsDrmvfg4/Oicjy2Itc87 zdK+KocE6mhDPzsdyyIXjV+z5hhxUZzZ0gAKnE+eNt/YHLiS5o0lH+eFqI0VlFwo7zkA UcqnGMkA4p9b0gbSBbDoKKGdlddCZfjAOO51jkS/5970zDV0dM3s2Mg1ng0hx9GO2fxb TdPFwB9b/I53ri5ivrwFNHVqXIAhAzrT86V5mruCq/GHkHjUJKSwHAlFnevtk9ThSGrz BESA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232899; x=1704837699; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pynuwdwrgTDQTs9EZPEkGjuJgTxII1R0VKB5FEKmh8A=; b=wottk8h8G4C1P8lsCiIqW13yen13XOL0SnRugvCdmDO2Q9+NQpnq+ce1Ba60ESUgpa ni15v4nmudmTUdUWHIqfGjyvVGjTCOH1IRmz1YGnryBp1JSom7vl9wtzl+DzEUSFoACF efoTHw/WtbZ0eDgYrgjQIJ5m064nn06XmbnCaYLFzCkBbaE0M6+506b5lS160774Dcow paUHD9WqxNwu+hEpJaU+FBOw9B3ZDB3yJ8w8WcYYbkARFGV7ESZ/eicj9I7j7pcV+6ub ElWM/waItUdpSe8MGfEuQJEzdxJ4ZnXD1O80SgRtUzkACflgsti31mfdHdZI6f9XHIzP KVpQ== X-Gm-Message-State: AOJu0YxYG9JPktKx3bsz+OhxzyJNRD4E4cj6wwNnHcV1lVu0vwvwq4Th 62bFR3zMPaLdv+p6dX26DkL675YXNmL/NA== X-Google-Smtp-Source: AGHT+IEYPya8PYA19IRLK0MnT0T3isXxWKz5hXZR0znVEsgxQFvYhbUCwOfyfHQYZwqQ+xa53Aj2UQ== X-Received: by 2002:a17:90b:3ec2:b0:28c:1b98:e6e7 with SMTP id rm2-20020a17090b3ec200b0028c1b98e6e7mr8644807pjb.47.1704232899208; Tue, 02 Jan 2024 14:01:39 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:38 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 02/12] riscv: Use IPIs for remote cache/TLB flushes by default Date: Tue, 2 Jan 2024 14:00:39 -0800 Message-ID: <20240102220134.3229156-3-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240102_140140_653516_C54EF5DB X-CRM114-Status: GOOD ( 21.86 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org An IPI backend is always required in an SMP configuration, but an SBI implementation is not. For example, SBI will be unavailable when the kernel runs in M mode. Generally, IPIs are assumed to be faster than SBI calls due to the SBI context switch overhead. However, when SBI is used as the IPI backend, then the context switch cost must be paid anyway, and performing the cache/TLB flush directly in the SBI implementation is more efficient than inserting an interrupt to the kernel. This is the only scenario where riscv_ipi_set_virq_range()'s use_for_rfence parameter is false. Thus, it makes sense for remote fences to use IPIs by default, and make the SBI remote fence extension the special case. sbi_ipi_init() already checks riscv_ipi_have_virq_range(), so it only calls riscv_ipi_set_virq_range() when no other IPI device is available. So we can move the static key and drop the use_for_rfence parameter. Furthermore, the static branch only makes sense when CONFIG_RISCV_SBI is enabled. Optherwise, IPIs must be used. Add a fallback definition of riscv_use_sbi_for_rfence() which handles this case and removes the need to check CONFIG_RISCV_SBI elsewhere, such as in cacheflush.c. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- Changes in v4: - New patch for v4 arch/riscv/include/asm/sbi.h | 4 ++++ arch/riscv/include/asm/smp.h | 15 ++------------- arch/riscv/kernel/sbi-ipi.c | 11 ++++++++++- arch/riscv/kernel/smp.c | 11 +---------- arch/riscv/mm/cacheflush.c | 5 ++--- arch/riscv/mm/tlbflush.c | 31 ++++++++++++++----------------- drivers/clocksource/timer-clint.c | 2 +- 7 files changed, 34 insertions(+), 45 deletions(-) diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index 0892f4421bc4..aeee0127df76 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -339,8 +339,12 @@ unsigned long riscv_cached_marchid(unsigned int cpu_id); unsigned long riscv_cached_mimpid(unsigned int cpu_id); #if IS_ENABLED(CONFIG_SMP) && IS_ENABLED(CONFIG_RISCV_SBI) +DECLARE_STATIC_KEY_FALSE(riscv_sbi_for_rfence); +#define riscv_use_sbi_for_rfence() \ + static_branch_unlikely(&riscv_sbi_for_rfence) void sbi_ipi_init(void); #else +static inline bool riscv_use_sbi_for_rfence(void) { return false; } static inline void sbi_ipi_init(void) { } #endif diff --git a/arch/riscv/include/asm/smp.h b/arch/riscv/include/asm/smp.h index 0d555847cde6..7ac80e9f2288 100644 --- a/arch/riscv/include/asm/smp.h +++ b/arch/riscv/include/asm/smp.h @@ -49,12 +49,7 @@ void riscv_ipi_disable(void); bool riscv_ipi_have_virq_range(void); /* Set the IPI interrupt numbers for arch (called by irqchip drivers) */ -void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence); - -/* Check if we can use IPIs for remote FENCEs */ -DECLARE_STATIC_KEY_FALSE(riscv_ipi_for_rfence); -#define riscv_use_ipi_for_rfence() \ - static_branch_unlikely(&riscv_ipi_for_rfence) +void riscv_ipi_set_virq_range(int virq, int nr); /* Check other CPUs stop or not */ bool smp_crash_stop_failed(void); @@ -104,16 +99,10 @@ static inline bool riscv_ipi_have_virq_range(void) return false; } -static inline void riscv_ipi_set_virq_range(int virq, int nr, - bool use_for_rfence) +static inline void riscv_ipi_set_virq_range(int virq, int nr) { } -static inline bool riscv_use_ipi_for_rfence(void) -{ - return false; -} - #endif /* CONFIG_SMP */ #if defined(CONFIG_HOTPLUG_CPU) && (CONFIG_SMP) diff --git a/arch/riscv/kernel/sbi-ipi.c b/arch/riscv/kernel/sbi-ipi.c index a4559695ce62..1026e22955cc 100644 --- a/arch/riscv/kernel/sbi-ipi.c +++ b/arch/riscv/kernel/sbi-ipi.c @@ -13,6 +13,9 @@ #include #include +DEFINE_STATIC_KEY_FALSE(riscv_sbi_for_rfence); +EXPORT_SYMBOL_GPL(riscv_sbi_for_rfence); + static int sbi_ipi_virq; static void sbi_ipi_handle(struct irq_desc *desc) @@ -72,6 +75,12 @@ void __init sbi_ipi_init(void) "irqchip/sbi-ipi:starting", sbi_ipi_starting_cpu, NULL); - riscv_ipi_set_virq_range(virq, BITS_PER_BYTE, false); + riscv_ipi_set_virq_range(virq, BITS_PER_BYTE); pr_info("providing IPIs using SBI IPI extension\n"); + + /* + * Use the SBI remote fence extension to avoid + * the extra context switch needed to handle IPIs. + */ + static_branch_enable(&riscv_sbi_for_rfence); } diff --git a/arch/riscv/kernel/smp.c b/arch/riscv/kernel/smp.c index 40420afbb1a0..1d06df04eb71 100644 --- a/arch/riscv/kernel/smp.c +++ b/arch/riscv/kernel/smp.c @@ -171,10 +171,7 @@ bool riscv_ipi_have_virq_range(void) return (ipi_virq_base) ? true : false; } -DEFINE_STATIC_KEY_FALSE(riscv_ipi_for_rfence); -EXPORT_SYMBOL_GPL(riscv_ipi_for_rfence); - -void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence) +void riscv_ipi_set_virq_range(int virq, int nr) { int i, err; @@ -197,12 +194,6 @@ void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence) /* Enabled IPIs for boot CPU immediately */ riscv_ipi_enable(); - - /* Update RFENCE static key */ - if (use_for_rfence) - static_branch_enable(&riscv_ipi_for_rfence); - else - static_branch_disable(&riscv_ipi_for_rfence); } static const char * const ipi_names[] = { diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index 55a34f2020a8..47c485bc7df0 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -21,7 +21,7 @@ void flush_icache_all(void) { local_flush_icache_all(); - if (IS_ENABLED(CONFIG_RISCV_SBI) && !riscv_use_ipi_for_rfence()) + if (riscv_use_sbi_for_rfence()) sbi_remote_fence_i(NULL); else on_each_cpu(ipi_remote_fence_i, NULL, 1); @@ -69,8 +69,7 @@ void flush_icache_mm(struct mm_struct *mm, bool local) * with flush_icache_deferred(). */ smp_mb(); - } else if (IS_ENABLED(CONFIG_RISCV_SBI) && - !riscv_use_ipi_for_rfence()) { + } else if (riscv_use_sbi_for_rfence()) { sbi_remote_fence_i(&others); } else { on_each_cpu_mask(&others, ipi_remote_fence_i, NULL, 1); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index e6659d7368b3..09b03bf71e6a 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -73,10 +73,10 @@ static void __ipi_flush_tlb_all(void *info) void flush_tlb_all(void) { - if (riscv_use_ipi_for_rfence()) - on_each_cpu(__ipi_flush_tlb_all, NULL, 1); - else + if (riscv_use_sbi_for_rfence()) sbi_remote_sfence_vma_asid(NULL, 0, FLUSH_TLB_MAX_SIZE, FLUSH_TLB_NO_ASID); + else + on_each_cpu(__ipi_flush_tlb_all, NULL, 1); } struct flush_tlb_range_data { @@ -96,7 +96,6 @@ static void __ipi_flush_tlb_range_asid(void *info) static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, unsigned long size, unsigned long stride) { - struct flush_tlb_range_data ftd; const struct cpumask *cmask; unsigned long asid = FLUSH_TLB_NO_ASID; bool broadcast; @@ -119,20 +118,18 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, broadcast = true; } - if (broadcast) { - if (riscv_use_ipi_for_rfence()) { - ftd.asid = asid; - ftd.start = start; - ftd.size = size; - ftd.stride = stride; - on_each_cpu_mask(cmask, - __ipi_flush_tlb_range_asid, - &ftd, 1); - } else - sbi_remote_sfence_vma_asid(cmask, - start, size, asid); - } else { + if (!broadcast) { local_flush_tlb_range_asid(start, size, stride, asid); + } else if (riscv_use_sbi_for_rfence()) { + sbi_remote_sfence_vma_asid(cmask, start, size, asid); + } else { + struct flush_tlb_range_data ftd; + + ftd.asid = asid; + ftd.start = start; + ftd.size = size; + ftd.stride = stride; + on_each_cpu_mask(cmask, __ipi_flush_tlb_range_asid, &ftd, 1); } if (mm) diff --git a/drivers/clocksource/timer-clint.c b/drivers/clocksource/timer-clint.c index 9a55e733ae99..7ccc16dd6a76 100644 --- a/drivers/clocksource/timer-clint.c +++ b/drivers/clocksource/timer-clint.c @@ -251,7 +251,7 @@ static int __init clint_timer_init_dt(struct device_node *np) } irq_set_chained_handler(clint_ipi_irq, clint_ipi_interrupt); - riscv_ipi_set_virq_range(rc, BITS_PER_BYTE, true); + riscv_ipi_set_virq_range(rc, BITS_PER_BYTE); clint_clear_ipi(); #endif From patchwork Tue Jan 2 22:00:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13509547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 59987C4725D for ; Tue, 2 Jan 2024 22:02:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=mo9gJJi7Z+yz2KjdlT3YSJtt7HxAnmBA43+v9ViQ0vc=; b=WeRiMBtrTny233 BeGm6xQMBHRlg5bEN9PcP0GKAvmz5uRMw3v+Da9ZjJoRW1or5FrTst3GbuG5vH1OXc1bqFdw9ar8M E4CgZGu6oR8xX1EBSy3xL4/FWqs4RAlHrWu3gnKSH2uUEs9l2RHM7W8MJ2epggMcKtxFKzStjuB+S /0J4Zsb/CjlQnZPFOpogLXVVooFHooSXHXkg4irXts4LCHranIsh52h7CzNp+KTAU581N36VmL21R 4OpVQJ8gZUeDz3NKBbfqWKvfuRReL7nEf5lig3AJ7D0BNHQn42NNgWIuxxehU4wOIviXvn7OS5dMz sXOtQCq2bjujv78lqSmg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpW-0098Sa-1w; Tue, 02 Jan 2024 22:01:46 +0000 Received: from mail-pg1-x52d.google.com ([2607:f8b0:4864:20::52d]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpR-0098Oo-1V for linux-riscv@lists.infradead.org; Tue, 02 Jan 2024 22:01:43 +0000 Received: by mail-pg1-x52d.google.com with SMTP id 41be03b00d2f7-5ce2aada130so1632181a12.1 for ; Tue, 02 Jan 2024 14:01:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232900; x=1704837700; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XA8TOHyyeKTc95PFXJRpYatfRR5oPHMuobG+iziT3HE=; b=bid0IVD1UhOWLJjZNRJ2sv9L3VbbzQR6I483xDi0pYPPzzvvLg0NtNx/2T8O42Ryjp e6ZBH7Xg1tJS+7EMGMW4TlgRTrV4dObEt/8gDD9IMwJh57G+Zh4HL0AhMabizFD//slk ou5pz3gAwpJf/aTrGEZ3S9lavQux1ov4Cn6fdsUpvv60BcezTG8lu4vMLkxB/awVTer5 zhJpd6zW2CihF/gy+lRCQwgeVBHaRhO7pQL9uU+DuI75GIM63zUlv2pkA/BzFzAXraQB TRp5yn2lengE6zjNpTbGG7WaY7oUgKs+hYoJTUMuULz8sylYRV0a2VouFRyT0ftXrTF3 GYiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232900; x=1704837700; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XA8TOHyyeKTc95PFXJRpYatfRR5oPHMuobG+iziT3HE=; b=a+AXH19U1P7wRodaamzfUDZJVtH7vzot+flJ9x6SrChJyEiaKpYFn9tmL3WbvYKhXg trjVzgb5rcS5HXcffa7yE6lFXA44EGwmp8af6NavsSxL3stIqTT2hFHshZoirDKcCUxy 2MxgMN43hl83yweO63ogyUo52gU552c+DBQQJCI3sOvve//eCdYD0VlIH4ly7uubAvfx DHKBf8Ni1Ga3jNKJ+XOUR1LGgJ3ccYBDF4JI9GFcin6BG8ndFBruFQTxZAXeTBoQFWa+ Lwqah+LdsfBDAVWtdD99+Nj+fo4x8Qyin8t0YP7wjXqDKaW985kuYBjdm4pmQUx3UY3K Ye+A== X-Gm-Message-State: AOJu0YwwCnmiF9BBhrsgvLRnsCnFzY07HncbRknj/Z8H8cYjDj0AaHYp B8CB0SNPjbz4LxxULXM69m/D+asnvYKRrioIYEgpPBG4ZsI= X-Google-Smtp-Source: AGHT+IGXoPWHR5PafwwvCfsK03qnPYeIgtaQPhGwO0vZvHcHemW0jssRCcGF3figFCmbRtQxmnrgGQ== X-Received: by 2002:a17:90a:974c:b0:28c:accc:b1db with SMTP id i12-20020a17090a974c00b0028cacccb1dbmr2272863pjw.44.1704232900428; Tue, 02 Jan 2024 14:01:40 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:40 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 03/12] riscv: mm: Broadcast kernel TLB flushes only when needed Date: Tue, 2 Jan 2024 14:00:40 -0800 Message-ID: <20240102220134.3229156-4-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240102_140141_505585_2C630689 X-CRM114-Status: GOOD ( 13.98 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org __flush_tlb_range() avoids broadcasting TLB flushes when an mm context is only active on the local CPU. Apply this same optimization to TLB flushes of kernel memory when only one CPU is online. This check can be constant-folded when SMP is disabled. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- Changes in v4: - New patch for v4 arch/riscv/mm/tlbflush.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 09b03bf71e6a..2f18fe6fc4f3 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -98,27 +98,23 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, { const struct cpumask *cmask; unsigned long asid = FLUSH_TLB_NO_ASID; - bool broadcast; + unsigned int cpu; if (mm) { - unsigned int cpuid; - cmask = mm_cpumask(mm); if (cpumask_empty(cmask)) return; - cpuid = get_cpu(); - /* check if the tlbflush needs to be sent to other CPUs */ - broadcast = cpumask_any_but(cmask, cpuid) < nr_cpu_ids; - if (static_branch_unlikely(&use_asid_allocator)) asid = atomic_long_read(&mm->context.id) & asid_mask; } else { cmask = cpu_online_mask; - broadcast = true; } - if (!broadcast) { + cpu = get_cpu(); + + /* Check if the TLB flush needs to be sent to other CPUs. */ + if (cpumask_any_but(cmask, cpu) >= nr_cpu_ids) { local_flush_tlb_range_asid(start, size, stride, asid); } else if (riscv_use_sbi_for_rfence()) { sbi_remote_sfence_vma_asid(cmask, start, size, asid); @@ -132,8 +128,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, on_each_cpu_mask(cmask, __ipi_flush_tlb_range_asid, &ftd, 1); } - if (mm) - put_cpu(); + put_cpu(); } void flush_tlb_mm(struct mm_struct *mm) From patchwork Tue Jan 2 22:00:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13509550 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7781DC47079 for ; Tue, 2 Jan 2024 22:02:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=lxextqT6+s4SnCLpRzGYJXl0X87N89GWg68rIw3Q/48=; b=fGBSANTX+v8kiS 1Z+I0DyDm0KytOqENe9JNssG9h0pHR5LwCiqTHe8qbcSls5F0TMloSbJLY02O71rdFA3ByOWv+4uJ cvtPgAV/nL4TKP9sQA6CcFfmDprFS0GoGbHRiPzQLTwT6hgv4w8oAlfygL2XqQtSXEz6XmLpaCQTo qImjoj4IT1TTMWc2l0I/XWbqMBTzlhFgahvQ8+wk7+JDMu+x9EdIHvl5TCJ/CfSaFfbbYQEK/dXM1 jvGCOGbSDxMxLA8IyX7IdSDMRLBo55YeDoVX/GaV4cOF8lS3lIYKQyrKaC9nQMxfg3dv8x6fvmOQA KjJQ4TdJ5UOM2KptVhiQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpZ-0098Ul-14; Tue, 02 Jan 2024 22:01:49 +0000 Received: from mail-pg1-x535.google.com ([2607:f8b0:4864:20::535]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpS-0098PD-2s for linux-riscv@lists.infradead.org; Tue, 02 Jan 2024 22:01:45 +0000 Received: by mail-pg1-x535.google.com with SMTP id 41be03b00d2f7-5ceb3fe708eso151271a12.3 for ; Tue, 02 Jan 2024 14:01:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232901; x=1704837701; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YDpWqQ4mc+jX/Roh6ocibeJ1LmLlK7pqWIKc4/L4AE8=; b=LVC8ZIbUTtTWg+I2pMGYo3NGPoicU1EHMxQO4qtoQb9yNqQjXdM8bRNxt3dACH8Fgi SYEaTRDZXP41Qy+zGZmwqzAQMQa929S9ztIcXwaaL8l+qLUL/w2GhDKcaA2EEvbsbvPz WbUapkY3A58iii431igJQNWj6ThK206oRvQZgbfnu5LxFoswg0CX3wPaMhBVOS42tcFK khyI8tI+qcHU3j+ZZZMwJakqSc8sGhdVhDuvCRs2J+sypCAd5w+cRnql1nzuqls/bhaQ qqhXz7Jr4xzth2xj16aAyOChQfnTFm5r8lmd/U6OfTwV4IQvd6Ig1Jk5nzl+if5RdUkH PjWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232901; x=1704837701; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YDpWqQ4mc+jX/Roh6ocibeJ1LmLlK7pqWIKc4/L4AE8=; b=q7wbq1OasL7lsx5MuwMGO3fwhV6TlA5Irq9F343AtJ/5JuoxKDd6LWtgEFYgBe3M3m IrUwzcb5+0yzt5PHnjmXtUuoq7yEwUMFts4UmWl11Uj15ZsiTScSycf/VZ3WoI6F+ezp 0dVLgjUY3lLRwb13XJ2oaURq7Lo4UZtzD+C1mftC7bexJwdxTNKOcifCkGqMNuVzTEA0 izKCvc5m9dbYN4RbR0qXVJUdXJ4Ng4N+bZuj9FtQDv40W7tfEr0kWFpFC50PJ7OcyPBI 9ADekdiaR5m2t5DZAXp8xiY8TyPf/Vg03xY0Q2mtsjmCTU6+H+NLEjc5cMVfMjL+0dPB RHWw== X-Gm-Message-State: AOJu0YzywGDDSaO8/SofFajg004UrIe9y25eWGumUtclRPx6Nqf64T2E DuLmQWi4fGrwK6PFEehS5G/6d+lZ27VR/A== X-Google-Smtp-Source: AGHT+IGhgK6idMxI6CkoCNiy9Y4cs5Xn9ml4r4H30wFrLDF3Z3r2uMsg0CAIjypKVmYj0k77Mk3d6A== X-Received: by 2002:a17:90a:6ba6:b0:28c:640b:ce0b with SMTP id w35-20020a17090a6ba600b0028c640bce0bmr4536054pjj.33.1704232901387; Tue, 02 Jan 2024 14:01:41 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:41 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 04/12] riscv: Only send remote fences when some other CPU is online Date: Tue, 2 Jan 2024 14:00:41 -0800 Message-ID: <20240102220134.3229156-5-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240102_140142_965737_9B38C86D X-CRM114-Status: GOOD ( 12.25 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org If no other CPU is online, a local cache or TLB flush is sufficient. These checks can be constant-folded when SMP is disabled. Signed-off-by: Samuel Holland --- Changes in v4: - New patch for v4 arch/riscv/mm/cacheflush.c | 4 +++- arch/riscv/mm/tlbflush.c | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index 47c485bc7df0..f7933ae88a55 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -21,7 +21,9 @@ void flush_icache_all(void) { local_flush_icache_all(); - if (riscv_use_sbi_for_rfence()) + if (num_online_cpus() < 2) + return; + else if (riscv_use_sbi_for_rfence()) sbi_remote_fence_i(NULL); else on_each_cpu(ipi_remote_fence_i, NULL, 1); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 2f18fe6fc4f3..37b3c93e3c30 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -73,7 +73,9 @@ static void __ipi_flush_tlb_all(void *info) void flush_tlb_all(void) { - if (riscv_use_sbi_for_rfence()) + if (num_online_cpus() < 2) + local_flush_tlb_all(); + else if (riscv_use_sbi_for_rfence()) sbi_remote_sfence_vma_asid(NULL, 0, FLUSH_TLB_MAX_SIZE, FLUSH_TLB_NO_ASID); else on_each_cpu(__ipi_flush_tlb_all, NULL, 1); From patchwork Tue Jan 2 22:00:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13509552 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EE0C7C4707B for ; Tue, 2 Jan 2024 22:02:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Ls1C07FBg1CnEUPS6mO2x68gE/7aWV9l3ZK/jEhzHb0=; b=GfdqBmNFDt4TFD tqMuK8tQyUj0GVpsxQLV9DrP11NH6/rDhVVLdyoWDbwq0FTYNyf2o6KTADRz36ARhTDDp1wmTg8mE NgWsx4xttzozM84ddsJ7bfzOGTzLYY87JA8NbgbqcHVAfDeEIKw2WOCNj405D3F9q7yW0fmpDOfhv s2z+bkxEX5jRLUL+SW/MZuplRCmkBJyoiBK7U4rSHKVBvMplqXRKeWOY0V+x1h1qGu0EF1OojvjSn DQ6hFZBK8JY7/ECnub7Y3qWzoER3OAtJPLPaKQvVpEN9NlYAYVOl0vZmoWFL/nAyG2GfcK9DenZM5 6bvglRYh12P6pYNK71iw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpY-0098U3-1N; Tue, 02 Jan 2024 22:01:48 +0000 Received: from mail-pg1-x535.google.com ([2607:f8b0:4864:20::535]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpS-0098Pe-2t for linux-riscv@lists.infradead.org; Tue, 02 Jan 2024 22:01:45 +0000 Received: by mail-pg1-x535.google.com with SMTP id 41be03b00d2f7-5ce0efd60ddso2158168a12.0 for ; Tue, 02 Jan 2024 14:01:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232902; x=1704837702; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LtzHkWRF5gALofCSRDnlYgSzHUyaPaF++esfIcBOkBw=; b=MUJN1kVp6oplOOOxzV1x/ark5BdO8ph7rr8loGkhWyNJGCo0mdsCA8qsXvwEtCEaoL WEzX/AJGIAOiebdPpZeW1OdTfr7xjCYfZ/GdTUGkzSj9YtobvCA0jd0x2GGLDidVSpSK muAwrOVxk9vHeH3un53O/4Rl1MmZ0o63VnflE18nVB/m4fYjd6DW6M2Dow3B6t2qPIpr aWruJaJ0nslum0OXb6ydqlPorbjoWBZkWsf2TlBHkxzUYBA5iqaow2JZDaON3eIUYiql UmFO8FfSyRDXcYpbrVahsGGZ4azOaRzwbiucdxDcu2OvzQ0YijA66GEGdSPlmiAoQ7jh 9v5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232902; x=1704837702; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LtzHkWRF5gALofCSRDnlYgSzHUyaPaF++esfIcBOkBw=; b=YZEz1BnyWzvIYVfcQPgIMFq1yNSMbhBOxGhdfBoepOirgY4D6/okVwpq5QrhKDZ+SI yTMbvSzhRWkil6oygjcEuR2w9FTOCr+N+d/2jQDSErowtS14b4CqTYzJCwuuVePUVqKu K9ktiVu0p+/K/8OhrxCPwE/DD5gcCLk/T8BlBB4awZBrFJBiZNoKlpIS2EpP9Gn8WKHz Vcy4bb2dPw8wEzvd5FWRd9YEBCw1Px7ZR9e0E20T+IBdQ1R0ecuxwhSwXJOoIQjoufgz Q51C9C9SqDxw9K/9BhKzKacfFuQVnZHnGPna9z5yzRS4a2VJowo+DVCWiYVOgUQddjGC xgHg== X-Gm-Message-State: AOJu0YwSl/UR2Dix8QAV+RI/0FNkH4gES3WLG0t43VHx3JXpf0sKdcA9 k6imqA7pilPlkVm2U/f0QrJ8TzJKhOIDAQ== X-Google-Smtp-Source: AGHT+IExkJSZEsbRdHGUNd8kiI0pICSa1vMynIFUPWzzc2fqMiijvWM2jfH6XllBTjKhdLtvimrjxg== X-Received: by 2002:a17:90a:740a:b0:286:7f0d:6254 with SMTP id a10-20020a17090a740a00b002867f0d6254mr6006468pjg.63.1704232902374; Tue, 02 Jan 2024 14:01:42 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:42 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 05/12] riscv: mm: Combine the SMP and UP TLB flush code Date: Tue, 2 Jan 2024 14:00:42 -0800 Message-ID: <20240102220134.3229156-6-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240102_140142_967178_7D61F304 X-CRM114-Status: GOOD ( 12.93 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In SMP configurations, all TLB flushing narrower than flush_tlb_all() goes through __flush_tlb_range(). Do the same in UP configurations. This allows UP configurations to take advantage of recent improvements to the code in tlbflush.c, such as support for huge pages and flushing multiple-page ranges. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- Changes in v4: - Merge the two copies of __flush_tlb_range() and rely on the compiler to optimize out the broadcast path (both clang and gcc do this) - Merge the two copies of flush_tlb_all() and rely on constant folding Changes in v2: - Move the SMP/UP merge earlier in the series to avoid build issues - Make a copy of __flush_tlb_range() instead of adding ifdefs inside - local_flush_tlb_all() is the only function used on !MMU (smpboot.c) arch/riscv/include/asm/tlbflush.h | 29 +++-------------------------- arch/riscv/mm/Makefile | 5 +---- 2 files changed, 4 insertions(+), 30 deletions(-) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 8f3418c5f172..7712ffe2f6c4 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -27,12 +27,7 @@ static inline void local_flush_tlb_page(unsigned long addr) { ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) : "memory")); } -#else /* CONFIG_MMU */ -#define local_flush_tlb_all() do { } while (0) -#define local_flush_tlb_page(addr) do { } while (0) -#endif /* CONFIG_MMU */ -#if defined(CONFIG_SMP) && defined(CONFIG_MMU) void flush_tlb_all(void); void flush_tlb_mm(struct mm_struct *mm); void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, @@ -46,26 +41,8 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end); void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); #endif -#else /* CONFIG_SMP && CONFIG_MMU */ - -#define flush_tlb_all() local_flush_tlb_all() -#define flush_tlb_page(vma, addr) local_flush_tlb_page(addr) - -static inline void flush_tlb_range(struct vm_area_struct *vma, - unsigned long start, unsigned long end) -{ - local_flush_tlb_all(); -} - -/* Flush a range of kernel pages */ -static inline void flush_tlb_kernel_range(unsigned long start, - unsigned long end) -{ - local_flush_tlb_all(); -} - -#define flush_tlb_mm(mm) flush_tlb_all() -#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() -#endif /* !CONFIG_SMP || !CONFIG_MMU */ +#else /* CONFIG_MMU */ +#define local_flush_tlb_all() do { } while (0) +#endif /* CONFIG_MMU */ #endif /* _ASM_RISCV_TLBFLUSH_H */ diff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile index 3a4dfc8babcf..96e65c571ce8 100644 --- a/arch/riscv/mm/Makefile +++ b/arch/riscv/mm/Makefile @@ -13,15 +13,12 @@ endif KCOV_INSTRUMENT_init.o := n obj-y += init.o -obj-$(CONFIG_MMU) += extable.o fault.o pageattr.o +obj-$(CONFIG_MMU) += extable.o fault.o pageattr.o tlbflush.o obj-y += cacheflush.o obj-y += context.o obj-y += pgtable.o obj-y += pmem.o -ifeq ($(CONFIG_MMU),y) -obj-$(CONFIG_SMP) += tlbflush.o -endif obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_KASAN) += kasan_init.o From patchwork Tue Jan 2 22:00:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13509546 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 36112C47074 for ; Tue, 2 Jan 2024 22:02:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=2xrtIk3GBKAEHdHQaWtzlHE8TY7iZe2kzPb7kxBT/VQ=; b=W0cvSoXxk6Y/UA eldH1dbSwsvWCSDClhqHBfTAP+9cJsr5Y6N122BmXa7ffxXM34WRb1IQZyAsgtAradB9Rhf+Wdrm4 iLjGEswS7SesJyqd+gRcuASP0V+Qc5YOOFjZE4gzBnxjIELvN2N7L2pASwKjXIe4bOekYXKx6uNCa qizUVpeny0oWwlHE3807YG8NLNTFzcP2CwvvqeDn/lypj4BFPy1k9k5ACwVX43znyG4LmqWUDyEze LuVPyxAdc6xpNdw8A+LURtq9Je/UxwgRyA8TnRq/VP1A4MqBZeVFmSwDx30EuK4b9RV9uMc0UTvjC VVLiq1OTpqM9Y75JRCtg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpa-0098Uw-0E; Tue, 02 Jan 2024 22:01:50 +0000 Received: from mail-pl1-x629.google.com ([2607:f8b0:4864:20::629]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpU-0098QZ-1Q for linux-riscv@lists.infradead.org; Tue, 02 Jan 2024 22:01:46 +0000 Received: by mail-pl1-x629.google.com with SMTP id d9443c01a7336-1d408c2908cso23253195ad.1 for ; Tue, 02 Jan 2024 14:01:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232903; x=1704837703; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6c+wYTacIgINHvokNTIzxqzbFxSPL5G6VDrdebR45nU=; b=df8FMgAiRYGhgh4t1+Y5NqhiGbGPeMPVtPMNBv3TmxTMixBbO3YupJIBJnn2XTmn3r 39W8isLekOiusQFQgqUmufbFeJUUmGC9AUD3jVm84xmVMQYm1eU7mH0jHJoKKYw+5MzY 64q953slGqGrf/a1qucQUmuyA6tFoU5CnyeaHzsEmdMWHczqvubVQFd3aQzqV5I7Y0qa sr+G/wEr7OnOjOgeWEb0sX6Fkm8nXd1H10tO/bMi+XW2sDO+/e9bWyfKPvyQRwEGWa5u IC6yWGTVwL+Zessgcv0LZN9WisfxoMSI9OdbPCLqayJPXV67goC15HZw2qBKMCgpFeQ0 4uuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232903; x=1704837703; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6c+wYTacIgINHvokNTIzxqzbFxSPL5G6VDrdebR45nU=; b=iboQID+Mg9eLauIDQmWEncVpIzC7iTY6ilupGWa9uGLLoSOQlpWMiC1HIopCt2mzws 21gkhaC5RJipIMLhOkMVgibjhRhor2/lQabwxEvATzHSRbVZVzjJvA1MiBIf/HL6sUzO wHAgtPJvcbTLAtRHwdb/kpOZILv0g7g2p3TiB2fJvGBiwtif7NT+zp8kz8U65D+YugTu vOa/hB90b0ckUoXCCGW2Q+3cysFtmxXDZE/3QJzVpUBh5Tenn/1gddKEILye8jeXlEv/ SVkwseBgz8oRyKboXCwRPuKPBzCwgY6CXFWhlixkUdBgWSHK29gnpVlntk+pEXu7/ZZL 8MlA== X-Gm-Message-State: AOJu0YwGGOtMDyywkt+U8Xo/L0MzBSegUdT2IPuaNCJKV5oou/Wbl5Fk 8v1DVQZyYXItTgdHbaEPvZ+in0NWjbgrhQ== X-Google-Smtp-Source: AGHT+IE2GdHLRUcW4ROKdXldqRiWcx2M7fdl7/RqRT5d/m4h5bCpV21xGk3IklfQQqw3IhSZ5sLqag== X-Received: by 2002:a17:902:d584:b0:1d4:5f9:d3d7 with SMTP id k4-20020a170902d58400b001d405f9d3d7mr8348439plh.127.1704232903384; Tue, 02 Jan 2024 14:01:43 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:43 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 06/12] riscv: Apply SiFive CIP-1200 workaround to single-ASID sfence.vma Date: Tue, 2 Jan 2024 14:00:43 -0800 Message-ID: <20240102220134.3229156-7-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240102_140144_517113_FC58C732 X-CRM114-Status: GOOD ( 14.35 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org commit 3f1e782998cd ("riscv: add ASID-based tlbflushing methods") added calls to the sfence.vma instruction with rs2 != x0. These single-ASID instruction variants are also affected by SiFive errata CIP-1200. Until now, the errata workaround was not needed for the single-ASID sfence.vma variants, because they were only used when the ASID allocator was enabled, and the affected SiFive platforms do not support multiple ASIDs. However, we are going to start using those sfence.vma variants regardless of ASID support, so now we need alternatives covering them. Signed-off-by: Samuel Holland --- (no changes since v2) Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/include/asm/errata_list.h | 12 +++++++++++- arch/riscv/include/asm/tlbflush.h | 19 ++++++++++++++++++- arch/riscv/mm/tlbflush.c | 23 ----------------------- 3 files changed, 29 insertions(+), 25 deletions(-) diff --git a/arch/riscv/include/asm/errata_list.h b/arch/riscv/include/asm/errata_list.h index 83ed25e43553..6781460ae564 100644 --- a/arch/riscv/include/asm/errata_list.h +++ b/arch/riscv/include/asm/errata_list.h @@ -44,11 +44,21 @@ ALTERNATIVE(__stringify(RISCV_PTR do_page_fault), \ CONFIG_ERRATA_SIFIVE_CIP_453) #else /* !__ASSEMBLY__ */ -#define ALT_FLUSH_TLB_PAGE(x) \ +#define ALT_SFENCE_VMA_ASID(asid) \ +asm(ALTERNATIVE("sfence.vma x0, %0", "sfence.vma", SIFIVE_VENDOR_ID, \ + ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ + : : "r" (asid) : "memory") + +#define ALT_SFENCE_VMA_ADDR(addr) \ asm(ALTERNATIVE("sfence.vma %0", "sfence.vma", SIFIVE_VENDOR_ID, \ ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ : : "r" (addr) : "memory") +#define ALT_SFENCE_VMA_ADDR_ASID(addr, asid) \ +asm(ALTERNATIVE("sfence.vma %0, %1", "sfence.vma", SIFIVE_VENDOR_ID, \ + ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ + : : "r" (addr), "r" (asid) : "memory") + /* * _val is marked as "will be overwritten", so need to set it to 0 * in the default case. diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 7712ffe2f6c4..002c4c2620f3 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -22,10 +22,27 @@ static inline void local_flush_tlb_all(void) __asm__ __volatile__ ("sfence.vma" : : : "memory"); } +static inline void local_flush_tlb_all_asid(unsigned long asid) +{ + if (asid != FLUSH_TLB_NO_ASID) + ALT_SFENCE_VMA_ASID(asid); + else + local_flush_tlb_all(); +} + /* Flush one page from local TLB */ static inline void local_flush_tlb_page(unsigned long addr) { - ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) : "memory")); + ALT_SFENCE_VMA_ADDR(addr); +} + +static inline void local_flush_tlb_page_asid(unsigned long addr, + unsigned long asid) +{ + if (asid != FLUSH_TLB_NO_ASID) + ALT_SFENCE_VMA_ADDR_ASID(addr, asid); + else + local_flush_tlb_page(addr); } void flush_tlb_all(void); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 37b3c93e3c30..292d7cf3c4f6 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -7,29 +7,6 @@ #include #include -static inline void local_flush_tlb_all_asid(unsigned long asid) -{ - if (asid != FLUSH_TLB_NO_ASID) - __asm__ __volatile__ ("sfence.vma x0, %0" - : - : "r" (asid) - : "memory"); - else - local_flush_tlb_all(); -} - -static inline void local_flush_tlb_page_asid(unsigned long addr, - unsigned long asid) -{ - if (asid != FLUSH_TLB_NO_ASID) - __asm__ __volatile__ ("sfence.vma %0, %1" - : - : "r" (addr), "r" (asid) - : "memory"); - else - local_flush_tlb_page(addr); -} - /* * Flush entire TLB if number of entries to be flushed is greater * than the threshold below. From patchwork Tue Jan 2 22:00:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13509549 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6A486C47255 for ; Tue, 2 Jan 2024 22:02:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=0pI9soNogU80XiyZbUwuLGYcabKkKxZNfXSAT7E6OUo=; b=qPkfK6jMfBFMrI zV8ZeicX+2Z+4raf+trWES/Fd6bXBbMowCCFooiJImsM+UJNkDwvjt6ZrCBsrmETJCJC6mVakXziF lXvwhOjF9Lfy+ei/5tMUkaNIUTeO4IRXPmLX5P060aHCA2IiKd9zUMK+n5IRm/dRrLXkJi9Q8G3si vxwW0e5CW7DAyyuvpzxTQ5+r43Ax5fOAqfB42CSj3qyOfHjF82WS0+nPwcQLNggintwl7miX76/3F Ka5eX/pimn0nSYxhksRjf//d0fyGhAYbHAz1ek0djdXFrguF6sQ1w/KHC7xvnvlm4JBLKzmBM5ABy z8RDz0mHUFOmdSs90HzA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpa-0098Ve-3D; Tue, 02 Jan 2024 22:01:50 +0000 Received: from mail-pg1-x52b.google.com ([2607:f8b0:4864:20::52b]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpV-0098RL-0y for linux-riscv@lists.infradead.org; Tue, 02 Jan 2024 22:01:47 +0000 Received: by mail-pg1-x52b.google.com with SMTP id 41be03b00d2f7-5cdfa676fa3so5020530a12.1 for ; Tue, 02 Jan 2024 14:01:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232904; x=1704837704; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1tIOva8KFaIwh3i+98q+ysPtJyT8vpZl2kFy86xHGzc=; b=dwkeTbwtiguGYVFXDVPu9wXqcEp1q+C2cYPW5HFjclsgF95hH9DpDLtjo/TH1Qq/cb r+3hIvpaa9pipvMGBQsM+LK0u9ZSQ9SOr97fQ1gRFNQlEssk6aZi+1PLdjhaHfvGxxL8 RGdvIRDxbK5pFaO2tIIXA6lrg7lGfLAMqVaN+a3xoQZhf9K72r6bTqBt0s/+8VmmFzIl sGzPqQHf20pjUGYdGVRpovozTkf0Ih45PaY3a5syEFPFdf3dL+eyJ+WR1mlJ+cI0X7Ht 1yhI/0Jmko9XalsmN1t0j9hrCT81kr1IMEtt2Tq2/g9Yv9SOqO6YWkBE6bSQ49VxHZov +Jmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232904; x=1704837704; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1tIOva8KFaIwh3i+98q+ysPtJyT8vpZl2kFy86xHGzc=; b=ExXfWqkZDlWEFWNgByBxRN9ISc+rMMSoGA3XlBU2IVYjZjoYFAAXHPYjPWxLHySu3X GERAOv5qFPQY6iEbb1OYW4s5y4RL5VblAZm0GEU/GGIh6ypqsOD9PRcMIGBZw9+ZjCh3 l2Or2d7Xpnl3a+MTDNrbkS1hOU3JMq9hNRNmK/8foL1y3J41wY2A5vobGa3s4/2IqOTp S5tAshCfZtiBsgNw69F5At7L5xPHML6oYrenYfl0Hu4oc/9iH+3qBJIk0v09Hl8Z+z3s yXfbUpS9otm9DJqNH+2KffxiCRlBynTKs17kwXyQafvpixyGWBuZ8BKjjuxEB7Svtjid xfLw== X-Gm-Message-State: AOJu0YyRNIXZs4B4IrwUieD1gF5NcBROFCPsxQuryXFnEs/wLs19MnWY 0JrNmFvWnywlRjsGcCa1i/sWLcpfmUvFsg== X-Google-Smtp-Source: AGHT+IERqoXqHCzKpOJ5RqmrnOEOyFJpME6yqKz2yoKuSVbxPLF5GZg8UdvpqpiYsRMOZbO3DMheoA== X-Received: by 2002:a17:90a:d90b:b0:28c:300d:d858 with SMTP id c11-20020a17090ad90b00b0028c300dd858mr127429pjv.20.1704232904383; Tue, 02 Jan 2024 14:01:44 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:44 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 07/12] riscv: Avoid TLB flush loops when affected by SiFive CIP-1200 Date: Tue, 2 Jan 2024 14:00:44 -0800 Message-ID: <20240102220134.3229156-8-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240102_140145_380556_9D2AE95C X-CRM114-Status: GOOD ( 13.52 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Since implementations affected by SiFive errata CIP-1200 always use the global variant of the sfence.vma instruction, they only need to execute the instruction once. The range-based loop only hurts performance. Signed-off-by: Samuel Holland --- Changes in v4: - Only set tlb_flush_all_threshold when CONFIG_MMU=y. Changes in v3: - New patch for v3 arch/riscv/errata/sifive/errata.c | 5 +++++ arch/riscv/include/asm/tlbflush.h | 2 ++ arch/riscv/mm/tlbflush.c | 2 +- 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/riscv/errata/sifive/errata.c b/arch/riscv/errata/sifive/errata.c index 3d9a32d791f7..716cfedad3a2 100644 --- a/arch/riscv/errata/sifive/errata.c +++ b/arch/riscv/errata/sifive/errata.c @@ -42,6 +42,11 @@ static bool errata_cip_1200_check_func(unsigned long arch_id, unsigned long imp return false; if ((impid & 0xffffff) > 0x200630 || impid == 0x1200626) return false; + +#ifdef CONFIG_MMU + tlb_flush_all_threshold = 0; +#endif + return true; } diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 002c4c2620f3..d9913590f82e 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -58,6 +58,8 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end); void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); #endif + +extern unsigned long tlb_flush_all_threshold; #else /* CONFIG_MMU */ #define local_flush_tlb_all() do { } while (0) #endif /* CONFIG_MMU */ diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 292d7cf3c4f6..76b24d4ed4ab 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -11,7 +11,7 @@ * Flush entire TLB if number of entries to be flushed is greater * than the threshold below. */ -static unsigned long tlb_flush_all_threshold __read_mostly = 64; +unsigned long tlb_flush_all_threshold __read_mostly = 64; static void local_flush_tlb_range_threshold_asid(unsigned long start, unsigned long size, From patchwork Tue Jan 2 22:00:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13509545 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 32059C47258 for ; Tue, 2 Jan 2024 22:02:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=JUutasPFwWnVEO4GtxO4unoTMRmmUpK1GEK3sK0f4fA=; b=P3yvX+PYA2IgYs VbVLTWMqrinmE3socZ7TtrtKDOlDwb/vBuxVsIyg4rmvD60jLq5UT11PfHxckcBUVjClpvQKTVbWh djahBbCGw+g8VxZgnXc0eCWpcYO6fNWQEznrI0WXqJeICtMbesS+y2O/XS6oGDxVef4SypQ9BfqO6 e6VWsQIXaJ/1maPdJQ5+QSmtuDk6ru7F0xt1FOKixsn1uixqDEce8k7SPd0QhBTvtB15biNxPQw2h 5I1ZwSGqRLJKZb+8bsWyd9e/Bt63ZBaJphSk16IVdzgrf0wmjbjIDmmYGJtN6fD4jtEHqwVMvglxX uIkiO54YLvhkAnDepe7A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpb-0098Wu-2D; Tue, 02 Jan 2024 22:01:51 +0000 Received: from mail-pg1-x52a.google.com ([2607:f8b0:4864:20::52a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpW-0098S8-2T for linux-riscv@lists.infradead.org; Tue, 02 Jan 2024 22:01:48 +0000 Received: by mail-pg1-x52a.google.com with SMTP id 41be03b00d2f7-5ce942efda5so1210830a12.2 for ; Tue, 02 Jan 2024 14:01:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232905; x=1704837705; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=d7sRColwM16ZDpkHsKhbx/H1UjX1V2ZP/gk6hKZVk/k=; b=EOtzAXUVdwmViT4+Zy3az9nK3rYPrvCNv7Rp20/1SF7P+QiLvApnUyFMpDuwaWweMV ngdc6IS5sfOhYWURLZZOPtl/MBlxVXIYiH6UqyVVkpnCB1Ld6k8oEBmJNNu/zNUgIutx izCGslPh/83XnOEHz3v1uTotwadkQZHODIOG/17JV9W3bqAX/25ftk9jOJGLbkrWzya3 OzD+jsf3KfZkK+eKJnV/h3iLXfsyOtbbSjZEkGxBncWtST0SumW/8h/4aVaXywXuPbSG I+NCVISxSbIGcAGi1CSly0kP5yr+LQsb3HoCe8lXK53zjVygTAqNb7pMNY7iS2BQXx7F gucg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232905; x=1704837705; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d7sRColwM16ZDpkHsKhbx/H1UjX1V2ZP/gk6hKZVk/k=; b=ZJ8B4me/X1asOfSXpyS3ekVKAK1lS4sAoq1+3vfxVQ5LEtn6UDkb6WW2yBIhVER9eb Ea4oJcFQ8A9WZ6C7IZ/+lc+cywAS9nvCL65rwzTAKSmBLaTmcCrelPXRrDUL0z3F5ToS IpjKiIkcA/pfBps/k6tYLEkP0A03gR4y4SmWwf+ldyrabdLqmoeSz+odk8Gga4oSSLPy 2N4QbAu7rpeOffHc7mmo9xRDYGQPwG+IWBT9Ymy8zTAPOHJTyP20xRNiFw2Mw7jqCmTG CtxaWXBUefdqa5GG2oFnDm1SPahsjPXvUxkKmOJtQKUqcHP8yMuDt+t4y74e9wWw8tz2 rnoA== X-Gm-Message-State: AOJu0YzxknBNez2LXU2SMrzNcpqa1uG1aijd331RzTn1Vv0wvS3fTsqH EhRdePGtCbG0m2s+tvTk+aFp3Nwwc03/0Q== X-Google-Smtp-Source: AGHT+IEo3bZM2aLuSMOGwMaVus9GR4XwWujHLJvE5N9ecURPZ+yptZqxQ7nXUllBGYAXlfndapu+WA== X-Received: by 2002:a05:6a20:a107:b0:197:2fc0:7e25 with SMTP id q7-20020a056a20a10700b001972fc07e25mr2866566pzk.3.1704232905452; Tue, 02 Jan 2024 14:01:45 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:45 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 08/12] riscv: mm: Introduce cntx2asid/cntx2version helper macros Date: Tue, 2 Jan 2024 14:00:45 -0800 Message-ID: <20240102220134.3229156-9-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240102_140146_817892_19FF4871 X-CRM114-Status: GOOD ( 15.23 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org When using the ASID allocator, the MM context ID contains two values: the ASID in the lower bits, and the allocator version number in the remaining bits. Use macros to make this separation more obvious. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- (no changes since v1) arch/riscv/include/asm/mmu.h | 3 +++ arch/riscv/mm/context.c | 12 ++++++------ arch/riscv/mm/tlbflush.c | 2 +- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index 355504b37f8e..a550fbf770be 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -26,6 +26,9 @@ typedef struct { #endif } mm_context_t; +#define cntx2asid(cntx) ((cntx) & asid_mask) +#define cntx2version(cntx) ((cntx) & ~asid_mask) + void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, pgprot_t prot); #endif /* __ASSEMBLY__ */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 217fd4de6134..43d005f63253 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -81,7 +81,7 @@ static void __flush_context(void) if (cntx == 0) cntx = per_cpu(reserved_context, i); - __set_bit(cntx & asid_mask, context_asid_map); + __set_bit(cntx2asid(cntx), context_asid_map); per_cpu(reserved_context, i) = cntx; } @@ -102,7 +102,7 @@ static unsigned long __new_context(struct mm_struct *mm) lockdep_assert_held(&context_lock); if (cntx != 0) { - unsigned long newcntx = ver | (cntx & asid_mask); + unsigned long newcntx = ver | cntx2asid(cntx); /* * If our current CONTEXT was active during a rollover, we @@ -115,7 +115,7 @@ static unsigned long __new_context(struct mm_struct *mm) * We had a valid CONTEXT in a previous life, so try to * re-use it if possible. */ - if (!__test_and_set_bit(cntx & asid_mask, context_asid_map)) + if (!__test_and_set_bit(cntx2asid(cntx), context_asid_map)) return newcntx; } @@ -168,7 +168,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) */ old_active_cntx = atomic_long_read(&per_cpu(active_context, cpu)); if (old_active_cntx && - ((cntx & ~asid_mask) == atomic_long_read(¤t_version)) && + (cntx2version(cntx) == atomic_long_read(¤t_version)) && atomic_long_cmpxchg_relaxed(&per_cpu(active_context, cpu), old_active_cntx, cntx)) goto switch_mm_fast; @@ -177,7 +177,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) /* Check that our ASID belongs to the current_version. */ cntx = atomic_long_read(&mm->context.id); - if ((cntx & ~asid_mask) != atomic_long_read(¤t_version)) { + if (cntx2version(cntx) != atomic_long_read(¤t_version)) { cntx = __new_context(mm); atomic_long_set(&mm->context.id, cntx); } @@ -191,7 +191,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) switch_mm_fast: csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | - ((cntx & asid_mask) << SATP_ASID_SHIFT) | + (cntx2asid(cntx) << SATP_ASID_SHIFT) | satp_mode); if (need_flush_tlb) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 76b24d4ed4ab..5ec621545c69 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -85,7 +85,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, return; if (static_branch_unlikely(&use_asid_allocator)) - asid = atomic_long_read(&mm->context.id) & asid_mask; + asid = cntx2asid(atomic_long_read(&mm->context.id)); } else { cmask = cpu_online_mask; } From patchwork Tue Jan 2 22:00:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13509540 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 516B1C47074 for ; Tue, 2 Jan 2024 22:02:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=IZkkL+Zghn7On6IzhM5WXFXYnjQ0LbDJtaLr3nlOYdk=; b=qoDObPA4vexAZX oWA0x/+Rsfw8a471wkJTbhpMQHgQ//624iaNqThsHgnzNzRgSc0Sg/SrAyiZAG+A02oqHAtF6rIwZ 5tBL3gVAT3nJTh4saoFQRK6njyN7xarmv9AUm3i9odre6xEaluL3S0AH06fNZIXw20DPOgNOe3oSp Ygm8Kn8XbCUN0I1PKt1/Mp2rxu/kxmx+zhnjy+t9A/xzRPMKJu5Wlsu6DySFDuPRV3cMdUqgbuO5V D9PwUAk+HQf51auAciK/uYnaiHpwnu8/VLdPEtVZlbs8wB82TRAw0v4WItPhbbr2xfOzpVQYPW7Mu w+M6ktrAxmD8qVsWeHWQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpc-0098XI-1N; Tue, 02 Jan 2024 22:01:52 +0000 Received: from mail-pj1-x1030.google.com ([2607:f8b0:4864:20::1030]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpX-0098Se-2b for linux-riscv@lists.infradead.org; Tue, 02 Jan 2024 22:01:49 +0000 Received: by mail-pj1-x1030.google.com with SMTP id 98e67ed59e1d1-28cd3f9cd40so717868a91.1 for ; Tue, 02 Jan 2024 14:01:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232906; x=1704837706; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=g2cmoIK/tf9yPohv/YvO5QQvGunkgXtONjIGDQZEX58=; b=W038j35yetw21ID6bC6RAus+bjNuPFEntfeyKJlGYYBD6r7atS5ke/JHjzIcdJ5JeJ ez6azwAsFJVwArAexSMIKpBp6WPdIXnLPui7WbSXn2Juo5CqMI7/i3O5TYMRfASiuhep YMJaLuCUbMb7huXSqEoVDkjOEvY7QETziJODmpaXHew4I7a1ZiO97zkQH0jZTWfeXpVl RcklVOqSjBN7APnQHCILS3tvtWwL+fucVZ9OhZ9A5khqD+sRl7uDkZvkA6MQ8T5DwcvJ pROXOSQ2mYJ2jCSrpv7LNyVPhir9cHvamw4AqKKADOz5Nrf6OW4VVLUBb7nRUg08I0aG tvVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232906; x=1704837706; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=g2cmoIK/tf9yPohv/YvO5QQvGunkgXtONjIGDQZEX58=; b=QFKzQAUasatOF9+E0k8cTgCKwnldQrig00rqygRlz3sVLsDif+iK6Bqsc2661i1HAC Sjum742Ybkuzqy/RIPRvVhH1TsQVy55sLpPJH4OqKP4RXgu11R24ihkBUQ9bMlyeZpMk p6d8IhhoZEzI8ENic4oXIB0VfNNot4h3dmOCdRirstgsigYE8sdRoLEl2hKCw8nEkXQ3 LAd+Dxggb+jDsYgnmxbXPXdI7ZuY9TKtaZtWxNRNFJG/fq4AV07cflzYQWomhlyAJcBT 56Hed5PaufkGDAf5VgiRrzFoVwTAHFOZb7MTlsBeT2JAP2bRKU6WobZdN/KzoGbRNOtG y8xg== X-Gm-Message-State: AOJu0YxOhmHxT3Ng/kVYFNLw1K2jIfs/20gu1B32xr58214JPXMT/TK8 v+czR6vKjXGNOW8X0YZvZ9+VLgc0PN8QKg== X-Google-Smtp-Source: AGHT+IG8w/0THQ/uqOPivBsiw22hWysSlz1giatwLj9QOMmwVtysp2KaHolkndd/R0ziY6DtdT9KEQ== X-Received: by 2002:a17:90a:134f:b0:28c:b1bd:3c42 with SMTP id y15-20020a17090a134f00b0028cb1bd3c42mr2130687pjf.55.1704232906433; Tue, 02 Jan 2024 14:01:46 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:46 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 09/12] riscv: mm: Use a fixed layout for the MM context ID Date: Tue, 2 Jan 2024 14:00:46 -0800 Message-ID: <20240102220134.3229156-10-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240102_140147_872123_645B9069 X-CRM114-Status: GOOD ( 12.70 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Currently, the size of the ASID field in the MM context ID dynamically depends on the number of hardware-supported ASID bits. This requires reading a global variable to extract either field from the context ID. Instead, allocate the maximum possible number of bits to the ASID field, so the layout of the context ID is known at compile-time. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- (no changes since v1) arch/riscv/include/asm/mmu.h | 4 ++-- arch/riscv/include/asm/tlbflush.h | 2 -- arch/riscv/mm/context.c | 6 ++---- 3 files changed, 4 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index a550fbf770be..dc0273f7905f 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -26,8 +26,8 @@ typedef struct { #endif } mm_context_t; -#define cntx2asid(cntx) ((cntx) & asid_mask) -#define cntx2version(cntx) ((cntx) & ~asid_mask) +#define cntx2asid(cntx) ((cntx) & SATP_ASID_MASK) +#define cntx2version(cntx) ((cntx) & ~SATP_ASID_MASK) void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, pgprot_t prot); diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index d9913590f82e..5bfd37cfd8c3 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -15,8 +15,6 @@ #define FLUSH_TLB_NO_ASID ((unsigned long)-1) #ifdef CONFIG_MMU -extern unsigned long asid_mask; - static inline void local_flush_tlb_all(void) { __asm__ __volatile__ ("sfence.vma" : : : "memory"); diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 43d005f63253..b5170ac1b742 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -22,7 +22,6 @@ DEFINE_STATIC_KEY_FALSE(use_asid_allocator); static unsigned long asid_bits; static unsigned long num_asids; -unsigned long asid_mask; static atomic_long_t current_version; @@ -128,7 +127,7 @@ static unsigned long __new_context(struct mm_struct *mm) goto set_asid; /* We're out of ASIDs, so increment current_version */ - ver = atomic_long_add_return_relaxed(num_asids, ¤t_version); + ver = atomic_long_add_return_relaxed(BIT(SATP_ASID_BITS), ¤t_version); /* Flush everything */ __flush_context(); @@ -247,7 +246,6 @@ static int __init asids_init(void) /* Pre-compute ASID details */ if (asid_bits) { num_asids = 1 << asid_bits; - asid_mask = num_asids - 1; } /* @@ -255,7 +253,7 @@ static int __init asids_init(void) * at-least twice more than CPUs */ if (num_asids > (2 * num_possible_cpus())) { - atomic_long_set(¤t_version, num_asids); + atomic_long_set(¤t_version, BIT(SATP_ASID_BITS)); context_asid_map = bitmap_zalloc(num_asids, GFP_KERNEL); if (!context_asid_map) From patchwork Tue Jan 2 22:00:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13509543 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 62B37C46CD2 for ; Tue, 2 Jan 2024 22:02:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=n4FUUJWMeQ50MhgT9WfVWFZlC3MyJ9tf2sNwhOkJThc=; b=xwdTaqHSb0Ovtq 8l3AR7qyqFvyfLEll9Y9VZTdrub62yPZCepgkxCc/5Ztk26qyjh2UEnaRmgYre9JDdp1CzWrz/jGt J4I/jwiD0priZZWJYG+ZpX/ECl+hJvuz5y0ZvwjcrnPue48tFLNsW4ccaVt3qoGZNXqXdzFavnrE0 tCKJYNJleuLdq63XoT3mPPJsJgkOBMUb7ddhm8p1Ui83lWyhfDh29q4C6DQGtp3MzWUfJGKFNx5r/ kyyjicj/shFea5k0c6DuXMLN+8cT8685Q7PSDF5dgScpHuOaKHA/H1z6pNNddEi/5D/zlIITOCOSK Kc4Xa1BmZGOOsbDgp3kw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpd-0098Xl-0i; Tue, 02 Jan 2024 22:01:53 +0000 Received: from mail-pj1-x102e.google.com ([2607:f8b0:4864:20::102e]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpY-0098Tg-2S for linux-riscv@lists.infradead.org; Tue, 02 Jan 2024 22:01:50 +0000 Received: by mail-pj1-x102e.google.com with SMTP id 98e67ed59e1d1-28c0565df34so3429737a91.0 for ; Tue, 02 Jan 2024 14:01:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232907; x=1704837707; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O8uQuAx5HbeGrqX4Ip/qQY7nv6lrxw4qacCJFbpKbko=; b=HNDASZSE0Sbn4BoMGTwmDKDz+XMSsxDzbaa6wpuAB1KP+SJ1y7a3+lcHwTtVmonZ26 D7QOgFJcldWWxNNSTuxxmQozcxRG3YvLFaLnWy5Hn70FsolNIMd9qR+PATiz8fMvgq7f jaqsyJQhyRezN3hTQKyC7lmJ/e21CORmc6ONiK6TPZuoFsu3OZBumhrG968vJQ5oErE+ GuQEE7sLgM7CUinSuNI1373s+K9FW9iLMq9CguAdEnBwdrjPbxRx5RKJYayDhGzrQOew QG9Ztv4dvNUQPeMUOEZL2n50fCkGpRppZcgYRHZC081Jwgxot5UkFgUNdcNNPy40vhEP 4ssg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232907; x=1704837707; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O8uQuAx5HbeGrqX4Ip/qQY7nv6lrxw4qacCJFbpKbko=; b=LlXZn1tHQvs/c43+VdZSAoTid8CgQ8+++/kZH2Vkdq4gNXCEYWFDl2qoxMjxzMTvg9 2jj3JfmV9yMI5CrN1KlckL5T/hHoBN9iQQHa42T4NoNqPRPDka65VekMCy8tODRsjsxq RM3y0YyowS14JX9EqYUKi4VQby6AoqUgg7iR5IIfoynQkM6ISBXndfhTgYsUx/Ryycvi gw02WNMRO/pVZ48RsTgRs8F63yQX9V42K5Kl/8cZWmxn1Zt11nBXF4bH1dCMDiuizhq0 kKBefHcPYAFyhUtd9alb+wqoFBHC7CzCaM5mbYbUfCRvcVnftLxjkdJqOAE/lNlYxMch AgGw== X-Gm-Message-State: AOJu0Yx+suCJ+fXPRMRgq/BZDKF6tv5PDYkHG+NfGyD7lvduXzy3NMt2 gvFEskpUCWw9xFz3n3rFS9Ekc1PTa9PiRA== X-Google-Smtp-Source: AGHT+IEU78MnSVmG+tTKoY5SsP+KmgJHfZS8cB0cAVh744mS+tR8J0bWreXJl70FWHY5n2NbigpZXg== X-Received: by 2002:a17:90b:33cc:b0:28a:f2cc:e008 with SMTP id lk12-20020a17090b33cc00b0028af2cce008mr6619129pjb.54.1704232907387; Tue, 02 Jan 2024 14:01:47 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:47 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 10/12] riscv: mm: Make asid_bits a local variable Date: Tue, 2 Jan 2024 14:00:47 -0800 Message-ID: <20240102220134.3229156-11-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240102_140148_796778_92F5CC1E X-CRM114-Status: UNSURE ( 9.30 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This variable is only used inside asids_init(). Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- (no changes since v1) arch/riscv/mm/context.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index b5170ac1b742..43a8bc2d5af4 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -20,7 +20,6 @@ DEFINE_STATIC_KEY_FALSE(use_asid_allocator); -static unsigned long asid_bits; static unsigned long num_asids; static atomic_long_t current_version; @@ -226,7 +225,7 @@ static inline void set_mm(struct mm_struct *prev, static int __init asids_init(void) { - unsigned long old; + unsigned long asid_bits, old; /* Figure-out number of ASID bits in HW */ old = csr_read(CSR_SATP); From patchwork Tue Jan 2 22:00:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13509542 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 28DC1C47255 for ; Tue, 2 Jan 2024 22:02:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Li/VAFYbscj6f9+EV6EJIvnlY67sDrtoWicRX6372nQ=; b=0bO57mWS+oXfcP NSMToxdaqyBljJuzUv9qqR3Q9mdCJ0W9k/B+xcXAUTiT1J7+2HF9d7rttPObm/ZCPsHgW3Nw6UBbk VhEOg+eL/msSR+umB7JAtxHst+868/efjn+Bg2Hc/GBMSGZI0QpGGpcWoXFefjZt8Sw75PCJQAIcK YAilAF4DwEmnRgSLTPJB3nwZ2ifdH2nDmkuhfpEeQD/9V1h1CmvEgEGIx6v5XNPgTXFFrioBr576x T0wnvqCIft1r2BMWF/Q635Ujt60dx/9Bcx7NZcALvflpc6kDnj0LM4esvbZl1Tq1CYoRPZB45gOWh wipX1x+LXgqoecnkblgw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpe-0098YM-01; Tue, 02 Jan 2024 22:01:54 +0000 Received: from mail-pj1-x1033.google.com ([2607:f8b0:4864:20::1033]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpY-0098U9-2N for linux-riscv@lists.infradead.org; Tue, 02 Jan 2024 22:01:50 +0000 Received: by mail-pj1-x1033.google.com with SMTP id 98e67ed59e1d1-28bc7155755so3952313a91.2 for ; Tue, 02 Jan 2024 14:01:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232908; x=1704837708; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MROdnZxFJAn7brM6ye/jIKPJcvVCbRqlgsjUSxtP0PY=; b=JC/XTboG0Z+lnXJ/9qEPzwgW8aQcfd/HAYL1poXng8s5XiPkM30S6mdCW7FQ4ZHNIw ISZ+hYcdutjDSQ5McqQmX1hHwHwxO0OkszCjQW0sFKj0qQ91+xU6eeLvAuWBhFLUTqhw SnZ+qE+DAvjZUT2ByFKX5SQS1ZBEw7NNvJBMYueI8FAfAmjDt36p3VhJbwQjwt2jJcqC 87DfIWJrXigyIC+JKNI/xzue1uU1YqwFVoaSQ7BBERq6BV9AqhR4FYL8BPInyXwrndT6 tnv6C2i7K1tY15duSq7JkO4N1Dnq3xWc3NNoSG76dFEsvdqCrfBwSwuOaRfh1XegbjDS vhJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232908; x=1704837708; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MROdnZxFJAn7brM6ye/jIKPJcvVCbRqlgsjUSxtP0PY=; b=q49Dlxqr9Jt7lN1s2DhiaWSHi7bIJXVwfIyhYe7H6FphxZWdY/2WBGh+T7Br+e0Arc q5JMa4fz4Kadong1FoScQvJGfaj15dUDpPLr7lyTeYpKe0tgG55V+DL0GFetmKGy3Tj/ PkMRANfDo2TsDsy8tyMbY0mIV1pPpf1ZT+VXBBGSaBgbE1xOa2ADZtrwMgjNJSFkok/e IkT9xkJc/WeCYwd3vwk4MRmqvCbqB/KrYkfYZtyeOOgt8uYPQcCIa0TWfkOU+zix0YxW 8WChLKZDQ/LgNcP435YLrbaxqFNNhKpKpvMHrDiGYbRMibBhQ6kP0h1OcYVNg3kkxuVt qxXA== X-Gm-Message-State: AOJu0YzCcLOpgtcIjAiLVceY/OuDoOoast/3w9X/s1vxiAPNksAFUC0C gW12WB/iZI/e/VV2Yy/WgFyKGZO9JjFY+g== X-Google-Smtp-Source: AGHT+IFQ4MdRufOELZ2z13xg7yZ/1JoYbB5tE+2va7NZu/tN1uOevH1SKPh+nsWqBiMa4TjGPndiZg== X-Received: by 2002:a17:90a:5c82:b0:28c:2e1f:8bf2 with SMTP id r2-20020a17090a5c8200b0028c2e1f8bf2mr4930306pji.84.1704232908323; Tue, 02 Jan 2024 14:01:48 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:48 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 11/12] riscv: mm: Preserve global TLB entries when switching contexts Date: Tue, 2 Jan 2024 14:00:48 -0800 Message-ID: <20240102220134.3229156-12-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240102_140148_773278_35FF2325 X-CRM114-Status: GOOD ( 14.86 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org If the CPU does not support multiple ASIDs, all MM contexts use ASID 0. In this case, it is still beneficial to flush the TLB by ASID, as the single-ASID variant of the sfence.vma instruction preserves TLB entries for global (kernel) pages. This optimization is recommended by the RISC-V privileged specification: If the implementation does not provide ASIDs, or software chooses to always use ASID 0, then after every satp write, software should execute SFENCE.VMA with rs1=x0. In the common case that no global translations have been modified, rs2 should be set to a register other than x0 but which contains the value zero, so that global translations are not flushed. It is not possible to apply this optimization when using the ASID allocator, because that code must flush the TLB for all ASIDs at once when incrementing the version number. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- (no changes since v1) arch/riscv/mm/context.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 43a8bc2d5af4..3ca9b653df7d 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -200,7 +200,7 @@ static void set_mm_noasid(struct mm_struct *mm) { /* Switch the page table and blindly nuke entire local TLB */ csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | satp_mode); - local_flush_tlb_all(); + local_flush_tlb_all_asid(0); } static inline void set_mm(struct mm_struct *prev, From patchwork Tue Jan 2 22:00:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13509541 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E0D1CC4707B for ; Tue, 2 Jan 2024 22:02:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=/lp0+H4CrN5+X/+hWWNaR/K52KLlk1E30geDBLaRXbQ=; b=TqA5oM2tIkp/Gz 3gxSTLcBCnxGcWKuQs5qN15BqvTBBnkcJ8phMkBhl1TCIb52h8TRq3dfuxOq+oYVWZrtUpyoEzwDz AziUoXYQllr0L5rVLPp+QqRBju9i3lXwtmWGaM4lX+8ktIyTKj07ic3n9wjsD/GO5sleA5hhrXFK8 tNdP6+aTrGOl98gsnLDpM/CmxKwI1y4pCcUdLGBV+KHHncnzPaTlAXSqsYEUutAKjnVRqh7OAbqHu N8sU1Mj51h/1GgaMu7TXTFhp8tmzd3g9O85WI4RuD0OdhLtsZoelPTzm/UrRcF5WF+VyvGVqzipD6 WgYVTmW1FAHn2yp+7bdg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpe-0098Yd-2H; Tue, 02 Jan 2024 22:01:54 +0000 Received: from mail-pl1-x634.google.com ([2607:f8b0:4864:20::634]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rKmpa-0098Un-2K for linux-riscv@lists.infradead.org; Tue, 02 Jan 2024 22:01:52 +0000 Received: by mail-pl1-x634.google.com with SMTP id d9443c01a7336-1d3f2985425so36164955ad.3 for ; Tue, 02 Jan 2024 14:01:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232909; x=1704837709; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ntXY1BdgwMpYCVbC/qFvmg3dz3t/ZzKIeqKl93B+ekg=; b=DpRm75I7F3nitvKfPEWkfdTdE38CcRA6YXjZCFR+OjIAr93v6NzyB+1LwxGUtbjq4D 02bUBUp94anemJ3TSsEcfMED01agYUY52NoV5sv9MeHuiiLyzh/7MWkyfyYAn49RPihq dCBsITs8FxvV7wURjR8xvrRWkyfhgi/H+Ispv2W30ku9uBm2VJGBuxUkFPICFSzJZ20Z wwpEWK0TKCPc0TGl8EBHrpyYrQYSckA45eLV/UrIUYICNlD4A9clQld8o3cBZX11GT+x wmuluZqoi9pbmJtykqn6x0JcBzy7hFPwDFZCL304pMMa3q3ajjfjXj8ybHqf4KmijpSr nymA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232909; x=1704837709; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ntXY1BdgwMpYCVbC/qFvmg3dz3t/ZzKIeqKl93B+ekg=; b=jMfkUIJKEg//cBUiDGNitIj9RU3lWJHDltegZW1v6y8P86DEPuQdIDIPDde7ql+cTD R0aTZSBR4BmHdtwA/jS3CwjGlFgfx0gGlbBmJtcPkqIz8rN3tbK52f2jITV4xNgomoJ1 JiVJxFfGqh1nXlWPwDH1Gs1gDfNisCIyeSqeokH+1sEqdeyB8Dy9wLY170Nmf4/vJ85V BHiKdVbUK2BTiZLU8kZ8a3dULyGPU1QQiTq7ZamphKojclfITC2m9MnltYoyGZCh6YXF GFoFQbNwpHBtAGjpHmDxieWf7GlDEU4/DxMBzaAg+utsf4Nfhr3PN3o2ZFYMEl9cpNEr D+ow== X-Gm-Message-State: AOJu0YwBhyMTiXGNnt5Jz4m8sSGVnZEaCmqrPG51YkLnojnUwQWjIZuJ TioatD4r0hDo2jgtBwFFHbovFrUVB8Mxaw== X-Google-Smtp-Source: AGHT+IE6m2P6rWzQ+Gmcf0Ghh7A4crRVy8Bq5ePskuN3Kp4kY6ZC52OLPfUohVtqOdFfc+/yJvJ6pQ== X-Received: by 2002:a17:90a:bc46:b0:28b:31f5:9668 with SMTP id t6-20020a17090abc4600b0028b31f59668mr5241043pjv.30.1704232909372; Tue, 02 Jan 2024 14:01:49 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:49 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 12/12] riscv: mm: Always use an ASID to flush mm contexts Date: Tue, 2 Jan 2024 14:00:49 -0800 Message-ID: <20240102220134.3229156-13-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240102_140150_759195_4AD929B5 X-CRM114-Status: GOOD ( 13.93 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Even if multiple ASIDs are not supported, using the single-ASID variant of the sfence.vma instruction preserves TLB entries for global (kernel) pages. So it is always more efficient to use the single-ASID code path. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- Changes in v4: - There is now only one copy of __flush_tlb_range() Changes in v2: - Update both copies of __flush_tlb_range() arch/riscv/include/asm/mmu_context.h | 2 -- arch/riscv/mm/context.c | 3 +-- arch/riscv/mm/tlbflush.c | 3 +-- 3 files changed, 2 insertions(+), 6 deletions(-) diff --git a/arch/riscv/include/asm/mmu_context.h b/arch/riscv/include/asm/mmu_context.h index 7030837adc1a..b0659413a080 100644 --- a/arch/riscv/include/asm/mmu_context.h +++ b/arch/riscv/include/asm/mmu_context.h @@ -33,8 +33,6 @@ static inline int init_new_context(struct task_struct *tsk, return 0; } -DECLARE_STATIC_KEY_FALSE(use_asid_allocator); - #include #endif /* _ASM_RISCV_MMU_CONTEXT_H */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 3ca9b653df7d..20057085ab8a 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -18,8 +18,7 @@ #ifdef CONFIG_MMU -DEFINE_STATIC_KEY_FALSE(use_asid_allocator); - +static DEFINE_STATIC_KEY_FALSE(use_asid_allocator); static unsigned long num_asids; static atomic_long_t current_version; diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 5ec621545c69..39d80f56d292 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -84,8 +84,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, if (cpumask_empty(cmask)) return; - if (static_branch_unlikely(&use_asid_allocator)) - asid = cntx2asid(atomic_long_read(&mm->context.id)); + asid = cntx2asid(atomic_long_read(&mm->context.id)); } else { cmask = cpu_online_mask; }