From patchwork Thu Feb 29 23:21:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13577708 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B2D3C54798 for ; Thu, 29 Feb 2024 23:22:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 889BB6B0099; Thu, 29 Feb 2024 18:22:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 838146B009B; Thu, 29 Feb 2024 18:22:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 661A16B009C; Thu, 29 Feb 2024 18:22:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 547D96B0099 for ; Thu, 29 Feb 2024 18:22:19 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 2ADA0C03B6 for ; Thu, 29 Feb 2024 23:22:19 +0000 (UTC) X-FDA: 81846417198.27.06C585A Received: from mail-il1-f181.google.com (mail-il1-f181.google.com [209.85.166.181]) by imf15.hostedemail.com (Postfix) with ESMTP id 365A6A0009 for ; Thu, 29 Feb 2024 23:22:17 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=QjMNTgSR; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf15.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.166.181 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709248937; a=rsa-sha256; cv=none; b=tZi7uDP6JCCgIuyJKk9ZYa8+9J9WOZsNoE3L+O+0NHnHAIYRLVdT6thA9mx8OAcQs0imyq wkcrydV2NrTXpWe9y8AtAcAsknNKJ0Bg6QfR/Ax6O42a1Qgz96U1jA8DYvwe1njVbjhbeo +vue/1R7J0sj1CcaSPAAPqbnMBkKsAY= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=QjMNTgSR; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf15.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.166.181 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709248937; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nL3CU8jW9vVfZeENVgIQs2Z0lmYtCUXWU5kIlhxMA+M=; b=c6xAj19bbjMJEvv8ikZHpZd5/MefokTkNwoLdJOw1psBCV/PYGVXPzZCWmW8LZ71Pw8tM4 ptGUMZIPv4yuOfUuItsa69XpZs2ha1tgDm/cifp7YcpSRdR3KE39ewnAwfi3Kb3yUrGdV/ I/g+LJ4p0MyJEhWqmJbRxqlXaUIDWwc= Received: by mail-il1-f181.google.com with SMTP id e9e14a558f8ab-3651c1e1022so7353665ab.0 for ; Thu, 29 Feb 2024 15:22:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1709248936; x=1709853736; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nL3CU8jW9vVfZeENVgIQs2Z0lmYtCUXWU5kIlhxMA+M=; b=QjMNTgSRwARh/B/NjAem7Lv33vyp9lvCU1+4XjhGyfR4IMjNfAocmqCBsrUhFr8NFH 7Ztfs6mPHE16Jf6WzYBpmOfRVWXtSS5zH8SlxtW0OjYlGVKwKfRdjV4iJDkRvc+n5K+Y XLv0CnuU7xXKHZOvcnhieT7mJ2Wu8D6hqDsVwYCc6obTIBn03KuRrZgRrUjfxBFHYoYJ EoYSb2GA54ACQVd+mKi/5ECFgK3UMjs2henoVmBJENvILf+jXu06meV02Og9wpTyvboP T0cVGSoXMnmkxmoakoY+LyHHTlK0rXPvhmSOo1aFFwtyspDJEHIY/M5+J1FmSecuK4FV OxHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709248936; x=1709853736; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nL3CU8jW9vVfZeENVgIQs2Z0lmYtCUXWU5kIlhxMA+M=; b=uAunNNStMMzi1z47T3ElWPqx7wOUK1oGTLt8UYJF9afWYbLupL6NpBq36Xt6q+yPE2 sDaGmbw0YbHJEx9tLIv1vXfXTGFMPUBi2E59Xfvpu3HgC0MEp2Rc5VaF3lH5vMCj+nkq cLeWqDPhM0dDPlaGQRA0RjIi2y7bnTl38vpyuiZvoHZ2DSn7KR3sVbvj8oQ4YBnuSqyf qZGAluFLa50QherfywRXDDBGML+icLqN/QnIFhYxWSZkLzsVZsNWeaRADBB4ZqbEZ2Od ymjsx5k7DGhFU1C34248lKULeyD1RGo0lMbh5M/icVS6YfD4wTkPBBhajDnQD0OtmDKz bmSw== X-Forwarded-Encrypted: i=1; AJvYcCUCcZgUs3KXSn5k0VW9CDu6V6Vvcu7uX9M311RIWiOgX5K5/Ups9afyqO80n4NZLdtbcyTcVs8VOejGAFmGjz6mb9Y= X-Gm-Message-State: AOJu0Yw65vTlTgny9ZOATqGkZpMlRKeBsKZKDKlAsoiTpvFmL3YQCd/6 1xTsT7S5nPhL43Ko9n0BKxov8Z/XzkzuFpLZKzgYPBHYMLAKUNPZP4Q+XCw5UDw= X-Google-Smtp-Source: AGHT+IEuQOuQ50R8qoZP0RbIoJTNSdVCKrs4lRWLzJ+znFdHc+eSLoYj/Ypw4BmFNiQ++vK2RkPm8w== X-Received: by 2002:a05:6e02:1b8f:b0:365:ae26:5827 with SMTP id h15-20020a056e021b8f00b00365ae265827mr159548ili.5.1709248936255; Thu, 29 Feb 2024 15:22:16 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id c6-20020aa78806000000b006e55aa75d6csm1779719pfo.122.2024.02.29.15.22.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Feb 2024 15:22:15 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v5 03/13] riscv: Use IPIs for remote cache/TLB flushes by default Date: Thu, 29 Feb 2024 15:21:44 -0800 Message-ID: <20240229232211.161961-4-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240229232211.161961-1-samuel.holland@sifive.com> References: <20240229232211.161961-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 365A6A0009 X-Stat-Signature: 5pshztxuu6tut91ci4dh7ygkxteknouc X-HE-Tag: 1709248937-465014 X-HE-Meta: U2FsdGVkX1+0iFrkvODnhGB8+BdWAHZQc05r4rK/z7yVH376ixn5DywRUCTbvURBUoStTGrPJ2zc1cINVQTjGsDN8MJVwWGkYSyo/mJa6Tb5GmgxuZB4JKUSIIp4Eux8DyDoqd8Bxr6luxDiSHaSlVrvQmUPkPcixnDK12ZUM0PKA5q0HDWc33x1fZkhiL2C+1TFo1NklXcWEUbgtPjMsd3Cu4NZxq+I1NnGKrD+cYMDP2r+oMHt3ddXt/UOBx1wqDTezv5S+qjP8XvHo4Y3oc/aVMa57uUH1hsMGTstCT7owroJSQDsm4B6ryOIYKTmvXhMoIxO06vDPZXg8DIC1VbcHfBHW8TbnfENaijarCm05PI78kWjaCFWASeT5MN6lqvRZqlVqcJDtdpmaJdIjquNhJUD3zIay437S8cUFT4JknDfo+MtT7NaImmywGkq0RbGGQufK9ZbCMsvz6FquDXC2VjLJtDbMtg7aFQwUbkK/QiI6p0QGY3o4rmPrB+WFZjJYQwNj9ukhaTaTrbbuMKjFyGslE5qAMMKGgRhrSl7vC1R/VXXvBx95zPBwrQatMt6ZIhd5t/0NHsjJpuloco9vd04jgD0Rx3fsX/Cl3szrgSHM/0FlaJuL+uQPW5TQqJGX6FG+cB9Q3O2Xk026inADzeSan/OYOTTHQ8x3SLYDKRQuwc0kK7F6dPLJDv2LWGSqRNc+66HwTWlUIMUteXlHu1w6DnzQkRGIyla8jeWDADDRNExfneuA6GNmxtuZsVRLtTGHhsVMmYl0vfNISMYGqsCByJCgUYGcbrUFFmacBNRuJwg6R+sCCfXUt52uwdIypg30csOOM/F4DNZoVM4WkzViz8Af2U8IeFakLkKi8rtJQ3YPqY5Zj03hlS8rDz15u+MOkXZCaSwABWG2P5lpl2ka8XaCjll1VzFnOJjzpo9JJfLC+H2yKohWJjeo2L+Lb7y7aYaWxtxI1p auk00okl VpUXd85MSa0uKlAwR4vtO8MEJ9CXVTwNDnUxOyonJcd5/yz269WBLViAKA2wxd/6prm6evR4uCkOLQgTERhQy7rZ2mktaEBHyK0philbobHt7mZNe0wEngKUs0bSRj+YH/n5K7WFGCXl9tbXjHDFL3GvBgHfWWacxBKhL4iqII1rgPN80eIQEDAqJqutyX3wddjqplt0p+4ojW5tiNDXy83jawyqJiOm644LfKgsaKZE6QHio908rFuVb7TjMXSSiqHZDclgeZ9gycN79Ba9oa3y7ktrp63eq+O3iGN8Fc6oUJF4wEH9V1L/2kqhZEHU6aLdWLEBQKbxwiTZbHVpEVcf/wZH/+zfkvD+XTkF6D5mtxDylTkVuqA9A7VvjdM8ld3e7B013VbBxEQE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: An IPI backend is always required in an SMP configuration, but an SBI implementation is not. For example, SBI will be unavailable when the kernel runs in M mode. Generally, IPIs are assumed to be faster than SBI calls due to the SBI context switch overhead. However, when SBI is used as the IPI backend, then the context switch cost must be paid anyway, and performing the cache/TLB flush directly in the SBI implementation is more efficient than inserting an interrupt to the kernel. This is the only scenario where riscv_ipi_set_virq_range()'s use_for_rfence parameter is false. Thus, it makes sense for remote fences to use IPIs by default, and make the SBI remote fence extension the special case. sbi_ipi_init() already checks riscv_ipi_have_virq_range(), so it only calls riscv_ipi_set_virq_range() when no other IPI device is available. So we can move the static key and drop the use_for_rfence parameter. Furthermore, the static branch only makes sense when CONFIG_RISCV_SBI is enabled. Optherwise, IPIs must be used. Add a fallback definition of riscv_use_sbi_for_rfence() which handles this case and removes the need to check CONFIG_RISCV_SBI elsewhere, such as in cacheflush.c. Signed-off-by: Samuel Holland Reviewed-by: Anup Patel --- Changes in v5: - Also switch to riscv_use_sbi_for_rfence() in asm/pgalloc.h Changes in v4: - New patch for v4 arch/riscv/include/asm/pgalloc.h | 7 ++++--- arch/riscv/include/asm/sbi.h | 4 ++++ arch/riscv/include/asm/smp.h | 15 ++------------- arch/riscv/kernel/sbi-ipi.c | 11 ++++++++++- arch/riscv/kernel/smp.c | 11 +---------- arch/riscv/mm/cacheflush.c | 5 ++--- arch/riscv/mm/tlbflush.c | 31 ++++++++++++++----------------- drivers/clocksource/timer-clint.c | 2 +- 8 files changed, 38 insertions(+), 48 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 87468f67951a..6578054977ef 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -8,6 +8,7 @@ #define _ASM_RISCV_PGALLOC_H #include +#include #include #ifdef CONFIG_MMU @@ -90,10 +91,10 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) { - if (riscv_use_ipi_for_rfence()) - tlb_remove_page_ptdesc(tlb, pt); - else + if (riscv_use_sbi_for_rfence()) tlb_remove_ptdesc(tlb, pt); + else + tlb_remove_page_ptdesc(tlb, pt); } #define pud_free pud_free diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index 6e68f8dff76b..ea84392ca9d7 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -375,8 +375,12 @@ unsigned long riscv_cached_marchid(unsigned int cpu_id); unsigned long riscv_cached_mimpid(unsigned int cpu_id); #if IS_ENABLED(CONFIG_SMP) && IS_ENABLED(CONFIG_RISCV_SBI) +DECLARE_STATIC_KEY_FALSE(riscv_sbi_for_rfence); +#define riscv_use_sbi_for_rfence() \ + static_branch_unlikely(&riscv_sbi_for_rfence) void sbi_ipi_init(void); #else +static inline bool riscv_use_sbi_for_rfence(void) { return false; } static inline void sbi_ipi_init(void) { } #endif diff --git a/arch/riscv/include/asm/smp.h b/arch/riscv/include/asm/smp.h index 0d555847cde6..7ac80e9f2288 100644 --- a/arch/riscv/include/asm/smp.h +++ b/arch/riscv/include/asm/smp.h @@ -49,12 +49,7 @@ void riscv_ipi_disable(void); bool riscv_ipi_have_virq_range(void); /* Set the IPI interrupt numbers for arch (called by irqchip drivers) */ -void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence); - -/* Check if we can use IPIs for remote FENCEs */ -DECLARE_STATIC_KEY_FALSE(riscv_ipi_for_rfence); -#define riscv_use_ipi_for_rfence() \ - static_branch_unlikely(&riscv_ipi_for_rfence) +void riscv_ipi_set_virq_range(int virq, int nr); /* Check other CPUs stop or not */ bool smp_crash_stop_failed(void); @@ -104,16 +99,10 @@ static inline bool riscv_ipi_have_virq_range(void) return false; } -static inline void riscv_ipi_set_virq_range(int virq, int nr, - bool use_for_rfence) +static inline void riscv_ipi_set_virq_range(int virq, int nr) { } -static inline bool riscv_use_ipi_for_rfence(void) -{ - return false; -} - #endif /* CONFIG_SMP */ #if defined(CONFIG_HOTPLUG_CPU) && (CONFIG_SMP) diff --git a/arch/riscv/kernel/sbi-ipi.c b/arch/riscv/kernel/sbi-ipi.c index a4559695ce62..1026e22955cc 100644 --- a/arch/riscv/kernel/sbi-ipi.c +++ b/arch/riscv/kernel/sbi-ipi.c @@ -13,6 +13,9 @@ #include #include +DEFINE_STATIC_KEY_FALSE(riscv_sbi_for_rfence); +EXPORT_SYMBOL_GPL(riscv_sbi_for_rfence); + static int sbi_ipi_virq; static void sbi_ipi_handle(struct irq_desc *desc) @@ -72,6 +75,12 @@ void __init sbi_ipi_init(void) "irqchip/sbi-ipi:starting", sbi_ipi_starting_cpu, NULL); - riscv_ipi_set_virq_range(virq, BITS_PER_BYTE, false); + riscv_ipi_set_virq_range(virq, BITS_PER_BYTE); pr_info("providing IPIs using SBI IPI extension\n"); + + /* + * Use the SBI remote fence extension to avoid + * the extra context switch needed to handle IPIs. + */ + static_branch_enable(&riscv_sbi_for_rfence); } diff --git a/arch/riscv/kernel/smp.c b/arch/riscv/kernel/smp.c index 45dd4035416e..8e6eb64459af 100644 --- a/arch/riscv/kernel/smp.c +++ b/arch/riscv/kernel/smp.c @@ -171,10 +171,7 @@ bool riscv_ipi_have_virq_range(void) return (ipi_virq_base) ? true : false; } -DEFINE_STATIC_KEY_FALSE(riscv_ipi_for_rfence); -EXPORT_SYMBOL_GPL(riscv_ipi_for_rfence); - -void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence) +void riscv_ipi_set_virq_range(int virq, int nr) { int i, err; @@ -197,12 +194,6 @@ void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence) /* Enabled IPIs for boot CPU immediately */ riscv_ipi_enable(); - - /* Update RFENCE static key */ - if (use_for_rfence) - static_branch_enable(&riscv_ipi_for_rfence); - else - static_branch_disable(&riscv_ipi_for_rfence); } static const char * const ipi_names[] = { diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index 55a34f2020a8..47c485bc7df0 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -21,7 +21,7 @@ void flush_icache_all(void) { local_flush_icache_all(); - if (IS_ENABLED(CONFIG_RISCV_SBI) && !riscv_use_ipi_for_rfence()) + if (riscv_use_sbi_for_rfence()) sbi_remote_fence_i(NULL); else on_each_cpu(ipi_remote_fence_i, NULL, 1); @@ -69,8 +69,7 @@ void flush_icache_mm(struct mm_struct *mm, bool local) * with flush_icache_deferred(). */ smp_mb(); - } else if (IS_ENABLED(CONFIG_RISCV_SBI) && - !riscv_use_ipi_for_rfence()) { + } else if (riscv_use_sbi_for_rfence()) { sbi_remote_fence_i(&others); } else { on_each_cpu_mask(&others, ipi_remote_fence_i, NULL, 1); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 8d12b26f5ac3..0373661bd1c4 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -78,10 +78,10 @@ static void __ipi_flush_tlb_all(void *info) void flush_tlb_all(void) { - if (riscv_use_ipi_for_rfence()) - on_each_cpu(__ipi_flush_tlb_all, NULL, 1); - else + if (riscv_use_sbi_for_rfence()) sbi_remote_sfence_vma_asid(NULL, 0, FLUSH_TLB_MAX_SIZE, FLUSH_TLB_NO_ASID); + else + on_each_cpu(__ipi_flush_tlb_all, NULL, 1); } struct flush_tlb_range_data { @@ -102,7 +102,6 @@ static void __flush_tlb_range(struct cpumask *cmask, unsigned long asid, unsigned long start, unsigned long size, unsigned long stride) { - struct flush_tlb_range_data ftd; bool broadcast; if (cpumask_empty(cmask)) @@ -118,20 +117,18 @@ static void __flush_tlb_range(struct cpumask *cmask, unsigned long asid, broadcast = true; } - if (broadcast) { - if (riscv_use_ipi_for_rfence()) { - ftd.asid = asid; - ftd.start = start; - ftd.size = size; - ftd.stride = stride; - on_each_cpu_mask(cmask, - __ipi_flush_tlb_range_asid, - &ftd, 1); - } else - sbi_remote_sfence_vma_asid(cmask, - start, size, asid); - } else { + if (!broadcast) { local_flush_tlb_range_asid(start, size, stride, asid); + } else if (riscv_use_sbi_for_rfence()) { + sbi_remote_sfence_vma_asid(cmask, start, size, asid); + } else { + struct flush_tlb_range_data ftd; + + ftd.asid = asid; + ftd.start = start; + ftd.size = size; + ftd.stride = stride; + on_each_cpu_mask(cmask, __ipi_flush_tlb_range_asid, &ftd, 1); } if (cmask != cpu_online_mask) diff --git a/drivers/clocksource/timer-clint.c b/drivers/clocksource/timer-clint.c index 09fd292eb83d..0bdd9d7ec545 100644 --- a/drivers/clocksource/timer-clint.c +++ b/drivers/clocksource/timer-clint.c @@ -251,7 +251,7 @@ static int __init clint_timer_init_dt(struct device_node *np) } irq_set_chained_handler(clint_ipi_irq, clint_ipi_interrupt); - riscv_ipi_set_virq_range(rc, BITS_PER_BYTE, true); + riscv_ipi_set_virq_range(rc, BITS_PER_BYTE); clint_clear_ipi(); #endif