From patchwork Wed Mar 27 04:49:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13605547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0C3FC54E67 for ; Wed, 27 Mar 2024 04:50:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3D8EA6B0096; Wed, 27 Mar 2024 00:50:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 341766B0098; Wed, 27 Mar 2024 00:50:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13EF16B0099; Wed, 27 Mar 2024 00:50:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E98826B0096 for ; Wed, 27 Mar 2024 00:50:47 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 86FD81203EA for ; Wed, 27 Mar 2024 04:50:47 +0000 (UTC) X-FDA: 81941593734.06.5D70BC6 Received: from mail-yw1-f178.google.com (mail-yw1-f178.google.com [209.85.128.178]) by imf13.hostedemail.com (Postfix) with ESMTP id BCD4D20016 for ; Wed, 27 Mar 2024 04:50:45 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=b0OiK0IU; spf=pass (imf13.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.128.178 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com; dmarc=pass (policy=reject) header.from=sifive.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711515045; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=G6C+MSH7tzUf74TudLETMF5DBzv2rpp8bzfwEKTodUs=; b=M/Pc+XE+IPjb+OX+/axr1YCpj1L0IQEWQJxVOLNXqBqmwmMcNStrcKfCWUJMbRg2/L/qmw EBHv7uprcnTjQ6u9YpPD6UcTAt+QDuxJ1zaYzkwLZEFgO41ZI1rUG+N0aqY/a6RX6aD3oV ZHR/Ssw8zewYVAl53LRJ/+ZhCvTi5PE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711515045; a=rsa-sha256; cv=none; b=M6MH/6l84DAJHnILfdjSSqr0Xfa51WQl0/JoLUaxnI8i0hHvUiFgs9n2vRdcKIKuOma7gs 6zDtNVaxLr3dUXky6Ohp62GCOiTBML7unVwBupg5PPCH+JlUYl814yg58NoifHivdrYGVN h/hhhdGTXjRAt2iKAnVdGS/jMKm0zf4= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=b0OiK0IU; spf=pass (imf13.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.128.178 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com; dmarc=pass (policy=reject) header.from=sifive.com Received: by mail-yw1-f178.google.com with SMTP id 00721157ae682-60a0599f631so55694577b3.2 for ; Tue, 26 Mar 2024 21:50:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515045; x=1712119845; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=G6C+MSH7tzUf74TudLETMF5DBzv2rpp8bzfwEKTodUs=; b=b0OiK0IUkDnk1hOdJYJMFLf8hIpzIMxO2lR+qoBiB0JyWRnhFIKSN9bw4cN7iotOy9 emRhm9M9h1EimXMmDTpWGtelor732AbfYHZlm23AGx95+Jp/nPnWMOQt9DlzWg2QZIbT vB7BUmHBqCEE8fVoAMeACd09V1k5a36lv5Kb+5gk4eJoIcXH45ftZP0ZXeYTB+o83wqq MSe5jXiVA4RsxY4ZbYCXesCCiWwphNmz0ArRMLmc2A7LqDsnWwPTzc+KZy9scQ942ifg Nzx45GHcTolLR49TG6G+UWp3fPBTSVhfzygBiK7DFnNh6wAnc32NJ61GQU5s8kb75W64 KTKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515045; x=1712119845; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G6C+MSH7tzUf74TudLETMF5DBzv2rpp8bzfwEKTodUs=; b=qBFKNbr+EL40w7dXalgIK9ydOjDGpW/zV4I8ythoYBeXzKViqhJsi3JPZ/3uNsQJMp zNtnTdbFdGpOwQVBkIeVEkVEnvHC6d7ihy0w/V1fmdsAg+W9ON9m307dOyjm+auB5dMW Y49H8p0Sy284ZDnNAwk1evIrw1KS3NtjbKOfSSkmKiAnSF3V4ebj7xFu2Z+OoVLo4Xeu NEnaJiW+EhT2CSCljQeYtG/OOqzxaYg5IlmxD1j4jaeFgAbwYHvJAYDG4PFd88g/hZ9N QSmVLEMbBwcN7dDHIr7NNBEF5Oszb5CzddJePU9FlESQrDoZkIXEp7Uv/MP+XJq71ELx Oo/w== X-Forwarded-Encrypted: i=1; AJvYcCWPZo9n75uExnP9M73MXJQgH2eodmXlxahd4BqJZuSMOCRZ8gkeeoDo/GN4BtjKc8wsuzbs7eLAzryzxXlko9ntiDs= X-Gm-Message-State: AOJu0YyiDJYJO31ABLu99MZD2tfvTN181OSLsxXXgE2ZL1X+3ywFfJXM +vRAPiuyRD9SXqEbyIf31Uhh9jMH2lpZK0GBR09RfesI/L9WGwnMNRzWCEwQcB0= X-Google-Smtp-Source: AGHT+IH+jSoiPAnTh406w99fZLlqeG+exqQi31XclYwr8IOVXBgedOOFyirc6G8eGQn9SYpg/+yAIQ== X-Received: by 2002:a81:8084:0:b0:611:9be5:4e17 with SMTP id q126-20020a818084000000b006119be54e17mr18863ywf.51.1711515044780; Tue, 26 Mar 2024 21:50:44 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:43 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland , Anup Patel Subject: [PATCH v6 03/13] riscv: Use IPIs for remote cache/TLB flushes by default Date: Tue, 26 Mar 2024 21:49:44 -0700 Message-ID: <20240327045035.368512-4-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-Stat-Signature: oynn5prf3fghfuyyjbqdxmf3qyg1w451 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: BCD4D20016 X-Rspam-User: X-HE-Tag: 1711515045-5767 X-HE-Meta: U2FsdGVkX18kWpHsCNqD6zsAOGACFp2qmhGxRLU5O3w38ZCFVqxlczGTg9RBd1Tdvf5tg0YRfJSiw+I6iEbaMTkCUf9dA6HfNyXqcDf0O1TC9myJRyUCPjf8XugwndMM/qSZltfQy4pKYMlO/f3H++mwj2NeTbRVAcG7uhIQd4aPrV+Xtt1XW1y7D4SCacMDH43z5nOJpC/eSz7a8o60JYNtYEaAJQcSfCgD8iZIroa7XMy55tCybo3RTLAc8sWCU7DcMTgFCyTWsK1vGpTFQEqpBrK2KqBIBgcQSkzb84TzALPykfzwv7l1qeArvvj4XBZ9Bg816XDqCywEZsd6g7n+wU2EMfGAZfRQyDN1XwTnBisnUq8Sc1+wwZ/d+VNT05KsJVJH1J4ZaV1kXcM57+TqmVWfuwBXGLewbKRKsXPuWec9N8IanCoIA454GyCZg4ey5e8hzjiDQ4dz2ccCija6DBWLDOZTPbjlzUD4B334yjoOUHJQUFkZNIfXEX51atgetXqJzepdW9/lxZaY5TfAAfg/x0RWkQKoDCURkwsesCIm+ZCO8eXkj+3tEONedZF+0gO+Ld/skxsMlA3uAvoMEkuSHriJpaDmKuEjcpUvym3by0fKvNtTr3B2FWEEdx2Q0WF12nOTpby922KRbbosSoFTs4TjabHywB1Wjm2SdhmNr+QR+F4PuHglf3l6vbTSX5eqleCAk3DQAc+GqjTi/em7oLI7W9zKW4iQr3MuD5msBE/K+0enB5Os+JWi1IanwtZaQeGeuW1yaLGzAxLbNEdMbz19yArn5XQk6udvGzCDVQvi67G/vwDqW3Z+yez/rqtWwZDPL68DUsbBIg5v6lSUR+ln+tu3fa8kYWhAgiGJQwUlEiQWfp49YolDfFXN/fjoFbCgL2j5YQOW8yJlu3BMNzmlLodWMBsGxXa566N6saCmC4KiSBhlZF+KnY3oZbA9b2YteMDxLXN A6mEGtpL yRJ2xt59r+ZtIrc9WUnsqo+gfTGD6OfnA9hJYLybkyMm7c0Zwu7amqwzfrpicY1ZCAnzWxG77FZZxEYJ6Go2yCmeJRoTdHF8Vh9wjHbSqRcEv8KwkRqwrF66cwhJ1vGwh83WTHgkzorPvFTM8aIZGIa9lPFYdR0nExNOoYkEZWYatlFDFIQ8MyH0tQKpp492/Gt3b/OneFRKlclvYfpD4zYC1kPWrmcVIrUzjpOYkdOjb6qSZT92FK5Z3WMDKI9ppo28NsKqRkHR1spLLT3NFH1CbHLykJW6xYiR9JQEwdO20L607Wi7gmGtha65lZUHV0Dgd9rpkNKvhx07ENOgo3BSh2KBE0MXdBQBFEu8Z3WRYXwOTc01KzwArGZDtt4bnGgQMQnSQc8gxwQ3LWB2L6Ui+/H5//CpGvAqOnig3Pa8rRyRGhta0m/xtEsTGLX7KigdO+OV1J83yFE8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: An IPI backend is always required in an SMP configuration, but an SBI implementation is not. For example, SBI will be unavailable when the kernel runs in M mode. For this reason, consider IPI delivery of cache and TLB flushes to be the base case, and any other implementation (such as the SBI remote fence extension) to be an optimization. Generally, if IPIs can be delivered without firmware assistance, they are assumed to be faster than SBI calls due to the SBI context switch overhead. However, when SBI is used as the IPI backend, then the context switch cost must be paid anyway, and performing the cache/TLB flush directly in the SBI implementation is more efficient than injecting an interrupt to S-mode. This is the only existing scenario where riscv_ipi_set_virq_range() is called with use_for_rfence set to false. sbi_ipi_init() already checks riscv_ipi_have_virq_range(), so it only calls riscv_ipi_set_virq_range() when no other IPI device is available. This allows moving the static key and dropping the use_for_rfence parameter. This decouples the static key from the irqchip driver probe order. Furthermore, the static branch only makes sense when CONFIG_RISCV_SBI is enabled. Optherwise, IPIs must be used. Add a fallback definition of riscv_use_sbi_for_rfence() which handles this case and removes the need to check CONFIG_RISCV_SBI elsewhere, such as in cacheflush.c. Reviewed-by: Anup Patel Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- Changes in v6: - Clarify the commit message for patch 3 based on ML discussion Changes in v5: - Also switch to riscv_use_sbi_for_rfence() in asm/pgalloc.h Changes in v4: - New patch for v4 arch/riscv/include/asm/pgalloc.h | 7 ++++--- arch/riscv/include/asm/sbi.h | 4 ++++ arch/riscv/include/asm/smp.h | 15 ++------------- arch/riscv/kernel/sbi-ipi.c | 11 ++++++++++- arch/riscv/kernel/smp.c | 11 +---------- arch/riscv/mm/cacheflush.c | 5 ++--- arch/riscv/mm/tlbflush.c | 31 ++++++++++++++----------------- drivers/clocksource/timer-clint.c | 2 +- 8 files changed, 38 insertions(+), 48 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index b34587da8882..f52264304f77 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -8,6 +8,7 @@ #define _ASM_RISCV_PGALLOC_H #include +#include #include #ifdef CONFIG_MMU @@ -17,10 +18,10 @@ static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) { - if (riscv_use_ipi_for_rfence()) - tlb_remove_page_ptdesc(tlb, pt); - else + if (riscv_use_sbi_for_rfence()) tlb_remove_ptdesc(tlb, pt); + else + tlb_remove_page_ptdesc(tlb, pt); } static inline void pmd_populate_kernel(struct mm_struct *mm, diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index 6e68f8dff76b..ea84392ca9d7 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -375,8 +375,12 @@ unsigned long riscv_cached_marchid(unsigned int cpu_id); unsigned long riscv_cached_mimpid(unsigned int cpu_id); #if IS_ENABLED(CONFIG_SMP) && IS_ENABLED(CONFIG_RISCV_SBI) +DECLARE_STATIC_KEY_FALSE(riscv_sbi_for_rfence); +#define riscv_use_sbi_for_rfence() \ + static_branch_unlikely(&riscv_sbi_for_rfence) void sbi_ipi_init(void); #else +static inline bool riscv_use_sbi_for_rfence(void) { return false; } static inline void sbi_ipi_init(void) { } #endif diff --git a/arch/riscv/include/asm/smp.h b/arch/riscv/include/asm/smp.h index 0d555847cde6..7ac80e9f2288 100644 --- a/arch/riscv/include/asm/smp.h +++ b/arch/riscv/include/asm/smp.h @@ -49,12 +49,7 @@ void riscv_ipi_disable(void); bool riscv_ipi_have_virq_range(void); /* Set the IPI interrupt numbers for arch (called by irqchip drivers) */ -void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence); - -/* Check if we can use IPIs for remote FENCEs */ -DECLARE_STATIC_KEY_FALSE(riscv_ipi_for_rfence); -#define riscv_use_ipi_for_rfence() \ - static_branch_unlikely(&riscv_ipi_for_rfence) +void riscv_ipi_set_virq_range(int virq, int nr); /* Check other CPUs stop or not */ bool smp_crash_stop_failed(void); @@ -104,16 +99,10 @@ static inline bool riscv_ipi_have_virq_range(void) return false; } -static inline void riscv_ipi_set_virq_range(int virq, int nr, - bool use_for_rfence) +static inline void riscv_ipi_set_virq_range(int virq, int nr) { } -static inline bool riscv_use_ipi_for_rfence(void) -{ - return false; -} - #endif /* CONFIG_SMP */ #if defined(CONFIG_HOTPLUG_CPU) && (CONFIG_SMP) diff --git a/arch/riscv/kernel/sbi-ipi.c b/arch/riscv/kernel/sbi-ipi.c index a4559695ce62..1026e22955cc 100644 --- a/arch/riscv/kernel/sbi-ipi.c +++ b/arch/riscv/kernel/sbi-ipi.c @@ -13,6 +13,9 @@ #include #include +DEFINE_STATIC_KEY_FALSE(riscv_sbi_for_rfence); +EXPORT_SYMBOL_GPL(riscv_sbi_for_rfence); + static int sbi_ipi_virq; static void sbi_ipi_handle(struct irq_desc *desc) @@ -72,6 +75,12 @@ void __init sbi_ipi_init(void) "irqchip/sbi-ipi:starting", sbi_ipi_starting_cpu, NULL); - riscv_ipi_set_virq_range(virq, BITS_PER_BYTE, false); + riscv_ipi_set_virq_range(virq, BITS_PER_BYTE); pr_info("providing IPIs using SBI IPI extension\n"); + + /* + * Use the SBI remote fence extension to avoid + * the extra context switch needed to handle IPIs. + */ + static_branch_enable(&riscv_sbi_for_rfence); } diff --git a/arch/riscv/kernel/smp.c b/arch/riscv/kernel/smp.c index 45dd4035416e..8e6eb64459af 100644 --- a/arch/riscv/kernel/smp.c +++ b/arch/riscv/kernel/smp.c @@ -171,10 +171,7 @@ bool riscv_ipi_have_virq_range(void) return (ipi_virq_base) ? true : false; } -DEFINE_STATIC_KEY_FALSE(riscv_ipi_for_rfence); -EXPORT_SYMBOL_GPL(riscv_ipi_for_rfence); - -void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence) +void riscv_ipi_set_virq_range(int virq, int nr) { int i, err; @@ -197,12 +194,6 @@ void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence) /* Enabled IPIs for boot CPU immediately */ riscv_ipi_enable(); - - /* Update RFENCE static key */ - if (use_for_rfence) - static_branch_enable(&riscv_ipi_for_rfence); - else - static_branch_disable(&riscv_ipi_for_rfence); } static const char * const ipi_names[] = { diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index bc61ee5975e4..d76fc73e594b 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -21,7 +21,7 @@ void flush_icache_all(void) { local_flush_icache_all(); - if (IS_ENABLED(CONFIG_RISCV_SBI) && !riscv_use_ipi_for_rfence()) + if (riscv_use_sbi_for_rfence()) sbi_remote_fence_i(NULL); else on_each_cpu(ipi_remote_fence_i, NULL, 1); @@ -69,8 +69,7 @@ void flush_icache_mm(struct mm_struct *mm, bool local) * with flush_icache_deferred(). */ smp_mb(); - } else if (IS_ENABLED(CONFIG_RISCV_SBI) && - !riscv_use_ipi_for_rfence()) { + } else if (riscv_use_sbi_for_rfence()) { sbi_remote_fence_i(&others); } else { on_each_cpu_mask(&others, ipi_remote_fence_i, NULL, 1); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 893566e004b7..0435605b07d0 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -79,10 +79,10 @@ static void __ipi_flush_tlb_all(void *info) void flush_tlb_all(void) { - if (riscv_use_ipi_for_rfence()) - on_each_cpu(__ipi_flush_tlb_all, NULL, 1); - else + if (riscv_use_sbi_for_rfence()) sbi_remote_sfence_vma_asid(NULL, 0, FLUSH_TLB_MAX_SIZE, FLUSH_TLB_NO_ASID); + else + on_each_cpu(__ipi_flush_tlb_all, NULL, 1); } struct flush_tlb_range_data { @@ -103,7 +103,6 @@ static void __flush_tlb_range(struct cpumask *cmask, unsigned long asid, unsigned long start, unsigned long size, unsigned long stride) { - struct flush_tlb_range_data ftd; bool broadcast; if (cpumask_empty(cmask)) @@ -119,20 +118,18 @@ static void __flush_tlb_range(struct cpumask *cmask, unsigned long asid, broadcast = true; } - if (broadcast) { - if (riscv_use_ipi_for_rfence()) { - ftd.asid = asid; - ftd.start = start; - ftd.size = size; - ftd.stride = stride; - on_each_cpu_mask(cmask, - __ipi_flush_tlb_range_asid, - &ftd, 1); - } else - sbi_remote_sfence_vma_asid(cmask, - start, size, asid); - } else { + if (!broadcast) { local_flush_tlb_range_asid(start, size, stride, asid); + } else if (riscv_use_sbi_for_rfence()) { + sbi_remote_sfence_vma_asid(cmask, start, size, asid); + } else { + struct flush_tlb_range_data ftd; + + ftd.asid = asid; + ftd.start = start; + ftd.size = size; + ftd.stride = stride; + on_each_cpu_mask(cmask, __ipi_flush_tlb_range_asid, &ftd, 1); } if (cmask != cpu_online_mask) diff --git a/drivers/clocksource/timer-clint.c b/drivers/clocksource/timer-clint.c index 09fd292eb83d..0bdd9d7ec545 100644 --- a/drivers/clocksource/timer-clint.c +++ b/drivers/clocksource/timer-clint.c @@ -251,7 +251,7 @@ static int __init clint_timer_init_dt(struct device_node *np) } irq_set_chained_handler(clint_ipi_irq, clint_ipi_interrupt); - riscv_ipi_set_virq_range(rc, BITS_PER_BYTE, true); + riscv_ipi_set_virq_range(rc, BITS_PER_BYTE); clint_clear_ipi(); #endif