From patchwork Tue Jan 2 22:00:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13509529 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C029C47074 for ; Tue, 2 Jan 2024 22:01:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 969E06B02A6; Tue, 2 Jan 2024 17:01:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8CBA16B02A7; Tue, 2 Jan 2024 17:01:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 71C516B02A8; Tue, 2 Jan 2024 17:01:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 627256B02A6 for ; Tue, 2 Jan 2024 17:01:42 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 3ECE312070C for ; Tue, 2 Jan 2024 22:01:42 +0000 (UTC) X-FDA: 81635743644.28.0D5B31B Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf21.hostedemail.com (Postfix) with ESMTP id 526131C0006 for ; Tue, 2 Jan 2024 22:01:40 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=Cl8aLY14; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf21.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704232900; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pynuwdwrgTDQTs9EZPEkGjuJgTxII1R0VKB5FEKmh8A=; b=HZFku7Ehmb31Yhs2NYurvSfwzGN0JfmQPJhvgXHrGq3Wt8m9nXMFqURwbbBvy1R5O60Vo4 2bgpJada9bCHoAyEiBo5RM62Il2/PJdJzcyzM0lSlRmH7L1oBABa2hkl66nYCxRPMIQLm4 apwk6rj2TbE14sL1zpX98Y0pulny7Rg= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=Cl8aLY14; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf21.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704232900; a=rsa-sha256; cv=none; b=lS0hOLmoV6IhjaQmOKJMSItGCAoFhgEklft2ky3Y3lhoVKza9mbty6kOQuEE3jE0TBbqf4 feVCnpKY8LKkJpwjt5oWVbQHeFAlTTzNS6vEIUaMahaE1dTGXv58m0aQK7Ww+XBhjexs41 5pSJhkDQlDhXfq6Q9/M8NGRm8E6roCc= Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-28c075ad8e7so7041791a91.2 for ; Tue, 02 Jan 2024 14:01:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232899; x=1704837699; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pynuwdwrgTDQTs9EZPEkGjuJgTxII1R0VKB5FEKmh8A=; b=Cl8aLY14pqO+9kzBdsesRsy9o52xrxkQa1aNl9DjsJV6AnK82oRF38FtlWGrTKsGE9 Zb/or2LL9ME2Wx9qCkxC5ndatN9ClCAxpcAROaMX+Zd5anbhkGAIbVRi4TXWvHbczj5t Ms7MPgQx1R8wmvqW/gUDL4tyQ5A0Q7xmSmKMnZX2JwySqOVYEuwkBt7ROcv3YanoAA/S QqY8Lp1zcaZDd/sX0lUQ2jDZYz0er5rjHD3jcukEaAXKmTi6uqYEwtaIfNjWT9WZ1ak8 N/2Lnh2Alvr+MKCwIFjalm7LmDrzIMnQihfF3UUbd22xsd+NQQvHKvxvlORD41K0IwLS u+9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232899; x=1704837699; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pynuwdwrgTDQTs9EZPEkGjuJgTxII1R0VKB5FEKmh8A=; b=ouwr2SBwlgUx/47GBKRTGDWPOCZ/qmQ0KShnjoDVSGh9kpohCRWIB6GNvKDeYzeDDJ EbpIxrWh7O5yj017r8n1XGO7CUKhYRSig0ARkPljV94EA2Ee6IeLsnedUR5tV3aopx1J iagsiFO2ValpJAKlW2+elKSmYar5uennfuI+m3EEKkxrp4wENkXLGkCQvv9GI+w5T0ZV gMEMVc/OPXCpn7ZHZQgb48dueWUngSbP4xtUinZ4DM6+44Zd5RNcSl0cn+XHT4e5lT6d ZlYH1WZKp0+HtwEcAg53B8XZ62FTEuAeEasfWHFrCrTbc2NN3Clp3nSX6whL5N1uiXN0 wEpQ== X-Gm-Message-State: AOJu0Yx2f6Dgb8nxVGj+LE6HQV/pfHh191KoMYbJZCBoVQ9atr+7AKGp B3xzE3yGPYiH9SuYTLwPeFHN+4lDuQ/rKQ== X-Google-Smtp-Source: AGHT+IEYPya8PYA19IRLK0MnT0T3isXxWKz5hXZR0znVEsgxQFvYhbUCwOfyfHQYZwqQ+xa53Aj2UQ== X-Received: by 2002:a17:90b:3ec2:b0:28c:1b98:e6e7 with SMTP id rm2-20020a17090b3ec200b0028c1b98e6e7mr8644807pjb.47.1704232899208; Tue, 02 Jan 2024 14:01:39 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:38 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 02/12] riscv: Use IPIs for remote cache/TLB flushes by default Date: Tue, 2 Jan 2024 14:00:39 -0800 Message-ID: <20240102220134.3229156-3-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: crtfduawzirxidhmz8btnkdny3bsnkmo X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 526131C0006 X-HE-Tag: 1704232900-892435 X-HE-Meta: U2FsdGVkX1/jNmwrrbwekcUPXRExiVZGieCowR1skC/MZlyXi6EciKDys0lSSkDSGseSLdGnUb1m7lRZms7/bt10RQeeno3qttYe/1YZVz7zKU1fkrU+VU05S85evIjaCsZZ4W7zaChEyNe6KtEdNtd6LQn4ea0FGOFXUV0x2XjX8MjWGBSrHKKFIR9Ttw9f+hT1X3GmkxlFW+ri4AUPVBk/hvC5WBFFBh5jyY1TfcvhxiuuyF9Zv/pQnzqZB4GvZC6tnaRrHx7TGKEgfvT1S1D87SIES2pDG7iEnW3EY1dI06boU6frF6EMj5VpwShM0jJ7SVNJA02TxoYK7GXGyp6zzmLgM7pX1mtc8iRYrYrMi7rs4/f/Gbc53VUnp0XjuqwGB34OmyyBtbdy6YJ4qBSHSXqJF+OPSuX6pTWBkCtypXE908lwGfS1rY0cQxgRf7iXSh/sRhbojunVwPyiunK+bsPxWjN4ZSkxtP7xVQxoaGhLcHzTjRSV8HyPDWKNqkdATkWbqOEZEirTWFJYYwmyL4Lq9Yq1X5Jxzcjy3yCbEER4SiHnwqFAIdTFqIQZsNWb3KXtjPonC1ThwK/kss7xytpHVFriwEaDfe8Pl6hgYovFi/AtycL9qM46zq4AXSPYT+kgOoVa8PfKcBzMPVjLS45l07E+WQ9Ig3YrSglVHSM7m1AzKZH7MnzUh1h9oEePVASYSnx7yp8VDTKtZj5jqtITJjfmykAS4TxGcYTCW6xM083j2Zlw3TNJYzKxx06a/PPzxZwDJjtZsX5/f2kz2cH+iaRdoMLmNj6dnCuOimBx9N0a8s4bqFK5z2PsGBVc1I5lK0ILN9md0xiH+hLm5xopmvb6pp+xTsdxs5yRkc9DKkOM3zEAaZmYDYlS4yn+/c+MbS5M/7xxjmMug5Bx2bYg7Od3o1Nn82xqEwTAei6flba0w1ays8bhloHhHejUT2sVob1z0W6Rn5K 6lJUfylb yVLmQ2rc5i3aphElUspnJKJcMM5ty7P8l+Q64f9/jqw9aZBO8FgKUGGXVOJsOKGpzT6X/c4DM/h86GNiPZ/6DOtYbBLHwMVyGawPJr+B02E9igyRgbGm0Fj05i52yPegOrTzTfdZmCochav6bdyPmbW9tAfzHTYvcMm+5iP+jNSlDBaalcMQcK42ZYhJwVanm0kkcqlem7xE8omGeWnRO39IV2aFn+HHtXJ4yalY5mCoSstYj9qlB+a5gA6pcl63hm1rjIJfLPwtvtoU00ttTyd5i+TgYN7+2HEeMpn0OWuzcRODK+HrukZTZDktMDucW88jKqIkc/vXSsucu5w3zIaf7pVGjTQko+FTmIhOsx5sRB4xqqOC+74xnkLycRLl14tX9bDrh6NBWM3o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: An IPI backend is always required in an SMP configuration, but an SBI implementation is not. For example, SBI will be unavailable when the kernel runs in M mode. Generally, IPIs are assumed to be faster than SBI calls due to the SBI context switch overhead. However, when SBI is used as the IPI backend, then the context switch cost must be paid anyway, and performing the cache/TLB flush directly in the SBI implementation is more efficient than inserting an interrupt to the kernel. This is the only scenario where riscv_ipi_set_virq_range()'s use_for_rfence parameter is false. Thus, it makes sense for remote fences to use IPIs by default, and make the SBI remote fence extension the special case. sbi_ipi_init() already checks riscv_ipi_have_virq_range(), so it only calls riscv_ipi_set_virq_range() when no other IPI device is available. So we can move the static key and drop the use_for_rfence parameter. Furthermore, the static branch only makes sense when CONFIG_RISCV_SBI is enabled. Optherwise, IPIs must be used. Add a fallback definition of riscv_use_sbi_for_rfence() which handles this case and removes the need to check CONFIG_RISCV_SBI elsewhere, such as in cacheflush.c. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- Changes in v4: - New patch for v4 arch/riscv/include/asm/sbi.h | 4 ++++ arch/riscv/include/asm/smp.h | 15 ++------------- arch/riscv/kernel/sbi-ipi.c | 11 ++++++++++- arch/riscv/kernel/smp.c | 11 +---------- arch/riscv/mm/cacheflush.c | 5 ++--- arch/riscv/mm/tlbflush.c | 31 ++++++++++++++----------------- drivers/clocksource/timer-clint.c | 2 +- 7 files changed, 34 insertions(+), 45 deletions(-) diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index 0892f4421bc4..aeee0127df76 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -339,8 +339,12 @@ unsigned long riscv_cached_marchid(unsigned int cpu_id); unsigned long riscv_cached_mimpid(unsigned int cpu_id); #if IS_ENABLED(CONFIG_SMP) && IS_ENABLED(CONFIG_RISCV_SBI) +DECLARE_STATIC_KEY_FALSE(riscv_sbi_for_rfence); +#define riscv_use_sbi_for_rfence() \ + static_branch_unlikely(&riscv_sbi_for_rfence) void sbi_ipi_init(void); #else +static inline bool riscv_use_sbi_for_rfence(void) { return false; } static inline void sbi_ipi_init(void) { } #endif diff --git a/arch/riscv/include/asm/smp.h b/arch/riscv/include/asm/smp.h index 0d555847cde6..7ac80e9f2288 100644 --- a/arch/riscv/include/asm/smp.h +++ b/arch/riscv/include/asm/smp.h @@ -49,12 +49,7 @@ void riscv_ipi_disable(void); bool riscv_ipi_have_virq_range(void); /* Set the IPI interrupt numbers for arch (called by irqchip drivers) */ -void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence); - -/* Check if we can use IPIs for remote FENCEs */ -DECLARE_STATIC_KEY_FALSE(riscv_ipi_for_rfence); -#define riscv_use_ipi_for_rfence() \ - static_branch_unlikely(&riscv_ipi_for_rfence) +void riscv_ipi_set_virq_range(int virq, int nr); /* Check other CPUs stop or not */ bool smp_crash_stop_failed(void); @@ -104,16 +99,10 @@ static inline bool riscv_ipi_have_virq_range(void) return false; } -static inline void riscv_ipi_set_virq_range(int virq, int nr, - bool use_for_rfence) +static inline void riscv_ipi_set_virq_range(int virq, int nr) { } -static inline bool riscv_use_ipi_for_rfence(void) -{ - return false; -} - #endif /* CONFIG_SMP */ #if defined(CONFIG_HOTPLUG_CPU) && (CONFIG_SMP) diff --git a/arch/riscv/kernel/sbi-ipi.c b/arch/riscv/kernel/sbi-ipi.c index a4559695ce62..1026e22955cc 100644 --- a/arch/riscv/kernel/sbi-ipi.c +++ b/arch/riscv/kernel/sbi-ipi.c @@ -13,6 +13,9 @@ #include #include +DEFINE_STATIC_KEY_FALSE(riscv_sbi_for_rfence); +EXPORT_SYMBOL_GPL(riscv_sbi_for_rfence); + static int sbi_ipi_virq; static void sbi_ipi_handle(struct irq_desc *desc) @@ -72,6 +75,12 @@ void __init sbi_ipi_init(void) "irqchip/sbi-ipi:starting", sbi_ipi_starting_cpu, NULL); - riscv_ipi_set_virq_range(virq, BITS_PER_BYTE, false); + riscv_ipi_set_virq_range(virq, BITS_PER_BYTE); pr_info("providing IPIs using SBI IPI extension\n"); + + /* + * Use the SBI remote fence extension to avoid + * the extra context switch needed to handle IPIs. + */ + static_branch_enable(&riscv_sbi_for_rfence); } diff --git a/arch/riscv/kernel/smp.c b/arch/riscv/kernel/smp.c index 40420afbb1a0..1d06df04eb71 100644 --- a/arch/riscv/kernel/smp.c +++ b/arch/riscv/kernel/smp.c @@ -171,10 +171,7 @@ bool riscv_ipi_have_virq_range(void) return (ipi_virq_base) ? true : false; } -DEFINE_STATIC_KEY_FALSE(riscv_ipi_for_rfence); -EXPORT_SYMBOL_GPL(riscv_ipi_for_rfence); - -void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence) +void riscv_ipi_set_virq_range(int virq, int nr) { int i, err; @@ -197,12 +194,6 @@ void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence) /* Enabled IPIs for boot CPU immediately */ riscv_ipi_enable(); - - /* Update RFENCE static key */ - if (use_for_rfence) - static_branch_enable(&riscv_ipi_for_rfence); - else - static_branch_disable(&riscv_ipi_for_rfence); } static const char * const ipi_names[] = { diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index 55a34f2020a8..47c485bc7df0 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -21,7 +21,7 @@ void flush_icache_all(void) { local_flush_icache_all(); - if (IS_ENABLED(CONFIG_RISCV_SBI) && !riscv_use_ipi_for_rfence()) + if (riscv_use_sbi_for_rfence()) sbi_remote_fence_i(NULL); else on_each_cpu(ipi_remote_fence_i, NULL, 1); @@ -69,8 +69,7 @@ void flush_icache_mm(struct mm_struct *mm, bool local) * with flush_icache_deferred(). */ smp_mb(); - } else if (IS_ENABLED(CONFIG_RISCV_SBI) && - !riscv_use_ipi_for_rfence()) { + } else if (riscv_use_sbi_for_rfence()) { sbi_remote_fence_i(&others); } else { on_each_cpu_mask(&others, ipi_remote_fence_i, NULL, 1); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index e6659d7368b3..09b03bf71e6a 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -73,10 +73,10 @@ static void __ipi_flush_tlb_all(void *info) void flush_tlb_all(void) { - if (riscv_use_ipi_for_rfence()) - on_each_cpu(__ipi_flush_tlb_all, NULL, 1); - else + if (riscv_use_sbi_for_rfence()) sbi_remote_sfence_vma_asid(NULL, 0, FLUSH_TLB_MAX_SIZE, FLUSH_TLB_NO_ASID); + else + on_each_cpu(__ipi_flush_tlb_all, NULL, 1); } struct flush_tlb_range_data { @@ -96,7 +96,6 @@ static void __ipi_flush_tlb_range_asid(void *info) static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, unsigned long size, unsigned long stride) { - struct flush_tlb_range_data ftd; const struct cpumask *cmask; unsigned long asid = FLUSH_TLB_NO_ASID; bool broadcast; @@ -119,20 +118,18 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, broadcast = true; } - if (broadcast) { - if (riscv_use_ipi_for_rfence()) { - ftd.asid = asid; - ftd.start = start; - ftd.size = size; - ftd.stride = stride; - on_each_cpu_mask(cmask, - __ipi_flush_tlb_range_asid, - &ftd, 1); - } else - sbi_remote_sfence_vma_asid(cmask, - start, size, asid); - } else { + if (!broadcast) { local_flush_tlb_range_asid(start, size, stride, asid); + } else if (riscv_use_sbi_for_rfence()) { + sbi_remote_sfence_vma_asid(cmask, start, size, asid); + } else { + struct flush_tlb_range_data ftd; + + ftd.asid = asid; + ftd.start = start; + ftd.size = size; + ftd.stride = stride; + on_each_cpu_mask(cmask, __ipi_flush_tlb_range_asid, &ftd, 1); } if (mm) diff --git a/drivers/clocksource/timer-clint.c b/drivers/clocksource/timer-clint.c index 9a55e733ae99..7ccc16dd6a76 100644 --- a/drivers/clocksource/timer-clint.c +++ b/drivers/clocksource/timer-clint.c @@ -251,7 +251,7 @@ static int __init clint_timer_init_dt(struct device_node *np) } irq_set_chained_handler(clint_ipi_irq, clint_ipi_interrupt); - riscv_ipi_set_virq_range(rc, BITS_PER_BYTE, true); + riscv_ipi_set_virq_range(rc, BITS_PER_BYTE); clint_clear_ipi(); #endif