From patchwork Wed Nov 22 01:07:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13463796 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F3FD9C61D98 for ; Wed, 22 Nov 2023 01:08:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=L6cV1L383SFCkQk53le6a3x0uI3kmtC+kOSUi5kGbDw=; b=iP4TccCRdt4e9C THkxQmvHi+kNn4oh4+OtszIVR8GZCB4EY9A+J73s1nDt1CrPZofdU/58kdbfsTaJGcjIp51k9Sooa wGYA2EPfWHCXpSXwXLf4w1IgtKAFKR755iJOLSDwnjcHddrbbsE6rAd7nqClgRpvT6SBg55U8RVa9 pQq8YGh2cNFh/zhe68S0YZA5UqMMxO1cBnAKv6aKNr2KVdQU7avvMEjXvBGf7KN0iZ5QLJXiSN3KB L7UdaXvLF3VFLOTt5X8cQlhCi+l2hjHjB76EI95qZ5OAV/xCisHWjp2o14XSg82myy6xXHDVNxvcd jujz/CcEUxJctfx2O+Mw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r5bj5-000N98-0L; Wed, 22 Nov 2023 01:08:23 +0000 Received: from mail-pf1-x42c.google.com ([2607:f8b0:4864:20::42c]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r5bj1-000N7R-2b for linux-riscv@lists.infradead.org; Wed, 22 Nov 2023 01:08:21 +0000 Received: by mail-pf1-x42c.google.com with SMTP id d2e1a72fcca58-6cbc8199a2aso601571b3a.1 for ; Tue, 21 Nov 2023 17:08:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1700615298; x=1701220098; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/6X+mOMZoGJuf/KhFjronMx0sFEsDU8bO3mA0YdbfIg=; b=F+DpzMPnYsEmkjvgJRwfREzR7hNOm5jG2qqw91KphWTdl55n35xGt2QPnQS26di+d2 YCOF87Rl1HvxeTlEjnUfmv+6ekrTZyjrnzYDqRYxUcouQBOJISEIkDuAZhPwF8tp3YLh HbBg+i5lzWvi+Wf2NbJlclnqiG+dhkKbtzlXUHsDxxdni1fHHAYOUrqV7jW6ekzo2aFs GwwUKbAzvqqPLiqwP4HEcaGgTF+Bb6StZvxqtxBeOc99FX6anuKEPrBXGcuz/cAAUzcq Nb3RkKqDZrS8NPkCoPGPMDexqYhuxa4M9fLFgqMMauj/7JR0kMpFpZL7vl9/LB7C6VGZ ZRyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700615298; x=1701220098; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/6X+mOMZoGJuf/KhFjronMx0sFEsDU8bO3mA0YdbfIg=; b=mXCqnx+P1Y2/xZrmLS4lyEfGkXVJNE+ZSxRy7gjIJraP8Um4N2wADXbOf/YeP3dNLn PiiuZVi9MnaFTr0+v9igrphMQVRJ4SXIWtDV5NL/jQ+xq4s8TFSO5fioRCYaQLQPu2xw b2uprAP9voV/XoqqlQtHAl/C5UWsivv/nAs06G95rFr89dnvPL9SvptFT6hd8rox5u9D jnqe7rYWr2OFwneTq6pqJ3ReMHKPfO9StG6j6uoPdfwcT2nEdHuOUIElsgFkuUTk217D TpJV/cYlW83mB5R3QUq7NqcNdOGP9/y3JZ6eSTWR8qQ8/IXkL7WKJ7yiK/rgIYIPgGHx djTw== X-Gm-Message-State: AOJu0Yy3XmxAApFnav8rQmy0oxPc44G3vxEv5NAhIl63mQXSemW8a0/8 VxQtWZ4yCTIm0e2/1bqHUvp/Xw== X-Google-Smtp-Source: AGHT+IF0MPSUCqiTsnl1eQc+NOEkZdfylg4Cinzvui2qig7vBCcLaky4/GlXwa1fL7VF+JjfOGRcVA== X-Received: by 2002:a05:6a20:431a:b0:18a:da5a:3825 with SMTP id h26-20020a056a20431a00b0018ada5a3825mr948843pzk.34.1700615298368; Tue, 21 Nov 2023 17:08:18 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id bn5-20020a056a00324500b006be047268d5sm8713961pfb.174.2023.11.21.17.08.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Nov 2023 17:08:18 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v3 1/8] riscv: mm: Combine the SMP and UP TLB flush code Date: Tue, 21 Nov 2023 17:07:12 -0800 Message-ID: <20231122010815.3545294-2-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231122010815.3545294-1-samuel.holland@sifive.com> References: <20231122010815.3545294-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231121_170819_846084_0648494A X-CRM114-Status: GOOD ( 13.53 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In SMP configurations, all TLB flushing narrower than flush_tlb_all() goes through __flush_tlb_range(). Do the same in UP configurations. This allows UP configurations to take advantage of recent improvements to the code in tlbflush.c, such as support for huge pages and flushing multiple-page ranges. Signed-off-by: Samuel Holland --- (no changes since v2) Changes in v2: - Move the SMP/UP merge earlier in the series to avoid build issues - Make a copy of __flush_tlb_range() instead of adding ifdefs inside - local_flush_tlb_all() is the only function used on !MMU (smpboot.c) arch/riscv/include/asm/tlbflush.h | 33 +++++++------------------------ arch/riscv/mm/Makefile | 5 +---- arch/riscv/mm/tlbflush.c | 13 ++++++++++++ 3 files changed, 21 insertions(+), 30 deletions(-) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 8f3418c5f172..317a1811aa51 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -27,13 +27,12 @@ static inline void local_flush_tlb_page(unsigned long addr) { ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) : "memory")); } -#else /* CONFIG_MMU */ -#define local_flush_tlb_all() do { } while (0) -#define local_flush_tlb_page(addr) do { } while (0) -#endif /* CONFIG_MMU */ -#if defined(CONFIG_SMP) && defined(CONFIG_MMU) +#ifdef CONFIG_SMP void flush_tlb_all(void); +#else +#define flush_tlb_all() local_flush_tlb_all() +#endif void flush_tlb_mm(struct mm_struct *mm); void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int page_size); @@ -46,26 +45,8 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end); void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); #endif -#else /* CONFIG_SMP && CONFIG_MMU */ - -#define flush_tlb_all() local_flush_tlb_all() -#define flush_tlb_page(vma, addr) local_flush_tlb_page(addr) - -static inline void flush_tlb_range(struct vm_area_struct *vma, - unsigned long start, unsigned long end) -{ - local_flush_tlb_all(); -} - -/* Flush a range of kernel pages */ -static inline void flush_tlb_kernel_range(unsigned long start, - unsigned long end) -{ - local_flush_tlb_all(); -} - -#define flush_tlb_mm(mm) flush_tlb_all() -#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() -#endif /* !CONFIG_SMP || !CONFIG_MMU */ +#else /* CONFIG_MMU */ +#define local_flush_tlb_all() do { } while (0) +#endif /* CONFIG_MMU */ #endif /* _ASM_RISCV_TLBFLUSH_H */ diff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile index 3a4dfc8babcf..96e65c571ce8 100644 --- a/arch/riscv/mm/Makefile +++ b/arch/riscv/mm/Makefile @@ -13,15 +13,12 @@ endif KCOV_INSTRUMENT_init.o := n obj-y += init.o -obj-$(CONFIG_MMU) += extable.o fault.o pageattr.o +obj-$(CONFIG_MMU) += extable.o fault.o pageattr.o tlbflush.o obj-y += cacheflush.o obj-y += context.o obj-y += pgtable.o obj-y += pmem.o -ifeq ($(CONFIG_MMU),y) -obj-$(CONFIG_SMP) += tlbflush.o -endif obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_KASAN) += kasan_init.o diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index e6659d7368b3..22d7ed5abf8e 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -66,6 +66,7 @@ static inline void local_flush_tlb_range_asid(unsigned long start, local_flush_tlb_range_threshold_asid(start, size, stride, asid); } +#ifdef CONFIG_SMP static void __ipi_flush_tlb_all(void *info) { local_flush_tlb_all(); @@ -138,6 +139,18 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, if (mm) put_cpu(); } +#else +static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, + unsigned long size, unsigned long stride) +{ + unsigned long asid = FLUSH_TLB_NO_ASID; + + if (mm && static_branch_unlikely(&use_asid_allocator)) + asid = atomic_long_read(&mm->context.id) & asid_mask; + + local_flush_tlb_range_asid(start, size, stride, asid); +} +#endif void flush_tlb_mm(struct mm_struct *mm) { From patchwork Wed Nov 22 01:07:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13463798 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 352C2C61DA4 for ; Wed, 22 Nov 2023 01:08:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=sMpjcQMktjz1kdAD+2C72GiGKAWcZq/ShxjaUbbuqtk=; b=d0JotTS34nyy38 VlPl7yNhm5j1pAP2a5xrLLcV0IdG1jFYFIlNNmx344DwoFFyTHon7lz3JTPkAs44u97ndK7ha4cs+ 8AFOMeDeOjKZUnq+Wd6fF80xgtssUJl7UXt7+mFCPyqQxA38GrmqLBo5xWi9ovA5NT8UrFWLjefOp KIYjt0M0mxF8uNkT0do1wLLaGwYDVoDKVqxvqXYnxBMp3mq2qlr3P4BjmfposGn5vYb4vw108HAvR 4ZI0pZHGaK08q8/fBWD1PWBdSajoTsiHnNmS+5kmJgHHhadDK3C3jNJNkW3cVipZ9lRDRSt4p1nmj 9XxxeoCoZScMcZJVaMew==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r5bj7-000NAy-2e; Wed, 22 Nov 2023 01:08:25 +0000 Received: from mail-oo1-xc2c.google.com ([2607:f8b0:4864:20::c2c]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r5bj2-000N7s-2r for linux-riscv@lists.infradead.org; Wed, 22 Nov 2023 01:08:22 +0000 Received: by mail-oo1-xc2c.google.com with SMTP id 006d021491bc7-5875c300becso3768739eaf.0 for ; Tue, 21 Nov 2023 17:08:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1700615300; x=1701220100; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4E3lnavwhWkbsPt/8jollKuN1Wp7270MVQq6Nf32IZI=; b=j2gUQTCNQEG+nKL68cSMLaqRwZZmtVGHyeiTthPI4+a1JC6VQsQAooBjLqnu/GuYag gMSkv1N0UpEIdGWnEOkLnTFWxYoEwMgRHQBNkT27pgXXAASD62x5eaCgqtHiaKmpedTA 6E/plJONzUj7kcgeMNVK0a6XWI4EzSL6OdvpTrJYxm+WEUmO7jjWCLwtXmYGuarXOP+X zdq/c//GKEytA6hELtpnTUU5gfFfmoDqN9/rxw4y6DRLAWxsJHvreTwFxtWlDf+d+Ecd 5bWWryVAC4tqFdjE80+OASKILPSzs+zsXiC1wrNwgAnA7TdWLFgS0edA30hutRWf1zHt jvbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700615300; x=1701220100; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4E3lnavwhWkbsPt/8jollKuN1Wp7270MVQq6Nf32IZI=; b=lsinxY9ZIg/GWTdsFpVf5VaV5pZ1cw3WKexFz8CmTVbcWODemEckBieYc92/H3oO47 wZAyeQV63U8h07APg4a9ON4cfdpFbtlrdOTIYg3F+wqz5aMjWR8u4SeF7XUlfXdJ3Ll+ fEn1dtlUSP/GSq2n1NwzUf+waSN/wIiJlZOrsAZM2U+OCIqyLHL05vh9WzPEgb98wqmx cLl9tkuOVFDLMDRFnWPo9hCOxw0Dt/icqkCa+sdEnUxdjhIUtw6wHj7xJeLh1cxVmkhl HHWAvHK6jgLx2NoqBXmuNyomS/v2b4Y+QS7X9c37W/IAiibu7R7UTRHxt6gRfITzJTf5 hQkg== X-Gm-Message-State: AOJu0Yx7H2RsqQuhG8KUr7aYwDF92ed6z/lKuNTo4q/OHj4lWrNJ6BHJ 5wdHYp6bBwlNK4oJGaD8NHqTJA== X-Google-Smtp-Source: AGHT+IFhS1aXruOMSeKbF/kr742mxFr/4ERJTn5q1RSxs6qFscJwCL7KnIKBT8DwBL+aJBuJSXtucQ== X-Received: by 2002:a05:6871:e805:b0:1d6:8b7f:ce89 with SMTP id qd5-20020a056871e80500b001d68b7fce89mr1213468oac.51.1700615299719; Tue, 21 Nov 2023 17:08:19 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id bn5-20020a056a00324500b006be047268d5sm8713961pfb.174.2023.11.21.17.08.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Nov 2023 17:08:19 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v3 2/8] riscv: Apply SiFive CIP-1200 workaround to single-ASID sfence.vma Date: Tue, 21 Nov 2023 17:07:13 -0800 Message-ID: <20231122010815.3545294-3-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231122010815.3545294-1-samuel.holland@sifive.com> References: <20231122010815.3545294-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231121_170820_921214_14ED47C4 X-CRM114-Status: GOOD ( 13.93 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org commit 3f1e782998cd ("riscv: add ASID-based tlbflushing methods") added calls to the sfence.vma instruction with rs2 != x0. These single-ASID instruction variants are also affected by SiFive errata CIP-1200. Until now, the errata workaround was not needed for the single-ASID sfence.vma variants, because they were only used when the ASID allocator was enabled, and the affected SiFive platforms do not support multiple ASIDs. However, we are going to start using those sfence.vma variants regardless of ASID support, so now we need alternatives covering them. Signed-off-by: Samuel Holland --- (no changes since v2) Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/include/asm/errata_list.h | 12 +++++++++++- arch/riscv/include/asm/tlbflush.h | 19 ++++++++++++++++++- arch/riscv/mm/tlbflush.c | 23 ----------------------- 3 files changed, 29 insertions(+), 25 deletions(-) diff --git a/arch/riscv/include/asm/errata_list.h b/arch/riscv/include/asm/errata_list.h index 83ed25e43553..6781460ae564 100644 --- a/arch/riscv/include/asm/errata_list.h +++ b/arch/riscv/include/asm/errata_list.h @@ -44,11 +44,21 @@ ALTERNATIVE(__stringify(RISCV_PTR do_page_fault), \ CONFIG_ERRATA_SIFIVE_CIP_453) #else /* !__ASSEMBLY__ */ -#define ALT_FLUSH_TLB_PAGE(x) \ +#define ALT_SFENCE_VMA_ASID(asid) \ +asm(ALTERNATIVE("sfence.vma x0, %0", "sfence.vma", SIFIVE_VENDOR_ID, \ + ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ + : : "r" (asid) : "memory") + +#define ALT_SFENCE_VMA_ADDR(addr) \ asm(ALTERNATIVE("sfence.vma %0", "sfence.vma", SIFIVE_VENDOR_ID, \ ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ : : "r" (addr) : "memory") +#define ALT_SFENCE_VMA_ADDR_ASID(addr, asid) \ +asm(ALTERNATIVE("sfence.vma %0, %1", "sfence.vma", SIFIVE_VENDOR_ID, \ + ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ + : : "r" (addr), "r" (asid) : "memory") + /* * _val is marked as "will be overwritten", so need to set it to 0 * in the default case. diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 317a1811aa51..e529a643be17 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -22,10 +22,27 @@ static inline void local_flush_tlb_all(void) __asm__ __volatile__ ("sfence.vma" : : : "memory"); } +static inline void local_flush_tlb_all_asid(unsigned long asid) +{ + if (asid != FLUSH_TLB_NO_ASID) + ALT_SFENCE_VMA_ASID(asid); + else + local_flush_tlb_all(); +} + /* Flush one page from local TLB */ static inline void local_flush_tlb_page(unsigned long addr) { - ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) : "memory")); + ALT_SFENCE_VMA_ADDR(addr); +} + +static inline void local_flush_tlb_page_asid(unsigned long addr, + unsigned long asid) +{ + if (asid != FLUSH_TLB_NO_ASID) + ALT_SFENCE_VMA_ADDR_ASID(addr, asid); + else + local_flush_tlb_page(addr); } #ifdef CONFIG_SMP diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 22d7ed5abf8e..0feccb8932d2 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -7,29 +7,6 @@ #include #include -static inline void local_flush_tlb_all_asid(unsigned long asid) -{ - if (asid != FLUSH_TLB_NO_ASID) - __asm__ __volatile__ ("sfence.vma x0, %0" - : - : "r" (asid) - : "memory"); - else - local_flush_tlb_all(); -} - -static inline void local_flush_tlb_page_asid(unsigned long addr, - unsigned long asid) -{ - if (asid != FLUSH_TLB_NO_ASID) - __asm__ __volatile__ ("sfence.vma %0, %1" - : - : "r" (addr), "r" (asid) - : "memory"); - else - local_flush_tlb_page(addr); -} - /* * Flush entire TLB if number of entries to be flushed is greater * than the threshold below. From patchwork Wed Nov 22 01:07:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13463799 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DC0B8C61D97 for ; Wed, 22 Nov 2023 01:08:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=KQuKemOUoWG6ZhOtkkMqpseGOSldcmUZePuabmzjUIc=; b=Jp2ar/OwjBfPxg PzGLAyTTlHHt5dyZdt3IL2Azk+826XWXLJwdo3fvpIEqxi0R9JzrXpj24wzhIkzKYdBuFRRkOizOH x37uliPivCPYXXx+8L0iUclsgGNthbKLd9dpGHV97bwyrKcbx4FnOAgPE678C+UdUlBVSPf4I4EBQ EbLnLUXDWDIj3er/edrpM9/oSXbFIZh/BfJ8nGnppvqvWbI6sTkjvVIbTwyWxhFd5PuHllL3lBmoO P79/0dxe4s8n3Ol/Y8+0ADvKG1FQzZKJkMZUecALrnM9DUy5+qQN1HqZWKu0q3ErO9Ww2O1YcuYfi /n2dog3JUvwKe6pxOJYw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r5bj9-000NCU-1i; Wed, 22 Nov 2023 01:08:27 +0000 Received: from mail-pf1-x42a.google.com ([2607:f8b0:4864:20::42a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r5bj4-000N8K-2g for linux-riscv@lists.infradead.org; Wed, 22 Nov 2023 01:08:25 +0000 Received: by mail-pf1-x42a.google.com with SMTP id d2e1a72fcca58-6b20577ef7bso5508896b3a.3 for ; Tue, 21 Nov 2023 17:08:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1700615301; x=1701220101; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VvpOAl9p0Aiam4sGuchXKDW+3q/tf43lzqxId4c/moI=; b=ikbktjTG/B5LtFPgV3jkHwZm4h+bBL0FEuMOUh41hY91sBeLGk4BqbRzqIY9x0sknX 4zNdFgU8F3UJdTIUn6DfrUnTlAczqZw0npLa74nwKqBzR9Q9jbhJ1+eZ24iteHNUkUcc uMvuoFQc2LhBwHh62ty2CgivNKAB1hhbevCC/YvR6kz4TWH2Lv1g404IHk58lbmlz/El 1p6NYxUIy8Q+ESjCrGW3VnQhkN8TLnoKrYnQmViAaX49HlgpBuPupBHRl58jYDScyCAu FBNmpd/CPNE2HsZ5b96OLvtzg8o2B4Kh9vSoR6oDkkwiTy7a86L9cmMWeZayZszEFJ8I AVyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700615301; x=1701220101; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VvpOAl9p0Aiam4sGuchXKDW+3q/tf43lzqxId4c/moI=; b=fzgfz51wkqMeamGi9cBZ2GqHXpuVivAQvnknEVexd6eI+pYqf4p2U4mykqVwcSgvR4 mVAHc2/eVCED6zcEOkrHlXExtIn1eTeNtYqsuzHVY6h3tD7eU4iNHZoe22W/QJMJSKac 2NwhqhK82Opj8SaqChP48O0+ozdY/A+bSbTDQQH337ytFCxcLmYvbOmGjwxBZYOCURzE v1iE1XJP0SavxLxflCJa/ghmhyoorbs4LZCjxHvrG0/qMPOw2gnFjJO8Zf1hulzqvRK1 IPZq74Jlf8KVT3N0lGc6d+ecGVknDqp3LyDFyxWL0JzrDo4p+4LbCPRfvcE/GnyXxRrz w+Vw== X-Gm-Message-State: AOJu0YzQMv56098aoUZ7psHJs9QCBmlP+uqw9FVKGgC41lrMt/gNlj6q rbpxcV8452NKUvFY+ls7l25GyA== X-Google-Smtp-Source: AGHT+IF9d9fQIS/asTDDunhMGjFlgWHJ5Uekq1PrtWCEzSD057jeC559ik7d8wrAC6IaVqDr4/Nfcw== X-Received: by 2002:a05:6a00:3927:b0:6c6:aaa4:5d68 with SMTP id fh39-20020a056a00392700b006c6aaa45d68mr871292pfb.27.1700615301365; Tue, 21 Nov 2023 17:08:21 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id bn5-20020a056a00324500b006be047268d5sm8713961pfb.174.2023.11.21.17.08.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Nov 2023 17:08:20 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v3 3/8] riscv: Avoid TLB flush loops when affected by SiFive CIP-1200 Date: Tue, 21 Nov 2023 17:07:14 -0800 Message-ID: <20231122010815.3545294-4-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231122010815.3545294-1-samuel.holland@sifive.com> References: <20231122010815.3545294-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231121_170822_873810_49D6B487 X-CRM114-Status: GOOD ( 13.24 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Since implementations affected by SiFive errata CIP-1200 always use the global variant of the sfence.vma instruction, they only need to execute the instruction once. The range-based loop only hurts performance. Signed-off-by: Samuel Holland --- Changes in v3: - New patch for v3 arch/riscv/errata/sifive/errata.c | 3 +++ arch/riscv/include/asm/tlbflush.h | 2 ++ arch/riscv/mm/tlbflush.c | 2 +- 3 files changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/riscv/errata/sifive/errata.c b/arch/riscv/errata/sifive/errata.c index 3d9a32d791f7..00e011d78866 100644 --- a/arch/riscv/errata/sifive/errata.c +++ b/arch/riscv/errata/sifive/errata.c @@ -42,6 +42,9 @@ static bool errata_cip_1200_check_func(unsigned long arch_id, unsigned long imp return false; if ((impid & 0xffffff) > 0x200630 || impid == 0x1200626) return false; + + tlb_flush_all_threshold = 0; + return true; } diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index e529a643be17..3b393f765805 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -62,6 +62,8 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end); void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); #endif + +extern unsigned long tlb_flush_all_threshold; #else /* CONFIG_MMU */ #define local_flush_tlb_all() do { } while (0) #endif /* CONFIG_MMU */ diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 0feccb8932d2..27b3744b5673 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -11,7 +11,7 @@ * Flush entire TLB if number of entries to be flushed is greater * than the threshold below. */ -static unsigned long tlb_flush_all_threshold __read_mostly = 64; +unsigned long tlb_flush_all_threshold __read_mostly = 64; static void local_flush_tlb_range_threshold_asid(unsigned long start, unsigned long size, From patchwork Wed Nov 22 01:07:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13463800 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8D389C61D98 for ; Wed, 22 Nov 2023 01:08:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ElsvYYAffHA994l7YOd/Ci8D03rLSidKN4QkGybUrLc=; b=SbMMxdB1/PUrz+ K4ONqqdkXqGPYsWP4HGVJ62XAwaKpRmeluLBab27M6PKAsJii5TkFXISBUFBDI7TqB16chPplxaQG ENlVnrqPqXCUiMQjawqhvCWF9lMCVuTI8PgK1b2zDqPqsVytMjn8Mi5MOWNDuqjC0QTcTroGbL+/F 9cZAdCVwUZP477lf1808FJ9kaO8kHTxdN2pwutE2PBwIyyEX38awJoQJd8FclpPeY5UIgfhVjUc7g 9VUxJTFeU7HmX4NPq/1s5rXEVMm+uRDpuDzrwrCj2XS935/roldixuoGZFHZKI5QV+c6pIF89Dlpr 09P0vmsEGl0hqJ3xnLZw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r5bjA-000NCv-0c; Wed, 22 Nov 2023 01:08:28 +0000 Received: from mail-pg1-x529.google.com ([2607:f8b0:4864:20::529]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r5bj5-000N94-36 for linux-riscv@lists.infradead.org; Wed, 22 Nov 2023 01:08:25 +0000 Received: by mail-pg1-x529.google.com with SMTP id 41be03b00d2f7-5bdb0be3591so4763528a12.2 for ; Tue, 21 Nov 2023 17:08:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1700615302; x=1701220102; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qsEPuyPCg+yq6nUonOiNDjvoYO+XNuK3/GzQIQmVNGs=; b=fbvKluR0YmdoTHESPrEEhRNqzS6TNSi1yviu/GDlf6aQhDWtb6VP/g9pJHyh+w2dbO vmDF7dM+bJICjDE3ns3FGuHtWZdXiyjnm0U//8IfKJJkitpwHnOGNJ9vI5xZqEE29OTY JVGBcgFj3a6iPA6I5VMCQPrIubXccGUygdwgrJwRRzgvb7Dh871GitrxAT3RZcPO7p67 nf6svjUy8Hg5+0gn6o2M9SCxsNGkfYQ+Zl8iY7Jidh3gtqARrRnG2zDHDUJAJ1s+Zq2x 0KJX3uPttioz+Q2HwtotVbm+ghRcXFHu3OpQEvEo0GZ/W2hx+iZWnzlIJwl6m3ldSn6u EzRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700615302; x=1701220102; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qsEPuyPCg+yq6nUonOiNDjvoYO+XNuK3/GzQIQmVNGs=; b=EAlSVamO9XjU1YR+nVg7HPNcHKsn1GEbpM8nG4WdT/BGvdmFl1ljR/mwivn6Q35wkl kkSKVdPBqgIRe4XeNmADT23tIjzT1wi0VFo2Zj2WFSpkqA7tieUpI8XNM4APLyrKzApS rYp3QID2GKLSl0Ha+LMbalGmi425BFIy26+DzgGh/q2uiAnAEUUPBeLR0dQRUajWWBXg TPIm6xsJg/VxujVkEbTwPfhjoh++hpJgLsZfBrntFdLR6xdhwRmZ5hWQg/y6e3OmrV3j CIyeUnvbPX5rJoRKhoWog9xDvJKHK4OG2gpp1mDAZY7UlJDf2zgl7gmoOmsEqcFhJWi2 91RA== X-Gm-Message-State: AOJu0Ywu6LbDIyT3a1V/yKydPEdzN2OS2lpRVH6udEGAGPnle4Rdc9TQ innSje5C0WP6PoA5QZEihW44SA== X-Google-Smtp-Source: AGHT+IF7ribPLGqSLRhKn+k/5RoFarRyyKm41YCW2vHm9TG9JHEkBRqiRle7Sp7LVXO54oVTBXbYlQ== X-Received: by 2002:a05:6a20:c6cd:b0:188:75c:e69a with SMTP id gw13-20020a056a20c6cd00b00188075ce69amr779809pzb.4.1700615302563; Tue, 21 Nov 2023 17:08:22 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id bn5-20020a056a00324500b006be047268d5sm8713961pfb.174.2023.11.21.17.08.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Nov 2023 17:08:22 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v3 4/8] riscv: mm: Introduce cntx2asid/cntx2version helper macros Date: Tue, 21 Nov 2023 17:07:15 -0800 Message-ID: <20231122010815.3545294-5-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231122010815.3545294-1-samuel.holland@sifive.com> References: <20231122010815.3545294-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231121_170823_998514_C57CE672 X-CRM114-Status: GOOD ( 15.32 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org When using the ASID allocator, the MM context ID contains two values: the ASID in the lower bits, and the allocator version number in the remaining bits. Use macros to make this separation more obvious. Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/include/asm/mmu.h | 3 +++ arch/riscv/mm/context.c | 12 ++++++------ arch/riscv/mm/tlbflush.c | 4 ++-- 3 files changed, 11 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index 355504b37f8e..a550fbf770be 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -26,6 +26,9 @@ typedef struct { #endif } mm_context_t; +#define cntx2asid(cntx) ((cntx) & asid_mask) +#define cntx2version(cntx) ((cntx) & ~asid_mask) + void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, pgprot_t prot); #endif /* __ASSEMBLY__ */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 217fd4de6134..43d005f63253 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -81,7 +81,7 @@ static void __flush_context(void) if (cntx == 0) cntx = per_cpu(reserved_context, i); - __set_bit(cntx & asid_mask, context_asid_map); + __set_bit(cntx2asid(cntx), context_asid_map); per_cpu(reserved_context, i) = cntx; } @@ -102,7 +102,7 @@ static unsigned long __new_context(struct mm_struct *mm) lockdep_assert_held(&context_lock); if (cntx != 0) { - unsigned long newcntx = ver | (cntx & asid_mask); + unsigned long newcntx = ver | cntx2asid(cntx); /* * If our current CONTEXT was active during a rollover, we @@ -115,7 +115,7 @@ static unsigned long __new_context(struct mm_struct *mm) * We had a valid CONTEXT in a previous life, so try to * re-use it if possible. */ - if (!__test_and_set_bit(cntx & asid_mask, context_asid_map)) + if (!__test_and_set_bit(cntx2asid(cntx), context_asid_map)) return newcntx; } @@ -168,7 +168,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) */ old_active_cntx = atomic_long_read(&per_cpu(active_context, cpu)); if (old_active_cntx && - ((cntx & ~asid_mask) == atomic_long_read(¤t_version)) && + (cntx2version(cntx) == atomic_long_read(¤t_version)) && atomic_long_cmpxchg_relaxed(&per_cpu(active_context, cpu), old_active_cntx, cntx)) goto switch_mm_fast; @@ -177,7 +177,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) /* Check that our ASID belongs to the current_version. */ cntx = atomic_long_read(&mm->context.id); - if ((cntx & ~asid_mask) != atomic_long_read(¤t_version)) { + if (cntx2version(cntx) != atomic_long_read(¤t_version)) { cntx = __new_context(mm); atomic_long_set(&mm->context.id, cntx); } @@ -191,7 +191,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) switch_mm_fast: csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | - ((cntx & asid_mask) << SATP_ASID_SHIFT) | + (cntx2asid(cntx) << SATP_ASID_SHIFT) | satp_mode); if (need_flush_tlb) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 27b3744b5673..23409d70440f 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -91,7 +91,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, broadcast = cpumask_any_but(cmask, cpuid) < nr_cpu_ids; if (static_branch_unlikely(&use_asid_allocator)) - asid = atomic_long_read(&mm->context.id) & asid_mask; + asid = cntx2asid(atomic_long_read(&mm->context.id)); } else { cmask = cpu_online_mask; broadcast = true; @@ -123,7 +123,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, unsigned long asid = FLUSH_TLB_NO_ASID; if (mm && static_branch_unlikely(&use_asid_allocator)) - asid = atomic_long_read(&mm->context.id) & asid_mask; + asid = cntx2asid(atomic_long_read(&mm->context.id)); local_flush_tlb_range_asid(start, size, stride, asid); } From patchwork Wed Nov 22 01:07:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13463802 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CB862C61D99 for ; Wed, 22 Nov 2023 01:08:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=EryK/siS6cp26WR4PINEBb7uP5yP/3IYm3UElUq/3bo=; b=h1Cp4own9vN8ki c2K0M4WxuxUGG/gW4PdZvCCrpnFKBKJzbeNqIE9Du9dgfilbiLKo/0CQo5Pwpko0JghHjxswjZx/k bVQoZHFt6YADkaGJQG4WDJkOeMBhOt6/UMMvYrW1zpEYEk8VsT6TOPFhXdXV5CnSltDaBhMOPRHDv lcI8HatUfS4BnsUGE9eoHcRD5UMMH03Ur6MXucYez/zjYct6Wzv+ShA2WxwMoCaZBgMUQtuMEhhab /9JRwaeEQT3rdkk7aXOQNw+f4TISpSeL0BNf1VcfYuUs3/E+d3BtDshKDHEzpLuAOid7VLfAMLOgY DW0l+rcuOySZEs6yOqLQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r5bjD-000NFB-0M; Wed, 22 Nov 2023 01:08:31 +0000 Received: from mail-il1-x136.google.com ([2607:f8b0:4864:20::136]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r5bj7-000NAL-1P for linux-riscv@lists.infradead.org; Wed, 22 Nov 2023 01:08:26 +0000 Received: by mail-il1-x136.google.com with SMTP id e9e14a558f8ab-357ccaf982eso19996625ab.0 for ; Tue, 21 Nov 2023 17:08:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1700615304; x=1701220104; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=b39ZSDVnn3/TpDDYcLNRgBYhWVyekPNc8OuxKxdgheo=; b=gZ9oWzmwIs1krlSbt8wD13G2DgXdoqDVl02guB65Dt0/GgbBZw60/jZeaU2dShF4gG RZbVHEs/c+NcmRigyjfu6y/7axnmR0BYlu+tWuTvLWXU+aSK2uU7/VANI88FQQnswfiO PI81UYOW4ItgtbgtgM4oGj9ln5+IBq6tA1ppOdwkjsKq3ZL990VU3yuuBkitMKZS8HoX 1VWKPAQ63LOoDopv3huhkr9RJTqXkBWvzRRjfXjVMeWXsAcXnzNsb6m6CHTLhRkK5joA X6bNKS+VFbGgkB3X1GQWVm4jVsyvO8aKSbNvMoLsAiWClND5RyGk+TB6w01ws7f9W5KS gw+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700615304; x=1701220104; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=b39ZSDVnn3/TpDDYcLNRgBYhWVyekPNc8OuxKxdgheo=; b=ImrdNOh1tiiGbJtzE/c0m2sAoPq3nrZram59ej/qrekQuwW52tsC5OBRqPO0jOGwhX WFczNoYLhGlU+0D8uRANIkrViWgEuZwDZLDCuJhUxVa8Fw9hia/eSSLPgV6yY0PqdTwS 1D94QYLI2HQdb8EEm97aWOAeaYegTOq5c2UGo8NxMXA9p5VJzuhLVkl4q23Q0HvSSXGJ L2vSLcMt1t4/uohGRq5DRP6eK2+NvM+gxLyxbw/NQIk421tVU2qUnTZXbgoOVMV8qLJO tXp0uzmEaWWW+BStWoJUsOgiPaBwOWDBi9DfafXFv8q6UVATKb/ToAuBkJje13sMD/if ov3g== X-Gm-Message-State: AOJu0YxlsbWc3dv9b2bbNSgdmcA9R05ZgCGrGPqzXbZw4xNEZskgs02I P2xJqzX97EkvTGIXmKTQtgEJ2A== X-Google-Smtp-Source: AGHT+IGi2JpVuCgk8T+HgcOfCC/VHmmgf8iSnDwMNXS0Bc2add3cL906GsPfWT4iX9QyNn2xttBJig== X-Received: by 2002:a05:6e02:b48:b0:35b:2b49:d816 with SMTP id f8-20020a056e020b4800b0035b2b49d816mr721861ilu.13.1700615303918; Tue, 21 Nov 2023 17:08:23 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id bn5-20020a056a00324500b006be047268d5sm8713961pfb.174.2023.11.21.17.08.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Nov 2023 17:08:23 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v3 5/8] riscv: mm: Use a fixed layout for the MM context ID Date: Tue, 21 Nov 2023 17:07:16 -0800 Message-ID: <20231122010815.3545294-6-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231122010815.3545294-1-samuel.holland@sifive.com> References: <20231122010815.3545294-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231121_170825_474865_AFDD524F X-CRM114-Status: GOOD ( 12.19 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Currently, the size of the ASID field in the MM context ID dynamically depends on the number of hardware-supported ASID bits. This requires reading a global variable to extract either field from the context ID. Instead, allocate the maximum possible number of bits to the ASID field, so the layout of the context ID is known at compile-time. Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/include/asm/mmu.h | 4 ++-- arch/riscv/include/asm/tlbflush.h | 2 -- arch/riscv/mm/context.c | 6 ++---- 3 files changed, 4 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index a550fbf770be..dc0273f7905f 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -26,8 +26,8 @@ typedef struct { #endif } mm_context_t; -#define cntx2asid(cntx) ((cntx) & asid_mask) -#define cntx2version(cntx) ((cntx) & ~asid_mask) +#define cntx2asid(cntx) ((cntx) & SATP_ASID_MASK) +#define cntx2version(cntx) ((cntx) & ~SATP_ASID_MASK) void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, pgprot_t prot); diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 3b393f765805..4448d907f2c9 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -15,8 +15,6 @@ #define FLUSH_TLB_NO_ASID ((unsigned long)-1) #ifdef CONFIG_MMU -extern unsigned long asid_mask; - static inline void local_flush_tlb_all(void) { __asm__ __volatile__ ("sfence.vma" : : : "memory"); diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 43d005f63253..b5170ac1b742 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -22,7 +22,6 @@ DEFINE_STATIC_KEY_FALSE(use_asid_allocator); static unsigned long asid_bits; static unsigned long num_asids; -unsigned long asid_mask; static atomic_long_t current_version; @@ -128,7 +127,7 @@ static unsigned long __new_context(struct mm_struct *mm) goto set_asid; /* We're out of ASIDs, so increment current_version */ - ver = atomic_long_add_return_relaxed(num_asids, ¤t_version); + ver = atomic_long_add_return_relaxed(BIT(SATP_ASID_BITS), ¤t_version); /* Flush everything */ __flush_context(); @@ -247,7 +246,6 @@ static int __init asids_init(void) /* Pre-compute ASID details */ if (asid_bits) { num_asids = 1 << asid_bits; - asid_mask = num_asids - 1; } /* @@ -255,7 +253,7 @@ static int __init asids_init(void) * at-least twice more than CPUs */ if (num_asids > (2 * num_possible_cpus())) { - atomic_long_set(¤t_version, num_asids); + atomic_long_set(¤t_version, BIT(SATP_ASID_BITS)); context_asid_map = bitmap_zalloc(num_asids, GFP_KERNEL); if (!context_asid_map) From patchwork Wed Nov 22 01:07:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13463801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7E7EDC61D85 for ; Wed, 22 Nov 2023 01:08:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=n4FUUJWMeQ50MhgT9WfVWFZlC3MyJ9tf2sNwhOkJThc=; b=UuLgwkHYDU7arp aTOvVw+trzX9fSp6DT8zwe3rs9bYc9vQDNjfSuKmTY9wpHYQEAdglPc/MgTZ1W4kayjTuvDe3ltSd +QEYuUqQ23KX91Ox2p5BBCM+oY6U8eoTf4IOnSRO8QoTVxvSszYj1JVc3k7EVb4nqvwCZmkYKMdgt qKjS43UhKQtqt1BCBFvos28vB1c7DbIHhQAznNUT14wjF/PbyAfkD56dSTwTIP/tzDGXWMF5v5C11 RBSkmeXMw1E6zijI+O/s1wySyS7/cbFpR70WvCe0FmcmwClYF1f+qXxwNqLdXyXi0+F8mOcSrU1zk wSAYcLM179OL9KRej7vQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r5bjD-000NFk-2F; Wed, 22 Nov 2023 01:08:31 +0000 Received: from mail-oa1-x2d.google.com ([2001:4860:4864:20::2d]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r5bj8-000NAd-0q for linux-riscv@lists.infradead.org; Wed, 22 Nov 2023 01:08:27 +0000 Received: by mail-oa1-x2d.google.com with SMTP id 586e51a60fabf-1f937a7b8aaso1562404fac.0 for ; Tue, 21 Nov 2023 17:08:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1700615305; x=1701220105; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O8uQuAx5HbeGrqX4Ip/qQY7nv6lrxw4qacCJFbpKbko=; b=doxzn3P39GoLzIANEE7xyZlYT0g/EbxUXomzhZ1Lh5g02/awPYOsg3LJW6PsOi8hvo WWVAXSpxfdU8WeZM7Ew7I8m7NWpBN7FEQb7cR7CVS3c/pJT9KO1xMvzvZSvbetk79C48 uFWBRy4jHtEJ9ZL0KV64RXB2Edj+p/lBqZeT7D11lgyAzdhj4eDjYIrzrr32GD8lfm0U 4jV6W06d/a+LTAGbZjgd8Go9iSpMEBMqu9YDwMca1BjG4idpg/5oNjaLScm0J/hHNo8z jO4FVbXgc3OsmQLMZwc45/Oz2W8CDvz7yDsn+SO/kBjQwbVVoiSRYWERd9o56R4B6nVP qdQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700615305; x=1701220105; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O8uQuAx5HbeGrqX4Ip/qQY7nv6lrxw4qacCJFbpKbko=; b=fLbcX6HkiR/8yoYPg/0NDe/oZiRRIkg2+jPld5Uza9aUm43uVNV2o4jfnnakskTWYI uzJ3Xc0RoJ8WpiAwgG4xsKqXxTAJJneF4o/zEg1/JSCFX4kEWi4IaUoRLi6F/CVUvEW6 H0g38Jeuj5hYkyCprqFxuSz13wA5begTnsf7V2tMdT4MzZxXA4y1vAO7wM6RyGnY6/ZB HSNyeIe+x5bJoGGNkf/zywDlqrYx/RoC51wFWcUuRqyLONl7mQO1+q/3aaNWNzVTP+t4 WdxgEsoBp1TSjmmKyPGwe7sDx5oB786BhUIorbX9hgZ/hQqBCdDNCAho34FAOnY+dimc tn7A== X-Gm-Message-State: AOJu0Yx7icA4WZx6LFxUKtAw8QrFS4JEfe2iLAzY65U8BSc+Tgnpe5ml AbpOKKtKKhX9Pv0Aoxmy/I0V0A== X-Google-Smtp-Source: AGHT+IHzY35P/bmGNlxsboL33Yxa5yN9S+7CxJ9PS0wbWCSnN9yRamcO2jnn4/Cd8sidyrAKu3XrJw== X-Received: by 2002:a05:6870:9b0c:b0:1f5:b5ca:435e with SMTP id hq12-20020a0568709b0c00b001f5b5ca435emr1104182oab.52.1700615304951; Tue, 21 Nov 2023 17:08:24 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id bn5-20020a056a00324500b006be047268d5sm8713961pfb.174.2023.11.21.17.08.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Nov 2023 17:08:24 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v3 6/8] riscv: mm: Make asid_bits a local variable Date: Tue, 21 Nov 2023 17:07:17 -0800 Message-ID: <20231122010815.3545294-7-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231122010815.3545294-1-samuel.holland@sifive.com> References: <20231122010815.3545294-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231121_170826_294700_D08963E1 X-CRM114-Status: UNSURE ( 8.79 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This variable is only used inside asids_init(). Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/mm/context.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index b5170ac1b742..43a8bc2d5af4 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -20,7 +20,6 @@ DEFINE_STATIC_KEY_FALSE(use_asid_allocator); -static unsigned long asid_bits; static unsigned long num_asids; static atomic_long_t current_version; @@ -226,7 +225,7 @@ static inline void set_mm(struct mm_struct *prev, static int __init asids_init(void) { - unsigned long old; + unsigned long asid_bits, old; /* Figure-out number of ASID bits in HW */ old = csr_read(CSR_SATP); From patchwork Wed Nov 22 01:07:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13463803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D669DC61D9B for ; Wed, 22 Nov 2023 01:08:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Li/VAFYbscj6f9+EV6EJIvnlY67sDrtoWicRX6372nQ=; b=4+ku44dz9B0Fs+ Ih6dX+TGlnWuOjUcfLHl2CyFoetgXkMC9cUJyEZQWtco7XMafvUTU3+iMyge8/iB69zMsM8LbimRJ me4ZkGpgrgOqBHlLZmpIp2bDuqnIu1lk4XI9+pY/gvu6fm3y1RYQFZfXueBTy4iRR5HjWxohekMKp OZDqg2GxbWpQhatQY4Mrs1P3sYDVGx8cLKKPrKkjx572uBzvcMJNymzPceXBqoelH9GodZ9IXnA4u akORMLAys58VPW6RMDwdeSmvJDCpyEQBq9QC1DUYYGP7TpQeSscN/QN9W0WC2gZA7fSzWLQVlkqTc xAX2H4Pq2+KX+bY4Yg0Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r5bjE-000NGP-1M; Wed, 22 Nov 2023 01:08:32 +0000 Received: from mail-pf1-x42c.google.com ([2607:f8b0:4864:20::42c]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r5bj8-000NBU-1b for linux-riscv@lists.infradead.org; Wed, 22 Nov 2023 01:08:27 +0000 Received: by mail-pf1-x42c.google.com with SMTP id d2e1a72fcca58-6b2018a11efso6318582b3a.0 for ; Tue, 21 Nov 2023 17:08:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1700615306; x=1701220106; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MROdnZxFJAn7brM6ye/jIKPJcvVCbRqlgsjUSxtP0PY=; b=Ie3IVsVQbnTEWMjEUZhnQHztatCc7hKq6ZfVplNfGDEIBDL+DOkycGJ40jaRmZgges neqN6kEDZ2HBysRy1/aUrpYlvwkDOrAif67Cc3XZKEEo8O5mM5nJop0RSlQ5xRS+t6/a 6gbxqflwIikkpd3OO4SOtGgK8Nr9w8o4mNMB8JHxn/sV8dyi2wgscErofSoazSPKYiJf 3YnDYUkrSmlNPv4mSh4Ves9ODjM5Zg7jzVqFaCCU1i3Rr6c2WIcT67wDOp/zpcHgQiyk M+Xt7q/RokyD9Mr6E1fdq2aqoradbwSf9N8SBLdIWbBCA5eiOCvyeqSK5f8aZ5hJrst7 Zp1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700615306; x=1701220106; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MROdnZxFJAn7brM6ye/jIKPJcvVCbRqlgsjUSxtP0PY=; b=d3Y6Oc39KdDEQuovu4khE0TBCzvsBO2jsv0Tu0GDR7Iy1sS/4v3Qmp1bjFhnE6+wr8 jtB6jPI7tqsZUh8h8r8WV/lytqPn9YQ2HekLUM/Mf9AXrE4TvoZF8o9OQ1Mgit8twU0r KQYwf9yW/mjEu9PGXSQ6ghM4qk3CvV3PiyTtmE2z4wGymHxP9JYzfgFugDdXlR9mikv6 CaJkdOL1h7jYXcozO2TAnUCfMovUdkMx8ETcRIXmKJa78f6+i47N7BViQGlJ61Zovvaq k/sL7lZSzwvD7IxWI3Qg5Mgf0F1Ycr4jKZclL78pseyRB21NTb9YG9XZAX38WAi34ICG HJsQ== X-Gm-Message-State: AOJu0Yxia9PkuCiNl+1gF48Nbb92WhwJVbejJ7+/dp6PwcVvv+gsK2Wq 1iubdSw+/rqvuwxc4RQpojY0Sw== X-Google-Smtp-Source: AGHT+IFVZu+XEimHfZXOsjvtj11LrkIBAYyBlYwN/UfpJ7aqLGYZgxb4tCN+mMPanboANiNPoJtK3w== X-Received: by 2002:a05:6a00:b54:b0:6cb:a1fe:5217 with SMTP id p20-20020a056a000b5400b006cba1fe5217mr1041980pfo.16.1700615306009; Tue, 21 Nov 2023 17:08:26 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id bn5-20020a056a00324500b006be047268d5sm8713961pfb.174.2023.11.21.17.08.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Nov 2023 17:08:25 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v3 7/8] riscv: mm: Preserve global TLB entries when switching contexts Date: Tue, 21 Nov 2023 17:07:18 -0800 Message-ID: <20231122010815.3545294-8-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231122010815.3545294-1-samuel.holland@sifive.com> References: <20231122010815.3545294-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231121_170826_533753_DC3B7818 X-CRM114-Status: GOOD ( 14.45 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org If the CPU does not support multiple ASIDs, all MM contexts use ASID 0. In this case, it is still beneficial to flush the TLB by ASID, as the single-ASID variant of the sfence.vma instruction preserves TLB entries for global (kernel) pages. This optimization is recommended by the RISC-V privileged specification: If the implementation does not provide ASIDs, or software chooses to always use ASID 0, then after every satp write, software should execute SFENCE.VMA with rs1=x0. In the common case that no global translations have been modified, rs2 should be set to a register other than x0 but which contains the value zero, so that global translations are not flushed. It is not possible to apply this optimization when using the ASID allocator, because that code must flush the TLB for all ASIDs at once when incrementing the version number. Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/mm/context.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 43a8bc2d5af4..3ca9b653df7d 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -200,7 +200,7 @@ static void set_mm_noasid(struct mm_struct *mm) { /* Switch the page table and blindly nuke entire local TLB */ csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | satp_mode); - local_flush_tlb_all(); + local_flush_tlb_all_asid(0); } static inline void set_mm(struct mm_struct *prev, From patchwork Wed Nov 22 01:07:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13463804 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3949EC61D98 for ; Wed, 22 Nov 2023 01:08:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=1//ZmfbzvQWrgeHwJSjY0RPFLAEhUje1i5lKuU+SekI=; b=B7cNjRhMrydA9K cf5ZyEOJiEJ1CP05dljoFeFDoXePwcqbtF8KCWWrtCbWj+K/YIcp3gwhuLjM8LpEwi7qO+2ktw82I 5t1ODIdxv9+MYBEISV7J8AGJyFHOOYMjOOsQEuspe9KsC+PE3tm/dTVRW7i7zaulYBxPUBICEkPbr ffMo36PvgMM4MoZnigGgzRaU4JKOZ+6qlxTagkntsAL2j5cgcFBle+qMRjQmZ4AjVi0RLP2WqJvTo oHG4yiEFq1QvDZSw/5NZzNP+8FYafR9EUJwVdCpGYX3Dk0geYKO1XS89TKIOn5kzRGkHeFSbCSQj1 pPHUTWf3oeT7++bri6kw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r5bjF-000NGt-0E; Wed, 22 Nov 2023 01:08:33 +0000 Received: from mail-oi1-x229.google.com ([2607:f8b0:4864:20::229]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r5bjA-000NCS-0T for linux-riscv@lists.infradead.org; Wed, 22 Nov 2023 01:08:29 +0000 Received: by mail-oi1-x229.google.com with SMTP id 5614622812f47-3b6a837a2e1so3635626b6e.0 for ; Tue, 21 Nov 2023 17:08:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1700615307; x=1701220107; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Wj2ICW1Fptg7hPvDVbuVBoYZZibiQ7SyAcM7RK2jyBY=; b=iZXwdjbOkGaR0SiH6TAV5H8KWJTvonLdpoRy1ZqKJdNjm1Vq15mx+ROUUH7V1QPv2K RQDSvIeB1W05CjbGesdrj+1crPueCy8rAKgvE4GoEHOlOKWx9AiMw38pHk/yQWBIkUUb YdGyIMuZZEpJ5dqj8Xqo66iiJUh1LFyT6CXIdacCxQMQ6k6nCRDaMSlSqGebTfCnNfZ4 ND84v7EUGDpwjKHsJZFpI+2iQ1LZBg+Nxz95aGnw8TEpFWaTapc9L1iBHTG06UZwWNnt G2czdw7z4oAnkZHv/+86vOd+3XJZrhVlPl4rZuP7t+7aJqJ0Ns5Gy9bJ76VaT/U9QEiQ waZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700615307; x=1701220107; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Wj2ICW1Fptg7hPvDVbuVBoYZZibiQ7SyAcM7RK2jyBY=; b=mMdL0lcgC23lZrUt0sQYBHo+INH7Q3U2I0NeIqwqnVM05cRB3nYXi/95zGiydyURWL hSgx4+Eq3LlR0WlSodPeNH2JgyQXV+WVRjSMKITTYgc7d/IdrBUj3buJwuPLQ2S2mkAu 1KiTiwZUMtYWfQ+iy92iqsMEtsTXOzUOE9LzrfCQL+Y38jvUov95VUcC/raMmoYPn/sY hTI+Tf282Efeom0CHzoJjDsMk9upn8JFM+phXhYk4O+kbdbK79gH1wlnpQC2Hso52+sP uykQ/UDmoO1CU4HpDvwvSognRUVGuKXgAeQldY/XZ9L9TLa1XPjhD6HU3C4aJ3aRtgAN Iayw== X-Gm-Message-State: AOJu0YyLdose4vRYkZU4LuIZRoODcRBndqi3x6oRt6RsZnpyKM9eUP4l fZhlE672fOoVyCgLX3QAnw24xQAPAAPCAp+rYWg= X-Google-Smtp-Source: AGHT+IE4b2JAkpIlmqFLR7/QsgdBYQzfOQCASKlXtamU7cDqbXpB+e41Rju/TG/nF3J4VXpKvwiJIw== X-Received: by 2002:a05:6808:699:b0:3b7:673:8705 with SMTP id k25-20020a056808069900b003b706738705mr885196oig.18.1700615307054; Tue, 21 Nov 2023 17:08:27 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id bn5-20020a056a00324500b006be047268d5sm8713961pfb.174.2023.11.21.17.08.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Nov 2023 17:08:26 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v3 8/8] riscv: mm: Always use ASID to flush MM contexts Date: Tue, 21 Nov 2023 17:07:19 -0800 Message-ID: <20231122010815.3545294-9-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231122010815.3545294-1-samuel.holland@sifive.com> References: <20231122010815.3545294-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231121_170828_185921_4E71040D X-CRM114-Status: GOOD ( 12.72 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Even if multiple ASIDs are not supported, using the single-ASID variant of the sfence.vma instruction preserves TLB entries for global (kernel) pages. So it is always more efficient to use the single-ASID code path. Signed-off-by: Samuel Holland --- (no changes since v2) Changes in v2: - Update both copies of __flush_tlb_range() arch/riscv/include/asm/mmu_context.h | 2 -- arch/riscv/mm/context.c | 3 +-- arch/riscv/mm/tlbflush.c | 5 ++--- 3 files changed, 3 insertions(+), 7 deletions(-) diff --git a/arch/riscv/include/asm/mmu_context.h b/arch/riscv/include/asm/mmu_context.h index 7030837adc1a..b0659413a080 100644 --- a/arch/riscv/include/asm/mmu_context.h +++ b/arch/riscv/include/asm/mmu_context.h @@ -33,8 +33,6 @@ static inline int init_new_context(struct task_struct *tsk, return 0; } -DECLARE_STATIC_KEY_FALSE(use_asid_allocator); - #include #endif /* _ASM_RISCV_MMU_CONTEXT_H */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 3ca9b653df7d..20057085ab8a 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -18,8 +18,7 @@ #ifdef CONFIG_MMU -DEFINE_STATIC_KEY_FALSE(use_asid_allocator); - +static DEFINE_STATIC_KEY_FALSE(use_asid_allocator); static unsigned long num_asids; static atomic_long_t current_version; diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 23409d70440f..d6619be10341 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -90,8 +90,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, /* check if the tlbflush needs to be sent to other CPUs */ broadcast = cpumask_any_but(cmask, cpuid) < nr_cpu_ids; - if (static_branch_unlikely(&use_asid_allocator)) - asid = cntx2asid(atomic_long_read(&mm->context.id)); + asid = cntx2asid(atomic_long_read(&mm->context.id)); } else { cmask = cpu_online_mask; broadcast = true; @@ -122,7 +121,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, { unsigned long asid = FLUSH_TLB_NO_ASID; - if (mm && static_branch_unlikely(&use_asid_allocator)) + if (mm) asid = cntx2asid(atomic_long_read(&mm->context.id)); local_flush_tlb_range_asid(start, size, stride, asid);