From patchwork Thu Dec 7 15:03:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13483398 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92F22C4167B for ; Thu, 7 Dec 2023 15:05:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 305EB6B008A; Thu, 7 Dec 2023 10:05:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B5776B008C; Thu, 7 Dec 2023 10:05:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 17D5D6B0092; Thu, 7 Dec 2023 10:05:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 091546B008A for ; Thu, 7 Dec 2023 10:05:38 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id CC95C1A011E for ; Thu, 7 Dec 2023 15:05:37 +0000 (UTC) X-FDA: 81540346314.23.518DA30 Received: from mail-wr1-f50.google.com (mail-wr1-f50.google.com [209.85.221.50]) by imf19.hostedemail.com (Postfix) with ESMTP id 20FF51A0029 for ; Thu, 7 Dec 2023 15:04:53 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=ShYsdKZx; dmarc=none; spf=pass (imf19.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.221.50 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701961494; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pjO7lbfVOd3z5ovuMphYRbL/oKKYVGVFcCWOfr+CdbY=; b=0Z96thzpD37ZDLR+sq1KGVJabY3qjyhNWSytkgEM/B2zEyt2FQY3pH9bDZHvwB9zhtvkx6 hi+zOCz8td0D+YKSAzBTUYq1V/Zi54cmRhA3wqLXkg8rPNMoo+cDsyotzzYRPrMPmVLd2v xOmnYsy2AmbAi3GaU3XkCkxvLkZm190= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=ShYsdKZx; dmarc=none; spf=pass (imf19.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.221.50 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701961494; a=rsa-sha256; cv=none; b=L44bKzbyhO1mwE55cLSfmgEKOoAyT0nsQ63/fMNwN49q04HoTQP2W0btXwoRqRzKlRxtFS 3VG5R/DfXJFvFb6lw3mFaQAbT1c3fdUR8tgVcJ82x8eyEacCFOr7hDHxeBJxFRwq+EuDdq 6pBlD6xHRWMKfVHFOdEP5cS0H9FOw+0= Received: by mail-wr1-f50.google.com with SMTP id ffacd0b85a97d-3332ad5b3e3so1058105f8f.2 for ; Thu, 07 Dec 2023 07:04:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1701961492; x=1702566292; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pjO7lbfVOd3z5ovuMphYRbL/oKKYVGVFcCWOfr+CdbY=; b=ShYsdKZxAcekeo317bmfe74JiKH8sI3GB8KfA7PSxMpTBKszu01bttCAUXQnUNWbB0 thOfU0Vko4TnCABoxrwy72jjyf8uURuDsTz/X9pJ2bNgWBZlv++zmO+wCCGOYkP0iZ3n NKxPZRPZgsk/bqecLx6v7wtv0S0B8rFk/ALemvnags/7bZuLkVNgcKMIT+4JFuPOFYP8 hpBWaM7ELw8GVY6Fz7n5WP8CMlHwSCBMLkoS+TgBsNRmTWHOZJ+zYONXLkP9Cra7UK3k hWmVTVCK7h1Az7/X9ZRGJ24x0tKb2WgzVzdS5WWLibEPl3dJazqMg3lyO4bBmIhmT5eJ L73A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701961492; x=1702566292; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pjO7lbfVOd3z5ovuMphYRbL/oKKYVGVFcCWOfr+CdbY=; b=e+qdAd2niIsM3Gu7vtZknmIeOQnpBmQ/O1jvISPqJmLL6Yuhcc4XCWVxKgswX218Gd otZj0MWsmqklFq+Yre2etcdv4p/OpE95z8GnUdQMXxji943RzP8NRYA2Jv9NklJ0288W QliFt2/+Mvbh0iprlczs1IUcEUpoeE158cfKAgHKeP8KDnYjPKPVIIa+Z8EOkSZPBkIw mGJoyBlpWLyyMYN5CBaDggDjC1oPSjkzMmw3mqgHxWoR+N3wk5DqBGVX47HUYWWVCOic LwD0f0DE6BjIxn1ak/Q5OMboBdnotDthwuQpV09x/vgRT/vOS/TJGhfn1SgrLJOF9zr6 YqBg== X-Gm-Message-State: AOJu0YyQuExwJOFn5/lY4L9MKBUucF48/tDmHyjyoI/Lj30wH9UnzuLl N9IEbUS2rSq+/5XM94kRL66Oaw== X-Google-Smtp-Source: AGHT+IHSI/0t5tXA9bwS1sCwP7hAVbXTwuyEMVxXnQWCRneQmLBavNOOSXDtGDpxWg7dQ2EGSdeIhg== X-Received: by 2002:a05:600c:152:b0:40c:711:f492 with SMTP id w18-20020a05600c015200b0040c0711f492mr1593121wmm.181.1701961492195; Thu, 07 Dec 2023 07:04:52 -0800 (PST) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id gw18-20020a05600c851200b004053e9276easm2206857wmb.32.2023.12.07.07.04.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 07:04:51 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Thomas Bogendoerfer , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , Ved Shanbhogue , Matt Evans , Dylan Jhong , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH RFC/RFT 1/4] riscv: Stop emitting preventive sfence.vma for new vmalloc mappings Date: Thu, 7 Dec 2023 16:03:45 +0100 Message-Id: <20231207150348.82096-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231207150348.82096-1-alexghiti@rivosinc.com> References: <20231207150348.82096-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 20FF51A0029 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: ubktnqfj3coq9c19ixi66a6wxok69ww6 X-HE-Tag: 1701961493-431084 X-HE-Meta: U2FsdGVkX1//78MybNWGnsKVkfbssAaue3wm88dS9FaydXBx5ralQ/R71fzGt4QsIKeSOIpApWUPMnrYsmwHY+1wpltz1dNH9zFynA9afRL+mvXJAuihiG/p1yMO3vpspj+7Aw1ozzxBKHW1ZoiSSXNSduUp2O/3rywQ99JB6Dh/DMSbDKQ5T7Js6bWpO1aXnmQV8O3rxVBErn6KvBp0fmX2SLi7P4L6Lm67JV6ITtOHUthRTQn9sa7ljYKGpWN521S2WAfoepo5EsLYBau5dpmeTTbap8KQbNkn9HxtIpyFoCJ0uEKYKPzJ+aw6SV1K+EWFUJtYG/cPyoGfet+nfT+/it3OMNa1gJoLuXUe124fbwlWiSNHWniVy1t3X84pU3iPvHa/jO9xLZW0mL426CWnAkFaPH+8Hn+eujRU3rJRNHQiaw1s5dB2f8M95+XdXchtLdFRBFmU15/oyDws+xJNbSQ9QLtyCsMeXxGfXatCNmTIkItHbwpzfohiBpzQARpCh7+x5qpSjR8WGrc1YFRqf58x8KCHJXiaj0rfkAK88igZa9Ap9XFnhO/JFfY+X4T7R5NFqoLoQTRuXx5btMjAU4Ggd6dsY26YQPQDzZZ6pszkOp7WgclZEgYdymGbfkdIHusZaH9G9f9Vyt45ie1I6bSp9vFqnObonFOrOrVvar109I57mPDyQoZCBKl3E7tLdSIFGN6/kAG9BFv8DJPeuZsAmeHzBjpz/uduFPkcn218EL1ugrHJfYl0BDreSuuk2o1NNZDudaPyQv5XEWbJf1tcK+Bz9SXs3/fXiE9Pxa+rwGKbryyYSQDoGlYOKUOUmKCLCFMKy69g2oRdv/IizKC2nIP7Gjzv7qe6/w1xuL+mQMe8jlhryIP2mLBPrRBH5eQFVIGXFK4CgkTDmjc29mnlKT6VPePl/RklgqV/JNGVYqkfOIfCUCMYkrw4/PGnkoVgTwpOsOapS57 mPLjhAyG XsUJ6kPuRCILIyuTg7xGFUcaoEcZpKs9bTEafotG9jCLz6JgqP0/KSyY+kbs+QbL8VDLFYOPOu+1Q5cxdM42C3Z1i3GIDxWqd4WApXrhCOIAwjBb2SwG7smQTi5N6JJftK62nHEjeghFz3JqgRc4jSmU5fvgIWCsWT4evpwEGBeacxJ4i0OD6gcAskNMJ2vdgJ8jDFlUhAXRnUZ29SNjdrwIe/LCtUh1Qwbc3hwGVUCxp6CjXjlBXdMADPXhfjA8W1eJWDyCo7/Iu+j/NObPypYPleh/Dx/hq5kxs0GEin6LlP2TzruGT6T52BjpM+kt2Y6ab4kcUiVX0y2fB+GPLEK1X660O0YVAs8J3kyAdNgwjIR0FEVdX3hkakENhDRHYeDCyp+QMEIxsnO+GAZXH/do8bV6WGHQxfWywj0ZB3i+bjDOFfCFY5eEWHkSEwGabafVy9ifPZfd0bWi3KqZ7ZcB9Lwr4EEboOgLQsdPGlCHbtes8/dXilqwHAPxEJcZYy85u1nzf6xDTzaJMMVl9DgKkqUEjxzqzPOFGsmOYjrPkXFVLgCqGGrEO8N0PZeuk1swymn+fQW8UxqaFEAMcVY6sXx/+29JoU1rjV+9GOQhEv/Aj8iLRjb3J7p1nCYTZN0jK5BRk5lxr5HXUjDEKJnYIMzDirEIhiUF5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In 6.5, we removed the vmalloc fault path because that can't work (see [1] [2]). Then in order to make sure that new page table entries were seen by the page table walker, we had to preventively emit a sfence.vma on all harts [3] but this solution is very costly since it relies on IPI. And even there, we could end up in a loop of vmalloc faults if a vmalloc allocation is done in the IPI path (for example if it is traced, see [4]), which could result in a kernel stack overflow. Those preventive sfence.vma needed to be emitted because: - if the uarch caches invalid entries, the new mapping may not be observed by the page table walker and an invalidation may be needed. - if the uarch does not cache invalid entries, a reordered access could "miss" the new mapping and traps: in that case, we would actually only need to retry the access, no sfence.vma is required. So this patch removes those preventive sfence.vma and actually handles the possible (and unlikely) exceptions. And since the kernel stacks mappings lie in the vmalloc area, this handling must be done very early when the trap is taken, at the very beginning of handle_exception: this also rules out the vmalloc allocations in the fault path. Note that for now, we emit a sfence.vma even for uarchs that do not cache invalid entries as we have no means to know that: that will be fixed in the next patch. Link: https://lore.kernel.org/linux-riscv/20230531093817.665799-1-bjorn@kernel.org/ [1] Link: https://lore.kernel.org/linux-riscv/20230801090927.2018653-1-dylan@andestech.com [2] Link: https://lore.kernel.org/linux-riscv/20230725132246.817726-1-alexghiti@rivosinc.com/ [3] Link: https://lore.kernel.org/lkml/20200508144043.13893-1-joro@8bytes.org/ [4] Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/cacheflush.h | 19 +++++- arch/riscv/include/asm/thread_info.h | 5 ++ arch/riscv/kernel/asm-offsets.c | 5 ++ arch/riscv/kernel/entry.S | 94 ++++++++++++++++++++++++++++ arch/riscv/mm/init.c | 2 + 5 files changed, 124 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/cacheflush.h b/arch/riscv/include/asm/cacheflush.h index 3cb53c4df27c..a916cbc69d47 100644 --- a/arch/riscv/include/asm/cacheflush.h +++ b/arch/riscv/include/asm/cacheflush.h @@ -37,7 +37,24 @@ static inline void flush_dcache_page(struct page *page) flush_icache_mm(vma->vm_mm, 0) #ifdef CONFIG_64BIT -#define flush_cache_vmap(start, end) flush_tlb_kernel_range(start, end) +extern u64 new_vmalloc[]; +extern char _end[]; +#define flush_cache_vmap flush_cache_vmap +static inline void flush_cache_vmap(unsigned long start, unsigned long end) +{ + if ((start < VMALLOC_END && end > VMALLOC_START) || + (start < MODULES_END && end > MODULES_VADDR)) { + int i; + + /* + * We don't care if concurrently a cpu resets this value since + * the only place this can happen is in handle_exception() where + * an sfence.vma is emitted. + */ + for (i = 0; i < NR_CPUS / sizeof(u64) + 1; ++i) + new_vmalloc[i] = -1ULL; + } +} #endif #ifndef CONFIG_SMP diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index 1833beb00489..8fe12fa6c329 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -60,6 +60,11 @@ struct thread_info { long user_sp; /* User stack pointer */ int cpu; unsigned long syscall_work; /* SYSCALL_WORK_ flags */ + /* + * Used in handle_exception() to save a0, a1 and a2 before knowing if we + * can access the kernel stack. + */ + unsigned long a0, a1, a2; }; /* diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index d6a75aac1d27..340c1c84560d 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -34,10 +34,15 @@ void asm_offsets(void) OFFSET(TASK_THREAD_S9, task_struct, thread.s[9]); OFFSET(TASK_THREAD_S10, task_struct, thread.s[10]); OFFSET(TASK_THREAD_S11, task_struct, thread.s[11]); + + OFFSET(TASK_TI_CPU, task_struct, thread_info.cpu); OFFSET(TASK_TI_FLAGS, task_struct, thread_info.flags); OFFSET(TASK_TI_PREEMPT_COUNT, task_struct, thread_info.preempt_count); OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp); OFFSET(TASK_TI_USER_SP, task_struct, thread_info.user_sp); + OFFSET(TASK_TI_A0, task_struct, thread_info.a0); + OFFSET(TASK_TI_A1, task_struct, thread_info.a1); + OFFSET(TASK_TI_A2, task_struct, thread_info.a2); OFFSET(TASK_THREAD_F0, task_struct, thread.fstate.f[0]); OFFSET(TASK_THREAD_F1, task_struct, thread.fstate.f[1]); diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 143a2bb3e697..3a3c7b563816 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -14,6 +14,88 @@ #include #include +.macro new_vmalloc_check + REG_S a0, TASK_TI_A0(tp) + REG_S a1, TASK_TI_A1(tp) + REG_S a2, TASK_TI_A2(tp) + + csrr a0, CSR_CAUSE + /* Exclude IRQs */ + blt a0, zero, _new_vmalloc_restore_context + /* Only check new_vmalloc if we are in page/protection fault */ + li a1, EXC_LOAD_PAGE_FAULT + beq a0, a1, _new_vmalloc_kernel_address + li a1, EXC_STORE_PAGE_FAULT + beq a0, a1, _new_vmalloc_kernel_address + li a1, EXC_INST_PAGE_FAULT + bne a0, a1, _new_vmalloc_restore_context + +_new_vmalloc_kernel_address: + /* Is it a kernel address? */ + csrr a0, CSR_TVAL + bge a0, zero, _new_vmalloc_restore_context + + /* Check if a new vmalloc mapping appeared that could explain the trap */ + + /* + * Computes: + * a0 = &new_vmalloc[BIT_WORD(cpu)] + * a1 = BIT_MASK(cpu) + */ + REG_L a2, TASK_TI_CPU(tp) + /* + * Compute the new_vmalloc element position: + * (cpu / 64) * 8 = (cpu >> 6) << 3 + */ + srli a1, a2, 6 + slli a1, a1, 3 + la a0, new_vmalloc + add a0, a0, a1 + /* + * Compute the bit position in the new_vmalloc element: + * bit_pos = cpu % 64 = cpu - (cpu / 64) * 64 = cpu - (cpu >> 6) << 6 + * = cpu - ((cpu >> 6) << 3) << 3 + */ + slli a1, a1, 3 + sub a1, a2, a1 + /* Compute the "get mask": 1 << bit_pos */ + li a2, 1 + sll a1, a2, a1 + + /* Check the value of new_vmalloc for this cpu */ + ld a2, 0(a0) + and a2, a2, a1 + beq a2, zero, _new_vmalloc_restore_context + + ld a2, 0(a0) + not a1, a1 + and a1, a2, a1 + sd a1, 0(a0) + + /* Only emit a sfence.vma if the uarch caches invalid entries */ + la a0, tlb_caching_invalid_entries + lb a0, 0(a0) + beqz a0, _new_vmalloc_no_caching_invalid_entries + sfence.vma +_new_vmalloc_no_caching_invalid_entries: + // debug + la a0, nr_sfence_vma_handle_exception + li a1, 1 + amoadd.w a0, a1, (a0) + // end debug + REG_L a0, TASK_TI_A0(tp) + REG_L a1, TASK_TI_A1(tp) + REG_L a2, TASK_TI_A2(tp) + csrw CSR_SCRATCH, x0 + sret + +_new_vmalloc_restore_context: + REG_L a0, TASK_TI_A0(tp) + REG_L a1, TASK_TI_A1(tp) + REG_L a2, TASK_TI_A2(tp) +.endm + + SYM_CODE_START(handle_exception) /* * If coming from userspace, preserve the user thread pointer and load @@ -25,6 +107,18 @@ SYM_CODE_START(handle_exception) _restore_kernel_tpsp: csrr tp, CSR_SCRATCH + + /* + * The RISC-V kernel does not eagerly emit a sfence.vma after each + * new vmalloc mapping, which may result in exceptions: + * - if the uarch caches invalid entries, the new mapping would not be + * observed by the page table walker and an invalidation is needed. + * - if the uarch does not cache invalid entries, a reordered access + * could "miss" the new mapping and traps: in that case, we only need + * to retry the access, no sfence.vma is required. + */ + new_vmalloc_check + REG_S sp, TASK_TI_KERNEL_SP(tp) #ifdef CONFIG_VMAP_STACK diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 0798bd861dcb..379403de6c6f 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -36,6 +36,8 @@ #include "../kernel/head.h" +u64 new_vmalloc[NR_CPUS / sizeof(u64) + 1]; + struct kernel_mapping kernel_map __ro_after_init; EXPORT_SYMBOL(kernel_map); #ifdef CONFIG_XIP_KERNEL From patchwork Thu Dec 7 15:03:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13483399 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F2ECC4167B for ; Thu, 7 Dec 2023 15:06:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B1C206B0092; Thu, 7 Dec 2023 10:06:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AA4BF6B0093; Thu, 7 Dec 2023 10:06:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F7B36B0095; Thu, 7 Dec 2023 10:06:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 76D406B0092 for ; Thu, 7 Dec 2023 10:06:17 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 4C789A023C for ; Thu, 7 Dec 2023 15:06:17 +0000 (UTC) X-FDA: 81540347994.03.65D4DAC Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) by imf27.hostedemail.com (Postfix) with ESMTP id 0CDBF4000B for ; Thu, 7 Dec 2023 15:05:54 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=SSDtWNrD; dmarc=none; spf=pass (imf27.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.221.47 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701961555; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LVN+VzhbTBys9mVhAQGqZxkaIjV/7TRqzMN+a14FBcU=; b=HcOU2U6azEG6440nkBUgKEkqovLWQ/kcJKYvvk5ZV0KZxst5BlJKgrVk3nikO+W+QRq7RJ Z0vThU1bDtcHKJCWK7d5DiCWqGMCOk/bF8mAOh/bR34dj3wZgndrpx5V7T/tna9fh9eVp+ dDsx/+1vOV36ywO9XfbR5p6jQOG5TKU= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=SSDtWNrD; dmarc=none; spf=pass (imf27.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.221.47 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701961555; a=rsa-sha256; cv=none; b=w8PlaXyN6KBfvPQt9H3ez84Dr2fzK4nqnvfW0xVBBi4Rxk0ii2iWr7ZytosMZHSRAuraaf DAqCR1B0J00lDDfxzPB+ptlMJLJn55HtCRplDw3t9K5G9/NNeXMd7okmvK+OG8GZcguuG9 0tUUi6bhUkCavMxN9czEZflDEI1TJ80= Received: by mail-wr1-f47.google.com with SMTP id ffacd0b85a97d-3332ad5b3e3so1059514f8f.2 for ; Thu, 07 Dec 2023 07:05:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1701961553; x=1702566353; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LVN+VzhbTBys9mVhAQGqZxkaIjV/7TRqzMN+a14FBcU=; b=SSDtWNrDem95axoCY9b+Xo5xwIg3OrfrzpExL7rPgm6Mvrqd58CgzeSA5BC9OSsOEZ saxXSGToJ/CailYtZZUUnU5gDtnMc58ccnpiNYLki8POOPitRVEyZpTcRVZ79Y7q+fva O65DzSk/7aC6lTtQMAzOtUQYEMLB4mMibBXWIRoYfdLTChkPBgM7sI2nwp5ngk2Jm8ob P7vlb6tNyrVPqWBqd516E3ubbBzcHSTA3kyXt2UWsh5NFoBDlGv86MuixuO4NC9fqd30 sJPvjsgKSYoSNZQa+Ex4dgXDjV6phgSNQB51BeY6nSTPgreKEQ4IhCU93zP0DMRDxcgm MPDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701961553; x=1702566353; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LVN+VzhbTBys9mVhAQGqZxkaIjV/7TRqzMN+a14FBcU=; b=BK+k5wBTIE268KBqMnb4pzLVVAsh7XVF5nBfQ+IBmk/EZJq8LikdzN9U9BQgo3BZ5N v0IscFDvqFI+cRGKc3w+Tf5eK+LCwKqpFPAUXjY+pHRH1UyE5Q1S2KBPmo4lLwdUqASK PEovqfy8HbbBw1aIYU3hAensAJq1vtLc36ITz2vdG8/+Vh9z7kxFYcJ8Aw6jacTpLgn0 h6GzVBdz9huvDTBiQ/nkc8UMdjKEuWlwYT4MX0S78Le4CC0FbxARnKVjI0o4kdsp7p9S VV61p10FBfXtB2Wp2p3VYRUb2GI51opCJp79WGRA69sflXy0mlFIZDkIqkRK03ugTI/3 CWng== X-Gm-Message-State: AOJu0Yx5jEvKwuuq7mOPfplYXRu+5nZsoPTFL7RKdpn1jDxv6iTYAN8Y xCVWwby9pyoa5DldlI12Wpcv5A== X-Google-Smtp-Source: AGHT+IEj2QTBGpOf3OHBs9Yni4aQareAokfWZfEZoiM8ehZh4/VLAukwcmzFcEIzO5CULnzn0YcNZw== X-Received: by 2002:adf:ea82:0:b0:333:1907:c2a3 with SMTP id s2-20020adfea82000000b003331907c2a3mr1503404wrm.21.1701961553264; Thu, 07 Dec 2023 07:05:53 -0800 (PST) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id p13-20020a056000018d00b00333415503a7sm1644486wrx.22.2023.12.07.07.05.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 07:05:53 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Thomas Bogendoerfer , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , Ved Shanbhogue , Matt Evans , Dylan Jhong , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH RFC/RFT 2/4] riscv: Add a runtime detection of invalid TLB entries caching Date: Thu, 7 Dec 2023 16:03:46 +0100 Message-Id: <20231207150348.82096-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231207150348.82096-1-alexghiti@rivosinc.com> References: <20231207150348.82096-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0CDBF4000B X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: nhpp3uoawhe3fdyj138qjcb1kpop5tg3 X-HE-Tag: 1701961554-922309 X-HE-Meta: U2FsdGVkX1+FlsYhol75AvnTVVrK9n6TX39qJI+pRhqpkwP7jClT5dNS3eAQhLQAxiRGVN1IVFh/Pu5T/dcpQxIR8Pjtx+5TGLOFPmlgT7EZBoIzZDT+cyYNLPElHxnI/fR3KRXI7ybHzdSbcFgMymgcv9ICgc8SxNck28YC81mIS9to4FWmusEV3Q3BiYVBHRS9H2vGum7wuMfrL9igAowquD1TrRhDaBLXDGRErqK8O/fJu4jDqSWaK8sIRVvzUTgM8de5SRWv5LRoY+PFFs4kaVxf+iOcDS223sgSm0phxlZU2zkCuRPWIvr8VWzmCmo7xpMGG+Eyf4wARxyiu9s5V75I1URauYmVWKYVh8KBlBv0jZVQNZjAmTaKpxvMMX4ILvCV7/T6+xalMKstDcp9mRB93X1+/fAbevYwmji9w8YZlIzWNtwWrDTsqweLbgekPEFqFRy7zGMv05zxydoZ39VfaNNWRaC6p6hBJRSpVUO71X45KqNriFZdOZ+cbl9gQ2tZjdaUA4w5ShC7RwXxJtshjkAqTVn550gjWOKYT0aIpCZ2K6yKxnGBnEYXjaU/EEQ1kQ6yfDZxMX5qlIXSPpRqHhUt4ppPvKu7qOtgKRTDTVek/Xs6SLbXud/5NikRoqolv0VppPzHbvVvzOdgNi4M0YeU003yvIKdtbZwNTE4fijkOLAVwy19BfWr9nosaEjXncvWnDQDyeddvbML+c9llJqJp63PMBmZDd2rhy9LdOQ/LXQdYtOnCnUJ1F/fshMMnzftLALp5mKTjPT6+baKOQOlyhuYeJGtnGC5YORLWg99OqJpGrksg4s3LYPfu4tc7NXaCOqU15wseo6Fqs/HkP+Rw2yF8VsSklj+5oq2hBaKPEhOqKGbdpM/2/qnivLjTT3GjjHo/X8/ajsyq682WoLdXamyfEr/ldI0J2YpFocyqeCM9n0OKEwFbD05AOjcQrwhbL7b0r+ 5I8law20 DWNipoqn2WKcovy+vuBHi1455iHtQKaL5RuKGkVnI/DOLwlMQ8tFDXqiGg88dG9kNyyWJebnmjFCM68E/Q5sMi0vwVd6cHpvDhCRZyf3z/N/STUKycJH0EAdDXVIkTsuEIG/8hg5EebVejjB/Rt6dZ7sVt22r3IB9PTKy4R8hpdl9spWco7vM4BTU71Ywfxa5pRiBfbyuZKrq5g3bzxVBQ1mE4rGuV3xcj3zvtQmMF/in0e1PN94vBXF466YYB7v5gzgYTWvCUOCnjq6WKuoJkWNwhHk9FKRWDNT/xWCtAXByHbluUq4/beQ+fsyxDN1amgAvMJGcVabQ6nVBoLhJ96p8hrabDrb9pavRvb/c5Nd8p1p13+LN4BFGwGLiK+X4evnRV5ynQDiiBtsMWIo9f+ve5kWhbFmKHoCD1TPpHd2buPMyzq28HSaP/v/OaGbvYQsBZuQJpmec4tc5xEovz79C90ouBvWIjQH7TEj/5N0pqmE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This mechanism allows to completely bypass the sfence.vma introduced by the previous commit for uarchs that do not cache invalid TLB entries. Signed-off-by: Alexandre Ghiti --- arch/riscv/mm/init.c | 124 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 124 insertions(+) diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 379403de6c6f..2e854613740c 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -56,6 +56,8 @@ bool pgtable_l5_enabled = IS_ENABLED(CONFIG_64BIT) && !IS_ENABLED(CONFIG_XIP_KER EXPORT_SYMBOL(pgtable_l4_enabled); EXPORT_SYMBOL(pgtable_l5_enabled); +bool tlb_caching_invalid_entries; + phys_addr_t phys_ram_base __ro_after_init; EXPORT_SYMBOL(phys_ram_base); @@ -750,6 +752,18 @@ static void __init disable_pgtable_l4(void) satp_mode = SATP_MODE_39; } +static void __init enable_pgtable_l5(void) +{ + pgtable_l5_enabled = true; + satp_mode = SATP_MODE_57; +} + +static void __init enable_pgtable_l4(void) +{ + pgtable_l4_enabled = true; + satp_mode = SATP_MODE_48; +} + static int __init print_no4lvl(char *p) { pr_info("Disabled 4-level and 5-level paging"); @@ -826,6 +840,112 @@ static __init void set_satp_mode(uintptr_t dtb_pa) memset(early_pud, 0, PAGE_SIZE); memset(early_pmd, 0, PAGE_SIZE); } + +/* Determine at runtime if the uarch caches invalid TLB entries */ +static __init void set_tlb_caching_invalid_entries(void) +{ +#define NR_RETRIES_CACHING_INVALID_ENTRIES 50 + uintptr_t set_tlb_caching_invalid_entries_pmd = ((unsigned long)set_tlb_caching_invalid_entries) & PMD_MASK; + // TODO the test_addr as defined below could go into another pud... + uintptr_t test_addr = set_tlb_caching_invalid_entries_pmd + 2 * PMD_SIZE; + pmd_t valid_pmd; + u64 satp; + int i = 0; + + /* To ease the page table creation */ + disable_pgtable_l5(); + disable_pgtable_l4(); + + /* Establish a mapping for set_tlb_caching_invalid_entries() in sv39 */ + create_pgd_mapping(early_pg_dir, + set_tlb_caching_invalid_entries_pmd, + (uintptr_t)early_pmd, + PGDIR_SIZE, PAGE_TABLE); + + /* Handle the case where set_tlb_caching_invalid_entries straddles 2 PMDs */ + create_pmd_mapping(early_pmd, + set_tlb_caching_invalid_entries_pmd, + set_tlb_caching_invalid_entries_pmd, + PMD_SIZE, PAGE_KERNEL_EXEC); + create_pmd_mapping(early_pmd, + set_tlb_caching_invalid_entries_pmd + PMD_SIZE, + set_tlb_caching_invalid_entries_pmd + PMD_SIZE, + PMD_SIZE, PAGE_KERNEL_EXEC); + + /* Establish an invalid mapping */ + create_pmd_mapping(early_pmd, test_addr, 0, PMD_SIZE, __pgprot(0)); + + /* Precompute the valid pmd here because the mapping for pfn_pmd() won't exist */ + valid_pmd = pfn_pmd(PFN_DOWN(set_tlb_caching_invalid_entries_pmd), PAGE_KERNEL); + + local_flush_tlb_all(); + satp = PFN_DOWN((uintptr_t)&early_pg_dir) | SATP_MODE_39; + csr_write(CSR_SATP, satp); + + /* + * Set stvec to after the trapping access, access this invalid mapping + * and legitimately trap + */ + // TODO: Should I save the previous stvec? +#define ASM_STR(x) __ASM_STR(x) + asm volatile( + "la a0, 1f \n" + "csrw " ASM_STR(CSR_TVEC) ", a0 \n" + "ld a0, 0(%0) \n" + ".align 2 \n" + "1: \n" + : + : "r" (test_addr) + : "a0" + ); + + /* Now establish a valid mapping to check if the invalid one is cached */ + early_pmd[pmd_index(test_addr)] = valid_pmd; + + /* + * Access the valid mapping multiple times: indeed, we can't use + * sfence.vma as a barrier to make sure the cpu did not reorder accesses + * so we may trap even if the uarch does not cache invalid entries. By + * trying a few times, we make sure that those uarchs will see the right + * mapping at some point. + */ + + i = NR_RETRIES_CACHING_INVALID_ENTRIES; + +#define ASM_STR(x) __ASM_STR(x) + asm_volatile_goto( + "la a0, 1f \n" + "csrw " ASM_STR(CSR_TVEC) ", a0 \n" + ".align 2 \n" + "1: \n" + "addi %0, %0, -1 \n" + "blt %0, zero, %l[caching_invalid_entries] \n" + "ld a0, 0(%1) \n" + : + : "r" (i), "r" (test_addr) + : "a0" + : caching_invalid_entries + ); + + csr_write(CSR_SATP, 0ULL); + local_flush_tlb_all(); + + /* If we don't trap, the uarch does not cache invalid entries! */ + tlb_caching_invalid_entries = false; + goto clean; + +caching_invalid_entries: + csr_write(CSR_SATP, 0ULL); + local_flush_tlb_all(); + + tlb_caching_invalid_entries = true; +clean: + memset(early_pg_dir, 0, PAGE_SIZE); + memset(early_pmd, 0, PAGE_SIZE); + + enable_pgtable_l4(); + enable_pgtable_l5(); +} #endif /* @@ -1072,6 +1192,7 @@ asmlinkage void __init setup_vm(uintptr_t dtb_pa) #endif #if defined(CONFIG_64BIT) && !defined(CONFIG_XIP_KERNEL) + set_tlb_caching_invalid_entries(); set_satp_mode(dtb_pa); #endif @@ -1322,6 +1443,9 @@ static void __init setup_vm_final(void) local_flush_tlb_all(); pt_ops_set_late(); + + pr_info("uarch caches invalid entries: %s", + tlb_caching_invalid_entries ? "yes" : "no"); } #else asmlinkage void __init setup_vm(uintptr_t dtb_pa) From patchwork Thu Dec 7 15:03:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13483400 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C55FC4167B for ; Thu, 7 Dec 2023 15:06:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 96FC66B0095; Thu, 7 Dec 2023 10:06:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8F80D6B0096; Thu, 7 Dec 2023 10:06:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7714C6B0099; Thu, 7 Dec 2023 10:06:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 61CE36B0095 for ; Thu, 7 Dec 2023 10:06:58 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 378191A026A for ; Thu, 7 Dec 2023 15:06:58 +0000 (UTC) X-FDA: 81540349716.30.E931CC0 Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) by imf26.hostedemail.com (Postfix) with ESMTP id 2387A140022 for ; Thu, 7 Dec 2023 15:06:55 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=j4rMe67f; spf=pass (imf26.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.221.42 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701961616; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EWcoco88tut+CY63Y/9PXHuBFD03h05uf7rbyZFiPT4=; b=gH3NR5uj+R/Y9IxnoWRFEpQL7fhnbymd/wFX5b6n16/o8j4jyiJ+E5EIyoaHf5dcrABks1 rZ7qbH1Wm60WGrEq6TWi5hpithn/X6zPgcM7AqgUDdL49MRqakbA7lF63IcSKeLZqmLpud bnxo0V13qNqYiWVCk0CC/RIJ7Zm374U= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701961616; a=rsa-sha256; cv=none; b=C3DtadWpX8ZFL9ox3n576WamP9DokCi9KqK6HYEk+7k1gvYr+ckayXxgcWxM9V/NP7n+Tb XLXvTN1RAmI/+NR+sAYhcF+l0i1xFo6it+NMggiTq0QODfB64fVXbGKW0eVuXuSgJHQrtM 4M/BeZkmM3yGNWaQ8kIFB2uzu9vYc6c= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=j4rMe67f; spf=pass (imf26.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.221.42 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none Received: by mail-wr1-f42.google.com with SMTP id ffacd0b85a97d-3316bb1303bso764163f8f.0 for ; Thu, 07 Dec 2023 07:06:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1701961614; x=1702566414; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EWcoco88tut+CY63Y/9PXHuBFD03h05uf7rbyZFiPT4=; b=j4rMe67fSOgDKDDv6uyuPumoup9xUSUor3nnIX1VvTdS+9FPEYNZaKmBys9F3qfuhz JFPmmcCA6O7jomRxjJVDMjJlD1mO1ejHlRvkQpmdp1xRc5CcNpjbgWeBpv8TUVo5/IkS 2JBNSWNRKb4BcEIwgJ4t8FamAZCfAjSXqeFrCZaM4t7NGwHbw/GLAwN7SagxG18aQq64 2Mw1zLbn5Y49QTjOyQ7EFtS0mx37nUT2S4Q7s5AnDKr8IO7qR529wTVh3nUWlRj+4/wv jAOdqf203+2uyX+hhA7rbV0Rvb70mOIXAv6UYXZKVusViGLdbq534Wg0utC/9WqJ1wGG j73Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701961614; x=1702566414; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EWcoco88tut+CY63Y/9PXHuBFD03h05uf7rbyZFiPT4=; b=ZQtjaHqhqOwGtOn7lZbyUn073BGS5vvzFwyo5RrJqRVqPQ+4YGCLb/UB6QsN8L/RMi 3gTZYu+ngymX5Z0pqntXQhQEA17n3X4fmF6lWj1rXrwUFtYQPs9aRLr90Dxckb/VIj4r 5psOnKrp1s0rN9a3h7gokHzCZuaTd0RpbQ79iID2zo0pmGHvNQuwCZo5CabSJSGBm6JD am5hI3aaZcQ/pNko9H7CkN5FUcDVDV/S+XEZJuux737XF+r3vGCnVhU1xR57AYP3FiLV g4m/Mlh1hv3y9/m8nS7MUsvqpwowYG8WyAb+ERqVhIVh/mP8f/dy6BHW6ME/+xXAO3WK ptyA== X-Gm-Message-State: AOJu0YzAHFy0rApuoGVxRgV+vti+Nww7ghgpYJ8S2oJLk9vX4Tf2PTck JE/DmGyewbnSQidhOj+jA5N9OA== X-Google-Smtp-Source: AGHT+IHTy8+IvueJsQn+KurIJffNNmKiqFVQtVzbm19iX0i45eZHV4tLMcBmveaKizCMPPNo4tYVVQ== X-Received: by 2002:a5d:67c5:0:b0:333:3867:c5aa with SMTP id n5-20020a5d67c5000000b003333867c5aamr1932506wrw.20.1701961614467; Thu, 07 Dec 2023 07:06:54 -0800 (PST) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id a8-20020adffb88000000b003335e67e574sm1649359wrr.78.2023.12.07.07.06.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 07:06:54 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Thomas Bogendoerfer , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , Ved Shanbhogue , Matt Evans , Dylan Jhong , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH RFC/RFT 3/4] riscv: Stop emitting preventive sfence.vma for new userspace mappings Date: Thu, 7 Dec 2023 16:03:47 +0100 Message-Id: <20231207150348.82096-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231207150348.82096-1-alexghiti@rivosinc.com> References: <20231207150348.82096-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2387A140022 X-Rspam-User: X-Stat-Signature: ubk9d9q76uad4jrh8iqh59n76aom4kma X-Rspamd-Server: rspam03 X-HE-Tag: 1701961615-558375 X-HE-Meta: U2FsdGVkX183DUoZoOxkrwGaE0qOEaKfAdqYvnsTaU9iVDrkX+ocfID9203Vn9r0XeZpIFuj9j9ea27OXoWiSM6Xy3JEHJdxpgpjqPB4mbOw/S2TkZLzG3toXiGMZ+g5+qQdMVZNyUFNdwjVsUopjzjOdtRMCoiWpJ0embYBVbnNyok1kc4nbPhrYixXy0BKiMwR7rMYcSereRHoh9xYY0YQUJ9F43ij3RDOPEXLlhyEUTN63I7NUNmyP9nh/6mR0n+rqfd/TTatooKOzp5U8D9EoCb23bMXmQaEYOQ5JToqoiJoW3xkcEsHdqGNbRjcVjDM/S2l/vd7Ud10vTe2yVQ4cksThwN1WWboENUheXyHjBQ1tF02vAQB6cjlPvXmvhtYm1RMvtJ6kfJydLGsw0xes1SgOREX8Rg9fapTOfPNlswKV7nGM67QAeshSjddHftyf4U0hWK2ePvU0Rq46oPbZmmrANpfR2hPgJevf/d5Ib8+3VlzaIoRE/+YuuDz/s9I0fpnJS4FZL2LqkvTiQ3PXp1t1A9Sln8MxOmbxhOoss8h/6bP0sAkLtSTvR4M2cLj6QnC174d5vJ/41i3ik0wEHDl1NgXaZzwKYdewOOlcZpa9mko5lP0i26Rr7o8SR/02DHcf7cAeUKLoJr/UhOJb5aR861TYru3Vd5W+ByDDiaCM4+mJmt3UdYI1G3NghJAT5WtWYl6KAMZGrR0xbM8G2jdeTyt9mzNdUYw0Hbfvgl4UiWFGpt2xCdLTkY31vwHZF3pTZDI3eXrW++OvlzP3t1H7qzdOjOcS0qNUPj6sXeBmNwUSahO+NN1QosGCJawrokh2mRrAnsqeDYBG9S8rNtItI4J1Nu6s8NsKKEyS/UU9EN+RTQvyxOLBLZ5ddTx4HbyWY2kLrUSab5zKiNPVW6zDVoCHtWwRJwsYSjdVis+4aU+whe64WNuCaSQVQ+sFW0wMMZwjJMjFNy BY/Xj9Vb XEwevsF72RIA33wnVrUHwwRU68p1V2XgltRiB/Hg8ma2yQxramATm2pFn2jwCqBWeVAOPajfMwFbSM6c94ixfmmsyJJpuFntyv6x1R06aTg3BQDSbGfkeLGG1cBULbCtgp6+OIcI05165dnFxtjdSs+9OwmsIFZMnrd3JOePTBRldpCSC0lmrqtu+FFMRwXSgyPz9oB2+5UBB7Gawcpju/ukMUWhNWjKp1FGuNbhZLWPvie6c/IfcJzSDMoz/tBmW8akpffn6iLbAYl9frklXQQxxGUdAU+IouXj9E24lmYcfLDrFrj7Pz432eqa+zS/Xa/75c8d93eURbNJ2+MxAIoZIPYlcfyHXyOyzkSASpCvZX3dBaAskm4RJEbkDnNYwLjgWq4GDSuVUKa0FVgxK+zaUkdtKaPvmubbGXSzY7eDw9cVaETAEpgZI8r4QT/SmhA8IDt56necRSbB0NFy90pB4gYftTVebKJcuF87JyCKk4R4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The preventive sfence.vma were emitted because new mappings must be made visible to the page table walker, either the uarch caches invalid entries or not. Actually, there is no need to preventively sfence.vma on new mappings for userspace, this should be handled only in the page fault path. This allows to drastically reduce the number of sfence.vma emitted: * Ubuntu boot to login: Before: ~630k sfence.vma After: ~200k sfence.vma * ltp - mmapstress01 Before: ~45k After: ~6.3k * lmbench - lat_pagefault Before: ~665k After: 832 (!) * lmbench - lat_mmap Before: ~546k After: 718 (!) The only issue with the removal of sfence.vma in update_mmu_cache() is that on uarchs that cache invalid entries, those won't be invalidated until the process takes a fault: so that's an additional fault in those cases. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 2 +- arch/mips/include/asm/pgtable.h | 6 +-- arch/powerpc/include/asm/book3s/64/tlbflush.h | 8 ++-- arch/riscv/include/asm/pgtable.h | 43 +++++++++++-------- include/linux/pgtable.h | 8 +++- mm/memory.c | 12 +++++- 6 files changed, 48 insertions(+), 31 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 7f7d9b1df4e5..728f25f529a5 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -57,7 +57,7 @@ static inline bool arch_thp_swp_supported(void) * fault on one CPU which has been handled concurrently by another CPU * does not need to perform additional invalidation. */ -#define flush_tlb_fix_spurious_fault(vma, address, ptep) do { } while (0) +#define flush_tlb_fix_spurious_write_fault(vma, address, ptep) do { } while (0) /* * ZERO_PAGE is a global shared page that is always zero: used diff --git a/arch/mips/include/asm/pgtable.h b/arch/mips/include/asm/pgtable.h index 430b208c0130..84439fe6ed29 100644 --- a/arch/mips/include/asm/pgtable.h +++ b/arch/mips/include/asm/pgtable.h @@ -478,9 +478,9 @@ static inline pgprot_t pgprot_writecombine(pgprot_t _prot) return __pgprot(prot); } -static inline void flush_tlb_fix_spurious_fault(struct vm_area_struct *vma, - unsigned long address, - pte_t *ptep) +static inline void flush_tlb_fix_spurious_write_fault(struct vm_area_struct *vma, + unsigned long address, + pte_t *ptep) { } diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush.h b/arch/powerpc/include/asm/book3s/64/tlbflush.h index 1950c1b825b4..7166d56f90db 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush.h @@ -128,10 +128,10 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, #define flush_tlb_page(vma, addr) local_flush_tlb_page(vma, addr) #endif /* CONFIG_SMP */ -#define flush_tlb_fix_spurious_fault flush_tlb_fix_spurious_fault -static inline void flush_tlb_fix_spurious_fault(struct vm_area_struct *vma, - unsigned long address, - pte_t *ptep) +#define flush_tlb_fix_spurious_write_fault flush_tlb_fix_spurious_write_fault +static inline void flush_tlb_fix_spurious_write_fault(struct vm_area_struct *vma, + unsigned long address, + pte_t *ptep) { /* * Book3S 64 does not require spurious fault flushes because the PTE diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index b2ba3f79cfe9..89aa5650f104 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -472,28 +472,20 @@ static inline void update_mmu_cache_range(struct vm_fault *vmf, struct vm_area_struct *vma, unsigned long address, pte_t *ptep, unsigned int nr) { - /* - * The kernel assumes that TLBs don't cache invalid entries, but - * in RISC-V, SFENCE.VMA specifies an ordering constraint, not a - * cache flush; it is necessary even after writing invalid entries. - * Relying on flush_tlb_fix_spurious_fault would suffice, but - * the extra traps reduce performance. So, eagerly SFENCE.VMA. - */ - while (nr--) - local_flush_tlb_page(address + nr * PAGE_SIZE); } #define update_mmu_cache(vma, addr, ptep) \ update_mmu_cache_range(NULL, vma, addr, ptep, 1) #define __HAVE_ARCH_UPDATE_MMU_TLB -#define update_mmu_tlb update_mmu_cache +static inline void update_mmu_tlb(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep) +{ + flush_tlb_range(vma, address, address + PAGE_SIZE); +} static inline void update_mmu_cache_pmd(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) { - pte_t *ptep = (pte_t *)pmdp; - - update_mmu_cache(vma, address, ptep); } #define __HAVE_ARCH_PTE_SAME @@ -548,13 +540,26 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty) { - if (!pte_same(*ptep, entry)) + if (!pte_same(*ptep, entry)) { __set_pte_at(ptep, entry); - /* - * update_mmu_cache will unconditionally execute, handling both - * the case that the PTE changed and the spurious fault case. - */ - return true; + /* Here only not svadu is impacted */ + flush_tlb_page(vma, address); + return true; + } + + return false; +} + +extern u64 nr_sfence_vma_handle_exception; +extern bool tlb_caching_invalid_entries; + +#define flush_tlb_fix_spurious_read_fault flush_tlb_fix_spurious_read_fault +static inline void flush_tlb_fix_spurious_read_fault(struct vm_area_struct *vma, + unsigned long address, + pte_t *ptep) +{ + if (tlb_caching_invalid_entries) + flush_tlb_page(vma, address); } #define __HAVE_ARCH_PTEP_GET_AND_CLEAR diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index af7639c3b0a3..7abaf42ef612 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -931,8 +931,12 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) # define pte_accessible(mm, pte) ((void)(pte), 1) #endif -#ifndef flush_tlb_fix_spurious_fault -#define flush_tlb_fix_spurious_fault(vma, address, ptep) flush_tlb_page(vma, address) +#ifndef flush_tlb_fix_spurious_write_fault +#define flush_tlb_fix_spurious_write_fault(vma, address, ptep) flush_tlb_page(vma, address) +#endif + +#ifndef flush_tlb_fix_spurious_read_fault +#define flush_tlb_fix_spurious_read_fault(vma, address, ptep) #endif /* diff --git a/mm/memory.c b/mm/memory.c index 517221f01303..5cb0ccf0c03f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5014,8 +5014,16 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) * with threads. */ if (vmf->flags & FAULT_FLAG_WRITE) - flush_tlb_fix_spurious_fault(vmf->vma, vmf->address, - vmf->pte); + flush_tlb_fix_spurious_write_fault(vmf->vma, vmf->address, + vmf->pte); + else + /* + * With the pte_same(ptep_get(vmf->pte), entry) check + * that calls update_mmu_tlb() above, multiple threads + * faulting at the same time won't get there. + */ + flush_tlb_fix_spurious_read_fault(vmf->vma, vmf->address, + vmf->pte); } unlock: pte_unmap_unlock(vmf->pte, vmf->ptl); From patchwork Thu Dec 7 15:03:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13483415 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8ABA0C10F05 for ; Thu, 7 Dec 2023 15:08:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 21D446B0099; Thu, 7 Dec 2023 10:08:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A5D66B009A; Thu, 7 Dec 2023 10:08:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F3B206B009B; Thu, 7 Dec 2023 10:07:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id DDABE6B0099 for ; Thu, 7 Dec 2023 10:07:59 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 4B1C91C06DA for ; Thu, 7 Dec 2023 15:07:59 +0000 (UTC) X-FDA: 81540352278.20.94EAB43 Received: from mail-lf1-f44.google.com (mail-lf1-f44.google.com [209.85.167.44]) by imf19.hostedemail.com (Postfix) with ESMTP id 52E1D1A0043 for ; Thu, 7 Dec 2023 15:07:57 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=KCZItFAf; dmarc=none; spf=pass (imf19.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.167.44 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701961677; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Copp2ndGZ68t6KATBAuS2R34UMtC0q0chiu6mkWyRdw=; b=0fGw6pCOqm3UU5QoetjJbm/SnHUQ3/H3okgyKMM6Z3YHTHpDUTNzF1hdhKwj78ncPBJA97 E972/KAJ48mL1Q96xqp3Uxs9PY+kgpCKVq3dPy5jVIfIB2Pj1uHDJ3DcHNZAXRfm8bY/jR mfjph5fK2s4Vmcn5aAIQDN54n2F8wqc= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=KCZItFAf; dmarc=none; spf=pass (imf19.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.167.44 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701961677; a=rsa-sha256; cv=none; b=BrQd0rBdDoPulQP24tq5cH8Am6ipCuZkNHlV7i+BbUf5wxk66EaXdynkoA9vSFPORJXrjR khUDZQSbrL5Kiz3hP4p9QM07Zb3zkJgoUaiXtN4u5PE2kzMivIKrHzrXfikdOYNY9/e7ix 8KP7RfI6a9uxMkJuxoPV8vaoDfTSq/g= Received: by mail-lf1-f44.google.com with SMTP id 2adb3069b0e04-50bf3efe2cbso964850e87.2 for ; Thu, 07 Dec 2023 07:07:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1701961675; x=1702566475; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Copp2ndGZ68t6KATBAuS2R34UMtC0q0chiu6mkWyRdw=; b=KCZItFAfcMedYUNMcDa9hIjmsIjQ0VJ3zmm3wDE//ZN383Io8sIz81bCrinEJEWPk1 ctn+L75qsOn+SmYkqj00O4exw5jH12mndlbDDLtCvVmIHuF5H3iUM4RRoK5/2HZLm/mt 4GjOOzqu2zuO/hg2MmFkx6BI2DQLGFSyIBKO5rfnsIy/Hl/hd8zBSXi62vkZb3XyVzUD BZ1pHnrqH7xWglQPuW2AUHgTKxJN7AzfiHWEK0kYl9XCO+9Yz98108ZIa9vVTrmYa/be Kc2jzt9pL2A4QUFlLWGH8I96tDWmz23jjLlLJ41oxR8TQ/KScSxjo+gXFuwEcdk1u/M/ 7AKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701961675; x=1702566475; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Copp2ndGZ68t6KATBAuS2R34UMtC0q0chiu6mkWyRdw=; b=U0A9RmiYrVuFnxEQFJRFJUwgFKc2wVWtLUnS/HS+m2nZx9yCSDuEUuhGh/tndZH25U NurKVWu3OkBNPOPPkgw+eEInMW04xSDfjK2PSVOlcS/Gfdzeu+slxl9MT4WvEemwQHrE PXGQzj002OSh2pVwaJ7BW/E+TuEmQH4vLFJnLWJy0TobZ6/ILjblaD9kQ++ueKwSTLu1 1knNn1LvtCfvIrcEfHs6olmAlUE1T5zMJKfOzTJsZoFnhE9ipVun/wLBnaw+honZlysK ZQA/TOGIC6HKOktYC3tH/T3im76xGV57KLtPBSAT+nDhK14YMO7QdaK+8uRXvs8mnTSP slqQ== X-Gm-Message-State: AOJu0YwBsusQVF/w2WMWrJS5p23q97Xy141RwOqOa8zzTmluAuiBnS/A IwxOp0SoShgakWv1zeMa1pGCpA== X-Google-Smtp-Source: AGHT+IF3ts9ek/mYmRNdXx3GxmP46nrPUW7Zl/dOKYDi965OgtHxqAPAmVkPP3Q02kKv6wkmHjBDJg== X-Received: by 2002:ac2:5147:0:b0:50b:debe:d35d with SMTP id q7-20020ac25147000000b0050bdebed35dmr702137lfd.133.1701961675502; Thu, 07 Dec 2023 07:07:55 -0800 (PST) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id i11-20020a5d438b000000b003332ef77db4sm1647628wrq.44.2023.12.07.07.07.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 07:07:55 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Thomas Bogendoerfer , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , Ved Shanbhogue , Matt Evans , Dylan Jhong , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH RFC/RFT 4/4] TEMP: riscv: Add debugfs interface to retrieve #sfence.vma Date: Thu, 7 Dec 2023 16:03:48 +0100 Message-Id: <20231207150348.82096-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231207150348.82096-1-alexghiti@rivosinc.com> References: <20231207150348.82096-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 52E1D1A0043 X-Stat-Signature: bsa97pbhhaei1sif6o68q788384g6a9a X-Rspam-User: X-HE-Tag: 1701961677-854697 X-HE-Meta: U2FsdGVkX19EWdfsyXJipTPQPfm0QE1V+SbijmFm031WX/1xF95wETeigsFNqOAKqxqPIGYJmwm6zFWku+yI9sYlkqXpDtwSeyY8Ki+N1K2rsO7zGbF08rBXwRq1BCwEtEmN8Xjxvth1NDAG6K6Np9BRIGBWea1/AVcQEh1u79Hb8AIk83twzFMt+qufXDpn21WkZ8ep3bGK0j4FVpmtWXpU2lsLzYYHBaBn2mNhG6TvMz3/6qe7bMztNzn072xclIWkFWg3MEFesma4l/fDBhx6ANkNMBPZwOSWLtYNAW/XXGMrrzuXm2PgwevPuhG9FsjKVW/3QqFVo/hvVIH8z7bxGyrloBrSYEs6Jv32hmyPbnfme9PDKI75YLHJPODRWL9ZdtyJ5bpQo4mSwcM89+bsd7FAzH/6o4Ae/C0ViiCu+AYD5beKHNsituSoF3wmKdaUr0hk83RmRF7oNTkfgE8zg07SepsUdNL8DoRFvJCVJWj72PVyKAzlMYb0nTiT2tXvdSK+PHhIUhxp1g5lEehuG76DeBJdM7v2G+rLSIH3Yhv12ZaVtqzmWagOCQa8eem9nF8hGKnAM0TWv1jNAe+PesiqI984Q/xQVnixy8X/2+5JR3/NN6r64ZEW+kiCZ3/SG7u3krOiwX304Uk+54Sn0kvVbBi7bXifgLCLQZQ24JAV2alBqzvnNJIdyF3J/CXvfuGYyra9E2r4sC85FxTzXHCf5iPXUKuvu/Q45C2svN6qyokFn2kBZ7/bTiWTE4ymassgvqN/GMk1bUTyQo04OOxSPVFmdu2++FJXW9y5TEiPUfq7MmO2vw2PPw0GsrpQ5DpJxrOo63acu14iLeJ/+GdEIk5q59wT2Hllx0mjosv9uqO/kCFQjyvfyYbP6nkAjmvOBTMjaRxpT0E1zNRO5MdcPXJV6sRYHxQdur1ciIbYa0tzb5/q7GZHa+y69AVGe56yXdFI5hL3HnJ p3roB1Ge Zi4WO0gq/B5ux7SKERlXoadPa5t7Oy9X52IP9IFe0Da5WE4dvMYfMueGjn7buaY8XurH8I+WLa29XKKIqX146OQBHU4u68n/IVbiNjp06+l6UcYmCUxe+74UQU1JQC+ZjQ9x1hNtGO0NRPBrYXFLnz2dELXBkp2X+9sD0iZSVhGEgNADb5v2r+J/XrX7TcnelsfhGspg/5QT38q1FKP34CyCKm17k5VYYLrgE7U1rsaWUvpdhERplhXeil1EmNysY4y9poZOXq9ycesJf8B0L7L4sBReZKeeWwxOosRxVbOUYtjS29ta9pifQRlTUNCaN4QmIVrD3/dhAxwGSeYnfanfJHGmNOXMQi1lWBgwalTChebEywk0c2M4X1MPm6Rkj1RnU+jfClFnd/Ien/AzqDbPB7zdqARxQUkfhbKW4+pmHZsexFA4hWUOazpGXZipeCt1ZbLqhwvrdRxDJ+1FvBprgbDyVnhlAy7Z2i3piUbYVg7Y= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is useful for testing/benchmarking. Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/pgtable.h | 6 ++++-- arch/riscv/include/asm/tlbflush.h | 4 ++++ arch/riscv/kernel/sbi.c | 12 ++++++++++++ arch/riscv/mm/tlbflush.c | 17 +++++++++++++++++ 4 files changed, 37 insertions(+), 2 deletions(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 89aa5650f104..b0855a620cfd 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -550,7 +550,7 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, return false; } -extern u64 nr_sfence_vma_handle_exception; +extern u64 nr_sfence_vma_spurious_read; extern bool tlb_caching_invalid_entries; #define flush_tlb_fix_spurious_read_fault flush_tlb_fix_spurious_read_fault @@ -558,8 +558,10 @@ static inline void flush_tlb_fix_spurious_read_fault(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { - if (tlb_caching_invalid_entries) + if (tlb_caching_invalid_entries) { + __sync_fetch_and_add(&nr_sfence_vma_spurious_read, 1UL); flush_tlb_page(vma, address); + } } #define __HAVE_ARCH_PTEP_GET_AND_CLEAR diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index a09196f8de68..f419ec9d2207 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -14,14 +14,18 @@ #ifdef CONFIG_MMU extern unsigned long asid_mask; +extern u64 nr_sfence_vma, nr_sfence_vma_all, nr_sfence_vma_all_asid; + static inline void local_flush_tlb_all(void) { + __sync_fetch_and_add(&nr_sfence_vma_all, 1UL); __asm__ __volatile__ ("sfence.vma" : : : "memory"); } /* Flush one page from local TLB */ static inline void local_flush_tlb_page(unsigned long addr) { + __sync_fetch_and_add(&nr_sfence_vma, 1UL); ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) : "memory")); } #else /* CONFIG_MMU */ diff --git a/arch/riscv/kernel/sbi.c b/arch/riscv/kernel/sbi.c index c672c8ba9a2a..ac1617759583 100644 --- a/arch/riscv/kernel/sbi.c +++ b/arch/riscv/kernel/sbi.c @@ -376,6 +376,8 @@ int sbi_remote_fence_i(const struct cpumask *cpu_mask) } EXPORT_SYMBOL(sbi_remote_fence_i); +extern u64 nr_sfence_vma, nr_sfence_vma_all, nr_sfence_vma_all_asid; + /** * sbi_remote_sfence_vma() - Execute SFENCE.VMA instructions on given remote * harts for the specified virtual address range. @@ -389,6 +391,11 @@ int sbi_remote_sfence_vma(const struct cpumask *cpu_mask, unsigned long start, unsigned long size) { + if (size == (unsigned long)-1) + __sync_fetch_and_add(&nr_sfence_vma_all, 1UL); + else + __sync_fetch_and_add(&nr_sfence_vma, ALIGN(size, PAGE_SIZE) / PAGE_SIZE); + return __sbi_rfence(SBI_EXT_RFENCE_REMOTE_SFENCE_VMA, cpu_mask, start, size, 0, 0); } @@ -410,6 +417,11 @@ int sbi_remote_sfence_vma_asid(const struct cpumask *cpu_mask, unsigned long size, unsigned long asid) { + if (size == (unsigned long)-1) + __sync_fetch_and_add(&nr_sfence_vma_all_asid, 1UL); + else + __sync_fetch_and_add(&nr_sfence_vma, ALIGN(size, PAGE_SIZE) / PAGE_SIZE); + return __sbi_rfence(SBI_EXT_RFENCE_REMOTE_SFENCE_VMA_ASID, cpu_mask, start, size, asid, 0); } diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 77be59aadc73..75a3e2dff16a 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -3,11 +3,16 @@ #include #include #include +#include #include #include +u64 nr_sfence_vma, nr_sfence_vma_all, nr_sfence_vma_all_asid, + nr_sfence_vma_handle_exception, nr_sfence_vma_spurious_read; + static inline void local_flush_tlb_all_asid(unsigned long asid) { + __sync_fetch_and_add(&nr_sfence_vma_all_asid, 1); __asm__ __volatile__ ("sfence.vma x0, %0" : : "r" (asid) @@ -17,6 +22,7 @@ static inline void local_flush_tlb_all_asid(unsigned long asid) static inline void local_flush_tlb_page_asid(unsigned long addr, unsigned long asid) { + __sync_fetch_and_add(&nr_sfence_vma, 1); __asm__ __volatile__ ("sfence.vma %0, %1" : : "r" (addr), "r" (asid) @@ -149,3 +155,14 @@ void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, __flush_tlb_range(vma->vm_mm, start, end - start, PMD_SIZE); } #endif + +static int debugfs_nr_sfence_vma(void) +{ + debugfs_create_u64("nr_sfence_vma", 0444, NULL, &nr_sfence_vma); + debugfs_create_u64("nr_sfence_vma_all", 0444, NULL, &nr_sfence_vma_all); + debugfs_create_u64("nr_sfence_vma_all_asid", 0444, NULL, &nr_sfence_vma_all_asid); + debugfs_create_u64("nr_sfence_vma_handle_exception", 0444, NULL, &nr_sfence_vma_handle_exception); + debugfs_create_u64("nr_sfence_vma_spurious_read", 0444, NULL, &nr_sfence_vma_spurious_read); + return 0; +} +device_initcall(debugfs_nr_sfence_vma);