[v11,00/12] AMD broadcast TLB invalidation

Message ID	20250213161423.449435-1-riel@surriel.com (mailing list archive)
Headers	show Return-Path: <owner-linux-mm@kvack.org> From: Rik van Riel <riel@surriel.com> To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, bp@alien8.de, peterz@infradead.org, dave.hansen@linux.intel.com, zhengqi.arch@bytedance.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, kernel-team@meta.com, linux-mm@kvack.org, akpm@linux-foundation.org, jackmanb@google.com, jannh@google.com, mhklinux@outlook.com, andrew.cooper3@citrix.com Subject: [PATCH v11 00/12] AMD broadcast TLB invalidation Date: Thu, 13 Feb 2025 11:13:51 -0500 Message-ID: <20250213161423.449435-1-riel@surriel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: owner-linux-mm@kvack.org Precedence: bulk
Series	AMD broadcast TLB invalidation \| expand [v11,00/12] AMD broadcast TLB invalidation [v11,01/12] x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional [v11,02/12] x86/mm: remove pv_ops.mmu.tlb_remove_table call [v11,03/12] x86/mm: consolidate full flush threshold decision [v11,04/12] x86/mm: get INVLPGB count max from CPUID [v11,05/12] x86/mm: add INVLPGB support code [v11,06/12] x86/mm: use INVLPGB for kernel TLB flushes [v11,07/12] x86/mm: use INVLPGB in flush_tlb_all [v11,08/12] x86/mm: use broadcast TLB flushing for page reclaim TLB flushing [v11,09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes [v11,10/12] x86/mm: do targeted broadcast flushing from tlbbatch code [v11,11/12] x86/mm: enable AMD translation cache extensions [v11,12/12] x86/mm: only invalidate final translations with INVLPGB

Rik van Riel Feb. 13, 2025, 4:13 p.m. UTC

Add support for broadcast TLB invalidation using AMD's INVLPGB instruction.

This allows the kernel to invalidate TLB entries on remote CPUs without
needing to send IPIs, without having to wait for remote CPUs to handle
those interrupts, and with less interruption to what was running on
those CPUs.

Because x86 PCID space is limited, and there are some very large
systems out there, broadcast TLB invalidation is only used for
processes that are active on 3 or more CPUs, with the threshold
being gradually increased the more the PCID space gets exhausted.

Combined with the removal of unnecessary lru_add_drain calls
(see https://lkml.org/lkml/2024/12/19/1388) this results in a
nice performance boost for the will-it-scale tlb_flush2_threads
test on an AMD Milan system with 36 cores:

- vanilla kernel:           527k loops/second
- lru_add_drain removal:    731k loops/second
- only INVLPGB:             527k loops/second
- lru_add_drain + INVLPGB: 1157k loops/second

Profiling with only the INVLPGB changes showed while
TLB invalidation went down from 40% of the total CPU
time to only around 4% of CPU time, the contention
simply moved to the LRU lock.

Fixing both at the same time about doubles the
number of iterations per second from this case.

Some numbers closer to real world performance
can be found at Phoronix, thanks to Michael:

https://www.phoronix.com/news/AMD-INVLPGB-Linux-Benefits

My current plan is to implement support for Intel's RAR
(Remote Action Request) TLB flushing in a follow-up series,
after this thing has been merged into -tip. Making things
any larger would just be unwieldy for reviewers.

v11:
 - resolve conflict with CONFIG_PT_RECLAIM code
 - a few more cleanups (Peter, Brendan, Nadav)
v10:
 - simplify partial pages with min(nr, 1) in the invlpgb loop (Peter)
 - document x86 paravirt, AMD invlpgb, and ARM64 flush without IPI (Brendan)
 - remove IS_ENABLED(CONFIG_X86_BROADCAST_TLB_FLUSH) (Brendan)
 - various cleanups (Brendan)
v9:
 - print warning when start or end address was rounded (Peter)
 - in the reclaim code, tlbsync at context switch time (Peter)
 - fix !CONFIG_CPU_SUP_AMD compile error in arch_tlbbatch_add_pending (Jan)
v8:
 - round start & end to handle non-page-aligned callers (Steven & Jan)
 - fix up changelog & add tested-by tags (Manali)
v7:
 - a few small code cleanups (Nadav)
 - fix spurious VM_WARN_ON_ONCE in mm_global_asid
 - code simplifications & better barriers (Peter & Dave)
v6:
 - fix info->end check in flush_tlb_kernel_range (Michael)
 - disable broadcast TLB flushing on 32 bit x86
v5:
 - use byte assembly for compatibility with older toolchains (Borislav, Michael)
 - ensure a panic on an invalid number of extra pages (Dave, Tom)
 - add cant_migrate() assertion to tlbsync (Jann)
 - a bunch more cleanups (Nadav)
 - key TCE enabling off X86_FEATURE_TCE (Andrew)
 - fix a race between reclaim and ASID transition (Jann)
v4:
 - Use only bitmaps to track free global ASIDs (Nadav)
 - Improved AMD initialization (Borislav & Tom)
 - Various naming and documentation improvements (Peter, Nadav, Tom, Dave)
 - Fixes for subtle race conditions (Jann)
v3:
 - Remove paravirt tlb_remove_table call (thank you Qi Zheng)
 - More suggested cleanups and changelog fixes by Peter and Nadav
v2:
 - Apply suggestions by Peter and Borislav (thank you!)
 - Fix bug in arch_tlbbatch_flush, where we need to do both
   the TLBSYNC, and flush the CPUs that are in the cpumask.
 - Some updates to comments and changelogs based on questions.

Brendan Jackman Feb. 13, 2025, 6:31 p.m. UTC | #1

On Thu, 13 Feb 2025 at 17:20, Rik van Riel <riel@surriel.com> wrote:
>
> Add support for broadcast TLB invalidation using AMD's INVLPGB instruction.

Oh - what if the user sets nopcid. We need to disable invlpgb in that
case right?

(I assume it could be made to work, but doesn't seem like a case worth
optimising).

Brendan Jackman Feb. 13, 2025, 6:38 p.m. UTC | #2

On Thu, 13 Feb 2025 at 19:31, Brendan Jackman <jackmanb@google.com> wrote:
>
> On Thu, 13 Feb 2025 at 17:20, Rik van Riel <riel@surriel.com> wrote:
> >
> > Add support for broadcast TLB invalidation using AMD's INVLPGB instruction.
>
> Oh - what if the user sets nopcid. We need to disable invlpgb in that
> case right?
>
> (I assume it could be made to work, but doesn't seem like a case worth
> optimising).

BTW there are still a couple of unresolved threads from previous versions.

KPTI comment:

https://lore.kernel.org/linux-kernel/CA+i-1C2zuctxx6oPVVu0zBJ=Q=Hs73mgrWs5jsp8obARNcUS9g@mail.gmail.com/
and
https://lore.kernel.org/linux-kernel/20250212104239.GF19118@noisy.programming.kicks-ass.net/

Clarify context-switch sync:

https://lore.kernel.org/linux-kernel/CA+i-1C2CcQmrbdv0wLUJFD6m85D7LekejU=VdgMxnqZW=DRPxA@mail.gmail.com/

(there's another mention of it somewhere that I can't find)

Suggestion for WARN_ON_ONCE:

https://lore.kernel.org/linux-kernel/CA+i-1C3-F-VAy_JccwKx_AcD1mXsVcGHGwUFvi_ruAiqusiXZQ@mail.gmail.com/

Rik van Riel Feb. 13, 2025, 8:02 p.m. UTC | #3

On Thu, 2025-02-13 at 19:31 +0100, Brendan Jackman wrote:
> On Thu, 13 Feb 2025 at 17:20, Rik van Riel <riel@surriel.com> wrote:
> > 
> > Add support for broadcast TLB invalidation using AMD's INVLPGB
> > instruction.
> 
> Oh - what if the user sets nopcid. We need to disable invlpgb in that
> case right?
> 
That's automatic, when !X86_FEATURE_PCID,
choose_new_asid() always returns 0, and
we never assign a global ASID to a task
on any CPU.

I suppose we could make "nopcid" a little
more efficient by short-circuiting the
code in consider_global_asid() as well,
but it should already work correctly.

Peter, do you prefer a v12, or should
additional small fixes and improvements
just be sent in follow-up patches at
this point?

Peter Zijlstra Feb. 14, 2025, 9:36 a.m. UTC | #4

On Thu, Feb 13, 2025 at 03:02:40PM -0500, Rik van Riel wrote:

> Peter, do you prefer a v12, or should
> additional small fixes and improvements
> just be sent in follow-up patches at
> this point?

I think I'm mostly okay at this point, but DaveH was wanting to have a
look at this thing. Dave did you get around to reading one of these
recent versions, wdyt?

Brendan Jackman Feb. 14, 2025, 9:54 a.m. UTC | #5

On Fri, 14 Feb 2025 at 10:36, Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Thu, Feb 13, 2025 at 03:02:40PM -0500, Rik van Riel wrote:
>
> > Peter, do you prefer a v12, or should
> > additional small fixes and improvements
> > just be sent in follow-up patches at
> > this point?
>
> I think I'm mostly okay at this point, but DaveH was wanting to have a
> look at this thing. Dave did you get around to reading one of these
> recent versions, wdyt?

I was waiting until my nitpicks were resolved but if you merge it
before then, please add my:

Reviewed-by: Brendan Jackman <jackmanb@google.com>

[v11,00/12] AMD broadcast TLB invalidation

Message

Comments