From patchwork Wed Aug 22 15:30:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 10573021 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1311C1390 for ; Wed, 22 Aug 2018 15:46:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 00EBF2B791 for ; Wed, 22 Aug 2018 15:46:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E942E2B7A1; Wed, 22 Aug 2018 15:46:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E08662B79C for ; Wed, 22 Aug 2018 15:46:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F21D76B251D; Wed, 22 Aug 2018 11:46:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E827B6B251E; Wed, 22 Aug 2018 11:46:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD4EB6B251F; Wed, 22 Aug 2018 11:46:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-wm0-f72.google.com (mail-wm0-f72.google.com [74.125.82.72]) by kanga.kvack.org (Postfix) with ESMTP id 6C96D6B251D for ; Wed, 22 Aug 2018 11:46:07 -0400 (EDT) Received: by mail-wm0-f72.google.com with SMTP id s18-v6so2116534wmc.5 for ; Wed, 22 Aug 2018 08:46:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:message-id:user-agent:date:from :to:cc:subject:references:mime-version; bh=gZDohBpEaaBbHawy57xcUUrmp9F4gx1HYZY7ap6aJeI=; b=Pa5rBTCsMuH/cbcczi+PqnBBtVgrsEkPIK3yWGE1fhOekU9CSnIENzCvLfh7OzcQcl CTPlUCcbnfve6IF4G8q9MK0obA7peqjI8Kuh39NtSDluoWzba43IMg1J4qfhusrPTR/l KTlMUNzNrjSyQfeVdfMVMtKSmQ9gL7eEjBGVdMixjfMZa+ptusnEZ7XZKgBETj5Cia9+ 4B1EnBaC/uWszUku4wG7i4zAro/zyjZ0ZoZXaiACtLzwlbpSj2JA8jAwpumNF2D+pQVW EGc+pSC7JF5ryG9sZ/IKBdOlpsNu7s20B1Q5mzbSbGAsp331OoRqAksnZDs3ba4Y3gxP RmZg== X-Gm-Message-State: APzg51Cy4GSd7CGFr8c6iXTpd7aoWsXXorysB+wTlbfBRGHdmSoOgylY hBTzG7Rsu6z8Wkaz98gv3lZ3Qh4Tx7bdorcNepUDFJIXtZ34eQPnEjLkhC34OMAK1UzJvcjrXxQ xfmF+oCayiFvB2LFWuTKTCDZsi3gPpcqNv9aueDsu3Ynlw1F5E4ED1UX7noyRoYuz9A== X-Received: by 2002:adf:a599:: with SMTP id g25-v6mr13292270wrc.88.1534952766375; Wed, 22 Aug 2018 08:46:06 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYy96imSBCvZ82xZ8Bd6Vc47TwaCeCa7GtbHz5JcYg31MGjEJVOIydqpXAJpmfMOjGGpg1W X-Received: by 2002:adf:a599:: with SMTP id g25-v6mr13292220wrc.88.1534952765037; Wed, 22 Aug 2018 08:46:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1534952765; cv=none; d=google.com; s=arc-20160816; b=pxs2xnxv/CsMk7YTrZR86e7v8/TRlgRMtXH0XdT+B0zoimvP50f3zOB/PdtZL9VNKl FnnOtD3tRnlQ2hArv047OVJJxa19dgZAVXoHkVo11Osoriezy3A7iIR/9V9ssFSpWvqF aHNt+Pv2If7a/R7s9E1CqTLXP7s5paiZiMYI6Q4azrq5zmtqwsTWs+MvnGX9T1OVdiyO 7MFV8BNTf1fFh3LFvt4FBvykloAhBJyd8ctZZH7NphZk7SY9M0O7vLcvpofpBsAfULpL oTPrU7MN3Ek1aSil7+ZyuPL5y91LS3//v/nzR/symltxhJgwz5zENrLbEVQLOubF3n9o BMbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:subject:cc:to:from:date:user-agent :message-id:dkim-signature:arc-authentication-results; bh=gZDohBpEaaBbHawy57xcUUrmp9F4gx1HYZY7ap6aJeI=; b=ykNikMfwBiALQ2hn3Pnxvi5McyuqeaCkcTkdfM9a+MZCkIjtaXtKGvIuGFHhnd11gF jV7b3a6hc9UKK4C3dr859ozsUE+q96dbAw41E2eoOSYkKykmqE1n3k58lxfoOWvzhmjC KXNPKSYBVLXSUmuaPKm7v4VrGf6d0VPxpvjtM3y2Dg3R16r5XFCUzIK5Pv97oFlce7tU B+Ri2ZEQPXgUrzHqgCUUZtOpnOibbQ4Dzffr8Pv3r/vZGPE937LXH9tBxdBSz7OFs8E6 zgZUTSzoFxfDEahVtdbHmFkpbYhTbiJTaxvNnZ0VY1TiyjDniI30/zb9XhAeBc09mVRY S7pQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=merlin.20170209 header.b=zKV660Uo; spf=pass (google.com: best guess record for domain of peterz@infradead.org designates 2001:8b0:10b:1231::1 as permitted sender) smtp.mailfrom=peterz@infradead.org Received: from merlin.infradead.org (merlin.infradead.org. [2001:8b0:10b:1231::1]) by mx.google.com with ESMTPS id 7-v6si1707347wri.221.2018.08.22.08.46.04 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 22 Aug 2018 08:46:05 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of peterz@infradead.org designates 2001:8b0:10b:1231::1 as permitted sender) client-ip=2001:8b0:10b:1231::1; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=merlin.20170209 header.b=zKV660Uo; spf=pass (google.com: best guess record for domain of peterz@infradead.org designates 2001:8b0:10b:1231::1 as permitted sender) smtp.mailfrom=peterz@infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=gZDohBpEaaBbHawy57xcUUrmp9F4gx1HYZY7ap6aJeI=; b=zKV660Uo1CQqaWqB6Osuyhtbu4 hLxuBjafWs6Q/FHLdR8GjnFpCtBIlp2w/g45UDGZO1jNRMu4hK1iL1FV3s/TFvycF/MDBphryZfwq R7nmjMDBPSxHSxbCl4MpvajUstEqALqeG5HLHfGYgbshCt3747wlzkzGsJYLVQm0z4tSDI2sD8nA4 Fw2qZBApFOwXrFG3iMLWcURTn7Su8YEo7Exm97Ym0dQyeIKzgy+bzgM4lbDmrJPDVR01KTIb2J6Gy XhTJN8SwxBRbNy7WVIc9ggkH9kaKCaJQCgXafNL/oddQK6LKfh4KQQHVEc1xqZSACgj0wC6RXN6lx mbS9YlCw==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=hirez.programming.kicks-ass.net) by merlin.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1fsVKN-0000WV-LX; Wed, 22 Aug 2018 15:45:48 +0000 Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id BA6F32025D75D; Wed, 22 Aug 2018 17:45:44 +0200 (CEST) Message-ID: <20180822154046.717610121@infradead.org> User-Agent: quilt/0.65 Date: Wed, 22 Aug 2018 17:30:13 +0200 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: peterz@infradead.org, luto@kernel.org, x86@kernel.org, bp@alien8.de, will.deacon@arm.com, riel@surriel.com, jannh@google.com, ascannell@google.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 1/4] x86/mm/tlb: Revert the recent lazy TLB patches References: <20180822153012.173508681@infradead.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Revert commits: 95b0e6357d3e x86/mm/tlb: Always use lazy TLB mode 64482aafe55f x86/mm/tlb: Only send page table free TLB flush to lazy TLB CPUs ac0315896970 x86/mm/tlb: Make lazy TLB mode lazier 61d0beb5796a x86/mm/tlb: Restructure switch_mm_irqs_off() 2ff6ddf19c0e x86/mm/tlb: Leave lazy TLB mode at page table free time In order to simplify the TLB invalidate fixes for x86. We'll try again later. Signed-off-by: Peter Zijlstra (Intel) --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -148,6 +148,22 @@ static inline unsigned long build_cr3_no #define __flush_tlb_one_user(addr) __native_flush_tlb_one_user(addr) #endif +static inline bool tlb_defer_switch_to_init_mm(void) +{ + /* + * If we have PCID, then switching to init_mm is reasonably + * fast. If we don't have PCID, then switching to init_mm is + * quite slow, so we try to defer it in the hopes that we can + * avoid it entirely. The latter approach runs the risk of + * receiving otherwise unnecessary IPIs. + * + * This choice is just a heuristic. The tlb code can handle this + * function returning true or false regardless of whether we have + * PCID. + */ + return !static_cpu_has(X86_FEATURE_PCID); +} + struct tlb_context { u64 ctx_id; u64 tlb_gen; @@ -538,9 +554,4 @@ extern void arch_tlbbatch_flush(struct a native_flush_tlb_others(mask, info) #endif -extern void tlb_flush_remove_tables(struct mm_struct *mm); -extern void tlb_flush_remove_tables_local(void *arg); - -#define HAVE_TLB_FLUSH_REMOVE_TABLES - #endif /* _ASM_X86_TLBFLUSH_H */ --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -7,7 +7,6 @@ #include #include #include -#include #include #include @@ -186,11 +185,8 @@ void switch_mm_irqs_off(struct mm_struct { struct mm_struct *real_prev = this_cpu_read(cpu_tlbstate.loaded_mm); u16 prev_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); - bool was_lazy = this_cpu_read(cpu_tlbstate.is_lazy); unsigned cpu = smp_processor_id(); u64 next_tlb_gen; - bool need_flush; - u16 new_asid; /* * NB: The scheduler will call us with prev == next when switching @@ -244,41 +240,20 @@ void switch_mm_irqs_off(struct mm_struct next->context.ctx_id); /* - * Even in lazy TLB mode, the CPU should stay set in the - * mm_cpumask. The TLB shootdown code can figure out from - * from cpu_tlbstate.is_lazy whether or not to send an IPI. + * We don't currently support having a real mm loaded without + * our cpu set in mm_cpumask(). We have all the bookkeeping + * in place to figure out whether we would need to flush + * if our cpu were cleared in mm_cpumask(), but we don't + * currently use it. */ if (WARN_ON_ONCE(real_prev != &init_mm && !cpumask_test_cpu(cpu, mm_cpumask(next)))) cpumask_set_cpu(cpu, mm_cpumask(next)); - /* - * If the CPU is not in lazy TLB mode, we are just switching - * from one thread in a process to another thread in the same - * process. No TLB flush required. - */ - if (!was_lazy) - return; - - /* - * Read the tlb_gen to check whether a flush is needed. - * If the TLB is up to date, just use it. - * The barrier synchronizes with the tlb_gen increment in - * the TLB shootdown code. - */ - smp_mb(); - next_tlb_gen = atomic64_read(&next->context.tlb_gen); - if (this_cpu_read(cpu_tlbstate.ctxs[prev_asid].tlb_gen) == - next_tlb_gen) - return; - - /* - * TLB contents went out of date while we were in lazy - * mode. Fall through to the TLB switching code below. - */ - new_asid = prev_asid; - need_flush = true; + return; } else { + u16 new_asid; + bool need_flush; u64 last_ctx_id = this_cpu_read(cpu_tlbstate.last_ctx_id); /* @@ -329,41 +304,41 @@ void switch_mm_irqs_off(struct mm_struct next_tlb_gen = atomic64_read(&next->context.tlb_gen); choose_new_asid(next, next_tlb_gen, &new_asid, &need_flush); - } - if (need_flush) { - this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id); - this_cpu_write(cpu_tlbstate.ctxs[new_asid].tlb_gen, next_tlb_gen); - load_new_mm_cr3(next->pgd, new_asid, true); + if (need_flush) { + this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id); + this_cpu_write(cpu_tlbstate.ctxs[new_asid].tlb_gen, next_tlb_gen); + load_new_mm_cr3(next->pgd, new_asid, true); + + /* + * NB: This gets called via leave_mm() in the idle path + * where RCU functions differently. Tracing normally + * uses RCU, so we need to use the _rcuidle variant. + * + * (There is no good reason for this. The idle code should + * be rearranged to call this before rcu_idle_enter().) + */ + trace_tlb_flush_rcuidle(TLB_FLUSH_ON_TASK_SWITCH, TLB_FLUSH_ALL); + } else { + /* The new ASID is already up to date. */ + load_new_mm_cr3(next->pgd, new_asid, false); + + /* See above wrt _rcuidle. */ + trace_tlb_flush_rcuidle(TLB_FLUSH_ON_TASK_SWITCH, 0); + } /* - * NB: This gets called via leave_mm() in the idle path - * where RCU functions differently. Tracing normally - * uses RCU, so we need to use the _rcuidle variant. - * - * (There is no good reason for this. The idle code should - * be rearranged to call this before rcu_idle_enter().) + * Record last user mm's context id, so we can avoid + * flushing branch buffer with IBPB if we switch back + * to the same user. */ - trace_tlb_flush_rcuidle(TLB_FLUSH_ON_TASK_SWITCH, TLB_FLUSH_ALL); - } else { - /* The new ASID is already up to date. */ - load_new_mm_cr3(next->pgd, new_asid, false); + if (next != &init_mm) + this_cpu_write(cpu_tlbstate.last_ctx_id, next->context.ctx_id); - /* See above wrt _rcuidle. */ - trace_tlb_flush_rcuidle(TLB_FLUSH_ON_TASK_SWITCH, 0); + this_cpu_write(cpu_tlbstate.loaded_mm, next); + this_cpu_write(cpu_tlbstate.loaded_mm_asid, new_asid); } - /* - * Record last user mm's context id, so we can avoid - * flushing branch buffer with IBPB if we switch back - * to the same user. - */ - if (next != &init_mm) - this_cpu_write(cpu_tlbstate.last_ctx_id, next->context.ctx_id); - - this_cpu_write(cpu_tlbstate.loaded_mm, next); - this_cpu_write(cpu_tlbstate.loaded_mm_asid, new_asid); - load_mm_cr4(next); switch_ldt(real_prev, next); } @@ -386,7 +361,20 @@ void enter_lazy_tlb(struct mm_struct *mm if (this_cpu_read(cpu_tlbstate.loaded_mm) == &init_mm) return; - this_cpu_write(cpu_tlbstate.is_lazy, true); + if (tlb_defer_switch_to_init_mm()) { + /* + * There's a significant optimization that may be possible + * here. We have accurate enough TLB flush tracking that we + * don't need to maintain coherence of TLB per se when we're + * lazy. We do, however, need to maintain coherence of + * paging-structure caches. We could, in principle, leave our + * old mm loaded and only switch to init_mm when + * tlb_remove_page() happens. + */ + this_cpu_write(cpu_tlbstate.is_lazy, true); + } else { + switch_mm(NULL, &init_mm, NULL); + } } /* @@ -473,9 +461,6 @@ static void flush_tlb_func_common(const * paging-structure cache to avoid speculatively reading * garbage into our TLB. Since switching to init_mm is barely * slower than a minimal flush, just switch to init_mm. - * - * This should be rare, with native_flush_tlb_others skipping - * IPIs to lazy TLB mode CPUs. */ switch_mm_irqs_off(NULL, &init_mm, NULL); return; @@ -582,9 +567,6 @@ static void flush_tlb_func_remote(void * void native_flush_tlb_others(const struct cpumask *cpumask, const struct flush_tlb_info *info) { - cpumask_var_t lazymask; - unsigned int cpu; - count_vm_tlb_event(NR_TLB_REMOTE_FLUSH); if (info->end == TLB_FLUSH_ALL) trace_tlb_flush(TLB_REMOTE_SEND_IPI, TLB_FLUSH_ALL); @@ -608,6 +590,8 @@ void native_flush_tlb_others(const struc * that UV should be updated so that smp_call_function_many(), * etc, are optimal on UV. */ + unsigned int cpu; + cpu = smp_processor_id(); cpumask = uv_flush_tlb_others(cpumask, info); if (cpumask) @@ -615,29 +599,8 @@ void native_flush_tlb_others(const struc (void *)info, 1); return; } - - /* - * A temporary cpumask is used in order to skip sending IPIs - * to CPUs in lazy TLB state, while keeping them in mm_cpumask(mm). - * If the allocation fails, simply IPI every CPU in mm_cpumask. - */ - if (!alloc_cpumask_var(&lazymask, GFP_ATOMIC)) { - smp_call_function_many(cpumask, flush_tlb_func_remote, + smp_call_function_many(cpumask, flush_tlb_func_remote, (void *)info, 1); - return; - } - - cpumask_copy(lazymask, cpumask); - - for_each_cpu(cpu, lazymask) { - if (per_cpu(cpu_tlbstate.is_lazy, cpu)) - cpumask_clear_cpu(cpu, lazymask); - } - - smp_call_function_many(lazymask, flush_tlb_func_remote, - (void *)info, 1); - - free_cpumask_var(lazymask); } /* @@ -690,68 +653,6 @@ void flush_tlb_mm_range(struct mm_struct put_cpu(); } -void tlb_flush_remove_tables_local(void *arg) -{ - struct mm_struct *mm = arg; - - if (this_cpu_read(cpu_tlbstate.loaded_mm) == mm && - this_cpu_read(cpu_tlbstate.is_lazy)) { - /* - * We're in lazy mode. We need to at least flush our - * paging-structure cache to avoid speculatively reading - * garbage into our TLB. Since switching to init_mm is barely - * slower than a minimal flush, just switch to init_mm. - */ - switch_mm_irqs_off(NULL, &init_mm, NULL); - } -} - -static void mm_fill_lazy_tlb_cpu_mask(struct mm_struct *mm, - struct cpumask *lazy_cpus) -{ - int cpu; - - for_each_cpu(cpu, mm_cpumask(mm)) { - if (!per_cpu(cpu_tlbstate.is_lazy, cpu)) - cpumask_set_cpu(cpu, lazy_cpus); - } -} - -void tlb_flush_remove_tables(struct mm_struct *mm) -{ - int cpu = get_cpu(); - cpumask_var_t lazy_cpus; - - if (cpumask_any_but(mm_cpumask(mm), cpu) >= nr_cpu_ids) { - put_cpu(); - return; - } - - if (!zalloc_cpumask_var(&lazy_cpus, GFP_ATOMIC)) { - /* - * If the cpumask allocation fails, do a brute force flush - * on all the CPUs that have this mm loaded. - */ - smp_call_function_many(mm_cpumask(mm), - tlb_flush_remove_tables_local, (void *)mm, 1); - put_cpu(); - return; - } - - /* - * CPUs with !is_lazy either received a TLB flush IPI while the user - * pages in this address range were unmapped, or have context switched - * and reloaded %CR3 since then. - * - * Shootdown IPIs at page table freeing time only need to be sent to - * CPUs that may have out of date TLB contents. - */ - mm_fill_lazy_tlb_cpu_mask(mm, lazy_cpus); - smp_call_function_many(lazy_cpus, - tlb_flush_remove_tables_local, (void *)mm, 1); - free_cpumask_var(lazy_cpus); - put_cpu(); -} static void do_flush_tlb_all(void *info) { --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -303,14 +303,4 @@ static inline void tlb_remove_check_page #define tlb_migrate_finish(mm) do {} while (0) -/* - * Used to flush the TLB when page tables are removed, when lazy - * TLB mode may cause a CPU to retain intermediate translations - * pointing to about-to-be-freed page table memory. - */ -#ifndef HAVE_TLB_FLUSH_REMOVE_TABLES -#define tlb_flush_remove_tables(mm) do {} while (0) -#define tlb_flush_remove_tables_local(mm) do {} while (0) -#endif - #endif /* _ASM_GENERIC__TLB_H */ --- a/mm/memory.c +++ b/mm/memory.c @@ -326,20 +326,16 @@ bool __tlb_remove_page_size(struct mmu_g #ifdef CONFIG_HAVE_RCU_TABLE_FREE +/* + * See the comment near struct mmu_table_batch. + */ + static void tlb_remove_table_smp_sync(void *arg) { - struct mm_struct __maybe_unused *mm = arg; - /* - * On most architectures this does nothing. Simply delivering the - * interrupt is enough to prevent races with software page table - * walking like that done in get_user_pages_fast. - * - * See the comment near struct mmu_table_batch. - */ - tlb_flush_remove_tables_local(mm); + /* Simply deliver the interrupt */ } -static void tlb_remove_table_one(void *table, struct mmu_gather *tlb) +static void tlb_remove_table_one(void *table) { /* * This isn't an RCU grace period and hence the page-tables cannot be @@ -348,7 +344,7 @@ static void tlb_remove_table_one(void *t * It is however sufficient for software page-table walkers that rely on * IRQ disabling. See the comment near struct mmu_table_batch. */ - smp_call_function(tlb_remove_table_smp_sync, tlb->mm, 1); + smp_call_function(tlb_remove_table_smp_sync, NULL, 1); __tlb_remove_table(table); } @@ -369,8 +365,6 @@ void tlb_table_flush(struct mmu_gather * { struct mmu_table_batch **batch = &tlb->batch; - tlb_flush_remove_tables(tlb->mm); - if (*batch) { call_rcu_sched(&(*batch)->rcu, tlb_remove_table_rcu); *batch = NULL; @@ -393,7 +387,7 @@ void tlb_remove_table(struct mmu_gather if (*batch == NULL) { *batch = (struct mmu_table_batch *)__get_free_page(GFP_NOWAIT | __GFP_NOWARN); if (*batch == NULL) { - tlb_remove_table_one(table, tlb); + tlb_remove_table_one(table); return; } (*batch)->nr = 0; From patchwork Wed Aug 22 15:30:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 10573025 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 736B617E0 for ; Wed, 22 Aug 2018 15:47:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 63F2E2B791 for ; Wed, 22 Aug 2018 15:47:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 57BAE2B7AA; Wed, 22 Aug 2018 15:47:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D98DD2B791 for ; Wed, 22 Aug 2018 15:46:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 004396B2521; Wed, 22 Aug 2018 11:46:59 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id ECF5D6B2522; Wed, 22 Aug 2018 11:46:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D70AB6B2523; Wed, 22 Aug 2018 11:46:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-wm0-f71.google.com (mail-wm0-f71.google.com [74.125.82.71]) by kanga.kvack.org (Postfix) with ESMTP id 7ACA46B2521 for ; Wed, 22 Aug 2018 11:46:58 -0400 (EDT) Received: by mail-wm0-f71.google.com with SMTP id m129-v6so2086860wma.8 for ; Wed, 22 Aug 2018 08:46:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:message-id:user-agent:date:from :to:cc:subject:references:mime-version; bh=DtDo4cQQlYpJctQYVAg287e1LDGKHg+PXr5W1nBz53Q=; b=o2jm7dgGNjlprlmklYPK0kCblDyzNnKjXXbl8Iuofy3wGsdnKEtjuyMzOYuEzkjEqc /9uLO0TLphy6Cy0XPsyBCCccd3Y/8Wfk8tWlFVmL6idU81OXN4lPCGraeww+492vVoJH QfNZm26IBADGA6rhmmoOwwfnRK7wynD0m79ucIJj+RSGcAOUSFTKL/kwllVvU79/Ap0x 4nLPm7VSaag3tRiZil5Z2Zig1vJL4pz6XaxSLWKbfVIYfkGsS7HQo4uyB3uPWfqC5+X+ jcttuwZA3WaSfl/k7fHrv/bylp+IeeUYPqrjQwMMvS5GN7G3P2PtMhdCvHcNVh8j90uh 8kVg== X-Gm-Message-State: APzg51DrHBrx/VZkzg96OPKS3Xx6GielRJJNcAHcYCqSZ9L0PwNRJqPP 9gW49qTd3xMwx1b3UdyaFl5LOPjDdNndiGIM2HIrs40/SoT2JX0niRCDDOsrSvtzoHIV4yzRG7K StZMIUU6HjlOAk/d/AaWhe6mPFos370Dwvp5LnYG5LWAAmjH0IaqXg9iyg15DINowHg== X-Received: by 2002:a5d:4c45:: with SMTP id n5-v6mr10345766wrt.71.1534952818010; Wed, 22 Aug 2018 08:46:58 -0700 (PDT) X-Google-Smtp-Source: ANB0Vda+rnGr1RkWh0K4Asb3a0cXysXbi5dp4cdehMayKVzRzDqe5w5UHwp49FaXvmCy61GMXPBz X-Received: by 2002:a5d:4c45:: with SMTP id n5-v6mr10345742wrt.71.1534952817341; Wed, 22 Aug 2018 08:46:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1534952817; cv=none; d=google.com; s=arc-20160816; b=H/e8bRJOtj5fnx1F7I8i5WMHkZe+b1eUFUqbXv5rX2fCHEjFIvpeH1Zxdg4TwIAykH OQNOoNNGpkkGO2Oi0TupqjNSXyFmGprj5MsWhMG+dYoXiqOvgTdPvdKcCOL/1/kLHEcv P26OLlarzVmHkRAPXjEOV2hzU/BeRhzbfzW2Hn1KJyM5Ds/h+lz1XZPwNIQV6/bwpgT9 qJ2TgJxb6QsYSqfCrirKt5O3KPcodkOdnxiJ9dBquTdnJ35ueE/PF8fTYPOQ9elIVMk5 28eIEbNiLSdx2FCRLcEVk4LbKpCkfs/cyqAU3sbaZbBbcTMH29UZZ1CEWNgqzD5f3zRp u/jQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:subject:cc:to:from:date:user-agent :message-id:dkim-signature:arc-authentication-results; bh=DtDo4cQQlYpJctQYVAg287e1LDGKHg+PXr5W1nBz53Q=; b=pOhdpyBJUelPKQwYM7LUbs7ScEUXtyv1OY0wvCRk1830MIjnokMb13U53M8LnekHG0 n21He4nHGhKqS+a29Aw91nqkTyWW7hmNswBIYcH8nzm5IeXP7FMikRENCgIwKfytJqBW DS4zn+MZdFGRnSgLvfsV5VGHqR0S570F6THUWHDa3ioNWxbjNrs0KWVG/2+0a/Q5A9uJ KpPF1VOy3f/AojoUDCXT+xRgWvglbNb7/yXtKaUKsR/3boXC34+1WgYFJ6Yx+BRQjint zmy/3/ULs1iKUmVgw8up1vQwYdl3PmCLWDUtUkIH+o8zcEhrdEwK6UuC0k5uqvzBkPJ8 WXuQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=merlin.20170209 header.b=AOQ8g0mF; spf=pass (google.com: best guess record for domain of peterz@infradead.org designates 2001:8b0:10b:1231::1 as permitted sender) smtp.mailfrom=peterz@infradead.org Received: from merlin.infradead.org (merlin.infradead.org. [2001:8b0:10b:1231::1]) by mx.google.com with ESMTPS id 64-v6si1699618wrf.122.2018.08.22.08.46.57 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 22 Aug 2018 08:46:57 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of peterz@infradead.org designates 2001:8b0:10b:1231::1 as permitted sender) client-ip=2001:8b0:10b:1231::1; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=merlin.20170209 header.b=AOQ8g0mF; spf=pass (google.com: best guess record for domain of peterz@infradead.org designates 2001:8b0:10b:1231::1 as permitted sender) smtp.mailfrom=peterz@infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=DtDo4cQQlYpJctQYVAg287e1LDGKHg+PXr5W1nBz53Q=; b=AOQ8g0mFgDBL61alQiOuAyiDxh qqZq25A0DVaTD3yaYdSTSnEDjNmpw0WdTTRf42Yi9IMBX8j/O8lpk+AayLo6D9dDe+iH5JhnxZ493 n6VTifk+gF+9jwc27RckF1whb4ZVc+IeVJG8kM9iDdrgQm+ziYX/zSh/eevlYCzpJehiC0/bRWylx CijMfNi6t4TBllQhy81e3Q/O3jHPowDI55bNlkEQL3vX9BOVDqW0YR/7Q5fd5r+R7Sh1HL/pIxxjl 9NDmWTQcaIu8MgUCci7y134gKJUW+NxpVCNo+KYqZdwHk88xq56AM1Fu6KzbbA3NBBZ8GGuwFTalr ZOjj6vsg==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=hirez.programming.kicks-ass.net) by merlin.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1fsVKN-0000WT-Ig; Wed, 22 Aug 2018 15:45:47 +0000 Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id BD93D2024D723; Wed, 22 Aug 2018 17:45:44 +0200 (CEST) Message-ID: <20180822154046.772017055@infradead.org> User-Agent: quilt/0.65 Date: Wed, 22 Aug 2018 17:30:14 +0200 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: peterz@infradead.org, luto@kernel.org, x86@kernel.org, bp@alien8.de, will.deacon@arm.com, riel@surriel.com, jannh@google.com, ascannell@google.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Nicholas Piggin , David Miller , Martin Schwidefsky , Michael Ellerman Subject: [PATCH 2/4] mm/tlb: Remove tlb_remove_table() non-concurrent condition References: <20180822153012.173508681@infradead.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Will noted that only checking mm_users is incorrect; we should also check mm_count in order to cover CPUs that have a lazy reference to this mm (and could do speculative TLB operations). If removing this turns out to be a performance issue, we can re-instate a more complete check, but in tlb_table_flush() eliding the call_rcu_sched(). Cc: stable@kernel.org Cc: Nicholas Piggin Cc: David Miller Cc: Will Deacon Cc: Martin Schwidefsky Cc: Michael Ellerman Fixes: 267239116987 ("mm, powerpc: move the RCU page-table freeing into generic code") Reported-by: Will Deacon Signed-off-by: Peter Zijlstra (Intel) Acked-by: Will Deacon --- mm/memory.c | 9 --------- 1 file changed, 9 deletions(-) --- a/mm/memory.c +++ b/mm/memory.c @@ -375,15 +375,6 @@ void tlb_remove_table(struct mmu_gather { struct mmu_table_batch **batch = &tlb->batch; - /* - * When there's less then two users of this mm there cannot be a - * concurrent page-table walk. - */ - if (atomic_read(&tlb->mm->mm_users) < 2) { - __tlb_remove_table(table); - return; - } - if (*batch == NULL) { *batch = (struct mmu_table_batch *)__get_free_page(GFP_NOWAIT | __GFP_NOWARN); if (*batch == NULL) { From patchwork Wed Aug 22 15:30:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 10573019 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 90385112E for ; Wed, 22 Aug 2018 15:46:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7F7D92B783 for ; Wed, 22 Aug 2018 15:46:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7333B2B79C; Wed, 22 Aug 2018 15:46:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E51032B783 for ; Wed, 22 Aug 2018 15:46:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 733FA6B251A; Wed, 22 Aug 2018 11:46:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6BF606B251B; Wed, 22 Aug 2018 11:46:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 498A76B251C; Wed, 22 Aug 2018 11:46:02 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id F1CEF6B251A for ; Wed, 22 Aug 2018 11:46:01 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id j15-v6so1375542pfi.10 for ; Wed, 22 Aug 2018 08:46:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:message-id:user-agent:date:from :to:cc:subject:references:mime-version; bh=zUKxfxjpgWzZcjbYNf+Wl4oxw1QTfZ/a5hMhTBA6A2c=; b=pGDlnUD899YzcAY/ONAfg1BM+tJqMbe4UcOVjn2zWYHG/yK+Q9kzBKJaFreWE+xzO5 312doScnSZl5GTjfEYN0Z03ukY0n+92BQoJF4dGvU4jbyYnAj9GBL9vZXMWv0JhkpdNT qyFvTwejauaOnq29gC2F4nPxV1UFmAw/9Z7ocQ58OR7RBgo8Va79i7gIYTpMfDKBzLrl O3X5xqKFcLKkqAzpSY5rKuHUYtVmyaw8VnyKteBNvzW4hfzuorzd1jspziBO66O/K/+K uCSQkJw7ltCuah/msMkkTfed8P0qEWTqWmPXm1Wr63qwXtpNMcwMIcz1GiqckPBmwKs4 rQtA== X-Gm-Message-State: AOUpUlHXdjBaxX/UBVDOU4P7jib/qpRgUM8FtowpoWZaHacKzf7UFhXX +nHa5PihWBfb8B3wI81ELngFK7MYq3wdVYSbl/xG9g9wT9Qli7XEqbbPXXo05FQM7seZNNoiIaD Jp0fr0O/ZevvE/7AGG9wP0lLuifgM0shHDSdjwk4lDzgfjD6hph3K5T0SjphGU1egLA== X-Received: by 2002:a63:91:: with SMTP id 139-v6mr21042234pga.389.1534952761654; Wed, 22 Aug 2018 08:46:01 -0700 (PDT) X-Google-Smtp-Source: AA+uWPyBHP4zPa9e64XNAfyqiYSZZchdA4cURrJP6E5FYaSJf4/5l93HWsi5yYAu9+ForLoeuEeB X-Received: by 2002:a63:91:: with SMTP id 139-v6mr21042185pga.389.1534952760841; Wed, 22 Aug 2018 08:46:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1534952760; cv=none; d=google.com; s=arc-20160816; b=m6Pg7nUn2ynWddhEgqj4oCNS4Eq8HBiwGy9zS/yLjn5qLn2i0+oCkPRhmq3aNmDSbe hCyEJRvdO5ZbERhdQK+IkX5F3KqSl2LvFK8cOGF5GKMEerCNps58OrBVihMrL5BfxMjC 8N62WOZIkTE86+dTq1Ck41Ap1m8AE5no7Qrxtwz6cHoutUOC7wNYvWCrH3U3vxUKo6D9 3ZoFXrTBgOIRpt+5dLIHQ7EIkXqrCSnUFi7+gvE0RUCMSZxLiSI+4Pr34REIvA524DBW PsS8QYHOO1IRzXONhBtwFRF4vUpmZNiF3aS+ffQ9k08mGPSJFHokxzUwCTBH7q0BGs2U ATkA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:subject:cc:to:from:date:user-agent :message-id:dkim-signature:arc-authentication-results; bh=zUKxfxjpgWzZcjbYNf+Wl4oxw1QTfZ/a5hMhTBA6A2c=; b=BwSc04sy22Kg/pUOFnlUlFHA0J7MyApBTKlqLv+iFdRmdoQNLfimFsY1R1RQ3QyN42 a4XkX6qC4VkUROFQqvmTa6XDLETK/7HbiSE0cYmrySXuu/0JOt9a2bIzJS9Q0F5bX/Kw hS6B91MGWDn3xmG6d2JvFdgywS2OYjYQ57DvJY97FyhLtLeygDPZK0ng38y7dNb5GGuF AWVqNS+4kVeYcCO5mMV5TNk1joapDMdq8RBfI6b1jp+YxNRKKRN/547591qEo5/gbP7X +yWV4NFZ6lv8CxoxZ2NRg7JiWeQ0qIzrJ6aQQ5Fc/Jh0wrgiAd6hFEN2qBDkc+LF5hOx 1v3w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=bombadil.20170209 header.b=LihpBYe8; spf=pass (google.com: best guess record for domain of peterz@infradead.org designates 2607:7c80:54:e::133 as permitted sender) smtp.mailfrom=peterz@infradead.org Received: from bombadil.infradead.org (bombadil.infradead.org. [2607:7c80:54:e::133]) by mx.google.com with ESMTPS id w7-v6si1842346plq.198.2018.08.22.08.46.00 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 22 Aug 2018 08:46:00 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of peterz@infradead.org designates 2607:7c80:54:e::133 as permitted sender) client-ip=2607:7c80:54:e::133; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=bombadil.20170209 header.b=LihpBYe8; spf=pass (google.com: best guess record for domain of peterz@infradead.org designates 2607:7c80:54:e::133 as permitted sender) smtp.mailfrom=peterz@infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=zUKxfxjpgWzZcjbYNf+Wl4oxw1QTfZ/a5hMhTBA6A2c=; b=LihpBYe80MmgqB361ej9IoCREs /vL0G5e1PRqx3qNsuISx1zrXxTXiwALJu3yr//2iVpu6gFCuZk0FVc5JqI/LLGa0AnyCNUDhaLQT3 VkkNNdswlRc8kYqTusppEvH8gakOYOTvLrxneAyZVTu5X8U5hicLVNSp6ElFMGr6Mmtls06D+ybiy K2Fd6Y9uFC2CpTlckZzCAtAZUZXkodQz/hQLG3ZaJfvgpgP7QCaGi6ijjI9l9wj3Op5XZHFZVQ1/P S1k9FIfD/VLv6KZSJZ8bANA4+yQ3V4DYguO8M/Cv+kgBe5Xd0O5/v2CqEFbFcZnfX9QoVDAB28b+G fk0zJHHw==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=hirez.programming.kicks-ass.net) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1fsVKM-0004RB-E1; Wed, 22 Aug 2018 15:45:46 +0000 Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id C2A5C2024D725; Wed, 22 Aug 2018 17:45:44 +0200 (CEST) Message-ID: <20180822154046.823850812@infradead.org> User-Agent: quilt/0.65 Date: Wed, 22 Aug 2018 17:30:15 +0200 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: peterz@infradead.org, luto@kernel.org, x86@kernel.org, bp@alien8.de, will.deacon@arm.com, riel@surriel.com, jannh@google.com, ascannell@google.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Nicholas Piggin , David Miller , Martin Schwidefsky , Michael Ellerman Subject: [PATCH 3/4] mm/tlb, x86/mm: Support invalidating TLB caches for RCU_TABLE_FREE References: <20180822153012.173508681@infradead.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Jann reported that x86 was missing required TLB invalidates when he hit the !*batch slow path in tlb_remove_table(). This is indeed the case; RCU_TABLE_FREE does not provide TLB (cache) invalidates, the PowerPC-hash where this code originated and the Sparc-hash where this was subsequently used did not need that. ARM which later used this put an explicit TLB invalidate in their __p*_free_tlb() functions, and PowerPC-radix followed that example. But when we hooked up x86 we failed to consider this. Fix this by (optionally) hooking tlb_remove_table() into the TLB invalidate code. NOTE: s390 was also needing something like this and might now be able to use the generic code again. Cc: stable@kernel.org Cc: Nicholas Piggin Cc: David Miller Cc: Will Deacon Cc: Martin Schwidefsky Cc: Michael Ellerman Fixes: 9e52fc2b50de ("x86/mm: Enable RCU based page table freeing (CONFIG_HAVE_RCU_TABLE_FREE=y)") Reported-by: Jann Horn Signed-off-by: Peter Zijlstra (Intel) --- arch/Kconfig | 3 +++ arch/x86/Kconfig | 1 + mm/memory.c | 27 +++++++++++++++++++++++++-- 3 files changed, 29 insertions(+), 2 deletions(-) --- a/arch/Kconfig +++ b/arch/Kconfig @@ -362,6 +362,9 @@ config HAVE_ARCH_JUMP_LABEL config HAVE_RCU_TABLE_FREE bool +config HAVE_RCU_TABLE_INVALIDATE + bool + config ARCH_HAVE_NMI_SAFE_CMPXCHG bool --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -180,6 +180,7 @@ config X86 select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP select HAVE_RCU_TABLE_FREE + select HAVE_RCU_TABLE_INVALIDATE if HAVE_RCU_TABLE_FREE select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RELIABLE_STACKTRACE if X86_64 && (UNWINDER_FRAME_POINTER || UNWINDER_ORC) && STACK_VALIDATION select HAVE_STACKPROTECTOR if CC_HAS_SANE_STACKPROTECTOR --- a/mm/memory.c +++ b/mm/memory.c @@ -238,17 +238,22 @@ void arch_tlb_gather_mmu(struct mmu_gath __tlb_reset_range(tlb); } -static void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) +static void __tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) { if (!tlb->end) return; tlb_flush(tlb); mmu_notifier_invalidate_range(tlb->mm, tlb->start, tlb->end); + __tlb_reset_range(tlb); +} + +static void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) +{ + __tlb_flush_mmu_tlbonly(tlb); #ifdef CONFIG_HAVE_RCU_TABLE_FREE tlb_table_flush(tlb); #endif - __tlb_reset_range(tlb); } static void tlb_flush_mmu_free(struct mmu_gather *tlb) @@ -330,6 +335,21 @@ bool __tlb_remove_page_size(struct mmu_g * See the comment near struct mmu_table_batch. */ +/* + * If we want tlb_remove_table() to imply TLB invalidates. + */ +static inline void tlb_table_invalidate(struct mmu_gather *tlb) +{ +#ifdef CONFIG_HAVE_RCU_TABLE_INVALIDATE + /* + * Invalidate page-table caches used by hardware walkers. Then we still + * need to RCU-sched wait while freeing the pages because software + * walkers can still be in-flight. + */ + __tlb_flush_mmu_tlbonly(tlb); +#endif +} + static void tlb_remove_table_smp_sync(void *arg) { /* Simply deliver the interrupt */ @@ -366,6 +386,7 @@ void tlb_table_flush(struct mmu_gather * struct mmu_table_batch **batch = &tlb->batch; if (*batch) { + tlb_table_invalidate(tlb); call_rcu_sched(&(*batch)->rcu, tlb_remove_table_rcu); *batch = NULL; } @@ -378,11 +399,13 @@ void tlb_remove_table(struct mmu_gather if (*batch == NULL) { *batch = (struct mmu_table_batch *)__get_free_page(GFP_NOWAIT | __GFP_NOWARN); if (*batch == NULL) { + tlb_table_invalidate(tlb); tlb_remove_table_one(table); return; } (*batch)->nr = 0; } + (*batch)->tables[(*batch)->nr++] = table; if ((*batch)->nr == MAX_TABLE_BATCH) tlb_table_flush(tlb); From patchwork Wed Aug 22 15:30:16 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 10573023 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5AA8A1390 for ; Wed, 22 Aug 2018 15:46:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 49E192B783 for ; Wed, 22 Aug 2018 15:46:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3C4562B79C; Wed, 22 Aug 2018 15:46:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8A53B2B783 for ; Wed, 22 Aug 2018 15:46:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1177E6B251E; Wed, 22 Aug 2018 11:46:09 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0A0496B251F; Wed, 22 Aug 2018 11:46:09 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E5B936B2520; Wed, 22 Aug 2018 11:46:08 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-wm0-f70.google.com (mail-wm0-f70.google.com [74.125.82.70]) by kanga.kvack.org (Postfix) with ESMTP id 894C96B251E for ; Wed, 22 Aug 2018 11:46:08 -0400 (EDT) Received: by mail-wm0-f70.google.com with SMTP id y18-v6so2068565wma.9 for ; Wed, 22 Aug 2018 08:46:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:message-id:user-agent:date:from :to:cc:subject:references:mime-version; bh=gBeaEcB0CRaE9iudyXRMqxnJyX6OHIKa2QWqyXhblwA=; b=KFFln+2v5Wg69GTtytOoG9LG7xmIRuz3iW9LxJaBe/D5558+k1AJsOws0euW3tiTyE GqY/r5O8P2JSyiv+visEhKN/xiMZ8ekRtUW62Bxyrtg19f13PMWgDF9Cccz9cGo3mffR 3lZaZH+d0PhDtL0Ev07zauWCjfYuT8Why/AlNO65VlAbYqMwHPSHXbIEdCzN4gMijAj0 SNutEhVOoc51xWG04x9+FWVXXNXgQUoZ0gAxVdd1gORjKbza6kz9Sm2fzVXWnp0A1Su7 7wVISi535cCVr6dkYMLa4QHenVyUcDFk9Ym1UwSEao2HdhiNy0wnZ3FEsvSkGHMSTW0n XOmg== X-Gm-Message-State: APzg51CSf4sjkg3S3x2qhPJCnbmztjfMUQvfW+ghz4pqjvxrcFvreXy3 RTtwkW7LL9sznjP5cWh9fJinEk4QJ1f0445JMS2ZWhKLBZZ2pE1kvKFCYu9o8ijgL3hFCoFSegf 3VeXDHRHDWFCMe4J93f0EHbO7MB2Yket0hnbkdyF1nzjAudDyZlLekDJ6wKq+Pg3Y/A== X-Received: by 2002:a1c:e146:: with SMTP id y67-v6mr2617333wmg.108.1534952768071; Wed, 22 Aug 2018 08:46:08 -0700 (PDT) X-Google-Smtp-Source: ANB0VdaG58KKlg9vK8yEE3FqCsLnuNd3BW6hIf1Efb9W/kHjoCZ1x0fCy0YyrdbVJUJQ3fYVQpb4 X-Received: by 2002:a1c:e146:: with SMTP id y67-v6mr2617295wmg.108.1534952767179; Wed, 22 Aug 2018 08:46:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1534952767; cv=none; d=google.com; s=arc-20160816; b=jb4WnKOdZEOYZhX32Bbzc9XSqfaIL/IiRYbFgSv8c09dmhbsfgUnwQV2mxr8u3v4D+ 3oo1YAH9G9cySlRhPzaWAcQQe1VkPn1Qd23N+HAVF6mZfkBGq+fmF8FceLbXAqTwiKRl 8sXTAr/iqh54qqJ0RL4QaHuK9dSp2tkDX0sGtRgU/O+M9Lx8mNLKzqrFTC/nI8DhwDCP tWZL8ELQCCr06rHcuysGrglXS/mJQxekhRXhcVl07QGV/Blvck5c13swUBKB6x6vY8Ud U/8p/QZb2+BzwpeW/5ZiH4nHyteBFQmarevFh1m4osm/TouPjiNt/Wb70Z0W0+MzVqK1 7TXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:subject:cc:to:from:date:user-agent :message-id:dkim-signature:arc-authentication-results; bh=gBeaEcB0CRaE9iudyXRMqxnJyX6OHIKa2QWqyXhblwA=; b=Lt5xVOWStKpvdRYQ1nxDYwAn/TfcvJe+dxd5niRJKLg1Lc/5zwcT6NT7SYzcpn6bOk ZsBgN6CYoASueAcf/qBJRIv7sq8JHSaI4pzZfoCcrF76l35V/V3OoQb9ATMcR5Au02b5 /QZ8F9+hkkyCvaskE9R0eIrg7CtB2qVi4dWCh6JcBYVQas45nI7HR4IzRHPaQ4r0o7+p sDdKhzJirZWyY3EtH10k2TWzI5xZ26CLmkqkcmjGoGUjBsVyHcDqlaCqVH/7347Pg4ns vTUo2uRDbyxMqEF13mDG3CfjGeCtyfPQ5/RDYFUGSmeE8CzlaDxewxn2xNACumn4Pg+l hqLg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=merlin.20170209 header.b=p8G9EifM; spf=pass (google.com: best guess record for domain of peterz@infradead.org designates 2001:8b0:10b:1231::1 as permitted sender) smtp.mailfrom=peterz@infradead.org Received: from merlin.infradead.org (merlin.infradead.org. [2001:8b0:10b:1231::1]) by mx.google.com with ESMTPS id w73-v6si1662086wmw.15.2018.08.22.08.46.07 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 22 Aug 2018 08:46:07 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of peterz@infradead.org designates 2001:8b0:10b:1231::1 as permitted sender) client-ip=2001:8b0:10b:1231::1; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=merlin.20170209 header.b=p8G9EifM; spf=pass (google.com: best guess record for domain of peterz@infradead.org designates 2001:8b0:10b:1231::1 as permitted sender) smtp.mailfrom=peterz@infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=gBeaEcB0CRaE9iudyXRMqxnJyX6OHIKa2QWqyXhblwA=; b=p8G9EifMms1fnFzIqWxv4jM6D0 0B4AEcLzmWE82yYl6NkQstHOzRaDl6nz148SfhDGz2POBDPUnTAsFeykKD15z9nlHnl3UPh4STzDz /IvBYWeIjwOLR2oH53h0kGxmGAFj5Xa6jRNvfNOaqqrhI0br6H3XbCckmWOd7dxwjWxwuDcmmhWEX xSDrnlqpqemTVvJNSZGc21VZjQXWL55S+9DTWiL53eJUr/FgkOZEWzcgTw0PiO35IB+YVmYf7RbFL bv68dVEqmevxLaN2YagmWBfXpkVAK6PL+iC9Ja20Ms1U52qXQvkSpjJt7XmBbzcr4KeSi3zBf2xks zJsIiR3A==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=hirez.programming.kicks-ass.net) by merlin.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1fsVKN-0000WU-K1; Wed, 22 Aug 2018 15:45:47 +0000 Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id C627D2024D726; Wed, 22 Aug 2018 17:45:44 +0200 (CEST) Message-ID: <20180822154046.877071284@infradead.org> User-Agent: quilt/0.65 Date: Wed, 22 Aug 2018 17:30:16 +0200 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: peterz@infradead.org, luto@kernel.org, x86@kernel.org, bp@alien8.de, will.deacon@arm.com, riel@surriel.com, jannh@google.com, ascannell@google.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 4/4] x86/mm: Only use tlb_remove_table() for paravirt References: <20180822153012.173508681@infradead.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP If we don't use paravirt; don't play unnecessary and complicated games to free page-tables. Suggested-by: Linus Torvalds Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/Kconfig | 2 +- arch/x86/hyperv/mmu.c | 2 ++ arch/x86/include/asm/paravirt.h | 5 +++++ arch/x86/include/asm/paravirt_types.h | 3 +++ arch/x86/include/asm/tlbflush.h | 3 +++ arch/x86/kernel/kvm.c | 5 ++++- arch/x86/kernel/paravirt.c | 2 ++ arch/x86/mm/pgtable.c | 8 ++++---- arch/x86/xen/mmu_pv.c | 1 + 9 files changed, 25 insertions(+), 6 deletions(-) --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -179,7 +179,7 @@ config X86 select HAVE_HARDLOCKUP_DETECTOR_PERF if PERF_EVENTS && HAVE_PERF_EVENTS_NMI select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP - select HAVE_RCU_TABLE_FREE + select HAVE_RCU_TABLE_FREE if PARAVIRT select HAVE_RCU_TABLE_INVALIDATE if HAVE_RCU_TABLE_FREE select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RELIABLE_STACKTRACE if X86_64 && (UNWINDER_FRAME_POINTER || UNWINDER_ORC) && STACK_VALIDATION --- a/arch/x86/hyperv/mmu.c +++ b/arch/x86/hyperv/mmu.c @@ -9,6 +9,7 @@ #include #include #include +#include #define CREATE_TRACE_POINTS #include @@ -231,4 +232,5 @@ void hyperv_setup_mmu_ops(void) pr_info("Using hypercall for remote TLB flush\n"); pv_mmu_ops.flush_tlb_others = hyperv_flush_tlb_others; + pv_mmu_ops.tlb_remove_table = tlb_remove_table; } --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -309,6 +309,11 @@ static inline void flush_tlb_others(cons PVOP_VCALL2(pv_mmu_ops.flush_tlb_others, cpumask, info); } +static inline void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table) +{ + PVOP_VCALL2(pv_mmu_ops.tlb_remove_table, tlb, table); +} + static inline int paravirt_pgd_alloc(struct mm_struct *mm) { return PVOP_CALL1(int, pv_mmu_ops.pgd_alloc, mm); --- a/arch/x86/include/asm/paravirt_types.h +++ b/arch/x86/include/asm/paravirt_types.h @@ -54,6 +54,7 @@ struct desc_struct; struct task_struct; struct cpumask; struct flush_tlb_info; +struct mmu_gather; /* * Wrapper type for pointers to code which uses the non-standard @@ -222,6 +223,8 @@ struct pv_mmu_ops { void (*flush_tlb_others)(const struct cpumask *cpus, const struct flush_tlb_info *info); + void (*tlb_remove_table)(struct mmu_gather *tlb, void *table); + /* Hooks for allocating and freeing a pagetable top-level */ int (*pgd_alloc)(struct mm_struct *mm); void (*pgd_free)(struct mm_struct *mm, pgd_t *pgd); --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -552,6 +552,9 @@ extern void arch_tlbbatch_flush(struct a #ifndef CONFIG_PARAVIRT #define flush_tlb_others(mask, info) \ native_flush_tlb_others(mask, info) + +#define paravirt_tlb_remove_table(tlb, page) \ + tlb_remove_page(tlb, (void *)(page)) #endif #endif /* _ASM_X86_TLBFLUSH_H */ --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -45,6 +45,7 @@ #include #include #include +#include static int kvmapf = 1; @@ -636,8 +637,10 @@ static void __init kvm_guest_init(void) if (kvm_para_has_feature(KVM_FEATURE_PV_TLB_FLUSH) && !kvm_para_has_hint(KVM_HINTS_REALTIME) && - kvm_para_has_feature(KVM_FEATURE_STEAL_TIME)) + kvm_para_has_feature(KVM_FEATURE_STEAL_TIME)) { pv_mmu_ops.flush_tlb_others = kvm_flush_tlb_others; + pv_mmu_ops.tlb_remove_table = tlb_remove_table; + } if (kvm_para_has_feature(KVM_FEATURE_PV_EOI)) apic_set_eoi_write(kvm_guest_apic_eoi_write); --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -41,6 +41,7 @@ #include #include #include +#include /* * nop stub, which must not clobber anything *including the stack* to @@ -409,6 +410,7 @@ struct pv_mmu_ops pv_mmu_ops __ro_after_ .flush_tlb_kernel = native_flush_tlb_global, .flush_tlb_one_user = native_flush_tlb_one_user, .flush_tlb_others = native_flush_tlb_others, + .tlb_remove_table = (void (*)(struct mmu_gather *, void *))tlb_remove_page, .pgd_alloc = __paravirt_pgd_alloc, .pgd_free = paravirt_nop, --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -63,7 +63,7 @@ void ___pte_free_tlb(struct mmu_gather * { pgtable_page_dtor(pte); paravirt_release_pte(page_to_pfn(pte)); - tlb_remove_table(tlb, pte); + paravirt_tlb_remove_table(tlb, pte); } #if CONFIG_PGTABLE_LEVELS > 2 @@ -79,21 +79,21 @@ void ___pmd_free_tlb(struct mmu_gather * tlb->need_flush_all = 1; #endif pgtable_pmd_page_dtor(page); - tlb_remove_table(tlb, page); + paravirt_tlb_remove_table(tlb, page); } #if CONFIG_PGTABLE_LEVELS > 3 void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); - tlb_remove_table(tlb, virt_to_page(pud)); + paravirt_tlb_remove_table(tlb, virt_to_page(pud)); } #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); - tlb_remove_table(tlb, virt_to_page(p4d)); + paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } #endif /* CONFIG_PGTABLE_LEVELS > 4 */ #endif /* CONFIG_PGTABLE_LEVELS > 3 */ --- a/arch/x86/xen/mmu_pv.c +++ b/arch/x86/xen/mmu_pv.c @@ -2397,6 +2397,7 @@ static const struct pv_mmu_ops xen_mmu_o .flush_tlb_kernel = xen_flush_tlb, .flush_tlb_one_user = xen_flush_tlb_one_user, .flush_tlb_others = xen_flush_tlb_others, + .tlb_remove_table = tlb_remove_table, .pgd_alloc = xen_pgd_alloc, .pgd_free = xen_pgd_free,