From patchwork Sat Jun 5 01:42:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12301127 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E764C47082 for ; Sat, 5 Jun 2021 04:46:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E5B266120F for ; Sat, 5 Jun 2021 04:46:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E5B266120F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3C3346B0036; Sat, 5 Jun 2021 00:46:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 34C116B006C; Sat, 5 Jun 2021 00:46:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1780C6B006E; Sat, 5 Jun 2021 00:46:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0118.hostedemail.com [216.40.44.118]) by kanga.kvack.org (Postfix) with ESMTP id D136B6B0036 for ; Sat, 5 Jun 2021 00:46:56 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 6D065ABEF for ; Sat, 5 Jun 2021 04:46:56 +0000 (UTC) X-FDA: 78218435232.02.6EC82AB Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177]) by imf16.hostedemail.com (Postfix) with ESMTP id 9F29E801937E for ; Sat, 5 Jun 2021 04:46:55 +0000 (UTC) Received: by mail-oi1-f177.google.com with SMTP id x196so11510244oif.10 for ; Fri, 04 Jun 2021 21:46:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ecwzSsruDCg5HQ2i3NCpkmbXtWN83qkbLVUmQqDCWlk=; b=K84jdMCLYtlUEgBdPd1FB73VBjM6s0HtzHQUEKQuqVbWUPLu6QNiGSyAiZKtAnSXZI qa4yEvXbNmIMTYUakiYMlhfQjzIRQyLNh9Jrtbve9f/zthPJ5CoYO/U1vCsbE68/oQo3 GqQ4ijTqbXVuk2mA1T65SdOnvXOHd72h0KEHCwafBFdOtVmnSxI6Nf2iyCXf+O+7QEyR 4o/pLi9n+6oidBSD8apK3ir/f7FzIJ/jZctGYehvbOABo4abVcwekvo8LzyZA/cgWtzR XCGf6kZDt3iIatDop/2zSlU9nQRfONbTF6CCJaHYWykAvm3j3mJ3esA/tPCsbIgWOq7U orFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ecwzSsruDCg5HQ2i3NCpkmbXtWN83qkbLVUmQqDCWlk=; b=REkNl9y84+2ZWepO1DwFZTVJiQ9nKRJ72+UuVEfnFO+/I+Q2znyZA0O6al42jTZf8l IiQNM3zpzu2LB0F33lgDeQhkvWXXWTrl9tw2LZ/10Wjf2fGO2RqYteh61CPhSO6XZT8D +iC6W7rrNNbRcrMxT68woka5wIgX+losRkdW8SQiarKboT9HypIlqaH66Gw5Vug+YGxU NjbKPBbThbWFUdnCtWuwQ0nEY6uwPJR7Ojid9AHFHOUXxZpKiauOkyQCoOpBgqbWxT28 pp+3FcsDyVQuS5oYla/aUQNH3XcnlxldoLzJN6LMFXNKg4AtaVtWK6HnsxZc5pjI7Wob 17Pw== X-Gm-Message-State: AOAM532tNI1zuf5L+T0XVLsjLpf0yXTcBn8zB3agYwNcza5m4uPcknC4 Jr742/O8MIWVFCVItQkaJJlux14KOwE= X-Google-Smtp-Source: ABdhPJxLEzbHIWZWXXi4Xnd53BEeXOfGW7hwSRiz+7W58BsNagoAhoouQQKJuY6psVrgcnmdrfBJRg== X-Received: by 2002:a17:90a:6343:: with SMTP id v3mr19806273pjs.61.1622857350102; Fri, 04 Jun 2021 18:42:30 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (60-242-147-73.tpgi.com.au. [60.242.147.73]) by smtp.gmail.com with ESMTPSA id q68sm5779056pjq.45.2021.06.04.18.42.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Jun 2021 18:42:29 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Randy Dunlap , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, Anton Blanchard , Andy Lutomirski Subject: [PATCH v4 1/4] lazy tlb: introduce lazy mm refcount helper functions Date: Sat, 5 Jun 2021 11:42:13 +1000 Message-Id: <20210605014216.446867-2-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210605014216.446867-1-npiggin@gmail.com> References: <20210605014216.446867-1-npiggin@gmail.com> MIME-Version: 1.0 Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=K84jdMCL; spf=pass (imf16.hostedemail.com: domain of npiggin@gmail.com designates 209.85.167.177 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 9F29E801937E X-Stat-Signature: 79ote7zhn8zre6kig4fz1ekpdexr1mdu X-HE-Tag: 1622868415-587129 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add explicit _lazy_tlb annotated functions for lazy mm refcounting. This makes lazy mm references more obvious, and allows explicit refcounting to be removed if it is not used. Signed-off-by: Nicholas Piggin Signed-off-by: Nicholas Piggin --- arch/arm/mach-rpc/ecard.c | 2 +- arch/powerpc/kernel/smp.c | 2 +- arch/powerpc/mm/book3s64/radix_tlb.c | 4 ++-- fs/exec.c | 4 ++-- include/linux/sched/mm.h | 11 +++++++++++ kernel/cpu.c | 2 +- kernel/exit.c | 2 +- kernel/kthread.c | 11 +++++++---- kernel/sched/core.c | 15 ++++++++------- 9 files changed, 34 insertions(+), 19 deletions(-) diff --git a/arch/arm/mach-rpc/ecard.c b/arch/arm/mach-rpc/ecard.c index 827b50f1c73e..1b4a41aad793 100644 --- a/arch/arm/mach-rpc/ecard.c +++ b/arch/arm/mach-rpc/ecard.c @@ -253,7 +253,7 @@ static int ecard_init_mm(void) current->mm = mm; current->active_mm = mm; activate_mm(active_mm, mm); - mmdrop(active_mm); + mmdrop_lazy_tlb(active_mm); ecard_init_pgtables(mm); return 0; } diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c index 2e05c783440a..fb0bdfc67366 100644 --- a/arch/powerpc/kernel/smp.c +++ b/arch/powerpc/kernel/smp.c @@ -1541,7 +1541,7 @@ void start_secondary(void *unused) { unsigned int cpu = raw_smp_processor_id(); - mmgrab(&init_mm); + mmgrab_lazy_tlb(&init_mm); current->active_mm = &init_mm; smp_store_cpu_info(cpu); diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 409e61210789..2962082787c0 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -663,10 +663,10 @@ void exit_lazy_flush_tlb(struct mm_struct *mm, bool always_flush) if (current->active_mm == mm) { WARN_ON_ONCE(current->mm != NULL); /* Is a kernel thread and is using mm as the lazy tlb */ - mmgrab(&init_mm); + mmgrab_lazy_tlb(&init_mm); current->active_mm = &init_mm; switch_mm_irqs_off(mm, &init_mm, current); - mmdrop(mm); + mmdrop_lazy_tlb(mm); } /* diff --git a/fs/exec.c b/fs/exec.c index 18594f11c31f..ca0f8b1af23a 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -1029,9 +1029,9 @@ static int exec_mmap(struct mm_struct *mm) setmax_mm_hiwater_rss(&tsk->signal->maxrss, old_mm); mm_update_next_owner(old_mm); mmput(old_mm); - return 0; + } else { + mmdrop_lazy_tlb(active_mm); } - mmdrop(active_mm); return 0; } diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index e24b1fe348e3..bfd1baca5266 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -49,6 +49,17 @@ static inline void mmdrop(struct mm_struct *mm) __mmdrop(mm); } +/* Helpers for lazy TLB mm refcounting */ +static inline void mmgrab_lazy_tlb(struct mm_struct *mm) +{ + mmgrab(mm); +} + +static inline void mmdrop_lazy_tlb(struct mm_struct *mm) +{ + mmdrop(mm); +} + /** * mmget() - Pin the address space associated with a &struct mm_struct. * @mm: The address space to pin. diff --git a/kernel/cpu.c b/kernel/cpu.c index e538518556f4..e87a89824e6c 100644 --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -602,7 +602,7 @@ static int finish_cpu(unsigned int cpu) */ if (mm != &init_mm) idle->active_mm = &init_mm; - mmdrop(mm); + mmdrop_lazy_tlb(mm); return 0; } diff --git a/kernel/exit.c b/kernel/exit.c index fd1c04193e18..8e87ec5f6be2 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -476,7 +476,7 @@ static void exit_mm(void) __set_current_state(TASK_RUNNING); mmap_read_lock(mm); } - mmgrab(mm); + mmgrab_lazy_tlb(mm); BUG_ON(mm != current->active_mm); /* more a memory barrier than a real lock */ task_lock(current); diff --git a/kernel/kthread.c b/kernel/kthread.c index fe3f2a40d61e..b70e28431a01 100644 --- a/kernel/kthread.c +++ b/kernel/kthread.c @@ -1314,14 +1314,14 @@ void kthread_use_mm(struct mm_struct *mm) WARN_ON_ONCE(!(tsk->flags & PF_KTHREAD)); WARN_ON_ONCE(tsk->mm); + mmgrab(mm); + task_lock(tsk); /* Hold off tlb flush IPIs while switching mm's */ local_irq_disable(); active_mm = tsk->active_mm; - if (active_mm != mm) { - mmgrab(mm); + if (active_mm != mm) tsk->active_mm = mm; - } tsk->mm = mm; membarrier_update_current_mm(mm); switch_mm_irqs_off(active_mm, mm, tsk); @@ -1341,7 +1341,7 @@ void kthread_use_mm(struct mm_struct *mm) * mmdrop(), or explicitly with smp_mb(). */ if (active_mm != mm) - mmdrop(active_mm); + mmdrop_lazy_tlb(active_mm); else smp_mb(); @@ -1375,10 +1375,13 @@ void kthread_unuse_mm(struct mm_struct *mm) local_irq_disable(); tsk->mm = NULL; membarrier_update_current_mm(NULL); + mmgrab_lazy_tlb(mm); /* active_mm is still 'mm' */ enter_lazy_tlb(mm, tsk); local_irq_enable(); task_unlock(tsk); + + mmdrop(mm); } EXPORT_SYMBOL_GPL(kthread_unuse_mm); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 5226cc26a095..e359c76ea2e2 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4229,13 +4229,14 @@ static struct rq *finish_task_switch(struct task_struct *prev) * rq->curr, before returning to userspace, so provide them here: * * - a full memory barrier for {PRIVATE,GLOBAL}_EXPEDITED, implicitly - * provided by mmdrop(), + * provided by mmdrop_lazy_tlb(), * - a sync_core for SYNC_CORE. */ if (mm) { membarrier_mm_sync_core_before_usermode(mm); - mmdrop(mm); + mmdrop_lazy_tlb(mm); } + if (unlikely(prev_state == TASK_DEAD)) { if (prev->sched_class->task_dead) prev->sched_class->task_dead(prev); @@ -4299,9 +4300,9 @@ context_switch(struct rq *rq, struct task_struct *prev, /* * kernel -> kernel lazy + transfer active - * user -> kernel lazy + mmgrab() active + * user -> kernel lazy + mmgrab_lazy_tlb() active * - * kernel -> user switch + mmdrop() active + * kernel -> user switch + mmdrop_lazy_tlb() active * user -> user switch */ if (!next->mm) { // to kernel @@ -4309,7 +4310,7 @@ context_switch(struct rq *rq, struct task_struct *prev, next->active_mm = prev->active_mm; if (prev->mm) // from user - mmgrab(prev->active_mm); + mmgrab_lazy_tlb(prev->active_mm); else prev->active_mm = NULL; } else { // to user @@ -4325,7 +4326,7 @@ context_switch(struct rq *rq, struct task_struct *prev, switch_mm_irqs_off(prev->active_mm, next->mm, next); if (!prev->mm) { // from kernel - /* will mmdrop() in finish_task_switch(). */ + /* will mmdrop_lazy_tlb() in finish_task_switch(). */ rq->prev_mm = prev->active_mm; prev->active_mm = NULL; } @@ -8239,7 +8240,7 @@ void __init sched_init(void) /* * The boot idle thread does lazy MMU switching as well: */ - mmgrab(&init_mm); + mmgrab_lazy_tlb(&init_mm); enter_lazy_tlb(&init_mm, current); /* From patchwork Sat Jun 5 01:42:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12301137 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2695DC4743E for ; Sat, 5 Jun 2021 05:18:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B200F6138C for ; Sat, 5 Jun 2021 05:18:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B200F6138C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4FF796B0036; Sat, 5 Jun 2021 01:18:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D60A6B006C; Sat, 5 Jun 2021 01:18:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 376C36B006E; Sat, 5 Jun 2021 01:18:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0235.hostedemail.com [216.40.44.235]) by kanga.kvack.org (Postfix) with ESMTP id 0790C6B0036 for ; Sat, 5 Jun 2021 01:18:19 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id A09449982 for ; Sat, 5 Jun 2021 05:18:19 +0000 (UTC) X-FDA: 78218514318.26.2345A0E Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf05.hostedemail.com (Postfix) with ESMTP id 655D4E0004EC for ; Sat, 5 Jun 2021 05:18:18 +0000 (UTC) Received: by mail-pl1-f176.google.com with SMTP id v12so5692490plo.10 for ; Fri, 04 Jun 2021 22:18:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=W8uxX4t8g+DxGYDGoM+1rG8fWNFKPCHDobEAy7P79JU=; b=t/BsozX+HKFZD2zebtgF3JGrIwCTcVj4ffmsJERRbi/MNaFuO3WFo3bmI7M7Pw7IDj 5k7ho9NFtodvy2hED8NQ029kOKEcm+emddzxclcV+A/9uFGxWtrrqyVwZBqBZQayVv8o uvzpSo3qbCEaRJsIsMf83kmaSWU82D3nQyk4UoworXY/8rBuy0UT3gajZ92D9fUUpVvL nkRMAM3yjYbObExBGVtCTrrn8upzvtdmLXXteWz7/0ii5TkWeGOPk21w9hgHzbsJtd4n 8a+HzVHQmMqiendjAtp1CC0CeIVqy3BZs61INz/EoX+TblkTnx0ukJD5mdWH0Ta+yWQq 0Sng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=W8uxX4t8g+DxGYDGoM+1rG8fWNFKPCHDobEAy7P79JU=; b=CgXcf2l+Xicqa6oOsiNqC5w2SEH4LeXBVTEuinQmg0+utroyT/4ZDc3kuk9164X7UF 0qf+2H5f/wzVy04ikenIXDLN5UZU2evzVJYVea/wlHlUQ4j+rie7l39bEM/Gili/ql2B YpbtzbpJsULIAOpCwMhVDgSTd97vr4rNoerFFDr4LcGa85rorTCqq1ZHVoxu9aF4FWL3 cdd7Lh53Oj1L73SK3WlAzDQH1oCd3jD2JO894BEeLQvmqDgVOiYeLdmAfA0qFkMEiTZI lhj73Q9vtWBw5Uci1RrB5f5mwH9+NOTPP2fSZIqCj7doQO3Bn68+Icd+nSMaHyRfe2Qi oStQ== X-Gm-Message-State: AOAM532n5LbWeCTyjzoGMzbYz7MoKenhF9CDa8I+S1WzLMPZ8iWKkWiT WCxOdGu9kOkHX2Mr0md14waIBrDmDpc= X-Google-Smtp-Source: ABdhPJyaiJ+cifU73dwUkfKSCUG8g6xUPympTQc18g/8gDitQusrenWMupQEWYNpZsq0HZ+33JFCOQ== X-Received: by 2002:aa7:900f:0:b029:2ec:82d2:d23 with SMTP id m15-20020aa7900f0000b02902ec82d20d23mr5855353pfo.16.1622857354192; Fri, 04 Jun 2021 18:42:34 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (60-242-147-73.tpgi.com.au. [60.242.147.73]) by smtp.gmail.com with ESMTPSA id q68sm5779056pjq.45.2021.06.04.18.42.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Jun 2021 18:42:33 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Randy Dunlap , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, Anton Blanchard , Andy Lutomirski Subject: [PATCH v4 2/4] lazy tlb: allow lazy tlb mm refcounting to be configurable Date: Sat, 5 Jun 2021 11:42:14 +1000 Message-Id: <20210605014216.446867-3-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210605014216.446867-1-npiggin@gmail.com> References: <20210605014216.446867-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 655D4E0004EC Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b="t/BsozX+"; spf=pass (imf05.hostedemail.com: domain of npiggin@gmail.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: n9yyphwbgm8fom7qops3msdiqs14mpwp X-HE-Tag: 1622870298-6057 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add CONFIG_MMU_TLB_REFCOUNT which enables refcounting of the lazy tlb mm when it is context switched. This can be disabled by architectures that don't require this refcounting if they clean up lazy tlb mms when the last refcount is dropped. Currently this is always enabled, which is what existing code does, so the patch is effectively a no-op. Rename rq->prev_mm to rq->prev_lazy_mm, because that's what it is. Signed-off-by: Nicholas Piggin Signed-off-by: Nicholas Piggin --- arch/Kconfig | 4 ++++ include/linux/sched/mm.h | 13 +++++++++++-- kernel/sched/core.c | 22 ++++++++++++++++++---- kernel/sched/sched.h | 4 +++- 4 files changed, 36 insertions(+), 7 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index c45b770d3579..1cff045cdde6 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -418,6 +418,10 @@ config ARCH_WANT_IRQS_OFF_ACTIVATE_MM irqs disabled over activate_mm. Architectures that do IPI based TLB shootdowns should enable this. +# Use normal mm refcounting for MMU_LAZY_TLB kernel thread references. +config MMU_LAZY_TLB_REFCOUNT + def_bool y + config ARCH_HAVE_NMI_SAFE_CMPXCHG bool diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index bfd1baca5266..29e4638ad124 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -52,12 +52,21 @@ static inline void mmdrop(struct mm_struct *mm) /* Helpers for lazy TLB mm refcounting */ static inline void mmgrab_lazy_tlb(struct mm_struct *mm) { - mmgrab(mm); + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) + mmgrab(mm); } static inline void mmdrop_lazy_tlb(struct mm_struct *mm) { - mmdrop(mm); + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) { + mmdrop(mm); + } else { + /* + * mmdrop_lazy_tlb must provide a full memory barrier, see the + * membarrier comment finish_task_switch which relies on this. + */ + smp_mb(); + } } /** diff --git a/kernel/sched/core.c b/kernel/sched/core.c index e359c76ea2e2..5e10cb712be3 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4171,7 +4171,7 @@ static struct rq *finish_task_switch(struct task_struct *prev) __releases(rq->lock) { struct rq *rq = this_rq(); - struct mm_struct *mm = rq->prev_mm; + struct mm_struct *mm = NULL; long prev_state; /* @@ -4190,7 +4190,10 @@ static struct rq *finish_task_switch(struct task_struct *prev) current->comm, current->pid, preempt_count())) preempt_count_set(FORK_PREEMPT_COUNT); - rq->prev_mm = NULL; +#ifdef CONFIG_MMU_LAZY_TLB_REFCOUNT + mm = rq->prev_lazy_mm; + rq->prev_lazy_mm = NULL; +#endif /* * A task struct has one reference for the use as "current". @@ -4326,9 +4329,20 @@ context_switch(struct rq *rq, struct task_struct *prev, switch_mm_irqs_off(prev->active_mm, next->mm, next); if (!prev->mm) { // from kernel - /* will mmdrop_lazy_tlb() in finish_task_switch(). */ - rq->prev_mm = prev->active_mm; +#ifdef CONFIG_MMU_LAZY_TLB_REFCOUNT + /* Will mmdrop_lazy_tlb() in finish_task_switch(). */ + rq->prev_lazy_mm = prev->active_mm; prev->active_mm = NULL; +#else + /* + * Without MMU_LAZY_TLB_REFCOUNT there is no lazy + * tracking (because no rq->prev_lazy_mm) in + * finish_task_switch, so no mmdrop_lazy_tlb(), so no + * memory barrier for membarrier (see the membarrier + * comment in finish_task_switch()). Do it here. + */ + smp_mb(); +#endif } } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index a189bec13729..0729cf19a987 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -961,7 +961,9 @@ struct rq { struct task_struct *idle; struct task_struct *stop; unsigned long next_balance; - struct mm_struct *prev_mm; +#ifdef CONFIG_MMU_LAZY_TLB_REFCOUNT + struct mm_struct *prev_lazy_mm; +#endif unsigned int clock_update_flags; u64 clock; From patchwork Sat Jun 5 01:42:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12301125 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A9D3C47082 for ; Sat, 5 Jun 2021 04:34:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B8A706141B for ; Sat, 5 Jun 2021 04:34:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B8A706141B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5A6E96B006C; Sat, 5 Jun 2021 00:34:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 57D7C6B006E; Sat, 5 Jun 2021 00:34:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 41E666B0070; Sat, 5 Jun 2021 00:34:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0123.hostedemail.com [216.40.44.123]) by kanga.kvack.org (Postfix) with ESMTP id 11C076B006C for ; Sat, 5 Jun 2021 00:34:11 -0400 (EDT) Received: from smtpin32.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id A16B4180ACEE8 for ; Sat, 5 Jun 2021 04:34:10 +0000 (UTC) X-FDA: 78218403060.32.AF51C2A Received: from mail-io1-f50.google.com (mail-io1-f50.google.com [209.85.166.50]) by imf04.hostedemail.com (Postfix) with ESMTP id EBF1599F for ; Sat, 5 Jun 2021 04:34:08 +0000 (UTC) Received: by mail-io1-f50.google.com with SMTP id p66so10469039iod.8 for ; Fri, 04 Jun 2021 21:34:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=o79xAv8VGlL9P0cnIMgEnd0kABY+0aVXPPf79qRUn38=; b=EfkqsrLHsB7OMvz7oioK9BBbvfX//VN0Qw/rqIoO6VTLg+nNlnZ/KwQq3Ak+aOlAJ3 4S1x1y+m8CXRSw/uKOh3GHNLGFovqdRQUIFX0Dcq6pC7OE+yBjpBJpGqqn+CRcKbBPAY 4mnCAWNE1DNFy3YE0B8updP/ljldtNDies0KZ25LEesppi4qXSwnW2uLmd/t/dWRN9Ia Jk1T9dWv2C5leuoyrJr5BQMj4Z9qUj/tleWND28vrxEqsj8DZ2aXJfQL9kYMwI7CfeIl Yzl9twL/atYxeXecU4ZgZwgwq00za2QJ/RHuxgZxyIRXUCMIvtLRYq7hcluVMYREzs8J MBPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=o79xAv8VGlL9P0cnIMgEnd0kABY+0aVXPPf79qRUn38=; b=QoWs7l/iFSnKyeZrYvc2HU+rxhe8qNP4lj2aLqJDZe+UZ8OJLzzjYcnN6SlvFnocp8 Vj0YF2twlNO9uhrfcezaBndWqbXe9xuERA8CXvBy6qkJ6UlsshbPMLcRhpdeumGZvJSx 7W/HE8HNfT/1YsYmPk4B7H4FmBrGzk5NdRD2ipAiohXqBc2PtgBvX2SfGPoro2MOMvKx f96eAVaGrXx+b6OpmmtQBsxiQ7h2/Axa5qH/gCj7kQZC9sJCuqZndp+pypCOBmPRj1S5 NnR/PgPxgwUecEUk6ZigqjGrXstkorj/v0oXCNUynTmgODhNiHZ39iaE55O1yMw4DKbR KiGQ== X-Gm-Message-State: AOAM531j7MzzxuvroZqWIiElE+Bc1Of4IZPgeQiYDDoEExlrOOkGsP9q t5xPVAVDX49SdM+EBdU/+Z5XcrTrXMI= X-Google-Smtp-Source: ABdhPJxId09rwktS+wz23oEw5BLQX2VX0E1Io1PESo6MYBadnFs6NbNcFmDkmt3wim2iUcQ6z1nzfw== X-Received: by 2002:a63:1626:: with SMTP id w38mr7676348pgl.420.1622857358356; Fri, 04 Jun 2021 18:42:38 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (60-242-147-73.tpgi.com.au. [60.242.147.73]) by smtp.gmail.com with ESMTPSA id q68sm5779056pjq.45.2021.06.04.18.42.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Jun 2021 18:42:38 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Randy Dunlap , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, Anton Blanchard , Andy Lutomirski Subject: [PATCH v4 3/4] lazy tlb: shoot lazies, a non-refcounting lazy tlb option Date: Sat, 5 Jun 2021 11:42:15 +1000 Message-Id: <20210605014216.446867-4-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210605014216.446867-1-npiggin@gmail.com> References: <20210605014216.446867-1-npiggin@gmail.com> MIME-Version: 1.0 Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=EfkqsrLH; spf=pass (imf04.hostedemail.com: domain of npiggin@gmail.com designates 209.85.166.50 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: EBF1599F X-Stat-Signature: sfrhyy4kz7qm7inqooc1ix6y9mipn7wj X-HE-Tag: 1622867648-798749 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On big systems, the mm refcount can become highly contented when doing a lot of context switching with threaded applications (particularly switching between the idle thread and an application thread). Abandoning lazy tlb slows switching down quite a bit in the important user->idle->user cases, so instead implement a non-refcounted scheme that causes __mmdrop() to IPI all CPUs in the mm_cpumask and shoot down any remaining lazy ones. Shootdown IPIs are some concern, but they have not been observed to be a big problem with this scheme (the powerpc implementation generated 314 additional interrupts on a 144 CPU system during a kernel compile). There are a number of strategies that could be employed to reduce IPIs if they turn out to be a problem for some workload. Signed-off-by: Nicholas Piggin Signed-off-by: Nicholas Piggin --- arch/Kconfig | 13 +++++++++++++ kernel/fork.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+) diff --git a/arch/Kconfig b/arch/Kconfig index 1cff045cdde6..f8136c893991 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -421,6 +421,19 @@ config ARCH_WANT_IRQS_OFF_ACTIVATE_MM # Use normal mm refcounting for MMU_LAZY_TLB kernel thread references. config MMU_LAZY_TLB_REFCOUNT def_bool y + depends on !MMU_LAZY_TLB_SHOOTDOWN + +# Instead of refcounting the lazy mm struct for kernel thread references +# (which can cause contention with multi-threaded apps on large multiprocessor +# systems), this option causes __mmdrop to IPI all CPUs in the mm_cpumask and +# switch to init_mm if they were using the to-be-freed mm as the lazy tlb. To +# implement this, architectures must use _lazy_tlb variants of mm refcounting +# when releasing kernel thread mm references, and mm_cpumask must include at +# least all possible CPUs in which the mm might be lazy, at the time of the +# final mmdrop. mmgrab/mmdrop in arch/ code must be switched to _lazy_tlb +# postfix as necessary. +config MMU_LAZY_TLB_SHOOTDOWN + bool config ARCH_HAVE_NMI_SAFE_CMPXCHG bool diff --git a/kernel/fork.c b/kernel/fork.c index dc06afd725cb..8085ff33c7f6 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -674,6 +674,53 @@ static void check_mm(struct mm_struct *mm) #define allocate_mm() (kmem_cache_alloc(mm_cachep, GFP_KERNEL)) #define free_mm(mm) (kmem_cache_free(mm_cachep, (mm))) +static void do_shoot_lazy_tlb(void *arg) +{ + struct mm_struct *mm = arg; + + if (current->active_mm == mm) { + WARN_ON_ONCE(current->mm); + current->active_mm = &init_mm; + switch_mm(mm, &init_mm, current); + } +} + +static void do_check_lazy_tlb(void *arg) +{ + struct mm_struct *mm = arg; + + WARN_ON_ONCE(current->active_mm == mm); +} + +static void shoot_lazy_tlbs(struct mm_struct *mm) +{ + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_SHOOTDOWN)) { + /* + * IPI overheads have not found to be expensive, but they could + * be reduced in a number of possible ways, for example (in + * roughly increasing order of complexity): + * - A batch of mms requiring IPIs could be gathered and freed + * at once. + * - CPUs could store their active mm somewhere that can be + * remotely checked without a lock, to filter out + * false-positives in the cpumask. + * - After mm_users or mm_count reaches zero, switching away + * from the mm could clear mm_cpumask to reduce some IPIs + * (some batching or delaying would help). + * - A delayed freeing and RCU-like quiescing sequence based on + * mm switching to avoid IPIs completely. + */ + on_each_cpu_mask(mm_cpumask(mm), do_shoot_lazy_tlb, (void *)mm, 1); + if (IS_ENABLED(CONFIG_DEBUG_VM)) + on_each_cpu(do_check_lazy_tlb, (void *)mm, 1); + } else { + /* + * In this case, lazy tlb mms are refounted and would not reach + * __mmdrop until all CPUs have switched away and mmdrop()ed. + */ + } +} + /* * Called when the last reference to the mm * is dropped: either by a lazy thread or by @@ -683,6 +730,10 @@ void __mmdrop(struct mm_struct *mm) { BUG_ON(mm == &init_mm); WARN_ON_ONCE(mm == current->mm); + + /* Ensure no CPUs are using this as their lazy tlb mm */ + shoot_lazy_tlbs(mm); + WARN_ON_ONCE(mm == current->active_mm); mm_free_pgd(mm); destroy_context(mm); From patchwork Sat Jun 5 01:42:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12301121 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE4DAC47082 for ; Sat, 5 Jun 2021 04:28:30 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 34F246140F for ; Sat, 5 Jun 2021 04:28:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 34F246140F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5FA396B0036; Sat, 5 Jun 2021 00:28:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5AAB46B006C; Sat, 5 Jun 2021 00:28:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 425286B006E; Sat, 5 Jun 2021 00:28:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0116.hostedemail.com [216.40.44.116]) by kanga.kvack.org (Postfix) with ESMTP id 12EB06B0036 for ; Sat, 5 Jun 2021 00:28:29 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 959BC8248047 for ; Sat, 5 Jun 2021 04:28:28 +0000 (UTC) X-FDA: 78218388696.05.BA03086 Received: from mail-qv1-f53.google.com (mail-qv1-f53.google.com [209.85.219.53]) by imf02.hostedemail.com (Postfix) with ESMTP id 05037421108F for ; Sat, 5 Jun 2021 04:28:27 +0000 (UTC) Received: by mail-qv1-f53.google.com with SMTP id x2so5404063qvo.8 for ; Fri, 04 Jun 2021 21:28:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=PZkiKtXqsSUWsFPxoaMD52BWVXcB9txjh0r70N9gfwg=; b=WW4xgRVmullOrJwODXcaph9gCP7wl4/aB45NW6pjILZwERzHDpOslub+OyR7hv6vhX MQF6HrAFrzeiZgWNqLnh7i5L5IUeCDNz4A5ttNpDA19cMy9/mGXcEJT0WbPyaayBmVSB bVOuiJ/6Rsgfw8dJqWBlu/YAJYV8e3PSbOsa4f4K9eWKr2Xgo39e1CSlwj7VCjuDyqVY S0LwDvr6wJcda9KLJjMBlr2BLG4r6PfTk2tg/qAZCN5xG7q7k/UhBG0CDq4/lfxkpqkm 0WECU+QAbnudJ7jXiUtvPGGgerN8eW0cwsDgFdV4SroQ2C6/rCsFRueDEuusyXuNkrHY BuTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PZkiKtXqsSUWsFPxoaMD52BWVXcB9txjh0r70N9gfwg=; b=QG1fRozjuvPXGuFxUwJUmcSMmfDJRDfF6TPYbd1Z2FMs+3neWZC6S5u0aHgypaTPTm tCsOSjGwL+B5HUG7I/khGq4UJXdo/I4VfztAIeeZkd4GGnNTiuVhdBl9QtI+ua5WiHG/ kel/rbAzBTqrn9ov+eQ9Rgv93RVZoB2sBH2G2a6nwf/UghromZEJtyAG0x4/DO4jEcDC PwP4L0dxC8gEp2Obv21IEf0WykgoagyDBHg7svKc+f+EeM+8BWudRW2YPdza5n45HERp R151trs7s8NZuIrbu+pTKp+2rWQFMC+oFAs/kz+nn1khee7k2o62COgE6s1cL0A6XLx9 KLuA== X-Gm-Message-State: AOAM532FSGJq6qnHON0mYMvJaH+KPBVjcr3zEl9kkLXlF+V05FXLiflU rMfCJ+buevT21qk60006DO5fIYB9Gcw= X-Google-Smtp-Source: ABdhPJyb9zARpDDElBpR8QCJlNYPOxW79PfjHchqfbuf/vsgpBZ0bkxI9OwGFQqJfm5GHitbPo8Vnw== X-Received: by 2002:a62:3782:0:b029:2de:903d:8640 with SMTP id e124-20020a6237820000b02902de903d8640mr7135267pfa.40.1622857362398; Fri, 04 Jun 2021 18:42:42 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (60-242-147-73.tpgi.com.au. [60.242.147.73]) by smtp.gmail.com with ESMTPSA id q68sm5779056pjq.45.2021.06.04.18.42.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Jun 2021 18:42:42 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Randy Dunlap , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, Anton Blanchard , Andy Lutomirski Subject: [PATCH v4 4/4] powerpc/64s: enable MMU_LAZY_TLB_SHOOTDOWN Date: Sat, 5 Jun 2021 11:42:16 +1000 Message-Id: <20210605014216.446867-5-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210605014216.446867-1-npiggin@gmail.com> References: <20210605014216.446867-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 05037421108F X-Stat-Signature: m7o777r8aewq3up5nk34pbuwymj5s4nn Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=WW4xgRVm; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of npiggin@gmail.com designates 209.85.219.53 as permitted sender) smtp.mailfrom=npiggin@gmail.com X-HE-Tag: 1622867307-616313 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On a 16-socket 192-core POWER8 system, a context switching benchmark with as many software threads as CPUs (so each switch will go in and out of idle), upstream can achieve a rate of about 1 million context switches per second. After this patch it goes up to 118 million. Signed-off-by: Nicholas Piggin --- arch/powerpc/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 088dd2afcfe4..8a092eedc692 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -252,6 +252,7 @@ config PPC select IRQ_FORCED_THREADING select MMU_GATHER_PAGE_SIZE select MMU_GATHER_RCU_TABLE_FREE + select MMU_LAZY_TLB_SHOOTDOWN if PPC_BOOK3S_64 select MODULES_USE_ELF_RELA select NEED_DMA_MAP_STATE if PPC64 || NOT_COHERENT_CACHE select NEED_SG_DMA_LENGTH