From patchwork Tue Jun 1 06:23:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12290463 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 606AAC47080 for ; Tue, 1 Jun 2021 06:23:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 078AD61375 for ; Tue, 1 Jun 2021 06:23:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 078AD61375 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9B9A26B006C; Tue, 1 Jun 2021 02:23:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9907F8E0001; Tue, 1 Jun 2021 02:23:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 831376B0070; Tue, 1 Jun 2021 02:23:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0119.hostedemail.com [216.40.44.119]) by kanga.kvack.org (Postfix) with ESMTP id 4FFC16B006C for ; Tue, 1 Jun 2021 02:23:17 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id DA269A8C7 for ; Tue, 1 Jun 2021 06:23:16 +0000 (UTC) X-FDA: 78204162792.09.77F3EAF Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by imf05.hostedemail.com (Postfix) with ESMTP id CA623E000255 for ; Tue, 1 Jun 2021 06:23:00 +0000 (UTC) Received: by mail-pf1-f173.google.com with SMTP id y15so6651734pfl.4 for ; Mon, 31 May 2021 23:23:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ecwzSsruDCg5HQ2i3NCpkmbXtWN83qkbLVUmQqDCWlk=; b=YlsXlVsDWoYMGPxW+QyS762YHdFOjE7CxBcZfRMKkUnI2PFDKCy9EUcSexkxG0c6tc JEOxq2/lzF3ffdvVbpz5kWXSrVKFyifnzzZzYWyhirC+Wl3O2KuMjuZe1++feSbhj221 a7jXtO00cFZrq5ZPUvJy+EZIuhgvZs8ffjQUkcF3fs5vG0jR70dh7b3/nPaDuA5S3xTF IYMaQMFZcwU8LY4YhFmqsekoeC1QlNfSLRQlkxdl6VXaguYbigTo1wp/2pO3dXha7wM7 XnJrwdwqNReQ9QySyKo9rj2h0BQJIDEw9ISx0Rd4lrqILj5rW4LgeYMIdludaJaGFZyP FpBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ecwzSsruDCg5HQ2i3NCpkmbXtWN83qkbLVUmQqDCWlk=; b=miCkWZr036Nz3l4DfCuKPW1iTxLXVhjBa/gZ7gHofiB2lIuTNVUg3Y9RXNEo3GDOr3 1c2ogRKPuxBK598kjj28/rq+E7vMpnpLSfHmGCyejEx57g5HB4iu+jB542dIu28+RORK i8ZyKrx0o/TqV51J/yBNGugNGi1Ob9wgOcJV9HT4Y+bl9rv/itnpaeLbU6DhnJ+sEpJt gSvYwg7oQ4Qemp5CAdNn1DYPfBd2UK70/VYsi4tc2FJL8JfQtvqSbuMFjO4olEVZZ/cN gbjqlVCPYIF4ahy6t2Xy44+DAWNMtq5zDism41lktk+ONf2Yx4M+eEa5TPJulvkEaCs8 77qQ== X-Gm-Message-State: AOAM5317iO1iPXCdjAV54MTqEz1I3DPK+SN95RQm8UUY9rsCH5VJJCx+ g+L7cHk2dJdSBoJ7LDOnrZA= X-Google-Smtp-Source: ABdhPJwCAsEtEKpT8K7Nr300NGB/ZvSukjJyFZA0oMZFNmXMIB8cLbka5YfgBrjsXWlgMYSF+dBsCQ== X-Received: by 2002:a63:da0a:: with SMTP id c10mr26585212pgh.255.1622528595531; Mon, 31 May 2021 23:23:15 -0700 (PDT) Received: from bobo.ibm.com (60-241-69-122.static.tpgi.com.au. [60.241.69.122]) by smtp.gmail.com with ESMTPSA id h1sm12519100pfh.72.2021.05.31.23.23.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 May 2021 23:23:15 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Randy Dunlap , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, Anton Blanchard , Andy Lutomirski Subject: [PATCH v3 1/4] lazy tlb: introduce lazy mm refcount helper functions Date: Tue, 1 Jun 2021 16:23:00 +1000 Message-Id: <20210601062303.3932513-2-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210601062303.3932513-1-npiggin@gmail.com> References: <20210601062303.3932513-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: CA623E000255 Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=YlsXlVsD; spf=pass (imf05.hostedemail.com: domain of npiggin@gmail.com designates 209.85.210.173 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam04 X-Stat-Signature: 47j38uq6cjh9ao9ir1y9pr3zqk6ztb9u X-HE-Tag: 1622528580-587909 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add explicit _lazy_tlb annotated functions for lazy mm refcounting. This makes lazy mm references more obvious, and allows explicit refcounting to be removed if it is not used. Signed-off-by: Nicholas Piggin --- arch/arm/mach-rpc/ecard.c | 2 +- arch/powerpc/kernel/smp.c | 2 +- arch/powerpc/mm/book3s64/radix_tlb.c | 4 ++-- fs/exec.c | 4 ++-- include/linux/sched/mm.h | 11 +++++++++++ kernel/cpu.c | 2 +- kernel/exit.c | 2 +- kernel/kthread.c | 11 +++++++---- kernel/sched/core.c | 15 ++++++++------- 9 files changed, 34 insertions(+), 19 deletions(-) diff --git a/arch/arm/mach-rpc/ecard.c b/arch/arm/mach-rpc/ecard.c index 827b50f1c73e..1b4a41aad793 100644 --- a/arch/arm/mach-rpc/ecard.c +++ b/arch/arm/mach-rpc/ecard.c @@ -253,7 +253,7 @@ static int ecard_init_mm(void) current->mm = mm; current->active_mm = mm; activate_mm(active_mm, mm); - mmdrop(active_mm); + mmdrop_lazy_tlb(active_mm); ecard_init_pgtables(mm); return 0; } diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c index 2e05c783440a..fb0bdfc67366 100644 --- a/arch/powerpc/kernel/smp.c +++ b/arch/powerpc/kernel/smp.c @@ -1541,7 +1541,7 @@ void start_secondary(void *unused) { unsigned int cpu = raw_smp_processor_id(); - mmgrab(&init_mm); + mmgrab_lazy_tlb(&init_mm); current->active_mm = &init_mm; smp_store_cpu_info(cpu); diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 409e61210789..2962082787c0 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -663,10 +663,10 @@ void exit_lazy_flush_tlb(struct mm_struct *mm, bool always_flush) if (current->active_mm == mm) { WARN_ON_ONCE(current->mm != NULL); /* Is a kernel thread and is using mm as the lazy tlb */ - mmgrab(&init_mm); + mmgrab_lazy_tlb(&init_mm); current->active_mm = &init_mm; switch_mm_irqs_off(mm, &init_mm, current); - mmdrop(mm); + mmdrop_lazy_tlb(mm); } /* diff --git a/fs/exec.c b/fs/exec.c index 18594f11c31f..ca0f8b1af23a 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -1029,9 +1029,9 @@ static int exec_mmap(struct mm_struct *mm) setmax_mm_hiwater_rss(&tsk->signal->maxrss, old_mm); mm_update_next_owner(old_mm); mmput(old_mm); - return 0; + } else { + mmdrop_lazy_tlb(active_mm); } - mmdrop(active_mm); return 0; } diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index e24b1fe348e3..bfd1baca5266 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -49,6 +49,17 @@ static inline void mmdrop(struct mm_struct *mm) __mmdrop(mm); } +/* Helpers for lazy TLB mm refcounting */ +static inline void mmgrab_lazy_tlb(struct mm_struct *mm) +{ + mmgrab(mm); +} + +static inline void mmdrop_lazy_tlb(struct mm_struct *mm) +{ + mmdrop(mm); +} + /** * mmget() - Pin the address space associated with a &struct mm_struct. * @mm: The address space to pin. diff --git a/kernel/cpu.c b/kernel/cpu.c index e538518556f4..e87a89824e6c 100644 --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -602,7 +602,7 @@ static int finish_cpu(unsigned int cpu) */ if (mm != &init_mm) idle->active_mm = &init_mm; - mmdrop(mm); + mmdrop_lazy_tlb(mm); return 0; } diff --git a/kernel/exit.c b/kernel/exit.c index fd1c04193e18..8e87ec5f6be2 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -476,7 +476,7 @@ static void exit_mm(void) __set_current_state(TASK_RUNNING); mmap_read_lock(mm); } - mmgrab(mm); + mmgrab_lazy_tlb(mm); BUG_ON(mm != current->active_mm); /* more a memory barrier than a real lock */ task_lock(current); diff --git a/kernel/kthread.c b/kernel/kthread.c index fe3f2a40d61e..b70e28431a01 100644 --- a/kernel/kthread.c +++ b/kernel/kthread.c @@ -1314,14 +1314,14 @@ void kthread_use_mm(struct mm_struct *mm) WARN_ON_ONCE(!(tsk->flags & PF_KTHREAD)); WARN_ON_ONCE(tsk->mm); + mmgrab(mm); + task_lock(tsk); /* Hold off tlb flush IPIs while switching mm's */ local_irq_disable(); active_mm = tsk->active_mm; - if (active_mm != mm) { - mmgrab(mm); + if (active_mm != mm) tsk->active_mm = mm; - } tsk->mm = mm; membarrier_update_current_mm(mm); switch_mm_irqs_off(active_mm, mm, tsk); @@ -1341,7 +1341,7 @@ void kthread_use_mm(struct mm_struct *mm) * mmdrop(), or explicitly with smp_mb(). */ if (active_mm != mm) - mmdrop(active_mm); + mmdrop_lazy_tlb(active_mm); else smp_mb(); @@ -1375,10 +1375,13 @@ void kthread_unuse_mm(struct mm_struct *mm) local_irq_disable(); tsk->mm = NULL; membarrier_update_current_mm(NULL); + mmgrab_lazy_tlb(mm); /* active_mm is still 'mm' */ enter_lazy_tlb(mm, tsk); local_irq_enable(); task_unlock(tsk); + + mmdrop(mm); } EXPORT_SYMBOL_GPL(kthread_unuse_mm); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 5226cc26a095..e359c76ea2e2 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4229,13 +4229,14 @@ static struct rq *finish_task_switch(struct task_struct *prev) * rq->curr, before returning to userspace, so provide them here: * * - a full memory barrier for {PRIVATE,GLOBAL}_EXPEDITED, implicitly - * provided by mmdrop(), + * provided by mmdrop_lazy_tlb(), * - a sync_core for SYNC_CORE. */ if (mm) { membarrier_mm_sync_core_before_usermode(mm); - mmdrop(mm); + mmdrop_lazy_tlb(mm); } + if (unlikely(prev_state == TASK_DEAD)) { if (prev->sched_class->task_dead) prev->sched_class->task_dead(prev); @@ -4299,9 +4300,9 @@ context_switch(struct rq *rq, struct task_struct *prev, /* * kernel -> kernel lazy + transfer active - * user -> kernel lazy + mmgrab() active + * user -> kernel lazy + mmgrab_lazy_tlb() active * - * kernel -> user switch + mmdrop() active + * kernel -> user switch + mmdrop_lazy_tlb() active * user -> user switch */ if (!next->mm) { // to kernel @@ -4309,7 +4310,7 @@ context_switch(struct rq *rq, struct task_struct *prev, next->active_mm = prev->active_mm; if (prev->mm) // from user - mmgrab(prev->active_mm); + mmgrab_lazy_tlb(prev->active_mm); else prev->active_mm = NULL; } else { // to user @@ -4325,7 +4326,7 @@ context_switch(struct rq *rq, struct task_struct *prev, switch_mm_irqs_off(prev->active_mm, next->mm, next); if (!prev->mm) { // from kernel - /* will mmdrop() in finish_task_switch(). */ + /* will mmdrop_lazy_tlb() in finish_task_switch(). */ rq->prev_mm = prev->active_mm; prev->active_mm = NULL; } @@ -8239,7 +8240,7 @@ void __init sched_init(void) /* * The boot idle thread does lazy MMU switching as well: */ - mmgrab(&init_mm); + mmgrab_lazy_tlb(&init_mm); enter_lazy_tlb(&init_mm, current); /* From patchwork Tue Jun 1 06:23:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12290465 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65A1FC4708F for ; Tue, 1 Jun 2021 06:23:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DFD4461159 for ; Tue, 1 Jun 2021 06:23:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DFD4461159 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7E50D8E0002; Tue, 1 Jun 2021 02:23:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7BBDC8E0001; Tue, 1 Jun 2021 02:23:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E6E98E0002; Tue, 1 Jun 2021 02:23:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0216.hostedemail.com [216.40.44.216]) by kanga.kvack.org (Postfix) with ESMTP id 2BA1B8E0001 for ; Tue, 1 Jun 2021 02:23:21 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id B279C8249980 for ; Tue, 1 Jun 2021 06:23:20 +0000 (UTC) X-FDA: 78204162960.08.184438C Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by imf07.hostedemail.com (Postfix) with ESMTP id EECBDA00025A for ; Tue, 1 Jun 2021 06:23:07 +0000 (UTC) Received: by mail-pj1-f42.google.com with SMTP id k22-20020a17090aef16b0290163512accedso1028159pjz.0 for ; Mon, 31 May 2021 23:23:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=B2ZM54o9Bp1W7ui49t2FYRDuyLvPHwC8Il/uTHhhdMs=; b=XDJ2kYvmv/Z5M24qrG7KDMUE9p7aJ0RlQOBX1p/t2w4hIRyxR6UjTzA19/wWm8wdfC 2bssCYG5blnj9LMkj0KYjULkjvgmSvquyWjePNQYm//MxqhrHejkK0QcMGRIOOJbMpox 7lTAot474lVNko8WQIRuPXUlRf3A83o42kAVvfnakMYQ5V5QvGDcEzfdLv1MpTxUKbNv e+S+jSKkwHUTddDHEL3mGYjuaQDAWQaGCy1sXxnPFkwfKdci8juvdPKPN8bYXqfSQPrI UD3u5SitNYnPXIjG3reVtT1Z66MrR1ddVVxfrlRrOkUzRhQFpbERqoAxcac1BEAeYuRK nDEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=B2ZM54o9Bp1W7ui49t2FYRDuyLvPHwC8Il/uTHhhdMs=; b=jE6YvGqB8Ahq4h7jN+nNzovLrlxJfYbw9Bvs0gJ15zfyUL7gBSVogJKKmEO+I4kTy1 x9wf6viEXGSIqAHX5ZVB+F7pDNGaj56VR9hwdCXN9edIae8YAvtuYY8Iq75p9Qd4gunm /9f+75u+HZQeWcdhBtRUiMNzcIZSSDastljLpHz08LiWcXPbkbVqr/WALULtfdfcS99/ VB3d7IO5QlGx1vwlAgXZIS5G/XjvoWTx3h3HfHR3xAtlwTI9JK0XFMQlLQcYGXNVjht1 jr5gHlATYgLIr8I8/bqE+QRGAlkfXjRdtnb5SGEI3zHzj0LPD4kMQrVoXFDZHlt7dvJI U/Uw== X-Gm-Message-State: AOAM530rKFrspi7FekTqImTAY5Wg0++VuV06eHZrcGIrTg/j/+jFn+jY yrvLFXQLdbSNYneHOCmCedw= X-Google-Smtp-Source: ABdhPJy2DUM8gibAhLieX5iGh16ca9VDqhVgs0t9sh7k8/bnT7er59ZtGvorelTu6yvv6DxaKgwUsw== X-Received: by 2002:a17:90b:3587:: with SMTP id mm7mr21958255pjb.71.1622528599607; Mon, 31 May 2021 23:23:19 -0700 (PDT) Received: from bobo.ibm.com (60-241-69-122.static.tpgi.com.au. [60.241.69.122]) by smtp.gmail.com with ESMTPSA id h1sm12519100pfh.72.2021.05.31.23.23.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 May 2021 23:23:19 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Randy Dunlap , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, Anton Blanchard , Andy Lutomirski Subject: [PATCH v3 2/4] lazy tlb: allow lazy tlb mm switching to be configurable Date: Tue, 1 Jun 2021 16:23:01 +1000 Message-Id: <20210601062303.3932513-3-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210601062303.3932513-1-npiggin@gmail.com> References: <20210601062303.3932513-1-npiggin@gmail.com> MIME-Version: 1.0 Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=XDJ2kYvm; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf07.hostedemail.com: domain of npiggin@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=npiggin@gmail.com X-Stat-Signature: fy1kqwx9es3ei85ocatsoa64widdp4je X-Rspamd-Queue-Id: EECBDA00025A X-Rspamd-Server: rspam02 X-HE-Tag: 1622528587-382474 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add CONFIG_MMU_LAZY_TLB which can be configured out to disable the lazy tlb mechanism entirely, and switches to init_mm when switching to a kernel thread. NOMMU systems could easily go without this and save a bit of code and the refcount atomics, because their mm switch is a no-op. They have not been switched over by default because the arch code needs to be audited and tested for lazy tlb mm refcounting and converted to _lazy_tlb refcounting if necessary. CONFIG_MMU_LAZY_TLB_REFCOUNT is also added, but it must always be enabled if CONFIG_MMU_LAZY_TLB is enabled until the next patch which provides an alternate scheme. Signed-off-by: Nicholas Piggin --- arch/Kconfig | 26 ++++++++++++++ include/linux/sched/mm.h | 13 +++++-- kernel/sched/core.c | 75 ++++++++++++++++++++++++++++++---------- kernel/sched/sched.h | 4 ++- 4 files changed, 96 insertions(+), 22 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index c45b770d3579..276e1c1c0219 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -418,6 +418,32 @@ config ARCH_WANT_IRQS_OFF_ACTIVATE_MM irqs disabled over activate_mm. Architectures that do IPI based TLB shootdowns should enable this. +# Enable "lazy TLB", which means a user->kernel thread context switch does not +# switch the mm to init_mm and the kernel thread takes a reference to the user +# mm to provide its kernel mapping. This is how Linux has traditionally worked +# (see Documentation/vm/active_mm.rst), for performance. Switching to and from +# idle thread is a performance-critical case. +# +# If mm context switches are inexpensive or free (in the case of NOMMU) then +# this could be disabled. +# +# It would make sense to have this depend on MMU, but need to audit and test +# the NOMMU architectures for lazy mm refcounting first. +config MMU_LAZY_TLB + def_bool y + depends on !NO_MMU_LAZY_TLB + +# This allows archs to disable MMU_LAZY_TLB. mmgrab/mmdrop in arch/ code has +# to be audited and switched to _lazy_tlb postfix as necessary. +config NO_MMU_LAZY_TLB + def_bool n + +# Use normal mm refcounting for MMU_LAZY_TLB kernel thread references. +# For now, this must be enabled if MMU_LAZY_TLB is enabled. +config MMU_LAZY_TLB_REFCOUNT + def_bool y + depends on MMU_LAZY_TLB + config ARCH_HAVE_NMI_SAFE_CMPXCHG bool diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index bfd1baca5266..29e4638ad124 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -52,12 +52,21 @@ static inline void mmdrop(struct mm_struct *mm) /* Helpers for lazy TLB mm refcounting */ static inline void mmgrab_lazy_tlb(struct mm_struct *mm) { - mmgrab(mm); + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) + mmgrab(mm); } static inline void mmdrop_lazy_tlb(struct mm_struct *mm) { - mmdrop(mm); + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) { + mmdrop(mm); + } else { + /* + * mmdrop_lazy_tlb must provide a full memory barrier, see the + * membarrier comment finish_task_switch which relies on this. + */ + smp_mb(); + } } /** diff --git a/kernel/sched/core.c b/kernel/sched/core.c index e359c76ea2e2..299c3eb12b2b 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4171,7 +4171,7 @@ static struct rq *finish_task_switch(struct task_struct *prev) __releases(rq->lock) { struct rq *rq = this_rq(); - struct mm_struct *mm = rq->prev_mm; + struct mm_struct *mm = NULL; long prev_state; /* @@ -4190,7 +4190,10 @@ static struct rq *finish_task_switch(struct task_struct *prev) current->comm, current->pid, preempt_count())) preempt_count_set(FORK_PREEMPT_COUNT); - rq->prev_mm = NULL; +#ifdef CONFIG_MMU_LAZY_TLB_REFCOUNT + mm = rq->prev_lazy_mm; + rq->prev_lazy_mm = NULL; +#endif /* * A task struct has one reference for the use as "current". @@ -4282,22 +4285,10 @@ asmlinkage __visible void schedule_tail(struct task_struct *prev) calculate_sigpending(); } -/* - * context_switch - switch to the new MM and the new thread's register state. - */ -static __always_inline struct rq * -context_switch(struct rq *rq, struct task_struct *prev, - struct task_struct *next, struct rq_flags *rf) +static __always_inline void +context_switch_mm(struct rq *rq, struct task_struct *prev, + struct task_struct *next) { - prepare_task_switch(rq, prev, next); - - /* - * For paravirt, this is coupled with an exit in switch_to to - * combine the page table reload and the switch backend into - * one hypercall. - */ - arch_start_context_switch(prev); - /* * kernel -> kernel lazy + transfer active * user -> kernel lazy + mmgrab_lazy_tlb() active @@ -4326,11 +4317,57 @@ context_switch(struct rq *rq, struct task_struct *prev, switch_mm_irqs_off(prev->active_mm, next->mm, next); if (!prev->mm) { // from kernel - /* will mmdrop_lazy_tlb() in finish_task_switch(). */ - rq->prev_mm = prev->active_mm; +#ifdef CONFIG_MMU_LAZY_TLB_REFCOUNT + /* Will mmdrop_lazy_tlb() in finish_task_switch(). */ + rq->prev_lazy_mm = prev->active_mm; prev->active_mm = NULL; +#else + /* + * Without MMU_LAZY_REFCOUNT there is no lazy + * tracking (because no rq->prev_lazy_mm) in + * finish_task_switch, so no mmdrop_lazy_tlb(), + * so no memory barrier for membarrier (see the + * membarrier comment in finish_task_switch()). + * Do it here. + */ + smp_mb(); +#endif } } +} + +static __always_inline void +context_switch_mm_nolazy(struct rq *rq, struct task_struct *prev, + struct task_struct *next) +{ + if (!next->mm) + next->active_mm = &init_mm; + membarrier_switch_mm(rq, prev->active_mm, next->active_mm); + switch_mm_irqs_off(prev->active_mm, next->active_mm, next); + if (!prev->mm) + prev->active_mm = NULL; +} + +/* + * context_switch - switch to the new MM and the new thread's register state. + */ +static __always_inline struct rq * +context_switch(struct rq *rq, struct task_struct *prev, + struct task_struct *next, struct rq_flags *rf) +{ + prepare_task_switch(rq, prev, next); + + /* + * For paravirt, this is coupled with an exit in switch_to to + * combine the page table reload and the switch backend into + * one hypercall. + */ + arch_start_context_switch(prev); + + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB)) + context_switch_mm(rq, prev, next); + else + context_switch_mm_nolazy(rq, prev, next); rq->clock_update_flags &= ~(RQCF_ACT_SKIP|RQCF_REQ_SKIP); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index a189bec13729..0729cf19a987 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -961,7 +961,9 @@ struct rq { struct task_struct *idle; struct task_struct *stop; unsigned long next_balance; - struct mm_struct *prev_mm; +#ifdef CONFIG_MMU_LAZY_TLB_REFCOUNT + struct mm_struct *prev_lazy_mm; +#endif unsigned int clock_update_flags; u64 clock; From patchwork Tue Jun 1 06:23:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12290467 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FF8FC4708F for ; Tue, 1 Jun 2021 06:23:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EDCDF61159 for ; Tue, 1 Jun 2021 06:23:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EDCDF61159 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8CFB98E0001; Tue, 1 Jun 2021 02:23:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8AA5E940007; Tue, 1 Jun 2021 02:23:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 746E48E0003; Tue, 1 Jun 2021 02:23:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0035.hostedemail.com [216.40.44.35]) by kanga.kvack.org (Postfix) with ESMTP id 466AB8E0001 for ; Tue, 1 Jun 2021 02:23:25 -0400 (EDT) Received: from smtpin31.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id CC294A76B for ; Tue, 1 Jun 2021 06:23:24 +0000 (UTC) X-FDA: 78204163128.31.7D9580F Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf29.hostedemail.com (Postfix) with ESMTP id 39C43371 for ; Tue, 1 Jun 2021 06:23:09 +0000 (UTC) Received: by mail-pl1-f179.google.com with SMTP id t21so6325462plo.2 for ; Mon, 31 May 2021 23:23:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7SY6QzuWUSlWj33T26/GRZSVwi+Ld4IHiibBoCb52p4=; b=ePlorr20RYdU83BpwKX/s0RfyvSJGLZuiSfUt1Q5Tf+Uj6s0wJkGx8MFmTlzTq06Ly /QdSin1Qqs2l0J37C68YVuBoCY22xVUnKkbWT5c/25xPZUE4hwRl5CveuKHLOW9KwoLz WMUnhDGfB+gMrKJPdIvTLtMDs5EgLCysNJyB17ago3WfOtWSwQRzdwE9GUfx+u24xoXb YvUh+XKO7EACTncuCcfNhm9MUjn8I6F0IEFKp7B02dAPD0YE4xz1HbQBw17+5uziSWyH 6Hbt03PuVPQDsas/U9JfW6GikTKlAqSmRjtpSQ956IQanhxIOkALHMsFIlrry/0xO4L8 Jh1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7SY6QzuWUSlWj33T26/GRZSVwi+Ld4IHiibBoCb52p4=; b=dVu3ZjxMA8TXKtw1SvXRLy2PX4W7CU/NbRfjetLxuaJe/0LnRw4xIi60QYfWYDOJXQ 4BGOCy0eQUFla/sxqgCzsbHGHhPvqrVcQtviJH6nRNYaQKheU82Qy4EQrSbOEgnFrxBb R90uAkVwivS7n3TdQFh+h1yJrkrU+KzU9WTdqxUktWBymADKCF6AolPkUIojPXlz2B9s v8C36f+iEAIYIaLurghDu5UxDfb5gdo8GnkkIsOtenTOtOkVcK954RmjlwxOEH/zOxqy QYjT6V03lsl1FwzYbfrnXFjvrGyx3hyVWBNhMHqkNwD7XnhwyLexFV9HboYKjaCXo9SW SASQ== X-Gm-Message-State: AOAM530A8ru279KqplE0Q8JvGBhw7nwW66tSvU9nEPJJDDqPT5pGgegZ 33F55NI2b4J1x0hAX3aAX6k4BVvKQWc= X-Google-Smtp-Source: ABdhPJw09TNZqBmKx3WQDrsKTZlhlpstv4mz2E1lJZnWicTzFCQBo5ZLvZlMnKhALwZzNq6+juRuaA== X-Received: by 2002:a17:90a:5106:: with SMTP id t6mr3315886pjh.216.1622528603683; Mon, 31 May 2021 23:23:23 -0700 (PDT) Received: from bobo.ibm.com (60-241-69-122.static.tpgi.com.au. [60.241.69.122]) by smtp.gmail.com with ESMTPSA id h1sm12519100pfh.72.2021.05.31.23.23.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 May 2021 23:23:23 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Randy Dunlap , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, Anton Blanchard , Andy Lutomirski Subject: [PATCH v3 3/4] lazy tlb: shoot lazies, a non-refcounting lazy tlb option Date: Tue, 1 Jun 2021 16:23:02 +1000 Message-Id: <20210601062303.3932513-4-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210601062303.3932513-1-npiggin@gmail.com> References: <20210601062303.3932513-1-npiggin@gmail.com> MIME-Version: 1.0 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=ePlorr20; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of npiggin@gmail.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=npiggin@gmail.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 39C43371 X-Stat-Signature: moz91ab7988cenjyanuxzmjz699xzjwp X-HE-Tag: 1622528589-452177 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On big systems, the mm refcount can become highly contented when doing a lot of context switching with threaded applications (particularly switching between the idle thread and an application thread). Abandoning lazy tlb slows switching down quite a bit in the important user->idle->user cases, so instead implement a non-refcounted scheme that causes __mmdrop() to IPI all CPUs in the mm_cpumask and shoot down any remaining lazy ones. Shootdown IPIs are some concern, but they have not been observed to be a big problem with this scheme (the powerpc implementation generated 314 additional interrupts on a 144 CPU system during a kernel compile). There are a number of strategies that could be employed to reduce IPIs if they turn out to be a problem for some workload. Signed-off-by: Nicholas Piggin --- arch/Kconfig | 14 +++++++++++++- kernel/fork.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 65 insertions(+), 1 deletion(-) diff --git a/arch/Kconfig b/arch/Kconfig index 276e1c1c0219..91e1882e3284 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -439,11 +439,23 @@ config NO_MMU_LAZY_TLB def_bool n # Use normal mm refcounting for MMU_LAZY_TLB kernel thread references. -# For now, this must be enabled if MMU_LAZY_TLB is enabled. config MMU_LAZY_TLB_REFCOUNT def_bool y depends on MMU_LAZY_TLB +# Instead of refcounting the lazy mm struct for kernel thread references +# (which can cause contention with multi-threaded apps on large multiprocessor +# systems), this option causes __mmdrop to IPI all CPUs in the mm_cpumask and +# switch to init_mm if they were using the to-be-freed mm as the lazy tlb. To +# implement this, architectures must use _lazy_tlb variants of mm refcounting +# when releasing kernel thread mm references, and mm_cpumask must include at +# least all possible CPUs in which the mm might be lazy, at the time of the +# final mmdrop. mmgrab/mmdrop in arch/ code must be switched to _lazy_tlb +# postfix as necessary. +config MMU_LAZY_TLB_SHOOTDOWN + bool + depends on MMU_LAZY_TLB + config ARCH_HAVE_NMI_SAFE_CMPXCHG bool diff --git a/kernel/fork.c b/kernel/fork.c index dc06afd725cb..d485c24426a0 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -674,6 +674,53 @@ static void check_mm(struct mm_struct *mm) #define allocate_mm() (kmem_cache_alloc(mm_cachep, GFP_KERNEL)) #define free_mm(mm) (kmem_cache_free(mm_cachep, (mm))) +static void do_shoot_lazy_tlb(void *arg) +{ + struct mm_struct *mm = arg; + + if (current->active_mm == mm) { + WARN_ON_ONCE(current->mm); + current->active_mm = &init_mm; + switch_mm(mm, &init_mm, current); + } +} + +static void do_check_lazy_tlb(void *arg) +{ + struct mm_struct *mm = arg; + + WARN_ON_ONCE(current->active_mm == mm); +} + +static void shoot_lazy_tlbs(struct mm_struct *mm) +{ + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_SHOOTDOWN)) { + /* + * IPI overheads have not found to be expensive, but they could + * be reduced in a number of possible ways, for example (in + * roughly increasing order of complexity): + * - A batch of mms requiring IPIs could be gathered and freed + * at once. + * - CPUs could store their active mm somewhere that can be + * remotely checked without a lock, to filter out + * false-positives in the cpumask. + * - After mm_users or mm_count reaches zero, switching away + * from the mm could clear mm_cpumask to reduce some IPIs + * (some batching or delaying would help). + * - A delayed freeing and RCU-like quiescing sequence based on + * mm switching to avoid IPIs completely. + */ + on_each_cpu_mask(mm_cpumask(mm), do_shoot_lazy_tlb, (void *)mm, 1); + if (IS_ENABLED(CONFIG_DEBUG_VM)) + on_each_cpu(do_check_lazy_tlb, (void *)mm, 1); + } else { + /* + * In this case, lazy tlb mms are refounted and would not reach + * __mmdrop until all CPUs have switched away and mmdrop()ed. + */ + } +} + /* * Called when the last reference to the mm * is dropped: either by a lazy thread or by @@ -683,7 +730,12 @@ void __mmdrop(struct mm_struct *mm) { BUG_ON(mm == &init_mm); WARN_ON_ONCE(mm == current->mm); + + /* Ensure no CPUs are using this as their lazy tlb mm */ + shoot_lazy_tlbs(mm); + WARN_ON_ONCE(mm == current->active_mm); + mm_free_pgd(mm); destroy_context(mm); mmu_notifier_subscriptions_destroy(mm); From patchwork Tue Jun 1 06:23:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12290469 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F24CC47096 for ; Tue, 1 Jun 2021 06:23:30 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EB61961159 for ; Tue, 1 Jun 2021 06:23:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EB61961159 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 859E7940008; Tue, 1 Jun 2021 02:23:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 83098940007; Tue, 1 Jun 2021 02:23:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6FA5D940008; Tue, 1 Jun 2021 02:23:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0044.hostedemail.com [216.40.44.44]) by kanga.kvack.org (Postfix) with ESMTP id 37096940007 for ; Tue, 1 Jun 2021 02:23:29 -0400 (EDT) Received: from smtpin31.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id C9D47180AD804 for ; Tue, 1 Jun 2021 06:23:28 +0000 (UTC) X-FDA: 78204163296.31.EC59741 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf23.hostedemail.com (Postfix) with ESMTP id 2D753A000261 for ; Tue, 1 Jun 2021 06:23:18 +0000 (UTC) Received: by mail-pj1-f43.google.com with SMTP id i22so1172920pju.0 for ; Mon, 31 May 2021 23:23:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=PZkiKtXqsSUWsFPxoaMD52BWVXcB9txjh0r70N9gfwg=; b=TMuuo1b+x5/KjM8834zLhSTP9eYuo7t0tW46rm7j4SnXrgLEDWE0TOlUaFMqu8a+wI vbs8XSGlHrRcHcTNztnJ6FzCBPFzIdXlEHvPisjfSNJtPoi22YsCPwpfGpRVCfOzBEyA qIpgYtAQFvdrhvVyPuJFVlv69Dqy2++lkPLlvSfYFcFCmgn46h+CL735k7uWPO2iR6ib F/kGzqky6oCb6NGOI9e7WIPMpMGVye8FitjlLICcOtIpc9ld3zB5Wx8ELsL60gT36dHI 97jMnbIhzGPVji0u7wu9KoHMx8MMSH5LNriPf5FvHIo0aVVgBAQXAvYl2+p+Q7UssMwq afpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PZkiKtXqsSUWsFPxoaMD52BWVXcB9txjh0r70N9gfwg=; b=XCgeGmFxCbSZWy6FXWbTT1fM8DERR8uY78FSdVWuT8h2gH8wIus/txhMld9E8pcAUU YFNoxRM3L4Fzsdc/zNSDMzo5J6uVHiGTAQOj5+S1duUD0zoTxmmc/BKRApGa8ZomOJHG CcvTX3AYEdE0xSMPonVxiMx/7xbN7UwvhUScjaVUDA5flZGajTOBbfFv2uOhvO6d84tw GeVPF+tkrNlK0GNH+IV/2KzMM3PQePumTZ+FDPgBcUMk3JFgjO1mh75/8d1MZhUvzTml 0mdxpu9RmBH6WZbDW0fcFj5yKrSQ1yGg7nJ6vwHFVNF8dwFIPnXPLErNZeF63OaZZCN7 DJWw== X-Gm-Message-State: AOAM533NezPivDTrcBQO+3ieeZkbL6qxopjJJ4CjUoCTsnOPsiKsRzkY GUJgFX6iyfwT/b76drvxAhE= X-Google-Smtp-Source: ABdhPJzO7M0Hh974CEgDAeZuPzkos1it6HgA9FTrTsKtUq98J+3jqEoDd6tOI7VI/NdE9fIIB6URYw== X-Received: by 2002:a17:90b:1094:: with SMTP id gj20mr14135765pjb.45.1622528607822; Mon, 31 May 2021 23:23:27 -0700 (PDT) Received: from bobo.ibm.com (60-241-69-122.static.tpgi.com.au. [60.241.69.122]) by smtp.gmail.com with ESMTPSA id h1sm12519100pfh.72.2021.05.31.23.23.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 May 2021 23:23:27 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Randy Dunlap , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, Anton Blanchard , Andy Lutomirski Subject: [PATCH v3 4/4] powerpc/64s: enable MMU_LAZY_TLB_SHOOTDOWN Date: Tue, 1 Jun 2021 16:23:03 +1000 Message-Id: <20210601062303.3932513-5-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210601062303.3932513-1-npiggin@gmail.com> References: <20210601062303.3932513-1-npiggin@gmail.com> MIME-Version: 1.0 Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=TMuuo1b+; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf23.hostedemail.com: domain of npiggin@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=npiggin@gmail.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 2D753A000261 X-Stat-Signature: kpn61gkpzzsyi664e4of8uiwz4ew5ket X-HE-Tag: 1622528598-335067 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On a 16-socket 192-core POWER8 system, a context switching benchmark with as many software threads as CPUs (so each switch will go in and out of idle), upstream can achieve a rate of about 1 million context switches per second. After this patch it goes up to 118 million. Signed-off-by: Nicholas Piggin --- arch/powerpc/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 088dd2afcfe4..8a092eedc692 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -252,6 +252,7 @@ config PPC select IRQ_FORCED_THREADING select MMU_GATHER_PAGE_SIZE select MMU_GATHER_RCU_TABLE_FREE + select MMU_LAZY_TLB_SHOOTDOWN if PPC_BOOK3S_64 select MODULES_USE_ELF_RELA select NEED_DMA_MAP_STATE if PPC64 || NOT_COHERENT_CACHE select NEED_SG_DMA_LENGTH