From patchwork Sat Jun 5 01:42:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12301137 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2695DC4743E for ; Sat, 5 Jun 2021 05:18:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B200F6138C for ; Sat, 5 Jun 2021 05:18:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B200F6138C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4FF796B0036; Sat, 5 Jun 2021 01:18:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D60A6B006C; Sat, 5 Jun 2021 01:18:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 376C36B006E; Sat, 5 Jun 2021 01:18:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0235.hostedemail.com [216.40.44.235]) by kanga.kvack.org (Postfix) with ESMTP id 0790C6B0036 for ; Sat, 5 Jun 2021 01:18:19 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id A09449982 for ; Sat, 5 Jun 2021 05:18:19 +0000 (UTC) X-FDA: 78218514318.26.2345A0E Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf05.hostedemail.com (Postfix) with ESMTP id 655D4E0004EC for ; Sat, 5 Jun 2021 05:18:18 +0000 (UTC) Received: by mail-pl1-f176.google.com with SMTP id v12so5692490plo.10 for ; Fri, 04 Jun 2021 22:18:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=W8uxX4t8g+DxGYDGoM+1rG8fWNFKPCHDobEAy7P79JU=; b=t/BsozX+HKFZD2zebtgF3JGrIwCTcVj4ffmsJERRbi/MNaFuO3WFo3bmI7M7Pw7IDj 5k7ho9NFtodvy2hED8NQ029kOKEcm+emddzxclcV+A/9uFGxWtrrqyVwZBqBZQayVv8o uvzpSo3qbCEaRJsIsMf83kmaSWU82D3nQyk4UoworXY/8rBuy0UT3gajZ92D9fUUpVvL nkRMAM3yjYbObExBGVtCTrrn8upzvtdmLXXteWz7/0ii5TkWeGOPk21w9hgHzbsJtd4n 8a+HzVHQmMqiendjAtp1CC0CeIVqy3BZs61INz/EoX+TblkTnx0ukJD5mdWH0Ta+yWQq 0Sng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=W8uxX4t8g+DxGYDGoM+1rG8fWNFKPCHDobEAy7P79JU=; b=CgXcf2l+Xicqa6oOsiNqC5w2SEH4LeXBVTEuinQmg0+utroyT/4ZDc3kuk9164X7UF 0qf+2H5f/wzVy04ikenIXDLN5UZU2evzVJYVea/wlHlUQ4j+rie7l39bEM/Gili/ql2B YpbtzbpJsULIAOpCwMhVDgSTd97vr4rNoerFFDr4LcGa85rorTCqq1ZHVoxu9aF4FWL3 cdd7Lh53Oj1L73SK3WlAzDQH1oCd3jD2JO894BEeLQvmqDgVOiYeLdmAfA0qFkMEiTZI lhj73Q9vtWBw5Uci1RrB5f5mwH9+NOTPP2fSZIqCj7doQO3Bn68+Icd+nSMaHyRfe2Qi oStQ== X-Gm-Message-State: AOAM532n5LbWeCTyjzoGMzbYz7MoKenhF9CDa8I+S1WzLMPZ8iWKkWiT WCxOdGu9kOkHX2Mr0md14waIBrDmDpc= X-Google-Smtp-Source: ABdhPJyaiJ+cifU73dwUkfKSCUG8g6xUPympTQc18g/8gDitQusrenWMupQEWYNpZsq0HZ+33JFCOQ== X-Received: by 2002:aa7:900f:0:b029:2ec:82d2:d23 with SMTP id m15-20020aa7900f0000b02902ec82d20d23mr5855353pfo.16.1622857354192; Fri, 04 Jun 2021 18:42:34 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (60-242-147-73.tpgi.com.au. [60.242.147.73]) by smtp.gmail.com with ESMTPSA id q68sm5779056pjq.45.2021.06.04.18.42.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Jun 2021 18:42:33 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Randy Dunlap , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, Anton Blanchard , Andy Lutomirski Subject: [PATCH v4 2/4] lazy tlb: allow lazy tlb mm refcounting to be configurable Date: Sat, 5 Jun 2021 11:42:14 +1000 Message-Id: <20210605014216.446867-3-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210605014216.446867-1-npiggin@gmail.com> References: <20210605014216.446867-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 655D4E0004EC Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b="t/BsozX+"; spf=pass (imf05.hostedemail.com: domain of npiggin@gmail.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: n9yyphwbgm8fom7qops3msdiqs14mpwp X-HE-Tag: 1622870298-6057 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add CONFIG_MMU_TLB_REFCOUNT which enables refcounting of the lazy tlb mm when it is context switched. This can be disabled by architectures that don't require this refcounting if they clean up lazy tlb mms when the last refcount is dropped. Currently this is always enabled, which is what existing code does, so the patch is effectively a no-op. Rename rq->prev_mm to rq->prev_lazy_mm, because that's what it is. Signed-off-by: Nicholas Piggin Signed-off-by: Nicholas Piggin --- arch/Kconfig | 4 ++++ include/linux/sched/mm.h | 13 +++++++++++-- kernel/sched/core.c | 22 ++++++++++++++++++---- kernel/sched/sched.h | 4 +++- 4 files changed, 36 insertions(+), 7 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index c45b770d3579..1cff045cdde6 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -418,6 +418,10 @@ config ARCH_WANT_IRQS_OFF_ACTIVATE_MM irqs disabled over activate_mm. Architectures that do IPI based TLB shootdowns should enable this. +# Use normal mm refcounting for MMU_LAZY_TLB kernel thread references. +config MMU_LAZY_TLB_REFCOUNT + def_bool y + config ARCH_HAVE_NMI_SAFE_CMPXCHG bool diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index bfd1baca5266..29e4638ad124 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -52,12 +52,21 @@ static inline void mmdrop(struct mm_struct *mm) /* Helpers for lazy TLB mm refcounting */ static inline void mmgrab_lazy_tlb(struct mm_struct *mm) { - mmgrab(mm); + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) + mmgrab(mm); } static inline void mmdrop_lazy_tlb(struct mm_struct *mm) { - mmdrop(mm); + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) { + mmdrop(mm); + } else { + /* + * mmdrop_lazy_tlb must provide a full memory barrier, see the + * membarrier comment finish_task_switch which relies on this. + */ + smp_mb(); + } } /** diff --git a/kernel/sched/core.c b/kernel/sched/core.c index e359c76ea2e2..5e10cb712be3 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4171,7 +4171,7 @@ static struct rq *finish_task_switch(struct task_struct *prev) __releases(rq->lock) { struct rq *rq = this_rq(); - struct mm_struct *mm = rq->prev_mm; + struct mm_struct *mm = NULL; long prev_state; /* @@ -4190,7 +4190,10 @@ static struct rq *finish_task_switch(struct task_struct *prev) current->comm, current->pid, preempt_count())) preempt_count_set(FORK_PREEMPT_COUNT); - rq->prev_mm = NULL; +#ifdef CONFIG_MMU_LAZY_TLB_REFCOUNT + mm = rq->prev_lazy_mm; + rq->prev_lazy_mm = NULL; +#endif /* * A task struct has one reference for the use as "current". @@ -4326,9 +4329,20 @@ context_switch(struct rq *rq, struct task_struct *prev, switch_mm_irqs_off(prev->active_mm, next->mm, next); if (!prev->mm) { // from kernel - /* will mmdrop_lazy_tlb() in finish_task_switch(). */ - rq->prev_mm = prev->active_mm; +#ifdef CONFIG_MMU_LAZY_TLB_REFCOUNT + /* Will mmdrop_lazy_tlb() in finish_task_switch(). */ + rq->prev_lazy_mm = prev->active_mm; prev->active_mm = NULL; +#else + /* + * Without MMU_LAZY_TLB_REFCOUNT there is no lazy + * tracking (because no rq->prev_lazy_mm) in + * finish_task_switch, so no mmdrop_lazy_tlb(), so no + * memory barrier for membarrier (see the membarrier + * comment in finish_task_switch()). Do it here. + */ + smp_mb(); +#endif } } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index a189bec13729..0729cf19a987 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -961,7 +961,9 @@ struct rq { struct task_struct *idle; struct task_struct *stop; unsigned long next_balance; - struct mm_struct *prev_mm; +#ifdef CONFIG_MMU_LAZY_TLB_REFCOUNT + struct mm_struct *prev_lazy_mm; +#endif unsigned int clock_update_flags; u64 clock;