From patchwork Fri Aug 28 10:00:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11742531 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D2E11138A for ; Fri, 28 Aug 2020 10:00:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 94524208CA for ; Fri, 28 Aug 2020 10:00:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CQod5hZ8" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 94524208CA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BF84D6B000C; Fri, 28 Aug 2020 06:00:39 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BAA4C8D0002; Fri, 28 Aug 2020 06:00:39 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ABF6A8D0001; Fri, 28 Aug 2020 06:00:39 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0082.hostedemail.com [216.40.44.82]) by kanga.kvack.org (Postfix) with ESMTP id 9528B6B000C for ; Fri, 28 Aug 2020 06:00:39 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 63C89ABF7 for ; Fri, 28 Aug 2020 10:00:39 +0000 (UTC) X-FDA: 77199532998.11.space37_16107b227074 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id 24087180F8B82 for ; Fri, 28 Aug 2020 10:00:39 +0000 (UTC) X-Spam-Summary: 1,0,0,3d2d607be28d345f,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:1801:2194:2196:2198:2199:2200:2201:2393:2553:2559:2562:2689:2895:2914:3138:3139:3140:3141:3142:3355:3865:3866:3867:3868:3870:3871:3872:3874:4117:4250:4321:4385:4605:5007:6119:6120:6261:6653:7514:7901:7903:8999:9000:9149:9413:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:12986:13255:13894:14096:14181:14394:14687:14721:21080:21433:21444:21451:21627:21666:21966:30045:30054:30090,0,RBL:209.85.210.196:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100;04yf65ejokfpoht4jpz8aqssce3rzyp1kgxyhsgt9yo97dz9iodrdeg9fk7a5dq.ntthco1cqkys7eidfts34knozrireyqkbn83j1h44pe9wdn5nytmz3ejrxfajij.c-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SU MMARY:no X-HE-Tag: space37_16107b227074 X-Filterd-Recvd-Size: 6928 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Fri, 28 Aug 2020 10:00:38 +0000 (UTC) Received: by mail-pf1-f196.google.com with SMTP id k15so391436pfc.12 for ; Fri, 28 Aug 2020 03:00:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=PMGQxaBKPbON3ssZWMqrEm63rTUeWUDOhP6COLqPYSc=; b=CQod5hZ8nMSXxmoYAACVHF/qI22QxGeFSjZUjfpoMAoBOV9QseFdRM4mT4eXynXUTC Srct4/znp2udP6D2fWV0bzQ+pU+XXOtJaMi2g+drMUXj2K60C7ryDET6zClkdldor8i/ E7b0vIa8R3BWPgXPD+PJJc+AOPD+vqs7DgiqbTvdaT+K3ptqwzdJmNc9JYAun+j0QBae 7AboNgbfU8v9VPxj5nmgHoLhu4gnUy6QW/u+UtbhElPKKl+uWSB8sno/j9ZONdogYAlc aQLufAMyCIBONhDco8RGsYXEqd6c77U8MUJIv054R9Y1Pl38buDYzYUONBBZjC6y1qyg Lk4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PMGQxaBKPbON3ssZWMqrEm63rTUeWUDOhP6COLqPYSc=; b=kG9RdUTHLL1Hjc9tmBOD58d15Oyltexj41sxA1ElRhj7b5J/UM+u9sBNThFd1Au2r0 HUhYfjb0MOzCbctCWTVKP7wo6ezdVken3kFJuOKJqWMkw2fuv6ydEL0YlrK1gEuAhgYL UlTH3nWBviO+ZLUskhqZQC7OccIDIA7QZtJDFoSPiMnlJZ0yGML0k7Lpgiyho8+X8/Yw bJE4Qq3c9qG9XA8AvKN1sYaQAcnBmjk1rSwZ4GV9aNw3Wi36s0NV34xQAHKB2mN47RAy //fTfbWywuMM/soI85OpNvw0MVXBXwvrBooF8nnedCMPECdPlq2FOnLzSVDcKm088S1u vTHQ== X-Gm-Message-State: AOAM5310K4jtVa0MnjseKHV4GdvgqBYs0u7HlbHlILDLYomgUDEnfB/1 bb700/F4uwS/3AQAkQYiPhGM6pt/+es= X-Google-Smtp-Source: ABdhPJzJadmdcr+E5dPdAeWoEtRrezvpoEws2ofoV8cHCzMpxKfGsMWs/5DPALVlJiaUIB6ACL7lKA== X-Received: by 2002:aa7:96f4:: with SMTP id i20mr619172pfq.312.1598608837398; Fri, 28 Aug 2020 03:00:37 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id 78sm1068608pfv.200.2020.08.28.03.00.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2020 03:00:37 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, "Aneesh Kumar K.V" , Andrew Morton , Jens Axboe , Peter Zijlstra , "David S. Miller" Subject: [PATCH 1/4] mm: fix exec activate_mm vs TLB shootdown and lazy tlb switching race Date: Fri, 28 Aug 2020 20:00:19 +1000 Message-Id: <20200828100022.1099682-2-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200828100022.1099682-1-npiggin@gmail.com> References: <20200828100022.1099682-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 24087180F8B82 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Reading and modifying current->mm and current->active_mm and switching mm should be done with irqs off, to prevent races seeing an intermediate state. This is similar to commit 38cf307c1f20 ("mm: fix kthread_use_mm() vs TLB invalidate"). At exec-time when the new mm is activated, the old one should usually be single-threaded and no longer used, unless something else is holding an mm_users reference (which may be possible). Absent other mm_users, there is also a race with preemption and lazy tlb switching. Consider the kernel_execve case where the current thread is using a lazy tlb active mm: call_usermodehelper() kernel_execve() old_mm = current->mm; active_mm = current->active_mm; *** preempt *** --------------------> schedule() prev->active_mm = NULL; mmdrop(prev active_mm); ... <-------------------- schedule() current->mm = mm; current->active_mm = mm; if (!old_mm) mmdrop(active_mm); If we switch back to the kernel thread from a different mm, there is a double free of the old active_mm, and a missing free of the new one. Closing this race only requires interrupts to be disabled while ->mm and ->active_mm are being switched, but the TLB problem requires also holding interrupts off over activate_mm. Unfortunately not all archs can do that yet, e.g., arm defers the switch if irqs are disabled and expects finish_arch_post_lock_switch() to be called to complete the flush; um takes a blocking lock in activate_mm(). So as a first step, disable interrupts across the mm/active_mm updates to close the lazy tlb preempt race, and provide an arch option to extend that to activate_mm which allows architectures doing IPI based TLB shootdowns to close the second race. This is a bit ugly, but in the interest of fixing the bug and backporting before all architectures are converted this is a compromise. Signed-off-by: Nicholas Piggin --- arch/Kconfig | 7 +++++++ fs/exec.c | 17 +++++++++++++++-- 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index af14a567b493..94821e3f94d1 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -414,6 +414,13 @@ config MMU_GATHER_NO_GATHER bool depends on MMU_GATHER_TABLE_FREE +config ARCH_WANT_IRQS_OFF_ACTIVATE_MM + bool + help + Temporary select until all architectures can be converted to have + irqs disabled over activate_mm. Architectures that do IPI based TLB + shootdowns should enable this. + config ARCH_HAVE_NMI_SAFE_CMPXCHG bool diff --git a/fs/exec.c b/fs/exec.c index a91003e28eaa..d4fb18baf1fb 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -1130,11 +1130,24 @@ static int exec_mmap(struct mm_struct *mm) } task_lock(tsk); - active_mm = tsk->active_mm; membarrier_exec_mmap(mm); - tsk->mm = mm; + + local_irq_disable(); + active_mm = tsk->active_mm; tsk->active_mm = mm; + tsk->mm = mm; + /* + * This prevents preemption while active_mm is being loaded and + * it and mm are being updated, which could cause problems for + * lazy tlb mm refcounting when these are updated by context + * switches. Not all architectures can handle irqs off over + * activate_mm yet. + */ + if (!IS_ENABLED(CONFIG_ARCH_WANT_IRQS_OFF_ACTIVATE_MM)) + local_irq_enable(); activate_mm(active_mm, mm); + if (IS_ENABLED(CONFIG_ARCH_WANT_IRQS_OFF_ACTIVATE_MM)) + local_irq_enable(); tsk->mm->vmacache_seqnum = 0; vmacache_flush(tsk); task_unlock(tsk); From patchwork Fri Aug 28 10:00:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11742533 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AF890138A for ; Fri, 28 Aug 2020 10:00:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7B1442078A for ; Fri, 28 Aug 2020 10:00:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bRRBgNBj" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7B1442078A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A4A988D0001; Fri, 28 Aug 2020 06:00:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9FCC26B000E; Fri, 28 Aug 2020 06:00:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 911758D0001; Fri, 28 Aug 2020 06:00:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0002.hostedemail.com [216.40.44.2]) by kanga.kvack.org (Postfix) with ESMTP id 7C7EB6B000D for ; Fri, 28 Aug 2020 06:00:44 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 3F098B7AD for ; Fri, 28 Aug 2020 10:00:44 +0000 (UTC) X-FDA: 77199533208.12.cause78_400c78627074 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id 096061801532F for ; Fri, 28 Aug 2020 10:00:44 +0000 (UTC) X-Spam-Summary: 1,0,0,3f3a1020fbca8b83,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3867:3870:3871:4250:4321:4605:5007:6119:6261:6653:7514:7903:9413:10004:11026:11473:11657:11658:11914:12043:12114:12296:12297:12438:12517:12519:12555:12895:12986:13069:13311:13357:13894:14096:14181:14384:14394:14687:14721:21080:21433:21444:21451:21627:21666:21987:30054:30089,0,RBL:209.85.214.193:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.50.0.100;04yf9b3ko71ra5ghzgko8cjd8w8xrocozshj5ze8f34c73bbr9gjgr7eriahjep.phoy9de4jbh4yk4igz8jsnkfxse3abzd6c7pcsko33cfq8bc9oqta1ik81krz7k.g-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: cause78_400c78627074 X-Filterd-Recvd-Size: 4630 Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) by imf34.hostedemail.com (Postfix) with ESMTP for ; Fri, 28 Aug 2020 10:00:43 +0000 (UTC) Received: by mail-pl1-f193.google.com with SMTP id c15so245269plq.4 for ; Fri, 28 Aug 2020 03:00:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NQ1kOxEaL9FVx1zYJ6CutcXNmCnj+fa5P2Lt3GvzHyI=; b=bRRBgNBjutZV34MKXKG3E/VwB3IKH7AOxXMifwoUlmRbz0QhDAlR12gCR3RgEp/4kJ GHdQnSrAbNR0hYRWAxQJ2UTdupNx3n7YuRnohk7oeBcdDM09JguHSVOUfX2qmpfI7F7+ u6SNJ+4RfnJrk//q1ccvcWLjfV5ugaWXI7HwDTBw9LQDMrlTpVUkfGveH3lmvTExpQu5 O4MinePSx7ves2sQvMrRYW4Gz7PEA9nDzz3ThtjN/h3oEsPzaCd0GqRFAPTmhc0NBBu6 98et6/pdtFvL/SFmG/Wi+MPArcGOyX2UV83ogTXzHMq71tgmKgvXDU62rTcVXUUQxflh 6Jxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NQ1kOxEaL9FVx1zYJ6CutcXNmCnj+fa5P2Lt3GvzHyI=; b=uGhAIivXhUhqnN7NRu3CmyXSi4brLCIgeDJMvt21KvzL3lnAUdDJ4YgVw5uxXfux46 ip32kCuuD9WM72Lp3GBQUQjPq3DwTQvE7G++2DfDQfgXAlm1UunOG9LWMN6FgogMV/2j XsZB8jcq7C1+1/5TOuShDxfAL2qUY2who7Kwq1rMp6QEkdU/LN9PwAlW2ghMUJwR34ei HsNhcgC7IHS9eUW0I41FCBcnIvDIh7/SihTHqTq52fJHuLNyYo9279CUxQIqS38Eng1f Fe7D2KgUQr6SX66nhj0pOdqpeH/UuKi4nQeqSqeeTbD7Dm0S0SHL7BG9Hd+L60uszG+5 sg9w== X-Gm-Message-State: AOAM531NV/iUG6li8RsR+d/5+dfRYXxnpjrBRcNhq/8i6AZ6AYD69uWF yv6UQFgRa/VYXc4E8pz5lUZS/l3GOVU= X-Google-Smtp-Source: ABdhPJxa/QRl5EZgnlzwuYW8jlGMI75oSzDlqSUeAaqnONi5FlxmVIhKGxJLmK+WQt5JYnUs+sDK6A== X-Received: by 2002:a17:902:b588:: with SMTP id a8mr712053pls.96.1598608842326; Fri, 28 Aug 2020 03:00:42 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id 78sm1068608pfv.200.2020.08.28.03.00.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2020 03:00:41 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, "Aneesh Kumar K.V" , Andrew Morton , Jens Axboe , Peter Zijlstra , "David S. Miller" Subject: [PATCH 2/4] powerpc: select ARCH_WANT_IRQS_OFF_ACTIVATE_MM Date: Fri, 28 Aug 2020 20:00:20 +1000 Message-Id: <20200828100022.1099682-3-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200828100022.1099682-1-npiggin@gmail.com> References: <20200828100022.1099682-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 096061801532F X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: powerpc uses IPIs in some situations to switch a kernel thread away from a lazy tlb mm, which is subject to the TLB flushing race described in the changelog introducing ARCH_WANT_IRQS_OFF_ACTIVATE_MM. Signed-off-by: Nicholas Piggin --- arch/powerpc/Kconfig | 1 + arch/powerpc/include/asm/mmu_context.h | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 1f48bbfb3ce9..65cb32211574 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -149,6 +149,7 @@ config PPC select ARCH_USE_QUEUED_RWLOCKS if PPC_QUEUED_SPINLOCKS select ARCH_USE_QUEUED_SPINLOCKS if PPC_QUEUED_SPINLOCKS select ARCH_WANT_IPC_PARSE_VERSION + select ARCH_WANT_IRQS_OFF_ACTIVATE_MM select ARCH_WEAK_RELEASE_ACQUIRE select BINFMT_ELF select BUILDTIME_TABLE_SORT diff --git a/arch/powerpc/include/asm/mmu_context.h b/arch/powerpc/include/asm/mmu_context.h index 7f3658a97384..e02aa793420b 100644 --- a/arch/powerpc/include/asm/mmu_context.h +++ b/arch/powerpc/include/asm/mmu_context.h @@ -244,7 +244,7 @@ static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next, */ static inline void activate_mm(struct mm_struct *prev, struct mm_struct *next) { - switch_mm(prev, next, current); + switch_mm_irqs_off(prev, next, current); } /* We don't currently use enter_lazy_tlb() for anything */ From patchwork Fri Aug 28 10:00:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11742535 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1E510138A for ; Fri, 28 Aug 2020 10:00:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CDBB62078A for ; Fri, 28 Aug 2020 10:00:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SVMi9H9M" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CDBB62078A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0ADBB8E0003; Fri, 28 Aug 2020 06:00:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 05E906B000E; Fri, 28 Aug 2020 06:00:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E67B38E0003; Fri, 28 Aug 2020 06:00:50 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0093.hostedemail.com [216.40.44.93]) by kanga.kvack.org (Postfix) with ESMTP id D34D46B000D for ; Fri, 28 Aug 2020 06:00:50 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 98B9CB2BE for ; Fri, 28 Aug 2020 10:00:50 +0000 (UTC) X-FDA: 77199533460.17.baby77_4b10d0627074 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id EC180180D0181 for ; Fri, 28 Aug 2020 10:00:48 +0000 (UTC) X-Spam-Summary: 1,0,0,87f971f8890b3467,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:2:41:69:355:379:541:800:960:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1730:1747:1777:1792:2393:2559:2562:2693:2895:3138:3139:3140:3141:3142:3165:3865:3866:3867:3868:3870:3871:3872:3873:3874:4037:4049:4120:4250:4321:4362:4605:5007:6119:6261:6653:7514:7903:8660:9008:9010:9413:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12683:12895:13148:13161:13229:13230:13894:14096:14394:14687:21080:21433:21444:21451:21627:21666:21740:21795:21809:21939:21972:21990:30003:30012:30034:30045:30051:30054:30056:30070:30075,0,RBL:209.85.216.68:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100;04yfua6xnj3kb4eycicuebajxw4fhycy93auxsoaxwm7ge8wa73pgdf38t5zebo.g7a6hyw818a37jhuzjnzq34tywjx1qinjyft3pa5we54e7sbbbsg9cayw13199k.y-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_ rules:0: X-HE-Tag: baby77_4b10d0627074 X-Filterd-Recvd-Size: 9523 Received: from mail-pj1-f68.google.com (mail-pj1-f68.google.com [209.85.216.68]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Fri, 28 Aug 2020 10:00:48 +0000 (UTC) Received: by mail-pj1-f68.google.com with SMTP id nv17so288966pjb.3 for ; Fri, 28 Aug 2020 03:00:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=t1XpS/HyClfw6pT/eMXqAbIXlcXWXqL02aliRjQU+Do=; b=SVMi9H9MJSwkxZS4sgc+XDPQgZVOYrHw/JEZlNn2qzPuodctA5rT7WpDP8xuiTQjly X0D2OJjTMOoqbxFDG/g8DkoZpjgu7jIBQFJ80Slq1BkwHqZaWF9oyKtnqCKmYKd61nxI Yp5xE8WBYoOmHck6mFx5BQ7Y+Dcivra0zWhSwoIGE69lPX5uSwulVVD6ttCYihuretrf ZPbBHnedxpLGc/cTg26SzEDCf6l6FZ7x5uZlMwGyF0Hk1fu/TVbYwGppEsB3dsB5H0gk 4T9dQC8KAAD6JEJi8n5bGxKTB/C3tGLdvMJ0k7fy9HPp2irNvVgYsbIih8sf9HtKzHOO jM5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=t1XpS/HyClfw6pT/eMXqAbIXlcXWXqL02aliRjQU+Do=; b=hGXpN1femoupVKFBquQk2qhtiatbAiofKAFtBQu87VIvOJJJmhNaRy0p/n0MIEJtin TGxHdDbrMmvCBe+2JPeSwJDm/xhWCKKyfzGdCWj5OWQnBt+ab/yrLf9HzOoBZtqut0Ts HOAG+/9bf15tcrb8qXtBY8qMmln9bqp/S8DVle/C9qGdwJ6h7GRuFbhBNvnl+/RUj9al o3fIiyUWKxmsWc+QDuo+JAX2RUnvud4zm+GOIuJ3S3lpYQSMrHBCF+muCuAjisEI6iht aQuLQkG5XaUl4bIvVjQs0TXe+N7YrP4AoOIUc8PetYBw2Z9sAume4XNr1ccIGEvRt3nl UY3A== X-Gm-Message-State: AOAM531dmJeyGndcQYWG8AP/Uc1YbWj4LEX6tYLZ7FbxxTJq4Smdz3sN 32MjVmF2RdlV0tM/j25p16djhWBEK90= X-Google-Smtp-Source: ABdhPJy4OYhts/oK1ycV+d0OGCx/rtyS/ekQliVuUahTMGfIyFq0gV8RJk/8jgwzJpUyRd+KfS/ZuQ== X-Received: by 2002:a17:902:a503:: with SMTP id s3mr722211plq.190.1598608847187; Fri, 28 Aug 2020 03:00:47 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id 78sm1068608pfv.200.2020.08.28.03.00.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2020 03:00:46 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, "Aneesh Kumar K.V" , Andrew Morton , Jens Axboe , Peter Zijlstra , "David S. Miller" , sparclinux@vger.kernel.org Subject: [PATCH 3/4] sparc64: remove mm_cpumask clearing to fix kthread_use_mm race Date: Fri, 28 Aug 2020 20:00:21 +1000 Message-Id: <20200828100022.1099682-4-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200828100022.1099682-1-npiggin@gmail.com> References: <20200828100022.1099682-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: EC180180D0181 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The de facto (and apparently uncommented) standard for using an mm had, thanks to this code in sparc if nothing else, been that you must have a reference on mm_users *and that reference must have been obtained with mmget()*, i.e., from a thread with a reference to mm_users that had used the mm. The introduction of mmget_not_zero() in commit d2005e3f41d4 ("userfaultfd: don't pin the user memory in userfaultfd_file_create()") allowed mm_count holders to aoperate on user mappings asynchronously from the actual threads using the mm, but they were not to load those mappings into their TLB (i.e., walking vmas and page tables is okay, kthread_use_mm() is not). io_uring 2b188cc1bb857 ("Add io_uring IO interface") added code which does a kthread_use_mm() from a mmget_not_zero() refcount. The problem with this is code which previously assumed mm == current->mm and mm->mm_users == 1 implies the mm will remain single-threaded at least until this thread creates another mm_users reference, has now broken. arch/sparc/kernel/smp_64.c: if (atomic_read(&mm->mm_users) == 1) { cpumask_copy(mm_cpumask(mm), cpumask_of(cpu)); goto local_flush_and_out; } vs fs/io_uring.c if (unlikely(!(ctx->flags & IORING_SETUP_SQPOLL) || !mmget_not_zero(ctx->sqo_mm))) return -EFAULT; kthread_use_mm(ctx->sqo_mm); mmget_not_zero() could come in right after the mm_users == 1 test, then kthread_use_mm() which sets its CPU in the mm_cpumask. That update could be lost if cpumask_copy() occurs afterward. I propose we fix this by allowing mmget_not_zero() to be a first-class reference, and not have this obscure undocumented and unchecked restriction. The basic fix for sparc64 is to remove its mm_cpumask clearing code. The optimisation could be effectively restored by sending IPIs to mm_cpumask members and having them remove themselves from mm_cpumask. This is more tricky so I leave it as an exercise for someone with a sparc64 SMP. powerpc has a (currently similarly broken) example. Cc: sparclinux@vger.kernel.org Signed-off-by: Nicholas Piggin --- arch/sparc/kernel/smp_64.c | 65 ++++++++------------------------------ 1 file changed, 14 insertions(+), 51 deletions(-) diff --git a/arch/sparc/kernel/smp_64.c b/arch/sparc/kernel/smp_64.c index e286e2badc8a..e38d8bf454e8 100644 --- a/arch/sparc/kernel/smp_64.c +++ b/arch/sparc/kernel/smp_64.c @@ -1039,38 +1039,9 @@ void smp_fetch_global_pmu(void) * are flush_tlb_*() routines, and these run after flush_cache_*() * which performs the flushw. * - * The SMP TLB coherency scheme we use works as follows: - * - * 1) mm->cpu_vm_mask is a bit mask of which cpus an address - * space has (potentially) executed on, this is the heuristic - * we use to avoid doing cross calls. - * - * Also, for flushing from kswapd and also for clones, we - * use cpu_vm_mask as the list of cpus to make run the TLB. - * - * 2) TLB context numbers are shared globally across all processors - * in the system, this allows us to play several games to avoid - * cross calls. - * - * One invariant is that when a cpu switches to a process, and - * that processes tsk->active_mm->cpu_vm_mask does not have the - * current cpu's bit set, that tlb context is flushed locally. - * - * If the address space is non-shared (ie. mm->count == 1) we avoid - * cross calls when we want to flush the currently running process's - * tlb state. This is done by clearing all cpu bits except the current - * processor's in current->mm->cpu_vm_mask and performing the - * flush locally only. This will force any subsequent cpus which run - * this task to flush the context from the local tlb if the process - * migrates to another cpu (again). - * - * 3) For shared address spaces (threads) and swapping we bite the - * bullet for most cases and perform the cross call (but only to - * the cpus listed in cpu_vm_mask). - * - * The performance gain from "optimizing" away the cross call for threads is - * questionable (in theory the big win for threads is the massive sharing of - * address space state across processors). + * mm->cpu_vm_mask is a bit mask of which cpus an address + * space has (potentially) executed on, this is the heuristic + * we use to limit cross calls. */ /* This currently is only used by the hugetlb arch pre-fault @@ -1080,18 +1051,13 @@ void smp_fetch_global_pmu(void) void smp_flush_tlb_mm(struct mm_struct *mm) { u32 ctx = CTX_HWBITS(mm->context); - int cpu = get_cpu(); - if (atomic_read(&mm->mm_users) == 1) { - cpumask_copy(mm_cpumask(mm), cpumask_of(cpu)); - goto local_flush_and_out; - } + get_cpu(); smp_cross_call_masked(&xcall_flush_tlb_mm, ctx, 0, 0, mm_cpumask(mm)); -local_flush_and_out: __flush_tlb_mm(ctx, SECONDARY_CONTEXT); put_cpu(); @@ -1114,17 +1080,15 @@ void smp_flush_tlb_pending(struct mm_struct *mm, unsigned long nr, unsigned long { u32 ctx = CTX_HWBITS(mm->context); struct tlb_pending_info info; - int cpu = get_cpu(); + + get_cpu(); info.ctx = ctx; info.nr = nr; info.vaddrs = vaddrs; - if (mm == current->mm && atomic_read(&mm->mm_users) == 1) - cpumask_copy(mm_cpumask(mm), cpumask_of(cpu)); - else - smp_call_function_many(mm_cpumask(mm), tlb_pending_func, - &info, 1); + smp_call_function_many(mm_cpumask(mm), tlb_pending_func, + &info, 1); __flush_tlb_pending(ctx, nr, vaddrs); @@ -1134,14 +1098,13 @@ void smp_flush_tlb_pending(struct mm_struct *mm, unsigned long nr, unsigned long void smp_flush_tlb_page(struct mm_struct *mm, unsigned long vaddr) { unsigned long context = CTX_HWBITS(mm->context); - int cpu = get_cpu(); - if (mm == current->mm && atomic_read(&mm->mm_users) == 1) - cpumask_copy(mm_cpumask(mm), cpumask_of(cpu)); - else - smp_cross_call_masked(&xcall_flush_tlb_page, - context, vaddr, 0, - mm_cpumask(mm)); + get_cpu(); + + smp_cross_call_masked(&xcall_flush_tlb_page, + context, vaddr, 0, + mm_cpumask(mm)); + __flush_tlb_page(context, vaddr); put_cpu(); From patchwork Fri Aug 28 10:00:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11742537 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 48235138A for ; Fri, 28 Aug 2020 10:00:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 145D42078A for ; Fri, 28 Aug 2020 10:00:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AP/FuUUz" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 145D42078A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1251C6B000D; Fri, 28 Aug 2020 06:00:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0D4368E0006; Fri, 28 Aug 2020 06:00:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F2CEB6B0010; Fri, 28 Aug 2020 06:00:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0199.hostedemail.com [216.40.44.199]) by kanga.kvack.org (Postfix) with ESMTP id DFD0E6B000D for ; Fri, 28 Aug 2020 06:00:53 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 85748180AD815 for ; Fri, 28 Aug 2020 10:00:53 +0000 (UTC) X-FDA: 77199533586.15.mind94_4401f6827074 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin15.hostedemail.com (Postfix) with ESMTP id 52B5D1814B0C7 for ; Fri, 28 Aug 2020 10:00:53 +0000 (UTC) X-Spam-Summary: 1,0,0,3c892ad2b3a8fba5,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2198:2199:2393:2553:2559:2562:2693:2901:2904:3138:3139:3140:3141:3142:3165:3355:3865:3866:3867:3868:3870:3871:3872:3874:4118:4250:4321:4605:5007:6119:6261:6653:7514:7901:7903:9413:10004:11026:11232:11473:11657:11658:11914:12043:12296:12297:12438:12517:12519:12555:12679:12895:12986:13894:14096:14181:14394:14687:14721:21080:21433:21444:21451:21627:21666:21990:30012:30054:30070:30090,0,RBL:209.85.216.66:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.50.0.100;04yr3ra8bjoedat13cbibfwhhdx5jypyhwtjxw3a146md1sou8x5b9za3ku8dr3.ygjcax3qegpah6iptzgq9m9smm1z1gxdsj1rkxqsxm3rezzpw5tzefd4qym6zsh.a-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:26,LUA_SUMMARY:none X-HE-Tag: mind94_4401f6827074 X-Filterd-Recvd-Size: 7010 Received: from mail-pj1-f66.google.com (mail-pj1-f66.google.com [209.85.216.66]) by imf30.hostedemail.com (Postfix) with ESMTP for ; Fri, 28 Aug 2020 10:00:52 +0000 (UTC) Received: by mail-pj1-f66.google.com with SMTP id i13so295685pjv.0 for ; Fri, 28 Aug 2020 03:00:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1Y/faACy8cJZqe+8jbvX5+3MT7IMXQLldBAzW7OQGFU=; b=AP/FuUUz2c24R7h91evEMtU7s8yHhwXL6DM9u/2Mj6P86m31kOa2XEFxhYVFf9tZrE jv91cc3rMnttdOv5nUM25o4L9TDkox0OKTI4NdHKWJ2t8G5X6kuCjT11G/p+JGAnetfj aAdB92JLJZ+7ALLMOIx0G2O39sEKINQW7bkStFpXhtzHjTjJGrWOw521+Rp98GWmDpbD fJqSEgzfi/V8ctIwTqI7EUTk+Eu0WQf2y+CmgiuYKQkw+ftlvW12QmS43WzjL5wegPaU +VekSmy9gjIfJo4DEQR7zjgtk3AuxjdbAtl+UiSUVXtR4ga0IDGrVC4DCumTFiQTzqNp WBZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1Y/faACy8cJZqe+8jbvX5+3MT7IMXQLldBAzW7OQGFU=; b=TGpc3zzzw/CCWGjtQCe8L3w6tps5wFvfpK+A7oE4iVNMbC+WC48arWgvvdv6vu2awH sCoz5u3jiQjgEg8+CR2h5Ibzqa7l0+//DSCJADJ+ucG9xi1+cUyk897vtZRgpDRgzfaW vtm/tGZZaAqve0qYHksM7LJypJVXJVeOW7bNlU4UN4ZdaNK4EmxAgJuFOUIqF1JhZXRs bFD8hoE4jEPXTU+calFNMYkPFOs8Sx25O1Y4JJFTRm+s2QPgUisriHcSHM8K6SQDdI94 IdtSp7RKWt9RhSd4DvJVs3TcIfdzWuswn8E0LvW83AKkqBkAg/94QsnsTlYgKECqVyxd yxpA== X-Gm-Message-State: AOAM530JFhMZEX95znySxTLExT2vzkIhxZJJqyxv0Sopeup+ByJqqQqL +a4rms5GqNmp3GNwLWpX5R70mCqdrhs= X-Google-Smtp-Source: ABdhPJxv4t8TEETyZrq5vtkYIY4wLkQzivB6f4HHzIzllMBpn1clbhbAwA9g7HebVxjFpblwPrb97w== X-Received: by 2002:a17:90b:100e:: with SMTP id gm14mr556462pjb.39.1598608851815; Fri, 28 Aug 2020 03:00:51 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id 78sm1068608pfv.200.2020.08.28.03.00.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2020 03:00:51 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, "Aneesh Kumar K.V" , Andrew Morton , Jens Axboe , Peter Zijlstra , "David S. Miller" Subject: [PATCH 4/4] powerpc/64s/radix: Fix mm_cpumask trimming race vs kthread_use_mm Date: Fri, 28 Aug 2020 20:00:22 +1000 Message-Id: <20200828100022.1099682-5-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200828100022.1099682-1-npiggin@gmail.com> References: <20200828100022.1099682-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 52B5D1814B0C7 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Commit 0cef77c7798a7 ("powerpc/64s/radix: flush remote CPUs out of single-threaded mm_cpumask") added a mechanism to trim the mm_cpumask of a process under certain conditions. One of the assumptions is that mm_users would not be incremented via a reference outside the process context with mmget_not_zero() then go on to kthread_use_mm() via that reference. That invariant was broken by io_uring code (see previous sparc64 fix), but I'll point Fixes: to the original powerpc commit because we are changing that assumption going forward, so this will make backports match up. Fix this by no longer relying on that assumption, but by having each CPU check the mm is not being used, and clearing their own bit from the mask if it's okay. This fix relies on commit 38cf307c1f20 ("mm: fix kthread_use_mm() vs TLB invalidate") to disable irqs over the mm switch, and ARCH_WANT_IRQS_OFF_ACTIVATE_MM to be enabled. Fixes: 0cef77c7798a7 ("powerpc/64s/radix: flush remote CPUs out of single-threaded mm_cpumask") Signed-off-by: Nicholas Piggin Reviewed-by: Michael Ellerman --- arch/powerpc/include/asm/tlb.h | 13 ------------- arch/powerpc/mm/book3s64/radix_tlb.c | 23 ++++++++++++++++------- 2 files changed, 16 insertions(+), 20 deletions(-) diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index fbc6f3002f23..d97f061fecac 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -66,19 +66,6 @@ static inline int mm_is_thread_local(struct mm_struct *mm) return false; return cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm)); } -static inline void mm_reset_thread_local(struct mm_struct *mm) -{ - WARN_ON(atomic_read(&mm->context.copros) > 0); - /* - * It's possible for mm_access to take a reference on mm_users to - * access the remote mm from another thread, but it's not allowed - * to set mm_cpumask, so mm_users may be > 1 here. - */ - WARN_ON(current->mm != mm); - atomic_set(&mm->context.active_cpus, 1); - cpumask_clear(mm_cpumask(mm)); - cpumask_set_cpu(smp_processor_id(), mm_cpumask(mm)); -} #else /* CONFIG_PPC_BOOK3S_64 */ static inline int mm_is_thread_local(struct mm_struct *mm) { diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 0d233763441f..a421a0e3f930 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -645,19 +645,29 @@ static void do_exit_flush_lazy_tlb(void *arg) struct mm_struct *mm = arg; unsigned long pid = mm->context.id; + /* + * A kthread could have done a mmget_not_zero() after the flushing CPU + * checked mm_users == 1, and be in the process of kthread_use_mm when + * interrupted here. In that case, current->mm will be set to mm, + * because kthread_use_mm() setting ->mm and switching to the mm is + * done with interrupts off. + */ if (current->mm == mm) - return; /* Local CPU */ + goto out_flush; if (current->active_mm == mm) { - /* - * Must be a kernel thread because sender is single-threaded. - */ - BUG_ON(current->mm); + WARN_ON_ONCE(current->mm != NULL); + /* Is a kernel thread and is using mm as the lazy tlb */ mmgrab(&init_mm); - switch_mm(mm, &init_mm, current); current->active_mm = &init_mm; + switch_mm_irqs_off(mm, &init_mm, current); mmdrop(mm); } + + atomic_dec(&mm->context.active_cpus); + cpumask_clear_cpu(smp_processor_id(), mm_cpumask(mm)); + +out_flush: _tlbiel_pid(pid, RIC_FLUSH_ALL); } @@ -672,7 +682,6 @@ static void exit_flush_lazy_tlbs(struct mm_struct *mm) */ smp_call_function_many(mm_cpumask(mm), do_exit_flush_lazy_tlb, (void *)mm, 1); - mm_reset_thread_local(mm); } void radix__flush_tlb_mm(struct mm_struct *mm)