From patchwork Mon Sep 14 04:52:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11772821 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 493546CA for ; Mon, 14 Sep 2020 04:52:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F26CC21655 for ; Mon, 14 Sep 2020 04:52:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="E6t5ATL8" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F26CC21655 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 33D5C6B0037; Mon, 14 Sep 2020 00:52:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 313CD6B0055; Mon, 14 Sep 2020 00:52:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 204F26B005A; Mon, 14 Sep 2020 00:52:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0186.hostedemail.com [216.40.44.186]) by kanga.kvack.org (Postfix) with ESMTP id 0BC1B6B0037 for ; Mon, 14 Sep 2020 00:52:43 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id B2DBD181AEF1D for ; Mon, 14 Sep 2020 04:52:42 +0000 (UTC) X-FDA: 77260446564.17.print54_0106bc927105 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id 7B520180D0180 for ; Mon, 14 Sep 2020 04:52:42 +0000 (UTC) X-Spam-Summary: 1,0,0,3d2d607be28d345f,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:1801:2194:2196:2198:2199:2200:2201:2393:2553:2559:2562:2689:2895:2914:3138:3139:3140:3141:3142:3355:3865:3866:3867:3868:3870:3871:3872:3874:4118:4250:4321:4385:4605:5007:6119:6120:6261:6653:7514:7901:7903:8999:9000:9149:9413:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:12986:13255:13894:14096:14181:14394:14687:14721:21080:21433:21444:21451:21627:21666:21966:30045:30054:30090,0,RBL:209.85.215.196:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.50.0.100;04yrn9ff67wjpyrz5xq95suuhpre1yp1kgxyhsgt9yo97dz9iodrdeg9fk7a5dq.ntthco1cqkys7eidfts34knozrireyqkbn83j1h44pe9wdn5nytmz3ejrxfajij.c-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SU MMARY:no X-HE-Tag: print54_0106bc927105 X-Filterd-Recvd-Size: 7018 Received: from mail-pg1-f196.google.com (mail-pg1-f196.google.com [209.85.215.196]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Mon, 14 Sep 2020 04:52:41 +0000 (UTC) Received: by mail-pg1-f196.google.com with SMTP id u13so10563807pgh.1 for ; Sun, 13 Sep 2020 21:52:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=PMGQxaBKPbON3ssZWMqrEm63rTUeWUDOhP6COLqPYSc=; b=E6t5ATL8gx3qXPR8MutckEOtFp8N/QqyJhBR7KZvwdtkFuAEyC2lhZ2/emCtjTKBmo B0kb2CydVdVNlaob1xulncfmSHsG5MpRJYpG32az5As5jUlalJv5Z1u24HGgsukfOob0 pStFmcZQbhM6RQl6+oGHRZ/T4R+TM1Eyv8T8Wo3EQa0oEPAufWykBkJADQ737q0p9bQY 2R33AJCLbtCHvjVUHJysl92cDr8E/Gps9+AWbHKehyystd+wcHrhzluGm/ArtKvSmZCu bm9w6OFAfNAC+nOJ6NR0QDHwYy2vZD6WeoHYvXZICjq3HwmMDhUuXmKAzk9mSX/KjWa1 XD6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PMGQxaBKPbON3ssZWMqrEm63rTUeWUDOhP6COLqPYSc=; b=RHnuba9xddBcohyEmK0ZeJPfmEZe3Wly9YgJYnJRQa/LSLbzkGAfIACnz6tGWtsqRi bER3KSCU/AwzzeNXjLlVtl7KdKA4ZUOYBziwWXU9kx+R21JNKEok5plQuF8z86MsV7wC Q7um2REtibVySAOgIZk42RoaIej//6A5tTKYuf14vwFIUJ8BVcoGxIxmRfQlplWCroXK e20+9hXg0BgrV8QCguzDA6xYSNCJvLFzkwLiXcRFHrUn8zOhyxc8KArSvSd6sJR0ePkT gRBiasHcMrAgAI24anUC4k6YgHBPFYiaFoYFKgFDL7nc19ShXA9NqIKkev3wE3vuSVBa jLvA== X-Gm-Message-State: AOAM532HQp5/n+iK11knuZxYnV0p6EwAZ+ni78vk8OEcjJgBnZsEQ+AL OBO0XCrbfinEPwSBjGdXhEsNYl5Vqho= X-Google-Smtp-Source: ABdhPJyPI7rDf+d21q9NzzsfoVdINl/UoWB/tNDVu6ib1rFaHHxGEc6qLhJCwAhX5Wm+j4pnSX9vTQ== X-Received: by 2002:a17:902:eec7:b029:d1:c2e4:6b58 with SMTP id h7-20020a170902eec7b02900d1c2e46b58mr4791803plb.4.1600059161060; Sun, 13 Sep 2020 21:52:41 -0700 (PDT) Received: from bobo.ozlabs.ibm.com ([203.185.249.227]) by smtp.gmail.com with ESMTPSA id a13sm6945312pgq.41.2020.09.13.21.52.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 13 Sep 2020 21:52:40 -0700 (PDT) From: Nicholas Piggin To: "linux-mm @ kvack . org" Cc: Nicholas Piggin , linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org, "Aneesh Kumar K . V" , Andrew Morton , Jens Axboe , Peter Zijlstra , "David S . Miller" Subject: [PATCH v2 1/4] mm: fix exec activate_mm vs TLB shootdown and lazy tlb switching race Date: Mon, 14 Sep 2020 14:52:16 +1000 Message-Id: <20200914045219.3736466-2-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200914045219.3736466-1-npiggin@gmail.com> References: <20200914045219.3736466-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 7B520180D0180 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Reading and modifying current->mm and current->active_mm and switching mm should be done with irqs off, to prevent races seeing an intermediate state. This is similar to commit 38cf307c1f20 ("mm: fix kthread_use_mm() vs TLB invalidate"). At exec-time when the new mm is activated, the old one should usually be single-threaded and no longer used, unless something else is holding an mm_users reference (which may be possible). Absent other mm_users, there is also a race with preemption and lazy tlb switching. Consider the kernel_execve case where the current thread is using a lazy tlb active mm: call_usermodehelper() kernel_execve() old_mm = current->mm; active_mm = current->active_mm; *** preempt *** --------------------> schedule() prev->active_mm = NULL; mmdrop(prev active_mm); ... <-------------------- schedule() current->mm = mm; current->active_mm = mm; if (!old_mm) mmdrop(active_mm); If we switch back to the kernel thread from a different mm, there is a double free of the old active_mm, and a missing free of the new one. Closing this race only requires interrupts to be disabled while ->mm and ->active_mm are being switched, but the TLB problem requires also holding interrupts off over activate_mm. Unfortunately not all archs can do that yet, e.g., arm defers the switch if irqs are disabled and expects finish_arch_post_lock_switch() to be called to complete the flush; um takes a blocking lock in activate_mm(). So as a first step, disable interrupts across the mm/active_mm updates to close the lazy tlb preempt race, and provide an arch option to extend that to activate_mm which allows architectures doing IPI based TLB shootdowns to close the second race. This is a bit ugly, but in the interest of fixing the bug and backporting before all architectures are converted this is a compromise. Signed-off-by: Nicholas Piggin Acked-by: Peter Zijlstra (Intel) --- arch/Kconfig | 7 +++++++ fs/exec.c | 17 +++++++++++++++-- 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index af14a567b493..94821e3f94d1 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -414,6 +414,13 @@ config MMU_GATHER_NO_GATHER bool depends on MMU_GATHER_TABLE_FREE +config ARCH_WANT_IRQS_OFF_ACTIVATE_MM + bool + help + Temporary select until all architectures can be converted to have + irqs disabled over activate_mm. Architectures that do IPI based TLB + shootdowns should enable this. + config ARCH_HAVE_NMI_SAFE_CMPXCHG bool diff --git a/fs/exec.c b/fs/exec.c index a91003e28eaa..d4fb18baf1fb 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -1130,11 +1130,24 @@ static int exec_mmap(struct mm_struct *mm) } task_lock(tsk); - active_mm = tsk->active_mm; membarrier_exec_mmap(mm); - tsk->mm = mm; + + local_irq_disable(); + active_mm = tsk->active_mm; tsk->active_mm = mm; + tsk->mm = mm; + /* + * This prevents preemption while active_mm is being loaded and + * it and mm are being updated, which could cause problems for + * lazy tlb mm refcounting when these are updated by context + * switches. Not all architectures can handle irqs off over + * activate_mm yet. + */ + if (!IS_ENABLED(CONFIG_ARCH_WANT_IRQS_OFF_ACTIVATE_MM)) + local_irq_enable(); activate_mm(active_mm, mm); + if (IS_ENABLED(CONFIG_ARCH_WANT_IRQS_OFF_ACTIVATE_MM)) + local_irq_enable(); tsk->mm->vmacache_seqnum = 0; vmacache_flush(tsk); task_unlock(tsk); From patchwork Mon Sep 14 04:52:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11772823 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 18AA459D for ; Mon, 14 Sep 2020 04:52:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B828D208DB for ; Mon, 14 Sep 2020 04:52:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZhI/Ifh3" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B828D208DB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id ED4146B0055; Mon, 14 Sep 2020 00:52:47 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E853B6B005A; Mon, 14 Sep 2020 00:52:47 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D77726B005C; Mon, 14 Sep 2020 00:52:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0002.hostedemail.com [216.40.44.2]) by kanga.kvack.org (Postfix) with ESMTP id B71216B0055 for ; Mon, 14 Sep 2020 00:52:47 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 7921A231AD for ; Mon, 14 Sep 2020 04:52:47 +0000 (UTC) X-FDA: 77260446774.06.hope80_2b0a01d27105 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id 463101048A00E for ; Mon, 14 Sep 2020 04:52:47 +0000 (UTC) X-Spam-Summary: 1,0,0,c991358b91cbaee3,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3867:3870:3871:4250:4321:4605:5007:6119:6261:6653:7514:7903:9413:10004:11026:11473:11657:11658:11914:12043:12114:12296:12297:12438:12517:12519:12555:12895:12986:13069:13311:13357:13894:14096:14181:14384:14394:14687:14721:21080:21433:21444:21451:21627:21666:21987:30054:30089,0,RBL:209.85.215.193:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.50.0.100;04yg5cjrk8suwzyfu67oqjjzjungcycozshj5ze8bayfipohd6gjgr7eriahjep.phoy9dd3yaaubsu3gz8jsnkfxi3ojhhwij5pcsko33cfq8bc9oqta1zc5hi1rwq.n-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: hope80_2b0a01d27105 X-Filterd-Recvd-Size: 4697 Received: from mail-pg1-f193.google.com (mail-pg1-f193.google.com [209.85.215.193]) by imf47.hostedemail.com (Postfix) with ESMTP for ; Mon, 14 Sep 2020 04:52:46 +0000 (UTC) Received: by mail-pg1-f193.google.com with SMTP id k14so172358pgi.9 for ; Sun, 13 Sep 2020 21:52:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qhSlIbBbaKajKS/IeKMTbHGZ3Q2g5fhRWyCc6GgGFOw=; b=ZhI/Ifh3Hh6S9ezSZcFBjkvaYu4u3PPYdLMSfBHM9LS26Zi7yErlcuHKgA3qBwnTfV iGDS/wM/bY3O58Mf9zXr4CxP6mSXYPzQgzApyoZV2tEVf9gLml7tNnK38HNgZ9lhZ1PJ O/UmN55TOjc2KeHwMoARlpP1+PF+6o7rHRkXoljfAxCOBNXpYx5t3IZz46zUM3pG8DuF GExC97I0wWEvqfV3A1vzgIZUAUrYlPpZAi9y12aoahe8F0r4WS6opEhiQVoBywwgh1vW kd74xEfqE4D83mFc0YL7uL6vFIp+rLW+oHi7rtFTbHgufUqXHYwzY4kr7WBnW0BHfBbx zhtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qhSlIbBbaKajKS/IeKMTbHGZ3Q2g5fhRWyCc6GgGFOw=; b=GzBna5PDOyHhb+dTx7nO+kG4/yHhatf9q7MtO0RcYNi232EDX047j57kaKT3DdWGzB SE1trmxuDMauG6s9gPgtuuLjhYehJC9zGJXYRQieEFYaYGdqWYz0wB8wrz0tOypH2t+F ifqUwR5cLQLh5qx70B5t2qnXjOlV5Am/+ddGcmnC/qDu24KqSG8IkyFx3aHSg7NqVOSI s0kmiyP+lZBRWJoaZuoQWUjwDAGngsreZ49s2LlyJgRO1t33rYT8/PLifoHlHV8HovWU ruVwR4zqHnvftUX4uF3fotPLYvbYwET9S81DfmnmM3wTuVlOtv/aBfvH2nbmYgCunKRY vPzQ== X-Gm-Message-State: AOAM531nVe6SI56Ha/FANSGxqg/zZdNuR3lSVncoyE7LGA4iORRVOR5L NsSQDSxLYjGUIKXPNoz8KSldjBfaHwE= X-Google-Smtp-Source: ABdhPJxbiaPuklpxkmh3T4+ZXi9ibcsoZFsVvSnpom8ez8y/HozIvGy1egzT8jlrpKnmg5m7ICoLVQ== X-Received: by 2002:a17:902:7883:: with SMTP id q3mr840986pll.117.1600059165913; Sun, 13 Sep 2020 21:52:45 -0700 (PDT) Received: from bobo.ozlabs.ibm.com ([203.185.249.227]) by smtp.gmail.com with ESMTPSA id a13sm6945312pgq.41.2020.09.13.21.52.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 13 Sep 2020 21:52:45 -0700 (PDT) From: Nicholas Piggin To: "linux-mm @ kvack . org" Cc: Nicholas Piggin , linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org, "Aneesh Kumar K . V" , Andrew Morton , Jens Axboe , Peter Zijlstra , "David S . Miller" Subject: [PATCH v2 2/4] powerpc: select ARCH_WANT_IRQS_OFF_ACTIVATE_MM Date: Mon, 14 Sep 2020 14:52:17 +1000 Message-Id: <20200914045219.3736466-3-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200914045219.3736466-1-npiggin@gmail.com> References: <20200914045219.3736466-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 463101048A00E X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: powerpc uses IPIs in some situations to switch a kernel thread away from a lazy tlb mm, which is subject to the TLB flushing race described in the changelog introducing ARCH_WANT_IRQS_OFF_ACTIVATE_MM. Signed-off-by: Nicholas Piggin --- arch/powerpc/Kconfig | 1 + arch/powerpc/include/asm/mmu_context.h | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 65bed1fdeaad..587ba8352d01 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -149,6 +149,7 @@ config PPC select ARCH_USE_QUEUED_RWLOCKS if PPC_QUEUED_SPINLOCKS select ARCH_USE_QUEUED_SPINLOCKS if PPC_QUEUED_SPINLOCKS select ARCH_WANT_IPC_PARSE_VERSION + select ARCH_WANT_IRQS_OFF_ACTIVATE_MM select ARCH_WEAK_RELEASE_ACQUIRE select BINFMT_ELF select BUILDTIME_TABLE_SORT diff --git a/arch/powerpc/include/asm/mmu_context.h b/arch/powerpc/include/asm/mmu_context.h index a3a12a8341b2..b42813359f49 100644 --- a/arch/powerpc/include/asm/mmu_context.h +++ b/arch/powerpc/include/asm/mmu_context.h @@ -244,7 +244,7 @@ static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next, #define activate_mm activate_mm static inline void activate_mm(struct mm_struct *prev, struct mm_struct *next) { - switch_mm(prev, next, current); + switch_mm_irqs_off(prev, next, current); } /* We don't currently use enter_lazy_tlb() for anything */ From patchwork Mon Sep 14 04:52:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11772825 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4586C6CA for ; Mon, 14 Sep 2020 04:52:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EF4C2208DB for ; Mon, 14 Sep 2020 04:52:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lm4Mggd2" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EF4C2208DB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0ABBC6B005A; Mon, 14 Sep 2020 00:52:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 054D96B005C; Mon, 14 Sep 2020 00:52:53 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EAE466B005D; Mon, 14 Sep 2020 00:52:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0177.hostedemail.com [216.40.44.177]) by kanga.kvack.org (Postfix) with ESMTP id D2E1C6B005A for ; Mon, 14 Sep 2020 00:52:52 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 90AAA82EBA15 for ; Mon, 14 Sep 2020 04:52:52 +0000 (UTC) X-FDA: 77260446984.04.rod96_070741a27105 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin04.hostedemail.com (Postfix) with ESMTP id 5E60C80846B9 for ; Mon, 14 Sep 2020 04:52:52 +0000 (UTC) X-Spam-Summary: 1,0,0,bd6616e7e27a103a,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:2:41:69:355:379:541:800:960:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1730:1747:1777:1792:2393:2559:2562:2693:2895:3138:3139:3140:3141:3142:3165:3865:3866:3867:3868:3870:3871:3872:3873:3874:4037:4049:4120:4250:4321:4362:4605:5007:6119:6261:6653:7514:7903:8660:9008:9010:9413:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12683:12895:13148:13161:13229:13230:13894:14096:14394:14687:21080:21433:21444:21451:21627:21666:21740:21795:21809:21939:21972:21990:30003:30012:30034:30045:30051:30054:30056:30070:30075,0,RBL:209.85.216.67:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100;04yrj9arwpeaaoo6j6ni76d8smkadycy93auxsoaxwm7ge8wa73pgdf38t5zebo.g7a6hyw818a37jhuzjnzq34tywjx1qinjyfpfqd7ntz9r7sbbbsg9cayw13199k.y-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_ rules:0: X-HE-Tag: rod96_070741a27105 X-Filterd-Recvd-Size: 9501 Received: from mail-pj1-f67.google.com (mail-pj1-f67.google.com [209.85.216.67]) by imf15.hostedemail.com (Postfix) with ESMTP for ; Mon, 14 Sep 2020 04:52:51 +0000 (UTC) Received: by mail-pj1-f67.google.com with SMTP id fa1so4813515pjb.0 for ; Sun, 13 Sep 2020 21:52:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BsgBvEdcPlC4FOpZ0B4ZRJc7oZi/0lTX9jrx1MeIFsE=; b=lm4Mggd2pzqrGBMVYnKwM7b/EYs+tQEgSODr69h0iLSy8WoVDZK4CDne8dWpI9DpM6 D7lHzUn1CY1LOkA3CpZ/wATmlG5/iwf0Wkf5TtwYG9G4Usond68GfMqQA4O898mUIaeR 3zIjx82y1PSgmAO26tZtmQ0n5flDErynkErvw5Lejp/HbFchEBel+ZXI5a5yAcs8DBAN YRPyFl1OkgYqPFNp6tr/WKTXbnVpTecPa0/OAopkNhdqBzqHeAKxUoLP2ODmdTkjHSwD ZlwklUEpqRI97FeYUv0aDOxgxM9ruMntwNpm9kf3/e7pKQBU+eM9UBVlO1C9BfETCUoJ JUpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BsgBvEdcPlC4FOpZ0B4ZRJc7oZi/0lTX9jrx1MeIFsE=; b=iD0PhyjNShJB0/8Tl5v0/F7KmWN9cMukpgG8boIAd0UUlYP8vROXB/JM54EJQ09b06 0egc6Ot/HFGIy9/VIg61LiGV0bzlwfaGadw2ICSd1ButboxoDauDXu7soG8uVkw4P4Cv VTiHPEWVfGGI+DEDsy+msQlxAuRbBEdo4bfZP+qUzZkqQq9Oj6VyDIbYpnIg9Y3qta08 xpjzSmvqyaShgFaJTmsscfXIkSEYXD62J+t0cxllQxRrYm+wdZND7GsqKMISrhXO75Gh fAjHvBs0lF3p5MJec3A5kukymjiT9dKnPezcYHRCIdX0YrLiySVsppKUPpdECgwTZPSy 2DUQ== X-Gm-Message-State: AOAM5333Dwgz6whMTuP2FSIZp+jG7sjywP6egMDXkyP3IZMWd5SoSkQj eDuv+YMBlMpus0AGXlzC/bHlggKtG3Q= X-Google-Smtp-Source: ABdhPJx3KPGLeRQoSE8dRCssQcbtGYQHCGhUf48Xq9kBcMqMZXfkcdYWdvgTWFqkpt5j1OQSPOfkNQ== X-Received: by 2002:a17:90b:3708:: with SMTP id mg8mr12159476pjb.39.1600059170766; Sun, 13 Sep 2020 21:52:50 -0700 (PDT) Received: from bobo.ozlabs.ibm.com ([203.185.249.227]) by smtp.gmail.com with ESMTPSA id a13sm6945312pgq.41.2020.09.13.21.52.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 13 Sep 2020 21:52:50 -0700 (PDT) From: Nicholas Piggin To: "linux-mm @ kvack . org" Cc: Nicholas Piggin , linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org, "Aneesh Kumar K . V" , Andrew Morton , Jens Axboe , Peter Zijlstra , "David S . Miller" Subject: [PATCH v2 3/4] sparc64: remove mm_cpumask clearing to fix kthread_use_mm race Date: Mon, 14 Sep 2020 14:52:18 +1000 Message-Id: <20200914045219.3736466-4-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200914045219.3736466-1-npiggin@gmail.com> References: <20200914045219.3736466-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5E60C80846B9 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The de facto (and apparently uncommented) standard for using an mm had, thanks to this code in sparc if nothing else, been that you must have a reference on mm_users *and that reference must have been obtained with mmget()*, i.e., from a thread with a reference to mm_users that had used the mm. The introduction of mmget_not_zero() in commit d2005e3f41d4 ("userfaultfd: don't pin the user memory in userfaultfd_file_create()") allowed mm_count holders to aoperate on user mappings asynchronously from the actual threads using the mm, but they were not to load those mappings into their TLB (i.e., walking vmas and page tables is okay, kthread_use_mm() is not). io_uring 2b188cc1bb857 ("Add io_uring IO interface") added code which does a kthread_use_mm() from a mmget_not_zero() refcount. The problem with this is code which previously assumed mm == current->mm and mm->mm_users == 1 implies the mm will remain single-threaded at least until this thread creates another mm_users reference, has now broken. arch/sparc/kernel/smp_64.c: if (atomic_read(&mm->mm_users) == 1) { cpumask_copy(mm_cpumask(mm), cpumask_of(cpu)); goto local_flush_and_out; } vs fs/io_uring.c if (unlikely(!(ctx->flags & IORING_SETUP_SQPOLL) || !mmget_not_zero(ctx->sqo_mm))) return -EFAULT; kthread_use_mm(ctx->sqo_mm); mmget_not_zero() could come in right after the mm_users == 1 test, then kthread_use_mm() which sets its CPU in the mm_cpumask. That update could be lost if cpumask_copy() occurs afterward. I propose we fix this by allowing mmget_not_zero() to be a first-class reference, and not have this obscure undocumented and unchecked restriction. The basic fix for sparc64 is to remove its mm_cpumask clearing code. The optimisation could be effectively restored by sending IPIs to mm_cpumask members and having them remove themselves from mm_cpumask. This is more tricky so I leave it as an exercise for someone with a sparc64 SMP. powerpc has a (currently similarly broken) example. Signed-off-by: Nicholas Piggin Acked-by: David S. Miller --- arch/sparc/kernel/smp_64.c | 65 ++++++++------------------------------ 1 file changed, 14 insertions(+), 51 deletions(-) diff --git a/arch/sparc/kernel/smp_64.c b/arch/sparc/kernel/smp_64.c index e286e2badc8a..e38d8bf454e8 100644 --- a/arch/sparc/kernel/smp_64.c +++ b/arch/sparc/kernel/smp_64.c @@ -1039,38 +1039,9 @@ void smp_fetch_global_pmu(void) * are flush_tlb_*() routines, and these run after flush_cache_*() * which performs the flushw. * - * The SMP TLB coherency scheme we use works as follows: - * - * 1) mm->cpu_vm_mask is a bit mask of which cpus an address - * space has (potentially) executed on, this is the heuristic - * we use to avoid doing cross calls. - * - * Also, for flushing from kswapd and also for clones, we - * use cpu_vm_mask as the list of cpus to make run the TLB. - * - * 2) TLB context numbers are shared globally across all processors - * in the system, this allows us to play several games to avoid - * cross calls. - * - * One invariant is that when a cpu switches to a process, and - * that processes tsk->active_mm->cpu_vm_mask does not have the - * current cpu's bit set, that tlb context is flushed locally. - * - * If the address space is non-shared (ie. mm->count == 1) we avoid - * cross calls when we want to flush the currently running process's - * tlb state. This is done by clearing all cpu bits except the current - * processor's in current->mm->cpu_vm_mask and performing the - * flush locally only. This will force any subsequent cpus which run - * this task to flush the context from the local tlb if the process - * migrates to another cpu (again). - * - * 3) For shared address spaces (threads) and swapping we bite the - * bullet for most cases and perform the cross call (but only to - * the cpus listed in cpu_vm_mask). - * - * The performance gain from "optimizing" away the cross call for threads is - * questionable (in theory the big win for threads is the massive sharing of - * address space state across processors). + * mm->cpu_vm_mask is a bit mask of which cpus an address + * space has (potentially) executed on, this is the heuristic + * we use to limit cross calls. */ /* This currently is only used by the hugetlb arch pre-fault @@ -1080,18 +1051,13 @@ void smp_fetch_global_pmu(void) void smp_flush_tlb_mm(struct mm_struct *mm) { u32 ctx = CTX_HWBITS(mm->context); - int cpu = get_cpu(); - if (atomic_read(&mm->mm_users) == 1) { - cpumask_copy(mm_cpumask(mm), cpumask_of(cpu)); - goto local_flush_and_out; - } + get_cpu(); smp_cross_call_masked(&xcall_flush_tlb_mm, ctx, 0, 0, mm_cpumask(mm)); -local_flush_and_out: __flush_tlb_mm(ctx, SECONDARY_CONTEXT); put_cpu(); @@ -1114,17 +1080,15 @@ void smp_flush_tlb_pending(struct mm_struct *mm, unsigned long nr, unsigned long { u32 ctx = CTX_HWBITS(mm->context); struct tlb_pending_info info; - int cpu = get_cpu(); + + get_cpu(); info.ctx = ctx; info.nr = nr; info.vaddrs = vaddrs; - if (mm == current->mm && atomic_read(&mm->mm_users) == 1) - cpumask_copy(mm_cpumask(mm), cpumask_of(cpu)); - else - smp_call_function_many(mm_cpumask(mm), tlb_pending_func, - &info, 1); + smp_call_function_many(mm_cpumask(mm), tlb_pending_func, + &info, 1); __flush_tlb_pending(ctx, nr, vaddrs); @@ -1134,14 +1098,13 @@ void smp_flush_tlb_pending(struct mm_struct *mm, unsigned long nr, unsigned long void smp_flush_tlb_page(struct mm_struct *mm, unsigned long vaddr) { unsigned long context = CTX_HWBITS(mm->context); - int cpu = get_cpu(); - if (mm == current->mm && atomic_read(&mm->mm_users) == 1) - cpumask_copy(mm_cpumask(mm), cpumask_of(cpu)); - else - smp_cross_call_masked(&xcall_flush_tlb_page, - context, vaddr, 0, - mm_cpumask(mm)); + get_cpu(); + + smp_cross_call_masked(&xcall_flush_tlb_page, + context, vaddr, 0, + mm_cpumask(mm)); + __flush_tlb_page(context, vaddr); put_cpu(); From patchwork Mon Sep 14 04:52:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11772827 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 625D659D for ; Mon, 14 Sep 2020 04:52:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 18283208DB for ; Mon, 14 Sep 2020 04:52:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="uNv5AhJ0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 18283208DB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2EE436B005C; Mon, 14 Sep 2020 00:52:58 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2C5546B005D; Mon, 14 Sep 2020 00:52:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1DBAA6B0062; Mon, 14 Sep 2020 00:52:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0039.hostedemail.com [216.40.44.39]) by kanga.kvack.org (Postfix) with ESMTP id 094BD6B005C for ; Mon, 14 Sep 2020 00:52:58 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id A9FF6181AEF1D for ; Mon, 14 Sep 2020 04:52:57 +0000 (UTC) X-FDA: 77260447194.16.bed32_500d87a27105 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin16.hostedemail.com (Postfix) with ESMTP id 7F6381048A013 for ; Mon, 14 Sep 2020 04:52:57 +0000 (UTC) X-Spam-Summary: 1,0,0,d3884a970209d701,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1544:1711:1730:1747:1777:1792:2198:2199:2393:2553:2559:2562:2693:2901:2904:3138:3139:3140:3141:3142:3165:3355:3865:3866:3867:3868:3870:3871:3872:3874:4118:4250:4321:4605:5007:6119:6261:6653:6742:7514:7901:7903:9413:10004:11026:11232:11473:11657:11658:11914:12043:12296:12297:12438:12517:12519:12555:12679:12895:12986:13255:13894:14096:14181:14394:14687:14721:21080:21433:21444:21451:21627:21666:21990:30012:30054:30070:30090,0,RBL:209.85.210.196:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100;04ygo1djebmgafxosmte3bdi9snxcocnk4t65kafk46md1sou8x5b9za3ku8dr3.ygjcaxsty1s6cecnxroqb63hu4ckj1hy6d5kswo7uxc6g9t9mg4kucd4qym6zsh.a-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: bed32_500d87a27105 X-Filterd-Recvd-Size: 7269 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Mon, 14 Sep 2020 04:52:56 +0000 (UTC) Received: by mail-pf1-f196.google.com with SMTP id k15so11533052pfc.12 for ; Sun, 13 Sep 2020 21:52:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=R15E1zKDEleMSpcLsh3CZt5cwnsxHHwg3les0S/1HZQ=; b=uNv5AhJ0SYv5S+1R/EdoTvDqPZ5eXsxLSyL8+onv+hZ/ZRcPZqRzqqm6XO3h5Nj6Py sKBbwnnGJu/HqHJUmlYkl+GiPJ8B8mrJS5k0iRTMEr0bvt4hkHbDzeeVFA3PHO/w5v9j DutoQlniOFqwgAaQkImqtu5I4FY5mSAlT6K0jQbybp3oSKsaoixC3ZjFM6tOjIQVYZqF +onq2SwiDTGpV8J1qnA382eEVQG+9lVu3Fz6arc+56QRPz4i/OKVOMr+OH7kxCGOJXtF 5is1TXx+TzvrmSlZ0XC8FTocgM6LhhrgTYxk66zt3maGFX2EV3KuWoosS3Ba8Je3Dykt UM7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=R15E1zKDEleMSpcLsh3CZt5cwnsxHHwg3les0S/1HZQ=; b=oCtl/5yG8za+QKdb5tS6kySvl4IA2gfQUi9+XbHA7zV1+Gwkw91CJbmBAjjFB62AGa FfLsZR2rBABQr9Pr5uKukeg/CbzTmFjnlfoS5oCCxUNQ8laGlgNXW0fQrwyQ3R4gBWhf O2YMNY7jvNaOQOMYYctcKxQncdCveRJ5F369HUfYwoO0V/0k8c90EYi6wAf4SPkWFhYY WiPrgcdYJdtMHAq2GOFXyWU70kdkfAnREHAWcquWsgMY9cw/+GS1vfEkp1s22/Qw82HG I1ZROupgMN0awgrL0KpA2rVnJpW8lE+DCbgUU5vCpBr5uDQZMHV7R7aWzfMZVspPDdVg 6THQ== X-Gm-Message-State: AOAM532OgaNCbM0U5KJ1wP4h2YozE7EqKg7yp/0RD+uv4Rs9Wg99GYGL vpVvO4bzoHkQ2+Hr6ULbvWvRqLip3iI= X-Google-Smtp-Source: ABdhPJyC3zHA2fO7ys2+MpyPfXJxUsoxMUIwuOQM00TR8Qj8lFaNM2scvwlKBeqI7feKRAikOHoVjg== X-Received: by 2002:a63:5561:: with SMTP id f33mr9503684pgm.13.1600059175888; Sun, 13 Sep 2020 21:52:55 -0700 (PDT) Received: from bobo.ozlabs.ibm.com ([203.185.249.227]) by smtp.gmail.com with ESMTPSA id a13sm6945312pgq.41.2020.09.13.21.52.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 13 Sep 2020 21:52:55 -0700 (PDT) From: Nicholas Piggin To: "linux-mm @ kvack . org" Cc: Nicholas Piggin , linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org, "Aneesh Kumar K . V" , Andrew Morton , Jens Axboe , Peter Zijlstra , "David S . Miller" , Michael Ellerman Subject: [PATCH v2 4/4] powerpc/64s/radix: Fix mm_cpumask trimming race vs kthread_use_mm Date: Mon, 14 Sep 2020 14:52:19 +1000 Message-Id: <20200914045219.3736466-5-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200914045219.3736466-1-npiggin@gmail.com> References: <20200914045219.3736466-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 7F6381048A013 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Commit 0cef77c7798a7 ("powerpc/64s/radix: flush remote CPUs out of single-threaded mm_cpumask") added a mechanism to trim the mm_cpumask of a process under certain conditions. One of the assumptions is that mm_users would not be incremented via a reference outside the process context with mmget_not_zero() then go on to kthread_use_mm() via that reference. That invariant was broken by io_uring code (see previous sparc64 fix), but I'll point Fixes: to the original powerpc commit because we are changing that assumption going forward, so this will make backports match up. Fix this by no longer relying on that assumption, but by having each CPU check the mm is not being used, and clearing their own bit from the mask only if it hasn't been switched-to by the time the IPI is processed. This relies on commit 38cf307c1f20 ("mm: fix kthread_use_mm() vs TLB invalidate") and ARCH_WANT_IRQS_OFF_ACTIVATE_MM to disable irqs over mm switch sequences. Reviewed-by: Michael Ellerman Depends-on: 38cf307c1f20 ("mm: fix kthread_use_mm() vs TLB invalidate") Fixes: 0cef77c7798a7 ("powerpc/64s/radix: flush remote CPUs out of single-threaded mm_cpumask") Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/tlb.h | 13 ------------- arch/powerpc/mm/book3s64/radix_tlb.c | 23 ++++++++++++++++------- 2 files changed, 16 insertions(+), 20 deletions(-) diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index fbc6f3002f23..d97f061fecac 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -66,19 +66,6 @@ static inline int mm_is_thread_local(struct mm_struct *mm) return false; return cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm)); } -static inline void mm_reset_thread_local(struct mm_struct *mm) -{ - WARN_ON(atomic_read(&mm->context.copros) > 0); - /* - * It's possible for mm_access to take a reference on mm_users to - * access the remote mm from another thread, but it's not allowed - * to set mm_cpumask, so mm_users may be > 1 here. - */ - WARN_ON(current->mm != mm); - atomic_set(&mm->context.active_cpus, 1); - cpumask_clear(mm_cpumask(mm)); - cpumask_set_cpu(smp_processor_id(), mm_cpumask(mm)); -} #else /* CONFIG_PPC_BOOK3S_64 */ static inline int mm_is_thread_local(struct mm_struct *mm) { diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 0d233763441f..143b4fd396f0 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -645,19 +645,29 @@ static void do_exit_flush_lazy_tlb(void *arg) struct mm_struct *mm = arg; unsigned long pid = mm->context.id; + /* + * A kthread could have done a mmget_not_zero() after the flushing CPU + * checked mm_is_singlethreaded, and be in the process of + * kthread_use_mm when interrupted here. In that case, current->mm will + * be set to mm, because kthread_use_mm() setting ->mm and switching to + * the mm is done with interrupts off. + */ if (current->mm == mm) - return; /* Local CPU */ + goto out_flush; if (current->active_mm == mm) { - /* - * Must be a kernel thread because sender is single-threaded. - */ - BUG_ON(current->mm); + WARN_ON_ONCE(current->mm != NULL); + /* Is a kernel thread and is using mm as the lazy tlb */ mmgrab(&init_mm); - switch_mm(mm, &init_mm, current); current->active_mm = &init_mm; + switch_mm_irqs_off(mm, &init_mm, current); mmdrop(mm); } + + atomic_dec(&mm->context.active_cpus); + cpumask_clear_cpu(smp_processor_id(), mm_cpumask(mm)); + +out_flush: _tlbiel_pid(pid, RIC_FLUSH_ALL); } @@ -672,7 +682,6 @@ static void exit_flush_lazy_tlbs(struct mm_struct *mm) */ smp_call_function_many(mm_cpumask(mm), do_exit_flush_lazy_tlb, (void *)mm, 1); - mm_reset_thread_local(mm); } void radix__flush_tlb_mm(struct mm_struct *mm)