From patchwork Fri Aug 28 10:00:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11742537 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 48235138A for ; Fri, 28 Aug 2020 10:00:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 145D42078A for ; Fri, 28 Aug 2020 10:00:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AP/FuUUz" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 145D42078A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1251C6B000D; Fri, 28 Aug 2020 06:00:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0D4368E0006; Fri, 28 Aug 2020 06:00:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F2CEB6B0010; Fri, 28 Aug 2020 06:00:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0199.hostedemail.com [216.40.44.199]) by kanga.kvack.org (Postfix) with ESMTP id DFD0E6B000D for ; Fri, 28 Aug 2020 06:00:53 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 85748180AD815 for ; Fri, 28 Aug 2020 10:00:53 +0000 (UTC) X-FDA: 77199533586.15.mind94_4401f6827074 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin15.hostedemail.com (Postfix) with ESMTP id 52B5D1814B0C7 for ; Fri, 28 Aug 2020 10:00:53 +0000 (UTC) X-Spam-Summary: 1,0,0,3c892ad2b3a8fba5,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2198:2199:2393:2553:2559:2562:2693:2901:2904:3138:3139:3140:3141:3142:3165:3355:3865:3866:3867:3868:3870:3871:3872:3874:4118:4250:4321:4605:5007:6119:6261:6653:7514:7901:7903:9413:10004:11026:11232:11473:11657:11658:11914:12043:12296:12297:12438:12517:12519:12555:12679:12895:12986:13894:14096:14181:14394:14687:14721:21080:21433:21444:21451:21627:21666:21990:30012:30054:30070:30090,0,RBL:209.85.216.66:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.50.0.100;04yr3ra8bjoedat13cbibfwhhdx5jypyhwtjxw3a146md1sou8x5b9za3ku8dr3.ygjcax3qegpah6iptzgq9m9smm1z1gxdsj1rkxqsxm3rezzpw5tzefd4qym6zsh.a-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:26,LUA_SUMMARY:none X-HE-Tag: mind94_4401f6827074 X-Filterd-Recvd-Size: 7010 Received: from mail-pj1-f66.google.com (mail-pj1-f66.google.com [209.85.216.66]) by imf30.hostedemail.com (Postfix) with ESMTP for ; Fri, 28 Aug 2020 10:00:52 +0000 (UTC) Received: by mail-pj1-f66.google.com with SMTP id i13so295685pjv.0 for ; Fri, 28 Aug 2020 03:00:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1Y/faACy8cJZqe+8jbvX5+3MT7IMXQLldBAzW7OQGFU=; b=AP/FuUUz2c24R7h91evEMtU7s8yHhwXL6DM9u/2Mj6P86m31kOa2XEFxhYVFf9tZrE jv91cc3rMnttdOv5nUM25o4L9TDkox0OKTI4NdHKWJ2t8G5X6kuCjT11G/p+JGAnetfj aAdB92JLJZ+7ALLMOIx0G2O39sEKINQW7bkStFpXhtzHjTjJGrWOw521+Rp98GWmDpbD fJqSEgzfi/V8ctIwTqI7EUTk+Eu0WQf2y+CmgiuYKQkw+ftlvW12QmS43WzjL5wegPaU +VekSmy9gjIfJo4DEQR7zjgtk3AuxjdbAtl+UiSUVXtR4ga0IDGrVC4DCumTFiQTzqNp WBZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1Y/faACy8cJZqe+8jbvX5+3MT7IMXQLldBAzW7OQGFU=; b=TGpc3zzzw/CCWGjtQCe8L3w6tps5wFvfpK+A7oE4iVNMbC+WC48arWgvvdv6vu2awH sCoz5u3jiQjgEg8+CR2h5Ibzqa7l0+//DSCJADJ+ucG9xi1+cUyk897vtZRgpDRgzfaW vtm/tGZZaAqve0qYHksM7LJypJVXJVeOW7bNlU4UN4ZdaNK4EmxAgJuFOUIqF1JhZXRs bFD8hoE4jEPXTU+calFNMYkPFOs8Sx25O1Y4JJFTRm+s2QPgUisriHcSHM8K6SQDdI94 IdtSp7RKWt9RhSd4DvJVs3TcIfdzWuswn8E0LvW83AKkqBkAg/94QsnsTlYgKECqVyxd yxpA== X-Gm-Message-State: AOAM530JFhMZEX95znySxTLExT2vzkIhxZJJqyxv0Sopeup+ByJqqQqL +a4rms5GqNmp3GNwLWpX5R70mCqdrhs= X-Google-Smtp-Source: ABdhPJxv4t8TEETyZrq5vtkYIY4wLkQzivB6f4HHzIzllMBpn1clbhbAwA9g7HebVxjFpblwPrb97w== X-Received: by 2002:a17:90b:100e:: with SMTP id gm14mr556462pjb.39.1598608851815; Fri, 28 Aug 2020 03:00:51 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id 78sm1068608pfv.200.2020.08.28.03.00.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2020 03:00:51 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, "Aneesh Kumar K.V" , Andrew Morton , Jens Axboe , Peter Zijlstra , "David S. Miller" Subject: [PATCH 4/4] powerpc/64s/radix: Fix mm_cpumask trimming race vs kthread_use_mm Date: Fri, 28 Aug 2020 20:00:22 +1000 Message-Id: <20200828100022.1099682-5-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200828100022.1099682-1-npiggin@gmail.com> References: <20200828100022.1099682-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 52B5D1814B0C7 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Commit 0cef77c7798a7 ("powerpc/64s/radix: flush remote CPUs out of single-threaded mm_cpumask") added a mechanism to trim the mm_cpumask of a process under certain conditions. One of the assumptions is that mm_users would not be incremented via a reference outside the process context with mmget_not_zero() then go on to kthread_use_mm() via that reference. That invariant was broken by io_uring code (see previous sparc64 fix), but I'll point Fixes: to the original powerpc commit because we are changing that assumption going forward, so this will make backports match up. Fix this by no longer relying on that assumption, but by having each CPU check the mm is not being used, and clearing their own bit from the mask if it's okay. This fix relies on commit 38cf307c1f20 ("mm: fix kthread_use_mm() vs TLB invalidate") to disable irqs over the mm switch, and ARCH_WANT_IRQS_OFF_ACTIVATE_MM to be enabled. Fixes: 0cef77c7798a7 ("powerpc/64s/radix: flush remote CPUs out of single-threaded mm_cpumask") Signed-off-by: Nicholas Piggin Reviewed-by: Michael Ellerman --- arch/powerpc/include/asm/tlb.h | 13 ------------- arch/powerpc/mm/book3s64/radix_tlb.c | 23 ++++++++++++++++------- 2 files changed, 16 insertions(+), 20 deletions(-) diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index fbc6f3002f23..d97f061fecac 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -66,19 +66,6 @@ static inline int mm_is_thread_local(struct mm_struct *mm) return false; return cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm)); } -static inline void mm_reset_thread_local(struct mm_struct *mm) -{ - WARN_ON(atomic_read(&mm->context.copros) > 0); - /* - * It's possible for mm_access to take a reference on mm_users to - * access the remote mm from another thread, but it's not allowed - * to set mm_cpumask, so mm_users may be > 1 here. - */ - WARN_ON(current->mm != mm); - atomic_set(&mm->context.active_cpus, 1); - cpumask_clear(mm_cpumask(mm)); - cpumask_set_cpu(smp_processor_id(), mm_cpumask(mm)); -} #else /* CONFIG_PPC_BOOK3S_64 */ static inline int mm_is_thread_local(struct mm_struct *mm) { diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 0d233763441f..a421a0e3f930 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -645,19 +645,29 @@ static void do_exit_flush_lazy_tlb(void *arg) struct mm_struct *mm = arg; unsigned long pid = mm->context.id; + /* + * A kthread could have done a mmget_not_zero() after the flushing CPU + * checked mm_users == 1, and be in the process of kthread_use_mm when + * interrupted here. In that case, current->mm will be set to mm, + * because kthread_use_mm() setting ->mm and switching to the mm is + * done with interrupts off. + */ if (current->mm == mm) - return; /* Local CPU */ + goto out_flush; if (current->active_mm == mm) { - /* - * Must be a kernel thread because sender is single-threaded. - */ - BUG_ON(current->mm); + WARN_ON_ONCE(current->mm != NULL); + /* Is a kernel thread and is using mm as the lazy tlb */ mmgrab(&init_mm); - switch_mm(mm, &init_mm, current); current->active_mm = &init_mm; + switch_mm_irqs_off(mm, &init_mm, current); mmdrop(mm); } + + atomic_dec(&mm->context.active_cpus); + cpumask_clear_cpu(smp_processor_id(), mm_cpumask(mm)); + +out_flush: _tlbiel_pid(pid, RIC_FLUSH_ALL); } @@ -672,7 +682,6 @@ static void exit_flush_lazy_tlbs(struct mm_struct *mm) */ smp_call_function_many(mm_cpumask(mm), do_exit_flush_lazy_tlb, (void *)mm, 1); - mm_reset_thread_local(mm); } void radix__flush_tlb_mm(struct mm_struct *mm)