From patchwork Mon Sep 14 04:52:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11772827 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 625D659D for ; Mon, 14 Sep 2020 04:52:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 18283208DB for ; Mon, 14 Sep 2020 04:52:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="uNv5AhJ0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 18283208DB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2EE436B005C; Mon, 14 Sep 2020 00:52:58 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2C5546B005D; Mon, 14 Sep 2020 00:52:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1DBAA6B0062; Mon, 14 Sep 2020 00:52:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0039.hostedemail.com [216.40.44.39]) by kanga.kvack.org (Postfix) with ESMTP id 094BD6B005C for ; Mon, 14 Sep 2020 00:52:58 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id A9FF6181AEF1D for ; Mon, 14 Sep 2020 04:52:57 +0000 (UTC) X-FDA: 77260447194.16.bed32_500d87a27105 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin16.hostedemail.com (Postfix) with ESMTP id 7F6381048A013 for ; Mon, 14 Sep 2020 04:52:57 +0000 (UTC) X-Spam-Summary: 1,0,0,d3884a970209d701,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1544:1711:1730:1747:1777:1792:2198:2199:2393:2553:2559:2562:2693:2901:2904:3138:3139:3140:3141:3142:3165:3355:3865:3866:3867:3868:3870:3871:3872:3874:4118:4250:4321:4605:5007:6119:6261:6653:6742:7514:7901:7903:9413:10004:11026:11232:11473:11657:11658:11914:12043:12296:12297:12438:12517:12519:12555:12679:12895:12986:13255:13894:14096:14181:14394:14687:14721:21080:21433:21444:21451:21627:21666:21990:30012:30054:30070:30090,0,RBL:209.85.210.196:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100;04ygo1djebmgafxosmte3bdi9snxcocnk4t65kafk46md1sou8x5b9za3ku8dr3.ygjcaxsty1s6cecnxroqb63hu4ckj1hy6d5kswo7uxc6g9t9mg4kucd4qym6zsh.a-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: bed32_500d87a27105 X-Filterd-Recvd-Size: 7269 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Mon, 14 Sep 2020 04:52:56 +0000 (UTC) Received: by mail-pf1-f196.google.com with SMTP id k15so11533052pfc.12 for ; Sun, 13 Sep 2020 21:52:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=R15E1zKDEleMSpcLsh3CZt5cwnsxHHwg3les0S/1HZQ=; b=uNv5AhJ0SYv5S+1R/EdoTvDqPZ5eXsxLSyL8+onv+hZ/ZRcPZqRzqqm6XO3h5Nj6Py sKBbwnnGJu/HqHJUmlYkl+GiPJ8B8mrJS5k0iRTMEr0bvt4hkHbDzeeVFA3PHO/w5v9j DutoQlniOFqwgAaQkImqtu5I4FY5mSAlT6K0jQbybp3oSKsaoixC3ZjFM6tOjIQVYZqF +onq2SwiDTGpV8J1qnA382eEVQG+9lVu3Fz6arc+56QRPz4i/OKVOMr+OH7kxCGOJXtF 5is1TXx+TzvrmSlZ0XC8FTocgM6LhhrgTYxk66zt3maGFX2EV3KuWoosS3Ba8Je3Dykt UM7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=R15E1zKDEleMSpcLsh3CZt5cwnsxHHwg3les0S/1HZQ=; b=oCtl/5yG8za+QKdb5tS6kySvl4IA2gfQUi9+XbHA7zV1+Gwkw91CJbmBAjjFB62AGa FfLsZR2rBABQr9Pr5uKukeg/CbzTmFjnlfoS5oCCxUNQ8laGlgNXW0fQrwyQ3R4gBWhf O2YMNY7jvNaOQOMYYctcKxQncdCveRJ5F369HUfYwoO0V/0k8c90EYi6wAf4SPkWFhYY WiPrgcdYJdtMHAq2GOFXyWU70kdkfAnREHAWcquWsgMY9cw/+GS1vfEkp1s22/Qw82HG I1ZROupgMN0awgrL0KpA2rVnJpW8lE+DCbgUU5vCpBr5uDQZMHV7R7aWzfMZVspPDdVg 6THQ== X-Gm-Message-State: AOAM532OgaNCbM0U5KJ1wP4h2YozE7EqKg7yp/0RD+uv4Rs9Wg99GYGL vpVvO4bzoHkQ2+Hr6ULbvWvRqLip3iI= X-Google-Smtp-Source: ABdhPJyC3zHA2fO7ys2+MpyPfXJxUsoxMUIwuOQM00TR8Qj8lFaNM2scvwlKBeqI7feKRAikOHoVjg== X-Received: by 2002:a63:5561:: with SMTP id f33mr9503684pgm.13.1600059175888; Sun, 13 Sep 2020 21:52:55 -0700 (PDT) Received: from bobo.ozlabs.ibm.com ([203.185.249.227]) by smtp.gmail.com with ESMTPSA id a13sm6945312pgq.41.2020.09.13.21.52.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 13 Sep 2020 21:52:55 -0700 (PDT) From: Nicholas Piggin To: "linux-mm @ kvack . org" Cc: Nicholas Piggin , linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org, "Aneesh Kumar K . V" , Andrew Morton , Jens Axboe , Peter Zijlstra , "David S . Miller" , Michael Ellerman Subject: [PATCH v2 4/4] powerpc/64s/radix: Fix mm_cpumask trimming race vs kthread_use_mm Date: Mon, 14 Sep 2020 14:52:19 +1000 Message-Id: <20200914045219.3736466-5-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200914045219.3736466-1-npiggin@gmail.com> References: <20200914045219.3736466-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 7F6381048A013 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Commit 0cef77c7798a7 ("powerpc/64s/radix: flush remote CPUs out of single-threaded mm_cpumask") added a mechanism to trim the mm_cpumask of a process under certain conditions. One of the assumptions is that mm_users would not be incremented via a reference outside the process context with mmget_not_zero() then go on to kthread_use_mm() via that reference. That invariant was broken by io_uring code (see previous sparc64 fix), but I'll point Fixes: to the original powerpc commit because we are changing that assumption going forward, so this will make backports match up. Fix this by no longer relying on that assumption, but by having each CPU check the mm is not being used, and clearing their own bit from the mask only if it hasn't been switched-to by the time the IPI is processed. This relies on commit 38cf307c1f20 ("mm: fix kthread_use_mm() vs TLB invalidate") and ARCH_WANT_IRQS_OFF_ACTIVATE_MM to disable irqs over mm switch sequences. Reviewed-by: Michael Ellerman Depends-on: 38cf307c1f20 ("mm: fix kthread_use_mm() vs TLB invalidate") Fixes: 0cef77c7798a7 ("powerpc/64s/radix: flush remote CPUs out of single-threaded mm_cpumask") Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/tlb.h | 13 ------------- arch/powerpc/mm/book3s64/radix_tlb.c | 23 ++++++++++++++++------- 2 files changed, 16 insertions(+), 20 deletions(-) diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index fbc6f3002f23..d97f061fecac 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -66,19 +66,6 @@ static inline int mm_is_thread_local(struct mm_struct *mm) return false; return cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm)); } -static inline void mm_reset_thread_local(struct mm_struct *mm) -{ - WARN_ON(atomic_read(&mm->context.copros) > 0); - /* - * It's possible for mm_access to take a reference on mm_users to - * access the remote mm from another thread, but it's not allowed - * to set mm_cpumask, so mm_users may be > 1 here. - */ - WARN_ON(current->mm != mm); - atomic_set(&mm->context.active_cpus, 1); - cpumask_clear(mm_cpumask(mm)); - cpumask_set_cpu(smp_processor_id(), mm_cpumask(mm)); -} #else /* CONFIG_PPC_BOOK3S_64 */ static inline int mm_is_thread_local(struct mm_struct *mm) { diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 0d233763441f..143b4fd396f0 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -645,19 +645,29 @@ static void do_exit_flush_lazy_tlb(void *arg) struct mm_struct *mm = arg; unsigned long pid = mm->context.id; + /* + * A kthread could have done a mmget_not_zero() after the flushing CPU + * checked mm_is_singlethreaded, and be in the process of + * kthread_use_mm when interrupted here. In that case, current->mm will + * be set to mm, because kthread_use_mm() setting ->mm and switching to + * the mm is done with interrupts off. + */ if (current->mm == mm) - return; /* Local CPU */ + goto out_flush; if (current->active_mm == mm) { - /* - * Must be a kernel thread because sender is single-threaded. - */ - BUG_ON(current->mm); + WARN_ON_ONCE(current->mm != NULL); + /* Is a kernel thread and is using mm as the lazy tlb */ mmgrab(&init_mm); - switch_mm(mm, &init_mm, current); current->active_mm = &init_mm; + switch_mm_irqs_off(mm, &init_mm, current); mmdrop(mm); } + + atomic_dec(&mm->context.active_cpus); + cpumask_clear_cpu(smp_processor_id(), mm_cpumask(mm)); + +out_flush: _tlbiel_pid(pid, RIC_FLUSH_ALL); } @@ -672,7 +682,6 @@ static void exit_flush_lazy_tlbs(struct mm_struct *mm) */ smp_call_function_many(mm_cpumask(mm), do_exit_flush_lazy_tlb, (void *)mm, 1); - mm_reset_thread_local(mm); } void radix__flush_tlb_mm(struct mm_struct *mm)