From patchwork Tue Aug 18 03:34:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11719759 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3DC5B722 for ; Tue, 18 Aug 2020 03:34:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F188F2076D for ; Tue, 18 Aug 2020 03:34:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JTsxTQCW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F188F2076D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0D73A6B0002; Mon, 17 Aug 2020 23:34:17 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 05F5E6B0005; Mon, 17 Aug 2020 23:34:16 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E42256B0006; Mon, 17 Aug 2020 23:34:16 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CAB846B0002 for ; Mon, 17 Aug 2020 23:34:16 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 88D2E2470 for ; Tue, 18 Aug 2020 03:34:16 +0000 (UTC) X-FDA: 77162271312.05.beast85_2008ca82701c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id 61EB518017807 for ; Tue, 18 Aug 2020 03:34:16 +0000 (UTC) X-Spam-Summary: 1,0,0,1bbd12d9004457ab,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:2:41:69:355:379:541:800:960:968:973:988:989:1260:1311:1314:1345:1437:1515:1535:1605:1606:1730:1747:1777:1792:2393:2559:2562:2693:2895:3138:3139:3140:3141:3142:3165:3865:3866:3867:3868:3870:3871:3872:3873:3874:4037:4119:4250:4321:4362:4605:5007:6119:6120:6261:6653:7514:7903:8660:9008:9010:9413:9592:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12683:12895:13148:13161:13229:13230:13894:14096:14687:21080:21433:21444:21451:21627:21666:21740:21795:21809:21939:21972:21990:30003:30012:30034:30045:30051:30054:30056:30070:30075,0,RBL:209.85.214.195:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.50.0.100;04yrzt7n1yp4jez156e9je18bdryuycy93auxsoacrs6okwrqydwidkumyhdnfb.mxg3qdr818a37jhuzjnzq34tywjx1qinjyft3pa5we54e7sbbbsg9c75ztqqoi7.h-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_ rules:0: X-HE-Tag: beast85_2008ca82701c X-Filterd-Recvd-Size: 8928 Received: from mail-pl1-f195.google.com (mail-pl1-f195.google.com [209.85.214.195]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Tue, 18 Aug 2020 03:34:15 +0000 (UTC) Received: by mail-pl1-f195.google.com with SMTP id f5so8554447plr.9 for ; Mon, 17 Aug 2020 20:34:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=kXKR3cXpwpt0q9jgFz26UJPnEar0RH5gNFSzQ9D4WUQ=; b=JTsxTQCWOkw6T9c3YtWdB1NVIAooJ/0TYLt2HeP3tRrpo9IrPH5fcSJC/W1TF6hUrK 36Q88tCA5xip+UxdciaZbVfDYgfeyDGUIlbtktjwoxbZTeYrUnbZdG9gFRAnIwKqJwdP CmGqb7y9E+ke6lvgFU5x9I3iM9wDvEHVONeLH5m/QC2OyjTuFzQr+1ki+1vWIx4c71RY cYA4y4P1JW8KO+6gnUICfuT2g5W25/HCbpSLRvLO85Mdpzi+D1wSk/sEAtEssZ3j+vfB n34t1a9YyqYAUV0FR8/FV1KruetVFbpMilo3/y6737GJKAD+w5XJS50Ge4nPNk/NYv2B 7DIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=kXKR3cXpwpt0q9jgFz26UJPnEar0RH5gNFSzQ9D4WUQ=; b=psBSj1OpbLH2G5ZzpngWgTuEJ5YEJaL0HmjAlIA7Ps5AmGFs1RBYtQa04A1dle89xd yAwvTNm1LL3444m1ZkJXbow7K5MfTlmmWzfkCvU7v/ccbqhkzYTym5vj1iDVc5ZL0IZ2 tRabmYWdrcDLa4Un/btDKbFoKB/d8SCtepPrV77NgXd2OKUxluulq6dtilNo3bZpZErB Kv59/cFbb9DqDcQ079/w1sdTIONjqrurEuEyuGcCyzi/2Wys/fusDX3NcWrXmOBfdQBl jWSi8r0VeiRml0X3rRiNKvE45hHZnT1OM9mX7KG86EgRSf00uqhJckIHI5LRTIM+ABHM rBBQ== X-Gm-Message-State: AOAM530x/BZ3/HXvmyd+Kl1SGrRxyfWgFtUIilvxWLGWeQ+sT0z8wzBv Cx5DOl1k9tzf2LDvur3r5FI= X-Google-Smtp-Source: ABdhPJyW1IrlExwvUFobL+ijBXLtHQla2kejcW1bkPDfbMej7nAW5/8WB8glAdU0v5nRmt+I6vLSLA== X-Received: by 2002:a17:90b:349:: with SMTP id fh9mr14332259pjb.73.1597721654856; Mon, 17 Aug 2020 20:34:14 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (193-116-193-175.tpgi.com.au. [193.116.193.175]) by smtp.gmail.com with ESMTPSA id b18sm19823352pgj.12.2020.08.17.20.34.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Aug 2020 20:34:14 -0700 (PDT) From: Nicholas Piggin To: "David S. Miller" Cc: Nicholas Piggin , Jens Axboe , "Aneesh Kumar K.V" , linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, sparclinux@vger.kernel.org Subject: [RFC PATCH 1/2] sparc64: remove mm_cpumask clearing to fix kthread_use_mm race Date: Tue, 18 Aug 2020 13:34:04 +1000 Message-Id: <20200818033405.170457-1-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 MIME-Version: 1.0 X-Rspamd-Queue-Id: 61EB518017807 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The de facto (and apparently uncommented) standard for using an mm had, thanks to this code in sparc if nothing else, been that you must have a reference on mm_users *and that reference must have been obtained with mmget()*, i.e., from a thread with a reference to mm_users that had used the mm. The introduction of mmget_not_zero() in commit d2005e3f41d4 ("userfaultfd: don't pin the user memory in userfaultfd_file_create()") allowed mm_count holders to aoperate on user mappings asynchronously from the actual threads using the mm, but they were not to load those mappings into their TLB (i.e., walking vmas and page tables is okay, kthread_use_mm() is not). io_uring 2b188cc1bb857 ("Add io_uring IO interface") added code which does a kthread_use_mm() from a mmget_not_zero() refcount. The problem with this is code which previously assumed mm == current->mm and mm->mm_users == 1 implies the mm will remain single-threaded at least until this thread creates another mm_users reference, has now broken. arch/sparc/kernel/smp_64.c: if (atomic_read(&mm->mm_users) == 1) { cpumask_copy(mm_cpumask(mm), cpumask_of(cpu)); goto local_flush_and_out; } vs fs/io_uring.c if (unlikely(!(ctx->flags & IORING_SETUP_SQPOLL) || !mmget_not_zero(ctx->sqo_mm))) return -EFAULT; kthread_use_mm(ctx->sqo_mm); mmget_not_zero() could come in right after the mm_users == 1 test, then kthread_use_mm() which sets its CPU in the mm_cpumask. That update could be lost if cpumask_copy() occurs afterward. I propose we fix this by allowing mmget_not_zero() to be a first-class reference, and not have this obscure undocumented and unchecked restriction. The basic fix for sparc64 is to remove its mm_cpumask clearing code. The optimisation could be effectively restored by sending IPIs to mm_cpumask members and having them remove themselves from mm_cpumask. This is more tricky so I leave it as an exercise for someone with a sparc64 SMP. powerpc has a (currently similarly broken) example. not-yet-Signed-off-by: Nicholas Piggin --- arch/sparc/kernel/smp_64.c | 58 ++++++-------------------------------- 1 file changed, 9 insertions(+), 49 deletions(-) diff --git a/arch/sparc/kernel/smp_64.c b/arch/sparc/kernel/smp_64.c index e286e2badc8a..41423c6077d0 100644 --- a/arch/sparc/kernel/smp_64.c +++ b/arch/sparc/kernel/smp_64.c @@ -1039,38 +1039,9 @@ void smp_fetch_global_pmu(void) * are flush_tlb_*() routines, and these run after flush_cache_*() * which performs the flushw. * - * The SMP TLB coherency scheme we use works as follows: - * - * 1) mm->cpu_vm_mask is a bit mask of which cpus an address - * space has (potentially) executed on, this is the heuristic - * we use to avoid doing cross calls. - * - * Also, for flushing from kswapd and also for clones, we - * use cpu_vm_mask as the list of cpus to make run the TLB. - * - * 2) TLB context numbers are shared globally across all processors - * in the system, this allows us to play several games to avoid - * cross calls. - * - * One invariant is that when a cpu switches to a process, and - * that processes tsk->active_mm->cpu_vm_mask does not have the - * current cpu's bit set, that tlb context is flushed locally. - * - * If the address space is non-shared (ie. mm->count == 1) we avoid - * cross calls when we want to flush the currently running process's - * tlb state. This is done by clearing all cpu bits except the current - * processor's in current->mm->cpu_vm_mask and performing the - * flush locally only. This will force any subsequent cpus which run - * this task to flush the context from the local tlb if the process - * migrates to another cpu (again). - * - * 3) For shared address spaces (threads) and swapping we bite the - * bullet for most cases and perform the cross call (but only to - * the cpus listed in cpu_vm_mask). - * - * The performance gain from "optimizing" away the cross call for threads is - * questionable (in theory the big win for threads is the massive sharing of - * address space state across processors). + * mm->cpu_vm_mask is a bit mask of which cpus an address + * space has (potentially) executed on, this is the heuristic + * we use to limit cross calls. */ /* This currently is only used by the hugetlb arch pre-fault @@ -1082,16 +1053,10 @@ void smp_flush_tlb_mm(struct mm_struct *mm) u32 ctx = CTX_HWBITS(mm->context); int cpu = get_cpu(); - if (atomic_read(&mm->mm_users) == 1) { - cpumask_copy(mm_cpumask(mm), cpumask_of(cpu)); - goto local_flush_and_out; - } - smp_cross_call_masked(&xcall_flush_tlb_mm, ctx, 0, 0, mm_cpumask(mm)); -local_flush_and_out: __flush_tlb_mm(ctx, SECONDARY_CONTEXT); put_cpu(); @@ -1120,11 +1085,8 @@ void smp_flush_tlb_pending(struct mm_struct *mm, unsigned long nr, unsigned long info.nr = nr; info.vaddrs = vaddrs; - if (mm == current->mm && atomic_read(&mm->mm_users) == 1) - cpumask_copy(mm_cpumask(mm), cpumask_of(cpu)); - else - smp_call_function_many(mm_cpumask(mm), tlb_pending_func, - &info, 1); + smp_call_function_many(mm_cpumask(mm), tlb_pending_func, + &info, 1); __flush_tlb_pending(ctx, nr, vaddrs); @@ -1136,12 +1098,10 @@ void smp_flush_tlb_page(struct mm_struct *mm, unsigned long vaddr) unsigned long context = CTX_HWBITS(mm->context); int cpu = get_cpu(); - if (mm == current->mm && atomic_read(&mm->mm_users) == 1) - cpumask_copy(mm_cpumask(mm), cpumask_of(cpu)); - else - smp_cross_call_masked(&xcall_flush_tlb_page, - context, vaddr, 0, - mm_cpumask(mm)); + smp_cross_call_masked(&xcall_flush_tlb_page, + context, vaddr, 0, + mm_cpumask(mm)); + __flush_tlb_page(context, vaddr); put_cpu(); From patchwork Tue Aug 18 03:34:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11719761 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B0E85722 for ; Tue, 18 Aug 2020 03:34:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 78F2F207BB for ; Tue, 18 Aug 2020 03:34:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="uRdD8KLt" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 78F2F207BB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 803766B0005; Mon, 17 Aug 2020 23:34:20 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 769356B0006; Mon, 17 Aug 2020 23:34:20 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 606016B0007; Mon, 17 Aug 2020 23:34:20 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0104.hostedemail.com [216.40.44.104]) by kanga.kvack.org (Postfix) with ESMTP id 429C06B0005 for ; Mon, 17 Aug 2020 23:34:20 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 00F22180AD81F for ; Tue, 18 Aug 2020 03:34:19 +0000 (UTC) X-FDA: 77162271480.27.teeth94_07127752701c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin27.hostedemail.com (Postfix) with ESMTP id C98293D669 for ; Tue, 18 Aug 2020 03:34:19 +0000 (UTC) X-Spam-Summary: 1,0,0,23e8bd5df5ee9a40,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2198:2199:2393:2553:2559:2562:2693:2901:2904:3138:3139:3140:3141:3142:3165:3355:3865:3866:3867:3868:3870:3871:3872:3874:4117:4250:4321:4605:5007:6119:6120:6261:6653:7514:7901:7903:9413:10004:11026:11232:11473:11657:11658:11914:12043:12296:12297:12438:12517:12519:12555:12679:12895:12986:13255:13894:14096:14181:14687:14721:21080:21433:21444:21451:21627:21666:21990:30012:30054:30070:30090,0,RBL:209.85.214.195:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.50.0.100;04yggiz3n6qxs8ucgfaywejzhwrs6ypyhwtjxw3a146md1sou8x5b9za3ku8dr3.ygjcaxsty1s6cecd7imrs68xpm1z1gxdsj1rkxqsxm3rezzpw5tzefd4qym6zsh.a-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: teeth94_07127752701c X-Filterd-Recvd-Size: 6880 Received: from mail-pl1-f195.google.com (mail-pl1-f195.google.com [209.85.214.195]) by imf32.hostedemail.com (Postfix) with ESMTP for ; Tue, 18 Aug 2020 03:34:19 +0000 (UTC) Received: by mail-pl1-f195.google.com with SMTP id r4so8570184pls.2 for ; Mon, 17 Aug 2020 20:34:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=t5NIMSla2q4H4CYrgEoypiSInrv82zNsRbmwKfnLv98=; b=uRdD8KLtueArRi5YgefMeJDNmk3262hrEcmU4d2F6wq++v7Ma9rUefUixF2COxCtLc td5Rwz0puO391VwBccL0iLXviBugIL3u84rPg8rn+K+h4YofYGJhIsC8G6APa8gVgzX5 2APhZ1DyLJRboh/HWU4dTWnJYXK0nLTNC8tH9fSFizTp9Ip7QLOgDFv0zUDT46SeFoAc yQStkqIILuLhbKNuf8KDIetfzrKboLdpW02QzOvqhM0RvSPIi0r7Wmlh/Nq4Q+AJ0A6q Tfg69C3isdqL8HsOzA8eHcZZF6FPHrfmwUviJRUc1g0N2OtBxFLf7tCrbRwIxKg42dX/ ZzJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=t5NIMSla2q4H4CYrgEoypiSInrv82zNsRbmwKfnLv98=; b=CszcP/ryFB0WRU1KvpVECzf8vUQ9HMqQ2AVPHmr2JTHSNwchtizvJ1OZ1tpen1/I0v Ax6qlXds+ZA/oFT1TGbddZI8jw+iPKhecrf16HuamJgvG/9LxjLC4yQq3QxwCQ8HqqJM rM+rI56Yt0G4nsQF14EOrUUF3v20+gy+Achm4/eP84HKSygATx0Et0ZLia7Mmp4Jvl3U 15oaHAphcVe2spjg/MjkhXRs6A3dmTp896M9n5VEHFCv6NlEnzr2ft6Mqqw0Q5L0g5Cm 6/bZ4kuo2CNxXuvYRRqEVZz1xIdER9mF33PXyGy8HrxiMFmf51cSRUxjDaeNm3zasHab k5kg== X-Gm-Message-State: AOAM531zQxMi996UL9YAX5LBk2svBFzy0S+jinS8kTC7hgUdry7iKWN4 m8K9JytTqGnHHGDGrQYzp1I= X-Google-Smtp-Source: ABdhPJy6d0Cu933mn+84pD243uhmVtbLS27aP0ei6qwpVknrf2wE9OmynUNrAFXe9REc+s5VjsOUuA== X-Received: by 2002:a17:90a:4214:: with SMTP id o20mr4177105pjg.232.1597721658610; Mon, 17 Aug 2020 20:34:18 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (193-116-193-175.tpgi.com.au. [193.116.193.175]) by smtp.gmail.com with ESMTPSA id b18sm19823352pgj.12.2020.08.17.20.34.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Aug 2020 20:34:18 -0700 (PDT) From: Nicholas Piggin To: "David S. Miller" Cc: Nicholas Piggin , Jens Axboe , "Aneesh Kumar K.V" , linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, sparclinux@vger.kernel.org Subject: [RFC PATCH 2/2] powerpc/64s/radix: Fix mm_cpumask trimming race vs kthread_use_mm Date: Tue, 18 Aug 2020 13:34:05 +1000 Message-Id: <20200818033405.170457-2-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200818033405.170457-1-npiggin@gmail.com> References: <20200818033405.170457-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: C98293D669 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Commit 0cef77c7798a7 ("powerpc/64s/radix: flush remote CPUs out of single-threaded mm_cpumask") added a mechanism to trim the mm_cpumask of a process under certain conditions. One of the assumptions is that mm_users would not be incremented via a reference outside the process context with mmget_not_zero() then go on to kthread_use_mm() via that reference. That invariant was broken by io_uring code (see previous sparc64 fix), but I'll point Fixes: to the original powerpc commit because we are changing that assumption going forward, so this will make backports match up. Fix this by no longer relying on that assumption, but by having each CPU check the mm is not being used, and clearing their own bit from the mask if it's okay. This relies on commit 38cf307c1f20 ("mm: fix kthread_use_mm() vs TLB invalidate") to disable irqs over the mm switch. Fixes: 0cef77c7798a7 ("powerpc/64s/radix: flush remote CPUs out of single-threaded mm_cpumask") not-yet-Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/tlb.h | 13 ------------- arch/powerpc/mm/book3s64/radix_tlb.c | 23 ++++++++++++++++------- 2 files changed, 16 insertions(+), 20 deletions(-) diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index fbc6f3002f23..d97f061fecac 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -66,19 +66,6 @@ static inline int mm_is_thread_local(struct mm_struct *mm) return false; return cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm)); } -static inline void mm_reset_thread_local(struct mm_struct *mm) -{ - WARN_ON(atomic_read(&mm->context.copros) > 0); - /* - * It's possible for mm_access to take a reference on mm_users to - * access the remote mm from another thread, but it's not allowed - * to set mm_cpumask, so mm_users may be > 1 here. - */ - WARN_ON(current->mm != mm); - atomic_set(&mm->context.active_cpus, 1); - cpumask_clear(mm_cpumask(mm)); - cpumask_set_cpu(smp_processor_id(), mm_cpumask(mm)); -} #else /* CONFIG_PPC_BOOK3S_64 */ static inline int mm_is_thread_local(struct mm_struct *mm) { diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 0d233763441f..a421a0e3f930 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -645,19 +645,29 @@ static void do_exit_flush_lazy_tlb(void *arg) struct mm_struct *mm = arg; unsigned long pid = mm->context.id; + /* + * A kthread could have done a mmget_not_zero() after the flushing CPU + * checked mm_users == 1, and be in the process of kthread_use_mm when + * interrupted here. In that case, current->mm will be set to mm, + * because kthread_use_mm() setting ->mm and switching to the mm is + * done with interrupts off. + */ if (current->mm == mm) - return; /* Local CPU */ + goto out_flush; if (current->active_mm == mm) { - /* - * Must be a kernel thread because sender is single-threaded. - */ - BUG_ON(current->mm); + WARN_ON_ONCE(current->mm != NULL); + /* Is a kernel thread and is using mm as the lazy tlb */ mmgrab(&init_mm); - switch_mm(mm, &init_mm, current); current->active_mm = &init_mm; + switch_mm_irqs_off(mm, &init_mm, current); mmdrop(mm); } + + atomic_dec(&mm->context.active_cpus); + cpumask_clear_cpu(smp_processor_id(), mm_cpumask(mm)); + +out_flush: _tlbiel_pid(pid, RIC_FLUSH_ALL); } @@ -672,7 +682,6 @@ static void exit_flush_lazy_tlbs(struct mm_struct *mm) */ smp_call_function_many(mm_cpumask(mm), do_exit_flush_lazy_tlb, (void *)mm, 1); - mm_reset_thread_local(mm); } void radix__flush_tlb_mm(struct mm_struct *mm)