From patchwork Wed Jan 18 08:00:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 13105810 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 001AAC004D4 for ; Wed, 18 Jan 2023 08:00:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D5DF6B0074; Wed, 18 Jan 2023 03:00:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 885CB6B007E; Wed, 18 Jan 2023 03:00:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 74E036B0080; Wed, 18 Jan 2023 03:00:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6839E6B007E for ; Wed, 18 Jan 2023 03:00:45 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E8947160541 for ; Wed, 18 Jan 2023 08:00:44 +0000 (UTC) X-FDA: 80367173208.26.D44AA38 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf02.hostedemail.com (Postfix) with ESMTP id 30E6380025 for ; Wed, 18 Jan 2023 08:00:43 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=WcdwzEYp; spf=pass (imf02.hostedemail.com: domain of npiggin@gmail.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674028843; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cadyF8CD/E3oK1yfSJSphX1/cZjS+kj7x9tW6+ar6sc=; b=MRM/B5HN2LacKeqt9gLLjVyUVumTXuZMAOHEfXWl/9rA0KBjDPqyGU81C7g5WUMlc9wCc5 QSclt4xBouUdfv/ve5RSMeD8/cmQlgaPBZIskLy3KDKViGnhRyTKZkaEaAaD/2ZiKrDict jZzsn448s33HfCNw1JexvT6y9ozIobM= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=WcdwzEYp; spf=pass (imf02.hostedemail.com: domain of npiggin@gmail.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674028843; a=rsa-sha256; cv=none; b=6V4atMrBxomoVbRhCjlClWxsrITI1MWtBONxvfV9XZzzZ7fKfQXvSJDY47EeWKqhk9HdAv +ECY41nvNWXJlRaYce+M5V/BJ+y+qXtfGAmqaqB01lUXjFvKujq1PGZBqZ7Kq++6j87ub6 PF6pPYyb5NjXltcH/PuhWa4+V5s38pY= Received: by mail-pl1-f172.google.com with SMTP id d9so36081416pll.9 for ; Wed, 18 Jan 2023 00:00:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cadyF8CD/E3oK1yfSJSphX1/cZjS+kj7x9tW6+ar6sc=; b=WcdwzEYpRynxGJWgQlKjA9zr5Ugl+tLrV/aeoKmaEyWm5I+eofmuFjC8v+O5CZJC8w tOZB+tAJR4+wiFQ1iaQWMCDXko5KxSau59i0CeKJSVmyvX/0i6O5Zr39SstJW0Z3rE1B Fyu5ijjz/jR31RZrsNZLyh0clWoOshZNgMIiETyrsUXvb3eq/iyNx1mkClYVLAY7+64k zYOIM7lzO4YVQSlFFkESwH80wBohrcEE2EqtboWnWa9tbyNk7GkZ4n+1ZHeUM7FQ71rU UAR97RL8R4zHQ490YhXWXkQbBJPh185TldI1Fz1/+yE5zazmCzPESNz0kF9nXmNoebc3 uPoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cadyF8CD/E3oK1yfSJSphX1/cZjS+kj7x9tW6+ar6sc=; b=NiEPr/P/sm9SGtZdsIC9x9sxzpV90rc935Vz+cirKAZT2dp4EiKu0fg18hGThFptSl j1/cTBsScyXd5lK3oFi+WnOt8dSY/CBIdsSc9ImrG7PXLmtaLXFnt3HLxsKpkHL8uMJ+ C+YSKvW3sj+kRqvrMDIscQo4FO2MEouF7HZ+nmOGI8sfypNYStXmukjLMu5YxlBwAELZ gvfYMO1IdS2bbSMxLuSz24JcK9eLISWXpNV1NtJjT+jVt3ukh9mdFavD3z76NpAoU6TI jdyRdikyVJfZoUAb+ro84loRMX93/KvUFBrIg6fAuKzUdx0L8oJUYUa6/CTivku8NDcg KiOw== X-Gm-Message-State: AFqh2kqKmrId+xciA+iCfvz8nM9tknR53hUJ6KRyoej7ksCT4v5DdNhg Gyj7OhjC9O0L/c5WFLfuGzs= X-Google-Smtp-Source: AMrXdXvru18rhSMREurCixBPKsRCor9uxoV61Ex5rHoJVQ7gmSRE7YHzcemWD4ARplQOcaVlqvp+VQ== X-Received: by 2002:a17:90a:3fca:b0:227:161a:6318 with SMTP id u10-20020a17090a3fca00b00227161a6318mr6162197pjm.47.1674028842208; Wed, 18 Jan 2023 00:00:42 -0800 (PST) Received: from bobo.ibm.com (193-116-102-45.tpgi.com.au. [193.116.102.45]) by smtp.gmail.com with ESMTPSA id y2-20020a17090a16c200b002272616d3e1sm738462pje.40.2023.01.18.00.00.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Jan 2023 00:00:41 -0800 (PST) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Andy Lutomirski , Linus Torvalds , linux-arch , linux-mm , linuxppc-dev@lists.ozlabs.org Subject: [PATCH v6 5/5] powerpc/64s/radix: combine final TLB flush and lazy tlb mm shootdown IPIs Date: Wed, 18 Jan 2023 18:00:11 +1000 Message-Id: <20230118080011.2258375-6-npiggin@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230118080011.2258375-1-npiggin@gmail.com> References: <20230118080011.2258375-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 30E6380025 X-Stat-Signature: wpd8kndcxk8z38b8gtsacuywgsb1dhyc X-Rspam-User: X-HE-Tag: 1674028842-496516 X-HE-Meta: U2FsdGVkX18pCBq/QkpQUQ8NbHograEEKsvRMUmwk6x/8T01dN+8/7QjVSDvXSwWdeVREfo/zfIWkIU9XXVDoSZmPWf5VNyT+uOzaLBaSq3fWTV9BujVoHuWom0Huz/a0Jios8BfgS6eIhAgzYZ0ktFdDohNSkASkF/7xT5DpMl+TJ/NfMxveXrIEgVyANMU4WlPoAcV6IT3Srz4tSeefEaM1VKPvfhVQv8xD/w9EcT9UAopdBtYTN0QOtsw7+U1bL3BZbawaqUtaBh96C3m+RDld8uwNPCRMCVHxryWzTLyke6ba+K89ly1OsgTF8Q0M39oUiMZ8hLZCDRVR/HkYjlSUngofiSJCxlPTBNPtnxIC1iKQ3WUFiTSUdp12HutgZELpRNUNu0KLIbUDedZWSglYAognAm2kQQc8yd+YRpFDEcUuQmON5axmAs3hzSp7iU4m9hDxBq3LjRj1LzcxQpbXFYMAt8gYNSr9qyBTyaih/PEsDNowoFqsq0bMy9Ka23lVHnXYT8seSM1cK9eub9L0KtJtr+fjKC2N2Pf3lyInFdjU+r0zrHjoebdBbIVcrU62Tr/1x2eNqszV5JCkYY+mK9gc7FW3hZH7QALYxyhG51StQAeFxKOAFKPuXuB2C3c8MAeSwT32slc9ZUthj4g1TOM1YgkS7V9XHKwv5FXA/Odek2/0f6nsKMh92rVlP+rooRyHwEaF5xOVb6V2uG9diiUImQFWZ2iq3+8T7VUfs2KkzAG7RXxHVfY2026eicYVE+spJisonJGlN2DoHvNa9J8dASz+Sh+OfXaoiTCcu6KD4BOB8wO9u4hqvYDWa6a/tfP9WtW6jJNswMa+Koer3JvXJvhrMTuWscHayzXkc5i8KwjYbJ8ZxlaSPTik59qYy//Mlt2BzOynEpzmiT3F+3YYIsPWtaKqMcSgRwXIN0VWIIPMJVdW8v9QlCxw/hYaTjI0k1aqq0ovNG O8/nKCgm oq+ekzEqVQsd0+3kP7n7TPXo7/IXsyDVDoOLUjvtDn/3xwdqmx33tsjpz5KBQ9db0zobqlUxD+3oI65iYirYEGLKueu0UTN/5GEaZya5xBezDUVvwURvc5l2z5uLbwh9y2gXE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: ** Not for merge ** CONFIG_MMU_LAZY_TLB_SHOOTDOWN that requires IPIs to clear the "lazy tlb" references to an mm that is being freed. With the radix MMU, the final userspace exit TLB flush can be performed with IPIs, and those IPIs can also clear lazy tlb mm references, which mostly eliminates the final IPIs required by MMU_LAZY_TLB_SHOOTDOWN. This does mean the final TLB flush is not done with TLBIE, which can be faster than IPI+TLBIEL, but we would have to do those IPIs for lazy shootdown so using TLBIEL should be a win. The final cpumask test and possible IPIs are still needed to clean up some rare race cases. We could prevent those entirely (e.g., prevent new lazy tlb mm references if userspace has gone away, or move the final TLB flush later), but I'd have to see actual numbers that matter before adding any more complexity for it. I can't imagine it would ever be worthwhile. This takes lazy tlb mm shootdown IPI interrupts from 314 to 3 on a 144 CPU system doing a kernel compile. It also takes care of the one potential problem workload which is a short-lived process with multiple CPU-bound threads that want to be spread to other CPUs, because the mm exit happens after the process is back to single-threaded. --- arch/powerpc/mm/book3s64/radix_tlb.c | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 282359ab525b..f34b78cb4c7d 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -1303,7 +1303,31 @@ void radix__tlb_flush(struct mmu_gather *tlb) * See the comment for radix in arch_exit_mmap(). */ if (tlb->fullmm || tlb->need_flush_all) { - __flush_all_mm(mm, true); + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_SHOOTDOWN)) { + /* + * Shootdown based lazy tlb mm refcounting means we + * have to IPI everyone in the mm_cpumask anyway soon + * when the mm goes away, so might as well do it as + * part of the final flush now. + * + * If lazy shootdown was improved to reduce IPIs (e.g., + * by batching), then it may end up being better to use + * tlbies here instead. + */ + smp_mb(); /* see radix__flush_tlb_mm */ + exit_flush_lazy_tlbs(mm); + _tlbiel_pid(mm->context.id, RIC_FLUSH_ALL); + + /* + * It should not be possible to have coprocessors still + * attached here. + */ + if (WARN_ON_ONCE(atomic_read(&mm->context.copros) > 0)) + __flush_all_mm(mm, true); + } else { + __flush_all_mm(mm, true); + } + } else if ( (psize = radix_get_mmu_psize(page_size)) == -1) { if (!tlb->freed_tables) radix__flush_tlb_mm(mm);