From patchwork Sun Feb 23 19:48:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rik van Riel X-Patchwork-Id: 13987232 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5D3EC021B2 for ; Sun, 23 Feb 2025 19:51:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D85AF6B007B; Sun, 23 Feb 2025 14:51:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AB55F6B009B; Sun, 23 Feb 2025 14:51:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 840E26B0096; Sun, 23 Feb 2025 14:51:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A13E96B0098 for ; Sun, 23 Feb 2025 14:51:16 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 65D271202A2 for ; Sun, 23 Feb 2025 19:51:16 +0000 (UTC) X-FDA: 83152253352.12.EB7C592 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) by imf06.hostedemail.com (Postfix) with ESMTP id C6B27180007 for ; Sun, 23 Feb 2025 19:51:14 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf06.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740340274; a=rsa-sha256; cv=none; b=7uCipucRdu7Gkm1/EnHGSyHRiXJsjk2Y+3yYRgs6bBZtcH/SDt9DiRuWEOot3vmSYf+cGY 7LKpHAyuBCuxThiTpPe5WMpaxxBe5tkREeAX4IdEKh70LjPbBrYU98DQrvMmjOc3/YYWYK mmMaKkd1Up8D7Kro4vNUet3Q66sjxes= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf06.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740340274; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HJW3QmIpSlqNaI883v50yEcBcnAci6RjnvBZXp2DQnU=; b=4Zr/MpAvf5K7p6j8KY94PevOvk2WDFKPoCrJ1kVvigUb2sbNNzFTOLwnXvnLSNVWkxZe5f vQarCJCo1eCKNmQhPgKOLcpy90xQzsU1JmwAjkvjyZRFQU0ZxUsWn5o34y8Z/7kIg8WIbw xwGHG6kaOrozjlhNJJFCcSpjdVjo63E= Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1tmHyy-000000001hX-3yln; Sun, 23 Feb 2025 14:49:44 -0500 From: Rik van Riel To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, bp@alien8.de, peterz@infradead.org, dave.hansen@linux.intel.com, zhengqi.arch@bytedance.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, kernel-team@meta.com, linux-mm@kvack.org, akpm@linux-foundation.org, jackmanb@google.com, jannh@google.com, mhklinux@outlook.com, andrew.cooper3@citrix.com, Manali.Shukla@amd.com, mingo@kernel.org, Rik van Riel Subject: [PATCH v13 06/14] x86/mm: use broadcast TLB flushing for page reclaim TLB flushing Date: Sun, 23 Feb 2025 14:48:56 -0500 Message-ID: <20250223194943.3518952-7-riel@surriel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250223194943.3518952-1-riel@surriel.com> References: <20250223194943.3518952-1-riel@surriel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: C6B27180007 X-Stat-Signature: mgd9q1q7y8d1har4f8eo4up1pn1oh9aq X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1740340274-418391 X-HE-Meta: U2FsdGVkX1/oZV2oh3BFeydCn3ad2+G9vbLrStJ8uZQbGSz7vldF95TtnyWjMpuhIu1STWn26D24Zp7bKYqT+HfPVMpCzUZ1nNjkWNw2ISeR/fgOEeVHHJqtB2ZH6lUdZbq1p+vRKUzaE7M77MQmIEeZg/IxksXlPROeulRfz3ptxvrvEMroGkxG+xVJJtsb/Bvp09zzUqH0McfIpSr7LEnMR9eV5DWNWeSrtWvXKqx+q+ILGsJtbyunlBN9mBh7Kv8s93LMv8zfrGUxNtDYLHxpAdqH24sitlO1iLnHuMwJB8p4go1QouAzMBjfIxqxJaCciKz2FtB/MnCeA+Q7R6SBMoiZ+AEO73onm38TvBBjgQhz7yPPcua0TPL/nSWZshEsFkTWqsvlYwXCEHfc7Iqx69fqnQMdrVh+a88OxHpYHWnO+apsKtqHtZHkVwJk9kkRyGrjd7CTq6o1rio69Ye9b4gbSdHhNVa8psdGcYiAbhjY6IlgLWheq91jhDnXhQA4AiWlL6qG+10B8m/azfFNHgZS2Wn88mPf6ZmO4JVPXgwwxs0b/9Qmvk4jKvGmZtU4FCHCnEKHTXG1qAgCe3aZQmYR6LSRgmuTWMS4MO9SGMgTj2Inf3Lggxi8ZKMQc5MofzEiu+lKi0iIxxI42hWyOLCGX6jfWnWB3NqKSQtXJq+JfX0z/gPkUtqSTqdyOJ5f/NrkyK6XLx3b2WSDV2BOhJ5moBtABRmt3I1KxXpO/YWq8c3Cn72hq14eGfeQey6CAjmngA2z/zMIUNacUYV2OV/1C0+2H1fgNZyJOfyvOfr3vmN/bimN75xMkBN3veLMSRlSsrP85RiBLVP0s4/TRaLsRXuVoY9gm9NfFQyJUQmBe9BucKRXjTvzywClyNtmGw7pt5LGdzpfbwyNWXfbFw4ldxlUV1z0LJ1mv4vtBi+DTkPLXNPf0G6dWXX6E3ACRiPEzBR0u9kesO6 DvBwp+Z3 FbEqdg1/YR3USTevmdPGnGDpOdxq3qdx44yliOUrrsQ0NnAkjxpK5PrdbbvDSUe2ztCehj9RmcVcee5GYw5FR1hyWSnQ4iiCBLBaGwpNNAGzmEv8lpUA3v3V0WwTr0JG6CHVkj2/1rlWgKZIzPX621F+FvPV/7djOrBfVP3oY3XTUv3kOTZgYnq1JUOK8W692n+/BPGjupUJ0OI+gwJ/52LHEdn/bqotektV+PU4oHHIp5O1KpjwHczSz1m6lwVSicg8VCo888USekUAUWaSoa5dcORgGn6LmDrUX2Dlhc4SsI/5tfjIUEAkXw7aIp4on8vVRzw/ONvXrdCBZkuA1sLYbZ0ySiU2frKnWTTr2uCUmKkfW1k5Sk+W3wpaWbvfEGn1kcX2kiYABtCz7/jlblxxionSZZq6sVJ8mQzV5SkwMpPStUXe2trV9wVY236H7zgm6mHVJSvzuYrY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the page reclaim code, we only track the CPU(s) where the TLB needs to be flushed, rather than all the individual mappings that may be getting invalidated. Use broadcast TLB flushing when that is available. This is a temporary hack to ensure that the PCID context for tasks in the next patch gets properly flushed from the page reclaim code, because the IPI based flushing in arch_tlbbatch_flush only flushes the currently loaded TLB context on each CPU. Patch 10 replaces this with the actual mechanism used to do broadcast TLB flushing from the page reclaim code. Signed-off-by: Rik van Riel Tested-by: Manali Shukla Tested-by: Brendan Jackman Tested-by: Michael Kelley --- arch/x86/mm/tlb.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 2d7ed0fda61f..16839651f67f 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1326,7 +1326,9 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) * a local TLB flush is needed. Optimize this use-case by calling * flush_tlb_func_local() directly in this case. */ - if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) { + invlpgb_flush_all_nonglobals(); + } else if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { flush_tlb_multi(&batch->cpumask, info); } else if (cpumask_test_cpu(cpu, &batch->cpumask)) { lockdep_assert_irqs_enabled();