From patchwork Tue Feb 11 21:08:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rik van Riel X-Patchwork-Id: 13970694 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 361A4C0219D for ; Tue, 11 Feb 2025 21:09:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C3AE16B0085; Tue, 11 Feb 2025 16:09:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BB2526B0093; Tue, 11 Feb 2025 16:09:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B6D86B0085; Tue, 11 Feb 2025 16:09:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6F9A66B0088 for ; Tue, 11 Feb 2025 16:09:05 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 08A31140279 for ; Tue, 11 Feb 2025 21:09:05 +0000 (UTC) X-FDA: 83108903850.17.0CC513C Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) by imf24.hostedemail.com (Postfix) with ESMTP id 7C06218000D for ; Tue, 11 Feb 2025 21:09:03 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf24.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739308143; a=rsa-sha256; cv=none; b=q05bT9A4p8ZUOioXAvYtz1f/OJYj8enVF12WSyweeONNWszY1GoQHprJRZItxTlrlF3TCY e/HonW4qk5PhUKw809JPNROvyuzQrnFgmHISXNx32SkxqZ6z1K000q/tV6+YW0FX5Rj+Ar 4lkc16HMdp4VT1qXOa6xF44nRdTL6P8= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf24.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739308143; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YwhRUul0Ead0E8FstHFdRR/pLtGJxHIOsTVrzdrugr4=; b=dgupN0TWFCYsH1vaBSDTu7GFmYadsEyqGbAPGttot32CcSxhJKjw1GPR+2Fbz/COHjgJup skEuSJvB0on+290aJh58JY61CCtarclNwm8tZ/nWv6Yc1GFm+XROToGjNQZfEuQVa5s/R2 tjfUVMYBMNTbJN2BV2Ddsz8Dw2cYlpE= Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1thxUX-000000008HU-3BaY; Tue, 11 Feb 2025 16:08:25 -0500 From: Rik van Riel To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, bp@alien8.de, peterz@infradead.org, dave.hansen@linux.intel.com, zhengqi.arch@bytedance.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, kernel-team@meta.com, linux-mm@kvack.org, akpm@linux-foundation.org, jackmanb@google.com, jannh@google.com, mhklinux@outlook.com, andrew.cooper3@citrix.com, Rik van Riel , Manali Shukla Subject: [PATCH v10 08/12] x86/mm: use broadcast TLB flushing for page reclaim TLB flushing Date: Tue, 11 Feb 2025 16:08:03 -0500 Message-ID: <20250211210823.242681-9-riel@surriel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250211210823.242681-1-riel@surriel.com> References: <20250211210823.242681-1-riel@surriel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 7C06218000D X-Stat-Signature: ikfucfjmox856tx1ekha1ptpeckugsfb X-Rspam-User: X-HE-Tag: 1739308143-757165 X-HE-Meta: U2FsdGVkX19lDrGL8VX385O4JjmlqBZhCyfLtgH/fQTNZmpTBMg3Mm5GnJf6wIzo7etpgZQ5ADVY9NCI6PIejxgqokRXdkE6OMpFzW91dwvMghsA493geR5LGwu1x1utmlr/y9stDtzGYrI+4LtYaz5HYLDnqLTlEIWnSg9pD573HFjec1Zhsq5VdTOS/4XgVWVIsOIw5aAcf6pIgBYseX4rCU+S+XBjqmkmYDK3wq2HwV2j55S1SF22fS4F5FakSXF/l1cQUe69b21QKk9YPLgMb7Vsj96V8FDc5HMkLdJtL6r2iPsmdzlAvnI8CHorKcQ/HkPv+hlWC98pgaUgk8mgMhBwzUcuOg7h1INzNH+ZPThjt/kuynDXR8MUpvOneAh5KdqKyfbkwlbr/91fDrRs0cDX5ZP/g++TZj03onPTKjC8i0Ffv3Xftgj39JvDZ58inFCE5t0yDBfQuiLQRXckm2Pog33Yi2QNreOC1aZJryWRDg+ah+kxdhY15Oe+oY5FyZFr/Q3Y9atiwxxhijmQgI+/ZQeH6ifChVi7s/UBevZ98W5okxZWu2mEDiSt9p06OyOwOXKfcKleDch/qGCmhJXkKv3shU8xvQjkbCAIF1NXbFwZo66jhYVNwOqEPO0QSQO4UahQST2K5bn6HALSVloDpHHTk7n4yO6auWVyoRshLb1e4CdG0anQfzFGz+mF2qS4jJgwJbUgpcvOInuG4X03vRRvsQk8LGEWr+B3hXxTQix7hyPfhU+ImpL2iDGhZ9MIbUJCo795CVfq/yd5LkrRdlRzbcm3EqhTywDSRX6yYjje+JNipArgu0qNDHI3gvh5q18uRhQIOu+E7lqc/HmZs2T190lJDPnRPpUqMeDUDF6ZJaUhQW/iV2vcCIt9WYY+21PZPOhAlHHECvGxG0SVxd8wZTZT5ybqGDvIkRCnqywiRUdKNQhzdoMmUsvjYOhY6rkf5VSM1dj RHQ2wZ4e w8pJjC2k0W0TmtYGGSkwMHAnvrKeZnlyABdNTwQ78wbkyR4g9ARgDHiSd+8/UiocSDJiRpos49/I3OTDj9DOMH4UK8KD5/1ImJhzajK56H5hU6PjPtcAgCX9bIvhmC9FTL5psmtmF0pQgMiLZ9kSUWh/y3+ViBITy/xa0UIClAU3/K4cL4t07Yabtty6aEg9/EJ049I+31fy7rNjcO0GhQKCDK57idTFeSvfsXiCy+Dy+8fxO4X1avSiq2AgLEZIChhz0lr3e+CCkYfZ8d9zIfSAdMuJfP7wZAI/X1WefSBvu6VyXw1PDeDLolfdIiN38mNguKVapbjzM2rsY8NkUF/PHbTb0Ney2YeNKHsl8+gJvLohQaXsB8w61uOV9koCbSBgpxroUxMPeUGLz2crVB2WD5nYVaRudiSJwawRJq/q9DBHmSQOQ94484o2cIRfSJDcYM6dy0kZbWoWHonB/An/lhP6H7iPigHLZWNvskHEsPcCBkP2ucJMgbVKnaVXgj0FptMLy2C31v0g= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the page reclaim code, we only track the CPU(s) where the TLB needs to be flushed, rather than all the individual mappings that may be getting invalidated. Use broadcast TLB flushing when that is available. Signed-off-by: Rik van Riel Tested-by: Manali Shukla Tested-by: Brendan Jackman --- arch/x86/mm/tlb.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 62b4eaf4e851..55c922083aa2 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1316,7 +1316,9 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) * a local TLB flush is needed. Optimize this use-case by calling * flush_tlb_func_local() directly in this case. */ - if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) { + invlpgb_flush_all_nonglobals(); + } else if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { flush_tlb_multi(&batch->cpumask, info); } else if (cpumask_test_cpu(cpu, &batch->cpumask)) { lockdep_assert_irqs_enabled();