From patchwork Wed Feb 26 03:00:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rik van Riel X-Patchwork-Id: 13991455 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD08DC021BE for ; Wed, 26 Feb 2025 03:02:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 714CA28000E; Tue, 25 Feb 2025 22:02:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6C53128000B; Tue, 25 Feb 2025 22:02:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5652128000E; Tue, 25 Feb 2025 22:02:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 327AD28000B for ; Tue, 25 Feb 2025 22:02:39 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E83AC1A0833 for ; Wed, 26 Feb 2025 03:02:38 +0000 (UTC) X-FDA: 83160597996.24.5F0F1AD Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) by imf11.hostedemail.com (Postfix) with ESMTP id 6F83D4000B for ; Wed, 26 Feb 2025 03:02:37 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740538957; a=rsa-sha256; cv=none; b=gLlRad2Tmp+cBdLO8hsaP/txc9t9J5MkNzx9BbMBzilGLwemchmNhIFqEpFp2RmnvHF030 iAUF3ASn82StvtfBd1/DF1IxWKNGaeTKK2w02s/AzSKrZl9uPt3t/hNbh0JU/Rj9hxNHwi jDDvcDOC7HZoYWJoez/89/gMIr3YTkA= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740538957; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=I9UTqQ0klEhWgfKYKeTC6jtFZ5gtL53ANv0XBAHaLPE=; b=iCv7x2U7Lz5BkZvEnO+okLOpF3a2/CfIFomMAtZYK0oCsg4NZRPtoKnGqhxrhQRhaWSbA+ o64PR+49aaGH7muEzSxh03CcHsQcKIbVaXPjox8xXROAD8c4JEQz882WdaAMnYnOKc+xRO yhcxSqPNW+USwLSmC+SQ9kIVP0WjmQM= Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1tn7fw-000000001Y5-0r2n; Tue, 25 Feb 2025 22:01:32 -0500 From: Rik van Riel To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, bp@alien8.de, peterz@infradead.org, dave.hansen@linux.intel.com, zhengqi.arch@bytedance.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, kernel-team@meta.com, linux-mm@kvack.org, akpm@linux-foundation.org, jackmanb@google.com, jannh@google.com, mhklinux@outlook.com, andrew.cooper3@citrix.com, Manali.Shukla@amd.com, mingo@kernel.org, Rik van Riel Subject: [PATCH v14 06/13] x86/mm: use broadcast TLB flushing for page reclaim TLB flushing Date: Tue, 25 Feb 2025 22:00:41 -0500 Message-ID: <20250226030129.530345-7-riel@surriel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250226030129.530345-1-riel@surriel.com> References: <20250226030129.530345-1-riel@surriel.com> MIME-Version: 1.0 X-Stat-Signature: h8ak8zody7eqi6kn1h5zqf74kyeresku X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 6F83D4000B X-Rspam-User: X-HE-Tag: 1740538957-22340 X-HE-Meta: U2FsdGVkX198u/J010eHHhqiu93kSV+UjkYZ9jwIceDUI+AB64cP2v7NcP67udIYbk2qO858Xo3ziNoed5WCEGkeWbnBB45vH1ELAScEJ/9+Z/aXtOfJicsR4DUY4q9Dk8mR6HaXfdcOE8oNZEE+vpm5Yk3scTWHFx99r8zyQ2ByH6RRllxuRQ9rgZnj9ejfKmlDwsdJ9tboJ6SIGGTT80WI/WNeaSs44aY+4Gm5//JJWPLRRoZh8+IcMSQvG8ZElDwHs6pwaAn9nHKPboPn09gq3Q20Wl00mW/6ETbICTtxBW7a2z5IRWOHfHndLSKdxaSHW5AK5UBg7/QixrZMT2RJVpeh3hbrVPdh1jocaGhdzH6MXov4RVWqHwwBnxQw3DpslXMQnStCcIfvp0u9fPgRIZi47iooIrk5cgrONLYxLxjyE6KJYua696D2RTYb0UNy3AKe1NbXTDkx+m3bIh0xMtKYssgoZpmqDZyfltz+5w7CTev0sCuFQUyJR+kpnSeyGVcNYX6Q9WDjtoG+badzL6dHr1JwVWby3dwYOnEp/NhtaqrBNuUsUCMzfqfDjtUqhlmSoUcJhtt1PSY9qjretlz1amU3DkS9gHuhp4ycJxFcbKNe9mE5a1iIRzIuwNnviro2hChXP83BnjwI6aCxpf+As2NSgZonSzDMicIXelJlOilyzxwKEFfUr9b+oBzyDYQdsXQmjNpGvnVnjaO6iOwCmVFMNjSSc6IMNjBlBrYxQ7ATqyTOdTQIAeQrebEsuYQafDnoPQnI5XrdFbyU/KLBOS2UmPiol6CfoQ8BhXI/0O5xbdHLQ/YjwIvUpmN0vj3OChoE0AVbGmUGgtufVZyHez81BQOJbG+GKjcDht7/MKcndq98I1HBDrk33VaKNq/lUtp2btZlgNbvRy5esGt/hAJ64e5Sf3viC+z48rGyARzN3CWeo+eZXJ/yuHmF/IhARDWUkD8hzBK w47tKXPb vRIZjXGxaS9lo9V9BgRQ5TbuLNIxxe6aOs3UL3SmZwiQ8ETxDBtEfvCoLPEz2KYSHj8M2DVfWKw+VE8XnKjpfaKAmCGpsogOaUY2Alny89aBS45x4VgutCik9e0szmnDl8dy+RlE4iwRKgyvoGxUFQM2mGLWJe8YTXDP6GljRBDFhlSms5ihZv0lIRhBbQrxX+uZXZWVvQvA/60usmJcJqZZ0KDTBn7/VQcFD1iGFJExiXWkn5JdHdXoc4UOW5Smr3BbonmXgnfHHRVtviNgD78uOQ0qiLBXN2IlYP1TI9N8wbzkH54AcLKUNwyfhEgPIDKDxm3uB0ofGVN2MpaM0LN8OjgMo1AC7QtKG4Fj/80fJcBnDOSZznzykkqsONirBPLT3yS+BeueqeB7CeeYaoI4P4K3cEVsbkIFcbSiT29mKWa0FpTBF5yON3+THm+MXsDBPss9Vo3O139E= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the page reclaim code, we only track the CPU(s) where the TLB needs to be flushed, rather than all the individual mappings that may be getting invalidated. Use broadcast TLB flushing when that is available. This is a temporary hack to ensure that the PCID context for tasks in the next patch gets properly flushed from the page reclaim code, because the IPI based flushing in arch_tlbbatch_flush only flushes the currently loaded TLB context on each CPU. Signed-off-by: Rik van Riel Tested-by: Manali Shukla Tested-by: Brendan Jackman Tested-by: Michael Kelley --- arch/x86/mm/tlb.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index a6cd61d5f423..1cc25e83bd34 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1316,7 +1316,9 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) * a local TLB flush is needed. Optimize this use-case by calling * flush_tlb_func_local() directly in this case. */ - if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) { + invlpgb_flush_all_nonglobals(); + } else if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { flush_tlb_multi(&batch->cpumask, info); } else if (cpumask_test_cpu(cpu, &batch->cpumask)) { lockdep_assert_irqs_enabled();