From patchwork Thu Jan 16 02:30:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rik van Riel X-Patchwork-Id: 13941173 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E545EC02180 for ; Thu, 16 Jan 2025 02:55:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6FF886B0082; Wed, 15 Jan 2025 21:55:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6881F6B0085; Wed, 15 Jan 2025 21:55:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 529156B0088; Wed, 15 Jan 2025 21:55:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 318386B0082 for ; Wed, 15 Jan 2025 21:55:43 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id E217BAEB6B for ; Thu, 16 Jan 2025 02:55:42 +0000 (UTC) X-FDA: 83011799724.24.67BC14C Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) by imf14.hostedemail.com (Postfix) with ESMTP id 62861100007 for ; Thu, 16 Jan 2025 02:55:41 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736996141; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cQd0Jxm3wPpHkQ2qCXJZ2BF12qzYrqztWPzol4AzppY=; b=gazXpqypeBZzLtRu9KdhIZXX7PUcg9k44aEQGNr6zRmb4kFqZwjEg60b46szPB7m9kd9IL h4x70qMcTbWI3XqJEIcAEH/pL0mdXWMUs51P/7S5HBO68NKHPO8Vl2B+ulAz5C08aKbvpO sE5q9RwYOPo/19ExKQPdSWxPoehuoHM= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736996141; a=rsa-sha256; cv=none; b=HWeIfteldIA3MAQ8l6WQqP62AiFVRP4fYV4x40fguDZVIWowC6iUDpsgzvyAk+Y0HJf6wK tyErNsgJ3je3rK27V6eJETbbVAKRsUKA5sIROZUoVTmK9orbrKOkdRnMyDDI3+Cj7EBgQR H+LRS97ylGN9gREgns4T98kZvhDssrQ= Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1tYFfO-000000003nd-3c0c; Wed, 15 Jan 2025 21:31:30 -0500 From: Rik van Riel To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, bp@alien8.de, peterz@infradead.org, dave.hansen@linux.intel.com, zhengqi.arch@bytedance.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, kernel-team@meta.com, linux-mm@kvack.org, akpm@linux-foundation.org, jannh@google.com, mhklinux@outlook.com, andrew.cooper3@citrix.com, Rik van Riel Subject: [PATCH v5 08/12] x86/mm: use broadcast TLB flushing for page reclaim TLB flushing Date: Wed, 15 Jan 2025 21:30:31 -0500 Message-ID: <20250116023127.1531583-9-riel@surriel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250116023127.1531583-1-riel@surriel.com> References: <20250116023127.1531583-1-riel@surriel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 62861100007 X-Stat-Signature: rg6yptebmk1c5xgpjru1b54g3jd1oh3y X-Rspam-User: X-HE-Tag: 1736996141-48568 X-HE-Meta: U2FsdGVkX19DOcJo1qGE13frstQUMzbjUep452AZaeIMOnCV1IbkCprWAWlRVit2X/wfIjm/KRZf9Pl7VvEIKbQzCX+5F1KkKt8dZ7lPXE28J9QmeE2dD2RmLgZh+0f6EbCTWgH4gIV3LPGI7CpEIKmGuhy+FPU7X6gFrcxnLQ0NDl4yCt6Qlk0dLPlIQwAV6DFcbcwrIEbjJhZYbIju3Z44BNEWkFW8baxijw3YYoIZ6kpjZrOtcw2tAapp2UQam72yDL2E4gHzkPh3WnJNaIHft0LFfeLcthPO+TyfFnRVKx8lTQv86PcNeC/HIa/SosgFkY2PCJ8GVW1KpnrHhHCgWEDBetD1uGCplCfvNDnke6FE7ePfZBNCT9oW3Gqt8PnC5SDnJwHQlBbYOLsEwc8/SlHaMDm/GnMhUl18bF5laRzSHpOf7PCBoljJ2cbIfeSPPFvaiaVS/+6+2WUse9GwTzJKvjoW9eXO6+lez2QtXNP+ZWCep2PpWo+uiTVtPK5GNnLS5+gaaUuY5mia6r2nRuwlEEYDOTdkPAC3qyK2S7m4mcJrKN228xtiPe5Ae6T8MSlRF6Fbwrm3+VdZj1MB04dT5I9kRM0lhdIGVf0IwNKVriQgf58SHzVG4U6YOYfVbhiH4jHm3ED+PctcXKw6f0TlVEWT0CaW7wFfzDLiHyy1ShkYqyVZdwIBE+NCgqEvuVmzgWhYEISLGDKwsnII0XHOpunJpdWOiGEg62L+A+70DqWNhQaUPwBAr0h4pq3dy5vrInL/lPLjs/k+UBmrnptqweYYdfoyvttU8S4QXU+rBYPDKndDa2EHJfebEs/8hdfbEWCv1UElPFbojlZCOluc+aQHuK8yQYTONs2IYucSX8rZ+Apf9W3IpeZgn4L9XxlvPhqw8ukAKElkXmnGGAedyvDo0ZSmrsd5mgST4R3JMA5QVICLWXlxWoHmvLlahVi4bkyo3fkgKwq mQel+X8L 1y/DkVYxicLi0xycMj7neJ1IqHSUGWQFHe+S8PS5Jnji0ICJRxDsm1+Z2vXSJnrGlqrKp5n8HPjkwLsbkTS7xEDCWssrV7YnRA2laiUWQG8XSEr/9Dw4AlhK31GGcfy7okOJX5lPj2XxRk/yvgNWFaC12ilpxBxQhxlWRQayoPehmJw1zmq76X1naRUFvL0khbLjru/F8o4azY1M3GYPjzr6cMb46G4LVFCEVFcSLb3OmpAZU3owWY0ffBzBY1IyY8ufx7UVOUkC6WEWEmDjrjhbEJO0T8kGYCd2OfmugC0aJU17AqsMpNhOqNV4UzZfxBE8tg0FGpmrm5KNbqiETgEmz/A6q4NkCzls8G0pz1lRvlAlTmDLcV4MS2zvgOPJjwNE2yLjceF1sFkLglbTulshr7dmzuD60WiPw2bkCI1Dj2mY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the page reclaim code, we only track the CPU(s) where the TLB needs to be flushed, rather than all the individual mappings that may be getting invalidated. Use broadcast TLB flushing when that is available. Signed-off-by: Rik van Riel --- arch/x86/mm/tlb.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 49b3e90503a3..746a89924f02 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1321,7 +1321,9 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) * a local TLB flush is needed. Optimize this use-case by calling * flush_tlb_func_local() directly in this case. */ - if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) { + invlpgb_flush_all_nonglobals(); + } else if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { flush_tlb_multi(&batch->cpumask, info); } else if (cpumask_test_cpu(cpu, &batch->cpumask)) { lockdep_assert_irqs_enabled();