From patchwork Sun Feb 23 19:48:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rik van Riel X-Patchwork-Id: 13987222 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67C82C021B6 for ; Sun, 23 Feb 2025 19:51:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AC4086B0089; Sun, 23 Feb 2025 14:51:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A10616B008A; Sun, 23 Feb 2025 14:51:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6DE786B0083; Sun, 23 Feb 2025 14:51:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3A8816B0082 for ; Sun, 23 Feb 2025 14:51:15 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id A77F9120291 for ; Sun, 23 Feb 2025 19:51:14 +0000 (UTC) X-FDA: 83152253268.11.270E337 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) by imf27.hostedemail.com (Postfix) with ESMTP id 2341E40005 for ; Sun, 23 Feb 2025 19:51:12 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=none; spf=pass (imf27.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740340272; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=H8e1q/ijWxutZBBUZ4egEFqPViZpFe2V2vsQQKwVkPs=; b=f7p6RSmis9yBQCzn8b0bG3G62T9Uq4CxrWDmK4YxWwK1oefCH99buHW74qAwpjgxzPJ3qR d7qUnZRvOGPlh7yS2QJgVKELR460HwkJQOhGAniC1ol+3R5UvIDBopfC2HFOASALRXwvGB xMIYxmiDCk5aExDKlmoGBINEtchfyiM= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=none; spf=pass (imf27.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740340272; a=rsa-sha256; cv=none; b=OSC2g8p2VTx15GLsERFKcsGc/yzMG9rKwyljSlQdl9KMLAr/qWMPZPejRc3dkuFEcK36HI 9IbrUS/nl54VQmvYMEeAYdWtQnmuXDw97XePnaj8IuQjT0s9LHloavLdN4ztXcy4P+L+wa 69Tmfqz35FMFOhi8u07phTkSEMCRJEQ= Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1tmHyy-000000001hX-3tae; Sun, 23 Feb 2025 14:49:44 -0500 From: Rik van Riel To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, bp@alien8.de, peterz@infradead.org, dave.hansen@linux.intel.com, zhengqi.arch@bytedance.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, kernel-team@meta.com, linux-mm@kvack.org, akpm@linux-foundation.org, jackmanb@google.com, jannh@google.com, mhklinux@outlook.com, andrew.cooper3@citrix.com, Manali.Shukla@amd.com, mingo@kernel.org, Rik van Riel Subject: [PATCH v13 05/14] x86/mm: use INVLPGB in flush_tlb_all Date: Sun, 23 Feb 2025 14:48:55 -0500 Message-ID: <20250223194943.3518952-6-riel@surriel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250223194943.3518952-1-riel@surriel.com> References: <20250223194943.3518952-1-riel@surriel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 2341E40005 X-Stat-Signature: 75e81m815c1gerczg3b9nnkssp83fbaj X-Rspam-User: X-HE-Tag: 1740340272-858071 X-HE-Meta: U2FsdGVkX1/T5j7ypYVlFSLgXbYBKehXg0tq/wPFUNx1MFHEf0/Ajpso4kTUqFN/rg+0pahd1olZd1iHJRwcPQ6BIrwpKtAtZO1Q9x0v0qo1iBYt9303V/hKvH8UylQUMdTLeoQj3HrGRBRQRwLW3b3Nx+nrcyVcSoKVpXAUozhc+4Mn+9pk3lElz/nD7N70oCQq5gd5pHVOxmPneRQi0CgxdpQprr3LLHWcCxwM1s2z4sljFOXSozvQVdYNZlD6tI+8GlYYwjU26oIYHTG848XR8fgPsgX0b+pebGugZU3hqM+aTpcciuNXN+nbBKqwVlpJ5/UYeAZvTxe2uBPBtFVK992GllxbEVUrrGiHNx8RlcesZlpRZ61Hi3SyTiqNJ8X6A7Yked2oTGhHevF18a5Aevqli6cXfQNzJMJws0Inn1Ve/PTQJn+zJKxAK6mqwM6exo2yMlnP/65cHk0WbkkepUYG8xoZ0RgJOuPwrALzYzHLA+t2H6rrb4ZfgR6qGUe1P5/EMFQsYs29Y3IP91yw9GTCD2PgmS/AcMZjRCwihS5QTp0rrVA42Hh5Ba74ZZUnfbpQIzBET0fW4KB1fdJcjBagKX0c1SAKXx9hTeM/G7BH8IXjpWDR1p+D1xRZSRp0uAJFJR4kDXRBISbbbXyBwGNsQHt6hy7kaPb6injF4Ya43bpv0++5jA9ypkgfFfBYFZbS2Tg4O2HjSXbwgegu11C7VGvXVVy5ZAFmkWdRUcIFHS9FP+LWo+r7hBTRBVdtatVRChRxQsOLVIA+KqLV7XnJHqGhSkCU3CXofewQS87EM2gDWGZXe8WzxqbWohBc4AbZ8ifTxbE4+r4NrrtvvOQ6mcJ+FJAR650KqlB4/iOirufypFHEm6FD3X9KdVNoUbgjFq2d8X7fQO020wOI6xXYXeG/OvxwKmHcmSqNSmXbU4UyhvuXB4j8OazncKk+5NzJgYNACYViZoT +Bl5PD0u V35OeED7acNy2h6g1tnzPrnNNdWG/ecmUpGVBsawj2S3sqfRFIITQQOqtpDtz22YNmoN1ntQYZuCLEPjDVrA0aYEdfDvRSEfHKJVhAAKJGID1HfFHuO9KZcg85L8Kr8Qddm3IjIJ2SjMtUjSwXhJmMTxa5kiv0qCbkGQoQWL+31GN2ZCo3QqWmluULjmDwLdDqQMNY1adCUytRYvnBvD4EMxSYSyPlcFkRPAaXbfrMk7jjDz64l1lnWbnhOTzbMLdC7Kmk/Fd+IOTr3277wosdEiKi2wBxZe/reugVi8clnhUv8QXqROgbvFywR8s+bP/0hNsdABI7uSdWAQFiIujhPVIXtj2EU5dM+1Y0gNBrEvwLGrvQs9vQ0IfInQTm4WIYiIwyUpv7Z/KTMAuW5ur+WEBOlnc/kSjUkkM69vmL2QEkv8IlwO4BRvTEHjRz+u1e5ZqUoNF7HziGl7eoFPFfdSvy5+nu4aoqK2og6EOJ/y/n8gmsbgO2mtisw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The flush_tlb_all() function is not used a whole lot, but we might as well use broadcast TLB flushing there, too. Signed-off-by: Rik van Riel Tested-by: Manali Shukla Tested-by: Brendan Jackman Tested-by: Michael Kelley --- arch/x86/mm/tlb.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 59396a3c6e9c..2d7ed0fda61f 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1065,6 +1065,16 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, } +static bool broadcast_flush_tlb_all(void) +{ + if (!cpu_feature_enabled(X86_FEATURE_INVLPGB)) + return false; + + guard(preempt)(); + invlpgb_flush_all(); + return true; +} + static void do_flush_tlb_all(void *info) { count_vm_tlb_event(NR_TLB_REMOTE_FLUSH_RECEIVED); @@ -1074,6 +1084,12 @@ static void do_flush_tlb_all(void *info) void flush_tlb_all(void) { count_vm_tlb_event(NR_TLB_REMOTE_FLUSH); + + /* First try (faster) hardware-assisted TLB invalidation. */ + if (broadcast_flush_tlb_all()) + return; + + /* Fall back to the IPI-based invalidation. */ on_each_cpu(do_flush_tlb_all, NULL, 1); }