From patchwork Thu Mar 25 10:29:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Borislav Petkov X-Patchwork-Id: 12163677 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D866DC433DB for ; Thu, 25 Mar 2021 10:30:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 87BBF61A25 for ; Thu, 25 Mar 2021 10:30:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230125AbhCYKaH (ORCPT ); Thu, 25 Mar 2021 06:30:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55446 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230076AbhCYKaE (ORCPT ); Thu, 25 Mar 2021 06:30:04 -0400 Received: from mail.skyhub.de (mail.skyhub.de [IPv6:2a01:4f8:190:11c2::b:1457]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 37B19C06174A; Thu, 25 Mar 2021 03:30:03 -0700 (PDT) Received: from zn.tnic (p200300ec2f0d5d00d5a461c7dd3b44f2.dip0.t-ipconnect.de [IPv6:2003:ec:2f0d:5d00:d5a4:61c7:dd3b:44f2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.skyhub.de (SuperMail on ZX Spectrum 128k) with ESMTPSA id AFE6D1EC0501; Thu, 25 Mar 2021 11:30:00 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alien8.de; s=dkim; t=1616668200; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=CAkDP25OFlShJCPDxQST7B40b5DDqLE/vRJZQKviFIQ=; b=GdX96tqMBdFMRuPKse/S5u+Fn7yuJhdOOt6Xq+BUnlxv1nfU9Ouittcw4nMEYk0s4wgFjl KsRna7lhZb8xuvYmGalsy8QTFdA1sKAVvm+hnk69HcRvajVPAZOIa6v1N4s2mjyV2InLaD HOQj5Arosp7WG6MWL/Xs5JTqzLUsuQg= Date: Thu, 25 Mar 2021 11:29:59 +0100 From: Borislav Petkov To: Hugh Dickins Cc: Babu Moger , Paolo Bonzini , Jim Mattson , Vitaly Kuznetsov , Wanpeng Li , kvm list , Joerg Roedel , the arch/x86 maintainers , LKML , Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Makarand Sonare , Sean Christopherson Subject: [PATCH] x86/tlb: Flush global mappings when KAISER is disabled Message-ID: <20210325102959.GD31322@zn.tnic> References: <2ca37e61-08db-3e47-f2b9-8a7de60757e6@amd.com> <20210311214013.GH5829@zn.tnic> <4a72f780-3797-229e-a938-6dc5b14bec8d@amd.com> <20210311235215.GI5829@zn.tnic> <20210324212139.GN5010@zn.tnic> <20210325095619.GC31322@zn.tnic> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20210325095619.GC31322@zn.tnic> Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Ok, I tried to be as specific as possible in the commit message so that we don't forget. Please lemme know if I've missed something. Babu, Jim, I'd appreciate it if you ran this to confirm. Thx. Reviewed-by: Paolo Bonzini Tested-by: Babu Moger Acked-by: Hugh Dickins Tested-by: Jim Mattson --- From: Borislav Petkov Date: Thu, 25 Mar 2021 11:02:31 +0100 Jim Mattson reported that Debian 9 guests using a 4.9-stable kernel are exploding during alternatives patching: kernel BUG at /build/linux-dqnRSc/linux-4.9.228/arch/x86/kernel/alternative.c:709! invalid opcode: 0000 [#1] SMP Modules linked in: CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.9.0-13-amd64 #1 Debian 4.9.228-1 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: swap_entry_free swap_entry_free text_poke_bp swap_entry_free arch_jump_label_transform set_debug_rodata __jump_label_update static_key_slow_inc frontswap_register_ops init_zswap init_frontswap do_one_initcall set_debug_rodata kernel_init_freeable rest_init kernel_init ret_from_fork triggering the BUG_ON in text_poke() which verifies whether patched instruction bytes have actually landed at the destination. Further debugging showed that the TLB flush before that check is insufficient because there could be global mappings left in the TLB, leading to a stale mapping getting used. I say "global mappings" because the hardware configuration is a new one: machine is an AMD, which means, KAISER/PTI doesn't need to be enabled there, which also means there's no user/kernel pagetables split and therefore the TLB can have global mappings. And the configuration is new one for a second reason: because that AMD machine supports PCID and INVPCID, which leads the CPU detection code to set the synthetic X86_FEATURE_INVPCID_SINGLE flag. Now, __native_flush_tlb_single() does invalidate global mappings when X86_FEATURE_INVPCID_SINGLE is *not* set and returns. When X86_FEATURE_INVPCID_SINGLE is set, however, it invalidates the requested address from both PCIDs in the KAISER-enabled case. But if KAISER is not enabled and the machine has global mappings in the TLB, then those global mappings do not get invalidated, which would lead to the above mismatch from using a stale TLB entry. So make sure to flush those global mappings in the KAISER disabled case. Co-debugged by Babu Moger . Reported-by: Jim Mattson Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/CALMp9eRDSW66%2BXvbHVF4ohL7XhThoPoT0BrB0TcS0cgk=dkcBg@mail.gmail.com --- arch/x86/include/asm/tlbflush.h | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index f5ca15622dc9..2bfa4deb8cae 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -245,12 +245,15 @@ static inline void __native_flush_tlb_single(unsigned long addr) * ASID. But, userspace flushes are probably much more * important performance-wise. * - * Make sure to do only a single invpcid when KAISER is - * disabled and we have only a single ASID. + * In the KAISER disabled case, do an INVLPG to make sure + * the mapping is flushed in case it is a global one. */ - if (kaiser_enabled) + if (kaiser_enabled) { invpcid_flush_one(X86_CR3_PCID_ASID_USER, addr); - invpcid_flush_one(X86_CR3_PCID_ASID_KERN, addr); + invpcid_flush_one(X86_CR3_PCID_ASID_KERN, addr); + } else { + asm volatile("invlpg (%0)" ::"r" (addr) : "memory"); + } } static inline void __flush_tlb_all(void)