From patchwork Tue Oct 10 21:36:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Woodhouse X-Patchwork-Id: 13416180 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A493CD98C4 for ; Tue, 10 Oct 2023 21:36:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343936AbjJJVgt (ORCPT ); Tue, 10 Oct 2023 17:36:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46692 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231224AbjJJVgs (ORCPT ); Tue, 10 Oct 2023 17:36:48 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 936E591 for ; Tue, 10 Oct 2023 14:36:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=MIME-Version:Content-Type:Date:Cc:To: From:Subject:Message-ID:Sender:Reply-To:Content-Transfer-Encoding:Content-ID: Content-Description:In-Reply-To:References; bh=GtamJgOMOys6lQJtmUDDFuPuo+d4pHCYcm7ggmWWGqU=; b=YPL787JKUxbhHKz/YsM4BelUyV lFELJ83++FUPv6aM8c7jJ54Fut0TMLkiWfZOgjS74gbw9dwBMUMaGVqnmtH+4QCvFYnwqHz1jBg2M zu9TRaWC5GMnNC/0pn73KBgF9vbl/mdb6R8On2gr+/GP3AYqXSHbvB2utHEvnHCXi0U6Oo/xBAW8J aXhLdMqr833kqwzQJky/BPTGPpwWjAH3gsR14ytAut9D3FkjNQLRYKXAdCvzCqPQN2QGHKxxUwHTL NasnHwV4ihitNvxm9HHJGUFPOSrAKu189n+Sd/RZkrOszWoSpc1a510oHAtVfr8SBF6QgvYA6dZsk g2bV3F3Q==; Received: from [2001:8b0:10b:5:2db5:a8b2:de2d:f60d] (helo=u3832b3a9db3152.ant.amazon.com) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1qqKPC-006vz6-3F; Tue, 10 Oct 2023 21:36:42 +0000 Message-ID: Subject: [RFC] KVM: x86: Don't wipe TDP MMU when guest sets %cr4 From: David Woodhouse To: kvm@vger.kernel.org, Ben Gardon , "Szczepanek, Bartosz" Cc: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" Date: Tue, 10 Oct 2023 22:36:41 +0100 User-Agent: Evolution 3.44.4-0ubuntu2 MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org If I understand things correctly, the point of the TDP MMU is to use page tables such as EPT for GPA → HPA translations, but let the virtualization support in the CPU handle all of the *virtual* addressing and page tables, including the non-root mode %cr3/%cr4. I have a guest which loves to flip the SMEP bit on and off in %cr4 all the time. The guest is actually Xen, in its 'PV shim' mode which enables it to support a single PV guest, while running in a true hardware virtual machine: https://lists.xenproject.org/archives/html/xen-devel/2018-01/msg00497.html The performance is *awful*, since as far as I can tell, on every flip KVM flushes the entire EPT. I understand why that might be necessary for the mode where KVM is building up a set of shadow page tables to directly map GVA → HPA and be loaded into %cr3 of a CPU that doesn't support native EPT translations. But I don't understand why the TDP MMU would need to do it. Surely we don't have to change anything in the EPT just because the stuff in the non-root-mode %cr3/%cr4 changes? So I tried this, and it went faster and nothing appears to have blown up. Am I missing something? Is this stupidly wrong? kvm_update_cpuid_runtime(vcpu); Also... if I have *two* vCPUs it doesn't go quite as slowly while Xen starts Grub and then Grub boots a Linux kernel. Until the kernel brings up its second vCPU and *then* it starts going really slowly again. Is that because the TDP roots are refcounted, and that idle vCPU holds onto the unused one and prevents it from being completely thrown away? Until the vCPU stops being idle and starts flipping SMEP on/off on Linux←→Xen transitions too? In practice, there's not a lot of point in Xen using SMEP when it's purely acting as a library for its *one* guest, living in an HVM container. The above patch speeds things up but telling Xen not to use SMEP at all makes things go *much* faster. But if I'm not being *entirely* stupid, there may be some generic improvements for KVM+TDPMMU here somewhere so it's worth making a fool of myself by asking...? --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -1072,7 +1074,8 @@ int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) if (kvm_x86_ops.set_cr4(vcpu, cr4)) return 1; - kvm_post_set_cr4(vcpu, old_cr4, cr4); + if (!vcpu->kvm->arch.tdp_mmu_enabled) + kvm_post_set_cr4(vcpu, old_cr4, cr4); if ((cr4 ^ old_cr4) & (X86_CR4_OSXSAVE | X86_CR4_PKE))