From patchwork Fri May 12 13:20:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathias Krause X-Patchwork-Id: 13239290 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05B6EC7EE2E for ; Fri, 12 May 2023 13:20:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241174AbjELNUq (ORCPT ); Fri, 12 May 2023 09:20:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42500 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240871AbjELNUm (ORCPT ); Fri, 12 May 2023 09:20:42 -0400 Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31EAEE72 for ; Fri, 12 May 2023 06:20:41 -0700 (PDT) Received: by mail-ej1-x632.google.com with SMTP id a640c23a62f3a-965fc25f009so1526961666b.3 for ; Fri, 12 May 2023 06:20:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grsecurity.net; s=grsec; t=1683897639; x=1686489639; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Qa8LWCnpnjsYiqtvKVSJdNLhlbxO/kudwlUnqlbTiAY=; b=Ddi1mJsYxyPgJ9MOFOIUY+tLhU8bv6TWfJe5fSq3tciDpFlLjNlbXydgtQYddAg5RV C5Quxc5YI+X+yDeNllZzwhUcDvihmkOX7zbqPWoB97Al/kk1Jhgwf1fdZml2XrUMa9Z8 BgP7+UVz22QO9k9GxyxsWY96ttQkiUPxNV2XGnuz8lpR02mFZWvJohAcp92U2XCBtsxe oeWFvFxZr/0FLJAUO3HcC7w0yaQku5YjcCZ3m43anrv/vWhxr9aFoMKshb4XtEKSxuLH 619ccjq0P5Wg1yTGzuwdnZEcHz7mZ5ljSyrTUq77zJEDKDGZ8RJD5/sNKgDAkwpRZUQ/ C9Tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683897639; x=1686489639; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Qa8LWCnpnjsYiqtvKVSJdNLhlbxO/kudwlUnqlbTiAY=; b=KnmB2/f8P7qZ855m07HqhT6Cb2LF2BENZ71oHz0BIFmTb6kaZQw3IfTvn5WtErt8c9 NOP5Isg9sdKVAe9bAdKy1+r0MTBRIMCQhAMnXyY4+OXYdbmi1nn/rXmPGqV98KSLG9ug LUGY4L/di4HdzPL1w+yerxdoZ4Z78jiN0B/IfKaqEiw7WbYXxarFqOHr5ElIpQBzUwCw Yux7e7jqfKU94fyvNGu0m3dOqAiwVo0EM3KWWZ6AB2qrtaz3+px4e2shL+TNVSRa1T9C yh8k1PEbYF81G7Q3h+egcH5wxvAbQXILcph8zg26YXaGsavk+47ao4Ap7n9ItiqetzUI HwWA== X-Gm-Message-State: AC+VfDyOWpo1NPQK9CUa+yVjithkHOyhCYQCyRupmdYN4hYuBa5mr3a7 od8eepMyDmdj2KTZGwK2w5VLYZaYvL/oBMUF5hs= X-Google-Smtp-Source: ACHHUZ4+gIHhXtbr3PVxk8DkcxbldJwX8tJ9ACY7/rrBQUFCyrEBrUWF9BrQa+ISOCpwSdqsfY9iOA== X-Received: by 2002:a17:907:3e9c:b0:966:471c:2565 with SMTP id hs28-20020a1709073e9c00b00966471c2565mr19629938ejc.48.1683897639582; Fri, 12 May 2023 06:20:39 -0700 (PDT) Received: from nuc.fritz.box (p200300f6af43a100a78da3f586d44204.dip0.t-ipconnect.de. [2003:f6:af43:a100:a78d:a3f5:86d4:4204]) by smtp.gmail.com with ESMTPSA id w21-20020a170907271500b00969dfd160aesm5077981ejk.109.2023.05.12.06.20.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 12 May 2023 06:20:39 -0700 (PDT) From: Mathias Krause To: stable@vger.kernel.org Cc: Paolo Bonzini , Sean Christopherson , kvm@vger.kernel.org, Mathias Krause Subject: [PATCH 6.3 1/5] KVM: x86/mmu: Avoid indirect call for get_cr3 Date: Fri, 12 May 2023 15:20:20 +0200 Message-Id: <20230512132024.4029-2-minipli@grsecurity.net> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230512132024.4029-1-minipli@grsecurity.net> References: <20230512132024.4029-1-minipli@grsecurity.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Paolo Bonzini [ Upstream commit 2fdcc1b324189b5fb20655baebd40cd82e2bdf0c ] Most of the time, calls to get_guest_pgd result in calling kvm_read_cr3 (the exception is only nested TDP). Hardcode the default instead of using the get_cr3 function, avoiding a retpoline if they are enabled. Signed-off-by: Paolo Bonzini Signed-off-by: Mathias Krause Link: https://lore.kernel.org/r/20230322013731.102955-2-minipli@grsecurity.net Signed-off-by: Sean Christopherson Signed-off-by: Mathias Krause # backport to v6.3.x --- arch/x86/kvm/mmu/mmu.c | 31 ++++++++++++++++++++----------- arch/x86/kvm/mmu/paging_tmpl.h | 2 +- 2 files changed, 21 insertions(+), 12 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index c8ebe542c565..18c0deeaa2ec 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -242,6 +242,20 @@ static struct kvm_mmu_role_regs vcpu_to_role_regs(struct kvm_vcpu *vcpu) return regs; } +static unsigned long get_guest_cr3(struct kvm_vcpu *vcpu) +{ + return kvm_read_cr3(vcpu); +} + +static inline unsigned long kvm_mmu_get_guest_pgd(struct kvm_vcpu *vcpu, + struct kvm_mmu *mmu) +{ + if (IS_ENABLED(CONFIG_RETPOLINE) && mmu->get_guest_pgd == get_guest_cr3) + return kvm_read_cr3(vcpu); + + return mmu->get_guest_pgd(vcpu); +} + static inline bool kvm_available_flush_tlb_with_range(void) { return kvm_x86_ops.tlb_remote_flush_with_range; @@ -3731,7 +3745,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) int quadrant, i, r; hpa_t root; - root_pgd = mmu->get_guest_pgd(vcpu); + root_pgd = kvm_mmu_get_guest_pgd(vcpu, mmu); root_gfn = root_pgd >> PAGE_SHIFT; if (mmu_check_root(vcpu, root_gfn)) @@ -4181,7 +4195,7 @@ static bool kvm_arch_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, arch.token = alloc_apf_token(vcpu); arch.gfn = gfn; arch.direct_map = vcpu->arch.mmu->root_role.direct; - arch.cr3 = vcpu->arch.mmu->get_guest_pgd(vcpu); + arch.cr3 = kvm_mmu_get_guest_pgd(vcpu, vcpu->arch.mmu); return kvm_setup_async_pf(vcpu, cr2_or_gpa, kvm_vcpu_gfn_to_hva(vcpu, gfn), &arch); @@ -4200,7 +4214,7 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work) return; if (!vcpu->arch.mmu->root_role.direct && - work->arch.cr3 != vcpu->arch.mmu->get_guest_pgd(vcpu)) + work->arch.cr3 != kvm_mmu_get_guest_pgd(vcpu, vcpu->arch.mmu)) return; kvm_mmu_do_page_fault(vcpu, work->cr2_or_gpa, 0, true); @@ -4604,11 +4618,6 @@ void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd) } EXPORT_SYMBOL_GPL(kvm_mmu_new_pgd); -static unsigned long get_cr3(struct kvm_vcpu *vcpu) -{ - return kvm_read_cr3(vcpu); -} - static bool sync_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn, unsigned int access) { @@ -5159,7 +5168,7 @@ static void init_kvm_tdp_mmu(struct kvm_vcpu *vcpu, context->page_fault = kvm_tdp_page_fault; context->sync_page = nonpaging_sync_page; context->invlpg = NULL; - context->get_guest_pgd = get_cr3; + context->get_guest_pgd = get_guest_cr3; context->get_pdptr = kvm_pdptr_read; context->inject_page_fault = kvm_inject_page_fault; @@ -5309,7 +5318,7 @@ static void init_kvm_softmmu(struct kvm_vcpu *vcpu, kvm_init_shadow_mmu(vcpu, cpu_role); - context->get_guest_pgd = get_cr3; + context->get_guest_pgd = get_guest_cr3; context->get_pdptr = kvm_pdptr_read; context->inject_page_fault = kvm_inject_page_fault; } @@ -5323,7 +5332,7 @@ static void init_kvm_nested_mmu(struct kvm_vcpu *vcpu, return; g_context->cpu_role.as_u64 = new_mode.as_u64; - g_context->get_guest_pgd = get_cr3; + g_context->get_guest_pgd = get_guest_cr3; g_context->get_pdptr = kvm_pdptr_read; g_context->inject_page_fault = kvm_inject_page_fault; diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 57f0b75c80f9..2ea2861bbb3c 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -324,7 +324,7 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, trace_kvm_mmu_pagetable_walk(addr, access); retry_walk: walker->level = mmu->cpu_role.base.level; - pte = mmu->get_guest_pgd(vcpu); + pte = kvm_mmu_get_guest_pgd(vcpu, mmu); have_ad = PT_HAVE_ACCESSED_DIRTY(mmu); #if PTTYPE == 64 From patchwork Fri May 12 13:20:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathias Krause X-Patchwork-Id: 13239291 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96901C77B75 for ; Fri, 12 May 2023 13:20:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241189AbjELNUs (ORCPT ); Fri, 12 May 2023 09:20:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42514 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241020AbjELNUn (ORCPT ); Fri, 12 May 2023 09:20:43 -0400 Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 011DA19A3 for ; Fri, 12 May 2023 06:20:41 -0700 (PDT) Received: by mail-ed1-x534.google.com with SMTP id 4fb4d7f45d1cf-50bc456cc39so14751011a12.1 for ; Fri, 12 May 2023 06:20:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grsecurity.net; s=grsec; t=1683897640; x=1686489640; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cNBZZe+hK/JPzfkEifbI0WOzbwUyqQzgKe8Z+nBTG/U=; b=N3T2E0clk/GWt9lw7qt3VgAO674vhkBlydpGiqPAVahLG3lfLWzYHcEYZdCFFgrM05 m4UK97gMqwwhN/+ObCrhEEvtvteqQYHpIg89c+f42c7ZW0W92oX7k4c/M7y/lPh/cw1a zbZmmG5yRWlJtGm91aVVQNK0AKutUHWvXFBu3jiLyQSHNFSl4QLiEGsbQHAv/kPxN7to KkpkohJgwZWJhBflVIZ8pn3M4k0NxlaVfA/opPIHNVYUdO3ahSFAFUlQeyUFNbwYftdy GV6Vc3cms+d57UeXeWMNxSqTDxqIfO87sZJEBk5ZTMtmvjqGN5/E92iX0QAH3rJ8yFaN 656A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683897640; x=1686489640; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cNBZZe+hK/JPzfkEifbI0WOzbwUyqQzgKe8Z+nBTG/U=; b=EpWp6CZjQ37rPwthNNM8PfLRo2FgS6jy96EcapgRBAvwPxA5fXBrM1OaNepLzCdXVB 022QroE3L7yvnSPZJnSxIgmfUQP2TmprWd5Lk7XUh+8jDgFKztbWqgumBHyKx1lgxqX8 E1T9pC9mF4bYi1ANE4Fk2uhUqJ/2Hnz2JuchGxuvqHQsZUgXPu1gk1z2ezM9VqLiaTRV POmjuA88Ggug/lPnhuRXuZsacAGyJmLt+RX604mWl+D2VdbUeHIonWXSQzZJMdmRYb69 wvmSI9yPOrM/o86zmdE3aICxnC/EvCvkH1ueJLwj7W6sOaXuSHjlUqzHkTBGiuNvIaCX ng4w== X-Gm-Message-State: AC+VfDyl7RZmhvcQW1SHeyBHrrnGKpaJNBxdmYDuPlSy3lDaxAJ/HAyz Srde4xCBtC6zjFNs6+SKusRhjg== X-Google-Smtp-Source: ACHHUZ4L0+19Klo2sUHt/UWdTBVCXPS1HNY3WJzIoK5W098pPdD3GcXlcFF+LLBnBfdSn45/GiCseQ== X-Received: by 2002:a17:907:9445:b0:96a:3e7:b588 with SMTP id dl5-20020a170907944500b0096a03e7b588mr10169506ejc.40.1683897640513; Fri, 12 May 2023 06:20:40 -0700 (PDT) Received: from nuc.fritz.box (p200300f6af43a100a78da3f586d44204.dip0.t-ipconnect.de. [2003:f6:af43:a100:a78d:a3f5:86d4:4204]) by smtp.gmail.com with ESMTPSA id w21-20020a170907271500b00969dfd160aesm5077981ejk.109.2023.05.12.06.20.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 12 May 2023 06:20:40 -0700 (PDT) From: Mathias Krause To: stable@vger.kernel.org Cc: Paolo Bonzini , Sean Christopherson , kvm@vger.kernel.org, Mathias Krause Subject: [PATCH 6.3 2/5] KVM: x86: Do not unload MMU roots when only toggling CR0.WP with TDP enabled Date: Fri, 12 May 2023 15:20:21 +0200 Message-Id: <20230512132024.4029-3-minipli@grsecurity.net> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230512132024.4029-1-minipli@grsecurity.net> References: <20230512132024.4029-1-minipli@grsecurity.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org [ Upstream commit 01b31714bd90be2784f7145bf93b7f78f3d081e1 ] There is no need to unload the MMU roots with TDP enabled when only CR0.WP has changed -- the paging structures are still valid, only the permission bitmap needs to be updated. One heavy user of toggling CR0.WP is grsecurity's KERNEXEC feature to implement kernel W^X. The optimization brings a huge performance gain for this case as the following micro-benchmark running 'ssdd 10 50000' from rt-tests[1] on a grsecurity L1 VM shows (runtime in seconds, lower is better): legacy TDP shadow kvm-x86/next@d8708b 8.43s 9.45s 70.3s +patch 5.39s 5.63s 70.2s For legacy MMU this is ~36% faster, for TDP MMU even ~40% faster. Also TDP and legacy MMU now both have a similar runtime which vanishes the need to disable TDP MMU for grsecurity. Shadow MMU sees no measurable difference and is still slow, as expected. [1] https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git Signed-off-by: Mathias Krause Link: https://lore.kernel.org/r/20230322013731.102955-3-minipli@grsecurity.net Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson Signed-off-by: Mathias Krause --- arch/x86/kvm/x86.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 3d852ce84920..999b2db0737b 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -906,6 +906,18 @@ EXPORT_SYMBOL_GPL(load_pdptrs); void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned long cr0) { + /* + * CR0.WP is incorporated into the MMU role, but only for non-nested, + * indirect shadow MMUs. If TDP is enabled, the MMU's metadata needs + * to be updated, e.g. so that emulating guest translations does the + * right thing, but there's no need to unload the root as CR0.WP + * doesn't affect SPTEs. + */ + if (tdp_enabled && (cr0 ^ old_cr0) == X86_CR0_WP) { + kvm_init_mmu(vcpu); + return; + } + if ((cr0 ^ old_cr0) & X86_CR0_PG) { kvm_clear_async_pf_completion_queue(vcpu); kvm_async_pf_hash_reset(vcpu); From patchwork Fri May 12 13:20:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathias Krause X-Patchwork-Id: 13239292 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F16FEC77B7F for ; Fri, 12 May 2023 13:20:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241209AbjELNUv (ORCPT ); Fri, 12 May 2023 09:20:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42514 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241136AbjELNUo (ORCPT ); Fri, 12 May 2023 09:20:44 -0400 Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E6A0F2688 for ; Fri, 12 May 2023 06:20:42 -0700 (PDT) Received: by mail-ej1-x632.google.com with SMTP id a640c23a62f3a-96ab81aa68dso76931866b.3 for ; Fri, 12 May 2023 06:20:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grsecurity.net; s=grsec; t=1683897641; x=1686489641; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=h/YBcFb+HEFX3gUhNTd2HIQMqv6srk/p3ypH72XDPVY=; b=TTaFDQvE6IP/SNx7SbFy4UfUD5zFQ5hM80OiHz/uAkx0MpodgOT1cyPEwuGZwODLwN q9PieYbGV8GaKbmSq3Lz7bCzwgyIgEHdwWIYH9ebyiG+7PCm16o7szMX2bdXFGLOVVuS /YF2m4Fn4bRau5UBguTFzoF5wIgec54ICe6TyDFAqrfekNCr1xJxCWyRqC+4+h3fE1zW Z+NUPIHJugv5EyzkrX0mObxE06lsLRPXdzAXYym0O9p4Tm4QscPHWXfvwtUmzCBgjFoY YCs93AnMQ6Ei9HuxIlz6eiddqK2Q8vXi/Kp8XHpD3nY1JzmCQk9rs4JjESCwiJ9yudum R8Cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683897641; x=1686489641; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=h/YBcFb+HEFX3gUhNTd2HIQMqv6srk/p3ypH72XDPVY=; b=Ph/WARqXH2Rw/DDHFII00j4dYn7cHwHobq0tFqgkGAn5sJW0lFboJZWEqAZcbwqxsv 2NjqERfzdUQbFZk/J+6tG8WS2TyyH39UIQY6MjykVjuNB3ILZl/jwzdqiOun4IfwSZJg qFSIwPN6yN5v/WOa0/dQy0OtWMnpW0D9BnzJoYaTXZLFvGqYXrW2AbX1XYJk8sJtwko0 cqdXx7+6pw070k/WpMp6UM/om+xHhO0q2x/Rg7GVudK3HJS+05W8YAWgFTilvE7b/57f nxOQ183JB5LK5EYg5mv6+dNdcDtmMgjn7gKOm1WzUQj2tWaSdn+aCwg41GpwaHgcvo/h fzMQ== X-Gm-Message-State: AC+VfDwJzfdv996JtpZWEFT1D9TdEdOpQxC6mFw7RMY6YSMYsZaJheni RLniN1vPAXL8CUpc7eOM+TFh6/tHajQvvSJRoIw= X-Google-Smtp-Source: ACHHUZ4vwgdN5BHzsHuPe0ttCXPWQ+6kaNHYiuz85o8K9kEmDVQCguagzO0uOE8FTIlBoBe53kctnw== X-Received: by 2002:a17:906:ee87:b0:965:6c67:11a1 with SMTP id wt7-20020a170906ee8700b009656c6711a1mr21252578ejb.21.1683897641439; Fri, 12 May 2023 06:20:41 -0700 (PDT) Received: from nuc.fritz.box (p200300f6af43a100a78da3f586d44204.dip0.t-ipconnect.de. [2003:f6:af43:a100:a78d:a3f5:86d4:4204]) by smtp.gmail.com with ESMTPSA id w21-20020a170907271500b00969dfd160aesm5077981ejk.109.2023.05.12.06.20.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 12 May 2023 06:20:41 -0700 (PDT) From: Mathias Krause To: stable@vger.kernel.org Cc: Paolo Bonzini , Sean Christopherson , kvm@vger.kernel.org, Mathias Krause Subject: [PATCH 6.3 3/5] KVM: x86: Make use of kvm_read_cr*_bits() when testing bits Date: Fri, 12 May 2023 15:20:22 +0200 Message-Id: <20230512132024.4029-4-minipli@grsecurity.net> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230512132024.4029-1-minipli@grsecurity.net> References: <20230512132024.4029-1-minipli@grsecurity.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org [ Upstream commit 74cdc836919bf34684ef66f995273f35e2189daf ] Make use of the kvm_read_cr{0,4}_bits() helper functions when we only want to know the state of certain bits instead of the whole register. This not only makes the intent cleaner, it also avoids a potential VMREAD in case the tested bits aren't guest owned. Signed-off-by: Mathias Krause Link: https://lore.kernel.org/r/20230322013731.102955-5-minipli@grsecurity.net Signed-off-by: Sean Christopherson Signed-off-by: Mathias Krause --- arch/x86/kvm/pmu.c | 4 ++-- arch/x86/kvm/vmx/vmx.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c index 612e6c70ce2e..f4aa170b5b97 100644 --- a/arch/x86/kvm/pmu.c +++ b/arch/x86/kvm/pmu.c @@ -540,9 +540,9 @@ int kvm_pmu_rdpmc(struct kvm_vcpu *vcpu, unsigned idx, u64 *data) if (!pmc) return 1; - if (!(kvm_read_cr4(vcpu) & X86_CR4_PCE) && + if (!(kvm_read_cr4_bits(vcpu, X86_CR4_PCE)) && (static_call(kvm_x86_get_cpl)(vcpu) != 0) && - (kvm_read_cr0(vcpu) & X86_CR0_PE)) + (kvm_read_cr0_bits(vcpu, X86_CR0_PE))) return 1; *data = pmc_read_counter(pmc) & mask; diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index dd92361f41b3..64b35223dc3d 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -5500,7 +5500,7 @@ static int handle_cr(struct kvm_vcpu *vcpu) break; case 3: /* lmsw */ val = (exit_qualification >> LMSW_SOURCE_DATA_SHIFT) & 0x0f; - trace_kvm_cr_write(0, (kvm_read_cr0(vcpu) & ~0xful) | val); + trace_kvm_cr_write(0, (kvm_read_cr0_bits(vcpu, ~0xful) | val)); kvm_lmsw(vcpu, val); return kvm_skip_emulated_instruction(vcpu); @@ -7558,7 +7558,7 @@ static u8 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio) if (!kvm_arch_has_noncoherent_dma(vcpu->kvm)) return (MTRR_TYPE_WRBACK << VMX_EPT_MT_EPTE_SHIFT) | VMX_EPT_IPAT_BIT; - if (kvm_read_cr0(vcpu) & X86_CR0_CD) { + if (kvm_read_cr0_bits(vcpu, X86_CR0_CD)) { if (kvm_check_has_quirk(vcpu->kvm, KVM_X86_QUIRK_CD_NW_CLEARED)) cache = MTRR_TYPE_WRBACK; else From patchwork Fri May 12 13:20:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathias Krause X-Patchwork-Id: 13239293 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D7E6C7EE26 for ; Fri, 12 May 2023 13:20:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241157AbjELNUx (ORCPT ); Fri, 12 May 2023 09:20:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42514 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240932AbjELNUp (ORCPT ); Fri, 12 May 2023 09:20:45 -0400 Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [IPv6:2a00:1450:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BC7ED173C for ; Fri, 12 May 2023 06:20:43 -0700 (PDT) Received: by mail-ej1-x629.google.com with SMTP id a640c23a62f3a-965d2749e2eso1465342466b.1 for ; Fri, 12 May 2023 06:20:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grsecurity.net; s=grsec; t=1683897642; x=1686489642; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vK90iJTB08dda6fvu5wKiqOE1X1cb8W54H9qEv6EPKY=; b=Xfq30YLnwkK+8xyzgFc1GGGshtpPW+TqCp8LPV4ORZloHX3KVNZHJZw1YzZPFCx6+/ qvMLTQrDZ90eTYkQ1Fg/SNmNO/y0bABn55ND5hr7lWFnY9s8emjgye/kzay83rlo3l6Z 4X818wboYmYsKdKtBgRYhRkshPLieHzRUjg9ze07jrTlp2AWyluLS7aeaZTCQwRWQu9X VoDLc+BZhuvIRg5cOUFIQ4Gt1l6k7Iobmhgoh9kwIIzlt5F1wFV+GL56uzEf3jfeP5Su likeGjbb+k2oBdjYSfag0JaHfX79W+YuzR7W9bmYZcFql2xsxQZux0Gmel5maNk2R3kY dZ8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683897642; x=1686489642; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vK90iJTB08dda6fvu5wKiqOE1X1cb8W54H9qEv6EPKY=; b=A0yG387Cu9ZHXdURhewnuO+GKBmFl3USZ9+Op73cZUGZQ8+bleQboumOvJeb6wZ2qN epJ1rwqemMdA33S2WYna7Vcw7BEfwrOgdVUlXm6JaUAzN6SnyyvAG6v+GYcMVueP61Eq 7oVGtWpspdpfSiQCfFlJ9Zfon3KWHjfZLfBP5cNUPxxR12SJqbBhDsX6H4NS5J1cVkh6 uU2vkhgqFUod6cjFFH4Zx+ppajmwRTodPkGBo+XFGyPjXjosnReAyutUL7vXnO5vFLem eWRTzPCbHhokhn076CdcC+hhkUNI4SXLqlRhDD0cZdTsFQxDQxAGHTg17/RR5z8fQp4H Ri8A== X-Gm-Message-State: AC+VfDyeMaPFkYPkfrWZiU/u50G/UVZ2fdKGb3VnBiLad5QYXXFDfQn4 iNyQqOkb/Srnhv35/9KlJ0VzJg== X-Google-Smtp-Source: ACHHUZ592gRRCbeuJBz2vuRGwiSMvO298YN/KLp4NL0URV+2ijG6wr2nijEFoXv29j3rFBZDLqUHlQ== X-Received: by 2002:a17:907:3faa:b0:966:17b2:5b15 with SMTP id hr42-20020a1709073faa00b0096617b25b15mr23183011ejc.7.1683897642260; Fri, 12 May 2023 06:20:42 -0700 (PDT) Received: from nuc.fritz.box (p200300f6af43a100a78da3f586d44204.dip0.t-ipconnect.de. [2003:f6:af43:a100:a78d:a3f5:86d4:4204]) by smtp.gmail.com with ESMTPSA id w21-20020a170907271500b00969dfd160aesm5077981ejk.109.2023.05.12.06.20.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 12 May 2023 06:20:41 -0700 (PDT) From: Mathias Krause To: stable@vger.kernel.org Cc: Paolo Bonzini , Sean Christopherson , kvm@vger.kernel.org, Mathias Krause Subject: [PATCH 6.3 4/5] KVM: VMX: Make CR0.WP a guest owned bit Date: Fri, 12 May 2023 15:20:23 +0200 Message-Id: <20230512132024.4029-5-minipli@grsecurity.net> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230512132024.4029-1-minipli@grsecurity.net> References: <20230512132024.4029-1-minipli@grsecurity.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org [ Upstream commit fb509f76acc8d42bed11bca308404f81c2be856a ] Guests like grsecurity that make heavy use of CR0.WP to implement kernel level W^X will suffer from the implied VMEXITs. With EPT there is no need to intercept a guest change of CR0.WP, so simply make it a guest owned bit if we can do so. This implies that a read of a guest's CR0.WP bit might need a VMREAD. However, the only potentially affected user seems to be kvm_init_mmu() which is a heavy operation to begin with. But also most callers already cache the full value of CR0 anyway, so no additional VMREAD is needed. The only exception is nested_vmx_load_cr3(). This change is VMX-specific, as SVM has no such fine grained control register intercept control. Suggested-by: Sean Christopherson Signed-off-by: Mathias Krause Link: https://lore.kernel.org/r/20230322013731.102955-7-minipli@grsecurity.net Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson Signed-off-by: Mathias Krause --- arch/x86/kvm/kvm_cache_regs.h | 2 +- arch/x86/kvm/vmx/nested.c | 4 ++-- arch/x86/kvm/vmx/vmx.c | 2 +- arch/x86/kvm/vmx/vmx.h | 18 ++++++++++++++++++ 4 files changed, 22 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/kvm_cache_regs.h b/arch/x86/kvm/kvm_cache_regs.h index 4c91f626c058..e50d353b5c1c 100644 --- a/arch/x86/kvm/kvm_cache_regs.h +++ b/arch/x86/kvm/kvm_cache_regs.h @@ -4,7 +4,7 @@ #include -#define KVM_POSSIBLE_CR0_GUEST_BITS X86_CR0_TS +#define KVM_POSSIBLE_CR0_GUEST_BITS (X86_CR0_TS | X86_CR0_WP) #define KVM_POSSIBLE_CR4_GUEST_BITS \ (X86_CR4_PVI | X86_CR4_DE | X86_CR4_PCE | X86_CR4_OSFXSR \ | X86_CR4_OSXMMEXCPT | X86_CR4_PGE | X86_CR4_TSD | X86_CR4_FSGSBASE) diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 768487611db7..89fa35fba3d8 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -4483,7 +4483,7 @@ static void load_vmcs12_host_state(struct kvm_vcpu *vcpu, * CR0_GUEST_HOST_MASK is already set in the original vmcs01 * (KVM doesn't change it); */ - vcpu->arch.cr0_guest_owned_bits = KVM_POSSIBLE_CR0_GUEST_BITS; + vcpu->arch.cr0_guest_owned_bits = vmx_l1_guest_owned_cr0_bits(); vmx_set_cr0(vcpu, vmcs12->host_cr0); /* Same as above - no reason to call set_cr4_guest_host_mask(). */ @@ -4634,7 +4634,7 @@ static void nested_vmx_restore_host_state(struct kvm_vcpu *vcpu) */ vmx_set_efer(vcpu, nested_vmx_get_vmcs01_guest_efer(vmx)); - vcpu->arch.cr0_guest_owned_bits = KVM_POSSIBLE_CR0_GUEST_BITS; + vcpu->arch.cr0_guest_owned_bits = vmx_l1_guest_owned_cr0_bits(); vmx_set_cr0(vcpu, vmcs_readl(CR0_READ_SHADOW)); vcpu->arch.cr4_guest_owned_bits = ~vmcs_readl(CR4_GUEST_HOST_MASK); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 64b35223dc3d..8ead0916e252 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -4773,7 +4773,7 @@ static void init_vmcs(struct vcpu_vmx *vmx) /* 22.2.1, 20.8.1 */ vm_entry_controls_set(vmx, vmx_vmentry_ctrl()); - vmx->vcpu.arch.cr0_guest_owned_bits = KVM_POSSIBLE_CR0_GUEST_BITS; + vmx->vcpu.arch.cr0_guest_owned_bits = vmx_l1_guest_owned_cr0_bits(); vmcs_writel(CR0_GUEST_HOST_MASK, ~vmx->vcpu.arch.cr0_guest_owned_bits); set_cr4_guest_host_mask(vmx); diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index 2acdc54bc34b..423e9d3c9c40 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -640,6 +640,24 @@ BUILD_CONTROLS_SHADOW(tertiary_exec, TERTIARY_VM_EXEC_CONTROL, 64) (1 << VCPU_EXREG_EXIT_INFO_1) | \ (1 << VCPU_EXREG_EXIT_INFO_2)) +static inline unsigned long vmx_l1_guest_owned_cr0_bits(void) +{ + unsigned long bits = KVM_POSSIBLE_CR0_GUEST_BITS; + + /* + * CR0.WP needs to be intercepted when KVM is shadowing legacy paging + * in order to construct shadow PTEs with the correct protections. + * Note! CR0.WP technically can be passed through to the guest if + * paging is disabled, but checking CR0.PG would generate a cyclical + * dependency of sorts due to forcing the caller to ensure CR0 holds + * the correct value prior to determining which CR0 bits can be owned + * by L1. Keep it simple and limit the optimization to EPT. + */ + if (!enable_ept) + bits &= ~X86_CR0_WP; + return bits; +} + static __always_inline struct kvm_vmx *to_kvm_vmx(struct kvm *kvm) { return container_of(kvm, struct kvm_vmx, kvm); From patchwork Fri May 12 13:20:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathias Krause X-Patchwork-Id: 13239294 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4339AC77B7F for ; Fri, 12 May 2023 13:20:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241226AbjELNUz (ORCPT ); Fri, 12 May 2023 09:20:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42500 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241180AbjELNUq (ORCPT ); Fri, 12 May 2023 09:20:46 -0400 Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 992C1273D for ; Fri, 12 May 2023 06:20:44 -0700 (PDT) Received: by mail-ej1-x631.google.com with SMTP id a640c23a62f3a-965e93f915aso1694915566b.2 for ; Fri, 12 May 2023 06:20:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grsecurity.net; s=grsec; t=1683897643; x=1686489643; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EAj9+wwvvQBZb98iVWDhS9msbB+DvyCsfBWvAkxMn64=; b=DPoOZm5Rl64bijgyVluptsPFEFFd2ZcMbhjgo9kJtSubQHcsnAmp6mg9cld22TEP/k 8gd0KwXkWIMvbu5SKWIzUOYciaRsxL+2TOIxqgAVw5OzjpBumNM0LKyA/Iz4/BgP9v5g cRamXKMEVGn7ScSVK2z3AzXDOogjtOJp6AYjoHcay1MzYt3b6/WVeisZdVVkTLzFxGwS KxgPjFvxXnljOSMUjf+7AGDzyk4kkZcm+NvQ43VY9/HnI8v43tCl/HyYtvAPanulIr/6 yBPrW9tqsRHxJ58CGj3Frvsl2QE2YdOj55MCzBO3V0TevenREPNAThyR9ihB35fnNBUM T6/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683897643; x=1686489643; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EAj9+wwvvQBZb98iVWDhS9msbB+DvyCsfBWvAkxMn64=; b=WLuErH4Woi2jLlNXKu7540njUUB7mZ/IhGDQDzeu96/FybiIb1ndeW6bt4D+IYrIs2 ob82DQQv/tfiwIIti3ClpPw6PrTpO/oIRNb/POIWfwxSaBFPZ3pDAifB/mvVc6eVtbR+ wEPIWfQYFeC/5NuixyQg/fcN0JJIKFWbiLyRjaxZ+vqBI53xeeHUWR1tQvjl3Y3w7wik pgVTm8rK65ZLxJqQMJVzKXMx70IQbxA0iSdFC+fTTn1PvGNWO4HXXY78U8LyIuQ74hS7 BJ7DztMKkBc05X8EwoxImglSCkdt76Z2Z7Mwfqfxt7/kYhw114Csh3JMFgXLgojE1Jzo 6nEg== X-Gm-Message-State: AC+VfDwQybs+YkoZO0aZr+OlXriNncBJa9ULxSQgAjnmnjcLGTzvv1SK 27QNKd6dNm9aRUrffzTlmgqovw== X-Google-Smtp-Source: ACHHUZ5Xdbv4TKYy6h0Rjoz+3eKhIg12ZBJAmVqbeZQ0LPWOkKCp5otUgJd7XKtatLLOz1txgOuE4A== X-Received: by 2002:a17:907:720e:b0:966:5a6c:752d with SMTP id dr14-20020a170907720e00b009665a6c752dmr16587933ejc.20.1683897643124; Fri, 12 May 2023 06:20:43 -0700 (PDT) Received: from nuc.fritz.box (p200300f6af43a100a78da3f586d44204.dip0.t-ipconnect.de. [2003:f6:af43:a100:a78d:a3f5:86d4:4204]) by smtp.gmail.com with ESMTPSA id w21-20020a170907271500b00969dfd160aesm5077981ejk.109.2023.05.12.06.20.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 12 May 2023 06:20:42 -0700 (PDT) From: Mathias Krause To: stable@vger.kernel.org Cc: Paolo Bonzini , Sean Christopherson , kvm@vger.kernel.org, Mathias Krause Subject: [PATCH 6.3 5/5] KVM: x86/mmu: Refresh CR0.WP prior to checking for emulated permission faults Date: Fri, 12 May 2023 15:20:24 +0200 Message-Id: <20230512132024.4029-6-minipli@grsecurity.net> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230512132024.4029-1-minipli@grsecurity.net> References: <20230512132024.4029-1-minipli@grsecurity.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Sean Christopherson [ Upstream commit cf9f4c0eb1699d306e348b1fd0225af7b2c282d3 ] Refresh the MMU's snapshot of the vCPU's CR0.WP prior to checking for permission faults when emulating a guest memory access and CR0.WP may be guest owned. If the guest toggles only CR0.WP and triggers emulation of a supervisor write, e.g. when KVM is emulating UMIP, KVM may consume a stale CR0.WP, i.e. use stale protection bits metadata. Note, KVM passes through CR0.WP if and only if EPT is enabled as CR0.WP is part of the MMU role for legacy shadow paging, and SVM (NPT) doesn't support per-bit interception controls for CR0. Don't bother checking for EPT vs. NPT as the "old == new" check will always be true under NPT, i.e. the only cost is the read of vcpu->arch.cr4 (SVM unconditionally grabs CR0 from the VMCB on VM-Exit). Reported-by: Mathias Krause Link: https://lkml.kernel.org/r/677169b4-051f-fcae-756b-9a3e1bb9f8fe%40grsecurity.net Fixes: fb509f76acc8 ("KVM: VMX: Make CR0.WP a guest owned bit") Tested-by: Mathias Krause Link: https://lore.kernel.org/r/20230405002608.418442-1-seanjc@google.com Signed-off-by: Sean Christopherson Signed-off-by: Mathias Krause # backport to v6.3.x --- - substitute lack of kvm_is_cr0_bit_set() with the older kvm_read_cr0_bits() arch/x86/kvm/mmu.h | 26 +++++++++++++++++++++++++- arch/x86/kvm/mmu/mmu.c | 15 +++++++++++++++ 2 files changed, 40 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index 168c46fd8dd1..0f38b78ab04b 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -113,6 +113,8 @@ void kvm_init_shadow_ept_mmu(struct kvm_vcpu *vcpu, bool execonly, bool kvm_can_do_async_pf(struct kvm_vcpu *vcpu); int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code, u64 fault_address, char *insn, int insn_len); +void __kvm_mmu_refresh_passthrough_bits(struct kvm_vcpu *vcpu, + struct kvm_mmu *mmu); int kvm_mmu_load(struct kvm_vcpu *vcpu); void kvm_mmu_unload(struct kvm_vcpu *vcpu); @@ -153,6 +155,24 @@ static inline void kvm_mmu_load_pgd(struct kvm_vcpu *vcpu) vcpu->arch.mmu->root_role.level); } +static inline void kvm_mmu_refresh_passthrough_bits(struct kvm_vcpu *vcpu, + struct kvm_mmu *mmu) +{ + /* + * When EPT is enabled, KVM may passthrough CR0.WP to the guest, i.e. + * @mmu's snapshot of CR0.WP and thus all related paging metadata may + * be stale. Refresh CR0.WP and the metadata on-demand when checking + * for permission faults. Exempt nested MMUs, i.e. MMUs for shadowing + * nEPT and nNPT, as CR0.WP is ignored in both cases. Note, KVM does + * need to refresh nested_mmu, a.k.a. the walker used to translate L2 + * GVAs to GPAs, as that "MMU" needs to honor L2's CR0.WP. + */ + if (!tdp_enabled || mmu == &vcpu->arch.guest_mmu) + return; + + __kvm_mmu_refresh_passthrough_bits(vcpu, mmu); +} + /* * Check if a given access (described through the I/D, W/R and U/S bits of a * page fault error code pfec) causes a permission fault with the given PTE @@ -184,8 +204,12 @@ static inline u8 permission_fault(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, u64 implicit_access = access & PFERR_IMPLICIT_ACCESS; bool not_smap = ((rflags & X86_EFLAGS_AC) | implicit_access) == X86_EFLAGS_AC; int index = (pfec + (not_smap << PFERR_RSVD_BIT)) >> 1; - bool fault = (mmu->permissions[index] >> pte_access) & 1; u32 errcode = PFERR_PRESENT_MASK; + bool fault; + + kvm_mmu_refresh_passthrough_bits(vcpu, mmu); + + fault = (mmu->permissions[index] >> pte_access) & 1; WARN_ON(pfec & (PFERR_PK_MASK | PFERR_RSVD_MASK)); if (unlikely(mmu->pkru_mask)) { diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 18c0deeaa2ec..d3812de54b02 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5121,6 +5121,21 @@ kvm_calc_cpu_role(struct kvm_vcpu *vcpu, const struct kvm_mmu_role_regs *regs) return role; } +void __kvm_mmu_refresh_passthrough_bits(struct kvm_vcpu *vcpu, + struct kvm_mmu *mmu) +{ + const bool cr0_wp = !!kvm_read_cr0_bits(vcpu, X86_CR0_WP); + + BUILD_BUG_ON((KVM_MMU_CR0_ROLE_BITS & KVM_POSSIBLE_CR0_GUEST_BITS) != X86_CR0_WP); + BUILD_BUG_ON((KVM_MMU_CR4_ROLE_BITS & KVM_POSSIBLE_CR4_GUEST_BITS)); + + if (is_cr0_wp(mmu) == cr0_wp) + return; + + mmu->cpu_role.base.cr0_wp = cr0_wp; + reset_guest_paging_metadata(vcpu, mmu); +} + static inline int kvm_mmu_get_tdp_level(struct kvm_vcpu *vcpu) { /* tdp_root_level is architecture forced level, use it if nonzero */