[19/89] KVM: arm64: Add pcpu fixmap infrastructure at EL2

Message ID	20220519134204.5379-20-will@kernel.org (mailing list archive)
State	New, archived
Headers	show Return-Path: <kvm-owner@kernel.org> From: Will Deacon <will@kernel.org> To: kvmarm@lists.cs.columbia.edu Cc: Will Deacon <will@kernel.org>, Ard Biesheuvel <ardb@kernel.org>, Sean Christopherson <seanjc@google.com>, Alexandru Elisei <alexandru.elisei@arm.com>, Andy Lutomirski <luto@amacapital.net>, Catalin Marinas <catalin.marinas@arm.com>, James Morse <james.morse@arm.com>, Chao Peng <chao.p.peng@linux.intel.com>, Quentin Perret <qperret@google.com>, Suzuki K Poulose <suzuki.poulose@arm.com>, Michael Roth <michael.roth@amd.com>, Mark Rutland <mark.rutland@arm.com>, Fuad Tabba <tabba@google.com>, Oliver Upton <oupton@google.com>, Marc Zyngier <maz@kernel.org>, kernel-team@android.com, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Subject: [PATCH 19/89] KVM: arm64: Add pcpu fixmap infrastructure at EL2 Date: Thu, 19 May 2022 14:40:54 +0100 Message-Id: <20220519134204.5379-20-will@kernel.org> In-Reply-To: <20220519134204.5379-1-will@kernel.org> References: <20220519134204.5379-1-will@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk
Series	KVM: arm64: Base support for the pKVM hypervisor at EL2 \| expand [00/89] KVM: arm64: Base support for the pKVM hypervisor at EL2 [01/89] KVM: arm64: Handle all ID registers trapped for a protected VM [02/89] KVM: arm64: Remove redundant hyp_assert_lock_held() assertions [03/89] KVM: arm64: Return error from kvm_arch_init_vm() on allocation failure [04/89] KVM: arm64: Ignore 'kvm-arm.mode=protected' when using VHE [05/89] KVM: arm64: Extend comment in has_vhe() [06/89] KVM: arm64: Drop stale comment [07/89] KVM: arm64: Move hyp refcount manipulation helpers [08/89] KVM: arm64: Back hyp_vmemmap for all of memory [09/89] KVM: arm64: Unify identifiers used to distinguish host and hypervisor [10/89] KVM: arm64: Implement do_donate() helper for donating memory [11/89] KVM: arm64: Prevent the donation of no-map pages [12/89] KVM: arm64: Add helpers to pin memory shared with hyp [13/89] KVM: arm64: Include asm/kvm_mmu.h in nvhe/mem_protect.h [14/89] KVM: arm64: Add hyp_spinlock_t static initializer [15/89] KVM: arm64: Introduce shadow VM state at EL2 [16/89] KVM: arm64: Instantiate VM shadow data from EL1 [17/89] KVM: arm64: Make hyp stage-1 refcnt correct on the whole range [18/89] KVM: arm64: Factor out private range VA allocation [19/89] KVM: arm64: Add pcpu fixmap infrastructure at EL2 [20/89] KVM: arm64: Provide I-cache invalidation by VA at EL2 [21/89] KVM: arm64: Allow non-coallescable pages in a hyp_pool [22/89] KVM: arm64: Add generic hyp_memcache helpers [23/89] KVM: arm64: Instantiate guest stage-2 page-tables at EL2 [24/89] KVM: arm64: Return guest memory from EL2 via dedicated teardown memcache [25/89] KVM: arm64: Add flags to struct hyp_page [26/89] KVM: arm64: Provide a hypercall for the host to reclaim guest memory [27/89] KVM: arm64: Extend memory sharing to allow host-to-guest transitions [28/89] KVM: arm64: Consolidate stage-2 init in one function [29/89] KVM: arm64: Check for PTE validity when checking for executable/cacheable [30/89] KVM: arm64: Do not allow memslot changes after first VM run under pKVM [31/89] KVM: arm64: Disallow dirty logging and RO memslots with pKVM [32/89] KVM: arm64: Use the shadow vCPU structure in handle___kvm_vcpu_run() [33/89] KVM: arm64: Handle guest stage-2 page-tables entirely at EL2 [34/89] KVM: arm64: Don't access kvm_arm_hyp_percpu_base at EL1 [35/89] KVM: arm64: Unmap kvm_arm_hyp_percpu_base from the host [36/89] KVM: arm64: Maintain a copy of 'kvm_arm_vmid_bits' at EL2 [37/89] KVM: arm64: Explicitly map kvm_vgic_global_state at EL2 [38/89] KVM: arm64: Don't map host sections in pkvm [39/89] KVM: arm64: Extend memory donation to allow host-to-guest transitions [40/89] KVM: arm64: Split up nvhe/fixed_config.h [41/89] KVM: arm64: Make vcpu_{read,write}_sys_reg available to HYP code [42/89] KVM: arm64: Simplify vgic-v3 hypercalls [43/89] KVM: arm64: Add the {flush,sync}_vgic_state() primitives [44/89] KVM: arm64: Introduce predicates to check for protected state [45/89] KVM: arm64: Add the {flush,sync}_timer_state() primitives [46/89] KVM: arm64: Introduce the pkvm_vcpu_{load,put} hypercalls [47/89] KVM: arm64: Add current vcpu and shadow_state lookup primitive [48/89] KVM: arm64: Skip __kvm_adjust_pc() for protected vcpus [49/89] KVM: arm64: Add hyp per_cpu variable to track current physical cpu number [50/89] KVM: arm64: Ensure that TLBs and I-cache are private to each vcpu [51/89] KVM: arm64: Introduce per-EC entry/exit handlers [52/89] KVM: arm64: Introduce lazy-ish state sync for non-protected VMs [53/89] KVM: arm64: Lazy host FP save/restore [54/89] KVM: arm64: Reduce host/shadow vcpu state copying [55/89] KVM: arm64: Do not pass the vcpu to __pkvm_host_map_guest() [56/89] KVM: arm64: Check directly whether the vcpu is protected [57/89] KVM: arm64: Trap debug break and watch from guest [58/89] KVM: arm64: Restrict protected VM capabilities [59/89] KVM: arm64: Do not support MTE for protected VMs [60/89] KVM: arm64: Refactor reset_mpidr to extract its computation [61/89] KVM: arm64: Reset sysregs for protected VMs [62/89] KVM: arm64: Move pkvm_vcpu_init_traps to shadow vcpu init [63/89] KVM: arm64: Fix initializing traps in protected mode [64/89] KVM: arm64: Advertise GICv3 sysreg interface to protected guests [65/89] KVM: arm64: Force injection of a data abort on NISV MMIO exit [66/89] KVM: arm64: Donate memory to protected guests [67/89] KVM: arm64: Add EL2 entry/exit handlers for pKVM guests [68/89] KVM: arm64: Move vgic state between host and shadow vcpu structures [69/89] KVM: arm64: Do not update virtual timer state for protected VMs [70/89] KVM: arm64: Refactor kvm_vcpu_enable_ptrauth() for hyp use [71/89] KVM: arm64: Initialize shadow vm state at hyp [72/89] KVM: arm64: Track the SVE state in the shadow vcpu [73/89] KVM: arm64: Add HVC handling for protected guests at EL2 [74/89] KVM: arm64: Move pstate reset values to kvm_arm.h [75/89] KVM: arm64: Move some kvm_psci functions to a shared header [76/89] KVM: arm64: Factor out vcpu_reset code for core registers and PSCI [77/89] KVM: arm64: Handle PSCI for protected VMs in EL2 [78/89] KVM: arm64: Don't expose TLBI hypercalls after de-privilege [79/89] KVM: arm64: Add is_pkvm_initialized() helper [80/89] KVM: arm64: Refactor enter_exception64() [81/89] KVM: arm64: Inject SIGSEGV on illegal accesses [82/89] KVM: arm64: Support TLB invalidation in guest context [83/89] KVM: arm64: Avoid BBM when changing only s/w bits in Stage-2 PTE [84/89] KVM: arm64: Extend memory sharing to allow guest-to-host transitions [85/89] KVM: arm64: Document the KVM/arm64-specific calls in hypercalls.rst [86/89] KVM: arm64: Reformat/beautify PTP hypercall documentation [87/89] KVM: arm64: Expose memory sharing hypercalls to protected guests [88/89] KVM: arm64: Introduce KVM_VM_TYPE_ARM_PROTECTED machine type for PVMs [89/89] Documentation: KVM: Add some documentation for Protected KVM on arm64

Message ID

20220519134204.5379-20-will@kernel.org (mailing list archive)

State

New, archived

Headers

From: Will Deacon <will@kernel.org>
To: kvmarm@lists.cs.columbia.edu
Cc: Will Deacon <will@kernel.org>, Ard Biesheuvel <ardb@kernel.org>,
        Sean Christopherson <seanjc@google.com>,
        Alexandru Elisei <alexandru.elisei@arm.com>,
        Andy Lutomirski <luto@amacapital.net>,
        Catalin Marinas <catalin.marinas@arm.com>,
        James Morse <james.morse@arm.com>,
        Chao Peng <chao.p.peng@linux.intel.com>,
        Quentin Perret <qperret@google.com>,
        Suzuki K Poulose <suzuki.poulose@arm.com>,
        Michael Roth <michael.roth@amd.com>,
        Mark Rutland <mark.rutland@arm.com>,
        Fuad Tabba <tabba@google.com>,
        Oliver Upton <oupton@google.com>,
        Marc Zyngier <maz@kernel.org>, kernel-team@android.com,
        kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org
Subject: [PATCH 19/89] KVM: arm64: Add pcpu fixmap infrastructure at EL2
Date: Thu, 19 May 2022 14:40:54 +0100
Message-Id: <20220519134204.5379-20-will@kernel.org>
In-Reply-To: <20220519134204.5379-1-will@kernel.org>
References: <20220519134204.5379-1-will@kernel.org>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk

Series

KVM: arm64: Base support for the pKVM hypervisor at EL2 | expand

Commit Message

Will Deacon May 19, 2022, 1:40 p.m. UTC

From: Quentin Perret <qperret@google.com>

We will soon need to temporarily map pages into the hypervisor stage-1
in nVHE protected mode. To do this efficiently, let's introduce a
per-cpu fixmap allowing to map a single page without needing to take any
lock or to allocate memory.

Signed-off-by: Quentin Perret <qperret@google.com>
---
 arch/arm64/kvm/hyp/include/nvhe/mem_protect.h |  2 +
 arch/arm64/kvm/hyp/include/nvhe/mm.h          |  4 +
 arch/arm64/kvm/hyp/nvhe/mem_protect.c         |  1 -
 arch/arm64/kvm/hyp/nvhe/mm.c                  | 85 +++++++++++++++++++
 arch/arm64/kvm/hyp/nvhe/setup.c               |  4 +
 5 files changed, 95 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h
index 3a0817b5c739..d11d9d68a680 100644
--- a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h
+++ b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h
@@ -59,6 +59,8 @@  enum pkvm_component_id {
 	PKVM_ID_HYP,
 };
 
+extern unsigned long hyp_nr_cpus;
+
 int __pkvm_prot_finalize(void);
 int __pkvm_host_share_hyp(u64 pfn);
 int __pkvm_host_unshare_hyp(u64 pfn);
diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h b/arch/arm64/kvm/hyp/include/nvhe/mm.h
index 73309ccc192e..45b04bfee171 100644
--- a/arch/arm64/kvm/hyp/include/nvhe/mm.h
+++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h
@@ -13,6 +13,10 @@ 
 extern struct kvm_pgtable pkvm_pgtable;
 extern hyp_spinlock_t pkvm_pgd_lock;
 
+int hyp_create_pcpu_fixmap(void);
+void *hyp_fixmap_map(phys_addr_t phys);
+int hyp_fixmap_unmap(void);
+
 int hyp_create_idmap(u32 hyp_va_bits);
 int hyp_map_vectors(void);
 int hyp_back_vmemmap(phys_addr_t back);
diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c
index 502bc0d04858..707bd832145f 100644
--- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c
+++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c
@@ -21,7 +21,6 @@ 
 
 #define KVM_HOST_S2_FLAGS (KVM_PGTABLE_S2_NOFWB | KVM_PGTABLE_S2_IDMAP)
 
-extern unsigned long hyp_nr_cpus;
 struct host_kvm host_kvm;
 
 static struct hyp_pool host_s2_pool;
diff --git a/arch/arm64/kvm/hyp/nvhe/mm.c b/arch/arm64/kvm/hyp/nvhe/mm.c
index 4377b067dc0e..bdb39897343f 100644
--- a/arch/arm64/kvm/hyp/nvhe/mm.c
+++ b/arch/arm64/kvm/hyp/nvhe/mm.c
@@ -14,6 +14,7 @@ 
 #include <nvhe/early_alloc.h>
 #include <nvhe/gfp.h>
 #include <nvhe/memory.h>
+#include <nvhe/mem_protect.h>
 #include <nvhe/mm.h>
 #include <nvhe/spinlock.h>
 
@@ -24,6 +25,7 @@  struct memblock_region hyp_memory[HYP_MEMBLOCK_REGIONS];
 unsigned int hyp_memblock_nr;
 
 static u64 __io_map_base;
+static DEFINE_PER_CPU(void *, hyp_fixmap_base);
 
 static int __pkvm_create_mappings(unsigned long start, unsigned long size,
 				  unsigned long phys, enum kvm_pgtable_prot prot)
@@ -198,6 +200,89 @@  int hyp_map_vectors(void)
 	return 0;
 }
 
+void *hyp_fixmap_map(phys_addr_t phys)
+{
+	void *addr = *this_cpu_ptr(&hyp_fixmap_base);
+	int ret = kvm_pgtable_hyp_map(&pkvm_pgtable, (u64)addr, PAGE_SIZE,
+				      phys, PAGE_HYP);
+	return ret ? NULL : addr;
+}
+
+int hyp_fixmap_unmap(void)
+{
+	void *addr = *this_cpu_ptr(&hyp_fixmap_base);
+	int ret = kvm_pgtable_hyp_unmap(&pkvm_pgtable, (u64)addr, PAGE_SIZE);
+
+	return (ret != PAGE_SIZE) ? -EINVAL : 0;
+}
+
+static int __pin_pgtable_cb(u64 addr, u64 end, u32 level, kvm_pte_t *ptep,
+			    enum kvm_pgtable_walk_flags flag, void * const arg)
+{
+	if (!kvm_pte_valid(*ptep) || level != KVM_PGTABLE_MAX_LEVELS - 1)
+		return -EINVAL;
+	hyp_page_ref_inc(hyp_virt_to_page(ptep));
+
+	return 0;
+}
+
+static int hyp_pin_pgtable_pages(u64 addr)
+{
+	struct kvm_pgtable_walker walker = {
+		.cb	= __pin_pgtable_cb,
+		.flags	= KVM_PGTABLE_WALK_LEAF,
+	};
+
+	return kvm_pgtable_walk(&pkvm_pgtable, addr, PAGE_SIZE, &walker);
+}
+
+int hyp_create_pcpu_fixmap(void)
+{
+	unsigned long i;
+	int ret = 0;
+	u64 addr;
+
+	hyp_spin_lock(&pkvm_pgd_lock);
+
+	for (i = 0; i < hyp_nr_cpus; i++) {
+		addr = hyp_alloc_private_va_range(PAGE_SIZE);
+		if (IS_ERR((void *)addr)) {
+			ret = -ENOMEM;
+			goto unlock;
+		}
+
+		/*
+		 * Create a dummy mapping, to get the intermediate page-table
+		 * pages allocated, then take a reference on the last level
+		 * page to keep it around at all times.
+		 */
+		ret = kvm_pgtable_hyp_map(&pkvm_pgtable, addr, PAGE_SIZE,
+					  __hyp_pa(__hyp_bss_start), PAGE_HYP);
+		if (ret) {
+			ret = -EINVAL;
+			goto unlock;
+		}
+
+		ret = hyp_pin_pgtable_pages(addr);
+		if (ret)
+			goto unlock;
+
+		ret = kvm_pgtable_hyp_unmap(&pkvm_pgtable, addr, PAGE_SIZE);
+		if (ret != PAGE_SIZE) {
+			ret = -EINVAL;
+			goto unlock;
+		} else {
+			ret = 0;
+		}
+
+		*per_cpu_ptr(&hyp_fixmap_base, i) = (void *)addr;
+	}
+unlock:
+	hyp_spin_unlock(&pkvm_pgd_lock);
+
+	return ret;
+}
+
 int hyp_create_idmap(u32 hyp_va_bits)
 {
 	unsigned long start, end;
diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c
index b306da2b5dae..59a478dde533 100644
--- a/arch/arm64/kvm/hyp/nvhe/setup.c
+++ b/arch/arm64/kvm/hyp/nvhe/setup.c
@@ -296,6 +296,10 @@  void __noreturn __pkvm_init_finalise(void)
 	if (ret)
 		goto out;
 
+	ret = hyp_create_pcpu_fixmap();
+	if (ret)
+		goto out;
+
 	hyp_shadow_table_init(shadow_table_base);
 out:
 	/*

[19/89] KVM: arm64: Add pcpu fixmap infrastructure at EL2

Commit Message

Patch