From patchwork Mon Feb 13 11:59:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Kai" X-Patchwork-Id: 13138314 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12B4FC636CC for ; Mon, 13 Feb 2023 12:01:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A4ABB6B007D; Mon, 13 Feb 2023 07:01:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D4BB6B0087; Mon, 13 Feb 2023 07:01:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 826F16B0089; Mon, 13 Feb 2023 07:01:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 70DBF6B0087 for ; Mon, 13 Feb 2023 07:01:51 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 37D0516164B for ; Mon, 13 Feb 2023 12:01:51 +0000 (UTC) X-FDA: 80462129622.22.CA75369 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf18.hostedemail.com (Postfix) with ESMTP id 9076C1C0028 for ; Mon, 13 Feb 2023 12:01:48 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=M3C1wsx5; spf=pass (imf18.hostedemail.com: domain of kai.huang@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=kai.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676289709; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8u9bePdMvnRhsuSfdz4Y/Sjlvhbyrm8i3jUTymnEv/E=; b=v1EbVtuFQNX5bUOGqoESMHUInYIIcBPjvMZl51KS0yd6WJDoOULoRcjxh2R60Dt+7FSbmi A9/T77dNP0e3Hkw36niqzQwaaco9+ZuRw9coYU+eXhvEUedA69hsUjwXfd0hM3rXL/Iapa BBojzYb19mulVvfT4G85vsNOn8TcW8c= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=M3C1wsx5; spf=pass (imf18.hostedemail.com: domain of kai.huang@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=kai.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676289709; a=rsa-sha256; cv=none; b=1kw6tAsiEvDac3gOE2eLtNabljTVGfDlu1tIkI4Z0LNPd4Jn63byBH9bmMHxlZmR2Y7N3J VeD40dywXIPZEBUtrqQez7vgPls/FF6tf4HpgD/4XJdxmmtTNWqJkjWYdNLzA3+8q/hhhF 2NLN7gVOT5B9/S8wHHXGjZQ4QCnZvY0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1676289708; x=1707825708; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=qRkFxe75ERnC5jK/b+M7BGDdGYgReE01fFl0GkUIxpw=; b=M3C1wsx5jJvtc9c6kvXXH4qZttt0HQ+KVJelPz4DWLTurwYIb2eRyuVK r2T1Atsnm+I296nsNuxvRbfQcmeODgFGL1pMwX3oxr/83zA97i7/j9pwZ 8tQcgY0CytDgF+B/QycxfW8M1d7FplN/ecjkilVQkuvIfx7ala0dil+1T BEdubLlp/zi5lmLxwNNQmfGffSKEx8hzngCssoGPnK04WrXqOS2AIpVfS 39BtgZiB2rFoCGhqpGdexSKohzLv/7MyRqdNXe3kOeH8pgb28ecryOgDU S0rv/Mrb4hqqoEqb1UKsFgR/qPbPGTtVg5iDEsV+IeKt+SM42END6KrEi w==; X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="358283555" X-IronPort-AV: E=Sophos;i="5.97,293,1669104000"; d="scan'208";a="358283555" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:01:48 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="701243571" X-IronPort-AV: E=Sophos;i="5.97,293,1669104000"; d="scan'208";a="701243571" Received: from wonger-mobl.amr.corp.intel.com (HELO khuang2-desk.gar.corp.intel.com) ([10.209.188.34]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:01:42 -0800 From: Kai Huang To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: linux-mm@kvack.org, dave.hansen@intel.com, peterz@infradead.org, tglx@linutronix.de, seanjc@google.com, pbonzini@redhat.com, dan.j.williams@intel.com, rafael.j.wysocki@intel.com, kirill.shutemov@linux.intel.com, ying.huang@intel.com, reinette.chatre@intel.com, len.brown@intel.com, tony.luck@intel.com, ak@linux.intel.com, isaku.yamahata@intel.com, chao.gao@intel.com, sathyanarayanan.kuppuswamy@linux.intel.com, david@redhat.com, bagasdotme@gmail.com, sagis@google.com, imammedo@redhat.com, kai.huang@intel.com Subject: [PATCH v9 18/18] Documentation/x86: Add documentation for TDX host support Date: Tue, 14 Feb 2023 00:59:25 +1300 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 9076C1C0028 X-Stat-Signature: f4w8kris9a7oxisz3omjm4bw5f1tbsk9 X-Rspam-User: X-HE-Tag: 1676289708-795578 X-HE-Meta: U2FsdGVkX19jcAtfTWgSe/P5XiZQvqbZZLzVxVlbOMMHUBeTuCNDHLUsdJ5htQd26vs+T9LXGFxBj067m5cJ56cUMFFLhaDSpkEJNPW4fbZoe1xMYzzK1c0SpcRxQUVCACfGxIJJ8boD3jro7MC8g9KH0xJwuDRnJvI9IGwyJN0HXdBlKLrUX9gFXv0vedrHVA+L6VRwb1l3BWKYbwcbFxVGjJdulHrpLnqYjGVmHELauyUTzJf7TCkfEEeJxAXp0NxL4QwiiBIAth2FfDG+sO/KyRstWEYIx4wvZYCyFYQ0Vc+bbLj0uga8PmaEtWJPqXdL5nQBO68aAJ9GqufaVVMOV/G33LndG75DaGWLrUwMDXFZkPpfAghsodl1d6rBT6ZG6HmiGIRRrPDWttke0kMzjj/1456TBbvnrsuIoYSqYLDnVeletoBEGSEFxzJ+OBR8vnbEAPMWpQ+NP12OUV69u6ztVt/HSzlHsQKLmGq9b+IP8BVR95SxrKSTHcpeHVOckVhKeOYLhlwZ5YAK8gOkhnJS9IFL+fe7qMz/yPJsz8x241BjKDvIf3M0DbCQw40hHWTQe7lNd8wIicRc/qKTMQf+ZK8PAOdq130V20512H7+9d1+aJJiKZivjWQeq++KgzQ448+zHCirMk9AKTHj9awjuJThmT12yD+8x7THtSl+yKdgWrhMm/3bzz5RLeRzDYJJeqZ5g7xOy85f1qiHXgmo4xxT85EX6aQ0QKT34vmmWEk3LMNGsfMj+jlt4F5fBKHlMc8dpS9Ka1gEQ2RVWLcxE5d9gMAva1w0+6QGGbC7ZfER1dpHqWUVfCp3lqni+ToFsaUXxIU1xy3HCe3DMPB18MLIxOqwJFgYmmbaEr0eMFBtate8qLsRdlxOGmhxqvYAJoHDTRSB/EnlvPpxjZwlLt+fwe7xbekZGg9Z3TleMsRcYT8sqsEXPYBKaGmcGfX7ZZGr13ncmr7 e5KexAMB EKIwYWvAR7Ce1nif625abK1IlGqjhyj5X4B0e5lQ/J1H1NpQR4oVJt2tnM6NylvrY1EuF3gO+YvR27s88YaiWeuR9mb0TDfIXcVcYotIwRyvusbl1OPGbDDs7IpTzp/0uc6T+MhhVghi4ocXoKinbi4AR5k3XX+RVoc4j+jllunXh+bU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add documentation for TDX host kernel support. There is already one file Documentation/x86/tdx.rst containing documentation for TDX guest internals. Also reuse it for TDX host kernel support. Introduce a new level menu "TDX Guest Support" and move existing materials under it, and add a new menu for TDX host kernel support. Signed-off-by: Kai Huang --- Documentation/x86/tdx.rst | 176 +++++++++++++++++++++++++++++++++++--- 1 file changed, 165 insertions(+), 11 deletions(-) diff --git a/Documentation/x86/tdx.rst b/Documentation/x86/tdx.rst index dc8d9fd2c3f7..8a84d7646bc3 100644 --- a/Documentation/x86/tdx.rst +++ b/Documentation/x86/tdx.rst @@ -10,6 +10,160 @@ encrypting the guest memory. In TDX, a special module running in a special mode sits between the host and the guest and manages the guest/host separation. +TDX Host Kernel Support +======================= + +TDX introduces a new CPU mode called Secure Arbitration Mode (SEAM) and +a new isolated range pointed by the SEAM Ranger Register (SEAMRR). A +CPU-attested software module called 'the TDX module' runs inside the new +isolated range to provide the functionalities to manage and run protected +VMs. + +TDX also leverages Intel Multi-Key Total Memory Encryption (MKTME) to +provide crypto-protection to the VMs. TDX reserves part of MKTME KeyIDs +as TDX private KeyIDs, which are only accessible within the SEAM mode. +BIOS is responsible for partitioning legacy MKTME KeyIDs and TDX KeyIDs. + +Before the TDX module can be used to create and run protected VMs, it +must be loaded into the isolated range and properly initialized. The TDX +architecture doesn't require the BIOS to load the TDX module, but the +kernel assumes it is loaded by the BIOS. + +TDX boot-time detection +----------------------- + +The kernel detects TDX by detecting TDX private KeyIDs during kernel +boot. Below dmesg shows when TDX is enabled by BIOS:: + + [..] tdx: BIOS enabled: private KeyID range: [16, 64). + +TDX module detection and initialization +--------------------------------------- + +There is no CPUID or MSR to detect the TDX module. The kernel detects it +by initializing it. + +The kernel talks to the TDX module via the new SEAMCALL instruction. The +TDX module implements SEAMCALL leaf functions to allow the kernel to +initialize it. + +Initializing the TDX module consumes roughly ~1/256th system RAM size to +use it as 'metadata' for the TDX memory. It also takes additional CPU +time to initialize those metadata along with the TDX module itself. Both +are not trivial. The kernel initializes the TDX module at runtime on +demand. The caller to call tdx_enable() to initialize the TDX module:: + + ret = tdx_enable(); + if (ret) + goto no_tdx; + // TDX is ready to use + +One step of initializing the TDX module requires at least one online cpu +for each package. The caller needs to guarantee this otherwise the +initialization will fail. + +Making SEAMCALL requires the CPU already being in VMX operation (VMXON +has been done). For now tdx_enable() doesn't handle VMXON internally, +but depends on the caller to guarantee that. So far only KVM calls +tdx_enable() and KVM already handles VMXON. + +User can consult dmesg to see the presence of the TDX module, and whether +it has been initialized. + +If the TDX module is not loaded, dmesg shows below:: + + [..] tdx: TDX module is not loaded. + +If the TDX module is initialized successfully, dmesg shows something +like below:: + + [..] tdx: TDX module: attributes 0x0, vendor_id 0x8086, major_version 1, minor_version 0, build_date 20211209, build_num 160 + [..] tdx: 262668 KBs allocated for PAMT. + [..] tdx: TDX module initialized. + +If the TDX module failed to initialize, dmesg also shows it failed to +initialize:: + + [..] tdx: initialization failed ... + +TDX Interaction to Other Kernel Components +------------------------------------------ + +TDX Memory Policy +~~~~~~~~~~~~~~~~~ + +TDX reports a list of "Convertible Memory Region" (CMR) to tell the +kernel which memory is TDX compatible. The kernel needs to build a list +of memory regions (out of CMRs) as "TDX-usable" memory and pass those +regions to the TDX module. Once this is done, those "TDX-usable" memory +regions are fixed during module's lifetime. + +To keep things simple, currently the kernel simply guarantees all pages +in the page allocator are TDX memory. Specifically, the kernel uses all +system memory in the core-mm at the time of initializing the TDX module +as TDX memory, and in the meantime, refuses to online any non-TDX-memory +in the memory hotplug. + +This can be enhanced in the future, i.e. by allowing adding non-TDX +memory to a separate NUMA node. In this case, the "TDX-capable" nodes +and the "non-TDX-capable" nodes can co-exist, but the kernel/userspace +needs to guarantee memory pages for TDX guests are always allocated from +the "TDX-capable" nodes. + +Physical Memory Hotplug +~~~~~~~~~~~~~~~~~~~~~~~ + +Note TDX assumes convertible memory is always physically present during +machine's runtime. A non-buggy BIOS should never support hot-removal of +any convertible memory. This implementation doesn't handle ACPI memory +removal but depends on the BIOS to behave correctly. + +CPU Hotplug +~~~~~~~~~~~ + +TDX module requires one SEAMCALL (TDH.SYS.LP.INIT) to do per-cpu module +initialization on one cpu before any other SEAMCALLs can be made on that +cpu, including those involved during the module initialization. + +Currently kernel simply guarantees all online cpus are "TDX-runnable" +(TDH.SYS.LP.INIT has been done successfully on them). During module +initialization, the SEAMCALL is done for all online cpus and CPU hotplug +is disabled during the entire module initialization. If any fails, TDX +is disabled. In CPU hotplug, the kernel provides another function +tdx_cpu_online() for the user of TDX (KVM for now) to call in it's own +CPU online callback, and reject to online the cpu if SEAMCALL fails. + +TDX doesn't support physical (ACPI) CPU hotplug. During machine boot, +TDX verifies all boot-time present logical CPUs are TDX compatible before +enabling TDX. A non-buggy BIOS should never support hot-add/removal of +physical CPU. Currently the kernel doesn't handle physical CPU hotplug, +but depends on the BIOS to behave correctly. + +Note TDX works with CPU logical online/offline, thus the kernel still +allows to offline logical CPU and online it again. + +Kexec() +~~~~~~~ + +There are two problems in terms of using kexec() to boot to a new kernel +when the old kernel has enabled TDX: 1) Part of the memory pages are +still TDX private pages; 2) There might be dirty cachelines associated +with TDX private pages. + +The first problem doesn't matter. KeyID 0 doesn't have integrity check. +Even the new kernel wants use any non-zero KeyID, it needs to convert +the memory to that KeyID and such conversion would work from any KeyID. + +However the old kernel needs to guarantee there's no dirty cacheline +left behind before booting to the new kernel to avoid silent corruption +from later cacheline writeback (Intel hardware doesn't guarantee cache +coherency across different KeyIDs). + +Similar to AMD SME, the kernel just uses wbinvd() to flush cache before +booting to the new kernel. + +TDX Guest Support +================= Since the host cannot directly access guest registers or memory, much normal functionality of a hypervisor must be moved into the guest. This is implemented using a Virtualization Exception (#VE) that is handled by the @@ -20,7 +174,7 @@ TDX includes new hypercall-like mechanisms for communicating from the guest to the hypervisor or the TDX module. New TDX Exceptions -================== +------------------ TDX guests behave differently from bare-metal and traditional VMX guests. In TDX guests, otherwise normal instructions or memory accesses can cause @@ -30,7 +184,7 @@ Instructions marked with an '*' conditionally cause exceptions. The details for these instructions are discussed below. Instruction-based #VE ---------------------- +~~~~~~~~~~~~~~~~~~~~~ - Port I/O (INS, OUTS, IN, OUT) - HLT @@ -41,7 +195,7 @@ Instruction-based #VE - CPUID* Instruction-based #GP ---------------------- +~~~~~~~~~~~~~~~~~~~~~ - All VMX instructions: INVEPT, INVVPID, VMCLEAR, VMFUNC, VMLAUNCH, VMPTRLD, VMPTRST, VMREAD, VMRESUME, VMWRITE, VMXOFF, VMXON @@ -52,7 +206,7 @@ Instruction-based #GP - RDMSR*,WRMSR* RDMSR/WRMSR Behavior --------------------- +~~~~~~~~~~~~~~~~~~~~ MSR access behavior falls into three categories: @@ -73,7 +227,7 @@ trapping and handling in the TDX module. Other than possibly being slow, these MSRs appear to function just as they would on bare metal. CPUID Behavior --------------- +~~~~~~~~~~~~~~ For some CPUID leaves and sub-leaves, the virtualized bit fields of CPUID return values (in guest EAX/EBX/ECX/EDX) are configurable by the @@ -93,7 +247,7 @@ not know how to handle. The guest kernel may ask the hypervisor for the value with a hypercall. #VE on Memory Accesses -====================== +---------------------- There are essentially two classes of TDX memory: private and shared. Private memory receives full TDX protections. Its content is protected @@ -107,7 +261,7 @@ entries. This helps ensure that a guest does not place sensitive information in shared memory, exposing it to the untrusted hypervisor. #VE on Shared Memory --------------------- +~~~~~~~~~~~~~~~~~~~~ Access to shared mappings can cause a #VE. The hypervisor ultimately controls whether a shared memory access causes a #VE, so the guest must be @@ -127,7 +281,7 @@ be careful not to access device MMIO regions unless it is also prepared to handle a #VE. #VE on Private Pages --------------------- +~~~~~~~~~~~~~~~~~~~~ An access to private mappings can also cause a #VE. Since all kernel memory is also private memory, the kernel might theoretically need to @@ -145,7 +299,7 @@ The hypervisor is permitted to unilaterally move accepted pages to a to handle the exception. Linux #VE handler -================= +----------------- Just like page faults or #GP's, #VE exceptions can be either handled or be fatal. Typically, an unhandled userspace #VE results in a SIGSEGV. @@ -167,7 +321,7 @@ While the block is in place, any #VE is elevated to a double fault (#DF) which is not recoverable. MMIO handling -============= +------------- In non-TDX VMs, MMIO is usually implemented by giving a guest access to a mapping which will cause a VMEXIT on access, and then the hypervisor @@ -189,7 +343,7 @@ MMIO access via other means (like structure overlays) may result in an oops. Shared Memory Conversions -========================= +------------------------- All TDX guest memory starts out as private at boot. This memory can not be accessed by the hypervisor. However, some kernel users like device