From patchwork Fri Apr 2 15:26:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12181187 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA9F2C43470 for ; Fri, 2 Apr 2021 15:27:04 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 74D23610E8 for ; Fri, 2 Apr 2021 15:27:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 74D23610E8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 644B26B0080; Fri, 2 Apr 2021 11:27:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 57E236B0081; Fri, 2 Apr 2021 11:27:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1851E6B0082; Fri, 2 Apr 2021 11:27:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0018.hostedemail.com [216.40.44.18]) by kanga.kvack.org (Postfix) with ESMTP id D47D66B007E for ; Fri, 2 Apr 2021 11:27:00 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 92C5B82F9B5F for ; Fri, 2 Apr 2021 15:27:00 +0000 (UTC) X-FDA: 77987805000.15.7B2B663 Received: from mail-lj1-f182.google.com (mail-lj1-f182.google.com [209.85.208.182]) by imf15.hostedemail.com (Postfix) with ESMTP id BC4A9A00038D for ; Fri, 2 Apr 2021 15:26:59 +0000 (UTC) Received: by mail-lj1-f182.google.com with SMTP id c6so4454092lji.8 for ; Fri, 02 Apr 2021 08:26:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov-name.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WavDtTX6TrwFdeBmXF60wrPd9BJChmTHA9LJTzuJkoU=; b=d48Z14vtQeTBnmY3KjiBOkvoSzWnEnzKL51lW9dQ/pvruzLz/a1kIWWaJwVfs6kDZs YGsq19NufLNS7oWgqkvDcXDssbIfw5QE2wgM6Z8Xtyebjke6D3u0lD07g7N5QFnGS4KY fVIcDYJaHOHIexJOKjE6YqJeJoksxdr+/9dYAc7bzG0GsfjphMTzDWIIgW2IKZzJpw6B OJY7QQsDq4EPZgDVbNJGDJQC2pBkR9bhlitGvowGSfd9mGiQzse9TYr6jA6U/8lA/HEt HILjBm4R6sWJxEx5Rmjyc0Y7wsAQRlTLqHAd0nesluCwm5hu1H1t/x1Qgxl1mWcxYvmQ F2mA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WavDtTX6TrwFdeBmXF60wrPd9BJChmTHA9LJTzuJkoU=; b=kPK/PRzNvhm+h8Gdg1Gs02O/5r9sk64zgbY5xre0nNQjPEnzCKJST7ovOzUB1e8C+R PP5ytleZDpv9hUiQAj1s/GcJX74GVyUPHz2VBDhbKr+0UKw+Rn4h+QFN6rCpFsEC+PpD 0BXWRHOuk01ajP+Sfm99jMAIdzDc4rEDLK2WEPeFSzc6Zs3dgi098OEmZvAEWWl9hMBP P4pNgBZbHCyfdJkCx2On4LsPnS6LuJsCY/WAvVMZ6PgchtMvtfDg7vPCN83BB+aV5iul Hw8Lh7iY+UgWqjBBp5gVqQ1kIxlEbgE4oQFGWpbknMISldLZfWNjG7J5zKrDBnoZb8mT uYQg== X-Gm-Message-State: AOAM533Kl8GSsQhWF6Tk6dZdEQRJAlPkCfT3nGq+Ipz7kelKSkEOxrar 9iBaaA2iRVKmV3i3chOXk/FI7Q== X-Google-Smtp-Source: ABdhPJyl7494iZYdytvBH3k7wfivOrumpFXeURKxUE8DWVLSKQoyXtMUpDn3OWXNrqBpjaPMpu1Dpg== X-Received: by 2002:a2e:509:: with SMTP id 9mr8710143ljf.170.1617377218622; Fri, 02 Apr 2021 08:26:58 -0700 (PDT) Received: from box.localdomain ([86.57.175.117]) by smtp.gmail.com with ESMTPSA id o26sm952073ljc.138.2021.04.02.08.26.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Apr 2021 08:26:57 -0700 (PDT) From: "Kirill A. Shutemov" X-Google-Original-From: "Kirill A. Shutemov" Received: by box.localdomain (Postfix, from userid 1000) id 246AB100A38; Fri, 2 Apr 2021 18:26:59 +0300 (+03) To: Dave Hansen , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Jim Mattson Cc: David Rientjes , "Edgecombe, Rick P" , "Kleen, Andi" , "Yamahata, Isaku" , x86@kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [RFCv1 1/7] x86/mm: Move force_dma_unencrypted() to common code Date: Fri, 2 Apr 2021 18:26:39 +0300 Message-Id: <20210402152645.26680-2-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210402152645.26680-1-kirill.shutemov@linux.intel.com> References: <20210402152645.26680-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BC4A9A00038D X-Stat-Signature: x98ekqto8uz58p3f5e9omsoaxqsp3yzt X-Rspamd-Server: rspam02 Received-SPF: none (shutemov.name>: No applicable sender policy available) receiver=imf15; identity=mailfrom; envelope-from=""; helo=mail-lj1-f182.google.com; client-ip=209.85.208.182 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617377219-57652 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: force_dma_unencrypted() has to return true for KVM guest with the memory protected enabled. Move it out of AMD SME code. Introduce new config option X86_MEM_ENCRYPT_COMMON that has to be selected by all x86 memory encryption features. This is preparation for the following patches. Signed-off-by: Kirill A. Shutemov --- arch/x86/Kconfig | 7 +++++- arch/x86/include/asm/io.h | 4 +++- arch/x86/mm/Makefile | 2 ++ arch/x86/mm/mem_encrypt.c | 30 ------------------------- arch/x86/mm/mem_encrypt_common.c | 38 ++++++++++++++++++++++++++++++++ 5 files changed, 49 insertions(+), 32 deletions(-) create mode 100644 arch/x86/mm/mem_encrypt_common.c diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 21f851179ff0..2b4ce1722dbd 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1520,14 +1520,19 @@ config X86_CPA_STATISTICS helps to determine the effectiveness of preserving large and huge page mappings when mapping protections are changed. +config X86_MEM_ENCRYPT_COMMON + select ARCH_HAS_FORCE_DMA_UNENCRYPTED + select DYNAMIC_PHYSICAL_MASK + def_bool n + config AMD_MEM_ENCRYPT bool "AMD Secure Memory Encryption (SME) support" depends on X86_64 && CPU_SUP_AMD select DMA_COHERENT_POOL - select DYNAMIC_PHYSICAL_MASK select ARCH_USE_MEMREMAP_PROT select ARCH_HAS_FORCE_DMA_UNENCRYPTED select INSTRUCTION_DECODER + select X86_MEM_ENCRYPT_COMMON help Say yes to enable support for the encryption of system memory. This requires an AMD processor that supports Secure Memory diff --git a/arch/x86/include/asm/io.h b/arch/x86/include/asm/io.h index d726459d08e5..6dc51b31cb0e 100644 --- a/arch/x86/include/asm/io.h +++ b/arch/x86/include/asm/io.h @@ -256,10 +256,12 @@ static inline void slow_down_io(void) #endif -#ifdef CONFIG_AMD_MEM_ENCRYPT #include extern struct static_key_false sev_enable_key; + +#ifdef CONFIG_AMD_MEM_ENCRYPT + static inline bool sev_key_active(void) { return static_branch_unlikely(&sev_enable_key); diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index 5864219221ca..b31cb52bf1bd 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -52,6 +52,8 @@ obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS) += pkeys.o obj-$(CONFIG_RANDOMIZE_MEMORY) += kaslr.o obj-$(CONFIG_PAGE_TABLE_ISOLATION) += pti.o +obj-$(CONFIG_X86_MEM_ENCRYPT_COMMON) += mem_encrypt_common.o + obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_identity.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_boot.o diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c index c3d5f0236f35..9ca477b9b8ba 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c @@ -15,10 +15,6 @@ #include #include #include -#include -#include -#include -#include #include #include @@ -390,32 +386,6 @@ bool noinstr sev_es_active(void) return sev_status & MSR_AMD64_SEV_ES_ENABLED; } -/* Override for DMA direct allocation check - ARCH_HAS_FORCE_DMA_UNENCRYPTED */ -bool force_dma_unencrypted(struct device *dev) -{ - /* - * For SEV, all DMA must be to unencrypted addresses. - */ - if (sev_active()) - return true; - - /* - * For SME, all DMA must be to unencrypted addresses if the - * device does not support DMA to addresses that include the - * encryption mask. - */ - if (sme_active()) { - u64 dma_enc_mask = DMA_BIT_MASK(__ffs64(sme_me_mask)); - u64 dma_dev_mask = min_not_zero(dev->coherent_dma_mask, - dev->bus_dma_limit); - - if (dma_dev_mask <= dma_enc_mask) - return true; - } - - return false; -} - void __init mem_encrypt_free_decrypted_mem(void) { unsigned long vaddr, vaddr_end, npages; diff --git a/arch/x86/mm/mem_encrypt_common.c b/arch/x86/mm/mem_encrypt_common.c new file mode 100644 index 000000000000..dd791352f73f --- /dev/null +++ b/arch/x86/mm/mem_encrypt_common.c @@ -0,0 +1,38 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * AMD Memory Encryption Support + * + * Copyright (C) 2016 Advanced Micro Devices, Inc. + * + * Author: Tom Lendacky + */ + +#include +#include +#include + +/* Override for DMA direct allocation check - ARCH_HAS_FORCE_DMA_UNENCRYPTED */ +bool force_dma_unencrypted(struct device *dev) +{ + /* + * For SEV, all DMA must be to unencrypted/shared addresses. + */ + if (sev_active()) + return true; + + /* + * For SME, all DMA must be to unencrypted addresses if the + * device does not support DMA to addresses that include the + * encryption mask. + */ + if (sme_active()) { + u64 dma_enc_mask = DMA_BIT_MASK(__ffs64(sme_me_mask)); + u64 dma_dev_mask = min_not_zero(dev->coherent_dma_mask, + dev->bus_dma_limit); + + if (dma_dev_mask <= dma_enc_mask) + return true; + } + + return false; +} From patchwork Fri Apr 2 15:26:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12181185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1468BC43461 for ; Fri, 2 Apr 2021 15:27:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9C6C261163 for ; Fri, 2 Apr 2021 15:27:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9C6C261163 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1D6D16B007D; Fri, 2 Apr 2021 11:27:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1AC376B0074; Fri, 2 Apr 2021 11:27:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F40E96B0081; Fri, 2 Apr 2021 11:27:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0040.hostedemail.com [216.40.44.40]) by kanga.kvack.org (Postfix) with ESMTP id D2AA86B0074 for ; Fri, 2 Apr 2021 11:27:00 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 8C28AA2AA for ; Fri, 2 Apr 2021 15:27:00 +0000 (UTC) X-FDA: 77987805000.26.3FAC79A Received: from mail-lf1-f46.google.com (mail-lf1-f46.google.com [209.85.167.46]) by imf04.hostedemail.com (Postfix) with ESMTP id 4126A3C3 for ; Fri, 2 Apr 2021 15:26:59 +0000 (UTC) Received: by mail-lf1-f46.google.com with SMTP id o10so8034301lfb.9 for ; Fri, 02 Apr 2021 08:26:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov-name.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=XV0Rr4q4z3pF6mu76yp48SNzLyhgb7peTnE0lBez02o=; b=EodbBshMCTRvKidcqtpUFzszeT8suAvHyZt41AkxUb25cC6X8OxPA4C8/OE6qI0Wjz tGJ7L/D4+6zUo57hATMXVzY4G4BtZTZtr3m9+BU5jCKirdrDSrgW2CSPuPh2020wSAna ggycoLovCsjjRgBc62sTAV+iYgDECMJIgU+ouZmYqeFDjU/XRt4Bejb4ue4PKz0GNzSW WIqrXtJa76zzFYufZBZYLnTt3DIhL/J42GpK3YM2KKeTpitbGRHQHMU8soEKWqBo5gnd e+XjG4bFYId8r3XpyapntpWDDeSM73zLUToBPPYtskMgWgutXsNbQFPPKzehRVJ/j51u wc5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XV0Rr4q4z3pF6mu76yp48SNzLyhgb7peTnE0lBez02o=; b=hzdw28MtjNSjB9k3itu1ok/fEdUQUnbHZYRlQ8mdjNUzEIDy+PSQ6Ab+P9ciHva+HH hKtW0+bQrsg69DBLZQ9MwcBsrsSr79aja2GiG56oRujw/aIN6Lek9VkXlJqRydBQLZT0 uCqj2+lo1ZtH8fpB8EchNQgeF7IHgOzSB6eCiCU4tbMZ6B0anLkAmFLlmMFDEnOi1Or4 NK55MhSR3ljnQUKB006wGJZ0W02hJI/4j3JhcW+7wY8paGYr0A+lrLkrIl21HMy4O5NQ h/xFvybGkygpqQ7LHj5/TTvGg+LEDfbcZEK4sNJhVqairggCg8Ai6EXVxe4AJbOzyvpg D3ww== X-Gm-Message-State: AOAM531mW45VPhyFjQnt92J/4U5Cac2iKjTjUIjQ3gdoCY/Mj2bbUBcA 6QsoBpOnywEb07O47c/1T2IY9Q== X-Google-Smtp-Source: ABdhPJwwhltQnBLsjblMC1m6HNOlB5i9WDQyil9E+Sy2m1NJ/ThtwWraH/J1f/w7kSgdubujWpX0Fw== X-Received: by 2002:a05:6512:118d:: with SMTP id g13mr9157808lfr.36.1617377218932; Fri, 02 Apr 2021 08:26:58 -0700 (PDT) Received: from box.localdomain ([86.57.175.117]) by smtp.gmail.com with ESMTPSA id d18sm957906ljo.51.2021.04.02.08.26.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Apr 2021 08:26:57 -0700 (PDT) From: "Kirill A. Shutemov" X-Google-Original-From: "Kirill A. Shutemov" Received: by box.localdomain (Postfix, from userid 1000) id 2D218102672; Fri, 2 Apr 2021 18:26:59 +0300 (+03) To: Dave Hansen , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Jim Mattson Cc: David Rientjes , "Edgecombe, Rick P" , "Kleen, Andi" , "Yamahata, Isaku" , x86@kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [RFCv1 2/7] x86/kvm: Introduce KVM memory protection feature Date: Fri, 2 Apr 2021 18:26:40 +0300 Message-Id: <20210402152645.26680-3-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210402152645.26680-1-kirill.shutemov@linux.intel.com> References: <20210402152645.26680-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4126A3C3 X-Stat-Signature: xweq6nk3xt4kbm7yghdakduj1piofaau X-Rspamd-Server: rspam02 Received-SPF: none (shutemov.name>: No applicable sender policy available) receiver=imf04; identity=mailfrom; envelope-from=""; helo=mail-lf1-f46.google.com; client-ip=209.85.167.46 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617377219-55795 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Provide basic helpers, KVM_FEATURE, CPUID flag and a hypercall. Host side doesn't provide the feature yet, so it is a dead code for now. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/kvm_para.h | 5 +++++ arch/x86/include/uapi/asm/kvm_para.h | 3 ++- arch/x86/kernel/kvm.c | 18 ++++++++++++++++++ include/uapi/linux/kvm_para.h | 3 ++- 5 files changed, 28 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 84b887825f12..5b6f23e6edc4 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -238,6 +238,7 @@ #define X86_FEATURE_VMW_VMMCALL ( 8*32+19) /* "" VMware prefers VMMCALL hypercall instruction */ #define X86_FEATURE_SEV_ES ( 8*32+20) /* AMD Secure Encrypted Virtualization - Encrypted State */ #define X86_FEATURE_VM_PAGE_FLUSH ( 8*32+21) /* "" VM Page Flush MSR is supported */ +#define X86_FEATURE_KVM_MEM_PROTECTED ( 8*32+22) /* KVM memory protection extenstion */ /* Intel-defined CPU features, CPUID level 0x00000007:0 (EBX), word 9 */ #define X86_FEATURE_FSGSBASE ( 9*32+ 0) /* RDFSBASE, WRFSBASE, RDGSBASE, WRGSBASE instructions*/ diff --git a/arch/x86/include/asm/kvm_para.h b/arch/x86/include/asm/kvm_para.h index 338119852512..74aea18f3130 100644 --- a/arch/x86/include/asm/kvm_para.h +++ b/arch/x86/include/asm/kvm_para.h @@ -11,11 +11,16 @@ extern void kvmclock_init(void); #ifdef CONFIG_KVM_GUEST bool kvm_check_and_clear_guest_paused(void); +bool kvm_mem_protected(void); #else static inline bool kvm_check_and_clear_guest_paused(void) { return false; } +static inline bool kvm_mem_protected(void) +{ + return false; +} #endif /* CONFIG_KVM_GUEST */ #define KVM_HYPERCALL \ diff --git a/arch/x86/include/uapi/asm/kvm_para.h b/arch/x86/include/uapi/asm/kvm_para.h index 950afebfba88..8d32c41861c9 100644 --- a/arch/x86/include/uapi/asm/kvm_para.h +++ b/arch/x86/include/uapi/asm/kvm_para.h @@ -28,11 +28,12 @@ #define KVM_FEATURE_PV_UNHALT 7 #define KVM_FEATURE_PV_TLB_FLUSH 9 #define KVM_FEATURE_ASYNC_PF_VMEXIT 10 -#define KVM_FEATURE_PV_SEND_IPI 11 +#define KVM_FEATURE_PV_SEND_IPI 11 #define KVM_FEATURE_POLL_CONTROL 12 #define KVM_FEATURE_PV_SCHED_YIELD 13 #define KVM_FEATURE_ASYNC_PF_INT 14 #define KVM_FEATURE_MSI_EXT_DEST_ID 15 +#define KVM_FEATURE_MEM_PROTECTED 16 #define KVM_HINTS_REALTIME 0 diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index 5e78e01ca3b4..e6989e1b74eb 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -39,6 +39,13 @@ #include #include +static bool mem_protected; + +bool kvm_mem_protected(void) +{ + return mem_protected; +} + DEFINE_STATIC_KEY_FALSE(kvm_async_pf_enabled); static int kvmapf = 1; @@ -749,6 +756,17 @@ static void __init kvm_init_platform(void) { kvmclock_init(); x86_platform.apic_post_init = kvm_apic_init; + + if (kvm_para_has_feature(KVM_FEATURE_MEM_PROTECTED)) { + if (kvm_hypercall0(KVM_HC_ENABLE_MEM_PROTECTED)) { + pr_err("Failed to enable KVM memory protection\n"); + return; + } + + pr_info("KVM memory protection enabled\n"); + mem_protected = true; + setup_force_cpu_cap(X86_FEATURE_KVM_MEM_PROTECTED); + } } #if defined(CONFIG_AMD_MEM_ENCRYPT) diff --git a/include/uapi/linux/kvm_para.h b/include/uapi/linux/kvm_para.h index 8b86609849b9..1a216f32e572 100644 --- a/include/uapi/linux/kvm_para.h +++ b/include/uapi/linux/kvm_para.h @@ -27,8 +27,9 @@ #define KVM_HC_MIPS_EXIT_VM 7 #define KVM_HC_MIPS_CONSOLE_OUTPUT 8 #define KVM_HC_CLOCK_PAIRING 9 -#define KVM_HC_SEND_IPI 10 +#define KVM_HC_SEND_IPI 10 #define KVM_HC_SCHED_YIELD 11 +#define KVM_HC_ENABLE_MEM_PROTECTED 12 /* * hypercalls use architecture specific From patchwork Fri Apr 2 15:26:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12181189 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3909C43462 for ; Fri, 2 Apr 2021 15:27:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 401D361166 for ; Fri, 2 Apr 2021 15:27:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 401D361166 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A14F06B007E; Fri, 2 Apr 2021 11:27:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9C53D8D0002; Fri, 2 Apr 2021 11:27:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7C7816B0083; Fri, 2 Apr 2021 11:27:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0034.hostedemail.com [216.40.44.34]) by kanga.kvack.org (Postfix) with ESMTP id 54E256B007E for ; Fri, 2 Apr 2021 11:27:01 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 139C3A2AA for ; Fri, 2 Apr 2021 15:27:01 +0000 (UTC) X-FDA: 77987805042.28.8DC1371 Received: from mail-lj1-f170.google.com (mail-lj1-f170.google.com [209.85.208.170]) by imf12.hostedemail.com (Postfix) with ESMTP id 65E6D132 for ; Fri, 2 Apr 2021 15:26:59 +0000 (UTC) Received: by mail-lj1-f170.google.com with SMTP id r20so5961280ljk.4 for ; Fri, 02 Apr 2021 08:27:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov-name.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=aho3YzbAYOK3xFgDwaA/KoZASR7zBJ1wJNmUpntSsWc=; b=2QzvDuEvCR3I2ZDGC2q/r4bNtqnH2e/nkaqSMvgaY8t1XXblCZNLVwizVwriqp4ctq CuEbDoBOax44+oLmnKITnvspc5PpazSknGo20E7xxQUgEL0BMY+U4SOPirIT02WkkJ/w wrAY+AmwUvUFXoqRvjBzonIuQt3gNbeytFCVwJuDxuQrO9BxYGmZTXg0Nt7gEsMJptNO Ivjl0iZK4X/qBiwWdxuaKlUU5GzHNIbUoaBRC7viRsnEds4bK5Selt6pbN7neSbHwSap g32WgF2ulA5zJiMifrWUkKbeNzEZUyS9ygY05adoG52zdthFiBJxjcrkFR5TiLVRXDKW 6UpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aho3YzbAYOK3xFgDwaA/KoZASR7zBJ1wJNmUpntSsWc=; b=nf1oydCzLSq9opH8gSDOC9u9kF5D8YNp8iLyfD3Ke13EmlWe5AMQexvb23a/+zJNfb kWunXRZRi95cA/RzfrxgJbdlySI22XmFWYlsZRjjEnFnGmxjqFbCUGV2HuzRejXu+8tB mRrf7u4dx0B+hK2l9QkQn/WmF4BGuZzURiXuQn/RT1OacTfAwMgTxLptb+NLu7AQ2W38 auBYJdN9rMZ9VEnj8Q5lKy1/+8+JXbsEB14eShWzRI1M78RcOK+Py4Jx74gJwSfIrCo8 nkDtlOzj3rtgmjrGyvynoZJ52xy/MsX2VJmRhfpsOPZYdA5f97ADXtQ4WJStWvzGjek5 z1vw== X-Gm-Message-State: AOAM530EE9d1iqcbtha2l6xOnAH0oW/37FpJivbF9VOlyTfKx3nLyLIb vuf6eWJhVZ73aauB/FMkrM6C9A== X-Google-Smtp-Source: ABdhPJxPXdliVU29L50xs/trh5bjPIkQyC1OfYRcYmB3fubXDmftI2jmDxrfsCutGi24mgL0K0++ww== X-Received: by 2002:a2e:9157:: with SMTP id q23mr8978027ljg.298.1617377219398; Fri, 02 Apr 2021 08:26:59 -0700 (PDT) Received: from box.localdomain ([86.57.175.117]) by smtp.gmail.com with ESMTPSA id v11sm947533ljp.63.2021.04.02.08.26.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Apr 2021 08:26:57 -0700 (PDT) From: "Kirill A. Shutemov" X-Google-Original-From: "Kirill A. Shutemov" Received: by box.localdomain (Postfix, from userid 1000) id 3598F102674; Fri, 2 Apr 2021 18:26:59 +0300 (+03) To: Dave Hansen , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Jim Mattson Cc: David Rientjes , "Edgecombe, Rick P" , "Kleen, Andi" , "Yamahata, Isaku" , x86@kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [RFCv1 3/7] x86/kvm: Make DMA pages shared Date: Fri, 2 Apr 2021 18:26:41 +0300 Message-Id: <20210402152645.26680-4-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210402152645.26680-1-kirill.shutemov@linux.intel.com> References: <20210402152645.26680-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 65E6D132 X-Stat-Signature: miym7g3pnfghegwcww5fpwdz8awpzg5z Received-SPF: none (shutemov.name>: No applicable sender policy available) receiver=imf12; identity=mailfrom; envelope-from=""; helo=mail-lj1-f170.google.com; client-ip=209.85.208.170 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617377219-166858 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make force_dma_unencrypted() return true for KVM to get DMA pages mapped as shared. __set_memory_enc_dec() now informs the host via hypercall if the state of the page has changed from shared to private or back. Signed-off-by: Kirill A. Shutemov --- arch/x86/Kconfig | 1 + arch/x86/mm/mem_encrypt_common.c | 5 +++-- arch/x86/mm/pat/set_memory.c | 10 ++++++++++ include/uapi/linux/kvm_para.h | 1 + 4 files changed, 15 insertions(+), 2 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 2b4ce1722dbd..d197b3beb904 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -811,6 +811,7 @@ config KVM_GUEST select PARAVIRT_CLOCK select ARCH_CPUIDLE_HALTPOLL select X86_HV_CALLBACK_VECTOR + select X86_MEM_ENCRYPT_COMMON default y help This option enables various optimizations for running under the KVM diff --git a/arch/x86/mm/mem_encrypt_common.c b/arch/x86/mm/mem_encrypt_common.c index dd791352f73f..6bf0718bb72a 100644 --- a/arch/x86/mm/mem_encrypt_common.c +++ b/arch/x86/mm/mem_encrypt_common.c @@ -10,14 +10,15 @@ #include #include #include +#include /* Override for DMA direct allocation check - ARCH_HAS_FORCE_DMA_UNENCRYPTED */ bool force_dma_unencrypted(struct device *dev) { /* - * For SEV, all DMA must be to unencrypted/shared addresses. + * For SEV and KVM, all DMA must be to unencrypted/shared addresses. */ - if (sev_active()) + if (sev_active() || kvm_mem_protected()) return true; /* diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index 16f878c26667..4b312d80033d 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include @@ -1977,6 +1978,15 @@ static int __set_memory_enc_dec(unsigned long addr, int numpages, bool enc) struct cpa_data cpa; int ret; + if (kvm_mem_protected()) { + /* TODO: Unsharing memory back */ + if (WARN_ON_ONCE(enc)) + return -ENOSYS; + + return kvm_hypercall2(KVM_HC_MEM_SHARE, + __pa(addr) >> PAGE_SHIFT, numpages); + } + /* Nothing to do if memory encryption is not active */ if (!mem_encrypt_active()) return 0; diff --git a/include/uapi/linux/kvm_para.h b/include/uapi/linux/kvm_para.h index 1a216f32e572..09d36683ee0a 100644 --- a/include/uapi/linux/kvm_para.h +++ b/include/uapi/linux/kvm_para.h @@ -30,6 +30,7 @@ #define KVM_HC_SEND_IPI 10 #define KVM_HC_SCHED_YIELD 11 #define KVM_HC_ENABLE_MEM_PROTECTED 12 +#define KVM_HC_MEM_SHARE 13 /* * hypercalls use architecture specific From patchwork Fri Apr 2 15:26:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12181191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 595BEC43460 for ; Fri, 2 Apr 2021 15:27:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E514261151 for ; Fri, 2 Apr 2021 15:27:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E514261151 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EF7BF6B0081; Fri, 2 Apr 2021 11:27:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ECD926B0082; Fri, 2 Apr 2021 11:27:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C5D8A6B0083; Fri, 2 Apr 2021 11:27:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0061.hostedemail.com [216.40.44.61]) by kanga.kvack.org (Postfix) with ESMTP id A71D96B0081 for ; Fri, 2 Apr 2021 11:27:01 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 6E6A5184875A3 for ; Fri, 2 Apr 2021 15:27:01 +0000 (UTC) X-FDA: 77987805042.12.7BC2659 Received: from mail-lj1-f177.google.com (mail-lj1-f177.google.com [209.85.208.177]) by imf08.hostedemail.com (Postfix) with ESMTP id 10FAF80192DA for ; Fri, 2 Apr 2021 15:26:57 +0000 (UTC) Received: by mail-lj1-f177.google.com with SMTP id r20so5961292ljk.4 for ; Fri, 02 Apr 2021 08:27:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov-name.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=l49+x9wha/OETvCMFi8m/d/uZbBJ4UwRqG9iOGtI1aM=; b=OuvK3SzQ6dQOhzxZUcyuQgdblcFLTBXZYao++de77CPnXGNcR10xHoWIXV714ZZa74 zm96h3Cz2GA40FA54aDesXzmTrnZJF1HLc9EAULShBa27yLlOgl9iReCZ0QEHO+reMoF B4+of1sK2aUj5FmsnomD7aj4PkP4jDjeSB78UIvj368hJY+40pj3xhR4saWRxBReRivX nfu+VrY64R4QZf+jRHnrEq/I26vU5cle4esowVbrFYusNbiM0Y52RUVpFIwbGPuoDs47 9Lmyb9ywzUINcVApb4zPN2tvnZpcGqEqv8BrJdW0vt1jTqKRXZvtOz7XLJ8Ph18W7ONK DcXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=l49+x9wha/OETvCMFi8m/d/uZbBJ4UwRqG9iOGtI1aM=; b=rLimx74gB9UVbRbn6HPHm/eaFz/sOuOJGyoaCINsB734F4sFanHp6RoZWomY+qJ6/B SWcAzereljh2iEqA8AvLMmiUWO2n6uo18g9MiD5jruGQ/N08fdQ1Ze+A3uogEj0b3f9w toMRXgziD1f1Tl7iuLJ8r82SOAjewoZageMBwLS/PZ65NqlKff+if7XVHenif6V5Qp6d VBfrBQlJ2i5JBb5psSeQkMRyAo1T1hPoAAeXR34JT2HDhxf/1J4KulUwINaiXBUUmgjW qE2jmbUc9CJIviI1Qm01yLuQ1hmpi21+0hxhQzb3AvXOgKhHaTXcLgSTrOSyG5Wzs0uz mU7A== X-Gm-Message-State: AOAM533jk3vUcrQunseui1QY/gdIWLqMnM/bRY7/2Th5ktFNnjg1+shc SptbmTUVrXG4qgBY+DdrNeV9tw== X-Google-Smtp-Source: ABdhPJzssft7IoVOjuBqHnu5o2WtjJ8q2hIrHHkfmVBUMr0CRXudxVHzj/x/Sx3degpbvNmcq7wWxw== X-Received: by 2002:a2e:8ed4:: with SMTP id e20mr8402684ljl.129.1617377219683; Fri, 02 Apr 2021 08:26:59 -0700 (PDT) Received: from box.localdomain ([86.57.175.117]) by smtp.gmail.com with ESMTPSA id c2sm891480lfc.221.2021.04.02.08.26.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Apr 2021 08:26:57 -0700 (PDT) From: "Kirill A. Shutemov" X-Google-Original-From: "Kirill A. Shutemov" Received: by box.localdomain (Postfix, from userid 1000) id 3E0D3102675; Fri, 2 Apr 2021 18:26:59 +0300 (+03) To: Dave Hansen , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Jim Mattson Cc: David Rientjes , "Edgecombe, Rick P" , "Kleen, Andi" , "Yamahata, Isaku" , x86@kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [RFCv1 4/7] x86/kvm: Use bounce buffers for KVM memory protection Date: Fri, 2 Apr 2021 18:26:42 +0300 Message-Id: <20210402152645.26680-5-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210402152645.26680-1-kirill.shutemov@linux.intel.com> References: <20210402152645.26680-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 10FAF80192DA X-Stat-Signature: e89qh4k1k79opdgft8a49k6ekr6jh45t Received-SPF: none (shutemov.name>: No applicable sender policy available) receiver=imf08; identity=mailfrom; envelope-from=""; helo=mail-lj1-f177.google.com; client-ip=209.85.208.177 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617377217-542673 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Mirror SEV, use SWIOTLB always if KVM memory protection is enabled. Signed-off-by: Kirill A. Shutemov --- arch/x86/Kconfig | 1 + arch/x86/include/asm/mem_encrypt.h | 7 +++-- arch/x86/kernel/kvm.c | 2 ++ arch/x86/kernel/pci-swiotlb.c | 3 +- arch/x86/mm/mem_encrypt.c | 44 --------------------------- arch/x86/mm/mem_encrypt_common.c | 48 ++++++++++++++++++++++++++++++ 6 files changed, 57 insertions(+), 48 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index d197b3beb904..c51d14db5620 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -812,6 +812,7 @@ config KVM_GUEST select ARCH_CPUIDLE_HALTPOLL select X86_HV_CALLBACK_VECTOR select X86_MEM_ENCRYPT_COMMON + select SWIOTLB default y help This option enables various optimizations for running under the KVM diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index 31c4df123aa0..a748b30c2f23 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -47,10 +47,8 @@ int __init early_set_memory_encrypted(unsigned long vaddr, unsigned long size); void __init mem_encrypt_free_decrypted_mem(void); -/* Architecture __weak replacement functions */ -void __init mem_encrypt_init(void); - void __init sev_es_init_vc_handling(void); + bool sme_active(void); bool sev_active(void); bool sev_es_active(void); @@ -91,6 +89,9 @@ static inline void mem_encrypt_free_decrypted_mem(void) { } #endif /* CONFIG_AMD_MEM_ENCRYPT */ +/* Architecture __weak replacement functions */ +void __init mem_encrypt_init(void); + /* * The __sme_pa() and __sme_pa_nodebug() macros are meant for use when * writing to or comparing values from the cr3 register. Having the diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index e6989e1b74eb..45aee29e4294 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include #include @@ -766,6 +767,7 @@ static void __init kvm_init_platform(void) pr_info("KVM memory protection enabled\n"); mem_protected = true; setup_force_cpu_cap(X86_FEATURE_KVM_MEM_PROTECTED); + swiotlb_force = SWIOTLB_FORCE; } } diff --git a/arch/x86/kernel/pci-swiotlb.c b/arch/x86/kernel/pci-swiotlb.c index c2cfa5e7c152..814060a6ceb0 100644 --- a/arch/x86/kernel/pci-swiotlb.c +++ b/arch/x86/kernel/pci-swiotlb.c @@ -13,6 +13,7 @@ #include #include #include +#include int swiotlb __read_mostly; @@ -49,7 +50,7 @@ int __init pci_swiotlb_detect_4gb(void) * buffers are allocated and used for devices that do not support * the addressing range required for the encryption mask. */ - if (sme_active()) + if (sme_active() || kvm_mem_protected()) swiotlb = 1; return swiotlb; diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c index 9ca477b9b8ba..3478f20fb46f 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c @@ -409,47 +409,3 @@ void __init mem_encrypt_free_decrypted_mem(void) free_init_pages("unused decrypted", vaddr, vaddr_end); } - -static void print_mem_encrypt_feature_info(void) -{ - pr_info("AMD Memory Encryption Features active:"); - - /* Secure Memory Encryption */ - if (sme_active()) { - /* - * SME is mutually exclusive with any of the SEV - * features below. - */ - pr_cont(" SME\n"); - return; - } - - /* Secure Encrypted Virtualization */ - if (sev_active()) - pr_cont(" SEV"); - - /* Encrypted Register State */ - if (sev_es_active()) - pr_cont(" SEV-ES"); - - pr_cont("\n"); -} - -/* Architecture __weak replacement functions */ -void __init mem_encrypt_init(void) -{ - if (!sme_me_mask) - return; - - /* Call into SWIOTLB to update the SWIOTLB DMA buffers */ - swiotlb_update_mem_attributes(); - - /* - * With SEV, we need to unroll the rep string I/O instructions. - */ - if (sev_active()) - static_branch_enable(&sev_enable_key); - - print_mem_encrypt_feature_info(); -} - diff --git a/arch/x86/mm/mem_encrypt_common.c b/arch/x86/mm/mem_encrypt_common.c index 6bf0718bb72a..351b77361a5d 100644 --- a/arch/x86/mm/mem_encrypt_common.c +++ b/arch/x86/mm/mem_encrypt_common.c @@ -11,6 +11,7 @@ #include #include #include +#include /* Override for DMA direct allocation check - ARCH_HAS_FORCE_DMA_UNENCRYPTED */ bool force_dma_unencrypted(struct device *dev) @@ -37,3 +38,50 @@ bool force_dma_unencrypted(struct device *dev) return false; } + +static void print_mem_encrypt_feature_info(void) +{ + if (kvm_mem_protected()) { + pr_info("KVM memory protection enabled\n"); + return; + } + + pr_info("AMD Memory Encryption Features active:"); + + /* Secure Memory Encryption */ + if (sme_active()) { + /* + * SME is mutually exclusive with any of the SEV + * features below. + */ + pr_cont(" SME\n"); + return; + } + + /* Secure Encrypted Virtualization */ + if (sev_active()) + pr_cont(" SEV"); + + /* Encrypted Register State */ + if (sev_es_active()) + pr_cont(" SEV-ES"); + + pr_cont("\n"); +} + +void __init mem_encrypt_init(void) +{ + if (!sme_me_mask && !kvm_mem_protected()) + return; + + /* Call into SWIOTLB to update the SWIOTLB DMA buffers */ + swiotlb_update_mem_attributes(); + + /* + * With SEV, we need to unroll the rep string I/O instructions. + */ + if (sev_active()) + static_branch_enable(&sev_enable_key); + + print_mem_encrypt_feature_info(); +} From patchwork Fri Apr 2 15:26:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12181193 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25179C433B4 for ; Fri, 2 Apr 2021 15:27:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AA6D46115B for ; Fri, 2 Apr 2021 15:27:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AA6D46115B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 866F56B0082; Fri, 2 Apr 2021 11:27:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 83F126B0083; Fri, 2 Apr 2021 11:27:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6DE2B6B0085; Fri, 2 Apr 2021 11:27:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0048.hostedemail.com [216.40.44.48]) by kanga.kvack.org (Postfix) with ESMTP id 4E17F6B0082 for ; Fri, 2 Apr 2021 11:27:02 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id F4106184875AB for ; Fri, 2 Apr 2021 15:27:01 +0000 (UTC) X-FDA: 77987805084.20.B7635C8 Received: from mail-lf1-f47.google.com (mail-lf1-f47.google.com [209.85.167.47]) by imf05.hostedemail.com (Postfix) with ESMTP id 410C3E00010F for ; Fri, 2 Apr 2021 15:27:01 +0000 (UTC) Received: by mail-lf1-f47.google.com with SMTP id o126so8115986lfa.0 for ; Fri, 02 Apr 2021 08:27:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov-name.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=V63UJPQR8b6EdyZmr/3VnRg5RBlmH2w6E7wlm6bMVko=; b=AuePuZL2bSIcibmAXbe3zRQxKxvR7sR9qfnsCDZXQjjYH2y03TELhv0GebMP8hJhJV NdD0lX0LVuiK5iFVcEkhiqhX3Tu/wwne8RjfT1xLXVafjf5Rli7LniDVUZEM0SYLkp7l XkFieKDi3UOd1cBwCHVl2eVma9pWmLYpNi2k1nfenF5yzKF+9OkpbkIoH2vepOv+TQoC OPuUiz5gYqYLuzvtOt5iHfoCbSyCqMShaXS/gtgBCABTb5ZPVjn2AUjwrXw0ouX/nvPj Uvu1WZiHdAdsEnA8S7jgQom9BqntEETfUpqIfIuN9t+tBOMNC/GMvRVHrWHEFxyShl9N VQ6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=V63UJPQR8b6EdyZmr/3VnRg5RBlmH2w6E7wlm6bMVko=; b=BkZ4lUkzw2+/neQKL+DqmsejI0yXZvPoqWSo1wEVi9HwD/qrwvX3lq3RE76/x/08sF c6jHwzGQTDY8g3M37+rZo6cG5J75pJZ6V2PLc1etetK5RVP88IEbh7jlDkZ6FK5zU/JX S4AA1i6kpl6fS0OD7IX1iWbIEajpXZpbiLhdJzaV57z+LtLK4XyLQEcVnOIRJ3kRyLJb HAH1solbmsgR0QqNMM3rYwkS0Ks90JpG72bhN2Kv0ZKiBWazzhIdSTzvzrez25sfUZSn aBj/YpB0NISOatKOWjAmwomZMXFqcjMMyLoNFRYk8fdUAKxI0fXOp6CZ2z1yNujTfWZ1 yZ9w== X-Gm-Message-State: AOAM531PEged3SBq+2ISN8Ys6Jjw+vZGf7Ke7DxbWSlzT6FbGciW/YBo U6RtvvYtL0fEEaD8oW3ttR6jGQ== X-Google-Smtp-Source: ABdhPJz3LE/aOX7pwxBVzyg+rwnlssxsmbNBVvY5MbgA2FQBPHQiqlCe2VwBUkDjdtmr83SspcKJDA== X-Received: by 2002:ac2:4c0b:: with SMTP id t11mr9139607lfq.241.1617377220305; Fri, 02 Apr 2021 08:27:00 -0700 (PDT) Received: from box.localdomain ([86.57.175.117]) by smtp.gmail.com with ESMTPSA id h62sm885645lfd.234.2021.04.02.08.26.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Apr 2021 08:26:59 -0700 (PDT) From: "Kirill A. Shutemov" X-Google-Original-From: "Kirill A. Shutemov" Received: by box.localdomain (Postfix, from userid 1000) id 465C4102677; Fri, 2 Apr 2021 18:26:59 +0300 (+03) To: Dave Hansen , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Jim Mattson Cc: David Rientjes , "Edgecombe, Rick P" , "Kleen, Andi" , "Yamahata, Isaku" , x86@kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [RFCv1 5/7] x86/kvmclock: Share hvclock memory with the host Date: Fri, 2 Apr 2021 18:26:43 +0300 Message-Id: <20210402152645.26680-6-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210402152645.26680-1-kirill.shutemov@linux.intel.com> References: <20210402152645.26680-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 410C3E00010F X-Stat-Signature: s1d9tcm9it15hinx4fx4sshywqx6e3u4 Received-SPF: none (shutemov.name>: No applicable sender policy available) receiver=imf05; identity=mailfrom; envelope-from=""; helo=mail-lf1-f47.google.com; client-ip=209.85.167.47 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617377221-764006 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: hvclock is shared between the guest and the hypervisor. It has to be accessible by host. Signed-off-by: Kirill A. Shutemov --- arch/x86/kernel/kvmclock.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kernel/kvmclock.c b/arch/x86/kernel/kvmclock.c index aa593743acf6..3d004b278dba 100644 --- a/arch/x86/kernel/kvmclock.c +++ b/arch/x86/kernel/kvmclock.c @@ -252,7 +252,7 @@ static void __init kvmclock_init_mem(void) * hvclock is shared between the guest and the hypervisor, must * be mapped decrypted. */ - if (sev_active()) { + if (sev_active() || kvm_mem_protected()) { r = set_memory_decrypted((unsigned long) hvclock_mem, 1UL << order); if (r) { From patchwork Fri Apr 2 15:26:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12181195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E63A8C43461 for ; Fri, 2 Apr 2021 15:27:11 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 86DF1610E8 for ; Fri, 2 Apr 2021 15:27:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 86DF1610E8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DE43C6B0083; Fri, 2 Apr 2021 11:27:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D95BD6B0085; Fri, 2 Apr 2021 11:27:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C37726B0087; Fri, 2 Apr 2021 11:27:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0101.hostedemail.com [216.40.44.101]) by kanga.kvack.org (Postfix) with ESMTP id 8AA776B0085 for ; Fri, 2 Apr 2021 11:27:02 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 20C84184875A3 for ; Fri, 2 Apr 2021 15:27:02 +0000 (UTC) X-FDA: 77987805084.04.1C18EE9 Received: from mail-lf1-f53.google.com (mail-lf1-f53.google.com [209.85.167.53]) by imf12.hostedemail.com (Postfix) with ESMTP id 65824DA for ; Fri, 2 Apr 2021 15:27:00 +0000 (UTC) Received: by mail-lf1-f53.google.com with SMTP id w28so8042077lfn.2 for ; Fri, 02 Apr 2021 08:27:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov-name.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NRukntRqIH9oIHyEudEHW0ud1MS+kv1ItAvshsRDqdY=; b=OpnsWKAM4VA/3UajjTCQJv0IEmv/0iPE0cJ2j+RQr1BHXgXYYceO6wtUvATOrb8yhE ROIE0NF0cuH3XzqzsGoFP1JH5Z9yBF0PahBLj0EICTqrjAVYZO4E2fXsJt2FgeUg/8U9 ZRGRbz90+MPk3bQ4bB+VUAa43wHu4myuLLxb01b9PPkMBQm771dpsGaPFoITfHu7WFsv iM0kGJIR3O5KSYl9ajKvDg7zS0KcRszn2eh6ZuRc4lo67X7SLMiHDVAUTOAF6p4iwgt+ WdofJyWwhCixbMTTAehF+H2czWQ42ICOYt0u0Kz7T8yHx9byTn7DeA7quwp3YYB0vhrh bE7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NRukntRqIH9oIHyEudEHW0ud1MS+kv1ItAvshsRDqdY=; b=nxYz3rE60bkmRVJpJtWKltGccM1Wg2aKs4XRwfhKErOKVynBufGQSa0i9N9u2PD/D+ tu3lj1LMF/+qrA4jQ4f8CYnbwqhP3LiGtLK9S0D9lO1qbZvb7ou+uZh+xRbjxIVsjeGe O0fIw8snc50PD3DWHEhMnf48g57B7ok45lac2yNBXfRYo46fv5NVhwgnWyRKRprnpqHS 4NxNGP3WD5RTTbSZ44oYpT8XuriUbwbVhT4C5toJoYJyddPaRiyPAaqo2JM75W3YsEYq HEBm45n3FtP0CUd0VijrlAgE6qRtxMgZmCcPir7QnByDkBQKBZ6r7Q/98XjdyOsGvg48 Dl6g== X-Gm-Message-State: AOAM530dbmPC3bK7kFnF8qaEnx0UaCVxJi18NUplnPQCF/rSFOFWG/Og 0P8WtzfiQhS/r9dEIKyggKmKJA== X-Google-Smtp-Source: ABdhPJw++Ib9aIsPBwn/2Eq095u9hzk548gclpFwNAHjmRz4tdPGJM905f0xXlmoFAawzYOuJioEtg== X-Received: by 2002:a19:b49:: with SMTP id 70mr9524337lfl.342.1617377220450; Fri, 02 Apr 2021 08:27:00 -0700 (PDT) Received: from box.localdomain ([86.57.175.117]) by smtp.gmail.com with ESMTPSA id w23sm953636lji.127.2021.04.02.08.26.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Apr 2021 08:26:59 -0700 (PDT) From: "Kirill A. Shutemov" X-Google-Original-From: "Kirill A. Shutemov" Received: by box.localdomain (Postfix, from userid 1000) id 4E65C102678; Fri, 2 Apr 2021 18:26:59 +0300 (+03) To: Dave Hansen , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Jim Mattson Cc: David Rientjes , "Edgecombe, Rick P" , "Kleen, Andi" , "Yamahata, Isaku" , x86@kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [RFCv1 6/7] x86/realmode: Share trampoline area if KVM memory protection enabled Date: Fri, 2 Apr 2021 18:26:44 +0300 Message-Id: <20210402152645.26680-7-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210402152645.26680-1-kirill.shutemov@linux.intel.com> References: <20210402152645.26680-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 65824DA X-Stat-Signature: egya58kdpkf7499zqiypg5gi38trphrk Received-SPF: none (shutemov.name>: No applicable sender policy available) receiver=imf12; identity=mailfrom; envelope-from=""; helo=mail-lf1-f53.google.com; client-ip=209.85.167.53 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617377220-787203 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If KVM memory protection is active, the trampoline area will need to be in shared memory. Signed-off-by: Kirill A. Shutemov --- arch/x86/realmode/init.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/arch/x86/realmode/init.c b/arch/x86/realmode/init.c index 22fda7d99159..f3b54b5da693 100644 --- a/arch/x86/realmode/init.c +++ b/arch/x86/realmode/init.c @@ -10,6 +10,7 @@ #include #include #include +#include struct real_mode_header *real_mode_header; u32 *trampoline_cr4_features; @@ -75,11 +76,11 @@ static void __init setup_real_mode(void) base = (unsigned char *)real_mode_header; /* - * If SME is active, the trampoline area will need to be in - * decrypted memory in order to bring up other processors + * If SME or KVM memory protection is active, the trampoline area will + * need to be in decrypted memory in order to bring up other processors * successfully. This is not needed for SEV. */ - if (sme_active()) + if (sme_active() || kvm_mem_protected()) set_memory_decrypted((unsigned long)base, size >> PAGE_SHIFT); memcpy(base, real_mode_blob, size); From patchwork Fri Apr 2 15:26:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12181197 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7F2AC43462 for ; Fri, 2 Apr 2021 15:27:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4358B61165 for ; Fri, 2 Apr 2021 15:27:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4358B61165 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5A7976B0085; Fri, 2 Apr 2021 11:27:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5815E6B0087; Fri, 2 Apr 2021 11:27:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 35F758D0002; Fri, 2 Apr 2021 11:27:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0172.hostedemail.com [216.40.44.172]) by kanga.kvack.org (Postfix) with ESMTP id 0A60B6B0087 for ; Fri, 2 Apr 2021 11:27:03 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 86EAD18481CCD for ; Fri, 2 Apr 2021 15:27:02 +0000 (UTC) X-FDA: 77987805084.13.B85E607 Received: from mail-lf1-f49.google.com (mail-lf1-f49.google.com [209.85.167.49]) by imf06.hostedemail.com (Postfix) with ESMTP id 8F9ABC0001FE for ; Fri, 2 Apr 2021 15:27:02 +0000 (UTC) Received: by mail-lf1-f49.google.com with SMTP id d13so8055697lfg.7 for ; Fri, 02 Apr 2021 08:27:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov-name.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=HR3F/UAJDKjuF9Hl3a5y3RW0XzXLCZLHeNx4ieqs1Ks=; b=OFCJ7C88v9T3ZO5FuIYZ4WIRU5HzJdSZ1SrnGSxqXG0JIRO7fdUSBe12lbgfVjrRrs 33I5UWqpEXf8bMQMoXTC+zFHmulioWoCYamTDLQx/E44uzS7VwKJfiYOyAFdbvP2GbYF j4NxZdYwvrgFPaiRHOQIeuRqDoIs77acL205ScFv0qnp1SINI72VldrxQLzap3Ku6RGW PRbjp5iWVRXzUtB+X0KdlxhO3BbB3YbfdCoO6fzmTsem/oXkGYwNbySMtSx8SOP9Lfh0 AnNRbGe8VbXOtX2DrKRr4IbQRFOQvSPRWDDkHqpO4t0DVYfbTJH5NAYR+q6UYA5zjYCa 7nOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HR3F/UAJDKjuF9Hl3a5y3RW0XzXLCZLHeNx4ieqs1Ks=; b=jdqwz1KAa/UnoTKi6VmOdjtEQ8kC7+CVLeRcqZEm1NubLCmMr5ow1YTJo+ZYwDHQmL qFB9aVQjhu535hbWdMTerFmO7+FgsKNnjaopzJWruVKKNYyY3olkOSvDzw/4w/nMIidL 5Jmzk2A/IG/fSjYWGtn5HdL4SJ/94gRZcKazj3e4WeEJf3W0VF5wHVheopBjAieXYQPo u6nejbdbu3ULdaNKMYCjK4HTeHolgiYxjkCK4yO9w5cmGqjQbcpkQfR7KUktuUjOImH/ xjDDhSnKw5tulYkexbcEx2NkfbdhQ/MOhtXZaJ11zYK14PKMQ6PZ1QuTNOvRRC+GJtK1 68gw== X-Gm-Message-State: AOAM530rIhhsDHvAN6/ba92cISxHOSX4yCDHP8znkTSqBbbBzExNrW82 x3unK3Cxde3PVF6VBxyq8mAlOA== X-Google-Smtp-Source: ABdhPJza487W4PANJskXRHpy88EZKj+PehJI/iZExRzRuFwVX6dYkWpTyngQQRbKtaDSI0NNHf/b8Q== X-Received: by 2002:ac2:4205:: with SMTP id y5mr9143399lfh.375.1617377220699; Fri, 02 Apr 2021 08:27:00 -0700 (PDT) Received: from box.localdomain ([86.57.175.117]) by smtp.gmail.com with ESMTPSA id i4sm884729lfv.161.2021.04.02.08.26.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Apr 2021 08:26:59 -0700 (PDT) From: "Kirill A. Shutemov" X-Google-Original-From: "Kirill A. Shutemov" Received: by box.localdomain (Postfix, from userid 1000) id 574C3102679; Fri, 2 Apr 2021 18:26:59 +0300 (+03) To: Dave Hansen , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Jim Mattson Cc: David Rientjes , "Edgecombe, Rick P" , "Kleen, Andi" , "Yamahata, Isaku" , x86@kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [RFCv1 7/7] KVM: unmap guest memory using poisoned pages Date: Fri, 2 Apr 2021 18:26:45 +0300 Message-Id: <20210402152645.26680-8-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210402152645.26680-1-kirill.shutemov@linux.intel.com> References: <20210402152645.26680-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Stat-Signature: botfhp16saekit5hoe9tt8pmnyabhjna X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 8F9ABC0001FE Received-SPF: none (shutemov.name>: No applicable sender policy available) receiver=imf06; identity=mailfrom; envelope-from=""; helo=mail-lf1-f49.google.com; client-ip=209.85.167.49 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617377222-241912 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: TDX architecture aims to provide resiliency against confidentiality and integrity attacks. Towards this goal, the TDX architecture helps enforce the enabling of memory integrity for all TD-private memory. The CPU memory controller computes the integrity check value (MAC) for the data (cache line) during writes, and it stores the MAC with the memory as meta-data. A 28-bit MAC is stored in the ECC bits. Checking of memory integrity is performed during memory reads. If integrity check fails, CPU poisones cache line. On a subsequent consumption (read) of the poisoned data by software, there are two possible scenarios: - Core determines that the execution can continue and it treats poison with exception semantics signaled as a #MCE - Core determines execution cannot continue,and it does an unbreakable shutdown For more details, see Chapter 14 of Intel TDX Module EAS[1] As some of integrity check failures may lead to system shutdown host kernel must not allow any writes to TD-private memory. This requirment clashes with KVM design: KVM expects the guest memory to be mapped into host userspace (e.g. QEMU). This patch aims to start discussion on how we can approach the issue. For now I intentionally keep TDX out of picture here and try to find a generic way to unmap KVM guest memory from host userspace. Hopefully, it makes the patch more approachable. And anyone can try it out. To the proposal: Looking into existing codepaths I've discovered that we already have semantics we want. That's PG_hwpoison'ed pages and SWP_HWPOISON swap entries in page tables: - If an application touches a page mapped with the SWP_HWPOISON, it will get SIGBUS. - GUP will fail with -EFAULT; Access the poisoned memory via page cache doesn't match required semantics right now, but it shouldn't be too hard to make it work: access to poisoned dirty pages should give -EIO or -EHWPOISON. My idea is that we can mark page as poisoned when we make it TD-private and replace all PTEs that map the page with SWP_HWPOISON. The patch is proof-of-concept and has known issues: - Limited to swap-backed pages for now: anon or tmpfs/shmem - No THP support - Need a new FOLL_XXX flags to access such pages from KVM code. - Page unpoisoning is not implemented. It proved to be more difficult than I expected. I'm looking into solution. - Poisoned pages must be tied to KVM instance and another KVM must not be able to map the page into guest. [1] https://software.intel.com/content/dam/develop/external/us/en/documents/intel-tdx-module-1eas.pdf Not-signed-off-by: Kirill A. Shutemov --- arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/cpuid.c | 3 +- arch/x86/kvm/mmu/mmu.c | 15 ++- arch/x86/kvm/mmu/paging_tmpl.h | 10 +- arch/x86/kvm/x86.c | 6 ++ include/linux/kvm_host.h | 12 +++ include/linux/swapops.h | 20 ++++ include/uapi/linux/kvm_para.h | 1 + mm/gup.c | 31 ++++--- mm/memory.c | 45 ++++++++- mm/page_vma_mapped.c | 8 +- mm/rmap.c | 2 +- mm/shmem.c | 7 ++ virt/kvm/Kconfig | 3 + virt/kvm/kvm_main.c | 164 +++++++++++++++++++++++++++++---- 15 files changed, 290 insertions(+), 38 deletions(-) diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig index 7ac592664c52..b7db1c455e7c 100644 --- a/arch/x86/kvm/Kconfig +++ b/arch/x86/kvm/Kconfig @@ -46,6 +46,7 @@ config KVM select KVM_GENERIC_DIRTYLOG_READ_PROTECT select KVM_VFIO select SRCU + select HAVE_KVM_PROTECTED_MEMORY help Support hosting fully virtualized guest machines using hardware virtualization extensions. You will need a fairly recent diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index 38172ca627d3..1457692c1080 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -796,7 +796,8 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) (1 << KVM_FEATURE_PV_SEND_IPI) | (1 << KVM_FEATURE_POLL_CONTROL) | (1 << KVM_FEATURE_PV_SCHED_YIELD) | - (1 << KVM_FEATURE_ASYNC_PF_INT); + (1 << KVM_FEATURE_ASYNC_PF_INT) | + (1 << KVM_FEATURE_MEM_PROTECTED); if (sched_info_on()) entry->eax |= (1 << KVM_FEATURE_STEAL_TIME); diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 6d16481aa29d..53a69c8c59f1 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -43,6 +43,7 @@ #include #include #include +#include #include #include @@ -2758,7 +2759,8 @@ static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep) if (sp->role.level > PG_LEVEL_4K) return; - __direct_pte_prefetch(vcpu, sp, sptep); + if (!vcpu->kvm->mem_protected) + __direct_pte_prefetch(vcpu, sp, sptep); } static int host_pfn_mapping_level(struct kvm_vcpu *vcpu, gfn_t gfn, @@ -3723,6 +3725,17 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code, if (handle_abnormal_pfn(vcpu, is_tdp ? 0 : gpa, gfn, pfn, ACC_ALL, &r)) return r; + if (vcpu->kvm->mem_protected && unlikely(!is_noslot_pfn(pfn)) && + !gfn_is_shared(vcpu->kvm, gfn)) { + struct page *page = pfn_to_page(pfn); + lock_page(page); + VM_BUG_ON_PAGE(!PageSwapBacked(page) && !PageReserved(page), page); + /* Recheck gfn_is_shared() under page lock */ + if (!gfn_is_shared(vcpu->kvm, gfn) && !TestSetPageHWPoison(page)) + try_to_unmap(page, TTU_IGNORE_MLOCK); + unlock_page(page); + } + r = RET_PF_RETRY; spin_lock(&vcpu->kvm->mmu_lock); if (mmu_notifier_retry(vcpu->kvm, mmu_seq)) diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 50e268eb8e1a..26b0494a1207 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -397,8 +397,14 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, goto error; ptep_user = (pt_element_t __user *)((void *)host_addr + offset); - if (unlikely(__get_user(pte, ptep_user))) - goto error; + if (vcpu->kvm->mem_protected) { + if (copy_from_guest(vcpu->kvm, &pte, host_addr + offset, + sizeof(pte))) + goto error; + } else { + if (unlikely(__get_user(pte, ptep_user))) + goto error; + } walker->ptep_user[walker->level - 1] = ptep_user; trace_kvm_mmu_paging_element(pte, walker->level); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 1b404e4d7dd8..f8183386abe7 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -8170,6 +8170,12 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu) kvm_sched_yield(vcpu->kvm, a0); ret = 0; break; + case KVM_HC_ENABLE_MEM_PROTECTED: + ret = kvm_protect_memory(vcpu->kvm); + break; + case KVM_HC_MEM_SHARE: + ret = kvm_share_memory(vcpu->kvm, a0, a1); + break; default: ret = -KVM_ENOSYS; break; diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index f3b1013fb22c..f941bcbefb79 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -436,6 +436,8 @@ static inline int kvm_arch_vcpu_memslots_id(struct kvm_vcpu *vcpu) } #endif +#define KVM_NR_SHARED_RANGES 32 + /* * Note: * memslots are not sorted by id anymore, please use id_to_memslot() @@ -513,6 +515,9 @@ struct kvm { pid_t userspace_pid; unsigned int max_halt_poll_ns; u32 dirty_ring_size; + bool mem_protected; + int nr_shared_ranges; + struct range shared_ranges[KVM_NR_SHARED_RANGES]; }; #define kvm_err(fmt, ...) \ @@ -709,6 +714,10 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm); void kvm_arch_flush_shadow_memslot(struct kvm *kvm, struct kvm_memory_slot *slot); +int kvm_protect_memory(struct kvm *kvm); +int kvm_share_memory(struct kvm *kvm, unsigned long gfn, unsigned long npages); +bool gfn_is_shared(struct kvm *kvm, unsigned long gfn); + int gfn_to_page_many_atomic(struct kvm_memory_slot *slot, gfn_t gfn, struct page **pages, int nr_pages); @@ -718,6 +727,9 @@ unsigned long gfn_to_hva_prot(struct kvm *kvm, gfn_t gfn, bool *writable); unsigned long gfn_to_hva_memslot(struct kvm_memory_slot *slot, gfn_t gfn); unsigned long gfn_to_hva_memslot_prot(struct kvm_memory_slot *slot, gfn_t gfn, bool *writable); +int copy_from_guest(struct kvm *kvm, void *data, unsigned long hva, int len); +int copy_to_guest(struct kvm *kvm, unsigned long hva, const void *data, int len); + void kvm_release_page_clean(struct page *page); void kvm_release_page_dirty(struct page *page); void kvm_set_page_accessed(struct page *page); diff --git a/include/linux/swapops.h b/include/linux/swapops.h index d9b7c9132c2f..520589b12fb3 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -323,6 +323,16 @@ static inline int is_hwpoison_entry(swp_entry_t entry) return swp_type(entry) == SWP_HWPOISON; } +static inline unsigned long hwpoison_entry_to_pfn(swp_entry_t entry) +{ + return swp_offset(entry); +} + +static inline struct page *hwpoison_entry_to_page(swp_entry_t entry) +{ + return pfn_to_page(hwpoison_entry_to_pfn(entry)); +} + static inline void num_poisoned_pages_inc(void) { atomic_long_inc(&num_poisoned_pages); @@ -345,6 +355,16 @@ static inline int is_hwpoison_entry(swp_entry_t swp) return 0; } +static inline unsigned long hwpoison_entry_to_pfn(swp_entry_t entry) +{ + return 0; +} + +static inline struct page *hwpoison_entry_to_page(swp_entry_t entry) +{ + return NULL; +} + static inline void num_poisoned_pages_inc(void) { } diff --git a/include/uapi/linux/kvm_para.h b/include/uapi/linux/kvm_para.h index 09d36683ee0a..743e621111f0 100644 --- a/include/uapi/linux/kvm_para.h +++ b/include/uapi/linux/kvm_para.h @@ -17,6 +17,7 @@ #define KVM_E2BIG E2BIG #define KVM_EPERM EPERM #define KVM_EOPNOTSUPP 95 +#define KVM_EINTR EINTR #define KVM_HC_VAPIC_POLL_IRQ 1 #define KVM_HC_MMU_OP 2 diff --git a/mm/gup.c b/mm/gup.c index e4c224cd9661..ce4fdf213455 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -384,22 +384,31 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, ptep = pte_offset_map_lock(mm, pmd, address, &ptl); pte = *ptep; if (!pte_present(pte)) { - swp_entry_t entry; + swp_entry_t entry = pte_to_swp_entry(pte); + + if (pte_none(pte)) + goto no_page; + /* * KSM's break_ksm() relies upon recognizing a ksm page * even while it is being migrated, so for that case we * need migration_entry_wait(). */ - if (likely(!(flags & FOLL_MIGRATION))) - goto no_page; - if (pte_none(pte)) - goto no_page; - entry = pte_to_swp_entry(pte); - if (!is_migration_entry(entry)) - goto no_page; - pte_unmap_unlock(ptep, ptl); - migration_entry_wait(mm, pmd, address); - goto retry; + if (is_migration_entry(entry) && (flags & FOLL_MIGRATION)) { + pte_unmap_unlock(ptep, ptl); + migration_entry_wait(mm, pmd, address); + goto retry; + } + + if (is_hwpoison_entry(entry)) { + page = hwpoison_entry_to_page(entry); + if (PageHWPoison(page) /* && (flags & FOLL_ALLOW_POISONED) */) { + get_page(page); + goto out; + } + } + + goto no_page; } if ((flags & FOLL_NUMA) && pte_protnone(pte)) goto no_page; diff --git a/mm/memory.c b/mm/memory.c index feff48e1465a..524dce15a087 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -767,6 +767,9 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, pte = pte_swp_mkuffd_wp(pte); set_pte_at(src_mm, addr, src_pte, pte); } + } else if (is_hwpoison_entry(entry)) { + page = hwpoison_entry_to_page(entry); + get_page(page); } set_pte_at(dst_mm, addr, dst_pte, pte); return 0; @@ -1305,6 +1308,9 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, page = migration_entry_to_page(entry); rss[mm_counter(page)]--; + + } else if (is_hwpoison_entry(entry)) { + put_page(hwpoison_entry_to_page(entry)); } if (unlikely(!free_swap_and_cache(entry))) print_bad_pte(vma, addr, ptent, NULL); @@ -3274,7 +3280,43 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vmf->page = device_private_entry_to_page(entry); ret = vmf->page->pgmap->ops->migrate_to_ram(vmf); } else if (is_hwpoison_entry(entry)) { - ret = VM_FAULT_HWPOISON; + page = hwpoison_entry_to_page(entry); + + locked = lock_page_or_retry(page, vma->vm_mm, vmf->flags); + if (!locked) { + ret = VM_FAULT_RETRY; + goto out; + } + + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); + + if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) { + ret = 0; + } else if (PageHWPoison(page)) { + ret = VM_FAULT_HWPOISON; + } else { + /* + * The page is unpoisoned. Replace hwpoison + * entry with a present PTE. + */ + + inc_mm_counter(vma->vm_mm, mm_counter(page)); + pte = mk_pte(page, vma->vm_page_prot); + + if (PageAnon(page)) { + page_add_anon_rmap(page, vma, + vmf->address, false); + } else { + page_add_file_rmap(page, false); + } + + set_pte_at(vma->vm_mm, vmf->address, + vmf->pte, pte); + } + + pte_unmap_unlock(vmf->pte, vmf->ptl); + unlock_page(page); } else { print_bad_pte(vma, vmf->address, vmf->orig_pte, NULL); ret = VM_FAULT_SIGBUS; @@ -3282,7 +3324,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out; } - delayacct_set_flag(DELAYACCT_PF_SWAPIN); page = lookup_swap_cache(entry, vma, vmf->address); swapcache = page; diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 86e3a3688d59..8fffae175104 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -93,10 +93,12 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw) return false; entry = pte_to_swp_entry(*pvmw->pte); - if (!is_migration_entry(entry)) + if (is_migration_entry(entry)) + pfn = migration_entry_to_pfn(entry); + else if (is_hwpoison_entry(entry)) + pfn = hwpoison_entry_to_pfn(entry); + else return false; - - pfn = migration_entry_to_pfn(entry); } else if (is_swap_pte(*pvmw->pte)) { swp_entry_t entry; diff --git a/mm/rmap.c b/mm/rmap.c index 08c56aaf72eb..f08d1fc28522 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1575,7 +1575,7 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, dec_mm_counter(mm, mm_counter(page)); set_pte_at(mm, address, pvmw.pte, pteval); } - + get_page(page); } else if (pte_unused(pteval) && !userfaultfd_armed(vma)) { /* * The guest indicated that the page content is of no diff --git a/mm/shmem.c b/mm/shmem.c index 7c6b6d8f6c39..d29a0c9be19c 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1832,6 +1832,13 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, if (page) hindex = page->index; + + if (page && PageHWPoison(page)) { + unlock_page(page); + put_page(page); + return -EIO; + } + if (page && sgp == SGP_WRITE) mark_page_accessed(page); diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig index 1c37ccd5d402..50d7422386aa 100644 --- a/virt/kvm/Kconfig +++ b/virt/kvm/Kconfig @@ -63,3 +63,6 @@ config HAVE_KVM_NO_POLL config KVM_XFER_TO_GUEST_WORK bool + +config HAVE_KVM_PROTECTED_MEMORY + bool diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 8367d88ce39b..f182c54bfa34 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -51,6 +51,7 @@ #include #include #include +#include #include #include @@ -2333,19 +2334,85 @@ static int next_segment(unsigned long len, int offset) return len; } -static int __kvm_read_guest_page(struct kvm_memory_slot *slot, gfn_t gfn, - void *data, int offset, int len) +int copy_from_guest(struct kvm *kvm, void *data, unsigned long hva, int len) +{ + int offset = offset_in_page(hva); + struct page *page; + int npages, seg; + void *vaddr; + + if (!IS_ENABLED(CONFIG_HAVE_KVM_PROTECTED_MEMORY) || + !kvm->mem_protected) { + return __copy_from_user(data, (void __user *)hva, len); + } + + might_fault(); + kasan_check_write(data, len); + check_object_size(data, len, false); + + while ((seg = next_segment(len, offset)) != 0) { + npages = get_user_pages_unlocked(hva, 1, &page, 0); + if (npages != 1) + return -EFAULT; + + vaddr = kmap_atomic(page); + memcpy(data, vaddr + offset, seg); + kunmap_atomic(vaddr); + + put_page(page); + len -= seg; + hva += seg; + data += seg; + offset = 0; + } + + return 0; +} + +int copy_to_guest(struct kvm *kvm, unsigned long hva, const void *data, int len) +{ + int offset = offset_in_page(hva); + struct page *page; + int npages, seg; + void *vaddr; + + if (!IS_ENABLED(CONFIG_HAVE_KVM_PROTECTED_MEMORY) || + !kvm->mem_protected) { + return __copy_to_user((void __user *)hva, data, len); + } + + might_fault(); + kasan_check_read(data, len); + check_object_size(data, len, true); + + while ((seg = next_segment(len, offset)) != 0) { + npages = get_user_pages_unlocked(hva, 1, &page, FOLL_WRITE); + if (npages != 1) + return -EFAULT; + + vaddr = kmap_atomic(page); + memcpy(vaddr + offset, data, seg); + kunmap_atomic(vaddr); + + put_page(page); + len -= seg; + hva += seg; + data += seg; + offset = 0; + } + + return 0; +} + +static int __kvm_read_guest_page(struct kvm *kvm, struct kvm_memory_slot *slot, + gfn_t gfn, void *data, int offset, int len) { - int r; unsigned long addr; addr = gfn_to_hva_memslot_prot(slot, gfn, NULL); if (kvm_is_error_hva(addr)) return -EFAULT; - r = __copy_from_user(data, (void __user *)addr + offset, len); - if (r) - return -EFAULT; - return 0; + return copy_from_guest(kvm, data, addr + offset, len); } int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset, @@ -2353,7 +2420,7 @@ int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset, { struct kvm_memory_slot *slot = gfn_to_memslot(kvm, gfn); - return __kvm_read_guest_page(slot, gfn, data, offset, len); + return __kvm_read_guest_page(kvm, slot, gfn, data, offset, len); } EXPORT_SYMBOL_GPL(kvm_read_guest_page); @@ -2362,7 +2429,7 @@ int kvm_vcpu_read_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data, { struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn); - return __kvm_read_guest_page(slot, gfn, data, offset, len); + return __kvm_read_guest_page(vcpu->kvm, slot, gfn, data, offset, len); } EXPORT_SYMBOL_GPL(kvm_vcpu_read_guest_page); @@ -2444,7 +2511,8 @@ static int __kvm_write_guest_page(struct kvm *kvm, addr = gfn_to_hva_memslot(memslot, gfn); if (kvm_is_error_hva(addr)) return -EFAULT; - r = __copy_to_user((void __user *)addr + offset, data, len); + + r = copy_to_guest(kvm, addr + offset, data, len); if (r) return -EFAULT; mark_page_dirty_in_slot(kvm, memslot, gfn); @@ -2581,7 +2649,7 @@ int kvm_write_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc, if (unlikely(!ghc->memslot)) return kvm_write_guest(kvm, gpa, data, len); - r = __copy_to_user((void __user *)ghc->hva + offset, data, len); + r = copy_to_guest(kvm, ghc->hva + offset, data, len); if (r) return -EFAULT; mark_page_dirty_in_slot(kvm, ghc->memslot, gpa >> PAGE_SHIFT); @@ -2602,7 +2670,6 @@ int kvm_read_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc, unsigned long len) { struct kvm_memslots *slots = kvm_memslots(kvm); - int r; gpa_t gpa = ghc->gpa + offset; BUG_ON(len + offset > ghc->len); @@ -2618,11 +2685,7 @@ int kvm_read_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc, if (unlikely(!ghc->memslot)) return kvm_read_guest(kvm, gpa, data, len); - r = __copy_from_user(data, (void __user *)ghc->hva + offset, len); - if (r) - return -EFAULT; - - return 0; + return copy_from_guest(kvm, data, ghc->hva + offset, len); } EXPORT_SYMBOL_GPL(kvm_read_guest_offset_cached); @@ -2688,6 +2751,73 @@ void kvm_vcpu_mark_page_dirty(struct kvm_vcpu *vcpu, gfn_t gfn) } EXPORT_SYMBOL_GPL(kvm_vcpu_mark_page_dirty); +int kvm_protect_memory(struct kvm *kvm) +{ + if (mmap_write_lock_killable(kvm->mm)) + return -KVM_EINTR; + kvm->mem_protected = true; + kvm_arch_flush_shadow_all(kvm); + mmap_write_unlock(kvm->mm); + + return 0; +} + +bool gfn_is_shared(struct kvm *kvm, unsigned long gfn) +{ + bool ret = false; + int i; + + spin_lock(&kvm->mmu_lock); + for (i = 0; i < kvm->nr_shared_ranges; i++) { + if (gfn < kvm->shared_ranges[i].start) + continue; + if (gfn >= kvm->shared_ranges[i].end) + continue; + + ret = true; + break; + } + spin_unlock(&kvm->mmu_lock); + + return ret; +} + +int kvm_share_memory(struct kvm *kvm, unsigned long gfn, unsigned long npages) +{ + unsigned long end = gfn + npages; + + if (!npages) + return 0; + + /* + * Out of slots. + * Still worth to proceed: the new range may merge with an existing + * one. + */ + WARN_ON_ONCE(kvm->nr_shared_ranges == ARRAY_SIZE(kvm->shared_ranges)); + + spin_lock(&kvm->mmu_lock); + kvm->nr_shared_ranges = add_range_with_merge(kvm->shared_ranges, + ARRAY_SIZE(kvm->shared_ranges), + kvm->nr_shared_ranges, gfn, end); + kvm->nr_shared_ranges = clean_sort_range(kvm->shared_ranges, + ARRAY_SIZE(kvm->shared_ranges)); + spin_unlock(&kvm->mmu_lock); + + for (; gfn < end; gfn++) { + struct page *page = gfn_to_page(kvm, gfn); + + if (page == KVM_ERR_PTR_BAD_PAGE) + continue; + lock_page(page); + ClearPageHWPoison(page); + unlock_page(page); + put_page(page); + } + + return 0; +} + void kvm_sigset_activate(struct kvm_vcpu *vcpu) { if (!vcpu->sigset_active)