From patchwork Mon May 13 14:38:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941143 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 91CE61390 for ; Mon, 13 May 2019 14:42:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8547A27861 for ; Mon, 13 May 2019 14:42:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 79A722832D; Mon, 13 May 2019 14:42:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2666627861 for ; Mon, 13 May 2019 14:42:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730780AbfEMOmg (ORCPT ); Mon, 13 May 2019 10:42:36 -0400 Received: from aserp2130.oracle.com ([141.146.126.79]:36808 "EHLO aserp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730180AbfEMOji (ORCPT ); Mon, 13 May 2019 10:39:38 -0400 Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DESwxh183032; Mon, 13 May 2019 14:38:50 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=1/A8dPg6N3BCp1dVVHZldE7IkiS+8kPsjdwBuin2g4s=; b=IIHbvoJP5+K/6hJmjo6mwDCr+XYe5WkcmTj/jZgsHQd++1A3nrRugabuYhfepHXTNKMa V+W5LQ4oABEgMC2/bRAoDFfs1URTJrv1FkjV3nZfJhEh3aTydJM7O9zVMLE+XCLnCNo0 mgaCYQLJsGwxxqQvTsOBwgrZT+TlGys5Y+xXj8Q2b2tjQKrJs5qkBiGxQsdVfxbT4cvm AvsvkAn4MmTs1z4WmXxi/1KuzDaA9HLi5wH7yN5hYhVDuggPf0x5mUO+bX8AKL3WJ1Pd Vh+iMejI0gUwJY+npXZCvzmO6WQnPg//1CrX7b7e8J9CRrNauyV7gxxNmcg6wsYTegjc qw== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by aserp2130.oracle.com with ESMTP id 2sdkwdfksf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:38:50 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQ4022780; Mon, 13 May 2019 14:38:42 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 01/27] kernel: Export memory-management symbols required for KVM address space isolation Date: Mon, 13 May 2019 16:38:09 +0200 Message-Id: <1557758315-12667-2-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130102 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Liran Alon Export symbols needed to create, manage, populate and switch a mm from a kernel module (kvm in this case). This is a hacky way for now to start. This should be changed to some suitable memory-management API. Signed-off-by: Liran Alon Signed-off-by: Alexandre Chartre --- arch/x86/kernel/ldt.c | 1 + arch/x86/mm/tlb.c | 3 ++- mm/memory.c | 5 +++++ 3 files changed, 8 insertions(+), 1 deletions(-) diff --git a/arch/x86/kernel/ldt.c b/arch/x86/kernel/ldt.c index b2463fc..19a86e0 100644 --- a/arch/x86/kernel/ldt.c +++ b/arch/x86/kernel/ldt.c @@ -401,6 +401,7 @@ void destroy_context_ldt(struct mm_struct *mm) free_ldt_struct(mm->context.ldt); mm->context.ldt = NULL; } +EXPORT_SYMBOL_GPL(destroy_context_ldt); void ldt_arch_exit_mmap(struct mm_struct *mm) { diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 7f61431..a4db7f5 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -70,7 +70,7 @@ static void clear_asid_other(void) } atomic64_t last_mm_ctx_id = ATOMIC64_INIT(1); - +EXPORT_SYMBOL_GPL(last_mm_ctx_id); static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, u16 *new_asid, bool *need_flush) @@ -159,6 +159,7 @@ void switch_mm(struct mm_struct *prev, struct mm_struct *next, switch_mm_irqs_off(prev, next, tsk); local_irq_restore(flags); } +EXPORT_SYMBOL_GPL(switch_mm); static void sync_current_stack_to_mm(struct mm_struct *mm) { diff --git a/mm/memory.c b/mm/memory.c index 36aac68..ede9335 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -434,6 +434,7 @@ int __pte_alloc(struct mm_struct *mm, pmd_t *pmd) pte_free(mm, new); return 0; } +EXPORT_SYMBOL_GPL(__pte_alloc); int __pte_alloc_kernel(pmd_t *pmd) { @@ -453,6 +454,7 @@ int __pte_alloc_kernel(pmd_t *pmd) pte_free_kernel(&init_mm, new); return 0; } +EXPORT_SYMBOL_GPL(__pte_alloc_kernel); static inline void init_rss_vec(int *rss) { @@ -4007,6 +4009,7 @@ int __p4d_alloc(struct mm_struct *mm, pgd_t *pgd, unsigned long address) spin_unlock(&mm->page_table_lock); return 0; } +EXPORT_SYMBOL_GPL(__p4d_alloc); #endif /* __PAGETABLE_P4D_FOLDED */ #ifndef __PAGETABLE_PUD_FOLDED @@ -4039,6 +4042,7 @@ int __pud_alloc(struct mm_struct *mm, p4d_t *p4d, unsigned long address) spin_unlock(&mm->page_table_lock); return 0; } +EXPORT_SYMBOL_GPL(__pud_alloc); #endif /* __PAGETABLE_PUD_FOLDED */ #ifndef __PAGETABLE_PMD_FOLDED @@ -4072,6 +4076,7 @@ int __pmd_alloc(struct mm_struct *mm, pud_t *pud, unsigned long address) spin_unlock(ptl); return 0; } +EXPORT_SYMBOL_GPL(__pmd_alloc); #endif /* __PAGETABLE_PMD_FOLDED */ static int __follow_pte_pmd(struct mm_struct *mm, unsigned long address, From patchwork Mon May 13 14:38:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941147 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 73D8A6C5 for ; Mon, 13 May 2019 14:42:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 655DA27861 for ; Mon, 13 May 2019 14:42:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 583992832D; Mon, 13 May 2019 14:42:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F337727861 for ; Mon, 13 May 2019 14:42:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730151AbfEMOje (ORCPT ); Mon, 13 May 2019 10:39:34 -0400 Received: from userp2120.oracle.com ([156.151.31.85]:59240 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730136AbfEMOjd (ORCPT ); Mon, 13 May 2019 10:39:33 -0400 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DESr12184826; Mon, 13 May 2019 14:38:48 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=03NJAuy8gQRMw5NMdO5oIpncwVrdcNhEn/qnXsnIh+Q=; b=g5WaOGjMCQGcw3yFQID9Xz5SR+orOYV1jEvqta9pP1dW+3vk6S7rk3Xf0Xqs9c72Ro1w /TCodkQB+5Z4YQjeExvs0QvsKAbjAzfxUXW21GiZ7ZoPR7nSWQ/5N76ENeSlLIaBPOuq oraTRR4s8rulSGRtJmfzQ6I6tr45ficozvt9Zf9IR0tS1V0R/j/qCwAykrzu4TBWku12 vyAVK6nNFxTgtyozBfwL3irrf/S3fPxxNjChwJoT7jqQ+3BTuw9SWcfOTckw2b1l05Nd oG/FOBhG00v9BsVeDYzcuV3eTXQRhjHKRHxD7Luw1u+N5LvXr+eEUi8yT9Gj8XmE4JX0 Bg== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2sdq1q7aqq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:38:48 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQ5022780; Mon, 13 May 2019 14:38:45 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 02/27] KVM: x86: Introduce address_space_isolation module parameter Date: Mon, 13 May 2019 16:38:10 +0200 Message-Id: <1557758315-12667-3-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130102 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Liran Alon Add the address_space_isolation parameter to the kvm module. When set to true, KVM #VMExit handlers run in isolated address space which maps only KVM required code and per-VM information instead of entire kernel address space. This mechanism is meant to mitigate memory-leak side-channels CPU vulnerabilities (e.g. Spectre, L1TF and etc.) but can also be viewed as security in-depth as it also helps generically against info-leaks vulnerabilities in KVM #VMExit handlers and reduce the available gadgets for ROP attacks. This is set to false by default because it incurs a performance hit which some users will not want to take for security gain. Signed-off-by: Liran Alon Signed-off-by: Alexandre Chartre --- arch/x86/kvm/Makefile | 2 +- arch/x86/kvm/isolation.c | 26 ++++++++++++++++++++++++++ 2 files changed, 27 insertions(+), 1 deletions(-) create mode 100644 arch/x86/kvm/isolation.c diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile index 31ecf7a..9f404e9 100644 --- a/arch/x86/kvm/Makefile +++ b/arch/x86/kvm/Makefile @@ -10,7 +10,7 @@ kvm-$(CONFIG_KVM_ASYNC_PF) += $(KVM)/async_pf.o kvm-y += x86.o mmu.o emulate.o i8259.o irq.o lapic.o \ i8254.o ioapic.o irq_comm.o cpuid.o pmu.o mtrr.o \ - hyperv.o page_track.o debugfs.o + hyperv.o page_track.o debugfs.o isolation.o kvm-intel-y += vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o vmx/evmcs.o vmx/nested.o kvm-amd-y += svm.o pmu_amd.o diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c new file mode 100644 index 0000000..e25f663 --- /dev/null +++ b/arch/x86/kvm/isolation.c @@ -0,0 +1,26 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved. + * + * KVM Address Space Isolation + */ + +#include +#include + +/* + * When set to true, KVM #VMExit handlers run in isolated address space + * which maps only KVM required code and per-VM information instead of + * entire kernel address space. + * + * This mechanism is meant to mitigate memory-leak side-channels CPU + * vulnerabilities (e.g. Spectre, L1TF and etc.) but can also be viewed + * as security in-depth as it also helps generically against info-leaks + * vulnerabilities in KVM #VMExit handlers and reduce the available + * gadgets for ROP attacks. + * + * This is set to false by default because it incurs a performance hit + * which some users will not want to take for security gain. + */ +static bool __read_mostly address_space_isolation; +module_param(address_space_isolation, bool, 0444); From patchwork Mon May 13 14:38:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941053 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 47CDB6C5 for ; Mon, 13 May 2019 14:39:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 38AE82817F for ; Mon, 13 May 2019 14:39:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 29D2B28305; Mon, 13 May 2019 14:39:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A258227FAE for ; Mon, 13 May 2019 14:39:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730043AbfEMOj2 (ORCPT ); Mon, 13 May 2019 10:39:28 -0400 Received: from userp2120.oracle.com ([156.151.31.85]:59132 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728225AbfEMOj2 (ORCPT ); Mon, 13 May 2019 10:39:28 -0400 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DESuQq184844; Mon, 13 May 2019 14:38:56 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=N+bhv6OONCtHadzhgbU1e3MAiku0pVi0en6iW2c0TrE=; b=P2E6fPrVnhroaq8muel0nn2IGVthj4KsZusNGeNWJzt9GowNKpeWiTFc/b69anfJ0Jmu oveuhNt89Ikw71ZvGW5vaL++kydq+j4PLd4quL0ofYa2FFOqRahZ4/VLfHaGhpQ4vHLA m5ceIwPWsPji5If/gseVw1NXQQCm7CUpEO9gRgNKuigMbubyHEiBRLZ2oXHfdXvE5+yg 4Vxc4Q0CWq9Ih+MZt7QCjMIpS8BEJfgPaEsyU1ULUJzIGUT76jHo7eJhX4dTAjgX0kjG sJYESl+2wGBWqLkwymFi6MuDOCTCpC/bVqNxgtfnEiuBaKSkXU7jAwqSWP/dnuQqoRnn 9A== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2sdq1q7as5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:38:56 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQ6022780; Mon, 13 May 2019 14:38:47 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 03/27] KVM: x86: Introduce KVM separate virtual address space Date: Mon, 13 May 2019 16:38:11 +0200 Message-Id: <1557758315-12667-4-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130102 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Liran Alon Create a separate mm for KVM that will be active when KVM #VMExit handlers run. Up until the point which we architectully need to access host (or other VM) sensitive data. This patch just create kvm_mm but never makes it active yet. This will be done by next commits. Signed-off-by: Liran Alon Signed-off-by: Alexandre Chartre --- arch/x86/kvm/isolation.c | 95 ++++++++++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/isolation.h | 8 ++++ arch/x86/kvm/x86.c | 10 ++++- 3 files changed, 112 insertions(+), 1 deletions(-) create mode 100644 arch/x86/kvm/isolation.h diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index e25f663..74bc0cd 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -7,6 +7,21 @@ #include #include +#include + +#include +#include + +#include "isolation.h" + +struct mm_struct kvm_mm = { + .mm_rb = RB_ROOT, + .mm_users = ATOMIC_INIT(2), + .mm_count = ATOMIC_INIT(1), + .mmap_sem = __RWSEM_INITIALIZER(kvm_mm.mmap_sem), + .page_table_lock = __SPIN_LOCK_UNLOCKED(kvm_mm.page_table_lock), + .mmlist = LIST_HEAD_INIT(kvm_mm.mmlist), +}; /* * When set to true, KVM #VMExit handlers run in isolated address space @@ -24,3 +39,83 @@ */ static bool __read_mostly address_space_isolation; module_param(address_space_isolation, bool, 0444); + +static int kvm_isolation_init_mm(void) +{ + pgd_t *kvm_pgd; + gfp_t gfp_mask; + + gfp_mask = GFP_KERNEL | __GFP_ZERO; + kvm_pgd = (pgd_t *)__get_free_pages(gfp_mask, PGD_ALLOCATION_ORDER); + if (!kvm_pgd) + return -ENOMEM; + +#ifdef CONFIG_PAGE_TABLE_ISOLATION + /* + * With PTI, we have two PGDs: one the kernel page table, and one + * for the user page table. The PGD with the kernel page table has + * to be the entire kernel address space because paranoid faults + * will unconditionally use it. So we define the KVM address space + * in the user table space, although it will be used in the kernel. + */ + + /* initialize the kernel page table */ + memcpy(kvm_pgd, current->active_mm->pgd, sizeof(pgd_t) * PTRS_PER_PGD); + + /* define kvm_mm with the user page table */ + kvm_mm.pgd = kernel_to_user_pgdp(kvm_pgd); +#else /* CONFIG_PAGE_TABLE_ISOLATION */ + kvm_mm.pgd = kvm_pgd; +#endif /* CONFIG_PAGE_TABLE_ISOLATION */ + mm_init_cpumask(&kvm_mm); + init_new_context(NULL, &kvm_mm); + + return 0; +} + +static void kvm_isolation_uninit_mm(void) +{ + pgd_t *kvm_pgd; + + BUG_ON(current->active_mm == &kvm_mm); + + destroy_context(&kvm_mm); + +#ifdef CONFIG_PAGE_TABLE_ISOLATION + /* + * With PTI, the KVM address space is defined in the user + * page table space, but the full PGD starts with the kernel + * page table space. + */ + kvm_pgd = user_to_kernel_pgdp(kvm_pgd); +#else /* CONFIG_PAGE_TABLE_ISOLATION */ + kvm_pgd = kvm_mm.pgd; +#endif /* CONFIG_PAGE_TABLE_ISOLATION */ + kvm_mm.pgd = NULL; + free_pages((unsigned long)kvm_pgd, PGD_ALLOCATION_ORDER); +} + +int kvm_isolation_init(void) +{ + int r; + + if (!address_space_isolation) + return 0; + + r = kvm_isolation_init_mm(); + if (r) + return r; + + pr_info("KVM: x86: Running with isolated address space\n"); + + return 0; +} + +void kvm_isolation_uninit(void) +{ + if (!address_space_isolation) + return; + + kvm_isolation_uninit_mm(); + pr_info("KVM: x86: End of isolated address space\n"); +} diff --git a/arch/x86/kvm/isolation.h b/arch/x86/kvm/isolation.h new file mode 100644 index 0000000..cf8c7d4 --- /dev/null +++ b/arch/x86/kvm/isolation.h @@ -0,0 +1,8 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef ARCH_X86_KVM_ISOLATION_H +#define ARCH_X86_KVM_ISOLATION_H + +extern int kvm_isolation_init(void); +extern void kvm_isolation_uninit(void); + +#endif diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index b5edc8e..4b7cec2 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -29,6 +29,7 @@ #include "cpuid.h" #include "pmu.h" #include "hyperv.h" +#include "isolation.h" #include #include @@ -6972,10 +6973,14 @@ int kvm_arch_init(void *opaque) goto out_free_x86_fpu_cache; } - r = kvm_mmu_module_init(); + r = kvm_isolation_init(); if (r) goto out_free_percpu; + r = kvm_mmu_module_init(); + if (r) + goto out_uninit_isolation; + kvm_set_mmio_spte_mask(); kvm_x86_ops = ops; @@ -7000,6 +7005,8 @@ int kvm_arch_init(void *opaque) return 0; +out_uninit_isolation: + kvm_isolation_uninit(); out_free_percpu: free_percpu(shared_msrs); out_free_x86_fpu_cache: @@ -7024,6 +7031,7 @@ void kvm_arch_exit(void) #ifdef CONFIG_X86_64 pvclock_gtod_unregister_notifier(&pvclock_gtod_notifier); #endif + kvm_isolation_uninit(); kvm_x86_ops = NULL; kvm_mmu_module_exit(); free_percpu(shared_msrs); From patchwork Mon May 13 14:38:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941071 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7629F15AB for ; Mon, 13 May 2019 14:39:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 674421FF62 for ; Mon, 13 May 2019 14:39:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5B353200E7; Mon, 13 May 2019 14:39:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0841F2012F for ; Mon, 13 May 2019 14:39:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730405AbfEMOj6 (ORCPT ); Mon, 13 May 2019 10:39:58 -0400 Received: from userp2130.oracle.com ([156.151.31.86]:45680 "EHLO userp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730320AbfEMOj5 (ORCPT ); Mon, 13 May 2019 10:39:57 -0400 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DESlCI171427; Mon, 13 May 2019 14:38:54 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=GS3GihsPyz7Y99NHOiQDTtmA0apAWQh5Zi8mHVLNyoo=; b=mJcTtw571vBY/DqchePtoK+a2fMaLqNsWff99ts41cz3uW81AVh8k2kIjQIXfkadX5Q3 6F/E6e4/O6myTDqiwNYcUSpAOVMeeIBt5z6sBwlWmvIiYKdfzx0Qg4nsfGs0iR0Ee/ws 8fyDT/ryZpc/Obi/2zRq7DJqs0810bOvF26HLCD7ir3wtStKFUFAa7Ws6PKz+Pu1UJQL dU9UJa5EK8+LYETRyQ5DuH2DbGMtHYVtA2jgoPUK3ziD1XjXnCkAD9TDmm7V4635R9i5 XflzqnTQJehHeZrV5CJoVBXa8ETCmtisHC5rPPYYhukBl0cMbZnPTDNSWhzzDSLkexDm Fw== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2130.oracle.com with ESMTP id 2sdnttfecf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:38:54 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQ7022780; Mon, 13 May 2019 14:38:50 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 04/27] KVM: x86: Switch to KVM address space on entry to guest Date: Mon, 13 May 2019 16:38:12 +0200 Message-Id: <1557758315-12667-5-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130102 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Liran Alon Switch to KVM address space on entry to guest and switch out on immediately at exit (before enabling host interrupts). For now, this is not effectively switching, we just remain on the kernel address space. In addition, we switch back as soon as we exit guest, which makes KVM #VMExit handlers still run with full host address space. However, this introduces the entry points and places for switching. Next commits will change switch to happen only when necessary. Signed-off-by: Liran Alon Signed-off-by: Alexandre Chartre --- arch/x86/kvm/isolation.c | 20 ++++++++++++++++++++ arch/x86/kvm/isolation.h | 2 ++ arch/x86/kvm/x86.c | 8 ++++++++ 3 files changed, 30 insertions(+), 0 deletions(-) diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index 74bc0cd..35aa659 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -119,3 +119,23 @@ void kvm_isolation_uninit(void) kvm_isolation_uninit_mm(); pr_info("KVM: x86: End of isolated address space\n"); } + +void kvm_isolation_enter(void) +{ + if (address_space_isolation) { + /* + * Switches to kvm_mm should happen from vCPU thread, + * which should not be a kernel thread with no mm + */ + BUG_ON(current->active_mm == NULL); + /* TODO: switch to kvm_mm */ + } +} + +void kvm_isolation_exit(void) +{ + if (address_space_isolation) { + /* TODO: Kick sibling hyperthread before switch to host mm */ + /* TODO: switch back to original mm */ + } +} diff --git a/arch/x86/kvm/isolation.h b/arch/x86/kvm/isolation.h index cf8c7d4..595f62c 100644 --- a/arch/x86/kvm/isolation.h +++ b/arch/x86/kvm/isolation.h @@ -4,5 +4,7 @@ extern int kvm_isolation_init(void); extern void kvm_isolation_uninit(void); +extern void kvm_isolation_enter(void); +extern void kvm_isolation_exit(void); #endif diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 4b7cec2..85700e0 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -7896,6 +7896,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) goto cancel_injection; } + kvm_isolation_enter(); + if (req_immediate_exit) { kvm_make_request(KVM_REQ_EVENT, vcpu); kvm_x86_ops->request_immediate_exit(vcpu); @@ -7946,6 +7948,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) vcpu->arch.last_guest_tsc = kvm_read_l1_tsc(vcpu, rdtsc()); + /* + * TODO: Move this to where we architectually need to access + * host (or other VM) sensitive data + */ + kvm_isolation_exit(); + vcpu->mode = OUTSIDE_GUEST_MODE; smp_wmb(); From patchwork Mon May 13 14:38:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941125 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 43C666C5 for ; Mon, 13 May 2019 14:41:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 36B7C28306 for ; Mon, 13 May 2019 14:41:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2A38D2832D; Mon, 13 May 2019 14:41:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C91DD28334 for ; Mon, 13 May 2019 14:41:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730739AbfEMOlj (ORCPT ); Mon, 13 May 2019 10:41:39 -0400 Received: from userp2120.oracle.com ([156.151.31.85]:33390 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729639AbfEMOlj (ORCPT ); Mon, 13 May 2019 10:41:39 -0400 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEd3QK194955; Mon, 13 May 2019 14:39:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=ivlXRYid08bCLk7EeeAFLog7R/4eiJfJ3h7LO8cUdeI=; b=z94gwv/h6a2eKxGWAbRUGkTU6KQJtXJ5d9W6ZAsILmq9BIFlV0LuM3OrSOh/UH7Pz5NG 96qNkm9FPETMU1CK7gybTPwq+ay4+eI/5ZBk7+Q+y/kQOV/x7sjuvp8KNOiVPI3sWHJp aavXu55k8608Ik9cqQUYuPI190YDw9stqWSKV2ovjULIm12NLn2fEyPVAmeiTcuHgcon 8059HBgFrgSpkKo1gDcIsSnBF1vNWz4rpcg52v7RzoR3yGYo4g+LLYhNJsd8LIgK7av1 KVYEFQoIDdxdVwMIjw1yVhHummBBRuJpdrOk80g5n+iCI4jKYptxDL+3/5eTCqCqtgeN 4g== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2sdq1q7atx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:39:03 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQ8022780; Mon, 13 May 2019 14:38:53 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 05/27] KVM: x86: Add handler to exit kvm isolation Date: Mon, 13 May 2019 16:38:13 +0200 Message-Id: <1557758315-12667-6-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Liran Alon Interrupt handlers will need this handler to switch from the KVM address space back to the kernel address space on their prelog. Signed-off-by: Liran Alon Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/irq.h | 1 + arch/x86/kernel/irq.c | 11 +++++++++++ arch/x86/kvm/isolation.c | 13 +++++++++++++ 3 files changed, 25 insertions(+), 0 deletions(-) diff --git a/arch/x86/include/asm/irq.h b/arch/x86/include/asm/irq.h index 8f95686..eb32abc 100644 --- a/arch/x86/include/asm/irq.h +++ b/arch/x86/include/asm/irq.h @@ -29,6 +29,7 @@ static inline int irq_canonicalize(int irq) extern __visible void smp_kvm_posted_intr_ipi(struct pt_regs *regs); extern __visible void smp_kvm_posted_intr_wakeup_ipi(struct pt_regs *regs); extern __visible void smp_kvm_posted_intr_nested_ipi(struct pt_regs *regs); +extern void kvm_set_isolation_exit_handler(void (*handler)(void)); #endif extern void (*x86_platform_ipi_callback)(void); diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index 59b5f2e..e68483b 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -295,6 +295,17 @@ void kvm_set_posted_intr_wakeup_handler(void (*handler)(void)) } EXPORT_SYMBOL_GPL(kvm_set_posted_intr_wakeup_handler); +void (*kvm_isolation_exit_handler)(void) = dummy_handler; + +void kvm_set_isolation_exit_handler(void (*handler)(void)) +{ + if (handler) + kvm_isolation_exit_handler = handler; + else + kvm_isolation_exit_handler = dummy_handler; +} +EXPORT_SYMBOL_GPL(kvm_set_isolation_exit_handler); + /* * Handler for POSTED_INTERRUPT_VECTOR. */ diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index 35aa659..22ff9c2 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -5,6 +5,7 @@ * KVM Address Space Isolation */ +#include #include #include #include @@ -95,6 +96,16 @@ static void kvm_isolation_uninit_mm(void) free_pages((unsigned long)kvm_pgd, PGD_ALLOCATION_ORDER); } +static void kvm_isolation_set_handlers(void) +{ + kvm_set_isolation_exit_handler(kvm_isolation_exit); +} + +static void kvm_isolation_clear_handlers(void) +{ + kvm_set_isolation_exit_handler(NULL); +} + int kvm_isolation_init(void) { int r; @@ -106,6 +117,7 @@ int kvm_isolation_init(void) if (r) return r; + kvm_isolation_set_handlers(); pr_info("KVM: x86: Running with isolated address space\n"); return 0; @@ -116,6 +128,7 @@ void kvm_isolation_uninit(void) if (!address_space_isolation) return; + kvm_isolation_clear_handlers(); kvm_isolation_uninit_mm(); pr_info("KVM: x86: End of isolated address space\n"); } From patchwork Mon May 13 14:38:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941105 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E52C01390 for ; Mon, 13 May 2019 14:40:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D7D0228346 for ; Mon, 13 May 2019 14:40:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CBEFD2835B; Mon, 13 May 2019 14:40:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 77F4F28355 for ; Mon, 13 May 2019 14:40:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730650AbfEMOkm (ORCPT ); Mon, 13 May 2019 10:40:42 -0400 Received: from userp2120.oracle.com ([156.151.31.85]:60450 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730646AbfEMOkl (ORCPT ); Mon, 13 May 2019 10:40:41 -0400 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEd4le195008; Mon, 13 May 2019 14:39:04 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=k+F2ikPI91eVYLszQvHNmFnwWVR/o1Jhouj4HTjs9uc=; b=sPuRCB7AsbZZWT9m3DPio3N73cCl1qZSGvYcOtO5wcwsHy2zHWKroJG67uw/Upj6MB8h as5U+fXDw92G4CC+u+VRhy4iiGPx4okU0TKnt3mZX/RUGaqFhGWJu4ijIoE5Zs9d9SNL u//amNhRfak4bvPH2HOHsboCM+M+dxOBIVZGLn3TN6a2pmSvuViBD6jXP4r86MOGViz9 jN0fBYhsPujDrb6O6Mzol1ztWZq7wn6RT/LFtqlQE5JCw2X/1/QkZ/0lzXFxFT78JDD8 PNKqvAwiyWmQ5SmRKDUE6R4j7mSNKvwv3s4EEuyCXYkMX1kBdTCqgLL8P3W8I/5vm5L5 rw== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2sdq1q7ata-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:39:04 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQ9022780; Mon, 13 May 2019 14:38:56 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 06/27] KVM: x86: Exit KVM isolation on IRQ entry Date: Mon, 13 May 2019 16:38:14 +0200 Message-Id: <1557758315-12667-7-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Liran Alon Next commits will change most of KVM #VMExit handlers to run in KVM isolated address space. Any interrupt handler raised during execution in KVM address space needs to switch back to host address space. This patch makes sure that IRQ handlers will run in full host address space instead of KVM isolated address space. Signed-off-by: Liran Alon Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/apic.h | 4 ++-- arch/x86/include/asm/hardirq.h | 10 ++++++++++ arch/x86/kernel/smp.c | 2 +- arch/x86/platform/uv/tlb_uv.c | 2 +- 4 files changed, 14 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/apic.h b/arch/x86/include/asm/apic.h index 130e81e..606da8f 100644 --- a/arch/x86/include/asm/apic.h +++ b/arch/x86/include/asm/apic.h @@ -515,7 +515,7 @@ static inline unsigned int read_apic_id(void) static inline void entering_irq(void) { irq_enter(); - kvm_set_cpu_l1tf_flush_l1d(); + kvm_cpu_may_access_sensitive_data(); } static inline void entering_ack_irq(void) @@ -528,7 +528,7 @@ static inline void ipi_entering_ack_irq(void) { irq_enter(); ack_APIC_irq(); - kvm_set_cpu_l1tf_flush_l1d(); + kvm_cpu_may_access_sensitive_data(); } static inline void exiting_irq(void) diff --git a/arch/x86/include/asm/hardirq.h b/arch/x86/include/asm/hardirq.h index d9069bb..e082ecb 100644 --- a/arch/x86/include/asm/hardirq.h +++ b/arch/x86/include/asm/hardirq.h @@ -80,4 +80,14 @@ static inline bool kvm_get_cpu_l1tf_flush_l1d(void) static inline void kvm_set_cpu_l1tf_flush_l1d(void) { } #endif /* IS_ENABLED(CONFIG_KVM_INTEL) */ +#ifdef CONFIG_HAVE_KVM +extern void (*kvm_isolation_exit_handler)(void); + +static inline void kvm_cpu_may_access_sensitive_data(void) +{ + kvm_set_cpu_l1tf_flush_l1d(); + kvm_isolation_exit_handler(); +} +#endif + #endif /* _ASM_X86_HARDIRQ_H */ diff --git a/arch/x86/kernel/smp.c b/arch/x86/kernel/smp.c index 04adc8d..b99fda0 100644 --- a/arch/x86/kernel/smp.c +++ b/arch/x86/kernel/smp.c @@ -261,7 +261,7 @@ __visible void __irq_entry smp_reschedule_interrupt(struct pt_regs *regs) { ack_APIC_irq(); inc_irq_stat(irq_resched_count); - kvm_set_cpu_l1tf_flush_l1d(); + kvm_cpu_may_access_sensitive_data(); if (trace_resched_ipi_enabled()) { /* diff --git a/arch/x86/platform/uv/tlb_uv.c b/arch/x86/platform/uv/tlb_uv.c index 1297e18..83a17ca 100644 --- a/arch/x86/platform/uv/tlb_uv.c +++ b/arch/x86/platform/uv/tlb_uv.c @@ -1285,7 +1285,7 @@ void uv_bau_message_interrupt(struct pt_regs *regs) struct msg_desc msgdesc; ack_APIC_irq(); - kvm_set_cpu_l1tf_flush_l1d(); + kvm_cpu_may_access_sensitive_data(); time_start = get_cycles(); bcp = &per_cpu(bau_control, smp_processor_id()); From patchwork Mon May 13 14:38:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941141 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A93FD6C5 for ; Mon, 13 May 2019 14:42:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9D34427861 for ; Mon, 13 May 2019 14:42:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 915792832D; Mon, 13 May 2019 14:42:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2D38C27861 for ; Mon, 13 May 2019 14:42:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730256AbfEMOjp (ORCPT ); Mon, 13 May 2019 10:39:45 -0400 Received: from userp2130.oracle.com ([156.151.31.86]:45408 "EHLO userp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730235AbfEMOjo (ORCPT ); Mon, 13 May 2019 10:39:44 -0400 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEd6qH181584; Mon, 13 May 2019 14:39:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=wqdeci0HyBGB7YKvZMkvhcTkiB8MUENeuIRahUviMQQ=; b=qAq8bRmqjLI4ZmGhxqxLQzFJ+Y7ZGe9B4tzqFa4rSPttIDYPwO8CyJSVFyohA4WGqJmF PiEooyxx92pLb5i0rofSV4dZw0lAf84hAXIH/yxG0GncuXG6InmO4dJUyzjA+ryXZa3+ kpAk+nls/3Op+xDAehiDEtWmVkdLl1RWyWJ2r/dqXCkn+On4KPyzKvKpd3t0q+Al/+8t OnREbE+Cqw+DpqL4613DUOyPi8UvLz/4anaHm7nytPek57wpfSd6QSOe3Wz14bi6e5IO vMHSdlEGcaFkmj7ADxMP0X19oYbCmNWG6TEextjbaF9y0FnEIo95HqGb3w9D+rgQ3QCW PQ== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2130.oracle.com with ESMTP id 2sdnttfeff-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:39:07 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQA022780; Mon, 13 May 2019 14:38:59 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 07/27] KVM: x86: Switch to host address space when may access sensitive data Date: Mon, 13 May 2019 16:38:15 +0200 Message-Id: <1557758315-12667-8-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=851 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Liran Alon Before this patch, we exited from KVM isolated address space to host address space as soon as we exit guest. Change code such that most of KVM #VMExit handlers will run in KVM isolated address space and switch back to host address space only before accessing sensitive data. Sensitive data is defined as either host data or other VM data. Currently, we switch from kvm_mm to host_mm on the following scenarios: 1) When handling guest page-faults: As this will access SPTs which contains host PFNs. 2) On schedule-out of vCPU thread 3) On write to guest virtual memory (kvm_write_guest_virt_system() can pull in tons of pages) 4) On return to userspace (e.g. QEMU) 5) On prelog of IRQ handlers Signed-off-by: Liran Alon Signed-off-by: Alexandre Chartre --- arch/x86/kvm/isolation.c | 7 ++++++- arch/x86/kvm/isolation.h | 3 +++ arch/x86/kvm/mmu.c | 3 ++- arch/x86/kvm/x86.c | 12 +++++------- 4 files changed, 16 insertions(+), 9 deletions(-) diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index 22ff9c2..eeb60c4 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -5,7 +5,6 @@ * KVM Address Space Isolation */ -#include #include #include #include @@ -133,6 +132,12 @@ void kvm_isolation_uninit(void) pr_info("KVM: x86: End of isolated address space\n"); } +void kvm_may_access_sensitive_data(struct kvm_vcpu *vcpu) +{ + vcpu->arch.l1tf_flush_l1d = true; + kvm_isolation_exit(); +} + void kvm_isolation_enter(void) { if (address_space_isolation) { diff --git a/arch/x86/kvm/isolation.h b/arch/x86/kvm/isolation.h index 595f62c..1290d32 100644 --- a/arch/x86/kvm/isolation.h +++ b/arch/x86/kvm/isolation.h @@ -2,9 +2,12 @@ #ifndef ARCH_X86_KVM_ISOLATION_H #define ARCH_X86_KVM_ISOLATION_H +#include + extern int kvm_isolation_init(void); extern void kvm_isolation_uninit(void); extern void kvm_isolation_enter(void); extern void kvm_isolation_exit(void); +extern void kvm_may_access_sensitive_data(struct kvm_vcpu *vcpu); #endif diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c index d9c7b45..a2b38de 100644 --- a/arch/x86/kvm/mmu.c +++ b/arch/x86/kvm/mmu.c @@ -23,6 +23,7 @@ #include "x86.h" #include "kvm_cache_regs.h" #include "cpuid.h" +#include "isolation.h" #include #include @@ -4059,7 +4060,7 @@ int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code, { int r = 1; - vcpu->arch.l1tf_flush_l1d = true; + kvm_may_access_sensitive_data(vcpu); switch (vcpu->arch.apf.host_apf_reason) { default: trace_kvm_page_fault(fault_address, error_code); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 85700e0..1db72c3 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -3307,6 +3307,8 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) * guest. do_debug expects dr6 to be cleared after it runs, do the same. */ set_debugreg(0, 6); + + kvm_may_access_sensitive_data(vcpu); } static int kvm_vcpu_ioctl_get_lapic(struct kvm_vcpu *vcpu, @@ -5220,7 +5222,7 @@ int kvm_write_guest_virt_system(struct kvm_vcpu *vcpu, gva_t addr, void *val, unsigned int bytes, struct x86_exception *exception) { /* kvm_write_guest_virt_system can pull in tons of pages. */ - vcpu->arch.l1tf_flush_l1d = true; + kvm_may_access_sensitive_data(vcpu); return kvm_write_guest_virt_helper(addr, val, bytes, vcpu, PFERR_WRITE_MASK, exception); @@ -7948,12 +7950,6 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) vcpu->arch.last_guest_tsc = kvm_read_l1_tsc(vcpu, rdtsc()); - /* - * TODO: Move this to where we architectually need to access - * host (or other VM) sensitive data - */ - kvm_isolation_exit(); - vcpu->mode = OUTSIDE_GUEST_MODE; smp_wmb(); @@ -8086,6 +8082,8 @@ static int vcpu_run(struct kvm_vcpu *vcpu) srcu_read_unlock(&kvm->srcu, vcpu->srcu_idx); + kvm_may_access_sensitive_data(vcpu); + return r; } From patchwork Mon May 13 14:38:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941137 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9B7C06C5 for ; Mon, 13 May 2019 14:42:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8D2B227861 for ; Mon, 13 May 2019 14:42:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 80BB22832D; Mon, 13 May 2019 14:42:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2E5F227861 for ; Mon, 13 May 2019 14:42:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730310AbfEMOjp (ORCPT ); Mon, 13 May 2019 10:39:45 -0400 Received: from userp2120.oracle.com ([156.151.31.85]:59418 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730144AbfEMOjo (ORCPT ); Mon, 13 May 2019 10:39:44 -0400 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEd2YY194903; Mon, 13 May 2019 14:39:05 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=BckiUE2qwd6ce7s4e94ZCEH08ViWVb2Uhow4MmaOmmg=; b=MqxRlocDO4PBVSKI4njGqiBJNQ3Dwy7bXoxOg336yVqnxDPoOo4C8bDcoA+hRinUelY/ IfBGNpsi5hC0VByASVpiWMEgObQsMVJz2ebCqcSbAU7+bP+dXw4f4OP72FR/CyrWZMzm C5l9JlMKOqhD3L7yBz6qxyBISnDBFSOI418qbfcikgzUXjTPisA4nha1WqGfH00pB+fR MAPL3suB/BEgJ7y+NOdtYN4R08kIlmFj7UCcos/Hh23ufmYdF9aqYoILoddmclvElloo MtVyLdQhfuOp3oe6RJY+QcC2ZMynhm3bA4JOkdVlLXIFA7MOvRKhnhe+hFi0uBip4m9o lw== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2sdq1q7aum-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:39:05 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQB022780; Mon, 13 May 2019 14:39:02 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 08/27] KVM: x86: Optimize branches which checks if address space isolation enabled Date: Mon, 13 May 2019 16:38:16 +0200 Message-Id: <1557758315-12667-9-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Liran Alon As every entry to guest checks if should switch from host_mm to kvm_mm, these branches is at very hot path. Optimize them by using static_branch. Signed-off-by: Liran Alon Signed-off-by: Alexandre Chartre --- arch/x86/kvm/isolation.c | 11 ++++++++--- arch/x86/kvm/isolation.h | 7 +++++++ 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index eeb60c4..43fd924 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -23,6 +23,9 @@ struct mm_struct kvm_mm = { .mmlist = LIST_HEAD_INIT(kvm_mm.mmlist), }; +DEFINE_STATIC_KEY_FALSE(kvm_isolation_enabled); +EXPORT_SYMBOL(kvm_isolation_enabled); + /* * When set to true, KVM #VMExit handlers run in isolated address space * which maps only KVM required code and per-VM information instead of @@ -118,15 +121,17 @@ int kvm_isolation_init(void) kvm_isolation_set_handlers(); pr_info("KVM: x86: Running with isolated address space\n"); + static_branch_enable(&kvm_isolation_enabled); return 0; } void kvm_isolation_uninit(void) { - if (!address_space_isolation) + if (!kvm_isolation()) return; + static_branch_disable(&kvm_isolation_enabled); kvm_isolation_clear_handlers(); kvm_isolation_uninit_mm(); pr_info("KVM: x86: End of isolated address space\n"); @@ -140,7 +145,7 @@ void kvm_may_access_sensitive_data(struct kvm_vcpu *vcpu) void kvm_isolation_enter(void) { - if (address_space_isolation) { + if (kvm_isolation()) { /* * Switches to kvm_mm should happen from vCPU thread, * which should not be a kernel thread with no mm @@ -152,7 +157,7 @@ void kvm_isolation_enter(void) void kvm_isolation_exit(void) { - if (address_space_isolation) { + if (kvm_isolation()) { /* TODO: Kick sibling hyperthread before switch to host mm */ /* TODO: switch back to original mm */ } diff --git a/arch/x86/kvm/isolation.h b/arch/x86/kvm/isolation.h index 1290d32..aa5e979 100644 --- a/arch/x86/kvm/isolation.h +++ b/arch/x86/kvm/isolation.h @@ -4,6 +4,13 @@ #include +DECLARE_STATIC_KEY_FALSE(kvm_isolation_enabled); + +static inline bool kvm_isolation(void) +{ + return static_branch_likely(&kvm_isolation_enabled); +} + extern int kvm_isolation_init(void); extern void kvm_isolation_uninit(void); extern void kvm_isolation_enter(void); From patchwork Mon May 13 14:38:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941113 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 32AC06C5 for ; Mon, 13 May 2019 14:41:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 259F827861 for ; Mon, 13 May 2019 14:41:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1917028334; Mon, 13 May 2019 14:41:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9E3F027861 for ; Mon, 13 May 2019 14:41:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730669AbfEMOkz (ORCPT ); Mon, 13 May 2019 10:40:55 -0400 Received: from aserp2130.oracle.com ([141.146.126.79]:38294 "EHLO aserp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729639AbfEMOky (ORCPT ); Mon, 13 May 2019 10:40:54 -0400 Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEd2hT193056; Mon, 13 May 2019 14:39:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=lqyn+UMxH2H0mFstbeYlyK/O28QxJu29o92QL7NqRo0=; b=Jr5TCZD/u/xNJPZvrSFuZJzX/+fkjXUeKcGKj+LAFsPLKhL5qgqU35GRnUUc8xQu8pLv kJ2cgm2Uaj8fWZEV+R26AC0h34/IHr0xch9O9VC8DBfJZs45MDHArPHOrKH2BZgkZn0g Vo1Qsw+WtWbru1nSWeUWRDzk0S8BMwbmJOSxY9hDJpCKuZ4Rjcl9KR+8ZAUPwwUnSucl F6NkZr7jZnrzkGGfd72bek1kv1u5Rb/uWnD+/v6jia+yUhHtbKzvABFRbX3KACRbuEhG GYu3aCArrPJv0oQqS9LptBfUuYpEaXqIKNe+OKRLzFVYsoDNcshbQCQH2UMO7meQY53G qQ== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by aserp2130.oracle.com with ESMTP id 2sdkwdfkvq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:39:08 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQC022780; Mon, 13 May 2019 14:39:05 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 09/27] kvm/isolation: function to track buffers allocated for the KVM page table Date: Mon, 13 May 2019 16:38:17 +0200 Message-Id: <1557758315-12667-10-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The KVM page table will have direct references to the kernel page table, at different levels (PGD, P4D, PUD, PMD). When freeing the KVM page table, we should make sure that we free parts actually allocated for the KVM page table, and not parts of the kernel page table referenced from the KVM page table. To do so, we will keep track of buffers when building the KVM page table. Signed-off-by: Alexandre Chartre --- arch/x86/kvm/isolation.c | 119 ++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 119 insertions(+), 0 deletions(-) diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index 43fd924..1efdab1 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -8,12 +8,60 @@ #include #include #include +#include #include #include #include "isolation.h" + +enum page_table_level { + PGT_LEVEL_PTE, + PGT_LEVEL_PMD, + PGT_LEVEL_PUD, + PGT_LEVEL_P4D, + PGT_LEVEL_PGD +}; + +/* + * The KVM page table can have direct references to the kernel page table, + * at different levels (PGD, P4D, PUD, PMD). When freeing the KVM page + * table, we should make sure that we free parts actually allocated for + * the KVM page table, and not parts of the kernel page table referenced + * from the KVM page table. + * + * To do so, page table directories (struct pgt_directory) are used to keep + * track of buffers allocated when building the KVM page table. Also, as + * a page table can have many buffers, page table directory groups (struct + * (pgt_directory_group) are used to group page table directories and save + * some space (instead of allocating each directory individually). + */ + +#define PGT_DIRECTORY_GROUP_SIZE 64 + +struct pgt_directory { + enum page_table_level level; + void *ptr; +}; + +struct pgt_directory_group { + struct list_head list; + int count; + struct pgt_directory directory[PGT_DIRECTORY_GROUP_SIZE]; +}; + +static LIST_HEAD(kvm_pgt_dgroup_list); +static DEFINE_MUTEX(kvm_pgt_dgroup_lock); + +/* + * Get the pointer to the beginning of a page table directory from a page + * table directory entry. + */ +#define PGTD_ALIGN(entry) \ + ((typeof(entry))(((unsigned long)(entry)) & PAGE_MASK)) + + struct mm_struct kvm_mm = { .mm_rb = RB_ROOT, .mm_users = ATOMIC_INIT(2), @@ -43,6 +91,77 @@ struct mm_struct kvm_mm = { static bool __read_mostly address_space_isolation; module_param(address_space_isolation, bool, 0444); + +static struct pgt_directory_group *pgt_directory_group_create(void) +{ + struct pgt_directory_group *dgroup; + + dgroup = kzalloc(sizeof(struct pgt_directory_group), GFP_KERNEL); + if (!dgroup) + return NULL; + + INIT_LIST_HEAD(&dgroup->list); + dgroup->count = 0; + + return dgroup; +} + +static bool kvm_add_pgt_directory(void *ptr, enum page_table_level level) +{ + struct pgt_directory_group *dgroup; + int index; + + mutex_lock(&kvm_pgt_dgroup_lock); + + if (list_empty(&kvm_pgt_dgroup_list)) + dgroup = NULL; + else + dgroup = list_entry(kvm_pgt_dgroup_list.next, + struct pgt_directory_group, list); + + if (!dgroup || dgroup->count >= PGT_DIRECTORY_GROUP_SIZE) { + dgroup = pgt_directory_group_create(); + if (!dgroup) { + mutex_unlock(&kvm_pgt_dgroup_lock); + return false; + } + list_add_tail(&dgroup->list, &kvm_pgt_dgroup_list); + } + + index = dgroup->count; + dgroup->directory[index].level = level; + dgroup->directory[index].ptr = PGTD_ALIGN(ptr); + dgroup->count = index + 1; + + mutex_unlock(&kvm_pgt_dgroup_lock); + + return true; +} + +static bool kvm_valid_pgt_entry(void *ptr) +{ + struct pgt_directory_group *dgroup; + int i; + + mutex_lock(&kvm_pgt_dgroup_lock); + + ptr = PGTD_ALIGN(ptr); + list_for_each_entry(dgroup, &kvm_pgt_dgroup_list, list) { + for (i = 0; i < dgroup->count; i++) { + if (dgroup->directory[i].ptr == ptr) { + mutex_unlock(&kvm_pgt_dgroup_lock); + return true; + } + } + } + + mutex_unlock(&kvm_pgt_dgroup_lock); + + return false; + +} + + static int kvm_isolation_init_mm(void) { pgd_t *kvm_pgd; From patchwork Mon May 13 14:38:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941139 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B0A6D6C5 for ; Mon, 13 May 2019 14:42:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A361B27861 for ; Mon, 13 May 2019 14:42:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 96A8D2832D; Mon, 13 May 2019 14:42:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4FA8627861 for ; Mon, 13 May 2019 14:42:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730298AbfEMOjp (ORCPT ); Mon, 13 May 2019 10:39:45 -0400 Received: from userp2120.oracle.com ([156.151.31.85]:59426 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730253AbfEMOjo (ORCPT ); Mon, 13 May 2019 10:39:44 -0400 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEd84M195057; Mon, 13 May 2019 14:39:11 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=4cIDR6PZl2rutLAEYSA22jeItW1PJPdhgsRn6XGFWEA=; b=VC6jjd4Q/95/JOy5PgCZ8AdKsMFlE4fQCQOC2dyJBtX/9iwCN4mgh4f7m5XZlJT41CUT CWiyU1PZYsAhyoP4PcbGhS++vcss56Xjgewi7dXzHWILBy4ssid7xPlSLghegxyz3MzM lRsyU6oTrYGTNnLa5CF5DZEESUPecwfArax8wnk/8fA+g1/lxpeUfKmiNGTDKimHFL0k BUWpMG9d2DNNstgdU8bqlCEnHiwjnEh1BuYVAwNxdf4+s9KmtLq6O/FzAGHQaEbNfMUf 6YUvBPOPUK+XTc5r2FauxEIQJUNK5TtoYUDMw6GnP5h/kShdnJWw0oXvReVp36peQCHR rw== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2sdq1q7avk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:39:11 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQD022780; Mon, 13 May 2019 14:39:08 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 10/27] kvm/isolation: add KVM page table entry free functions Date: Mon, 13 May 2019 16:38:18 +0200 Message-Id: <1557758315-12667-11-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP These functions are wrappers around the p4d/pud/pmd/pte free function which can be used with any pointer in the directory. Signed-off-by: Alexandre Chartre --- arch/x86/kvm/isolation.c | 26 ++++++++++++++++++++++++++ 1 files changed, 26 insertions(+), 0 deletions(-) diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index 1efdab1..61df750 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -161,6 +161,32 @@ static bool kvm_valid_pgt_entry(void *ptr) } +/* + * kvm_pXX_free() functions are equivalent to kernel pXX_free() + * functions but they can be used with any PXX pointer in the + * directory. + */ + +static inline void kvm_pte_free(struct mm_struct *mm, pte_t *pte) +{ + pte_free_kernel(mm, PGTD_ALIGN(pte)); +} + +static inline void kvm_pmd_free(struct mm_struct *mm, pmd_t *pmd) +{ + pmd_free(mm, PGTD_ALIGN(pmd)); +} + +static inline void kvm_pud_free(struct mm_struct *mm, pud_t *pud) +{ + pud_free(mm, PGTD_ALIGN(pud)); +} + +static inline void kvm_p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + p4d_free(mm, PGTD_ALIGN(p4d)); +} + static int kvm_isolation_init_mm(void) { From patchwork Mon May 13 14:38:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941069 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0FFBB6C5 for ; Mon, 13 May 2019 14:39:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 025621FF62 for ; Mon, 13 May 2019 14:39:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EA896200E7; Mon, 13 May 2019 14:39:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9C2B71FF62 for ; Mon, 13 May 2019 14:39:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730378AbfEMOjx (ORCPT ); Mon, 13 May 2019 10:39:53 -0400 Received: from userp2120.oracle.com ([156.151.31.85]:59466 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730320AbfEMOjr (ORCPT ); Mon, 13 May 2019 10:39:47 -0400 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEd4lg195008; Mon, 13 May 2019 14:39:14 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=gIeFN0ziPc52a8KLvMF4BYUuKgdMORhtc90dfJFFcK8=; b=hG+EotkwfW0afgyQ6GQFcjNoUZvzVSUEOYaoXlfVoV2cUsog6OXPvWLhJn7Oel/wqLNU q93k3vhzcUlchxQbCpQ1datYttq5bU2CVnCcfgAzotf47Y8hQH+8rBgoN2OmMBnGwcKO LEQEsJMpMtvD5hREw/USDtI1pRBmvG/plElU+Xrot/aCMJUcLyH6vUNMyX9iYSD4MPrI Y0R0Lk4RBx9aTbJnKTE11caIJKBYXFjCKw9NZy0zBsZERP6ZtLnDVZy0YizHnM93iBtI eV66j3vQdt+zdewHBfSeob4gdmSsVVx7BhZaz3q+MBojcv1hUk06rRc6rPcuwhVdDGFV rg== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2sdq1q7avt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:39:14 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQE022780; Mon, 13 May 2019 14:39:11 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 11/27] kvm/isolation: add KVM page table entry offset functions Date: Mon, 13 May 2019 16:38:19 +0200 Message-Id: <1557758315-12667-12-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP These functions are wrappers are the p4d/pud/pmd/pte offset functions which ensure that page table pointers are in the KVM page table. Signed-off-by: Alexandre Chartre --- arch/x86/kvm/isolation.c | 61 ++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 61 insertions(+), 0 deletions(-) diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index 61df750..b29a09b 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -162,6 +162,67 @@ static bool kvm_valid_pgt_entry(void *ptr) } /* + * kvm_pXX_offset() functions are equivalent to kernel pXX_offset() + * functions but, in addition, they ensure that page table pointers + * are in the KVM page table. Otherwise an error is returned. + */ + +static pte_t *kvm_pte_offset(pmd_t *pmd, unsigned long addr) +{ + pte_t *pte; + + pte = pte_offset_map(pmd, addr); + if (!kvm_valid_pgt_entry(pte)) { + pr_err("PTE %px is not in KVM page table\n", pte); + return ERR_PTR(-EINVAL); + } + + return pte; +} + +static pmd_t *kvm_pmd_offset(pud_t *pud, unsigned long addr) +{ + pmd_t *pmd; + + pmd = pmd_offset(pud, addr); + if (!kvm_valid_pgt_entry(pmd)) { + pr_err("PMD %px is not in KVM page table\n", pmd); + return ERR_PTR(-EINVAL); + } + + return pmd; +} + +static pud_t *kvm_pud_offset(p4d_t *p4d, unsigned long addr) +{ + pud_t *pud; + + pud = pud_offset(p4d, addr); + if (!kvm_valid_pgt_entry(pud)) { + pr_err("PUD %px is not in KVM page table\n", pud); + return ERR_PTR(-EINVAL); + } + + return pud; +} + +static p4d_t *kvm_p4d_offset(pgd_t *pgd, unsigned long addr) +{ + p4d_t *p4d; + + p4d = p4d_offset(pgd, addr); + /* + * p4d is the same has pgd if we don't have a 5-level page table. + */ + if ((p4d != (p4d_t *)pgd) && !kvm_valid_pgt_entry(p4d)) { + pr_err("P4D %px is not in KVM page table\n", p4d); + return ERR_PTR(-EINVAL); + } + + return p4d; +} + +/* * kvm_pXX_free() functions are equivalent to kernel pXX_free() * functions but they can be used with any PXX pointer in the * directory. From patchwork Mon May 13 14:38:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941121 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 43E8E1390 for ; Mon, 13 May 2019 14:41:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3670D27861 for ; Mon, 13 May 2019 14:41:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2A27C2832D; Mon, 13 May 2019 14:41:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CE01F27861 for ; Mon, 13 May 2019 14:41:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730452AbfEMOkB (ORCPT ); Mon, 13 May 2019 10:40:01 -0400 Received: from aserp2130.oracle.com ([141.146.126.79]:37252 "EHLO aserp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730420AbfEMOkA (ORCPT ); Mon, 13 May 2019 10:40:00 -0400 Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEd28F193025; Mon, 13 May 2019 14:39:16 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=3oQZsnmQ7NshrUK5gBFOiQdwJCmvyMk7rTDgLZ4eiR8=; b=cackC5e/rN14Atf5QlnsuYve24xrmHKk5OAv4LozaAp9vhiUL7E7xPH3kbnOM8VYLH+M qQZDMDajYAWzXDX4pYfxSi/cFR9+miTzGtpQvwsGANTuO936Hac7155wikox7cO9zHM6 D6VSW0wFlH4kS9hhjEHDuhBzZL61Y1YPaaQ9QapuqevDj5phnii9KR4rMc6oFyvj+0cK chnwMcSACXVNjOAa1xqmZO14HcVVu9E4v0xi+ZQhdjbaq/Q1oelYLJg/hz12bN7CBmtg 1H1C0ZDiqvGmCSUtMdjVlrxOudApkaffggciBhogmTfWK5aks2i24a2IdgYkJ3LSxlPk OA== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by aserp2130.oracle.com with ESMTP id 2sdkwdfkwq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:39:16 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQF022780; Mon, 13 May 2019 14:39:13 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 12/27] kvm/isolation: add KVM page table entry allocation functions Date: Mon, 13 May 2019 16:38:20 +0200 Message-Id: <1557758315-12667-13-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP These functions allocate p4d/pud/pmd/pte pages and ensure that pages are in the KVM page table. Signed-off-by: Alexandre Chartre --- arch/x86/kvm/isolation.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 94 insertions(+), 0 deletions(-) diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index b29a09b..6ec86df 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -248,6 +248,100 @@ static inline void kvm_p4d_free(struct mm_struct *mm, p4d_t *p4d) p4d_free(mm, PGTD_ALIGN(p4d)); } +/* + * kvm_pXX_alloc() functions are equivalent to kernel pXX_alloc() + * functions but, in addition, they ensure that page table pointers + * are in the KVM page table. Otherwise an error is returned. + */ + +static pte_t *kvm_pte_alloc(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr) +{ + pte_t *pte; + + if (pmd_none(*pmd)) { + pte = pte_alloc_kernel(pmd, addr); + if (!pte) { + pr_debug("PTE: ERR ALLOC\n"); + return ERR_PTR(-ENOMEM); + } + if (!kvm_add_pgt_directory(pte, PGT_LEVEL_PTE)) { + kvm_pte_free(mm, pte); + return ERR_PTR(-EINVAL); + } + } else { + pte = kvm_pte_offset(pmd, addr); + } + + return pte; +} + +static pmd_t *kvm_pmd_alloc(struct mm_struct *mm, pud_t *pud, + unsigned long addr) +{ + pmd_t *pmd; + + if (pud_none(*pud)) { + pmd = pmd_alloc(mm, pud, addr); + if (!pmd) { + pr_debug("PMD: ERR ALLOC\n"); + return ERR_PTR(-ENOMEM); + } + if (!kvm_add_pgt_directory(pmd, PGT_LEVEL_PMD)) { + kvm_pmd_free(mm, pmd); + return ERR_PTR(-EINVAL); + } + } else { + pmd = kvm_pmd_offset(pud, addr); + } + + return pmd; +} + +static pud_t *kvm_pud_alloc(struct mm_struct *mm, p4d_t *p4d, + unsigned long addr) +{ + pud_t *pud; + + if (p4d_none(*p4d)) { + pud = pud_alloc(mm, p4d, addr); + if (!pud) { + pr_debug("PUD: ERR ALLOC\n"); + return ERR_PTR(-ENOMEM); + } + if (!kvm_add_pgt_directory(pud, PGT_LEVEL_PUD)) { + kvm_pud_free(mm, pud); + return ERR_PTR(-EINVAL); + } + } else { + pud = kvm_pud_offset(p4d, addr); + } + + return pud; +} + +static p4d_t *kvm_p4d_alloc(struct mm_struct *mm, pgd_t *pgd, + unsigned long addr) +{ + p4d_t *p4d; + + if (pgd_none(*pgd)) { + p4d = p4d_alloc(mm, pgd, addr); + if (!p4d) { + pr_debug("P4D: ERR ALLOC\n"); + return ERR_PTR(-ENOMEM); + } + if (!kvm_add_pgt_directory(p4d, PGT_LEVEL_P4D)) { + kvm_p4d_free(mm, p4d); + return ERR_PTR(-EINVAL); + } + } else { + p4d = kvm_p4d_offset(pgd, addr); + } + + return p4d; +} + static int kvm_isolation_init_mm(void) { From patchwork Mon May 13 14:38:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941075 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 68D946C5 for ; Mon, 13 May 2019 14:40:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 58E0427BA5 for ; Mon, 13 May 2019 14:40:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4C9D927FA1; Mon, 13 May 2019 14:40:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E0F0D27BA5 for ; Mon, 13 May 2019 14:40:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730514AbfEMOkC (ORCPT ); Mon, 13 May 2019 10:40:02 -0400 Received: from aserp2130.oracle.com ([141.146.126.79]:37302 "EHLO aserp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730447AbfEMOkC (ORCPT ); Mon, 13 May 2019 10:40:02 -0400 Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEd3jQ193102; Mon, 13 May 2019 14:39:24 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=5Qtz3s/XQzDItLwgII6a7yqyolD8+y7jKItUNNZ5jHw=; b=bc0UAXJuEUDdxvyu9LwCdXHqD4qh6pNojWMqcM9jSLBAxGM6igASeeKPL3IWePb5ch0s fbCFf/kSYwqFzCOuWD0pbNuB78gAyLFK5Ed4veTVvUvCkVI2DR43VeRRk0g9ufQAr0sw e6QhuGNeEnywPiXsoWdU2luDBNsmNNUqdcfqgUJyVy51fsmTOMsXHwAuC4cslhey/L9A UYFWoKFzlhctdRm9y9d9iC7Djqvsf9+FCNhU/NbXFxMpjmDvfMOaaoR1Y5lpKRNKz4fW VMMMFGBaxysR4n+y/P893dNcj5PpeJwOEyHs1Tnsryx4DvPOgcmS7i7F6ug4vAd8pZZL 3A== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by aserp2130.oracle.com with ESMTP id 2sdkwdfkxg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:39:24 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQG022780; Mon, 13 May 2019 14:39:16 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 13/27] kvm/isolation: add KVM page table entry set functions Date: Mon, 13 May 2019 16:38:21 +0200 Message-Id: <1557758315-12667-14-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add wrappers around the page table entry (pgd/p4d/pud/pmd) set function to check that an existing entry is not being overwritten. Signed-off-by: Alexandre Chartre --- arch/x86/kvm/isolation.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 107 insertions(+), 0 deletions(-) diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index 6ec86df..b681e4f 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -342,6 +342,113 @@ static inline void kvm_p4d_free(struct mm_struct *mm, p4d_t *p4d) return p4d; } +/* + * kvm_set_pXX() functions are equivalent to kernel set_pXX() functions + * but, in addition, they ensure that they are not overwriting an already + * existing reference in the page table. Otherwise an error is returned. + * + * Note that this is not used for PTE because a PTE entry points to page + * frames containing the actual user data, and not to another entry in the + * page table. However this is used for PGD. + */ + +static int kvm_set_pmd(pmd_t *pmd, pmd_t pmd_value) +{ +#ifdef DEBUG + /* + * The pmd pointer should come from kvm_pmd_alloc() or kvm_pmd_offset() + * both of which check if the pointer is in the KVM page table. So this + * is a paranoid check to ensure the pointer is really in the KVM page + * table. + */ + if (!kvm_valid_pgt_entry(pmd)) { + pr_err("PMD %px is not in KVM page table\n", pmd); + return -EINVAL; + } +#endif + if (pmd_val(*pmd) == pmd_val(pmd_value)) + return 0; + + if (!pmd_none(*pmd)) { + pr_err("PMD %px: overwriting %lx with %lx\n", + pmd, pmd_val(*pmd), pmd_val(pmd_value)); + return -EBUSY; + } + + set_pmd(pmd, pmd_value); + + return 0; +} + +static int kvm_set_pud(pud_t *pud, pud_t pud_value) +{ +#ifdef DEBUG + /* + * The pud pointer should come from kvm_pud_alloc() or kvm_pud_offset() + * both of which check if the pointer is in the KVM page table. So this + * is a paranoid check to ensure the pointer is really in the KVM page + * table. + */ + if (!kvm_valid_pgt_entry(pud)) { + pr_err("PUD %px is not in KVM page table\n", pud); + return -EINVAL; + } +#endif + if (pud_val(*pud) == pud_val(pud_value)) + return 0; + + if (!pud_none(*pud)) { + pr_err("PUD %px: overwriting %lx\n", pud, pud_val(*pud)); + return -EBUSY; + } + + set_pud(pud, pud_value); + + return 0; +} + +static int kvm_set_p4d(p4d_t *p4d, p4d_t p4d_value) +{ +#ifdef DEBUG + /* + * The p4d pointer should come from kvm_p4d_alloc() or kvm_p4d_offset() + * both of which check if the pointer is in the KVM page table. So this + * is a paranoid check to ensure the pointer is really in the KVM page + * table. + */ + if (!kvm_valid_pgt_entry(p4d)) { + pr_err("P4D %px is not in KVM page table\n", p4d); + return -EINVAL; + } +#endif + if (p4d_val(*p4d) == p4d_val(p4d_value)) + return 0; + + if (!p4d_none(*p4d)) { + pr_err("P4D %px: overwriting %lx\n", p4d, p4d_val(*p4d)); + return -EBUSY; + } + + set_p4d(p4d, p4d_value); + + return 0; +} + +static int kvm_set_pgd(pgd_t *pgd, pgd_t pgd_value) +{ + if (pgd_val(*pgd) == pgd_val(pgd_value)) + return 0; + + if (!pgd_none(*pgd)) { + pr_err("PGD %px: overwriting %lx\n", pgd, pgd_val(*pgd)); + return -EBUSY; + } + + set_pgd(pgd, pgd_value); + + return 0; +} + static int kvm_isolation_init_mm(void) { From patchwork Mon May 13 14:38:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941131 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 71D436C5 for ; Mon, 13 May 2019 14:42:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6499828306 for ; Mon, 13 May 2019 14:42:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 58D6C28334; Mon, 13 May 2019 14:42:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B736528306 for ; Mon, 13 May 2019 14:42:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730436AbfEMOld (ORCPT ); Mon, 13 May 2019 10:41:33 -0400 Received: from userp2120.oracle.com ([156.151.31.85]:59706 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730424AbfEMOkA (ORCPT ); Mon, 13 May 2019 10:40:00 -0400 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEd2pd194892; Mon, 13 May 2019 14:39:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=HIfaAaJqDpQgZqOlqv93OYEN222yx/NWgo2RgdnE7Ys=; b=rJqFVfy/BraQo2VV+vTf4vEU8uKTxNMJYqNUMPJntKe3SPbYIqJswWfe8FSX9DOUDUvq NMeBPV/EFppUnPmGQYbDoMnmg/dC+wvoFJX/MAxWhRp9eR7c70Ir05eJ9ObgYhuTZIN0 D6O2EwvBsR4WuKaiukLz+63YyUEe8qIjPm7QE5ehY3t9v0MmShxoS6X2fFxg5nPXKcjD 5e42nLFl9jyGEOT3WQUgTW0ZEa0BSwnYUeiOjo0jQFfif/6jx2982K9zu7DiWOea0CSj W4Xl7iGJz5g3uHt6WP7akNTkktkKiod+MAJFVIj50sV+mpTRrNCEYkdwKX02CtXlEBbR ag== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2sdq1q7aws-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:39:23 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQH022780; Mon, 13 May 2019 14:39:19 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 14/27] kvm/isolation: functions to copy page table entries for a VA range Date: Mon, 13 May 2019 16:38:22 +0200 Message-Id: <1557758315-12667-15-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP These functions are based on the copy_pxx_range() functions defined in mm/memory.c. The main difference is that a level parameter is specified to indicate the page table level (PGD, P4D, PUD PMD, PTE) at which the copy should be done. Also functions don't use a vma parameter, and don't alter the source page table even if an entry is bad. Also kvm_copy_pte_range() can be called with a non page-aligned buffer, so the buffer should be aligned with the page start so that the entire buffer is mapped if the end of buffer crosses a page. These functions will be used to populate the KVM page table. Signed-off-by: Alexandre Chartre --- arch/x86/kvm/isolation.c | 229 ++++++++++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/isolation.h | 1 + 2 files changed, 230 insertions(+), 0 deletions(-) diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index b681e4f..4f1b511 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -450,6 +450,235 @@ static int kvm_set_pgd(pgd_t *pgd, pgd_t pgd_value) } +static int kvm_copy_pte_range(struct mm_struct *dst_mm, + struct mm_struct *src_mm, pmd_t *dst_pmd, + pmd_t *src_pmd, unsigned long addr, + unsigned long end) +{ + pte_t *src_pte, *dst_pte; + + dst_pte = kvm_pte_alloc(dst_mm, dst_pmd, addr); + if (IS_ERR(dst_pte)) + return PTR_ERR(dst_pte); + + addr &= PAGE_MASK; + src_pte = pte_offset_map(src_pmd, addr); + + do { + pr_debug("PTE: %lx/%lx set[%lx] = %lx\n", + addr, addr + PAGE_SIZE, (long)dst_pte, pte_val(*src_pte)); + set_pte_at(dst_mm, addr, dst_pte, *src_pte); + + } while (dst_pte++, src_pte++, addr += PAGE_SIZE, addr < end); + + return 0; +} + +static int kvm_copy_pmd_range(struct mm_struct *dst_mm, + struct mm_struct *src_mm, + pud_t *dst_pud, pud_t *src_pud, + unsigned long addr, unsigned long end, + enum page_table_level level) +{ + pmd_t *src_pmd, *dst_pmd; + unsigned long next; + int err; + + dst_pmd = kvm_pmd_alloc(dst_mm, dst_pud, addr); + if (IS_ERR(dst_pmd)) + return PTR_ERR(dst_pmd); + + src_pmd = pmd_offset(src_pud, addr); + + do { + next = pmd_addr_end(addr, end); + if (level == PGT_LEVEL_PMD || pmd_none(*src_pmd)) { + pr_debug("PMD: %lx/%lx set[%lx] = %lx\n", + addr, next, (long)dst_pmd, pmd_val(*src_pmd)); + err = kvm_set_pmd(dst_pmd, *src_pmd); + if (err) + return err; + continue; + } + + if (!pmd_present(*src_pmd)) { + pr_warn("PMD: not present for [%lx,%lx]\n", + addr, next - 1); + pmd_clear(dst_pmd); + continue; + } + + if (pmd_trans_huge(*src_pmd) || pmd_devmap(*src_pmd)) { + pr_debug("PMD: %lx/%lx set[%lx] = %lx (huge/devmap)\n", + addr, next, (long)dst_pmd, pmd_val(*src_pmd)); + err = kvm_set_pmd(dst_pmd, *src_pmd); + if (err) + return err; + continue; + } + + err = kvm_copy_pte_range(dst_mm, src_mm, dst_pmd, src_pmd, + addr, next); + if (err) { + pr_err("PMD: ERR PTE addr=%lx next=%lx\n", addr, next); + return err; + } + + } while (dst_pmd++, src_pmd++, addr = next, addr < end); + + return 0; +} + +static int kvm_copy_pud_range(struct mm_struct *dst_mm, + struct mm_struct *src_mm, + p4d_t *dst_p4d, p4d_t *src_p4d, + unsigned long addr, unsigned long end, + enum page_table_level level) +{ + pud_t *src_pud, *dst_pud; + unsigned long next; + int err; + + dst_pud = kvm_pud_alloc(dst_mm, dst_p4d, addr); + if (IS_ERR(dst_pud)) + return PTR_ERR(dst_pud); + + src_pud = pud_offset(src_p4d, addr); + + do { + next = pud_addr_end(addr, end); + if (level == PGT_LEVEL_PUD || pud_none(*src_pud)) { + pr_debug("PUD: %lx/%lx set[%lx] = %lx\n", + addr, next, (long)dst_pud, pud_val(*src_pud)); + err = kvm_set_pud(dst_pud, *src_pud); + if (err) + return err; + continue; + } + + if (pud_trans_huge(*src_pud) || pud_devmap(*src_pud)) { + pr_debug("PUD: %lx/%lx set[%lx] = %lx (huge/devmap)\n", + addr, next, (long)dst_pud, pud_val(*src_pud)); + err = kvm_set_pud(dst_pud, *src_pud); + if (err) + return err; + continue; + } + + err = kvm_copy_pmd_range(dst_mm, src_mm, dst_pud, src_pud, + addr, next, level); + if (err) { + pr_err("PUD: ERR PMD addr=%lx next=%lx\n", addr, next); + return err; + } + + } while (dst_pud++, src_pud++, addr = next, addr < end); + + return 0; +} + +static int kvm_copy_p4d_range(struct mm_struct *dst_mm, + struct mm_struct *src_mm, + pgd_t *dst_pgd, pgd_t *src_pgd, + unsigned long addr, unsigned long end, + enum page_table_level level) +{ + p4d_t *src_p4d, *dst_p4d; + unsigned long next; + int err; + + dst_p4d = kvm_p4d_alloc(dst_mm, dst_pgd, addr); + if (IS_ERR(dst_p4d)) + return PTR_ERR(dst_p4d); + + src_p4d = p4d_offset(src_pgd, addr); + + do { + next = p4d_addr_end(addr, end); + if (level == PGT_LEVEL_P4D || p4d_none(*src_p4d)) { + pr_debug("P4D: %lx/%lx set[%lx] = %lx\n", + addr, next, (long)dst_p4d, p4d_val(*src_p4d)); + + err = kvm_set_p4d(dst_p4d, *src_p4d); + if (err) + return err; + continue; + } + + err = kvm_copy_pud_range(dst_mm, src_mm, dst_p4d, src_p4d, + addr, next, level); + if (err) { + pr_err("P4D: ERR PUD addr=%lx next=%lx\n", addr, next); + return err; + } + + } while (dst_p4d++, src_p4d++, addr = next, addr < end); + + return 0; +} + +static int kvm_copy_pgd_range(struct mm_struct *dst_mm, + struct mm_struct *src_mm, unsigned long addr, + unsigned long end, enum page_table_level level) +{ + pgd_t *src_pgd, *dst_pgd; + unsigned long next; + int err; + + dst_pgd = pgd_offset(dst_mm, addr); + src_pgd = pgd_offset(src_mm, addr); + + do { + next = pgd_addr_end(addr, end); + if (level == PGT_LEVEL_PGD || pgd_none(*src_pgd)) { + pr_debug("PGD: %lx/%lx set[%lx] = %lx\n", + addr, next, (long)dst_pgd, pgd_val(*src_pgd)); + err = kvm_set_pgd(dst_pgd, *src_pgd); + if (err) + return err; + continue; + } + + err = kvm_copy_p4d_range(dst_mm, src_mm, dst_pgd, src_pgd, + addr, next, level); + if (err) { + pr_err("PGD: ERR P4D addr=%lx next=%lx\n", addr, next); + return err; + } + + } while (dst_pgd++, src_pgd++, addr = next, addr < end); + + return 0; +} + +/* + * Copy page table entries from the current page table (i.e. from the + * kernel page table) to the KVM page table. The level parameter specifies + * the page table level (PGD, P4D, PUD PMD, PTE) at which the copy should + * be done. + */ +static int kvm_copy_mapping(void *ptr, size_t size, enum page_table_level level) +{ + unsigned long addr = (unsigned long)ptr; + unsigned long end = addr + ((unsigned long)size); + + BUG_ON(current->mm == &kvm_mm); + pr_debug("KERNMAP COPY addr=%px size=%lx\n", ptr, size); + return kvm_copy_pgd_range(&kvm_mm, current->mm, addr, end, level); +} + + +/* + * Copy page table PTE entries from the current page table to the KVM + * page table. + */ +int kvm_copy_ptes(void *ptr, unsigned long size) +{ + return kvm_copy_mapping(ptr, size, PGT_LEVEL_PTE); +} +EXPORT_SYMBOL(kvm_copy_ptes); + + static int kvm_isolation_init_mm(void) { pgd_t *kvm_pgd; diff --git a/arch/x86/kvm/isolation.h b/arch/x86/kvm/isolation.h index aa5e979..e8c018a 100644 --- a/arch/x86/kvm/isolation.h +++ b/arch/x86/kvm/isolation.h @@ -16,5 +16,6 @@ static inline bool kvm_isolation(void) extern void kvm_isolation_enter(void); extern void kvm_isolation_exit(void); extern void kvm_may_access_sensitive_data(struct kvm_vcpu *vcpu); +extern int kvm_copy_ptes(void *ptr, unsigned long size); #endif From patchwork Mon May 13 14:38:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941133 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E06871390 for ; Mon, 13 May 2019 14:42:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D330A27861 for ; Mon, 13 May 2019 14:42:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C79152832D; Mon, 13 May 2019 14:42:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6DAE127861 for ; Mon, 13 May 2019 14:42:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730415AbfEMOj7 (ORCPT ); Mon, 13 May 2019 10:39:59 -0400 Received: from userp2130.oracle.com ([156.151.31.86]:45696 "EHLO userp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730379AbfEMOj6 (ORCPT ); Mon, 13 May 2019 10:39:58 -0400 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEd3Xk181510; Mon, 13 May 2019 14:39:26 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=bWRO35KV4FFQ3ievQKGZUgx84GSk+HyQdow97r7zwRo=; b=m4RXoI1fsAKJ65HwcxYTTiRielpSLksHyApCP0jeXXPvtewp5rwRVYDUYtxPM3t1rURV OUQPSWm2qaD3jmmQMsTFovUD4DnaSZ9GrLWp+D/fBi4LLvQVebxgVSc9OJHFf2CRAqzr jMs2DWV9OiHzkIdKAyWvrDZ3MS42p42Wdt2fSuBYwE1/w/asqQj5umS4w9GtEYvIA23u 3rWtsxf8YqaVnsN0/d1amhOkTseV2bABVDPs67GwZ9yZi2jVBvqt2G58KXhAHzSc5cet 2eTD9xR3JIRNVohIEeFBP6T1HxJGuEpp2qFDS0MQNFcDFncFDyVl1u6MqmF255tAofwm Xw== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2130.oracle.com with ESMTP id 2sdnttfeh6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:39:25 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQI022780; Mon, 13 May 2019 14:39:22 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 15/27] kvm/isolation: keep track of VA range mapped in KVM address space Date: Mon, 13 May 2019 16:38:23 +0200 Message-Id: <1557758315-12667-16-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This will be used when we have to clear mappings to ensure the same range is cleared at the same page table level it was copied. Signed-off-by: Alexandre Chartre --- arch/x86/kvm/isolation.c | 86 ++++++++++++++++++++++++++++++++++++++++++++- 1 files changed, 84 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index 4f1b511..c8358a9 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -61,6 +61,20 @@ struct pgt_directory_group { #define PGTD_ALIGN(entry) \ ((typeof(entry))(((unsigned long)(entry)) & PAGE_MASK)) +/* + * Variables to keep track of address ranges mapped into the KVM + * address space. + */ +struct kvm_range_mapping { + struct list_head list; + void *ptr; + size_t size; + enum page_table_level level; +}; + +static LIST_HEAD(kvm_range_mapping_list); +static DEFINE_MUTEX(kvm_range_mapping_lock); + struct mm_struct kvm_mm = { .mm_rb = RB_ROOT, @@ -91,6 +105,52 @@ struct mm_struct kvm_mm = { static bool __read_mostly address_space_isolation; module_param(address_space_isolation, bool, 0444); +static struct kvm_range_mapping *kvm_get_range_mapping_locked(void *ptr, + bool *subset) +{ + struct kvm_range_mapping *range; + + list_for_each_entry(range, &kvm_range_mapping_list, list) { + if (range->ptr == ptr) { + if (subset) + *subset = false; + return range; + } + if (ptr > range->ptr && ptr < range->ptr + range->size) { + if (subset) + *subset = true; + return range; + } + } + + return NULL; +} + +static struct kvm_range_mapping *kvm_get_range_mapping(void *ptr, bool *subset) +{ + struct kvm_range_mapping *range; + + mutex_lock(&kvm_range_mapping_lock); + range = kvm_get_range_mapping_locked(ptr, subset); + mutex_unlock(&kvm_range_mapping_lock); + + return range; +} + +static void kvm_free_all_range_mapping(void) +{ + struct kvm_range_mapping *range, *range_next; + + mutex_lock(&kvm_range_mapping_lock); + + list_for_each_entry_safe(range, range_next, + &kvm_range_mapping_list, list) { + list_del(&range->list); + kfree(range); + } + + mutex_unlock(&kvm_range_mapping_lock); +} static struct pgt_directory_group *pgt_directory_group_create(void) { @@ -661,10 +721,30 @@ static int kvm_copy_mapping(void *ptr, size_t size, enum page_table_level level) { unsigned long addr = (unsigned long)ptr; unsigned long end = addr + ((unsigned long)size); + struct kvm_range_mapping *range_mapping; + bool subset; + int err; BUG_ON(current->mm == &kvm_mm); - pr_debug("KERNMAP COPY addr=%px size=%lx\n", ptr, size); - return kvm_copy_pgd_range(&kvm_mm, current->mm, addr, end, level); + pr_debug("KERNMAP COPY addr=%px size=%lx level=%d\n", ptr, size, level); + + range_mapping = kmalloc(sizeof(struct kvm_range_mapping), GFP_KERNEL); + if (!range_mapping) + return -ENOMEM; + + err = kvm_copy_pgd_range(&kvm_mm, current->mm, addr, end, level); + if (err) { + kfree(range_mapping); + return err; + } + + INIT_LIST_HEAD(&range_mapping->list); + range_mapping->ptr = ptr; + range_mapping->size = size; + range_mapping->level = level; + list_add(&range_mapping->list, &kvm_range_mapping_list); + + return 0; } @@ -720,6 +800,8 @@ static void kvm_isolation_uninit_mm(void) destroy_context(&kvm_mm); + kvm_free_all_range_mapping(); + #ifdef CONFIG_PAGE_TABLE_ISOLATION /* * With PTI, the KVM address space is defined in the user From patchwork Mon May 13 14:38:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941115 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 605926C5 for ; Mon, 13 May 2019 14:41:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4EA2427861 for ; Mon, 13 May 2019 14:41:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 424312832D; Mon, 13 May 2019 14:41:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9B19927861 for ; Mon, 13 May 2019 14:41:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730554AbfEMOkL (ORCPT ); Mon, 13 May 2019 10:40:11 -0400 Received: from aserp2130.oracle.com ([141.146.126.79]:37468 "EHLO aserp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730525AbfEMOkK (ORCPT ); Mon, 13 May 2019 10:40:10 -0400 Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEd2tv193095; Mon, 13 May 2019 14:39:28 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=kkqkMbnz+iek9dkeudKGJuXmnkn48lWfRSXOX7tv6jE=; b=2ar9gUe8PEkk2N7E5R2hXBN+S8EK1bK+MAG3miC+Uo2QLiQpRtXqL81wyT3WFUlvaGIV yFj+PnJKxx3M+EJiG2N5Wy2Ca0GxXFOvdr9H+PUOpN9ibTPdSwSw+PUxnSclUc/j8iqx wwwphVvx1m+Qjgrj3A5zXZqSIA9u83IyoHXzPJ4+e91MJFKjD0UNIlFLxXBq84f1OvIs RoXbf7KaYRZRzbEmfMM66PYK9PkQLGNLAw9R0GErJlNjc4FdSPRQilul0tfbzsHbejxk biccPs2shvHGp4MSsXlJ7ox721gE8q5sCTOHoWye2NivkamyCR56V4G1+yL8MEIsUd/r kQ== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by aserp2130.oracle.com with ESMTP id 2sdkwdfkxs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:39:28 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQJ022780; Mon, 13 May 2019 14:39:25 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 16/27] kvm/isolation: functions to clear page table entries for a VA range Date: Mon, 13 May 2019 16:38:24 +0200 Message-Id: <1557758315-12667-17-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP These functions will be used to unmapped memory from the KVM address space. When clearing mapping in the KVM page table, check that the clearing effectively happens in the KVM page table and there is no crossing of the KVM page table boundary (with references to the kernel page table), so that the kernel page table isn't mistakenly modified. Information (address, size, page table level) about address ranges mapped to the KVM page table is tracked, so mapping clearing is done with just specified the start address of the range. Signed-off-by: Alexandre Chartre --- arch/x86/kvm/isolation.c | 172 ++++++++++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/isolation.h | 1 + 2 files changed, 173 insertions(+), 0 deletions(-) diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index c8358a9..e494a15 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -758,6 +758,178 @@ int kvm_copy_ptes(void *ptr, unsigned long size) } EXPORT_SYMBOL(kvm_copy_ptes); +static void kvm_clear_pte_range(pmd_t *pmd, unsigned long addr, + unsigned long end) +{ + pte_t *pte; + + pte = kvm_pte_offset(pmd, addr); + if (IS_ERR(pte)) { + pr_debug("PTE not found, skip clearing\n"); + return; + } + do { + pr_debug("PTE: %lx/%lx clear[%lx]\n", addr, end, (long)pte); + pte_clear(NULL, addr, pte); + } while (pte++, addr += PAGE_SIZE, addr < end); +} + +static void kvm_clear_pmd_range(pud_t *pud, unsigned long addr, + unsigned long end, enum page_table_level level) +{ + pmd_t *pmd; + unsigned long next; + + pmd = kvm_pmd_offset(pud, addr); + if (IS_ERR(pmd)) { + pr_debug("PMD not found, skip clearing\n"); + return; + } + do { + next = pmd_addr_end(addr, end); + if (pmd_none(*pmd)) + continue; + BUG_ON(!pmd_present(*pmd)); + if (level == PGT_LEVEL_PMD || pmd_trans_huge(*pmd) || + pmd_devmap(*pmd)) { + pr_debug("PMD: %lx/%lx clear[%lx]\n", + addr, end, (long)pmd); + pmd_clear(pmd); + continue; + } + kvm_clear_pte_range(pmd, addr, next); + } while (pmd++, addr = next, addr < end); +} + +static void kvm_clear_pud_range(p4d_t *p4d, unsigned long addr, + unsigned long end, enum page_table_level level) +{ + pud_t *pud; + unsigned long next; + + pud = kvm_pud_offset(p4d, addr); + if (IS_ERR(pud)) { + pr_debug("PUD not found, skip clearing\n"); + return; + } + do { + next = pud_addr_end(addr, end); + if (pud_none(*pud)) + continue; + if (level == PGT_LEVEL_PUD || pud_trans_huge(*pud) || + pud_devmap(*pud)) { + pr_debug("PUD: %lx/%lx clear[%lx]\n", + addr, end, (long)pud); + pud_clear(pud); + continue; + } + kvm_clear_pmd_range(pud, addr, next, level); + } while (pud++, addr = next, addr < end); +} + +static void kvm_clear_p4d_range(pgd_t *pgd, unsigned long addr, + unsigned long end, enum page_table_level level) +{ + p4d_t *p4d; + unsigned long next; + + p4d = kvm_p4d_offset(pgd, addr); + if (IS_ERR(p4d)) { + pr_debug("P4D not found, skip clearing\n"); + return; + } + + do { + next = p4d_addr_end(addr, end); + if (p4d_none(*p4d)) + continue; + if (level == PGT_LEVEL_P4D) { + pr_debug("P4D: %lx/%lx clear[%lx]\n", + addr, end, (long)p4d); + p4d_clear(p4d); + continue; + } + kvm_clear_pud_range(p4d, addr, next, level); + } while (p4d++, addr = next, addr < end); +} + +static void kvm_clear_pgd_range(struct mm_struct *mm, unsigned long addr, + unsigned long end, enum page_table_level level) +{ + pgd_t *pgd; + unsigned long next; + + pgd = pgd_offset(mm, addr); + do { + next = pgd_addr_end(addr, end); + if (pgd_none(*pgd)) + continue; + if (level == PGT_LEVEL_PGD) { + pr_debug("PGD: %lx/%lx clear[%lx]\n", + addr, end, (long)pgd); + pgd_clear(pgd); + continue; + } + kvm_clear_p4d_range(pgd, addr, next, level); + } while (pgd++, addr = next, addr < end); +} + +/* + * Clear page table entries in the KVM page table. The level parameter + * specifies the page table level (PGD, P4D, PUD PMD, PTE) at which the + * clear should be done. + * + * WARNING: The KVM page table can have direct references to the kernel + * page table, at different levels (PGD, P4D, PUD, PMD). When clearing + * such references, if the level is incorrect (for example, clear at the + * PTE level while the mapping was done at PMD level), then the clearing + * will occur in the kernel page table and the system will likely crash + * on an unhandled page fault. + */ +static void kvm_clear_mapping(void *ptr, size_t size, + enum page_table_level level) +{ + unsigned long start = (unsigned long)ptr; + unsigned long end = start + ((unsigned long)size); + + pr_debug("CLEAR %px, %lx [%lx,%lx], level=%d\n", + ptr, size, start, end, level); + kvm_clear_pgd_range(&kvm_mm, start, end, level); +} + +/* + * Clear a range mapping in the KVM page table. + */ +void kvm_clear_range_mapping(void *ptr) +{ + struct kvm_range_mapping *range_mapping; + bool subset; + + mutex_lock(&kvm_range_mapping_lock); + + range_mapping = kvm_get_range_mapping_locked(ptr, &subset); + if (!range_mapping) { + mutex_unlock(&kvm_range_mapping_lock); + pr_debug("CLEAR %px - range not found\n", ptr); + return; + } + if (subset) { + mutex_unlock(&kvm_range_mapping_lock); + pr_debug("CLEAR %px - ignored, subset of %px/%lx/%d\n", + ptr, range_mapping->ptr, range_mapping->size, + range_mapping->level); + return; + } + + kvm_clear_mapping(range_mapping->ptr, range_mapping->size, + range_mapping->level); + list_del(&range_mapping->list); + mutex_unlock(&kvm_range_mapping_lock); + + kfree(range_mapping); +} +EXPORT_SYMBOL(kvm_clear_range_mapping); + static int kvm_isolation_init_mm(void) { diff --git a/arch/x86/kvm/isolation.h b/arch/x86/kvm/isolation.h index e8c018a..7d3c985 100644 --- a/arch/x86/kvm/isolation.h +++ b/arch/x86/kvm/isolation.h @@ -17,5 +17,6 @@ static inline bool kvm_isolation(void) extern void kvm_isolation_exit(void); extern void kvm_may_access_sensitive_data(struct kvm_vcpu *vcpu); extern int kvm_copy_ptes(void *ptr, unsigned long size); +extern void kvm_clear_range_mapping(void *ptr); #endif From patchwork Mon May 13 14:38:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941119 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2BC8C1390 for ; Mon, 13 May 2019 14:41:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1F0D627861 for ; Mon, 13 May 2019 14:41:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 132772832D; Mon, 13 May 2019 14:41:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A5DA227861 for ; Mon, 13 May 2019 14:41:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730543AbfEMOkK (ORCPT ); Mon, 13 May 2019 10:40:10 -0400 Received: from userp2120.oracle.com ([156.151.31.85]:59876 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730542AbfEMOkJ (ORCPT ); Mon, 13 May 2019 10:40:09 -0400 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEd2Gn194925; Mon, 13 May 2019 14:39:31 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=+MaSC7D5NYyFHcoLlt21rqwn+ky+e8tS8duegev3cnQ=; b=z6slNgJ9MrFp8uPsBPJmJU6GEaQ8wt3ZidHa0OAjWLM8dA3+hvZO3Q2zqNmvKu+Qfvha f39YKjJiNdGzsIVZtiZmHgCAk3/RyTNp7XRWQJE/vV0AXPAhhrT+G+j8gHuyTYUQNX0f AIhcCp0PH4W5K5mBi7O15eBaUEQLLjjyWKwdghJjGta7Pwvvxw11nROudMT9WBKTJRLS bc5hFg0tiztOk+LrZZH2/SQPAOQVHBG6LvLcBcZDJcfhj/iVjW1L4APtEZtPI5sxvMGw RUUksDo71NnDOv56zatbXlgzV01aZsNu7msINVw4f0f0NjXKNnCv8fOY+KxWkWNogtPN 5Q== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2sdq1q7axg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:39:31 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQK022780; Mon, 13 May 2019 14:39:28 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 17/27] kvm/isolation: improve mapping copy when mapping is already present Date: Mon, 13 May 2019 16:38:25 +0200 Message-Id: <1557758315-12667-18-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP A mapping can already exist if a buffer was mapped in the KVM address space, and then the buffer was freed but there was no request to unmap from the KVM address space. In that case, clear the existing mapping before mapping the new buffer. Also if the new mapping is a subset of an already larger mapped range, then remap the entire larger map. Signed-off-by: Alexandre Chartre --- arch/x86/kvm/isolation.c | 67 +++++++++++++++++++++++++++++++++++++++++++--- 1 files changed, 63 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index e494a15..539e287 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -88,6 +88,9 @@ struct mm_struct kvm_mm = { DEFINE_STATIC_KEY_FALSE(kvm_isolation_enabled); EXPORT_SYMBOL(kvm_isolation_enabled); +static void kvm_clear_mapping(void *ptr, size_t size, + enum page_table_level level); + /* * When set to true, KVM #VMExit handlers run in isolated address space * which maps only KVM required code and per-VM information instead of @@ -721,6 +724,7 @@ static int kvm_copy_mapping(void *ptr, size_t size, enum page_table_level level) { unsigned long addr = (unsigned long)ptr; unsigned long end = addr + ((unsigned long)size); + unsigned long range_addr, range_end; struct kvm_range_mapping *range_mapping; bool subset; int err; @@ -728,22 +732,77 @@ static int kvm_copy_mapping(void *ptr, size_t size, enum page_table_level level) BUG_ON(current->mm == &kvm_mm); pr_debug("KERNMAP COPY addr=%px size=%lx level=%d\n", ptr, size, level); - range_mapping = kmalloc(sizeof(struct kvm_range_mapping), GFP_KERNEL); - if (!range_mapping) - return -ENOMEM; + mutex_lock(&kvm_range_mapping_lock); + + /* + * A mapping can already exist if the buffer was mapped and then + * freed but there was no request to unmap it. We might also be + * trying to map a subset of an already mapped buffer. + */ + range_mapping = kvm_get_range_mapping_locked(ptr, &subset); + if (range_mapping) { + if (subset) { + pr_debug("range %px/%lx/%d is a subset of %px/%lx/%d already mapped, remapping\n", + ptr, size, level, range_mapping->ptr, + range_mapping->size, range_mapping->level); + range_addr = (unsigned long)range_mapping->ptr; + range_end = range_addr + + ((unsigned long)range_mapping->size); + err = kvm_copy_pgd_range(&kvm_mm, current->mm, + range_addr, range_end, + range_mapping->level); + if (end <= range_end) { + /* + * We effectively have a subset, fully contained + * in the superset. So we are done. + */ + mutex_unlock(&kvm_range_mapping_lock); + return err; + } + /* + * The new range is larger than the existing mapped + * range. So we need an extra mapping to map the end + * of the range. + */ + addr = range_end; + range_mapping = NULL; + pr_debug("adding extra range %lx-%lx (%d)\n", addr, + end, level); + } else { + pr_debug("range %px size=%lx level=%d already mapped, clearing\n", + range_mapping->ptr, range_mapping->size, + range_mapping->level); + kvm_clear_mapping(range_mapping->ptr, + range_mapping->size, + range_mapping->level); + list_del(&range_mapping->list); + } + } + + if (!range_mapping) { + range_mapping = kmalloc(sizeof(struct kvm_range_mapping), + GFP_KERNEL); + if (!range_mapping) { + mutex_unlock(&kvm_range_mapping_lock); + return -ENOMEM; + } + INIT_LIST_HEAD(&range_mapping->list); + } err = kvm_copy_pgd_range(&kvm_mm, current->mm, addr, end, level); if (err) { + mutex_unlock(&kvm_range_mapping_lock); kfree(range_mapping); return err; } - INIT_LIST_HEAD(&range_mapping->list); range_mapping->ptr = ptr; range_mapping->size = size; range_mapping->level = level; list_add(&range_mapping->list, &kvm_range_mapping_list); + mutex_unlock(&kvm_range_mapping_lock); + return 0; } From patchwork Mon May 13 14:38:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941117 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AEF7D1390 for ; Mon, 13 May 2019 14:41:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A1EFB28306 for ; Mon, 13 May 2019 14:41:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 95C6E28334; Mon, 13 May 2019 14:41:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 430F928306 for ; Mon, 13 May 2019 14:41:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730685AbfEMOlM (ORCPT ); Mon, 13 May 2019 10:41:12 -0400 Received: from userp2130.oracle.com ([156.151.31.86]:47098 "EHLO userp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730664AbfEMOlL (ORCPT ); Mon, 13 May 2019 10:41:11 -0400 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEd3dB181544; Mon, 13 May 2019 14:39:39 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=bOd6NTuOO3w4Tse88wsGsS3lTxF7t3wBy812EA6WdBU=; b=Y9nXfUJNbdNXiy0WXZfnXcvJMfJnbW6Wd/jhOsfaH+Nku6LO73tegfk4kUXz6qGARmee jxG1wNCbZiHDX8SNNrTewc3cLIlDZsttlVSwGklecgW/HROVU+I3mtazS3Yvqh11anbo Zv4jHKjLPjfduxRLu4X8LAzZ3P36HyfHmP1W/DyLiXegpYJeMKnqKZFwITYW5qzlSHm5 RtesqMSGG/6fDOLSNgWvXF9bfkYj0YrLp3UqSNuvKowRK44FvGuaRvjBi3RpZ0NPhj/l QmdYQ9IaU3TVKzTuW8I6wo8hxvktW9E/r6qI9QWOuBYsblTK3/WgBv+O9a9uDwsYhX/x cg== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2130.oracle.com with ESMTP id 2sdnttfeja-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:39:39 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQL022780; Mon, 13 May 2019 14:39:31 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 18/27] kvm/isolation: function to copy page table entries for percpu buffer Date: Mon, 13 May 2019 16:38:26 +0200 Message-Id: <1557758315-12667-19-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP pcpu_base_addr is already mapped to the KVM address space, but this represents the first percpu chunk. To access a per-cpu buffer not allocated in the first chunk, add a function which maps all cpu buffers corresponding to that per-cpu buffer. Also add function to clear page table entries for a percpu buffer. Signed-off-by: Alexandre Chartre --- arch/x86/kvm/isolation.c | 34 ++++++++++++++++++++++++++++++++++ arch/x86/kvm/isolation.h | 2 ++ 2 files changed, 36 insertions(+), 0 deletions(-) diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index 539e287..2052abf 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -990,6 +990,40 @@ void kvm_clear_range_mapping(void *ptr) EXPORT_SYMBOL(kvm_clear_range_mapping); +void kvm_clear_percpu_mapping(void *percpu_ptr) +{ + void *ptr; + int cpu; + + pr_debug("PERCPU CLEAR percpu=%px\n", percpu_ptr); + for_each_possible_cpu(cpu) { + ptr = per_cpu_ptr(percpu_ptr, cpu); + kvm_clear_range_mapping(ptr); + } +} +EXPORT_SYMBOL(kvm_clear_percpu_mapping); + +int kvm_copy_percpu_mapping(void *percpu_ptr, size_t size) +{ + void *ptr; + int cpu, err; + + pr_debug("PERCPU COPY percpu=%px size=%lx\n", percpu_ptr, size); + for_each_possible_cpu(cpu) { + ptr = per_cpu_ptr(percpu_ptr, cpu); + pr_debug("PERCPU COPY cpu%d addr=%px\n", cpu, ptr); + err = kvm_copy_ptes(ptr, size); + if (err) { + kvm_clear_range_mapping(percpu_ptr); + return err; + } + } + + return 0; +} +EXPORT_SYMBOL(kvm_copy_percpu_mapping); + + static int kvm_isolation_init_mm(void) { pgd_t *kvm_pgd; diff --git a/arch/x86/kvm/isolation.h b/arch/x86/kvm/isolation.h index 7d3c985..3ef2060 100644 --- a/arch/x86/kvm/isolation.h +++ b/arch/x86/kvm/isolation.h @@ -18,5 +18,7 @@ static inline bool kvm_isolation(void) extern void kvm_may_access_sensitive_data(struct kvm_vcpu *vcpu); extern int kvm_copy_ptes(void *ptr, unsigned long size); extern void kvm_clear_range_mapping(void *ptr); +extern int kvm_copy_percpu_mapping(void *percpu_ptr, size_t size); +extern void kvm_clear_percpu_mapping(void *percpu_ptr); #endif From patchwork Mon May 13 14:38:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941111 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 145971390 for ; Mon, 13 May 2019 14:40:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 074D828355 for ; Mon, 13 May 2019 14:40:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EC5C22835B; Mon, 13 May 2019 14:40:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 610CD28334 for ; Mon, 13 May 2019 14:40:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730590AbfEMOkQ (ORCPT ); Mon, 13 May 2019 10:40:16 -0400 Received: from aserp2130.oracle.com ([141.146.126.79]:37590 "EHLO aserp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730579AbfEMOkP (ORCPT ); Mon, 13 May 2019 10:40:15 -0400 Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEd28I193025; Mon, 13 May 2019 14:39:36 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=l5aKMAk3Fh+gtpNvYGvrecFqK8ir23pbpb3HmvcDshc=; b=NPL0tURBc+BNNRYWXut9QJPxP0shoD0HzAqoBR5E6EV0J0revf1EigBcQsuJWtb30CCC upBrgbB61tjujiQwsK2QAD4JDlF2ymOmSTZCeesv44XBs0QlN2E2lJ976YC4UOUGYYBG yj4HV+wTXtHjArhgs933THSajLYTjqu3Pm7eFTZXAp/pY1YUghyA/Rlag0ritPGg8VZh 0vV3GFARwn18IkWwCCTC4TZudBcx6LFynGTBEWs9PS6Pa3VOJP3RXhwmMCToyY+MwubI lgNu9vsCkFz4pSpo1VcWzLZNKsLSUinM/ps8GUa/cBmdF5gC/ozgE5MprNsOpOKW+J8h TA== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by aserp2130.oracle.com with ESMTP id 2sdkwdfkye-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:39:36 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQM022780; Mon, 13 May 2019 14:39:33 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 19/27] kvm/isolation: initialize the KVM page table with core mappings Date: Mon, 13 May 2019 16:38:27 +0200 Message-Id: <1557758315-12667-20-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The KVM page table is initialized with adding core memory mappings: the kernel text, the per-cpu memory, the kvm module, the cpu_entry_area, %esp fixup stacks, IRQ stacks. Signed-off-by: Alexandre Chartre --- arch/x86/kernel/cpu/common.c | 2 + arch/x86/kvm/isolation.c | 131 ++++++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/isolation.h | 10 +++ include/linux/percpu.h | 2 + mm/percpu.c | 6 +- 5 files changed, 149 insertions(+), 2 deletions(-) diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 3764054..0fa44b1 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1511,6 +1511,8 @@ static __init int setup_clearcpuid(char *arg) EXPORT_PER_CPU_SYMBOL(current_task); DEFINE_PER_CPU(struct irq_stack *, hardirq_stack_ptr); +EXPORT_PER_CPU_SYMBOL_GPL(hardirq_stack_ptr); + DEFINE_PER_CPU(unsigned int, irq_count) __visible = -1; DEFINE_PER_CPU(int, __preempt_count) = INIT_PREEMPT_COUNT; diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index 2052abf..cf5ee0d 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -10,6 +10,8 @@ #include #include +#include +#include #include #include @@ -88,6 +90,8 @@ struct mm_struct kvm_mm = { DEFINE_STATIC_KEY_FALSE(kvm_isolation_enabled); EXPORT_SYMBOL(kvm_isolation_enabled); +static void kvm_isolation_uninit_page_table(void); +static void kvm_isolation_uninit_mm(void); static void kvm_clear_mapping(void *ptr, size_t size, enum page_table_level level); @@ -1024,10 +1028,130 @@ int kvm_copy_percpu_mapping(void *percpu_ptr, size_t size) EXPORT_SYMBOL(kvm_copy_percpu_mapping); +static int kvm_isolation_init_page_table(void) +{ + void *stack; + int cpu, rv; + + /* + * Copy the mapping for all the kernel text. We copy at the PMD + * level since the PUD is shared with the module mapping space. + */ + rv = kvm_copy_mapping((void *)__START_KERNEL_map, KERNEL_IMAGE_SIZE, + PGT_LEVEL_PMD); + if (rv) + goto out_uninit_page_table; + + /* copy the mapping of per cpu memory */ + rv = kvm_copy_mapping(pcpu_base_addr, pcpu_unit_size * pcpu_nr_units, + PGT_LEVEL_PMD); + if (rv) + goto out_uninit_page_table; + + /* + * Copy the mapping for cpu_entry_area and %esp fixup stacks + * (this is based on the PTI userland address space, but probably + * not needed because the KVM address space is not directly + * enterered from userspace). They can both be copied at the P4D + * level since they each have a dedicated P4D entry. + */ + rv = kvm_copy_mapping((void *)CPU_ENTRY_AREA_PER_CPU, P4D_SIZE, + PGT_LEVEL_P4D); + if (rv) + goto out_uninit_page_table; + +#ifdef CONFIG_X86_ESPFIX64 + rv = kvm_copy_mapping((void *)ESPFIX_BASE_ADDR, P4D_SIZE, + PGT_LEVEL_P4D); + if (rv) + goto out_uninit_page_table; +#endif + +#ifdef CONFIG_VMAP_STACK + /* + * Interrupt stacks are vmap'ed with guard pages, so we need to + * copy mappings. + */ + for_each_possible_cpu(cpu) { + stack = per_cpu(hardirq_stack_ptr, cpu); + pr_debug("IRQ Stack %px\n", stack); + if (!stack) + continue; + rv = kvm_copy_ptes(stack - IRQ_STACK_SIZE, IRQ_STACK_SIZE); + if (rv) + goto out_uninit_page_table; + } + +#endif + + /* copy mapping of the current module (kvm) */ + rv = kvm_copy_module_mapping(); + if (rv) + goto out_uninit_page_table; + + return 0; + +out_uninit_page_table: + kvm_isolation_uninit_page_table(); + return rv; +} + +/* + * Free all buffers used by the kvm page table. These buffers are stored + * in the kvm_pgt_dgroup_list. + */ +static void kvm_isolation_uninit_page_table(void) +{ + struct pgt_directory_group *dgroup, *dgroup_next; + enum page_table_level level; + void *ptr; + int i; + + mutex_lock(&kvm_pgt_dgroup_lock); + + list_for_each_entry_safe(dgroup, dgroup_next, + &kvm_pgt_dgroup_list, list) { + + for (i = 0; i < dgroup->count; i++) { + ptr = dgroup->directory[i].ptr; + level = dgroup->directory[i].level; + + switch (dgroup->directory[i].level) { + + case PGT_LEVEL_PTE: + kvm_pte_free(NULL, ptr); + break; + + case PGT_LEVEL_PMD: + kvm_pmd_free(NULL, ptr); + break; + + case PGT_LEVEL_PUD: + kvm_pud_free(NULL, ptr); + break; + + case PGT_LEVEL_P4D: + kvm_p4d_free(NULL, ptr); + break; + + default: + pr_err("unexpected page directory %d for %px\n", + level, ptr); + } + } + + list_del(&dgroup->list); + kfree(dgroup); + } + + mutex_unlock(&kvm_pgt_dgroup_lock); +} + static int kvm_isolation_init_mm(void) { pgd_t *kvm_pgd; gfp_t gfp_mask; + int rv; gfp_mask = GFP_KERNEL | __GFP_ZERO; kvm_pgd = (pgd_t *)__get_free_pages(gfp_mask, PGD_ALLOCATION_ORDER); @@ -1054,6 +1178,12 @@ static int kvm_isolation_init_mm(void) mm_init_cpumask(&kvm_mm); init_new_context(NULL, &kvm_mm); + rv = kvm_isolation_init_page_table(); + if (rv) { + kvm_isolation_uninit_mm(); + return rv; + } + return 0; } @@ -1065,6 +1195,7 @@ static void kvm_isolation_uninit_mm(void) destroy_context(&kvm_mm); + kvm_isolation_uninit_page_table(); kvm_free_all_range_mapping(); #ifdef CONFIG_PAGE_TABLE_ISOLATION diff --git a/arch/x86/kvm/isolation.h b/arch/x86/kvm/isolation.h index 3ef2060..1f79e28 100644 --- a/arch/x86/kvm/isolation.h +++ b/arch/x86/kvm/isolation.h @@ -3,6 +3,16 @@ #define ARCH_X86_KVM_ISOLATION_H #include +#include + +/* + * Copy the memory mapping for the current module. This is defined as a + * macro to ensure it is expanded in the module making the call so that + * THIS_MODULE has the correct value. + */ +#define kvm_copy_module_mapping() \ + (kvm_copy_ptes(THIS_MODULE->core_layout.base, \ + THIS_MODULE->core_layout.size)) DECLARE_STATIC_KEY_FALSE(kvm_isolation_enabled); diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 70b7123..fb0ab9a 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -70,6 +70,8 @@ extern void *pcpu_base_addr; extern const unsigned long *pcpu_unit_offsets; +extern int pcpu_unit_size; +extern int pcpu_nr_units; struct pcpu_group_info { int nr_units; /* aligned # of units */ diff --git a/mm/percpu.c b/mm/percpu.c index 68dd2e7..b68b3d8 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -119,8 +119,10 @@ #endif /* CONFIG_SMP */ static int pcpu_unit_pages __ro_after_init; -static int pcpu_unit_size __ro_after_init; -static int pcpu_nr_units __ro_after_init; +int pcpu_unit_size __ro_after_init; +EXPORT_SYMBOL(pcpu_unit_size); +int pcpu_nr_units __ro_after_init; +EXPORT_SYMBOL(pcpu_nr_units); static int pcpu_atom_size __ro_after_init; int pcpu_nr_slots __ro_after_init; static size_t pcpu_chunk_struct_size __ro_after_init; From patchwork Mon May 13 14:38:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941099 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8DA916C5 for ; Mon, 13 May 2019 14:40:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 80B282832D for ; Mon, 13 May 2019 14:40:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 747782833E; Mon, 13 May 2019 14:40:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 27C1D2832D for ; Mon, 13 May 2019 14:40:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730610AbfEMOkV (ORCPT ); Mon, 13 May 2019 10:40:21 -0400 Received: from aserp2130.oracle.com ([141.146.126.79]:37628 "EHLO aserp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730584AbfEMOkQ (ORCPT ); Mon, 13 May 2019 10:40:16 -0400 Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEdHlu193231; Mon, 13 May 2019 14:39:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=gJ5tt/4IYtOQ64SjXc4XK/hg4MlEde9v0jqeQXYD/eQ=; b=ZVeO12rORSnjNn30oKP4IfrRQOnpuWsZZpTFew29AW/A1AopB4vNTvJPzoBUNnLwecR5 RxfhLqEFwtp98NkffYHLWW/xGFZAMdCGFC1Cm5FhGyQSMCaCSB6w20I2qahvkzKCsFa5 m/f/wbDM5y5zJuEnz5wG8g9nfWjmPhU8RB2bXp/ASv1kj1gRX0Hrz8Moy8R160Qf+K6H gpnLK5eK57XeoRIkkZcloR/MsoTBMRliLnztAGJlyy4fIbv6fkl19QVpaSKUSvXbYn60 PuCO41ERbdVUgxGByWPWdcyJSJL7JyT3ve6bcXI+B2FmQnOupQrAXYuphyQi5YY9ctn+ Fg== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by aserp2130.oracle.com with ESMTP id 2sdkwdfm05-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:39:44 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQN022780; Mon, 13 May 2019 14:39:36 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 20/27] kvm/isolation: initialize the KVM page table with vmx specific data Date: Mon, 13 May 2019 16:38:28 +0200 Message-Id: <1557758315-12667-21-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In addition of core memory mappings, the KVM page table has to be initialized with vmx specific data. Signed-off-by: Alexandre Chartre --- arch/x86/kvm/vmx/vmx.c | 19 +++++++++++++++++++ 1 files changed, 19 insertions(+), 0 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 0c955bb..f181b3c 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -63,6 +63,7 @@ #include "vmcs12.h" #include "vmx.h" #include "x86.h" +#include "isolation.h" MODULE_AUTHOR("Qumranet"); MODULE_LICENSE("GPL"); @@ -7830,6 +7831,24 @@ static int __init vmx_init(void) } } + if (kvm_isolation()) { + pr_debug("mapping vmx init"); + /* copy mapping of the current module (kvm_intel) */ + r = kvm_copy_module_mapping(); + if (r) { + vmx_exit(); + return r; + } + if (vmx_l1d_flush_pages) { + r = kvm_copy_ptes(vmx_l1d_flush_pages, + PAGE_SIZE << L1D_CACHE_ORDER); + if (r) { + vmx_exit(); + return r; + } + } + } + #ifdef CONFIG_KEXEC_CORE rcu_assign_pointer(crash_vmclear_loaded_vmcss, crash_vmclear_local_loaded_vmcss); From patchwork Mon May 13 14:38:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941109 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 95B586C5 for ; Mon, 13 May 2019 14:40:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8837828346 for ; Mon, 13 May 2019 14:40:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7BAAD2834A; Mon, 13 May 2019 14:40:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0740B28334 for ; Mon, 13 May 2019 14:40:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730623AbfEMOkj (ORCPT ); Mon, 13 May 2019 10:40:39 -0400 Received: from userp2130.oracle.com ([156.151.31.86]:46126 "EHLO userp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730525AbfEMOkU (ORCPT ); Mon, 13 May 2019 10:40:20 -0400 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEd3dD181544; Mon, 13 May 2019 14:39:43 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=OxanbwcMAhQGR2MgwGG1WhL6eejs6scvJBA++WEfeAY=; b=sU2LVxvsJrs8djlyFSIdZHIRCpnGERg2UYPNUhkjBZS5y33p5PwL4YfHc1GeuRzDdQDc tI0T4yhU0dK1zriLrw7oc9ar2kWJb8ndOWDLgtO2S3p47ONlJjMT8DUNuiH6bnk/HBS8 4pRTCY+QUM4kQQE3oKH19lJQLedFyMmlQevJV2qxd27nI0pcuQGDzWA0LWRRFA/SWQEU nPdKdbh2dWUUqgGbATEXLToM31SW1rDh84aMrX31dZ5ifOINbXk1GaZ1ybT260MDzhmO Rv/V6YNnfjMVWcuyQ7jnbONxErhqEaY2u+iG8myukAxzaJ4GJAWzjMX/sNhelawAFCXL Aw== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2130.oracle.com with ESMTP id 2sdnttfejj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:39:42 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQO022780; Mon, 13 May 2019 14:39:39 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 21/27] kvm/isolation: initialize the KVM page table with vmx VM data Date: Mon, 13 May 2019 16:38:29 +0200 Message-Id: <1557758315-12667-22-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=2 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Map VM data, in particular the kvm structure data. Signed-off-by: Alexandre Chartre --- arch/x86/kvm/isolation.c | 17 +++++++++++++++++ arch/x86/kvm/isolation.h | 2 ++ arch/x86/kvm/vmx/vmx.c | 31 ++++++++++++++++++++++++++++++- arch/x86/kvm/x86.c | 12 ++++++++++++ include/linux/kvm_host.h | 1 + virt/kvm/arm/arm.c | 4 ++++ virt/kvm/kvm_main.c | 2 +- 7 files changed, 67 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index cf5ee0d..d3ac014 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -1222,6 +1222,23 @@ static void kvm_isolation_clear_handlers(void) kvm_set_isolation_exit_handler(NULL); } +int kvm_isolation_init_vm(struct kvm *kvm) +{ + if (!kvm_isolation()) + return 0; + + return (kvm_copy_percpu_mapping(kvm->srcu.sda, + sizeof(struct srcu_data))); +} + +void kvm_isolation_destroy_vm(struct kvm *kvm) +{ + if (!kvm_isolation()) + return; + + kvm_clear_percpu_mapping(kvm->srcu.sda); +} + int kvm_isolation_init(void) { int r; diff --git a/arch/x86/kvm/isolation.h b/arch/x86/kvm/isolation.h index 1f79e28..33e9a87 100644 --- a/arch/x86/kvm/isolation.h +++ b/arch/x86/kvm/isolation.h @@ -23,6 +23,8 @@ static inline bool kvm_isolation(void) extern int kvm_isolation_init(void); extern void kvm_isolation_uninit(void); +extern int kvm_isolation_init_vm(struct kvm *kvm); +extern void kvm_isolation_destroy_vm(struct kvm *kvm); extern void kvm_isolation_enter(void); extern void kvm_isolation_exit(void); extern void kvm_may_access_sensitive_data(struct kvm_vcpu *vcpu); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index f181b3c..5b52e8c 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6523,6 +6523,33 @@ static void vmx_vcpu_run(struct kvm_vcpu *vcpu) vmx_complete_interrupts(vmx); } +static void vmx_unmap_vm(struct kvm *kvm) +{ + struct kvm_vmx *kvm_vmx = to_kvm_vmx(kvm); + + if (!kvm_isolation()) + return; + + pr_debug("unmapping kvm %p", kvm_vmx); + kvm_clear_range_mapping(kvm_vmx); +} + +static int vmx_map_vm(struct kvm *kvm) +{ + struct kvm_vmx *kvm_vmx = to_kvm_vmx(kvm); + + if (!kvm_isolation()) + return 0; + + pr_debug("mapping kvm %p", kvm_vmx); + /* + * Only copy kvm_vmx struct mapping because other + * attributes (like kvm->srcu) are not initialized + * yet. + */ + return kvm_copy_ptes(kvm_vmx, sizeof(struct kvm_vmx)); +} + static struct kvm *vmx_vm_alloc(void) { struct kvm_vmx *kvm_vmx = __vmalloc(sizeof(struct kvm_vmx), @@ -6533,6 +6560,7 @@ static void vmx_vcpu_run(struct kvm_vcpu *vcpu) static void vmx_vm_free(struct kvm *kvm) { + vmx_unmap_vm(kvm); vfree(to_kvm_vmx(kvm)); } @@ -6702,7 +6730,8 @@ static int vmx_vm_init(struct kvm *kvm) break; } } - return 0; + + return (vmx_map_vm(kvm)); } static void __init vmx_check_processor_compat(void *rtn) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 1db72c3..e1cc3a6 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -9207,6 +9207,17 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) return 0; } +void kvm_arch_vm_postcreate(struct kvm *kvm) +{ + /* + * The kvm structure is mapped in vmx.c so that the full kvm_vmx + * structure can be mapped. Attributes allocated in the kvm + * structure (like kvm->srcu) are mapped by kvm_isolation_init_vm() + * because they are not initialized when vmx.c maps the kvm structure. + */ + kvm_isolation_init_vm(kvm); +} + static void kvm_unload_vcpu_mmu(struct kvm_vcpu *vcpu) { vcpu_load(vcpu); @@ -9320,6 +9331,7 @@ void kvm_arch_destroy_vm(struct kvm *kvm) x86_set_memory_region(kvm, IDENTITY_PAGETABLE_PRIVATE_MEMSLOT, 0, 0); x86_set_memory_region(kvm, TSS_PRIVATE_MEMSLOT, 0, 0); } + kvm_isolation_destroy_vm(kvm); if (kvm_x86_ops->vm_destroy) kvm_x86_ops->vm_destroy(kvm); kvm_pic_destroy(kvm); diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 640a036..ad24d9e 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -932,6 +932,7 @@ static inline bool kvm_arch_intc_initialized(struct kvm *kvm) int kvm_arch_init_vm(struct kvm *kvm, unsigned long type); void kvm_arch_destroy_vm(struct kvm *kvm); +void kvm_arch_vm_postcreate(struct kvm *kvm); void kvm_arch_sync_events(struct kvm *kvm); int kvm_cpu_has_pending_timer(struct kvm_vcpu *vcpu); diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c index f412ebc..0921cb3 100644 --- a/virt/kvm/arm/arm.c +++ b/virt/kvm/arm/arm.c @@ -156,6 +156,10 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) return ret; } +void kvm_arch_vm_postcreate(struct kvm *kvm) +{ +} + bool kvm_arch_has_vcpu_debugfs(void) { return false; diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index a704d1f..3c0c3db 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -3366,7 +3366,7 @@ static int kvm_dev_ioctl_create_vm(unsigned long type) return -ENOMEM; } kvm_uevent_notify_change(KVM_EVENT_CREATE_VM, kvm); - + kvm_arch_vm_postcreate(kvm); fd_install(r, file); return r; From patchwork Mon May 13 14:38:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941097 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E07AE1390 for ; Mon, 13 May 2019 14:40:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D2B9828306 for ; Mon, 13 May 2019 14:40:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C67E928334; Mon, 13 May 2019 14:40:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 77FEC28306 for ; Mon, 13 May 2019 14:40:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728704AbfEMOkZ (ORCPT ); Mon, 13 May 2019 10:40:25 -0400 Received: from userp2120.oracle.com ([156.151.31.85]:60134 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730620AbfEMOkX (ORCPT ); Mon, 13 May 2019 10:40:23 -0400 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEd2Tx194906; Mon, 13 May 2019 14:39:45 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=K8i6wzsowphEeVDFXNpunuRUppmUJV415MqqD0nUaBI=; b=W22kvYLfLPm+lj8NF2wS7sDx9KGLeptevfeDTUzv+GZagg1XtUp9vZNWcihPEkWLSaPG Lx/9JGXoffT/Eb0lGLC1sDrq7K90utKJBv6oHFBVipO5+pMWF/TFSJYeuAR0q5E3kmrb Z8y5MymYgg/n3HDgLeCgwZcunQle8i2tJ4Yh1yNjfZX7YXAq/W6CISOEnlueSKA6ARpD 7dxEJ5PcvUpwerEg1uRhKhIlwA7KzrsVOb2FbKrBGoKH6oKfCK78Z0vuFk+sXe8ZpF4h vFYm9C8pwOMgg5LaQcKAYKtRiG+FEuIFLEcRxD9EaawYkjfOPClZpb0QoK/QdH7HCyaA Yg== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2sdq1q7ayc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:39:45 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQP022780; Mon, 13 May 2019 14:39:42 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 22/27] kvm/isolation: initialize the KVM page table with vmx cpu data Date: Mon, 13 May 2019 16:38:30 +0200 Message-Id: <1557758315-12667-23-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Map vmx cpu to the KVM address space when a vmx cpu is created, and unmap when it is freed. Signed-off-by: Alexandre Chartre --- arch/x86/kvm/vmx/vmx.c | 65 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 65 insertions(+), 0 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 5b52e8c..cbbaf58 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6564,10 +6564,69 @@ static void vmx_vm_free(struct kvm *kvm) vfree(to_kvm_vmx(kvm)); } +static void vmx_unmap_vcpu(struct vcpu_vmx *vmx) +{ + pr_debug("unmapping vmx %p", vmx); + + kvm_clear_range_mapping(vmx); + if (enable_pml) + kvm_clear_range_mapping(vmx->pml_pg); + kvm_clear_range_mapping(vmx->guest_msrs); + kvm_clear_range_mapping(vmx->vmcs01.vmcs); + kvm_clear_range_mapping(vmx->vmcs01.msr_bitmap); + kvm_clear_range_mapping(vmx->vcpu.arch.pio_data); + kvm_clear_range_mapping(vmx->vcpu.arch.apic); +} + +static int vmx_map_vcpu(struct vcpu_vmx *vmx) +{ + int rv; + + pr_debug("mapping vmx %p", vmx); + + rv = kvm_copy_ptes(vmx, sizeof(struct vcpu_vmx)); + if (rv) + goto out_unmap_vcpu; + + if (enable_pml) { + rv = kvm_copy_ptes(vmx->pml_pg, PAGE_SIZE); + if (rv) + goto out_unmap_vcpu; + } + + rv = kvm_copy_ptes(vmx->guest_msrs, PAGE_SIZE); + if (rv) + goto out_unmap_vcpu; + + rv = kvm_copy_ptes(vmx->vmcs01.vmcs, PAGE_SIZE << vmcs_config.order); + if (rv) + goto out_unmap_vcpu; + + rv = kvm_copy_ptes(vmx->vmcs01.msr_bitmap, PAGE_SIZE); + if (rv) + goto out_unmap_vcpu; + + rv = kvm_copy_ptes(vmx->vcpu.arch.pio_data, PAGE_SIZE); + if (rv) + goto out_unmap_vcpu; + + rv = kvm_copy_ptes(vmx->vcpu.arch.apic, sizeof(struct kvm_lapic)); + if (rv) + goto out_unmap_vcpu; + + return 0; + +out_unmap_vcpu: + vmx_unmap_vcpu(vmx); + return rv; +} + static void vmx_free_vcpu(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx = to_vmx(vcpu); + if (kvm_isolation()) + vmx_unmap_vcpu(vmx); if (enable_pml) vmx_destroy_pml_buffer(vmx); free_vpid(vmx->vpid); @@ -6679,6 +6738,12 @@ static void vmx_free_vcpu(struct kvm_vcpu *vcpu) vmx->ept_pointer = INVALID_PAGE; + if (kvm_isolation()) { + err = vmx_map_vcpu(vmx); + if (err) + goto free_vmcs; + } + return &vmx->vcpu; free_vmcs: From patchwork Mon May 13 14:38:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941135 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AC9656C5 for ; Mon, 13 May 2019 14:42:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9F2CD27861 for ; Mon, 13 May 2019 14:42:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 92F692832D; Mon, 13 May 2019 14:42:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E879B27861 for ; Mon, 13 May 2019 14:42:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729695AbfEMOmW (ORCPT ); Mon, 13 May 2019 10:42:22 -0400 Received: from aserp2130.oracle.com ([141.146.126.79]:40302 "EHLO aserp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728993AbfEMOmV (ORCPT ); Mon, 13 May 2019 10:42:21 -0400 Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEd2cM193026; Mon, 13 May 2019 14:39:48 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=Dw7jnDOehrItQ8IBhhkiUXlVuHtLBnaP+UdwDVV0Wkc=; b=zSrYKyEoOAdq0QWpvZ0qd6kNFKnCiopVoHRqhqc1ZV4bi48RNUFvLgth3il0pLxTxJ2n CePloH/migBtqYtd6xbrITTyQDKGmELS+eGvp8q4r6EsdjFT9Aha0voKsbZbmYzkK/aH 4zsWyBkbuScAy0o39X/qZ0peTs4L6A5vj+f8j0R0RpMOLecXpCbYiaN7tvdV6Zj7SkBb LqMBBHLoFSVmQtxPLbBms9KAeMYrLYP9+8QQeRm3CM/TahwAKsdeOMTJKAgWYkztBied FDur+uo2FG0gAOb5iBm4+gebU8igLetDFFzIk9XWaWLWwgf4BEpCgqcLaJ6fAkuU2QB3 PQ== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by aserp2130.oracle.com with ESMTP id 2sdkwdfm0d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:39:47 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQQ022780; Mon, 13 May 2019 14:39:45 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 23/27] kvm/isolation: initialize the KVM page table with the vcpu tasks Date: Mon, 13 May 2019 16:38:31 +0200 Message-Id: <1557758315-12667-24-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Tasks which are going to be running with the KVM address space have to be mapped with their core data (stack, mm, pgd..) so that they can (at least) switch back to the kernel address space. For now, assume that these tasks are the ones running vcpu, and that there's a 1:1 mapping between a task and vcpu. This should eventually be improved to be independent of any task/vcpu mapping. Also check that the task effectively entering the KVM address space is mapped. Signed-off-by: Alexandre Chartre --- arch/x86/kvm/isolation.c | 182 ++++++++++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/isolation.h | 2 + arch/x86/kvm/vmx/vmx.c | 8 ++ include/linux/sched.h | 5 + 4 files changed, 197 insertions(+), 0 deletions(-) diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index d3ac014..e7979b3 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -64,6 +64,20 @@ struct pgt_directory_group { ((typeof(entry))(((unsigned long)(entry)) & PAGE_MASK)) /* + * Variables to keep track of tasks mapped into the KVM address space. + */ +struct kvm_task_mapping { + struct list_head list; + struct task_struct *task; + void *stack; + struct mm_struct *mm; + pgd_t *pgd; +}; + +static LIST_HEAD(kvm_task_mapping_list); +static DEFINE_MUTEX(kvm_task_mapping_lock); + +/* * Variables to keep track of address ranges mapped into the KVM * address space. */ @@ -1027,6 +1041,160 @@ int kvm_copy_percpu_mapping(void *percpu_ptr, size_t size) } EXPORT_SYMBOL(kvm_copy_percpu_mapping); +static void kvm_clear_task_mapping(struct kvm_task_mapping *task_mapping) +{ + if (task_mapping->task) { + kvm_clear_range_mapping(task_mapping->task); + task_mapping->task = NULL; + } + if (task_mapping->stack) { + kvm_clear_range_mapping(task_mapping->stack); + task_mapping->stack = NULL; + } + if (task_mapping->mm) { + kvm_clear_range_mapping(task_mapping->mm); + task_mapping->mm = NULL; + } + if (task_mapping->pgd) { + kvm_clear_range_mapping(task_mapping->pgd); + task_mapping->pgd = NULL; + } +} + +static int kvm_copy_task_mapping(struct task_struct *tsk, + struct kvm_task_mapping *task_mapping) +{ + int err; + + err = kvm_copy_ptes(tsk, sizeof(struct task_struct)); + if (err) + goto out_clear_task_mapping; + task_mapping->task = tsk; + + err = kvm_copy_ptes(tsk->stack, THREAD_SIZE); + if (err) + goto out_clear_task_mapping; + task_mapping->stack = tsk->stack; + + err = kvm_copy_ptes(tsk->active_mm, sizeof(struct mm_struct)); + if (err) + goto out_clear_task_mapping; + task_mapping->mm = tsk->active_mm; + + err = kvm_copy_ptes(tsk->active_mm->pgd, + PAGE_SIZE << PGD_ALLOCATION_ORDER); + if (err) + goto out_clear_task_mapping; + task_mapping->pgd = tsk->active_mm->pgd; + + return 0; + +out_clear_task_mapping: + kvm_clear_task_mapping(task_mapping); + return err; +} + +int kvm_add_task_mapping(struct task_struct *tsk) +{ + struct kvm_task_mapping *task_mapping; + int err; + + mutex_lock(&kvm_task_mapping_lock); + + if (tsk->kvm_mapped) { + mutex_unlock(&kvm_task_mapping_lock); + return 0; + } + + task_mapping = kzalloc(sizeof(struct kvm_task_mapping), GFP_KERNEL); + if (!task_mapping) { + mutex_unlock(&kvm_task_mapping_lock); + return -ENOMEM; + } + INIT_LIST_HEAD(&task_mapping->list); + + /* + * Ensure that the task and its stack are mapped into the KVM + * address space. Also map the task mm to be able to switch back + * to the original mm, and its PGD directory. + */ + pr_debug("mapping task %px\n", tsk); + err = kvm_copy_task_mapping(tsk, task_mapping); + if (err) { + kfree(task_mapping); + mutex_unlock(&kvm_task_mapping_lock); + return err; + } + + get_task_struct(tsk); + list_add(&task_mapping->list, &kvm_task_mapping_list); + tsk->kvm_mapped = true; + + mutex_unlock(&kvm_task_mapping_lock); + + return 0; +} +EXPORT_SYMBOL(kvm_add_task_mapping); + +static struct kvm_task_mapping *kvm_find_task_mapping(struct task_struct *tsk) +{ + struct kvm_task_mapping *task_mapping; + + list_for_each_entry(task_mapping, &kvm_task_mapping_list, list) { + if (task_mapping->task == tsk) + return task_mapping; + } + return NULL; +} + +void kvm_cleanup_task_mapping(struct task_struct *tsk) +{ + struct kvm_task_mapping *task_mapping; + + if (!tsk->kvm_mapped) + return; + + task_mapping = kvm_find_task_mapping(tsk); + if (!task_mapping) { + pr_debug("KVM isolation: mapping not found for mapped task %px\n", + tsk); + tsk->kvm_mapped = false; + mutex_unlock(&kvm_task_mapping_lock); + return; + } + + pr_debug("unmapping task %px\n", tsk); + + list_del(&task_mapping->list); + kvm_clear_task_mapping(task_mapping); + kfree(task_mapping); + tsk->kvm_mapped = false; + put_task_struct(tsk); + mutex_unlock(&kvm_task_mapping_lock); +} +EXPORT_SYMBOL(kvm_cleanup_task_mapping); + +/* + * Mark all tasks which have being mapped into the KVM address space + * as not mapped. This only clears the mapping attribute in the task + * structure, but page table mappings remain in the KVM page table. + * They will be effectively removed when deleting the KVM page table. + */ +static void kvm_reset_all_task_mapping(void) +{ + struct kvm_task_mapping *task_mapping; + struct task_struct *tsk; + + mutex_lock(&kvm_task_mapping_lock); + list_for_each_entry(task_mapping, &kvm_task_mapping_list, list) { + tsk = task_mapping->task; + pr_debug("clear mapping for task %px\n", tsk); + tsk->kvm_mapped = false; + put_task_struct(tsk); + } + mutex_unlock(&kvm_task_mapping_lock); +} + static int kvm_isolation_init_page_table(void) { @@ -1195,6 +1363,7 @@ static void kvm_isolation_uninit_mm(void) destroy_context(&kvm_mm); + kvm_reset_all_task_mapping(); kvm_isolation_uninit_page_table(); kvm_free_all_range_mapping(); @@ -1227,6 +1396,8 @@ int kvm_isolation_init_vm(struct kvm *kvm) if (!kvm_isolation()) return 0; + pr_debug("mapping kvm srcu sda\n"); + return (kvm_copy_percpu_mapping(kvm->srcu.sda, sizeof(struct srcu_data))); } @@ -1236,6 +1407,8 @@ void kvm_isolation_destroy_vm(struct kvm *kvm) if (!kvm_isolation()) return; + pr_debug("unmapping kvm srcu sda\n"); + kvm_clear_percpu_mapping(kvm->srcu.sda); } @@ -1276,12 +1449,21 @@ void kvm_may_access_sensitive_data(struct kvm_vcpu *vcpu) void kvm_isolation_enter(void) { + int err; + if (kvm_isolation()) { /* * Switches to kvm_mm should happen from vCPU thread, * which should not be a kernel thread with no mm */ BUG_ON(current->active_mm == NULL); + + err = kvm_add_task_mapping(current); + if (err) { + pr_err("KVM isolation cancelled (failed to map task %px)", + current); + return; + } /* TODO: switch to kvm_mm */ } } diff --git a/arch/x86/kvm/isolation.h b/arch/x86/kvm/isolation.h index 33e9a87..2d7d016 100644 --- a/arch/x86/kvm/isolation.h +++ b/arch/x86/kvm/isolation.h @@ -32,5 +32,7 @@ static inline bool kvm_isolation(void) extern void kvm_clear_range_mapping(void *ptr); extern int kvm_copy_percpu_mapping(void *percpu_ptr, size_t size); extern void kvm_clear_percpu_mapping(void *percpu_ptr); +extern int kvm_add_task_mapping(struct task_struct *tsk); +extern void kvm_cleanup_task_mapping(struct task_struct *tsk); #endif diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index cbbaf58..9ed31c2 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6576,6 +6576,9 @@ static void vmx_unmap_vcpu(struct vcpu_vmx *vmx) kvm_clear_range_mapping(vmx->vmcs01.msr_bitmap); kvm_clear_range_mapping(vmx->vcpu.arch.pio_data); kvm_clear_range_mapping(vmx->vcpu.arch.apic); + + /* XXX assume there's a 1:1 mapping between a task and a vcpu */ + kvm_cleanup_task_mapping(current); } static int vmx_map_vcpu(struct vcpu_vmx *vmx) @@ -6614,6 +6617,11 @@ static int vmx_map_vcpu(struct vcpu_vmx *vmx) if (rv) goto out_unmap_vcpu; + /* XXX assume there's a 1:1 mapping between a task and a vcpu */ + rv = kvm_add_task_mapping(current); + if (rv) + goto out_unmap_vcpu; + return 0; out_unmap_vcpu: diff --git a/include/linux/sched.h b/include/linux/sched.h index 50606a6..80e1d75 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1199,6 +1199,11 @@ struct task_struct { unsigned long prev_lowest_stack; #endif +#ifdef CONFIG_HAVE_KVM + /* Is the task mapped into the KVM address space? */ + bool kvm_mapped; +#endif + /* * New fields for task_struct should be added above here, so that * they are included in the randomized portion of task_struct. From patchwork Mon May 13 14:38:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941127 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5668D6C5 for ; Mon, 13 May 2019 14:41:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 48F2927861 for ; Mon, 13 May 2019 14:41:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3D2BE2832D; Mon, 13 May 2019 14:41:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7D69827861 for ; Mon, 13 May 2019 14:41:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730715AbfEMOle (ORCPT ); Mon, 13 May 2019 10:41:34 -0400 Received: from aserp2130.oracle.com ([141.146.126.79]:39128 "EHLO aserp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729639AbfEMOle (ORCPT ); Mon, 13 May 2019 10:41:34 -0400 Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEdhZb193417; Mon, 13 May 2019 14:39:50 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=I1kUHuKiSH9QzRVSTdzQMnQ4Y46T3IjTtH8oraLYUKM=; b=ajrelu+23EpPoZzOm46NJouVlFfpY/7VuA8j/AA2maWWZv6PEWO74sb0JEOk8WTm7MlX zkCzIK+uulNLm3Uc5+fKPE0rSLiDjjJLejX1G+rGS4A2aaX6wwImF2XGsend6ZiCm1fm P8RiX9tW+RIc4zR0FuNImR/j0P1fr1IsIkAxcMSKq1XNd8xla45EpEAzSsqq0KaMJbcU mGTuD8FX0uYid5B79ED/WxOVxHhFvK/EkRkDNZuXjosZo4JvYLUzGo5D2aW2dE1YBaP0 wQ7ypO/K2a5rX9NiNfuwOPDx8wO0Hl7bt9/GOcZqO+YBs4IdO/Hb+PDVBfNbd7Ra1TXZ Dw== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by aserp2130.oracle.com with ESMTP id 2sdkwdfm0s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:39:50 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQR022780; Mon, 13 May 2019 14:39:47 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 24/27] kvm/isolation: KVM page fault handler Date: Mon, 13 May 2019 16:38:32 +0200 Message-Id: <1557758315-12667-25-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The KVM page fault handler handles page fault occurring while using the KVM address space by switching to the kernel address space and retrying the access (except if the fault occurs while switching to the kernel address space). Processing of page faults occurring while using the kernel address space is unchanged. Page fault log is cleared when creating a vm so that page fault information doesn't persist when qemu is stopped and restarted. The KVM module parameter page_fault_stack can be used to disable dumping stack trace when a page fault occurs while using the KVM address space. The fault will still be reported but without the stack trace. Signed-off-by: Alexandre Chartre --- arch/x86/kernel/dumpstack.c | 1 + arch/x86/kvm/isolation.c | 202 +++++++++++++++++++++++++++++++++++++++++++ arch/x86/mm/fault.c | 12 +++ 3 files changed, 215 insertions(+), 0 deletions(-) diff --git a/arch/x86/kernel/dumpstack.c b/arch/x86/kernel/dumpstack.c index 2b58864..aa28763 100644 --- a/arch/x86/kernel/dumpstack.c +++ b/arch/x86/kernel/dumpstack.c @@ -292,6 +292,7 @@ void show_stack(struct task_struct *task, unsigned long *sp) show_trace_log_lvl(task, NULL, sp, KERN_DEFAULT); } +EXPORT_SYMBOL(show_stack); void show_stack_regs(struct pt_regs *regs) { diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index e7979b3..db0a7ce 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include @@ -17,6 +18,9 @@ #include "isolation.h" +extern bool (*kvm_page_fault_handler)(struct pt_regs *regs, + unsigned long error_code, + unsigned long address); enum page_table_level { PGT_LEVEL_PTE, @@ -91,6 +95,25 @@ struct kvm_range_mapping { static LIST_HEAD(kvm_range_mapping_list); static DEFINE_MUTEX(kvm_range_mapping_lock); +/* + * When a page fault occurs, while running with the KVM address space, + * the KVM page fault handler prints information about the fault (in + * particular the stack trace), and it switches back to the kernel + * address space. + * + * Information printed by the KVM page fault handler can be used to find + * out data not mapped in the KVM address space. Then the KVM address + * space can be augmented to include the missing mapping so that we don't + * fault at that same place anymore. + * + * The following variables keep track of page faults occurring while running + * with the KVM address space to prevent displaying the same information. + */ + +#define KVM_LAST_FAULT_COUNT 128 + +static unsigned long kvm_last_fault[KVM_LAST_FAULT_COUNT]; + struct mm_struct kvm_mm = { .mm_rb = RB_ROOT, @@ -126,6 +149,14 @@ static void kvm_clear_mapping(void *ptr, size_t size, static bool __read_mostly address_space_isolation; module_param(address_space_isolation, bool, 0444); +/* + * When set to true, KVM dumps the stack when a page fault occurs while + * running with the KVM address space. Otherwise the page fault is still + * reported but without the stack trace. + */ +static bool __read_mostly page_fault_stack = true; +module_param(page_fault_stack, bool, 0444); + static struct kvm_range_mapping *kvm_get_range_mapping_locked(void *ptr, bool *subset) { @@ -1195,6 +1226,173 @@ static void kvm_reset_all_task_mapping(void) mutex_unlock(&kvm_task_mapping_lock); } +static int bad_address(void *p) +{ + unsigned long dummy; + + return probe_kernel_address((unsigned long *)p, dummy); +} + +static void kvm_dump_pagetable(pgd_t *base, unsigned long address) +{ + pgd_t *pgd = base + pgd_index(address); + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + + pr_info("BASE %px ", base); + + if (bad_address(pgd)) + goto bad; + + pr_cont("PGD %lx ", pgd_val(*pgd)); + + if (!pgd_present(*pgd)) + goto out; + + p4d = p4d_offset(pgd, address); + if (bad_address(p4d)) + goto bad; + + pr_cont("P4D %lx ", p4d_val(*p4d)); + if (!p4d_present(*p4d) || p4d_large(*p4d)) + goto out; + + pud = pud_offset(p4d, address); + if (bad_address(pud)) + goto bad; + + pr_cont("PUD %lx ", pud_val(*pud)); + if (!pud_present(*pud) || pud_large(*pud)) + goto out; + + pmd = pmd_offset(pud, address); + if (bad_address(pmd)) + goto bad; + + pr_cont("PMD %lx ", pmd_val(*pmd)); + if (!pmd_present(*pmd) || pmd_large(*pmd)) + goto out; + + pte = pte_offset_kernel(pmd, address); + if (bad_address(pte)) + goto bad; + + pr_cont("PTE %lx", pte_val(*pte)); +out: + pr_cont("\n"); + return; +bad: + pr_info("BAD\n"); +} + +static void kvm_clear_page_fault(void) +{ + int i; + + for (i = 0; i < KVM_LAST_FAULT_COUNT; i++) + kvm_last_fault[i] = 0; +} + +static void kvm_log_page_fault(struct pt_regs *regs, unsigned long error_code, + unsigned long address) +{ + int i; + + /* + * Log information about the fault only if this is a fault + * we don't know about yet (or if the fault tracking buffer + * is full). + */ + for (i = 0; i < KVM_LAST_FAULT_COUNT; i++) { + if (!kvm_last_fault[i]) { + kvm_last_fault[i] = regs->ip; + break; + } + if (kvm_last_fault[i] == regs->ip) + return; + } + + if (i >= KVM_LAST_FAULT_COUNT) + pr_warn("KVM isolation: fault tracking buffer is full [%d]\n", + i); + + pr_info("KVM isolation: page fault #%d (%ld) at %pS on %px (%pS)\n", + i, error_code, (void *)regs->ip, + (void *)address, (void *)address); + if (page_fault_stack) + show_stack(NULL, (unsigned long *)regs->sp); +} + +/* + * KVM Page Fault Handler. The handler handles two simple cases: + * + * - If the fault occurs while using the kernel address space, then let + * the kernel handles the fault normally. + * + * - If the fault occurs while using the KVM address space, then switch + * to the kernel address space, and retry. + * + * It also handles a tricky case: if the fault occurs when using the KVM + * address space but while switching to the kernel address space then the + * switch is failing and we can't recover. In that case, we force switching + * to the kernel address space, print information and let the kernel + * handles the fault. + */ +static bool kvm_page_fault(struct pt_regs *regs, unsigned long error_code, + unsigned long address) +{ + struct mm_struct *active_mm = current->active_mm; + unsigned long cr3; + + /* + * First, do a quick and simple test to see if we are using + * the KVM address space. If we do then exit KVM isolation, + * log the fault and report that we have handled the fault. + */ + if (likely(active_mm == &kvm_mm)) { + kvm_isolation_exit(); + kvm_log_page_fault(regs, error_code, address); + return true; + } + + /* + * Verify that we are effectively using the kernel address space. + * When switching address space, active_mm is not necessarily up + * to date as it can already be set with the next mm while %cr3 + * has not been updated yet. So check loaded_mm which is updated + * after %cr3. + * + * If we are effectively using the kernel address space then report + * that we haven't handled the fault. + */ + if (this_cpu_read(cpu_tlbstate.loaded_mm) != &kvm_mm) + return false; + + /* + * We are actually using the KVM address space and faulting while + * switching address space. Force swiching to the kernel address + * space, log information and reported that we haven't handled + * the fault. + */ + cr3 = __read_cr3(); + write_cr3(build_cr3(active_mm->pgd, 0)); + kvm_dump_pagetable(kvm_mm.pgd, address); + kvm_dump_pagetable(active_mm->pgd, address); + printk(KERN_DEFAULT "KVM isolation: page fault %ld at %pS on %lx (%pS) while switching mm\n" + " cr3=%lx\n" + " kvm_mm=%px pgd=%px\n" + " active_mm=%px pgd=%px\n", + error_code, (void *)regs->ip, address, (void *)address, + cr3, + &kvm_mm, kvm_mm.pgd, + active_mm, active_mm->pgd); + dump_stack(); + + return false; +} + static int kvm_isolation_init_page_table(void) { @@ -1384,11 +1582,13 @@ static void kvm_isolation_uninit_mm(void) static void kvm_isolation_set_handlers(void) { kvm_set_isolation_exit_handler(kvm_isolation_exit); + kvm_page_fault_handler = kvm_page_fault; } static void kvm_isolation_clear_handlers(void) { kvm_set_isolation_exit_handler(NULL); + kvm_page_fault_handler = NULL; } int kvm_isolation_init_vm(struct kvm *kvm) @@ -1396,6 +1596,8 @@ int kvm_isolation_init_vm(struct kvm *kvm) if (!kvm_isolation()) return 0; + kvm_clear_page_fault(); + pr_debug("mapping kvm srcu sda\n"); return (kvm_copy_percpu_mapping(kvm->srcu.sda, diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 46df4c6..317e105 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -33,6 +33,10 @@ #define CREATE_TRACE_POINTS #include +bool (*kvm_page_fault_handler)(struct pt_regs *regs, unsigned long error_code, + unsigned long address); +EXPORT_SYMBOL(kvm_page_fault_handler); + /* * Returns 0 if mmiotrace is disabled, or if the fault is not * handled by mmiotrace: @@ -1253,6 +1257,14 @@ static int fault_in_kernel_space(unsigned long address) WARN_ON_ONCE(hw_error_code & X86_PF_PK); /* + * KVM might be able to handle the fault when running with the + * KVM address space. + */ + if (kvm_page_fault_handler && + kvm_page_fault_handler(regs, hw_error_code, address)) + return; + + /* * We can fault-in kernel-space virtual memory on-demand. The * 'reference' page table is init_mm.pgd. * From patchwork Mon May 13 14:38:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941091 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 307881390 for ; Mon, 13 May 2019 14:40:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 213EA28334 for ; Mon, 13 May 2019 14:40:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1547B28346; Mon, 13 May 2019 14:40:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 944F528334 for ; Mon, 13 May 2019 14:40:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730632AbfEMOk2 (ORCPT ); Mon, 13 May 2019 10:40:28 -0400 Received: from aserp2130.oracle.com ([141.146.126.79]:37798 "EHLO aserp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730623AbfEMOk0 (ORCPT ); Mon, 13 May 2019 10:40:26 -0400 Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEdHlv193231; Mon, 13 May 2019 14:39:53 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=LfHNGKdGVZ8SXafCfcR3Zof+WadTo/bKWh5zIaesw/M=; b=m2arcl/OoSz25ufQlPkSDyXa5f1frMRq2QeB0JzA4PV/uhaSpAPsJfmP6YeH0R+Y7+PC 50HJ+Yx7Z8gUn3m69WqI9tJvyt5ZPxzpbW+60gErtelRwYfFJw+X3V78E3pi6uU708Zl uEVUq4MhUxXUf1DFx84dYgGV/93RXZwfztVCXTVvknZhM6YZw2KSvN8qBdHsbd2ELVmx /7qs4qjDnvxh+fcskoo9PGQvTX2vY0RinWTrcvKjzeXlWoP9/wWiom7xFgGIssxbulhP IvPfd4bMncK92V8msleNSWffh5EcxE1AX3Pw6AsvfuevV3vA7E7XlcOF1Gt+oJHTRqAq Ww== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by aserp2130.oracle.com with ESMTP id 2sdkwdfm13-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:39:53 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQS022780; Mon, 13 May 2019 14:39:50 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 25/27] kvm/isolation: implement actual KVM isolation enter/exit Date: Mon, 13 May 2019 16:38:33 +0200 Message-Id: <1557758315-12667-26-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Liran Alon KVM isolation enter/exit is done by switching between the KVM address space and the kernel address space. Signed-off-by: Liran Alon Signed-off-by: Alexandre Chartre --- arch/x86/kvm/isolation.c | 30 ++++++++++++++++++++++++------ arch/x86/mm/tlb.c | 1 + include/linux/sched.h | 1 + 3 files changed, 26 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index db0a7ce..b0c789f 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -1383,11 +1383,13 @@ static bool kvm_page_fault(struct pt_regs *regs, unsigned long error_code, printk(KERN_DEFAULT "KVM isolation: page fault %ld at %pS on %lx (%pS) while switching mm\n" " cr3=%lx\n" " kvm_mm=%px pgd=%px\n" - " active_mm=%px pgd=%px\n", + " active_mm=%px pgd=%px\n" + " kvm_prev_mm=%px pgd=%px\n", error_code, (void *)regs->ip, address, (void *)address, cr3, &kvm_mm, kvm_mm.pgd, - active_mm, active_mm->pgd); + active_mm, active_mm->pgd, + current->kvm_prev_mm, current->kvm_prev_mm->pgd); dump_stack(); return false; @@ -1649,11 +1651,27 @@ void kvm_may_access_sensitive_data(struct kvm_vcpu *vcpu) kvm_isolation_exit(); } +static void kvm_switch_mm(struct mm_struct *mm) +{ + unsigned long flags; + + /* + * Disable interrupt before updating active_mm, otherwise if an + * interrupt occurs during the switch then the interrupt handler + * can be mislead about the mm effectively in use. + */ + local_irq_save(flags); + current->kvm_prev_mm = current->active_mm; + current->active_mm = mm; + switch_mm_irqs_off(current->kvm_prev_mm, mm, NULL); + local_irq_restore(flags); +} + void kvm_isolation_enter(void) { int err; - if (kvm_isolation()) { + if (kvm_isolation() && current->active_mm != &kvm_mm) { /* * Switches to kvm_mm should happen from vCPU thread, * which should not be a kernel thread with no mm @@ -1666,14 +1684,14 @@ void kvm_isolation_enter(void) current); return; } - /* TODO: switch to kvm_mm */ + kvm_switch_mm(&kvm_mm); } } void kvm_isolation_exit(void) { - if (kvm_isolation()) { + if (kvm_isolation() && current->active_mm == &kvm_mm) { /* TODO: Kick sibling hyperthread before switch to host mm */ - /* TODO: switch back to original mm */ + kvm_switch_mm(current->kvm_prev_mm); } } diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index a4db7f5..7ad5ad1 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -444,6 +444,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, switch_ldt(real_prev, next); } } +EXPORT_SYMBOL_GPL(switch_mm_irqs_off); /* * Please ignore the name of this function. It should be called diff --git a/include/linux/sched.h b/include/linux/sched.h index 80e1d75..b03680d 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1202,6 +1202,7 @@ struct task_struct { #ifdef CONFIG_HAVE_KVM /* Is the task mapped into the KVM address space? */ bool kvm_mapped; + struct mm_struct *kvm_prev_mm; #endif /* From patchwork Mon May 13 14:38:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941129 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B92641390 for ; Mon, 13 May 2019 14:42:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AC9E42832D for ; Mon, 13 May 2019 14:42:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9E80A28334; Mon, 13 May 2019 14:42:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 340D628306 for ; Mon, 13 May 2019 14:42:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729081AbfEMOl4 (ORCPT ); Mon, 13 May 2019 10:41:56 -0400 Received: from aserp2130.oracle.com ([141.146.126.79]:39134 "EHLO aserp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729742AbfEMOle (ORCPT ); Mon, 13 May 2019 10:41:34 -0400 Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEdhZc193417; Mon, 13 May 2019 14:40:01 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=WgSqF1N9AL8OkCOWJMjKijUQDrf/NZQ7sjOAy3KvX18=; b=F3ixlg5Del/kZFpkm5V339YqcYZIvytCmmZa2JMsPdT9dTNJdzNgJr2su3ZaFPX7+xSo OgE5doXVsWLJE6lg24N70KQRsqrw7KDlX0Lmv11xaiukFvSmC1PL9Z+pb6KOsPd3f9/Q TysNHNbYW17DUsb88Xzm+0xLKL38OFY7GPCscqM5Sr2RGXWabgevtm0BY8LIkE1hebHi ZDG3ZUmmf/4/2FsM+aInuyKqWofpo40CdOK/HySYEBTuzbf6QiQlJM7oxHIAKfyG/xd8 7z7TkAUzLmadWyiur41dp0GSyhTRruaBBuRMWSLXH6YSLiSP0FUqhXsgwlGrUBetwxxp pQ== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by aserp2130.oracle.com with ESMTP id 2sdkwdfm1k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:40:01 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQT022780; Mon, 13 May 2019 14:39:53 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 26/27] kvm/isolation: initialize the KVM page table with KVM memslots Date: Mon, 13 May 2019 16:38:34 +0200 Message-Id: <1557758315-12667-27-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=970 adultscore=15 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP KVM memslots can change after they have been created so new memslots have to be mapped when they are created. TODO: we currently don't unmapped old memslots, they should be unmapped when they are freed. Signed-off-by: Alexandre Chartre --- arch/x86/kvm/isolation.c | 39 +++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/isolation.h | 1 + arch/x86/kvm/x86.c | 3 +++ 3 files changed, 43 insertions(+), 0 deletions(-) diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index b0c789f..255b2da 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -1593,13 +1593,45 @@ static void kvm_isolation_clear_handlers(void) kvm_page_fault_handler = NULL; } +void kvm_isolation_check_memslots(struct kvm *kvm) +{ + struct kvm_range_mapping *rmapping; + int i, err; + + if (!kvm_isolation()) + return; + + for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) { + rmapping = kvm_get_range_mapping(kvm->memslots[i], NULL); + if (rmapping) + continue; + pr_debug("remapping kvm memslots[%d]\n", i); + err = kvm_copy_ptes(kvm->memslots[i], + sizeof(struct kvm_memslots)); + if (err) + pr_debug("failed to map kvm memslots[%d]\n", i); + } + +} + int kvm_isolation_init_vm(struct kvm *kvm) { + int err, i; + if (!kvm_isolation()) return 0; kvm_clear_page_fault(); + pr_debug("mapping kvm memslots\n"); + + for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) { + err = kvm_copy_ptes(kvm->memslots[i], + sizeof(struct kvm_memslots)); + if (err) + return err; + } + pr_debug("mapping kvm srcu sda\n"); return (kvm_copy_percpu_mapping(kvm->srcu.sda, @@ -1608,9 +1640,16 @@ int kvm_isolation_init_vm(struct kvm *kvm) void kvm_isolation_destroy_vm(struct kvm *kvm) { + int i; + if (!kvm_isolation()) return; + pr_debug("unmapping kvm memslots\n"); + + for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) + kvm_clear_range_mapping(kvm->memslots[i]); + pr_debug("unmapping kvm srcu sda\n"); kvm_clear_percpu_mapping(kvm->srcu.sda); diff --git a/arch/x86/kvm/isolation.h b/arch/x86/kvm/isolation.h index 2d7d016..1e55799 100644 --- a/arch/x86/kvm/isolation.h +++ b/arch/x86/kvm/isolation.h @@ -32,6 +32,7 @@ static inline bool kvm_isolation(void) extern void kvm_clear_range_mapping(void *ptr); extern int kvm_copy_percpu_mapping(void *percpu_ptr, size_t size); extern void kvm_clear_percpu_mapping(void *percpu_ptr); +extern void kvm_isolation_check_memslots(struct kvm *kvm); extern int kvm_add_task_mapping(struct task_struct *tsk); extern void kvm_cleanup_task_mapping(struct task_struct *tsk); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index e1cc3a6..7d98e9f 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -9438,6 +9438,7 @@ void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen) * mmio generation may have reached its maximum value. */ kvm_mmu_invalidate_mmio_sptes(kvm, gen); + kvm_isolation_check_memslots(kvm); } int kvm_arch_prepare_memory_region(struct kvm *kvm, @@ -9537,6 +9538,8 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, */ if (change != KVM_MR_DELETE) kvm_mmu_slot_apply_flags(kvm, (struct kvm_memory_slot *) new); + + kvm_isolation_check_memslots(kvm); } void kvm_arch_flush_shadow_all(struct kvm *kvm) From patchwork Mon May 13 14:38:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 10941123 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 623451390 for ; Mon, 13 May 2019 14:41:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5575F27861 for ; Mon, 13 May 2019 14:41:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 49A9F2832D; Mon, 13 May 2019 14:41:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CB39227861 for ; Mon, 13 May 2019 14:41:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730762AbfEMOln (ORCPT ); Mon, 13 May 2019 10:41:43 -0400 Received: from userp2130.oracle.com ([156.151.31.86]:47814 "EHLO userp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727786AbfEMOlm (ORCPT ); Mon, 13 May 2019 10:41:42 -0400 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x4DEd2K2181455; Mon, 13 May 2019 14:40:04 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=bVamqx26WofIGHx47Rok5tY+OeXg0YuA7C3C4gCjugU=; b=phOTgVyRSmfJlShasjXR81DKx3htuPesB+i0d59gwRf4Pvj7Dim335VGDly6TaryasbH p79YnJpLHxV6ux2E6qwXe0qzqxgl9bjTmvmDzhmAzNh5lJ8rrhJ6ObfkWiFmfAq3lRJz K05CgDIVBI1/f6xo2nFMOruVDoE6XxhauS8j3PiqHmyqCTNH4xnpgnaVEUHrFTBXW7f2 VLrhtdpEyD7KijA3Esx4FX4nbFn/WkBHai46PD3BIoP4/Hkwmpdc6QjQUAlSMhB2yCbA Ui+mFwRtvI4wzRar5LmOK42XboaGtSbIhchHkhsgVIP9GVewhcqNUP1Lh2Qu6jIn+TwR /w== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2130.oracle.com with ESMTP id 2sdnttfemt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 May 2019 14:40:04 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x4DEcZQU022780; Mon, 13 May 2019 14:39:56 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, alexandre.chartre@oracle.com Subject: [RFC KVM 27/27] kvm/isolation: initialize the KVM page table with KVM buses Date: Mon, 13 May 2019 16:38:35 +0200 Message-Id: <1557758315-12667-28-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> References: <1557758315-12667-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9255 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905130103 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP KVM buses can change after they have been created so new buses have to be mapped when they are created. Signed-off-by: Alexandre Chartre --- arch/x86/kvm/isolation.c | 37 +++++++++++++++++++++++++++++++++++++ arch/x86/kvm/isolation.h | 1 + arch/x86/kvm/x86.c | 13 ++++++++++++- include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 2 ++ 5 files changed, 53 insertions(+), 1 deletions(-) diff --git a/arch/x86/kvm/isolation.c b/arch/x86/kvm/isolation.c index 255b2da..329e769 100644 --- a/arch/x86/kvm/isolation.c +++ b/arch/x86/kvm/isolation.c @@ -1614,6 +1614,29 @@ void kvm_isolation_check_memslots(struct kvm *kvm) } +void kvm_isolation_check_buses(struct kvm *kvm) +{ + struct kvm_range_mapping *rmapping; + struct kvm_io_bus *bus; + int i, err; + + if (!kvm_isolation()) + return; + + for (i = 0; i < KVM_NR_BUSES; i++) { + bus = kvm->buses[i]; + rmapping = kvm_get_range_mapping(bus, NULL); + if (rmapping) + continue; + pr_debug("remapping kvm buses[%d]\n", i); + err = kvm_copy_ptes(bus, sizeof(*bus) + bus->dev_count * + sizeof(struct kvm_io_range)); + if (err) + pr_debug("failed to map kvm buses[%d]\n", i); + } + +} + int kvm_isolation_init_vm(struct kvm *kvm) { int err, i; @@ -1632,6 +1655,15 @@ int kvm_isolation_init_vm(struct kvm *kvm) return err; } + pr_debug("mapping kvm buses\n"); + + for (i = 0; i < KVM_NR_BUSES; i++) { + err = kvm_copy_ptes(kvm->buses[i], + sizeof(struct kvm_io_bus)); + if (err) + return err; + } + pr_debug("mapping kvm srcu sda\n"); return (kvm_copy_percpu_mapping(kvm->srcu.sda, @@ -1650,6 +1682,11 @@ void kvm_isolation_destroy_vm(struct kvm *kvm) for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) kvm_clear_range_mapping(kvm->memslots[i]); + pr_debug("unmapping kvm buses\n"); + + for (i = 0; i < KVM_NR_BUSES; i++) + kvm_clear_range_mapping(kvm->buses[i]); + pr_debug("unmapping kvm srcu sda\n"); kvm_clear_percpu_mapping(kvm->srcu.sda); diff --git a/arch/x86/kvm/isolation.h b/arch/x86/kvm/isolation.h index 1e55799..b048946 100644 --- a/arch/x86/kvm/isolation.h +++ b/arch/x86/kvm/isolation.h @@ -33,6 +33,7 @@ static inline bool kvm_isolation(void) extern int kvm_copy_percpu_mapping(void *percpu_ptr, size_t size); extern void kvm_clear_percpu_mapping(void *percpu_ptr); extern void kvm_isolation_check_memslots(struct kvm *kvm); +extern void kvm_isolation_check_buses(struct kvm *kvm); extern int kvm_add_task_mapping(struct task_struct *tsk); extern void kvm_cleanup_task_mapping(struct task_struct *tsk); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 7d98e9f..3ba1996 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -9253,6 +9253,13 @@ void kvm_arch_sync_events(struct kvm *kvm) cancel_delayed_work_sync(&kvm->arch.kvmclock_sync_work); cancel_delayed_work_sync(&kvm->arch.kvmclock_update_work); kvm_free_pit(kvm); + /* + * Note that kvm_isolation_destroy_vm() has to be called from + * here, and not from kvm_arch_destroy_vm() because it will unmap + * buses which are already destroyed when kvm_arch_destroy_vm() + * is invoked. + */ + kvm_isolation_destroy_vm(kvm); } int __x86_set_memory_region(struct kvm *kvm, int id, gpa_t gpa, u32 size) @@ -9331,7 +9338,6 @@ void kvm_arch_destroy_vm(struct kvm *kvm) x86_set_memory_region(kvm, IDENTITY_PAGETABLE_PRIVATE_MEMSLOT, 0, 0); x86_set_memory_region(kvm, TSS_PRIVATE_MEMSLOT, 0, 0); } - kvm_isolation_destroy_vm(kvm); if (kvm_x86_ops->vm_destroy) kvm_x86_ops->vm_destroy(kvm); kvm_pic_destroy(kvm); @@ -9909,6 +9915,11 @@ bool kvm_vector_hashing_enabled(void) } EXPORT_SYMBOL_GPL(kvm_vector_hashing_enabled); +void kvm_arch_buses_updated(struct kvm *kvm, struct kvm_io_bus *bus) +{ + kvm_isolation_check_buses(kvm); +} + EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_exit); EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_fast_mmio); EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_inj_virq); diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index ad24d9e..1291d8d 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -199,6 +199,7 @@ void kvm_io_bus_unregister_dev(struct kvm *kvm, enum kvm_bus bus_idx, struct kvm_io_device *dev); struct kvm_io_device *kvm_io_bus_get_dev(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t addr); +void kvm_arch_buses_updated(struct kvm *kvm, struct kvm_io_bus *bus); #ifdef CONFIG_KVM_ASYNC_PF struct kvm_async_pf { diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 3c0c3db..374e79f 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -3749,6 +3749,8 @@ int kvm_io_bus_register_dev(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t addr, synchronize_srcu_expedited(&kvm->srcu); kfree(bus); + kvm_arch_buses_updated(kvm, new_bus); + return 0; }