From patchwork Mon Oct 8 05:30:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Mackerras X-Patchwork-Id: 10630009 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 131F514BD for ; Mon, 8 Oct 2018 05:31:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EEDE728BAB for ; Mon, 8 Oct 2018 05:31:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DAB6528B3D; Mon, 8 Oct 2018 05:31:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3535928B3D for ; Mon, 8 Oct 2018 05:31:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726013AbeJHMlY (ORCPT ); Mon, 8 Oct 2018 08:41:24 -0400 Received: from ozlabs.org ([203.11.71.1]:60773 "EHLO ozlabs.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725842AbeJHMlY (ORCPT ); Mon, 8 Oct 2018 08:41:24 -0400 Received: from authenticated.ozlabs.org (localhost [127.0.0.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPSA id 42T89W5gRbz9sCV; Mon, 8 Oct 2018 16:31:27 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=ozlabs.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ozlabs.org; s=201707; t=1538976688; bh=AXY7PeAuI/p6JfB3jENisBVQeaBugOhnNSJLk9pvxYs=; h=From:To:Cc:Subject:Date:From; b=iIQIevQhkdS0PjbAhLNwAD9SValHgY/ZbfyiICzqHyXKpViNJs2Valmb2oVct1xrP mQ3jwFIyMrKCIqnqvmwF4pyJ/RDMwSufnecbeB+0RbwlYhD62BUButLBjK2hiD2Ytw Bww99lj3ETAM0LKTv93fV8eeRXFiBLJEhQsQgH+J3cqsYPhHavPl1vpdjXkn2XQ3n+ SI+ZVMSBq0VPbaL20kV+CGayxtvwAM4uYgVntATLDBqtw/V8NlKHkMGedy7WXY0+q3 fhSupnnSHNARE0+lvaP96wxjXlu4SXqZrYwyjzLrdSCVrId+6aRhrVm1Q7lEnd1BnB 5S4UYi/KplLgQ== From: Paul Mackerras To: kvm@vger.kernel.org, kvm-ppc@vger.kernel.org Cc: David Gibson , linuxppc-dev@ozlabs.org Subject: [PATCH v5 00/33] KVM: PPC: Book3S HV: Nested HV virtualization Date: Mon, 8 Oct 2018 16:30:46 +1100 Message-Id: <1538976679-1363-1-git-send-email-paulus@ozlabs.org> X-Mailer: git-send-email 2.7.4 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This patch series implements nested virtualization in the KVM-HV module for radix guests on POWER9 systems. Unlike PR KVM, nested guests are able to run in supervisor mode, meaning that performance is much better than with PR KVM, and is very close to the performance of a non-nested guests for most things. The way this works is that each nested guest is also a guest of the real hypervisor, also known as the level 0 or L0 hypervisor, which runs in the CPU's hypervisor mode. Its guests are at level 1, and when a L1 system wants to run a nested guest, it performs hypercalls to L0 to set up a virtual partition table in its (L1's) memory and to enter the L2 guest. The L0 hypervisor maintains a shadow partition-scoped page table for the L2 guest and demand-faults entries into it by translating the L1 real addresses in the partition-scoped page table in L1 memory into L0 real addresses and puts them in the shadow partition-scoped page table for L2. Essentially what this is doing is providing L1 with the ability to do (some) hypervisor functions using paravirtualization; optionally, TLB invalidations can be done through emulation of the tlbie instruction rather than a hypercall. Along the way, this implements a new guest entry/exit path for radix guests on POWER9 systems which is written almost entirely in C and does not do any of the inter-thread coordination that the existing entry/exit path does. It is only used for radix guests and when indep_threads_mode=Y (the default). The limitations of this scheme are: - Host and all nested hypervisors and their guests must be in radix mode. - Nested hypervisors cannot use indep_threads_mode=N. - If the host (i.e. the L0 hypervisor) has indep_threads_mode=N then only one nested vcpu can be run on any core at any given time; the secondary threads will do nothing. - A nested hypervisor can't use a smaller page size than the base page size of the hypervisor(s) above it. - A nested hypervisor is limited to having at most 1023 guests below it, each of which can have at most NR_CPUS virtual CPUs (and the virtual CPU ids have to be < NR_CPUS as well). This patch series is against the kvm tree's next branch. Changes in this version since version 4: - Added KVM_PPC_NO_HASH to flags field of struct kvm_ppc_smmu_info rather than disabling the KVM_PPC_GET_SMMU_INFO ioctl entirely. - Make sure the hypercalls for controlling nested guests will fail if the guest is in HPT mode. - Made the KVM_CAP_PPC_MMU_HASH_V3 capability report false in a nested hypervisor. - Fixed a bug causing HPT guests to fail to execute real-mode hypercalls correctly. - Fix crashes seen on VM exit or when switching from HPT to radix, due to leftover rmap values. - Removed enable/disable flag from KVM_ENABLE_CAP on the KVM_CAP_PPC_NESTED_HV capability; it always enables. - Fixed a bug causing memory corruption on nested guest startup, and a bug where we were never clearing bits in the cpu_in_guest cpumask. - Simplified some code in kvmhv_run_single_vcpu. Paul. Documentation/virtual/kvm/api.txt | 19 + arch/powerpc/include/asm/asm-prototypes.h | 21 + arch/powerpc/include/asm/book3s/64/mmu-hash.h | 12 + .../powerpc/include/asm/book3s/64/tlbflush-radix.h | 1 + arch/powerpc/include/asm/hvcall.h | 41 + arch/powerpc/include/asm/kvm_asm.h | 4 +- arch/powerpc/include/asm/kvm_book3s.h | 45 +- arch/powerpc/include/asm/kvm_book3s_64.h | 118 +- arch/powerpc/include/asm/kvm_book3s_asm.h | 3 + arch/powerpc/include/asm/kvm_booke.h | 4 +- arch/powerpc/include/asm/kvm_host.h | 16 +- arch/powerpc/include/asm/kvm_ppc.h | 4 + arch/powerpc/include/asm/ppc-opcode.h | 1 + arch/powerpc/include/asm/reg.h | 2 + arch/powerpc/include/uapi/asm/kvm.h | 1 + arch/powerpc/kernel/asm-offsets.c | 5 +- arch/powerpc/kernel/cpu_setup_power.S | 4 +- arch/powerpc/kvm/Makefile | 3 +- arch/powerpc/kvm/book3s.c | 43 +- arch/powerpc/kvm/book3s_64_mmu_hv.c | 7 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 718 ++++++++--- arch/powerpc/kvm/book3s_emulate.c | 13 +- arch/powerpc/kvm/book3s_hv.c | 852 ++++++++++++- arch/powerpc/kvm/book3s_hv_builtin.c | 92 +- arch/powerpc/kvm/book3s_hv_interrupts.S | 95 +- arch/powerpc/kvm/book3s_hv_nested.c | 1291 ++++++++++++++++++++ arch/powerpc/kvm/book3s_hv_ras.c | 10 + arch/powerpc/kvm/book3s_hv_rm_xics.c | 13 +- arch/powerpc/kvm/book3s_hv_rmhandlers.S | 823 +++++++------ arch/powerpc/kvm/book3s_hv_tm.c | 6 +- arch/powerpc/kvm/book3s_hv_tm_builtin.c | 5 +- arch/powerpc/kvm/book3s_pr.c | 5 +- arch/powerpc/kvm/book3s_xics.c | 14 +- arch/powerpc/kvm/book3s_xive.c | 63 + arch/powerpc/kvm/book3s_xive_template.c | 8 - arch/powerpc/kvm/bookehv_interrupts.S | 8 +- arch/powerpc/kvm/emulate_loadstore.c | 1 - arch/powerpc/kvm/powerpc.c | 15 +- arch/powerpc/kvm/tm.S | 250 ++-- arch/powerpc/kvm/trace_book3s.h | 1 - arch/powerpc/mm/tlb-radix.c | 9 + include/uapi/linux/kvm.h | 2 + tools/perf/arch/powerpc/util/book3s_hv_exits.h | 1 - 43 files changed, 3806 insertions(+), 843 deletions(-)