From patchwork Tue Jun 16 02:23:15 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Rutherford X-Patchwork-Id: 6612701 Return-Path: X-Original-To: patchwork-kvm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 3FA0C9F399 for ; Tue, 16 Jun 2015 02:28:52 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id CEC5720786 for ; Tue, 16 Jun 2015 02:28:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5443520780 for ; Tue, 16 Jun 2015 02:28:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752060AbbFPC2q (ORCPT ); Mon, 15 Jun 2015 22:28:46 -0400 Received: from mail-ig0-f181.google.com ([209.85.213.181]:38453 "EHLO mail-ig0-f181.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751690AbbFPC2o (ORCPT ); Mon, 15 Jun 2015 22:28:44 -0400 Received: by igblz2 with SMTP id lz2so4764219igb.1 for ; Mon, 15 Jun 2015 19:28:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=from:to:subject:date:message-id; bh=j9BQt03Kyvj1K9JnZ1PduRrLdJN5QgtCzi9JdBeIL7k=; b=TcyC04xCq9B8M263vALstTOC1MlQ/fw5UjQBWYNWLRBSFvb7DL2UjW7j/cE/ebj90v MMYJaOBmZ4qJbQWAS1uNrvTog9m2act5Ra5qNdIvM1qe6BM17ddNFbB/DodkDp0YBUqk seg3EzP0Ecc0UEz09CKWT7RcEdpY4tv6YeA6kkTCeWglkPTZmRJEYeuSY0K3+ESUxq5d c6kQE9VPCPJ5af0ZZB0fLrRv0ai5AMs8eFausBXUh1XdpnCMNOAMfK97QhTS2v+tLwR/ 8a+4vbs6P9+FNJgFBAk5F47fYTOXoZxXgJGkkZRHIy46U53R1875++W0mEy4koTG5UXJ 3bIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:subject:date:message-id; bh=j9BQt03Kyvj1K9JnZ1PduRrLdJN5QgtCzi9JdBeIL7k=; b=XU6dyx5SvYNp2xzA4W+v+FfWbGvNLN+hPPYmiLtGsUCHSz9euyYwqLRh7T0QyWuLGE kUJrUuCyWCNa9T6KlmF9e489mJ1u9cYN5LtO1pa9SUflor9H1ppq272eEMn8jb4bmUSG bT4KfpaLrBcnK+kaDa3kW1OhEyfNKy+G2vZvbVg3j3kn2LpFZtcZ8aLL/33zISRukBWA VTjqfPJFJOEaU4iWqyL4xT66w9i4e0WL2xeguf2m/vnKSc4mmrTS3eVWqUq8Ay13Y8yL ZPal3j7FypwVCCpiSSdsyDNaKS86oO5gFQ3AGGDUn8eLuxWbvH9Q8zbbzkIXSaPmAzhs GiEg== X-Gm-Message-State: ALoCoQnaO2RCLEzR8IQDQsXLbj0qCsAhqVvpTxM3RoZn3WJh/Ul/QXDVqn7IVBHg03HbzUfnXVYY X-Received: by 10.50.147.10 with SMTP id tg10mr25062701igb.36.1434421723399; Mon, 15 Jun 2015 19:28:43 -0700 (PDT) Received: from entropic.kir.corp.google.com ([172.31.8.128]) by mx.google.com with ESMTPSA id g3sm223036igi.10.2015.06.15.19.28.41 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 15 Jun 2015 19:28:42 -0700 (PDT) From: Steve Rutherford To: kvm@vger.kernel.org Subject: [PATCH v4 1/4] KVM: x86: Split the APIC from the rest of IRQCHIP. Date: Mon, 15 Jun 2015 19:23:15 -0700 Message-Id: <1434421398-6613-1-git-send-email-srutherford@google.com> X-Mailer: git-send-email 2.2.0.rc0.207.ga3a616c Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Spam-Status: No, score=-7.2 required=5.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, T_DKIM_INVALID, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP First patch in a series which enables the relocation of the PIC/IOAPIC to userspace. Adds capability KVM_CAP_SPLIT_IRQCHIP; KVM_CAP_SPLIT_IRQCHIP enables the construction of LAPICs without the rest of the irqchip. Compile tested for x86. Signed-off-by: Steve Rutherford Suggested-by: Andrew Honig --- Documentation/virtual/kvm/api.txt | 16 ++++++++++++++++ arch/powerpc/kvm/irq.h | 1 - arch/s390/kvm/irq.h | 1 - arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/i8254.c | 5 ++++- arch/x86/kvm/ioapic.h | 9 +++++++++ arch/x86/kvm/irq.h | 6 ++++++ arch/x86/kvm/irq_comm.c | 9 ++++++++- arch/x86/kvm/lapic.c | 9 ++++++--- arch/x86/kvm/vmx.c | 4 ++-- arch/x86/kvm/x86.c | 23 +++++++++++++++++++++-- include/kvm/arm_vgic.h | 1 + include/linux/kvm_host.h | 1 + include/uapi/linux/kvm.h | 1 + 14 files changed, 77 insertions(+), 11 deletions(-) diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt index 6955444..3143f9e 100644 --- a/Documentation/virtual/kvm/api.txt +++ b/Documentation/virtual/kvm/api.txt @@ -2979,6 +2979,7 @@ len must be a multiple of sizeof(struct kvm_s390_irq). It must be > 0 and it must not exceed (max_vcpus + 32) * sizeof(struct kvm_s390_irq), which is the maximum number of possibly pending cpu-local interrupts. + 5. The kvm_run structure ------------------------ @@ -3575,6 +3576,21 @@ struct { KVM handlers should exit to userspace with rc = -EREMOTE. +7.5 KVM_SPLIT_IRQCHIP + +Architectures: x86 +Parameters: None +Returns: 0 on success, -1 on error + +Create a local apic for each processor in the kernel. With this capability +enabled, the userspace VMM is expected to emulate the IOAPIC and PIC. + +This supersedes KVM_CREATE_IRQCHIP, creating only local APICs, but no in kernel +IOAPIC or PIC. This also enables in kernel routing of interrupt requests. + +Fails if VCPU has already been created, or if the irqchip is already in the +kernel (i.e. KVM_CREATE_IRQCHIP has already been called). + 8. Other capabilities. ---------------------- diff --git a/arch/powerpc/kvm/irq.h b/arch/powerpc/kvm/irq.h index 5a9a10b..772fa8c 100644 --- a/arch/powerpc/kvm/irq.h +++ b/arch/powerpc/kvm/irq.h @@ -16,5 +16,4 @@ static inline int irqchip_in_kernel(struct kvm *kvm) smp_rmb(); return ret; } - #endif diff --git a/arch/s390/kvm/irq.h b/arch/s390/kvm/irq.h index d98e415..9a21a86 100644 --- a/arch/s390/kvm/irq.h +++ b/arch/s390/kvm/irq.h @@ -18,5 +18,4 @@ static inline int irqchip_in_kernel(struct kvm *kvm) { return 1; } - #endif diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 7276107..af3225a 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -639,6 +639,8 @@ struct kvm_arch { bool boot_vcpu_runs_old_kvmclock; u64 disabled_quirks; + + bool irqchip_split; }; struct kvm_vm_stat { diff --git a/arch/x86/kvm/i8254.c b/arch/x86/kvm/i8254.c index 4dce6f8..de06e7e 100644 --- a/arch/x86/kvm/i8254.c +++ b/arch/x86/kvm/i8254.c @@ -35,6 +35,7 @@ #include #include +#include "ioapic.h" #include "irq.h" #include "i8254.h" #include "x86.h" @@ -333,7 +334,9 @@ static void create_pit_timer(struct kvm *kvm, u32 val, int is_period) struct kvm_kpit_state *ps = &kvm->arch.vpit->pit_state; s64 interval; - if (!irqchip_in_kernel(kvm) || ps->flags & KVM_PIT_FLAGS_HPET_LEGACY) + if (!irqchip_in_kernel(kvm) || + !ioapic_in_kernel(kvm) || + ps->flags & KVM_PIT_FLAGS_HPET_LEGACY) return; interval = muldiv64(val, NSEC_PER_SEC, KVM_PIT_FREQ); diff --git a/arch/x86/kvm/ioapic.h b/arch/x86/kvm/ioapic.h index ca0b0b4..d8cc54b 100644 --- a/arch/x86/kvm/ioapic.h +++ b/arch/x86/kvm/ioapic.h @@ -98,6 +98,15 @@ static inline struct kvm_ioapic *ioapic_irqchip(struct kvm *kvm) return kvm->arch.vioapic; } +static inline int ioapic_in_kernel(struct kvm *kvm) +{ + int ret; + + ret = (ioapic_irqchip(kvm) != NULL); + smp_rmb(); + return ret; +} + static inline bool kvm_ioapic_handles_vector(struct kvm *kvm, int vector) { struct kvm_ioapic *ioapic = kvm->arch.vioapic; diff --git a/arch/x86/kvm/irq.h b/arch/x86/kvm/irq.h index ad68c73..2f13dd5 100644 --- a/arch/x86/kvm/irq.h +++ b/arch/x86/kvm/irq.h @@ -83,11 +83,17 @@ static inline struct kvm_pic *pic_irqchip(struct kvm *kvm) return kvm->arch.vpic; } +static inline int irqchip_split(struct kvm *kvm) +{ + return kvm->arch.irqchip_split; +} + static inline int irqchip_in_kernel(struct kvm *kvm) { int ret; ret = (pic_irqchip(kvm) != NULL); + ret |= irqchip_split(kvm); smp_rmb(); return ret; } diff --git a/arch/x86/kvm/irq_comm.c b/arch/x86/kvm/irq_comm.c index 9efff9e..67f6b62 100644 --- a/arch/x86/kvm/irq_comm.c +++ b/arch/x86/kvm/irq_comm.c @@ -208,7 +208,7 @@ void kvm_free_irq_source_id(struct kvm *kvm, int irq_source_id) goto unlock; } clear_bit(irq_source_id, &kvm->arch.irq_sources_bitmap); - if (!irqchip_in_kernel(kvm)) + if (!ioapic_in_kernel(kvm)) goto unlock; kvm_ioapic_clear_all(kvm->arch.vioapic, irq_source_id); @@ -328,3 +328,10 @@ int kvm_setup_default_irq_routing(struct kvm *kvm) return kvm_set_irq_routing(kvm, default_routing, ARRAY_SIZE(default_routing), 0); } + +static const struct kvm_irq_routing_entry empty_routing[] = {}; + +int kvm_setup_empty_irq_routing(struct kvm *kvm) +{ + return kvm_set_irq_routing(kvm, empty_routing, 0, 0); +} diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index c789e00..717b4e2 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -209,7 +209,8 @@ out: if (old) kfree_rcu(old, rcu); - kvm_vcpu_request_scan_ioapic(kvm); + if (!irqchip_split(kvm)) + kvm_vcpu_request_scan_ioapic(kvm); } static inline void apic_set_spiv(struct kvm_lapic *apic, u32 val) @@ -1827,7 +1828,8 @@ void kvm_apic_post_state_restore(struct kvm_vcpu *vcpu, kvm_x86_ops->hwapic_isr_update(vcpu->kvm, apic_find_highest_isr(apic)); kvm_make_request(KVM_REQ_EVENT, vcpu); - kvm_rtc_eoi_tracking_restore_one(vcpu); + if (!irqchip_split(vcpu->kvm)) + kvm_rtc_eoi_tracking_restore_one(vcpu); } void __kvm_migrate_apic_timer(struct kvm_vcpu *vcpu) @@ -1910,7 +1912,8 @@ static void apic_sync_pv_eoi_to_guest(struct kvm_vcpu *vcpu, /* Cache not set: could be safe but we don't bother. */ apic->highest_isr_cache == -1 || /* Need EOI to update ioapic. */ - kvm_ioapic_handles_vector(vcpu->kvm, apic->highest_isr_cache)) { + kvm_ioapic_handles_vector(vcpu->kvm, apic->highest_isr_cache) || + irqchip_split(vcpu->kvm)) { /* * PV EOI was disabled by apic_sync_pv_eoi_from_guest * so we need not do anything here. diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c index 9cf5030..b4a239c 100644 --- a/arch/x86/kvm/vmx.c +++ b/arch/x86/kvm/vmx.c @@ -948,7 +948,7 @@ static inline bool cpu_has_vmx_tpr_shadow(void) static inline bool vm_need_tpr_shadow(struct kvm *kvm) { - return (cpu_has_vmx_tpr_shadow()) && (irqchip_in_kernel(kvm)); + return (cpu_has_vmx_tpr_shadow()) && irqchip_in_kernel(kvm); } static inline bool cpu_has_secondary_exec_ctrls(void) @@ -9419,7 +9419,7 @@ static void prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12) /* vmcs12's VM_ENTRY_LOAD_IA32_EFER and VM_ENTRY_IA32E_MODE are * emulated by vmx_set_efer(), below. */ - vm_entry_controls_init(vmx, + vm_entry_controls_init(vmx, (vmcs12->vm_entry_controls & ~VM_ENTRY_LOAD_IA32_EFER & ~VM_ENTRY_IA32E_MODE) | (vmcs_config.vmentry_ctrl & ~VM_ENTRY_IA32E_MODE)); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 79dde16..1cc6a65 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -2866,6 +2866,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) case KVM_CAP_TSC_DEADLINE_TIMER: case KVM_CAP_ENABLE_CAP_VM: case KVM_CAP_DISABLE_QUIRKS: + case KVM_CAP_SPLIT_IRQCHIP: #ifdef CONFIG_KVM_DEVICE_ASSIGNMENT case KVM_CAP_ASSIGN_DEV_IRQ: case KVM_CAP_PCI_2_3: @@ -3926,6 +3927,23 @@ static int kvm_vm_ioctl_enable_cap(struct kvm *kvm, kvm->arch.disabled_quirks = cap->args[0]; r = 0; break; + case KVM_CAP_SPLIT_IRQCHIP: { + mutex_lock(&kvm->lock); + r = -EEXIST; + if (irqchip_in_kernel(kvm)) + goto split_irqchip_unlock; + r = -EINVAL; + if (atomic_read(&kvm->online_vcpus)) + goto split_irqchip_unlock; + r = kvm_setup_empty_irq_routing(kvm); + if (r) + goto split_irqchip_unlock; + kvm->arch.irqchip_split = true; + r = 0; +split_irqchip_unlock: + mutex_unlock(&kvm->lock); + break; + } default: r = -EINVAL; break; @@ -4044,7 +4062,7 @@ long kvm_arch_vm_ioctl(struct file *filp, } r = -ENXIO; - if (!irqchip_in_kernel(kvm)) + if (!irqchip_in_kernel(kvm) || !ioapic_in_kernel(kvm)) goto get_irqchip_out; r = kvm_vm_ioctl_get_irqchip(kvm, chip); if (r) @@ -4068,7 +4086,7 @@ long kvm_arch_vm_ioctl(struct file *filp, } r = -ENXIO; - if (!irqchip_in_kernel(kvm)) + if (!irqchip_in_kernel(kvm) || !ioapic_in_kernel(kvm)) goto set_irqchip_out; r = kvm_vm_ioctl_set_irqchip(kvm, chip); if (r) @@ -4194,6 +4212,7 @@ long kvm_arch_vm_ioctl(struct file *filp, r = kvm_vm_ioctl_enable_cap(kvm, &cap); break; } + default: r = kvm_vm_ioctl_assigned_device(kvm, ioctl, arg); } diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index 133ea00..ffe1f4e 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -329,6 +329,7 @@ int kvm_vgic_vcpu_pending_irq(struct kvm_vcpu *vcpu); int kvm_vgic_vcpu_active_irq(struct kvm_vcpu *vcpu); #define irqchip_in_kernel(k) (!!((k)->arch.vgic.in_kernel)) +#define lapic_in_kernel(k) (irqchip_in_kernel(k)) #define vgic_initialized(k) (!!((k)->arch.vgic.nr_cpus)) #define vgic_ready(k) ((k)->arch.vgic.ready) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index a8bcbc9..7e2b41a 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -935,6 +935,7 @@ static inline int mmu_notifier_retry(struct kvm *kvm, unsigned long mmu_seq) #endif int kvm_setup_default_irq_routing(struct kvm *kvm); +int kvm_setup_empty_irq_routing(struct kvm *kvm); int kvm_set_irq_routing(struct kvm *kvm, const struct kvm_irq_routing_entry *entries, unsigned nr, diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index 75bd9f7..1e6f6c3 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -815,6 +815,7 @@ struct kvm_ppc_smmu_info { #define KVM_CAP_S390_IRQ_STATE 114 #define KVM_CAP_PPC_HWRNG 115 #define KVM_CAP_DISABLE_QUIRKS 116 +#define KVM_CAP_SPLIT_IRQCHIP 117 #ifdef KVM_CAP_IRQ_ROUTING