From patchwork Tue Jan 30 12:46:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoffer Dall X-Patchwork-Id: 10191791 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 53C5460388 for ; Tue, 30 Jan 2018 12:46:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4204B1FFB2 for ; Tue, 30 Jan 2018 12:46:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 35AC5288D2; Tue, 30 Jan 2018 12:46:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id A376C1FFB2 for ; Tue, 30 Jan 2018 12:46:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=pDuzBM0m53Fj6rAQD0LFzZZ/47bx2EOf68l2GvG82v0=; b=tHb 3PdeCXRed1Fss9eYTFPRkQizWIhfjTRMy4M8aLgR8ZTBvySi3lMOiaMgpyghD/6XpAN8B+TPTrymp zW2jFxEUje0uW0VneJMobUwA4jw1/M/xowcriENJ8r1HXunlJ5UinprakXp/DBrtxnP844ECV+ZeI h9UsRkv7ohtH2Vlgz/NlCu3YBDR7aMRPcy23vug67DJ/NO51HYfjjbBSIeFm8C3EcPq4tgvcaLcRF pZsgf+0I+IlBTNPXm+XLhkUyinZx3rQFk1KCVOIQRK4XfeN6CwTT7RyvzmV0vNI2YGHY1MLLJLnxC eWRf6sU2qKqqXsoDoN1/Wh68oXU1dkg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.89 #1 (Red Hat Linux)) id 1egVIy-0001ff-0G; Tue, 30 Jan 2018 12:46:28 +0000 Received: from mail-wm0-x241.google.com ([2a00:1450:400c:c09::241]) by bombadil.infradead.org with esmtps (Exim 4.89 #1 (Red Hat Linux)) id 1egVIu-0001fW-PU for linux-arm-kernel@lists.infradead.org; Tue, 30 Jan 2018 12:46:26 +0000 Received: by mail-wm0-x241.google.com with SMTP id b21so857111wme.4 for ; Tue, 30 Jan 2018 04:46:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id; bh=BYNSzmsKn3OnhgqpVRSX4IIDcxng2hrVookjaSDVbFY=; b=fjkeveAyTli6K8+NgY1stIBmu+FNw4Ihmtdtax+pCud5RagCShe/g9PixURXTnaMDL +anj/sqYaEzKhqx8ahydIFjqdP5x34sysMHibTlca+NiNWXN3XYBF69PJMC9hzRFqM7K zOtrVFIgKQDCmCXbSv4rJjUbBDmp4z0Ae9+4s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=BYNSzmsKn3OnhgqpVRSX4IIDcxng2hrVookjaSDVbFY=; b=oSuTiyjl7lr8sQ1ZJq7CbvcnRrtTPs8uEswh0guxiWPEjwDS2WXkzPk7J95zEHbmvl w9F6RrkJzahhN1q4br7Zc+oGkafVGIaUQhsWcMjpclWAN2dvqff3sa9K0uQszjKTG/ii Fdbud85GX2uE/XTI+ZQwe2aMzKnN1F1jMNEBwCu9YPLEcmUiNHFa01GrE1X4uwwM6Cko FkuSXENqJasfy60MU7YLCHbHFjZNEE1yxRfoFI0Vs/TRJyZbkpQxobpyK3JbyUgmFGL9 q12LJYX9gY1L0qjSXGkgbLx96wjCfUWEuj6IdvlsXSOw4vSFRnKFfyqX14n+lBKTHeyE RZdw== X-Gm-Message-State: AKwxyte97dnyefIO73nbByM37CEsuxNFds+cR8iOksDhyExe0L8I8TZK 7Wv/YQX4m4Bd0uc2N/LDK0YUeAB2wvE= X-Google-Smtp-Source: AH8x225uC8FJT9arlSHHOGdu4zmMqiiaSKYGL5vvJEakQNmvxaD0JOytlIeyjktzvbspVqfgq+jAVA== X-Received: by 10.80.148.217 with SMTP id t25mr51952745eda.121.1517316372638; Tue, 30 Jan 2018 04:46:12 -0800 (PST) Received: from localhost.localdomain (x50d2404e.cust.hiper.dk. [80.210.64.78]) by smtp.gmail.com with ESMTPSA id i6sm1849396edl.57.2018.01.30.04.46.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 30 Jan 2018 04:46:11 -0800 (PST) From: Christoffer Dall To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu Subject: [PATCH] KVM: arm/arm64: Fix arch timers with userspace irqchips Date: Tue, 30 Jan 2018 13:46:09 +0100 Message-Id: <20180130124609.15076-1-christoffer.dall@linaro.org> X-Mailer: git-send-email 2.14.2 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Marc Zyngier , stable@vger.kernel.org, Christoffer Dall , kvm@vger.kernel.org, Alexander Graf MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP When introducing support for irqchip in userspace we needed a way to mask the timer signal to prevent the guest continuously exiting due to a screaming timer. We did this by disabling the corresponding percpu interrupt on the host interrupt controller, because we cannot rely on the host system having a GIC, and therefore cannot make any assumptions about having an active state to hide the timer signal. Unfortunately, when introducing this feature, it became entirely possible that a VCPU which belongs to a VM that has a userspace irqchip can disable the vtimer irq on the host on some physical CPU, and then go away without ever enabling the vimter irq on that physical CPU again. This means that using irqchips in userspace on a system that also supports running VMs with an in-kernel GIC can prevent forward progress from in-kernel GIC VMs. Later on, when we started taking virtual timer interrupts in the arch timer code, we would also leave this timer state active for userspace irqchip VMs, because we leave it up to a VGIC-enabled guest to deactivate the hardware IRQ using the HW bit in the LR. Both issues are solved by only using the enable/disable trick on systems that do not have a host GIC which supports the active state, because all VMs on such systems must use irqchips in userspace. Systems that have a working GIC with support for an active state use the active state to mask the timer signal for both userspace an in-kernel irqchips. Cc: Alexander Graf Cc: # v4.12+ Fixes: d9e139778376 ("KVM: arm/arm64: Support arch timers with a userspace gic") Signed-off-by: Christoffer Dall Reviewed-by: Marc Zyngier --- This conflicts horribly with everything when applied to either kvmarm/queue or kvmarm/master. Therefore, this patch is written for (and applies to) v4.15 with kvmarm/queue merged and should therefore apply cleanly after v4.16-rc1. An example with this patch applied can be found on kvmarm/temp-for-v4.16-rc2. I plan on sending this along with any other potential fixes post v4.16-rc1. virt/kvm/arm/arch_timer.c | 77 ++++++++++++++++++++++++++--------------------- 1 file changed, 42 insertions(+), 35 deletions(-) diff --git a/virt/kvm/arm/arch_timer.c b/virt/kvm/arm/arch_timer.c index 70268c0bec79..228906ceb722 100644 --- a/virt/kvm/arm/arch_timer.c +++ b/virt/kvm/arm/arch_timer.c @@ -35,6 +35,7 @@ static struct timecounter *timecounter; static unsigned int host_vtimer_irq; static u32 host_vtimer_irq_flags; +static bool has_gic_active_state; static const struct kvm_irq_level default_ptimer_irq = { .irq = 30, @@ -69,25 +70,6 @@ static void soft_timer_cancel(struct hrtimer *hrt, struct work_struct *work) cancel_work_sync(work); } -static void kvm_vtimer_update_mask_user(struct kvm_vcpu *vcpu) -{ - struct arch_timer_context *vtimer = vcpu_vtimer(vcpu); - - /* - * When using a userspace irqchip with the architected timers, we must - * prevent continuously exiting from the guest, and therefore mask the - * physical interrupt by disabling it on the host interrupt controller - * when the virtual level is high, such that the guest can make - * forward progress. Once we detect the output level being - * de-asserted, we unmask the interrupt again so that we exit from the - * guest when the timer fires. - */ - if (vtimer->irq.level) - disable_percpu_irq(host_vtimer_irq); - else - enable_percpu_irq(host_vtimer_irq, 0); -} - static irqreturn_t kvm_arch_timer_handler(int irq, void *dev_id) { struct kvm_vcpu *vcpu = *(struct kvm_vcpu **)dev_id; @@ -107,8 +89,8 @@ static irqreturn_t kvm_arch_timer_handler(int irq, void *dev_id) kvm_timer_update_irq(vcpu, true, vtimer); if (static_branch_unlikely(&userspace_irqchip_in_use) && - unlikely(!irqchip_in_kernel(vcpu->kvm))) - kvm_vtimer_update_mask_user(vcpu); + unlikely(!irqchip_in_kernel(vcpu->kvm)) && !has_gic_active_state) + disable_percpu_irq(host_vtimer_irq); return IRQ_HANDLED; } @@ -460,13 +442,16 @@ static void set_cntvoff(u64 cntvoff) kvm_call_hyp(__kvm_timer_set_cntvoff, low, high); } -static void kvm_timer_vcpu_load_vgic(struct kvm_vcpu *vcpu) +static void kvm_timer_vcpu_load_gic(struct kvm_vcpu *vcpu) { struct arch_timer_context *vtimer = vcpu_vtimer(vcpu); bool phys_active; int ret; - phys_active = kvm_vgic_map_is_active(vcpu, vtimer->irq.irq); + if (irqchip_in_kernel(vcpu->kvm)) + phys_active = kvm_vgic_map_is_active(vcpu, vtimer->irq.irq); + else + phys_active = vtimer->irq.level; ret = irq_set_irqchip_state(host_vtimer_irq, IRQCHIP_STATE_ACTIVE, @@ -474,9 +459,24 @@ static void kvm_timer_vcpu_load_vgic(struct kvm_vcpu *vcpu) WARN_ON(ret); } -static void kvm_timer_vcpu_load_user(struct kvm_vcpu *vcpu) +static void kvm_timer_vcpu_load_nogic(struct kvm_vcpu *vcpu) { - kvm_vtimer_update_mask_user(vcpu); + struct arch_timer_context *vtimer = vcpu_vtimer(vcpu); + + /* + * When using a userspace irqchip with the architected timers and a + * host interrupt controller that doesn't support an active state, we + * must still we must prevent continuously exiting from the guest, and + * therefore mask the physical interrupt by disabling it on the host + * interrupt controller when the virtual level is high, such that the + * guest can make forward progress. Once we detect the output level + * being de-asserted, we unmask the interrupt again so that we exit + * from the guest when the timer fires. + */ + if (vtimer->irq.level) + disable_percpu_irq(host_vtimer_irq); + else + enable_percpu_irq(host_vtimer_irq, host_vtimer_irq_flags); } void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) @@ -487,10 +487,10 @@ void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) if (unlikely(!timer->enabled)) return; - if (unlikely(!irqchip_in_kernel(vcpu->kvm))) - kvm_timer_vcpu_load_user(vcpu); + if (has_gic_active_state) + kvm_timer_vcpu_load_gic(vcpu); else - kvm_timer_vcpu_load_vgic(vcpu); + kvm_timer_vcpu_load_nogic(vcpu); set_cntvoff(vtimer->cntvoff); @@ -555,18 +555,23 @@ static void unmask_vtimer_irq_user(struct kvm_vcpu *vcpu) { struct arch_timer_context *vtimer = vcpu_vtimer(vcpu); - if (unlikely(!irqchip_in_kernel(vcpu->kvm))) { - __timer_snapshot_state(vtimer); - if (!kvm_timer_should_fire(vtimer)) { - kvm_timer_update_irq(vcpu, false, vtimer); - kvm_vtimer_update_mask_user(vcpu); - } + __timer_snapshot_state(vtimer); + if (!kvm_timer_should_fire(vtimer)) { + kvm_timer_update_irq(vcpu, false, vtimer); + if (!has_gic_active_state) + enable_percpu_irq(host_vtimer_irq, host_vtimer_irq_flags); } } void kvm_timer_sync_hwstate(struct kvm_vcpu *vcpu) { - unmask_vtimer_irq_user(vcpu); + struct arch_timer_cpu *timer = &vcpu->arch.timer_cpu; + + if (unlikely(!timer->enabled)) + return; + + if (unlikely(!irqchip_in_kernel(vcpu->kvm))) + unmask_vtimer_irq_user(vcpu); } int kvm_timer_vcpu_reset(struct kvm_vcpu *vcpu) @@ -753,6 +758,8 @@ int kvm_timer_hyp_init(bool has_gic) kvm_err("kvm_arch_timer: error setting vcpu affinity\n"); goto out_free_irq; } + + has_gic_active_state = true; } kvm_info("virtual timer IRQ%d\n", host_vtimer_irq);