From patchwork Thu Jun 6 16:54:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 10980103 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AF0011398 for ; Thu, 6 Jun 2019 16:55:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 910822766D for ; Thu, 6 Jun 2019 16:55:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 84A6328A17; Thu, 6 Jun 2019 16:55:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 28AF82892D for ; Thu, 6 Jun 2019 16:55:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=VY/mqcWr9CCrRQbOA0L92DfTTDv9cpWXMsh/2Eyhu1E=; b=DhtyHbs6+dZsmr i+XZHmtRZRQUcZW7ci01ZpdSSAMDeYUAoVyhaGerrweTmhanogDSTm9N5k+zLxJ1hcuquMwaoBSzG mYzjNJuQdhmdOpTbp/vSY1Xj3H6HBMBlnguVBKACaB/FZuckW2h6z02xNZaMhGYtzE3tNjN2kN5tH I2s7rWv8lov22HJOvD1HFb20Wg+w13N5llnHB5dCjF9AiF3ZNtIwotiPIG0cucC0+O/HEvZmSwZJF ffgXFhBjHRX+kn1FhYPt7XEEiQMrNIXcuEAm3ysay5OwtF39kzKbavPetPjjZLxd8g7QgiG4ZGJz1 A9oL8nHp3ofFp2oMy+aA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hYvfQ-0005oQ-My; Thu, 06 Jun 2019 16:55:08 +0000 Received: from foss.arm.com ([217.140.101.70]) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hYvfN-0005EO-Ky for linux-arm-kernel@lists.infradead.org; Thu, 06 Jun 2019 16:55:07 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3F5AF374; Thu, 6 Jun 2019 09:55:05 -0700 (PDT) Received: from filthy-habits.cambridge.arm.com (filthy-habits.cambridge.arm.com [10.1.197.61]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 70B2D3F690; Thu, 6 Jun 2019 09:55:03 -0700 (PDT) From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Subject: [PATCH 0/8] KVM: arm/arm64: vgic: ITS translation cache Date: Thu, 6 Jun 2019 17:54:47 +0100 Message-Id: <20190606165455.162478-1-marc.zyngier@arm.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190606_095505_702707_F7B0A22F X-CRM114-Status: GOOD ( 13.60 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Suzuki K Poulose , "Raslan, KarimAllah" , Julien Thierry , Christoffer Dall , Eric Auger , James Morse , Zenghui Yu Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP It recently became apparent[1] that our LPI injection path is not as efficient as it could be when injecting interrupts coming from a VFIO assigned device. Although the proposed patch wasn't 100% correct, it outlined at least two issues: (1) Injecting an LPI from VFIO always results in a context switch to a worker thread: no good (2) We have no way of amortising the cost of translating a DID+EID pair to an LPI number The reason for (1) is that we may sleep when translating an LPI, so we do need a context process. A way to fix that is to implement a small LPI translation cache that could be looked up from an atomic context. It would also solve (2). This is what this small series proposes. it implements a very basic LRU cache of pre-translated LPIs, which gets used to implement kvm_arch_set_irq_inatomic. The size of the cache is currently hard-coded at 4 times the number of vcpus, a number I have randomly chosen with the utmost care. Does it work? well, it doesn't crash, and is thus perfect. More seriously, I don't really have a way to benchmark it directly, so my observations are indirect: On a TX2 system, I run a 4 vcpu VM with an Ethernet interface passed to it directly. From the host, I inject interrupts using debugfs. In parallel, I look at the number of context switch, and the number of interrupts on the host. Without this series, I get the same number for both IRQ and CS (about half a million of each per second is pretty easy to reach). With this series, the number of context switches drops to something pretty small (in the low 2k), while the number of interrupts stays the same. Yes, this is a pretty rubbish benchmark, what did you expect? ;-) So I'm putting this out for people with real workloads to try out and report what they see. [1] https://lore.kernel.org/lkml/1552833373-19828-1-git-send-email-yuzenghui@huawei.com/ Marc Zyngier (8): KVM: arm/arm64: vgic: Add LPI translation cache definition KVM: arm/arm64: vgic: Add __vgic_put_lpi_locked primitive KVM: arm/arm64: vgic-its: Cache successful MSI->LPI translation KVM: arm/arm64: vgic-its: Add kvm parameter to vgic_its_free_collection KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on specific commands KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on disabling LPIs KVM: arm/arm64: vgic-its: Check the LPI translation cache on MSI injection KVM: arm/arm64: vgic-irqfd: Implement kvm_arch_set_irq_inatomic include/kvm/arm_vgic.h | 10 +++ virt/kvm/arm/vgic/vgic-init.c | 34 ++++++++ virt/kvm/arm/vgic/vgic-irqfd.c | 36 ++++++-- virt/kvm/arm/vgic/vgic-its.c | 143 +++++++++++++++++++++++++++++-- virt/kvm/arm/vgic/vgic-mmio-v3.c | 4 +- virt/kvm/arm/vgic/vgic.c | 26 ++++-- virt/kvm/arm/vgic/vgic.h | 6 ++ 7 files changed, 238 insertions(+), 21 deletions(-)