From patchwork Wed Jan 24 20:48:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 13529650 Received: from out-179.mta1.migadu.com (out-179.mta1.migadu.com [95.215.58.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC0871350D6 for ; Wed, 24 Jan 2024 20:49:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129370; cv=none; b=LDKShtdUGHbOmGd2RAkc6tPiYvhDUPYTtMKPcJpyttk3/UVf6Zi/evCbW7dZApNQd0s+l8RcOAwRHHJBjmMwXoQzV3Psmgn/Nq2oQ9Jya0GwKNzvj4H2+6oLSe93VnuUCNbunbHMXWpGJTjaFGgMI5NADqtj3/YBc6uiGRAq8ZY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129370; c=relaxed/simple; bh=PdNfihDw36AYZxr6ju2QEwA0g69yapalLlGfGIOnoZU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=s8fRPORK/ssOxQ0NrRclKFA60RReyjAJLQeH6ldnEXJlh1jNSF5L32stjloV+05uLLl58uLxM/FXnuLlGuBJk+UnBIU7v9HihhsmcDJnfNFrl+S180jfs08I0AsPO6SeZtBJBG7eWJaSOQMSaOma2mLjQrUJjLQ/OdOpCfowBDc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=bxCj97vD; arc=none smtp.client-ip=95.215.58.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="bxCj97vD" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706129363; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WnI9sUjZ8H7mxRuV1VRi9CrF4BbT2F07UGflwBcqAWw=; b=bxCj97vDt8gAugnT3u4Gbfgyeju+jDaY6QCzqsEOwuHbQqhZNeUjQYcsuX26D1pHyl1fqs bl8c1LaPh9cN6VSRHPCDhQGt+UaDN47RpJloQuRnRYTaynPpXAzCcOGdwLov7xgiSmuWRT PthOCBvVCPqr+rjQCW4F6JanfNiNxnE= From: Oliver Upton To: kvmarm@lists.linux.dev Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Suzuki K Poulose , Zenghui Yu , Raghavendra Rao Ananta , Jing Zhang , Oliver Upton Subject: [PATCH 01/15] KVM: arm64: vgic: Store LPIs in an xarray Date: Wed, 24 Jan 2024 20:48:55 +0000 Message-ID: <20240124204909.105952-2-oliver.upton@linux.dev> In-Reply-To: <20240124204909.105952-1-oliver.upton@linux.dev> References: <20240124204909.105952-1-oliver.upton@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Using a linked-list for LPIs is less than ideal as it of course requires iterative searches to find a particular entry. An xarray is a better data structure for this use case, as it provides faster searches and can still handle a potentially sparse range of INTID allocations. Start by storing LPIs in an xarray, punting usage of the xarray to a subsequent change. Signed-off-by: Oliver Upton --- arch/arm64/kvm/vgic/vgic-init.c | 3 +++ arch/arm64/kvm/vgic/vgic-its.c | 13 +++++++++++++ arch/arm64/kvm/vgic/vgic.c | 1 + include/kvm/arm_vgic.h | 2 ++ 4 files changed, 19 insertions(+) diff --git a/arch/arm64/kvm/vgic/vgic-init.c b/arch/arm64/kvm/vgic/vgic-init.c index e949e1d0fd9f..411719053107 100644 --- a/arch/arm64/kvm/vgic/vgic-init.c +++ b/arch/arm64/kvm/vgic/vgic-init.c @@ -56,6 +56,7 @@ void kvm_vgic_early_init(struct kvm *kvm) INIT_LIST_HEAD(&dist->lpi_list_head); INIT_LIST_HEAD(&dist->lpi_translation_cache); raw_spin_lock_init(&dist->lpi_list_lock); + xa_init_flags(&dist->lpi_xa, XA_FLAGS_LOCK_IRQ); } /* CREATION */ @@ -366,6 +367,8 @@ static void kvm_vgic_dist_destroy(struct kvm *kvm) if (vgic_supports_direct_msis(kvm)) vgic_v4_teardown(kvm); + + xa_destroy(&dist->lpi_xa); } static void __kvm_vgic_vcpu_destroy(struct kvm_vcpu *vcpu) diff --git a/arch/arm64/kvm/vgic/vgic-its.c b/arch/arm64/kvm/vgic/vgic-its.c index e2764d0ffa9f..f152d670113f 100644 --- a/arch/arm64/kvm/vgic/vgic-its.c +++ b/arch/arm64/kvm/vgic/vgic-its.c @@ -52,6 +52,12 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid, if (!irq) return ERR_PTR(-ENOMEM); + ret = xa_reserve_irq(&dist->lpi_xa, intid, GFP_KERNEL_ACCOUNT); + if (ret) { + kfree(irq); + return ERR_PTR(ret); + } + INIT_LIST_HEAD(&irq->lpi_list); INIT_LIST_HEAD(&irq->ap_list); raw_spin_lock_init(&irq->irq_lock); @@ -86,6 +92,13 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid, goto out_unlock; } + ret = xa_err(xa_store(&dist->lpi_xa, intid, irq, 0)); + if (ret) { + xa_release(&dist->lpi_xa, intid); + kfree(irq); + return ERR_PTR(ret); + } + list_add_tail(&irq->lpi_list, &dist->lpi_list_head); dist->lpi_list_count++; diff --git a/arch/arm64/kvm/vgic/vgic.c b/arch/arm64/kvm/vgic/vgic.c index db2a95762b1b..c126014f8395 100644 --- a/arch/arm64/kvm/vgic/vgic.c +++ b/arch/arm64/kvm/vgic/vgic.c @@ -131,6 +131,7 @@ void __vgic_put_lpi_locked(struct kvm *kvm, struct vgic_irq *irq) return; list_del(&irq->lpi_list); + xa_erase(&dist->lpi_xa, irq->intid); dist->lpi_list_count--; kfree(irq); diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index 8cc38e836f54..795b35656b54 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -275,6 +276,7 @@ struct vgic_dist { /* Protects the lpi_list and the count value below. */ raw_spinlock_t lpi_list_lock; + struct xarray lpi_xa; struct list_head lpi_list_head; int lpi_list_count; From patchwork Wed Jan 24 20:48:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 13529652 Received: from out-176.mta1.migadu.com (out-176.mta1.migadu.com [95.215.58.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC96E1350E0 for ; Wed, 24 Jan 2024 20:49:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129371; cv=none; b=THr+UVh+oWQne6iFJvrg3SIhFFaJ/6ctOVJDdx8sOXpeeMGie3GQm4kX84u5AAs3icEZt4G9vZPkhKGdVtFj6WWr/nW6kc06+DQ62DbBiERtZ5u6ZrmydgPWC8eJpaIrNz60jO0/P/H6y04EcdZWC6+yxDb+G0t+MMG8NK+nVW8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129371; c=relaxed/simple; bh=s/7p2y0MuJJ2yjCghMin92mq2VbvJX2KFI0tYjDUUaI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jgZy8Q34w+bBzGd1Twpj7J0W5WuUTMun+a+9khGWsLXtVJ1Hr1aCm2eG7aTxFufh8A6U+W0EZUfOF/NdG+APYZrWeONTv8nDjoIjB7glUPsukyxoBHBPQ3+XXVBTtXLNNxTfqkOjsXInsXQJKFeZd8RPIEzUwbSxW//oOvE+qEQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=V/OJnTnl; arc=none smtp.client-ip=95.215.58.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="V/OJnTnl" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706129365; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SBBQZKa3HE+jnKMX9AT3xH561Qb4a05EumcmqYcHZtQ=; b=V/OJnTnlZZInSqrkwJ+S5kImPcNYXsDJnD7LwwwmFMyZqHTTkClxwmvA4aVBNfXdzX/AjM NwUpC5bi++ja91vcM/MWbG4sCWP3fra3DMqjoQlb7SLwGjA2TL9CTeIqXEb3qC0dQlzIZn 2tIEN4WDY4rM/Lx4XB0lAS8HtEuT80M= From: Oliver Upton To: kvmarm@lists.linux.dev Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Suzuki K Poulose , Zenghui Yu , Raghavendra Rao Ananta , Jing Zhang , Oliver Upton Subject: [PATCH 02/15] KVM: arm64: vgic: Use xarray to find LPI in vgic_get_lpi() Date: Wed, 24 Jan 2024 20:48:56 +0000 Message-ID: <20240124204909.105952-3-oliver.upton@linux.dev> In-Reply-To: <20240124204909.105952-1-oliver.upton@linux.dev> References: <20240124204909.105952-1-oliver.upton@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Iterating over the LPI linked-list is less than ideal when the desired index is already known. Use the INTID to index the LPI xarray instead. Signed-off-by: Oliver Upton --- arch/arm64/kvm/vgic/vgic.c | 19 +++++-------------- 1 file changed, 5 insertions(+), 14 deletions(-) diff --git a/arch/arm64/kvm/vgic/vgic.c b/arch/arm64/kvm/vgic/vgic.c index c126014f8395..d90c42ff051d 100644 --- a/arch/arm64/kvm/vgic/vgic.c +++ b/arch/arm64/kvm/vgic/vgic.c @@ -54,8 +54,9 @@ struct vgic_global kvm_vgic_global_state __ro_after_init = { */ /* - * Iterate over the VM's list of mapped LPIs to find the one with a - * matching interrupt ID and return a reference to the IRQ structure. + * Index the VM's xarray of mapped LPIs and return a reference to the IRQ + * structure. The caller is expected to call vgic_put_irq() later once it's + * finished with the IRQ. */ static struct vgic_irq *vgic_get_lpi(struct kvm *kvm, u32 intid) { @@ -65,20 +66,10 @@ static struct vgic_irq *vgic_get_lpi(struct kvm *kvm, u32 intid) raw_spin_lock_irqsave(&dist->lpi_list_lock, flags); - list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) { - if (irq->intid != intid) - continue; - - /* - * This increases the refcount, the caller is expected to - * call vgic_put_irq() later once it's finished with the IRQ. - */ + irq = xa_load(&dist->lpi_xa, intid); + if (irq) vgic_get_irq_kref(irq); - goto out_unlock; - } - irq = NULL; -out_unlock: raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags); return irq; From patchwork Wed Jan 24 20:48:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 13529651 Received: from out-185.mta1.migadu.com (out-185.mta1.migadu.com [95.215.58.185]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CB7661339B9 for ; Wed, 24 Jan 2024 20:49:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.185 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129370; cv=none; b=Z9lVcUJdm4HP8O/qRkAYRZjwdlHmNDao3Upduu+MxO9rT5GbE6l+JBWPru7KFECTjzxOCB79CSVg4A1zJKnleJCWr4x1QwVe9p9+6J2QBc6WodungVVdXnLbClpaLvHeOyxaNbahBITJPeoNWb30Ym2FoIZxtbIFDBXTcv0SGWI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129370; c=relaxed/simple; bh=GMgH9J5hFBKgsWvx97GgZsEcvW0lAaU3NZWwr1EKK64=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SyrEfvrMD/Q8wFMXU+6ELKtwTcdeEuF1MEmPT5KSENWopm+ncY+TbxistHhtZNIKsdvr7WethKWChTpk6hV3eMzhEH9OLUkToULV2bhbCo3SpMhApsfm1FYe5O1Q5eftpS2XtZvfp/JTb4Rup/VNoQm0DkYC1+5aXtgFvKp5DBc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=cB7dMMs0; arc=none smtp.client-ip=95.215.58.185 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="cB7dMMs0" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706129366; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TLZEdrKWhCuJ5JOctfy+1PuB2n+NWvfNuRcYY3F4nLM=; b=cB7dMMs07aU4BoOiS0SKvW5LqHgKDTGC/xJRVCTHk2g4nANCfNh3++/bvRvClHEJ2OANlO La/GV0DUT1iNY6SszC+9hLfb66yct+5+0+rWYUfc4QVaL7k4w/aP8ySHReJ0uGUyZnaSks 7WAhiv8cxN+CaB9+iq14Zs+4aXV3geU= From: Oliver Upton To: kvmarm@lists.linux.dev Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Suzuki K Poulose , Zenghui Yu , Raghavendra Rao Ananta , Jing Zhang , Oliver Upton Subject: [PATCH 03/15] KVM: arm64: vgic-v3: Iterate the xarray to find pending LPIs Date: Wed, 24 Jan 2024 20:48:57 +0000 Message-ID: <20240124204909.105952-4-oliver.upton@linux.dev> In-Reply-To: <20240124204909.105952-1-oliver.upton@linux.dev> References: <20240124204909.105952-1-oliver.upton@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Start walking the LPI xarray to find pending LPIs in preparation for the removal of the LPI linked-list. Note that the 'basic' iterator is chosen here as each iteration needs to drop the xarray read lock (RCU) as reads/writes to guest memory can potentially block. Signed-off-by: Oliver Upton --- arch/arm64/kvm/vgic/vgic-v3.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/vgic/vgic-v3.c b/arch/arm64/kvm/vgic/vgic-v3.c index 9465d3706ab9..4ea3340786b9 100644 --- a/arch/arm64/kvm/vgic/vgic-v3.c +++ b/arch/arm64/kvm/vgic/vgic-v3.c @@ -380,6 +380,7 @@ int vgic_v3_save_pending_tables(struct kvm *kvm) struct vgic_irq *irq; gpa_t last_ptr = ~(gpa_t)0; bool vlpi_avail = false; + unsigned long index; int ret = 0; u8 val; @@ -396,7 +397,7 @@ int vgic_v3_save_pending_tables(struct kvm *kvm) vlpi_avail = true; } - list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) { + xa_for_each(&dist->lpi_xa, index, irq) { int byte_offset, bit_nr; struct kvm_vcpu *vcpu; gpa_t pendbase, ptr; From patchwork Wed Jan 24 20:48:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 13529654 Received: from out-188.mta1.migadu.com (out-188.mta1.migadu.com [95.215.58.188]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B63B1350C5 for ; Wed, 24 Jan 2024 20:49:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.188 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129372; cv=none; b=aYMKZ0hCoJYvofzPyLRIfVAdSCYxUDexdFsJirMu2NJyoow8ykc7E38XK5oCUKLIxS/Q3AiC2/3EsFoT9b6LoRy4Kr2hGDglPv9CbHsW8VvFbFl2q2LGYIvDPrtdRuogdK+XKi7VDjCTHHRp9t9UM2ORsY3MBLS7NOuNueHbSyQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129372; c=relaxed/simple; bh=EQxOcqkChok4sxfbnzwEFj1fVU3Aa1XVGuAjzKFsnoo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oAnubGH3jPtP4Qoc4Bh5BXbAbOb2/uQp0a8wm16qCjwt9FyuFUU6eUaDcY4E14g1ea0VWYZqOaJkcFE+bi1ianrUiOo6n/MOcq9bRn8jYatehUyktndb4iQwmtJ3kEIDR7HrjBUmJgSQ/eCr5mo7ph8eQTHBYVkPUIjt/zqqjZ4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=GqkyWNab; arc=none smtp.client-ip=95.215.58.188 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="GqkyWNab" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706129368; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UgcFzuCZCGBGq43AzcrLHC5Pd72zyqu8lxZidsgbsEI=; b=GqkyWNabnKnkeObvMJ/FRzoCT6oKeXtbNT5mUjYJrbgYJtHOAm65oO8fkeCI7syffofLXL ooJtHG8vAa9CtBbrL+DrwtAFIsfGEisWrb2SDY+I0iTrgTUL1ycfLvlKG+fitri6mmMnzB MhXOGyzCrXAn5QZ2PbudfeZP+HGxhxA= From: Oliver Upton To: kvmarm@lists.linux.dev Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Suzuki K Poulose , Zenghui Yu , Raghavendra Rao Ananta , Jing Zhang , Oliver Upton Subject: [PATCH 04/15] KVM: arm64: vgic-its: Walk the LPI xarray in vgic_copy_lpi_list() Date: Wed, 24 Jan 2024 20:48:58 +0000 Message-ID: <20240124204909.105952-5-oliver.upton@linux.dev> In-Reply-To: <20240124204909.105952-1-oliver.upton@linux.dev> References: <20240124204909.105952-1-oliver.upton@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Start iterating the LPI xarray in anticipation of removing the LPI linked-list. Signed-off-by: Oliver Upton --- arch/arm64/kvm/vgic/vgic-its.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/vgic/vgic-its.c b/arch/arm64/kvm/vgic/vgic-its.c index f152d670113f..a2d95a279798 100644 --- a/arch/arm64/kvm/vgic/vgic-its.c +++ b/arch/arm64/kvm/vgic/vgic-its.c @@ -332,6 +332,7 @@ static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq, int vgic_copy_lpi_list(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 **intid_ptr) { struct vgic_dist *dist = &kvm->arch.vgic; + XA_STATE(xas, &dist->lpi_xa, 0); struct vgic_irq *irq; unsigned long flags; u32 *intids; @@ -350,7 +351,9 @@ int vgic_copy_lpi_list(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 **intid_ptr) return -ENOMEM; raw_spin_lock_irqsave(&dist->lpi_list_lock, flags); - list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) { + rcu_read_lock(); + + xas_for_each(&xas, irq, U32_MAX) { if (i == irq_count) break; /* We don't need to "get" the IRQ, as we hold the list lock. */ @@ -358,6 +361,8 @@ int vgic_copy_lpi_list(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 **intid_ptr) continue; intids[i++] = irq->intid; } + + rcu_read_unlock(); raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags); *intid_ptr = intids; From patchwork Wed Jan 24 20:48:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 13529655 Received: from out-177.mta1.migadu.com (out-177.mta1.migadu.com [95.215.58.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A96041353F1 for ; Wed, 24 Jan 2024 20:49:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129374; cv=none; b=gCSl49pAfDF3V4TABcye89rArfUXGqxCH/485WTYupH+VoRJpGRTjY7cKsmj4R3nrhYD24CPzCE7rrq7W7hwhKj1ryPTwFqTMeOkJ3jAyA50Yn3eEWoEul2I6aYx6K1sYigEfc6CDyLxJmmtoAqnT3yKBo9y0DW2GQAaE4wquTY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129374; c=relaxed/simple; bh=OWQPlGbJfv74O6Ejj1hlEx603Lza1Ej43wwVmD20d6g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BpohHk7ktI7GiTcztjXHRvvYTsCwfKol8XSUfc6S+m8Cb4/NTCRbuZdvZCxGk5vksUi0IaEyTG9P+D78FxLYb/npbyZCrJMdszoWqB+T2ofbS7qsQVZtSCxes7wyhVoUvVNiY8rZAEm6Bn1yrEG0OHHCU+sAlwa8caFNUGJzLJ8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=gFQ+wblN; arc=none smtp.client-ip=95.215.58.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="gFQ+wblN" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706129370; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gMHb8qwOgYdG295rXfmcu6C9eixNCueL5GSPuTMDr/M=; b=gFQ+wblN+DmTLfOxsU/cOMUDyxDFW0xoVLAf2Jm7QHEUY17RoeBiXHE9NhjmcsVxvLDzTU AL8jxKTuIA8AMTw+YFAb2VHW6n4BAPi4BClJ/iVglMe0Cuz+/MYPiMMOioLpy4LXMBp0US hbJHrg86q579EmLXYL6RG99jEhxkLN4= From: Oliver Upton To: kvmarm@lists.linux.dev Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Suzuki K Poulose , Zenghui Yu , Raghavendra Rao Ananta , Jing Zhang , Oliver Upton Subject: [PATCH 05/15] KVM: arm64: vgic: Get rid of the LPI linked-list Date: Wed, 24 Jan 2024 20:48:59 +0000 Message-ID: <20240124204909.105952-6-oliver.upton@linux.dev> In-Reply-To: <20240124204909.105952-1-oliver.upton@linux.dev> References: <20240124204909.105952-1-oliver.upton@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT All readers of LPI configuration have been transitioned to use the LPI xarray. Get rid of the linked-list altogether. Signed-off-by: Oliver Upton --- arch/arm64/kvm/vgic/vgic-init.c | 1 - arch/arm64/kvm/vgic/vgic-its.c | 7 ++----- arch/arm64/kvm/vgic/vgic.c | 1 - include/kvm/arm_vgic.h | 1 - 4 files changed, 2 insertions(+), 8 deletions(-) diff --git a/arch/arm64/kvm/vgic/vgic-init.c b/arch/arm64/kvm/vgic/vgic-init.c index 411719053107..e25672d6e846 100644 --- a/arch/arm64/kvm/vgic/vgic-init.c +++ b/arch/arm64/kvm/vgic/vgic-init.c @@ -53,7 +53,6 @@ void kvm_vgic_early_init(struct kvm *kvm) { struct vgic_dist *dist = &kvm->arch.vgic; - INIT_LIST_HEAD(&dist->lpi_list_head); INIT_LIST_HEAD(&dist->lpi_translation_cache); raw_spin_lock_init(&dist->lpi_list_lock); xa_init_flags(&dist->lpi_xa, XA_FLAGS_LOCK_IRQ); diff --git a/arch/arm64/kvm/vgic/vgic-its.c b/arch/arm64/kvm/vgic/vgic-its.c index a2d95a279798..0486d3779d11 100644 --- a/arch/arm64/kvm/vgic/vgic-its.c +++ b/arch/arm64/kvm/vgic/vgic-its.c @@ -74,10 +74,8 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid, * There could be a race with another vgic_add_lpi(), so we need to * check that we don't add a second list entry with the same LPI. */ - list_for_each_entry(oldirq, &dist->lpi_list_head, lpi_list) { - if (oldirq->intid != intid) - continue; - + oldirq = xa_load(&dist->lpi_xa, intid); + if (oldirq) { /* Someone was faster with adding this LPI, lets use that. */ kfree(irq); irq = oldirq; @@ -99,7 +97,6 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid, return ERR_PTR(ret); } - list_add_tail(&irq->lpi_list, &dist->lpi_list_head); dist->lpi_list_count++; out_unlock: diff --git a/arch/arm64/kvm/vgic/vgic.c b/arch/arm64/kvm/vgic/vgic.c index d90c42ff051d..e58ce68e325c 100644 --- a/arch/arm64/kvm/vgic/vgic.c +++ b/arch/arm64/kvm/vgic/vgic.c @@ -121,7 +121,6 @@ void __vgic_put_lpi_locked(struct kvm *kvm, struct vgic_irq *irq) if (!kref_put(&irq->refcount, vgic_irq_release)) return; - list_del(&irq->lpi_list); xa_erase(&dist->lpi_xa, irq->intid); dist->lpi_list_count--; diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index 795b35656b54..39037db3fa90 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -277,7 +277,6 @@ struct vgic_dist { /* Protects the lpi_list and the count value below. */ raw_spinlock_t lpi_list_lock; struct xarray lpi_xa; - struct list_head lpi_list_head; int lpi_list_count; /* LPI translation cache */ From patchwork Wed Jan 24 20:49:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 13529656 Received: from out-171.mta1.migadu.com (out-171.mta1.migadu.com [95.215.58.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 802A21350CA for ; Wed, 24 Jan 2024 20:49:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129377; cv=none; b=CJInGtTUh0FE2EpgnEftOmRtSjaSS6CyOFOaH2wK1GUeTgTGDpzDILzo9aaiQ9KWAF0T0R1L5dk7n4fmItUkmQNhhV6wc+dx5ccKkcio7sryDJfE2sNrVhmsASAsKAjrkluE/h31HKY2QnupGwGbCpefD9W8EBUu8mQLCi3oxHY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129377; c=relaxed/simple; bh=Yci9UjKWnyQCjg1gteyPivkD77+UxH3VnJgIayCPj4k=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tK/OYcLJh3iI4GTr2sU7pdX0y0xrk6CS+lt9FEw5bKnYNoqbY0g5fUKU9zUisUFli3JfRX5FNwWESSMVDmoJBffO9LK8k2pYpg6LOh6ylapmmErvOsCJ1uuVsI95maTnfi/rNg7aatSrp6/03fYC8iqyrvg9f+fPMGXBKuFQCjo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=XcVgE3Vg; arc=none smtp.client-ip=95.215.58.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="XcVgE3Vg" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706129372; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PrQ+PVEC9XhGPAIG43CquUH33xiU8286LZeimDf+tfE=; b=XcVgE3VgULstQEd6lTi1Iv7You3egtz/rN9kfGrDlz3W7rAqPQKIIz1QetaUv4w8sd0kbc 0ZduumdAP8sR03fdgB7AYV9ov9ti8tKwOb3Nl+c0mSRQEhBT3byFUHK6SVt2Xnkl/Ti5jN GwJ/1ZrumKK/p3h/6uLrK50NDKkNd0c= From: Oliver Upton To: kvmarm@lists.linux.dev Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Suzuki K Poulose , Zenghui Yu , Raghavendra Rao Ananta , Jing Zhang , Oliver Upton Subject: [PATCH 06/15] KVM: arm64: vgic: Use atomics to count LPIs Date: Wed, 24 Jan 2024 20:49:00 +0000 Message-ID: <20240124204909.105952-7-oliver.upton@linux.dev> In-Reply-To: <20240124204909.105952-1-oliver.upton@linux.dev> References: <20240124204909.105952-1-oliver.upton@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Switch to using atomics for LPI accounting, allowing vgic_irq references to be dropped in parallel. Signed-off-by: Oliver Upton --- arch/arm64/kvm/vgic/vgic-debug.c | 2 +- arch/arm64/kvm/vgic/vgic-its.c | 4 ++-- arch/arm64/kvm/vgic/vgic.c | 2 +- include/kvm/arm_vgic.h | 4 ++-- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/arch/arm64/kvm/vgic/vgic-debug.c b/arch/arm64/kvm/vgic/vgic-debug.c index 85606a531dc3..389025ce7749 100644 --- a/arch/arm64/kvm/vgic/vgic-debug.c +++ b/arch/arm64/kvm/vgic/vgic-debug.c @@ -149,7 +149,7 @@ static void print_dist_state(struct seq_file *s, struct vgic_dist *dist) seq_printf(s, "vgic_model:\t%s\n", v3 ? "GICv3" : "GICv2"); seq_printf(s, "nr_spis:\t%d\n", dist->nr_spis); if (v3) - seq_printf(s, "nr_lpis:\t%d\n", dist->lpi_list_count); + seq_printf(s, "nr_lpis:\t%d\n", atomic_read(&dist->lpi_count)); seq_printf(s, "enabled:\t%d\n", dist->enabled); seq_printf(s, "\n"); diff --git a/arch/arm64/kvm/vgic/vgic-its.c b/arch/arm64/kvm/vgic/vgic-its.c index 0486d3779d11..1d912a595b71 100644 --- a/arch/arm64/kvm/vgic/vgic-its.c +++ b/arch/arm64/kvm/vgic/vgic-its.c @@ -97,7 +97,7 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid, return ERR_PTR(ret); } - dist->lpi_list_count++; + atomic_inc(&dist->lpi_count); out_unlock: raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags); @@ -342,7 +342,7 @@ int vgic_copy_lpi_list(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 **intid_ptr) * command). If coming from another path (such as enabling LPIs), * we must be careful not to overrun the array. */ - irq_count = READ_ONCE(dist->lpi_list_count); + irq_count = atomic_read(&dist->lpi_count); intids = kmalloc_array(irq_count, sizeof(intids[0]), GFP_KERNEL_ACCOUNT); if (!intids) return -ENOMEM; diff --git a/arch/arm64/kvm/vgic/vgic.c b/arch/arm64/kvm/vgic/vgic.c index e58ce68e325c..5988d162b765 100644 --- a/arch/arm64/kvm/vgic/vgic.c +++ b/arch/arm64/kvm/vgic/vgic.c @@ -122,7 +122,7 @@ void __vgic_put_lpi_locked(struct kvm *kvm, struct vgic_irq *irq) return; xa_erase(&dist->lpi_xa, irq->intid); - dist->lpi_list_count--; + atomic_dec(&dist->lpi_count); kfree(irq); } diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index 39037db3fa90..e944536feee8 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -274,10 +274,10 @@ struct vgic_dist { */ u64 propbaser; - /* Protects the lpi_list and the count value below. */ + /* Protects the lpi_list. */ raw_spinlock_t lpi_list_lock; struct xarray lpi_xa; - int lpi_list_count; + atomic_t lpi_count; /* LPI translation cache */ struct list_head lpi_translation_cache; From patchwork Wed Jan 24 20:49:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 13529657 Received: from out-179.mta1.migadu.com (out-179.mta1.migadu.com [95.215.58.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AE32D1350C2 for ; Wed, 24 Jan 2024 20:49:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129378; cv=none; b=SkOzYgYTsvwLYv8oxUZiV9GT7jW7S3iVsCWAVic6bjXmIx+cjuEnWPsn7dmNTo/NQ3ZQnLkrhKdxDIyOBfYNEsuPeJB0TJfUgGRlYRrNojW7jnVCKe/E7WD+JmIkQyjIm5kD43i+wOpBrEoIvMJLNPiPAy1K4MiruWSm0Irgwk8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129378; c=relaxed/simple; bh=qY4V4V8zZDoaf3u8iSxofhASVbe4oRXpUzZDgzVWLOQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XVog6CM4sqk15DYpUFnbqhDIGcJ5wnJzkYBxJiH67QRGi84qDP7g7nOj2jtNuSBZRf9zcAkkGdhppv91NQ8pn4qUcXPNdNyA4LIeY641NJ0mePrmB7+laQpSQA2ms8hfOJzPWw8rv2e3lCFKq++fLeJTHfVpgmUfIp22rC1DQNE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=uq7HGbrw; arc=none smtp.client-ip=95.215.58.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="uq7HGbrw" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706129374; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yJV18VS+eQbuqvjOmYfLuHXF00F9ZRH1HOLRMBo/vo0=; b=uq7HGbrwbZDoDVYt6UeY5TwwNhOF9A0NTJQ/FkIx2MBncZWJU44UbMD36SUD9uJG2dvLYE Up0RemD5n/PvVlvHFuDO8LA8qfo2qCe3lMJXnT4IB7m1tsPueOYSbU9YQ6c6vzKTdtpeUT o1/SB2w8sLkWvcCjDaeN7ITpuHi5X08= From: Oliver Upton To: kvmarm@lists.linux.dev Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Suzuki K Poulose , Zenghui Yu , Raghavendra Rao Ananta , Jing Zhang , Oliver Upton Subject: [PATCH 07/15] KVM: arm64: vgic: Free LPI vgic_irq structs in an RCU-safe manner Date: Wed, 24 Jan 2024 20:49:01 +0000 Message-ID: <20240124204909.105952-8-oliver.upton@linux.dev> In-Reply-To: <20240124204909.105952-1-oliver.upton@linux.dev> References: <20240124204909.105952-1-oliver.upton@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Free the vgic_irq structs in an RCU-safe manner to allow reads of the LPI configuration data to happen in parallel with the release of LPIs. Signed-off-by: Oliver Upton --- arch/arm64/kvm/vgic/vgic.c | 2 +- include/kvm/arm_vgic.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/vgic/vgic.c b/arch/arm64/kvm/vgic/vgic.c index 5988d162b765..be3ed4c5e1fa 100644 --- a/arch/arm64/kvm/vgic/vgic.c +++ b/arch/arm64/kvm/vgic/vgic.c @@ -124,7 +124,7 @@ void __vgic_put_lpi_locked(struct kvm *kvm, struct vgic_irq *irq) xa_erase(&dist->lpi_xa, irq->intid); atomic_dec(&dist->lpi_count); - kfree(irq); + kfree_rcu(irq, rcu); } void vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq) diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index e944536feee8..a6f6c1583662 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -117,6 +117,7 @@ struct irq_ops { struct vgic_irq { raw_spinlock_t irq_lock; /* Protects the content of the struct */ + struct rcu_head rcu; struct list_head lpi_list; /* Used to link all LPIs together */ struct list_head ap_list; From patchwork Wed Jan 24 20:49:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 13529658 Received: from out-172.mta1.migadu.com (out-172.mta1.migadu.com [95.215.58.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6ADCC1350E8 for ; Wed, 24 Jan 2024 20:49:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129381; cv=none; b=m4RigZE1KmxXUZ95yc68vQxD58L6DSK0Tafn7WiQOiefzyJ2s7F8qa+/bL0e8I5pAf4nbRV32uhsg0igaG5C+9OkZoX6AXaskD/G5jAU065nbY9lNGuhyatx1WbIX4f0i+fEI/QuDhxm6Y/lhwxC9F4zGvhY2UXw3trPxg40L7Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129381; c=relaxed/simple; bh=DAkL2Y9gxAm/I7me//FA38b8lKO9PwoCRKLKU4/roWc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eyE5DH+YIn7DSn4au5tJfslNGOxWd8An3s0Gg9EdfagTMGwH9VGu2+a9zvumzE50fWL4jieV/eHv0kc+HbiWOpPVOMaSj04fclmEM68RbVLx1SaVlRCvHQ1sMcVXmD7+KuoevbqWc4x2kVOzUQ2s1nsjrDZdXR6QO/k+NjDVy3U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=SqbU3ANK; arc=none smtp.client-ip=95.215.58.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="SqbU3ANK" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706129376; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t+qjGFCXzXN4Bwc8iaYXwhCvz8/RNxUojqqBTdUJ0KM=; b=SqbU3ANKSAqHlEzCx0w6jK0PtznQH31HZOLK8yjb43D0hZl042x2NxiiTABADbuelBSBCv EW3i/orakJzSy5v1io8w5tQ+XghZRna9t02DIuJ8caJDLUbylgSUrD4voVXCBMdxBm8dBA pmlNRtkmpdXaHvrR99MoTP10smAyWZA= From: Oliver Upton To: kvmarm@lists.linux.dev Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Suzuki K Poulose , Zenghui Yu , Raghavendra Rao Ananta , Jing Zhang , Oliver Upton Subject: [PATCH 08/15] KVM: arm64: vgic: Rely on RCU protection in vgic_get_lpi() Date: Wed, 24 Jan 2024 20:49:02 +0000 Message-ID: <20240124204909.105952-9-oliver.upton@linux.dev> In-Reply-To: <20240124204909.105952-1-oliver.upton@linux.dev> References: <20240124204909.105952-1-oliver.upton@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Stop acquiring the lpi_list_lock in favor of RCU for protecting the read-side critical section in vgic_get_lpi(). In order for this to be safe, we also need to be careful not to take a reference on an irq with a refcount of 0, as it is about to be freed. Signed-off-by: Oliver Upton --- arch/arm64/kvm/vgic/vgic.c | 9 ++++----- arch/arm64/kvm/vgic/vgic.h | 11 ++++++++--- 2 files changed, 12 insertions(+), 8 deletions(-) diff --git a/arch/arm64/kvm/vgic/vgic.c b/arch/arm64/kvm/vgic/vgic.c index be3ed4c5e1fa..949af87bb599 100644 --- a/arch/arm64/kvm/vgic/vgic.c +++ b/arch/arm64/kvm/vgic/vgic.c @@ -62,15 +62,14 @@ static struct vgic_irq *vgic_get_lpi(struct kvm *kvm, u32 intid) { struct vgic_dist *dist = &kvm->arch.vgic; struct vgic_irq *irq = NULL; - unsigned long flags; - raw_spin_lock_irqsave(&dist->lpi_list_lock, flags); + rcu_read_lock(); irq = xa_load(&dist->lpi_xa, intid); - if (irq) - vgic_get_irq_kref(irq); + if (irq && !vgic_try_get_irq_kref(irq)) + irq = NULL; - raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags); + rcu_read_unlock(); return irq; } diff --git a/arch/arm64/kvm/vgic/vgic.h b/arch/arm64/kvm/vgic/vgic.h index 8d134569d0a1..20886f57416a 100644 --- a/arch/arm64/kvm/vgic/vgic.h +++ b/arch/arm64/kvm/vgic/vgic.h @@ -220,12 +220,17 @@ void vgic_v2_vmcr_sync(struct kvm_vcpu *vcpu); void vgic_v2_save_state(struct kvm_vcpu *vcpu); void vgic_v2_restore_state(struct kvm_vcpu *vcpu); -static inline void vgic_get_irq_kref(struct vgic_irq *irq) +static inline bool vgic_try_get_irq_kref(struct vgic_irq *irq) { if (irq->intid < VGIC_MIN_LPI) - return; + return true; + + return kref_get_unless_zero(&irq->refcount); +} - kref_get(&irq->refcount); +static inline void vgic_get_irq_kref(struct vgic_irq *irq) +{ + WARN_ON_ONCE(!vgic_try_get_irq_kref(irq)); } void vgic_v3_fold_lr_state(struct kvm_vcpu *vcpu); From patchwork Wed Jan 24 20:49:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 13529659 Received: from out-188.mta1.migadu.com (out-188.mta1.migadu.com [95.215.58.188]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D06AD135402 for ; Wed, 24 Jan 2024 20:49:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.188 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129384; cv=none; b=BmsozUJ88/wIkMWPvABNQUQnLwgIAN05ddb8CzOSHAkBpJm6QEKnSkRQ5tZRZI/tvlD2J9VwNlL2c/YcFRHY0ztG0yr6dbeSFsc80SgmNseNSpxFhcL9bJPvcIAn4V7jpYG7UA2daZ3YQcshZJe8pMpiWLWfElm//dmVGC8z2gk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129384; c=relaxed/simple; bh=AaUwQqfc4Ku5CZrkEkRiMftjazrtuY7bOsUzVzb64Mc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IwB5FQu7Xsz4XH7NCqs2L49Z0vYMLSAmlNkSnf7oMTwHPPX2EV2IRGTzyTLAAb68F/b8tHSnP4mh7go30tsqMP5ppfcpaVhL34o+GzcEGP3YTA0sklI0uvUURXuXrDDLuMnc1T2VURlR8nJ6KN2OsVEUyR4e6oV+iPUggR3i+1Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=cMlnq2Zn; arc=none smtp.client-ip=95.215.58.188 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="cMlnq2Zn" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706129378; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LxxqEpTwO+zOxGEx/S+lcOqJ8/x3pVGgz89u4x3XNnQ=; b=cMlnq2ZnRx9gK/QCKmQw4yHzghedLSRfeCu9QAVj+8ifMd90nhmuiaAnAb2IP73EGYHFqK m3voIsdj+ElcUbBYHsyPeyOB/apHaOdUC4cJHM8Wt0FfXdiHBJCQ9zS/N6r2xB2Ih6AXEz V32y2tpxGfvLw1/V0796ACB+1DxdDR4= From: Oliver Upton To: kvmarm@lists.linux.dev Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Suzuki K Poulose , Zenghui Yu , Raghavendra Rao Ananta , Jing Zhang , Oliver Upton Subject: [PATCH 09/15] KVM: arm64: vgic: Ensure the irq refcount is nonzero when taking a ref Date: Wed, 24 Jan 2024 20:49:03 +0000 Message-ID: <20240124204909.105952-10-oliver.upton@linux.dev> In-Reply-To: <20240124204909.105952-1-oliver.upton@linux.dev> References: <20240124204909.105952-1-oliver.upton@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT It will soon be possible for get() and put() calls to happen in parallel, which means in most cases we must ensure the refcount is nonzero when taking a new reference. Switch to using vgic_try_get_irq_kref() where necessary, and document the few conditions where an IRQ's refcount is guaranteed to be nonzero. Signed-off-by: Oliver Upton --- arch/arm64/kvm/vgic/vgic-its.c | 18 ++++++++---------- arch/arm64/kvm/vgic/vgic.c | 3 ++- 2 files changed, 10 insertions(+), 11 deletions(-) diff --git a/arch/arm64/kvm/vgic/vgic-its.c b/arch/arm64/kvm/vgic/vgic-its.c index 1d912a595b71..7219f4a0a93d 100644 --- a/arch/arm64/kvm/vgic/vgic-its.c +++ b/arch/arm64/kvm/vgic/vgic-its.c @@ -75,18 +75,11 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid, * check that we don't add a second list entry with the same LPI. */ oldirq = xa_load(&dist->lpi_xa, intid); - if (oldirq) { + if (oldirq && vgic_try_get_irq_kref(oldirq)) { /* Someone was faster with adding this LPI, lets use that. */ kfree(irq); irq = oldirq; - /* - * This increases the refcount, the caller is expected to - * call vgic_put_irq() on the returned pointer once it's - * finished with the IRQ. - */ - vgic_get_irq_kref(irq); - goto out_unlock; } @@ -607,8 +600,8 @@ static struct vgic_irq *vgic_its_check_cache(struct kvm *kvm, phys_addr_t db, raw_spin_lock_irqsave(&dist->lpi_list_lock, flags); irq = __vgic_its_check_cache(dist, db, devid, eventid); - if (irq) - vgic_get_irq_kref(irq); + if (irq && !vgic_try_get_irq_kref(irq)) + irq = NULL; raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags); @@ -654,6 +647,11 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, if (cte->irq) __vgic_put_lpi_locked(kvm, cte->irq); + /* + * The irq refcount is guaranteed to be nonzero while holding the + * its_lock, as the ITE (and the reference it holds) cannot be freed. + */ + lockdep_assert_held(&its->its_lock); vgic_get_irq_kref(irq); cte->db = db; diff --git a/arch/arm64/kvm/vgic/vgic.c b/arch/arm64/kvm/vgic/vgic.c index 949af87bb599..7e0d84906a12 100644 --- a/arch/arm64/kvm/vgic/vgic.c +++ b/arch/arm64/kvm/vgic/vgic.c @@ -394,7 +394,8 @@ bool vgic_queue_irq_unlock(struct kvm *kvm, struct vgic_irq *irq, /* * Grab a reference to the irq to reflect the fact that it is - * now in the ap_list. + * now in the ap_list. This is safe as the caller must already hold a + * reference on the irq. */ vgic_get_irq_kref(irq); list_add_tail(&irq->ap_list, &vcpu->arch.vgic_cpu.ap_list_head); From patchwork Wed Jan 24 20:49:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 13529660 Received: from out-178.mta1.migadu.com (out-178.mta1.migadu.com [95.215.58.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AAC3A1350DA for ; Wed, 24 Jan 2024 20:49:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129385; cv=none; b=CcnJMFss1Pm/ZsggoLpaFfL9NB9uViRDvwAi+bULepyHWcnfQGxTWaf+kC4fUfhtRt/C0h3A1rEzC6Kp9i52JgDR/Hkyaa5eavKzgrNx/6JplNC78j5ga7yYfbJPP1VwuHTWNwjsEjU6MU63dbB2QP74ztzmj3z+bjndOhy98n8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129385; c=relaxed/simple; bh=zn+NEwCWV2J3fnlNAv5UMIsfb2daqDGOeyb3cOTr5Hs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CJG1nAxtOxMOfrkLH1/GQNlgHgJP8B64KZ51D0GIJjK/UsFU9jhpah7aBC+O3eWCJAQIs/NF0Epwg9IvCcajFyvE8PkwILn3FXRHGmEiwiqShwlDbcuZATbkTW0pM71LUu+BSdjL0jzpIP3U8z9Qy88An2bmuI8cLwiE5hwFhjw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=oOIhrA5D; arc=none smtp.client-ip=95.215.58.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="oOIhrA5D" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706129380; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JSRHFG0+mRsPGBx4WvzyykyzgB4/zirusbi7pLa/qNU=; b=oOIhrA5D3lJg4GSUFQDL3QLVro9vo0pypba9bifO/3h2NcrwEERhgaf/6BMGpVfhsyoAK1 LTDWMeu5jXahn7CDfs2JTb8E1TK3eoWSN+PggYvbylA1EAMAXbErepMD7O34yeg4Nwezip TViXME2XGctJrav32UkOpVt4lfq8wbk= From: Oliver Upton To: kvmarm@lists.linux.dev Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Suzuki K Poulose , Zenghui Yu , Raghavendra Rao Ananta , Jing Zhang , Oliver Upton Subject: [PATCH 10/15] KVM: arm64: vgic: Don't acquire the lpi_list_lock in vgic_put_irq() Date: Wed, 24 Jan 2024 20:49:04 +0000 Message-ID: <20240124204909.105952-11-oliver.upton@linux.dev> In-Reply-To: <20240124204909.105952-1-oliver.upton@linux.dev> References: <20240124204909.105952-1-oliver.upton@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT The LPI xarray's xa_lock is sufficient for synchronizing writers when freeing a given LPI. Furthermore, readers can only take a new reference on an IRQ if it was already nonzero. Stop taking the lpi_list_lock unnecessarily and get rid of __vgic_put_lpi_locked(). Signed-off-by: Oliver Upton --- arch/arm64/kvm/vgic/vgic-its.c | 4 ++-- arch/arm64/kvm/vgic/vgic.c | 21 ++++----------------- arch/arm64/kvm/vgic/vgic.h | 1 - 3 files changed, 6 insertions(+), 20 deletions(-) diff --git a/arch/arm64/kvm/vgic/vgic-its.c b/arch/arm64/kvm/vgic/vgic-its.c index 7219f4a0a93d..8c026a530018 100644 --- a/arch/arm64/kvm/vgic/vgic-its.c +++ b/arch/arm64/kvm/vgic/vgic-its.c @@ -645,7 +645,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, * was in the cache, and increment it on the new interrupt. */ if (cte->irq) - __vgic_put_lpi_locked(kvm, cte->irq); + vgic_put_irq(kvm, cte->irq); /* * The irq refcount is guaranteed to be nonzero while holding the @@ -682,7 +682,7 @@ void vgic_its_invalidate_cache(struct kvm *kvm) if (!cte->irq) break; - __vgic_put_lpi_locked(kvm, cte->irq); + vgic_put_irq(kvm, cte->irq); cte->irq = NULL; } diff --git a/arch/arm64/kvm/vgic/vgic.c b/arch/arm64/kvm/vgic/vgic.c index 7e0d84906a12..6b8f440ca0fd 100644 --- a/arch/arm64/kvm/vgic/vgic.c +++ b/arch/arm64/kvm/vgic/vgic.c @@ -110,13 +110,13 @@ static void vgic_irq_release(struct kref *ref) { } -/* - * Drop the refcount on the LPI. Must be called with lpi_list_lock held. - */ -void __vgic_put_lpi_locked(struct kvm *kvm, struct vgic_irq *irq) +void vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq) { struct vgic_dist *dist = &kvm->arch.vgic; + if (irq->intid < VGIC_MIN_LPI) + return; + if (!kref_put(&irq->refcount, vgic_irq_release)) return; @@ -126,19 +126,6 @@ void __vgic_put_lpi_locked(struct kvm *kvm, struct vgic_irq *irq) kfree_rcu(irq, rcu); } -void vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq) -{ - struct vgic_dist *dist = &kvm->arch.vgic; - unsigned long flags; - - if (irq->intid < VGIC_MIN_LPI) - return; - - raw_spin_lock_irqsave(&dist->lpi_list_lock, flags); - __vgic_put_lpi_locked(kvm, irq); - raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags); -} - void vgic_flush_pending_lpis(struct kvm_vcpu *vcpu) { struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; diff --git a/arch/arm64/kvm/vgic/vgic.h b/arch/arm64/kvm/vgic/vgic.h index 20886f57416a..b0a2754eae93 100644 --- a/arch/arm64/kvm/vgic/vgic.h +++ b/arch/arm64/kvm/vgic/vgic.h @@ -180,7 +180,6 @@ vgic_get_mmio_region(struct kvm_vcpu *vcpu, struct vgic_io_device *iodev, gpa_t addr, int len); struct vgic_irq *vgic_get_irq(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 intid); -void __vgic_put_lpi_locked(struct kvm *kvm, struct vgic_irq *irq); void vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq); bool vgic_get_phys_line_level(struct vgic_irq *irq); void vgic_irq_set_phys_pending(struct vgic_irq *irq, bool pending); From patchwork Wed Jan 24 20:49:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 13529661 Received: from out-186.mta1.migadu.com (out-186.mta1.migadu.com [95.215.58.186]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 84EBC1350EC for ; Wed, 24 Jan 2024 20:49:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.186 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129386; cv=none; b=fwxyD8o74Dcy0zjn/kO5/NmuTVSSuriIDxHduCBiVTfcgbSZe7AIUXJfoHKbBchv82zhtinF4i1Bw6K3kW//ymjl1SjZ9eeivoCC5gZgKl6WtwVxKyYDcAwtjWhO3JPyzsdE8WxgvjyHMoGdGhhuSRqNwiYWrK1Rx1+3TfZGe0c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129386; c=relaxed/simple; bh=oTyMnOeo4+hRi1tWu/Z+DFCpY7MaGTv5BzKw4vMNwPE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ekN+HlbPCTm/WeZyURYXGHJ95Wqu3tovDk2fxNpDmMm8RvvhIF/zeASdUI/5w5/a+prY0iClNSk2FeBktweT5Omoy/goSzx/NfmQNrmRHP65B/Bnz9UHu80841akaEYcpXBQBAIdrnkqYATK6tbfHjas272eMfUN4a6Hp00Fa0g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=JLpaQYjc; arc=none smtp.client-ip=95.215.58.186 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="JLpaQYjc" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706129382; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KW0iJG5RQGiIM6NdrK4b2D5XgcaBzPy7qVtrdzoHvOs=; b=JLpaQYjcQQrygzAoulCFPLVg7Z6i4rruEf8kIMEhvn7I6gut4CPwudnCZlAaA1sNMfil3d pJQEbxnlMVk5S/7mY5hgg6tzeTCnRpcSvAvWa6k+rIb/luD2YlENn/gpEXrTl/5CNP8MG2 yH9TQQTmI6zYi4KRPjJbe/in0/8O+xY= From: Oliver Upton To: kvmarm@lists.linux.dev Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Suzuki K Poulose , Zenghui Yu , Raghavendra Rao Ananta , Jing Zhang , Oliver Upton Subject: [PATCH 11/15] KVM: arm64: vgic-its: Lazily allocate LPI translation cache Date: Wed, 24 Jan 2024 20:49:05 +0000 Message-ID: <20240124204909.105952-12-oliver.upton@linux.dev> In-Reply-To: <20240124204909.105952-1-oliver.upton@linux.dev> References: <20240124204909.105952-1-oliver.upton@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Reusing translation cache entries within a read-side critical section is fundamentally incompatible with an rculist. As such, we need to allocate a new entry to replace an eviction and free the removed entry afterwards. Take this as an opportunity to remove the eager allocation of translation cache entries altogether in favor of a lazy allocation model on cache miss. Signed-off-by: Oliver Upton --- arch/arm64/kvm/vgic/vgic-init.c | 3 -- arch/arm64/kvm/vgic/vgic-its.c | 86 ++++++++++++++------------------- include/kvm/arm_vgic.h | 1 + 3 files changed, 38 insertions(+), 52 deletions(-) diff --git a/arch/arm64/kvm/vgic/vgic-init.c b/arch/arm64/kvm/vgic/vgic-init.c index e25672d6e846..660d5ce3b610 100644 --- a/arch/arm64/kvm/vgic/vgic-init.c +++ b/arch/arm64/kvm/vgic/vgic-init.c @@ -305,9 +305,6 @@ int vgic_init(struct kvm *kvm) } } - if (vgic_has_its(kvm)) - vgic_lpi_translation_cache_init(kvm); - /* * If we have GICv4.1 enabled, unconditionnaly request enable the * v4 support so that we get HW-accelerated vSGIs. Otherwise, only diff --git a/arch/arm64/kvm/vgic/vgic-its.c b/arch/arm64/kvm/vgic/vgic-its.c index 8c026a530018..aec82d9a1b3c 100644 --- a/arch/arm64/kvm/vgic/vgic-its.c +++ b/arch/arm64/kvm/vgic/vgic-its.c @@ -608,12 +608,20 @@ static struct vgic_irq *vgic_its_check_cache(struct kvm *kvm, phys_addr_t db, return irq; } +/* Default is 16 cached LPIs per vcpu */ +#define LPI_DEFAULT_PCPU_CACHE_SIZE 16 + +static unsigned int vgic_its_max_cache_size(struct kvm *kvm) +{ + return atomic_read(&kvm->online_vcpus) * LPI_DEFAULT_PCPU_CACHE_SIZE; +} + static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, u32 devid, u32 eventid, struct vgic_irq *irq) { + struct vgic_translation_cache_entry *new, *victim; struct vgic_dist *dist = &kvm->arch.vgic; - struct vgic_translation_cache_entry *cte; unsigned long flags; phys_addr_t db; @@ -621,10 +629,11 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, if (irq->hw) return; - raw_spin_lock_irqsave(&dist->lpi_list_lock, flags); + new = victim = kzalloc(sizeof(*new), GFP_KERNEL_ACCOUNT); + if (!new) + return; - if (unlikely(list_empty(&dist->lpi_translation_cache))) - goto out; + raw_spin_lock_irqsave(&dist->lpi_list_lock, flags); /* * We could have raced with another CPU caching the same @@ -635,17 +644,15 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, if (__vgic_its_check_cache(dist, db, devid, eventid)) goto out; - /* Always reuse the last entry (LRU policy) */ - cte = list_last_entry(&dist->lpi_translation_cache, - typeof(*cte), entry); - - /* - * Caching the translation implies having an extra reference - * to the interrupt, so drop the potential reference on what - * was in the cache, and increment it on the new interrupt. - */ - if (cte->irq) - vgic_put_irq(kvm, cte->irq); + if (dist->lpi_cache_count >= vgic_its_max_cache_size(kvm)) { + /* Always reuse the last entry (LRU policy) */ + victim = list_last_entry(&dist->lpi_translation_cache, + typeof(*cte), entry); + list_del(&victim->entry); + dist->lpi_cache_count--; + } else { + victim = NULL; + } /* * The irq refcount is guaranteed to be nonzero while holding the @@ -654,16 +661,26 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, lockdep_assert_held(&its->its_lock); vgic_get_irq_kref(irq); - cte->db = db; - cte->devid = devid; - cte->eventid = eventid; - cte->irq = irq; + new->db = db; + new->devid = devid; + new->eventid = eventid; + new->irq = irq; /* Move the new translation to the head of the list */ - list_move(&cte->entry, &dist->lpi_translation_cache); + list_add(&new->entry, &dist->lpi_translation_cache); out: raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags); + + /* + * Caching the translation implies having an extra reference + * to the interrupt, so drop the potential reference on what + * was in the cache, and increment it on the new interrupt. + */ + if (victim && victim->irq) + vgic_put_irq(kvm, victim->irq); + + kfree(victim); } void vgic_its_invalidate_cache(struct kvm *kvm) @@ -1905,33 +1922,6 @@ static int vgic_register_its_iodev(struct kvm *kvm, struct vgic_its *its, return ret; } -/* Default is 16 cached LPIs per vcpu */ -#define LPI_DEFAULT_PCPU_CACHE_SIZE 16 - -void vgic_lpi_translation_cache_init(struct kvm *kvm) -{ - struct vgic_dist *dist = &kvm->arch.vgic; - unsigned int sz; - int i; - - if (!list_empty(&dist->lpi_translation_cache)) - return; - - sz = atomic_read(&kvm->online_vcpus) * LPI_DEFAULT_PCPU_CACHE_SIZE; - - for (i = 0; i < sz; i++) { - struct vgic_translation_cache_entry *cte; - - /* An allocation failure is not fatal */ - cte = kzalloc(sizeof(*cte), GFP_KERNEL_ACCOUNT); - if (WARN_ON(!cte)) - break; - - INIT_LIST_HEAD(&cte->entry); - list_add(&cte->entry, &dist->lpi_translation_cache); - } -} - void vgic_lpi_translation_cache_destroy(struct kvm *kvm) { struct vgic_dist *dist = &kvm->arch.vgic; @@ -1978,8 +1968,6 @@ static int vgic_its_create(struct kvm_device *dev, u32 type) kfree(its); return ret; } - - vgic_lpi_translation_cache_init(dev->kvm); } mutex_init(&its->its_lock); diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index a6f6c1583662..70490a2a300d 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -282,6 +282,7 @@ struct vgic_dist { /* LPI translation cache */ struct list_head lpi_translation_cache; + unsigned int lpi_cache_count; /* used by vgic-debug */ struct vgic_state_iter *iter; From patchwork Wed Jan 24 20:49:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 13529662 Received: from out-187.mta1.migadu.com (out-187.mta1.migadu.com [95.215.58.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7EBA3135A57 for ; Wed, 24 Jan 2024 20:49:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.187 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129388; cv=none; b=Aazz4IT5Odve7LfDwyvDKBkwq6g9BUXTqXvSj5/H+z17ntO2WZtxj+shp0rOZ2tQZbtgb1KD3BgvW7eaRPRVR1HzbkYzrOKp8RuHm2nD/BYx3WYysoeq7q6JuNt/xq7gCX18T9dxZz/O/b+2nrLQNhZKeEU8OlCGcZXgBgpRieE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129388; c=relaxed/simple; bh=V+Aqf0jjqyGa1E2qOjT62lbfoHRkavh2wOZwXZb04x8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Swuve9FUTFrIMpsMPNlba0IHDbDJiV9Qy4jAV4aYAAtp1NmOlue/xzSPeuk8b7XMeQmGU6YajH3cAeVeUj8OkEZPVuNTZEofcmIXzZzVsCbpGew9gqS3Rl0GzThbenhq84Ab2mVg4YtOJsb+MyH8eN6VB0cJ3xpeDN5v/vmGvZU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=iYYzZbwW; arc=none smtp.client-ip=95.215.58.187 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="iYYzZbwW" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706129384; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6+2oom+tyocopN2F9E6bClPUUgz5VYwJmjrToVciJR0=; b=iYYzZbwWmhBMA5YW9/oAmiXQdY3D2dtGEYhfBuIagJoHWJjhRZEeG3LzXuqrzfASC4+Dr4 +4n8tOkAAO5EzNw/BYxPJtmYEbyc2X/vHnDn4Dpw2Nr7Xuahg+1EvGqcoZhrm2Krk1cNhd 0MqKB8njnktRx9R6wjkLqvCvFUNfFiM= From: Oliver Upton To: kvmarm@lists.linux.dev Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Suzuki K Poulose , Zenghui Yu , Raghavendra Rao Ananta , Jing Zhang , Oliver Upton Subject: [PATCH 12/15] KVM: arm64: vgic-its: Pick cache victim based on usage count Date: Wed, 24 Jan 2024 20:49:06 +0000 Message-ID: <20240124204909.105952-13-oliver.upton@linux.dev> In-Reply-To: <20240124204909.105952-1-oliver.upton@linux.dev> References: <20240124204909.105952-1-oliver.upton@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT To date the translation cache LRU policy relies on the ordering of the linked-list to pick the victim, as entries are moved to the head of the list on every cache hit. These sort of transformations are incompatible with an rculist, necessitating a different strategy for recording usage in-place. Count the number of cache hits since the last translation cache miss for every entry. The preferences for selecting a victim are as follows: - Invalid entries over valid entries - Valid entry with the lowest usage count - In the case of a tie, pick the entry closest to the tail (oldest) Signed-off-by: Oliver Upton --- arch/arm64/kvm/vgic/vgic-its.c | 42 ++++++++++++++++++++++++++-------- 1 file changed, 32 insertions(+), 10 deletions(-) diff --git a/arch/arm64/kvm/vgic/vgic-its.c b/arch/arm64/kvm/vgic/vgic-its.c index aec82d9a1b3c..ed0c6c333a6c 100644 --- a/arch/arm64/kvm/vgic/vgic-its.c +++ b/arch/arm64/kvm/vgic/vgic-its.c @@ -154,6 +154,7 @@ struct vgic_translation_cache_entry { u32 devid; u32 eventid; struct vgic_irq *irq; + atomic64_t usage_count; }; /** @@ -577,13 +578,7 @@ static struct vgic_irq *__vgic_its_check_cache(struct vgic_dist *dist, cte->eventid != eventid) continue; - /* - * Move this entry to the head, as it is the most - * recently used. - */ - if (!list_is_first(&cte->entry, &dist->lpi_translation_cache)) - list_move(&cte->entry, &dist->lpi_translation_cache); - + atomic64_inc(&cte->usage_count); return cte->irq; } @@ -616,6 +611,30 @@ static unsigned int vgic_its_max_cache_size(struct kvm *kvm) return atomic_read(&kvm->online_vcpus) * LPI_DEFAULT_PCPU_CACHE_SIZE; } +static struct vgic_translation_cache_entry *vgic_its_cache_victim(struct vgic_dist *dist) +{ + struct vgic_translation_cache_entry *cte, *victim = NULL; + u64 min, tmp; + + /* + * Find the least used cache entry since the last cache miss, preferring + * older entries in the case of a tie. Note that usage accounting is + * deliberately non-atomic, so this is all best-effort. + */ + list_for_each_entry(cte, &dist->lpi_translation_cache, entry) { + if (!cte->irq) + return cte; + + tmp = atomic64_xchg_relaxed(&cte->usage_count, 0); + if (!victim || tmp <= min) { + victim = cte; + min = tmp; + } + } + + return victim; +} + static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, u32 devid, u32 eventid, struct vgic_irq *irq) @@ -645,9 +664,12 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, goto out; if (dist->lpi_cache_count >= vgic_its_max_cache_size(kvm)) { - /* Always reuse the last entry (LRU policy) */ - victim = list_last_entry(&dist->lpi_translation_cache, - typeof(*cte), entry); + victim = vgic_its_cache_victim(dist); + if (WARN_ON_ONCE(!victim)) { + victim = new; + goto out; + } + list_del(&victim->entry); dist->lpi_cache_count--; } else { From patchwork Wed Jan 24 20:49:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 13529663 Received: from out-177.mta1.migadu.com (out-177.mta1.migadu.com [95.215.58.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3188A135A60 for ; Wed, 24 Jan 2024 20:49:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129390; cv=none; b=QNjRc5bPy5SlWOC8ERbXGvJIU8ZnIXdKsVnHKcrQXVKDp9q+qIckshhLRMObQYZl8MGLK4wPQg0J4kkooDdzWXV4GngBebAZkke6bbxDgmwLP03GXi/XOApooGGUUw68QAV8wz5OiJcyNNS8mwos9AGaRI+4HfL74fjFxkzaZsY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129390; c=relaxed/simple; bh=LORJbe3ID83LnrxDienlFK/6XzGGKHhuqfuKs+CQORc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZadVWgZIwaxtsX1MmxlH2d7EYqOx9VWk3jSdtI7QM44U/uGNwsQ35I15u9krUFYQjcpKvFioD4Kgl9v2buOfwgXOdip/sZLbmKzXOnyc1g/1Kl2LaLAnFnPrxVV0u/MyVyh0dxGMwfvlGiZhHIUNlR876+919SPmDtMYE+ASdlE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=AmSufSwg; arc=none smtp.client-ip=95.215.58.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="AmSufSwg" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706129386; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=63nbbU4EZtlCwl4weak8AD8fCWTldEW9ao3X+RGpuoY=; b=AmSufSwgUm2HFHsMVWwFSzd10TloPd+vqBg8wXFah6BUnjCwaFO3VMVvyxiUJe/+BMQm09 3v03q0ZJ1Ui82Sem/qAa3Yz230lFyZURagEQKTm1xaLTgn1gz6JKT2UXKfSSJL+Jjh/Q8F 24kqDd58srdJr3/v+akoXwF6QYvhOKE= From: Oliver Upton To: kvmarm@lists.linux.dev Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Suzuki K Poulose , Zenghui Yu , Raghavendra Rao Ananta , Jing Zhang , Oliver Upton Subject: [PATCH 13/15] KVM: arm64: vgic-its: Protect cached vgic_irq pointers with RCU Date: Wed, 24 Jan 2024 20:49:07 +0000 Message-ID: <20240124204909.105952-14-oliver.upton@linux.dev> In-Reply-To: <20240124204909.105952-1-oliver.upton@linux.dev> References: <20240124204909.105952-1-oliver.upton@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT RCU readers of the LPI translation cache will be able to run in parallel with a cache invalidation, which clears the RCU pointer. Start using RCU protection on the cached irq pointer in light of this. Signed-off-by: Oliver Upton --- arch/arm64/kvm/vgic/vgic-its.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kvm/vgic/vgic-its.c b/arch/arm64/kvm/vgic/vgic-its.c index ed0c6c333a6c..79b35fdaa1cd 100644 --- a/arch/arm64/kvm/vgic/vgic-its.c +++ b/arch/arm64/kvm/vgic/vgic-its.c @@ -153,7 +153,7 @@ struct vgic_translation_cache_entry { phys_addr_t db; u32 devid; u32 eventid; - struct vgic_irq *irq; + struct vgic_irq __rcu *irq; atomic64_t usage_count; }; @@ -571,7 +571,7 @@ static struct vgic_irq *__vgic_its_check_cache(struct vgic_dist *dist, * If we hit a NULL entry, there is nothing after this * point. */ - if (!cte->irq) + if (!rcu_access_pointer(cte->irq)) break; if (cte->db != db || cte->devid != devid || @@ -579,7 +579,7 @@ static struct vgic_irq *__vgic_its_check_cache(struct vgic_dist *dist, continue; atomic64_inc(&cte->usage_count); - return cte->irq; + return rcu_dereference(cte->irq); } return NULL; @@ -622,7 +622,7 @@ static struct vgic_translation_cache_entry *vgic_its_cache_victim(struct vgic_di * deliberately non-atomic, so this is all best-effort. */ list_for_each_entry(cte, &dist->lpi_translation_cache, entry) { - if (!cte->irq) + if (!rcu_access_pointer(cte->irq)) return cte; tmp = atomic64_xchg_relaxed(&cte->usage_count, 0); @@ -653,6 +653,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, return; raw_spin_lock_irqsave(&dist->lpi_list_lock, flags); + rcu_read_lock(); /* * We could have raced with another CPU caching the same @@ -686,12 +687,13 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, new->db = db; new->devid = devid; new->eventid = eventid; - new->irq = irq; + rcu_assign_pointer(new->irq, irq); /* Move the new translation to the head of the list */ list_add(&new->entry, &dist->lpi_translation_cache); out: + rcu_read_unlock(); raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags); /* @@ -712,19 +714,21 @@ void vgic_its_invalidate_cache(struct kvm *kvm) unsigned long flags; raw_spin_lock_irqsave(&dist->lpi_list_lock, flags); + rcu_read_lock(); list_for_each_entry(cte, &dist->lpi_translation_cache, entry) { /* * If we hit a NULL entry, there is nothing after this * point. */ - if (!cte->irq) + if (!rcu_access_pointer(cte->irq)) break; vgic_put_irq(kvm, cte->irq); - cte->irq = NULL; + rcu_assign_pointer(cte->irq, NULL); } + rcu_read_unlock(); raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags); } From patchwork Wed Jan 24 20:49:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 13529664 Received: from out-170.mta1.migadu.com (out-170.mta1.migadu.com [95.215.58.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 51C0C135A57 for ; Wed, 24 Jan 2024 20:49:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129391; cv=none; b=DPhKtC3WYF1+2SvDdBvb1yBhkFE0ZOv/sMJYq9gWL048zx/U1y5V9CbALuFYZbWGZPu8UFU7+VUU0XKBnn4cbkGECRwAjfNb80JTVsNfp7pYwKDTN/QUAyyahlt5Ma6xbJrwlY81MKa/9HyEEedpe4QuOTAo1qgz0Vy8Po2Nte0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129391; c=relaxed/simple; bh=zLh5rirk9xIasi9yvIQwcfwIRUE75kHp5mBYXmy5RQI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kZpZSsMvhzpNkIt0XGUBcjej43vYHGQcVvE4sHcfSwNOHqripCv146C8XRJYfcvUkEtx/SFtf0B+/FLkQDEaNBqk6lOjcZp5bEJYCBXXdjzvFzy7hOgXv9r+T0cUzSAUSn2wkde9Sgf9hfis0bVXbKdkzHe6bECX8ctRcbt7+Hs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=UCFzyySB; arc=none smtp.client-ip=95.215.58.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="UCFzyySB" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706129388; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5NDypmTFyIFRVIwUeZ2Km9VEABncd2W2wdVuF/Yz7UI=; b=UCFzyySBUeZTb9dKijn5GnBl22cCNrJ0pFPnbyHC/LUe3+t7CrTeN62wS+ZFf7knalSfxU TMf1EdjRnt0pJeIxaMY4NpgDT1dtU00/jPmLsP5Be8a5HvpsakbFVGlZtiWT+YYEQnNn9r 83NyZLxRTkCj9nzJVy6rgRX/2lY4qc0= From: Oliver Upton To: kvmarm@lists.linux.dev Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Suzuki K Poulose , Zenghui Yu , Raghavendra Rao Ananta , Jing Zhang , Oliver Upton Subject: [PATCH 14/15] KVM: arm64: vgic-its: Treat the LPI translation cache as an rculist Date: Wed, 24 Jan 2024 20:49:08 +0000 Message-ID: <20240124204909.105952-15-oliver.upton@linux.dev> In-Reply-To: <20240124204909.105952-1-oliver.upton@linux.dev> References: <20240124204909.105952-1-oliver.upton@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Convert the LPI translation cache to an rculist such that readers can walk it while only holding the RCU read lock. Signed-off-by: Oliver Upton --- arch/arm64/kvm/vgic/vgic-its.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kvm/vgic/vgic-its.c b/arch/arm64/kvm/vgic/vgic-its.c index 79b35fdaa1cd..1670b452c682 100644 --- a/arch/arm64/kvm/vgic/vgic-its.c +++ b/arch/arm64/kvm/vgic/vgic-its.c @@ -566,7 +566,7 @@ static struct vgic_irq *__vgic_its_check_cache(struct vgic_dist *dist, { struct vgic_translation_cache_entry *cte; - list_for_each_entry(cte, &dist->lpi_translation_cache, entry) { + list_for_each_entry_rcu(cte, &dist->lpi_translation_cache, entry) { /* * If we hit a NULL entry, there is nothing after this * point. @@ -671,7 +671,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, goto out; } - list_del(&victim->entry); + list_del_rcu(&victim->entry); dist->lpi_cache_count--; } else { victim = NULL; @@ -690,7 +690,8 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, rcu_assign_pointer(new->irq, irq); /* Move the new translation to the head of the list */ - list_add(&new->entry, &dist->lpi_translation_cache); + list_add_rcu(&new->entry, &dist->lpi_translation_cache); + dist->lpi_cache_count++; out: rcu_read_unlock(); @@ -704,6 +705,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, if (victim && victim->irq) vgic_put_irq(kvm, victim->irq); + synchronize_rcu(); kfree(victim); } From patchwork Wed Jan 24 20:49:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 13529665 Received: from out-179.mta1.migadu.com (out-179.mta1.migadu.com [95.215.58.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2A759135A79 for ; Wed, 24 Jan 2024 20:49:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129393; cv=none; b=nBmAi77msfvax3NRHvIseBPcNWBmwU86Lx96e2nFlbxGIpfxSBym6JBEglRCz7Hu0D/6iHqi580CwpP2f1eOtPH2ubyGGOMjX+xuP0/5l2RpnPEXDt4rubs5+UiOkmLTvnNjrLGGylTYOX6QIpzjhxIIEhQiuwQLlzJ9c48HjNI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706129393; c=relaxed/simple; bh=3DFCwiB6MeI2Ico7jMywLcbYPxCd+D2TwpKxJ/Zr86k=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VgM3mvkGZyW5mARA6tc2fSABbdQsfdGpNGG3zoXnRoAM9QT4kskqywGEqgWg0aH1XZWpKjBM8mQLUBE5DO83n+/YhYGfDpLucDZHpPyo0bsuwbQI7NGfzYO3/qJHGEqz+qIAFPJp6wH+pIZrI+mo8BvB6/sNDoxAZ71uoTnOzVM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=KWeOQxnF; arc=none smtp.client-ip=95.215.58.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="KWeOQxnF" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706129390; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kczms5RnsRrZgA05jJgqcXdZsBvDLLGbHzeZIIdOh7w=; b=KWeOQxnF7FEi1R519+74lreCzlafBL4c2mQrnfaHQHtiUYzbUSeZW7KwBQiiUhv4ZQAL+G dd4BTDM1zzV/s1EXxPZig5yLkDQ4ZCY0/yXRmwe1ZpzEhHPa0V+cMD7A7xVqr9TBFSnShL Cg+VrxgFrUxSEKdZyc2+sMQ2MkM3SK0= From: Oliver Upton To: kvmarm@lists.linux.dev Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Suzuki K Poulose , Zenghui Yu , Raghavendra Rao Ananta , Jing Zhang , Oliver Upton Subject: [PATCH 15/15] KVM: arm64: vgic-its: Rely on RCU to protect translation cache reads Date: Wed, 24 Jan 2024 20:49:09 +0000 Message-ID: <20240124204909.105952-16-oliver.upton@linux.dev> In-Reply-To: <20240124204909.105952-1-oliver.upton@linux.dev> References: <20240124204909.105952-1-oliver.upton@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Stop taking the lpi_list_lock when reading the LPI translation cache. Signed-off-by: Oliver Upton --- arch/arm64/kvm/vgic/vgic-its.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kvm/vgic/vgic-its.c b/arch/arm64/kvm/vgic/vgic-its.c index 1670b452c682..46ff21fea785 100644 --- a/arch/arm64/kvm/vgic/vgic-its.c +++ b/arch/arm64/kvm/vgic/vgic-its.c @@ -590,15 +590,14 @@ static struct vgic_irq *vgic_its_check_cache(struct kvm *kvm, phys_addr_t db, { struct vgic_dist *dist = &kvm->arch.vgic; struct vgic_irq *irq; - unsigned long flags; - raw_spin_lock_irqsave(&dist->lpi_list_lock, flags); + rcu_read_lock(); irq = __vgic_its_check_cache(dist, db, devid, eventid); if (irq && !vgic_try_get_irq_kref(irq)) irq = NULL; - raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags); + rcu_read_unlock(); return irq; }