From patchwork Thu Jun 21 14:57:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Martin X-Patchwork-Id: 10480043 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id E407760532 for ; Thu, 21 Jun 2018 15:23:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D0CE5284D1 for ; Thu, 21 Jun 2018 15:23:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C53AF28688; Thu, 21 Jun 2018 15:23:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 18569284D1 for ; Thu, 21 Jun 2018 15:23:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=evg/nueOl5SrNrK7jrWYY6DBwtPF5YcbZhAjAH8cE7E=; b=UmUapoaAYIisDu9YUnOLokRu0L ZL4VE7096Shq1bgMCDfeMnfm3f59XO74tUTe/O5cf6e4kuben6BnJJfs0yJ0/sNC+Xn7gKUe9secc snpPoigYRBkAGUJPeF8PTOj1Avz4FYaMZCTyctF6bTstr1YD264j0oAy5i7LCIFp47gbsgnt9ZxZx msHrxBWb40r7tOoDr1YeTc4/I1RAaQZZ256NIchmz4sHfIEtkpP4lAddBzFWUIErBxY+2YZyLiSsG Y8n0WQKgNCIZFcoO/Yf08Rurw9i3XBjLND/GZuRJEEU55tu3mTmd+EZOnLLIyPzsy+nZcJeKhaABJ ik5L2H2Q==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fW1Qa-0004T8-6U; Thu, 21 Jun 2018 15:23:16 +0000 Received: from merlin.infradead.org ([2001:8b0:10b:1231::1]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1fW1NX-0000uR-2k for linux-arm-kernel@bombadil.infradead.org; Thu, 21 Jun 2018 15:20:07 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=mnKPuA95XGxLkrQTNKVmKkBo251VngdJljcV1WTkaKE=; b=wuIuNJqvTpuw5+6er59g6cVl2 Zsw8XFSkElQ79hyZShYW3tapO2q5+UGCXVf9QPIPSeBklvoTStncUvB4NoLTT0J3xnodXntl10nkl TWLuJUqx+64p71rqYNDaSnkRDtK5Q4NS7FVG2nnO0hHIe8iaiYB6Z1iL4nEZrBu5/T6YTCqFUWyv2 5+suSdeef/sYbcFrTDRElEVE+noqisGQT3n1TTk0LXNhURT6iV6j9Ujcw0a92TQaAo2sdG8Etw3se od78WA0UQHBKu6wCqGzZem868q5Wzs5NUxLiNWHRtqE5XRS3TEEl7QudGN51LYAQWvxHna9jyhQd1 KICoX02Jg==; Received: from foss.arm.com ([217.140.101.70]) by merlin.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fW132-0007kB-87 for linux-arm-kernel@lists.infradead.org; Thu, 21 Jun 2018 14:58:58 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8B8091650; Thu, 21 Jun 2018 07:58:44 -0700 (PDT) Received: from e103592.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E46D43F246; Thu, 21 Jun 2018 07:58:42 -0700 (PDT) From: Dave Martin To: kvmarm@lists.cs.columbia.edu Subject: [RFC PATCH 06/16] arm64/sve: Determine virtualisation-friendly vector lengths Date: Thu, 21 Jun 2018 15:57:30 +0100 Message-Id: <1529593060-542-7-git-send-email-Dave.Martin@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1529593060-542-1-git-send-email-Dave.Martin@arm.com> References: <1529593060-542-1-git-send-email-Dave.Martin@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180621_105856_483031_F283897E X-CRM114-Status: GOOD ( 28.42 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Okamoto Takayuki , Christoffer Dall , Ard Biesheuvel , Marc Zyngier , Catalin Marinas , Will Deacon , =?UTF-8?q?Alex=20Benn=C3=A9e?= , linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Software at EL1 is permitted to assume that when programming ZCR_EL1.LEN, the effective vector length is nonetheless a vector length supported by the CPU. Thus, software may rely on the effective vector length being different from that programmed via ZCR_EL1.LEN in some situations. However, KVM does not tightly bind vcpus to individual underlying physical CPUs. As a result, vcpus can migrate from one CPU to another. This means that in order to preserve the guarantee described in the previous paragraph, the set of supported vector lengths must appear to be the same for the vcpu at all times, irrespective of which physical CPU the vcpu is currently running on. The Arm SVE architecture allows the maximum vector length visible to EL1 to be restricted by programming ZCR_EL2.LEN. This provides a means to hide from guests any vector lengths that are not supported by every physical CPU in the system. However, there is no way to hide a particular vector length while some greater vector length is exposed to EL1. This patch determines the maximum vector length (sve_max_virtualisable_vl) for which the set of supported vector lengths not exceeding it is identical for all CPUs. When KVM is available, the set of vector lengths supported by each late secondary CPU is verified to be consistent with those of the early CPUs, in order to ensure that the value chosen for sve_max_virtualisable_vl remains globally valid, and ensure that all created vcpus continue to behave correctly. sve_secondary_vq_map is used as scratch space for these computations, rendering its name misleading. This patch renames this bitmap to sve_tmp_vq_map in order to make its purpose clearer. Signed-off-by: Dave Martin --- arch/arm64/include/asm/fpsimd.h | 1 + arch/arm64/kernel/cpufeature.c | 2 +- arch/arm64/kernel/fpsimd.c | 86 ++++++++++++++++++++++++++++++++++------- 3 files changed, 75 insertions(+), 14 deletions(-) diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h index fa92747..3ad4607 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -85,6 +85,7 @@ extern void sve_kernel_enable(const struct arm64_cpu_capabilities *__unused); extern u64 read_zcr_features(void); extern int __ro_after_init sve_max_vl; +extern int __ro_after_init sve_max_virtualisable_vl; #ifdef CONFIG_ARM64_SVE diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index d2856b1..f493a2f 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1511,7 +1511,7 @@ static void verify_sve_features(void) unsigned int len = zcr & ZCR_ELx_LEN_MASK; if (len < safe_len || sve_verify_vq_map()) { - pr_crit("CPU%d: SVE: required vector length(s) missing\n", + pr_crit("CPU%d: SVE: vector length support mismatch\n", smp_processor_id()); cpu_die_early(); } diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 6b1ddae..390afb4 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -18,6 +18,7 @@ */ #include +#include #include #include #include @@ -48,6 +49,7 @@ #include #include #include +#include #define FPEXC_IOF (1 << 0) #define FPEXC_DZF (1 << 1) @@ -130,14 +132,18 @@ static int sve_default_vl = -1; /* Maximum supported vector length across all CPUs (initially poisoned) */ int __ro_after_init sve_max_vl = SVE_VL_MIN; +int __ro_after_init sve_max_virtualisable_vl = SVE_VL_MIN; /* Set of available vector lengths, as vq_to_bit(vq): */ static __ro_after_init DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX); +/* Set of vector lengths present on at least one cpu: */ +static __ro_after_init DECLARE_BITMAP(sve_vq_partial_map, SVE_VQ_MAX); static void __percpu *efi_sve_state; #else /* ! CONFIG_ARM64_SVE */ /* Dummy declaration for code that will be optimised out: */ extern __ro_after_init DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX); +extern __ro_after_init DECLARE_BITMAP(sve_vq_partial_map, SVE_VQ_MAX); extern void __percpu *efi_sve_state; #endif /* ! CONFIG_ARM64_SVE */ @@ -642,11 +648,8 @@ int sve_get_current_vl(void) return sve_prctl_status(0); } -/* - * Bitmap for temporary storage of the per-CPU set of supported vector lengths - * during secondary boot. - */ -static DECLARE_BITMAP(sve_secondary_vq_map, SVE_VQ_MAX); +/* Bitmaps for temporary storage during manipulation of vector length sets */ +static DECLARE_BITMAP(sve_tmp_vq_map, SVE_VQ_MAX); static void sve_probe_vqs(DECLARE_BITMAP(map, SVE_VQ_MAX)) { @@ -669,6 +672,7 @@ static void sve_probe_vqs(DECLARE_BITMAP(map, SVE_VQ_MAX)) void __init sve_init_vq_map(void) { sve_probe_vqs(sve_vq_map); + bitmap_copy(sve_vq_partial_map, sve_vq_map, SVE_VQ_MAX); } /* @@ -677,24 +681,60 @@ void __init sve_init_vq_map(void) */ void sve_update_vq_map(void) { - sve_probe_vqs(sve_secondary_vq_map); - bitmap_and(sve_vq_map, sve_vq_map, sve_secondary_vq_map, SVE_VQ_MAX); + sve_probe_vqs(sve_tmp_vq_map); + bitmap_and(sve_vq_map, sve_vq_map, sve_tmp_vq_map, + SVE_VQ_MAX); + bitmap_or(sve_vq_partial_map, sve_vq_partial_map, sve_tmp_vq_map, + SVE_VQ_MAX); } /* Check whether the current CPU supports all VQs in the committed set */ int sve_verify_vq_map(void) { - int ret = 0; + int ret = -EINVAL; + unsigned long b; - sve_probe_vqs(sve_secondary_vq_map); - bitmap_andnot(sve_secondary_vq_map, sve_vq_map, sve_secondary_vq_map, - SVE_VQ_MAX); - if (!bitmap_empty(sve_secondary_vq_map, SVE_VQ_MAX)) { + sve_probe_vqs(sve_tmp_vq_map); + + bitmap_complement(sve_tmp_vq_map, sve_tmp_vq_map, SVE_VQ_MAX); + if (bitmap_intersects(sve_tmp_vq_map, sve_vq_map, SVE_VQ_MAX)) { pr_warn("SVE: cpu%d: Required vector length(s) missing\n", smp_processor_id()); - ret = -EINVAL; + goto error; } + if (!IS_ENABLED(CONFIG_KVM) || !is_hyp_mode_available()) + goto ok; + + /* + * For KVM, it is necessary to ensure that this CPU doesn't + * support any vector length that guests may have probed as + * unsupported. + */ + + /* Recover the set of supported VQs: */ + bitmap_complement(sve_tmp_vq_map, sve_tmp_vq_map, SVE_VQ_MAX); + /* Find VQs supported that are not globally supported: */ + bitmap_andnot(sve_tmp_vq_map, sve_tmp_vq_map, sve_vq_map, SVE_VQ_MAX); + + /* Find the lowest such VQ, if any: */ + b = find_last_bit(sve_tmp_vq_map, SVE_VQ_MAX); + if (b >= SVE_VQ_MAX) + goto ok; /* no mismatches */ + + /* + * Mismatches above sve_max_virtualisable_vl are fine, since + * no guest is allowed to configure ZCR_EL2.LEN to exceed this: + */ + if (sve_vl_from_vq(bit_to_vq(b)) <= sve_max_virtualisable_vl) { + pr_warn("SVE: cpu%d: Unsupported vector length(s) present\n", + smp_processor_id()); + goto error; + } + +ok: + ret = 0; +error: return ret; } @@ -762,6 +802,7 @@ u64 read_zcr_features(void) void __init sve_setup(void) { u64 zcr; + unsigned long b; if (!system_supports_sve()) return; @@ -790,10 +831,29 @@ void __init sve_setup(void) */ sve_default_vl = find_supported_vector_length(64); + bitmap_andnot(sve_tmp_vq_map, sve_vq_partial_map, sve_vq_map, + SVE_VQ_MAX); + + b = find_last_bit(sve_tmp_vq_map, SVE_VQ_MAX); + if (b >= SVE_VQ_MAX) + /* No non-virtualisable VLs found */ + sve_max_virtualisable_vl = SVE_VQ_MAX; + else if (WARN_ON(b == SVE_VQ_MAX - 1)) + /* No virtualisable VLs? This is architecturally forbidden. */ + sve_max_virtualisable_vl = SVE_VQ_MIN; + else /* b + 1 < SVE_VQ_MAX */ + sve_max_virtualisable_vl = sve_vl_from_vq(bit_to_vq(b + 1)); + + if (sve_max_virtualisable_vl > sve_max_vl) + sve_max_virtualisable_vl = sve_max_vl; + pr_info("SVE: maximum available vector length %u bytes per vector\n", sve_max_vl); pr_info("SVE: default vector length %u bytes per vector\n", sve_default_vl); + if (sve_max_virtualisable_vl < sve_max_vl) + pr_info("SVE: vector lengths greater than %u bytes not virtualisable\n", + sve_max_virtualisable_vl); sve_efi_setup(); }