From patchwork Thu Dec 1 10:49:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akihiko Odaki X-Patchwork-Id: 13061207 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 66873C43217 for ; Thu, 1 Dec 2022 10:51:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=mFb/8a9jhUvTtNvJwoIiBmmGWr5MSA9X2iXqoCdqciE=; b=LhMmXb9Epa7BUl gT6EpIedmGwMuKovS28Bk1zQrtdihgeE9gct0GEkA3/yFCBGzyfUCNNIMsOjq8SW8caxl96y4J91w dlQvD5SSfZacrAAd8G5Yn4AGuyFhVdHcLqck00geRToQijnG11vGf8D4J+kuogF0d381gNFy7wgOz 9biQbu2HiUcpfkMvQ9jM/geojdPCz4kYbXJ6xsjVMrpO8RV2rAQT29GQ4QHI6JCWcyx+FHk1d4LCi mZgSXYgDbhAQu3TH3MtDW2nv2tODiPXkvBYh8UCRVQkz/WRkkXFntPkYRskkEzONN1uZ0mQM9tSPD ryXjx3bD5dEC5VPEz7eA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1p0h8j-006qSg-RV; Thu, 01 Dec 2022 10:50:01 +0000 Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1p0h8b-006qNt-Db for linux-arm-kernel@lists.infradead.org; Thu, 01 Dec 2022 10:49:55 +0000 Received: by mail-pl1-x62e.google.com with SMTP id y17so1282378plp.3 for ; Thu, 01 Dec 2022 02:49:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=daynix-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9d6GIsRnHRwJeo4c8vwktBHVtkrfvERt/xtyaBPDy4o=; b=UF7drim7moVNa+AO0IAYUNeTBgBFvWm60+kHGmRNakQh4mdoH4iKggMg+4EuZZ3MXB IT1R1Ry/USDLKLT9/zJlCeUXXIfUngZFtUHVN+I4fjNpX4kQh0hfFESY+fgXXpr1TQE7 ymN49ybc4SPnqqpomMNQe4FB6r84YZphIB6HwgMx8a7LtOnhQZnRJQHD8vPEpswTT91J 7tJ0YRJwil8e/laIxfg1mJzooYKONCPnhl9QWxKHixN10B0cqzp0zcNwS9kOBOM+7fUN 8G2jD408lr4K+wH5xpIub+G6cQ2K/qp8zKvZips9UrraQ0mE9zP237bys0iNHMbGAAHT 4paA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9d6GIsRnHRwJeo4c8vwktBHVtkrfvERt/xtyaBPDy4o=; b=y5whQ+X8fFWn3O4bFq9fAAav1UlPOeako4SWz56XITNM0XsMGQKJu9cBDzuaHIEe49 l+8LOzQ9nxvHa0nIcNdVL548yW6tQbF5QmGPtYh9IUp7yOcVqcijMQ8OCkJxJmjCf2UL KLomiI654z6dH8aENgVY1dLRnjnTgReVDZHW6uOfoqviBc6lT8bvsJDI9HkkeDx8iANh 3UPCimfknZMSqP6betNxd3jlhbxeUvAtidrA2NAIJzxzu4zMSKtW8i7J4MNbMrWMu+25 eNj4CzcLk05/CrEBE9UvrQzbs1Z/8Wsm0J5g3OnezcIoOLYHKqKNZkx9MybKBSyhUO8N UmhA== X-Gm-Message-State: ANoB5pmf2ADGfDJRgFHJ7K0OKIyfkZYkBGmUXHiBr6RvT075K5nxuX5u 1pKbiEBAb3z6Bcg5s35ixzNLMg== X-Google-Smtp-Source: AA0mqf5I7QVq0C0CSAaOPDG0zKpfuEuQQ+mRZCxYhiDAqNchwNBAU9nCqkYFzRdnVyrzfG7tR327QQ== X-Received: by 2002:a17:902:ab89:b0:185:3659:1ce9 with SMTP id f9-20020a170902ab8900b0018536591ce9mr44908324plr.26.1669891790333; Thu, 01 Dec 2022 02:49:50 -0800 (PST) Received: from fedora.flets-east.jp ([2400:4050:c360:8200:8ae8:3c4:c0da:7419]) by smtp.gmail.com with ESMTPSA id 4-20020a630804000000b004785a63b44bsm2320580pgi.43.2022.12.01.02.49.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Dec 2022 02:49:50 -0800 (PST) From: Akihiko Odaki To: Cc: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, Mathieu Poirier , Oliver Upton , Suzuki K Poulose , Alexandru Elisei , James Morse , Marc Zyngier , Will Deacon , Catalin Marinas , asahi@lists.linux.dev, Alyssa Rosenzweig , Sven Peter , Hector Martin , Akihiko Odaki Subject: [PATCH 1/3] KVM: arm64: Make CCSIDRs consistent Date: Thu, 1 Dec 2022 19:49:12 +0900 Message-Id: <20221201104914.28944-2-akihiko.odaki@daynix.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221201104914.28944-1-akihiko.odaki@daynix.com> References: <20221201104914.28944-1-akihiko.odaki@daynix.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221201_024953_495536_3027B008 X-CRM114-Status: GOOD ( 16.98 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org A vCPU sees masked CCSIDRs when the physical CPUs has mismatched cache types or the vCPU has 32-bit EL1. Perform the same masking for ioctls too so that ioctls shows values consistent with the values the vCPU actually sees. Signed-off-by: Akihiko Odaki --- arch/arm64/include/asm/kvm_emulate.h | 9 ++++-- arch/arm64/kvm/sys_regs.c | 45 ++++++++++++++-------------- 2 files changed, 30 insertions(+), 24 deletions(-) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 9bdba47f7e14..b45cf8903190 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -61,6 +61,12 @@ static __always_inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu) } #endif +static inline bool vcpu_cache_overridden(struct kvm_vcpu *vcpu) +{ + return cpus_have_const_cap(ARM64_MISMATCHED_CACHE_TYPE) || + vcpu_el1_is_32bit(vcpu); +} + static inline void vcpu_reset_hcr(struct kvm_vcpu *vcpu) { vcpu->arch.hcr_el2 = HCR_GUEST_FLAGS; @@ -88,8 +94,7 @@ static inline void vcpu_reset_hcr(struct kvm_vcpu *vcpu) if (vcpu_el1_is_32bit(vcpu)) vcpu->arch.hcr_el2 &= ~HCR_RW; - if (cpus_have_const_cap(ARM64_MISMATCHED_CACHE_TYPE) || - vcpu_el1_is_32bit(vcpu)) + if (vcpu_cache_overridden(vcpu)) vcpu->arch.hcr_el2 |= HCR_TID2; if (kvm_has_mte(vcpu->kvm)) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index f4a7c5abcbca..273ed1aaa6b3 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -88,7 +88,7 @@ static u32 cache_levels; #define CSSELR_MAX 14 /* Which cache CCSIDR represents depends on CSSELR value. */ -static u32 get_ccsidr(u32 csselr) +static u32 get_ccsidr(struct kvm_vcpu *vcpu, u32 csselr) { u32 ccsidr; @@ -99,6 +99,21 @@ static u32 get_ccsidr(u32 csselr) ccsidr = read_sysreg(ccsidr_el1); local_irq_enable(); + /* + * Guests should not be doing cache operations by set/way at all, and + * for this reason, we trap them and attempt to infer the intent, so + * that we can flush the entire guest's address space at the appropriate + * time. + * To prevent this trapping from causing performance problems, let's + * expose the geometry of all data and unified caches (which are + * guaranteed to be PIPT and thus non-aliasing) as 1 set and 1 way. + * [If guests should attempt to infer aliasing properties from the + * geometry (which is not permitted by the architecture), they would + * only do so for virtually indexed caches.] + */ + if (vcpu_cache_overridden(vcpu) && !(csselr & 1)) // data or unified cache + ccsidr &= ~GENMASK(27, 3); + return ccsidr; } @@ -1300,22 +1315,8 @@ static bool access_ccsidr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, return write_to_read_only(vcpu, p, r); csselr = vcpu_read_sys_reg(vcpu, CSSELR_EL1); - p->regval = get_ccsidr(csselr); + p->regval = get_ccsidr(vcpu, csselr); - /* - * Guests should not be doing cache operations by set/way at all, and - * for this reason, we trap them and attempt to infer the intent, so - * that we can flush the entire guest's address space at the appropriate - * time. - * To prevent this trapping from causing performance problems, let's - * expose the geometry of all data and unified caches (which are - * guaranteed to be PIPT and thus non-aliasing) as 1 set and 1 way. - * [If guests should attempt to infer aliasing properties from the - * geometry (which is not permitted by the architecture), they would - * only do so for virtually indexed caches.] - */ - if (!(csselr & 1)) // data or unified cache - p->regval &= ~GENMASK(27, 3); return true; } @@ -2686,7 +2687,7 @@ static bool is_valid_cache(u32 val) } } -static int demux_c15_get(u64 id, void __user *uaddr) +static int demux_c15_get(struct kvm_vcpu *vcpu, u64 id, void __user *uaddr) { u32 val; u32 __user *uval = uaddr; @@ -2705,13 +2706,13 @@ static int demux_c15_get(u64 id, void __user *uaddr) if (!is_valid_cache(val)) return -ENOENT; - return put_user(get_ccsidr(val), uval); + return put_user(get_ccsidr(vcpu, val), uval); default: return -ENOENT; } } -static int demux_c15_set(u64 id, void __user *uaddr) +static int demux_c15_set(struct kvm_vcpu *vcpu, u64 id, void __user *uaddr) { u32 val, newval; u32 __user *uval = uaddr; @@ -2734,7 +2735,7 @@ static int demux_c15_set(u64 id, void __user *uaddr) return -EFAULT; /* This is also invariant: you can't change it. */ - if (newval != get_ccsidr(val)) + if (newval != get_ccsidr(vcpu, val)) return -EINVAL; return 0; default: @@ -2773,7 +2774,7 @@ int kvm_arm_sys_reg_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg int err; if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_DEMUX) - return demux_c15_get(reg->id, uaddr); + return demux_c15_get(vcpu, reg->id, uaddr); err = get_invariant_sys_reg(reg->id, uaddr); if (err != -ENOENT) @@ -2817,7 +2818,7 @@ int kvm_arm_sys_reg_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg int err; if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_DEMUX) - return demux_c15_set(reg->id, uaddr); + return demux_c15_set(vcpu, reg->id, uaddr); err = set_invariant_sys_reg(reg->id, uaddr); if (err != -ENOENT) From patchwork Thu Dec 1 10:49:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akihiko Odaki X-Patchwork-Id: 13061209 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1A45AC43217 for ; Thu, 1 Dec 2022 10:51:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ISsTD1sbu0IBJwlSYiUdcbejthbrxP/eg7XWzO6MNl8=; b=J2CXIeVYAiOaLC 6iEz5cI+odpFgkVtPvXgHj19JAfyoz7+6f6aD3RFw4QeD3qaAHwguQnpv0qmMl281M0/SGJCQKCFr GHvpNTO3L3+n9/RhVijxFdu2f0+EYUtqZVaxrKKGb5zmVK3EOzina6eCab1eYVV0l2jk9adN8sBvj XiSm62nUhoYwE/VNf/ciMh4NvPzwhMKdlylkq+/iT49tduDTqx8iVVwV45Zesxxqrs/rCG3KcWBDM +FTdgW7zRxt0+KyvlWLknce90lrrdFxvIOYVPoXsrntb5WnwOWxo2YjxWzt5QGEZQ8sy3m9WJSBVN bQyzRdCwnHs8+mCaV9+g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1p0h8s-006qVH-GE; Thu, 01 Dec 2022 10:50:10 +0000 Received: from mail-pl1-x62f.google.com ([2607:f8b0:4864:20::62f]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1p0h8i-006qPx-G2 for linux-arm-kernel@lists.infradead.org; Thu, 01 Dec 2022 10:50:02 +0000 Received: by mail-pl1-x62f.google.com with SMTP id 4so1313770pli.0 for ; Thu, 01 Dec 2022 02:49:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=daynix-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TzHwXsUVcNKW6PtJQ1J0+WaE2jc/6muUSDkeS4NHJ3A=; b=zaJLFADO9dpcOwd8VIhLCtDevcWu59j47Ph4/0tK9r6GAyRUKtSxenx+7EClnbg6fr 6IyGsDvGPi5Z+IAfrS0dw4dnH9lCGonGwIRhn9f88nTNTV1+KB1+LSQPoH7H9q/qui0X 0DhFOb/5MLv/+3qfOvNXPBTPp9I1oLOYHVV9vsVBn201NcZ1pBZ1q9pDEmfnqn6s67z5 A+768GnVUJyzWP3tPFdo0KWGx9LaROqLzgKLpFIXFSO1e4nU3p2rR8hjlFq5Jwbbgehj 9kAyQ9K601ktg6ACOKCWSXqfMOevNCZ3XnL9r7jGSCSK6/6ZL070AydGSLUbthFtRTBz iIIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TzHwXsUVcNKW6PtJQ1J0+WaE2jc/6muUSDkeS4NHJ3A=; b=3RMO/TP5xi8Ad+mWJE8U75RqUDbaxNkJ8gsHf+wBTpQKwR+C+57r5U5Owumjd+Ayg1 BU8MLzeNfnp6vTod9cjKMMen3Oi9CA0y+ZHy49OAToXM3uHRz99X+fHfTOZIxzUpsK26 c1V6uZSeYiKA2RAGCnqifqGu66rmqh5Q4cV2mDKqN7eVwNKbSA8qPaw88ZgRWGEkrq2l Jud43g+AiKbOzcjwXwzkA4TpryGOs9pj86puRHYMQqUCXLx9lTScwe10q+WNQMhZFt1X YwDJITcR2NXpvvWvarhKrUGsgDGx4ENSa9NuEM90q+Wv3mHvxK2xg/6a0dIg6pIvMETf +1Iw== X-Gm-Message-State: ANoB5pmoH3J2UgKqX0ELnv7JKBvsU75Xb/YoX0dyGZHE2LcLN9CMIhEG GoqvvxaQ+D2PK735S6jCiO0LXA== X-Google-Smtp-Source: AA0mqf5ReIh59TFhDLI1b4fbtC++P8aYNKOnfrq/q12L+UGEiXe5pO5lFhjAZ+SxS6tvsWAlH5LMiw== X-Received: by 2002:a17:903:cc:b0:189:7441:1ada with SMTP id x12-20020a17090300cc00b0018974411adamr26762017plc.13.1669891794164; Thu, 01 Dec 2022 02:49:54 -0800 (PST) Received: from fedora.flets-east.jp ([2400:4050:c360:8200:8ae8:3c4:c0da:7419]) by smtp.gmail.com with ESMTPSA id 4-20020a630804000000b004785a63b44bsm2320580pgi.43.2022.12.01.02.49.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Dec 2022 02:49:53 -0800 (PST) From: Akihiko Odaki To: Cc: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, Mathieu Poirier , Oliver Upton , Suzuki K Poulose , Alexandru Elisei , James Morse , Marc Zyngier , Will Deacon , Catalin Marinas , asahi@lists.linux.dev, Alyssa Rosenzweig , Sven Peter , Hector Martin , Akihiko Odaki Subject: [PATCH 2/3] arm64: errata: Check for mismatched cache associativity Date: Thu, 1 Dec 2022 19:49:13 +0900 Message-Id: <20221201104914.28944-3-akihiko.odaki@daynix.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221201104914.28944-1-akihiko.odaki@daynix.com> References: <20221201104914.28944-1-akihiko.odaki@daynix.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221201_025000_561175_2C4A4CDA X-CRM114-Status: GOOD ( 24.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org M2 MacBook Air has mismatched CCSIDR associativity bits, which makes the bits a KVM vCPU sees inconsistent when migrating. Record such mismatches so that KVM can use the information later to avoid the problem. Signed-off-by: Akihiko Odaki --- arch/arm64/include/asm/cache.h | 3 ++ arch/arm64/include/asm/cpu.h | 1 + arch/arm64/include/asm/cpufeature.h | 8 +++++ arch/arm64/include/asm/sysreg.h | 7 ++++ arch/arm64/kernel/cacheinfo.c | 4 +-- arch/arm64/kernel/cpu_errata.c | 52 +++++++++++++++++++++++++++++ arch/arm64/kernel/cpufeature.c | 4 +++ arch/arm64/kernel/cpuinfo.c | 30 +++++++++++++++++ arch/arm64/kvm/sys_regs.c | 4 +-- arch/arm64/tools/cpucaps | 1 + 10 files changed, 110 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/cache.h b/arch/arm64/include/asm/cache.h index c0b178d1bb4f..eeab2b8c7e71 100644 --- a/arch/arm64/include/asm/cache.h +++ b/arch/arm64/include/asm/cache.h @@ -72,6 +72,8 @@ static inline u32 cache_type_cwg(void) #define __read_mostly __section(".data..read_mostly") +#define MAX_CACHE_LEVEL 7 /* Max 7 level supported */ + static inline int cache_line_size_of_cpu(void) { u32 cwg = cache_type_cwg(); @@ -80,6 +82,7 @@ static inline int cache_line_size_of_cpu(void) } int cache_line_size(void); +enum cache_type get_cache_type(int level); /* * Read the effective value of CTR_EL0. diff --git a/arch/arm64/include/asm/cpu.h b/arch/arm64/include/asm/cpu.h index fd7a92219eea..b8d4f31ed59b 100644 --- a/arch/arm64/include/asm/cpu.h +++ b/arch/arm64/include/asm/cpu.h @@ -41,6 +41,7 @@ struct cpuinfo_arm64 { struct cpu cpu; struct kobject kobj; u64 reg_ctr; + struct ccsidr reg_ccsidr[MAX_CACHE_LEVEL + 1]; u64 reg_cntfrq; u64 reg_dczid; u64 reg_midr; diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index f73f11b55042..104483151362 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -7,6 +7,7 @@ #define __ASM_CPUFEATURE_H #include +#include #include #include #include @@ -917,6 +918,13 @@ extern struct arm64_ftr_override id_aa64isar2_override; u32 get_kvm_ipa_limit(void); void dump_cpu_features(void); +struct ccsidr { + u64 data; + u64 inst; +}; + +extern struct ccsidr ccsidr[MAX_CACHE_LEVEL + 1]; + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 7d301700d1a9..e796f14fdc2a 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -941,6 +941,13 @@ #define HFGxTR_EL2_nSMPRI_EL1_SHIFT 54 #define HFGxTR_EL2_nSMPRI_EL1_MASK BIT_MASK(HFGxTR_EL2_nSMPRI_EL1_SHIFT) +/* CCSIDR_EL1 bit definitions */ +#define CCSIDR_ASSOCIATIVITY_BITS_MASK GENMASK(27, 3) + +/* CSSELR_EL1 */ +#define CSSELR_IN 1 +#define CSSELR_LEVEL_SHIFT 1 + #define ARM64_FEATURE_FIELD_BITS 4 /* Create a mask for the feature bits of the specified feature. */ diff --git a/arch/arm64/kernel/cacheinfo.c b/arch/arm64/kernel/cacheinfo.c index 97c42be71338..2e808ccc15bf 100644 --- a/arch/arm64/kernel/cacheinfo.c +++ b/arch/arm64/kernel/cacheinfo.c @@ -10,7 +10,6 @@ #include #include -#define MAX_CACHE_LEVEL 7 /* Max 7 level supported */ /* Ctypen, bits[3(n - 1) + 2 : 3(n - 1)], for n = 1 to 7 */ #define CLIDR_CTYPE_SHIFT(level) (3 * (level - 1)) #define CLIDR_CTYPE_MASK(level) (7 << CLIDR_CTYPE_SHIFT(level)) @@ -26,7 +25,7 @@ int cache_line_size(void) } EXPORT_SYMBOL_GPL(cache_line_size); -static inline enum cache_type get_cache_type(int level) +enum cache_type get_cache_type(int level) { u64 clidr; @@ -35,6 +34,7 @@ static inline enum cache_type get_cache_type(int level) clidr = read_sysreg(clidr_el1); return CLIDR_CTYPE(clidr, level); } +EXPORT_SYMBOL_GPL(get_cache_type); static void ci_leaf_init(struct cacheinfo *this_leaf, enum cache_type type, unsigned int level) diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c index 89ac00084f38..5caccf602fc0 100644 --- a/arch/arm64/kernel/cpu_errata.c +++ b/arch/arm64/kernel/cpu_errata.c @@ -8,6 +8,8 @@ #include #include #include +#include +#include #include #include #include @@ -87,6 +89,50 @@ has_mismatched_cache_type(const struct arm64_cpu_capabilities *entry, return (ctr_real != sys) && (ctr_raw != sys); } +static bool +has_mismatched_cache_associativity(const struct arm64_cpu_capabilities *entry, + int scope) +{ + u64 mask = CCSIDR_ASSOCIATIVITY_BITS_MASK; + u64 real; + bool mismatched = false; + enum cache_type cache_type; + unsigned int i; + + WARN_ON(scope != SCOPE_LOCAL_CPU || preemptible()); + + local_irq_disable(); + + for (i = 0; i <= MAX_CACHE_LEVEL; i++) { + cache_type = get_cache_type(i); + + if ((cache_type & (CACHE_TYPE_DATA | CACHE_TYPE_UNIFIED))) { + write_sysreg(i << CSSELR_LEVEL_SHIFT, csselr_el1); + isb(); + real = read_sysreg(ccsidr_el1); + if ((ccsidr[i].data & mask) != (real & mask)) { + mismatched = true; + break; + } + } + + if ((cache_type & CACHE_TYPE_INST)) { + write_sysreg((i << CSSELR_LEVEL_SHIFT) | CSSELR_IN, + csselr_el1); + isb(); + real = read_sysreg(ccsidr_el1); + if ((ccsidr[i].inst & mask) != (real & mask)) { + mismatched = true; + break; + } + } + } + + local_irq_enable(); + + return mismatched; +} + static void cpu_enable_trap_ctr_access(const struct arm64_cpu_capabilities *cap) { @@ -499,6 +545,12 @@ const struct arm64_cpu_capabilities arm64_errata[] = { ERRATA_MIDR_RANGE_LIST(cavium_erratum_30115_cpus), }, #endif + { + .desc = "Mismatched cache associativity", + .capability = ARM64_MISMATCHED_CACHE_ASSOCIATIVITY, + .matches = has_mismatched_cache_associativity, + .type = ARM64_CPUCAP_LOCAL_CPU_ERRATUM, + }, { .desc = "Mismatched cache type (CTR_EL0)", .capability = ARM64_MISMATCHED_CACHE_TYPE, diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index b3f37e2209ad..ef259396aa4c 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -930,6 +930,8 @@ static void init_cpu_ftr_reg(u32 sys_reg, u64 new) reg->user_mask = user_mask; } +struct ccsidr ccsidr[MAX_CACHE_LEVEL + 1]; + extern const struct arm64_cpu_capabilities arm64_errata[]; static const struct arm64_cpu_capabilities arm64_features[]; @@ -1039,6 +1041,8 @@ void __init init_cpu_features(struct cpuinfo_arm64 *info) * after we have initialised the CPU feature infrastructure. */ setup_boot_cpu_capabilities(); + + memcpy(ccsidr, info->reg_ccsidr, sizeof(ccsidr)); } static void update_cpu_ftr_reg(struct arm64_ftr_reg *reg, u64 new) diff --git a/arch/arm64/kernel/cpuinfo.c b/arch/arm64/kernel/cpuinfo.c index 28d4f442b0bc..b1ea276b9d10 100644 --- a/arch/arm64/kernel/cpuinfo.c +++ b/arch/arm64/kernel/cpuinfo.c @@ -13,6 +13,7 @@ #include #include +#include #include #include #include @@ -47,6 +48,34 @@ static inline const char *icache_policy_str(int l1ip) } } +static void read_ccsidr(struct ccsidr *ccsidr) +{ + enum cache_type cache_type; + unsigned int i; + + local_irq_disable(); + + for (i = 0; i <= MAX_CACHE_LEVEL; i++) { + cache_type = get_cache_type(i); + + if ((cache_type & (CACHE_TYPE_DATA | CACHE_TYPE_UNIFIED))) { + write_sysreg(i << CSSELR_LEVEL_SHIFT, csselr_el1); + isb(); + ccsidr[i].data = read_sysreg(ccsidr_el1); + break; + } + + if ((cache_type & CACHE_TYPE_INST)) { + write_sysreg((i << CSSELR_LEVEL_SHIFT) | CSSELR_IN, + csselr_el1); + isb(); + ccsidr[i].inst = read_sysreg(ccsidr_el1); + } + } + + local_irq_enable(); +} + unsigned long __icache_flags; static const char *const hwcap_str[] = { @@ -440,6 +469,7 @@ static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info) if (id_aa64pfr0_32bit_el0(info->reg_id_aa64pfr0)) __cpuinfo_store_cpu_32bit(&info->aarch32); + read_ccsidr(info->reg_ccsidr); cpuinfo_detect_icache_policy(info); } diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 273ed1aaa6b3..1f0cb015e81c 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -111,8 +111,8 @@ static u32 get_ccsidr(struct kvm_vcpu *vcpu, u32 csselr) * geometry (which is not permitted by the architecture), they would * only do so for virtually indexed caches.] */ - if (vcpu_cache_overridden(vcpu) && !(csselr & 1)) // data or unified cache - ccsidr &= ~GENMASK(27, 3); + if (vcpu_cache_overridden(vcpu) && !(csselr & CSSELR_IN)) // data or unified cache + ccsidr &= ~CCSIDR_ASSOCIATIVITY_BITS_MASK; return ccsidr; } diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index f1c0347ec31a..061c93319295 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -44,6 +44,7 @@ HAS_VIRT_HOST_EXTN HAS_WFXT HW_DBM KVM_PROTECTED_MODE +MISMATCHED_CACHE_ASSOCIATIVITY MISMATCHED_CACHE_TYPE MTE MTE_ASYMM From patchwork Thu Dec 1 10:49:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akihiko Odaki X-Patchwork-Id: 13061210 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8CAE8C43217 for ; Thu, 1 Dec 2022 10:51:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=cq13ARCNdUzWzPnXP7vgGycWn/ofKGl6LVAqcW2qwCo=; b=jxYB7MuOLGUI1A Egkzw1XjARz8wlTAs1C4WiLi33Whke+c8i5+SFfM9ULcP8slU3ppZTUtDAT3oN778p3EM9UT1OZjD ZaG0FEyH522tfy8vZbzJpTOcpxiGkK9i4V4v7MfxK4My67wV90CERL5L3uRVECqxXmCnViyNcMMJW NfC5G04dEf+usZLBh8iVgrimlZYbIydGQ12cf3a4eJHV7gzIbu/X0JedGCdcU0WD5ZTjgGCVFW6H+ +cU3m8Ddpz2Y3f+enRhRQTVdK/CU8jRMxfvP4+5+gj4Pg+8GgvMW/Y9wXsHsvJpqjE4bBXEwaMg1E ApEw0J7UbULJtuwh7JXA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1p0h92-006qZX-MN; Thu, 01 Dec 2022 10:50:20 +0000 Received: from mail-pj1-x102b.google.com ([2607:f8b0:4864:20::102b]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1p0h8v-006qRr-JW for linux-arm-kernel@lists.infradead.org; Thu, 01 Dec 2022 10:50:17 +0000 Received: by mail-pj1-x102b.google.com with SMTP id t11-20020a17090a024b00b0021932afece4so4823462pje.5 for ; Thu, 01 Dec 2022 02:49:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=daynix-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JPjVjf+i0nuKjzq4xuTtWcDbzlk974FEzySz6jaQavQ=; b=707jnRzZEQf01TpjqN9DSN/NFPM/VMAM0ZKeLKPJRYCiI6ojaLlzyyjRPUpTLHEb7+ t/85QUAWoI1i3uBqrU+h1j5F4G8R54m7oP3WPD6Z75LxFEs8MFosOJnDoLoJMShEhD6T M0uSLQjMILQC+3vSNj7yMjn7Uf63q85nOteYpM/WFq3kNP/C4aTX+hOPClaIklzFJcn5 9uFkhbiVD8TmmdCAgFtsMbaJH9BYrMzT0ak6Zy/dNho9mC3EqFLBduH3k6r/R8W0Ehwh 3gwT9d0maH8rr+D9FUCYBsu4HRCMmow03xksU5//CMHjWJ9+r3fRPZkEhMzUnLtP4A4y iGtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JPjVjf+i0nuKjzq4xuTtWcDbzlk974FEzySz6jaQavQ=; b=3LqBHhxWfC3STgnRrCVUYIIkolA8+ay9Af3Q3z7Ar23xW2/DbwTD2T4v7szz7lQ2aB nxogznWAH5hqN+os+3G4mXKVMzEndD6eHV59X1iNtMMsmb2skKOBL3NriPdOysv42vB5 LHoH1Air/WpslayH8YBqgUEgsw+wS98KeWvKRcYKDW37RHHK6uBnBU4ziUmnCSRnN7fo P4iR/K5m/gVs2m1YxxzCZU2edni05a+JoRfS6swEW1fEaWf47hH8FJMh+/lD4Hh+cmC0 rkuHV4/lqwARqYMuBMcOKxsoGmM+Nb45hszcksdFGM+QQo94SyvWSvKKvJRRHFLwsbdn Sugg== X-Gm-Message-State: ANoB5pkhZC/nF7tjdvRN/9+PH39GCnQeLaFj4xuuWFeGLzvxxL4rHMbP wVK79Yi9GnnufKG2T2wcIDLM8A== X-Google-Smtp-Source: AA0mqf5Yhsy8Ypvr0GCLP2Zg6ZsUX9UGviILhc7n8Tbc14XMJHu8PCzO14QcWmW7k5EhIGo/uGorQw== X-Received: by 2002:a17:902:8d98:b0:189:8e39:3c88 with SMTP id v24-20020a1709028d9800b001898e393c88mr19470197plo.102.1669891797955; Thu, 01 Dec 2022 02:49:57 -0800 (PST) Received: from fedora.flets-east.jp ([2400:4050:c360:8200:8ae8:3c4:c0da:7419]) by smtp.gmail.com with ESMTPSA id 4-20020a630804000000b004785a63b44bsm2320580pgi.43.2022.12.01.02.49.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Dec 2022 02:49:57 -0800 (PST) From: Akihiko Odaki To: Cc: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, Mathieu Poirier , Oliver Upton , Suzuki K Poulose , Alexandru Elisei , James Morse , Marc Zyngier , Will Deacon , Catalin Marinas , asahi@lists.linux.dev, Alyssa Rosenzweig , Sven Peter , Hector Martin , Akihiko Odaki Subject: [PATCH 3/3] KVM: arm64: Handle CCSIDR associativity mismatches Date: Thu, 1 Dec 2022 19:49:14 +0900 Message-Id: <20221201104914.28944-4-akihiko.odaki@daynix.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221201104914.28944-1-akihiko.odaki@daynix.com> References: <20221201104914.28944-1-akihiko.odaki@daynix.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221201_025013_718337_4D40820D X-CRM114-Status: GOOD ( 14.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org CCSIDR associativity mismatches among the physical CPUs causes a vCPU see inconsistent values when it is migrated among physical CPUs. It also makes QEMU fail restoring the vCPU registers because QEMU saves and restores all of the registers including CCSIDRs, and if the vCPU migrated among physical CPUs between saving and restoring, it tries to restore CCSIDR values that mismatch with the current physical CPU, which causes EFAULT. Trap CCSIDRs if there are CCSIDR value msimatches, and override the associativity bits when handling the trap. Signed-off-by: Akihiko Odaki --- arch/arm64/include/asm/kvm_emulate.h | 1 + arch/arm64/kvm/sys_regs.c | 7 ++++++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index b45cf8903190..df2bab867e12 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -64,6 +64,7 @@ static __always_inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu) static inline bool vcpu_cache_overridden(struct kvm_vcpu *vcpu) { return cpus_have_const_cap(ARM64_MISMATCHED_CACHE_TYPE) || + cpus_have_const_cap(ARM64_MISMATCHED_CACHE_ASSOCIATIVITY) || vcpu_el1_is_32bit(vcpu); } diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 1f0cb015e81c..181a5b215a0e 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -110,8 +110,13 @@ static u32 get_ccsidr(struct kvm_vcpu *vcpu, u32 csselr) * [If guests should attempt to infer aliasing properties from the * geometry (which is not permitted by the architecture), they would * only do so for virtually indexed caches.] + * + * This also makes sure the associativity bits of the CCSIDRs, including + * the ones of CCSIDRs for instruction caches, are overridden when the + * physical CPUs have a heterogeneous configuration so that a vCPU sees + * the consistent values if it is migrated among physical CPUs. */ - if (vcpu_cache_overridden(vcpu) && !(csselr & CSSELR_IN)) // data or unified cache + if (vcpu_cache_overridden(vcpu)) ccsidr &= ~CCSIDR_ASSOCIATIVITY_BITS_MASK; return ccsidr;