From patchwork Thu Jun 13 23:36:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697614 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CD203C27C4F for ; Thu, 13 Jun 2024 23:39:27 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu1T-00053x-JM; Thu, 13 Jun 2024 19:38:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu1R-00053Q-4Y; Thu, 13 Jun 2024 19:38:25 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu1P-0002y6-2S; Thu, 13 Jun 2024 19:38:24 -0400 Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0f2F27fMz67kr9; Fri, 14 Jun 2024 07:36:53 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id B9FA7140594; Fri, 14 Jun 2024 07:38:16 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:37:54 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 01/29] arm/virt, target/arm: Add new ARMCPU {socket, cluster, core, thread}-id property Date: Fri, 14 Jun 2024 00:36:11 +0100 Message-ID: <20240613233639.202896-2-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This shall be used to store user specified topology{socket,cluster,core,thread} and shall be converted to a unique 'vcpu-id' which is used as slot-index during hot(un)plug of vCPU. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 10 ++++++++++ include/hw/arm/virt.h | 28 ++++++++++++++++++++++++++++ target/arm/cpu.c | 4 ++++ target/arm/cpu.h | 4 ++++ 4 files changed, 46 insertions(+) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 3c93c0c0a6..11fc7fc318 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2215,6 +2215,14 @@ static void machvirt_init(MachineState *machine) &error_fatal); aarch64 &= object_property_get_bool(cpuobj, "aarch64", NULL); + object_property_set_int(cpuobj, "socket-id", + virt_get_socket_id(machine, n), NULL); + object_property_set_int(cpuobj, "cluster-id", + virt_get_cluster_id(machine, n), NULL); + object_property_set_int(cpuobj, "core-id", + virt_get_core_id(machine, n), NULL); + object_property_set_int(cpuobj, "thread-id", + virt_get_thread_id(machine, n), NULL); if (!vms->secure) { object_property_set_bool(cpuobj, "has_el3", false, NULL); @@ -2708,6 +2716,7 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms) { int n; unsigned int max_cpus = ms->smp.max_cpus; + unsigned int smp_threads = ms->smp.threads; VirtMachineState *vms = VIRT_MACHINE(ms); MachineClass *mc = MACHINE_GET_CLASS(vms); @@ -2721,6 +2730,7 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms) ms->possible_cpus->len = max_cpus; for (n = 0; n < ms->possible_cpus->len; n++) { ms->possible_cpus->cpus[n].type = ms->cpu_type; + ms->possible_cpus->cpus[n].vcpus_count = smp_threads; ms->possible_cpus->cpus[n].arch_id = virt_cpu_mp_affinity(vms, n); diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index bb486d36b1..6f9a7bb60b 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -209,4 +209,32 @@ static inline int virt_gicv3_redist_region_count(VirtMachineState *vms) vms->highmem_redists) ? 2 : 1; } +static inline int virt_get_socket_id(const MachineState *ms, int cpu_index) +{ + assert(cpu_index >= 0 && cpu_index < ms->possible_cpus->len); + + return ms->possible_cpus->cpus[cpu_index].props.socket_id; +} + +static inline int virt_get_cluster_id(const MachineState *ms, int cpu_index) +{ + assert(cpu_index >= 0 && cpu_index < ms->possible_cpus->len); + + return ms->possible_cpus->cpus[cpu_index].props.cluster_id; +} + +static inline int virt_get_core_id(const MachineState *ms, int cpu_index) +{ + assert(cpu_index >= 0 && cpu_index < ms->possible_cpus->len); + + return ms->possible_cpus->cpus[cpu_index].props.core_id; +} + +static inline int virt_get_thread_id(const MachineState *ms, int cpu_index) +{ + assert(cpu_index >= 0 && cpu_index < ms->possible_cpus->len); + + return ms->possible_cpus->cpus[cpu_index].props.thread_id; +} + #endif /* QEMU_ARM_VIRT_H */ diff --git a/target/arm/cpu.c b/target/arm/cpu.c index 77f8c9c748..abc4ed0842 100644 --- a/target/arm/cpu.c +++ b/target/arm/cpu.c @@ -2582,6 +2582,10 @@ static Property arm_cpu_properties[] = { DEFINE_PROP_UINT64("mp-affinity", ARMCPU, mp_affinity, ARM64_AFFINITY_INVALID), DEFINE_PROP_INT32("node-id", ARMCPU, node_id, CPU_UNSET_NUMA_NODE_ID), + DEFINE_PROP_INT32("socket-id", ARMCPU, socket_id, 0), + DEFINE_PROP_INT32("cluster-id", ARMCPU, cluster_id, 0), + DEFINE_PROP_INT32("core-id", ARMCPU, core_id, 0), + DEFINE_PROP_INT32("thread-id", ARMCPU, thread_id, 0), DEFINE_PROP_INT32("core-count", ARMCPU, core_count, -1), /* True to default to the backward-compat old CNTFRQ rather than 1Ghz */ DEFINE_PROP_BOOL("backcompat-cntfrq", ARMCPU, backcompat_cntfrq, false), diff --git a/target/arm/cpu.h b/target/arm/cpu.h index c17264c239..208c719db3 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -1076,6 +1076,10 @@ struct ArchCPU { QLIST_HEAD(, ARMELChangeHook) el_change_hooks; int32_t node_id; /* NUMA node this CPU belongs to */ + int32_t socket_id; + int32_t cluster_id; + int32_t core_id; + int32_t thread_id; /* Used to synchronize KVM and QEMU in-kernel device levels */ uint8_t device_irq_level; From patchwork Thu Jun 13 23:36:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697618 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6C641C27C75 for ; Thu, 13 Jun 2024 23:40:08 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu1s-0005CS-6n; Thu, 13 Jun 2024 19:38:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu1l-00057j-97; Thu, 13 Jun 2024 19:38:45 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu1h-0002zT-No; Thu, 13 Jun 2024 19:38:44 -0400 Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0f4D540Wz6K6Bt; Fri, 14 Jun 2024 07:38:36 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 8C601140594; Fri, 14 Jun 2024 07:38:39 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:38:17 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 02/29] cpu-common: Add common CPU utility for possible vCPUs Date: Fri, 14 Jun 2024 00:36:12 +0100 Message-ID: <20240613233639.202896-3-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This patch adds various utility functions that may be required to fetch or check the state of possible vCPUs. It also introduces the concept of *disabled* vCPUs, which are part of the *possible* vCPUs but are not enabled. This state will be used during machine initialization and later during the plugging or unplugging of vCPUs. We release the QOM CPU objects for all disabled vCPUs. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- cpu-common.c | 31 +++++++++++++++++++++++++ include/hw/core/cpu.h | 54 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 85 insertions(+) diff --git a/cpu-common.c b/cpu-common.c index ce78273af5..49d2a50835 100644 --- a/cpu-common.c +++ b/cpu-common.c @@ -24,6 +24,7 @@ #include "sysemu/cpus.h" #include "qemu/lockable.h" #include "trace/trace-root.h" +#include "hw/boards.h" QemuMutex qemu_cpu_list_lock; static QemuCond exclusive_cond; @@ -107,6 +108,36 @@ void cpu_list_remove(CPUState *cpu) cpu_list_generation_id++; } +CPUState *qemu_get_possible_cpu(int index) +{ + MachineState *ms = MACHINE(qdev_get_machine()); + const CPUArchIdList *possible_cpus = ms->possible_cpus; + + assert((index >= 0) && (index < possible_cpus->len)); + + return CPU(possible_cpus->cpus[index].cpu); +} + +bool qemu_present_cpu(CPUState *cpu) +{ + return cpu; +} + +bool qemu_enabled_cpu(CPUState *cpu) +{ + return cpu && !cpu->disabled; +} + +uint64_t qemu_get_cpu_archid(int cpu_index) +{ + MachineState *ms = MACHINE(qdev_get_machine()); + const CPUArchIdList *possible_cpus = ms->possible_cpus; + + assert((cpu_index >= 0) && (cpu_index < possible_cpus->len)); + + return possible_cpus->cpus[cpu_index].arch_id; +} + CPUState *qemu_get_cpu(int index) { CPUState *cpu; diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index 60b160d0b4..60b4778da9 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -528,6 +528,18 @@ struct CPUState { CPUPluginState *plugin_state; #endif + /* + * Some architectures do not allow the *presence* of vCPUs to be changed + * after the guest has booted, based on information specified by the + * VMM/firmware via ACPI MADT at boot time. Thus, to enable vCPU hotplug on + * these architectures, possible vCPUs can have a CPUState object in a + * 'disabled' state or may not have a CPUState object at all. This is + * possible when vCPU hotplug is supported, and vCPUs are + * 'yet-to-be-plugged' in the QOM or have been hot-unplugged. By default, + * every CPUState is enabled across all architectures. + */ + bool disabled; + /* TODO Move common fields from CPUArchState here. */ int cpu_index; int cluster_index; @@ -914,6 +926,48 @@ static inline bool cpu_in_exclusive_context(const CPUState *cpu) */ CPUState *qemu_get_cpu(int index); +/** + * qemu_get_possible_cpu: + * @index: The CPUState@cpu_index value of the CPU to obtain. + * Input index MUST be in range [0, Max Possible CPUs) + * + * If CPUState object exists,then it gets a CPU matching + * @index in the possible CPU array. + * + * Returns: The possible CPU or %NULL if CPU does not exist. + */ +CPUState *qemu_get_possible_cpu(int index); + +/** + * qemu_present_cpu: + * @cpu: The vCPU to check + * + * Checks if the vCPU is amongst the present possible vcpus. + * + * Returns: True if it is present possible vCPU else false + */ +bool qemu_present_cpu(CPUState *cpu); + +/** + * qemu_enabled_cpu: + * @cpu: The vCPU to check + * + * Checks if the vCPU is enabled. + * + * Returns: True if it is 'enabled' else false + */ +bool qemu_enabled_cpu(CPUState *cpu); + +/** + * qemu_get_cpu_archid: + * @cpu_index: possible vCPU for which arch-id needs to be retreived + * + * Fetches the vCPU arch-id from the present possible vCPUs. + * + * Returns: arch-id of the possible vCPU + */ +uint64_t qemu_get_cpu_archid(int cpu_index); + /** * cpu_exists: * @id: Guest-exposed CPU ID to lookup. From patchwork Thu Jun 13 23:36:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697616 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F3F1AC27C4F for ; Thu, 13 Jun 2024 23:39:56 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu2A-0005JY-Qn; Thu, 13 Jun 2024 19:39:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu27-0005Il-IL; Thu, 13 Jun 2024 19:39:07 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu25-00031Q-LD; Thu, 13 Jun 2024 19:39:07 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0dzJ075Cz6HJTn; Fri, 14 Jun 2024 07:34:20 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 526FF140B63; Fri, 14 Jun 2024 07:39:02 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:38:39 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 03/29] hw/arm/virt: Limit number of possible vCPUs for unsupported Accel or GIC Type Date: Fri, 14 Jun 2024 00:36:13 +0100 Message-ID: <20240613233639.202896-4-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org If Virtual CPU Hotplug support does not exist on a particular Accel platform or ARM GIC version, we should limit the possible vCPUs to those available during boot time (i.e SMP CPUs) and explicitly disable Virtual CPU Hotplug support. Signed-off-by: Salil Mehta --- hw/arm/virt.c | 66 +++++++++++++++++++++++++++++---------------------- 1 file changed, 38 insertions(+), 28 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 11fc7fc318..3e1c4d2d2f 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2082,8 +2082,6 @@ static void machvirt_init(MachineState *machine) unsigned int smp_cpus = machine->smp.cpus; unsigned int max_cpus = machine->smp.max_cpus; - possible_cpus = mc->possible_cpu_arch_ids(machine); - /* * In accelerated mode, the memory map is computed earlier in kvm_type() * to create a VM with the right number of IPA bits. @@ -2098,7 +2096,7 @@ static void machvirt_init(MachineState *machine) * we are about to deal with. Once this is done, get rid of * the object. */ - cpuobj = object_new(possible_cpus->cpus[0].type); + cpuobj = object_new(machine->cpu_type); armcpu = ARM_CPU(cpuobj); pa_bits = arm_pamax(armcpu); @@ -2113,6 +2111,43 @@ static void machvirt_init(MachineState *machine) */ finalize_gic_version(vms); + /* + * The maximum number of CPUs depends on the GIC version, or on how + * many redistributors we can fit into the memory map (which in turn + * depends on whether this is a GICv3 or v4). + */ + if (vms->gic_version == VIRT_GIC_VERSION_2) { + virt_max_cpus = GIC_NCPU; + } else { + virt_max_cpus = virt_redist_capacity(vms, VIRT_GIC_REDIST); + if (vms->highmem_redists) { + virt_max_cpus += virt_redist_capacity(vms, VIRT_HIGH_GIC_REDIST2); + } + } + + if (tcg_enabled() || hvf_enabled() || qtest_enabled() || + (vms->gic_version < VIRT_GIC_VERSION_3)) { + max_cpus = machine->smp.max_cpus = smp_cpus; + mc->has_hotpluggable_cpus = false; + if (vms->gic_version >= VIRT_GIC_VERSION_3) { + warn_report("cpu hotplug feature has been disabled"); + } + } + + if (max_cpus > virt_max_cpus) { + error_report("Number of SMP CPUs requested (%d) exceeds max CPUs " + "supported by machine 'mach-virt' (%d)", + max_cpus, virt_max_cpus); + if (vms->gic_version != VIRT_GIC_VERSION_2 && !vms->highmem_redists) { + error_printf("Try 'highmem-redists=on' for more CPUs\n"); + } + + exit(1); + } + + /* uses smp.max_cpus to initialize all possible vCPUs */ + possible_cpus = mc->possible_cpu_arch_ids(machine); + if (vms->secure) { /* * The Secure view of the world is the same as the NonSecure, @@ -2147,31 +2182,6 @@ static void machvirt_init(MachineState *machine) vms->psci_conduit = QEMU_PSCI_CONDUIT_HVC; } - /* - * The maximum number of CPUs depends on the GIC version, or on how - * many redistributors we can fit into the memory map (which in turn - * depends on whether this is a GICv3 or v4). - */ - if (vms->gic_version == VIRT_GIC_VERSION_2) { - virt_max_cpus = GIC_NCPU; - } else { - virt_max_cpus = virt_redist_capacity(vms, VIRT_GIC_REDIST); - if (vms->highmem_redists) { - virt_max_cpus += virt_redist_capacity(vms, VIRT_HIGH_GIC_REDIST2); - } - } - - if (max_cpus > virt_max_cpus) { - error_report("Number of SMP CPUs requested (%d) exceeds max CPUs " - "supported by machine 'mach-virt' (%d)", - max_cpus, virt_max_cpus); - if (vms->gic_version != VIRT_GIC_VERSION_2 && !vms->highmem_redists) { - error_printf("Try 'highmem-redists=on' for more CPUs\n"); - } - - exit(1); - } - if (vms->secure && (kvm_enabled() || hvf_enabled())) { error_report("mach-virt: %s does not support providing " "Security extensions (TrustZone) to the guest CPU", From patchwork Thu Jun 13 23:36:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697617 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 13A85C27C4F for ; Thu, 13 Jun 2024 23:40:00 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu2i-0006QC-4Q; Thu, 13 Jun 2024 19:39:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu2T-0006Cn-Pf; Thu, 13 Jun 2024 19:39:30 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu2R-000341-5i; Thu, 13 Jun 2024 19:39:29 -0400 Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0dzk4lrHz6HJ5h; Fri, 14 Jun 2024 07:34:42 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id F0C211400D9; Fri, 14 Jun 2024 07:39:24 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:39:02 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 04/29] hw/arm/virt: Move setting of common CPU properties in a function Date: Fri, 14 Jun 2024 00:36:14 +0100 Message-ID: <20240613233639.202896-5-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Factor out CPU properties code common for {hot,cold}-plugged CPUs. This allows code reuse. Signed-off-by: Salil Mehta --- hw/arm/virt.c | 261 ++++++++++++++++++++++++++++-------------- include/hw/arm/virt.h | 4 + 2 files changed, 182 insertions(+), 83 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 3e1c4d2d2f..2e0ec7d869 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -1753,6 +1753,46 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) return arm_build_mp_affinity(idx, clustersz); } +static CPUArchId *virt_find_cpu_slot(MachineState *ms, int vcpuid) +{ + VirtMachineState *vms = VIRT_MACHINE(ms); + CPUArchId *found_cpu; + uint64_t mp_affinity; + + assert(vcpuid >= 0 && vcpuid < ms->possible_cpus->len); + + mp_affinity = virt_cpu_mp_affinity(vms, vcpuid); + found_cpu = &ms->possible_cpus->cpus[vcpuid]; + + assert(found_cpu->arch_id == mp_affinity); + + /* + * RFC: Question: + * Slot-id is the index where vCPU with certain arch-id(=mpidr/ap-affinity) + * is plugged. For Host KVM, MPIDR for vCPU is derived using vcpu-id. + * As I understand, MPIDR and vcpu-id are property of vCPU but slot-id is + * more related to machine? Current code assumes slot-id and vcpu-id are + * same i.e. meaning of slot is bit vague. + * + * Q1: Is there any requirement to clearly represent slot and dissociate it + * from vcpu-id? + * Q2: Should we make MPIDR within host KVM user configurable? + * + * +----+----+----+----+----+----+----+----+ + * MPIDR ||| Res | Aff2 | Aff1 | Aff0 | + * +----+----+----+----+----+----+----+----+ + * \ \ \ | | + * \ 8bit \ 8bit \ |4bit| + * \<------->\<------->\ |<-->| + * \ \ \| | + * +----+----+----+----+----+----+----+----+ + * VCPU-ID | Byte4 | Byte2 | Byte1 | Byte0 | + * +----+----+----+----+----+----+----+----+ + */ + + return found_cpu; +} + static inline bool *virt_get_high_memmap_enabled(VirtMachineState *vms, int index) { @@ -2065,16 +2105,129 @@ static void virt_cpu_post_init(VirtMachineState *vms, MemoryRegion *sysmem) } } +static void virt_cpu_set_properties(Object *cpuobj, const CPUArchId *cpu_slot, + Error **errp) +{ + MachineState *ms = MACHINE(qdev_get_machine()); + VirtMachineState *vms = VIRT_MACHINE(ms); + Error *local_err = NULL; + VirtMachineClass *vmc; + + vmc = VIRT_MACHINE_GET_CLASS(ms); + + /* now, set the cpu object property values */ + numa_cpu_pre_plug(cpu_slot, DEVICE(cpuobj), &local_err); + if (local_err) { + goto out; + } + + object_property_set_int(cpuobj, "mp-affinity", cpu_slot->arch_id, NULL); + + if (!vms->secure) { + object_property_set_bool(cpuobj, "has_el3", false, NULL); + } + + if (!vms->virt && object_property_find(cpuobj, "has_el2")) { + object_property_set_bool(cpuobj, "has_el2", false, NULL); + } + + if (vmc->kvm_no_adjvtime && + object_property_find(cpuobj, "kvm-no-adjvtime")) { + object_property_set_bool(cpuobj, "kvm-no-adjvtime", true, NULL); + } + + if (vmc->no_kvm_steal_time && + object_property_find(cpuobj, "kvm-steal-time")) { + object_property_set_bool(cpuobj, "kvm-steal-time", false, NULL); + } + + if (vmc->no_pmu && object_property_find(cpuobj, "pmu")) { + object_property_set_bool(cpuobj, "pmu", false, NULL); + } + + if (vmc->no_tcg_lpa2 && object_property_find(cpuobj, "lpa2")) { + object_property_set_bool(cpuobj, "lpa2", false, NULL); + } + + if (object_property_find(cpuobj, "reset-cbar")) { + object_property_set_int(cpuobj, "reset-cbar", + vms->memmap[VIRT_CPUPERIPHS].base, + &local_err); + if (local_err) { + goto out; + } + } + + /* link already initialized {secure,tag}-memory regions to this cpu */ + object_property_set_link(cpuobj, "memory", OBJECT(vms->sysmem), &local_err); + if (local_err) { + goto out; + } + + if (vms->secure) { + object_property_set_link(cpuobj, "secure-memory", + OBJECT(vms->secure_sysmem), &local_err); + if (local_err) { + goto out; + } + } + + if (vms->mte) { + if (!object_property_find(cpuobj, "tag-memory")) { + error_setg(&local_err, "MTE requested, but not supported " + "by the guest CPU"); + if (local_err) { + goto out; + } + } + + object_property_set_link(cpuobj, "tag-memory", OBJECT(vms->tag_sysmem), + &local_err); + if (local_err) { + goto out; + } + + if (vms->secure) { + object_property_set_link(cpuobj, "secure-tag-memory", + OBJECT(vms->secure_tag_sysmem), + &local_err); + if (local_err) { + goto out; + } + } + } + + /* + * RFC: Question: this must only be called for the hotplugged cpus. For the + * cold booted secondary cpus this is being taken care in arm_load_kernel() + * in boot.c. Perhaps we should remove that code now? + */ + if (vms->psci_conduit != QEMU_PSCI_CONDUIT_DISABLED) { + object_property_set_int(cpuobj, "psci-conduit", vms->psci_conduit, + NULL); + + /* Secondary CPUs start in PSCI powered-down state */ + if (CPU(cpuobj)->cpu_index > 0) { + object_property_set_bool(cpuobj, "start-powered-off", true, NULL); + } + } + +out: + if (local_err) { + error_propagate(errp, local_err); + } +} + static void machvirt_init(MachineState *machine) { VirtMachineState *vms = VIRT_MACHINE(machine); VirtMachineClass *vmc = VIRT_MACHINE_GET_CLASS(machine); MachineClass *mc = MACHINE_GET_CLASS(machine); const CPUArchIdList *possible_cpus; - MemoryRegion *sysmem = get_system_memory(); + MemoryRegion *secure_tag_sysmem = NULL; MemoryRegion *secure_sysmem = NULL; MemoryRegion *tag_sysmem = NULL; - MemoryRegion *secure_tag_sysmem = NULL; + MemoryRegion *sysmem; int n, virt_max_cpus; bool firmware_loaded; bool aarch64 = true; @@ -2148,6 +2301,8 @@ static void machvirt_init(MachineState *machine) /* uses smp.max_cpus to initialize all possible vCPUs */ possible_cpus = mc->possible_cpu_arch_ids(machine); + sysmem = vms->sysmem = get_system_memory(); + if (vms->secure) { /* * The Secure view of the world is the same as the NonSecure, @@ -2155,7 +2310,7 @@ static void machvirt_init(MachineState *machine) * containing the system memory at low priority; any secure-only * devices go in at higher priority and take precedence. */ - secure_sysmem = g_new(MemoryRegion, 1); + secure_sysmem = vms->secure_sysmem = g_new(MemoryRegion, 1); memory_region_init(secure_sysmem, OBJECT(machine), "secure-memory", UINT64_MAX); memory_region_add_subregion_overlap(secure_sysmem, 0, sysmem, -1); @@ -2203,10 +2358,28 @@ static void machvirt_init(MachineState *machine) exit(1); } + if (vms->mte) { + /* Create the memory region only once, but link to all cpus later */ + tag_sysmem = vms->tag_sysmem = g_new(MemoryRegion, 1); + memory_region_init(tag_sysmem, OBJECT(machine), + "tag-memory", UINT64_MAX / 32); + + if (vms->secure) { + secure_tag_sysmem = vms->secure_tag_sysmem = g_new(MemoryRegion, 1); + memory_region_init(secure_tag_sysmem, OBJECT(machine), + "secure-tag-memory", UINT64_MAX / 32); + + /* As with ram, secure-tag takes precedence over tag. */ + memory_region_add_subregion_overlap(secure_tag_sysmem, 0, + tag_sysmem, -1); + } + } + create_fdt(vms); assert(possible_cpus->len == max_cpus); for (n = 0; n < possible_cpus->len; n++) { + CPUArchId *cpu_slot; Object *cpuobj; CPUState *cs; @@ -2215,15 +2388,10 @@ static void machvirt_init(MachineState *machine) } cpuobj = object_new(possible_cpus->cpus[n].type); - object_property_set_int(cpuobj, "mp-affinity", - possible_cpus->cpus[n].arch_id, NULL); cs = CPU(cpuobj); cs->cpu_index = n; - numa_cpu_pre_plug(&possible_cpus->cpus[cs->cpu_index], DEVICE(cpuobj), - &error_fatal); - aarch64 &= object_property_get_bool(cpuobj, "aarch64", NULL); object_property_set_int(cpuobj, "socket-id", virt_get_socket_id(machine, n), NULL); @@ -2234,81 +2402,8 @@ static void machvirt_init(MachineState *machine) object_property_set_int(cpuobj, "thread-id", virt_get_thread_id(machine, n), NULL); - if (!vms->secure) { - object_property_set_bool(cpuobj, "has_el3", false, NULL); - } - - if (!vms->virt && object_property_find(cpuobj, "has_el2")) { - object_property_set_bool(cpuobj, "has_el2", false, NULL); - } - - if (vmc->kvm_no_adjvtime && - object_property_find(cpuobj, "kvm-no-adjvtime")) { - object_property_set_bool(cpuobj, "kvm-no-adjvtime", true, NULL); - } - - if (vmc->no_kvm_steal_time && - object_property_find(cpuobj, "kvm-steal-time")) { - object_property_set_bool(cpuobj, "kvm-steal-time", false, NULL); - } - - if (vmc->no_pmu && object_property_find(cpuobj, "pmu")) { - object_property_set_bool(cpuobj, "pmu", false, NULL); - } - - if (vmc->no_tcg_lpa2 && object_property_find(cpuobj, "lpa2")) { - object_property_set_bool(cpuobj, "lpa2", false, NULL); - } - - if (object_property_find(cpuobj, "reset-cbar")) { - object_property_set_int(cpuobj, "reset-cbar", - vms->memmap[VIRT_CPUPERIPHS].base, - &error_abort); - } - - object_property_set_link(cpuobj, "memory", OBJECT(sysmem), - &error_abort); - if (vms->secure) { - object_property_set_link(cpuobj, "secure-memory", - OBJECT(secure_sysmem), &error_abort); - } - - if (vms->mte) { - /* Create the memory region only once, but link to all cpus. */ - if (!tag_sysmem) { - /* - * The property exists only if MemTag is supported. - * If it is, we must allocate the ram to back that up. - */ - if (!object_property_find(cpuobj, "tag-memory")) { - error_report("MTE requested, but not supported " - "by the guest CPU"); - exit(1); - } - - tag_sysmem = g_new(MemoryRegion, 1); - memory_region_init(tag_sysmem, OBJECT(machine), - "tag-memory", UINT64_MAX / 32); - - if (vms->secure) { - secure_tag_sysmem = g_new(MemoryRegion, 1); - memory_region_init(secure_tag_sysmem, OBJECT(machine), - "secure-tag-memory", UINT64_MAX / 32); - - /* As with ram, secure-tag takes precedence over tag. */ - memory_region_add_subregion_overlap(secure_tag_sysmem, 0, - tag_sysmem, -1); - } - } - - object_property_set_link(cpuobj, "tag-memory", OBJECT(tag_sysmem), - &error_abort); - if (vms->secure) { - object_property_set_link(cpuobj, "secure-tag-memory", - OBJECT(secure_tag_sysmem), - &error_abort); - } - } + cpu_slot = virt_find_cpu_slot(machine, cs->cpu_index); + virt_cpu_set_properties(cpuobj, cpu_slot, &error_fatal); qdev_realize(DEVICE(cpuobj), NULL, &error_fatal); object_unref(cpuobj); diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index 6f9a7bb60b..780bd53ceb 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -139,6 +139,10 @@ struct VirtMachineState { DeviceState *platform_bus_dev; FWCfgState *fw_cfg; PFlashCFI01 *flash[2]; + MemoryRegion *sysmem; + MemoryRegion *secure_sysmem; + MemoryRegion *tag_sysmem; + MemoryRegion *secure_tag_sysmem; bool secure; bool highmem; bool highmem_compact; From patchwork Thu Jun 13 23:36:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697619 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 524DBC27C75 for ; Thu, 13 Jun 2024 23:40:34 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu37-0007Mm-4G; Thu, 13 Jun 2024 19:40:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu2u-0006qz-NV; Thu, 13 Jun 2024 19:39:58 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu2r-00035V-T9; Thu, 13 Jun 2024 19:39:56 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0f402bDPz6H7Gh; Fri, 14 Jun 2024 07:38:24 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id CB992140AE5; Fri, 14 Jun 2024 07:39:47 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:39:25 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 05/29] arm/virt, target/arm: Machine init time change common to vCPU {cold|hot}-plug Date: Fri, 14 Jun 2024 00:36:15 +0100 Message-ID: <20240613233639.202896-6-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Introduce the common logic required during the initialization of both cold and hot-plugged vCPUs. Additionally, initialize the *disabled* state of the vCPUs, which will be used further during the initialization phases of various other components like GIC, PMU, ACPI, etc., as part of the virtual machine initialization. KVM vCPUs corresponding to unplugged or yet-to-be-plugged QOM CPUs are kept in a powered-off state in the KVM Host and do not run the guest code. Plugged vCPUs are also kept in a powered-off state, but vCPU threads exist and remain in a sleeping state. TBD: For cold-booted vCPUs, this change also exists in the `arm_load_kernel()` function in `boot.c`. However, for hot-plugged CPUs, this change should remain part of the pre-plug phase. We are duplicating the powering-off of the cold-booted CPUs. Should we remove the duplicate change from `boot.c`? Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta Reported-by: Gavin Shan [GS: pointed the assertion due to wrong range check] --- hw/arm/virt.c | 94 +++++++++++++++++++++++++++++++++++++++++++++- target/arm/cpu.c | 7 ++++ target/arm/cpu64.c | 14 +++++++ 3 files changed, 114 insertions(+), 1 deletion(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 2e0ec7d869..a285139165 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2817,6 +2817,26 @@ static int64_t virt_get_default_cpu_node_id(const MachineState *ms, int idx) return socket_id % ms->numa_state->num_nodes; } +static int +virt_get_cpu_id_from_cpu_topo(const MachineState *ms, DeviceState *dev) +{ + int cpu_id, sock_vcpu_num, clus_vcpu_num, core_vcpu_num; + ARMCPU *cpu = ARM_CPU(dev); + + /* calculate total logical cpus across socket/cluster/core */ + sock_vcpu_num = cpu->socket_id * (ms->smp.threads * ms->smp.cores * + ms->smp.clusters); + clus_vcpu_num = cpu->cluster_id * (ms->smp.threads * ms->smp.cores); + core_vcpu_num = cpu->core_id * ms->smp.threads; + + /* get vcpu-id(logical cpu index) for this vcpu from this topology */ + cpu_id = (sock_vcpu_num + clus_vcpu_num + core_vcpu_num) + cpu->thread_id; + + assert(cpu_id >= 0 && cpu_id < ms->possible_cpus->len); + + return cpu_id; +} + static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms) { int n; @@ -2899,6 +2919,72 @@ static void virt_memory_plug(HotplugHandler *hotplug_dev, dev, &error_abort); } +static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, + Error **errp) +{ + MachineState *ms = MACHINE(hotplug_dev); + ARMCPU *cpu = ARM_CPU(dev); + CPUState *cs = CPU(dev); + CPUArchId *cpu_slot; + + /* sanity check the cpu */ + if (!object_dynamic_cast(OBJECT(cpu), ms->cpu_type)) { + error_setg(errp, "Invalid CPU type, expected cpu type: '%s'", + ms->cpu_type); + return; + } + + if ((cpu->thread_id < 0) || (cpu->thread_id >= ms->smp.threads)) { + error_setg(errp, "Invalid thread-id %u specified, correct range 0:%u", + cpu->thread_id, ms->smp.threads - 1); + return; + } + + if ((cpu->core_id < 0) || (cpu->core_id >= ms->smp.cores)) { + error_setg(errp, "Invalid core-id %d specified, correct range 0:%u", + cpu->core_id, ms->smp.cores - 1); + return; + } + + if ((cpu->cluster_id < 0) || (cpu->cluster_id >= ms->smp.clusters)) { + error_setg(errp, "Invalid cluster-id %u specified, correct range 0:%u", + cpu->cluster_id, ms->smp.clusters - 1); + return; + } + + if ((cpu->socket_id < 0) || (cpu->socket_id >= ms->smp.sockets)) { + error_setg(errp, "Invalid socket-id %u specified, correct range 0:%u", + cpu->socket_id, ms->smp.sockets - 1); + return; + } + + cs->cpu_index = virt_get_cpu_id_from_cpu_topo(ms, dev); + + cpu_slot = virt_find_cpu_slot(ms, cs->cpu_index); + if (qemu_present_cpu(CPU(cpu_slot->cpu))) { + error_setg(errp, "cpu(id%d=%d:%d:%d:%d) with arch-id %" PRIu64 " exist", + cs->cpu_index, cpu->socket_id, cpu->cluster_id, cpu->core_id, + cpu->thread_id, cpu_slot->arch_id); + return; + } + virt_cpu_set_properties(OBJECT(cs), cpu_slot, errp); +} + +static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, + Error **errp) +{ + MachineState *ms = MACHINE(hotplug_dev); + CPUState *cs = CPU(dev); + CPUArchId *cpu_slot; + + /* insert the cold/hot-plugged vcpu in the slot */ + cpu_slot = virt_find_cpu_slot(ms, cs->cpu_index); + cpu_slot->cpu = CPU(dev); + + cs->disabled = false; + return; +} + static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { @@ -2906,6 +2992,8 @@ static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev, if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) { virt_memory_pre_plug(hotplug_dev, dev, errp); + } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) { + virt_cpu_pre_plug(hotplug_dev, dev, errp); } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MD_PCI)) { virtio_md_pci_pre_plug(VIRTIO_MD_PCI(dev), MACHINE(hotplug_dev), errp); } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) { @@ -2962,6 +3050,8 @@ static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev, virt_memory_plug(hotplug_dev, dev, errp); } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MD_PCI)) { virtio_md_pci_plug(VIRTIO_MD_PCI(dev), MACHINE(hotplug_dev), errp); + } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) { + virt_cpu_plug(hotplug_dev, dev, errp); } if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) { @@ -3046,7 +3136,8 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine, if (device_is_dynamic_sysbus(mc, dev) || object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM) || object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MD_PCI) || - object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) { + object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI) || + object_dynamic_cast(OBJECT(dev), TYPE_CPU)) { return HOTPLUG_HANDLER(machine); } return NULL; @@ -3150,6 +3241,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) mc->valid_cpu_types = valid_cpu_types; mc->get_default_cpu_node_id = virt_get_default_cpu_node_id; mc->kvm_type = virt_kvm_type; + mc->has_hotpluggable_cpus = true; assert(!mc->get_hotplug_handler); mc->get_hotplug_handler = virt_machine_get_hotplug_handler; hc->pre_plug = virt_machine_device_pre_plug_cb; diff --git a/target/arm/cpu.c b/target/arm/cpu.c index abc4ed0842..c92162fa97 100644 --- a/target/arm/cpu.c +++ b/target/arm/cpu.c @@ -2639,6 +2639,12 @@ static const TCGCPUOps arm_tcg_ops = { }; #endif /* CONFIG_TCG */ +static int64_t arm_cpu_get_arch_id(CPUState *cs) +{ + ARMCPU *cpu = ARM_CPU(cs); + return cpu->mp_affinity; +} + static void arm_cpu_class_init(ObjectClass *oc, void *data) { ARMCPUClass *acc = ARM_CPU_CLASS(oc); @@ -2658,6 +2664,7 @@ static void arm_cpu_class_init(ObjectClass *oc, void *data) cc->has_work = arm_cpu_has_work; cc->mmu_index = arm_cpu_mmu_index; cc->dump_state = arm_cpu_dump_state; + cc->get_arch_id = arm_cpu_get_arch_id; cc->set_pc = arm_cpu_set_pc; cc->get_pc = arm_cpu_get_pc; cc->gdb_read_register = arm_cpu_gdb_read_register; diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c index c15d086049..d6b48b3424 100644 --- a/target/arm/cpu64.c +++ b/target/arm/cpu64.c @@ -780,6 +780,17 @@ static void aarch64_cpu_set_aarch64(Object *obj, bool value, Error **errp) } } +static void aarch64_cpu_initfn(Object *obj) +{ + CPUState *cs = CPU(obj); + + /* + * we start every ARM64 vcpu as disabled possible vCPU. It needs to be + * enabled explicitly + */ + cs->disabled = true; +} + static void aarch64_cpu_finalizefn(Object *obj) { } @@ -792,7 +803,9 @@ static const gchar *aarch64_gdb_arch_name(CPUState *cs) static void aarch64_cpu_class_init(ObjectClass *oc, void *data) { CPUClass *cc = CPU_CLASS(oc); + DeviceClass *dc = DEVICE_CLASS(oc); + dc->user_creatable = true; cc->gdb_read_register = aarch64_cpu_gdb_read_register; cc->gdb_write_register = aarch64_cpu_gdb_write_register; cc->gdb_core_xml_file = "aarch64-core.xml"; @@ -837,6 +850,7 @@ void aarch64_cpu_register(const ARMCPUInfo *info) static const TypeInfo aarch64_cpu_type_info = { .name = TYPE_AARCH64_CPU, .parent = TYPE_ARM_CPU, + .instance_init = aarch64_cpu_initfn, .instance_finalize = aarch64_cpu_finalizefn, .abstract = true, .class_init = aarch64_cpu_class_init, From patchwork Thu Jun 13 23:36:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697620 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EF971C27C75 for ; Thu, 13 Jun 2024 23:40:44 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu3Y-0008A6-DV; Thu, 13 Jun 2024 19:40:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu3I-0007kb-CZ; Thu, 13 Jun 2024 19:40:22 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu3D-0003KP-NT; Thu, 13 Jun 2024 19:40:17 -0400 Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0f0c0zWJz6HJTY; Fri, 14 Jun 2024 07:35:28 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 6C9A8140594; Fri, 14 Jun 2024 07:40:10 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:39:48 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 06/29] arm/virt, kvm: Pre-create disabled possible vCPUs @machine init Date: Fri, 14 Jun 2024 00:36:16 +0100 Message-ID: <20240613233639.202896-7-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org In the ARMv8 architecture, the GIC must know all the CPUs it is connected to during its initialization, and this cannot change afterward. This must be ensured during the initialization of the VGIC as well in KVM, which requires all vCPUs to be created and present during its initialization. This is necessary because: 1. The association between GICC and MPIDR must be fixed at VM initialization time. This is represented by the register `GIC_TYPER(mp_affinity, proc_num)`. 2. GICC (CPU interfaces), GICR (redistributors), etc., must all be initialized at boot time. 3. Memory regions associated with GICR, etc., cannot be changed (added, deleted, or modified) after the VM has been initialized. This patch adds support to pre-create all possible vCPUs within the host using the KVM interface as part of the virtual machine initialization. These vCPUs can later be attached to QOM/ACPI when they are actually hot-plugged and made present. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta Reported-by: Vishnu Pajjuri [VP: Identified CPU stall issue & suggested probable fix] --- hw/arm/virt.c | 56 +++++++++++++++++++++++++++++++++++-------- include/hw/core/cpu.h | 1 + target/arm/cpu64.c | 1 + target/arm/kvm.c | 41 ++++++++++++++++++++++++++++++- target/arm/kvm_arm.h | 11 +++++++++ 5 files changed, 99 insertions(+), 11 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index a285139165..81e7a27786 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2383,14 +2383,8 @@ static void machvirt_init(MachineState *machine) Object *cpuobj; CPUState *cs; - if (n >= smp_cpus) { - break; - } - cpuobj = object_new(possible_cpus->cpus[n].type); - cs = CPU(cpuobj); - cs->cpu_index = n; aarch64 &= object_property_get_bool(cpuobj, "aarch64", NULL); object_property_set_int(cpuobj, "socket-id", @@ -2402,11 +2396,53 @@ static void machvirt_init(MachineState *machine) object_property_set_int(cpuobj, "thread-id", virt_get_thread_id(machine, n), NULL); - cpu_slot = virt_find_cpu_slot(machine, cs->cpu_index); - virt_cpu_set_properties(cpuobj, cpu_slot, &error_fatal); + if (n < smp_cpus) { + qdev_realize(DEVICE(cpuobj), NULL, &error_fatal); + object_unref(cpuobj); + } else { + /* handling for vcpus which are yet to be hot-plugged */ + cs->cpu_index = n; + cpu_slot = virt_find_cpu_slot(machine, cs->cpu_index); - qdev_realize(DEVICE(cpuobj), NULL, &error_fatal); - object_unref(cpuobj); + /* + * ARM host vCPU features need to be fixed at the boot time. But as + * per current approach this CPU object will be destroyed during + * cpu_post_init(). During hotplug of vCPUs these properties are + * initialized again. + */ + virt_cpu_set_properties(cpuobj, cpu_slot, &error_fatal); + + /* + * For KVM, we shall be pre-creating the now disabled/un-plugged + * possbile host vcpus and park them till the time they are + * actually hot plugged. This is required to pre-size the host + * GICC and GICR with the all possible vcpus for this VM. + */ + if (kvm_enabled()) { + kvm_arm_create_host_vcpu(ARM_CPU(cs)); + } + /* + * Add disabled vCPU to CPU slot during the init phase of the virt + * machine + * 1. We need this ARMCPU object during the GIC init. This object + * will facilitate in pre-realizing the GIC. Any info like + * mp-affinity(required to derive gicr_type) etc. could still be + * fetched while preserving QOM abstraction akin to realized + * vCPUs. + * 2. Now, after initialization of the virt machine is complete we + * could use two approaches to deal with this ARMCPU object: + * (i) re-use this ARMCPU object during hotplug of this vCPU. + * OR + * (ii) defer release this ARMCPU object after gic has been + * initialized or during pre-plug phase when a vCPU is + * hotplugged. + * + * We will use the (ii) approach and release the ARMCPU objects + * after GIC and machine has been fully initialized during + * machine_init_done() phase. + */ + cpu_slot->cpu = cs; + } } /* Now we've created the CPUs we can see if they have the hypvirt timer */ diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index 60b4778da9..62e68611c0 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -520,6 +520,7 @@ struct CPUState { uint64_t dirty_pages; int kvm_vcpu_stats_fd; bool vcpu_dirty; + VMChangeStateEntry *vmcse; /* Use by accel-block: CPU is executing an ioctl() */ QemuLockCnt in_ioctl_lock; diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c index d6b48b3424..9b7e8b032c 100644 --- a/target/arm/cpu64.c +++ b/target/arm/cpu64.c @@ -789,6 +789,7 @@ static void aarch64_cpu_initfn(Object *obj) * enabled explicitly */ cs->disabled = true; + cs->thread_id = 0; } static void aarch64_cpu_finalizefn(Object *obj) diff --git a/target/arm/kvm.c b/target/arm/kvm.c index 7cf5cf31de..01c83c1994 100644 --- a/target/arm/kvm.c +++ b/target/arm/kvm.c @@ -1003,6 +1003,38 @@ void kvm_arm_reset_vcpu(ARMCPU *cpu) write_list_to_cpustate(cpu); } +void kvm_arm_create_host_vcpu(ARMCPU *cpu) +{ + CPUState *cs = CPU(cpu); + unsigned long vcpu_id = cs->cpu_index; + int ret; + + ret = kvm_create_vcpu(cs); + if (ret < 0) { + error_report("Failed to create host vcpu %ld", vcpu_id); + abort(); + } + + /* + * Initialize the vCPU in the host. This will reset the sys regs + * for this vCPU and related registers like MPIDR_EL1 etc. also + * gets programmed during this call to host. These are referred + * later while setting device attributes of the GICR during GICv3 + * reset + */ + ret = kvm_arch_init_vcpu(cs); + if (ret < 0) { + error_report("Failed to initialize host vcpu %ld", vcpu_id); + abort(); + } + + /* + * park the created vCPU. shall be used during kvm_get_vcpu() when + * threads are created during realization of ARM vCPUs. + */ + kvm_park_vcpu(cs); +} + /* * Update KVM's MP_STATE based on what QEMU thinks it is */ @@ -1874,7 +1906,14 @@ int kvm_arch_init_vcpu(CPUState *cs) return -EINVAL; } - qemu_add_vm_change_state_handler(kvm_arm_vm_state_change, cpu); + /* + * Install VM change handler only when vCPU thread has been spawned + * i.e. vCPU is being realized + */ + if (cs->thread_id) { + cs->vmcse = qemu_add_vm_change_state_handler(kvm_arm_vm_state_change, + cpu); + } /* Determine init features for this CPU */ memset(cpu->kvm_init_features, 0, sizeof(cpu->kvm_init_features)); diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h index cfaa0d9bc7..0be7e896d2 100644 --- a/target/arm/kvm_arm.h +++ b/target/arm/kvm_arm.h @@ -96,6 +96,17 @@ void kvm_arm_cpu_post_load(ARMCPU *cpu); */ void kvm_arm_reset_vcpu(ARMCPU *cpu); +/** + * kvm_arm_create_host_vcpu: + * @cpu: ARMCPU + * + * Called at to pre create all possible kvm vCPUs within the the host at the + * virt machine init time. This will also init this pre-created vCPU and + * hence result in vCPU reset at host. These pre created and inited vCPUs + * shall be parked for use when ARM vCPUs are actually realized. + */ +void kvm_arm_create_host_vcpu(ARMCPU *cpu); + #ifdef CONFIG_KVM /** * kvm_arm_create_scratch_host_vcpu: From patchwork Thu Jun 13 23:36:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697621 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 25535C27C4F for ; Thu, 13 Jun 2024 23:40:50 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu3i-0000pt-7v; Thu, 13 Jun 2024 19:40:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu3f-0000Z2-6x; Thu, 13 Jun 2024 19:40:43 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu3c-0003SG-8u; Thu, 13 Jun 2024 19:40:42 -0400 Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0f1524dLz6HJTP; Fri, 14 Jun 2024 07:35:53 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 95E051400D9; Fri, 14 Jun 2024 07:40:35 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:40:10 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 07/29] arm/virt, gicv3: Changes to pre-size GIC with possible vcpus @machine init Date: Fri, 14 Jun 2024 00:36:17 +0100 Message-ID: <20240613233639.202896-8-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The GIC needs to be pre-sized with possible vCPUs at initialization time. This is necessary because memory regions and resources associated with GICC/GICR, etc., cannot be changed (added, deleted, or modified) after the VM has been initialized. Additionally, `GIC_TYPER` needs to be initialized with `mp_affinity` and CPU interface number association, which cannot be changed after the GIC has been initialized. Once all the CPU interfaces of the GIC have been initialized, it must be ensured that any updates to the GICC during reset only take place for the *enabled* vCPUs and not the disabled ones. Therefore, proper checks are required at various places. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta Signed-off-by: Jean-Philippe Brucker [changed the comment in arm_gicv3_icc_reset] --- hw/arm/virt.c | 15 ++++++++------- hw/intc/arm_gicv3_common.c | 7 +++++-- hw/intc/arm_gicv3_cpuif.c | 8 ++++++++ hw/intc/arm_gicv3_kvm.c | 34 +++++++++++++++++++++++++++++++--- include/hw/arm/virt.h | 2 +- 5 files changed, 53 insertions(+), 13 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 81e7a27786..ac53bfadca 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -751,6 +751,7 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem) const char *gictype; int i; unsigned int smp_cpus = ms->smp.cpus; + unsigned int max_cpus = ms->smp.max_cpus; uint32_t nb_redist_regions = 0; int revision; @@ -775,7 +776,7 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem) } vms->gic = qdev_new(gictype); qdev_prop_set_uint32(vms->gic, "revision", revision); - qdev_prop_set_uint32(vms->gic, "num-cpu", smp_cpus); + qdev_prop_set_uint32(vms->gic, "num-cpu", max_cpus); /* Note that the num-irq property counts both internal and external * interrupts; there are always 32 of the former (mandated by GIC spec). */ @@ -787,7 +788,7 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem) if (vms->gic_version != VIRT_GIC_VERSION_2) { QList *redist_region_count; uint32_t redist0_capacity = virt_redist_capacity(vms, VIRT_GIC_REDIST); - uint32_t redist0_count = MIN(smp_cpus, redist0_capacity); + uint32_t redist0_count = MIN(max_cpus, redist0_capacity); nb_redist_regions = virt_gicv3_redist_region_count(vms); @@ -798,7 +799,7 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem) virt_redist_capacity(vms, VIRT_HIGH_GIC_REDIST2); qlist_append_int(redist_region_count, - MIN(smp_cpus - redist0_count, redist1_capacity)); + MIN(max_cpus - redist0_count, redist1_capacity)); } qdev_prop_set_array(vms->gic, "redist-region-count", redist_region_count); @@ -871,7 +872,7 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem) } else if (vms->virt) { qemu_irq irq = qdev_get_gpio_in(vms->gic, intidbase + ARCH_GIC_MAINT_IRQ); - sysbus_connect_irq(gicbusdev, i + 4 * smp_cpus, irq); + sysbus_connect_irq(gicbusdev, i + 4 * max_cpus, irq); } qdev_connect_gpio_out_named(cpudev, "pmu-interrupt", 0, @@ -879,11 +880,11 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem) + VIRTUAL_PMU_IRQ)); sysbus_connect_irq(gicbusdev, i, qdev_get_gpio_in(cpudev, ARM_CPU_IRQ)); - sysbus_connect_irq(gicbusdev, i + smp_cpus, + sysbus_connect_irq(gicbusdev, i + max_cpus, qdev_get_gpio_in(cpudev, ARM_CPU_FIQ)); - sysbus_connect_irq(gicbusdev, i + 2 * smp_cpus, + sysbus_connect_irq(gicbusdev, i + 2 * max_cpus, qdev_get_gpio_in(cpudev, ARM_CPU_VIRQ)); - sysbus_connect_irq(gicbusdev, i + 3 * smp_cpus, + sysbus_connect_irq(gicbusdev, i + 3 * max_cpus, qdev_get_gpio_in(cpudev, ARM_CPU_VFIQ)); if (vms->gic_version != VIRT_GIC_VERSION_2) { diff --git a/hw/intc/arm_gicv3_common.c b/hw/intc/arm_gicv3_common.c index bd50a1b079..183d2de7eb 100644 --- a/hw/intc/arm_gicv3_common.c +++ b/hw/intc/arm_gicv3_common.c @@ -436,10 +436,13 @@ static void arm_gicv3_common_realize(DeviceState *dev, Error **errp) s->cpu = g_new0(GICv3CPUState, s->num_cpu); for (i = 0; i < s->num_cpu; i++) { - CPUState *cpu = qemu_get_cpu(i); + CPUState *cpu = qemu_get_possible_cpu(i); uint64_t cpu_affid; - s->cpu[i].cpu = cpu; + if (qemu_enabled_cpu(cpu)) { + s->cpu[i].cpu = cpu; + } + s->cpu[i].gic = s; /* Store GICv3CPUState in CPUARMState gicv3state pointer */ gicv3_set_gicv3state(cpu, &s->cpu[i]); diff --git a/hw/intc/arm_gicv3_cpuif.c b/hw/intc/arm_gicv3_cpuif.c index bdb13b00e9..2a8aff0b99 100644 --- a/hw/intc/arm_gicv3_cpuif.c +++ b/hw/intc/arm_gicv3_cpuif.c @@ -1052,6 +1052,10 @@ void gicv3_cpuif_update(GICv3CPUState *cs) ARMCPU *cpu = ARM_CPU(cs->cpu); CPUARMState *env = &cpu->env; + if (!qemu_enabled_cpu(cs->cpu)) { + return; + } + g_assert(bql_locked()); trace_gicv3_cpuif_update(gicv3_redist_affid(cs), cs->hppi.irq, @@ -2036,6 +2040,10 @@ static void icc_generate_sgi(CPUARMState *env, GICv3CPUState *cs, for (i = 0; i < s->num_cpu; i++) { GICv3CPUState *ocs = &s->cpu[i]; + if (!qemu_enabled_cpu(ocs->cpu)) { + continue; + } + if (irm) { /* IRM == 1 : route to all CPUs except self */ if (cs == ocs) { diff --git a/hw/intc/arm_gicv3_kvm.c b/hw/intc/arm_gicv3_kvm.c index 9ea6b8e218..8dbbd79e1b 100644 --- a/hw/intc/arm_gicv3_kvm.c +++ b/hw/intc/arm_gicv3_kvm.c @@ -24,6 +24,7 @@ #include "hw/intc/arm_gicv3_common.h" #include "qemu/error-report.h" #include "qemu/module.h" +#include "sysemu/cpus.h" #include "sysemu/kvm.h" #include "sysemu/runstate.h" #include "kvm_arm.h" @@ -458,6 +459,18 @@ static void kvm_arm_gicv3_put(GICv3State *s) GICv3CPUState *c = &s->cpu[ncpu]; int num_pri_bits; + /* + * To support hotplug of vcpus we need to make sure all gic cpuif/GICC + * are initialized at machvirt init time. Once the init is done we + * release the ARMCPU object for disabled vcpus but this leg could hit + * during reset of GICC later as well i.e. after init has happened and + * all of the cases we want to make sure we dont acess the GICC for + * the disabled VCPUs. + */ + if (!qemu_enabled_cpu(c->cpu)) { + continue; + } + kvm_gicc_access(s, ICC_SRE_EL1, ncpu, &c->icc_sre_el1, true); kvm_gicc_access(s, ICC_CTLR_EL1, ncpu, &c->icc_ctlr_el1[GICV3_NS], true); @@ -616,6 +629,11 @@ static void kvm_arm_gicv3_get(GICv3State *s) GICv3CPUState *c = &s->cpu[ncpu]; int num_pri_bits; + /* don't access GICC for the disabled vCPUs. */ + if (!qemu_enabled_cpu(c->cpu)) { + continue; + } + kvm_gicc_access(s, ICC_SRE_EL1, ncpu, &c->icc_sre_el1, false); kvm_gicc_access(s, ICC_CTLR_EL1, ncpu, &c->icc_ctlr_el1[GICV3_NS], false); @@ -695,10 +713,19 @@ static void arm_gicv3_icc_reset(CPUARMState *env, const ARMCPRegInfo *ri) return; } + /* + * This shall be called even when vcpu is being hotplugged or onlined and + * other vcpus might be running. Host kernel KVM code to handle device + * access of IOCTLs KVM_{GET|SET}_DEVICE_ATTR might fail due to inability to + * grab vcpu locks for all the vcpus. Hence, we need to pause all vcpus to + * facilitate locking within host. + */ + pause_all_vcpus(); /* Initialize to actual HW supported configuration */ kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS, KVM_VGIC_ATTR(ICC_CTLR_EL1, c->gicr_typer), &c->icc_ctlr_el1[GICV3_NS], false, &error_abort); + resume_all_vcpus(); c->icc_ctlr_el1[GICV3_S] = c->icc_ctlr_el1[GICV3_NS]; } @@ -813,9 +840,10 @@ static void kvm_arm_gicv3_realize(DeviceState *dev, Error **errp) gicv3_init_irqs_and_mmio(s, kvm_arm_gicv3_set_irq, NULL); for (i = 0; i < s->num_cpu; i++) { - ARMCPU *cpu = ARM_CPU(qemu_get_cpu(i)); - - define_arm_cp_regs(cpu, gicv3_cpuif_reginfo); + CPUState *cs = qemu_get_cpu(i); + if (qemu_enabled_cpu(cs)) { + define_arm_cp_regs(ARM_CPU(cs), gicv3_cpuif_reginfo); + } } /* Try to create the device via the device control API */ diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index 780bd53ceb..36ac5ff4a2 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -209,7 +209,7 @@ static inline int virt_gicv3_redist_region_count(VirtMachineState *vms) assert(vms->gic_version != VIRT_GIC_VERSION_2); - return (MACHINE(vms)->smp.cpus > redist0_capacity && + return (MACHINE(vms)->smp.max_cpus > redist0_capacity && vms->highmem_redists) ? 2 : 1; } From patchwork Thu Jun 13 23:36:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697622 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2CFFAC27C4F for ; Thu, 13 Jun 2024 23:41:17 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu40-0002SR-Lv; Thu, 13 Jun 2024 19:41:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu3w-0002Ho-8k; Thu, 13 Jun 2024 19:41:00 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu3u-0003al-AJ; Thu, 13 Jun 2024 19:40:59 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0f6s05Cnz687SH; Fri, 14 Jun 2024 07:40:53 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id D4665140AE5; Fri, 14 Jun 2024 07:40:55 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:40:33 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 08/29] arm/virt: Init PMU at host for all possible vcpus Date: Fri, 14 Jun 2024 00:36:18 +0100 Message-ID: <20240613233639.202896-9-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org PMU for all possible vCPUs must be initialized at the VM initialization time. Refactor existing code to accomodate possible vCPUs. This also assumes that all processor being used are identical. Past discussion for reference: Link: https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg00131.html Co-developed-by: Salil Mehta Signed-off-by: Salil Mehta Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 12 ++++++++---- include/hw/arm/virt.h | 1 + 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index ac53bfadca..57ec429022 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2045,12 +2045,14 @@ static void finalize_gic_version(VirtMachineState *vms) */ static void virt_cpu_post_init(VirtMachineState *vms, MemoryRegion *sysmem) { + CPUArchIdList *possible_cpus = vms->parent.possible_cpus; int max_cpus = MACHINE(vms)->smp.max_cpus; - bool aarch64, pmu, steal_time; + bool aarch64, steal_time; CPUState *cpu; + int n; aarch64 = object_property_get_bool(OBJECT(first_cpu), "aarch64", NULL); - pmu = object_property_get_bool(OBJECT(first_cpu), "pmu", NULL); + vms->pmu = object_property_get_bool(OBJECT(first_cpu), "pmu", NULL); steal_time = object_property_get_bool(OBJECT(first_cpu), "kvm-steal-time", NULL); @@ -2077,8 +2079,10 @@ static void virt_cpu_post_init(VirtMachineState *vms, MemoryRegion *sysmem) memory_region_add_subregion(sysmem, pvtime_reg_base, pvtime); } - CPU_FOREACH(cpu) { - if (pmu) { + for (n = 0; n < possible_cpus->len; n++) { + cpu = qemu_get_possible_cpu(n); + + if (vms->pmu) { assert(arm_feature(&ARM_CPU(cpu)->env, ARM_FEATURE_PMU)); if (kvm_irqchip_in_kernel()) { kvm_arm_pmu_set_irq(ARM_CPU(cpu), VIRTUAL_PMU_IRQ); diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index 36ac5ff4a2..d8dcc89a0d 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -155,6 +155,7 @@ struct VirtMachineState { bool ras; bool mte; bool dtb_randomness; + bool pmu; OnOffAuto acpi; VirtGICType gic_version; VirtIOMMUType iommu; From patchwork Thu Jun 13 23:36:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697630 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E11F3C27C4F for ; Thu, 13 Jun 2024 23:41:35 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu4N-0002xF-8O; Thu, 13 Jun 2024 19:41:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu4K-0002ld-Vj; Thu, 13 Jun 2024 19:41:25 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu4H-0003gv-Uw; Thu, 13 Jun 2024 19:41:23 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0f1w2Fbvz6H8xV; Fri, 14 Jun 2024 07:36:36 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 9B92414065C; Fri, 14 Jun 2024 07:41:18 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:40:56 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 09/29] arm/acpi: Enable ACPI support for vcpu hotplug Date: Fri, 14 Jun 2024 00:36:19 +0100 Message-ID: <20240613233639.202896-10-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org ACPI is required to interface QEMU with the guest. Roughly falls into below cases, 1. Convey the possible vcpus config at the machine init time to the guest using various DSDT tables like MADT etc. 2. Convey vcpu hotplug events to guest(using GED) 3. Assist in evaluation of various ACPI methods(like _EVT, _STA, _OST, _EJ0, _MAT etc.) 4. Provides ACPI cpu hotplug state and 12 Byte memory mapped cpu hotplug control register interface to the OSPM/guest corresponding to each possible vcpu. The register interface consists of various R/W fields and their handling operations. These are called when ever register fields or memory regions are accessed(i.e. read or written) by OSPM when ever it evaluates various ACPI methods. Note: lot of this framework code is inherited from the changes already done for x86 but still some minor changes are required to make it compatible with ARM64.) This patch enables the ACPI support for virtual cpu hotplug. ACPI changes required will follow in subsequent patches. Co-developed-by: Salil Mehta Signed-off-by: Salil Mehta Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/hw/arm/Kconfig b/hw/arm/Kconfig index 8b97683a45..e94f19a26a 100644 --- a/hw/arm/Kconfig +++ b/hw/arm/Kconfig @@ -33,6 +33,7 @@ config ARM_VIRT select ACPI_HW_REDUCED select ACPI_APEI select ACPI_VIOT + select ACPI_CPU_HOTPLUG select VIRTIO_MEM_SUPPORTED select ACPI_CXL select ACPI_HMAT From patchwork Thu Jun 13 23:36:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697631 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5BC13C27C4F for ; Thu, 13 Jun 2024 23:42:06 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu4j-00044x-6Y; Thu, 13 Jun 2024 19:41:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu4g-00044M-Sw; Thu, 13 Jun 2024 19:41:46 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu4e-0003mA-6O; Thu, 13 Jun 2024 19:41:46 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0f7k2xc6z6K69v; Fri, 14 Jun 2024 07:41:38 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 44AD114065C; Fri, 14 Jun 2024 07:41:41 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:41:18 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 10/29] arm/virt: Add cpu hotplug events to GED during creation Date: Fri, 14 Jun 2024 00:36:20 +0100 Message-ID: <20240613233639.202896-11-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add the CPU Hotplug event to the set of supported GED events during the creation of the GED device at VM initialization. Additionally, initialize the memory map for the CPU Hotplug control device, which is used for event exchanges between QEMU/VMM and the guest. Signed-off-by: Salil Mehta Reviewed-by: Gavin Shan --- hw/arm/virt.c | 5 ++++- include/hw/arm/virt.h | 1 + 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 57ec429022..918bcb9a1b 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -80,6 +80,7 @@ #include "hw/mem/pc-dimm.h" #include "hw/mem/nvdimm.h" #include "hw/acpi/generic_event_device.h" +#include "hw/acpi/cpu_hotplug.h" #include "hw/virtio/virtio-md-pci.h" #include "hw/virtio/virtio-iommu.h" #include "hw/char/pl011.h" @@ -176,6 +177,7 @@ static const MemMapEntry base_memmap[] = { [VIRT_NVDIMM_ACPI] = { 0x09090000, NVDIMM_ACPI_IO_LEN}, [VIRT_PVTIME] = { 0x090a0000, 0x00010000 }, [VIRT_SECURE_GPIO] = { 0x090b0000, 0x00001000 }, + [VIRT_CPUHP_ACPI] = { 0x090c0000, ACPI_CPU_HOTPLUG_REG_LEN}, [VIRT_MMIO] = { 0x0a000000, 0x00000200 }, /* ...repeating for a total of NUM_VIRTIO_TRANSPORTS, each of that size */ [VIRT_PLATFORM_BUS] = { 0x0c000000, 0x02000000 }, @@ -660,7 +662,7 @@ static inline DeviceState *create_acpi_ged(VirtMachineState *vms) DeviceState *dev; MachineState *ms = MACHINE(vms); int irq = vms->irqmap[VIRT_ACPI_GED]; - uint32_t event = ACPI_GED_PWR_DOWN_EVT; + uint32_t event = ACPI_GED_PWR_DOWN_EVT | ACPI_GED_CPU_HOTPLUG_EVT; if (ms->ram_slots) { event |= ACPI_GED_MEM_HOTPLUG_EVT; @@ -676,6 +678,7 @@ static inline DeviceState *create_acpi_ged(VirtMachineState *vms) sysbus_mmio_map(SYS_BUS_DEVICE(dev), 0, vms->memmap[VIRT_ACPI_GED].base); sysbus_mmio_map(SYS_BUS_DEVICE(dev), 1, vms->memmap[VIRT_PCDIMM_ACPI].base); + sysbus_mmio_map(SYS_BUS_DEVICE(dev), 3, vms->memmap[VIRT_CPUHP_ACPI].base); sysbus_connect_irq(SYS_BUS_DEVICE(dev), 0, qdev_get_gpio_in(vms->gic, irq)); return dev; diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index d8dcc89a0d..d711cab46d 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -75,6 +75,7 @@ enum { VIRT_PCDIMM_ACPI, VIRT_ACPI_GED, VIRT_NVDIMM_ACPI, + VIRT_CPUHP_ACPI, VIRT_PVTIME, VIRT_LOWMEMMAP_LAST, }; From patchwork Thu Jun 13 23:36:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697632 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 69122C27C4F for ; Thu, 13 Jun 2024 23:42:38 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu53-0004kZ-8X; Thu, 13 Jun 2024 19:42:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu51-0004k2-Q8; Thu, 13 Jun 2024 19:42:07 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu50-0003tm-7z; Thu, 13 Jun 2024 19:42:07 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0f6c2x72z67MmR; Fri, 14 Jun 2024 07:40:40 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id D6E7314065C; Fri, 14 Jun 2024 07:42:03 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:41:41 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 11/29] arm/virt: Create GED dev before *disabled* CPU Objs are destroyed Date: Fri, 14 Jun 2024 00:36:21 +0100 Message-ID: <20240613233639.202896-12-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org ACPI CPU hotplug state (is_present=_STA.PRESENT, is_enabled=_STA.ENABLED) for all the possible vCPUs MUST be initialized during machine init. This is done during the creation of the GED device. VMM/Qemu MUST expose/fake the ACPI state of the disabled vCPUs to the Guest kernel as 'present' (_STA.PRESENT) always i.e. ACPI persistent. if the 'disabled' vCPU objectes are destroyed before the GED device has been created then their ACPI hotplug state might not get initialized correctly as acpi_persistent flag is part of the CPUState. This will expose wrong status of the unplugged vCPUs to the Guest kernel. Hence, moving the GED device creation before disabled vCPU objects get destroyed as part of the post CPU init routine. Signed-off-by: Salil Mehta --- hw/arm/virt.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 918bcb9a1b..5f98162587 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2467,6 +2467,12 @@ static void machvirt_init(MachineState *machine) create_gic(vms, sysmem); + has_ged = has_ged && aarch64 && firmware_loaded && + virt_is_acpi_enabled(vms); + if (has_ged) { + vms->acpi_dev = create_acpi_ged(vms); + } + virt_cpu_post_init(vms, sysmem); fdt_add_pmu_nodes(vms); @@ -2489,9 +2495,7 @@ static void machvirt_init(MachineState *machine) create_pcie(vms); - if (has_ged && aarch64 && firmware_loaded && virt_is_acpi_enabled(vms)) { - vms->acpi_dev = create_acpi_ged(vms); - } else { + if (!has_ged) { create_gpio_devices(vms, VIRT_GPIO, sysmem); } From patchwork Thu Jun 13 23:36:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697633 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8B94AC27C4F for ; Thu, 13 Jun 2024 23:42:42 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu5W-0005K3-Q8; Thu, 13 Jun 2024 19:42:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu5P-00059d-05; Thu, 13 Jun 2024 19:42:32 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu5N-0003xX-2i; Thu, 13 Jun 2024 19:42:30 -0400 Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0f72734Tz6H7LJ; Fri, 14 Jun 2024 07:41:02 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 687561400D9; Fri, 14 Jun 2024 07:42:26 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:42:04 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 12/29] arm/virt/acpi: Build CPUs AML with CPU Hotplug support Date: Fri, 14 Jun 2024 00:36:22 +0100 Message-ID: <20240613233639.202896-13-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Support for Virtual CPU Hotplug requires a sequence of ACPI handshakes between QEMU and the guest kernel when a vCPU is plugged or unplugged. Most of the AML code to support these handshakes already exists. This AML needs to be built during VM initialization for the ARM architecture as well, if GED support exists. Signed-off-by: Salil Mehta --- hw/arm/virt-acpi-build.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c index c3ccfef026..2d44567df5 100644 --- a/hw/arm/virt-acpi-build.c +++ b/hw/arm/virt-acpi-build.c @@ -799,6 +799,7 @@ static void build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) { VirtMachineClass *vmc = VIRT_MACHINE_GET_CLASS(vms); + MachineClass *mc = MACHINE_GET_CLASS(vms); Aml *scope, *dsdt; MachineState *ms = MACHINE(vms); const MemMapEntry *memmap = vms->memmap; @@ -815,7 +816,18 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) * the RTC ACPI device at all when using UEFI. */ scope = aml_scope("\\_SB"); - acpi_dsdt_add_cpus(scope, vms); + /* if GED is enabled then cpus AML shall be added as part build_cpus_aml */ + if (vms->acpi_dev && mc->has_hotpluggable_cpus) { + CPUHotplugFeatures opts = { + .acpi_1_compatible = false, + .has_legacy_cphp = false + }; + + build_cpus_aml(scope, ms, opts, NULL, memmap[VIRT_CPUHP_ACPI].base, + "\\_SB", NULL, AML_SYSTEM_MEMORY); + } else { + acpi_dsdt_add_cpus(scope, vms); + } acpi_dsdt_add_uart(scope, &memmap[VIRT_UART], (irqmap[VIRT_UART] + ARM_SPI_BASE)); if (vmc->acpi_expose_flash) { From patchwork Thu Jun 13 23:36:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697634 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CF89BC27C75 for ; Thu, 13 Jun 2024 23:42:56 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu5n-0005yG-G4; Thu, 13 Jun 2024 19:42:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu5l-0005o5-2u; Thu, 13 Jun 2024 19:42:53 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu5j-0003zc-5d; Thu, 13 Jun 2024 19:42:52 -0400 Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0f921SBRz6K62D; Fri, 14 Jun 2024 07:42:46 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 10EC3140594; Fri, 14 Jun 2024 07:42:49 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:42:26 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 13/29] arm/virt: Make ARM vCPU *present* status ACPI *persistent* Date: Fri, 14 Jun 2024 00:36:23 +0100 Message-ID: <20240613233639.202896-14-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org ARM arch does not allow CPUs presence to be changed [1] after kernel has booted. Hence, firmware/ACPI/Qemu must ensure persistent view of the vCPUs to the Guest kernel even when they are not present in the QoM i.e. are unplugged or are yet-to-be-plugged References: [1] Check comment 5 in the bugzilla entry Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5 Signed-off-by: Salil Mehta --- cpu-common.c | 6 ++++++ hw/arm/virt.c | 7 +++++++ include/hw/core/cpu.h | 21 +++++++++++++++++++++ 3 files changed, 34 insertions(+) diff --git a/cpu-common.c b/cpu-common.c index 49d2a50835..e4b4dee99a 100644 --- a/cpu-common.c +++ b/cpu-common.c @@ -128,6 +128,12 @@ bool qemu_enabled_cpu(CPUState *cpu) return cpu && !cpu->disabled; } +bool qemu_persistent_cpu(CPUState *cpu) +{ + /* cpu state can be faked to the guest via acpi */ + return cpu && cpu->acpi_persistent; +} + uint64_t qemu_get_cpu_archid(int cpu_index) { MachineState *ms = MACHINE(qdev_get_machine()); diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 5f98162587..9d33f30a6a 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -3016,6 +3016,13 @@ static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, return; } virt_cpu_set_properties(OBJECT(cs), cpu_slot, errp); + + /* + * To give persistent presence view of vCPUs to the guest, ACPI might need + * to fake the presence of the vCPUs to the guest but keep them disabled. + * This shall be used during the init of ACPI Hotplug state and hot-unplug + */ + cs->acpi_persistent = true; } static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index 62e68611c0..e13e542177 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -540,6 +540,14 @@ struct CPUState { * every CPUState is enabled across all architectures. */ bool disabled; + /* + * On certain architectures, to provide a persistent view of the 'presence' + * of vCPUs to the guest, ACPI might need to fake the 'presence' of the + * vCPUs but keep them ACPI-disabled for the guest. This is achieved by + * returning `_STA.PRES=True` and `_STA.Ena=False` for the unplugged vCPUs + * in QEMU QoM. + */ + bool acpi_persistent; /* TODO Move common fields from CPUArchState here. */ int cpu_index; @@ -959,6 +967,19 @@ bool qemu_present_cpu(CPUState *cpu); */ bool qemu_enabled_cpu(CPUState *cpu); +/** + * qemu_persistent_cpu: + * @cpu: The vCPU to check + * + * Checks if the vCPU state should always be reflected as *present* via ACPI + * to the Guest. By default, this is False on all architectures and has to be + * explicity set during initialization. + * + * Returns: True if it is ACPI 'persistent' CPU + * + */ +bool qemu_persistent_cpu(CPUState *cpu); + /** * qemu_get_cpu_archid: * @cpu_index: possible vCPU for which arch-id needs to be retreived From patchwork Thu Jun 13 23:36:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697635 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9CE57C27C4F for ; Thu, 13 Jun 2024 23:43:27 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu6E-00072X-OG; Thu, 13 Jun 2024 19:43:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu6B-00071m-Pj; Thu, 13 Jun 2024 19:43:19 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu65-00045f-Qj; Thu, 13 Jun 2024 19:43:19 -0400 Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0f9S5t55z6K69v; Fri, 14 Jun 2024 07:43:08 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id A7338140C72; Fri, 14 Jun 2024 07:43:11 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:42:49 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 14/29] hw/acpi: ACPI/AML Changes to reflect the correct _STA.{PRES, ENA} Bits to Guest Date: Fri, 14 Jun 2024 00:36:24 +0100 Message-ID: <20240613233639.202896-15-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org ACPI AML changes are required to properly reflect the `_STA.PRES` and `_STA.ENA` bits to the guest during initialization, when CPUs are hot-plugged, and after CPUs are hot-unplugged. Signed-off-by: Salil Mehta --- hw/acpi/cpu.c | 53 ++++++++++++++++++++++++++++++---- hw/acpi/generic_event_device.c | 11 +++++++ include/hw/acpi/cpu.h | 2 ++ 3 files changed, 61 insertions(+), 5 deletions(-) diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c index 4c63514b16..40b8899125 100644 --- a/hw/acpi/cpu.c +++ b/hw/acpi/cpu.c @@ -64,10 +64,11 @@ static uint64_t cpu_hotplug_rd(void *opaque, hwaddr addr, unsigned size) cdev = &cpu_st->devs[cpu_st->selector]; switch (addr) { case ACPI_CPU_FLAGS_OFFSET_RW: /* pack and return is_* fields */ - val |= cdev->cpu ? 1 : 0; + val |= cdev->is_enabled ? 1 : 0; val |= cdev->is_inserting ? 2 : 0; val |= cdev->is_removing ? 4 : 0; val |= cdev->fw_remove ? 16 : 0; + val |= cdev->is_present ? 32 : 0; trace_cpuhp_acpi_read_flags(cpu_st->selector, val); break; case ACPI_CPU_CMD_DATA_OFFSET_RW: @@ -230,7 +231,23 @@ void cpu_hotplug_hw_init(MemoryRegion *as, Object *owner, state->dev_count = id_list->len; state->devs = g_new0(typeof(*state->devs), state->dev_count); for (i = 0; i < id_list->len; i++) { - state->devs[i].cpu = CPU(id_list->cpus[i].cpu); + struct CPUState *cpu = CPU(id_list->cpus[i].cpu); + /* + * In most archs, CPUs which are ACPI 'present' are also ACPI 'enabled' + * by default. And these states are consistent at QOM and ACPI level. + */ + if (qemu_enabled_cpu(cpu)) { + state->devs[i].cpu = cpu; + state->devs[i].is_present = true; + state->devs[i].is_enabled = true; + } else { + state->devs[i].is_enabled = false; + /* + * Some archs might expose 'disabled' QOM CPUs as ACPI 'present'. + * Hence, these states at QOM and ACPI level might be inconsistent. + */ + state->devs[i].is_present = qemu_present_cpu(cpu); + } state->devs[i].arch_id = id_list->cpus[i].arch_id; } memory_region_init_io(&state->ctrl_reg, owner, &cpu_hotplug_ops, state, @@ -263,6 +280,8 @@ void acpi_cpu_plug_cb(HotplugHandler *hotplug_dev, } cdev->cpu = CPU(dev); + cdev->is_present = true; + cdev->is_enabled = true; if (dev->hotplugged) { cdev->is_inserting = true; acpi_send_event(DEVICE(hotplug_dev), ACPI_CPU_HOTPLUG_STATUS); @@ -294,6 +313,11 @@ void acpi_cpu_unplug_cb(CPUHotplugState *cpu_st, return; } + cdev->is_enabled = false; + if (!qemu_persistent_cpu(CPU(dev))) { + cdev->is_present = false; + } + cdev->cpu = NULL; } @@ -304,6 +328,8 @@ static const VMStateDescription vmstate_cpuhp_sts = { .fields = (const VMStateField[]) { VMSTATE_BOOL(is_inserting, AcpiCpuStatus), VMSTATE_BOOL(is_removing, AcpiCpuStatus), + VMSTATE_BOOL(is_present, AcpiCpuStatus), + VMSTATE_BOOL(is_enabled, AcpiCpuStatus), VMSTATE_UINT32(ost_event, AcpiCpuStatus), VMSTATE_UINT32(ost_status, AcpiCpuStatus), VMSTATE_END_OF_LIST() @@ -341,6 +367,7 @@ const VMStateDescription vmstate_cpu_hotplug = { #define CPU_REMOVE_EVENT "CRMV" #define CPU_EJECT_EVENT "CEJ0" #define CPU_FW_EJECT_EVENT "CEJF" +#define CPU_PRESENT "CPRS" void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts, build_madt_cpu_fn build_madt_cpu, hwaddr base_addr, @@ -399,7 +426,9 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts, aml_append(field, aml_named_field(CPU_EJECT_EVENT, 1)); /* tell firmware to do device eject, write only */ aml_append(field, aml_named_field(CPU_FW_EJECT_EVENT, 1)); - aml_append(field, aml_reserved_field(3)); + /* 1 if present, read only */ + aml_append(field, aml_named_field(CPU_PRESENT, 1)); + aml_append(field, aml_reserved_field(2)); aml_append(field, aml_named_field(CPU_COMMAND, 8)); aml_append(cpu_ctrl_dev, field); @@ -429,6 +458,7 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts, Aml *ctrl_lock = aml_name("%s.%s", cphp_res_path, CPU_LOCK); Aml *cpu_selector = aml_name("%s.%s", cphp_res_path, CPU_SELECTOR); Aml *is_enabled = aml_name("%s.%s", cphp_res_path, CPU_ENABLED); + Aml *is_present = aml_name("%s.%s", cphp_res_path, CPU_PRESENT); Aml *cpu_cmd = aml_name("%s.%s", cphp_res_path, CPU_COMMAND); Aml *cpu_data = aml_name("%s.%s", cphp_res_path, CPU_DATA); Aml *ins_evt = aml_name("%s.%s", cphp_res_path, CPU_INSERT_EVENT); @@ -457,13 +487,26 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts, { Aml *idx = aml_arg(0); Aml *sta = aml_local(0); + Aml *ifctx2; + Aml *else_ctx; aml_append(method, aml_acquire(ctrl_lock, 0xFFFF)); aml_append(method, aml_store(idx, cpu_selector)); aml_append(method, aml_store(zero, sta)); - ifctx = aml_if(aml_equal(is_enabled, one)); + ifctx = aml_if(aml_equal(is_present, one)); { - aml_append(ifctx, aml_store(aml_int(0xF), sta)); + ifctx2 = aml_if(aml_equal(is_enabled, one)); + { + /* cpu is present and enabled */ + aml_append(ifctx2, aml_store(aml_int(0xF), sta)); + } + aml_append(ifctx, ifctx2); + else_ctx = aml_else(); + { + /* cpu is present but disabled */ + aml_append(else_ctx, aml_store(aml_int(0xD), sta)); + } + aml_append(ifctx, else_ctx); } aml_append(method, ifctx); aml_append(method, aml_release(ctrl_lock)); diff --git a/hw/acpi/generic_event_device.c b/hw/acpi/generic_event_device.c index 63226b0040..e92ce07955 100644 --- a/hw/acpi/generic_event_device.c +++ b/hw/acpi/generic_event_device.c @@ -333,6 +333,16 @@ static const VMStateDescription vmstate_memhp_state = { } }; +static const VMStateDescription vmstate_cpuhp_state = { + .name = "acpi-ged/cpuhp", + .version_id = 1, + .minimum_version_id = 1, + .fields = (VMStateField[]) { + VMSTATE_CPU_HOTPLUG(cpuhp_state, AcpiGedState), + VMSTATE_END_OF_LIST() + } +}; + static const VMStateDescription vmstate_ged_state = { .name = "acpi-ged-state", .version_id = 1, @@ -381,6 +391,7 @@ static const VMStateDescription vmstate_acpi_ged = { }, .subsections = (const VMStateDescription * const []) { &vmstate_memhp_state, + &vmstate_cpuhp_state, &vmstate_ghes_state, NULL } diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h index 48cded697c..07b524b713 100644 --- a/include/hw/acpi/cpu.h +++ b/include/hw/acpi/cpu.h @@ -24,6 +24,8 @@ typedef struct AcpiCpuStatus { uint64_t arch_id; bool is_inserting; bool is_removing; + bool is_present; + bool is_enabled; bool fw_remove; uint32_t ost_event; uint32_t ost_status; From patchwork Thu Jun 13 23:36:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697636 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E9DFCC27C4F for ; Thu, 13 Jun 2024 23:44:01 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu6V-0007FK-MJ; Thu, 13 Jun 2024 19:43:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu6U-0007EK-DA; Thu, 13 Jun 2024 19:43:38 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu6S-00047w-FB; Thu, 13 Jun 2024 19:43:38 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0f8L60scz67MmR; Fri, 14 Jun 2024 07:42:10 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 4D7F414065C; Fri, 14 Jun 2024 07:43:34 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:43:11 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 15/29] hw/arm: MADT Tbl change to size the guest with possible vCPUs Date: Fri, 14 Jun 2024 00:36:25 +0100 Message-ID: <20240613233639.202896-16-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Changes are required during the building of the MADT table by QEMU to accommodate disabled possible vCPUs. This information will be used by the guest kernel to size up its resources during boot time. The pre-sizing of the guest kernel based on possible vCPUs will facilitate the hotplug of the disabled vCPUs. This change also addresses ACPI MADT GIC CPU interface flag-related changes recently introduced in the UEFI ACPI 6.5 specification, which allows deferred virtual CPU online'ing in the guest kernel. Link: https://uefi.org/specs/ACPI/6.5/05_ACPI_Software_Programming_Model.html#gic-cpu-interface-gicc-structure Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt-acpi-build.c | 36 ++++++++++++++++++++++++++++++------ 1 file changed, 30 insertions(+), 6 deletions(-) diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c index 2d44567df5..4b4906f407 100644 --- a/hw/arm/virt-acpi-build.c +++ b/hw/arm/virt-acpi-build.c @@ -657,6 +657,29 @@ static void build_append_gicr(GArray *table_data, uint64_t base, uint32_t size) build_append_int_noprefix(table_data, size, 4); /* Discovery Range Length */ } +static uint32_t virt_acpi_get_gicc_flags(CPUState *cpu) +{ + MachineClass *mc = MACHINE_GET_CLASS(qdev_get_machine()); + + /* can only exist in 'enabled' state */ + if (!mc->has_hotpluggable_cpus) { + return 1; + } + + /* + * ARM GIC CPU Interface can be 'online-capable' or 'enabled' at boot + * We MUST set 'online-capable' bit for all hotpluggable CPUs except the + * first/boot CPU. Cold-booted CPUs without 'Id' can also be unplugged. + * Though as-of-now this is only used as a debugging feature. + * + * UEFI ACPI Specification 6.5 + * Section: 5.2.12.14. GIC CPU Interface (GICC) Structure + * Table: 5.37 GICC CPU Interface Flags + * Link: https://uefi.org/specs/ACPI/6.5 + */ + return cpu && !cpu->cpu_index ? 1 : (1 << 3); +} + static void build_madt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) { @@ -683,12 +706,13 @@ build_madt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) build_append_int_noprefix(table_data, vms->gic_version, 1); build_append_int_noprefix(table_data, 0, 3); /* Reserved */ - for (i = 0; i < MACHINE(vms)->smp.cpus; i++) { - ARMCPU *armcpu = ARM_CPU(qemu_get_cpu(i)); + for (i = 0; i < MACHINE(vms)->smp.max_cpus; i++) { + CPUState *cpu = qemu_get_possible_cpu(i); uint64_t physical_base_address = 0, gich = 0, gicv = 0; uint32_t vgic_interrupt = vms->virt ? ARCH_GIC_MAINT_IRQ : 0; - uint32_t pmu_interrupt = arm_feature(&armcpu->env, ARM_FEATURE_PMU) ? - VIRTUAL_PMU_IRQ : 0; + uint32_t pmu_interrupt = vms->pmu ? VIRTUAL_PMU_IRQ : 0; + uint32_t flags = virt_acpi_get_gicc_flags(cpu); + uint64_t mpidr = qemu_get_cpu_archid(i); if (vms->gic_version == VIRT_GIC_VERSION_2) { physical_base_address = memmap[VIRT_GIC_CPU].base; @@ -703,7 +727,7 @@ build_madt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) build_append_int_noprefix(table_data, i, 4); /* GIC ID */ build_append_int_noprefix(table_data, i, 4); /* ACPI Processor UID */ /* Flags */ - build_append_int_noprefix(table_data, 1, 4); /* Enabled */ + build_append_int_noprefix(table_data, flags, 4); /* Parking Protocol Version */ build_append_int_noprefix(table_data, 0, 4); /* Performance Interrupt GSIV */ @@ -717,7 +741,7 @@ build_madt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) build_append_int_noprefix(table_data, vgic_interrupt, 4); build_append_int_noprefix(table_data, 0, 8); /* GICR Base Address*/ /* MPIDR */ - build_append_int_noprefix(table_data, arm_cpu_mp_affinity(armcpu), 8); + build_append_int_noprefix(table_data, mpidr, 8); /* Processor Power Efficiency Class */ build_append_int_noprefix(table_data, 0, 1); /* Reserved */ From patchwork Thu Jun 13 23:36:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697637 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 715FBC27C75 for ; Thu, 13 Jun 2024 23:44:19 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu6t-0008KR-DG; Thu, 13 Jun 2024 19:44:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu6r-0008I0-Ea; Thu, 13 Jun 2024 19:44:01 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu6p-000494-JT; Thu, 13 Jun 2024 19:44:01 -0400 Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0fBL0Bl2z6K6Sd; Fri, 14 Jun 2024 07:43:54 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id DD488140C72; Fri, 14 Jun 2024 07:43:56 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:43:34 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 16/29] hw/acpi: Make _MAT method optional Date: Fri, 14 Jun 2024 00:36:26 +0100 Message-ID: <20240613233639.202896-17-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jean-Philippe Brucker The GICC interface on arm64 vCPUs is statically defined in the MADT, and doesn't require a _MAT entry. Although the GICC is indicated as present by the MADT entry, it can only be used from vCPU sysregs, which aren't accessible until hot-add. Co-developed-by: Jean-Philippe Brucker Signed-off-by: Jean-Philippe Brucker Co-developed-by: Jonathan Cameron Signed-off-by: Jonathan Cameron Reviewed-by: Gavin Shan Signed-off-by: Salil Mehta --- hw/acpi/cpu.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c index 40b8899125..2240c6e108 100644 --- a/hw/acpi/cpu.c +++ b/hw/acpi/cpu.c @@ -717,10 +717,13 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts, aml_append(dev, method); /* build _MAT object */ - build_madt_cpu(i, arch_ids, madt_buf, true); /* set enabled flag */ - aml_append(dev, aml_name_decl("_MAT", + if (build_madt_cpu) { + build_madt_cpu(i, arch_ids, madt_buf, + true); /* set enabled flag */ + aml_append(dev, aml_name_decl("_MAT", aml_buffer(madt_buf->len, (uint8_t *)madt_buf->data))); - g_array_free(madt_buf, true); + g_array_free(madt_buf, true); + } if (CPU(arch_ids->cpus[i].cpu) != first_cpu) { method = aml_method("_EJ0", 1, AML_NOTSERIALIZED); From patchwork Thu Jun 13 23:36:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697638 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D31F4C27C4F for ; Thu, 13 Jun 2024 23:44:39 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu7J-0000rb-Gx; Thu, 13 Jun 2024 19:44:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu7E-0000fa-5k; Thu, 13 Jun 2024 19:44:24 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu7C-0004C2-6l; Thu, 13 Jun 2024 19:44:23 -0400 Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0fBm4qtcz6K6QZ; Fri, 14 Jun 2024 07:44:16 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 871461400D9; Fri, 14 Jun 2024 07:44:19 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:43:57 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 17/29] arm/virt: Release objects for *disabled* possible vCPUs after init Date: Fri, 14 Jun 2024 00:36:27 +0100 Message-ID: <20240613233639.202896-18-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org During `machvirt_init()`, QOM ARMCPU objects are pre-created along with the corresponding KVM vCPUs in the host for all possible vCPUs. This is necessary due to the architectural constraint that KVM restricts the deferred creation of KVM vCPUs and VGIC initialization/sizing after VM initialization. Hence, VGIC is pre-sized with possible vCPUs. After the initialization of the machine is complete, the disabled possible KVM vCPUs are parked in the per-virt-machine list "kvm_parked_vcpus," and we release the QOM ARMCPU objects for the disabled vCPUs. These will be re-created when the vCPU is hotplugged again. The QOM ARMCPU object is then re-attached to the corresponding parked KVM vCPU. Alternatively, we could have chosen not to release the QOM CPU objects and kept reusing them. This approach might require some modifications to the `qdevice_add()` interface to retrieve the old ARMCPU object instead of creating a new one for the hotplug request. Each of these approaches has its own pros and cons. This prototype uses the first approach (suggestions are welcome!). Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 9d33f30a6a..a72cd3b20d 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2050,6 +2050,7 @@ static void virt_cpu_post_init(VirtMachineState *vms, MemoryRegion *sysmem) { CPUArchIdList *possible_cpus = vms->parent.possible_cpus; int max_cpus = MACHINE(vms)->smp.max_cpus; + MachineState *ms = MACHINE(vms); bool aarch64, steal_time; CPUState *cpu; int n; @@ -2111,6 +2112,37 @@ static void virt_cpu_post_init(VirtMachineState *vms, MemoryRegion *sysmem) } } } + + if (kvm_enabled() || tcg_enabled()) { + for (n = 0; n < possible_cpus->len; n++) { + cpu = qemu_get_possible_cpu(n); + + /* + * Now, GIC has been sized with possible CPUs and we dont require + * disabled vCPU objects to be represented in the QOM. Release the + * disabled ARMCPU objects earlier used during init for pre-sizing. + * + * We fake to the guest through ACPI about the presence(_STA.PRES=1) + * of these non-existent vCPUs at VMM/qemu and present these as + * disabled vCPUs(_STA.ENA=0) so that they cant be used. These vCPUs + * can be later added to the guest through hotplug exchanges when + * ARMCPU objects are created back again using 'device_add' QMP + * command. + */ + /* + * RFC: Question: Other approach could've been to keep them forever + * and release it only once when qemu exits as part of finalize or + * when new vCPU is hotplugged. In the later old could be released + * for the newly created object for the same vCPU? + */ + if (!qemu_enabled_cpu(cpu)) { + CPUArchId *cpu_slot; + cpu_slot = virt_find_cpu_slot(ms, cpu->cpu_index); + cpu_slot->cpu = NULL; + object_unref(OBJECT(cpu)); + } + } + } } static void virt_cpu_set_properties(Object *cpuobj, const CPUArchId *cpu_slot, From patchwork Thu Jun 13 23:36:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8E90CC27C4F for ; Thu, 13 Jun 2024 23:45:08 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu7e-0001S8-5M; Thu, 13 Jun 2024 19:44:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu7c-0001OJ-Lt; Thu, 13 Jun 2024 19:44:48 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu7a-0004Dp-Lb; Thu, 13 Jun 2024 19:44:48 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0fCF47Q5z6K69v; Fri, 14 Jun 2024 07:44:41 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 6E33214065C; Fri, 14 Jun 2024 07:44:44 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:44:19 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 18/29] arm/virt: Add/update basic hot-(un)plug framework Date: Fri, 14 Jun 2024 00:36:28 +0100 Message-ID: <20240613233639.202896-19-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add CPU hot-unplug hooks and update hotplug hooks with additional sanity checks for use in hotplug paths. Note: The functional contents of the hooks (currently left with TODO comments) will be gradually filled in subsequent patches in an incremental approach to patch and logic building, which would roughly include the following: 1. (Un)wiring of interrupts between vCPU<->GIC. 2. Sending events to the guest for hot-(un)plug so that the guest can take appropriate actions. 3. Notifying the GIC about the hot-(un)plug action so that the vCPU can be (un)stitched to the GIC CPU interface. 4. Updating the guest with next boot information for this vCPU in the firmware. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 105 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 105 insertions(+) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index a72cd3b20d..f6b8c21f26 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -85,6 +85,7 @@ #include "hw/virtio/virtio-iommu.h" #include "hw/char/pl011.h" #include "qemu/guest-random.h" +#include "qapi/qmp/qdict.h" static GlobalProperty arm_virt_compat[] = { { TYPE_VIRTIO_IOMMU_PCI, "aw-bits", "48" }, @@ -3002,11 +3003,23 @@ static void virt_memory_plug(HotplugHandler *hotplug_dev, static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { + VirtMachineState *vms = VIRT_MACHINE(hotplug_dev); MachineState *ms = MACHINE(hotplug_dev); + MachineClass *mc = MACHINE_GET_CLASS(ms); ARMCPU *cpu = ARM_CPU(dev); CPUState *cs = CPU(dev); CPUArchId *cpu_slot; + if (dev->hotplugged && !vms->acpi_dev) { + error_setg(errp, "GED acpi device does not exists"); + return; + } + + if (dev->hotplugged && !mc->has_hotpluggable_cpus) { + error_setg(errp, "CPU hotplug not supported on this machine"); + return; + } + /* sanity check the cpu */ if (!object_dynamic_cast(OBJECT(cpu), ms->cpu_type)) { error_setg(errp, "Invalid CPU type, expected cpu type: '%s'", @@ -3049,6 +3062,22 @@ static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, } virt_cpu_set_properties(OBJECT(cs), cpu_slot, errp); + /* + * Fix the GIC for this new vCPU being plugged. The QOM CPU object for the + * new vCPU need to be updated in the corresponding QOM GICv3CPUState object + * We also need to re-wire the IRQs for this new CPU object. This update + * is limited to the QOM only and does not affects the KVM. Later has + * already been pre-sized with possible CPU at VM init time. This is a + * workaround to the constraints posed by ARM architecture w.r.t supporting + * CPU Hotplug. Specification does not exist for the later. + * This patch-up is required both for {cold,hot}-plugged vCPUs. Cold-inited + * vCPUs have their GIC state initialized during machvit_init(). + */ + if (vms->acpi_dev) { + /* TODO: update GIC about this hotplug change here */ + /* TODO: wire the GIC<->CPU irqs */ + } + /* * To give persistent presence view of vCPUs to the guest, ACPI might need * to fake the presence of the vCPUs to the guest but keep them disabled. @@ -3060,6 +3089,7 @@ static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { + VirtMachineState *vms = VIRT_MACHINE(hotplug_dev); MachineState *ms = MACHINE(hotplug_dev); CPUState *cs = CPU(dev); CPUArchId *cpu_slot; @@ -3068,10 +3098,81 @@ static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, cpu_slot = virt_find_cpu_slot(ms, cs->cpu_index); cpu_slot->cpu = CPU(dev); + /* + * Update the ACPI Hotplug state both for vCPUs being {hot,cold}-plugged. + * vCPUs can be cold-plugged using '-device' option. For vCPUs being hot + * plugged, guest is also notified. + */ + if (vms->acpi_dev) { + /* TODO: update acpi hotplug state. Send cpu hotplug event to guest */ + /* TODO: register cpu for reset & update F/W info for the next boot */ + } + cs->disabled = false; return; } +static void virt_cpu_unplug_request(HotplugHandler *hotplug_dev, + DeviceState *dev, Error **errp) +{ + MachineClass *mc = MACHINE_GET_CLASS(qdev_get_machine()); + VirtMachineState *vms = VIRT_MACHINE(hotplug_dev); + ARMCPU *cpu = ARM_CPU(dev); + CPUState *cs = CPU(dev); + + if (!vms->acpi_dev || !dev->realized) { + error_setg(errp, "GED does not exists or device is not realized!"); + return; + } + + if (!mc->has_hotpluggable_cpus) { + error_setg(errp, "CPU hot(un)plug not supported on this machine"); + return; + } + + if (cs->cpu_index == first_cpu->cpu_index) { + error_setg(errp, "Boot CPU(id%d=%d:%d:%d:%d) hot-unplug not supported", + first_cpu->cpu_index, cpu->socket_id, cpu->cluster_id, + cpu->core_id, cpu->thread_id); + return; + } + + /* TODO: request cpu hotplug from guest */ + + return; +} + +static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, + Error **errp) +{ + VirtMachineState *vms = VIRT_MACHINE(hotplug_dev); + MachineState *ms = MACHINE(hotplug_dev); + CPUState *cs = CPU(dev); + CPUArchId *cpu_slot; + + if (!vms->acpi_dev || !dev->realized) { + error_setg(errp, "GED does not exists or device is not realized!"); + return; + } + + cpu_slot = virt_find_cpu_slot(ms, cs->cpu_index); + + /* TODO: update the acpi cpu hotplug state for cpu hot-unplug */ + + /* TODO: unwire the gic-cpu irqs here */ + /* TODO: update the GIC about this hot unplug change */ + + /* TODO: unregister cpu for reset & update F/W info for the next boot */ + + qobject_unref(dev->opts); + dev->opts = NULL; + + cpu_slot->cpu = NULL; + cs->disabled = true; + + return; +} + static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { @@ -3196,6 +3297,8 @@ static void virt_machine_device_unplug_request_cb(HotplugHandler *hotplug_dev, } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MD_PCI)) { virtio_md_pci_unplug_request(VIRTIO_MD_PCI(dev), MACHINE(hotplug_dev), errp); + } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) { + virt_cpu_unplug_request(hotplug_dev, dev, errp); } else { error_setg(errp, "device unplug request for unsupported device" " type: %s", object_get_typename(OBJECT(dev))); @@ -3209,6 +3312,8 @@ static void virt_machine_device_unplug_cb(HotplugHandler *hotplug_dev, virt_dimm_unplug(hotplug_dev, dev, errp); } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MD_PCI)) { virtio_md_pci_unplug(VIRTIO_MD_PCI(dev), MACHINE(hotplug_dev), errp); + } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) { + virt_cpu_unplug(hotplug_dev, dev, errp); } else { error_setg(errp, "virt: device unplug for unsupported device" " type: %s", object_get_typename(OBJECT(dev))); From patchwork Thu Jun 13 23:36:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697640 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2C813C27C4F for ; Thu, 13 Jun 2024 23:45:20 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu80-0002RZ-7y; Thu, 13 Jun 2024 19:45:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu7y-0002JO-41; Thu, 13 Jun 2024 19:45:10 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu7v-0004QQ-Gg; Thu, 13 Jun 2024 19:45:09 -0400 Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0fCd6sc2z6K6DJ; Fri, 14 Jun 2024 07:45:01 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id CAACD140594; Fri, 14 Jun 2024 07:45:04 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:44:42 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 19/29] arm/virt: Changes to (un)wire GICC<->vCPU IRQs during hot-(un)plug Date: Fri, 14 Jun 2024 00:36:29 +0100 Message-ID: <20240613233639.202896-20-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Refactors the existing GIC create code to extract common code to wire the vcpu<->gic interrupts. This function could be used with cold-plug case and also used when vCPU is hot-plugged. It also introduces a new function to unwire the vcpu<->gic interrupts for the vCPU hot-unplug cases. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta Reported-by: Vishnu Pajjuri [4/05/2024: Issue with total number of PPI available during create GIC] Suggested-by: Miguel Luis [5/05/2024: Fix the total number of PPIs available as per ARM BSA to avoid overflow] Signed-off-by: Salil Mehta --- hw/arm/virt.c | 154 ++++++++++++++++++++++++++++------------- hw/core/gpio.c | 2 +- include/hw/qdev-core.h | 2 + target/arm/cpu-qom.h | 18 +++-- 4 files changed, 118 insertions(+), 58 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index f6b8c21f26..1556c362f7 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -747,6 +747,107 @@ static bool gicv3_nmi_present(VirtMachineState *vms) (vms->gic_version != VIRT_GIC_VERSION_2); } +/* + * Mapping from the output timer irq lines from the CPU to the GIC PPI inputs + * we use for the virt board. + */ +const int timer_irq[] = { + [GTIMER_PHYS] = ARCH_TIMER_NS_EL1_IRQ, + [GTIMER_VIRT] = ARCH_TIMER_VIRT_IRQ, + [GTIMER_HYP] = ARCH_TIMER_NS_EL2_IRQ, + [GTIMER_SEC] = ARCH_TIMER_S_EL1_IRQ, +}; + +static void unwire_gic_cpu_irqs(VirtMachineState *vms, CPUState *cs) +{ + MachineState *ms = MACHINE(vms); + unsigned int max_cpus = ms->smp.max_cpus; + DeviceState *cpudev = DEVICE(cs); + DeviceState *gicdev = vms->gic; + int cpu = CPU(cs)->cpu_index; + int type = vms->gic_version; + int irq, num_gpio_in; + + for (irq = 0; irq < ARRAY_SIZE(timer_irq); irq++) { + qdev_disconnect_gpio_out_named(cpudev, NULL, irq); + } + + if (type != VIRT_GIC_VERSION_2) { + qdev_disconnect_gpio_out_named(cpudev, "gicv3-maintenance-interrupt", + 0); + } else if (vms->virt) { + qdev_disconnect_gpio_out_named(gicdev, SYSBUS_DEVICE_GPIO_IRQ, + cpu + 4 * max_cpus); + } + + /* + * RFC: Question: This currently does not takes care of intimating the + * devices which might be sitting on system bus. Do we need a + * sysbus_disconnect_irq() which also does the job of notification beside + * disconnection? + */ + qdev_disconnect_gpio_out_named(cpudev, "pmu-interrupt", 0); + + /* Unwire GIC's IRQ/FIQ/VIRQ/VFIQ/NMI/VINMI interrupt outputs to CPU */ + num_gpio_in = (vms->gic_version != VIRT_GIC_VERSION_2) ? + NUM_GPIO_IN : NUM_GICV2_GPIO_IN; + for (irq = 0; irq < num_gpio_in; irq++) { + qdev_disconnect_gpio_out_named(gicdev, SYSBUS_DEVICE_GPIO_IRQ, + cpu + irq * max_cpus); + } +} + +static void wire_gic_cpu_irqs(VirtMachineState *vms, CPUState *cs) +{ + MachineState *ms = MACHINE(vms); + unsigned int max_cpus = ms->smp.max_cpus; + DeviceState *cpudev = DEVICE(cs); + DeviceState *gicdev = vms->gic; + int cpu = CPU(cs)->cpu_index; + int type = vms->gic_version; + SysBusDevice *gicbusdev; + int intidbase; + int irqn; + + intidbase = NUM_IRQS + cpu * GIC_INTERNAL; + + for (irqn = 0; irqn < ARRAY_SIZE(timer_irq); irqn++) { + qdev_connect_gpio_out(cpudev, irqn, + qdev_get_gpio_in(gicdev, + intidbase + timer_irq[irqn])); + } + + gicbusdev = SYS_BUS_DEVICE(gicdev); + if (type != VIRT_GIC_VERSION_2) { + qemu_irq irq = qdev_get_gpio_in(gicdev, + intidbase + ARCH_GIC_MAINT_IRQ); + qdev_connect_gpio_out_named(cpudev, "gicv3-maintenance-interrupt", + 0, irq); + } else if (vms->virt) { + qemu_irq irq = qdev_get_gpio_in(gicdev, + intidbase + ARCH_GIC_MAINT_IRQ); + sysbus_connect_irq(gicbusdev, cpu + 4 * max_cpus, irq); + } + + qdev_connect_gpio_out_named(cpudev, "pmu-interrupt", 0, + qdev_get_gpio_in(gicdev, + intidbase + VIRTUAL_PMU_IRQ)); + + sysbus_connect_irq(gicbusdev, cpu, qdev_get_gpio_in(cpudev, ARM_CPU_IRQ)); + sysbus_connect_irq(gicbusdev, cpu + max_cpus, + qdev_get_gpio_in(cpudev, ARM_CPU_FIQ)); + sysbus_connect_irq(gicbusdev, cpu + 2 * max_cpus, + qdev_get_gpio_in(cpudev, ARM_CPU_VIRQ)); + sysbus_connect_irq(gicbusdev, cpu + 3 * max_cpus, + qdev_get_gpio_in(cpudev, ARM_CPU_VFIQ)); + if (vms->gic_version != VIRT_GIC_VERSION_2) { + sysbus_connect_irq(gicbusdev, cpu + 4 * max_cpus, + qdev_get_gpio_in(cpudev, ARM_CPU_NMI)); + sysbus_connect_irq(gicbusdev, cpu + 5 * max_cpus, + qdev_get_gpio_in(cpudev, ARM_CPU_VINMI)); + } +} + static void create_gic(VirtMachineState *vms, MemoryRegion *mem) { MachineState *ms = MACHINE(vms); @@ -849,54 +950,7 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem) * CPU's inputs. */ for (i = 0; i < smp_cpus; i++) { - DeviceState *cpudev = DEVICE(qemu_get_cpu(i)); - int intidbase = NUM_IRQS + i * GIC_INTERNAL; - /* Mapping from the output timer irq lines from the CPU to the - * GIC PPI inputs we use for the virt board. - */ - const int timer_irq[] = { - [GTIMER_PHYS] = ARCH_TIMER_NS_EL1_IRQ, - [GTIMER_VIRT] = ARCH_TIMER_VIRT_IRQ, - [GTIMER_HYP] = ARCH_TIMER_NS_EL2_IRQ, - [GTIMER_SEC] = ARCH_TIMER_S_EL1_IRQ, - [GTIMER_HYPVIRT] = ARCH_TIMER_NS_EL2_VIRT_IRQ, - }; - - for (unsigned irq = 0; irq < ARRAY_SIZE(timer_irq); irq++) { - qdev_connect_gpio_out(cpudev, irq, - qdev_get_gpio_in(vms->gic, - intidbase + timer_irq[irq])); - } - - if (vms->gic_version != VIRT_GIC_VERSION_2) { - qemu_irq irq = qdev_get_gpio_in(vms->gic, - intidbase + ARCH_GIC_MAINT_IRQ); - qdev_connect_gpio_out_named(cpudev, "gicv3-maintenance-interrupt", - 0, irq); - } else if (vms->virt) { - qemu_irq irq = qdev_get_gpio_in(vms->gic, - intidbase + ARCH_GIC_MAINT_IRQ); - sysbus_connect_irq(gicbusdev, i + 4 * max_cpus, irq); - } - - qdev_connect_gpio_out_named(cpudev, "pmu-interrupt", 0, - qdev_get_gpio_in(vms->gic, intidbase - + VIRTUAL_PMU_IRQ)); - - sysbus_connect_irq(gicbusdev, i, qdev_get_gpio_in(cpudev, ARM_CPU_IRQ)); - sysbus_connect_irq(gicbusdev, i + max_cpus, - qdev_get_gpio_in(cpudev, ARM_CPU_FIQ)); - sysbus_connect_irq(gicbusdev, i + 2 * max_cpus, - qdev_get_gpio_in(cpudev, ARM_CPU_VIRQ)); - sysbus_connect_irq(gicbusdev, i + 3 * max_cpus, - qdev_get_gpio_in(cpudev, ARM_CPU_VFIQ)); - - if (vms->gic_version != VIRT_GIC_VERSION_2) { - sysbus_connect_irq(gicbusdev, i + 4 * smp_cpus, - qdev_get_gpio_in(cpudev, ARM_CPU_NMI)); - sysbus_connect_irq(gicbusdev, i + 5 * smp_cpus, - qdev_get_gpio_in(cpudev, ARM_CPU_VINMI)); - } + wire_gic_cpu_irqs(vms, qemu_get_cpu(i)); } fdt_add_gic_node(vms); @@ -3075,7 +3129,7 @@ static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, */ if (vms->acpi_dev) { /* TODO: update GIC about this hotplug change here */ - /* TODO: wire the GIC<->CPU irqs */ + wire_gic_cpu_irqs(vms, cs); } /* @@ -3159,7 +3213,7 @@ static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, /* TODO: update the acpi cpu hotplug state for cpu hot-unplug */ - /* TODO: unwire the gic-cpu irqs here */ + unwire_gic_cpu_irqs(vms, cs); /* TODO: update the GIC about this hot unplug change */ /* TODO: unregister cpu for reset & update F/W info for the next boot */ diff --git a/hw/core/gpio.c b/hw/core/gpio.c index 80d07a6ec9..abb164d5c0 100644 --- a/hw/core/gpio.c +++ b/hw/core/gpio.c @@ -143,7 +143,7 @@ qemu_irq qdev_get_gpio_out_connector(DeviceState *dev, const char *name, int n) /* disconnect a GPIO output, returning the disconnected input (if any) */ -static qemu_irq qdev_disconnect_gpio_out_named(DeviceState *dev, +qemu_irq qdev_disconnect_gpio_out_named(DeviceState *dev, const char *name, int n) { char *propname = g_strdup_printf("%s[%d]", diff --git a/include/hw/qdev-core.h b/include/hw/qdev-core.h index 5336728a23..742e62e400 100644 --- a/include/hw/qdev-core.h +++ b/include/hw/qdev-core.h @@ -738,6 +738,8 @@ qemu_irq qdev_get_gpio_out_connector(DeviceState *dev, const char *name, int n); */ qemu_irq qdev_intercept_gpio_out(DeviceState *dev, qemu_irq icpt, const char *name, int n); +qemu_irq qdev_disconnect_gpio_out_named(DeviceState *dev, + const char *name, int n); BusState *qdev_get_child_bus(DeviceState *dev, const char *name); diff --git a/target/arm/cpu-qom.h b/target/arm/cpu-qom.h index b497667d61..e49fb096de 100644 --- a/target/arm/cpu-qom.h +++ b/target/arm/cpu-qom.h @@ -37,13 +37,17 @@ DECLARE_CLASS_CHECKERS(AArch64CPUClass, AARCH64_CPU, #define ARM_CPU_TYPE_NAME(name) (name ARM_CPU_TYPE_SUFFIX) /* Meanings of the ARMCPU object's seven inbound GPIO lines */ -#define ARM_CPU_IRQ 0 -#define ARM_CPU_FIQ 1 -#define ARM_CPU_VIRQ 2 -#define ARM_CPU_VFIQ 3 -#define ARM_CPU_NMI 4 -#define ARM_CPU_VINMI 5 -#define ARM_CPU_VFNMI 6 +enum { + ARM_CPU_IRQ = 0, + ARM_CPU_FIQ = 1, + ARM_CPU_VIRQ = 2, + ARM_CPU_VFIQ = 3, + NUM_GICV2_GPIO_IN = (ARM_CPU_VFIQ+1), + ARM_CPU_NMI = 4, + ARM_CPU_VINMI = 5, + /* ARM_CPU_VFNMI = 6, */ /* not used? */ + NUM_GPIO_IN = (ARM_CPU_VINMI+1), +}; /* For M profile, some registers are banked secure vs non-secure; * these are represented as a 2-element array where the first element From patchwork Thu Jun 13 23:36:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D8FD5C27C4F for ; Thu, 13 Jun 2024 23:45:50 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu8M-0003aa-Fy; Thu, 13 Jun 2024 19:45:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu8J-0003WG-O3; Thu, 13 Jun 2024 19:45:31 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu8H-0004Y5-Iw; Thu, 13 Jun 2024 19:45:31 -0400 Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0fD43bQZz6K6DJ; Fri, 14 Jun 2024 07:45:24 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 5B4521400D9; Fri, 14 Jun 2024 07:45:27 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:45:05 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 20/29] hw/arm, gicv3: Changes to update GIC with vCPU hot-plug notification Date: Fri, 14 Jun 2024 00:36:30 +0100 Message-ID: <20240613233639.202896-21-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Virtual CPU hot-(un)plug events MUST be notified to the GIC. Introduce a notfication mechanism to update any such events to GIC so that it can update its vCPU to GIC CPU interface association. This is required to implement a workaround to the limitations posed by the ARM architecture. For details about the constraints and workarounds please check below slides: Link: https://kvm-forum.qemu.org/2023/talk/9SMPDQ/ Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 27 +++++++++++++-- hw/intc/arm_gicv3_common.c | 54 +++++++++++++++++++++++++++++- hw/intc/arm_gicv3_cpuif_common.c | 5 +++ hw/intc/gicv3_internal.h | 1 + include/hw/arm/virt.h | 1 + include/hw/intc/arm_gicv3_common.h | 22 ++++++++++++ 6 files changed, 107 insertions(+), 3 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 1556c362f7..9f7e07bd8e 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -685,6 +685,16 @@ static inline DeviceState *create_acpi_ged(VirtMachineState *vms) return dev; } +static void virt_add_gic_cpuhp_notifier(VirtMachineState *vms) +{ + MachineClass *mc = MACHINE_GET_CLASS(vms); + + if (mc->has_hotpluggable_cpus) { + Notifier *cpuhp_notifier = gicv3_cpuhp_notifier(vms->gic); + notifier_list_add(&vms->cpuhp_notifiers, cpuhp_notifier); + } +} + static void create_its(VirtMachineState *vms) { const char *itsclass = its_class_name(); @@ -960,6 +970,9 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem) } else if (vms->gic_version == VIRT_GIC_VERSION_2) { create_v2m(vms); } + + /* add GIC CPU hot(un)plug update notifier */ + virt_add_gic_cpuhp_notifier(vms); } static void create_uart(const VirtMachineState *vms, int uart, @@ -2472,6 +2485,8 @@ static void machvirt_init(MachineState *machine) create_fdt(vms); + notifier_list_init(&vms->cpuhp_notifiers); + assert(possible_cpus->len == max_cpus); for (n = 0; n < possible_cpus->len; n++) { CPUArchId *cpu_slot; @@ -3054,6 +3069,14 @@ static void virt_memory_plug(HotplugHandler *hotplug_dev, dev, &error_abort); } +static void virt_update_gic(VirtMachineState *vms, CPUState *cs) +{ + GICv3CPUHotplugInfo gic_info = { .gic = vms->gic, .cpu = cs }; + + /* notify gic to stitch GICC to this new cpu */ + notifier_list_notify(&vms->cpuhp_notifiers, &gic_info); +} + static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { @@ -3128,7 +3151,7 @@ static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, * vCPUs have their GIC state initialized during machvit_init(). */ if (vms->acpi_dev) { - /* TODO: update GIC about this hotplug change here */ + virt_update_gic(vms, cs); wire_gic_cpu_irqs(vms, cs); } @@ -3214,7 +3237,7 @@ static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, /* TODO: update the acpi cpu hotplug state for cpu hot-unplug */ unwire_gic_cpu_irqs(vms, cs); - /* TODO: update the GIC about this hot unplug change */ + virt_update_gic(vms, cs); /* TODO: unregister cpu for reset & update F/W info for the next boot */ diff --git a/hw/intc/arm_gicv3_common.c b/hw/intc/arm_gicv3_common.c index 183d2de7eb..155342055b 100644 --- a/hw/intc/arm_gicv3_common.c +++ b/hw/intc/arm_gicv3_common.c @@ -33,7 +33,6 @@ #include "hw/arm/linux-boot-if.h" #include "sysemu/kvm.h" - static void gicv3_gicd_no_migration_shift_bug_post_load(GICv3State *cs) { if (cs->gicd_no_migration_shift_bug) { @@ -366,6 +365,56 @@ void gicv3_init_irqs_and_mmio(GICv3State *s, qemu_irq_handler handler, } } +static int arm_gicv3_get_proc_num(GICv3State *s, CPUState *cpu) +{ + uint64_t mp_affinity; + uint64_t gicr_typer; + uint64_t cpu_affid; + int i; + + mp_affinity = object_property_get_uint(OBJECT(cpu), "mp-affinity", NULL); + /* match the cpu mp-affinity to get the gic cpuif number */ + for (i = 0; i < s->num_cpu; i++) { + gicr_typer = s->cpu[i].gicr_typer; + cpu_affid = (gicr_typer >> 32) & 0xFFFFFF; + if (cpu_affid == mp_affinity) { + return i; + } + } + + return -1; +} + +static void arm_gicv3_cpu_update_notifier(Notifier *notifier, void * data) +{ + GICv3CPUHotplugInfo *gic_info = (GICv3CPUHotplugInfo *)data; + CPUState *cpu = gic_info->cpu; + int gic_cpuif_num; + GICv3State *s; + + s = ARM_GICV3_COMMON(gic_info->gic); + + /* this shall get us mapped gicv3 cpuif corresponding to mpidr */ + gic_cpuif_num = arm_gicv3_get_proc_num(s, cpu); + if (gic_cpuif_num < 0) { + error_report("Failed to associate cpu %d with any GIC cpuif", + cpu->cpu_index); + abort(); + } + + /* check if update is for vcpu hot-unplug */ + if (qemu_enabled_cpu(cpu)) { + s->cpu[gic_cpuif_num].cpu = NULL; + return; + } + + /* re-stitch the gic cpuif to this new cpu */ + gicv3_set_gicv3state(cpu, &s->cpu[gic_cpuif_num]); + gicv3_set_cpustate(&s->cpu[gic_cpuif_num], cpu); + + /* TODO: initialize the registers info for this newly added cpu */ +} + static void arm_gicv3_common_realize(DeviceState *dev, Error **errp) { GICv3State *s = ARM_GICV3_COMMON(dev); @@ -488,6 +537,8 @@ static void arm_gicv3_common_realize(DeviceState *dev, Error **errp) s->cpu[cpuidx - 1].gicr_typer |= GICR_TYPER_LAST; } + s->cpu_update_notifier.notify = arm_gicv3_cpu_update_notifier; + s->itslist = g_ptr_array_new(); } @@ -495,6 +546,7 @@ static void arm_gicv3_finalize(Object *obj) { GICv3State *s = ARM_GICV3_COMMON(obj); + notifier_remove(&s->cpu_update_notifier); g_free(s->redist_region_count); } diff --git a/hw/intc/arm_gicv3_cpuif_common.c b/hw/intc/arm_gicv3_cpuif_common.c index ff1239f65d..381cf2754b 100644 --- a/hw/intc/arm_gicv3_cpuif_common.c +++ b/hw/intc/arm_gicv3_cpuif_common.c @@ -20,3 +20,8 @@ void gicv3_set_gicv3state(CPUState *cpu, GICv3CPUState *s) env->gicv3state = (void *)s; }; + +void gicv3_set_cpustate(GICv3CPUState *s, CPUState *cpu) +{ + s->cpu = cpu; +} diff --git a/hw/intc/gicv3_internal.h b/hw/intc/gicv3_internal.h index bc9f518fe8..42441c19c6 100644 --- a/hw/intc/gicv3_internal.h +++ b/hw/intc/gicv3_internal.h @@ -861,5 +861,6 @@ static inline void gicv3_cache_all_target_cpustates(GICv3State *s) } void gicv3_set_gicv3state(CPUState *cpu, GICv3CPUState *s); +void gicv3_set_cpustate(GICv3CPUState *s, CPUState *cpu); #endif /* QEMU_ARM_GICV3_INTERNAL_H */ diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index d711cab46d..9c728ba042 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -181,6 +181,7 @@ struct VirtMachineState { char *oem_id; char *oem_table_id; bool ns_el2_virt_timer_irq; + NotifierList cpuhp_notifiers; }; #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM) diff --git a/include/hw/intc/arm_gicv3_common.h b/include/hw/intc/arm_gicv3_common.h index cd09bee3bc..496b198016 100644 --- a/include/hw/intc/arm_gicv3_common.h +++ b/include/hw/intc/arm_gicv3_common.h @@ -293,6 +293,7 @@ struct GICv3State { GICv3CPUState *gicd_irouter_target[GICV3_MAXIRQ]; uint32_t gicd_nsacr[DIV_ROUND_UP(GICV3_MAXIRQ, 16)]; + Notifier cpu_update_notifier; GICv3CPUState *cpu; /* List of all ITSes connected to this GIC */ GPtrArray *itslist; @@ -342,6 +343,27 @@ struct ARMGICv3CommonClass { void gicv3_init_irqs_and_mmio(GICv3State *s, qemu_irq_handler handler, const MemoryRegionOps *ops); +/** + * Structure used by GICv3 CPU hotplug notifier + */ +typedef struct GICv3CPUHotplugInfo { + DeviceState *gic; /* GICv3State */ + CPUState *cpu; +} GICv3CPUHotplugInfo; + +/** + * gicv3_cpuhp_notifier + * + * Returns CPU hotplug notifier which could be used to update GIC about any + * CPU hot(un)plug events. + * + * Returns: Notifier initialized with CPU Hot(un)plug update function + */ +static inline Notifier *gicv3_cpuhp_notifier(DeviceState *dev) +{ + GICv3State *s = ARM_GICV3_COMMON(dev); + return &s->cpu_update_notifier; +} /** * gicv3_class_name From patchwork Thu Jun 13 23:36:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697642 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2458AC27C4F for ; Thu, 13 Jun 2024 23:46:23 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu8y-0004G2-0D; Thu, 13 Jun 2024 19:46:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu8i-00045e-O2; Thu, 13 Jun 2024 19:45:58 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu8f-0004ZP-Er; Thu, 13 Jun 2024 19:45:55 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0fDW2G5tz6K5nF; Fri, 14 Jun 2024 07:45:47 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 2F28414065C; Fri, 14 Jun 2024 07:45:50 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:45:27 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 21/29] hw/intc/arm-gicv3*: Changes required to (re)init the vCPU register info Date: Fri, 14 Jun 2024 00:36:31 +0100 Message-ID: <20240613233639.202896-22-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org vCPU register info needs to be re-initialized each time vCPU is hot-plugged. This has to be done both for emulation/TCG and KVM case. This is done in context to the GIC update notification for any vCPU hot-(un)plug events. This change adds that support and re-factors existing to maximize the code re-use. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/intc/arm_gicv3.c | 1 + hw/intc/arm_gicv3_common.c | 7 +- hw/intc/arm_gicv3_cpuif.c | 261 +++++++++++++++-------------- hw/intc/arm_gicv3_kvm.c | 7 +- hw/intc/gicv3_internal.h | 1 + include/hw/intc/arm_gicv3_common.h | 1 + 6 files changed, 151 insertions(+), 127 deletions(-) diff --git a/hw/intc/arm_gicv3.c b/hw/intc/arm_gicv3.c index 58e18fff54..2a30625916 100644 --- a/hw/intc/arm_gicv3.c +++ b/hw/intc/arm_gicv3.c @@ -459,6 +459,7 @@ static void arm_gicv3_class_init(ObjectClass *klass, void *data) ARMGICv3Class *agc = ARM_GICV3_CLASS(klass); agcc->post_load = arm_gicv3_post_load; + agcc->init_cpu_reginfo = gicv3_init_cpu_reginfo; device_class_set_parent_realize(dc, arm_gic_realize, &agc->parent_realize); } diff --git a/hw/intc/arm_gicv3_common.c b/hw/intc/arm_gicv3_common.c index 155342055b..02e68a437e 100644 --- a/hw/intc/arm_gicv3_common.c +++ b/hw/intc/arm_gicv3_common.c @@ -389,10 +389,12 @@ static void arm_gicv3_cpu_update_notifier(Notifier *notifier, void * data) { GICv3CPUHotplugInfo *gic_info = (GICv3CPUHotplugInfo *)data; CPUState *cpu = gic_info->cpu; + ARMGICv3CommonClass *c; int gic_cpuif_num; GICv3State *s; s = ARM_GICV3_COMMON(gic_info->gic); + c = ARM_GICV3_COMMON_GET_CLASS(s); /* this shall get us mapped gicv3 cpuif corresponding to mpidr */ gic_cpuif_num = arm_gicv3_get_proc_num(s, cpu); @@ -412,7 +414,10 @@ static void arm_gicv3_cpu_update_notifier(Notifier *notifier, void * data) gicv3_set_gicv3state(cpu, &s->cpu[gic_cpuif_num]); gicv3_set_cpustate(&s->cpu[gic_cpuif_num], cpu); - /* TODO: initialize the registers info for this newly added cpu */ + /* initialize the registers info for this newly added cpu */ + if (c->init_cpu_reginfo) { + c->init_cpu_reginfo(cpu); + } } static void arm_gicv3_common_realize(DeviceState *dev, Error **errp) diff --git a/hw/intc/arm_gicv3_cpuif.c b/hw/intc/arm_gicv3_cpuif.c index 2a8aff0b99..1ace0f54d8 100644 --- a/hw/intc/arm_gicv3_cpuif.c +++ b/hw/intc/arm_gicv3_cpuif.c @@ -3033,143 +3033,138 @@ static void gicv3_cpuif_el_change_hook(ARMCPU *cpu, void *opaque) gicv3_cpuif_virt_irq_fiq_update(cs); } -void gicv3_init_cpuif(GICv3State *s) +void gicv3_init_cpu_reginfo(CPUState *cs) { - /* Called from the GICv3 realize function; register our system - * registers with the CPU - */ - int i; + ARMCPU *cpu = ARM_CPU(cs); + GICv3CPUState *gcs = icc_cs_from_env(&cpu->env); - for (i = 0; i < s->num_cpu; i++) { - ARMCPU *cpu = ARM_CPU(qemu_get_cpu(i)); - GICv3CPUState *cs = &s->cpu[i]; + /* + * If the CPU doesn't define a GICv3 configuration, probably because + * in real hardware it doesn't have one, then we use default values + * matching the one used by most Arm CPUs. This applies to: + * cpu->gic_num_lrs + * cpu->gic_vpribits + * cpu->gic_vprebits + * cpu->gic_pribits + */ - /* - * If the CPU doesn't define a GICv3 configuration, probably because - * in real hardware it doesn't have one, then we use default values - * matching the one used by most Arm CPUs. This applies to: - * cpu->gic_num_lrs - * cpu->gic_vpribits - * cpu->gic_vprebits - * cpu->gic_pribits - */ + /* + * Note that we can't just use the GICv3CPUState as an opaque pointer + * in define_arm_cp_regs_with_opaque(), because when we're called back + * it might be with code translated by CPU 0 but run by CPU 1, in + * which case we'd get the wrong value. + * So instead we define the regs with no ri->opaque info, and + * get back to the GICv3CPUState from the CPUARMState. + * + * These CP regs callbacks can be called from either TCG or HVF code. + */ + define_arm_cp_regs(cpu, gicv3_cpuif_reginfo); - /* Note that we can't just use the GICv3CPUState as an opaque pointer - * in define_arm_cp_regs_with_opaque(), because when we're called back - * it might be with code translated by CPU 0 but run by CPU 1, in - * which case we'd get the wrong value. - * So instead we define the regs with no ri->opaque info, and - * get back to the GICv3CPUState from the CPUARMState. - * - * These CP regs callbacks can be called from either TCG or HVF code. - */ - define_arm_cp_regs(cpu, gicv3_cpuif_reginfo); + /* + * If the CPU implements FEAT_NMI and FEAT_GICv3 it must also + * implement FEAT_GICv3_NMI, which is the CPU interface part + * of NMI support. This is distinct from whether the GIC proper + * (redistributors and distributor) have NMI support. In QEMU + * that is a property of the GIC device in s->nmi_support; + * cs->nmi_support indicates the CPU interface's support. + */ + if (cpu_isar_feature(aa64_nmi, cpu)) { + gcs->nmi_support = true; + define_arm_cp_regs(cpu, gicv3_cpuif_gicv3_nmi_reginfo); + } - /* - * If the CPU implements FEAT_NMI and FEAT_GICv3 it must also - * implement FEAT_GICv3_NMI, which is the CPU interface part - * of NMI support. This is distinct from whether the GIC proper - * (redistributors and distributor) have NMI support. In QEMU - * that is a property of the GIC device in s->nmi_support; - * cs->nmi_support indicates the CPU interface's support. - */ - if (cpu_isar_feature(aa64_nmi, cpu)) { - cs->nmi_support = true; - define_arm_cp_regs(cpu, gicv3_cpuif_gicv3_nmi_reginfo); - } + /* + * The CPU implementation specifies the number of supported + * bits of physical priority. For backwards compatibility + * of migration, we have a compat property that forces use + * of 8 priority bits regardless of what the CPU really has. + */ + if (gcs->gic->force_8bit_prio) { + gcs->pribits = 8; + } else { + gcs->pribits = cpu->gic_pribits ?: 5; + } - /* - * The CPU implementation specifies the number of supported - * bits of physical priority. For backwards compatibility - * of migration, we have a compat property that forces use - * of 8 priority bits regardless of what the CPU really has. - */ - if (s->force_8bit_prio) { - cs->pribits = 8; - } else { - cs->pribits = cpu->gic_pribits ?: 5; - } + /* + * The GICv3 has separate ID register fields for virtual priority + * and preemption bit values, but only a single ID register field + * for the physical priority bits. The preemption bit count is + * always the same as the priority bit count, except that 8 bits + * of priority means 7 preemption bits. We precalculate the + * preemption bits because it simplifies the code and makes the + * parallels between the virtual and physical bits of the GIC + * a bit clearer. + */ + gcs->prebits = gcs->pribits; + if (gcs->prebits == 8) { + gcs->prebits--; + } + /* + * Check that CPU code defining pribits didn't violate + * architectural constraints our implementation relies on. + */ + g_assert(gcs->pribits >= 4 && gcs->pribits <= 8); - /* - * The GICv3 has separate ID register fields for virtual priority - * and preemption bit values, but only a single ID register field - * for the physical priority bits. The preemption bit count is - * always the same as the priority bit count, except that 8 bits - * of priority means 7 preemption bits. We precalculate the - * preemption bits because it simplifies the code and makes the - * parallels between the virtual and physical bits of the GIC - * a bit clearer. - */ - cs->prebits = cs->pribits; - if (cs->prebits == 8) { - cs->prebits--; - } - /* - * Check that CPU code defining pribits didn't violate - * architectural constraints our implementation relies on. - */ - g_assert(cs->pribits >= 4 && cs->pribits <= 8); + /* + * gicv3_cpuif_reginfo[] defines ICC_AP*R0_EL1; add definitions + * for ICC_AP*R{1,2,3}_EL1 if the prebits value requires them. + */ + if (gcs->prebits >= 6) { + define_arm_cp_regs(cpu, gicv3_cpuif_icc_apxr1_reginfo); + } + if (gcs->prebits == 7) { + define_arm_cp_regs(cpu, gicv3_cpuif_icc_apxr23_reginfo); + } - /* - * gicv3_cpuif_reginfo[] defines ICC_AP*R0_EL1; add definitions - * for ICC_AP*R{1,2,3}_EL1 if the prebits value requires them. - */ - if (cs->prebits >= 6) { - define_arm_cp_regs(cpu, gicv3_cpuif_icc_apxr1_reginfo); - } - if (cs->prebits == 7) { - define_arm_cp_regs(cpu, gicv3_cpuif_icc_apxr23_reginfo); - } + if (arm_feature(&cpu->env, ARM_FEATURE_EL2)) { + int j; - if (arm_feature(&cpu->env, ARM_FEATURE_EL2)) { - int j; + gcs->num_list_regs = cpu->gic_num_lrs ?: 4; + gcs->vpribits = cpu->gic_vpribits ?: 5; + gcs->vprebits = cpu->gic_vprebits ?: 5; - cs->num_list_regs = cpu->gic_num_lrs ?: 4; - cs->vpribits = cpu->gic_vpribits ?: 5; - cs->vprebits = cpu->gic_vprebits ?: 5; - /* Check against architectural constraints: getting these - * wrong would be a bug in the CPU code defining these, - * and the implementation relies on them holding. - */ - g_assert(cs->vprebits <= cs->vpribits); - g_assert(cs->vprebits >= 5 && cs->vprebits <= 7); - g_assert(cs->vpribits >= 5 && cs->vpribits <= 8); + /* Check against architectural constraints: getting these + * wrong would be a bug in the CPU code defining these, + * and the implementation relies on them holding. + */ + g_assert(gcs->vprebits <= gcs->vpribits); + g_assert(gcs->vprebits >= 5 && gcs->vprebits <= 7); + g_assert(gcs->vpribits >= 5 && gcs->vpribits <= 8); - define_arm_cp_regs(cpu, gicv3_cpuif_hcr_reginfo); + define_arm_cp_regs(cpu, gicv3_cpuif_hcr_reginfo); - for (j = 0; j < cs->num_list_regs; j++) { - /* Note that the AArch64 LRs are 64-bit; the AArch32 LRs - * are split into two cp15 regs, LR (the low part, with the - * same encoding as the AArch64 LR) and LRC (the high part). - */ - ARMCPRegInfo lr_regset[] = { - { .name = "ICH_LRn_EL2", .state = ARM_CP_STATE_BOTH, - .opc0 = 3, .opc1 = 4, .crn = 12, - .crm = 12 + (j >> 3), .opc2 = j & 7, - .type = ARM_CP_IO | ARM_CP_NO_RAW, - .nv2_redirect_offset = 0x400 + 8 * j, - .access = PL2_RW, - .readfn = ich_lr_read, - .writefn = ich_lr_write, - }, - { .name = "ICH_LRCn_EL2", .state = ARM_CP_STATE_AA32, - .cp = 15, .opc1 = 4, .crn = 12, - .crm = 14 + (j >> 3), .opc2 = j & 7, - .type = ARM_CP_IO | ARM_CP_NO_RAW, - .access = PL2_RW, - .readfn = ich_lr_read, - .writefn = ich_lr_write, - }, - }; - define_arm_cp_regs(cpu, lr_regset); - } - if (cs->vprebits >= 6) { - define_arm_cp_regs(cpu, gicv3_cpuif_ich_apxr1_reginfo); - } - if (cs->vprebits == 7) { - define_arm_cp_regs(cpu, gicv3_cpuif_ich_apxr23_reginfo); - } + for (j = 0; j < gcs->num_list_regs; j++) { + /* Note that the AArch64 LRs are 64-bit; the AArch32 LRs + * are split into two cp15 regs, LR (the low part, with the + * same encoding as the AArch64 LR) and LRC (the high part). + */ + ARMCPRegInfo lr_regset[] = { + { .name = "ICH_LRn_EL2", .state = ARM_CP_STATE_BOTH, + .opc0 = 3, .opc1 = 4, .crn = 12, + .crm = 12 + (j >> 3), .opc2 = j & 7, + .type = ARM_CP_IO | ARM_CP_NO_RAW, + .nv2_redirect_offset = 0x400 + 8 * j, + .access = PL2_RW, + .readfn = ich_lr_read, + .writefn = ich_lr_write, + }, + { .name = "ICH_LRCn_EL2", .state = ARM_CP_STATE_AA32, + .cp = 15, .opc1 = 4, .crn = 12, + .crm = 14 + (j >> 3), .opc2 = j & 7, + .type = ARM_CP_IO | ARM_CP_NO_RAW, + .access = PL2_RW, + .readfn = ich_lr_read, + .writefn = ich_lr_write, + }, + }; + define_arm_cp_regs(cpu, lr_regset); + } + if (gcs->vprebits >= 6) { + define_arm_cp_regs(cpu, gicv3_cpuif_ich_apxr1_reginfo); + } + if (gcs->vprebits == 7) { + define_arm_cp_regs(cpu, gicv3_cpuif_ich_apxr23_reginfo); } if (tcg_enabled() || qtest_enabled()) { /* @@ -3177,10 +3172,26 @@ void gicv3_init_cpuif(GICv3State *s) * state only changes on EL changes involving EL2 or EL3, so for * the non-TCG case this is OK, as EL2 and EL3 can't exist. */ - arm_register_el_change_hook(cpu, gicv3_cpuif_el_change_hook, cs); + arm_register_el_change_hook(cpu, gicv3_cpuif_el_change_hook, gcs); } else { assert(!arm_feature(&cpu->env, ARM_FEATURE_EL2)); assert(!arm_feature(&cpu->env, ARM_FEATURE_EL3)); } } } + +void gicv3_init_cpuif(GICv3State *s) +{ + /* Called from the GICv3 realize function; register our system + * registers with the CPU + */ + int i; + + for (i = 0; i < s->num_cpu; i++) { + ARMCPU *cpu = ARM_CPU(qemu_get_cpu(i)); + + if (qemu_enabled_cpu(CPU(cpu))) { + gicv3_init_cpu_reginfo(CPU(cpu)); + } + } +} diff --git a/hw/intc/arm_gicv3_kvm.c b/hw/intc/arm_gicv3_kvm.c index 8dbbd79e1b..1c8e880117 100644 --- a/hw/intc/arm_gicv3_kvm.c +++ b/hw/intc/arm_gicv3_kvm.c @@ -804,6 +804,10 @@ static void vm_change_state_handler(void *opaque, bool running, } } +static void kvm_gicv3_init_cpu_reginfo(CPUState *cs) +{ + define_arm_cp_regs(ARM_CPU(cs), gicv3_cpuif_reginfo); +} static void kvm_arm_gicv3_realize(DeviceState *dev, Error **errp) { @@ -842,7 +846,7 @@ static void kvm_arm_gicv3_realize(DeviceState *dev, Error **errp) for (i = 0; i < s->num_cpu; i++) { CPUState *cs = qemu_get_cpu(i); if (qemu_enabled_cpu(cs)) { - define_arm_cp_regs(ARM_CPU(cs), gicv3_cpuif_reginfo); + kvm_gicv3_init_cpu_reginfo(cs); } } @@ -930,6 +934,7 @@ static void kvm_arm_gicv3_class_init(ObjectClass *klass, void *data) agcc->pre_save = kvm_arm_gicv3_get; agcc->post_load = kvm_arm_gicv3_put; + agcc->init_cpu_reginfo = kvm_gicv3_init_cpu_reginfo; device_class_set_parent_realize(dc, kvm_arm_gicv3_realize, &kgc->parent_realize); resettable_class_set_parent_phases(rc, NULL, kvm_arm_gicv3_reset_hold, NULL, diff --git a/hw/intc/gicv3_internal.h b/hw/intc/gicv3_internal.h index 42441c19c6..e568eef57f 100644 --- a/hw/intc/gicv3_internal.h +++ b/hw/intc/gicv3_internal.h @@ -722,6 +722,7 @@ void gicv3_redist_vinvall(GICv3CPUState *cs, uint64_t vptaddr); void gicv3_redist_send_sgi(GICv3CPUState *cs, int grp, int irq, bool ns); void gicv3_init_cpuif(GICv3State *s); +void gicv3_init_cpu_reginfo(CPUState *cs); /** * gicv3_cpuif_update: diff --git a/include/hw/intc/arm_gicv3_common.h b/include/hw/intc/arm_gicv3_common.h index 496b198016..b63abeae82 100644 --- a/include/hw/intc/arm_gicv3_common.h +++ b/include/hw/intc/arm_gicv3_common.h @@ -339,6 +339,7 @@ struct ARMGICv3CommonClass { void (*pre_save)(GICv3State *s); void (*post_load)(GICv3State *s); + void (*init_cpu_reginfo)(CPUState *cs); }; void gicv3_init_irqs_and_mmio(GICv3State *s, qemu_irq_handler handler, From patchwork Thu Jun 13 23:36:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697664 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 083BCC27C75 for ; Thu, 13 Jun 2024 23:46:42 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu98-0004WL-6Y; Thu, 13 Jun 2024 19:46:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu95-0004TR-4q; Thu, 13 Jun 2024 19:46:19 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu91-0004aO-E9; Thu, 13 Jun 2024 19:46:18 -0400 Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0fCP3vpSz6H784; Fri, 14 Jun 2024 07:44:49 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 046F5140594; Fri, 14 Jun 2024 07:46:13 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:45:50 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 22/29] arm/virt: Update the guest(via GED) about CPU hot-(un)plug events Date: Fri, 14 Jun 2024 00:36:32 +0100 Message-ID: <20240613233639.202896-23-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org During any vCPU hot-(un)plug, running guest VM needs to be intimated about the new vCPU being added or request the deletion of the vCPU which is already part of the guest VM. This is done using the ACPI GED event which eventually gets demultiplexed to a CPU hotplug event and further to specific hot-(un)plug event of a particular vCPU. This change adds the ACPI calls to the existing hot-(un)plug hooks to trigger ACPI GED events from QEMU to guest VM. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 34 ++++++++++++++++++++++++++-------- 1 file changed, 26 insertions(+), 8 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 9f7e07bd8e..4fa2b7d9e7 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -3169,6 +3169,7 @@ static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, VirtMachineState *vms = VIRT_MACHINE(hotplug_dev); MachineState *ms = MACHINE(hotplug_dev); CPUState *cs = CPU(dev); + Error *local_err = NULL; CPUArchId *cpu_slot; /* insert the cold/hot-plugged vcpu in the slot */ @@ -3181,12 +3182,18 @@ static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, * plugged, guest is also notified. */ if (vms->acpi_dev) { - /* TODO: update acpi hotplug state. Send cpu hotplug event to guest */ + HotplugHandlerClass *hhc; + /* update acpi hotplug state and send cpu hotplug event to guest */ + hhc = HOTPLUG_HANDLER_GET_CLASS(vms->acpi_dev); + hhc->plug(HOTPLUG_HANDLER(vms->acpi_dev), dev, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } /* TODO: register cpu for reset & update F/W info for the next boot */ } cs->disabled = false; - return; } static void virt_cpu_unplug_request(HotplugHandler *hotplug_dev, @@ -3194,8 +3201,10 @@ static void virt_cpu_unplug_request(HotplugHandler *hotplug_dev, { MachineClass *mc = MACHINE_GET_CLASS(qdev_get_machine()); VirtMachineState *vms = VIRT_MACHINE(hotplug_dev); + HotplugHandlerClass *hhc; ARMCPU *cpu = ARM_CPU(dev); CPUState *cs = CPU(dev); + Error *local_err = NULL; if (!vms->acpi_dev || !dev->realized) { error_setg(errp, "GED does not exists or device is not realized!"); @@ -3214,9 +3223,12 @@ static void virt_cpu_unplug_request(HotplugHandler *hotplug_dev, return; } - /* TODO: request cpu hotplug from guest */ - - return; + /* request cpu hotplug from guest */ + hhc = HOTPLUG_HANDLER_GET_CLASS(vms->acpi_dev); + hhc->unplug_request(HOTPLUG_HANDLER(vms->acpi_dev), dev, &local_err); + if (local_err) { + error_propagate(errp, local_err); + } } static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, @@ -3224,7 +3236,9 @@ static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, { VirtMachineState *vms = VIRT_MACHINE(hotplug_dev); MachineState *ms = MACHINE(hotplug_dev); + HotplugHandlerClass *hhc; CPUState *cs = CPU(dev); + Error *local_err = NULL; CPUArchId *cpu_slot; if (!vms->acpi_dev || !dev->realized) { @@ -3234,7 +3248,13 @@ static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, cpu_slot = virt_find_cpu_slot(ms, cs->cpu_index); - /* TODO: update the acpi cpu hotplug state for cpu hot-unplug */ + /* update the acpi cpu hotplug state for cpu hot-unplug */ + hhc = HOTPLUG_HANDLER_GET_CLASS(vms->acpi_dev); + hhc->unplug(HOTPLUG_HANDLER(vms->acpi_dev), dev, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } unwire_gic_cpu_irqs(vms, cs); virt_update_gic(vms, cs); @@ -3246,8 +3266,6 @@ static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, cpu_slot->cpu = NULL; cs->disabled = true; - - return; } static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev, From patchwork Thu Jun 13 23:36:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697665 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 72208C27C75 for ; Thu, 13 Jun 2024 23:46:58 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu9f-0005IW-4o; Thu, 13 Jun 2024 19:46:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu9Q-0004w7-1l; Thu, 13 Jun 2024 19:46:44 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu9N-0004dI-Ro; Thu, 13 Jun 2024 19:46:39 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0f812cTjz6HJTP; Fri, 14 Jun 2024 07:41:53 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id A30AA140AE5; Fri, 14 Jun 2024 07:46:35 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:46:13 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 23/29] hw/arm: Changes required for reset and to support next boot Date: Fri, 14 Jun 2024 00:36:33 +0100 Message-ID: <20240613233639.202896-24-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Updates the firmware config with the next boot cpus information and also registers the reset callback to be called when guest reboots to reset the cpu. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/boot.c | 2 +- hw/arm/virt.c | 17 ++++++++++++++--- include/hw/arm/boot.h | 2 ++ include/hw/arm/virt.h | 1 + 4 files changed, 18 insertions(+), 4 deletions(-) diff --git a/hw/arm/boot.c b/hw/arm/boot.c index d480a7da02..cb5c1e4848 100644 --- a/hw/arm/boot.c +++ b/hw/arm/boot.c @@ -682,7 +682,7 @@ fail: return -1; } -static void do_cpu_reset(void *opaque) +void do_cpu_reset(void *opaque) { ARMCPU *cpu = opaque; CPUState *cs = CPU(cpu); diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 4fa2b7d9e7..a2200099a1 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -45,6 +45,8 @@ #include "sysemu/device_tree.h" #include "sysemu/numa.h" #include "sysemu/runstate.h" +#include "sysemu/reset.h" +#include "sysemu/sysemu.h" #include "sysemu/tpm.h" #include "sysemu/tcg.h" #include "sysemu/kvm.h" @@ -1405,7 +1407,7 @@ static FWCfgState *create_fw_cfg(const VirtMachineState *vms, AddressSpace *as) char *nodename; fw_cfg = fw_cfg_init_mem_wide(base + 8, base, 8, base + 16, as); - fw_cfg_add_i16(fw_cfg, FW_CFG_NB_CPUS, (uint16_t)ms->smp.cpus); + fw_cfg_add_i16(fw_cfg, FW_CFG_NB_CPUS, vms->boot_cpus); nodename = g_strdup_printf("/fw-cfg@%" PRIx64, base); qemu_fdt_add_subnode(ms->fdt, nodename); @@ -3190,9 +3192,14 @@ static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, error_propagate(errp, local_err); return; } - /* TODO: register cpu for reset & update F/W info for the next boot */ + /* register this cpu for reset & update F/W info for the next boot */ + qemu_register_reset(do_cpu_reset, ARM_CPU(cs)); } + vms->boot_cpus++; + if (vms->fw_cfg) { + fw_cfg_modify_i16(vms->fw_cfg, FW_CFG_NB_CPUS, vms->boot_cpus); + } cs->disabled = false; } @@ -3259,7 +3266,11 @@ static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, unwire_gic_cpu_irqs(vms, cs); virt_update_gic(vms, cs); - /* TODO: unregister cpu for reset & update F/W info for the next boot */ + qemu_unregister_reset(do_cpu_reset, ARM_CPU(cs)); + vms->boot_cpus--; + if (vms->fw_cfg) { + fw_cfg_modify_i16(vms->fw_cfg, FW_CFG_NB_CPUS, vms->boot_cpus); + } qobject_unref(dev->opts); dev->opts = NULL; diff --git a/include/hw/arm/boot.h b/include/hw/arm/boot.h index 80c492d742..f81326a1dc 100644 --- a/include/hw/arm/boot.h +++ b/include/hw/arm/boot.h @@ -178,6 +178,8 @@ AddressSpace *arm_boot_address_space(ARMCPU *cpu, int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, hwaddr addr_limit, AddressSpace *as, MachineState *ms); +void do_cpu_reset(void *opaque); + /* Write a secure board setup routine with a dummy handler for SMCs */ void arm_write_secure_board_setup_dummy_smc(ARMCPU *cpu, const struct arm_boot_info *info, diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index 9c728ba042..8dce426cc0 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -167,6 +167,7 @@ struct VirtMachineState { MemMapEntry *memmap; char *pciehb_nodename; const int *irqmap; + uint16_t boot_cpus; int fdt_size; uint32_t clock_phandle; uint32_t gic_phandle; From patchwork Thu Jun 13 23:36:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697666 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2E7C3C27C4F for ; Thu, 13 Jun 2024 23:47:26 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu9q-0005zV-JB; Thu, 13 Jun 2024 19:47:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu9n-0005q6-1u; Thu, 13 Jun 2024 19:47:03 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu9k-0004kI-OV; Thu, 13 Jun 2024 19:47:02 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0f8S0zM6z67j6n; Fri, 14 Jun 2024 07:42:16 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 7279014065C; Fri, 14 Jun 2024 07:46:58 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:46:35 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 24/29] target/arm: Add support of *unrealize* ARMCPU during vCPU Hot-unplug Date: Fri, 14 Jun 2024 00:36:34 +0100 Message-ID: <20240613233639.202896-25-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org vCPU Hot-unplug will result in QOM CPU object unrealization which will do away with all the vCPU thread creations, allocations, registrations that happened as part of the realization process. This change introduces the ARM CPU unrealize function taking care of exactly that. Note, initialized KVM vCPUs are not destroyed in host KVM but their Qemu context is parked at the QEMU KVM layer. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta Reported-by: Vishnu Pajjuri [VP: Identified CPU stall issue & suggested probable fix] Signed-off-by: Salil Mehta --- target/arm/cpu.c | 101 +++++++++++++++++++++++++++++++++++++++++ target/arm/cpu.h | 14 ++++++ target/arm/gdbstub.c | 6 +++ target/arm/helper.c | 25 ++++++++++ target/arm/internals.h | 3 ++ target/arm/kvm.c | 5 ++ 6 files changed, 154 insertions(+) diff --git a/target/arm/cpu.c b/target/arm/cpu.c index c92162fa97..a3dc669309 100644 --- a/target/arm/cpu.c +++ b/target/arm/cpu.c @@ -157,6 +157,16 @@ void arm_register_pre_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, QLIST_INSERT_HEAD(&cpu->pre_el_change_hooks, entry, node); } +void arm_unregister_pre_el_change_hooks(ARMCPU *cpu) +{ + ARMELChangeHook *entry, *next; + + QLIST_FOREACH_SAFE(entry, &cpu->pre_el_change_hooks, node, next) { + QLIST_REMOVE(entry, node); + g_free(entry); + } +} + void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, void *opaque) { @@ -168,6 +178,16 @@ void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, QLIST_INSERT_HEAD(&cpu->el_change_hooks, entry, node); } +void arm_unregister_el_change_hooks(ARMCPU *cpu) +{ + ARMELChangeHook *entry, *next; + + QLIST_FOREACH_SAFE(entry, &cpu->el_change_hooks, node, next) { + QLIST_REMOVE(entry, node); + g_free(entry); + } +} + static void cp_reg_reset(gpointer key, gpointer value, gpointer opaque) { /* Reset a single ARMCPRegInfo register */ @@ -2552,6 +2572,85 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp) acc->parent_realize(dev, errp); } +static void arm_cpu_unrealizefn(DeviceState *dev) +{ + ARMCPUClass *acc = ARM_CPU_GET_CLASS(dev); + ARMCPU *cpu = ARM_CPU(dev); + CPUARMState *env = &cpu->env; + CPUState *cs = CPU(dev); + bool has_secure; + + has_secure = cpu->has_el3 || arm_feature(env, ARM_FEATURE_M_SECURITY); + + /* rock 'n' un-roll, whatever happened in the arm_cpu_realizefn cleanly */ + cpu_address_space_destroy(cs, ARMASIdx_NS); + + if (cpu->tag_memory != NULL) { + cpu_address_space_destroy(cs, ARMASIdx_TagNS); + if (has_secure) { + cpu_address_space_destroy(cs, ARMASIdx_TagS); + } + } + + if (has_secure) { + cpu_address_space_destroy(cs, ARMASIdx_S); + } + + destroy_cpreg_list(cpu); + arm_cpu_unregister_gdb_regs(cpu); + unregister_cp_regs_for_features(cpu); + + if (cpu->sau_sregion && arm_feature(env, ARM_FEATURE_M_SECURITY)) { + g_free(env->sau.rbar); + g_free(env->sau.rlar); + } + + if (arm_feature(env, ARM_FEATURE_PMSA) && + arm_feature(env, ARM_FEATURE_V7) && + cpu->pmsav7_dregion) { + if (arm_feature(env, ARM_FEATURE_V8)) { + g_free(env->pmsav8.rbar[M_REG_NS]); + g_free(env->pmsav8.rlar[M_REG_NS]); + if (arm_feature(env, ARM_FEATURE_M_SECURITY)) { + g_free(env->pmsav8.rbar[M_REG_S]); + g_free(env->pmsav8.rlar[M_REG_S]); + } + } else { + g_free(env->pmsav7.drbar); + g_free(env->pmsav7.drsr); + g_free(env->pmsav7.dracr); + } + if (cpu->pmsav8r_hdregion) { + g_free(env->pmsav8.hprbar); + g_free(env->pmsav8.hprlar); + } + } + + if (arm_feature(env, ARM_FEATURE_PMU)) { + if (!kvm_enabled()) { + arm_unregister_pre_el_change_hooks(cpu); + arm_unregister_el_change_hooks(cpu); + } + +#ifndef CONFIG_USER_ONLY + if (cpu->pmu_timer) { + timer_del(cpu->pmu_timer); + } +#endif + } + + cpu_remove_sync(CPU(dev)); + acc->parent_unrealize(dev); + +#ifndef CONFIG_USER_ONLY + timer_del(cpu->gt_timer[GTIMER_PHYS]); + timer_del(cpu->gt_timer[GTIMER_VIRT]); + timer_del(cpu->gt_timer[GTIMER_HYP]); + timer_del(cpu->gt_timer[GTIMER_SEC]); + timer_del(cpu->gt_timer[GTIMER_HYPVIRT]); +#endif +} + static ObjectClass *arm_cpu_class_by_name(const char *cpu_model) { ObjectClass *oc; @@ -2654,6 +2753,8 @@ static void arm_cpu_class_init(ObjectClass *oc, void *data) device_class_set_parent_realize(dc, arm_cpu_realizefn, &acc->parent_realize); + device_class_set_parent_unrealize(dc, arm_cpu_unrealizefn, + &acc->parent_unrealize); device_class_set_props(dc, arm_cpu_properties); diff --git a/target/arm/cpu.h b/target/arm/cpu.h index 208c719db3..a4a7555f7e 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -1118,6 +1118,7 @@ struct ARMCPUClass { const ARMCPUInfo *info; DeviceRealize parent_realize; + DeviceUnrealize parent_unrealize; ResettablePhases parent_phases; }; @@ -3228,6 +3229,13 @@ static inline AddressSpace *arm_addressspace(CPUState *cs, MemTxAttrs attrs) */ void arm_register_pre_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, void *opaque); +/** + * arm_unregister_pre_el_change_hook: + * unregister all pre EL change hook functions. Generally called during + * unrealize'ing leg + */ +void arm_unregister_pre_el_change_hooks(ARMCPU *cpu); + /** * arm_register_el_change_hook: * Register a hook function which will be called immediately after this @@ -3240,6 +3248,12 @@ void arm_register_pre_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, */ void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, void *opaque); +/** + * arm_unregister_el_change_hook: + * unregister all EL change hook functions. Generally called during + * unrealize'ing leg + */ +void arm_unregister_el_change_hooks(ARMCPU *cpu); /** * arm_rebuild_hflags: diff --git a/target/arm/gdbstub.c b/target/arm/gdbstub.c index a3bb73cfa7..948e40b981 100644 --- a/target/arm/gdbstub.c +++ b/target/arm/gdbstub.c @@ -555,3 +555,9 @@ void arm_cpu_register_gdb_regs_for_features(ARMCPU *cpu) } #endif /* CONFIG_TCG */ } + +void arm_cpu_unregister_gdb_regs(ARMCPU *cpu) +{ + CPUState *cs = CPU(cpu); + gdb_unregister_coprocessor_all(cs); +} diff --git a/target/arm/helper.c b/target/arm/helper.c index 7587635960..9a2468347a 100644 --- a/target/arm/helper.c +++ b/target/arm/helper.c @@ -264,6 +264,19 @@ void init_cpreg_list(ARMCPU *cpu) g_list_free(keys); } +void destroy_cpreg_list(ARMCPU *cpu) +{ + assert(cpu->cpreg_indexes); + assert(cpu->cpreg_values); + assert(cpu->cpreg_vmstate_indexes); + assert(cpu->cpreg_vmstate_values); + + g_free(cpu->cpreg_indexes); + g_free(cpu->cpreg_values); + g_free(cpu->cpreg_vmstate_indexes); + g_free(cpu->cpreg_vmstate_values); +} + static bool arm_pan_enabled(CPUARMState *env) { if (is_a64(env)) { @@ -9987,6 +10000,18 @@ void register_cp_regs_for_features(ARMCPU *cpu) #endif } +void unregister_cp_regs_for_features(ARMCPU *cpu) +{ + CPUARMState *env = &cpu->env; + if (arm_feature(env, ARM_FEATURE_M)) { + /* M profile has no coprocessor registers */ + return; + } + + /* empty it all. unregister all the coprocessor registers */ + g_hash_table_remove_all(cpu->cp_regs); +} + /* * Private utility function for define_one_arm_cp_reg_with_opaque(): * add a single reginfo struct to the hash table. diff --git a/target/arm/internals.h b/target/arm/internals.h index ee3ebd383e..34dab0bb02 100644 --- a/target/arm/internals.h +++ b/target/arm/internals.h @@ -353,9 +353,12 @@ void arm_cpu_register(const ARMCPUInfo *info); void aarch64_cpu_register(const ARMCPUInfo *info); void register_cp_regs_for_features(ARMCPU *cpu); +void unregister_cp_regs_for_features(ARMCPU *cpu); void init_cpreg_list(ARMCPU *cpu); +void destroy_cpreg_list(ARMCPU *cpu); void arm_cpu_register_gdb_regs_for_features(ARMCPU *cpu); +void arm_cpu_unregister_gdb_regs(ARMCPU *cpu); void arm_translate_init(void); void arm_restore_state_to_opc(CPUState *cs, diff --git a/target/arm/kvm.c b/target/arm/kvm.c index 01c83c1994..1121771c4a 100644 --- a/target/arm/kvm.c +++ b/target/arm/kvm.c @@ -1988,6 +1988,11 @@ int kvm_arch_init_vcpu(CPUState *cs) int kvm_arch_destroy_vcpu(CPUState *cs) { + /* vCPUs which are yet to be realized will not have handler */ + if (cs->thread_id) { + qemu_del_vm_change_state_handler(cs->vmcse); + } + return 0; } From patchwork Thu Jun 13 23:36:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697667 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 684B2C27C75 for ; Thu, 13 Jun 2024 23:47:47 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHuAC-0006qG-Cg; Thu, 13 Jun 2024 19:47:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHuAA-0006kk-7s; Thu, 13 Jun 2024 19:47:26 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHuA8-0004nN-Cw; Thu, 13 Jun 2024 19:47:25 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0f8t5WZ0z6H8xV; Fri, 14 Jun 2024 07:42:38 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 1A470140AE5; Fri, 14 Jun 2024 07:47:21 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:46:58 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 25/29] target/arm/kvm: Write CPU state back to KVM on reset Date: Fri, 14 Jun 2024 00:36:35 +0100 Message-ID: <20240613233639.202896-26-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jean-Philippe Brucker When a KVM vCPU is reset following a PSCI CPU_ON call, its power state is not synchronized with KVM at the moment. Because the vCPU is not marked dirty, we miss the call to kvm_arch_put_registers() that writes to KVM's MP_STATE. Force mp_state synchronization. Signed-off-by: Jean-Philippe Brucker Signed-off-by: Salil Mehta --- target/arm/kvm.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/target/arm/kvm.c b/target/arm/kvm.c index 1121771c4a..7acd83ce64 100644 --- a/target/arm/kvm.c +++ b/target/arm/kvm.c @@ -980,6 +980,7 @@ void kvm_arm_cpu_post_load(ARMCPU *cpu) void kvm_arm_reset_vcpu(ARMCPU *cpu) { int ret; + CPUState *cs = CPU(cpu); /* Re-init VCPU so that all registers are set to * their respective reset values. @@ -1001,6 +1002,12 @@ void kvm_arm_reset_vcpu(ARMCPU *cpu) * for the same reason we do so in kvm_arch_get_registers(). */ write_list_to_cpustate(cpu); + + /* + * Ensure we call kvm_arch_put_registers(). The vCPU isn't marked dirty if + * it was parked in KVM and is now booting from a PSCI CPU_ON call. + */ + cs->vcpu_dirty = true; } void kvm_arm_create_host_vcpu(ARMCPU *cpu) From patchwork Fri Jun 14 00:15:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697686 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 20666C27C4F for ; Fri, 14 Jun 2024 00:17:22 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHuc7-00028b-7n; Thu, 13 Jun 2024 20:16:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHuc4-00025q-1W; Thu, 13 Jun 2024 20:16:16 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHuc0-0001wy-FH; Thu, 13 Jun 2024 20:16:15 -0400 Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0fsr72RRz6H7Jk; Fri, 14 Jun 2024 08:14:40 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 783A7140C98; Fri, 14 Jun 2024 08:16:04 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 01:15:42 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 26/29] target/arm/kvm, tcg: Register/Handle SMCCC hypercall exits to VMM/Qemu Date: Fri, 14 Jun 2024 01:15:10 +0100 Message-ID: <20240614001510.202991-1-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Author Salil Mehta Add registration and Handling of HVC/SMC hypercall exits to VMM Co-developed-by: Jean-Philippe Brucker Signed-off-by: Jean-Philippe Brucker Signed-off-by: Salil Mehta --- target/arm/arm-powerctl.c | 51 ++++++++++++++++---- target/arm/helper.c | 2 +- target/arm/internals.h | 11 ----- target/arm/kvm.c | 93 +++++++++++++++++++++++++++++++++++++ target/arm/kvm_arm.h | 14 ++++++ target/arm/meson.build | 1 + target/arm/{tcg => }/psci.c | 8 ++++ target/arm/tcg/meson.build | 4 -- 8 files changed, 159 insertions(+), 25 deletions(-) rename target/arm/{tcg => }/psci.c (97%) diff --git a/target/arm/arm-powerctl.c b/target/arm/arm-powerctl.c index 2b2055c6ac..f567478e89 100644 --- a/target/arm/arm-powerctl.c +++ b/target/arm/arm-powerctl.c @@ -17,6 +17,7 @@ #include "qemu/main-loop.h" #include "sysemu/tcg.h" #include "target/arm/multiprocessing.h" +#include "hw/boards.h" #ifndef DEBUG_ARM_POWERCTL #define DEBUG_ARM_POWERCTL 0 @@ -29,18 +30,37 @@ } \ } while (0) +static CPUArchId *arm_get_archid_by_id(uint64_t id) +{ + int n; + CPUArchId *arch_id; + MachineState *ms = MACHINE(qdev_get_machine()); + + /* + * At this point disabled CPUs don't have a CPUState, but their CPUArchId + * exists. + * + * TODO: Is arch_id == mp_affinity? This needs work. + */ + for (n = 0; n < ms->possible_cpus->len; n++) { + arch_id = &ms->possible_cpus->cpus[n]; + + if (arch_id->arch_id == id) { + return arch_id; + } + } + return NULL; +} + CPUState *arm_get_cpu_by_id(uint64_t id) { - CPUState *cpu; + CPUArchId *arch_id; DPRINTF("cpu %" PRId64 "\n", id); - CPU_FOREACH(cpu) { - ARMCPU *armcpu = ARM_CPU(cpu); - - if (arm_cpu_mp_affinity(armcpu) == id) { - return cpu; - } + arch_id = arm_get_archid_by_id(id); + if (arch_id && arch_id->cpu) { + return CPU(arch_id->cpu); } qemu_log_mask(LOG_GUEST_ERROR, @@ -98,6 +118,7 @@ int arm_set_cpu_on(uint64_t cpuid, uint64_t entry, uint64_t context_id, { CPUState *target_cpu_state; ARMCPU *target_cpu; + CPUArchId *arch_id; struct CpuOnInfo *info; assert(bql_locked()); @@ -118,12 +139,24 @@ int arm_set_cpu_on(uint64_t cpuid, uint64_t entry, uint64_t context_id, } /* Retrieve the cpu we are powering up */ - target_cpu_state = arm_get_cpu_by_id(cpuid); - if (!target_cpu_state) { + arch_id = arm_get_archid_by_id(cpuid); + if (!arch_id) { /* The cpu was not found */ return QEMU_ARM_POWERCTL_INVALID_PARAM; } + target_cpu_state = CPU(arch_id->cpu); + if (!qemu_enabled_cpu(target_cpu_state)) { + /* + * The cpu is not plugged in or disabled. We should return appropriate + * value as introduced in DEN0022E PSCI 1.2 issue E + */ + qemu_log_mask(LOG_GUEST_ERROR, + "[ARM]%s: Denying attempt to online removed/disabled " + "CPU%" PRId64"\n", __func__, cpuid); + return QEMU_ARM_POWERCTL_IS_OFF; + } + target_cpu = ARM_CPU(target_cpu_state); if (target_cpu->power_state == PSCI_ON) { qemu_log_mask(LOG_GUEST_ERROR, diff --git a/target/arm/helper.c b/target/arm/helper.c index 9a2468347a..4ea0a42f52 100644 --- a/target/arm/helper.c +++ b/target/arm/helper.c @@ -11842,7 +11842,7 @@ void arm_cpu_do_interrupt(CPUState *cs) env->exception.syndrome); } - if (tcg_enabled() && arm_is_psci_call(cpu, cs->exception_index)) { + if (arm_is_psci_call(cpu, cs->exception_index)) { arm_handle_psci_call(cpu); qemu_log_mask(CPU_LOG_INT, "...handled as PSCI call\n"); return; diff --git a/target/arm/internals.h b/target/arm/internals.h index 34dab0bb02..f7b7f7966a 100644 --- a/target/arm/internals.h +++ b/target/arm/internals.h @@ -480,21 +480,10 @@ vaddr arm_adjust_watchpoint_address(CPUState *cs, vaddr addr, int len); /* Callback function for when a watchpoint or breakpoint triggers. */ void arm_debug_excp_handler(CPUState *cs); -#if defined(CONFIG_USER_ONLY) || !defined(CONFIG_TCG) -static inline bool arm_is_psci_call(ARMCPU *cpu, int excp_type) -{ - return false; -} -static inline void arm_handle_psci_call(ARMCPU *cpu) -{ - g_assert_not_reached(); -} -#else /* Return true if the r0/x0 value indicates that this SMC/HVC is a PSCI call. */ bool arm_is_psci_call(ARMCPU *cpu, int excp_type); /* Actually handle a PSCI call */ void arm_handle_psci_call(ARMCPU *cpu); -#endif /** * arm_clear_exclusive: clear the exclusive monitor diff --git a/target/arm/kvm.c b/target/arm/kvm.c index 7acd83ce64..eb1c623828 100644 --- a/target/arm/kvm.c +++ b/target/arm/kvm.c @@ -544,9 +544,51 @@ int kvm_arch_get_default_type(MachineState *ms) return fixed_ipa ? 0 : size; } +static bool kvm_arm_set_vm_attr(struct kvm_device_attr *attr, const char *name) +{ + int err; + + err = kvm_vm_ioctl(kvm_state, KVM_HAS_DEVICE_ATTR, attr); + if (err != 0) { + error_report("%s: KVM_HAS_DEVICE_ATTR: %s", name, strerror(-err)); + return false; + } + + err = kvm_vm_ioctl(kvm_state, KVM_SET_DEVICE_ATTR, attr); + if (err != 0) { + error_report("%s: KVM_SET_DEVICE_ATTR: %s", name, strerror(-err)); + return false; + } + + return true; +} + +int kvm_arm_set_smccc_filter(uint64_t func, uint8_t faction) +{ + struct kvm_smccc_filter filter = { + .base = func, + .nr_functions = 1, + .action = faction, + }; + struct kvm_device_attr attr = { + .group = KVM_ARM_VM_SMCCC_CTRL, + .attr = KVM_ARM_VM_SMCCC_FILTER, + .flags = 0, + .addr = (uintptr_t)&filter, + }; + + if (!kvm_arm_set_vm_attr(&attr, "SMCCC Filter")) { + error_report("failed to set SMCCC filter in KVM Host"); + return -1; + } + + return 0; +} + int kvm_arch_init(MachineState *ms, KVMState *s) { int ret = 0; + /* For ARM interrupt delivery is always asynchronous, * whether we are using an in-kernel VGIC or not. */ @@ -609,6 +651,22 @@ int kvm_arch_init(MachineState *ms, KVMState *s) hw_breakpoints = g_array_sized_new(true, true, sizeof(HWBreakpoint), max_hw_bps); + /* + * To be able to handle PSCI CPU ON calls in QEMU, we need to install SMCCC + * filter in the Host KVM. This is required to support features like + * virtual CPU Hotplug on ARM platforms. + */ + if (kvm_arm_set_smccc_filter(PSCI_0_2_FN64_CPU_ON, + KVM_SMCCC_FILTER_FWD_TO_USER)) { + error_report("CPU On PSCI-to-user-space fwd filter install failed"); + abort(); + } + if (kvm_arm_set_smccc_filter(PSCI_0_2_FN_CPU_OFF, + KVM_SMCCC_FILTER_FWD_TO_USER)) { + error_report("CPU Off PSCI-to-user-space fwd filter install failed"); + abort(); + } + return ret; } @@ -1459,6 +1517,38 @@ static bool kvm_arm_handle_debug(ARMCPU *cpu, return false; } +static int kvm_arm_handle_hypercall(CPUState *cs, struct kvm_run *run) +{ + ARMCPU *cpu = ARM_CPU(cs); + CPUARMState *env = &cpu->env; + + kvm_cpu_synchronize_state(cs); + + /* + * hard coding immediate to 0 as we dont expect non-zero value as of now + * This might change in future versions. Hence, KVM_GET_ONE_REG could be + * used in such cases but it must be enhanced then only synchronize will + * also fetch ESR_EL2 value. + */ + if (run->hypercall.flags == KVM_HYPERCALL_EXIT_SMC) { + cs->exception_index = EXCP_SMC; + env->exception.syndrome = syn_aa64_smc(0); + } else { + cs->exception_index = EXCP_HVC; + env->exception.syndrome = syn_aa64_hvc(0); + } + env->exception.target_el = 1; + bql_lock(); + arm_cpu_do_interrupt(cs); + bql_unlock(); + + /* + * For PSCI, exit the kvm_run loop and process the work. Especially + * important if this was a CPU_OFF command and we can't return to the guest. + */ + return EXCP_INTERRUPT; +} + int kvm_arch_handle_exit(CPUState *cs, struct kvm_run *run) { ARMCPU *cpu = ARM_CPU(cs); @@ -1475,6 +1565,9 @@ int kvm_arch_handle_exit(CPUState *cs, struct kvm_run *run) ret = kvm_arm_handle_dabt_nisv(cpu, run->arm_nisv.esr_iss, run->arm_nisv.fault_ipa); break; + case KVM_EXIT_HYPERCALL: + ret = kvm_arm_handle_hypercall(cs, run); + break; default: qemu_log_mask(LOG_UNIMP, "%s: un-handled exit reason %d\n", __func__, run->exit_reason); diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h index 0be7e896d2..b9c2b0f501 100644 --- a/target/arm/kvm_arm.h +++ b/target/arm/kvm_arm.h @@ -225,6 +225,15 @@ void kvm_arm_pvtime_init(ARMCPU *cpu, uint64_t ipa); int kvm_arm_set_irq(int cpu, int irqtype, int irq, int level); +/** + * kvm_arm_set_smccc_filter + * @func: funcion + * @faction: SMCCC filter action(handle, deny, fwd-to-user) to be deployed + * + * Sets the ARMs SMC-CC filter in KVM Host for selective hypercall exits + */ +int kvm_arm_set_smccc_filter(uint64_t func, uint8_t faction); + #else /* @@ -294,6 +303,11 @@ static inline uint32_t kvm_arm_sve_get_vls(ARMCPU *cpu) g_assert_not_reached(); } +static inline int kvm_arm_set_smccc_filter(uint64_t func, uint8_t faction) +{ + g_assert_not_reached(); +} + #endif #endif diff --git a/target/arm/meson.build b/target/arm/meson.build index 2e10464dbb..3e9f704f35 100644 --- a/target/arm/meson.build +++ b/target/arm/meson.build @@ -23,6 +23,7 @@ arm_system_ss.add(files( 'arm-qmp-cmds.c', 'cortex-regs.c', 'machine.c', + 'psci.c', 'ptw.c', )) diff --git a/target/arm/tcg/psci.c b/target/arm/psci.c similarity index 97% rename from target/arm/tcg/psci.c rename to target/arm/psci.c index 51d2ca3d30..b3fcb85079 100644 --- a/target/arm/tcg/psci.c +++ b/target/arm/psci.c @@ -21,7 +21,9 @@ #include "exec/helper-proto.h" #include "kvm-consts.h" #include "qemu/main-loop.h" +#include "qemu/error-report.h" #include "sysemu/runstate.h" +#include "sysemu/tcg.h" #include "internals.h" #include "arm-powerctl.h" #include "target/arm/multiprocessing.h" @@ -158,6 +160,11 @@ void arm_handle_psci_call(ARMCPU *cpu) case QEMU_PSCI_0_1_FN_CPU_SUSPEND: case QEMU_PSCI_0_2_FN_CPU_SUSPEND: case QEMU_PSCI_0_2_FN64_CPU_SUSPEND: + if (!tcg_enabled()) { + warn_report("CPU suspend not supported in non-tcg mode"); + break; + } +#ifdef CONFIG_TCG /* Affinity levels are not supported in QEMU */ if (param[1] & 0xfffe0000) { ret = QEMU_PSCI_RET_INVALID_PARAMS; @@ -170,6 +177,7 @@ void arm_handle_psci_call(ARMCPU *cpu) env->regs[0] = 0; } helper_wfi(env, 4); +#endif break; case QEMU_PSCI_1_0_FN_PSCI_FEATURES: switch (param[1]) { diff --git a/target/arm/tcg/meson.build b/target/arm/tcg/meson.build index 508932a249..5b43c84c40 100644 --- a/target/arm/tcg/meson.build +++ b/target/arm/tcg/meson.build @@ -54,9 +54,5 @@ arm_ss.add(when: 'TARGET_AARCH64', if_true: files( 'sve_helper.c', )) -arm_system_ss.add(files( - 'psci.c', -)) - arm_system_ss.add(when: 'CONFIG_ARM_V7M', if_true: files('cpu-v7m.c')) arm_user_ss.add(when: 'TARGET_AARCH64', if_false: files('cpu-v7m.c')) From patchwork Fri Jun 14 00:18:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id ED18AC27C4F for ; Fri, 14 Jun 2024 00:19:45 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHuf3-0003ga-MK; Thu, 13 Jun 2024 20:19:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHuf2-0003fq-2d; Thu, 13 Jun 2024 20:19:20 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHuf0-0002HR-1t; Thu, 13 Jun 2024 20:19:19 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0fxT4hWVz67MmR; Fri, 14 Jun 2024 08:17:49 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 2807714065C; Fri, 14 Jun 2024 08:19:13 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 01:18:50 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 27/29] hw/arm: Support hotplug capability check using _OSC method Date: Fri, 14 Jun 2024 01:18:36 +0100 Message-ID: <20240614001836.203018-1-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Physical CPU hotplug results in (un)setting of ACPI _STA.Present bit. AARCH64 platforms do not support physical CPU hotplug. Virtual CPU hotplug support being implemented toggles ACPI _STA.Enabled Bit to achieve hotplug functionality. This is not same as physical CPU hotplug support. In future, if ARM architecture supports physical CPU hotplug then the current design of virtual CPU hotplug can be used unchanged. Hence, there is a need for firmware/VMM/Qemu to support evaluation of platform wide capabilitiy related to the *type* of CPU hotplug support present on the platform. OSPM might need this during boot time to correctly initialize the CPUs and other related components in the kernel. NOTE: This implementation will be improved to add the support of *query* in the subsequent versions. This is very minimal support to assist kernel. ASL for the implemented _OSC method: Method (_OSC, 4, NotSerialized) // _OSC: Operating System Capabilities { CreateDWordField (Arg3, Zero, CDW1) If ((Arg0 == ToUUID ("0811b06e-4a27-44f9-8d60-3cbbc22e7b48") /* Platform-wide Capabilities */)) { CreateDWordField (Arg3, 0x04, CDW2) Local0 = CDW2 /* \_SB_._OSC.CDW2 */ If ((Arg1 != One)) { CDW1 |= 0x08 } Local0 &= 0x00800000 If ((CDW2 != Local0)) { CDW1 |= 0x10 } CDW2 = Local0 } Else { CDW1 |= 0x04 } Return (Arg3) } Signed-off-by: Salil Mehta --- hw/arm/virt-acpi-build.c | 52 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c index 4b4906f407..6cb613103f 100644 --- a/hw/arm/virt-acpi-build.c +++ b/hw/arm/virt-acpi-build.c @@ -818,6 +818,55 @@ static void build_fadt_rev6(GArray *table_data, BIOSLinker *linker, build_fadt(table_data, linker, &fadt, vms->oem_id, vms->oem_table_id); } +static void build_virt_osc_method(Aml *scope, VirtMachineState *vms) +{ + Aml *if_uuid, *else_uuid, *if_rev, *if_caps_masked, *method; + Aml *a_cdw1 = aml_name("CDW1"); + Aml *a_cdw2 = aml_local(0); + + method = aml_method("_OSC", 4, AML_NOTSERIALIZED); + aml_append(method, aml_create_dword_field(aml_arg(3), aml_int(0), "CDW1")); + + /* match UUID */ + if_uuid = aml_if(aml_equal( + aml_arg(0), aml_touuid("0811B06E-4A27-44F9-8D60-3CBBC22E7B48"))); + + aml_append(if_uuid, aml_create_dword_field(aml_arg(3), aml_int(4), "CDW2")); + aml_append(if_uuid, aml_store(aml_name("CDW2"), a_cdw2)); + + /* check unknown revision in arg(1) */ + if_rev = aml_if(aml_lnot(aml_equal(aml_arg(1), aml_int(1)))); + /* set revision error bits, DWORD1 Bit[3] */ + aml_append(if_rev, aml_or(a_cdw1, aml_int(0x08), a_cdw1)); + aml_append(if_uuid, if_rev); + + /* + * check support for vCPU hotplug type(=enabled) platform-wide capability + * in DWORD2 as sepcified in the below ACPI Specification ECR, + * # https://bugzilla.tianocore.org/show_bug.cgi?id=4481 + */ + if (vms->acpi_dev) { + aml_append(if_uuid, aml_and(a_cdw2, aml_int(0x800000), a_cdw2)); + /* check if OSPM specified hotplug capability bits were masked */ + if_caps_masked = aml_if(aml_lnot(aml_equal(aml_name("CDW2"), a_cdw2))); + aml_append(if_caps_masked, aml_or(a_cdw1, aml_int(0x10), a_cdw1)); + aml_append(if_uuid, if_caps_masked); + } + aml_append(if_uuid, aml_store(a_cdw2, aml_name("CDW2"))); + + aml_append(method, if_uuid); + else_uuid = aml_else(); + + /* set unrecognized UUID error bits, DWORD1 Bit[2] */ + aml_append(else_uuid, aml_or(a_cdw1, aml_int(4), a_cdw1)); + aml_append(method, else_uuid); + + aml_append(method, aml_return(aml_arg(3))); + aml_append(scope, method); + + return; +} + /* DSDT */ static void build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) @@ -852,6 +901,9 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) } else { acpi_dsdt_add_cpus(scope, vms); } + + build_virt_osc_method(scope, vms); + acpi_dsdt_add_uart(scope, &memmap[VIRT_UART], (irqmap[VIRT_UART] + ARM_SPI_BASE)); if (vmc->acpi_expose_flash) { From patchwork Fri Jun 14 00:19:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697688 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 90391C27C4F for ; Fri, 14 Jun 2024 00:21:08 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHugC-0004Su-W9; Thu, 13 Jun 2024 20:20:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHugB-0004ST-96; Thu, 13 Jun 2024 20:20:31 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHug9-0002g0-AP; Thu, 13 Jun 2024 20:20:31 -0400 Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0g0L2mDyz6K61G; Fri, 14 Jun 2024 08:20:18 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 521B7140594; Fri, 14 Jun 2024 08:20:21 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 01:19:58 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 28/29] tcg/mttcg: enable threads to unregister in tcg_ctxs[] Date: Fri, 14 Jun 2024 01:19:44 +0100 Message-ID: <20240614001944.203038-1-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Miguel Luis [BROKEN: This patch is just for reference. It has problems as it does not takes care of the TranslationBlocks and their assigned regions during CPU unrealize] When using TCG acceleration in a multi-threaded context each vCPU has its own thread registered in tcg_ctxs[] upon creation and tcg_cur_ctxs stores the current number of threads that got created. Although, the lack of a mechanism to unregister these threads is a problem when exercising vCPU hotplug/unplug due to the fact that tcg_cur_ctxs gets incremented everytime a vCPU gets hotplugged but never gets decremented everytime a vCPU gets unplugged, therefore breaking the assert stating tcg_cur_ctxs < tcg_max_ctxs after a certain amount of vCPU hotplugs. Suggested-by: Salil Mehta [SM: Check Things To Do Section, https://lore.kernel.org/all/20200613213629.21984-1-salil.mehta@huawei.com/] Signed-off-by: Miguel Luis --- accel/tcg/tcg-accel-ops-mttcg.c | 1 + include/tcg/startup.h | 7 +++++++ tcg/tcg.c | 24 ++++++++++++++++++++++++ 3 files changed, 32 insertions(+) diff --git a/accel/tcg/tcg-accel-ops-mttcg.c b/accel/tcg/tcg-accel-ops-mttcg.c index c552b45b8e..b6d7911a87 100644 --- a/accel/tcg/tcg-accel-ops-mttcg.c +++ b/accel/tcg/tcg-accel-ops-mttcg.c @@ -122,6 +122,7 @@ static void *mttcg_cpu_thread_fn(void *arg) bql_unlock(); rcu_remove_force_rcu_notifier(&force_rcu.notifier); rcu_unregister_thread(); + tcg_unregister_thread(); return NULL; } diff --git a/include/tcg/startup.h b/include/tcg/startup.h index f71305765c..dc35b24de5 100644 --- a/include/tcg/startup.h +++ b/include/tcg/startup.h @@ -45,6 +45,13 @@ void tcg_init(size_t tb_size, int splitwx, unsigned max_cpus); */ void tcg_register_thread(void); +/** + * tcg_unregister_thread: Unregister this thread with the TCG runtime + * + * TBD + */ +void tcg_unregister_thread(void); + /** * tcg_prologue_init(): Generate the code for the TCG prologue * diff --git a/tcg/tcg.c b/tcg/tcg.c index 34e3056380..e5bbe8dc07 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -761,6 +761,15 @@ QEMU_BUILD_BUG_ON((int)(offsetof(CPUNegativeOffsetState, tlb.f[0]) - < MIN_TLB_MASK_TABLE_OFS); #endif +/* TODO: vCPU Hotplug: Need to come back and fix the TCG */ +static void free_tcg_plugin_context(TCGContext *s) +{ +#ifdef CONFIG_PLUGIN + g_ptr_array_unref(s->plugin_tb->insns); + g_free(s->plugin_tb); +#endif +} + /* * All TCG threads except the parent (i.e. the one that called tcg_context_init * and registered the target's TCG globals) must register with this function @@ -810,6 +819,21 @@ void tcg_register_thread(void) tcg_ctx = s; } + +void tcg_unregister_thread(void) +{ + TCGContext *s = tcg_ctx; + unsigned int n; + + /* Unclaim an entry in tcg_ctxs */ + n = qatomic_fetch_dec(&tcg_cur_ctxs); + g_assert(n > 1); + qatomic_store_release(&tcg_ctxs[n - 1], 0); + + free_tcg_plugin_context(s); + + g_free(s); +} #endif /* !CONFIG_USER_ONLY */ /* pool based memory allocation */ From patchwork Fri Jun 14 00:20:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13697689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 18356C27C4F for ; Fri, 14 Jun 2024 00:22:06 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHuhO-0005G9-9O; Thu, 13 Jun 2024 20:21:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHuhH-0005CW-Md; Thu, 13 Jun 2024 20:21:39 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHuhF-0002pO-IE; Thu, 13 Jun 2024 20:21:39 -0400 Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0g0C36Qwz6H7Jk; Fri, 14 Jun 2024 08:20:11 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id E4C20140CF4; Fri, 14 Jun 2024 08:21:34 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 01:21:12 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 29/29] hw/arm/virt: Expose cold-booted CPUs as MADT GICC Enabled Date: Fri, 14 Jun 2024 01:20:58 +0100 Message-ID: <20240614002058.203058-1-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Hotpluggable CPUs MUST be exposed as 'online-capable' according to the new change. However, cold-booted CPUs, if marked as 'online-capable' during boot time, might not be detected by legacy operating systems. This could cause compatibility problems. Original Change Link: https://bugzilla.tianocore.org/show_bug.cgi?id=3706 Since updating the specification might take time, it is necessary to disable the support for unplugging any cold-booted CPUs to preserve compatibility with legacy operating systems. Signed-off-by: Salil Mehta --- hw/arm/virt-acpi-build.c | 29 ++++++++++++++++++++--------- hw/arm/virt.c | 16 ++++++++++++++++ include/hw/core/cpu.h | 2 ++ 3 files changed, 38 insertions(+), 9 deletions(-) diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c index 6cb613103f..322ed8e35b 100644 --- a/hw/arm/virt-acpi-build.c +++ b/hw/arm/virt-acpi-build.c @@ -667,17 +667,28 @@ static uint32_t virt_acpi_get_gicc_flags(CPUState *cpu) } /* - * ARM GIC CPU Interface can be 'online-capable' or 'enabled' at boot - * We MUST set 'online-capable' bit for all hotpluggable CPUs except the - * first/boot CPU. Cold-booted CPUs without 'Id' can also be unplugged. - * Though as-of-now this is only used as a debugging feature. + * The ARM GIC CPU Interface can be either 'online-capable' or 'enabled' at + * boot. We MUST set the 'online-capable' bit for all hotpluggable CPUs. * - * UEFI ACPI Specification 6.5 - * Section: 5.2.12.14. GIC CPU Interface (GICC) Structure - * Table: 5.37 GICC CPU Interface Flags - * Link: https://uefi.org/specs/ACPI/6.5 + * Change Link: https://bugzilla.tianocore.org/show_bug.cgi?id=3706 + * + * Refer to the UEFI ACPI Specification 6.5: + * Section: 5.2.12.14. GIC CPU Interface (GICC) Structure + * Table: 5.37 GICC CPU Interface Flags + * Link: https://uefi.org/specs/ACPI/6.5 + * + * Cold-booted CPUs, except for the first/boot CPU, SHOULD be allowed to be + * hot(un)plugged as well. However, for this to happen, these CPUs MUST have + * the 'online-capable' bit set. This creates a compatibility problem with + * legacy OS, as it might ignore 'online-capable' bits during boot time, and + * hence some CPUs might not get detected. + * + * To fix this, the MADT GIC CPU interface flag should allow both + * 'online-capable' and 'enabled' bits to be set together. This change will + * require an update to the UEFI ACPI standard. Until this update occurs, + * all cold-booted CPUs should be exposed as 'enabled' only. */ - return cpu && !cpu->cpu_index ? 1 : (1 << 3); + return cpu && cpu->cold_booted ? 1 : (1 << 3); } static void diff --git a/hw/arm/virt.c b/hw/arm/virt.c index a2200099a1..770b599acf 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -3163,6 +3163,10 @@ static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, * This shall be used during the init of ACPI Hotplug state and hot-unplug */ cs->acpi_persistent = true; + + if (!dev->hotplugged) { + cs->cold_booted = true; + } } static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, @@ -3223,6 +3227,18 @@ static void virt_cpu_unplug_request(HotplugHandler *hotplug_dev, return; } + /* + * UEFI ACPI standard change is required to make both 'enabled' and the + * 'online-capable' bit co-exist instead of being mutually exclusive. + * check virt_acpi_get_gicc_flags() for more details. + * + * Disable the unplugging of cold-booted vCPUs as a temporary mitigation. + */ + if (cs->cold_booted) { + error_setg(errp, "Hot-unplug of cold-booted CPU not supported!"); + return; + } + if (cs->cpu_index == first_cpu->cpu_index) { error_setg(errp, "Boot CPU(id%d=%d:%d:%d:%d) hot-unplug not supported", first_cpu->cpu_index, cpu->socket_id, cpu->cluster_id, diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index e13e542177..99b699b47f 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -556,6 +556,8 @@ struct CPUState { uint32_t halted; int32_t exception_index; + bool cold_booted; + AccelCPUState *accel; /* Used to keep track of an outstanding cpu throttle thread for migration