From patchwork Tue Oct 24 15:19:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 13434895 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9D3E1C07545 for ; Tue, 24 Oct 2023 15:22:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:Message-Id:MIME-Version:Subject:Cc :To:From:References:In-Reply-To:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=0JkAdjcYTTvN/F74wFC60mg1OcZcYzlVhbgARbRg0do=; b=e7uCcJZvAe7if2 lfK6i7V3ge/RliGk9d18rjhPpy51Sj3NwJtDydvGa58cYjE45T+FtHQVfXd1aQIzapiCiht87nbin VYT9aaTqkuA8aa7l7JE1MfsV4f5DdmNf8jG4YzOR4RAA91I6B4YoMFrbX35gDYQxPCBrRMfhhgW0I k4Nmpmpx7jf36WOCzy6J0bTggz6k3/U+JApgFvuDdXg89uBeyOo0HhNMijPDtFa1ET2FWrQQ388pq dX6OsH8nubgglXOSYm/Cy2Hb392yZDWLw4ASVlS5vsqxe1lfIxCuRfUB+GkASlj2WUMhD3BsM+uIK +RCKrwSl02u6Qvxy//+A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qvJEK-00AJO7-1U; Tue, 24 Oct 2023 15:22:04 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qvJBm-00AHED-1p; Tue, 24 Oct 2023 15:19:26 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description; bh=oCft4xPi3XDHU0UN3l9OsD70hFFh+BkwY8FvO/LnA90=; b=bAAJRN7EjhP4twiaDCuOmYwJ4E FRGmQZo6vM9hrjjL73iBJEk/1cHcCtyUWuKzSYuYq0mpywF4DXvOhpLBJ5WOv8KE1xVoqImnlkMJf puSxdfdMPIDHCNedCD5C4c0ydvoHVHd4ZlxJnLu2ZKYXR8Rc75dp5uIqe0eyxymZaIPQYc6jr1eWU cbREFHVeXz5zwPTmFI/Ab+FlcyE57VEPdb5c37CxKMcSHBSUBug/xTLVDdglq5jX2sMnZ24DI/DAy cKiF1UZQ2V06GK5/kNlhfz1f77KPCNrFvZIaixMtlEEIGKfdwR4vbbMpdQmcYcivWoLPnO/NrDYyf JuwRIVkg==; Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qvJBj-00FbLV-16; Tue, 24 Oct 2023 15:19:25 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=oCft4xPi3XDHU0UN3l9OsD70hFFh+BkwY8FvO/LnA90=; b=WogGkYar3Xyhom4+CPTJn22CDH Tt5whuXtcSNoiYeoc9S2lZnq5wQoVR8xqbtC0cDpKIBCN2lgOQG8CkaM8Wyzz9/O4xrS5Dr8zE4vd 8EBRz5FqHbEILZTdY2vtmVwK/aJ5PBxxpGBHJJYZgeAeU+G46cf6TiLYioCdbEVg1GBPdhjAQXkyg LNVCwEUkolxbKcCLONfgGj/yf6Pu7XEliLfg6/sYVggJmYv7evUj85xcZIoRyJCWrCOY9wB9CP5Zj bu4yDH29G2hZ8lmxUUipXQh4jwp+SbcvSnnl7N0g7nPIJ+BnuH+ABlwb/rGiQXV2foOL8QFoPQ+dR uZtqbnqw==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:60324 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1qvJBZ-0004Ws-0H; Tue, 24 Oct 2023 16:19:13 +0100 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1qvJBa-00AqSK-Hb; Tue, 24 Oct 2023 16:19:14 +0100 In-Reply-To: References: From: Russell King (Oracle) To: linux-pm@vger.kernel.org, loongarch@lists.linux.dev, linux-acpi@vger.kernel.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-riscv@lists.infradead.org, kvmarm@lists.linux.dev, x86@kernel.org, linux-csky@vger.kernel.org, linux-doc@vger.kernel.org, linux-ia64@vger.kernel.org, linux-parisc@vger.kernel.org Cc: Salil Mehta , Jean-Philippe Brucker , jianyong.wu@arm.com, justin.he@arm.com, James Morse , Catalin Marinas , Will Deacon , Jonathan Corbet Subject: [PATCH 36/39] arm64: document virtual CPU hotplug's expectations MIME-Version: 1.0 Content-Disposition: inline Message-Id: Date: Tue, 24 Oct 2023 16:19:14 +0100 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231024_161923_455702_A14ABD8F X-CRM114-Status: GOOD ( 24.28 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: James Morse Add a description of physical and virtual CPU hotplug, explain the differences and elaborate on what is required in ACPI for a working virtual hotplug system. Signed-off-by: James Morse --- Documentation/arch/arm64/cpu-hotplug.rst | 79 ++++++++++++++++++++++++ Documentation/arch/arm64/index.rst | 1 + 2 files changed, 80 insertions(+) create mode 100644 Documentation/arch/arm64/cpu-hotplug.rst diff --git a/Documentation/arch/arm64/cpu-hotplug.rst b/Documentation/arch/arm64/cpu-hotplug.rst new file mode 100644 index 000000000000..76ba8d932c72 --- /dev/null +++ b/Documentation/arch/arm64/cpu-hotplug.rst @@ -0,0 +1,79 @@ +.. SPDX-License-Identifier: GPL-2.0 +.. _cpuhp_index: + +==================== +CPU Hotplug and ACPI +==================== + +CPU hotplug in the arm64 world is commonly used to describe the kernel taking +CPUs online/offline using PSCI. This document is about ACPI firmware allowing +CPUs that were not available during boot to be added to the system later. + +``possible`` and ``present`` refer to the state of the CPU as seen by linux. + + +CPU Hotplug on physical systems - CPUs not present at boot +---------------------------------------------------------- + +Physical systems need to mark a CPU that is ``possible`` but not ``present`` as +being ``present``. An example would be a dual socket machine, where the package +in one of the sockets can be replaced while the system is running. + +This is not supported. + +In the arm64 world CPUs are not a single device but a slice of the system. +There are no systems that support the physical addition (or removal) of CPUs +while the system is running, and ACPI is not able to sufficiently describe +them. + +e.g. New CPUs come with new caches, but the platform's cache toplogy is +described in a static table, the PPTT. How caches are shared between CPUs is +not discoverable, and must be described by firmware. + +e.g. The GIC redistributor for each CPU must be accessed by the driver during +boot to discover the system wide supported features. ACPI's MADT GICC +structures can describe a redistributor associated with a disabled CPU, but +can't describe whether the redistributor is accessible, only that it is not +'always on'. + +arm64's ACPI tables assume that everything described is ``present``. + + +CPU Hotplug on virtual systems - CPUs not enabled at boot +--------------------------------------------------------- + +Virtual systems have the advantage that all the properties the system will +ever have can be described at boot. There are no power-domain considerations +as such devices are emulated. + +CPU Hotplug on virtual systems is supported. It is distinct from physical +CPU Hotplug as all resources are described as ``present``, but CPUs may be +marked as disabled by firmware. Only the CPU's online/offline behaviour is +influenced by firmware. An example is where a virtual machine boots with a +single CPU, and additional CPUs are added once a cloud orchestrator deploys +the workload. + +For a virtual machine, the VMM (e.g. Qemu) plays the part of firmware. + +Virtual hotplug is implemented as a firmware policy affecting which CPUs can be +brought online. Firmware can enforce its policy via PSCI's return codes. e.g. +``DENIED``. + +The ACPI tables must describe all the resources of the virtual machine. CPUs +that firmware wishes to disable either from boot (or later) should not be +``enabled`` in the MADT GICC structures, but should have the ``online capable`` +bit set, to indicate they can be enabled later. The boot CPU must be marked as +``enabled``. The 'always on' GICR structure must be used to describe the +redistributors. + +CPUs described as ``online capable`` but not ``enabled`` can be set to enabled +by the DSDT's Processor object's _STA method. On virtual systems the _STA method +must always report the CPU as ``present``. Changes to the firmware policy can +be notified to the OS via device-check or eject-request. + +CPUs described as ``enabled`` in the static table, should not have their _STA +modified dynamically by firmware. Soft-restart features such as kexec will +re-read the static properties of the system from these static tables, and +may malfunction if these no longer describe the running system. Linux will +re-discover the dynamic properties of the system from the _STA method later +during boot. diff --git a/Documentation/arch/arm64/index.rst b/Documentation/arch/arm64/index.rst index d08e924204bf..78544de0a8a9 100644 --- a/Documentation/arch/arm64/index.rst +++ b/Documentation/arch/arm64/index.rst @@ -13,6 +13,7 @@ ARM64 Architecture asymmetric-32bit booting cpu-feature-registers + cpu-hotplug elf_hwcaps hugetlbpage kdump