From patchwork Wed Aug 9 17:43:38 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mirela Simonovic X-Patchwork-Id: 9891409 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 6E972603F2 for ; Wed, 9 Aug 2017 17:47:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5B1FE289B2 for ; Wed, 9 Aug 2017 17:47:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4F93A289CC; Wed, 9 Aug 2017 17:47:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,RCVD_IN_SORBS_SPAM,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 27DF628A70 for ; Wed, 9 Aug 2017 17:47:26 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dfV1Z-0003nY-F0; Wed, 09 Aug 2017 17:44:05 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dfV1Y-0003nS-5T for xen-devel@lists.xen.org; Wed, 09 Aug 2017 17:44:04 +0000 Received: from [193.109.254.147] by server-8.bemta-6.messagelabs.com id 61/12-09901-3E94B895; Wed, 09 Aug 2017 17:44:03 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprFIsWRWlGSWpSXmKPExsXiVRukpfvIszv S4OlvHoslHxezODB6HN39mymAMYo1My8pvyKBNaO5+RJLwQGXisPLl7M3MF6y6mLk4hASmMAo seh/DyuIwyLwikXi2JfD7CCOhMA7FokXt58ydzFyAjlJEjOWHGGBsCslXp39xAZiCwloScy/v I4dYtQPRolTX+aygyTYBEwkpqxoZwSxRQSkJa59vgxmMwvMZZRoXpkAYgsL2Ep82vSOCcRmEV CVaPpwH8zmFXCQuNX2EWqZvMSVX/0sExj5FjAyrGJUL04tKkst0rXQSyrKTM8oyU3MzNE1NDD Ty00tLk5MT81JTCrWS87P3cQIDBUGINjBOPuy/yFGSQ4mJVHeTdqdkUJ8SfkplRmJxRnxRaU5 qcWHGGU4OJQkeBWAoSckWJSanlqRlpkDDFqYtAQHj5IIrwlImre4IDG3ODMdInWK0Z5jw+r1X 5g4bp3fAiQnHdgOJF9N+P+NSYglLz8vVUqclwukTQCkLaM0D24oLMouMcpKCfMyAp0pxFOQWp SbWYIq/4pRnINRSZj3rwfQFJ7MvBK43a+AzmICOivCtxPkrJJEhJRUA6ORp6eU2gXOf9+uGi1 elVH/WpGr8o23c2E6c+Lev3rfbYpum79xeD5HQuJ32XsW8+83lOxbDsvums2b8/jt5QfSNdof zLIVF0rMyeKQSLPO1pypwVutsSPT4r3ZXJlJ/PcfTg01uHn9NQvD5nTztzs+My5LmqKyYrFrT /Cpl8cz0/PKlP1N4pRYijMSDbWYi4oTAUkVDFutAgAA X-Env-Sender: mirela.simonovic@aggios.com X-Msg-Ref: server-4.tower-27.messagelabs.com!1502300642!110294992!1 X-Originating-IP: [74.125.82.42] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.4.45; banners=-,-,- X-VirusChecked: Checked Received: (qmail 20385 invoked from network); 9 Aug 2017 17:44:02 -0000 Received: from mail-wm0-f42.google.com (HELO mail-wm0-f42.google.com) (74.125.82.42) by server-4.tower-27.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 9 Aug 2017 17:44:02 -0000 Received: by mail-wm0-f42.google.com with SMTP id i66so2518512wmg.0 for ; Wed, 09 Aug 2017 10:44:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aggios-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id; bh=ddEZs/hJYFlzBSok6uTd5s+E95c3svg3mK+EBprI1p4=; b=gtoUfqMEn9wMXDuzISVLVSed4Fqxie1PjywAgS2+DlD8Hn54zVeWsWQZYgkQGQ1nCI UAZrYO5oy94JkBFlrjzhUj3LvWeroi6R2qCg8qTVPAYjI9O3KRZ/rWHxt35kzIi7NZtn RIEmHMFtZruFk2qP1CZ1qzhOStspMcBMDtxXabSlh6cV7xMcAAkl5KKf56+hiKmoqrDB I21IuKmVr/zehtlrVxTcRBfQS4xhZUcG9yxh8aO08/+ewOCYps3qroSyuwZyzBD+/W1D tjFTrjcUTnHzMfDQ53zi5TsBQPZaiZxCKUmwFiKfF4nC6jmVvfwTCsSxUBs1j1mJncYt wo0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=ddEZs/hJYFlzBSok6uTd5s+E95c3svg3mK+EBprI1p4=; b=LiF7tQvBmkTlRpGx0SIs4UH3rQ4jJ6wMPHRbMUMfA6NfAu9wRxjudVKqye1PxEljOd DrI/OJ9RsatJwJzXAiFSQiBvejq6ONLhaHcIppYwuHUaCaVAZBolLih5EnDCTckeqGFo iE6Y8OunsOxQFPtyWlLOrDRsaa28+8pXPqrzHKDrYQ0pMXTyj2cDUth147N5l8MwqAQm qm/4KYBC227o2xXMzy/V+GAfx4czXbw0ImKXQMHDg+twh40tNQXVtrUwIMJTex3yvOZk XLmis6ZfTednY8h/xFgEUZJjtudBKv5p/cAfwKmSW+9+QLg/11h/Etu/RVEls2W4cNV+ webw== X-Gm-Message-State: AHYfb5j+dIPYLwjWtynz5I/F2076nuyv0nRFsJ31uNCx1E2EgSJ7LFGI dwKMTdjigFZC83tGOik= X-Received: by 10.28.45.198 with SMTP id t189mr6963985wmt.72.1502300641428; Wed, 09 Aug 2017 10:44:01 -0700 (PDT) Received: from localhost.localdomain ([212.200.89.41]) by smtp.gmail.com with ESMTPSA id q21sm3969452wra.86.2017.08.09.10.44.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 09 Aug 2017 10:44:00 -0700 (PDT) From: Mirela Simonovic To: xen-devel@lists.xen.org Date: Wed, 9 Aug 2017 19:43:38 +0200 Message-Id: <20170809174338.10143-1-mirela.simonovic@aggios.com> X-Mailer: git-send-email 2.13.0 Cc: edgar.iglesias@xilinx.com, will.wong@xilinx.com, davorin.mista@aggios.com, nirmala.pelluri@xilinx.com, Mirela Simonovic Subject: [Xen-devel] [RFC] xen/arm: Suspend to RAM Support in Xen for ARM X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP This document contains our draft proposal for implementing "suspend to RAM" support for ARM in Xen, as discussed during the last Xen ARM community call. It covers the basic suspend to RAM mechanism based on ARM PSCI standard, that would allow individual guests and Xen itself to suspend. We would appreciate your feedback. Signed-off-by: Mirela Simonovic --- docs/misc/arm/suspend-to-ram.txt | 210 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 210 insertions(+) create mode 100644 docs/misc/arm/suspend-to-ram.txt diff --git a/docs/misc/arm/suspend-to-ram.txt b/docs/misc/arm/suspend-to-ram.txt new file mode 100644 index 0000000000..ec8080fc64 --- /dev/null +++ b/docs/misc/arm/suspend-to-ram.txt @@ -0,0 +1,210 @@ +% Suspend to RAM Support in Xen for ARM +% Revision 1.0 + +======== +Overview +======== + +Suspend to RAM (in the following text 'suspend') for ARM in Xen should be +coordinated using ARM PSCI standard [1]. + +EL1/2 should suspend in the following order: +1) Unprivileged guests (DomUs) suspend +2) Privileged guest (Dom0) suspends +3) Xen suspends + +Since this proposal is focused on implementing PSCI-based suspend mechanisms in +Xen, communication with or among the guests is not covered by this document. +The order of suspending the guests is assumed to be guaranteed by the software +running in EL1. + +----------------- +Suspending Guests +----------------- + +Suspend procedure for a guest consists of the following: +1) Suspending devices +2) Suspending non-boot CPUs +3) System suspend, performed by the boot CPU + +Each guest should suspend the devices it owns. Suspending of devices is not +covered by this document. The document covers only mechanisms for suspending +non-boot CPUs, as well as the system suspend. + +Guests should suspend their non-boot vCPUs using the hotplug mechanism. +Virtual CPUs should be put offline using the already implemented PSCI vCPU_OFF +call (prefix 'v' is added to distinguish PSCI calls made by guests to Xen, which +affect virtual machines; as opposed to PSCI calls made by Xen to the EL3, which +can affect power state of the physical machine). + +After suspending its non-boot vCPUs a guest should finalize the suspend by +making the vSYSTEM_SUSPEND PSCI call. The resume address is specified by the +guest via the vSYSTEM_SUSPEND entry_point_address argument. The vSYSTEM_SUSPEND +call is currently not implemented in Xen. + +It is expected that a guest leaves enabled all interrupts that should wake it +up. Other interrupts should be disabled by the guest prior to calling +vSYSTEM_SUSPEND. + +After an unprivileged guest suspends, Xen will not suspend. Xen would suspend +only after the Dom0 completes the system suspend. + +-------------- +Suspending Xen +-------------- + +Xen should start suspending itself upon receiving the vSYSTEM_SUSPEND call +from the last running guest (Dom0). At that moment all physical CPUs are still +online (taking offline a vCPU or suspending a VM does not affect physical CPUs). +Xen shall now put offline the non-boot pCPUs by making the CPU_OFF PSCI call +to EL3. The CPU_OFF PSCI function is currently not implemented in Xen. + +After putting offline the non-boot cores Xen must save the context and finalize +suspend by invoking SYSTEM_SUSPEND PSCI call, which is passed to EL3. +The resume point of Xen is specified by the entry_point_address argument of the +SYSTEM_SUSPEND call. The SYSTEM_SUSPEND function and context saving is not +implemented in Xen for ARM today. + +------------ +Resuming Xen +------------ + +Xen must be resumed prior to any software running in EL1. Starting from the +resume point, Xen should restore the context and resume Dom0. Dom0 shall always +be resumed whenever Xen resumes. +The whole Xen resume flow for the ARM architecture has to be implemented. + +--------------- +Resuming Guests +--------------- + +Resume of the privileged guest (Dom0) is always following the Xen resume. + +An unprivileged guest shall resume once a device it owns triggers a wake-up +interrupt, regardless of whether Xen was suspended when the wake-up interrupt +was triggered. If Xen was suspended, it is assumed that Dom0 will be running +before the DomU guest starts to resume. The synchronization mechanism to +enforce the assumed condition is TBD. + +If the ARM's GIC was powered down after the ARM subsystem suspended, it is +assumed that Xen needs to restore the GIC interface for a VM prior to handing +over control to the guest. However, the guest should restore its own context +upon entering the resume point (out of scope of this document). + +======================= +Implementation Proposal +======================= + +-------- +Overview +-------- + +In order to enable the suspend/resume of VMs and Xen itself, the following PSCI +calls have to be implemented and integrated in Xen: +1) vSYSTEM_SUSPEND +2) CPU_OFF +3) SYSTEM_SUSPEND + +In addition, the following have to be implemented: +* Save/restore of EL2 context +* Save/restore of GIC configuration for each VM + +Implementation details are provided in the sections below. Function names and +paths used below are consistent within the document but may not always match the +names used in future implementation. Existing functions and paths are named as +in Xen source tree. + +Note: The proposal is still incomplete and shall be refined in future revisions. +Specific issues that are not addressed are marked as "TBD". + +------------------------------------- +Suspend/Resume Implementation Details +------------------------------------- + +PSCI Implementation and Integration +----------------------------------- +vSYSTEM_SUSPEND +--------------- +vSYSTEM_SUSPEND shall be implemented in +* do_psci_system_suspend() in arch/arm/vpsci.c + +The implementation shall include the following steps: +* Block the current vCPU +* If the hardware domain made the call trigger Xen suspend, i.e. + call machine_suspend() which will be implemented in arch/arm/suspend.c + (similar as the machine_restart() is implemented in arch/arm/shutdown.c) + +The function do_psci_system_suspend() shall be called from +* do_trap_psci() in arch/arm/traps.c + +CPU_OFF (physical CPUs) +----------------------- +The CPU_OFF function shall be implemented in +* call_psci_cpu_off() in arch/arm/psci.c + +The implementation shall consist just of making the SMC call to EL3. + +This function needs to be called when Xen generic code disables non-boot CPUs, +which is done by +* disable_nonboot_cpus() in common/cpu.c +This function calls architecture specific +* __cpu_die() implemented in arch/arm/smpboot.c +The call_psci_cpu_off() shall be invoked when the respective CPU dies. To make +that happen, the +* arch_cpu_die() would be implemented in arch/arm/arm64/smpboot.c +and called from __cpu_die(). +Finally the call_psci_cpu_off() shall be invoked from arch_cpu_die(). + +Such a control flow would be similar to the already existing flow for enabling +non-boot CPUs, which looks like this: +enable_nonboot_cpus() -> cpu_up() -> __cpu_up() -> arch_cpu_up() -> +call_psci_cpu_on() + +SYSTEM_SUSPEND (physical) +------------------------- +The SYSTEM_SUSPEND function shall be implemented in +* call_psci_system_suspend() in arch/arm/psci.c + +The implementation shall consist just of making the SMC call to EL3. The +entry_point_address argument of the SMC call needs to be an ARM architecture +resume address. The call_psci_system_suspend() function does not return. + +The function needs to be called from machine_suspend() to finalize the suspend +procedure. + +------------------ +Additional Changes +------------------ + +Suspend Flow +------------ +The suspend procedure shall be implemented in +* machine_suspend() in arch/arm/suspend.c + +The implementation shall include the following steps: +* Set the system_state variable to SYS_STATE_suspend +* Freeze domains by calling domain_pause() for each domain (we assume this needs + to be done) +* Disable non-boot CPUs by calling disable_nonboot_cpus() +* Save ARM specific context + +Resume Flow +------------ +The resume procedure shall be implemented in +* machine_resume() in arch/arm/suspend.c + +The machine_resume() implementation shall include the following steps: +* Restore ARM specific context +* Enable non-boot CPUs by calling enable_nonboot_cpus() +* Thaw domains by calling domain_unpause() for each domain (we assume this needs + to be done) +* Set the system_state variable to SYS_STATE_resume +* TBD: how to resume Dom0, i.e. how to hand over control to Dom0? + +========== +References +========== + +[1] Power State Coordination Interface (ARM): +http://infocenter.arm.com/help/topic/com.arm.doc.den0022d/Power_State_Coordination_Interface_PDD_v1_1_DEN0022D.pdf +