From patchwork Tue Mar 2 20:47:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112201 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6EECC433DB for ; Tue, 2 Mar 2021 20:50:48 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 300D364F11 for ; Tue, 2 Mar 2021 20:50:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 300D364F11 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:34610 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHByh-00036y-0n for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 15:50:47 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45594) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwa-0001Mr-M4 for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:36 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:5850 helo=mx0a-001b2d01.pphosted.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwY-00084s-QU for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:36 -0500 Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122Kiva0019704; Tue, 2 Mar 2021 15:48:30 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=1FijDBwr/EkYnd5r4iVjvrLE8UHpNR8E6i0Rhewa7Rs=; b=nPEV3BLITGntiKJtJIBeLkJBpU2tE/iafk/E6MyGpCdI7/ZEFZOkqO0go1nU7GqhUt6I 81bl1F/MW47NfaiV2y2IU4d0SvsSnq/a6f0eWkWfofnZmQ6bjh+Xcz2XtLxbYXUkPvqj KqURtmjeVM5s51Ar1GXfbXqzRKgWXvSyifYqWAW9Q88hRjk7bMfOYYJaIxiWZrDsSEIg fzHvbn9kA3E+3dKJYOa+NjQ5YQusGF9EGxpRJN+ZOCUv2x0pRwoFI00OZzH8HBaKM4GJ CSXa1pviayw39AjvvIsbdUhMq124RZmSK/Vdc6OOukag6uK44rqtnFH+GMVXRSIHFZ+i 0A== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 371vnsr4mk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:30 -0500 Received: from m0098420.ppops.net (m0098420.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122Kj3aU020951; Tue, 2 Mar 2021 15:48:30 -0500 Received: from ppma04dal.us.ibm.com (7a.29.35a9.ip4.static.sl-reverse.com [169.53.41.122]) by mx0b-001b2d01.pphosted.com with ESMTP id 371vnsr4mc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:30 -0500 Received: from pps.filterd (ppma04dal.us.ibm.com [127.0.0.1]) by ppma04dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122KmRQP008408; Tue, 2 Mar 2021 20:48:29 GMT Received: from b01cxnp23032.gho.pok.ibm.com (b01cxnp23032.gho.pok.ibm.com [9.57.198.27]) by ppma04dal.us.ibm.com with ESMTP id 36ydq988jj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:29 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmSdw25755970 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:28 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3D4D128064; Tue, 2 Mar 2021 20:48:28 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0CC6428059; Tue, 2 Mar 2021 20:48:28 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:28 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 01/26] linux-headers: Add definitions of KVM page encryption bitmap ioctls Date: Tue, 2 Mar 2021 15:47:57 -0500 Message-Id: <20210302204822.81901-2-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 mlxlogscore=999 clxscore=1015 mlxscore=0 bulkscore=0 impostorscore=0 lowpriorityscore=0 spamscore=0 suspectscore=0 adultscore=0 priorityscore=1501 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.158.5; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Cornelia Huck , Brijesh Singh , "open list:Overall KVM CPUs" , "Michael S. Tsirkin" , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add support for two ioctls KVM_GET_PAGE_ENC_BITMAP and KVM_SET_PAGE_ENC_BITMAP used to record the encryption state of each guest page. This patch will be replaced by a new implementation based on shared regions list, or by user-space handling of the regions list. However, these changes do not affect the use of the page encryption indication in confidential guest migration flow. Signed-off-by: Dov Murik --- linux-headers/linux/kvm.h | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h index 020b62a619..836c3776c0 100644 --- a/linux-headers/linux/kvm.h +++ b/linux-headers/linux/kvm.h @@ -532,6 +532,16 @@ struct kvm_dirty_log { }; }; +/* for KVM_GET_PAGE_ENC_BITMAP */ +struct kvm_page_enc_bitmap { + __u64 start_gfn; + __u64 num_pages; + union { + void *enc_bitmap; /* one bit per page */ + __u64 padding2; + }; +}; + /* for KVM_CLEAR_DIRTY_LOG */ struct kvm_clear_dirty_log { __u32 slot; @@ -1557,6 +1567,9 @@ struct kvm_pv_cmd { /* Available with KVM_CAP_S390_PROTECTED */ #define KVM_S390_PV_COMMAND _IOWR(KVMIO, 0xc5, struct kvm_pv_cmd) +#define KVM_GET_PAGE_ENC_BITMAP _IOW(KVMIO, 0xc6, struct kvm_page_enc_bitmap) +#define KVM_SET_PAGE_ENC_BITMAP _IOW(KVMIO, 0xc7, struct kvm_page_enc_bitmap) + /* Available with KVM_CAP_X86_MSR_FILTER */ #define KVM_X86_SET_MSR_FILTER _IOW(KVMIO, 0xc6, struct kvm_msr_filter) From patchwork Tue Mar 2 20:47:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112211 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A907C433E6 for ; Tue, 2 Mar 2021 20:53:22 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F162160234 for ; Tue, 2 Mar 2021 20:53:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F162160234 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:43326 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHC1A-0006jl-NM for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 15:53:20 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45628) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwb-0001My-FE for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:37 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:5678) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwZ-00084v-1V for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:37 -0500 Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122KhLOJ124963; Tue, 2 Mar 2021 15:48:31 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=GgtUo7M6IAF8gKRdGxj4ROOGvmT7f45ORgvxTxyoVdg=; b=NvgFfs1SU0GWqsjSeKWooTviuvYZ5ddp0hWxiH87J9j5m7pXj+vsqr++9rJiALrFYsqo 9YONk1RJJrjLsUE1Zza/NOukAcTkgkWOPQyZagzjAaA7MVlGJPlM1aXFs+4qLuI6taav qXxGSU0kM4GG1Cim4Hx8EnQAF3hfYw+ZyhU0pwIKyImPqd7CK17GKVQP9LyzdTf7XUUb ZCIU4KHm49aG0u5YxlbDgqbEhQ5cCd/0HSyPvD4KfnbXCc7GdmeDxQskpbxVqVNhtc4B V6yjRGq2vFtX5i3N35uOLjGrOArIX/q/SchcRMvHXlgGbck/64L3ksS3tG2gfhxvX4sm og== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vn7r7bu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:31 -0500 Received: from m0098421.ppops.net (m0098421.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122KlbGf162811; Tue, 2 Mar 2021 15:48:30 -0500 Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vn7r7bh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:30 -0500 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122KgZad019202; Tue, 2 Mar 2021 20:48:29 GMT Received: from b01cxnp23032.gho.pok.ibm.com (b01cxnp23032.gho.pok.ibm.com [9.57.198.27]) by ppma02dal.us.ibm.com with ESMTP id 3710sqnccy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:29 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmSZD15860192 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:28 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9B51528059; Tue, 2 Mar 2021 20:48:28 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 58D1428067; Tue, 2 Mar 2021 20:48:28 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:28 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 02/26] kvm: add support to sync the page encryption state bitmap Date: Tue, 2 Mar 2021 15:47:58 -0500 Message-Id: <20210302204822.81901-3-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 mlxscore=0 clxscore=1011 priorityscore=1501 spamscore=0 bulkscore=0 adultscore=0 malwarescore=0 suspectscore=0 mlxlogscore=999 lowpriorityscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.158.5; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0b-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , "open list:Overall KVM CPUs" , Juan Quintela , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , "Dr. David Alan Gilbert" , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Brijesh Singh The SEV VMs have concept of private and shared memory. The private memory is encrypted with guest-specific key, while shared memory may be encrypted with hyperivosr key. The KVM_GET_PAGE_ENC_BITMAP can be used to get a bitmap indicating whether the guest page is private or shared. A private page must be transmitted using the SEV migration commands. Add a cpu_physical_memory_sync_encrypted_bitmap() which can be used to get the page encryption bitmap for a given memory region. The page encryption bitmap is not exactly same as dirty bitmap. The page encryption bitmap is a purely a matter of state about the page is encrypted or not. To avoid some confusion we clone few functions for clarity. [Dov changes: replace memcrypt-related checkers with confidential guest support in migration/ram.c and accel/kvm; rename atomic_* to qatomic_* in include/exec/ram_addr.h] Signed-off-by: Brijesh Singh Signed-off-by: Dov Murik --- include/exec/ram_addr.h | 197 ++++++++++++++++++++++++++++++++++++++++ include/exec/ramblock.h | 3 + include/exec/ramlist.h | 3 +- accel/kvm/kvm-all.c | 43 +++++++++ migration/ram.c | 17 ++++ 5 files changed, 262 insertions(+), 1 deletion(-) diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h index 3cb9791df3..aac5b5e393 100644 --- a/include/exec/ram_addr.h +++ b/include/exec/ram_addr.h @@ -284,6 +284,60 @@ static inline void cpu_physical_memory_set_dirty_flag(ram_addr_t addr, set_bit_atomic(offset, blocks->blocks[idx]); } +static inline void cpu_physical_memory_set_encrypted_range(ram_addr_t start, + ram_addr_t length, + unsigned long val) +{ + unsigned long page; + unsigned long * const *src; + + page = start >> TARGET_PAGE_BITS; + + rcu_read_lock(); + + src = qatomic_rcu_read( + &ram_list.dirty_memory[DIRTY_MEMORY_ENCRYPTED])->blocks; + + if (length) { + unsigned long idx = page / DIRTY_MEMORY_BLOCK_SIZE; + unsigned long offset = page % DIRTY_MEMORY_BLOCK_SIZE; + int m = (start) & (BITS_PER_LONG - 1); + int n = MIN(length, BITS_PER_LONG - m); + unsigned long old_val = qatomic_read(&src[idx][BIT_WORD(offset)]); + unsigned long mask; + + mask = (~0UL >> n); + mask = mask << m; + + old_val &= ~mask; + val &= mask; + + qatomic_xchg(&src[idx][BIT_WORD(offset)], old_val | val); + page += n; + length -= n; + } + + /* remaining bits */ + if (length) { + unsigned long idx = page / DIRTY_MEMORY_BLOCK_SIZE; + unsigned long offset = page % DIRTY_MEMORY_BLOCK_SIZE; + int m = (start) & (BITS_PER_LONG - 1); + int n = MIN(length, BITS_PER_LONG - m); + unsigned long old_val = qatomic_read(&src[idx][BIT_WORD(offset)]); + unsigned long mask; + + mask = (~0UL >> n); + mask = mask << m; + + old_val &= ~mask; + val &= mask; + + qatomic_xchg(&src[idx][BIT_WORD(offset)], old_val | val); + } + + rcu_read_unlock(); +} + static inline void cpu_physical_memory_set_dirty_range(ram_addr_t start, ram_addr_t length, uint8_t mask) @@ -335,6 +389,62 @@ static inline void cpu_physical_memory_set_dirty_range(ram_addr_t start, } #if !defined(_WIN32) +static inline void cpu_physical_memory_set_encrypted_lebitmap( + unsigned long *bitmap, + ram_addr_t start, + ram_addr_t pages) +{ + unsigned long i; + unsigned long hpratio = getpagesize() / TARGET_PAGE_SIZE; + unsigned long page = BIT_WORD(start >> TARGET_PAGE_BITS); + + /* start address is aligned at the start of a word? */ + if ((((page * BITS_PER_LONG) << TARGET_PAGE_BITS) == start) && + (hpratio == 1)) { + unsigned long **blocks[DIRTY_MEMORY_NUM]; + unsigned long idx; + unsigned long offset; + long k; + long nr = BITS_TO_LONGS(pages); + + idx = (start >> TARGET_PAGE_BITS) / DIRTY_MEMORY_BLOCK_SIZE; + offset = BIT_WORD((start >> TARGET_PAGE_BITS) % + DIRTY_MEMORY_BLOCK_SIZE); + + rcu_read_lock(); + + for (i = 0; i < DIRTY_MEMORY_NUM; i++) { + blocks[i] = qatomic_rcu_read(&ram_list.dirty_memory[i])->blocks; + } + + for (k = 0; k < nr; k++) { + if (bitmap[k]) { + unsigned long temp = leul_to_cpu(bitmap[k]); + + qatomic_xchg(&blocks[DIRTY_MEMORY_ENCRYPTED][idx][offset], temp); + } + + if (++offset >= BITS_TO_LONGS(DIRTY_MEMORY_BLOCK_SIZE)) { + offset = 0; + idx++; + } + } + + rcu_read_unlock(); + } else { + i = 0; + while (pages > 0) { + unsigned long len = MIN(pages, BITS_PER_LONG); + + cpu_physical_memory_set_encrypted_range(start, len, + leul_to_cpu(bitmap[i])); + start += len; + i++; + pages -= len; + } + } +} + static inline void cpu_physical_memory_set_dirty_lebitmap(unsigned long *bitmap, ram_addr_t start, ram_addr_t pages) @@ -438,6 +548,8 @@ static inline void cpu_physical_memory_clear_dirty_range(ram_addr_t start, cpu_physical_memory_test_and_clear_dirty(start, length, DIRTY_MEMORY_MIGRATION); cpu_physical_memory_test_and_clear_dirty(start, length, DIRTY_MEMORY_VGA); cpu_physical_memory_test_and_clear_dirty(start, length, DIRTY_MEMORY_CODE); + cpu_physical_memory_test_and_clear_dirty(start, length, + DIRTY_MEMORY_ENCRYPTED); } @@ -513,5 +625,90 @@ uint64_t cpu_physical_memory_sync_dirty_bitmap(RAMBlock *rb, return num_dirty; } + +static inline bool cpu_physical_memory_test_encrypted(ram_addr_t start, + ram_addr_t length) +{ + unsigned long end, page; + bool enc = false; + unsigned long * const *src; + + if (length == 0) { + return enc; + } + + end = TARGET_PAGE_ALIGN(start + length) >> TARGET_PAGE_BITS; + page = start >> TARGET_PAGE_BITS; + + rcu_read_lock(); + + src = qatomic_rcu_read( + &ram_list.dirty_memory[DIRTY_MEMORY_ENCRYPTED])->blocks; + + while (page < end) { + unsigned long idx = page / DIRTY_MEMORY_BLOCK_SIZE; + unsigned long offset = page % DIRTY_MEMORY_BLOCK_SIZE; + unsigned long num = MIN(end - page, DIRTY_MEMORY_BLOCK_SIZE - offset); + + enc |= qatomic_read(&src[idx][BIT_WORD(offset)]); + page += num; + } + + rcu_read_unlock(); + + return enc; +} + +static inline +void cpu_physical_memory_sync_encrypted_bitmap(RAMBlock *rb, + ram_addr_t start, + ram_addr_t length) +{ + ram_addr_t addr; + unsigned long word = BIT_WORD((start + rb->offset) >> TARGET_PAGE_BITS); + unsigned long *dest = rb->encbmap; + + /* start address and length is aligned at the start of a word? */ + if (((word * BITS_PER_LONG) << TARGET_PAGE_BITS) == + (start + rb->offset) && + !(length & ((BITS_PER_LONG << TARGET_PAGE_BITS) - 1))) { + int k; + int nr = BITS_TO_LONGS(length >> TARGET_PAGE_BITS); + unsigned long * const *src; + unsigned long idx = (word * BITS_PER_LONG) / DIRTY_MEMORY_BLOCK_SIZE; + unsigned long offset = BIT_WORD((word * BITS_PER_LONG) % + DIRTY_MEMORY_BLOCK_SIZE); + unsigned long page = BIT_WORD(start >> TARGET_PAGE_BITS); + + rcu_read_lock(); + + src = qatomic_rcu_read( + &ram_list.dirty_memory[DIRTY_MEMORY_ENCRYPTED])->blocks; + + for (k = page; k < page + nr; k++) { + unsigned long bits = qatomic_read(&src[idx][offset]); + dest[k] = bits; + + if (++offset >= BITS_TO_LONGS(DIRTY_MEMORY_BLOCK_SIZE)) { + offset = 0; + idx++; + } + } + + rcu_read_unlock(); + } else { + ram_addr_t offset = rb->offset; + + for (addr = 0; addr < length; addr += TARGET_PAGE_SIZE) { + long k = (start + addr) >> TARGET_PAGE_BITS; + if (cpu_physical_memory_test_encrypted(start + addr + offset, + TARGET_PAGE_SIZE)) { + set_bit(k, dest); + } else { + clear_bit(k, dest); + } + } + } +} #endif #endif diff --git a/include/exec/ramblock.h b/include/exec/ramblock.h index 07d50864d8..bcd7720826 100644 --- a/include/exec/ramblock.h +++ b/include/exec/ramblock.h @@ -59,6 +59,9 @@ struct RAMBlock { */ unsigned long *clear_bmap; uint8_t clear_bmap_shift; + + /* bitmap of page encryption state for an encrypted guest */ + unsigned long *encbmap; }; #endif #endif diff --git a/include/exec/ramlist.h b/include/exec/ramlist.h index 26704aa3b0..2422e5ce86 100644 --- a/include/exec/ramlist.h +++ b/include/exec/ramlist.h @@ -11,7 +11,8 @@ typedef struct RAMBlockNotifier RAMBlockNotifier; #define DIRTY_MEMORY_VGA 0 #define DIRTY_MEMORY_CODE 1 #define DIRTY_MEMORY_MIGRATION 2 -#define DIRTY_MEMORY_NUM 3 /* num of dirty bits */ +#define DIRTY_MEMORY_ENCRYPTED 3 +#define DIRTY_MEMORY_NUM 4 /* num of dirty bits */ /* The dirty memory bitmap is split into fixed-size blocks to allow growth * under RCU. The bitmap for a block can be accessed as follows: diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index 84c943fcdb..13350c1b9b 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -604,6 +604,43 @@ static void kvm_memslot_init_dirty_bitmap(KVMSlot *mem) mem->dirty_bmap = g_malloc0(bitmap_size); } +/* sync page_enc bitmap */ +static int kvm_sync_page_enc_bitmap(KVMMemoryListener *kml, + MemoryRegionSection *section, + KVMSlot *mem) +{ + unsigned long size; + KVMState *s = kvm_state; + struct kvm_page_enc_bitmap e = {}; + ram_addr_t pages = int128_get64(section->size) / getpagesize(); + ram_addr_t start = section->offset_within_region + + memory_region_get_ram_addr(section->mr); + + size = ALIGN(((mem->memory_size) >> TARGET_PAGE_BITS), 64) / 8; + e.enc_bitmap = g_malloc0(size); + e.start_gfn = mem->start_addr >> TARGET_PAGE_BITS; + e.num_pages = pages; + if (kvm_vm_ioctl(s, KVM_GET_PAGE_ENC_BITMAP, &e) == -1) { + DPRINTF("KVM_GET_PAGE_ENC_BITMAP ioctl failed %d\n", errno); + g_free(e.enc_bitmap); + return 1; + } + + cpu_physical_memory_set_encrypted_lebitmap(e.enc_bitmap, + start, pages); + + g_free(e.enc_bitmap); + + return 0; +} + +static inline bool confidential_guest(void) +{ + MachineState *ms = MACHINE(qdev_get_machine()); + + return ms->cgs; +} + /** * kvm_physical_sync_dirty_bitmap - Sync dirty bitmap from kernel space * @@ -659,6 +696,12 @@ static int kvm_physical_sync_dirty_bitmap(KVMMemoryListener *kml, slot_offset += slot_size; start_addr += slot_size; size -= slot_size; + + if (confidential_guest() && + kvm_sync_page_enc_bitmap(kml, section, mem)) { + g_free(d.dirty_bitmap); + return -1; + } } out: return ret; diff --git a/migration/ram.c b/migration/ram.c index 72143da0ac..997f90cc5b 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -61,6 +61,7 @@ #if defined(__linux__) #include "qemu/userfaultfd.h" #endif /* defined(__linux__) */ +#include "hw/boards.h" /***********************************************************/ /* ram save/restore */ @@ -81,6 +82,13 @@ /* 0x80 is reserved in migration.h start with 0x100 next */ #define RAM_SAVE_FLAG_COMPRESS_PAGE 0x100 +static inline bool memcrypt_enabled(void) +{ + MachineState *ms = MACHINE(qdev_get_machine()); + + return ms->cgs; +} + static inline bool is_zero_range(uint8_t *p, uint64_t size) { return buffer_is_zero(p, size); @@ -865,6 +873,9 @@ static void ramblock_sync_dirty_bitmap(RAMState *rs, RAMBlock *rb) rs->migration_dirty_pages += new_dirty_pages; rs->num_dirty_pages_period += new_dirty_pages; + if (memcrypt_enabled()) { + cpu_physical_memory_sync_encrypted_bitmap(rb, 0, rb->used_length); + } } /** @@ -2174,6 +2185,8 @@ static void ram_save_cleanup(void *opaque) block->clear_bmap = NULL; g_free(block->bmap); block->bmap = NULL; + g_free(block->encbmap); + block->encbmap = NULL; } xbzrle_cleanup(); @@ -2615,6 +2628,10 @@ static void ram_list_init_bitmaps(void) bitmap_set(block->bmap, 0, pages); block->clear_bmap_shift = shift; block->clear_bmap = bitmap_new(clear_bmap_size(pages, shift)); + if (memcrypt_enabled()) { + block->encbmap = bitmap_new(pages); + bitmap_set(block->encbmap, 0, pages); + } } } } From patchwork Tue Mar 2 20:47:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112223 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F9E6C433DB for ; Tue, 2 Mar 2021 20:57:09 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9F3E664F21 for ; Tue, 2 Mar 2021 20:57:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9F3E664F21 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:55216 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHC4p-0003GG-Hm for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 15:57:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45714) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwd-0001PR-Gc for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:39 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:26874) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwb-00085S-LV for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:39 -0500 Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122KiGWq092485; Tue, 2 Mar 2021 15:48:32 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=eJlrr4xPmIrD8Ngtz0c25Al/mZy2bTrSq4zDBFrl0X8=; b=YNatZsDsrfjHGyS5kC+3JrRSys9uVxW4fvTzqADEg/7pnVQ3gjq1QCiPoQKH0w+yfCv6 MgY7o8okAvw33igkhtRCPj88mdHcBnYf/R91CgGmLAmZUbyHKtNxbpCsqz+tiX6457u1 ivZf3F0lt68as9inccIA1YangjBdiLUIlJaNuEobC/hA36RuvtuVYyoNliovExL/B6np 7iqADbBS6bY8k0QTYPy181TKeTgvlimPEvdvmpr87uqftV9cj88B2ktOxeF1Ij60PV7O +ZH1vVEx1wAejwoCB5oBz1XRwnYaBzZKagL/+YGHH2QXRfx17gfebGXYlRQi3JPcbs/R iQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vnf0637-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:31 -0500 Received: from m0098399.ppops.net (m0098399.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122KiTIs100432; Tue, 2 Mar 2021 15:48:31 -0500 Received: from ppma04wdc.us.ibm.com (1a.90.2fa9.ip4.static.sl-reverse.com [169.47.144.26]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vnf0627-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:30 -0500 Received: from pps.filterd (ppma04wdc.us.ibm.com [127.0.0.1]) by ppma04wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122Km4K6009941; Tue, 2 Mar 2021 20:48:29 GMT Received: from b01cxnp23033.gho.pok.ibm.com (b01cxnp23033.gho.pok.ibm.com [9.57.198.28]) by ppma04wdc.us.ibm.com with ESMTP id 3712phhu7u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:29 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmTYu25690560 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:29 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DC3BF28059; Tue, 2 Mar 2021 20:48:28 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B2FEB2805A; Tue, 2 Mar 2021 20:48:28 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:28 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 03/26] machine: Add auxcpus=N suboption to -smp Date: Tue, 2 Mar 2021 15:47:59 -0500 Message-Id: <20210302204822.81901-4-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1011 mlxlogscore=999 malwarescore=0 impostorscore=0 phishscore=0 bulkscore=0 lowpriorityscore=0 mlxscore=0 suspectscore=0 priorityscore=1501 spamscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.156.1; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Eduardo Habkost , "Michael S. Tsirkin" , Richard Henderson , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add a notion of auxiliary vcpus to CpuTopology, which will allow to designate a few vcpus (normally 1) to helper tasks not related to main guest VM execution. Example usage for starting a 4-vcpu guest, of which 1 vcpu is marked as auxiliary: qemu-system-x86_64 -smp 4,auxcpus=1 ... Signed-off-by: Dov Murik --- include/hw/boards.h | 1 + hw/core/machine.c | 7 +++++++ hw/i386/pc.c | 7 +++++++ softmmu/vl.c | 3 +++ 4 files changed, 18 insertions(+) diff --git a/include/hw/boards.h b/include/hw/boards.h index a46dfe5d1a..7ee5c73510 100644 --- a/include/hw/boards.h +++ b/include/hw/boards.h @@ -246,6 +246,7 @@ typedef struct CpuTopology { unsigned int threads; unsigned int sockets; unsigned int max_cpus; + unsigned int aux_cpus; } CpuTopology; /** diff --git a/hw/core/machine.c b/hw/core/machine.c index 970046f438..08ea2cedea 100644 --- a/hw/core/machine.c +++ b/hw/core/machine.c @@ -722,6 +722,7 @@ static void smp_parse(MachineState *ms, QemuOpts *opts) unsigned sockets = qemu_opt_get_number(opts, "sockets", 0); unsigned cores = qemu_opt_get_number(opts, "cores", 0); unsigned threads = qemu_opt_get_number(opts, "threads", 0); + unsigned aux_cpus = qemu_opt_get_number(opts, "auxcpus", 0); /* compute missing values, prefer sockets over cores over threads */ if (cpus == 0 || sockets == 0) { @@ -767,10 +768,16 @@ static void smp_parse(MachineState *ms, QemuOpts *opts) exit(1); } + if (aux_cpus >= ms->smp.max_cpus) { + error_report("auxcpus must be lower than max_cpus"); + exit(1); + } + ms->smp.cpus = cpus; ms->smp.cores = cores; ms->smp.threads = threads; ms->smp.sockets = sockets; + ms->smp.aux_cpus = aux_cpus; } if (ms->smp.cpus > 1) { diff --git a/hw/i386/pc.c b/hw/i386/pc.c index 8aa85dec54..95d3769842 100644 --- a/hw/i386/pc.c +++ b/hw/i386/pc.c @@ -718,6 +718,7 @@ void pc_smp_parse(MachineState *ms, QemuOpts *opts) unsigned dies = qemu_opt_get_number(opts, "dies", 1); unsigned cores = qemu_opt_get_number(opts, "cores", 0); unsigned threads = qemu_opt_get_number(opts, "threads", 0); + unsigned aux_cpus = qemu_opt_get_number(opts, "auxcpus", 0); /* compute missing values, prefer sockets over cores over threads */ if (cpus == 0 || sockets == 0) { @@ -763,10 +764,16 @@ void pc_smp_parse(MachineState *ms, QemuOpts *opts) exit(1); } + if (aux_cpus >= ms->smp.max_cpus) { + error_report("auxcpus must be lower than max_cpus"); + exit(1); + } + ms->smp.cpus = cpus; ms->smp.cores = cores; ms->smp.threads = threads; ms->smp.sockets = sockets; + ms->smp.aux_cpus = aux_cpus; x86ms->smp_dies = dies; } diff --git a/softmmu/vl.c b/softmmu/vl.c index b219ce1f35..96f0ff8111 100644 --- a/softmmu/vl.c +++ b/softmmu/vl.c @@ -720,6 +720,9 @@ static QemuOptsList qemu_smp_opts = { }, { .name = "maxcpus", .type = QEMU_OPT_NUMBER, + }, { + .name = "auxcpus", + .type = QEMU_OPT_NUMBER, }, { /*End of list */ } }, From patchwork Tue Mar 2 20:48:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112203 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 230A4C433E0 for ; Tue, 2 Mar 2021 20:50:50 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3499B64F21 for ; Tue, 2 Mar 2021 20:50:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3499B64F21 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:34716 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHByh-00039m-V3 for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 15:50:47 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45642) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwb-0001N9-TZ for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:37 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:12584 helo=mx0a-001b2d01.pphosted.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwZ-00084w-A8 for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:37 -0500 Received: from pps.filterd (m0098416.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122KhIrW191148; Tue, 2 Mar 2021 15:48:31 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=VV2EcsYRdA7SCtuLO6OLWS5cJN2YfMieO7hR66jJjcA=; b=s2bdZwhCVRK/jHFFbkAvEz5D6E212OxVoy0OL77vCFEP1CK6+qh9hc6Q3YglN9CdXr/h 2KYbbYbPO/8n6ZqJ8ju6GvYN62de1MCIDOfgKq2xRGEoDKn/LSJSZQHMLzvcQKsjQ4W8 8chLjmBB6F3fIaz/yDwlhNOQUPLmtwbHJikj8AuNzzAf+1EydMy/PGUr7LxNaGuAZln+ JthVu0y6WqMF0g4XSsrIvZ76WPrzBhwg5OISQIPSRPhuiCyLnK/CRqFo9rL620xioB/N ouqDB4xw7xi1VVCxjTHTTJCsXMZIohQ+2C2Z55f2tqkbM80PhvqwqsOWKbzvEbMQm1UI vw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 371vn7r7te-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:31 -0500 Received: from m0098416.ppops.net (m0098416.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122KhM5h191662; Tue, 2 Mar 2021 15:48:30 -0500 Received: from ppma02wdc.us.ibm.com (aa.5b.37a9.ip4.static.sl-reverse.com [169.55.91.170]) by mx0b-001b2d01.pphosted.com with ESMTP id 371vn7r7sx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:30 -0500 Received: from pps.filterd (ppma02wdc.us.ibm.com [127.0.0.1]) by ppma02wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122KX2aI018077; Tue, 2 Mar 2021 20:48:30 GMT Received: from b01cxnp23033.gho.pok.ibm.com (b01cxnp23033.gho.pok.ibm.com [9.57.198.28]) by ppma02wdc.us.ibm.com with ESMTP id 3711dwtf6h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:30 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmT9g20709818 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:29 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 36B092806D; Tue, 2 Mar 2021 20:48:29 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 01D992805A; Tue, 2 Mar 2021 20:48:29 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:28 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 04/26] hw/boards: Add aux flag to CPUArchId Date: Tue, 2 Mar 2021 15:48:00 -0500 Message-Id: <20210302204822.81901-5-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 mlxlogscore=999 adultscore=0 suspectscore=0 spamscore=0 lowpriorityscore=0 mlxscore=0 phishscore=0 priorityscore=1501 bulkscore=0 impostorscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.158.5; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Eduardo Habkost , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The aux flag indicates whether a vcpu is auxiliary. Signed-off-by: Dov Murik --- include/hw/boards.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/include/hw/boards.h b/include/hw/boards.h index 7ee5c73510..4458b359c3 100644 --- a/include/hw/boards.h +++ b/include/hw/boards.h @@ -51,6 +51,7 @@ MemoryRegion *machine_consume_memdev(MachineState *machine, * @type - QOM class name of possible @cpu object * @props - CPU object properties, initialized by board * #vcpus_count - number of threads provided by @cpu object + * @aux - is this CPU auxiliary */ typedef struct CPUArchId { uint64_t arch_id; @@ -58,6 +59,7 @@ typedef struct CPUArchId { CpuInstanceProperties props; Object *cpu; const char *type; + bool aux; } CPUArchId; /** From patchwork Tue Mar 2 20:48:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112267 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA101C433DB for ; Tue, 2 Mar 2021 21:09:33 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2E06364F23 for ; Tue, 2 Mar 2021 21:09:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2E06364F23 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:57502 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHCGn-0008VD-UZ for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 16:09:29 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45994) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBx6-000218-07 for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:49:08 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:59058) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwp-00087M-DQ for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:49:07 -0500 Received: from pps.filterd (m0187473.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122Khgbn168482; Tue, 2 Mar 2021 15:48:38 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=0so7zAwdW56R4/Dmy2Bc0nTNU8ztVfE08VdYrOnCcNU=; b=sHnVbnD6mbvCoV2qIc1E1vTm9036MZ6rRX3vcrBcq3UuP0KSEm5s71GyDtwa4hXoKu1C iunGT9yAzD7/OlJhJcYZ4LLSpcPbKw43z8bRZ+8HbHVPNqg7UZ7ed04sKhfdxwKkwJ4D 7pObT2I+YXQHjub+boddCFSGAb5l6Anh1l67mLL6GMpNEe5fbdeutkZo660p2gZbN3V6 f96BEivVOlEOtFp8tglMCZHc1ZXq5uvDnCVQa9eTPbZecpEREBqBe/3Wqjv7ecFCPCjJ xh0egTo0p25+vl+qARvr3rPdfZx1S8ExCRDsA7RV5p1QJVoH/D51txYB+tUNYIEQ4Vlr Tw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vnd05jj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:37 -0500 Received: from m0187473.ppops.net (m0187473.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122Kht9g171154; Tue, 2 Mar 2021 15:48:34 -0500 Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vnd05gt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:33 -0500 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122KhDhj019585; Tue, 2 Mar 2021 20:48:30 GMT Received: from b01cxnp23033.gho.pok.ibm.com (b01cxnp23033.gho.pok.ibm.com [9.57.198.28]) by ppma02dal.us.ibm.com with ESMTP id 3710sqncd2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:30 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmThL29426060 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:29 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 841C52805E; Tue, 2 Mar 2021 20:48:29 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 47DAD2805A; Tue, 2 Mar 2021 20:48:29 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:29 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 05/26] hw/i386: Mark auxiliary vcpus in possible_cpus Date: Tue, 2 Mar 2021 15:48:01 -0500 Message-Id: <20210302204822.81901-6-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 impostorscore=0 spamscore=0 phishscore=0 lowpriorityscore=0 bulkscore=0 mlxscore=0 malwarescore=0 suspectscore=0 priorityscore=1501 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.156.1; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Eduardo Habkost , "Michael S. Tsirkin" , Richard Henderson , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Mark the last aux_cpus vcpus in the machine state's possible_cpus as auxiliary. Signed-off-by: Dov Murik --- hw/i386/x86.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/hw/i386/x86.c b/hw/i386/x86.c index 6329f90ef9..be23fad650 100644 --- a/hw/i386/x86.c +++ b/hw/i386/x86.c @@ -448,6 +448,7 @@ const CPUArchIdList *x86_possible_cpu_arch_ids(MachineState *ms) { X86MachineState *x86ms = X86_MACHINE(ms); unsigned int max_cpus = ms->smp.max_cpus; + unsigned int aux_cpus_start_at = max_cpus - ms->smp.aux_cpus; X86CPUTopoInfo topo_info; int i; @@ -475,6 +476,7 @@ const CPUArchIdList *x86_possible_cpu_arch_ids(MachineState *ms) x86_cpu_apic_id_from_index(x86ms, i); x86_topo_ids_from_apicid(ms->possible_cpus->cpus[i].arch_id, &topo_info, &topo_ids); + ms->possible_cpus->cpus[i].aux = i >= aux_cpus_start_at; ms->possible_cpus->cpus[i].props.has_socket_id = true; ms->possible_cpus->cpus[i].props.socket_id = topo_ids.pkg_id; if (x86ms->smp_dies > 1) { From patchwork Tue Mar 2 20:48:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112207 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48976C433E6 for ; Tue, 2 Mar 2021 20:50:51 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7A96060234 for ; Tue, 2 Mar 2021 20:50:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7A96060234 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:34876 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHByj-0003Dh-Ew for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 15:50:49 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45708) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwd-0001Oy-Ae for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:39 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:24122) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwb-00085I-8G for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:39 -0500 Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122KjIOn042909; Tue, 2 Mar 2021 15:48:32 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=Wx/vEAbRkX9QV8xxhblVvmPL07WccAn1wPbN6fuibac=; b=nfK0rfyPxBZ4l0BiBJkM4XqQxmarJnF/ezuMxGzZDkwcRXIogNKdUEY1cKZIwA8ZtzQQ OvPjfHV+xUow98QU6J5cVnIkDvT7A7t784x8EW5MiufMKoj7/6SaQKfghVCMVc/vknlg YH3W6qd9rNNo7jcTBxRXDj0BoTL4zEFX/AsNTlLZiC9tzdfixjSlHMN8KDoXrJbw57cT yt1Y63f5f3di4l2PFv0EmOEmXRttq6zlEcd5JSYZWxo2W0zN+6e05WzdgDZ+gw5bL9Uq LJntwytj94lkNmhiy2R7cbhhb+qyS63igoBKidmLBkhKxkggGkZ+WbwcbpfdE+zGGVtZ CQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vp083g1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:32 -0500 Received: from m0098409.ppops.net (m0098409.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122Kk8Ct045199; Tue, 2 Mar 2021 15:48:32 -0500 Received: from ppma04dal.us.ibm.com (7a.29.35a9.ip4.static.sl-reverse.com [169.53.41.122]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vp083f5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:32 -0500 Received: from pps.filterd (ppma04dal.us.ibm.com [127.0.0.1]) by ppma04dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122KmRQp008407; Tue, 2 Mar 2021 20:48:31 GMT Received: from b01cxnp23033.gho.pok.ibm.com (b01cxnp23033.gho.pok.ibm.com [9.57.198.28]) by ppma04dal.us.ibm.com with ESMTP id 36ydq988k0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:31 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmT3O25690586 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:30 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D564D28059; Tue, 2 Mar 2021 20:48:29 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9A8BE28065; Tue, 2 Mar 2021 20:48:29 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:29 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 06/26] hw/acpi: Don't include auxiliary vcpus in ACPI tables Date: Tue, 2 Mar 2021 15:48:02 -0500 Message-Id: <20210302204822.81901-7-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=860 clxscore=1015 mlxscore=0 malwarescore=0 adultscore=0 impostorscore=0 spamscore=0 bulkscore=0 phishscore=0 priorityscore=1501 suspectscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.156.1; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Eduardo Habkost , "Michael S. Tsirkin" , Richard Henderson , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini , Igor Mammedov Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Tobin Feldman-Fitzthum By excluding auxiliary vcpus from the ACPI tables, we hide them from the guest OS. This in turn allows OVMF to execute code on the auxiliary vcpus in parallel to the OS. Signed-off-by: Tobin Feldman-Fitzthum Signed-off-by: Dov Murik --- hw/acpi/cpu.c | 10 ++++++++++ hw/i386/acpi-build.c | 5 +++++ hw/i386/acpi-common.c | 5 +++++ 3 files changed, 20 insertions(+) diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c index e2317be546..d3f0a48da9 100644 --- a/hw/acpi/cpu.c +++ b/hw/acpi/cpu.c @@ -436,6 +436,11 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts, method = aml_method(CPU_NOTIFY_METHOD, 2, AML_NOTSERIALIZED); for (i = 0; i < arch_ids->len; i++) { + if (arch_ids->cpus[i].aux) { + /* don't build objects for auxiliary CPUs */ + continue; + } + Aml *cpu = aml_name(CPU_NAME_FMT, i); Aml *uid = aml_arg(0); Aml *event = aml_arg(1); @@ -651,6 +656,11 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts, /* build Processor object for each processor */ for (i = 0; i < arch_ids->len; i++) { + if (arch_ids->cpus[i].aux) { + /* don't build objects for auxiliary CPUs */ + continue; + } + Aml *dev; Aml *uid = aml_int(i); GArray *madt_buf = g_array_new(0, 1, 1); diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c index b9190b924a..8cc7e2af3a 100644 --- a/hw/i386/acpi-build.c +++ b/hw/i386/acpi-build.c @@ -1708,6 +1708,11 @@ build_srat(GArray *table_data, BIOSLinker *linker, MachineState *machine) srat->reserved1 = cpu_to_le32(1); for (i = 0; i < apic_ids->len; i++) { + if (apic_ids->cpus[i].aux) { + /* don't build objects for auxiliary CPUs */ + continue; + } + int node_id = apic_ids->cpus[i].props.node_id; uint32_t apic_id = apic_ids->cpus[i].arch_id; diff --git a/hw/i386/acpi-common.c b/hw/i386/acpi-common.c index 1f5947fcf9..7c73f3900b 100644 --- a/hw/i386/acpi-common.c +++ b/hw/i386/acpi-common.c @@ -91,6 +91,11 @@ void acpi_build_madt(GArray *table_data, BIOSLinker *linker, madt->flags = cpu_to_le32(1); for (i = 0; i < apic_ids->len; i++) { + if (apic_ids->cpus[i].aux) { + /* don't build objects for auxiliary CPUs */ + continue; + } + adevc->madt_cpu(adev, i, apic_ids, table_data); if (apic_ids->cpus[i].arch_id > 254) { x2apic_mode = true; From patchwork Tue Mar 2 20:48:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112217 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00FC1C433E0 for ; Tue, 2 Mar 2021 20:56:34 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4140F64F21 for ; Tue, 2 Mar 2021 20:56:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4140F64F21 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:51950 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHC4H-0001tI-1Y for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 15:56:33 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45738) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwe-0001Qz-89 for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:40 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:35114) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwc-00085r-EI for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:39 -0500 Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122Kinwe125403; Tue, 2 Mar 2021 15:48:35 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=3QI+ldJIrLBSOG2cd0JxX+2zevxf0LjjRw8OcOvyBlg=; b=LvTYt9wl+eWrWQarPSXdl65mEMs2I0yshY2suNe9BThL7erbx13fTjTSUxo9hSpFehAO bx1kCfSqy4SVOO3LHl8fz+w9H/7YiSd40DKunT9O6MYfYDhFwtVuZSHvwxu9aU7BTBDm 1nJ2rxxrvyNTTWZ6wGY9gZ7EvBdaupCi8vjusT9+vbNDlNx5BsAJ3N+gImUAd4g+N/M4 7qUB3IEhLRnpZpmppfYT9Ac682Bo3Vw3ZfI4eLPrBdDGCCmCIATVTuYPiruNgib32PCi dOLNSGu730txk/1MNNow8bkxdifrA4KPYBVLPyzgVBLIZi4QKcA6qKtlq1mRh6Ac9TwH 6g== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vnt055v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:35 -0500 Received: from m0098404.ppops.net (m0098404.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122Kj7DZ127729; Tue, 2 Mar 2021 15:48:32 -0500 Received: from ppma03wdc.us.ibm.com (ba.79.3fa9.ip4.static.sl-reverse.com [169.63.121.186]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vnt054s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:32 -0500 Received: from pps.filterd (ppma03wdc.us.ibm.com [127.0.0.1]) by ppma03wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122Kh8Ld015679; Tue, 2 Mar 2021 20:48:31 GMT Received: from b01cxnp23034.gho.pok.ibm.com (b01cxnp23034.gho.pok.ibm.com [9.57.198.29]) by ppma03wdc.us.ibm.com with ESMTP id 37128ga1vg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:31 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmUYC42140062 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:30 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 293DC2805E; Tue, 2 Mar 2021 20:48:30 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E73EF28067; Tue, 2 Mar 2021 20:48:29 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:29 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 07/26] cpu: Add boolean aux field to CPUState Date: Tue, 2 Mar 2021 15:48:03 -0500 Message-Id: <20210302204822.81901-8-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 mlxscore=0 suspectscore=0 mlxlogscore=999 priorityscore=1501 lowpriorityscore=0 clxscore=1015 adultscore=0 spamscore=0 phishscore=0 impostorscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.156.1; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Eduardo Habkost , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The aux field indicates auxiliary CPUs. This will allow QEMU to act differently on auxilirary CPUs, for example pausing the VM while keeping the auxiliary CPUs running. Signed-off-by: Dov Murik --- include/hw/core/cpu.h | 2 ++ hw/core/cpu.c | 1 + 2 files changed, 3 insertions(+) diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index c005d3dc2d..cdbb6462a3 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -430,6 +430,8 @@ struct CPUState { int hvf_fd; + bool aux; + /* track IOMMUs whose translations we've cached in the TCG TLB */ GArray *iommu_notifiers; }; diff --git a/hw/core/cpu.c b/hw/core/cpu.c index 00330ba07d..2100cb7f1a 100644 --- a/hw/core/cpu.c +++ b/hw/core/cpu.c @@ -387,6 +387,7 @@ static Property cpu_common_props[] = { MemoryRegion *), #endif DEFINE_PROP_BOOL("start-powered-off", CPUState, start_powered_off, false), + DEFINE_PROP_BOOL("aux", CPUState, aux, false), DEFINE_PROP_END_OF_LIST(), }; From patchwork Tue Mar 2 20:48:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112265 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03454C433E0 for ; Tue, 2 Mar 2021 21:07:46 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5785364F23 for ; Tue, 2 Mar 2021 21:07:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5785364F23 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:52560 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHCF1-0006LM-6g for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 16:07:44 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45806) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwg-0001WV-Bq for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:42 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:33836) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwe-00086k-8L for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:42 -0500 Received: from pps.filterd (m0187473.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122KhgMk168465; Tue, 2 Mar 2021 15:48:36 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=8hyH56LRkpvTv0mf4DTpS+OO6A4Mif12MqDVQqWX1jA=; b=BTQbiMYVbYG2ccAG1kS3FDFrbC44AusTVTtg21H0ibbNzSLg6qvLPioXcCTJtfEW+kSv keAqWtBHw907aHfZ+ncG/wVEis2WvyoXT7fypQ94Ajmmv2WWCjASS/H7umL302i2O1YB 6P+ZDQPrReEBAffxZTlH9hhAwNBFwNlBRtciIq0jZzJfCbZBdmac/TkmJVZK8acQ41MT j7ZV37set/qIF2Hl16VVv3SdBy65iQuTd3vb3bWmBx4UACB/RG3VtpT8BRlAX2rveGaD PIsCsdGzVd4JlBg0Hpg5tkTJzD3oKfgyD5gZr7BZo9BKGRMCm0P0VkXSWx1WqMSZbRz/ Sw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vnd05j4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:35 -0500 Received: from m0187473.ppops.net (m0187473.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122Khis3168579; Tue, 2 Mar 2021 15:48:33 -0500 Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vnd05h9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:33 -0500 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122Kgw7l019511; Tue, 2 Mar 2021 20:48:31 GMT Received: from b01cxnp23034.gho.pok.ibm.com (b01cxnp23034.gho.pok.ibm.com [9.57.198.29]) by ppma02dal.us.ibm.com with ESMTP id 3710sqncda-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:31 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmUX440042850 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:30 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 728292805C; Tue, 2 Mar 2021 20:48:30 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3C50728065; Tue, 2 Mar 2021 20:48:30 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:30 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 08/26] hw/i386: Set CPUState.aux=true for auxiliary vcpus Date: Tue, 2 Mar 2021 15:48:04 -0500 Message-Id: <20210302204822.81901-9-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 impostorscore=0 spamscore=0 phishscore=0 lowpriorityscore=0 bulkscore=0 mlxscore=0 malwarescore=0 suspectscore=0 priorityscore=1501 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.156.1; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Eduardo Habkost , "Michael S. Tsirkin" , Richard Henderson , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" On x86 machines, when initializing the CPUState structs, set the aux flag to true for auxiliary vcpus. Signed-off-by: Dov Murik --- include/hw/i386/x86.h | 2 +- hw/i386/x86.c | 8 ++++++-- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/include/hw/i386/x86.h b/include/hw/i386/x86.h index 56080bd1fb..f9ec6af9b7 100644 --- a/include/hw/i386/x86.h +++ b/include/hw/i386/x86.h @@ -85,7 +85,7 @@ void init_topo_info(X86CPUTopoInfo *topo_info, const X86MachineState *x86ms); uint32_t x86_cpu_apic_id_from_index(X86MachineState *pcms, unsigned int cpu_index); -void x86_cpu_new(X86MachineState *pcms, int64_t apic_id, Error **errp); +void x86_cpu_new(X86MachineState *pcms, int64_t apic_id, bool aux, Error **errp); void x86_cpus_init(X86MachineState *pcms, int default_cpu_version); CpuInstanceProperties x86_cpu_index_to_props(MachineState *ms, unsigned cpu_index); diff --git a/hw/i386/x86.c b/hw/i386/x86.c index be23fad650..bc17b53180 100644 --- a/hw/i386/x86.c +++ b/hw/i386/x86.c @@ -101,13 +101,16 @@ uint32_t x86_cpu_apic_id_from_index(X86MachineState *x86ms, } -void x86_cpu_new(X86MachineState *x86ms, int64_t apic_id, Error **errp) +void x86_cpu_new(X86MachineState *x86ms, int64_t apic_id, bool aux, Error **errp) { Object *cpu = object_new(MACHINE(x86ms)->cpu_type); if (!object_property_set_uint(cpu, "apic-id", apic_id, errp)) { goto out; } + if (!object_property_set_bool(cpu, "aux", aux, errp)) { + goto out; + } qdev_realize(DEVICE(cpu), NULL, errp); out: @@ -135,7 +138,8 @@ void x86_cpus_init(X86MachineState *x86ms, int default_cpu_version) ms->smp.max_cpus - 1) + 1; possible_cpus = mc->possible_cpu_arch_ids(ms); for (i = 0; i < ms->smp.cpus; i++) { - x86_cpu_new(x86ms, possible_cpus->cpus[i].arch_id, &error_fatal); + x86_cpu_new(x86ms, possible_cpus->cpus[i].arch_id, + possible_cpus->cpus[i].aux, &error_fatal); } } From patchwork Tue Mar 2 20:48:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112213 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5357EC433E0 for ; Tue, 2 Mar 2021 20:53:34 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B108964F11 for ; Tue, 2 Mar 2021 20:53:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B108964F11 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:43512 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHC1M-0006oG-ML for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 15:53:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45744) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwe-0001RY-GD for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:40 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:31544) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwb-00085Y-LP for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:40 -0500 Received: from pps.filterd (m0098393.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122KiFnJ104172; Tue, 2 Mar 2021 15:48:33 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=GRt7Z08VIR4C5ZF2ZcLcM0RHjunGkBrMkk+3l5llNds=; b=e0IQOVX2shfEB06+NJlccIr6QUkeqMUSuAu+9GjeX//JpKaZBh8xa1PnUIB8b8ipT/pB yurhZ0H5EmcU9tTMRBedQyDRdInhDNNhUHWgBc3rZkc7X/nwervaa+UrxkTjjcbZIauO NTmZqG1bzPSQ4YsZZGt35feofhgGuvsYov0tldDn7/sFv4Vof9ifWERyXb1yc0g0M7tb 9rOaVxtkR0ctFqQ4q7Laj769vLhzrJPqMkH/PYj6paORGd6lIkTUyypPOc2SRhdnmXy6 AlNkoFEljzFGInRZO/lL5CuhOv6CPQ/G2ERGxfhnyT0WfY8E/9QHdpHWCVO5qOqc5Dog 2Q== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vnh05xa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:33 -0500 Received: from m0098393.ppops.net (m0098393.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122KiPSI108827; Tue, 2 Mar 2021 15:48:32 -0500 Received: from ppma05wdc.us.ibm.com (1b.90.2fa9.ip4.static.sl-reverse.com [169.47.144.27]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vnh05wj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:32 -0500 Received: from pps.filterd (ppma05wdc.us.ibm.com [127.0.0.1]) by ppma05wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122KmLvt016520; Tue, 2 Mar 2021 20:48:31 GMT Received: from b01cxnp23034.gho.pok.ibm.com (b01cxnp23034.gho.pok.ibm.com [9.57.198.29]) by ppma05wdc.us.ibm.com with ESMTP id 371b00xpwk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:31 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmUdx40501556 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:30 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id BA0A528059; Tue, 2 Mar 2021 20:48:30 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8363028060; Tue, 2 Mar 2021 20:48:30 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:30 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 09/26] softmmu: Don't sync aux vcpus in pre_loadvm Date: Tue, 2 Mar 2021 15:48:05 -0500 Message-Id: <20210302204822.81901-10-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 bulkscore=0 adultscore=0 mlxlogscore=999 clxscore=1015 malwarescore=0 phishscore=0 lowpriorityscore=0 priorityscore=1501 suspectscore=0 impostorscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.156.1; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Richard Henderson , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" When preparing to load state into a migration target VM, don't sync the auxiliary vcpus to allow the migration helper to keep running there (it is used during the migration). Signed-off-by: Dov Murik --- softmmu/cpus.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/softmmu/cpus.c b/softmmu/cpus.c index a7ee431187..878cf2a421 100644 --- a/softmmu/cpus.c +++ b/softmmu/cpus.c @@ -162,7 +162,9 @@ void cpu_synchronize_all_pre_loadvm(void) CPUState *cpu; CPU_FOREACH(cpu) { - cpu_synchronize_pre_loadvm(cpu); + if (!cpu->aux) { + cpu_synchronize_pre_loadvm(cpu); + } } } From patchwork Tue Mar 2 20:48:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112273 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 709C5C433DB for ; Tue, 2 Mar 2021 21:18:25 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C437064F2A for ; Tue, 2 Mar 2021 21:18:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C437064F2A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:49562 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHCPP-0000eC-4H for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 16:18:23 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45972) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBx3-00020b-Ue for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:49:07 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:54112) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwo-00086s-Hx for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:49:04 -0500 Received: from pps.filterd (m0187473.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122Khgbm168482; Tue, 2 Mar 2021 15:48:37 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=VupSK9vxmDvSP6JLi0c6Yu9Pi+xtlZZyF6WflQwMfxs=; b=mik5DVqt6+15FtUzTJh+B8IMxC6Erkzl5npE8GplPfggh2RLXfnv16DFu1fQp+hYNVNT +trSzluFO8Eqxn/FNXjvQa12o0FQJuDUP+k9u3ZF8MN72U/OTlHuALvbTLMhK+hvSrIs 3hqNj+aYs5kU6SEvhql+ZLcVpktT4cZUat/u80LNhPOPH17eGGSaUxe32R7AfFYjHQBF +Md4CxwNtGV+bZ/yz5mdlobfK8pwFOPLcfva/f/ihurKyEuPzxQa0AmMdOcmbdleFKAW zCyuZ5FBTNBrITa+mWkkhLCVYnsUBwGlB/vCAz0NaD6dOH/i/w402oU1G1b0IwP/QKKz ug== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vnd05jd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:36 -0500 Received: from m0187473.ppops.net (m0187473.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122KhnZW168816; Tue, 2 Mar 2021 15:48:33 -0500 Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vnd05hr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:33 -0500 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122KkqnI028108; Tue, 2 Mar 2021 20:48:32 GMT Received: from b01cxnp22033.gho.pok.ibm.com (b01cxnp22033.gho.pok.ibm.com [9.57.198.23]) by ppma01dal.us.ibm.com with ESMTP id 371qmuagu8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:32 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmVDV25887066 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:31 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 04D9028058; Tue, 2 Mar 2021 20:48:31 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CCD072805E; Tue, 2 Mar 2021 20:48:30 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:30 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 10/26] softmmu: Add cpu_synchronize_without_aux_post_init Date: Tue, 2 Mar 2021 15:48:06 -0500 Message-Id: <20210302204822.81901-11-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 impostorscore=0 spamscore=0 phishscore=0 lowpriorityscore=0 bulkscore=0 mlxscore=0 malwarescore=0 suspectscore=0 priorityscore=1501 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.156.1; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Richard Henderson , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This will be used during migration on the target. Signed-off-by: Dov Murik --- include/sysemu/cpus.h | 1 + softmmu/cpus.c | 11 +++++++++++ 2 files changed, 12 insertions(+) diff --git a/include/sysemu/cpus.h b/include/sysemu/cpus.h index 868f1192de..dc24e38254 100644 --- a/include/sysemu/cpus.h +++ b/include/sysemu/cpus.h @@ -46,6 +46,7 @@ bool cpus_are_resettable(void); void cpu_synchronize_all_states(void); void cpu_synchronize_all_post_reset(void); void cpu_synchronize_all_post_init(void); +void cpu_synchronize_without_aux_post_init(void); void cpu_synchronize_all_pre_loadvm(void); #ifndef CONFIG_USER_ONLY diff --git a/softmmu/cpus.c b/softmmu/cpus.c index 878cf2a421..68fa4639a7 100644 --- a/softmmu/cpus.c +++ b/softmmu/cpus.c @@ -157,6 +157,17 @@ void cpu_synchronize_all_post_init(void) } } +void cpu_synchronize_without_aux_post_init(void) +{ + CPUState *cpu; + + CPU_FOREACH(cpu) { + if (!cpu->aux) { + cpu_synchronize_post_init(cpu); + } + } +} + void cpu_synchronize_all_pre_loadvm(void) { CPUState *cpu; From patchwork Tue Mar 2 20:48:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112205 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 253A7C433E9 for ; Tue, 2 Mar 2021 20:50:50 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3411260234 for ; Tue, 2 Mar 2021 20:50:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3411260234 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:34706 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHByi-00039a-1V for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 15:50:48 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45654) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwc-0001NS-5J for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:38 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:33296) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwa-00085E-Ez for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:37 -0500 Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122KhIIr124566; Tue, 2 Mar 2021 15:48:33 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=8ZOaX8pUhxLDYxV8GkmhAwqnKTZb51Fhp3sOw/Kv6v0=; b=eECjofdEJ9dmBtpre2h2C13AJKObi2Iy8hQG1RkQMjGWtECa2YCI3Jzqp653Z1/jTNeH FzzKSYr9nihpbB0FqzYd46dEANATAb+Qrp0h0c41lZSwrLWAIHRV1qeSYtEvrX7Ej6kX BD4zaNhyORLYhI4fjtI0KN7CfA7H233q7TEYFsiICIwLuvJF/+eh4m3CiXLRV9dS4VDy 5qaRJawDhhOpV8UkZXXugiy/FA/jVDDzAWuUa2mhUQw0BsUv29oOvBEXSjFhwFmCd76X gL7HqVpxbS3FJG8hJtZnJb+qywlp7WTw53D8M4mb8465V+tlpGaMb1v6Mz2Q3Os03zdu Qg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vn7r7cy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:33 -0500 Received: from m0098421.ppops.net (m0098421.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122KiJl2138163; Tue, 2 Mar 2021 15:48:32 -0500 Received: from ppma03wdc.us.ibm.com (ba.79.3fa9.ip4.static.sl-reverse.com [169.63.121.186]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vn7r7cm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:32 -0500 Received: from pps.filterd (ppma03wdc.us.ibm.com [127.0.0.1]) by ppma03wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122KhNUC016036; Tue, 2 Mar 2021 20:48:32 GMT Received: from b01cxnp22033.gho.pok.ibm.com (b01cxnp22033.gho.pok.ibm.com [9.57.198.23]) by ppma03wdc.us.ibm.com with ESMTP id 37128ga1vq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:32 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmV0t18022732 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:31 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4227E2805A; Tue, 2 Mar 2021 20:48:31 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2095F28066; Tue, 2 Mar 2021 20:48:31 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:31 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 11/26] softmmu: Add pause_all_vcpus_except_aux Date: Tue, 2 Mar 2021 15:48:07 -0500 Message-Id: <20210302204822.81901-12-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 mlxscore=0 clxscore=1015 priorityscore=1501 spamscore=0 bulkscore=0 adultscore=0 malwarescore=0 suspectscore=0 mlxlogscore=999 lowpriorityscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.158.5; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0b-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Richard Henderson , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Tobin Feldman-Fitzthum Introduce a function to pause all CPUs except the auxiliary CPUs. This will be used during migration when a migration handler is running on the auxiliary CPU. Co-Author: Dov Murik Signed-off-by: Dov Murik Signed-off-by: Tobin Feldman-Fitzthum --- include/sysemu/cpus.h | 1 + softmmu/cpus.c | 53 +++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 52 insertions(+), 2 deletions(-) diff --git a/include/sysemu/cpus.h b/include/sysemu/cpus.h index dc24e38254..e668570053 100644 --- a/include/sysemu/cpus.h +++ b/include/sysemu/cpus.h @@ -34,6 +34,7 @@ bool qemu_in_vcpu_thread(void); void qemu_init_cpu_loop(void); void resume_all_vcpus(void); void pause_all_vcpus(void); +void pause_all_vcpus_except_aux(void); void cpu_stop_current(void); extern int icount_align_option; diff --git a/softmmu/cpus.c b/softmmu/cpus.c index 68fa4639a7..3028b5d0d4 100644 --- a/softmmu/cpus.c +++ b/softmmu/cpus.c @@ -550,6 +550,19 @@ static bool all_vcpus_paused(void) return true; } +static bool all_vcpus_except_aux_paused(void) +{ + CPUState *cpu; + + CPU_FOREACH(cpu) { + if (!cpu->aux && !cpu->stopped) { + return false; + } + } + + return true; +} + void pause_all_vcpus(void) { CPUState *cpu; @@ -564,15 +577,51 @@ void pause_all_vcpus(void) } } + /* + * Drop the replay_lock so any vCPU threads woken up can finish their + * replay tasks + */ + replay_mutex_unlock(); + + while (!all_vcpus_paused()) { + qemu_cond_wait(&qemu_pause_cond, &qemu_global_mutex); + CPU_FOREACH(cpu) { + qemu_cpu_kick(cpu); + } + } + + qemu_mutex_unlock_iothread(); + replay_mutex_lock(); + qemu_mutex_lock_iothread(); +} + +void pause_all_vcpus_except_aux(void) +{ + CPUState *cpu; + + qemu_clock_enable(QEMU_CLOCK_VIRTUAL, false); + CPU_FOREACH(cpu) { + if (!cpu->aux) { + if (qemu_cpu_is_self(cpu)) { + qemu_cpu_stop(cpu, true); + } else { + cpu->stop = true; + qemu_cpu_kick(cpu); + } + } + } + /* We need to drop the replay_lock so any vCPU threads woken up * can finish their replay tasks */ replay_mutex_unlock(); - while (!all_vcpus_paused()) { + while (!all_vcpus_except_aux_paused()) { qemu_cond_wait(&qemu_pause_cond, &qemu_global_mutex); CPU_FOREACH(cpu) { - qemu_cpu_kick(cpu); + if (!cpu->aux) { + qemu_cpu_kick(cpu); + } } } From patchwork Tue Mar 2 20:48:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112215 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76022C433DB for ; Tue, 2 Mar 2021 20:54:08 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C57DE60234 for ; Tue, 2 Mar 2021 20:54:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C57DE60234 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:46606 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHC1u-00083W-N4 for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 15:54:06 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45676) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwc-0001Nl-KF for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:38 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:27904 helo=mx0a-001b2d01.pphosted.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwZ-000858-QA for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:38 -0500 Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122Khpa2055083; Tue, 2 Mar 2021 15:48:34 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=PK60rtl9R0bqzdhzZjLQ2LNHuZU+osf/T1velZPNeKw=; b=VKWsfUhvpbFYWa5F6whTp0wgSao4tpAThd12wBysMIopi0vxDAS1c0XJKRmTtz5o+Qgi IHg3vm1ubWF+ctrTCdnBAfXTPTbZ8K4BsLiMMQCE176njyiQ4oew9+PfiuopaXydY2e6 aDbO/yCncHNiQikZdDhTLtJSEmnVV0DVT6zFwbj2FhNYHA1HxtWGfUlJB8HEpQH8dwyZ 4WXvr6QBNSitfcLZNIcA8PPUCWGgR2/yHGiEsl0cFkbK6FhQQNFV+YkFwlC3ywQtHP96 6MwwFLxISEqXfaOz0pq0t9Gm5Co8k3qkzooXtTIzga9OUl2novHv4Hmpw/kKQd66EYSE EA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 371vn9g6nv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:33 -0500 Received: from m0098419.ppops.net (m0098419.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122KhwGJ058607; Tue, 2 Mar 2021 15:48:33 -0500 Received: from ppma01wdc.us.ibm.com (fd.55.37a9.ip4.static.sl-reverse.com [169.55.85.253]) by mx0b-001b2d01.pphosted.com with ESMTP id 371vn9g6nh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:33 -0500 Received: from pps.filterd (ppma01wdc.us.ibm.com [127.0.0.1]) by ppma01wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122Kc18O027219; Tue, 2 Mar 2021 20:48:32 GMT Received: from b01cxnp22033.gho.pok.ibm.com (b01cxnp22033.gho.pok.ibm.com [9.57.198.23]) by ppma01wdc.us.ibm.com with ESMTP id 36ydq91sa1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:32 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmV7q20185482 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:31 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 972C72805E; Tue, 2 Mar 2021 20:48:31 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 63EB828058; Tue, 2 Mar 2021 20:48:31 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:31 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 12/26] migration: Add helpers to save confidential RAM Date: Tue, 2 Mar 2021 15:48:08 -0500 Message-Id: <20210302204822.81901-13-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 impostorscore=0 suspectscore=0 priorityscore=1501 malwarescore=0 mlxscore=0 lowpriorityscore=0 clxscore=1015 spamscore=0 adultscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.158.5; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Juan Quintela , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , "Dr. David Alan Gilbert" , Hubertus Franke , Tobin Feldman-Fitzthum Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" QEMU cannot read the memory of memory-encrypted guests, which is required for sending RAM to the migration target. Instead, QEMU asks a migration helper running on an auxiliary vcpu in the guest to extract pages from memory; these pages are encrypted with a transfer key that is known to the source and target guests, but not to both QEMUs. The interaction with the guest migration helper is performed using two shared (unencrypted) pages which both QEMU and guest can read from and write to. The details of the mailbox protocol are described in migration/confidential-ram.c. Signed-off-by: Dov Murik --- migration/confidential-ram.h | 17 ++++ migration/confidential-ram.c | 184 +++++++++++++++++++++++++++++++++++ migration/meson.build | 6 +- migration/trace-events | 3 + 4 files changed, 209 insertions(+), 1 deletion(-) create mode 100644 migration/confidential-ram.h create mode 100644 migration/confidential-ram.c diff --git a/migration/confidential-ram.h b/migration/confidential-ram.h new file mode 100644 index 0000000000..0d49718d31 --- /dev/null +++ b/migration/confidential-ram.h @@ -0,0 +1,17 @@ +/* + * QEMU migration for confidential guest's RAM + */ + +#ifndef QEMU_CONFIDENTIAL_RAM_H +#define QEMU_CONFIDENTIAL_RAM_H + +#include "exec/cpu-common.h" +#include "qemu-file.h" + +void cgs_mh_init(void); +void cgs_mh_cleanup(void); + +int cgs_mh_save_encrypted_page(QEMUFile *f, ram_addr_t src_gpa, uint32_t size, + uint64_t *bytes_sent); + +#endif diff --git a/migration/confidential-ram.c b/migration/confidential-ram.c new file mode 100644 index 0000000000..65a588e7f6 --- /dev/null +++ b/migration/confidential-ram.c @@ -0,0 +1,184 @@ +#include "qemu/osdep.h" +#include "cpu.h" +#include "qemu/osdep.h" +#include "qemu/error-report.h" +#include "qemu/rcu.h" +#include "qemu/coroutine.h" +#include "qemu/timer.h" +#include "io/channel.h" +#include "qapi/error.h" +#include "exec/memory.h" +#include "trace.h" +#include "confidential-ram.h" + +enum cgs_mig_helper_cmd { + /* Initialize migration helper in guest */ + CGS_MIG_HELPER_CMD_INIT = 0, + + /* + * Fetch a page from gpa, encrypt it, and save result into the shared page + */ + CGS_MIG_HELPER_CMD_ENCRYPT, + + /* Read the shared page, decrypt it, and save result into gpa */ + CGS_MIG_HELPER_CMD_DECRYPT, + + /* Reset migration helper in guest */ + CGS_MIG_HELPER_CMD_RESET, + + CGS_MIG_HELPER_CMD_MAX +}; + +struct QEMU_PACKED CGSMigHelperCmdParams { + uint64_t cmd_type; + uint64_t gpa; + int32_t prefetch; + int32_t ret; + int32_t go; + int32_t done; +}; +typedef struct CGSMigHelperCmdParams CGSMigHelperCmdParams; + +struct QEMU_PACKED CGSMigHelperPageHeader { + uint32_t len; + uint8_t data[0]; +}; +typedef struct CGSMigHelperPageHeader CGSMigHelperPageHeader; + +struct CGSMigHelperState { + CGSMigHelperCmdParams *cmd_params; + CGSMigHelperPageHeader *io_page_hdr; + uint8_t *io_page; + bool initialized; +}; +typedef struct CGSMigHelperState CGSMigHelperState; + +static CGSMigHelperState cmhs = {0}; + +#define MH_BUSYLOOP_TIMEOUT 100000000LL +#define MH_REQUEST_TIMEOUT_MS 100 +#define MH_REQUEST_TIMEOUT_NS (MH_REQUEST_TIMEOUT_MS * 1000 * 1000) + +/* + * The migration helper shared area is hard-coded at gpa 0x820000 with size of + * 2 pages (0x2000 bytes). Instead of hard-coding, the address and size may be + * fetched from OVMF itself using a pc_system_ovmf_table_find call to query + * OVMF's GUIDed structure for a migration helper GUID. + */ +#define MH_SHARED_CMD_PARAMS_ADDR 0x820000 +#define MH_SHARED_IO_PAGE_HDR_ADDR (MH_SHARED_CMD_PARAMS_ADDR + 0x800) +#define MH_SHARED_IO_PAGE_ADDR (MH_SHARED_CMD_PARAMS_ADDR + 0x1000) + +void cgs_mh_init(void) +{ + RCU_READ_LOCK_GUARD(); + cmhs.cmd_params = qemu_map_ram_ptr(NULL, MH_SHARED_CMD_PARAMS_ADDR); + cmhs.io_page_hdr = qemu_map_ram_ptr(NULL, MH_SHARED_IO_PAGE_HDR_ADDR); + cmhs.io_page = qemu_map_ram_ptr(NULL, MH_SHARED_IO_PAGE_ADDR); +} + +static int send_command_to_cgs_mig_helper(uint64_t cmd_type, uint64_t gpa) +{ + /* + * The cmd_params struct is on a page shared with the guest migration + * helper. We use a volatile struct to force writes to memory so that the + * guest can see them. + */ + volatile CGSMigHelperCmdParams *params = cmhs.cmd_params; + int64_t counter, request_timeout_at; + + /* + * At this point io_page and io_page_hdr should be already filled according + * to the requested cmd_type. + */ + + params->cmd_type = cmd_type; + params->gpa = gpa; + params->prefetch = 0; + params->ret = -1; + params->done = 0; + + /* + * Force writes of all command parameters before writing the 'go' flag. + * The guest migration handler waits for the go flag and then reads the + * command parameters. + */ + smp_wmb(); + + /* Tell the migration helper to start working on this command */ + params->go = 1; + + /* + * Wait for the guest migration helper to process the command and mark the + * done flag + */ + request_timeout_at = qemu_clock_get_ns(QEMU_CLOCK_REALTIME) + + MH_REQUEST_TIMEOUT_NS; + do { + counter = 0; + while (!params->done && (counter < MH_BUSYLOOP_TIMEOUT)) { + counter++; + } + } while (!params->done && + qemu_clock_get_ns(QEMU_CLOCK_REALTIME) < request_timeout_at); + + if (!params->done) { + error_report("Migration helper command %" PRIu64 " timed-out for " + "gpa 0x%" PRIx64, cmd_type, gpa); + return -EIO; + } + + return params->ret; +} + +static void init_cgs_mig_helper_if_needed(void) +{ + int ret; + + if (cmhs.initialized) { + return; + } + + ret = send_command_to_cgs_mig_helper(CGS_MIG_HELPER_CMD_INIT, 0); + if (ret == 0) { + cmhs.initialized = true; + } +} + +void cgs_mh_cleanup(void) +{ + send_command_to_cgs_mig_helper(CGS_MIG_HELPER_CMD_RESET, 0); +} + +int cgs_mh_save_encrypted_page(QEMUFile *f, ram_addr_t src_gpa, uint32_t size, + uint64_t *bytes_sent) +{ + int ret; + + init_cgs_mig_helper_if_needed(); + + /* Ask the migration helper to encrypt the page at src_gpa */ + trace_encrypted_ram_save_page(size, src_gpa); + ret = send_command_to_cgs_mig_helper(CGS_MIG_HELPER_CMD_ENCRYPT, src_gpa); + if (ret) { + error_report("Error cgs_mh_save_encrypted_page ret=%d", ret); + return -1; + } + + /* Sanity check for response header */ + if (cmhs.io_page_hdr->len > 1024) { + error_report("confidential-ram: migration helper response is too large " + "(len=%u)", cmhs.io_page_hdr->len); + return -EINVAL; + } + + qemu_put_be32(f, cmhs.io_page_hdr->len); + qemu_put_buffer(f, cmhs.io_page_hdr->data, cmhs.io_page_hdr->len); + *bytes_sent = 4 + cmhs.io_page_hdr->len; + + qemu_put_be32(f, size); + qemu_put_buffer(f, cmhs.io_page, size); + *bytes_sent += 4 + size; + + return ret; +} diff --git a/migration/meson.build b/migration/meson.build index 9645f44005..95fe7bdd12 100644 --- a/migration/meson.build +++ b/migration/meson.build @@ -30,4 +30,8 @@ softmmu_ss.add(when: ['CONFIG_RDMA', rdma], if_true: files('rdma.c')) softmmu_ss.add(when: 'CONFIG_LIVE_BLOCK_MIGRATION', if_true: files('block.c')) softmmu_ss.add(when: zstd, if_true: files('multifd-zstd.c')) -specific_ss.add(when: 'CONFIG_SOFTMMU', if_true: files('dirtyrate.c', 'ram.c')) +specific_ss.add(when: 'CONFIG_SOFTMMU', if_true: files( + 'dirtyrate.c', + 'ram.c', + 'confidential-ram.c', +)) diff --git a/migration/trace-events b/migration/trace-events index 668c562fed..929de4ca98 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -343,3 +343,6 @@ migration_block_save_pending(uint64_t pending) "Enter save live pending %" PRIu # page_cache.c migration_pagecache_init(int64_t max_num_items) "Setting cache buckets to %" PRId64 migration_pagecache_insert(void) "Error allocating page" + +# confidential-ram.c +encrypted_ram_save_page(uint32_t size, uint64_t gpa) "size: %u, gpa: 0x%" PRIx64 From patchwork Tue Mar 2 20:48:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112231 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51068C433DB for ; Tue, 2 Mar 2021 21:00:33 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C18E660234 for ; Tue, 2 Mar 2021 21:00:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C18E660234 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:35428 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHC87-0006vy-Gh for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 16:00:31 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45734) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwe-0001Qg-2s for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:40 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:36240) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwc-00085g-5Q for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:39 -0500 Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122KiCZJ092135; Tue, 2 Mar 2021 15:48:35 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=cadysZuPx8vVG+UCWHCyjS7plfrBRgsJd08921TQ8hA=; b=BRl76bPHJpZaFTM1Fx/NpU1bNqkBm9RIHRw5Rn3pW4Cc/Ykd9KEoOLoia2khoNM4Wxw3 AhjHtEbAA9DDAFKb4mO4Ba8fvW27A97FUwT44cGMlog6ocdYrogJEenAUCdPh5/tUiTT 7jsNZYVfd9H/P+ld5LTSGk7UL4RDYiX2YsKojCdojG0cWDlCj1GK3wXq6fQ8zBwRapYy UOJZ0zeJ56GAinoZWPQml6LCtNSOTkv3V9A3Mx+vr+NKI/W97gcFpWTstLmeHzbWy63/ wGcg8cxnx6wG1jX5fgrMvGAR0ClEXwKVFQrQERFkF55le8a/e+nj1DHiLlhvqRBw/8nW hQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vnf0657-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:34 -0500 Received: from m0098399.ppops.net (m0098399.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122KkeTc112164; Tue, 2 Mar 2021 15:48:34 -0500 Received: from ppma04wdc.us.ibm.com (1a.90.2fa9.ip4.static.sl-reverse.com [169.47.144.26]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vnf064e-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:34 -0500 Received: from pps.filterd (ppma04wdc.us.ibm.com [127.0.0.1]) by ppma04wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122Km3td009935; Tue, 2 Mar 2021 20:48:32 GMT Received: from b01cxnp22033.gho.pok.ibm.com (b01cxnp22033.gho.pok.ibm.com [9.57.198.23]) by ppma04wdc.us.ibm.com with ESMTP id 3712phhu85-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:32 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmVwd15073740 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:32 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D7E2B28059; Tue, 2 Mar 2021 20:48:31 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AD2B328058; Tue, 2 Mar 2021 20:48:31 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:31 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 13/26] migration: Add helpers to load confidential RAM Date: Tue, 2 Mar 2021 15:48:09 -0500 Message-Id: <20210302204822.81901-14-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 mlxlogscore=999 malwarescore=0 impostorscore=0 phishscore=0 bulkscore=0 lowpriorityscore=0 mlxscore=0 suspectscore=0 priorityscore=1501 spamscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.156.1; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Juan Quintela , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , "Dr. David Alan Gilbert" , Hubertus Franke , Tobin Feldman-Fitzthum Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" QEMU cannot write directly to the memory of memory-encrypted guests; this breaks normal RAM-load in the migration target. Instead, QEMU asks a migration helper running on an auxiliary vcpu in the guest to restore encrypted pages as they were received from the source to a specific GPA. The migration helper running inside the guest can safely decrypt the pages arrived from the source and load them into their proper location in the guest's memory. Loading pages uses the same shared (unencrypted) pages which both QEMU and the guest can read from and write to. Signed-off-by: Dov Murik --- migration/confidential-ram.h | 2 ++ migration/confidential-ram.c | 37 ++++++++++++++++++++++++++++++++++++ migration/trace-events | 1 + 3 files changed, 40 insertions(+) diff --git a/migration/confidential-ram.h b/migration/confidential-ram.h index 0d49718d31..ebe4073bce 100644 --- a/migration/confidential-ram.h +++ b/migration/confidential-ram.h @@ -14,4 +14,6 @@ void cgs_mh_cleanup(void); int cgs_mh_save_encrypted_page(QEMUFile *f, ram_addr_t src_gpa, uint32_t size, uint64_t *bytes_sent); +int cgs_mh_load_encrypted_page(QEMUFile *f, ram_addr_t dest_gpa); + #endif diff --git a/migration/confidential-ram.c b/migration/confidential-ram.c index 65a588e7f6..fe317ee74b 100644 --- a/migration/confidential-ram.c +++ b/migration/confidential-ram.c @@ -182,3 +182,40 @@ int cgs_mh_save_encrypted_page(QEMUFile *f, ram_addr_t src_gpa, uint32_t size, return ret; } + +int cgs_mh_load_encrypted_page(QEMUFile *f, ram_addr_t dest_gpa) +{ + int ret = 1; + uint32_t page_hdr_len, enc_page_len; + + init_cgs_mig_helper_if_needed(); + + assert((dest_gpa & TARGET_PAGE_MASK) == dest_gpa); + + /* Read page header */ + page_hdr_len = qemu_get_be32(f); + if (page_hdr_len > 1024) { + error_report("confidential-ram: page header is too large (%d bytes) " + "when loading gpa %" PRIu64, page_hdr_len, dest_gpa); + return -EINVAL; + } + cmhs.io_page_hdr->len = page_hdr_len; + qemu_get_buffer(f, cmhs.io_page_hdr->data, page_hdr_len); + + /* Read encrypted page */ + enc_page_len = qemu_get_be32(f); + if (enc_page_len != TARGET_PAGE_SIZE) { + error_report("confidential-ram: encrypted page is too large (%d bytes) " + "when loading gpa %" PRIu64, enc_page_len, dest_gpa); + return -EINVAL; + } + qemu_get_buffer(f, cmhs.io_page, enc_page_len); + + trace_encrypted_ram_load_page(page_hdr_len, enc_page_len, dest_gpa); + ret = send_command_to_cgs_mig_helper(CGS_MIG_HELPER_CMD_DECRYPT, dest_gpa); + if (ret) { + error_report("confidential-ram: failed loading page at dest_gpa " + "%" PRIu64 ": ret=%d", dest_gpa, ret); + } + return ret; +} diff --git a/migration/trace-events b/migration/trace-events index 929de4ca98..ef31cf78cb 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -345,4 +345,5 @@ migration_pagecache_init(int64_t max_num_items) "Setting cache buckets to %" PRI migration_pagecache_insert(void) "Error allocating page" # confidential-ram.c +encrypted_ram_load_page(uint32_t hdr_len, uint32_t trans_len, uint64_t gpa) "hdr_len: %u, trans_len: %u, gpa: 0x%" PRIx64 encrypted_ram_save_page(uint32_t size, uint64_t gpa) "size: %u, gpa: 0x%" PRIx64 From patchwork Tue Mar 2 20:48:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112225 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F49CC433E0 for ; Tue, 2 Mar 2021 20:59:18 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DC31360234 for ; Tue, 2 Mar 2021 20:59:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DC31360234 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:60312 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHC6u-0005QU-Lr for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 15:59:16 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45760) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwe-0001ST-S2 for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:41 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:20996) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwc-00085u-KF for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:40 -0500 Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122KhB3U154823; Tue, 2 Mar 2021 15:48:34 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=nuSbDcYNLehDlCmcWpK7iKrZMp6Z9YiK3uABzIiWq18=; b=fXYNvf6BcGJwbP7cLZcyGdP0SYM7mVaiJgyDVnpOyhrLYa0BoR96dwUANdqsrm09o4Ed LTvED+1vwbgh4RWLFW2UzCPdmwEujB9cLP6YRUgtvOurf7HUd8C62zB9sdlLpNKZGHsg 5YXIECsDIXMiYqtUXZ8Gbq+o2KEpeKuXaaQy//czUOTvoAwsLhq8HWWjPxGkhKBRD72G qY+cPSNJj4PMdeiEX32egap+5w1EraIPgAxAsq2ZnoN4kr7AF3F+JRzITytKEQJKNVVb JWTgMdPvWulAKpDoECyw3X15/z6wXzJUgx5k/Cgowf+FtyDcauqXla9riQ9WubufZSpw cw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vn4r8vq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:34 -0500 Received: from m0098396.ppops.net (m0098396.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122KhcMO161657; Tue, 2 Mar 2021 15:48:34 -0500 Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vn4r8vb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:34 -0500 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122KmXcE030374; Tue, 2 Mar 2021 20:48:33 GMT Received: from b01cxnp22034.gho.pok.ibm.com (b01cxnp22034.gho.pok.ibm.com [9.57.198.24]) by ppma02dal.us.ibm.com with ESMTP id 3710sqncdh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:33 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmWvw26935602 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:32 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 20C9A28064; Tue, 2 Mar 2021 20:48:32 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id EA7C928058; Tue, 2 Mar 2021 20:48:31 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:31 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 14/26] migration: Introduce gpa_inside_migration_helper_shared_area Date: Tue, 2 Mar 2021 15:48:10 -0500 Message-Id: <20210302204822.81901-15-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 mlxlogscore=999 clxscore=1015 suspectscore=0 impostorscore=0 malwarescore=0 adultscore=0 bulkscore=0 spamscore=0 priorityscore=1501 phishscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.156.1; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Juan Quintela , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , "Dr. David Alan Gilbert" , Hubertus Franke , Tobin Feldman-Fitzthum Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The gpa_inside_migration_helper_shared_area will be used to skip migrating RAM pages that are used by the migration helper at the target. Signed-off-by: Dov Murik --- migration/confidential-ram.h | 2 ++ migration/confidential-ram.c | 6 ++++++ 2 files changed, 8 insertions(+) diff --git a/migration/confidential-ram.h b/migration/confidential-ram.h index ebe4073bce..9a1027bdaf 100644 --- a/migration/confidential-ram.h +++ b/migration/confidential-ram.h @@ -8,6 +8,8 @@ #include "exec/cpu-common.h" #include "qemu-file.h" +bool gpa_inside_migration_helper_shared_area(ram_addr_t gpa); + void cgs_mh_init(void); void cgs_mh_cleanup(void); diff --git a/migration/confidential-ram.c b/migration/confidential-ram.c index fe317ee74b..0b821af774 100644 --- a/migration/confidential-ram.c +++ b/migration/confidential-ram.c @@ -68,6 +68,12 @@ static CGSMigHelperState cmhs = {0}; #define MH_SHARED_CMD_PARAMS_ADDR 0x820000 #define MH_SHARED_IO_PAGE_HDR_ADDR (MH_SHARED_CMD_PARAMS_ADDR + 0x800) #define MH_SHARED_IO_PAGE_ADDR (MH_SHARED_CMD_PARAMS_ADDR + 0x1000) +#define MH_SHARED_LAST_BYTE (MH_SHARED_CMD_PARAMS_ADDR + 0x1fff) + +bool gpa_inside_migration_helper_shared_area(ram_addr_t gpa) +{ + return gpa >= MH_SHARED_CMD_PARAMS_ADDR && gpa <= MH_SHARED_LAST_BYTE; +} void cgs_mh_init(void) { From patchwork Tue Mar 2 20:48:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112269 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F0D2C433DB for ; Tue, 2 Mar 2021 21:11:34 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8DEF064F20 for ; Tue, 2 Mar 2021 21:11:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8DEF064F20 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:32882 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHCIm-0001ge-HV for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 16:11:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45828) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwh-0001bD-T2 for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:44 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:22236) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwc-000860-NQ for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:43 -0500 Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122KhCLd154864; Tue, 2 Mar 2021 15:48:35 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=J2RxgLr7JRwgCZoLLrGN0Mxfw0E2qRB7yyRuPjyEZjc=; b=LxKNC5AeTgsjNtepnt9VzW+we7L5Sivr2h/c4XEmlUQlKCKU6BoCf6AkJzaJOa5OEsMU JkEbNlBIURjKxRVrTgvuojc9dFwMA3nJC8mrrfgxnH/37Xd9VpBko4G+jV9eUBJTweL8 /mwy8z5dl5BMhlwUFDLoLhgR8gM36f/mAhJZ+ZaKv7O+pczuZFuMMRwCVfG3rfSBXZtV WKG/mWl9bh3rVxp0+0kh3KtgYeDf4UdMO0ebx+r9aKTz9N2XTUlfafONLjNJ7XX3zvtE +36+OG7yJ4333+UlPlzNl5jwD3y4tSEZoMNcLYBxN7xDBEy4LdQvuWFJE8879hMmF85W HA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vn4r8w2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:35 -0500 Received: from m0098396.ppops.net (m0098396.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122KhP7H159450; Tue, 2 Mar 2021 15:48:34 -0500 Received: from ppma04dal.us.ibm.com (7a.29.35a9.ip4.static.sl-reverse.com [169.53.41.122]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vn4r8v9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:34 -0500 Received: from pps.filterd (ppma04dal.us.ibm.com [127.0.0.1]) by ppma04dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122KmRMl008437; Tue, 2 Mar 2021 20:48:33 GMT Received: from b01cxnp22034.gho.pok.ibm.com (b01cxnp22034.gho.pok.ibm.com [9.57.198.24]) by ppma04dal.us.ibm.com with ESMTP id 36ydq988kb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:33 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmWsq43909560 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:32 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5D5942805E; Tue, 2 Mar 2021 20:48:32 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 32AA828058; Tue, 2 Mar 2021 20:48:32 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:32 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 15/26] migration: Save confidential guest RAM using migration helper Date: Tue, 2 Mar 2021 15:48:11 -0500 Message-Id: <20210302204822.81901-16-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 mlxlogscore=999 clxscore=1015 suspectscore=0 impostorscore=0 malwarescore=0 adultscore=0 bulkscore=0 spamscore=0 priorityscore=1501 phishscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.156.1; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Juan Quintela , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , "Dr. David Alan Gilbert" , Hubertus Franke , Tobin Feldman-Fitzthum Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" When saving RAM pages of a confidential guest, check whether a page is encrypted. If it is, ask the in-guest migration helper to encrypt the page for transmission. This relies on ability to track the encryption status of each page according to guest's reports, and thus requires the relevant patches in the guest OS and OVMF and the host KVM and QEMU. This is all encapsulated in is_page_encrypted; the implementation can be modified according to the underlying implementation of page encryption status tracking (bitmap / KVM shared regions list / user-side list) Signed-off-by: Dov Murik --- migration/ram.c | 91 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 997f90cc5b..8e55ed49fd 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -53,10 +53,13 @@ #include "block.h" #include "sysemu/sysemu.h" #include "sysemu/cpu-throttle.h" +#include "sysemu/kvm.h" #include "savevm.h" #include "qemu/iov.h" #include "multifd.h" #include "sysemu/runstate.h" +#include "hw/boards.h" +#include "confidential-ram.h" #if defined(__linux__) #include "qemu/userfaultfd.h" @@ -81,6 +84,7 @@ #define RAM_SAVE_FLAG_XBZRLE 0x40 /* 0x80 is reserved in migration.h start with 0x100 next */ #define RAM_SAVE_FLAG_COMPRESS_PAGE 0x100 +#define RAM_SAVE_FLAG_GUEST_ENCRYPTED_PAGE 0x200 static inline bool memcrypt_enabled(void) { @@ -94,6 +98,13 @@ static inline bool is_zero_range(uint8_t *p, uint64_t size) return buffer_is_zero(p, size); } +static inline bool confidential_guest(void) +{ + MachineState *ms = MACHINE(qdev_get_machine()); + + return ms->cgs; +} + XBZRLECacheStats xbzrle_counters; /* struct contains XBZRLE cache and a static page @@ -660,6 +671,23 @@ static void mig_throttle_guest_down(uint64_t bytes_dirty_period, } } +/** + * is_page_encrypted: check if the page is encrypted + * + * Returns a bool indicating whether the page is encrypted. + */ +static bool is_page_encrypted(RAMState *rs, RAMBlock *block, unsigned long page) +{ + /* ROM devices contain unencrypted data */ + if (memory_region_is_romd(block->mr) || + memory_region_is_rom(block->mr) || + !memory_region_is_ram(block->mr)) { + return false; + } + + return test_bit(page, block->encbmap); +} + /** * xbzrle_cache_zero_page: insert a zero page in the XBZRLE cache * @@ -1928,6 +1956,45 @@ static bool save_compress_page(RAMState *rs, RAMBlock *block, ram_addr_t offset) return false; } +/** + * ram_save_encrypted_page - send the given encrypted page to the stream + * + * Return the number of pages written (=1). + */ +static int ram_save_encrypted_page(RAMState *rs, PageSearchStatus *pss, + bool last_stage) +{ + int ret; + uint8_t *p; + RAMBlock *block = pss->block; + ram_addr_t offset = pss->page << TARGET_PAGE_BITS; + ram_addr_t gpa; + uint64_t bytes_sent; + + p = block->host + offset; + + /* Find the GPA of the page */ + if (!kvm_physical_memory_addr_from_host(kvm_state, p, &gpa)) { + error_report("%s failed to get gpa for offset %" PRIu64 " block %s", + __func__, offset, memory_region_name(block->mr)); + return -1; + } + + ram_counters.transferred += + save_page_header(rs, rs->f, block, + offset | RAM_SAVE_FLAG_GUEST_ENCRYPTED_PAGE); + + ret = cgs_mh_save_encrypted_page(rs->f, gpa, TARGET_PAGE_SIZE, &bytes_sent); + if (ret) { + return -1; + } + + ram_counters.transferred += bytes_sent; + ram_counters.normal++; + + return 1; +} + /** * ram_save_target_page: save one target page * @@ -1948,6 +2015,26 @@ static int ram_save_target_page(RAMState *rs, PageSearchStatus *pss, return res; } + /* + * If memory encryption is enabled then skip saving the data pages used by + * the migration handler. + */ + if (confidential_guest() && + gpa_inside_migration_helper_shared_area(offset)) { + return 0; + } + + /* + * If memory encryption is enabled then use memory encryption APIs + * to write the outgoing buffer to the wire. The encryption APIs + * will take care of accessing the guest memory and re-encrypt it + * for the transport purposes. + */ + if (confidential_guest() && + is_page_encrypted(rs, pss->block, pss->page)) { + return ram_save_encrypted_page(rs, pss, last_stage); + } + if (save_compress_page(rs, block, offset)) { return 1; } @@ -2776,6 +2863,10 @@ static int ram_save_setup(QEMUFile *f, void *opaque) return -1; } + if (confidential_guest()) { + cgs_mh_init(); + } + /* migration has already setup the bitmap, reuse it. */ if (!migration_in_colo_state()) { if (ram_init_all(rsp) != 0) { From patchwork Tue Mar 2 20:48:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112209 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C932BC433E0 for ; Tue, 2 Mar 2021 20:53:20 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3E2F760234 for ; Tue, 2 Mar 2021 20:53:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3E2F760234 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:43388 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHC17-0006l3-G2 for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 15:53:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45692) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwc-0001O9-W6 for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:39 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:39296) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwb-00085V-6q for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:38 -0500 Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122KhIto124551; Tue, 2 Mar 2021 15:48:34 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=stM31GpidGLQm+iESbp31GnriOMFeKwCOv8cg0AjbQk=; b=h5JCSHTRlTt6vqc6mIMT9EzGbPE1O/x2ylcHLGq9DfHlrWJUbmJER+Fd/AWr41ROyraU L4Tp8TUxihe1oe7v6lhZr55Amj33zHfiTz7ET62JEmLqbADCKGAn79GVPIEgbOy+4Lx1 F+l2nYhSZEiK9Alfhja6uiAzeaIVbckDfcHkju4EPuXBFVWv+QjnHQYvNqNy/Q6xJICn cq2fwDqpKh3d44d8PkjqiHDidCR2LwQYOdXBGUUnbvuGqv0T6aSG8achbGfS6jrtUBzt j1irMdJ0PMPy7LvwNI39F/hywoq3XNJduGU7Kslo9jfbmtEgWDDWdpxaGwWXDMborTcY RQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vn7r7dk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:34 -0500 Received: from m0098421.ppops.net (m0098421.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122KhTsN125901; Tue, 2 Mar 2021 15:48:34 -0500 Received: from ppma04wdc.us.ibm.com (1a.90.2fa9.ip4.static.sl-reverse.com [169.47.144.26]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vn7r7d6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:34 -0500 Received: from pps.filterd (ppma04wdc.us.ibm.com [127.0.0.1]) by ppma04wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122Km5K2009951; Tue, 2 Mar 2021 20:48:33 GMT Received: from b01cxnp22034.gho.pok.ibm.com (b01cxnp22034.gho.pok.ibm.com [9.57.198.24]) by ppma04wdc.us.ibm.com with ESMTP id 3712phhu8a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:33 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmWja26935610 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:32 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8F4622805A; Tue, 2 Mar 2021 20:48:32 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6E8BF28058; Tue, 2 Mar 2021 20:48:32 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:32 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 16/26] migration: Load confidential guest RAM using migration helper Date: Tue, 2 Mar 2021 15:48:12 -0500 Message-Id: <20210302204822.81901-17-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 mlxscore=0 clxscore=1015 priorityscore=1501 spamscore=0 bulkscore=0 adultscore=0 malwarescore=0 suspectscore=0 mlxlogscore=981 lowpriorityscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.158.5; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0b-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Juan Quintela , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , "Dr. David Alan Gilbert" , Hubertus Franke , Tobin Feldman-Fitzthum Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" When loading encrypted RAM pages of a confidential guest, ask the in-guest migration helper to decrypt the incoming page and place it correctly in the guest memory at the appropriate address. This way the page's plaintext content remains inaccessible to the host. Signed-off-by: Dov Murik --- migration/ram.c | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/migration/ram.c b/migration/ram.c index 8e55ed49fd..82a1d13f5f 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3519,6 +3519,10 @@ void colo_release_ram_cache(void) */ static int ram_load_setup(QEMUFile *f, void *opaque) { + if (confidential_guest()) { + cgs_mh_init(); + } + if (compress_threads_load_setup(f)) { return -1; } @@ -3812,6 +3816,8 @@ void colo_flush_ram_cache(void) static int ram_load_precopy(QEMUFile *f) { int flags = 0, ret = 0, invalid_flags = 0, len = 0, i = 0; + ram_addr_t gpa; + /* ADVISE is earlier, it shows the source has the postcopy capability on */ bool postcopy_advised = postcopy_is_advised(); if (!migrate_use_compression()) { @@ -3848,7 +3854,8 @@ static int ram_load_precopy(QEMUFile *f) } if (flags & (RAM_SAVE_FLAG_ZERO | RAM_SAVE_FLAG_PAGE | - RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) { + RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE | + RAM_SAVE_FLAG_GUEST_ENCRYPTED_PAGE)) { RAMBlock *block = ram_block_from_stream(f, flags); host = host_from_ram_block_offset(block, addr); @@ -3977,6 +3984,16 @@ static int ram_load_precopy(QEMUFile *f) break; } break; + + case RAM_SAVE_FLAG_GUEST_ENCRYPTED_PAGE: + if (!kvm_physical_memory_addr_from_host(kvm_state, host, &gpa)) { + error_report("%s: failed to get gpa for host %p", __func__, host); + ret = -EINVAL; + break; + } + ret = cgs_mh_load_encrypted_page(f, gpa); + break; + case RAM_SAVE_FLAG_EOS: /* normal exit */ multifd_recv_sync_main(); From patchwork Tue Mar 2 20:48:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 993D5C433E0 for ; Tue, 2 Mar 2021 21:04:23 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 18E7A64F23 for ; Tue, 2 Mar 2021 21:04:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 18E7A64F23 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:43638 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHCBq-0002Fa-1l for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 16:04:22 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45988) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBx5-000211-Sm for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:49:07 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:32602 helo=mx0a-001b2d01.pphosted.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBws-00089d-Iq for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:49:05 -0500 Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122Kj1i6020855; Tue, 2 Mar 2021 15:48:38 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=4b7gCV47XUV2KpXJjH0Ddco1GnBVILBnVnOdpnsotJE=; b=ekWIde5EIGIG8ESCnuW0gYAq7VUvx3RMOammkovqnU6+DVDBPRQtRg4Y6tNw2hTm0nXk oM2Ck+GTXPjaCumX5Z5CrB7IG/pXrQPpR2+N9c12HNXBK4cKSr3n4506OvvVucMcgt3m iHP5EJC3+mDeonMJTwaAAPKYs5/zAA67fVqxwm2ZSUftwI29xNzZzvyXmPtJJB+a73Vq cf4mnTPHr1r9w9G7oSM/jMJuVzMpe6rVZ4tyd7+rkRliQHiqlR2H1ZyZGxQoRzV+Q9nv nyW/4bR1HJ2PW+LqHGKWziKGdQZpCwjdIjrY0SZkx0adMAFNPMrC9VeVx+Lk1juk//yc Ig== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 371vnsr4sp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:37 -0500 Received: from m0098420.ppops.net (m0098420.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122Kk2Gf025109; Tue, 2 Mar 2021 15:48:36 -0500 Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0b-001b2d01.pphosted.com with ESMTP id 371vnsr4q8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:36 -0500 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122KkqMA028086; Tue, 2 Mar 2021 20:48:34 GMT Received: from b01cxnp22034.gho.pok.ibm.com (b01cxnp22034.gho.pok.ibm.com [9.57.198.24]) by ppma01dal.us.ibm.com with ESMTP id 371qmuagug-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:34 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmW9p40698236 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:33 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D88F528064; Tue, 2 Mar 2021 20:48:32 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B0C0928058; Tue, 2 Mar 2021 20:48:32 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:32 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 17/26] migration: Stop VM after loading confidential RAM Date: Tue, 2 Mar 2021 15:48:13 -0500 Message-Id: <20210302204822.81901-18-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 mlxlogscore=999 clxscore=1015 mlxscore=0 bulkscore=0 impostorscore=0 lowpriorityscore=0 spamscore=0 suspectscore=0 adultscore=0 priorityscore=1501 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.158.5; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Eduardo Habkost , Juan Quintela , Richard Henderson , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , "Dr. David Alan Gilbert" , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Register a dummy device state (EndOfConfidentialRAMState) with high priority so it is the first device which is loaded in the target. The post_load handler of this device stops the VM, which makes things easier when loading devices' states which expect the VM not to be running at the same time. Signed-off-by: Dov Murik --- migration/confidential-ram.h | 2 ++ migration/confidential-ram.c | 31 +++++++++++++++++++++++++++++++ softmmu/runstate.c | 1 + target/i386/sev.c | 2 ++ 4 files changed, 36 insertions(+) diff --git a/migration/confidential-ram.h b/migration/confidential-ram.h index 9a1027bdaf..2822c5ee3d 100644 --- a/migration/confidential-ram.h +++ b/migration/confidential-ram.h @@ -18,4 +18,6 @@ int cgs_mh_save_encrypted_page(QEMUFile *f, ram_addr_t src_gpa, uint32_t size, int cgs_mh_load_encrypted_page(QEMUFile *f, ram_addr_t dest_gpa); +void register_end_of_confidential_ram(void); + #endif diff --git a/migration/confidential-ram.c b/migration/confidential-ram.c index 0b821af774..982cf9b874 100644 --- a/migration/confidential-ram.c +++ b/migration/confidential-ram.c @@ -8,6 +8,8 @@ #include "io/channel.h" #include "qapi/error.h" #include "exec/memory.h" +#include "migration/vmstate.h" +#include "sysemu/runstate.h" #include "trace.h" #include "confidential-ram.h" @@ -225,3 +227,32 @@ int cgs_mh_load_encrypted_page(QEMUFile *f, ram_addr_t dest_gpa) } return ret; } + +typedef struct { + bool dummy; +} EndOfConfidentialRAMState; + +static EndOfConfidentialRAMState end_of_confidential_ram_state = { .dummy = false }; + +static int end_of_confidential_ram_post_load(void *opaque, int version_id) +{ + vm_stop(RUN_STATE_INMIGRATE); + return 0; +} + +static const VMStateDescription vmstate_end_of_confidential_ram = { + .name = "end-of-confidential-ram", + .priority = MIG_PRI_GICV3, /* TODO define new (higher) priority level */ + .version_id = 1, + .post_load = end_of_confidential_ram_post_load, + .fields = (VMStateField[]) { + VMSTATE_BOOL(dummy, EndOfConfidentialRAMState), + VMSTATE_END_OF_LIST() + }, +}; + +void register_end_of_confidential_ram(void) +{ + vmstate_register(NULL, 0, &vmstate_end_of_confidential_ram, + &end_of_confidential_ram_state); +} diff --git a/softmmu/runstate.c b/softmmu/runstate.c index 2874417b61..193413246d 100644 --- a/softmmu/runstate.c +++ b/softmmu/runstate.c @@ -131,6 +131,7 @@ static const RunStateTransition runstate_transitions_def[] = { { RUN_STATE_RUNNING, RUN_STATE_INTERNAL_ERROR }, { RUN_STATE_RUNNING, RUN_STATE_IO_ERROR }, { RUN_STATE_RUNNING, RUN_STATE_PAUSED }, + { RUN_STATE_RUNNING, RUN_STATE_INMIGRATE }, { RUN_STATE_RUNNING, RUN_STATE_FINISH_MIGRATE }, { RUN_STATE_RUNNING, RUN_STATE_RESTORE_VM }, { RUN_STATE_RUNNING, RUN_STATE_SAVE_VM }, diff --git a/target/i386/sev.c b/target/i386/sev.c index 0f414df02f..da2d0cc699 100644 --- a/target/i386/sev.c +++ b/target/i386/sev.c @@ -33,6 +33,7 @@ #include "exec/address-spaces.h" #include "monitor/monitor.h" #include "exec/confidential-guest-support.h" +#include "migration/confidential-ram.h" #include "hw/i386/pc.h" #define TYPE_SEV_GUEST "sev-guest" @@ -1011,6 +1012,7 @@ static void sev_register_types(void) { type_register_static(&sev_guest_info); + register_end_of_confidential_ram(); } type_init(sev_register_types); From patchwork Tue Mar 2 20:48:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112239 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AF55C433DB for ; Tue, 2 Mar 2021 21:04:27 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AF91B64F21 for ; Tue, 2 Mar 2021 21:04:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AF91B64F21 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:43982 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHCBt-0002OU-Np for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 16:04:25 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45792) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwf-0001UD-JK for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:41 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:53768) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwd-00086M-1V for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:41 -0500 Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122KhIIu124566; Tue, 2 Mar 2021 15:48:35 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=565rVDpXIrueRMe9M5o87ZC7PkPqDvOeL6fY0lx2glQ=; b=ViyO6owMuHUY4A5e8fcwqWKMo2y5270Z1L4uxh0mEc2odKZZaYt72NzW6UpONTWEuJm3 7zSloA06TQKBc/OB0DegD61Z//GwRJPvrODzmQggtO/FNtGPPVl+iFTd5Gh6KPF4wLwf uTmx6lyjev1LQUJv5KuUTiWmjSxDgNb/GuLUNbej8pSxcrvTppOwe7D5HytRLW5VVFPJ WsmfdPHGIgSzYnhZZNCd1OOoGgwq3IcXO7yzDFA+OXMBuUAhwnDwJbgT636MfhyBxFv/ tr9CrMeKK5bsthRNiHB+CwfF8BMkO3RNCDs0HJGo2kzxoBqxvajV4gUvFpylY+1MF+6E ig== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vn7r7e6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:35 -0500 Received: from m0098421.ppops.net (m0098421.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122KhI0H124650; Tue, 2 Mar 2021 15:48:34 -0500 Received: from ppma02wdc.us.ibm.com (aa.5b.37a9.ip4.static.sl-reverse.com [169.55.91.170]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vn7r7dd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:34 -0500 Received: from pps.filterd (ppma02wdc.us.ibm.com [127.0.0.1]) by ppma02wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122KX0ma018064; Tue, 2 Mar 2021 20:48:34 GMT Received: from b01cxnp22035.gho.pok.ibm.com (b01cxnp22035.gho.pok.ibm.com [9.57.198.25]) by ppma02wdc.us.ibm.com with ESMTP id 3711dwtf74-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:34 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22035.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmX0X24838436 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:33 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 29D2B28059; Tue, 2 Mar 2021 20:48:33 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id F2E4928058; Tue, 2 Mar 2021 20:48:32 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:32 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 18/26] migration: Stop non-aux vcpus before copying the last pages Date: Tue, 2 Mar 2021 15:48:14 -0500 Message-Id: <20210302204822.81901-19-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 mlxscore=0 clxscore=1015 priorityscore=1501 spamscore=0 bulkscore=0 adultscore=0 malwarescore=0 suspectscore=0 mlxlogscore=999 lowpriorityscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.158.5; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0b-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Juan Quintela , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , "Dr. David Alan Gilbert" , Hubertus Franke , Tobin Feldman-Fitzthum Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Tobin Feldman-Fitzthum Co-Author: Dov Murik Signed-off-by: Dov Murik Signed-off-by: Tobin Feldman-Fitzthum --- migration/migration.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index a5ddf43559..7ec25bd006 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -24,6 +24,7 @@ #include "sysemu/runstate.h" #include "sysemu/sysemu.h" #include "sysemu/cpu-throttle.h" +#include "sysemu/cpus.h" #include "rdma.h" #include "ram.h" #include "migration/global_state.h" @@ -3156,14 +3157,14 @@ static void migration_completion(MigrationState *s) qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER, NULL); s->vm_was_running = runstate_is_running(); ret = global_state_store(); + pause_all_vcpus_except_aux(); + qemu_mutex_unlock_iothread(); if (!ret) { bool inactivate = !migrate_colo_enabled(); - ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE); - if (ret >= 0) { - ret = migration_maybe_pause(s, ¤t_active_state, - MIGRATION_STATUS_DEVICE); - } + ret = migration_maybe_pause(s, ¤t_active_state, + MIGRATION_STATUS_DEVICE); + if (ret >= 0) { qemu_file_set_rate_limit(s->to_dst_file, INT64_MAX); ret = qemu_savevm_state_complete_precopy(s->to_dst_file, false, @@ -3173,7 +3174,7 @@ static void migration_completion(MigrationState *s) s->block_inactive = true; } } - qemu_mutex_unlock_iothread(); + runstate_set(RUN_STATE_FINISH_MIGRATE); if (ret < 0) { goto fail; From patchwork Tue Mar 2 20:48:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112221 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A91DC433E0 for ; Tue, 2 Mar 2021 20:56:37 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C047F64F23 for ; Tue, 2 Mar 2021 20:56:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C047F64F23 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:52116 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHC4J-0001xX-Jr for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 15:56:35 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45752) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwe-0001Rw-Lw for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:40 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:36354) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwc-00085t-Ju for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:40 -0500 Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122KhCC6154834; Tue, 2 Mar 2021 15:48:36 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=cETO28MIDTsxPUirV5M6A1eYKBZc0wVgX9s6/puG400=; b=NRfOHOG73A7ZHqf8QjsYseSDm18yNvocbkbkQO6Li08R7i5F3quBUU77Za+14Pj2bKSK 0qzr8owI+0PVCPIWpDDjcCo7FGU3MZxsu/RxN839m70thGM0WPyAhdCDwsBpz4ixLNyD P1TEk4psdc/MjdeWg0qXzmqjKvbkeavxoV5gi1MkH1AXktaGhtSisStdDsO0xQEN6Mus lAWwG+sWeaOqBq+XfnqsJsPRBm4wUbp5R1w+dv1iUHFrDYV9JhI8K5S3huTgBr6YTG8s xBOK2k51T/lX8Wg+WkvqHcWzJaLzerlqSK2DP/t1E9npLpprM5KmxZfPI+3jjmKyorp9 1g== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vn4r8wa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:35 -0500 Received: from m0098396.ppops.net (m0098396.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122KhTgs160186; Tue, 2 Mar 2021 15:48:35 -0500 Received: from ppma05wdc.us.ibm.com (1b.90.2fa9.ip4.static.sl-reverse.com [169.47.144.27]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vn4r8vp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:35 -0500 Received: from pps.filterd (ppma05wdc.us.ibm.com [127.0.0.1]) by ppma05wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122KmKTc016512; Tue, 2 Mar 2021 20:48:34 GMT Received: from b01cxnp22035.gho.pok.ibm.com (b01cxnp22035.gho.pok.ibm.com [9.57.198.25]) by ppma05wdc.us.ibm.com with ESMTP id 371b00xpwv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:34 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22035.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmXg225035224 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:33 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6409D28059; Tue, 2 Mar 2021 20:48:33 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 39D0828058; Tue, 2 Mar 2021 20:48:33 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:33 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 19/26] migration: Don't sync vcpus when migrating confidential guests Date: Tue, 2 Mar 2021 15:48:15 -0500 Message-Id: <20210302204822.81901-20-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 mlxlogscore=984 clxscore=1015 suspectscore=0 impostorscore=0 malwarescore=0 adultscore=0 bulkscore=0 spamscore=0 priorityscore=1501 phishscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.156.1; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Juan Quintela , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , "Dr. David Alan Gilbert" , Hubertus Franke , Tobin Feldman-Fitzthum Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" When loading incoming VM state to the migration target, don't sync the vcpus because it'll prevent the migration handler to keep running. Signed-off-by: Dov Murik --- migration/savevm.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/migration/savevm.c b/migration/savevm.c index 52e2d72e4b..c5252612c3 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -2673,6 +2673,13 @@ out: return ret; } +static inline bool confidential_guest(void) +{ + MachineState *ms = MACHINE(qdev_get_machine()); + + return ms->cgs; +} + int qemu_loadvm_state(QEMUFile *f) { MigrationIncomingState *mis = migration_incoming_get_current(); @@ -2693,7 +2700,9 @@ int qemu_loadvm_state(QEMUFile *f) return -EINVAL; } - cpu_synchronize_all_pre_loadvm(); + if (!confidential_guest()) { + cpu_synchronize_all_pre_loadvm(); + } ret = qemu_loadvm_state_main(f, mis); qemu_event_set(&mis->main_thread_load_event); From patchwork Tue Mar 2 20:48:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112271 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 794C3C433E0 for ; Tue, 2 Mar 2021 21:14:38 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F3CF464F21 for ; Tue, 2 Mar 2021 21:14:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F3CF464F21 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:41740 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHCLl-0005dF-0T for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 16:14:37 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45970) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBx3-00020Y-U8 for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:49:07 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:50282) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwo-00086p-9d for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:49:04 -0500 Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122KirAB125853; Tue, 2 Mar 2021 15:48:37 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=Fkv7qZFnMwolkrKf1/i5ogOSZd8zqMCoqz78cB/X9s8=; b=ryV1QKf2vYy2Y+KVgXT931Wc93pcxeHRcO0upSn0RbRI9XeqkAJSkWSM/gMGS6b2rqR5 9l3TtEfNe7zb1fXacdNRNPz7gI8CZHL650vuTqMtcK9ja9CRbLM/UUrweuo2b4DkvAfn /hHXfhs/JReSgtqtmUlMKiaK/Tt8ziMaKD3jRAy0pt6ZZfkTZ0zxQ2HnzhQ6qPIvQWSd jnJ/aS/jnmx0MpC+1wFHs4rvhJttHNvcO+zeDNCYrFHif6bB8zMk7/7/oy8XyF6T8J9w Lm0wLCV6MZlwfPIZjHmPYlvN0ecQSeu+udWQQnYk/TMVeuUErquNCs0Zj/SyIfL8cv19 ww== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vnt0590-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:37 -0500 Received: from m0098404.ppops.net (m0098404.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122Kjkt9132749; Tue, 2 Mar 2021 15:48:36 -0500 Received: from ppma03wdc.us.ibm.com (ba.79.3fa9.ip4.static.sl-reverse.com [169.63.121.186]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vnt0572-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:36 -0500 Received: from pps.filterd (ppma03wdc.us.ibm.com [127.0.0.1]) by ppma03wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122KmXVN025958; Tue, 2 Mar 2021 20:48:34 GMT Received: from b01cxnp22035.gho.pok.ibm.com (b01cxnp22035.gho.pok.ibm.com [9.57.198.25]) by ppma03wdc.us.ibm.com with ESMTP id 37128ga1w5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:34 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22035.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmX1o19988972 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:33 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B38E02805C; Tue, 2 Mar 2021 20:48:33 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7F8C628060; Tue, 2 Mar 2021 20:48:33 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:33 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 20/26] migration: When starting target, don't sync auxiliary vcpus Date: Tue, 2 Mar 2021 15:48:16 -0500 Message-Id: <20210302204822.81901-21-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 mlxscore=0 suspectscore=0 mlxlogscore=868 priorityscore=1501 lowpriorityscore=0 clxscore=1015 adultscore=0 spamscore=0 phishscore=0 impostorscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.156.1; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Juan Quintela , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , "Dr. David Alan Gilbert" , Hubertus Franke , Tobin Feldman-Fitzthum Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" If auxiliary vcpus are defined, they are used for running the migration helper inside the guest. We want to keep them running and not sync their state. This behaves exactly like cpu_synchronize_all_post_init() when there are no auxiliary vcpus. Signed-off-by: Dov Murik --- migration/savevm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/migration/savevm.c b/migration/savevm.c index c5252612c3..c6af1f7bba 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -2754,7 +2754,7 @@ int qemu_loadvm_state(QEMUFile *f) } qemu_loadvm_state_cleanup(); - cpu_synchronize_all_post_init(); + cpu_synchronize_without_aux_post_init(); return ret; } From patchwork Tue Mar 2 20:48:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112241 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6F38C433DB for ; Tue, 2 Mar 2021 21:06:23 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 43B7A64F23 for ; Tue, 2 Mar 2021 21:06:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 43B7A64F23 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:48918 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHCDk-0004XN-Hb for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 16:06:20 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:46262) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBxm-000345-5g for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:49:50 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:50616) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBxj-0008F3-GL for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:49:49 -0500 Received: from pps.filterd (m0187473.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122KhgkS168494; Tue, 2 Mar 2021 15:49:40 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=OaogKjzFLHSuOOIg+JgiSxMTnobul8jWMSZ5b0j5Hno=; b=c5txgLbU35IXdba7WRleSbLKL1ZU4x+2mUF+U54ZYoveEnf9pkFJBgMyGGi6QpRnzfrn d6zKVVwdGR87huQMKtLjN1Gnr+0e7R7+C4Dgle5RF3arRmwrun/IBrQmo49NauHmTxX0 jjYUGsKJIbIv6u23nZtazsNppmExe3v/pApNUBGTIYkJW+bkKQDRG26Hn9/qaCVmP8UM onL6TSk/6HU6d6ZVR2OQdodC4nZnlBWNJQjqkouqe0t8+nvUTdOO+JhGl0xs6+QqHD4O knee8BMgSURQbBmOnibc/COledD2BKVI1H/Wwdm3a/F9ozdy1YJILkK847p0uZUA2uLX /w== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vnd06pb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:49:38 -0500 Received: from m0187473.ppops.net (m0187473.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122Khna5168816; Tue, 2 Mar 2021 15:49:38 -0500 Received: from ppma02wdc.us.ibm.com (aa.5b.37a9.ip4.static.sl-reverse.com [169.55.91.170]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vnd06jg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:49:35 -0500 Received: from pps.filterd (ppma02wdc.us.ibm.com [127.0.0.1]) by ppma02wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122KX2h1018076; Tue, 2 Mar 2021 20:48:34 GMT Received: from b01cxnp22036.gho.pok.ibm.com (b01cxnp22036.gho.pok.ibm.com [9.57.198.26]) by ppma02wdc.us.ibm.com with ESMTP id 3711dwtf76-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:34 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22036.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmY6U6488698 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:34 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 03F6528058; Tue, 2 Mar 2021 20:48:34 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C436E28064; Tue, 2 Mar 2021 20:48:33 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:33 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 21/26] migration: Call migration handler cleanup routines Date: Tue, 2 Mar 2021 15:48:17 -0500 Message-Id: <20210302204822.81901-22-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 impostorscore=0 spamscore=0 phishscore=0 lowpriorityscore=0 bulkscore=0 mlxscore=0 malwarescore=0 suspectscore=0 priorityscore=1501 mlxlogscore=973 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.156.1; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Juan Quintela , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , "Dr. David Alan Gilbert" , Hubertus Franke , Tobin Feldman-Fitzthum Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Tobin Feldman-Fitzthum Signed-off-by: Tobin Feldman-Fitzthum Signed-off-by: Dov Murik --- migration/ram.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 82a1d13f5f..ce551c1d2f 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3054,6 +3054,10 @@ static int ram_save_complete(QEMUFile *f, void *opaque) ram_control_after_iterate(f, RAM_CONTROL_FINISH); } + if (confidential_guest()) { + cgs_mh_cleanup(); + } + if (ret >= 0) { multifd_send_sync_main(rs->f); qemu_put_be64(f, RAM_SAVE_FLAG_EOS); @@ -3549,6 +3553,10 @@ static int ram_load_cleanup(void *opaque) rb->receivedmap = NULL; } + if (confidential_guest()) { + cgs_mh_cleanup(); + } + return 0; } From patchwork Tue Mar 2 20:48:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112233 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8985C433DB for ; Tue, 2 Mar 2021 21:03:18 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3D7D964F1E for ; Tue, 2 Mar 2021 21:03:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3D7D964F1E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:40564 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHCAn-0000uI-51 for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 16:03:17 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:46218) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBxZ-0002vz-UP for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:49:37 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:22206) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBxY-0008EM-6G for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:49:37 -0500 Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122KiFMG064838; Tue, 2 Mar 2021 15:49:33 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=4N0Q4l7A0ce24AID9lZvV4+auzwXikjZ0o/Mh06aOIM=; b=ULYrT/tVHyPbgkxxRyLhIY68E8y9hdniI9TpkQ5RlZ6ksm/rlXX1nJzYw8bEXHKx1lW/ KhF8kqPTP6pLb1ipgvyrED79tGiUz7Eoin00vd34BiCxdIioKO8TJaBrIQEMYGde+dKn /YszpthyMBpPX4nBSEI3++0nFAforQn5QgeeyzlLxU9SxZIzDme1RoV86xChAYdE9Bsu s3dRN0Ar7a2JPxHAgpl/nAXGWGSCRMfgoKgpHSj/c7CifNUa1PO1NSE5+g0uLiDf+97P /Dtv9lc/33QTJFu2d6Q22sFUi+HVnIFTloFYXmP0nXaTyHwJM9ee8leyNs4+soMxwnp4 8w== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vnf87ab-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:49:32 -0500 Received: from m0098394.ppops.net (m0098394.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122Kig7o072018; Tue, 2 Mar 2021 15:49:32 -0500 Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vnf879d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:49:32 -0500 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122KmXcG030374; Tue, 2 Mar 2021 20:48:35 GMT Received: from b01cxnp22036.gho.pok.ibm.com (b01cxnp22036.gho.pok.ibm.com [9.57.198.26]) by ppma02dal.us.ibm.com with ESMTP id 3710sqncdt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:35 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22036.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmYAL7799416 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:34 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 490DA2805E; Tue, 2 Mar 2021 20:48:34 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1502528059; Tue, 2 Mar 2021 20:48:34 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:34 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 22/26] hw/isa/lpc_ich9: Allow updating an already-running VM Date: Tue, 2 Mar 2021 15:48:18 -0500 Message-Id: <20210302204822.81901-23-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 priorityscore=1501 bulkscore=0 adultscore=0 mlxlogscore=999 clxscore=1015 spamscore=0 impostorscore=0 mlxscore=0 lowpriorityscore=0 phishscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.156.1; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , "Michael S. Tsirkin" , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The post_load function crashed when we were loading the device state in to an already-running guest. This was because an existing memory region as not deleted in ich9_lpc_rcba_update. Signed-off-by: Dov Murik --- hw/isa/lpc_ich9.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/hw/isa/lpc_ich9.c b/hw/isa/lpc_ich9.c index d3145bf014..1fc1af3491 100644 --- a/hw/isa/lpc_ich9.c +++ b/hw/isa/lpc_ich9.c @@ -529,9 +529,10 @@ ich9_lpc_pmcon_update(ICH9LPCState *lpc) static int ich9_lpc_post_load(void *opaque, int version_id) { ICH9LPCState *lpc = opaque; + uint32_t rcba_old = pci_get_long(lpc->d.config + ICH9_LPC_RCBA); ich9_lpc_pmbase_sci_update(lpc); - ich9_lpc_rcba_update(lpc, 0 /* disabled ICH9_LPC_RCBA_EN */); + ich9_lpc_rcba_update(lpc, rcba_old); ich9_lpc_pmcon_update(lpc); return 0; } From patchwork Tue Mar 2 20:48:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112227 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CAFD0C433E6 for ; Tue, 2 Mar 2021 20:59:19 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3281460234 for ; Tue, 2 Mar 2021 20:59:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3281460234 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:60444 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHC6v-0005To-TT for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 15:59:17 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45798) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwg-0001VZ-1Z for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:42 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:38132 helo=mx0a-001b2d01.pphosted.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwd-00086a-Ir for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:41 -0500 Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122Khn6r054943; Tue, 2 Mar 2021 15:48:37 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=pU1YpA++e+Hrf7KQ5FtJYr10Pdn5ncGR8lkqHgrc2is=; b=Hn7M+aJjlFh0M1qMcDdm6Fh9KR/vvVfXU7HPzid9amrhjkprliQsxd6o/cYzSW57TN12 OeSbsLGS7X3y4MFIUppZoUMJr8B5vr7oc2BYGDGngpy0lO1W5I4N5/4SGtbdGYEdfdEX TNgVO+S3JAVDQrNcsV6LB1IsNS+dD/hQLyGUvbgffTx8N0S1wByyGE7yDD0v7loPs37+ +DjyugYkdTSEPfSa/EdMy9LuOM+WNb8T0jrmMNA4ZXZb+w8OQo8NXssOz+Beba0IISXK w3VpM6p5AELfE6cabB49vYDYnwLcSrTOQbzQhGG9sVYN1JPL1pxAXnaGifEZ2ZucvMHo ng== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 371vn9g6r7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:37 -0500 Received: from m0098419.ppops.net (m0098419.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122KiKp4064759; Tue, 2 Mar 2021 15:48:36 -0500 Received: from ppma04wdc.us.ibm.com (1a.90.2fa9.ip4.static.sl-reverse.com [169.47.144.26]) by mx0b-001b2d01.pphosted.com with ESMTP id 371vn9g6qe-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:36 -0500 Received: from pps.filterd (ppma04wdc.us.ibm.com [127.0.0.1]) by ppma04wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122Km5WF009944; Tue, 2 Mar 2021 20:48:35 GMT Received: from b01cxnp22036.gho.pok.ibm.com (b01cxnp22036.gho.pok.ibm.com [9.57.198.26]) by ppma04wdc.us.ibm.com with ESMTP id 3712phhu8j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:35 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22036.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmYBm10617264 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:34 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 867FA2805C; Tue, 2 Mar 2021 20:48:34 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5984A28059; Tue, 2 Mar 2021 20:48:34 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:34 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 23/26] target/i386: Re-sync kvm-clock after confidential guest migration Date: Tue, 2 Mar 2021 15:48:19 -0500 Message-Id: <20210302204822.81901-24-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 impostorscore=0 suspectscore=0 priorityscore=1501 malwarescore=0 mlxscore=0 lowpriorityscore=0 clxscore=1015 spamscore=0 adultscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.158.5; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Eduardo Habkost , Richard Henderson , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" If confidential guest support is active, set TSC to 0 on the target when loading the CPU state. This causes the guest OS to re-sync with kvm-clock. Without this change, the guest clocks after migration are stuck (don't advance), except the *_COARSE clocks which advance normally. Signed-off-by: Dov Murik --- target/i386/machine.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/target/i386/machine.c b/target/i386/machine.c index 3768a753af..36c52ec02e 100644 --- a/target/i386/machine.c +++ b/target/i386/machine.c @@ -297,8 +297,17 @@ static int cpu_post_load(void *opaque, int version_id) X86CPU *cpu = opaque; CPUState *cs = CPU(cpu); CPUX86State *env = &cpu->env; + MachineState *ms = MACHINE(qdev_get_machine()); int i; + /* + * When loading the state of a confidential guest, set TSC to zero at allow + * the guest OS to re-sync with kvmclock. + */ + if (ms->cgs) { + env->tsc = 0; + } + if (env->tsc_khz && env->user_tsc_khz && env->tsc_khz != env->user_tsc_khz) { error_report("Mismatch between user-specified TSC frequency and " From patchwork Tue Mar 2 20:48:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112229 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 038E5C433DB for ; Tue, 2 Mar 2021 20:59:20 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5BB7864F21 for ; Tue, 2 Mar 2021 20:59:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5BB7864F21 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:60510 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHC6w-0005VY-99 for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 15:59:18 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45826) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwh-0001Zs-Do for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:43 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:63666) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwe-00087C-R1 for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:43 -0500 Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122KhItv124551; Tue, 2 Mar 2021 15:48:38 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=3PvMeeTWa3MmHAR7tWHus/9tHidDa0gb9NCFGmTzJO0=; b=ME8UtmIJMQd7yBB3+kA7csh7F5aXh98pCr6NEbDLGCCPfyfiu7cYK4ly1cN2ZD/o6MMw geGrOgB+m+vM6WUvP3bhAyt4vEkSbzBsMdcrdgDR1cQzWeuk9HJfaiWhvIFDr29Vjtsk u0XOn8hlTkaVPtPakuumjWvOoUWqcSuUfNztX+1PtO+k/nwpRgRCnNK5e2veZxF4SPrZ 0rh0tb8Az/ylCnT2COKP6DktkZDjXA1+o/omoZWdZaTvHvgUuTLPLXle1KJ6mOwGzfk6 KE62Yggp4F3JnC4G0+ApF0ApiDD7rCg32gJWksBTwUI7ANTGfA+o3J4b2Z1+cofECQy4 9A== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vn7r7gk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:37 -0500 Received: from m0098421.ppops.net (m0098421.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122KhUen126025; Tue, 2 Mar 2021 15:48:36 -0500 Received: from ppma02wdc.us.ibm.com (aa.5b.37a9.ip4.static.sl-reverse.com [169.55.91.170]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vn7r7f1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:36 -0500 Received: from pps.filterd (ppma02wdc.us.ibm.com [127.0.0.1]) by ppma02wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122KX0Q8018061; Tue, 2 Mar 2021 20:48:35 GMT Received: from b01cxnp22036.gho.pok.ibm.com (b01cxnp22036.gho.pok.ibm.com [9.57.198.26]) by ppma02wdc.us.ibm.com with ESMTP id 3711dwtf7h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:35 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22036.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmYOT5833426 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:34 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D33EA2805C; Tue, 2 Mar 2021 20:48:34 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A3DF928058; Tue, 2 Mar 2021 20:48:34 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:34 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 24/26] migration: Add start-migrate-incoming QMP command Date: Tue, 2 Mar 2021 15:48:20 -0500 Message-Id: <20210302204822.81901-25-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 mlxscore=0 clxscore=1015 priorityscore=1501 spamscore=0 bulkscore=0 adultscore=0 malwarescore=0 suspectscore=0 mlxlogscore=999 lowpriorityscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.158.5; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0b-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Juan Quintela , Markus Armbruster , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , "Dr. David Alan Gilbert" , Hubertus Franke , Tobin Feldman-Fitzthum Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This command forces a running VM into a migrate-incoming state. When using guest-assisted migration (for confidential guests), the target must be started so that its auxiliary vcpu is running the migration helper; after it is ready we can start receiving the incoming migration connection. Signed-off-by: Dov Murik --- qapi/migration.json | 26 ++++++++++++++++++++++++++ migration/migration.c | 5 +++++ 2 files changed, 31 insertions(+) diff --git a/qapi/migration.json b/qapi/migration.json index 6e5943fbb4..c7361e0038 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -1569,6 +1569,32 @@ ## { 'command': 'migrate-incoming', 'data': {'uri': 'str' } } +## +# @start-migrate-incoming: +# +# Force start an incoming migration even in a running VM. This is used by the +# target VM in guest-assisted migration of a confidential guest. +# +# @uri: The Uniform Resource Identifier identifying the source or +# address to listen on +# +# Returns: nothing on success +# +# Since: 6.0 +# +# Notes: +# +# The uri format is the same as the -incoming command-line option. +# +# Example: +# +# -> { "execute": "start-migrate-incoming", +# "arguments": { "uri": "tcp::4446" } } +# <- { "return": {} } +# +## +{ 'command': 'start-migrate-incoming', 'data': {'uri': 'str' } } + ## # @xen-save-devices-state: # diff --git a/migration/migration.c b/migration/migration.c index 7ec25bd006..4729b89bef 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -2098,6 +2098,11 @@ void qmp_migrate_incoming(const char *uri, Error **errp) once = false; } +void qmp_start_migrate_incoming(const char *uri, Error **errp) +{ + qemu_start_incoming_migration(uri, errp); +} + void qmp_migrate_recover(const char *uri, Error **errp) { MigrationIncomingState *mis = migration_incoming_get_current(); From patchwork Tue Mar 2 20:48:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112219 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41444C433DB for ; Tue, 2 Mar 2021 20:56:37 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 956C364F21 for ; Tue, 2 Mar 2021 20:56:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 956C364F21 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:52098 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHC4J-0001x3-GE for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 15:56:35 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45818) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwh-0001Ye-0V for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:43 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:46076 helo=mx0a-001b2d01.pphosted.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwe-000874-MJ for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:42 -0500 Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122Kiv0r019672; Tue, 2 Mar 2021 15:48:37 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=14BKeqquuNbtx2IcmuWrKO9c62dlnNtxt5S70nonB2Q=; b=p9rDjJ4LIeHlC5gaiGZY12lY4oIn+bsMtUQi3bVbkbEoKcfJ1xzZl41fUrhSMfB+BUlX dJqoEc/otTqGppCWwT/v95qRah4QpOTfTZfJNfVMAcMaKmom9GI9771CtT2K2reSKiIL YwO+v8qL9cSJ6IpRJ8uF3oRJTzjuLHf5HTiuJxCEtlg8mAtdAsVTnr7p4RtC1mgxWZca /uwDsOPM1gM86BSiTauEV5BtjO6FrVrpi5rlOXLQgofx5kwXG3XmLsYdLClTrmQT0uY8 DETyDIhTsJmim7IkBCsNq31LDmyODtzibqTg++Vscs3xePOq4zi/wXuJWI0hf3CnLzh0 oA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 371vnsr4sx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:37 -0500 Received: from m0098420.ppops.net (m0098420.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 122Kir2j019571; Tue, 2 Mar 2021 15:48:37 -0500 Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0b-001b2d01.pphosted.com with ESMTP id 371vnsr4s8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:37 -0500 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122Km7iv013967; Tue, 2 Mar 2021 20:48:36 GMT Received: from b01cxnp23032.gho.pok.ibm.com (b01cxnp23032.gho.pok.ibm.com [9.57.198.27]) by ppma03dal.us.ibm.com with ESMTP id 37103w5sgf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:36 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmZGX26280300 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:35 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1C5D628058; Tue, 2 Mar 2021 20:48:35 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E3FC528060; Tue, 2 Mar 2021 20:48:34 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:34 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 25/26] target/i386: SEV: Allow migration unless there are no aux vcpus Date: Tue, 2 Mar 2021 15:48:21 -0500 Message-Id: <20210302204822.81901-26-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 mlxlogscore=997 clxscore=1015 mlxscore=0 bulkscore=0 impostorscore=0 lowpriorityscore=0 spamscore=0 suspectscore=0 adultscore=0 priorityscore=1501 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.158.5; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , Eduardo Habkost , Richard Henderson , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Memory-encrypted guests require a migration helper running on an auxiliary vcpu inside the guest in order to migrate RAM to the target. When there are no auxiliary vcpus, block migration attempts. Signed-off-by: Dov Murik --- target/i386/sev.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/target/i386/sev.c b/target/i386/sev.c index da2d0cc699..f22f9b29ea 100644 --- a/target/i386/sev.c +++ b/target/i386/sev.c @@ -32,6 +32,7 @@ #include "qom/object.h" #include "exec/address-spaces.h" #include "monitor/monitor.h" +#include "hw/boards.h" #include "exec/confidential-guest-support.h" #include "migration/confidential-ram.h" #include "hw/i386/pc.h" @@ -669,6 +670,7 @@ sev_launch_finish(SevGuestState *sev) { int ret, error; Error *local_err = NULL; + MachineState *ms = MACHINE(qdev_get_machine()); trace_kvm_sev_launch_finish(); ret = sev_ioctl(sev->sev_fd, KVM_SEV_LAUNCH_FINISH, 0, &error); @@ -680,14 +682,19 @@ sev_launch_finish(SevGuestState *sev) sev_set_guest_state(sev, SEV_STATE_RUNNING); - /* add migration blocker */ - error_setg(&sev_mig_blocker, - "SEV: Migration is not implemented"); - ret = migrate_add_blocker(sev_mig_blocker, &local_err); - if (local_err) { - error_report_err(local_err); - error_free(sev_mig_blocker); - exit(1); + /* + * SEV migration is not supported unless there's an auxiliary CPU running + * the guest-assisted migration helper. + */ + if (ms->smp.aux_cpus == 0) { + error_setg(&sev_mig_blocker, + "SEV: Migration is not implemented"); + ret = migrate_add_blocker(sev_mig_blocker, &local_err); + if (local_err) { + error_report_err(local_err); + error_free(sev_mig_blocker); + exit(1); + } } } From patchwork Tue Mar 2 20:48:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12112243 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9041CC433DB for ; Tue, 2 Mar 2021 21:06:33 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D3BA6614A7 for ; Tue, 2 Mar 2021 21:06:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D3BA6614A7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:49144 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHCDv-0004eW-I7 for qemu-devel@archiver.kernel.org; Tue, 02 Mar 2021 16:06:31 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45940) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBx1-0001yt-Fj for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:49:03 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:26374) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHBwp-000879-4Q for qemu-devel@nongnu.org; Tue, 02 Mar 2021 15:48:56 -0500 Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 122KiCZM092135; Tue, 2 Mar 2021 15:48:37 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=u7CCu3RoiU0rv7aH0pLqzRk11ep6jKjG3kUDpdwQ/1o=; b=H1wXrZk4hLr5Dh0X+ucZLT4hvghJV9ig7Aii1wBfI4bTr7jVFmfnC3QSH48g0wXsyMo+ 6KGFbMRf/QZA/QxAr/TxHjfHWYbK95jJT9LUwRPL2xNLPA00Yma8PC1q3utE1/UG0XNN ng9WL0OrqbKnq6E1TofHnVQsPWGiaI8dt97AU4IUAE8ivPASve8gGNQNMf02176Mzr0+ zEY9nzbTWJq3afEA31lFnCaA+HtrGEVzowQfQ69ZmFHFOVpdUWSJeBpkkm2Z8ekO2ZvR 6I9iq9yxItnATGjOnPw9Gg4P9lu8gbPpM+tAmokI6pFQQiyM7CklY/O2qfBbASi/6H1M 2A== Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com with ESMTP id 371vnf0678-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 15:48:37 -0500 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 122Kkp3P028076; Tue, 2 Mar 2021 20:48:36 GMT Received: from b01cxnp23032.gho.pok.ibm.com (b01cxnp23032.gho.pok.ibm.com [9.57.198.27]) by ppma01dal.us.ibm.com with ESMTP id 371qmuagux-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Mar 2021 20:48:36 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 122KmZU626280304 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Mar 2021 20:48:35 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5D8A428059; Tue, 2 Mar 2021 20:48:35 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2FCD22805A; Tue, 2 Mar 2021 20:48:35 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Mar 2021 20:48:35 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH 26/26] docs: Add confidential guest live migration documentation Date: Tue, 2 Mar 2021 15:48:22 -0500 Message-Id: <20210302204822.81901-27-dovmurik@linux.vnet.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> References: <20210302204822.81901-1-dovmurik@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-02_08:2021-03-01, 2021-03-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 mlxlogscore=999 malwarescore=0 impostorscore=0 phishscore=0 bulkscore=0 lowpriorityscore=0 mlxscore=0 suspectscore=0 priorityscore=1501 spamscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103020156 Received-SPF: none client-ip=148.163.156.1; envelope-from=dovmurik@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , James Bottomley , Jon Grimm , Tobin Feldman-Fitzthum , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The new page is linked from the main index, otherwise sphinx complains that "document isn't included in any toctree"; I assume there would be a better place for it in the documentation tree. Signed-off-by: Dov Murik --- docs/confidential-guest-live-migration.rst | 142 +++++++++++++++++++++ docs/confidential-guest-support.txt | 5 + docs/index.rst | 1 + 3 files changed, 148 insertions(+) create mode 100644 docs/confidential-guest-live-migration.rst diff --git a/docs/confidential-guest-live-migration.rst b/docs/confidential-guest-live-migration.rst new file mode 100644 index 0000000000..dc59df8f9c --- /dev/null +++ b/docs/confidential-guest-live-migration.rst @@ -0,0 +1,142 @@ +================================= +Confidential Guest Live Migration +================================= + +When migrating regular QEMU guests, QEMU reads the guest's RAM and sends it +over to the migration target host, where QEMU there writes it into the target +guest's RAM and starts the VM. This mechanism doesn't work when the guest +memory is encrypted or QEMU is prevented from reading it in another way. + +In order to support live migration in such scenarios, QEMU relies on an +in-guest migration helper which can securely extract RAM content from the +guest in order to send it to the target. The migration helper is implemented as +part of the VM's firmware in OVMF. + + +Migration flow +============== + +Source VM +--------- + +The source VM is started with an extra auxiliary vcpu which is not listed in the +ACPI tables. OVMF uses this vcpu and starts a dedicated migration helper on it; +the migration helper simply waits for commands from QEMU. When migration starts +using the ``migrate`` command, QEMU starts saving the state of the different +devices. When it reaches saving RAM pages, it'll check for each page whether it +is encrypted or not; for encrypted pages, it'll send a command to the migration +helper to extract the given page. The migration helper receives this command, +reads the page content, encrypts it with a transport key, and returns the +newly-encrypted page to QEMU. QEMU saves those pages to the outgoing migration +stream using a new page flag ``RAM_SAVE_FLAG_GUEST_ENCRYPTED_PAGE``. + +When QEMU reaches the last stage of RAM migration, it stops the source VM to +avoid dirtying the last pages of RAM. However, the auxiliary vcpu must be kept +running so the migration helper can still extract pages from the guest memory. + +Target VM +--------- + +Usually QEMU migration target VMs are started with the ``-incoming`` +command-line option which starts the VM paused. However, in order to migrate +confidential guests we must have the migration helper running inside the guest; +in such a case, we start the target with a special ``-fw_cfg`` value that tells +OVMF to enter a CPU dead loop on all vcpus except the auxiliary vcpu, which runs +the migration helper. After this short "boot" completes, QEMU can switch to the +"migration incoming" mode; we do that with the new ``start-migrate-incoming`` +QMP command that makes the target VM listen for incoming migration connections. + +QEMU will load the state of VM devices as it arrives from the incoming migration +stream. When it encounters a RAM page with the +``RAM_SAVE_FLAG_GUEST_ENCRYPTED_PAGE`` flag, it will send its +transport-encrypted content and guest physical address to the migration helper. +The migration helper running inside the guest will decrypt the page using the +transport key and place the content in memory (again, that memory page is not +accessible to host due to the confidential guest properties; for example, in SEV +it is hardware-encrypted with a VM-specific key). + + +Usage +===== + +In order to start the source and target VMs with auxiliary CPUs, the auxcpus= +option must be passed to ``-smp`` . For example:: + + # ${QEMU} -smp 5,auxcpus=1 ... + +This command starts a VM with 5 vcpus of which 4 are main vcpus (available for +the guest OS) and 1 is auxliary vcpu. + +Moreover, in both the source and target we need to instruct OVMF to start the +migration helper running in the auxiliary vcpu. This is achieved using the +following command-line option:: + + # ${QEMU} -fw_cfg name=opt/ovmf/PcdSevIsMigrationHelper,string=0 ... + +In the target VM we need to add another ``-fw_cfg`` entry to instruct OVMF to +start only the migration helepr, which will wait for incoming pages (the target +cannot be started with ``-incoming`` because that option completely pauses the +VM, not allowing the migration helper to run). Because the migration helper must +be running when the incoming RAM pages are received, starting the target VM with +the ``-incoming`` option doesn't work (with that option, the VM doesn't start +executing). Instead, start the target VM without ``-incoming`` but with the +following option:: + + # ${QEMU} -fw_cfg name=opt/ovmf/PcdSevIsMigrationTarget,string=1 ... + +After the VM boots into the migration helper, we instruct QEMU to listen for +incoming migration connections by sending the following QMP command:: + + { "execute": "start-migrate-incoming", + "arguments": { "uri": "tcp:0.0.0.0:6666" } } + +Now that the target is ready, we instruct the source VM to start migrating its +state using the regular ``migrate`` QMP command, supplying the target VMs +listening address:: + + { "execute": "migrate", + "arguments": { "uri": "tcp:192.168.111.222:6666" } } + + +Implementation details +====================== + +Migration helper <-> QEMU communication +--------------------------------------- + +The migration helper is running inside the guest (implemented as part of OVMF). +QEMU communicates with it using a mailbox protocol over two shared (unencrypted) +4K RAM pages. + +The first page contains a ``SevMigHelperCmdParams`` struct at offset 0x0 +(``cmd_params``) and a ``MigrationHelperHeader`` struct at offset 0x800 +(``io_hdr``). The second page (``io_page``) is dedicated for encrypted page +content. + +In order to save a confidential RAM page, QEMU will fill the ``cmd_params`` +struct to indicate the SEV_MIG_HELPER_CMD_ENCRYPT command and the requested gpa +(guest physical address), and then set the ``go`` field to 1. Meanwhile the +migration helper waits for the ``go`` field to become non-zero; after it notices +``go`` is 1 it'll read the gpa, read the content of the relevant page from the +guest's memory, encrypt it with the transport key, and store the +transport-encrypted page in the the ``io_page``. Additional envelope data like +encryption IV and other fields are stored in ``io_hdr``. After the migration is +done writing to ``io_page`` and ``io_hdr``, it sets the ``done`` field to 1. At +this point QEMU notices that the migration helper is done and can continue its +part, which is saving the header and page to the outgoing migration stream. + +Similar process is used when loading a confidential RAM from the incoming +migration stream. QEMU reads the header and the encrypted page from the stream, +and copies them into the shared areas ``io_hdr`` and ``io_page`` respectably. +It then fills the ``cmd_params`` struct to indicate the +SEV_MIG_HELPER_CMD_DECRYPT command and the gpa, and sets ``go`` to 1. The +migration helper will notice the command, will decrypt the page using the +transport key and will place the decrypted content in the requetsed gpa, and set +``done`` to 1 to allow QEMU to continue processing the next item in the incoming +migration stream. + +Shared pages address discovery +------------------------------ +In the current implementation the address of the two shared pages is hard-coded +in both OVMF and QEMU. We plan for OVMF to expose this address via its GUIDed +table and let QEMU discover it using ``pc_system_ovmf_table_find()``. diff --git a/docs/confidential-guest-support.txt b/docs/confidential-guest-support.txt index 71d07ba57a..bed1601fbb 100644 --- a/docs/confidential-guest-support.txt +++ b/docs/confidential-guest-support.txt @@ -47,3 +47,8 @@ s390x Protected Virtualization (PV) docs/system/s390x/protvirt.rst Other mechanisms may be supported in future. + +Live migration support +---------------------- +Details regarding confidential guest live migration are in: + docs/confidential-guest-live-migration.rst diff --git a/docs/index.rst b/docs/index.rst index 763e3d0426..6f797d050a 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -16,3 +16,4 @@ Welcome to QEMU's documentation! interop/index specs/index devel/index + confidential-guest-live-migration