From patchwork Mon Sep 4 01:35:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tao Su X-Patchwork-Id: 13373400 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D375FC83F3E for ; Mon, 4 Sep 2023 01:36:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350342AbjIDBgJ (ORCPT ); Sun, 3 Sep 2023 21:36:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59686 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350278AbjIDBgH (ORCPT ); Sun, 3 Sep 2023 21:36:07 -0400 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0811BF2 for ; Sun, 3 Sep 2023 18:36:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1693791362; x=1725327362; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=RoFlc4JBC10OMuq/moZm50NnjJ1GpnDjtaFUFC+K/N8=; b=Qbusm095FOmnf0H6f8ANlsXRDQh57pALa3xBrMawSK7RJpKCeSRDGjxV qtKEg+hfq/8ff20g9iZm1t6RgJFV2KGyFvDVMgBsVXZIRycLCA7ziDlVu 8DUtDxRhBXVvK2bxkbQHCTtj01GmsPME3fDv8zVV/StjDMuTrItCb91BN U3sZWxoVxbjit6NFGGeTyeE70IWmXWsxNgQfSpFaKKa38XAsA6JjvsP+0 XiqtiXDJBGJfy/P7VSaD0JgFv4XgjYkB95IuFEkd8/XbN1vd+m2fDrP12 IJqiQ0XV3WEM38Y3FAheRB/sYTWPX8pxnoIlloCiOKrAYsKLCvEOsF6ei g==; X-IronPort-AV: E=McAfee;i="6600,9927,10822"; a="407493240" X-IronPort-AV: E=Sophos;i="6.02,225,1688454000"; d="scan'208";a="407493240" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Sep 2023 18:36:01 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10822"; a="743764797" X-IronPort-AV: E=Sophos;i="6.02,225,1688454000"; d="scan'208";a="743764797" Received: from st-server.bj.intel.com ([10.240.193.102]) by fmsmga007.fm.intel.com with ESMTP; 03 Sep 2023 18:35:59 -0700 From: Tao Su To: kvm@vger.kernel.org Cc: seanjc@google.com, pbonzini@redhat.com, chao.gao@intel.com, guang.zeng@intel.com, yi1.lai@intel.com, tao1.su@linux.intel.com Subject: [PATCH 1/2] x86/apic: Introduce X2APIC_ICR_UNUSED_12 for x2APIC mode Date: Mon, 4 Sep 2023 09:35:54 +0800 Message-Id: <20230904013555.725413-2-tao1.su@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230904013555.725413-1-tao1.su@linux.intel.com> References: <20230904013555.725413-1-tao1.su@linux.intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org According to SDM, bit12 of ICR is no longer BUSY bit but UNUSED bit in x2APIC mode, which is the only difference of lower ICR between xAPIC and x2APIC mode. To avoid ambiguity, introduce X2APIC_ICR_UNUSED_12 for x2APIC mode. Signed-off-by: Tao Su --- arch/x86/include/asm/apicdef.h | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/include/asm/apicdef.h b/arch/x86/include/asm/apicdef.h index 4b125e5b3187..ea2725738b81 100644 --- a/arch/x86/include/asm/apicdef.h +++ b/arch/x86/include/asm/apicdef.h @@ -78,6 +78,7 @@ #define APIC_INT_LEVELTRIG 0x08000 #define APIC_INT_ASSERT 0x04000 #define APIC_ICR_BUSY 0x01000 +#define X2APIC_ICR_UNUSED_12 0x01000 #define APIC_DEST_LOGICAL 0x00800 #define APIC_DEST_PHYSICAL 0x00000 #define APIC_DM_FIXED 0x00000 From patchwork Mon Sep 4 01:35:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tao Su X-Patchwork-Id: 13373399 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0037ACA0FE3 for ; Mon, 4 Sep 2023 01:36:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350346AbjIDBgK (ORCPT ); Sun, 3 Sep 2023 21:36:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59688 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350282AbjIDBgI (ORCPT ); Sun, 3 Sep 2023 21:36:08 -0400 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D9211F1 for ; Sun, 3 Sep 2023 18:36:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1693791364; x=1725327364; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=63ah4cavcJ/lGN/GrD4kdJPA0fhSOQprjmv2IjSCMi4=; b=l3zzhwEIlVXsv7nSyEiiZG7LsGIdZi2McShWp1GWr2QIoW0ub2ATicDZ uisrfG5sGN8vC4hw9uMXvm6yVKZk56zTulTga9jmufqBYUcdL5yOPkL0z v6oB1PC72cRRO4Xa3xQaWtJWnSNZXVc84Ik5VJhF0AqPhxNTr8hLHnrVR hl/mSyC8L/8WrMWry4NyN7CRWe18jq/W4USc+LNZI0jUCGYFfmb7YWgq8 WN9PZT9enYArRZSZsHsGpRgr95kBGTm+dxHSr2JjCCwEqRwNxn9k2ZUDb 04Mc8GeXYVp3a9/5nWZBMyMCJ+RPg2oxWlR+1v8BEg+W3bT5ylaMwsjOF Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10822"; a="407493244" X-IronPort-AV: E=Sophos;i="6.02,225,1688454000"; d="scan'208";a="407493244" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Sep 2023 18:36:04 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10822"; a="743764817" X-IronPort-AV: E=Sophos;i="6.02,225,1688454000"; d="scan'208";a="743764817" Received: from st-server.bj.intel.com ([10.240.193.102]) by fmsmga007.fm.intel.com with ESMTP; 03 Sep 2023 18:36:01 -0700 From: Tao Su To: kvm@vger.kernel.org Cc: seanjc@google.com, pbonzini@redhat.com, chao.gao@intel.com, guang.zeng@intel.com, yi1.lai@intel.com, tao1.su@linux.intel.com Subject: [PATCH 2/2] KVM: x86: Clear X2APIC_ICR_UNUSED_12 after APIC-write VM-exit Date: Mon, 4 Sep 2023 09:35:55 +0800 Message-Id: <20230904013555.725413-3-tao1.su@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230904013555.725413-1-tao1.su@linux.intel.com> References: <20230904013555.725413-1-tao1.su@linux.intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When IPI virtualization is enabled, a WARN is triggered if bit12 of ICR MSR is set after APIC-write VM-exit. The reason is kvm_apic_send_ipi() thinks the APIC_ICR_BUSY bit should be cleared because KVM has no delay, but kvm_apic_write_nodecode() doesn't clear the APIC_ICR_BUSY bit. Since bit12 of ICR is no longer BUSY bit but UNUSED bit in x2APIC mode, and SDM has no detail about how hardware will handle the UNUSED bit12 set, we tested on Intel CPU (SRF/GNR) with IPI virtualization and found the UNUSED bit12 was also cleared by hardware without #GP. Therefore, the clearing of bit12 should be still kept being consistent with the hardware behavior. Fixes: 5413bcba7ed5 ("KVM: x86: Add support for vICR APIC-write VM-Exits in x2APIC mode") Signed-off-by: Tao Su Tested-by: Yi Lai --- arch/x86/kvm/lapic.c | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index a983a16163b1..09a376aeb4a0 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -1482,8 +1482,17 @@ void kvm_apic_send_ipi(struct kvm_lapic *apic, u32 icr_low, u32 icr_high) { struct kvm_lapic_irq irq; - /* KVM has no delay and should always clear the BUSY/PENDING flag. */ - WARN_ON_ONCE(icr_low & APIC_ICR_BUSY); + /* + * In non-x2apic mode, KVM has no delay and should always clear the + * BUSY/PENDING flag. In x2apic mode, KVM should clear the unused bit12 + * of ICR since hardware will also clear this bit. Although + * APIC_ICR_BUSY and X2APIC_ICR_UNUSED_12 are same, they mean different + * things in different modes. + */ + if (!apic_x2apic_mode(apic)) + WARN_ON_ONCE(icr_low & APIC_ICR_BUSY); + else + WARN_ON_ONCE(icr_low & X2APIC_ICR_UNUSED_12); irq.vector = icr_low & APIC_VECTOR_MASK; irq.delivery_mode = icr_low & APIC_MODE_MASK; @@ -2429,13 +2438,12 @@ void kvm_apic_write_nodecode(struct kvm_vcpu *vcpu, u32 offset) * ICR is a single 64-bit register when x2APIC is enabled. For legacy * xAPIC, ICR writes need to go down the common (slightly slower) path * to get the upper half from ICR2. + * + * TODO: optimize to just emulate side effect w/o one more write */ if (apic_x2apic_mode(apic) && offset == APIC_ICR) { - val = kvm_lapic_get_reg64(apic, APIC_ICR); - kvm_apic_send_ipi(apic, (u32)val, (u32)(val >> 32)); - trace_kvm_apic_write(APIC_ICR, val); + kvm_x2apic_icr_write(apic, val); } else { - /* TODO: optimize to just emulate side effect w/o one more write */ val = kvm_lapic_get_reg(apic, offset); kvm_lapic_reg_write(apic, offset, (u32)val); } @@ -3122,7 +3130,12 @@ int kvm_lapic_set_vapic_addr(struct kvm_vcpu *vcpu, gpa_t vapic_addr) int kvm_x2apic_icr_write(struct kvm_lapic *apic, u64 data) { - data &= ~APIC_ICR_BUSY; + /* + * The Delivery Status Bit(bit 12) is removed in x2apic mode, but this + * bit is also cleared by hardware, so keep consistent with hardware + * behavior to clear this bit. + */ + data &= ~X2APIC_ICR_UNUSED_12; kvm_apic_send_ipi(apic, (u32)data, (u32)(data >> 32)); kvm_lapic_set_reg64(apic, APIC_ICR, data);