From patchwork Wed Dec 4 10:44:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Roger_Pau_Monn=C3=A9?= X-Patchwork-Id: 11272733 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5533C138D for ; Wed, 4 Dec 2019 10:46:13 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2FA4C207DD for ; Wed, 4 Dec 2019 10:46:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="LCPmC8BO" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2FA4C207DD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1icS9M-0001q5-36; Wed, 04 Dec 2019 10:44:52 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1icS9K-0001pu-Uv for xen-devel@lists.xenproject.org; Wed, 04 Dec 2019 10:44:50 +0000 X-Inumbo-ID: 187926ae-1683-11ea-a0d2-bc764e2007e4 Received: from esa5.hc3370-68.iphmx.com (unknown [216.71.155.168]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 187926ae-1683-11ea-a0d2-bc764e2007e4; Wed, 04 Dec 2019 10:44:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1575456289; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=xv4s6xp4c54ioRBMxc59+ltO1VVHkuPuvM/98/apfxc=; b=LCPmC8BONl+CJcBx5qULWHKyL7EOvq1ihcfCDzrr5Psh4Yu1dREQUrBG nqVh4hGO88kQEOzAh8WZo/sNMUjjMEy1Fe6hcBvkYSj5/KaXmhczs3Bpb ysknu8mde3AyGN79AmL1bhrdjfHZqI9uoLSm22pjA4R4LbjbvKSDkiCF5 Q=; Authentication-Results: esa5.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=roger.pau@citrix.com; spf=Pass smtp.mailfrom=roger.pau@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: None (esa5.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa5.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa5.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: sZzdRLFP8RM5uV3caTO2m84C+LxxLC+W4Vh0X2Ao9hLuGBj9CXvfj29+gVyinmyS1BTQBSivC2 fdXKghggHg4T1p+1n1oHaZ5TZbFq5VBvbd4T0Pgs2SCniD8wtUTkEsaCEkWuTzjfRJT8CUqEH4 kMVXl8PiDzaQkJtNtq4eLE1wb92+F3jhkGPXbkX545ssYx9nhbKylSGyDzbl/6VtP87+9/PhNe jGF1boex3IlO+QRZVWbY4m6myUbAOHvkvN4fxJMkcpQ/XlmRIA7p1vgE6c0/0ZlZFDLGdYyvUQ Us8= X-SBRS: 2.7 X-MesageID: 9532434 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.69,277,1571716800"; d="scan'208";a="9532434" From: Roger Pau Monne To: Date: Wed, 4 Dec 2019 11:44:20 +0100 Message-ID: <20191204104420.34418-1-roger.pau@citrix.com> X-Mailer: git-send-email 2.24.0 MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v2] x86: do not enable global pages when virtualized on AMD hardware X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Stefano Stabellini , Julien Grall , Wei Liu , Konrad Rzeszutek Wilk , George Dunlap , Andrew Cooper , Ian Jackson , Jan Beulich , Roger Pau Monne Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" When using global pages a full tlb flush can only be performed by toggling the PGE bit in CR4, which is usually quite expensive in terms of performance when running virtualized. This is specially relevant on AMD hardware, which doesn't have the ability to do selective CR4 trapping, but can also be relevant on Intel if the underlying hypervisor also traps accesses to the PGE CR4 bit. In order to avoid this performance penalty, do not use global pages when running virtualized on AMD hardware. A command line option 'global-pages' is provided in order to allow the user to select whether global pages will be enabled for PV guests. The above figures are from a PV shim running on AMD hardware with 32 vCPUs: PGE enabled, x2APIC mode: (XEN) Global lock flush_lock: addr=ffff82d0804b01c0, lockval=1adb1adb, not locked (XEN) lock:1841883(1375128998543), block:1658716(10193054890781) Average lock time: 746588ns Average block time: 6145147ns PGE disabled, x2APIC mode: (XEN) Global lock flush_lock: addr=ffff82d0804af1c0, lockval=a8bfa8bf, not locked (XEN) lock:2730175(657505389886), block:2039716(2963768247738) Average lock time: 240829ns Average block time: 1453029ns As seen from the above figures the lock and block time of the flush lock is reduced to approximately 1/3 of the original value. Signed-off-by: Roger Pau Monné --- Changes since v1: - Provide command line option to enable/disable PGE. - Only disable PGE on AMD hardware when virtualized. - Document the global-pages option. --- docs/misc/xen-command-line.pandoc | 13 +++++++++++++ xen/arch/x86/pv/domain.c | 9 ++++++++- 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc index d9495ef6b9..7be30f2766 100644 --- a/docs/misc/xen-command-line.pandoc +++ b/docs/misc/xen-command-line.pandoc @@ -1087,6 +1087,19 @@ value settable via Xen tools. Dom0 is using this value for sizing its maptrack table. +### global-pages (x86) +> `= ` + +> Default: `true` unless running virtualized on AMD hardware + +Set whether the PGE bit in CR4 will be enabled for PV guests. This controls the +usage of global pages, and thus the need to perform tlb flushes by writing to +CR4. + +Note it's disabled by default when running virtualized on AMD hardware since +AMD SVM doesn't support selective trapping of CR4, so global pages are not +enabled in order to reduce the overhead of tlb flushes. + ### guest_loglvl > `= [/]` where level is `none | error | warning | info | debug | all` diff --git a/xen/arch/x86/pv/domain.c b/xen/arch/x86/pv/domain.c index 4b6f48dea2..93fb823d63 100644 --- a/xen/arch/x86/pv/domain.c +++ b/xen/arch/x86/pv/domain.c @@ -118,11 +118,18 @@ unsigned long pv_fixup_guest_cr4(const struct vcpu *v, unsigned long cr4) (mmu_cr4_features & PV_CR4_GUEST_VISIBLE_MASK)); } +static int opt_global_pages = -1; +boolean_runtime_param("global-pages", opt_global_pages); + unsigned long pv_make_cr4(const struct vcpu *v) { const struct domain *d = v->domain; unsigned long cr4 = mmu_cr4_features & ~(X86_CR4_PCIDE | X86_CR4_PGE | X86_CR4_TSD); + bool pge = opt_global_pages == -1 ? (!cpu_has_hypervisor || + boot_cpu_data.x86_vendor != + X86_VENDOR_AMD) + : !!opt_global_pages; /* * PCIDE or PGE depends on the PCID/XPTI settings, but must not both be @@ -130,7 +137,7 @@ unsigned long pv_make_cr4(const struct vcpu *v) */ if ( d->arch.pv.pcid ) cr4 |= X86_CR4_PCIDE; - else if ( !d->arch.pv.xpti ) + else if ( !d->arch.pv.xpti && pge ) cr4 |= X86_CR4_PGE; /*