From patchwork Wed Feb 15 21:37:50 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 9575089 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 1F5266045F for ; Wed, 15 Feb 2017 21:40:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 133942851B for ; Wed, 15 Feb 2017 21:40:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 07DD428544; Wed, 15 Feb 2017 21:40:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 7B0BF2851B for ; Wed, 15 Feb 2017 21:40:21 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ce7HH-0004pN-3n; Wed, 15 Feb 2017 21:38:19 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ce7HF-0004on-Qa for xen-devel@lists.xenproject.org; Wed, 15 Feb 2017 21:38:17 +0000 Received: from [85.158.143.35] by server-8.bemta-6.messagelabs.com id 60/05-21675-94AC4A85; Wed, 15 Feb 2017 21:38:17 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrELMWRWlGSWpSXmKPExsVysWW7jK77qSU RBj9WyFp83zKZyYHR4/CHKywBjFGsmXlJ+RUJrBlz/hxnL3goUzGvL6qBsV+ii5GLQ0hgN5PE 5j/nWCGcg4wS8/98B3I4OdgE1CT+3OoEs0UEjjNJ/J6hBFLELPCMWeJqbwd7FyMHh7BAjMSFS 7ogNSwCqhLz3+9kAbF5BRwlvm44yQJSwingJNHXzQUSFgIK/zl6nB3ElhDQlmiYNZkFZKSEQB +jxMrjG9knMPIsYGRYxahRnFpUllqka2iol1SUmZ5RkpuYmaNraGCml5taXJyYnpqTmFSsl5y fu4kR6HcGINjB+GlZwCFGSQ4mJVHeX/uWRAjxJeWnVGYkFmfEF5XmpBYfYpTh4FCS4PU/CZQT LEpNT61Iy8wBBiBMWoKDR0mEtwkkzVtckJhbnJkOkTrFqCglzhsGkhAASWSU5sG1wYL+EqOsl DAvI9AhQjwFqUW5mSWo8q8YxTkYlYR554NM4cnMK4Gb/gpoMRPQYta4hSCLSxIRUlINjD1X3i 6YONnHMV1oQ4Spdcn9ztOLdkS19hkvXR7/vlA7/JalopauTLyFnfHp4/EvPKau3H7q5e3E6fk 8t/v8NT8vaDywe9n3GKUCf2uDbpcnyeK5nrpfO750x59Pnnz293mBtLLjbu9fz3uzrOXm4tpJ JbosoXYrV2lNVwm7kDTV6Oa7KQYarkosxRmJhlrMRcWJABFi5ih1AgAA X-Env-Sender: longman@redhat.com X-Msg-Ref: server-2.tower-21.messagelabs.com!1487194694!44037664!1 X-Originating-IP: [209.132.183.28] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMjA5LjEzMi4xODMuMjggPT4gNTQwNjQ=\n X-StarScan-Received: X-StarScan-Version: 9.2.3; banners=-,-,- X-VirusChecked: Checked Received: (qmail 26575 invoked from network); 15 Feb 2017 21:38:15 -0000 Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by server-2.tower-21.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 15 Feb 2017 21:38:15 -0000 Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 48D6E80463; Wed, 15 Feb 2017 21:38:14 +0000 (UTC) Received: from llong.com (dhcp-17-128.bos.redhat.com [10.18.17.128]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v1FLc6DA008201; Wed, 15 Feb 2017 16:38:12 -0500 From: Waiman Long To: Jeremy Fitzhardinge , Chris Wright , Alok Kataria , Rusty Russell , Peter Zijlstra , Ingo Molnar , Thomas Gleixner , "H. Peter Anvin" Date: Wed, 15 Feb 2017 16:37:50 -0500 Message-Id: <1487194670-6319-3-git-send-email-longman@redhat.com> In-Reply-To: <1487194670-6319-1-git-send-email-longman@redhat.com> References: <1487194670-6319-1-git-send-email-longman@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Wed, 15 Feb 2017 21:38:14 +0000 (UTC) Cc: linux-arch@vger.kernel.org, Juergen Gross , kvm@vger.kernel.org, =?UTF-8?q?Radim=20Kr=C4=8Dm=C3=A1=C5=99?= , Pan Xinhui , x86@kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, Waiman Long , Paolo Bonzini , xen-devel@lists.xenproject.org, Boris Ostrovsky Subject: [Xen-devel] [PATCH v4 2/2] x86/kvm: Provide optimized version of vcpu_is_preempted() for x86-64 X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP It was found when running fio sequential write test with a XFS ramdisk on a KVM guest running on a 2-socket x86-64 system, the %CPU times as reported by perf were as follows: 69.75% 0.59% fio [k] down_write 69.15% 0.01% fio [k] call_rwsem_down_write_failed 67.12% 1.12% fio [k] rwsem_down_write_failed 63.48% 52.77% fio [k] osq_lock 9.46% 7.88% fio [k] __raw_callee_save___kvm_vcpu_is_preempt 3.93% 3.93% fio [k] __kvm_vcpu_is_preempted Making vcpu_is_preempted() a callee-save function has a relatively high cost on x86-64 primarily due to at least one more cacheline of data access from the saving and restoring of registers (8 of them) to and from stack as well as one more level of function call. To reduce this performance overhead, an optimized assembly version of the the __raw_callee_save___kvm_vcpu_is_preempt() function is provided for x86-64. With this patch applied on a KVM guest on a 2-socekt 16-core 32-thread system with 16 parallel jobs (8 on each socket), the aggregrate bandwidth of the fio test on an XFS ramdisk were as follows: I/O Type w/o patch with patch -------- --------- ---------- random read 8141.2 MB/s 8497.1 MB/s seq read 8229.4 MB/s 8304.2 MB/s random write 1675.5 MB/s 1701.5 MB/s seq write 1681.3 MB/s 1699.9 MB/s There are some increases in the aggregated bandwidth because of the patch. The perf data now became: 70.78% 0.58% fio [k] down_write 70.20% 0.01% fio [k] call_rwsem_down_write_failed 69.70% 1.17% fio [k] rwsem_down_write_failed 59.91% 55.42% fio [k] osq_lock 10.14% 10.14% fio [k] __kvm_vcpu_is_preempted The assembly code was verified by using a test kernel module to compare the output of C __kvm_vcpu_is_preempted() and that of assembly __raw_callee_save___kvm_vcpu_is_preempt() to verify that they matched. Suggested-by: Peter Zijlstra Signed-off-by: Waiman Long --- arch/x86/kernel/kvm.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index 85ed343..e423435 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -589,6 +589,7 @@ static void kvm_wait(u8 *ptr, u8 val) local_irq_restore(flags); } +#ifdef CONFIG_X86_32 __visible bool __kvm_vcpu_is_preempted(long cpu) { struct kvm_steal_time *src = &per_cpu(steal_time, cpu); @@ -597,11 +598,40 @@ __visible bool __kvm_vcpu_is_preempted(long cpu) } PV_CALLEE_SAVE_REGS_THUNK(__kvm_vcpu_is_preempted); +#else + +extern bool __raw_callee_save___kvm_vcpu_is_preempted(long); + +/* + * Hand-optimize version for x86-64 to avoid 8 64-bit register saving and + * restoring to/from the stack. It is assumed that the preempted value + * is at an offset of 16 from the beginning of the kvm_steal_time structure + * which is verified by the BUILD_BUG_ON() macro below. + */ +#define PREEMPTED_OFFSET 16 +asm( +".pushsection .text;" +".global __raw_callee_save___kvm_vcpu_is_preempted;" +".type __raw_callee_save___kvm_vcpu_is_preempted, @function;" +"__raw_callee_save___kvm_vcpu_is_preempted:" +"movq __per_cpu_offset(,%rdi,8), %rax;" +"cmpb $0, " __stringify(PREEMPTED_OFFSET) "+steal_time(%rax);" +"setne %al;" +"ret;" +".popsection"); + +#endif + /* * Setup pv_lock_ops to exploit KVM_FEATURE_PV_UNHALT if present. */ void __init kvm_spinlock_init(void) { +#ifdef CONFIG_X86_64 + BUILD_BUG_ON((offsetof(struct kvm_steal_time, preempted) + != PREEMPTED_OFFSET) || (sizeof(steal_time.preempted) != 1)); +#endif + if (!kvm_para_available()) return; /* Does host kernel support KVM_FEATURE_PV_UNHALT? */