From patchwork Tue Aug 31 01:59:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tianqiang Xu X-Patchwork-Id: 12466237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C7A6C432BE for ; Tue, 31 Aug 2021 02:00:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 71ECD6101C for ; Tue, 31 Aug 2021 02:00:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239336AbhHaCAy (ORCPT ); Mon, 30 Aug 2021 22:00:54 -0400 Received: from smtp181.sjtu.edu.cn ([202.120.2.181]:41020 "EHLO smtp181.sjtu.edu.cn" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235217AbhHaCAx (ORCPT ); Mon, 30 Aug 2021 22:00:53 -0400 Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id D47751008CBCD; Tue, 31 Aug 2021 09:59:55 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id C5B93228C9244; Tue, 31 Aug 2021 09:59:55 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id SBVDgW0gRBe3; Tue, 31 Aug 2021 09:59:55 +0800 (CST) Received: from sky.ipads-lab.se.sjtu.edu.cn (unknown [202.120.40.82]) (Authenticated sender: skyele@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id 40DB2228C9235; Tue, 31 Aug 2021 09:59:34 +0800 (CST) From: Tianqiang Xu To: x86@kernel.org Cc: pbonzini@redhat.com, seanjc@google.com, vkuznets@redhat.com, wanpengli@tencent.com, jmattson@google.com, joro@8bytes.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, kvm@vger.kernel.org, hpa@zytor.com, jarkko@kernel.org, dave.hansen@linux.intel.com, linux-kernel@vger.kernel.org, linux-sgx@vger.kernel.org, Tianqiang Xu Subject: [PATCH 1/4] KVM: x86: Introduce .pcpu_is_idle() stub infrastructure Date: Tue, 31 Aug 2021 09:59:16 +0800 Message-Id: <20210831015919.13006-1-skyele@sjtu.edu.cn> X-Mailer: git-send-email 2.17.1 Precedence: bulk List-ID: X-Mailing-List: linux-sgx@vger.kernel.org This patch series aims to fix performance issue caused by current para-virtualized scheduling design. The current para-virtualized scheduling design uses 'preempted' field of kvm_steal_time to avoid scheduling task on the preempted vCPU. However, when the pCPU where the preempted vCPU most recently run is idle, it will result in low cpu utilization, and consequently poor performance. The new field: 'is_idle' of kvm_steal_time can precisely reveal the status of pCPU where preempted vCPU most recently run, and then improve cpu utilization. pcpu_is_idle() is used to get the value of 'is_idle' of kvm_steal_time. Experiments on a VM with 16 vCPUs show that the patch can reduce around 50% to 80% execution time for most PARSEC benchmarks. This also holds true for a VM with 112 vCPUs. Experiments on 2 VMs with 112 vCPUs show that the patch can reduce around 20% to 80% execution time for most PARSEC benchmarks. Test environment: -- PowerEdge R740 -- 56C-112T CPU Intel(R) Xeon(R) Gold 6238R CPU -- Host 190G DRAM -- QEMU 5.0.0 -- PARSEC 3.0 Native Inputs -- Host is idle during the test -- Host and Guest kernel are both kernel-5.14.0 Results: 1. 1 VM, 16 VCPU, 16 THREAD. Host Topology: sockets=2 cores=28 threads=2 VM Topology: sockets=1 cores=16 threads=1 Command: /bin/parsecmgmt -a run -p -i native -n 16 Statistics below are the real time of running each benchmark.(lower is better) before patch after patch improvements bodytrack 52.866s 22.619s 57.21% fluidanimate 84.009s 38.148s 54.59% streamcluster 270.17s 42.726s 84.19% splash2x.ocean_cp 31.932s 9.539s 70.13% splash2x.ocean_ncp 36.063s 14.189s 60.65% splash2x.volrend 134.587s 21.79s 83.81% 2. 1VM, 112 VCPU. Some benchmarks require the number of threads to be the power of 2, so we run them with 64 threads and 128 threads. Host Topology: sockets=2 cores=28 threads=2 VM Topology: sockets=1 cores=112 threads=1 Command: /bin/parsecmgmt -a run -p -i native -n <64,112,128> Statistics below are the real time of running each benchmark.(lower is better) before patch after patch improvements fluidanimate(64 thread) 124.235s 27.924s 77.52% fluidanimate(128 thread) 169.127s 64.541s 61.84% streamcluster(112 thread) 861.879s 496.66s 42.37% splash2x.ocean_cp(64 thread) 46.415s 18.527s 60.08% splash2x.ocean_cp(128 thread) 53.647s 28.929s 46.08% splash2x.ocean_ncp(64 thread) 47.613s 19.576s 58.89% splash2x.ocean_ncp(128 thread) 54.94s 29.199s 46.85% splash2x.volrend(112 thread) 801.384s 144.824s 81.93% 3. 2VM, each VM: 112 VCPU. Some benchmarks require the number of threads to be the power of 2, so we run them with 64 threads and 128 threads. Host Topology: sockets=2 cores=28 threads=2 VM Topology: sockets=1 cores=112 threads=1 Command: /bin/parsecmgmt -a run -p -i native -n <64,112,128> Statistics below are the average real time of running each benchmark in 2 VMs.(lower is better) before patch after patch improvements fluidanimate(64 thread) 135.2125s 49.827s 63.15% fluidanimate(128 thread) 178.309s 86.964s 51.23% splash2x.ocean_cp(64 thread) 47.4505s 20.314s 57.19% splash2x.ocean_cp(128 thread) 55.5645s 30.6515s 44.84% splash2x.ocean_ncp(64 thread) 49.9775s 23.489s 53.00% splash2x.ocean_ncp(128 thread) 56.847s 28.545s 49.79% splash2x.volrend(112 thread) 838.939s 239.632s 71.44% For space limit, we list representative statistics here. --- Authors: Tianqiang Xu, Dingji Li, Zeyu Mi Shanghai Jiao Tong University Signed-off-by: Tianqiang Xu --- arch/x86/hyperv/hv_spinlock.c | 7 +++++++ arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/kvm_host.h | 1 + arch/x86/include/asm/paravirt.h | 8 ++++++++ arch/x86/include/asm/paravirt_types.h | 1 + arch/x86/include/asm/qspinlock.h | 7 +++++++ arch/x86/include/uapi/asm/kvm_para.h | 4 +++- arch/x86/kernel/asm-offsets_64.c | 1 + arch/x86/kernel/kvm.c | 21 +++++++++++++++++++++ arch/x86/kernel/paravirt-spinlocks.c | 15 +++++++++++++++ arch/x86/kernel/paravirt.c | 2 ++ 11 files changed, 67 insertions(+), 1 deletion(-) diff --git a/arch/x86/hyperv/hv_spinlock.c b/arch/x86/hyperv/hv_spinlock.c index 91cfe698bde0..9782e188904a 100644 --- a/arch/x86/hyperv/hv_spinlock.c +++ b/arch/x86/hyperv/hv_spinlock.c @@ -60,6 +60,12 @@ static void hv_qlock_wait(u8 *byte, u8 val) /* * Hyper-V does not support this so far. */ +__visible bool hv_pcpu_is_idle(int vcpu) +{ + return false; +} +PV_CALLEE_SAVE_REGS_THUNK(hv_pcpu_is_idle); + __visible bool hv_vcpu_is_preempted(int vcpu) { return false; @@ -82,6 +88,7 @@ void __init hv_init_spinlocks(void) pv_ops.lock.wait = hv_qlock_wait; pv_ops.lock.kick = hv_qlock_kick; pv_ops.lock.vcpu_is_preempted = PV_CALLEE_SAVE(hv_vcpu_is_preempted); + pv_ops.lock.pcpu_is_idle = PV_CALLEE_SAVE(hv_pcpu_is_idle); } static __init int hv_parse_nopvspin(char *arg) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index d0ce5cfd3ac1..efda9b9a4cad 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -238,6 +238,7 @@ #define X86_FEATURE_VMW_VMMCALL ( 8*32+19) /* "" VMware prefers VMMCALL hypercall instruction */ #define X86_FEATURE_PVUNLOCK ( 8*32+20) /* "" PV unlock function */ #define X86_FEATURE_VCPUPREEMPT ( 8*32+21) /* "" PV vcpu_is_preempted function */ +#define X86_FEATURE_PCPUISIDLE ( 8*32+22) /* "" PV pcpu_is_idle function */ /* Intel-defined CPU features, CPUID level 0x00000007:0 (EBX), word 9 */ #define X86_FEATURE_FSGSBASE ( 9*32+ 0) /* RDFSBASE, WRFSBASE, RDGSBASE, WRGSBASE instructions*/ diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index af6ce8d4c86a..705c55be0eed 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -742,6 +742,7 @@ struct kvm_vcpu_arch { struct { u8 preempted; + u8 is_idle; u64 msr_val; u64 last_steal; struct gfn_to_pfn_cache cache; diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h index da3a1ac82be5..f34dec6eb515 100644 --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -609,8 +609,16 @@ static __always_inline bool pv_vcpu_is_preempted(long cpu) ALT_NOT(X86_FEATURE_VCPUPREEMPT)); } +static __always_inline bool pv_pcpu_is_idle(long cpu) +{ + return PVOP_ALT_CALLEE1(bool, lock.pcpu_is_idle, cpu, + "xor %%" _ASM_AX ", %%" _ASM_AX ";", + ALT_NOT(X86_FEATURE_PCPUISIDLE)); +} + void __raw_callee_save___native_queued_spin_unlock(struct qspinlock *lock); bool __raw_callee_save___native_vcpu_is_preempted(long cpu); +bool __raw_callee_save___native_pcpu_is_idle(long cpu); #endif /* SMP && PARAVIRT_SPINLOCKS */ diff --git a/arch/x86/include/asm/paravirt_types.h b/arch/x86/include/asm/paravirt_types.h index d9d6b0203ec4..7d9b5906580c 100644 --- a/arch/x86/include/asm/paravirt_types.h +++ b/arch/x86/include/asm/paravirt_types.h @@ -257,6 +257,7 @@ struct pv_lock_ops { void (*kick)(int cpu); struct paravirt_callee_save vcpu_is_preempted; + struct paravirt_callee_save pcpu_is_idle; } __no_randomize_layout; /* This contains all the paravirt structures: we get a convenient diff --git a/arch/x86/include/asm/qspinlock.h b/arch/x86/include/asm/qspinlock.h index d86ab942219c..c32f2eb6186c 100644 --- a/arch/x86/include/asm/qspinlock.h +++ b/arch/x86/include/asm/qspinlock.h @@ -63,6 +63,13 @@ static inline bool vcpu_is_preempted(long cpu) } #endif +#define pcpu_is_idle pcpu_is_idle +static inline bool pcpu_is_idle(long cpu) +{ + return pv_pcpu_is_idle(cpu); +} +#endif + #ifdef CONFIG_PARAVIRT /* * virt_spin_lock_key - enables (by default) the virt_spin_lock() hijack. diff --git a/arch/x86/include/uapi/asm/kvm_para.h b/arch/x86/include/uapi/asm/kvm_para.h index 5146bbab84d4..82940e4b76d5 100644 --- a/arch/x86/include/uapi/asm/kvm_para.h +++ b/arch/x86/include/uapi/asm/kvm_para.h @@ -63,12 +63,14 @@ struct kvm_steal_time { __u32 version; __u32 flags; __u8 preempted; - __u8 u8_pad[3]; + __u8 is_idle; + __u8 u8_pad[2]; __u32 pad[11]; }; #define KVM_VCPU_PREEMPTED (1 << 0) #define KVM_VCPU_FLUSH_TLB (1 << 1) +#define KVM_PCPU_IS_IDLE (1 << 0) #define KVM_CLOCK_PAIRING_WALLCLOCK 0 struct kvm_clock_pairing { diff --git a/arch/x86/kernel/asm-offsets_64.c b/arch/x86/kernel/asm-offsets_64.c index b14533af7676..b587bbe44470 100644 --- a/arch/x86/kernel/asm-offsets_64.c +++ b/arch/x86/kernel/asm-offsets_64.c @@ -22,6 +22,7 @@ int main(void) #if defined(CONFIG_KVM_GUEST) && defined(CONFIG_PARAVIRT_SPINLOCKS) OFFSET(KVM_STEAL_TIME_preempted, kvm_steal_time, preempted); + OFFSET(KVM_STEAL_TIME_is_idle, kvm_steal_time, is_idle); BLANK(); #endif diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index a26643dc6bd6..b167589fffbc 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -900,11 +900,19 @@ __visible bool __kvm_vcpu_is_preempted(long cpu) } PV_CALLEE_SAVE_REGS_THUNK(__kvm_vcpu_is_preempted); +__visible bool __kvm_pcpu_is_idle(long cpu) +{ + struct kvm_steal_time *src = &per_cpu(steal_time, cpu); + + return !!(src->preempted & KVM_PCPU_IS_IDLE); +} +PV_CALLEE_SAVE_REGS_THUNK(__kvm_pcpu_is_idle); #else #include extern bool __raw_callee_save___kvm_vcpu_is_preempted(long); +extern bool __raw_callee_save___kvm_pcpu_is_idle(long); /* * Hand-optimize version for x86-64 to avoid 8 64-bit register saving and @@ -922,6 +930,17 @@ asm( ".size __raw_callee_save___kvm_vcpu_is_preempted, .-__raw_callee_save___kvm_vcpu_is_preempted;" ".popsection"); +asm( +".pushsection .text;" +".global __raw_callee_save___kvm_pcpu_is_idle;" +".type __raw_callee_save___kvm_pcpu_is_idle, @function;" +"__raw_callee_save___kvm_pcpu_is_idle:" +"movq __per_cpu_offset(,%rdi,8), %rax;" +"cmpb $0, " __stringify(KVM_STEAL_TIME_is_idle) "+steal_time(%rax);" +"setne %al;" +"ret;" +".size __raw_callee_save___kvm_pcpu_is_idle, .-__raw_callee_save___kvm_pcpu_is_idle;" +".popsection"); #endif /* @@ -970,6 +989,8 @@ void __init kvm_spinlock_init(void) if (kvm_para_has_feature(KVM_FEATURE_STEAL_TIME)) { pv_ops.lock.vcpu_is_preempted = PV_CALLEE_SAVE(__kvm_vcpu_is_preempted); + pv_ops.lock.pcpu_is_idle = + PV_CALLEE_SAVE(__kvm_pcpu_is_idle); } /* * When PV spinlock is enabled which is preferred over diff --git a/arch/x86/kernel/paravirt-spinlocks.c b/arch/x86/kernel/paravirt-spinlocks.c index 9e1ea99ad9df..d7f6a461d0a5 100644 --- a/arch/x86/kernel/paravirt-spinlocks.c +++ b/arch/x86/kernel/paravirt-spinlocks.c @@ -27,12 +27,24 @@ __visible bool __native_vcpu_is_preempted(long cpu) } PV_CALLEE_SAVE_REGS_THUNK(__native_vcpu_is_preempted); +__visible bool __native_pcpu_is_idle(long cpu) +{ + return false; +} +PV_CALLEE_SAVE_REGS_THUNK(__native_pcpu_is_idle); + bool pv_is_native_vcpu_is_preempted(void) { return pv_ops.lock.vcpu_is_preempted.func == __raw_callee_save___native_vcpu_is_preempted; } +bool pv_is_native_pcpu_is_idle(void) +{ + return pv_ops.lock.pcpu_is_idle.func == + __raw_callee_save___native_pcpu_is_idle; +} + void __init paravirt_set_cap(void) { if (!pv_is_native_spin_unlock()) @@ -40,4 +52,7 @@ void __init paravirt_set_cap(void) if (!pv_is_native_vcpu_is_preempted()) setup_force_cpu_cap(X86_FEATURE_VCPUPREEMPT); + + if (!pv_is_native_pcpu_is_idle()) + setup_force_cpu_cap(X86_FEATURE_PCPUISIDLE); } diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index 04cafc057bed..543d89856161 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -366,6 +366,8 @@ struct paravirt_patch_template pv_ops = { .lock.kick = paravirt_nop, .lock.vcpu_is_preempted = PV_CALLEE_SAVE(__native_vcpu_is_preempted), + .lock.pcpu_is_idle = + PV_CALLEE_SAVE(__native_pcpu_is_idle), #endif /* SMP */ #endif }; From patchwork Tue Aug 31 01:59:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tianqiang Xu X-Patchwork-Id: 12466239 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F8F8C432BE for ; Tue, 31 Aug 2021 02:00:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2CD586103E for ; Tue, 31 Aug 2021 02:00:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239325AbhHaCBU (ORCPT ); Mon, 30 Aug 2021 22:01:20 -0400 Received: from smtp181.sjtu.edu.cn ([202.120.2.181]:41048 "EHLO smtp181.sjtu.edu.cn" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239357AbhHaCBU (ORCPT ); Mon, 30 Aug 2021 22:01:20 -0400 Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id 5E22D1008CBCD; Tue, 31 Aug 2021 10:00:23 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id B8CB4228C9242; Tue, 31 Aug 2021 10:00:21 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id va3NV7d66HyK; Tue, 31 Aug 2021 10:00:21 +0800 (CST) Received: from sky.ipads-lab.se.sjtu.edu.cn (unknown [202.120.40.82]) (Authenticated sender: skyele@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id B4C51228C9235; Tue, 31 Aug 2021 09:59:58 +0800 (CST) From: Tianqiang Xu To: x86@kernel.org Cc: pbonzini@redhat.com, seanjc@google.com, vkuznets@redhat.com, wanpengli@tencent.com, jmattson@google.com, joro@8bytes.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, kvm@vger.kernel.org, hpa@zytor.com, jarkko@kernel.org, dave.hansen@linux.intel.com, linux-kernel@vger.kernel.org, linux-sgx@vger.kernel.org, Tianqiang Xu Subject: [PATCH 2/4] Scheduler changes Date: Tue, 31 Aug 2021 09:59:17 +0800 Message-Id: <20210831015919.13006-2-skyele@sjtu.edu.cn> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210831015919.13006-1-skyele@sjtu.edu.cn> References: <20210831015919.13006-1-skyele@sjtu.edu.cn> Precedence: bulk List-ID: X-Mailing-List: linux-sgx@vger.kernel.org available_idle_cpu_sched() invokes pcpu_is_idle() to detect whether a pCPU where the preempted vCPU most recently run is idle or not, which is used by the guest OS. get_cpu_nr_running() gets the run queue of current cpu, which is used by the host KVM to know whether a pCPU where the preempted vCPU mostly recently run is idle or not. --- Authors: Tianqiang Xu, Dingji Li, Zeyu Mi Shanghai Jiao Tong University Signed-off-by: Tianqiang Xu --- include/linux/sched.h | 1 + kernel/sched/core.c | 17 +++++++++++++++++ 2 files changed, 18 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index ec8d07d88641..dd4c41d2d8d3 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1736,6 +1736,7 @@ extern int can_nice(const struct task_struct *p, const int nice); extern int task_curr(const struct task_struct *p); extern int idle_cpu(int cpu); extern int available_idle_cpu(int cpu); +extern int available_idle_cpu_sched(int cpu); extern int sched_setscheduler(struct task_struct *, int, const struct sched_param *); extern int sched_setscheduler_nocheck(struct task_struct *, int, const struct sched_param *); extern void sched_set_fifo(struct task_struct *p); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index f3b27c6c5153..c777dbcbeb9c 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6752,6 +6752,17 @@ int available_idle_cpu(int cpu) return 1; } +int available_idle_cpu_sched(int cpu) +{ + if (!idle_cpu(cpu)) + return 0; + + if (!pcpu_is_idle(cpu)) + return 0; + + return 1; +} + /** * idle_task - return the idle task for a given CPU. * @cpu: the processor in question. @@ -10504,3 +10515,9 @@ void call_trace_sched_update_nr_running(struct rq *rq, int count) { trace_sched_update_nr_running_tp(rq, count); } + +int get_cpu_nr_running(int cpu) +{ + return cpu_rq(cpu)->nr_running; +} +EXPORT_SYMBOL_GPL(get_cpu_nr_running); From patchwork Tue Aug 31 01:59:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tianqiang Xu X-Patchwork-Id: 12466241 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.0 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 088E1C432BE for ; Tue, 31 Aug 2021 02:01:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D928E600AA for ; Tue, 31 Aug 2021 02:01:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239350AbhHaCCD (ORCPT ); Mon, 30 Aug 2021 22:02:03 -0400 Received: from smtp181.sjtu.edu.cn ([202.120.2.181]:41160 "EHLO smtp181.sjtu.edu.cn" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239310AbhHaCCD (ORCPT ); Mon, 30 Aug 2021 22:02:03 -0400 Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id C915E1008B3AD; Tue, 31 Aug 2021 10:01:06 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id 28CC2228C9242; Tue, 31 Aug 2021 10:00:48 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 16Zggdo9Kzq8; Tue, 31 Aug 2021 10:00:47 +0800 (CST) Received: from sky.ipads-lab.se.sjtu.edu.cn (unknown [202.120.40.82]) (Authenticated sender: skyele@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id 0B851228C9235; Tue, 31 Aug 2021 10:00:24 +0800 (CST) From: Tianqiang Xu To: x86@kernel.org Cc: pbonzini@redhat.com, seanjc@google.com, vkuznets@redhat.com, wanpengli@tencent.com, jmattson@google.com, joro@8bytes.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, kvm@vger.kernel.org, hpa@zytor.com, jarkko@kernel.org, dave.hansen@linux.intel.com, linux-kernel@vger.kernel.org, linux-sgx@vger.kernel.org, Tianqiang Xu Subject: [PATCH 3/4] KVM host implementation Date: Tue, 31 Aug 2021 09:59:18 +0800 Message-Id: <20210831015919.13006-3-skyele@sjtu.edu.cn> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210831015919.13006-1-skyele@sjtu.edu.cn> References: <20210831015919.13006-1-skyele@sjtu.edu.cn> Precedence: bulk List-ID: X-Mailing-List: linux-sgx@vger.kernel.org Host OS sets 'is_idle' field of kvm_steal_time to 1 if cpu_rq(this_cpu)->nr_running is 1 before a vCPU being scheduled out. On this condition, there is no other task on this pCPU to run. Thus, is_idle == 1 means the pCPU where the preempted vCPU most recently run is idle. Host OS invokes get_cpu_nr_running() to get the value of cpu_rq(this_cpu)->nr_running. --- Authors: Tianqiang Xu, Dingji Li, Zeyu Mi Shanghai Jiao Tong University Signed-off-by: Tianqiang Xu --- arch/x86/include/asm/qspinlock.h | 1 - arch/x86/kvm/x86.c | 88 +++++++++++++++++++++++++++++++- 2 files changed, 87 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/qspinlock.h b/arch/x86/include/asm/qspinlock.h index c32f2eb6186c..1832dd8308ca 100644 --- a/arch/x86/include/asm/qspinlock.h +++ b/arch/x86/include/asm/qspinlock.h @@ -61,7 +61,6 @@ static inline bool vcpu_is_preempted(long cpu) { return pv_vcpu_is_preempted(cpu); } -#endif #define pcpu_is_idle pcpu_is_idle static inline bool pcpu_is_idle(long cpu) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index e5d5c5ed7dd4..1fb1ab3d6fca 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -3181,6 +3181,72 @@ static void kvm_vcpu_flush_tlb_guest(struct kvm_vcpu *vcpu) static_call(kvm_x86_tlb_flush_guest)(vcpu); } +static void kvm_steal_time_set_is_idle(struct kvm_vcpu *vcpu) +{ + struct kvm_host_map map; + struct kvm_steal_time *st; + + if (!(vcpu->arch.st.msr_val & KVM_MSR_ENABLED)) + return; + + if (vcpu->arch.st.is_idle) + return; + + if (kvm_map_gfn(vcpu, vcpu->arch.st.msr_val >> PAGE_SHIFT, &map, + &vcpu->arch.st.cache, true)) + return; + + st = map.hva + + offset_in_page(vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS); + + st->is_idle = vcpu->arch.st.is_idle = KVM_PCPU_IS_IDLE; + + kvm_unmap_gfn(vcpu, &map, &vcpu->arch.st.cache, true, true); +} + +static void kvm_steal_time_clear_is_idle(struct kvm_vcpu *vcpu) +{ + struct kvm_host_map map; + struct kvm_steal_time *st; + + if (!(vcpu->arch.st.msr_val & KVM_MSR_ENABLED)) + return; + + if (vcpu->arch.st.is_idle) + return; + + if (kvm_map_gfn(vcpu, vcpu->arch.st.msr_val >> PAGE_SHIFT, &map, + &vcpu->arch.st.cache, false)) + return; + + st = map.hva + + offset_in_page(vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS); + + if (guest_pv_has(vcpu, KVM_FEATURE_PV_TLB_FLUSH)) + xchg(&st->is_idle, 0); + else + st->is_idle = 0; + + vcpu->arch.st.is_idle = 0; + + kvm_unmap_gfn(vcpu, &map, &vcpu->arch.st.cache, true, false); +} + + +static DEFINE_PER_CPU(struct kvm_vcpu *, this_cpu_pre_run_vcpu); + +static void vcpu_load_update_pre_vcpu_callback(struct kvm_vcpu *new_vcpu, struct kvm_steal_time *st) +{ + struct kvm_vcpu *old_vcpu = __this_cpu_read(this_cpu_pre_run_vcpu); + + if (!old_vcpu) + return; + if (old_vcpu != new_vcpu) + kvm_steal_time_clear_is_idle(old_vcpu); + else + st->is_idle = new_vcpu->arch.st.is_idle = KVM_PCPU_IS_IDLE; +} + static void record_steal_time(struct kvm_vcpu *vcpu) { struct kvm_host_map map; @@ -3219,6 +3285,8 @@ static void record_steal_time(struct kvm_vcpu *vcpu) vcpu->arch.st.preempted = 0; + vcpu_load_update_pre_vcpu_callback(vcpu, st); + if (st->version & 1) st->version += 1; /* first time write, random junk */ @@ -4290,6 +4358,8 @@ static void kvm_steal_time_set_preempted(struct kvm_vcpu *vcpu) kvm_unmap_gfn(vcpu, &map, &vcpu->arch.st.cache, true, true); } +extern int get_cpu_nr_running(int cpu); + void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) { int idx; @@ -4304,8 +4374,14 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) idx = srcu_read_lock(&vcpu->kvm->srcu); if (kvm_xen_msr_enabled(vcpu->kvm)) kvm_xen_runstate_set_preempted(vcpu); - else + else { kvm_steal_time_set_preempted(vcpu); + + if (get_cpu_nr_running(smp_processor_id()) <= 1) + kvm_steal_time_set_is_idle(vcpu); + else + kvm_steal_time_clear_is_idle(vcpu); + } srcu_read_unlock(&vcpu->kvm->srcu, idx); static_call(kvm_x86_vcpu_put)(vcpu); @@ -9693,6 +9769,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) local_irq_enable(); preempt_enable(); + __this_cpu_write(this_cpu_pre_run_vcpu, vcpu); + vcpu->srcu_idx = srcu_read_lock(&vcpu->kvm->srcu); /* @@ -11253,6 +11331,14 @@ void kvm_arch_pre_destroy_vm(struct kvm *kvm) void kvm_arch_destroy_vm(struct kvm *kvm) { + int cpu; + struct kvm_vcpu *vcpu; + + for_each_possible_cpu(cpu) { + vcpu = per_cpu(this_cpu_pre_run_vcpu, cpu); + if (vcpu && vcpu->kvm == kvm) + per_cpu(this_cpu_pre_run_vcpu, cpu) = NULL; + } + if (current->mm == kvm->mm) { /* * Free memory regions allocated on behalf of userspace, From patchwork Tue Aug 31 01:59:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tianqiang Xu X-Patchwork-Id: 12466243 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13C03C432BE for ; Tue, 31 Aug 2021 02:01:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EAA25600AA for ; Tue, 31 Aug 2021 02:01:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239377AbhHaCCL (ORCPT ); Mon, 30 Aug 2021 22:02:11 -0400 Received: from smtp180.sjtu.edu.cn ([202.120.2.180]:49658 "EHLO smtp180.sjtu.edu.cn" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239310AbhHaCCK (ORCPT ); Mon, 30 Aug 2021 22:02:10 -0400 Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp180.sjtu.edu.cn (Postfix) with ESMTPS id 8F2D5100869B1; Tue, 31 Aug 2021 10:01:13 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id 48F5D228C9242; Tue, 31 Aug 2021 10:01:11 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id XeyUk3hsZDdE; Tue, 31 Aug 2021 10:01:11 +0800 (CST) Received: from sky.ipads-lab.se.sjtu.edu.cn (unknown [202.120.40.82]) (Authenticated sender: skyele@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id D7C062036576A; Tue, 31 Aug 2021 10:00:48 +0800 (CST) From: Tianqiang Xu To: x86@kernel.org Cc: pbonzini@redhat.com, seanjc@google.com, vkuznets@redhat.com, wanpengli@tencent.com, jmattson@google.com, joro@8bytes.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, kvm@vger.kernel.org, hpa@zytor.com, jarkko@kernel.org, dave.hansen@linux.intel.com, linux-kernel@vger.kernel.org, linux-sgx@vger.kernel.org, Tianqiang Xu Subject: [PATCH 4/4] KVM guest implementation Date: Tue, 31 Aug 2021 09:59:19 +0800 Message-Id: <20210831015919.13006-4-skyele@sjtu.edu.cn> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210831015919.13006-1-skyele@sjtu.edu.cn> References: <20210831015919.13006-1-skyele@sjtu.edu.cn> Precedence: bulk List-ID: X-Mailing-List: linux-sgx@vger.kernel.org Guest OS uses 'is_idle' field of kvm_steal_time to know if a pCPU is idle and decides whether to schedule a task to a preempted vCPU or not. If the pCPU is idle, scheduling a task to this pCPU will improve cpu utilization. If not, avoiding scheduling a task to this preempted vCPU can avoid host/guest switch, hence improving performance. Guest OS invokes available_idle_cpu_sched() to get the value of 'is_idle' field of kvm_steal_time. Other modules in kernel except kernel/sched/fair.c which invokes available_idle_cpu() is left unchanged, because other modules in kernel need the semantic provided by 'preempted' field of kvm_steal_time. --- Authors: Tianqiang Xu, Dingji Li, Zeyu Mi Shanghai Jiao Tong University Signed-off-by: Tianqiang Xu --- kernel/sched/fair.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 44c452072a1b..f69f0a8d2abe 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5871,13 +5871,13 @@ wake_affine_idle(int this_cpu, int prev_cpu, int sync) * a cpufreq perspective, it's better to have higher utilisation * on one CPU. */ - if (available_idle_cpu(this_cpu) && cpus_share_cache(this_cpu, prev_cpu)) - return available_idle_cpu(prev_cpu) ? prev_cpu : this_cpu; + if (available_idle_cpu_sched(this_cpu) && cpus_share_cache(this_cpu, prev_cpu)) + return available_idle_cpu_sched(prev_cpu) ? prev_cpu : this_cpu; if (sync && cpu_rq(this_cpu)->nr_running == 1) return this_cpu; - if (available_idle_cpu(prev_cpu)) + if (available_idle_cpu_sched(prev_cpu)) return prev_cpu; return nr_cpumask_bits; @@ -5976,7 +5976,7 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this if (sched_idle_cpu(i)) return i; - if (available_idle_cpu(i)) { + if (available_idle_cpu_sched(i)) { struct cpuidle_state *idle = idle_get_state(rq); if (idle && idle->exit_latency < min_exit_latency) { /* @@ -6064,7 +6064,7 @@ static inline int find_idlest_cpu(struct sched_domain *sd, struct task_struct *p static inline int __select_idle_cpu(int cpu, struct task_struct *p) { - if ((available_idle_cpu(cpu) || sched_idle_cpu(cpu)) && + if ((available_idle_cpu_sched(cpu) || sched_idle_cpu(cpu)) && sched_cpu_cookie_match(cpu_rq(cpu), p)) return cpu; @@ -6115,7 +6115,7 @@ void __update_idle_core(struct rq *rq) if (cpu == core) continue; - if (!available_idle_cpu(cpu)) + if (!available_idle_cpu_sched(cpu)) goto unlock; } @@ -6138,7 +6138,7 @@ static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpu return __select_idle_cpu(core, p); for_each_cpu(cpu, cpu_smt_mask(core)) { - if (!available_idle_cpu(cpu)) { + if (!available_idle_cpu_sched(cpu)) { idle = false; if (*idle_cpu == -1) { if (sched_idle_cpu(cpu) && cpumask_test_cpu(cpu, p->cpus_ptr)) { @@ -6171,7 +6171,7 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t if (!cpumask_test_cpu(cpu, p->cpus_ptr) || !cpumask_test_cpu(cpu, sched_domain_span(sd))) continue; - if (available_idle_cpu(cpu) || sched_idle_cpu(cpu)) + if (available_idle_cpu_sched(cpu) || sched_idle_cpu(cpu)) return cpu; } @@ -6302,7 +6302,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) for_each_cpu_wrap(cpu, cpus, target) { unsigned long cpu_cap = capacity_of(cpu); - if (!available_idle_cpu(cpu) && !sched_idle_cpu(cpu)) + if (!available_idle_cpu_sched(cpu) && !sched_idle_cpu(cpu)) continue; if (fits_capacity(task_util, cpu_cap)) return cpu; @@ -6348,7 +6348,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) */ lockdep_assert_irqs_disabled(); - if ((available_idle_cpu(target) || sched_idle_cpu(target)) && + if ((available_idle_cpu_sched(target) || sched_idle_cpu(target)) && asym_fits_capacity(task_util, target)) return target; @@ -6356,7 +6356,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) * If the previous CPU is cache affine and idle, don't be stupid: */ if (prev != target && cpus_share_cache(prev, target) && - (available_idle_cpu(prev) || sched_idle_cpu(prev)) && + (available_idle_cpu_sched(prev) || sched_idle_cpu(prev)) && asym_fits_capacity(task_util, prev)) return prev; @@ -6379,7 +6379,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) if (recent_used_cpu != prev && recent_used_cpu != target && cpus_share_cache(recent_used_cpu, target) && - (available_idle_cpu(recent_used_cpu) || sched_idle_cpu(recent_used_cpu)) && + (available_idle_cpu_sched(recent_used_cpu) || sched_idle_cpu(recent_used_cpu)) && cpumask_test_cpu(p->recent_used_cpu, p->cpus_ptr) && asym_fits_capacity(task_util, recent_used_cpu)) { /*