| Message ID | 20190809145833.1020-1-jgross@suse.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | xen: add core scheduling support | expand |
Hi Juergen, The latest round of testing revealed the following 3 Xen crashes: 1. vcpu_sleep_sync() <-- vlapic_init_sipi_action() This was seen multiple times. It tends to happen on large Windows Server VMs (>= 12 vCPUs). https://paste.debian.net/1095844/ 2. vcpu_sleep_sync() <-- vcpu_set_periodic_timer() <-- pv_hypercall() This was seen on Sles 11.4 PV VM with 32 vCPUs and 6GB RAM https://paste.debian.net/1095843/ 3. ACPI S5: schedule.c#sched_slave() This one was during shutdown again. https://paste.debian.net/1095845/ There also were a number of Dom0 soft-lockups (together with RCU stalls) but unfortunately our crashdump tool was a bit broken so full Dom0 kernel logs were not captured this time. And CentOS 5.11 suspend regression (as reported previously) is still present. Thanks, Sergey
Hi Sergey, On 15.08.19 12:17, Sergey Dyasli wrote: > Hi Juergen, > > The latest round of testing revealed the following 3 Xen crashes: > > 1. vcpu_sleep_sync() <-- vlapic_init_sipi_action() > This was seen multiple times. It tends to happen on large Windows Server > VMs (>= 12 vCPUs). > https://paste.debian.net/1095844/ > > 2. vcpu_sleep_sync() <-- vcpu_set_periodic_timer() <-- pv_hypercall() > This was seen on Sles 11.4 PV VM with 32 vCPUs and 6GB RAM > https://paste.debian.net/1095843/ > > 3. ACPI S5: schedule.c#sched_slave() > This one was during shutdown again. > https://paste.debian.net/1095845/ > > There also were a number of Dom0 soft-lockups (together with RCU stalls) > but unfortunately our crashdump tool was a bit broken so full Dom0 kernel > logs were not captured this time. > > And CentOS 5.11 suspend regression (as reported previously) is still > present. I have found one bug in the v->is_running setting, which might be related to your observed hangs, but I couldn't verify that. I have rebased my series with the is_running fix to current staging. You can find that under git://github.com/jgross1/xen/ sched-v2b-2019-09-05 This branch includes the needed prerequisite series ("xen/sched: use new idle scheduler for free cpus"), and some additional debug enhancements I'd like you to use: - my series for debugtrace enhancements - my series for lock-profiling enhancements - and as the last patch some debugtrace hooks in the scheduler Please make sure to have CONFIG_DEBUG_TRACE and CONFIG_DEBUG_LOCK_PROFILE set when building the hypervisor. When testing please add: async-show-all=on watchdog=on debugtrace=cpu:16 to the hypervisor boot parameters. This should add a lot of additional output in case of a crash, which will (hopefully) help to find the reason(s) for hangs and crashes. Juergen
Add support for core- and socket-scheduling in the Xen hypervisor. Via boot parameter sched-gran=core (or sched-gran=socket) it is possible to change the scheduling granularity from cpu (the default) to either whole cores or even sockets. All logical cpus (threads) of the core or socket are always scheduled together. This means that on a core always vcpus of the same domain will be active, and those vcpus will always be scheduled at the same time. This is achieved by switching the scheduler to no longer see vcpus as the primary object to schedule, but "schedule units". Each schedule unit consists of as many vcpus as each core has threads on the current system. The vcpu->unit relation is fixed. I have done some very basic performance testing: on a 4 cpu system (2 cores with 2 threads each) I did a "make -j 4" for building the Xen hypervisor. With This test has been run on dom0, once with no other guest active and once with another guest with 4 vcpus running the same test. The results are (always elapsed time, system time, user time): sched-gran=cpu, no other guest: 116.10 177.65 207.84 sched-gran=core, no other guest: 114.04 175.47 207.45 sched-gran=cpu, other guest: 202.30 334.21 384.63 sched-gran=core, other guest: 207.24 293.04 371.37 The performance tests have been performed with credit2, the other schedulers are tested only briefly to be able to create a domain in a cpupool. Cpupools have been moderately tested (cpu add/remove, create, destroy, move domain). Cpu on-/offlining has been moderately tested, too. The series is based on the series: "xen/sched: use new idle scheduler for free cpus" which has been split off from V1. The complete patch series is available under: git://github.com/jgross1/xen/ sched-v2 Changes in V2: - comments addressed - some patches merged into one - idle scheduler related patches split off to own series - some patches are already applied - some bugs fixed (e.g. crashes when powering off) Changes in V1: - cpupools are working now - cpu on-/offlining working now - all schedulers working now - renamed "items" to "units" - introduction of "idle scheduler" - several new patches (see individual patches, mostly splits of former patches or cpupool and cpu on-/offlining support) - all review comments addressed - some minor changes (see individual patches) Changes in RFC V2: - ARM is building now - HVM domains are working now - idling will always be done with idle_vcpu active - other small changes see individual patches Juergen Gross (48): xen/sched: use new sched_unit instead of vcpu in scheduler interfaces xen/sched: move per-vcpu scheduler private data pointer to sched_unit xen/sched: build a linked list of struct sched_unit xen/sched: introduce struct sched_resource xen/sched: let pick_cpu return a scheduler resource xen/sched: switch schedule_data.curr to point at sched_unit xen/sched: move per cpu scheduler private data into struct sched_resource xen/sched: switch vcpu_schedule_lock to unit_schedule_lock xen/sched: move some per-vcpu items to struct sched_unit xen/sched: add scheduler helpers hiding vcpu xen/sched: rename scheduler related perf counters xen/sched: switch struct task_slice from vcpu to sched_unit xen/sched: add is_running indicator to struct sched_unit xen/sched: make null scheduler vcpu agnostic. xen/sched: make rt scheduler vcpu agnostic. xen/sched: make credit scheduler vcpu agnostic. xen/sched: make credit2 scheduler vcpu agnostic. xen/sched: make arinc653 scheduler vcpu agnostic. xen: add sched_unit_pause_nosync() and sched_unit_unpause() xen: let vcpu_create() select processor xen/sched: use sched_resource cpu instead smp_processor_id in schedulers xen/sched: switch schedule() from vcpus to sched_units xen/sched: switch sched_move_irqs() to take sched_unit as parameter xen: switch from for_each_vcpu() to for_each_sched_unit() xen/sched: add runstate counters to struct sched_unit xen/sched: rework and rename vcpu_force_reschedule() xen/sched: Change vcpu_migrate_*() to operate on schedule unit xen/sched: move struct task_slice into struct sched_unit xen/sched: add code to sync scheduling of all vcpus of a sched unit xen/sched: introduce unit_runnable_state() xen/sched: add support for multiple vcpus per sched unit where missing xen/sched: modify cpupool_domain_cpumask() to be an unit mask xen/sched: support allocating multiple vcpus into one sched unit xen/sched: add a percpu resource index xen/sched: add fall back to idle vcpu when scheduling unit xen/sched: make vcpu_wake() and vcpu_sleep() core scheduling aware xen/sched: carve out freeing sched_unit memory into dedicated function xen/sched: move per-cpu variable scheduler to struct sched_resource xen/sched: move per-cpu variable cpupool to struct sched_resource xen/sched: reject switching smt on/off with core scheduling active xen/sched: prepare per-cpupool scheduling granularity xen/sched: split schedule_cpu_switch() xen/sched: protect scheduling resource via rcu xen/sched: support multiple cpus per scheduling resource xen/sched: support differing granularity in schedule_cpu_[add/rm]() xen/sched: support core scheduling for moving cpus to/from cpupools xen/sched: disable scheduling when entering ACPI deep sleep states xen/sched: add scheduling granularity enum xen/arch/arm/domain.c | 2 +- xen/arch/arm/domain_build.c | 13 +- xen/arch/x86/Kconfig | 1 + xen/arch/x86/acpi/cpu_idle.c | 1 - xen/arch/x86/acpi/power.c | 4 + xen/arch/x86/cpu/mcheck/mce.c | 1 - xen/arch/x86/cpu/mcheck/mctelem.c | 1 - xen/arch/x86/dom0_build.c | 10 +- xen/arch/x86/domain.c | 26 +- xen/arch/x86/hvm/dom0_build.c | 9 +- xen/arch/x86/pv/dom0_build.c | 10 +- xen/arch/x86/pv/shim.c | 4 +- xen/arch/x86/setup.c | 1 - xen/arch/x86/smpboot.c | 1 - xen/arch/x86/sysctl.c | 5 + xen/common/Kconfig | 3 + xen/common/cpupool.c | 157 +++- xen/common/domain.c | 35 +- xen/common/domctl.c | 23 +- xen/common/keyhandler.c | 58 +- xen/common/sched_arinc653.c | 256 +++-- xen/common/sched_credit.c | 760 +++++++-------- xen/common/sched_credit2.c | 1119 +++++++++++----------- xen/common/sched_null.c | 466 +++++----- xen/common/sched_rt.c | 543 +++++------ xen/common/schedule.c | 1851 +++++++++++++++++++++++++++++-------- xen/common/softirq.c | 6 +- xen/include/asm-arm/current.h | 1 + xen/include/asm-x86/cpuidle.h | 11 - xen/include/asm-x86/current.h | 19 +- xen/include/asm-x86/dom0_build.h | 3 +- xen/include/asm-x86/smp.h | 3 + xen/include/xen/domain.h | 3 +- xen/include/xen/perfc_defn.h | 32 +- xen/include/xen/sched-if.h | 340 +++++-- xen/include/xen/sched.h | 93 +- xen/include/xen/softirq.h | 1 + 37 files changed, 3675 insertions(+), 2197 deletions(-)