[v5,00/19] xen: add core scheduling support

Message

Jürgen Groß Sept. 30, 2019, 5:21 a.m. UTC

Add support for core- and socket-scheduling in the Xen hypervisor.

Via boot parameter sched-gran=core (or sched-gran=socket)
it is possible to change the scheduling granularity from cpu (the
default) to either whole cores or even sockets.

All logical cpus (threads) of the core or socket are always scheduled
together. This means that on a core always vcpus of the same domain
will be active, and those vcpus will always be scheduled at the same
time.

This is achieved by switching the scheduler to no longer see vcpus as
the primary object to schedule, but "schedule units". Each schedule
unit consists of as many vcpus as each core has threads on the current
system. The vcpu->unit relation is fixed.

I have done some very basic performance testing: on a 4 cpu system
(2 cores with 2 threads each) I did a "make -j 4" for building the Xen
hypervisor. With This test has been run on dom0, once with no other
guest active and once with another guest with 4 vcpus running the same
test. The results are (always elapsed time, system time, user time):

sched-gran=cpu,    no other guest: 116.10 177.65 207.84
sched-gran=core,   no other guest: 114.04 175.47 207.45
sched-gran=cpu,    other guest:    202.30 334.21 384.63
sched-gran=core,   other guest:    207.24 293.04 371.37

The performance tests have been performed with credit2, the other
schedulers are tested only briefly to be able to create a domain in a
cpupool.

Cpupools have been moderately tested (cpu add/remove, create, destroy,
move domain).

Cpu on-/offlining has been moderately tested, too.

The series is based on:
"xen/sched: rework and rename vcpu_force_reschedule()"
which has been split off from V2 and:
"xen/sched: fix locking in a653sched_free_vdata()"
which is fixing a problem detected via review of V3.

The complete patch series (plus prereq patches and some debugging
additions in form of additional patches) is available under:

  git://github.com/jgross1/xen/ sched-v5

Changes in V5:
- dropped patches 1-27 as they already went in
- added comments in 2 patches

Changes in V4:
- comments addressed
- former patch 36 merged into patch 32

Changes in V3:
- comments addressed
- former patch 26 carved out and sent separately
- some minor bugs fixed

Changes in V2:
- comments addressed
- some patches merged into one
- idle scheduler related patches split off to own series
- some patches are already applied
- some bugs fixed (e.g. crashes when powering off)

Changes in V1:
- cpupools are working now
- cpu on-/offlining working now
- all schedulers working now
- renamed "items" to "units"
- introduction of "idle scheduler"
- several new patches (see individual patches, mostly splits of
  former patches or cpupool and cpu on-/offlining support)
- all review comments addressed
- some minor changes (see individual patches)

Changes in RFC V2:
- ARM is building now
- HVM domains are working now
- idling will always be done with idle_vcpu active
- other small changes see individual patches

Juergen Gross (19):
  xen/sched: add code to sync scheduling of all vcpus of a sched unit
  xen/sched: introduce unit_runnable_state()
  xen/sched: add support for multiple vcpus per sched unit where missing
  xen/sched: modify cpupool_domain_cpumask() to be an unit mask
  xen/sched: support allocating multiple vcpus into one sched unit
  xen/sched: add a percpu resource index
  xen/sched: add fall back to idle vcpu when scheduling unit
  xen/sched: make vcpu_wake() and vcpu_sleep() core scheduling aware
  xen/sched: move per-cpu variable scheduler to struct sched_resource
  xen/sched: move per-cpu variable cpupool to struct sched_resource
  xen/sched: reject switching smt on/off with core scheduling active
  xen/sched: prepare per-cpupool scheduling granularity
  xen/sched: split schedule_cpu_switch()
  xen/sched: protect scheduling resource via rcu
  xen/sched: support multiple cpus per scheduling resource
  xen/sched: support differing granularity in schedule_cpu_[add/rm]()
  xen/sched: support core scheduling for moving cpus to/from cpupools
  xen/sched: disable scheduling when entering ACPI deep sleep states
  xen/sched: add scheduling granularity enum

 xen/arch/arm/domain.c         |    2 +-
 xen/arch/x86/Kconfig          |    1 +
 xen/arch/x86/acpi/power.c     |    4 +
 xen/arch/x86/domain.c         |   26 +-
 xen/arch/x86/sysctl.c         |    5 +
 xen/common/Kconfig            |    3 +
 xen/common/cpupool.c          |  232 ++++++--
 xen/common/domain.c           |    8 +-
 xen/common/domctl.c           |    2 +-
 xen/common/sched_arinc653.c   |    4 +-
 xen/common/sched_credit.c     |   73 +--
 xen/common/sched_credit2.c    |   32 +-
 xen/common/sched_null.c       |   11 +-
 xen/common/sched_rt.c         |   18 +-
 xen/common/schedule.c         | 1300 +++++++++++++++++++++++++++++++++--------
 xen/common/softirq.c          |    6 +-
 xen/include/asm-arm/current.h |    1 +
 xen/include/asm-x86/current.h |   19 +-
 xen/include/asm-x86/smp.h     |    7 +
 xen/include/xen/sched-if.h    |   86 ++-
 xen/include/xen/sched.h       |   26 +-
 xen/include/xen/softirq.h     |    1 +
 22 files changed, 1504 insertions(+), 363 deletions(-)