[v2,3/3] docs: RTDS feature document.

Commit Message

Dario Faggioli Oct. 14, 2016, 10:02 a.m. UTC

Signed-off-by: Dario Faggioli <[e-mail]>
---
Cc: Meng Xu <[e-mail]>
Cc: George Dunlap <[e-mail]>
Cc: Wei Liu <[e-mail]>
Cc: Lars Kurth <[e-mail]>
Cc: Andrew Cooper <[e-mail]>
Cc: Ian Jackson <[e-mail]>
Cc: Jan Beulich <[e-mail]>
Cc: Konrad Rzeszutek Wilk <[e-mail]>
Cc: Stefano Stabellini <[e-mail]>
---
Changes from v1:
* file renamed from rtds.pandoc to sched_rtds.pandoc;
* feature name is now 'RTDS Scheduler';
* kill the 'e.g.'-s in the header (sorry!);
* removed 'Architecture' line from header, as this is arch independent;
* removed some text that was duplicated in other sched feature documents;
  it's redundant wrt what's in the wiki about schedulers, and the page
  is referenced.
---
 docs/features/sched_rtds.pandoc |  121 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 121 insertions(+)
 create mode 100644 docs/features/sched_rtds.pandoc

Comments

Meng Xu Oct. 18, 2016, 5:49 p.m. UTC | #1

Hi Dario,

Thank you very much for writing this nice document!
Sorry for the late reply...

I found two typos in the command example.
I saw the patch has been committed. Maybe I (or you) should send a
patch to correct the typo?
>
>
> +
> +For checking or changing a VM's scheduling parameters from xl, do
> +as follows:
> +    * `xl sched-rtds -d vm-rt`


It should be
xl sched-rtds -d vm-rt -v all


>
> +    * `xl sched-rtds -d vm-rt -t 10000 -b 25000`


There are three typos:
1) We don't support "-t" option.
2) We must specify -v in the latest Xen. (Once the per-vcpu parameter
is supported, we won't support setting a VM's parameter like this. The
toolstack will reports error.)
3) The budget 25000 is larger than the period.

It should be:
sudo xl sched-rtds -d vm-rt -v all -p 10000 -b 2500



>
> +
> +It is possible, for a multiple vCPUs VM, to change the parameters of
> +each vCPU individually:
> +    * `xl sched-rtds -d vm-rt -v 0 -p 20000 -b 10000 -v 1 -p 45000 -b 12000`


This is correct.

>
> +
> +# Technical details
> +
> +Implementation entirely lives in the hypervisor. Xen has a pluggable,
> +hook based, architecture for schedulers. Thanks to this, RTDS code
> +is all contained in `xen/common/sched_rtds.c`.
> +
> +In libxl, the availability of the RTDS scheduler is advertised by
> +the presence of the LIBXL\_HAVE\_SCHED\_RTDS symbol. The ability of
> +specifying different scheduling parameters for each vcpu has been
> +introduced later, and is available if the following symbols are defined:
> +    * `LIBXL\_HAVE\_VCPU\_SCHED\_PARAMS`,
> +    * `LIBXL\_HAVE\_SCHED\_RTDS\_VCPU\_PARAMS`.
> +
> +# Limitations
> +
> +RTDS is a special purpose scheduling. This is by design, and not at
> +all a limitation, but it is certainly something to keep in mind when
> +thinking about using it. The purpose of the scheduler is enabling
> +deterministic and statically analyzable behavior (as per the
> +real-time academic literature), according to the scheduling parameters
> +assigned to each vCPU.
> +
> +Using RTDS a the Xen scheduler, and/or for general purpose workloads
> +is definitely possible, but the vCPU scheduling parameters (of both
> +Domain0 and of the various VMs) would probably require tweaking, with
> +respect to their default values.
> +
> +# Testing
> +
> +Any change done in RTDS must be tested by doing the following:
> +
> +* create a cpupool with RTDS as its scheduler,
> +* create a few virtual machines a move them in and out of the pool,
> +* create a few virtual machines, directly inside the pool, and verify
> +  that they boot and can run some basic workload (e.g., login into them
> +  and run simple commands),
> +* shutdown/reboot the virtual machines,
> +
> +The fact that the system boots fine when passing `sched=rtds` to Xen
> +should also be verified.
> +
> +Finally, to check that the scheduler is working properly (although only
> +at a macroscopic level), the following should be done:
> +
> +* create a VM with 1 vCPU and put it in the RTDS cpupool,
> +* set the scheduling parameters such as it has a 50% reservation, with
> +  `xl sched-rtds -d vm -t 100000 -b 50000`,


-t here should be -p.
The -v all should be specified here.

Thanks,

Meng


-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

Wei Liu Oct. 18, 2016, 5:55 p.m. UTC | #2

Hi Meng,

I've committed this patch. Feel free to send update when you and
Dario have all the details hashed out.

Wei.

Dario Faggioli Oct. 18, 2016, 9:53 p.m. UTC | #3

[Trimming the Cc-list]

On Tue, 2016-10-18 at 13:49 -0400, Meng Xu wrote:
> > +For checking or changing a VM's scheduling parameters from xl, do
> > +as follows:
> > +    * `xl sched-rtds -d vm-rt`
> 
> 
> It should be
> xl sched-rtds -d vm-rt -v all
> 
Yeah, you're right.

I wrote this basing on my memory, then went there and check, but I
apparently have forgotten to update all the examples.

All you say in this mail makes sense. So, yes, do put together a patch
and send it here. :-)

Thanks and Regards,
Dario

Patch

diff --git a/docs/features/sched_rtds.pandoc b/docs/features/sched_rtds.pandoc
new file mode 100644
index 0000000..2843d97
--- /dev/null
+++ b/docs/features/sched_rtds.pandoc
@@ -0,0 +1,121 @@ 
+% RTDS Scheduler
+% Revision 1
+
+\clearpage
+
+# Basics
+---------------- ----------------------------------------------------
+         Status: **Experimental**
+
+      Component: Hypervisor
+---------------- ----------------------------------------------------
+
+# Overview
+
+RTDS is one of the virtual CPU (vCPU) scheduler available in the Xen
+hypervisor.
+
+RTDS is a real--time scheduler, so its purpose is enabling
+**deterministic** scheduling of the virtual machine's vCPUs. It has
+been originally developed in the context of the RT-Xen project.
+
+# User details
+
+RTDS is not in use by default. In order to use it as the Xen scheduler
+the following parameter should be passed to the hypervisor at boot:
+
+    `sched=rtds`
+
+Once the system is live, for creating a cpupool with RTDS as its
+scheduler, either compile a cpupool configuration file, as described
+in `docs/man/xlcpupool.cfg.pod.5` (and as exemplified in
+`tools/examples/cpupool`), or use just `xl` directly:
+
+    xl cpupool-create name=\"pool-rt\" sched=\"rtds\" cpus=[4,5,6,8]
+
+For checking or changing a VM's scheduling parameters from xl, do
+as follows:
+    * `xl sched-rtds -d vm-rt`
+    * `xl sched-rtds -d vm-rt -t 10000 -b 25000`
+
+It is possible, for a multiple vCPUs VM, to change the parameters of
+each vCPU individually:
+    * `xl sched-rtds -d vm-rt -v 0 -p 20000 -b 10000 -v 1 -p 45000 -b 12000`
+
+# Technical details
+
+Implementation entirely lives in the hypervisor. Xen has a pluggable,
+hook based, architecture for schedulers. Thanks to this, RTDS code
+is all contained in `xen/common/sched_rtds.c`.
+
+In libxl, the availability of the RTDS scheduler is advertised by
+the presence of the LIBXL\_HAVE\_SCHED\_RTDS symbol. The ability of
+specifying different scheduling parameters for each vcpu has been
+introduced later, and is available if the following symbols are defined:
+    * `LIBXL\_HAVE\_VCPU\_SCHED\_PARAMS`,
+    * `LIBXL\_HAVE\_SCHED\_RTDS\_VCPU\_PARAMS`.
+
+# Limitations
+
+RTDS is a special purpose scheduling. This is by design, and not at
+all a limitation, but it is certainly something to keep in mind when
+thinking about using it. The purpose of the scheduler is enabling
+deterministic and statically analyzable behavior (as per the
+real-time academic literature), according to the scheduling parameters
+assigned to each vCPU.
+
+Using RTDS a the Xen scheduler, and/or for general purpose workloads
+is definitely possible, but the vCPU scheduling parameters (of both
+Domain0 and of the various VMs) would probably require tweaking, with
+respect to their default values.
+
+# Testing
+
+Any change done in RTDS must be tested by doing the following:
+
+* create a cpupool with RTDS as its scheduler,
+* create a few virtual machines a move them in and out of the pool,
+* create a few virtual machines, directly inside the pool, and verify
+  that they boot and can run some basic workload (e.g., login into them
+  and run simple commands),
+* shutdown/reboot the virtual machines,
+
+The fact that the system boots fine when passing `sched=rtds` to Xen
+should also be verified.
+
+Finally, to check that the scheduler is working properly (although only
+at a macroscopic level), the following should be done:
+
+* create a VM with 1 vCPU and put it in the RTDS cpupool,
+* set the scheduling parameters such as it has a 50% reservation, with
+  `xl sched-rtds -d vm -t 100000 -b 50000`,
+* run a CPU-burning process inside the VM (e.g., `yes`),
+* check with `xentop` (in Domain0) that the VM is getting no more than
+  50% pCPU time.
+
+# Areas for improvement
+
+* Work-conserving mode to be added;
+* performance assessment, especially focusing on what level of real-time
+  behavior the scheduler enables.
+
+# Known issues
+
+* OSSTest reports occasional failures on ARM.
+
+# References
+
+* "RT-Xen: Real-Time Virtualization" [XPDS14 Presentation](http://events.linuxfoundation.org/sites/events/files/slides/2014_Xen_Developer_Summit_0.pdf)
+* "Scheduling in Xen" [XPDS15 Presentation](http://events.linuxfoundation.org/sites/events/files/slides/Faggioli_XenSummit.pdf)
+* [RT-Xen Project](https://sites.google.com/site/realtimexen/)
+* [RTDS-Based-Scheduler](https://wiki.xenproject.org/wiki/RTDS-Based-Scheduler)
+* "The RTDS Scheduler" [on the Xen-Project wiki](https://wiki.xenproject.org/wiki/RTDS-Based-Scheduler)
+* "Xen Project Schedulers" [on the Xen-Project wiki](https://wiki.xenproject.org/wiki/Xen_Project_Schedulers)
+
+# History
+
+------------------------------------------------------------------------
+Date       Revision Version  Notes
+---------- -------- -------- -------------------------------------------
[phone]        Xen 4.8  Document written
+---------- -------- -------- -------------------------------------------