From patchwork Thu Oct 13 11:04:08 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dario Faggioli X-Patchwork-Id: 9374795 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D93196075E for ; Thu, 13 Oct 2016 11:06:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CA16029FD3 for ; Thu, 13 Oct 2016 11:06:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BE71729FD6; Thu, 13 Oct 2016 11:06:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,RCVD_IN_SORBS_SPAM,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id CC46029FD3 for ; Thu, 13 Oct 2016 11:06:45 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1budoX-0002xj-Dk; Thu, 13 Oct 2016 11:04:41 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1budoW-0002xI-4N for xen-devel@lists.xenproject.org; Thu, 13 Oct 2016 11:04:40 +0000 Received: from [85.158.137.68] by server-12.bemta-3.messagelabs.com id B1/B2-04658-74A6FF75; Thu, 13 Oct 2016 11:04:39 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrJIsWRWlGSWpSXmKPExsVyMfS6i65b1v9 wg9bJwhbft0xmcmD0OPzhCksAYxRrZl5SfkUCa8bend3sBUd0KvYu+MzUwNiq1sXIxSEkMINR 4s/cK+wgDovAGlaJi6uvgTkSApdYJWYc+snYxcgJ5MRIrOybzgZhl0m86nvGCmILCahI3Ny+i gli1HwmiY1PljCBJIQF9CSOHP0BNIkDyDaU6P0rDBJmEzCQeLNjL1iviICSxL1Vk8F6mQVeMU m8aV/CDpJgEVCV6DlyhwXE5hVwlTg6ZSdYnFPATWLH059MEItdJTatXAV2kKiAnMTKyy2sEPW CEidnPmEB2cssoCmxfpc+SJhZQF5i+9s5zBMYRWYhqZqFUDULSdUCRuZVjBrFqUVlqUW6RqZ6 SUWZ6RkluYmZObqGBsZ6uanFxYnpqTmJScV6yfm5mxiBEVDPwMC4g7H1hN8hRkkOJiVR3l9+/ 8OF+JLyUyozEosz4otKc1KLDzHKcHAoSfAeygDKCRalpqdWpGXmAGMRJi3BwaMkwmuRCZTmLS 5IzC3OTIdInWI05tjy+9paJo5tU++tZRJiycvPS5US57UEKRUAKc0ozYMbBEsRlxhlpYR5GRk YGIR4ClKLcjNLUOVfMYpzMCoJ80qATOHJzCuB2/cK6BQmoFNsJv0BOaUkESEl1cCYo3xg5iF7 /oe7Jsr7eIr2R4aGFL8Lc67un/YxZ6Gt1ZxpfKVty9iz7QWSWo9Nuq1/0dJMgtE5//WjCAeng GO87AE7X+3T5/tprTXDPdNNpnfJqXtnig/zCRZHv61s8M2N9K9Qmxyb9FcoaNK8C8u4L4W2XN h/v+mCrcRH7QkFt+yjP8vHSimxFGckGmoxFxUnAgAWODg2DAMAAA== X-Env-Sender: raistlin.df@gmail.com X-Msg-Ref: server-11.tower-31.messagelabs.com!1476356677!35057203!1 X-Originating-IP: [209.85.215.68] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 8.84; banners=-,-,- X-VirusChecked: Checked Received: (qmail 53807 invoked from network); 13 Oct 2016 11:04:38 -0000 Received: from mail-lf0-f68.google.com (HELO mail-lf0-f68.google.com) (209.85.215.68) by server-11.tower-31.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 13 Oct 2016 11:04:38 -0000 Received: by mail-lf0-f68.google.com with SMTP id b75so11955001lfg.3 for ; Thu, 13 Oct 2016 04:04:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=frVEPcRYy0/Gf4M3T3TRA5Rj2OjxSOA3Lbj73pKf6T8=; b=dB5GtQsgLQIjdBuyy1oUbnr6IScYJ46TFXAt7uHkfi5lWslsjzy6W+3ncqL9RM6MCA RS7zvrnaLcDvxU9Fpb64iQ9d1fq5+pS9PgeuHpvbX0rSvPq3kXiLsKAJl1PbOuf8Giid s699NQ3kJxmB9Wa0dR57N0Uk6w4ZfiV7kVwxvdI/8YT1yY5YZN9UI5vbDJY3kAFsaBN7 VAnC0GyEr1E46pVXwXNq5BZq4bd40aZN1ArfWP5i1iF4PhIJ6LzeXQgjOOTi4OZ+32F3 YRp9qmZjCYCi2IAwrCCpX6+HP721adWlt8XAwZ9Eaf0giyUYpDGMuIvVbhbxiiGeozxP dklQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=frVEPcRYy0/Gf4M3T3TRA5Rj2OjxSOA3Lbj73pKf6T8=; b=Sn/5fI+w/DarLGPcVNOliC5GcBxW8je0fBdFCnh/x8nvZzkjWw7WL9p9xEQM82aRm/ Q9pI07Y5YfD9xlTVl4NzNPpp9+i9v1c9cTRU1K5gBnF2ydqxPKdFvKtMAwFrtQTXBDDs 4odjLjnQUbBVSi++M6km3j9NreAIpyqKLpCskFbZwYzTt+snuJOnpSqR9YTZvY0j0xhl eP1SQmCalbzxqLKE6WlL6pHX2CdH4Yay5IoIn17yUwccyTXYDOjJkOJje1L7NxQDXUwn WwLhVEZ0mjFBR4yyj+MP68JA1H4mY4oY+vcMBt8KpTouga0ZrzzZQ8R0hWcqq2ZA/ziR 2MGA== X-Gm-Message-State: AA6/9Rl+nadDsCnx0lBh2v0Zyaecra/WWzprcTUPQQcf2TTuCduzsQM+osBIP/tA7zFH2w== X-Received: by 10.28.102.9 with SMTP id a9mr1769427wmc.10.1476356677509; Thu, 13 Oct 2016 04:04:37 -0700 (PDT) Received: from [10.80.3.155] (default-46-102-197-194.interdsl.co.uk. [46.102.197.194]) by smtp.gmail.com with ESMTPSA id e2sm21608107wjw.14.2016.10.13.04.04.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Oct 2016 04:04:33 -0700 (PDT) From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Thu, 13 Oct 2016 12:04:08 +0100 Message-ID: <147635663645.17095.14072329395173107167.stgit@Palanthas> In-Reply-To: <147635565298.17095.17316867931713209620.stgit@Palanthas> References: <147635565298.17095.17316867931713209620.stgit@Palanthas> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Cc: Lars Kurth , Stefano Stabellini , Wei Liu , George Dunlap , Andrew Cooper , Ian Jackson , Meng Xu , Jan Beulich Subject: [Xen-devel] [PATCH 3/3] docs: RTDS feature document. X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Dario Faggioli --- Cc: Meng Xu Cc: George Dunlap Cc: Wei Liu Cc: Lars Kurth Cc: Andrew Cooper Cc: Ian Jackson Cc: Jan Beulich Cc: Konrad Rzeszutek Wilk Cc: Stefano Stabellini --- docs/features/rtds.pandoc | 125 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 125 insertions(+) create mode 100644 docs/features/rtds.pandoc diff --git a/docs/features/rtds.pandoc b/docs/features/rtds.pandoc new file mode 100644 index 0000000..76aa165 --- /dev/null +++ b/docs/features/rtds.pandoc @@ -0,0 +1,125 @@ +% RTDS +% Revision 1 + +\clearpage + +# Basics +---------------- ---------------------------------------------------- + Status: e.g. **Experimental** + +Architecture(s): e.g. x86, arm + + Component: e.g. Hypervisor +---------------- ---------------------------------------------------- + +# Overview + +RTDS is one of the virtual CPU (vCPU) scheduler available in the Xen +hypervisor. The job of an hypervisor's virtual CPU scheduler is to +decide, among all the various vCPUs of the various virtual machines, +which ones should execute on the host's physical CPUs (pCPUs), at any +given point in time. + +RTDS is a real--time scheduler, so its purpose is enabling +**deterministic** scheduling of the virtual machine's vCPUs. It has +been originally developed in the context of the RT-Xen project. + +# User details + +RTDS is not in use by default. In order to use it as the Xen scheduler +the following parameter should be passed to the hypervisor at boot: + + `sched=rtds` + +Once the system is live, for creating a cpupool with RTDS as its +scheduler, either compile a cpupool configuration file, as described +in `docs/man/xlcpupool.cfg.pod.5` (and as exemplified in +`tools/examples/cpupool`), or use just `xl` directly: + + xl cpupool-create name=\"pool-rt\" sched=\"rtds\" cpus=[4,5,6,8] + +For checking or changing a VM's scheduling parameters from xl, do +as follows: + * `xl sched-rtds -d vm-rt` + * `xl sched-rtds -d vm-rt -t 10000 -b 25000` + +It is possible, for a multiple vCPUs VM, to change the parameters of +each vCPU individually: + * `xl sched-rtds -d vm-rt -v 0 -p 20000 -b 10000 -v 1 -p 45000 -b 12000` + +# Technical details + +The implementation entirely lives in the hypervisor. Xen has a +pluggable, hook based, architecture for schedulers. RTDS code +is all inside one file: `xen/common/sched_rt.c` + +In libxl, the availability of the RTDS scheduler is advertised by +the presence of the LIBXL\_HAVE\_SCHED\_RTDS symbol. The ability of +specifying different scheduling parameters for each vcpu has been +introduced later, and is available if the following symbols are defined: + * `LIBXL\_HAVE\_VCPU\_SCHED\_PARAMS`, + * `LIBXL\_HAVE\_SCHED\_RTDS\_VCPU\_PARAMS`. + +# Limitations + +RTDS is a special purpose scheduling. This is by design, and not at +all a limitation, but it is certainly something to keep in mind when +thinking about using it. The purpose of the scheduler is enabling +deterministic and statically analyzable behavior (as per the +real-time academic literature), according to the scheduling parameters +assigned to each vCPU. + +Using RTDS a the Xen scheduler, and/or for general purpose workloads +is definitely possible, but the vCPU scheduling parameters (of both +Domain0 and of the various VMs) would probably require tweaking, with +respect to their default values. + +# Testing + +Any change done in RTDS must be tested by doing the following: + +* create a cpupool with RTDS as its scheduler, +* create a few virtual machines a move them in and out of the pool, +* create a few virtual machines, directly inside the pool, and verify + that they boot and can run some basic workload (e.g., login into them + and run simple commands), +* shutdown/reboot the virtual machines, + +The fact that the system boots fine when passing `sched=rtds` to Xen +should also be verified. + +Finally, to check that the scheduler is working properly (although only +at a macroscopic level), the following should be done: + +* create a VM with 1 vCPU and put it in the RTDS cpupool, +* set the scheduling parameters such as it has a 50% reservation, with + `xl sched-rtds -d vm -t 100000 -b 50000`, +* run a CPU-burning process inside the VM (e.g., `yes`), +* check with `xentop` (in Domain0) that the VM is getting no more than + 50% pCPU time. + +# Areas for improvement + +* Work-conserving mode to be added; +* performance assessment, especially focusing on what level of real-time + behavior the scheduler enables. + +# Known issues + +* OSSTest reports occasional failures on ARM. + +# References + +* "RT-Xen: Real-Time Virtualization" [XPDS14 Presentation](http://events.linuxfoundation.org/sites/events/files/slides/2014_Xen_Developer_Summit_0.pdf) +* "Scheduling in Xen" [XPDS15 Presentation](http://events.linuxfoundation.org/sites/events/files/slides/Faggioli_XenSummit.pdf) +* [RT-Xen Project](https://sites.google.com/site/realtimexen/) +* [RTDS-Based-Scheduler](https://wiki.xenproject.org/wiki/RTDS-Based-Scheduler) +* [Xen Project Scheduler](https://wiki.xenproject.org/wiki/Xen_Project_Schedulers) + +# History + +------------------------------------------------------------------------ +Date Revision Version Notes +---------- -------- -------- ------------------------------------------- +2016-10-13 1 Xen 4.8 Document written +---------- -------- -------- -------------------------------------------