From patchwork Fri Oct 14 10:01:40 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dario Faggioli X-Patchwork-Id: 9376351 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 5BB8A6075E for ; Fri, 14 Oct 2016 10:04:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4C8EC2A5F9 for ; Fri, 14 Oct 2016 10:04:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3F52F2A5FB; Fri, 14 Oct 2016 10:04:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,RCVD_IN_SORBS_SPAM,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 3D2522A5F9 for ; Fri, 14 Oct 2016 10:04:34 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1buzJc-0003dD-5L; Fri, 14 Oct 2016 10:02:12 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1buzJb-0003cx-3B for xen-devel@lists.xenproject.org; Fri, 14 Oct 2016 10:02:11 +0000 Received: from [193.109.254.147] by server-7.bemta-6.messagelabs.com id 69/CE-15404-22DA0085; Fri, 14 Oct 2016 10:02:10 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmplleJIrShJLcpLzFFi42K5GHrVWVdxLUO Ewaf73Bbft0xmcmD0OPzhCksAYxRrZl5SfkUCa8bP8z8ZC9r1KlZ3/GNqYHyg3sXIxSEkMJ1R 4syh96wgDovAGlaJxdtbmEAcCYFLrBJ7l71h7GLkBHJiJNr+XmGGsCsklmy5zwpiCwmoSNzcv ooJYtR8Joljr2+BJYQF9CSOHP3B3sXIAWSbS8zeZQsSZhMwkHizYy9YiYiAksS9VZPBepkFvj BJHJzezQaSYBFQlWjeNAvM5hVwkbj1/Bw7iM0p4Caxp3UWM8RiV4nDTS/AjhMVkJNYebmFFaJ eUOLkzCcsIHuZBTQl1u/SBwkzC8hLbH87h3kCo8gsJFWzEKpmIalawMi8ilGjOLWoLLVI19BI L6koMz2jJDcxM0fX0MBMLze1uDgxPTUnMalYLzk/dxMjMAIYgGAH4+WNAYcYJTmYlER5a/UYI oT4kvJTKjMSizPii0pzUosPMWpwcAhMODt3OpMUS15+XqqSBG/naqA6waLU9NSKtMwcYIzClE pw8CiJ8G4CSfMWFyTmFmemQ6ROMRpzbPl9bS0Tx7ap99YyCYFNkhLnPQpSKgBSmlGaBzcIljo uMcpKCfMyAp0pxFOQWpSbWYIq/4pRnINRSZh3P8gUnsy8Erh9r4BOYQI65UMb2CkliQgpqQbG lFPX7wkoHdlca7pOQafAY/XeN8zfzUR+HGh3aCpXaOGfcI3x1I/DDDf/1DnNvqywW2xFVgnjF aO0vFb2/MZkj1NqJdfmSRUW3BdQ3eKclvB18YeV1bUiP5ubt2wI/H9S+GzzQ1VlBsMm28RzO2 U6g67P26qzgU122xye5+KPVrR+SK5kWzhBiaU4I9FQi7moOBEAL13CihgDAAA= X-Env-Sender: raistlin.df@gmail.com X-Msg-Ref: server-8.tower-27.messagelabs.com!1476439328!54966522!1 X-Originating-IP: [209.85.213.67] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.0.13; banners=-,-,- X-VirusChecked: Checked Received: (qmail 64023 invoked from network); 14 Oct 2016 10:02:09 -0000 Received: from mail-vk0-f67.google.com (HELO mail-vk0-f67.google.com) (209.85.213.67) by server-8.tower-27.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 14 Oct 2016 10:02:09 -0000 Received: by mail-vk0-f67.google.com with SMTP id 130so2335735vkg.2 for ; Fri, 14 Oct 2016 03:02:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=gjYtDkGsR2C77RkIKC927hnVncAO59v5jANsw9s4z1g=; b=NBuae+GEj1xhr9k8NHBwYzOCQr5x4QrFANRwnxpNAQGPNRn/wB+hEqrdbl5TCGE0Un swz/rj9hyVwQgLfnORqpYyQuoClP/tqdr/hDwhe9PnRhJ7WOZdr9N5SADqge42ZRAsG6 +wpdeOjEtVQ1CIGd2oCEsVRhGtwFas3TJpFkt89xx4NYvIMIOHtKrNbXpboM8YAZW3/O 6P/gVskusYBDirJomzNXRq7MIzaycyOSHIcQ8Dq96BYwaHtc3xhosZExvhEj4T0qtnjW S3mJ/Zr6fr+11KzqgDA//NSmFPx/6+sK+QO2gTXxXYG22QkIP7pkK2GSrM/Z/PQyIUc6 dhKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=gjYtDkGsR2C77RkIKC927hnVncAO59v5jANsw9s4z1g=; b=hgEwn02MKBEZecXKQtVVuEBw6yCWqSxlaZRe0Z4McHJ+V6OimJpCN5OWZECFJBoPa9 99mACh5NOalbbFBB4VTwe4JbfJKkckk26KtwL/B94seMlR4WiOxkyT7QXfTfgctxLiit EToYJClYMIyj4wY2TP/cg4P28tr8U1N9gRFMt5MzRdq+cKl7GiWSHPgLcTKeSxOxzEyH WeFDDSMdwStJgua4JezprnZ7rq/YljpcVYPOtS65ui7YwfIXXa3+/2j/QYFUOPjsrOxu 6EBmH8P93x1rz8ZUdtEY7YTEateSjdrOIWI+Dxw/2+F1ZcYSBpyieYQPfz3uolNUra2n k55A== X-Gm-Message-State: AA6/9RnNg6h5eOd6DoItQMEDxQ0QgLhXQIjmoPTb6DLgt4n3EQ/9LCRUr7R+8flImDbqpA== X-Received: by 10.31.11.143 with SMTP id 137mr2970540vkl.60.1476439327849; Fri, 14 Oct 2016 03:02:07 -0700 (PDT) Received: from [10.80.3.155] (default-46-102-197-194.interdsl.co.uk. [46.102.197.194]) by smtp.gmail.com with ESMTPSA id s17sm7112274vka.26.2016.10.14.03.02.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 14 Oct 2016 03:02:07 -0700 (PDT) From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Fri, 14 Oct 2016 11:01:40 +0100 Message-ID: <147643928828.13134.6416222510198492144.stgit@Palanthas> In-Reply-To: <147643879557.13134.13090577186331555974.stgit@Palanthas> References: <147643879557.13134.13090577186331555974.stgit@Palanthas> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Cc: Lars Kurth , Stefano Stabellini , Wei Liu , George Dunlap , Andrew Cooper , Anshul Makkar , Ian Jackson , Jan Beulich Subject: [Xen-devel] [PATCH v2 2/3] docs: Credit2 feature document. X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Dario Faggioli --- Cc: George Dunlap Cc: Anshul Makkar Cc: Wei Liu Cc: Lars Kurth Cc: Andrew Cooper Cc: Ian Jackson Cc: Jan Beulich Cc: Konrad Rzeszutek Wilk Cc: Stefano Stabellini --- Changes from v1: * don't claim 'Supported' status; * file renamed from credit2.pandoc to sched_credit2.pandoc; * feature name is now 'Credit2 Scheduler'; * kill the 'e.g.'-s in the header (sorry!); * removed 'Architecture' line from header, as this is arch independent; * removed some text that was duplicated in other sched feature documents; it's redundant wrt what's in the wiki about schedulers, and the page is referenced. --- docs/features/sched_credit2.pandoc | 118 ++++++++++++++++++++++++++++++++++++ 1 file changed, 118 insertions(+) create mode 100644 docs/features/sched_credit2.pandoc diff --git a/docs/features/sched_credit2.pandoc b/docs/features/sched_credit2.pandoc new file mode 100644 index 0000000..8609d9c --- /dev/null +++ b/docs/features/sched_credit2.pandoc @@ -0,0 +1,118 @@ +% Credit2 Scheduler +% Revision 1 + +\clearpage + +# Basics +---------------- ---------------------------------------------------- + Status: **Experimental** + + Component: Hypervisor +---------------- ---------------------------------------------------- + +# Overview + +Credit2 is one of the virtual CPU (vCPU) scheduler available in the +Xen hypervisor. + +Credit2 was designed as a general purpose scheduler, with particular +focus on improving handling of mixed workloads, scalability and +support for low latency applications inside VMs, with respect to +Credit1. + +# User details + +Credit2 is not in use by default. In order to use it as the Xen +scheduler the following parameter should be passed to the hypervisor +at boot: + + `sched=credit2` + +Other parameters are available for tuning the behavior of Credit2 +(see `docs/misc/xen-command-line.markdown` for a complete list and +for their meaning). + +Once the system is live, for creating a cpupool with Credit2 as +its scheduler, either compile a cpupool configuration file, as +described in `docs/man/xlcpupool.cfg.pod.5` (and as exemplified +in `tools/examples/cpupool`), or use just `xl` directly: + + xl cpupool-create name=\"pool1\" sched=\"credit2\" cpus=[1,2] + +Two kind of interactions with the scheduler are possible: + +* checking or changing the global parameters, via, e.g.: + * `xl sched-credit2 -s` + * `xl sched-credit2 -s -p pool1` + * `xl sched-credit2 -s -r 100` +* checking or changing a VM scheduling parameters, via, e.g.: + * `xl sched-credit2 -d vm1` + * `xl sched-credit2 -d vm1 -w 1024` + +# Technical details + +Implementation entirely lives in the hypervisor. Xen has a pluggable, +hook based, architecture for schedulers. Thanks to this, Credit2 code +is all contained in `xen/common/sched_credit2.c`. + +Global scheduling parameters, such as context switching rate +limiting, is only available from Xen 4.8 onward. In libxl, the +`LIBXL\_HAVE\_SCHED\_CREDIT2\_PARAMS` symbol is introduced to +indicate their availability. + +# Limitations + +The Credit1 scheduler comes with vCPU hard-affinity, vCPU soft-affinity +and caps (see `docs/man/xl.pod.1.in` for more details). In Credit2, +vCPU hard affinity is supported starting from Xen 4.8, while soft-affinity +and caps, while being worked on, are not yet available in any released +hypervisor. + +# Testing + +Any change done in Credit2 wants to be tested by doing at least the +following: + +* boot the system with `sched=credit2`, +* create a few virtual machine and verify that they boot and can + run some basic workload (e.g., login into them and run simple commands), +* shutdown/reboot the virtual machines, +* shutdown/reboot the system. + +Ideally, all the above steps should **also** be performed in a configuration +where Credit2 is used as the scheduler of a cpupool, and by also doing the +following: + +* move a virtual machine inside and outside a Credit2 cpupool. + +# Areas for improvement + +* Close the feature gap with Credit1 (i.e., finishing implementing vCPU + soft-affinity and caps); +* vCPUs' reservations (similar to caps, but providing a vCPU with guarantees + about some pCPU time it will always be able to execute for); +* benchmarking for assessing the best combination of values for the various + parameters (`sched\_credit2\_migrate\_resist`, `credit2\_balance\_over`, + `credit2\_balance\_under`) + +# Known issues + +* I/O oriented benchmarks (like network and disk throughput) have given + contradictory and non-conclusive results so far. Need to run more of + those. + +# References + +* "Scheduler development update", XenSummit Asia 2009 [whitepaper](http://www-archive.xenproject.org/files/xensummit_intel09/George_Dunlap.pdf) +* "Scheduling in Xen" [XPDS15 Presentation](http://events.linuxfoundation.org/sites/events/files/slides/Faggioli_XenSummit.pdf) +* "Scope and Performance of Credit-2 Scheduler" [XPDS16 Presentation](http://www.slideshare.net/xen_com_mgr/xpds16-scope-and-performance-of-credit2-scheduler-anshul-makkar-ctirix-systems-uk-ltd) +* "The Credit2 Scheduler" [on the Xen-Project wiki](https://wiki.xenproject.org/wiki/Credit2_Scheduler_Development) +* "Xen Project Schedulers" [on the Xen-Project wiki](https://wiki.xenproject.org/wiki/Xen_Project_Schedulers) + +# History + +------------------------------------------------------------------------ +Date Revision Version Notes +---------- -------- -------- ------------------------------------------- +2016-10-14 1 Xen 4.8 Document written +---------- -------- -------- -------------------------------------------