From patchwork Fri Oct 14 10:00:55 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dario Faggioli X-Patchwork-Id: 9376349 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 937A76075E for ; Fri, 14 Oct 2016 10:03:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 820852A5F9 for ; Fri, 14 Oct 2016 10:03:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 75C982A5FB; Fri, 14 Oct 2016 10:03:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 7E24E2A5F9 for ; Fri, 14 Oct 2016 10:03:41 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1buzIt-0003WM-TB; Fri, 14 Oct 2016 10:01:27 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1buzIs-0003W7-Pc for xen-devel@lists.xenproject.org; Fri, 14 Oct 2016 10:01:26 +0000 Received: from [85.158.139.211] by server-12.bemta-5.messagelabs.com id 58/D5-09561-5FCA0085; Fri, 14 Oct 2016 10:01:25 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpgleJIrShJLcpLzFFi42K5GHrVUffLGoY Ig/UzpC2+b5nM5MDocfjDFZYAxijWzLyk/IoE1ozG6VOYCr5rVMx+ztrAuEm5i5GTQ0hgKqPE 8S9VIDaLwBpWiddTsroYuTgkBC6xSsx4PoMZJCEhECMx+94CVgi7UmJN9z02iGYViZvbVzGBN AgJLGCSWPD4JlhCWEBP4sjRH+xdjBxAtrnEjjW6IGE2AQOJNzv2gs0REVCSuLdqMlgvs8BJJo lHPf/ZQOpZBFQlvswRAanhFXCReH/hPdhITgE3iT2ts5gh9rpKHG56wQhiiwrISay83MIKUS8 ocXLmExaQMcwCmhLrd+mDhJkF5CW2v53DPIFRZBaSqlkIVbOQVC1gZF7FqF6cWlSWWqRrqpdU lJmeUZKbmJmja2hgqpebWlycmJ6ak5hUrJecn7uJERj2DECwg/FLv/MhRkkOJiVR3lo9hgghv qT8lMqMxOKM+KLSnNTiQ4waHBwCE87Onc4kxZKXn5eqJMHLDIwvIcGi1PTUirTMHGBkwpRKcP AoifBuWg2U5i0uSMwtzkyHSJ1iNObY8vvaWiaObVPvrWUSApskJc57FKRUAKQ0ozQPbhAsYVx ilJUS5mUEOlOIpyC1KDezBFX+FaM4B6OSMG8xyD08mXklcPteAZ3CBHTKhzawU0oSEVJSDYzh pysurTl7cHnamsOJU4SjM6Jvbz5z1r9gt+iBhZJJrGdvM2+pXyGQfqAtpd0saHey1faVq7PYf 5nNf9d8y77iuwW/nmzwh3MF5y1eLvStWb3h2ef/vk9lMrvrF31+dnD30zUsftV8DDc3v1wUvE poQZBYweO5y6Ui8z9GVWZ65oS4asiqcFgosRRnJBpqMRcVJwIAENpCyBMDAAA= X-Env-Sender: raistlin.df@gmail.com X-Msg-Ref: server-13.tower-206.messagelabs.com!1476439283!49265507!1 X-Originating-IP: [209.85.213.65] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 8.84; banners=-,-,- X-VirusChecked: Checked Received: (qmail 40270 invoked from network); 14 Oct 2016 10:01:24 -0000 Received: from mail-vk0-f65.google.com (HELO mail-vk0-f65.google.com) (209.85.213.65) by server-13.tower-206.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 14 Oct 2016 10:01:24 -0000 Received: by mail-vk0-f65.google.com with SMTP id 2so4785184vkb.1 for ; Fri, 14 Oct 2016 03:01:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=FD4pcMlPDvGZzHxUEWwrN9OT87dSH17G06GM8Vf7NZ8=; b=XI/AJ5DfYNhZWM87gfy7aZMlUUrLasxc7ydhT9innPrF4+j6jqOTzrITmXotw/qXkY jStLrX2cL8dbvfhUjN/nB1rm3bKyQ6iQ564mwBOsYTk6co8m0DuvBhccJVLtnXfQYrX1 hCUTmdq0TcBA8IA+utdxBm2ZviSkqS3ezC0UQ06IeBwFP5gC+qTE7QW1vPk3Q6O79coG iZlKxFxMHIj3OCF8715UmZPxN3liFH4ndGkWT2Y9oSuKI2gPTUnnPABFoGSS3HGIAhxS +A7mMt2zAAF4IyRnGRhdZ8ok0kpdZ7cC/gMrv01ymLYUMNKP7FU/gA26cawT+4CZ4pU3 d58A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=FD4pcMlPDvGZzHxUEWwrN9OT87dSH17G06GM8Vf7NZ8=; b=NN6Lo9zuKHJFq2xGi3bpQMjGuz1mHdx6HSUcNE3vxJwYI8U9EzQ6Y6K+fTuEQe11sU HF+gZ3CO/1rGUCjqnyHw9sVO/D68Ux8OhRLN3EPlDxsLbXP1o032QBpRXHlXFccop2ty 8NshsNV7ZwSMcq4NxBs1TAIWItqKWotby12Kn03VNvLasnmN2AK3UHggacAxVbWv1/9E y/nr0LWHoDf3bXZ0ks/VnrcVWsjrJRKKB+uwQN1IBjbhDBEtuEWqdQ642gBvYr/szdUq nrSIJiNvJjbUcAbVHx04etQPgylLCIlaAJw/VnGRsZ2W0KXOJbyazQUzP2zZcfyMfsSB TXXw== X-Gm-Message-State: AA6/9RlfFti8KWybVOXr1AFg9Dl+lsYB7NnY5OOlXh/8+UmRZVmWxl4ci/gctcEcpJ0NCw== X-Received: by 10.31.226.67 with SMTP id z64mr8406909vkg.141.1476439283131; Fri, 14 Oct 2016 03:01:23 -0700 (PDT) Received: from [10.80.3.155] (default-46-102-197-194.interdsl.co.uk. [46.102.197.194]) by smtp.gmail.com with ESMTPSA id k134sm4214321vkd.12.2016.10.14.03.01.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 14 Oct 2016 03:01:22 -0700 (PDT) From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Fri, 14 Oct 2016 11:00:55 +0100 Message-ID: <147643924366.13134.6747424247602583199.stgit@Palanthas> In-Reply-To: <147643879557.13134.13090577186331555974.stgit@Palanthas> References: <147643879557.13134.13090577186331555974.stgit@Palanthas> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Cc: Lars Kurth , Stefano Stabellini , Wei Liu , George Dunlap , Andrew Cooper , Ian Jackson , Jan Beulich Subject: [Xen-devel] [PATCH v2 1/3] docs: Credit1 feature document. X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Dario Faggioli --- Cc: George Dunlap Cc: Wei Liu Cc: Lars Kurth Cc: Andrew Cooper Cc: Ian Jackson Cc: Jan Beulich Cc: Konrad Rzeszutek Wilk Cc: Stefano Stabellini --- Changes from v1: * file renamed from credit.pandoc to sched_credit.pandoc; * feature name is now 'Credit Scheduler'; * kill the 'e.g.'-s in the header (sorry!); * removed 'Architecture' line from header, as this is arch independent; * removed some text that was duplicated in other sched feature documents; it's redundant wrt what's in the wiki about schedulers, and the page is referenced. --- docs/features/sched_credit.pandoc | 97 +++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 docs/features/sched_credit.pandoc diff --git a/docs/features/sched_credit.pandoc b/docs/features/sched_credit.pandoc new file mode 100644 index 0000000..79587d8 --- /dev/null +++ b/docs/features/sched_credit.pandoc @@ -0,0 +1,97 @@ +% Credit Scheduler +% Revision 1 + +\clearpage + +# Basics +---------------- ---------------------------------------------------- + Status: **Supported** + + Component: Hypervisor +---------------- ---------------------------------------------------- + +# Overview + +Credit (also known as Credit1) is the default virtual CPU (vCPU) scheduler +of the Xen hypervisor. + +It is a general purpose, weighted fair-share scheduler. + +# User details + +Xen supports multiple schedulers. As said, Credit is the default, so it +is used automatically, unless the `sched=$SCHED` (with `$SCHED` different +than `credit`) parameter is passed to Xen via the bootloader. + +Once the system is live, for creating a cpupool with Credit as its +scheduler, either compile a cpupool configuration file, as described +in `docs/man/xlcpupool.cfg.pod.5` (and as exemplified in +`tools/examples/cpupool`), or use just `xl` directly: + + xl cpupool-create name=\"pool1\" sched=\"credit\" cpus=[4,8] + +Two kind of interactions with the scheduler are possible: + +* checking or changing the global parameters, via, e.g.: + * `xl sched-credit -s` + * `xl sched-credit -s -p pool1` + * `xl sched-credit -s -t 20` +* checking or changing a VM's scheduling parameters, via, e.g.: + * `xl sched-credit -d vm1` + * `xl sched-credit -d vm1 -w 512` + +# Technical details + +Implementation entirely lives in the hypervisor. Xen has a pluggable, +hook based, architecture for schedulers. Thanks to this, Credit code +is all contained in `xen/common/sched_credit.c`. + +# Limitations + +In Credit, a vCPU has a priority, a status (i.e., active or inactive), +a weight and some credits... and all these things interact in a rather +involved way. Also, with years of use, things have gotten even more +complex (due to, e.g., the introduction of boosting, caps and vCPU +soft-affinity). + +Dealing with such complexity is starting to be an issue. Odd behavior +or subtle scheduling anomalies, that is not always possible to act upon, +have been identified already. [1][2][3] + +A certain lack of scalability and difficulties and weakness in dealing +with mixed workloads and VMs with low latency requirements are other +known problems. [4] For all these reasons, effort is ongoing to have +Credit2 become the new default scheduler. + +# Testing + +Any change to Credit code must to be tested by doing at least the following: + +* create a few virtual machine and verify that they boot and can + run some basic workload (e.g., login into them and run simple commands), +* shutdown/reboot the virtual machines, +* shutdown the system. + +Ideally, all the above steps should **also** be performed in a configuration +that includes cpupools, better if with pools using different schedulers, and +by also doing the following: + +* move the virtual machines between cpupools. + +# References + +* [potential non-ideal behavior on hyperthreaded systems](https://lists.xenproject.org/archives/html/xen-devel/2014-07/msg01848.html) [1] +* [long standing BOOST vs. migration bug](https://lists.xen.org/archives/html/xen-devel/2015-10/msg02851.html) [2] +* [priority handling issues](https://lists.xenproject.org/archives/html/xen-devel/2016-05/msg01362.html) [3] +* "Scheduler development update", XenSummit Asia 2009 [whitepaper](http://www-archive.xenproject.org/files/xensummit_intel09/George_Dunlap.pdf) [4] +* "Scheduling in Xen" [XPDS15 Presentation](http://events.linuxfoundation.org/sites/events/files/slides/Faggioli_XenSummit.pdf) +* "The Credit Scheduler" [on the Xen-Project wiki](https://wiki.xenproject.org/wiki/Credit_Scheduler) +* "Xen Project Schedulers" [on the Xen-Project wiki](https://wiki.xenproject.org/wiki/Xen_Project_Schedulers) + +# History + +------------------------------------------------------------------------ +Date Revision Version Notes +---------- -------- -------- ------------------------------------------- +2016-10-14 1 Xen 4.8 Document written +---------- -------- -------- -------------------------------------------