From patchwork Thu Oct 13 11:02:28 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dario Faggioli X-Patchwork-Id: 9374789 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 53B636075E for ; Thu, 13 Oct 2016 11:05:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 453BF297A7 for ; Thu, 13 Oct 2016 11:05:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 398F6297FE; Thu, 13 Oct 2016 11:05:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,RCVD_IN_SORBS_SPAM,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 30F1929FD5 for ; Thu, 13 Oct 2016 11:05:21 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1budn7-0002ij-FK; Thu, 13 Oct 2016 11:03:13 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1budn5-0002iZ-GP for xen-devel@lists.xenproject.org; Thu, 13 Oct 2016 11:03:11 +0000 Received: from [85.158.143.35] by server-3.bemta-6.messagelabs.com id 19/D9-04595-EE96FF75; Thu, 13 Oct 2016 11:03:10 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmphleJIrShJLcpLzFFi42K5GHrnoO7bzP/ hBj+7zCy+b5nM5MDocfjDFZYAxijWzLyk/IoE1owNc64wF/xTrfj9eQZbA+NLhS5GLg4hgZmM Ek/bTjGBOCwCa1glvs6+ytjFyMkhIXCJVeLiOxYIO0bi18K1TBB2lUTLx19gtpCAisTN7auYI CbNZ5JoOP+eHSQhLKAnceToDyjbRGLhzMdgNpuAgcSbHXtZQWwRASWJe6smgzUzC5xkknjU85 8NJMEioCpx4sFjZhCbV8BVYuKetWAXcQq4Sex4+hNqs6vEppWrwOpFBeQkVl5uYYWoF5Q4OfM J0NUcQEM1Jdbv0gcJMwvIS2x/O4d5AqPILCRVsxCqZiGpWsDIvIpRvTi1qCy1SNdML6koMz2j JDcxM0fX0MBMLze1uDgxPTUnMalYLzk/dxMjMPwZgGAH47wT/ocYJTmYlER5f/n9DxfiS8pPq cxILM6ILyrNSS0+xKjBwSEw4ezc6UxSLHn5ealKEryHMoDqBItS01Mr0jJzgBEKUyrBwaMkwu sMkuYtLkjMLc5Mh0idYjTm2PL72lomjm1T761lEgKbJCXOexykVACkNKM0D24QLHFcYpSVEuZ lBDpTiKcgtSg3swRV/hWjOAejkjCvBsgUnsy8Erh9r4BOYQI6xWbSH5BTShIRUlINjOEv5vgt bp+7Q3XuZ9G+/we4TwVnT4l0lRZ3fpp98iHPwfeBj5U1dCwPsXJsYj5oslzu2MTM80yXdv/7K 3fmYeDOF8n71us7zbn0WMq4v17HcGpX0SHpRpcP/KWHpT8ePzKp52t6dVJm3YpdtRxVUzjX8v Xw3RN1Lkt5KGyUp3hPKSbz47TFc5RYijMSDbWYi4oTAS7HbXAXAwAA X-Env-Sender: raistlin.df@gmail.com X-Msg-Ref: server-8.tower-21.messagelabs.com!1476356588!37809888!1 X-Originating-IP: [209.85.220.193] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.0.13; banners=-,-,- X-VirusChecked: Checked Received: (qmail 63392 invoked from network); 13 Oct 2016 11:03:09 -0000 Received: from mail-qk0-f193.google.com (HELO mail-qk0-f193.google.com) (209.85.220.193) by server-8.tower-21.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 13 Oct 2016 11:03:09 -0000 Received: by mail-qk0-f193.google.com with SMTP id v138so5632232qka.2 for ; Thu, 13 Oct 2016 04:03:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=WDQzrGlf59sLJAk3LRL0DXFbCqgodbsk0NLMzpJeI5Q=; b=Slbn0W+Eiw84M/bap5SsisQk6pagyhBlydrIuQI7g2cGqlZE8W406dp1DkEmqP+bXs wER1Z1bwbiSFFQct9dz5ZM/KJ8g/vNhDj8rTVQtQGOwC4HXqTpcU2OAUyxTmPVA22NxL bxitu1VqmYsQSaf/HOTqajKFOpHQEm9jvw0CYkY/7pQ8i6ZD08br7cr4K3TqEVzq7D/L Gno943bQeCYTXyQJxzckTNll8vF+dsWZBk16ZhQ66hVtTfyMPGC/GG3pX5t21ghx2hVj fu9QeEhFNXp3SzHk37wuu3inVUeQfBAhrKsbh0n0Fh1mTQw6+FKORCCQ6WZfaMYh8u4g 5NYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=WDQzrGlf59sLJAk3LRL0DXFbCqgodbsk0NLMzpJeI5Q=; b=Zx1MlDhUYyt7Nq7/1GXJB8a8XPK3VpqstHjx903aw8gdnCfpOWptWQzvGAvTuQPWFr DnpCfRtwY4CbS0YE597Jv3ElnYXp71XZNXS1X+V8rfPuIn0EzQPLFoRtRNXqKPn2AN04 zkbCxyq9zK8/LohvlsLA+QpVBL0uXz1WAMDXUdx9U1hh16fMuaZXPmXRWgCS4Qz0414D +N8jKv/3JyvP5BHGymhRPDlnKBf3tgnE+Nvm0f+22VETbY5L6rYnr45NMXzNWYYDbRkO r8OglFRukpaZrkleaKYgv3cLGSGoxn6Ya7jRjP1Cf1YH5XXI9bPSkLrKG7OGbSOe/Xqf Df9w== X-Gm-Message-State: AA6/9Rlx3NQ4Xjap0o/IrjLqklVcw8dbNSeeRZ4YKVpuEtea6U8NKXeZCEcYhUyPSl/2XQ== X-Received: by 10.194.201.227 with SMTP id kd3mr6628880wjc.74.1476356588426; Thu, 13 Oct 2016 04:03:08 -0700 (PDT) Received: from [10.80.3.155] (default-46-102-197-194.interdsl.co.uk. [46.102.197.194]) by smtp.gmail.com with ESMTPSA id j1sm11238276wjl.21.2016.10.13.04.03.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Oct 2016 04:03:07 -0700 (PDT) From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Thu, 13 Oct 2016 12:02:28 +0100 Message-ID: <147635653648.17095.14880201185403654377.stgit@Palanthas> In-Reply-To: <147635565298.17095.17316867931713209620.stgit@Palanthas> References: <147635565298.17095.17316867931713209620.stgit@Palanthas> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Cc: Lars Kurth , Stefano Stabellini , Wei Liu , George Dunlap , Andrew Cooper , Ian Jackson , Jan Beulich Subject: [Xen-devel] [PATCH 1/3] docs: Credit1 feature document. X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Dario Faggioli --- Cc: George Dunlap Cc: Wei Liu Cc: Lars Kurth Cc: Andrew Cooper Cc: Ian Jackson Cc: Jan Beulich Cc: Konrad Rzeszutek Wilk Cc: Stefano Stabellini --- docs/features/credit.pandoc | 99 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) create mode 100644 docs/features/credit.pandoc diff --git a/docs/features/credit.pandoc b/docs/features/credit.pandoc new file mode 100644 index 0000000..fed0da2 --- /dev/null +++ b/docs/features/credit.pandoc @@ -0,0 +1,99 @@ +% Credit +% Revision 1 + +\clearpage + +# Basics +---------------- ---------------------------------------------------- + Status: e.g. **Supported** + +Architecture(s): e.g. x86, arm + + Component: e.g. Hypervisor +---------------- ---------------------------------------------------- + +# Overview + +Credit (also known as Credit1) is the default virtual CPU (vCPU) scheduler +of the Xen hypervisor. The job of an hypervisor's scheduler is to decide, +among all the various vCPUs of the various virtual machines, which ones +should execute on the host's physical CPUs (pCPUs), at any given point in +time. + +# User details + +Xen supports multiple schedulers. As said, Credit is the default, so it +is used automatically, unless the `sched=$SCHED` (with `$SCHED` different +than `credit`) parameter is passed to Xen via the bootloader. + +Once the system is live, for creating a cpupool with Credit as its +scheduler, either compile a cpupool configuration file, as described +in `docs/man/xlcpupool.cfg.pod.5` (and as exemplified in +`tools/examples/cpupool`), or use just `xl` directly: + + xl cpupool-create name=\"pool1\" sched=\"credit\" cpus=[4,8] + +Two kind of interactions with the scheduler are possible: + +* checking or changing the global parameters, via, e.g.: + * `xl sched-credit -s` + * `xl sched-credit -s -p pool1` + * `xl sched-credit -s -t 20` +* checking or changing a VM's scheduling parameters, via, e.g.: + * `xl sched-credit -d vm1` + * `xl sched-credit -d vm1 -w 512` + +# Technical details + +Implementation entirely lives in the hypervisor. Xen has a pluggable, hook +based, architecture for schedulers. Actual Credit code is all inside one +file: `xen/common/sched_credit.c`. + +# Limitations + +In Credit, a vCPU has a priority, a status (i.e., active or inactive), +a weight and some credits... and all these things interact in a rather +involved way. Also, with years of use, things have gotten even more +complex (due to, e.g., the introduction of boosting, caps and vCPU +soft-affinity). + +Dealing with such complexity is starting to be an issue. Odd behavior +or subtle scheduling anomalies, that is not always possible to act upon, +have been identified already. [1][2][3] + +A certain lack of scalability and difficulties and weakness in dealing +with mixed workloads and VMs with low latency requirements are other +known problems. [4] For all these reasons, effort is ongoing to have +Credit2 become the new default scheduler. + +# Testing + +Any change to Credit code must to be tested by doing at least the following: + +* create a few virtual machine and verify that they boot and can + run some basic workload (e.g., login into them and run simple commands), +* shutdown/reboot the virtual machines, +* shutdown the system. + +Ideally, all the above steps should **also** be performed in a configuration +that includes cpupools, better if with pools using different schedulers, and +by also doing the following: + +* move the virtual machines between cpupools. + +# References + +* [potential non-ideal behavior on hyperthreaded systems](https://lists.xenproject.org/archives/html/xen-devel/2014-07/msg01848.html) [1] +* [long standing BOOST vs. migration bug](https://lists.xen.org/archives/html/xen-devel/2015-10/msg02851.html) [2] +* [priority handling issues](https://lists.xenproject.org/archives/html/xen-devel/2016-05/msg01362.html) [3] +* "Scheduler development update", XenSummit Asia 2009 [whitepaper](http://www-archive.xenproject.org/files/xensummit_intel09/George_Dunlap.pdf) [4] +* "Scheduling in Xen" [XPDS15 Presentation](http://events.linuxfoundation.org/sites/events/files/slides/Faggioli_XenSummit.pdf) +* [Xen Project Scheduler](https://wiki.xenproject.org/wiki/Xen_Project_Schedulers) + +# History + +------------------------------------------------------------------------ +Date Revision Version Notes +---------- -------- -------- ------------------------------------------- +2016-10-10 1 Xen 4.8 Document written +---------- -------- -------- -------------------------------------------