From patchwork Thu Oct 13 11:03:25 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dario Faggioli X-Patchwork-Id: 9374793 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 2A0FC6075E for ; Thu, 13 Oct 2016 11:05:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1A8FB29FB3 for ; Thu, 13 Oct 2016 11:05:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0F0D829FB8; Thu, 13 Oct 2016 11:05:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,RCVD_IN_SORBS_SPAM,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 626F129FB3 for ; Thu, 13 Oct 2016 11:05:58 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1budnq-0002pm-0U; Thu, 13 Oct 2016 11:03:58 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1budnp-0002pY-BE for xen-devel@lists.xenproject.org; Thu, 13 Oct 2016 11:03:57 +0000 Received: from [193.109.254.147] by server-6.bemta-6.messagelabs.com id 9E/1D-11175-C1A6FF75; Thu, 13 Oct 2016 11:03:56 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmplleJIrShJLcpLzFFi42K5GHrnoK5E1v9 wg6PHtS2+b5nM5MDocfjDFZYAxijWzLyk/IoE1oxlC1cyFew3rHjSt5etgfGQVhcjF4eQwExG iYun3jGCOCwCa1glmo4uYwJxJAQusUpM/vmYvYuRE8iJkbhxrw3KrpB4tL6PBcQWElCRuLl9F RPEqAVMEv9WvAYrEhbQkzhy9AeUbSKxvPUDmM0mYCDxZsdeVhBbREBJ4t6qyWDNzAJfmCQOTu 9mA0mwCKhKLFqwnxnE5hVwlehb9QPM5hRwk9jx9CcTxGZXiU0rV4HViwrISay83MIKUS8ocXL mE6DrOICGakqs36UPEmYWkJfY/nYO8wRGkVlIqmYhVM1CUrWAkXkVo3pxalFZapGuqV5SUWZ6 RkluYmaOrqGBmV5uanFxYnpqTmJSsV5yfu4mRmAEMADBDsbpl/0PMUpyMCmJ8v7y+x8uxJeUn 1KZkVicEV9UmpNafIhRg4NDYMLZudOZpFjy8vNSlSR4D2UA1QkWpaanVqRl5gBjFKZUgoNHSY S3ESTNW1yQmFucmQ6ROsVozLHl97W1TBzbpt5byyQENklKnPc4SKkASGlGaR7cIFjquMQoKyX Mywh0phBPQWpRbmYJqvwrRnEORiVh3jSQKTyZeSVw+14BncIEdIrNpD8gp5QkIqSkGhh392s8 uXdBf+KdAwlbYg3vH9zSJZxVu3nv/wabRVfSu1c+OGS9b9feX98SVVd3JAV6MivHhBQEL/zjy H1EarXbjxuBszxzWZsOci7Nr7I1kU649JfjxKa/k2UEi9a9yxL9vcq43iE5w1Ln+NxtZ/c63D G9Yt5aGi4ncG/+goe5HMLspSFqnDVKLMUZiYZazEXFiQD4xmXaGAMAAA== X-Env-Sender: raistlin.df@gmail.com X-Msg-Ref: server-9.tower-27.messagelabs.com!1476356631!65123589!1 X-Originating-IP: [209.85.220.193] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 8.84; banners=-,-,- X-VirusChecked: Checked Received: (qmail 58450 invoked from network); 13 Oct 2016 11:03:52 -0000 Received: from mail-qk0-f193.google.com (HELO mail-qk0-f193.google.com) (209.85.220.193) by server-9.tower-27.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 13 Oct 2016 11:03:52 -0000 Received: by mail-qk0-f193.google.com with SMTP id z190so5622326qkc.3 for ; Thu, 13 Oct 2016 04:03:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=qfRIJ6ZJ+DRyz/V39TweancxpV9cn4FMf9BDrn4ON4k=; b=sojsCNlqYRuAljFEgHy++3bMILL1JE457iNT2n7aL2IpIdIYBBR7XJ/aW30SdyGotQ 4PwLrFmD6xGHroD7wjhxKRvckQnpdz93hK0beVMv/cbNYZloh2HB+SncdGm616Aqzu2i XLwiUDJxCJdVLhBDNxPFcmmuOGzC6RQYjEFquwXAJkGqw2GQXOcsyOuOWGjqpwpWPv7U 2hhzVYjPnNupTLQbpPoZqCe8ufjbtO3v8r+W/ehfW8fC92e7XX/QBgiJEtFLvfshO+Zl Wlrqz4hWjqp8F49LDwsxG1BBiORQpumneZLdPVgxXRkKnLTEv/MkqWkH8V2Tp83lcbKe 1z0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=qfRIJ6ZJ+DRyz/V39TweancxpV9cn4FMf9BDrn4ON4k=; b=Q+ADQ4p0xKYc6LxOycC2oWJOQvmjFFAYNYgBzbH8rzmMEm8T6f/5EdMSUIYtrnsztd uCepuTG03FXgvAF/84uQnoUJnNdhWrxtj+Z5wgGoLnhb/k2EN+hsJ5FFr+xTMN24A8Z2 RA11RlveWCjYMqM8pK4M1w42+d4TjrgsMZx5DO88PFq4d2Jr9cTouWJdLRi3watesCUj ebgrmOVG1Arg3UgBvuMZHEcZJ6kucqltyMnDXnOlBc6wahIRcnnqjtngp0uvxtmRaXn8 Aq2GFq4zx3WRaq7ZWkRz7gcT9tEL8DQNlSRFF9yIhYShmfwpK8kd4aCzCcDPccmnRaH7 e08A== X-Gm-Message-State: AA6/9RngL1Bcn6uhxZztcccNT/mWsqSjO4yt27FmMHLxGi6XCACCSfrc/QlqBjOZnlC8WA== X-Received: by 10.194.81.133 with SMTP id a5mr6645440wjy.86.1476356631411; Thu, 13 Oct 2016 04:03:51 -0700 (PDT) Received: from [10.80.3.155] (default-46-102-197-194.interdsl.co.uk. [46.102.197.194]) by smtp.gmail.com with ESMTPSA id qo8sm21574803wjc.46.2016.10.13.04.03.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Oct 2016 04:03:50 -0700 (PDT) From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Thu, 13 Oct 2016 12:03:25 +0100 Message-ID: <147635659347.17095.16688986879634316365.stgit@Palanthas> In-Reply-To: <147635565298.17095.17316867931713209620.stgit@Palanthas> References: <147635565298.17095.17316867931713209620.stgit@Palanthas> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Cc: Lars Kurth , Stefano Stabellini , Wei Liu , George Dunlap , Andrew Cooper , Anshul Makkar , Ian Jackson , Jan Beulich Subject: [Xen-devel] [PATCH 2/3] docs: Credit2 feature document. X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Since we are marking the feature as 'Supported', remove the "this is experimental software" warning in the code at once. Signed-off-by: Dario Faggioli --- Cc: George Dunlap Cc: Anshul Makkar Cc: Wei Liu Cc: Lars Kurth Cc: Andrew Cooper Cc: Ian Jackson Cc: Jan Beulich Cc: Konrad Rzeszutek Wilk Cc: Stefano Stabellini --- docs/features/credit2.pandoc | 123 ++++++++++++++++++++++++++++++++++++++++++ xen/common/sched_credit2.c | 2 - 2 files changed, 123 insertions(+), 2 deletions(-) create mode 100644 docs/features/credit2.pandoc diff --git a/docs/features/credit2.pandoc b/docs/features/credit2.pandoc new file mode 100644 index 0000000..6203cec --- /dev/null +++ b/docs/features/credit2.pandoc @@ -0,0 +1,123 @@ +% Credit2 +% Revision 1 + +\clearpage + +# Basics +---------------- ---------------------------------------------------- + Status: e.g. **Supported** + +Architecture(s): e.g. x86, arm + + Component: e.g. Hypervisor +---------------- ---------------------------------------------------- + +# Overview + +Credit2 is one of the virtual CPU (vCPU) scheduler available in the +Xen hypervisor. The job of an hypervisor's virtual CPU scheduler is +to decide, among all the various vCPUs of the various virtual machines, +which ones should execute on the host's physical CPUs (pCPUs), at any +given point in time. + +Credit2 was designed as a general purpose scheduler, with particular +focus on improving handling of mixed workloads, scalability and +support for low latency applications inside VMs, with respect to +Credit1. + +# User details + +Credit2 is not in use by default. In order to use it as the Xen +scheduler the following parameter should be passed to the hypervisor +at boot: + + `sched=credit2` + +Other parameters are available for tuning the behavior of Credit2 +(see `docs/misc/xen-command-line.markdown` for a complete list and +for their meaning). + +Once the system is live, for creating a cpupool with Credit2 as +its scheduler, either compile a cpupool configuration file, as +described in `docs/man/xlcpupool.cfg.pod.5` (and as exemplified +in `tools/examples/cpupool`), or use just `xl` directly: + + xl cpupool-create name=\"pool1\" sched=\"credit2\" cpus=[1,2] + +Two kind of interactions with the scheduler are possible: + +* checking or changing the global parameters, via, e.g.: + * `xl sched-credit2 -s` + * `xl sched-credit2 -s -p pool1` + * `xl sched-credit2 -s -r 100` +* checking or changing a VM scheduling parameters, via, e.g.: + * `xl sched-credit2 -d vm1` + * `xl sched-credit2 -d vm1 -w 1024` + +# Technical details + +The implementation entirely lives in the hypervisor. Xen has a +pluggable, hook based, architecture for schedulers. Credit2 code +is all inside one file: `xen/common/sched_credit2.c` + +Global scheduling parameters, such as context switching rate +limiting, is only available from Xen 4.8 onward. In libxl, the +`LIBXL\_HAVE\_SCHED\_CREDIT2\_PARAMS` symbol is introduced to +indicate their availability. + +# Limitations + +The Credit1 scheduler supports vCPU hard-affinity, vCPU soft-affinity +and caps (see `docs/man/xl.pod.1.in` for more details). In Credit2, +vCPU hard affinity is supported starting from Xen 4.8, while soft-affinity +and caps are being implemented, but not supported yet in any released +hypervisor. + +# Testing + +Any change done in Credit2 wants to be tested by doing at least the +following: + +* boot the system with `sched=credit2`, +* create a few virtual machine and verify that they boot and can + run some basic workload (e.g., login into them and run simple commands), +* shutdown/reboot the virtual machines, +* shutdown/reboot the system. + +Ideally, all the above steps should **also** be performed in a configuration +where Credit2 is used as the scheduler of a cpupool, and by also doing the +following: + +* move a virtual machine inside and outside a Credit2 cpupool. + +# Areas for improvement + +* Close the feature gap with Credit1 (i.e., finishing implementing vCPU + soft-affinity and caps); +* vCPUs' reservations (similar to caps, but providing a vCPU with guarantees + about some pCPU time it will always be able to execute for); +* benchmarking for assessing the best combination of values for the various + parameters (`sched\_credit2\_migrate\_resist`, `credit2\_balance\_over`, + `credit2\_balance\_under`) + +# Known issues + +* I/O oriented benchmarks (like network and disk throughput) have given + contradictory and non-conclusive results so far. Need to run more of + those. + +# References + +* "Scheduler development update", XenSummit Asia 2009 [whitepaper](http://www-archive.xenproject.org/files/xensummit_intel09/George_Dunlap.pdf) +* "Scheduling in Xen" [XPDS15 Presentation](http://events.linuxfoundation.org/sites/events/files/slides/Faggioli_XenSummit.pdf) +* "Scope and Performance of Credit-2 Scheduler" [XPDS16 Presentation](http://www.slideshare.net/xen_com_mgr/xpds16-scope-and-performance-of-credit2-scheduler-anshul-makkar-ctirix-systems-uk-ltd) +* [Credit2 Scheduler Development](https://wiki.xenproject.org/wiki/Credit2_Scheduler_Development) +* [Xen Project Scheduler](https://wiki.xenproject.org/wiki/Xen_Project_Schedulers) + +# History + +------------------------------------------------------------------------ +Date Revision Version Notes +---------- -------- -------- ------------------------------------------- +2016-10-10 1 Xen 4.8 Document written +---------- -------- -------- ------------------------------------------- diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c index fe46e80..1f26553 100644 --- a/xen/common/sched_credit2.c +++ b/xen/common/sched_credit2.c @@ -2954,8 +2954,6 @@ csched2_init(struct scheduler *ops) struct csched2_private *prv; printk("Initializing Credit2 scheduler\n"); - printk(" WARNING: This is experimental software in development.\n" \ - " Use at your own risk.\n"); printk(XENLOG_INFO " load_precision_shift: %d\n" XENLOG_INFO " load_window_shift: %d\n"