From patchwork Fri Aug 18 15:51:01 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dario Faggioli X-Patchwork-Id: 9909529 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 84D3A603B5 for ; Fri, 18 Aug 2017 15:53:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7392428CFB for ; Fri, 18 Aug 2017 15:53:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6847228CFD; Fri, 18 Aug 2017 15:53:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,RCVD_IN_SORBS_SPAM,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 82EF728D05 for ; Fri, 18 Aug 2017 15:53:38 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dijYB-0008LC-6u; Fri, 18 Aug 2017 15:51:07 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dijYA-0008L1-2b for xen-devel@lists.xenproject.org; Fri, 18 Aug 2017 15:51:06 +0000 Received: from [85.158.137.68] by server-11.bemta-3.messagelabs.com id 40/C7-01724-9EC07995; Fri, 18 Aug 2017 15:51:05 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpmleJIrShJLcpLzFFi42K5GNpwRPcFz/R Ig+5dAhbft0xmcmD0OPzhCksAYxRrZl5SfkUCa8bhKQ8ZCxY6V0xc+4WpgXGtdhcjJ4eQwHRG iZMtWl2MXBwsAmtYJe4eXs4I4kgIXGKV2HlxM3sXIweQEydxbnEiSIOEQI3EuacrWSGaVSRub l/FBFIvJPCTUeL7pWVMIAlhAT2JI0d/sEPYXhKvLk5lA7HZBAwk3uzYC9YsIqAkcW/VZLBmZo EFTBInv3czgiRYBFQlVn+dCmbzCnhLLJ7ZB9bACWS//fSWBWKzl8TjHRPBlokKyEmsvNzCClE vKHFy5hMWkKOZBTQl1u/SBwkzC8hLbH87h3kCo8gsJFWzEKpmIalawMi8ilG9OLWoLLVI10Qv qSgzPaMkNzEzR9fQwFgvN7W4ODE9NScxqVgvOT93EyMw+BmAYAdj4xenQ4ySHExKory/Z02JF OJLyk+pzEgszogvKs1JLT7EqMHBITDh7NzpTFIsefl5qUoSvNXc0yOFBItS01Mr0jJzgPEJUy rBwaMkwpsOkuYtLkjMLc5Mh0idYjTmuHJl3RcmjikHtn9hEgKbJCXO+xKkVACkNKM0D24QLG1 cYpSVEuZlBDpTiKcgtSg3swRV/hWjOAejkjCvKTAJCfFk5pXA7XsFdAoT0CmGrdNATilJREhJ NTAu38jafJ8nYeaJlk7N5MOTb3AFv7qfcveYTaJLWfyCmXOmNDMdT1/iU9jeevy7wKK9x/Y+9 AieuMuvyEhotvCva0tMrFNbc5yPVpXmXLnmp9nK6KGaPq8zrkLL8eHZGzUdNu0BkrscDDQuBx e0XlrZXS1cs+b6N1fWeUcfsuXtZbsZYRXQsUSJpTgj0VCLuag4EQANJv7PFgMAAA== X-Env-Sender: raistlin.df@gmail.com X-Msg-Ref: server-9.tower-31.messagelabs.com!1503071464!55145225!1 X-Originating-IP: [209.85.128.196] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.4.45; banners=-,-,- X-VirusChecked: Checked Received: (qmail 37004 invoked from network); 18 Aug 2017 15:51:04 -0000 Received: from mail-wr0-f196.google.com (HELO mail-wr0-f196.google.com) (209.85.128.196) by server-9.tower-31.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 18 Aug 2017 15:51:04 -0000 Received: by mail-wr0-f196.google.com with SMTP id 49so7739847wrw.5 for ; Fri, 18 Aug 2017 08:51:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=op7i2QMLv+iba9ioJJ2Ryp+XXksQRYlvqAVR5DSna4I=; b=njw8OJDNJA812dbilkgCU2c2GaDFB9F7hBobOZH5MrGtco0DYlgrKguOT2lgN0Tsvr DRbR34pI8YN0dZVE3gIjfavIQI41dgQc92kHvusBjEk5R5kS2FhwyowIjCVffVawiXh6 cNfgBQ6bRLuYhgDk3AZsfMwLejS/ez2pOzv6fOtUiFYZBZL3WHYNobGkoX88zAEjgG9P xn83p3nL1m5ieGbprj2PBP7LzjmjKUX1nv8JFetRHAGr2kToD0hJ0sOcf7ZZ2R0JvvIV d5MlJN7GfBGkSZkbzyPTOTRfuVDKzk58UhhlCxeu4Kl3E78jsWaOamKdivxVPon9iWrQ yrQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=op7i2QMLv+iba9ioJJ2Ryp+XXksQRYlvqAVR5DSna4I=; b=JrqL5qLMk/VV9kt2yCR3XSxTb1CkhXzlTLfIgmVpiInvnx4iOnXjBrYM7oK3vuLHbw iHM2MrVxfdUBL/Y1veNXRzx3Wf26cG43G8alQjSuuKu7Vs2wgoWTQ4/0v23E5ZwPhV2h 2JBINo7E42H+y+PUQuEM3lka+RnPdqE7hHbXOJPxB9BTpA86J3LfSzS6AYrz1iHfB8vl +WMa3UwGqowxEB0N79/wvxO3Na8Th5nE3Kv7gKCrou8308s20DiaC+530ZghVqqanb3t bthN5YWBa8c81nGM1DYrs2SBgYaz85j9ykBMM4Ol4bFvts0fJAJRBi/lkpOY8PwN8Ofb vnYQ== X-Gm-Message-State: AHYfb5iwdhbCfo5EQLhqPXntZvs9mv3vM98ac4mJ3ocA6sUkm/z+d4lt ldMT/HGQvMZrsA== X-Received: by 10.223.168.110 with SMTP id l101mr5553367wrc.251.1503071463794; Fri, 18 Aug 2017 08:51:03 -0700 (PDT) Received: from Solace.fritz.box ([80.66.223.3]) by smtp.gmail.com with ESMTPSA id a134sm1117016wma.47.2017.08.18.08.51.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Aug 2017 08:51:03 -0700 (PDT) From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Fri, 18 Aug 2017 17:51:01 +0200 Message-ID: <150307146099.6642.12653443547432533087.stgit@Solace.fritz.box> In-Reply-To: <150307081385.6642.6516202758428761422.stgit@Solace.fritz.box> References: <150307081385.6642.6516202758428761422.stgit@Solace.fritz.box> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Cc: Wei Liu , George Dunlap , Andrew Cooper , Ian Jackson , Anshul Makkar , Jan Beulich Subject: [Xen-devel] [PATCH v2 2/4] xen: credit2: allow to set and get utilization cap X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP As cap is already present in Credit1, as a parameter, all the wiring is there already for it to be percolate down to csched2_dom_cntl() too. In this commit, we actually deal with it, and implement setting, changing or disabling the cap of a domain. Signed-off-by: Dario Faggioli Reviewed-by: George Dunlap --- Cc: George Dunlap Cc: Ian Jackson Cc: Wei Liu Cc: Andrew Cooper Cc: Jan Beulich Cc: Konrad Rzeszutek Wilk Cc: Anshul Makkar --- Changes from v1: - check that cap is below 100*nr_vcpus; - do multiplication first when computing the domain's budget, given the cap; - when disabling cap, take the budget lock for manipulating the list of parked vCPUs. Things would have been safe without it, but it's just more linear, more robust and more future-proof, to "do thing properly". --- xen/common/sched_credit2.c | 129 +++++++++++++++++++++++++++++++++++++++++-- xen/include/public/domctl.h | 1 2 files changed, 125 insertions(+), 5 deletions(-) diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c index 69a7679..ce70224 100644 --- a/xen/common/sched_credit2.c +++ b/xen/common/sched_credit2.c @@ -2772,30 +2772,35 @@ csched2_dom_cntl( struct csched2_dom * const sdom = csched2_dom(d); struct csched2_private *prv = csched2_priv(ops); unsigned long flags; + struct vcpu *v; int rc = 0; /* * Locking: * - we must take the private lock for accessing the weights of the - * vcpus of d, + * vcpus of d, and/or the cap; * - in the putinfo case, we also need the runqueue lock(s), for * updating the max waight of the runqueue(s). + * If changing the cap, we also need the budget_lock, for updating + * the value of the domain budget pool (and the runqueue lock, + * for adjusting the parameters and rescheduling any vCPU that is + * running at the time of the change). */ switch ( op->cmd ) { case XEN_DOMCTL_SCHEDOP_getinfo: read_lock_irqsave(&prv->lock, flags); op->u.credit2.weight = sdom->weight; + op->u.credit2.cap = sdom->cap; read_unlock_irqrestore(&prv->lock, flags); break; case XEN_DOMCTL_SCHEDOP_putinfo: + write_lock_irqsave(&prv->lock, flags); + /* Weight */ if ( op->u.credit2.weight != 0 ) { - struct vcpu *v; int old_weight; - write_lock_irqsave(&prv->lock, flags); - old_weight = sdom->weight; sdom->weight = op->u.credit2.weight; @@ -2813,9 +2818,123 @@ csched2_dom_cntl( vcpu_schedule_unlock(lock, svc->vcpu); } + } + /* Cap */ + if ( op->u.credit2.cap != 0 ) + { + /* Cap is only valid if it's below 100 * nr_of_vCPUS */ + if ( op->u.credit2.cap > 100 * sdom->nr_vcpus ) + { + rc = -EINVAL; + break; + } + + spin_lock(&sdom->budget_lock); + sdom->tot_budget = (CSCHED2_BDGT_REPL_PERIOD * op->u.credit2.cap); + sdom->tot_budget /= 100; + spin_unlock(&sdom->budget_lock); + + if ( sdom->cap == 0 ) + { + /* + * We give to the domain the budget to which it is entitled, + * and queue its first replenishment event. + * + * Since cap is currently disabled for this domain, we + * know no vCPU is messing with the domain's budget, and + * the replenishment timer is still off. + * For these reasons, it is safe to do the following without + * taking the budget_lock. + */ + sdom->budget = sdom->tot_budget; + sdom->next_repl = NOW() + CSCHED2_BDGT_REPL_PERIOD; + set_timer(sdom->repl_timer, sdom->next_repl); + + /* + * Now, let's enable budget accounting for all the vCPUs. + * For making sure that they will start to honour the domain's + * cap, we set their budget to 0. + * This way, as soon as they will try to run, they will have + * to get some budget. + * + * For the vCPUs that are already running, we trigger the + * scheduler on their pCPU. When, as a consequence of this, + * csched2_schedule() will run, it will figure out there is + * no budget, and the vCPU will try to get some (and be parked, + * if there's none, and we'll switch to someone else). + */ + for_each_vcpu ( d, v ) + { + struct csched2_vcpu *svc = csched2_vcpu(v); + spinlock_t *lock = vcpu_schedule_lock(svc->vcpu); + + if ( v->is_running ) + { + unsigned int cpu = v->processor; + struct csched2_runqueue_data *rqd = c2rqd(ops, cpu); + + ASSERT(curr_on_cpu(cpu) == v); + + /* + * We are triggering a reschedule on the vCPU's + * pCPU. That will run burn_credits() and, since + * the vCPU is capped now, it would charge all the + * execution time of this last round as budget as + * well. That will make the vCPU budget go negative, + * potentially by a large amount, and it's unfair. + * + * To avoid that, call burn_credit() here, to do the + * accounting of this current running instance now, + * with budgetting still disabled. This does not + * prevent some small amount of budget being charged + * to the vCPU (i.e., the amount of time it runs from + * now, to when scheduling happens). The budget will + * also go below 0, but a lot less than how it would + * if we don't do this. + */ + burn_credits(rqd, svc, NOW()); + __cpumask_set_cpu(cpu, &rqd->tickled); + ASSERT(!cpumask_test_cpu(cpu, &rqd->smt_idle)); + cpu_raise_softirq(cpu, SCHEDULE_SOFTIRQ); + } + svc->budget = 0; + vcpu_schedule_unlock(lock, svc->vcpu); + } + } + + sdom->cap = op->u.credit2.cap; + } + else if ( sdom->cap != 0 ) + { + LIST_HEAD(parked); + + stop_timer(sdom->repl_timer); + + /* Disable budget accounting for all the vCPUs. */ + for_each_vcpu ( d, v ) + { + struct csched2_vcpu *svc = csched2_vcpu(v); + spinlock_t *lock = vcpu_schedule_lock(svc->vcpu); + + svc->budget = STIME_MAX; + + vcpu_schedule_unlock(lock, svc->vcpu); + } + sdom->cap = 0; + /* + * We are disabling the cap for this domain, which may have + * vCPUs waiting for a replenishment, so we unpark them all. + * Note that, since we have already disabled budget accounting + * for all the vCPUs of the domain, no currently running vCPU + * will be added to the parked vCPUs list any longer. + */ + spin_lock(&sdom->budget_lock); + list_splice_init(&sdom->parked_vcpus, &parked); + spin_unlock(&sdom->budget_lock); - write_unlock_irqrestore(&prv->lock, flags); + unpark_parked_vcpus(ops, &parked); } + write_unlock_irqrestore(&prv->lock, flags); break; default: rc = -EINVAL; diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h index 0669c31..10c0015 100644 --- a/xen/include/public/domctl.h +++ b/xen/include/public/domctl.h @@ -355,6 +355,7 @@ typedef struct xen_domctl_sched_credit { typedef struct xen_domctl_sched_credit2 { uint16_t weight; + uint16_t cap; } xen_domctl_sched_credit2_t; typedef struct xen_domctl_sched_rtds {