From patchwork Fri Mar 18 19:04:39 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dario Faggioli X-Patchwork-Id: 8623281 Return-Path: X-Original-To: patchwork-xen-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 5E3779F6E1 for ; Fri, 18 Mar 2016 19:07:04 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 5569B20306 for ; Fri, 18 Mar 2016 19:07:03 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 521FB2022A for ; Fri, 18 Mar 2016 19:07:02 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1agzhU-0000CE-W0; Fri, 18 Mar 2016 19:04:44 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1agzhT-0000BR-LO for xen-devel@lists.xenproject.org; Fri, 18 Mar 2016 19:04:43 +0000 Received: from [85.158.137.68] by server-13.bemta-3.messagelabs.com id 27/64-03443-A415CE65; Fri, 18 Mar 2016 19:04:42 +0000 X-Env-Sender: raistlin.df@gmail.com X-Msg-Ref: server-3.tower-31.messagelabs.com!1458327882!30103315!1 X-Originating-IP: [74.125.82.67] X-SpamReason: No, hits=0.7 required=7.0 tests=BODY_RANDOM_LONG, RCVD_ILLEGAL_IP X-StarScan-Received: X-StarScan-Version: 8.11; banners=-,-,- X-VirusChecked: Checked Received: (qmail 12977 invoked from network); 18 Mar 2016 19:04:42 -0000 Received: from mail-wm0-f67.google.com (HELO mail-wm0-f67.google.com) (74.125.82.67) by server-3.tower-31.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 18 Mar 2016 19:04:42 -0000 Received: by mail-wm0-f67.google.com with SMTP id p65so8545399wmp.1 for ; Fri, 18 Mar 2016 12:04:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=vYVhsXtUCPDZ5ohuv7YicohkGi92BK5SeMpl3WyOx/U=; b=cJpQ4YT67TCk+szoWzyzFH3Fj/Jo4yItXy3A/17oiyIP3YPQf0gBhk8Wf2q2mcMwU2 lZKWFIHg0LUi2tc9MO7lXz+oWBniSpVsHPd6svMJdFaaAMQp/OGo462k4vEJCb0rzaxw y1zVCq336g2TNaSOeggDnrUX32ONcziStF7FnbhyV9T15gNIVDyY138Wschxc3ib9DCw GzjO8RQo+iqSNvkqfK3sIkOd6sC2tH+uMFPu08PgUYlSKCM1iKcDLIq20xaWQoJlLFwb UF+++R7Ovbp3VEoY5jr84XStW4VcF6POfL9RKWvSCjD9kMFG0QIsf5bRO99yNp/ozjcS FF9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=vYVhsXtUCPDZ5ohuv7YicohkGi92BK5SeMpl3WyOx/U=; b=WLXdQzbxdDt2adPoUlSC/1hUoc8SqBQyCxdFvcSSVvyLlp3pGKRnSJVrkOs0ZObf+c nhXfydxZo0QsboLNslZejDisiBU/3/3MGhyDNbMWJvjkq/66nqc/uPi0/818FdUjvEpZ yGe8Mqn5lK02NrYJi4c11hM0parTv1X4C/hshZC8mtp9kimOfB3ibuoYElIfhPtpK4vb iSC7eb7IHaczAeIZCcS9UFYZhqYsGn4XxZO6eNuKKkntuR2xKa3plX46ycMnAB/k6O6s uIZY3wkZzYuMAUcxi2xoYEGIHE6E4SGIYt56H9js2eZYfjDOH+iy4xetEaWZ0ANKJLqw ieHQ== X-Gm-Message-State: AD7BkJKz2KVbrGOsQut75kKWGe5LxIElBulqkh27nJGqdaqVn0H9b9Tn2gP64WLg2plqvQ== X-Received: by 10.28.93.79 with SMTP id r76mr995190wmb.32.1458327881805; Fri, 18 Mar 2016 12:04:41 -0700 (PDT) Received: from Solace.station (net-2-35-170-8.cust.vodafonedsl.it. [2.35.170.8]) by smtp.gmail.com with ESMTPSA id pd1sm13410143wjb.19.2016.03.18.12.04.40 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Mar 2016 12:04:41 -0700 (PDT) From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Fri, 18 Mar 2016 20:04:39 +0100 Message-ID: <20160318190439.8117.93021.stgit@Solace.station> In-Reply-To: <20160318185524.8117.74837.stgit@Solace.station> References: <20160318185524.8117.74837.stgit@Solace.station> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Cc: George Dunlap Subject: [Xen-devel] [PATCH 06/16] xen: sched: prepare a .switch_sched hook for Credit1 X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Spam-Status: No, score=-4.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED, T_DKIM_INVALID, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In fact, right now, if we switch cpu X from, say, Credit2 to Credit, we do: schedule_cpu_switch(x, csched2 --> csched): //scheduler[x] is csched2 //schedule_lock[x] is csched2_lock csched_alloc_pdata(x) csched_init_pdata(x) pcpu_schedule_lock(x) ----> takes csched2_lock scheduler[X] = csched pcpu_schedule_unlock(x) --> unlocks csched2_lock [*] csched2_free_pdata(x) pcpu_schedule_lock(x) --> takes csched2_lock schedule_lock[x] = csched_lock spin_unlock(csched2_lock) So, if anything scheduling related and involving CPU X happens, at time [*], we will: - take csched2_lock, - operate on Credit1 functions and data structures, which is no good! The problem arises because there is a window where the scheduler is already the new one, but the lock is still the old one. And, in this specific transition this is due to the fact that it is only csched2_free_pdata() that (re)sets the lock in the way that Credit1 needs it, instead of Credit1 itself doing it on its own. This patch, therefore, introduce a new hook in the scheduler interface, called switch_sched, meant at being used when switching scheduler on a CPU, and implements it for Credit1. This allows to do the various operations in the correct order and under the protection of the best suited (set of) lock(s). It is necessary to add the hook (as compare to keep doing things in generic code), because different schedulers may have different locking scheme. See, for instance, the different lock nesting rules in Credit1 and Credit2. Signed-off-by: Dario Faggioli --- Cc: George Dunlap --- xen/common/sched_credit.c | 38 ++++++++++++++++++++++++++++++++++++++ xen/include/xen/sched-if.h | 3 +++ 2 files changed, 41 insertions(+) diff --git a/xen/common/sched_credit.c b/xen/common/sched_credit.c index 4488d7c..929ba9c 100644 --- a/xen/common/sched_credit.c +++ b/xen/common/sched_credit.c @@ -583,6 +583,43 @@ csched_init_pdata(const struct scheduler *ops, void *pdata, int cpu) spin_unlock_irqrestore(&prv->lock, flags); } +/* Change the scheduler of cpu to us (Credit). */ +static void +csched_switch_sched(struct scheduler *ops, unsigned int cpu, + void *pdata, void *vdata) +{ + struct schedule_data *sd = &per_cpu(schedule_data, cpu); + struct csched_private *prv = CSCHED_PRIV(ops); + struct csched_vcpu *svc = vdata; + spinlock_t * old_lock; + + ASSERT(svc && is_idle_vcpu(svc->vcpu)); + + /* + * We may be acquiring the lock of another scheduler here (the one cpu + * still belongs to when calling this function). That is ok as, anyone + * trying to schedule on this cpu will block until when we release that + * lock (bottom of this function). When unblocked --because of the loop + * implemented by schedule_lock() functions-- he will notice the lock + * changed, and acquire ours before being able to proceed. + */ + old_lock = pcpu_schedule_lock_irq(cpu); + + idle_vcpu[cpu]->sched_priv = vdata; + + spin_lock(&prv->lock); + init_pdata(prv, pdata, cpu); + spin_unlock(&prv->lock); + + per_cpu(scheduler, cpu) = ops; + per_cpu(schedule_data, cpu).sched_priv = pdata; + /* (Re?)route the lock to the per pCPU lock. */ + sd->schedule_lock = &sd->_lock; + + /* _Not_ pcpu_schedule_unlock(): schedule_lock may have changed! */ + spin_unlock_irq(old_lock); +} + #ifndef NDEBUG static inline void __csched_vcpu_check(struct vcpu *vc) @@ -2064,6 +2101,7 @@ static const struct scheduler sched_credit_def = { .alloc_pdata = csched_alloc_pdata, .init_pdata = csched_init_pdata, .free_pdata = csched_free_pdata, + .switch_sched = csched_switch_sched, .alloc_domdata = csched_alloc_domdata, .free_domdata = csched_free_domdata, diff --git a/xen/include/xen/sched-if.h b/xen/include/xen/sched-if.h index 560cba5..0aa00de 100644 --- a/xen/include/xen/sched-if.h +++ b/xen/include/xen/sched-if.h @@ -138,6 +138,9 @@ struct scheduler { void (*free_domdata) (const struct scheduler *, void *); void * (*alloc_domdata) (const struct scheduler *, struct domain *); + void (*switch_sched) (struct scheduler *, unsigned int, + void *, void *); + int (*init_domain) (const struct scheduler *, struct domain *); void (*destroy_domain) (const struct scheduler *, struct domain *);