From patchwork Thu Mar 3 16:48:46 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?SsO8cmdlbiBHcm/Dnw==?= X-Patchwork-Id: 8494041 Return-Path: X-Original-To: patchwork-xen-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 7EF5EC0553 for ; Thu, 3 Mar 2016 16:51:38 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id EF42A20364 for ; Thu, 3 Mar 2016 16:51:36 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BFF2F20382 for ; Thu, 3 Mar 2016 16:51:35 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xen.org with esmtp (Exim 4.84) (envelope-from ) id 1abWR1-0005L2-Il; Thu, 03 Mar 2016 16:49:07 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xen.org with esmtp (Exim 4.84) (envelope-from ) id 1abWR0-0005KO-N7 for xen-devel@lists.xen.org; Thu, 03 Mar 2016 16:49:06 +0000 Received: from [85.158.137.68] by server-6.bemta-3.messagelabs.com id A5/43-23864-10B68D65; Thu, 03 Mar 2016 16:49:05 +0000 X-Env-Sender: jgross@suse.com X-Msg-Ref: server-3.tower-31.messagelabs.com!1457023745!26733249!1 X-Originating-IP: [195.135.220.15] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 8.11; banners=-,-,- X-VirusChecked: Checked Received: (qmail 6417 invoked from network); 3 Mar 2016 16:49:05 -0000 Received: from mx2.suse.de (HELO mx2.suse.de) (195.135.220.15) by server-3.tower-31.messagelabs.com with DHE-RSA-CAMELLIA256-SHA encrypted SMTP; 3 Mar 2016 16:49:05 -0000 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 70872ADD8; Thu, 3 Mar 2016 16:49:04 +0000 (UTC) From: Juergen Gross To: xen-devel@lists.xen.org Date: Thu, 3 Mar 2016 17:48:46 +0100 Message-Id: <1457023730-10997-3-git-send-email-jgross@suse.com> X-Mailer: git-send-email 2.6.2 In-Reply-To: <1457023730-10997-1-git-send-email-jgross@suse.com> References: <1457023730-10997-1-git-send-email-jgross@suse.com> Cc: Juergen Gross , Wei Liu , Stefano Stabellini , George Dunlap , Andrew Cooper , Dario Faggioli , Ian Jackson , David Vrabel , Jan Beulich Subject: [Xen-devel] [PATCH v3 2/6] xen: add hypercall option to override and restore vcpu affinity X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Some hardware (e.g. Dell studio 1555 laptops) require SMIs to be called on physical cpu 0 only. Linux drivers like dcdbas or i8k try to achieve this by pinning the running thread to cpu 0, but in Dom0 this is not enough: the vcpu must be pinned to physical cpu 0 via Xen, too. Add a stable hypercall option SCHEDOP_pin_override to the sched_op hypercall to achieve this. It is taking a physical cpu number as parameter. If pinning is possible (the calling domain has the privilege to make the call and the cpu is available in the domain's cpupool) the calling vcpu is pinned to the specified cpu. The old cpu affinity is saved. To undo the override pinning a negative cpu value is specified. This will restore the original cpu affinity of the vcpu. Cc: George Dunlap Cc: Dario Faggioli Cc: Jan Beulich Cc: Andrew Cooper Cc: Ian Jackson Cc: David Vrabel Cc: Stefano Stabellini Cc: Wei Liu Signed-off-by: Juergen Gross Acked-by: Jan Beulich Acked-by: Dario Faggioli --- V2: - limit operation to hardware domain as suggested by Jan Beulich - some style issues corrected as requested by Jan Beulich - use fixed width types in interface as requested by Jan Beulich - add compat layer checking as requested by Jan Beulich V3: - adjust coding style as requested by Jan Beulich - rename pin_temp to pin_override as requested by Jan Beulich and suggested by David Vrabel - change return type of do_pin_temp() to int as requested by Jan Beulich - swap error checks in do_sched_op() as requested by Jan Beulich - adjust comment in xen/include/public/sched.h as requested by Jan Beulich --- xen/common/compat/schedule.c | 4 ++ xen/common/schedule.c | 93 +++++++++++++++++++++++++++++++++++++++++--- xen/include/public/sched.h | 17 ++++++++ xen/include/xen/sched.h | 1 + xen/include/xlat.lst | 1 + 5 files changed, 111 insertions(+), 5 deletions(-) diff --git a/xen/common/compat/schedule.c b/xen/common/compat/schedule.c index 812c550..8b6e6f1 100644 --- a/xen/common/compat/schedule.c +++ b/xen/common/compat/schedule.c @@ -10,6 +10,10 @@ #define do_sched_op compat_sched_op +#define xen_sched_pin_override sched_pin_override +CHECK_sched_pin_override; +#undef xen_sched_pin_override + #define xen_sched_shutdown sched_shutdown CHECK_sched_shutdown; #undef xen_sched_shutdown diff --git a/xen/common/schedule.c b/xen/common/schedule.c index 13803ec..e57b659 100644 --- a/xen/common/schedule.c +++ b/xen/common/schedule.c @@ -271,6 +271,12 @@ int sched_move_domain(struct domain *d, struct cpupool *c) struct scheduler *old_ops; void *old_domdata; + for_each_vcpu ( d, v ) + { + if ( v->affinity_broken ) + return -EBUSY; + } + domdata = SCHED_OP(c->sched, alloc_domdata, d); if ( domdata == NULL ) return -ENOMEM; @@ -669,6 +675,14 @@ int cpu_disable_scheduler(unsigned int cpu) if ( cpumask_empty(&online_affinity) && cpumask_test_cpu(cpu, v->cpu_hard_affinity) ) { + if ( v->affinity_broken ) + { + /* The vcpu is temporarily pinned, can't move it. */ + vcpu_schedule_unlock_irqrestore(lock, flags, v); + ret = -EBUSY; + break; + } + if (system_state == SYS_STATE_suspend) { cpumask_copy(v->cpu_hard_affinity_saved, @@ -752,14 +766,22 @@ static int vcpu_set_affinity( struct vcpu *v, const cpumask_t *affinity, cpumask_t *which) { spinlock_t *lock; + int ret = 0; lock = vcpu_schedule_lock_irq(v); - cpumask_copy(which, affinity); + if ( v->affinity_broken ) + ret = -EBUSY; + else + { + cpumask_copy(which, affinity); - /* Always ask the scheduler to re-evaluate placement - * when changing the affinity */ - set_bit(_VPF_migrating, &v->pause_flags); + /* + * Always ask the scheduler to re-evaluate placement + * when changing the affinity. + */ + set_bit(_VPF_migrating, &v->pause_flags); + } vcpu_schedule_unlock_irq(lock, v); @@ -771,7 +793,7 @@ static int vcpu_set_affinity( vcpu_migrate(v); } - return 0; + return ret; } int vcpu_set_hard_affinity(struct vcpu *v, const cpumask_t *affinity) @@ -982,6 +1004,50 @@ void watchdog_domain_destroy(struct domain *d) kill_timer(&d->watchdog_timer[i]); } +int vcpu_pin_override(struct vcpu *v, int cpu) +{ + spinlock_t *lock; + int ret = -EINVAL; + + lock = vcpu_schedule_lock_irq(v); + + if ( cpu < 0 ) + { + if ( v->affinity_broken ) + { + cpumask_copy(v->cpu_hard_affinity, v->cpu_hard_affinity_saved); + v->affinity_broken = 0; + set_bit(_VPF_migrating, &v->pause_flags); + ret = 0; + } + } + else if ( cpu < nr_cpu_ids ) + { + if ( v->affinity_broken ) + ret = -EBUSY; + else if ( cpumask_test_cpu(cpu, VCPU2ONLINE(v)) ) + { + cpumask_copy(v->cpu_hard_affinity_saved, v->cpu_hard_affinity); + v->affinity_broken = 1; + cpumask_copy(v->cpu_hard_affinity, cpumask_of(cpu)); + set_bit(_VPF_migrating, &v->pause_flags); + ret = 0; + } + } + + vcpu_schedule_unlock_irq(lock, v); + + domain_update_node_affinity(v->domain); + + if ( v->pause_flags & VPF_migrating ) + { + vcpu_sleep_nosync(v); + vcpu_migrate(v); + } + + return ret; +} + typedef long ret_t; #endif /* !COMPAT */ @@ -1091,6 +1157,23 @@ ret_t do_sched_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg) break; } + case SCHEDOP_pin_override: + { + struct sched_pin_override sched_pin_override; + + ret = -EPERM; + if ( !is_hardware_domain(current->domain) ) + break; + + ret = -EFAULT; + if ( copy_from_guest(&sched_pin_override, arg, 1) ) + break; + + ret = vcpu_pin_override(current, sched_pin_override.pcpu); + + break; + } + default: ret = -ENOSYS; } diff --git a/xen/include/public/sched.h b/xen/include/public/sched.h index 2219696..82c338f 100644 --- a/xen/include/public/sched.h +++ b/xen/include/public/sched.h @@ -118,6 +118,17 @@ * With id != 0 and timeout != 0, poke watchdog timer and set new timeout. */ #define SCHEDOP_watchdog 6 + +/* + * Override the current vcpu affinity by pinning it to one physical cpu or undo + * this override restoring the previous affinity. + * @arg == pointer to sched_pin_override_t structure. + * + * A negative pcpu value will undo a previous pin override and restore the + * previous cpu affinity. + * This call is allowed for the hardware domain only. + */ +#define SCHEDOP_pin_override 7 /* ` } */ struct sched_shutdown { @@ -148,6 +159,12 @@ struct sched_watchdog { typedef struct sched_watchdog sched_watchdog_t; DEFINE_XEN_GUEST_HANDLE(sched_watchdog_t); +struct sched_pin_override { + int32_t pcpu; +}; +typedef struct sched_pin_override sched_pin_override_t; +DEFINE_XEN_GUEST_HANDLE(sched_pin_override_t); + /* * Reason codes for SCHEDOP_shutdown. These may be interpreted by control * software to determine the appropriate action. For the most part, Xen does diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index cc77d70..09fff5f 100644 --- a/xen/include/xen/sched.h +++ b/xen/include/xen/sched.h @@ -811,6 +811,7 @@ int cpu_disable_scheduler(unsigned int cpu); int vcpu_set_hard_affinity(struct vcpu *v, const cpumask_t *affinity); int vcpu_set_soft_affinity(struct vcpu *v, const cpumask_t *affinity); void restore_vcpu_affinity(struct domain *d); +int vcpu_pin_override(struct vcpu *v, int cpu); void vcpu_runstate_get(struct vcpu *v, struct vcpu_runstate_info *runstate); uint64_t get_cpu_idle_time(unsigned int cpu); diff --git a/xen/include/xlat.lst b/xen/include/xlat.lst index fda1137..23befb3 100644 --- a/xen/include/xlat.lst +++ b/xen/include/xlat.lst @@ -104,6 +104,7 @@ ? pmu_data pmu.h ? pmu_params pmu.h ! sched_poll sched.h +? sched_pin_override sched.h ? sched_remote_shutdown sched.h ? sched_shutdown sched.h ? tmem_oid tmem.h