From patchwork Fri Feb 24 16:14:37 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dario Faggioli X-Patchwork-Id: 9590699 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 7DDFA604A2 for ; Fri, 24 Feb 2017 16:16:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8689D286E0 for ; Fri, 24 Feb 2017 16:16:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7B67C286FD; Fri, 24 Feb 2017 16:16:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id F2207286E0 for ; Fri, 24 Feb 2017 16:16:55 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1chIW9-0005MU-8z; Fri, 24 Feb 2017 16:14:49 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1chIW7-0005K6-IZ for xen-devel@lists.xenproject.org; Fri, 24 Feb 2017 16:14:47 +0000 Received: from [85.158.143.35] by server-9.bemta-6.messagelabs.com id FD/E2-27165-6FB50B85; Fri, 24 Feb 2017 16:14:46 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrHIsWRWlGSWpSXmKPExsXitHSDve636A0 RBufvi1h83zKZyYHR4/CHKywBjFGsmXlJ+RUJrBknd8xmL3ioVXHp2GP2Bsb9Kl2MnBwSAiES G94fYOxi5ODgFTCU2LzVGCQsLFAjMffWPkYQm03AQOLNjr2sILaIgJ9E2+teli5GLg5mgS2ME nMnLmYBSbAIqEosutrKDmJzCthLnP60lRWkSEhgBqPEx2N7wCbxC0hK3PrykRnEZhaolpjx/A QTxBHaEj17f4DFeQUEJU7OfAI2VEhATWLG3MusExj5ZiFpmYWkDCKuKdG6/Tc7hK0tsWzha2Y I21Zi3br3UDU2EpuuLmCEsOUltr+dw7yAkX0Vo0ZxalFZapGusZFeUlFmekZJbmJmjq6hgZle bmpxcWJ6ak5iUrFecn7uJkZgODMAwQ7G0+sCDzFKcjApifKGBm+IEOJLyk+pzEgszogvKs1JL T7EKMPBoSTBeygKKCdYlJqeWpGWmQOMLJi0BAePkghvfiRQmre4IDG3ODMdInWKUVFKnDcQpE 8AJJFRmgfXBovmS4yyUsK8jECHCPEUpBblZpagyr9iFOdgVBLm7QSZwpOZVwI3/RXQYiagxZb Oa0EWlyQipKQaGCdlOymlKny/fPTr6l0zj1w5fi7i9xJtBZWV34uyXzCuC0jkOnqf7QjLXIat IndiSzkWua9deF1XS2LP34nP/jBdZdZIMm9Xec68imHBLq6DOufabn05G/wuU9/lxe6/dUfid c73lp7p1rfe+47p7Lu8mhD/rzGfXXbbvWxIE7s/Nahx2/2WJWVKLMUZiYZazEXFiQCxVbkD4Q IAAA== X-Env-Sender: prvs=221d86bfe=dario.faggioli@citrix.com X-Msg-Ref: server-3.tower-21.messagelabs.com!1487952884!58811527!1 X-Originating-IP: [66.165.176.63] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni42MyA9PiAzMDYwNDg=\n, received_headers: No Received headers X-StarScan-Received: X-StarScan-Version: 9.2.3; banners=-,-,- X-VirusChecked: Checked Received: (qmail 50707 invoked from network); 24 Feb 2017 16:14:46 -0000 Received: from smtp02.citrix.com (HELO SMTP02.CITRIX.COM) (66.165.176.63) by server-3.tower-21.messagelabs.com with RC4-SHA encrypted SMTP; 24 Feb 2017 16:14:46 -0000 X-IronPort-AV: E=Sophos;i="5.35,201,1484006400"; d="asc'?scan'208";a="418410923" Message-ID: <1487952877.5548.26.camel@citrix.com> From: Dario Faggioli To: Jan Beulich , Andrew Cooper Date: Fri, 24 Feb 2017 17:14:37 +0100 In-Reply-To: <58AD5DDA020000780013C9ED@prv-mh.provo.novell.com> References: <4b6337e3-74b3-8ecd-d7a6-aa8a451fb8d3@citrix.com> <58AD5DDA020000780013C9ED@prv-mh.provo.novell.com> Organization: Citrix Inc. X-Mailer: Evolution 3.20.5 (3.20.5-1.fc24) MIME-Version: 1.0 Cc: George Dunlap , xen-devel , osstest service owner , Juergen Gross Subject: [Xen-devel] RFC/PATCH: xen: race during domain destruction [Re: [xen-4.7-testing test] 105948: regressions - FAIL] X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP [Adding Juergen] On Wed, 2017-02-22 at 01:46 -0700, Jan Beulich wrote: > > > > On 22.02.17 at 01:02, wrote: > > (XEN) Xen call trace: > > (XEN)    [] > > sched_credit2.c#vcpu_is_migrateable+0x22/0x9a > > (XEN)    [] > > sched_credit2.c#csched2_schedule+0x823/0xb4e > > (XEN)    [] schedule.c#schedule+0x108/0x609 > > (XEN)    [] softirq.c#__do_softirq+0x7f/0x8a > > (XEN)    [] do_softirq+0x13/0x15 > > (XEN)    [] domain.c#idle_loop+0x55/0x62 > > (XEN) > > (XEN) > > (XEN) **************************************** > > (XEN) Panic on CPU 14: > > (XEN) Assertion 'd->cpupool != NULL' failed at > > ...5948.build-amd64/xen/xen/include/xen/sched-if.h:200 > > (XEN) **************************************** > > (XEN) > > (XEN) Manual reset required ('noreboot' specified) > > > > I am guessing the most recent credit2 backports weren't quite so > > safe? > Well, what I'd say we're facing is the surfacing of a latent bug. > However, comparing with the staging version of the file > (which is heavily different), the immediate code involved here isn't > all that different, so I wonder whether (a) this is a problem on > staging too or (b) we're missing another backport. Dario? > So, according to my investigation, this is a genuine race. It affects this branch as well as staging, but it manifests less frequently (or, I should say, very rarely) in the latter. The problem is that the Credit2's load balancer operates not only on runnable vCPUs, but also on blocked, sleeping, and paused ones (and that's by design). In this case, the original domain is in the process of being destroyed,  after migration completed, and reaches the point where, within domain_destroy(), we call cpupool_rm_domain(). This remove the domain from any cpupool, and sets d->cpupool = NULL. Then, on another pCPU --since the vCPUs of the domain are still around (until we call sched_destroy_vcpu(), which happens much later-- and they also are still assigned to a Credit2 runqueue, balance_load() picks up one of them for moving to another runqueue, and things explode when we realize that the vCPU is actually out of any pool! So, I've thought quite a bit of how to solve this. Possibilities are to act at the Credit2 level, or outside of it. I drafted a couple of solutions only affecting sched_credit2.c, but could not be satisfied with the results. And that's because I ultimately think it should be safe for a scheduler that it can play with a vCPU that it can reach out to, and that means the vCPU must be in a pool. And that's why I came up with the patch below. This is a draft and is on top of staging-4.7. I will properly submit it against staging, if you agree with me it's an ok thing to do. Basically, I anticipate a little bit calling sched_destroy_vcpu(), so that it happens before cpupool_rm_domain(). This ensures that vCPUs have valid cpupool information until the very last moment that they are accessible from a scheduler. --- Let me know. Thanks and Regards, Dario diff --git a/xen/common/domain.c b/xen/common/domain.c index 45273d4..4db7750 100644 --- a/xen/common/domain.c +++ b/xen/common/domain.c @@ -643,7 +643,10 @@ int domain_kill(struct domain *d) if ( cpupool_move_domain(d, cpupool0) ) return -ERESTART; for_each_vcpu ( d, v ) + { unmap_vcpu_info(v); + sched_destroy_vcpu(v); + } d->is_dying = DOMDYING_dead; /* Mem event cleanup has to go here because the rings * have to be put before we call put_domain. */ @@ -807,7 +810,6 @@ static void complete_domain_destroy(struct rcu_head *head) continue; tasklet_kill(&v->continue_hypercall_tasklet); vcpu_destroy(v); - sched_destroy_vcpu(v); destroy_waitqueue_vcpu(v); } ---