From patchwork Fri Oct 21 13:49:30 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dario Faggioli X-Patchwork-Id: 9389191 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 6DD00607D0 for ; Fri, 21 Oct 2016 13:52:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5EAAC2A1EF for ; Fri, 21 Oct 2016 13:52:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 529802A1F3; Fri, 21 Oct 2016 13:52:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,RCVD_IN_SORBS_SPAM,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 2ECF52A1EF for ; Fri, 21 Oct 2016 13:52:46 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bxaCX-0003cT-4f; Fri, 21 Oct 2016 13:49:37 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bxaCV-0003cN-LR for xen-devel@lists.xenproject.org; Fri, 21 Oct 2016 13:49:35 +0000 Received: from [85.158.143.35] by server-8.bemta-6.messagelabs.com id BC/3C-28497-EEC1A085; Fri, 21 Oct 2016 13:49:34 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrAIsWRWlGSWpSXmKPExsVyMfTOYd13Mlw RBk+vCll83zKZyYHR4/CHKywBjFGsmXlJ+RUJrBm/VqxjLehSqPi76wJTA+N8iS5GLg4hgZmM ErtWT2AEcVgEprJKLNwxCcyRENjIKrH+SjdzFyMnkBMjcef7BaAEB5BdJdF7NxkkLCSgInFz+ yomiEm/GCXWbOxkA0kIC+hJHDn6gx2kXlggQeLWCQ+QMJuAgcSbHXtZQWwRASWJe6smg/UyCx xklNg8uYUFJMEioCpx7cBkMJtXwFvi/aoVTCC2qICcxMrLLawQcUGJkzOfsIDMZxbQlFi/Sx8 kzCwgL7H97RzmCYxCs5BUzUKomoWkagEj8ypGjeLUorLUIl0jU72kosz0jJLcxMwcXUMDM73c 1OLixPTUnMSkYr3k/NxNjMBwZgCCHYyrFgQeYpTkYFIS5V00kTNCiC8pP6UyI7E4I76oNCe1+ BCjDAeHkgTvLWmuCCHBotT01Iq0zBxgZMGkJTh4lER4jUHSvMUFibnFmekQqVOMxhxbfl9by8 Sxbeq9tUxCLHn5ealS4rwJIKUCIKUZpXlwg2ARf4lRVkqYlxHoNCGegtSi3MwSVPlXjOIcjEr CvCzA9CHEk5lXArfvFdApTECn1KRxgJxSkoiQkmpgnPnp+yGTMIfQGBf+1CSt3pUcl372HD0u LnJqWVS9n9Vd793L92Rbz/SqqFsrLHBlqUqtR6Lw5giL+syVG+vOPj8pqed2+a2q1R1uZ8Y7X 3rX1N2IFEuI4JYuXHf+/SzN/aF9890W2H8Wd5C79f717O6t+v+6u29675B+IMUycwpb3s2z5w TslFiKMxINtZiLihMBFMMSffMCAAA= X-Env-Sender: raistlin.df@gmail.com X-Msg-Ref: server-8.tower-21.messagelabs.com!1477057773!39216792!1 X-Originating-IP: [209.85.220.195] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.0.13; banners=-,-,- X-VirusChecked: Checked Received: (qmail 48274 invoked from network); 21 Oct 2016 13:49:34 -0000 Received: from mail-qk0-f195.google.com (HELO mail-qk0-f195.google.com) (209.85.220.195) by server-8.tower-21.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 21 Oct 2016 13:49:34 -0000 Received: by mail-qk0-f195.google.com with SMTP id z190so7199233qkc.3 for ; Fri, 21 Oct 2016 06:49:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:subject:from:to:cc:date:message-id:user-agent:mime-version :content-transfer-encoding; bh=yvRR6WcxPKpcQmAGV1QwQg1qwbaSvxQlW/5LdrRbamw=; b=gH5z8f6dizEr2cZgmj4PHKne/qTYhXK2mfcDJ8CTI9ytxXAfR4Vo0Wl88r77wwAXrZ FbHWgR8/09y3raKVqT2U9GpqKZIcNoubYIfJdwgWFgEongvvLjTtyq+RxS5R0DvPBkdS iCyn6KRPuIzWB3Rqra0zRhFZAvhSig18+hkQCW+HHVuwcP6CbWuv0mz6FIq2RWLSShqM ufHl+/w9SC28iTkFlIT2+AHq8ZQRHXJKvYH2EGLFAiH5Fp8caK2jkaPD9AwPDNzdtC5S Px4OHHNsIj1qhPNS3OqHZ3/Or7cZOw9vzjD3H3XLqyWppdbF+EMftSh2pOpjT3QDVzEn xpUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :user-agent:mime-version:content-transfer-encoding; bh=yvRR6WcxPKpcQmAGV1QwQg1qwbaSvxQlW/5LdrRbamw=; b=NHoICJ049I06suNDiGJqtK3vtJbK0ozGPHz+XvUaiBYe14r9ZQbqxj/lGcBfUrYoXR RdOtzYFBKGHARqO1fnLYw6HPY4IdM3/nmEczfTaWyYwtA+7w99rYLJTiRxD70oncVHp+ SgrKDX5xv9od26LWAIWqq4GmGcf4ZcMdRq6WOpm904bB3G3voSxh4tGHmqFV7mf7CQb5 U++KpVJH15IerWOWUWzQWSfyQC8XYqkYoFJs7POhrTwOqEHAwkEPPPICbAhT09V6t3FT 9EAnTM/5GZX3xcNEjhAzp0GD3WIL6Gg6MZVNl2S3dvAy/TtULAeE7BIr8371Ftzu7KRU xS7g== X-Gm-Message-State: ABUngvfqkOn0I9G4n5I9rWy/un47BEiRWedhZ41CdGj0W4/ZWEtF3/t0vk/tyNNV9CweSw== X-Received: by 10.194.116.225 with SMTP id jz1mr865548wjb.224.1477057773092; Fri, 21 Oct 2016 06:49:33 -0700 (PDT) Received: from Solace.fritz.box ([80.66.223.32]) by smtp.gmail.com with ESMTPSA id w133sm4276558wmf.12.2016.10.21.06.49.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Oct 2016 06:49:32 -0700 (PDT) From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Fri, 21 Oct 2016 15:49:30 +0200 Message-ID: <147705777085.26968.8266354656960221924.stgit@Solace.fritz.box> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Cc: Juergen Gross , Wei Liu , Anshul Makkar , Ian Jackson , George Dunlap Subject: [Xen-devel] [PATCH v2] libxl: avoid considering pCPUs outside of the cpupool during NUMA placement X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP During NUMA automatic placement, the information of how many vCPUs can run on what NUMA nodes is used, in order to spread the load as evenly as possible. Such information is derived from vCPU hard and soft affinity, but that is not enough. In fact, affinity can be set to be a superset of the pCPUs that belongs to the cpupool in which a domain is but, of course, the domain will never run on pCPUs outside of its cpupool. Take this into account in the placement algorithm. Signed-off-by: Dario Faggioli Reported-by: George Dunlap Reviewed-by: Juergen Gross Reviewed-by: Wei Liu --- Cc: Ian Jackson Cc: Wei Liu Cc: George Dunlap Cc: Anshul Makkar --- Changes from v1: * moved libxl_cpupoolinfo_init() inside the loop, as requested; * removed the pointless vinfo=NULL assignment _at_the_end_ of the loop (and keep the one at the beginning, which is necssary). --- Wei, this is bugfix, so I think it should go in 4.8. Ian, this is bugfix, so I think it is a backporting candidate. Also, note that this function does not respect the libxl coding style, as far as error handling is concerned. However, given that I'm asking for it to go in now and to be backported, I've tried to keep the changes to the minimum. I'm up for a follow up patch for 4.9 to make the style compliant. Thanks, Dario --- tools/libxl/libxl_numa.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/tools/libxl/libxl_numa.c b/tools/libxl/libxl_numa.c index 33289d5..fd64c22 100644 --- a/tools/libxl/libxl_numa.c +++ b/tools/libxl/libxl_numa.c @@ -205,12 +205,21 @@ static int nr_vcpus_on_nodes(libxl__gc *gc, libxl_cputopology *tinfo, } for (i = 0; i < nr_doms; i++) { - libxl_vcpuinfo *vinfo; - int nr_dom_vcpus; + libxl_vcpuinfo *vinfo = NULL; + libxl_cpupoolinfo cpupool_info; + int cpupool, nr_dom_vcpus; + + libxl_cpupoolinfo_init(&cpupool_info); + + cpupool = libxl__domain_cpupool(gc, dinfo[i].domid); + if (cpupool < 0) + goto next; + if (libxl_cpupool_info(CTX, &cpupool_info, cpupool)) + goto next; vinfo = libxl_list_vcpu(CTX, dinfo[i].domid, &nr_dom_vcpus, &nr_cpus); if (vinfo == NULL) - continue; + goto next; /* Retrieve the domain's node-affinity map */ libxl_domain_get_nodeaffinity(CTX, dinfo[i].domid, &dom_nodemap); @@ -220,6 +229,12 @@ static int nr_vcpus_on_nodes(libxl__gc *gc, libxl_cputopology *tinfo, * For each vcpu of each domain, it must have both vcpu-affinity * and node-affinity to (a pcpu belonging to) a certain node to * cause an increment in the corresponding element of the array. + * + * Note that we also need to check whether the cpu actually + * belongs to the domain's cpupool (the cpupool of the domain + * being checked). In fact, it could be that the vcpu has affinity + * with cpus in suitable_cpumask, but that are not in its own + * cpupool, and we don't want to consider those! */ libxl_bitmap_set_none(&nodes_counted); libxl_for_each_set_bit(k, vinfo[j].cpumap) { @@ -228,6 +243,7 @@ static int nr_vcpus_on_nodes(libxl__gc *gc, libxl_cputopology *tinfo, int node = tinfo[k].node; if (libxl_bitmap_test(suitable_cpumap, k) && + libxl_bitmap_test(&cpupool_info.cpumap, k) && libxl_bitmap_test(&dom_nodemap, node) && !libxl_bitmap_test(&nodes_counted, node)) { libxl_bitmap_set(&nodes_counted, node); @@ -236,6 +252,8 @@ static int nr_vcpus_on_nodes(libxl__gc *gc, libxl_cputopology *tinfo, } } + next: + libxl_cpupoolinfo_dispose(&cpupool_info); libxl_vcpuinfo_list_free(vinfo, nr_dom_vcpus); }