From patchwork Wed Sep 13 16:21:37 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dario Faggioli X-Patchwork-Id: 9951657 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 17F9860360 for ; Wed, 13 Sep 2017 16:23:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 099D728B44 for ; Wed, 13 Sep 2017 16:23:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F2A4F28B5C; Wed, 13 Sep 2017 16:23:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4338F28B44 for ; Wed, 13 Sep 2017 16:23:50 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dsAQ3-0007YX-TI; Wed, 13 Sep 2017 16:21:43 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dsAQ2-0007YP-PR for xen-devel@lists.xenproject.org; Wed, 13 Sep 2017 16:21:42 +0000 Received: from [85.158.143.35] by server-5.bemta-6.messagelabs.com id 32/25-03454-61B59B95; Wed, 13 Sep 2017 16:21:42 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrDIsWRWlGSWpSXmKPExsVyMbRhg65I9M5 Ig52NYhbft0xmcmD0OPzhCksAYxRrZl5SfkUCa8bM9V/YCl7YV3yf94y9gXGzbhcjF4eQwAxG if9LN7GCOCwCU1klzt85wwTiSAhsZJVYtGMvYxcjJ5ATJ9G/+B4rhJ0mcfVQP5RdLXHz228mE FtIQEXi5vZVUPYvRom1mzRBbGEBPYkjR3+wQ9juErMmTAKrYRMwkHizYy/YHBEBJYl7qyaDxZ kFIiU6ljxmA7FZBFQl1nxoBrI5OHgFvCW2N9eChEUF5CRWXm4Ba+UVEJQ4OfMJC0gJs4CmxPp d+hBT5CW2v53DPIFReBaSqlkIVbOQVC1gZF7FqF6cWlSWWqRrppdUlJmeUZKbmJmja2hgppeb WlycmJ6ak5hUrJecn7uJERjiDECwg3HeCf9DjJIcTEqivHt1d0YK8SXlp1RmJBZnxBeV5qQWH 2LU4OAQmHB27nQmKZa8/LxUJQle5yigOsGi1PTUirTMHGAUwpRKcPAoifAqgqR5iwsSc4sz0y FSpxgtOS7cufSHiePYpstA8sCeW3+YhMDmSYnzyoA0CIA0ZJTmwY2DJYpLjLJSwryMQMcK8RS kFuVmlqDKv2IU52BUEuadBzKFJzOvBG7rK6CDmIAOOnN6B8hBJYkIKakGxutlW7b/yDh98rT2 PL9jwbZc/zQrXkZO2+ls8092Vkv+/Clmy50WPrpZ35O+PDRh3VqlsNuXZv3ysP8ovGrVreX/p CW3zl/q8zg868bkKe6muydf8hOcu+Ho4/NxccxqJxiWnt2vd2LenQTr9wfXL14hlsSycvUX3h Y3Bj3ZzrU7BcotKqqzvjgrsRRnJBpqMRcVJwIAKkiG+Q8DAAA= X-Env-Sender: raistlin.df@gmail.com X-Msg-Ref: server-15.tower-21.messagelabs.com!1505319700!82538961!1 X-Originating-IP: [209.85.128.176] X-SpamReason: No, hits=0.5 required=7.0 tests=BODY_RANDOM_LONG X-StarScan-Received: X-StarScan-Version: 9.4.45; banners=-,-,- X-VirusChecked: Checked Received: (qmail 32116 invoked from network); 13 Sep 2017 16:21:40 -0000 Received: from mail-wr0-f176.google.com (HELO mail-wr0-f176.google.com) (209.85.128.176) by server-15.tower-21.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 13 Sep 2017 16:21:40 -0000 Received: by mail-wr0-f176.google.com with SMTP id a43so1687382wrc.0 for ; Wed, 13 Sep 2017 09:21:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:user-agent:mime-version :content-transfer-encoding; bh=DaMf1vPJIf2EuR9SMmABA9UOOsCkVXcaYwsDqOXXYMA=; b=a5raKL8QJSLhhe9fY+GrcL5N0JMITgCTeIIJevN0W67uiuv4BK2IgzVudv42jNGvri a/9PTyMZ7GBUpaUqSHZW26S32QxafGvxZXtBUw3Ht9x/InteH/TDs11XFWVqdgY2BqjX Uufnko+nrUz4FcLGsZbdKlaP+0KgR7XCHTiRWxocK7HBFIflEfK0xhedroDGDK538RbA jNiogfBhB95wobZLfmY3BJiK/pNEv5XJkZ2ymp4OjWdzwmo2CLcmT5HpN+hcD93HICUW SZ/Yh0XHJY5FHGFXGtihIxYnwc9Aa8bpEUgTp5cI0etqXDzkux2XF8naaIr1WdKgtnKz abCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :user-agent:mime-version:content-transfer-encoding; bh=DaMf1vPJIf2EuR9SMmABA9UOOsCkVXcaYwsDqOXXYMA=; b=pdBd/cLiTqNCG549qCPm3r8xcSuRWw1eIOlD4lirVxr2j03u/BNOyEBSh4SUGy58Lj wLB39RNXdYe3xRca9cB6Z5LhzYxO1wIFaWWJ9sHtozJdNuNcPCsoWLrvommXwCBMKHPP cAASSoFseC7wQkmGZXuuTdVFk3RUGCbyBcJjmz7WWubXOnI40vR7llIIu6XaY6UCQH4J sRCqst3q15GspKVNmi2TPfn0DC8HGrK9iRt8AXg0QPxgD1anRVfX2PlFM7QAqrkV+pGx oECOkwfX86mZmK3Fl6taxUtR4qQhKq+D1QNtH1v0s1FOvX7Pr2eGYPz1pnGbaFm1wDgr MfCQ== X-Gm-Message-State: AHPjjUiHc5Bwbq7T6Vir66N5ZMLhFUatcKvIKpdWxkcEQEZ78/+ZQ4ve 95NXZnk9l5sbdQ== X-Google-Smtp-Source: ADKCNb5F2Y2Xnd4H9Iq9jlhHw9Bki52mWcrHUimWJI4q7w+nxP96TD5ZHy7EQ+lo1DHUTdLIbR7hWA== X-Received: by 10.223.177.18 with SMTP id l18mr14709761wra.167.1505319699769; Wed, 13 Sep 2017 09:21:39 -0700 (PDT) Received: from Solace.fritz.box ([80.66.223.52]) by smtp.gmail.com with ESMTPSA id f19sm25283045wrf.85.2017.09.13.09.21.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Sep 2017 09:21:38 -0700 (PDT) From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Wed, 13 Sep 2017 18:21:37 +0200 Message-ID: <150531969753.22283.6861569329752511399.stgit@Solace.fritz.box> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Cc: George Dunlap , Praveen Kumar Subject: [Xen-devel] [PATCH] xen: Credit2: enable fully custom runqueue arrangement X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP The patch introduces yet another runqueue arrangement option for Credit2. In fact, it allows the user to specify, explicitly and precisely, what pCPUs should belong to which runqueue. Signed-off-by: Dario Faggioli Signed-off-by: Praveen Kumar --- Cc: George Dunlap Cc: Praveen Kumar --- This is derived --although *heavily* reworked-- from what Praveen has submitted before, here: https://lists.xen.org/archives/html/xen-devel/2017-04/msg02402.html https://lists.xen.org/archives/html/xen-devel/2017-06/msg00201.html During my review of that, I suggested to make the array dynamically allocated, using nr_cpu_ids as its size. Turns out, that does not make much sense, as at the time the parameters are parsed, nr_cpu_ids is still equal to NR_CPUS. --- docs/misc/xen-command-line.markdown | 12 +++ xen/common/sched_credit2.c | 135 +++++++++++++++++++++++++++++++++++ 2 files changed, 146 insertions(+), 1 deletion(-) diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown index 9797c8d..dbf5d4c 100644 --- a/docs/misc/xen-command-line.markdown +++ b/docs/misc/xen-command-line.markdown @@ -567,7 +567,7 @@ also slow in responding to load changes. The default value of `1 sec` is rather long. ### credit2\_runqueue -> `= cpu | core | socket | node | all` +> `= cpu | core | socket | node | all | ` > Default: `socket` @@ -585,6 +585,16 @@ Available alternatives, with their meaning, are: * `node`: one runqueue per each NUMA node of the host; * `all`: just one runqueue shared by all the logical pCPUs of the host +* ``: one runqueue per each specified subset of logical + pCPUs of the host. Subsets are defined either as: + `[[0,1,][2,6][3,5][4,7]]`, or as: + `'0,1;2,6;3,5;4,7'`. That means + - pCPUs 0 and 1 belong to runqueue 0 + - pCPUs 2 and 6 belong to runqueue 1 + - pCPUs 3 and 5 belong to runqueue 2 + - pCPUs 4 and 7 belong to runqueue 3 + pCPUs that are present on the host, but are not + part of any subset, are assigned to runqueue 0. ### dbgp > `= ehci[ | @pci:. ]` diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c index 0f93ad5..10da084 100644 --- a/xen/common/sched_credit2.c +++ b/xen/common/sched_credit2.c @@ -12,6 +12,7 @@ #include #include +#include #include #include #include @@ -323,6 +324,17 @@ integer_param("credit2_balance_over", opt_overload_balance_tolerance); * (logical) processors of the host belong. This will happen if * the opt_runqueue parameter is set to 'all'. * + * - custom: meaning that there will be one runqueue per each specified + * subset, as shown in the following examples: + * - credit2_runqueue=[[0,1][3][4,5]] + * - credit2_runqueue='0,1;3;4,5' + * These (both) mean the following: + * - CPU 0 and CPU 1 belong to runqueue 0 + * - CPU 3 belongs to runqueue 1 + * - CPU 4 and CPU 5 belong to runqueue 2 + * CPUs that are present on the host, but are not part of any + * defined subset, will be assigned to runqueue 0. + * * Depending on the value of opt_runqueue, therefore, cpus that are part of * either the same physical core, the same physical socket, the same NUMA * node, or just all of them, will be put together to form runqueues. @@ -332,6 +344,7 @@ integer_param("credit2_balance_over", opt_overload_balance_tolerance); #define OPT_RUNQUEUE_SOCKET 2 #define OPT_RUNQUEUE_NODE 3 #define OPT_RUNQUEUE_ALL 4 +#define OPT_RUNQUEUE_CUSTOM 5 static const char *const opt_runqueue_str[] = { [OPT_RUNQUEUE_CPU] = "cpu", [OPT_RUNQUEUE_CORE] = "core", @@ -341,6 +354,115 @@ static const char *const opt_runqueue_str[] = { }; static int __read_mostly opt_runqueue = OPT_RUNQUEUE_SOCKET; +static unsigned int __read_mostly custom_cpu_runqueue[NR_CPUS]; + +static int parse_custom_runqueue(const char *s) +{ + unsigned int cpu, rqi = 0; + bool in_subset = false, ret = true; + cpumask_t cpus; + + /* + * If we are dealing with format 1 (i.e., [[0,1][3,5]]), first + * and last character must be '[' and ']', respectively. + * + * If we are dealing with format 2 (i.e., 0,1;3,5), first and + * last character must be numbers. + */ + if ( *s == '[' && *(s+strlen(s)-1) == ']' ) + s++; + else if ( !(isdigit(*s) && isdigit(s[strlen(s)-1])) ) + return false; + + cpumask_clear(&cpus); + while ( !(*s == ']' && *(s+1) == '\0') && *s != '\0' ) + { + /* + * We tolerate only the allowed characters (depending on the + * format). Also, we don't accept empty subsets. + */ + if ( *(s+strlen(s)-1) == ']' ) + { + /* Format 1 */ + if ( !(*s == '[' || *s == ']' || *s == ',' || isdigit(*s)) || + (*s == '[' && *(s+1) == ']') ) + return false; + } + else + { + /* Format 2 */ + if ( !(*s == ';' || *s == ',' || isdigit(*s)) || + (*s == ';' && *(s+1) == ';') ) + return false; + } + + /* Are we at the beginning of a subset, in format 1? */ + if ( *s == '[' ) + { + /* + * If we are opening a subset, we must have closed all the + * previously defined ones, or the string is malformed. + */ + if ( in_subset ) + return false; + + s++; + in_subset = true; + continue; + } + /* Are we at the end of a subset? */ + if ( *s == ']' || *s == ';' ) + { + /* + * If we are closing a subset, in format 1, we must have + * opened it before. If not, the string is malformed. + */ + if ( *s == ']' && !in_subset ) + return false; + + s++; + rqi++; + in_subset = false; + continue; + } + + /* + * At this point, we must be dealing with either a CPU ID (i.e., + * a number) or CPU separator, within a subset (i.e., ','). + */ + if ( *s == ',' ) + { + s++; + continue; + } + cpu = simple_strtoul(s, &s, 10); + + /* Is the cpu ID we found valid? */ + if ( cpu >= nr_cpu_ids ) + return false; + + /* + * CPU IDs are considered only the first time they're found in the + * string considere. Multiple subsequent occurrences are ignored. + */ + if ( cpumask_test_cpu(cpu, &cpus) ) + continue; + + cpumask_set_cpu(cpu, &cpus); + custom_cpu_runqueue[cpu] = rqi; + } + + /* + * If in_subset is true, it means we are in format 1, and we + * found a subset that was not closed with its ']', which + * means the string is malformed. + */ + if ( in_subset ) + return false; + + return ret; +} + static int parse_credit2_runqueue(const char *s) { unsigned int i; @@ -354,6 +476,13 @@ static int parse_credit2_runqueue(const char *s) } } + if ( parse_custom_runqueue(s) ) + { + opt_runqueue = OPT_RUNQUEUE_CUSTOM; + return 0; + } + + printk("WARNING, unrecognized value of credit2_runqueue option!\n"); return -EINVAL; } custom_param("credit2_runqueue", parse_credit2_runqueue); @@ -728,6 +857,12 @@ cpu_to_runqueue(struct csched2_private *prv, unsigned int cpu) struct csched2_runqueue_data *rqd; unsigned int rqi; + if ( opt_runqueue == OPT_RUNQUEUE_CUSTOM ) + { + ASSERT(custom_cpu_runqueue[cpu] < nr_cpu_ids); + return custom_cpu_runqueue[cpu]; + } + for ( rqi = 0; rqi < nr_cpu_ids; rqi++ ) { unsigned int peer_cpu;