From patchwork Thu Aug 24 01:14:41 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Sun X-Patchwork-Id: 9918827 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 37BD460349 for ; Thu, 24 Aug 2017 01:35:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2A62828AB8 for ; Thu, 24 Aug 2017 01:35:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1E09028AD9; Thu, 24 Aug 2017 01:35:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 650A628AB8 for ; Thu, 24 Aug 2017 01:34:59 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dkh0X-0000qn-Lz; Thu, 24 Aug 2017 01:32:29 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dkh0W-0000py-QS for xen-devel@lists.xenproject.org; Thu, 24 Aug 2017 01:32:29 +0000 Received: from [85.158.137.68] by server-3.bemta-3.messagelabs.com id A0/66-01987-CAC2E995; Thu, 24 Aug 2017 01:32:28 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrLLMWRWlGSWpSXmKPExsXS1tbhqLtKZ16 kwaezJhbft0xmcmD0OPzhCksAYxRrZl5SfkUCa8a+Ky/YClZHVRzaMo25gXGVUxcjF4eQwHRG ia77T9m7GDk5JAR4JY4sm8EKYftLPDv+lQmiqIFRYsKpWWwgCTYBdYnHX3uYQGwRASWJe6smg xUxC2xjknh/+QXYJGEBe4mO8+vAGlgEVCX6d1wCsjk4eAU8JNrOJUIskJM4eWwyK0iYU8BTYu 06X5CwEFDF3Xd3WScw8i5gZFjFqF6cWlSWWqRropdUlJmeUZKbmJmja2hgrJebWlycmJ6ak5h UrJecn7uJERgMDECwg7Hxi9MhRkkOJiVR3ifS8yKF+JLyUyozEosz4otKc1KLDzHKcHAoSfDK awPlBItS01Mr0jJzgGEJk5bg4FES4S0ASfMWFyTmFmemQ6ROMSpKifMmgiQEQBIZpXlwbbBYu MQoKyXMywh0iBBPQWpRbmYJqvwrRnEORiVh3ndaQFN4MvNK4Ka/AlrMBLR40ok5IItLEhFSUg 2MWbPamViMyz0nuvzMuGbLWXInU7ZhTc8DP8fN533Wvd75W3Hv/wdy35wtEpce28r7L/v+dvO vQaa3NsZcto/Y8YpjPZ+WiEhtxLqUA2YWp+00y9gvSstNlHkTlKfc0tx735Op/8CHLjvB2cpx mn0qk/bt+bfObid3Due8kifPt87NmX7BImG7EktxRqKhFnNRcSIAYSUiWoACAAA= X-Env-Sender: yi.y.sun@linux.intel.com X-Msg-Ref: server-11.tower-31.messagelabs.com!1503538328!80460649!8 X-Originating-IP: [134.134.136.65] X-SpamReason: No, hits=0.5 required=7.0 tests=BODY_RANDOM_LONG X-StarScan-Received: X-StarScan-Version: 9.4.45; banners=-,-,- X-VirusChecked: Checked Received: (qmail 7260 invoked from network); 24 Aug 2017 01:32:26 -0000 Received: from mga03.intel.com (HELO mga03.intel.com) (134.134.136.65) by server-11.tower-31.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 24 Aug 2017 01:32:26 -0000 Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 23 Aug 2017 18:32:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.41,419,1498546800"; d="scan'208";a="303775143" Received: from vmmmba-s2600wft.bj.intel.com ([10.240.193.80]) by fmsmga004.fm.intel.com with ESMTP; 23 Aug 2017 18:32:23 -0700 From: Yi Sun To: xen-devel@lists.xenproject.org Date: Thu, 24 Aug 2017 09:14:41 +0800 Message-Id: <1503537289-56036-8-git-send-email-yi.y.sun@linux.intel.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1503537289-56036-1-git-send-email-yi.y.sun@linux.intel.com> References: <1503537289-56036-1-git-send-email-yi.y.sun@linux.intel.com> Cc: kevin.tian@intel.com, wei.liu2@citrix.com, andrew.cooper3@citrix.com, dario.faggioli@citrix.com, ian.jackson@eu.citrix.com, Yi Sun , julien.grall@arm.com, mengxu@cis.upenn.edu, jbeulich@suse.com, chao.p.peng@linux.intel.com, roger.pau@citrix.com Subject: [Xen-devel] [PATCH v2 07/15] x86: implement set value flow for MBA X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP This patch implements set value flow for MBA including its callback function and domctl interface. It also changes the memebers in 'cos_write_info' to transfer the feature array, feature properties array and value array. Then, we can write all features values on the cos id into MSRs. Because multiple features may co-exist, we need handle all features to write values of them into a COS register with new COS ID. E.g: 1. L3 CAT and MBA co-exist. 2. Dom1 and Dom2 share a same COS ID (2). The L3 CAT CBM of Dom1 is 0x1ff, the MBA Thrtle of Dom1 is 0xa. 3. User wants to change MBA Thrtl of Dom1 to be 0x14. Because COS ID 2 is used by Dom2 too, we have to pick a new COS ID 3. The original values of Dom1 on COS ID 3 may be below: --------- | COS 3 | --------- L3 CAT | 0x7ff | --------- MBA | 0x0 | --------- 4. After setting, the L3 CAT CBM value of Dom1 should be kept and the new MBA Thrtl is set. So, the values on COS ID 3 should be below. --------- | COS 3 | --------- L3 CAT | 0x1ff | --------- MBA | 0x14 | --------- So, we should write all features values into their MSRs. That requires the feature array, feature properties array and value array are input. Signed-off-by: Yi Sun --- v2: - remove linear mode 'thrtl_max' check in 'mba_check_thrtl' because it has been checked in 'mba_init_feature'. (suggested by Chao Peng) - for non-linear mode, check if '*thrtl' is not 0 in 'mba_check_thrtl'. If it is 0, we do not need to change it. (suggested by Chao Peng) - move comments to explain changes of 'cos_write_info' from psr.c to commit message. (suggested by Chao Peng) --- xen/arch/x86/domctl.c | 6 ++ xen/arch/x86/psr.c | 150 ++++++++++++++++++++++++++++++-------------- xen/include/public/domctl.h | 1 + 3 files changed, 109 insertions(+), 48 deletions(-) diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c index 4936bcb..0ae4799 100644 --- a/xen/arch/x86/domctl.c +++ b/xen/arch/x86/domctl.c @@ -1468,6 +1468,12 @@ long arch_do_domctl( PSR_VAL_TYPE_L2_CBM); break; + case XEN_DOMCTL_PSR_MBA_OP_SET_THRTL: + ret = psr_set_val(d, domctl->u.psr_alloc_op.target, + domctl->u.psr_alloc_op.data, + PSR_VAL_TYPE_MBA); + break; + case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM: ret = psr_get_val(d, domctl->u.psr_alloc_op.target, &val32, PSR_VAL_TYPE_L3_CBM); diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c index 4a0c982..ce82975 100644 --- a/xen/arch/x86/psr.c +++ b/xen/arch/x86/psr.c @@ -138,6 +138,12 @@ static const struct feat_props { /* write_msr is used to write out feature MSR register. */ void (*write_msr)(unsigned int cos, uint32_t val, enum psr_val_type type); + + /* + * check_val is used to check if input val fulfills SDM requirement. + * Change it to valid value if SDM allows. + */ + bool (*check_val)(const struct feat_node *feat, unsigned long *val); } *feat_props[FEAT_TYPE_NUM]; /* @@ -275,29 +281,6 @@ static enum psr_feat_type psr_val_type_to_feat_type(enum psr_val_type type) return feat_type; } -static bool psr_check_cbm(unsigned int cbm_len, unsigned long cbm) -{ - unsigned int first_bit, zero_bit; - - /* Set bits should only in the range of [0, cbm_len]. */ - if ( cbm & (~0ul << cbm_len) ) - return false; - - /* At least one bit need to be set. */ - if ( cbm == 0 ) - return false; - - first_bit = find_first_bit(&cbm, cbm_len); - zero_bit = find_next_zero_bit(&cbm, cbm_len, first_bit); - - /* Set bits should be contiguous. */ - if ( zero_bit < cbm_len && - find_next_bit(&cbm, cbm_len, zero_bit) < cbm_len ) - return false; - - return true; -} - /* Implementation of allocation features' functions. */ static int cat_init_feature(const struct cpuid_leaf *regs, struct feat_node *feat, @@ -433,6 +416,30 @@ static bool cat_get_feat_info(const struct feat_node *feat, return true; } +static bool cat_check_cbm(const struct feat_node *feat, unsigned long *cbm) +{ + unsigned int first_bit, zero_bit; + unsigned int cbm_len = feat->cat_info.cbm_len; + + /* Set bits should only in the range of [0, cbm_len]. */ + if ( *cbm & (~0ul << cbm_len) ) + return false; + + /* At least one bit need to be set. */ + if ( *cbm == 0 ) + return false; + + first_bit = find_first_bit(cbm, cbm_len); + zero_bit = find_next_zero_bit(cbm, cbm_len, first_bit); + + /* Set bits should be contiguous. */ + if ( zero_bit < cbm_len && + find_next_bit(cbm, cbm_len, zero_bit) < cbm_len ) + return false; + + return true; +} + /* L3 CAT props */ static void l3_cat_write_msr(unsigned int cos, uint32_t val, enum psr_val_type type) @@ -446,6 +453,7 @@ static const struct feat_props l3_cat_props = { .alt_type = PSR_VAL_TYPE_UNKNOWN, .get_feat_info = cat_get_feat_info, .write_msr = l3_cat_write_msr, + .check_val = cat_check_cbm, }; /* L3 CDP props */ @@ -476,6 +484,7 @@ static const struct feat_props l3_cdp_props = { .alt_type = PSR_VAL_TYPE_L3_CBM, .get_feat_info = l3_cdp_get_feat_info, .write_msr = l3_cdp_write_msr, + .check_val = cat_check_cbm, }; /* L2 CAT props */ @@ -491,6 +500,7 @@ static const struct feat_props l2_cat_props = { .alt_type = PSR_VAL_TYPE_UNKNOWN, .get_feat_info = cat_get_feat_info, .write_msr = l2_cat_write_msr, + .check_val = cat_check_cbm, }; /* MBA props */ @@ -514,6 +524,40 @@ static bool mba_get_feat_info(const struct feat_node *feat, static void mba_write_msr(unsigned int cos, uint32_t val, enum psr_val_type type) { + wrmsrl(MSR_IA32_PSR_MBA_MASK(cos), val); +} + +static bool mba_check_thrtl(const struct feat_node *feat, unsigned long *thrtl) +{ + if ( *thrtl > feat->mba_info.thrtl_max ) + return false; + + /* + * Per SDM (chapter "Memory Bandwidth Allocation Configuration"): + * 1. Linear mode: In the linear mode the input precision is defined + * as 100-(MBA_MAX). For instance, if the MBA_MAX value is 90, the + * input precision is 10%. Values not an even multiple of the + * precision (e.g., 12%) will be rounded down (e.g., to 10% delay + * applied). + * 2. Non-linear mode: Input delay values are powers-of-two from zero + * to the MBA_MAX value from CPUID. In this case any values not a + * power of two will be rounded down the next nearest power of two. + */ + if ( feat->mba_info.linear ) + { + unsigned int mod; + + mod = *thrtl % (100 - feat->mba_info.thrtl_max); + *thrtl -= mod; + } + else + { + /* Not power of 2. */ + if ( *thrtl && (*thrtl & (*thrtl - 1)) ) + *thrtl = *thrtl & (1 << (flsl(*thrtl) - 1)); + } + + return true; } static const struct feat_props mba_props = { @@ -522,6 +566,7 @@ static const struct feat_props mba_props = { .alt_type = PSR_VAL_TYPE_UNKNOWN, .get_feat_info = mba_get_feat_info, .write_msr = mba_write_msr, + .check_val = mba_check_thrtl, }; static void __init parse_psr_bool(char *s, char *value, char *feature, @@ -942,6 +987,7 @@ static int insert_val_into_array(uint32_t val[], const struct feat_node *feat; const struct feat_props *props; unsigned int i; + unsigned long check_val = new_val; int ret; ASSERT(feat_type < FEAT_TYPE_NUM); @@ -966,9 +1012,11 @@ static int insert_val_into_array(uint32_t val[], if ( array_len < props->cos_num ) return -ENOSPC; - if ( !psr_check_cbm(feat->cat_info.cbm_len, new_val) ) + if ( !props->check_val(feat, &check_val) ) return -EINVAL; + new_val = check_val; + /* * Value setting position is same as feature array. * For CDP, user may set both DATA and CODE to same value. For such case, @@ -1198,25 +1246,42 @@ static unsigned int get_socket_cpu(unsigned int socket) struct cos_write_info { unsigned int cos; - struct feat_node *feature; + struct feat_node **features; const uint32_t *val; - const struct feat_props *props; + unsigned int array_len; + const struct feat_props **props; }; static void do_write_psr_msrs(void *data) { const struct cos_write_info *info = data; - struct feat_node *feat = info->feature; - const struct feat_props *props = info->props; - unsigned int i, cos = info->cos, cos_num = props->cos_num; + unsigned int i, j, index = 0, array_len = info->array_len, cos = info->cos; + const uint32_t *val_array = info->val; - for ( i = 0; i < cos_num; i++ ) + for ( i = 0; i < ARRAY_SIZE(feat_props); i++ ) { - if ( feat->cos_reg_val[cos * cos_num + i] != info->val[i] ) + struct feat_node *feat = info->features[i]; + const struct feat_props *props = info->props[i]; + unsigned int cos_num; + + if ( !feat || !props ) + continue; + + cos_num = props->cos_num; + if ( array_len < cos_num ) + return; + + for ( j = 0; j < cos_num; j++ ) { - feat->cos_reg_val[cos * cos_num + i] = info->val[i]; - props->write_msr(cos, info->val[i], props->type[i]); + if ( feat->cos_reg_val[cos * cos_num + j] != val_array[index + j] ) + { + feat->cos_reg_val[cos * cos_num + j] = val_array[index + j]; + props->write_msr(cos, val_array[index + j], props->type[j]); + } } + + array_len -= cos_num; + index += cos_num; } } @@ -1224,30 +1289,19 @@ static int write_psr_msrs(unsigned int socket, unsigned int cos, const uint32_t val[], unsigned int array_len, enum psr_feat_type feat_type) { - int ret; struct psr_socket_info *info = get_socket_info(socket); struct cos_write_info data = { .cos = cos, - .feature = info->features[feat_type], - .props = feat_props[feat_type], + .features = info->features, + .val = val, + .array_len = array_len, + .props = feat_props, }; if ( cos > info->features[feat_type]->cos_max ) return -EINVAL; - /* Skip to the feature's value head. */ - ret = skip_prior_features(&array_len, feat_type); - if ( ret < 0 ) - return ret; - - val += ret; - - if ( array_len < feat_props[feat_type]->cos_num ) - return -ENOSPC; - - data.val = val; - if ( socket == cpu_to_socket(smp_processor_id()) ) do_write_psr_msrs(&data); else diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h index a577a3e..8826cfb 100644 --- a/xen/include/public/domctl.h +++ b/xen/include/public/domctl.h @@ -1144,6 +1144,7 @@ struct xen_domctl_psr_alloc_op { #define XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA 5 #define XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM 6 #define XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM 7 +#define XEN_DOMCTL_PSR_MBA_OP_SET_THRTL 8 #define XEN_DOMCTL_PSR_MBA_OP_GET_THRTL 9 uint32_t cmd; /* IN: XEN_DOMCTL_PSR_CAT_OP_* */ uint32_t target; /* IN */