From patchwork Thu Jul 20 08:49:05 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Sun X-Patchwork-Id: 9854317 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 64E0260393 for ; Thu, 20 Jul 2017 09:06:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 59B9D28782 for ; Thu, 20 Jul 2017 09:06:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4DF9528785; Thu, 20 Jul 2017 09:06:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 773BB28783 for ; Thu, 20 Jul 2017 09:06:25 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dY7NH-0007Mb-9Y; Thu, 20 Jul 2017 09:03:59 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dY7NF-0007Lj-2H for xen-devel@lists.xenproject.org; Thu, 20 Jul 2017 09:03:57 +0000 Received: from [193.109.254.147] by server-3.bemta-6.messagelabs.com id 90/C5-03044-CF170795; Thu, 20 Jul 2017 09:03:56 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrJLMWRWlGSWpSXmKPExsXS1tYhr/u7sCD S4OFHTYvvWyYzOTB6HP5whSWAMYo1My8pvyKBNePx7c+MBTcSKq6ePcnWwDjJvYuRg0NIoEJi 4aOoLkZODgkBXokjy2awQtgBEjfPH2fqYuQCKmlglHg7ax8LSIJNQF3i8dceJhBbREBJ4t6qy WBFzALbmCTeX37BDpIQBuq+8+cpWAOLgKrEkYOf2UGW8Qp4SMz+4wexQE7i5LHJYMs4BTwlnn dvBmsVAiq5t/Iv8wRG3gWMDKsYNYpTi8pSi3QNjfWSijLTM0pyEzNzdA0NzPRyU4uLE9NTcxK TivWS83M3MQKDgQEIdjB+WRZwiFGSg0lJlJclsCBSiC8pP6UyI7E4I76oNCe1+BCjDAeHkgTv wwKgnGBRanpqRVpmDjAsYdISHDxKIhBp3uKCxNzizHSI1ClGXY5XE/5/YxJiycvPS5US510KU iQAUpRRmgc3AhYjlxhlpYR5GYGOEuIpSC3KzSxBlX/FKM7BqCTM+xxkCk9mXgncpldARzABHf HIDeyIkkSElFQD41KXLTU33WuKIl0/TOb6vTxRO4pVJraxv3G/++0jdxub9lqyP2q9esC2mMV JpmaL098zj2ZmZ4iddSo54H135t6ptz6vc136gv3VQakQCYe9nAmn89zOd3tdPDcl/DZ705eD 10KYTrlUmKxfZayT/UF7/hWJ580FOc/NcwOTtm7+eFafI3arrxJLcUaioRZzUXEiAGSKX3yMA gAA X-Env-Sender: yi.y.sun@linux.intel.com X-Msg-Ref: server-9.tower-27.messagelabs.com!1500541426!108131258!4 X-Originating-IP: [134.134.136.31] X-SpamReason: No, hits=0.5 required=7.0 tests=BODY_RANDOM_LONG X-StarScan-Received: X-StarScan-Version: 9.4.25; banners=-,-,- X-VirusChecked: Checked Received: (qmail 60125 invoked from network); 20 Jul 2017 09:03:55 -0000 Received: from mga06.intel.com (HELO mga06.intel.com) (134.134.136.31) by server-9.tower-27.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 20 Jul 2017 09:03:55 -0000 Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga104.jf.intel.com with ESMTP; 20 Jul 2017 02:03:54 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos; i="5.40,383,1496127600"; d="scan'208"; a="1153387001" Received: from vmmmba-s2600wft.bj.intel.com ([10.240.193.80]) by orsmga001.jf.intel.com with ESMTP; 20 Jul 2017 02:03:51 -0700 From: Yi Sun To: xen-devel@lists.xenproject.org Date: Thu, 20 Jul 2017 16:49:05 +0800 Message-Id: <1500540553-29199-5-git-send-email-yi.y.sun@linux.intel.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1500540553-29199-1-git-send-email-yi.y.sun@linux.intel.com> References: <1500540553-29199-1-git-send-email-yi.y.sun@linux.intel.com> Cc: kevin.tian@intel.com, wei.liu2@citrix.com, andrew.cooper3@citrix.com, dario.faggioli@citrix.com, ian.jackson@eu.citrix.com, Yi Sun , julien.grall@arm.com, mengxu@cis.upenn.edu, jbeulich@suse.com, chao.p.peng@linux.intel.com, roger.pau@citrix.com Subject: [Xen-devel] [RFC v2 04/12] x86: implement data structure and CPU init flow for MBA. X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP This patch implements main data structures of MBA. Like CAT features, MBA HW info has cos_max which means the max cos registers number, and thrtl_max which means the max throttle value (delay value). It also has a flag to represent if the throttle value is linear or not. One COS register of MBA stores a throttle value for one or more domains. The throttle value means the transaction time between L2 cache and next level memory to be delayed. This patch also implements init flow for MBA and register stub callback functions. Signed-off-by: Yi Sun --- xen/arch/x86/psr.c | 130 ++++++++++++++++++++++++++++++++-------- xen/include/asm-x86/msr-index.h | 1 + xen/include/asm-x86/psr.h | 2 + 3 files changed, 109 insertions(+), 24 deletions(-) diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c index d1d854f..d1ea5a4 100644 --- a/xen/arch/x86/psr.c +++ b/xen/arch/x86/psr.c @@ -27,13 +27,16 @@ * - CMT Cache Monitoring Technology * - COS/CLOS Class of Service. Also mean COS registers. * - COS_MAX Max number of COS for the feature (minus 1) + * - MBA Memory Bandwidth Allocation * - MSRs Machine Specific Registers * - PSR Intel Platform Shared Resource + * - THRTL_MAX Max throttle value (delay value) of MBA */ #define PSR_CMT (1<<0) #define PSR_CAT (1<<1) #define PSR_CDP (1<<2) +#define PSR_MBA (1<<3) #define CAT_CBM_LEN_MASK 0x1f #define CAT_COS_MAX_MASK 0xffff @@ -60,10 +63,14 @@ */ #define MAX_COS_NUM 2 +#define MBA_LINEAR (1<<2) +#define MBA_THRTL_MAX_MASK 0xfff + enum psr_feat_type { FEAT_TYPE_L3_CAT, FEAT_TYPE_L3_CDP, FEAT_TYPE_L2_CAT, + FEAT_TYPE_MBA, FEAT_TYPE_NUM, FEAT_TYPE_UNKNOWN, }; @@ -71,7 +78,6 @@ enum psr_feat_type { /* * This structure represents one feature. * cos_max - The max COS registers number got through CPUID. - * cbm_len - The length of CBM got through CPUID. * cos_reg_val - Array to store the values of COS registers. One entry stores * the value of one COS register. * For L3 CAT and L2 CAT, one entry corresponds to one COS_ID. @@ -80,9 +86,21 @@ enum psr_feat_type { * cos_reg_val[1] (Code). */ struct feat_node { - /* cos_max and cbm_len are common values for all features so far. */ + /* cos_max is common values for all features so far. */ unsigned int cos_max; - unsigned int cbm_len; + + /* Feature specific HW info. */ + union { + struct { + unsigned int cbm_len; + } cat_info; + + struct { + unsigned int thrtl_max; + unsigned int linear; + } mba_info; + }; + uint32_t cos_reg_val[MAX_COS_REG_CNT]; }; @@ -161,6 +179,7 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc); */ static struct feat_node *feat_l3; static struct feat_node *feat_l2_cat; +static struct feat_node *feat_mba; /* Common functions */ #define cat_default_val(len) (0xffffffff >> (32 - (len))) @@ -274,22 +293,22 @@ static bool psr_check_cbm(unsigned int cbm_len, unsigned long cbm) } /* CAT common functions implementation. */ -static int cat_init_feature(const struct cpuid_leaf *regs, - struct feat_node *feat, - struct psr_socket_info *info, - enum psr_feat_type type) +static int init_alloc_features(const struct cpuid_leaf *regs, + struct feat_node *feat, + struct psr_socket_info *info, + enum psr_feat_type type) { - const char * const cat_feat_name[FEAT_TYPE_NUM] = { + const char * const feat_name[FEAT_TYPE_NUM] = { "L3 CAT", "CDP", "L2 CAT", + "MBA", }; /* No valid value so do not enable feature. */ if ( !regs->a || !regs->d ) return -ENOENT; - feat->cbm_len = (regs->a & CAT_CBM_LEN_MASK) + 1; feat->cos_max = min(opt_cos_max, regs->d & CAT_COS_MAX_MASK); switch ( type ) @@ -299,13 +318,15 @@ static int cat_init_feature(const struct cpuid_leaf *regs, if ( feat->cos_max < 1 ) return -ENOENT; + feat->cat_info.cbm_len = (regs->a & CAT_CBM_LEN_MASK) + 1; + /* We reserve cos=0 as default cbm (all bits within cbm_len are 1). */ - feat->cos_reg_val[0] = cat_default_val(feat->cbm_len); + feat->cos_reg_val[0] = cat_default_val(feat->cat_info.cbm_len); wrmsrl((type == FEAT_TYPE_L3_CAT ? MSR_IA32_PSR_L3_MASK(0) : MSR_IA32_PSR_L2_MASK(0)), - cat_default_val(feat->cbm_len)); + cat_default_val(feat->cat_info.cbm_len)); break; @@ -316,15 +337,19 @@ static int cat_init_feature(const struct cpuid_leaf *regs, if ( feat->cos_max < 3 ) return -ENOENT; + feat->cat_info.cbm_len = (regs->a & CAT_CBM_LEN_MASK) + 1; + /* Cut half of cos_max when CDP is enabled. */ feat->cos_max = (feat->cos_max - 1) >> 1; /* We reserve cos=0 as default cbm (all bits within cbm_len are 1). */ - get_cdp_code(feat, 0) = cat_default_val(feat->cbm_len); - get_cdp_data(feat, 0) = cat_default_val(feat->cbm_len); + get_cdp_code(feat, 0) = cat_default_val(feat->cat_info.cbm_len); + get_cdp_data(feat, 0) = cat_default_val(feat->cat_info.cbm_len); - wrmsrl(MSR_IA32_PSR_L3_MASK(0), cat_default_val(feat->cbm_len)); - wrmsrl(MSR_IA32_PSR_L3_MASK(1), cat_default_val(feat->cbm_len)); + wrmsrl(MSR_IA32_PSR_L3_MASK(0), + cat_default_val(feat->cat_info.cbm_len)); + wrmsrl(MSR_IA32_PSR_L3_MASK(1), + cat_default_val(feat->cat_info.cbm_len)); rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val); wrmsrl(MSR_IA32_PSR_L3_QOS_CFG, val | (1ull << PSR_L3_QOS_CDP_ENABLE_BIT)); @@ -332,6 +357,20 @@ static int cat_init_feature(const struct cpuid_leaf *regs, break; } + case FEAT_TYPE_MBA: + if ( feat->cos_max < 1 ) + return -ENOENT; + + feat->mba_info.thrtl_max = (regs->a & MBA_THRTL_MAX_MASK) + 1; + + if ( regs->c & MBA_LINEAR ) + feat->mba_info.linear = 1; + + feat->cos_reg_val[0] = 0; + wrmsrl(MSR_IA32_PSR_MBA_MASK(0), 0); + + break; + default: return -ENOENT; } @@ -342,9 +381,14 @@ static int cat_init_feature(const struct cpuid_leaf *regs, if ( !opt_cpu_info ) return 0; - printk(XENLOG_INFO "%s: enabled on socket %u, cos_max:%u, cbm_len:%u\n", - cat_feat_name[type], cpu_to_socket(smp_processor_id()), - feat->cos_max, feat->cbm_len); + if ( type == FEAT_TYPE_MBA ) + printk(XENLOG_INFO "%s: enabled on socket %u, cos_max:%u, thrtl_max:%u, linear:%u.\n", + feat_name[type], cpu_to_socket(smp_processor_id()), + feat->cos_max, feat->mba_info.thrtl_max, feat->mba_info.linear); + else + printk(XENLOG_INFO "%s: enabled on socket %u, cos_max:%u, cbm_len:%u\n", + feat_name[type], cpu_to_socket(smp_processor_id()), + feat->cos_max, feat->cat_info.cbm_len); return 0; } @@ -356,7 +400,7 @@ static bool cat_get_feat_info(const struct feat_node *feat, return false; data[PSR_INFO_IDX_COS_MAX] = feat->cos_max; - data[PSR_INFO_IDX_CAT_CBM_LEN] = feat->cbm_len; + data[PSR_INFO_IDX_CAT_CBM_LEN] = feat->cat_info.cbm_len; data[PSR_INFO_IDX_CAT_FLAG] = 0; return true; @@ -422,6 +466,26 @@ static const struct feat_props l2_cat_props = { .write_msr = l2_cat_write_msr, }; +/* MBA props */ +static bool mba_get_feat_info(const struct feat_node *feat, + uint32_t data[], unsigned int array_len) +{ + return false; +} + +static void mba_write_msr(unsigned int cos, uint32_t val, + enum psr_val_type type) +{ +} + +static const struct feat_props mba_props = { + .cos_num = 1, + .type[0] = PSR_VAL_TYPE_MBA, + .alt_type = PSR_VAL_TYPE_UNKNOWN, + .get_feat_info = mba_get_feat_info, + .write_msr = mba_write_msr, +}; + static void __init parse_psr_bool(char *s, char *value, char *feature, unsigned int mask) { @@ -457,6 +521,7 @@ static void __init parse_psr_param(char *s) parse_psr_bool(s, val_str, "cmt", PSR_CMT); parse_psr_bool(s, val_str, "cat", PSR_CAT); parse_psr_bool(s, val_str, "cdp", PSR_CDP); + parse_psr_bool(s, val_str, "mba", PSR_MBA); if ( val_str && !strcmp(s, "rmid_max") ) opt_rmid_max = simple_strtoul(val_str, NULL, 0); @@ -863,7 +928,7 @@ static int insert_val_into_array(uint32_t val[], if ( array_len < props->cos_num ) return -ENOSPC; - if ( !psr_check_cbm(feat->cbm_len, new_val) ) + if ( !psr_check_cbm(feat->cat_info.cbm_len, new_val) ) return -EINVAL; /* @@ -1380,6 +1445,10 @@ static int psr_cpu_prepare(void) (feat_l2_cat = xzalloc(struct feat_node)) == NULL ) return -ENOMEM; + if ( feat_mba == NULL && + (feat_mba = xzalloc(struct feat_node)) == NULL ) + return -ENOMEM; + return 0; } @@ -1416,7 +1485,7 @@ static void psr_cpu_init(void) if ( (regs.c & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) ) { - if ( !cat_init_feature(®s, feat, info, FEAT_TYPE_L3_CDP) ) + if ( !init_alloc_features(®s, feat, info, FEAT_TYPE_L3_CDP) ) feat_props[FEAT_TYPE_L3_CDP] = &l3_cdp_props; else /* If CDP init fails, try to work as L3 CAT. */ @@ -1425,7 +1494,7 @@ static void psr_cpu_init(void) else { l3_cat_init: - if ( !cat_init_feature(®s, feat, info, FEAT_TYPE_L3_CAT) ) + if ( !init_alloc_features(®s, feat, info, FEAT_TYPE_L3_CAT) ) feat_props[FEAT_TYPE_L3_CAT] = &l3_cat_props; else feat_l3 = feat; @@ -1439,12 +1508,25 @@ static void psr_cpu_init(void) feat = feat_l2_cat; feat_l2_cat = NULL; - if ( !cat_init_feature(®s, feat, info, FEAT_TYPE_L2_CAT) ) + if ( !init_alloc_features(®s, feat, info, FEAT_TYPE_L2_CAT) ) feat_props[FEAT_TYPE_L2_CAT] = &l2_cat_props; else feat_l2_cat = feat; } + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s); + if ( regs.b & PSR_RESOURCE_TYPE_MBA ) + { + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 3, ®s); + + feat = feat_mba; + feat_mba = NULL; + if ( !init_alloc_features(®s, feat, info, FEAT_TYPE_MBA) ) + feat_props[FEAT_TYPE_MBA] = &mba_props; + else + feat_mba = feat; + } + info->feat_init = true; assoc_init: @@ -1504,7 +1586,7 @@ static int __init psr_presmp_init(void) if ( (opt_psr & PSR_CMT) && opt_rmid_max ) init_psr_cmt(opt_rmid_max); - if ( opt_psr & (PSR_CAT | PSR_CDP) ) + if ( opt_psr & (PSR_CAT | PSR_CDP | PSR_MBA) ) init_psr(); if ( psr_cpu_prepare() ) diff --git a/xen/include/asm-x86/msr-index.h b/xen/include/asm-x86/msr-index.h index 4e08de6..41f1677 100644 --- a/xen/include/asm-x86/msr-index.h +++ b/xen/include/asm-x86/msr-index.h @@ -348,6 +348,7 @@ #define MSR_IA32_PSR_L3_MASK_CODE(n) (0x00000c90 + (n) * 2 + 1) #define MSR_IA32_PSR_L3_MASK_DATA(n) (0x00000c90 + (n) * 2) #define MSR_IA32_PSR_L2_MASK(n) (0x00000d10 + (n)) +#define MSR_IA32_PSR_MBA_MASK(n) (0x00000d50 + (n)) /* Intel Model 6 */ #define MSR_P6_PERFCTR(n) (0x000000c1 + (n)) diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h index 1b6f22f..551ccf3 100644 --- a/xen/include/asm-x86/psr.h +++ b/xen/include/asm-x86/psr.h @@ -24,6 +24,7 @@ /* Resource Type Enumeration */ #define PSR_RESOURCE_TYPE_L3 0x2 #define PSR_RESOURCE_TYPE_L2 0x4 +#define PSR_RESOURCE_TYPE_MBA 0x8 /* L3 Monitoring Features */ #define PSR_CMT_L3_OCCUPANCY 0x1 @@ -58,6 +59,7 @@ enum psr_val_type { PSR_VAL_TYPE_L3_CODE, PSR_VAL_TYPE_L3_DATA, PSR_VAL_TYPE_L2, + PSR_VAL_TYPE_MBA, PSR_VAL_TYPE_UNKNOWN, };