From patchwork Sat Apr 1 13:53:32 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Yi Sun X-Patchwork-Id: 9657987 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 45D4F60349 for ; Sat, 1 Apr 2017 13:56:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3BC3A285C2 for ; Sat, 1 Apr 2017 13:56:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3079B28610; Sat, 1 Apr 2017 13:56:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id F3798285C2 for ; Sat, 1 Apr 2017 13:56:35 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cuJTF-0005Rc-FH; Sat, 01 Apr 2017 13:53:37 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cuJTE-0005RL-Bm for xen-devel@lists.xenproject.org; Sat, 01 Apr 2017 13:53:36 +0000 Received: from [85.158.137.68] by server-12.bemta-3.messagelabs.com id 1A/D6-12861-FD0BFD85; Sat, 01 Apr 2017 13:53:35 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrMIsWRWlGSWpSXmKPExsXS1tYhontvw/0 Ig5k/5Cy+b5nM5MDocfjDFZYAxijWzLyk/IoE1oy1C/czFjSsYKz4cngCawPj0qouRk4OFoFG Zon7h7i6GLk4hASmMUocaH7AApKQEOCVOLJsBiuE7S/x6Xg3M4gtJFAv0femiR3EZhNQl3j8t YcJxBYRUJK4t2oyE8ggZoGdTBLrTn9nBnGEBRoZJRZ8/c4IsU5VoufseTYQm1fAQ2LjlTVsEB vkJE4emwy2jVPAU+LKi30sENs8JFZNusQMUS8ocXLmE6A4B9AGdYn184RAwswC8hLNW2czT2A UnIWkahZC1SwkVQsYmVcxahSnFpWlFukaWuglFWWmZ5TkJmbm6BoaGOvlphYXJ6an5iQmFesl 5+duYgQGbj0DA+MOxt+nPQ8xSnIwKYnyfi++FyHEl5SfUpmRWJwRX1Sak1p8iFGGg0NJglcBG AlCgkWp6akVaZk5wBiCSUtw8CiJ8B5bD5TmLS5IzC3OTIdInWJUlBLnfQaSEABJZJTmwbXB4v YSo6yUMC8jAwODEE9BalFuZgmq/CtGcQ5GJWFeeZApPJl5JXDTXwEtZgJabPH1LsjikkSElFQ DY7r5scyE7L+FZ5rsvjaE//yev6esb1PiNh/rQJHjX/9c1bh4Pf7oLYvKy0ru5yOcr+WeSf+y 6NPnmUGNrwTaXZf3KJoW33qUkdq5xD/tyRY/5cx4s4sKD5VKHIvCTyb9tj7CsyC2P++opOyis wamjm/Fln/k68tYvNzz1tpE/g9Xpda8u+lTqcRSnJFoqMVcVJwIAGZexybWAgAA X-Env-Sender: yi.y.sun@linux.intel.com X-Msg-Ref: server-9.tower-31.messagelabs.com!1491054811!37933881!2 X-Originating-IP: [134.134.136.20] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTM0LjEzNC4xMzYuMjAgPT4gMzU1MzU4\n X-StarScan-Received: X-StarScan-Version: 9.2.3; banners=-,-,- X-VirusChecked: Checked Received: (qmail 49005 invoked from network); 1 Apr 2017 13:53:34 -0000 Received: from mga02.intel.com (HELO mga02.intel.com) (134.134.136.20) by server-9.tower-31.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 1 Apr 2017 13:53:34 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=intel.com; i=@intel.com; q=dns/txt; s=intel; t=1491054814; x=1522590814; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=sbeg3IghUpvgMH1XIDltryBIUWQaWWeugCLTYSc5TCg=; b=TfwEbvZaVRdp9YSZHfYOfHRnDY944ixX+5vMIp/DuCHinapJAJJB8TQO ERIjQ1XzMWeuZ6mEMaGZMmHyhlh/fQ==; Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 01 Apr 2017 06:53:33 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.36,258,1486454400"; d="scan'208";a="950544193" Received: from yisun1-ubuntu.bj.intel.com ([10.238.156.112]) by orsmga003.jf.intel.com with ESMTP; 01 Apr 2017 06:53:30 -0700 From: Yi Sun To: xen-devel@lists.xenproject.org Date: Sat, 1 Apr 2017 21:53:32 +0800 Message-Id: <1491054836-30488-2-git-send-email-yi.y.sun@linux.intel.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1491054836-30488-1-git-send-email-yi.y.sun@linux.intel.com> References: <1491054836-30488-1-git-send-email-yi.y.sun@linux.intel.com> MIME-Version: 1.0 Cc: kevin.tian@intel.com, wei.liu2@citrix.com, andrew.cooper3@citrix.com, dario.faggioli@citrix.com, he.chen@linux.intel.com, ian.jackson@eu.citrix.com, Yi Sun , mengxu@cis.upenn.edu, jbeulich@suse.com, chao.p.peng@linux.intel.com, roger.pau@citrix.com Subject: [Xen-devel] [PATCH v10 01/25] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP This patch creates CAT and CDP feature document in doc/features/. It describes key points to implement L3 CAT/CDP and L2 CAT which is described in details in Intel SDM "INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES". Signed-off-by: Yi Sun Reviewed-by: Konrad Rzeszutek Wilk Reviewed-by: Wei Liu --- v9: - add 'CMT' to the list of acronyms. (suggested by Wei Liu) - change feature list to feature array. - modify data structure descriptions according to latest codes. - modify revision. v8: - change revision info. (suggested by Konrad Rzeszutek Wilk) - add content int 'Areas for improvement'. (suggested by Konrad Rzeszutek Wilk) v7: - correct typo. (suggested by Konrad Rzeszutek Wilk) - replace application/VM to domain. (suggested by Konrad Rzeszutek Wilk) - amend description of `feat_mask` to make it clearer. (suggested by Konrad Rzeszutek Wilk) - update revision. (suggested by Konrad Rzeszutek Wilk) - other minor fixes. (suggested by Konrad Rzeszutek Wilk) v6: - write a new feature document to cover L3 CAT/CDP and L2 CAT. (suggested by Kevin Tian) - adjust 'Terminology' position in document. (suggested by Dario Faggioli) - fix wordings. (suggested by Dario Faggioli, Kevin Tian and Konrad Rzeszutek Wilk) - add SDM chapter title in commit message. (suggested by Konrad Rzeszutek Wilk) - add more explanations. (suggested by Kevin Tian) v4: - change file name to be more descriptive, 'intel_psr_l2_cat.pandoc'. (suggested by Dario Faggioli) - remove 'Ext' and 'New' prefixes. (suggested by Dario Faggioli) - remove change log in Revison part. (suggested by Dario Faggioli) - adjust Xen release number to 4.9 to show this feature targets 4.9. (suggested by Dario Faggioli) - provide 'Terminology' and more sections. (suggested by Dario Faggioli) - fix wordings. (suggested by Konrad Rzeszutek Wilk) - remove chapter number. (suggested by Konrad Rzeszutek Wilk) v3: - make design document be a patch. (suggested by Konrad Rzeszutek Wilk) v2: - provide chapter for the L2 CAT. (suggested by Meng Xu) --- docs/features/intel_psr_cat_cdp.pandoc | 469 +++++++++++++++++++++++++++++++++ 1 file changed, 469 insertions(+) create mode 100644 docs/features/intel_psr_cat_cdp.pandoc diff --git a/docs/features/intel_psr_cat_cdp.pandoc b/docs/features/intel_psr_cat_cdp.pandoc new file mode 100644 index 0000000..022fbdc --- /dev/null +++ b/docs/features/intel_psr_cat_cdp.pandoc @@ -0,0 +1,469 @@ +% Intel Cache Allocation Technology and Code and Data Prioritization Features +% Revision 1.9 + +\clearpage + +# Basics + +---------------- ---------------------------------------------------- + Status: **Tech Preview** + +Architecture(s): Intel x86 + + Component(s): Hypervisor, toolstack + + Hardware: L3 CAT: Haswell and beyond CPUs + CDP : Broadwell and beyond CPUs + L2 CAT: Atom codename Goldmont and beyond CPUs +---------------- ---------------------------------------------------- + +# Terminology + +* CAT Cache Allocation Technology +* CBM Capacity BitMasks +* CDP Code and Data Prioritization +* CMT Cache Monitoring Technology +* COS/CLOS Class of Service +* MSRs Machine Specific Registers +* PSR Intel Platform Shared Resource + +# Overview + +Intel provides a set of allocation capabilities including Cache Allocatation +Technology (CAT) and Code and Data Prioritization (CDP). + +CAT allows an OS or hypervisor to control allocation of a CPU's shared cache +based on application/domain priority or Class of Service (COS). Each COS is +configured using capacity bitmasks (CBMs) which represent cache capacity and +indicate the degree of overlap and isolation between classes. Once CAT is co- +nfigured, the processor allows access to portions of cache according to the +established COS. Intel Xeon processor E5 v4 family (and some others) introduce +capabilities to configure and make use of the CAT mechanism on the L3 cache. +Intel Goldmont processor provides support for control over the L2 cache. + +Code and Data Prioritization (CDP) Technology is an extension of CAT. CDP +enables isolation and separate prioritization of code and data fetches to +the L3 cache in a SW configurable manner, which can enable workload priorit- +ization and tuning of cache capacity to the characteristics of the workload. +CDP extends CAT by providing separate code and data masks per Class of Service +(COS). When SW configures to enable CDP, L3 CAT is disabled. + +# User details + +* Feature Enabling: + + Add "psr=cat" to boot line parameter to enable all supported level CAT featu- + res. Add "psr=cdp" to enable L3 CDP but disables L3 CAT by SW. + +* xl interfaces: + + 1. `psr-cat-show [OPTIONS] domain-id`: + + Show L2 CAT or L3 CAT/CDP CBM of the domain designated by Xen domain-id. + + Option `-l`: + `-l2`: Show cbm for L2 cache. + `-l3`: Show cbm for L3 cache. + + If `-lX` is specified and LX is not supported, print error. + If no `-l` is specified, level 3 is the default option. + + 2. `psr-cat-set [OPTIONS] domain-id cbm`: + + Set L2 CAT or L3 CAT/CDP CBM to the domain designated by Xen domain-id. + + Option `-s`: Specify the socket to process, otherwise all sockets are + processed. + + Option `-l`: + `-l2`: Specify cbm for L2 cache. + `-l3`: Specify cbm for L3 cache. + + If `-lX` is specified and LX is not supported, print error. + If no `-l` is specified, level 3 is the default option. + + Option `-c` or `-d`: + `-c`: Set L3 CDP code cbm. + `-d`: Set L3 CDP data cbm. + + 3. `psr-hwinfo [OPTIONS]`: + + Show CMT & L2 CAT & L3 CAT/CDP HW information on every socket. + + Option `-m, --cmt`: Show Cache Monitoring Technology (CMT) hardware info. + + Option `-a, --cat`: Show CAT/CDP hardware info. + +# Technical details + +L3 CAT/CDP and L2 CAT are all members of Intel PSR features, they share the base +PSR infrastructure in Xen. + +## Hardware perspective + + CAT/CDP defines a range of MSRs to assign different cache access patterns + which are known as CBMs, each CBM is associated with a COS. + + ``` + E.g. L2 CAT: + +----------------------------+----------------+ + IA32_PQR_ASSOC | MSR (per socket) | Address | + +----+---+-------+ +----------------------------+----------------+ + | |COS| | | IA32_L2_QOS_MASK_0 | 0xD10 | + +----+---+-------+ +----------------------------+----------------+ + └-------------> | ... | ... | + +----------------------------+----------------+ + | IA32_L2_QOS_MASK_n | 0xD10+n (n<64) | + +----------------------------+----------------+ + ``` + + L3 CAT/CDP uses a range of MSRs from 0xC90 ~ 0xC90+n (n<128). + + L2 CAT uses a range of MSRs from 0xD10 ~ 0xD10+n (n<64), following the L3 + CAT/CDP MSRs, setting different L2 cache accessing patterns from L3 cache is + supported. + + Every MSR stores a CBM value. A capacity bitmask (CBM) provides a hint to the + hardware indicating the cache space a domain should be limited to as well as + providing an indication of overlap and isolation in the CAT-capable cache from + other domains contending for the cache. + + Sample cache capacity bitmasks for a bitlength of 8 are shown below. Please + note that all (and only) contiguous '1' combinations are allowed (e.g. FFFFH, + 0FF0H, 003CH, etc.). + + ``` + +----+----+----+----+----+----+----+----+ + | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 | + +----+----+----+----+----+----+----+----+ + COS0 | A | A | A | A | A | A | A | A | Default Bitmask + +----+----+----+----+----+----+----+----+ + COS1 | A | A | A | A | A | A | A | A | + +----+----+----+----+----+----+----+----+ + COS2 | A | A | A | A | A | A | A | A | + +----+----+----+----+----+----+----+----+ + + +----+----+----+----+----+----+----+----+ + | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 | + +----+----+----+----+----+----+----+----+ + COS0 | A | A | A | A | A | A | A | A | Overlapped Bitmask + +----+----+----+----+----+----+----+----+ + COS1 | | | | | A | A | A | A | + +----+----+----+----+----+----+----+----+ + COS2 | | | | | | | A | A | + +----+----+----+----+----+----+----+----+ + + +----+----+----+----+----+----+----+----+ + | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 | + +----+----+----+----+----+----+----+----+ + COS0 | A | A | A | A | | | | | Isolated Bitmask + +----+----+----+----+----+----+----+----+ + COS1 | | | | | A | A | | | + +----+----+----+----+----+----+----+----+ + COS2 | | | | | | | A | A | + +----+----+----+----+----+----+----+----+ + ``` + + We can get the CBM length through CPUID. The default value of CBM is calcul- + ated by `(1ull << cbm_len) - 1`. That is a fully open bitmask, all ones bitm- + ask. The COS[0] always stores the default value without change. + + There is a `IA32_PQR_ASSOC` register which stores the COS ID of the VCPU. HW + enforces cache allocation according to the corresponding CBM. + +## The relationship between L3 CAT/CDP and L2 CAT + + HW may support all features. By default, CDP is disabled on the processor. + If the L3 CAT MSRs are used without enabling CDP, the processor operates in + a traditional CAT-only mode. When CDP is enabled: + * the CAT mask MSRs are re-mapped into interleaved pairs of mask MSRs for + data or code fetches. + * the range of COS for CAT is re-indexed, with the lower-half of the COS + range available for CDP. + + L2 CAT is independent of L3 CAT/CDP, which means L2 CAT can be enabled while + L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are both enabled. + + As a requirement, the bits of CBM of CAT/CDP must be continuous. + + N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same associate + register `IA32_PQR_ASSOC`, which means one COS is associated with a pair of + L2 CAT CBM and L3 CAT/CDP CBM. + + Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or other + PSR features in future). In some cases, a domain is permitted to have a COS + that is beyond one (or more) of PSR features but within the others. For + instance, let's assume the max COS of L2 CAT is 8 but the max COS of L3 + CAT is 16, when a domain is assigned 9 as COS, the L3 CAT CBM associated to + COS 9 would be enforced, but for L2 CAT, the HW works as default value is + set since COS 9 is beyond the max COS (8) of L2 CAT. + +## Design Overview + +* Core COS/CBM association + + When enforcing CAT/CDP, all cores of domains have the same default COS (COS0) + which is associated with the fully open CBM (all ones bitmask) to access all + cache. The default COS is used only in hypervisor and is transparent to tool + stack and user. + + System administrator can change PSR allocation policy at runtime by tool stack. + Since L2 CAT shares COS with L3 CAT/CDP, a COS corresponds to a 2-tuple, like + [L2 CBM, L3 CBM] with only-CAT enabled, when CDP is enabled, one COS correspo- + nds to a 3-tuple, like [L2 CBM, L3 Code_CBM, L3 Data_CBM]. If neither L3 CAT + nor L3 CDP is enabled, things would be easier, one COS corresponds to one L2 + CBM. + +* VCPU schedule + + When context switch happens, the COS of VCPU is written to per-thread MSR + `IA32_PQR_ASSOC`, and then hardware enforces cache allocation according to + the corresponding CBM. + +* Multi-sockets + + Different sockets may have different CAT/CDP capability (e.g. max COS) alth- + ough it is consistent on the same socket. So the capability of per-socket CAT/ + CDP is specified. + + 'psr-cat-set' can set CBM for one domain per socket. On each socket, we main- + tain a COS array for all domains. One domain uses one COS at one time. One COS + stores the CBM of the domain to work. So, when a VCPU of the domain is migrat- + ed from socket 1 to socket 2, it follows configuration on socket 2. + + E.g. user sets domain 1 CBM on socket 1 to 0x7f which uses COS 9 but sets do- + main 1 CBM on socket 2 to 0x3f which uses COS 7. When VCPU of this domain + is migrated from socket 1 to 2, the COS ID used is 7, that means 0x3f is the + CBM to work for this domain 1 now. + +## Implementation Description + +* Hypervisor interfaces: + + 1. Boot line parameter "psr=cat" enables L2 CAT and L3 CAT if hardware suppo- + rted. "psr=cdp" enables CDP if hardware supported. + + 2. SYSCTL: + - XEN_SYSCTL_PSR_CAT_get_l3_info: Get L3 CAT/CDP information. + - XEN_SYSCTL_PSR_CAT_get_l2_info: Get L2 CAT information. + + 3. DOMCTL: + - XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM: Get L3 CBM for a domain. + - XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM: Set L3 CBM for a domain. + - XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE: Get CDP Code CBM for a domain. + - XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE: Set CDP Code CBM for a domain. + - XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA: Get CDP Data CBM for a domain. + - XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA: Set CDP Data CBM for a domain. + - XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM: Get L2 CBM for a domain. + - XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM: Set L2 CBM for a domain. + +* xl interfaces: + + 1. psr-cat-show -lX domain-id + Show LX cbm for a domain. + => XEN_SYSCTL_PSR_CAT_get_l3_info / + XEN_SYSCTL_PSR_CAT_get_l2_info / + XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM / + XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE / + XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA / + XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM + + 2. psr-cat-set -lX domain-id cbm + Set LX cbm for a domain. + => XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM / + XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE / + XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA / + XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM + + 3. psr-hwinfo + Show PSR HW information, including L3 CAT/CDP/L2 CAT + => XEN_SYSCTL_PSR_CAT_get_l3_info / + XEN_SYSCTL_PSR_CAT_get_l2_info + +* Key data structure: + + 1. Feature HW info + + ``` + struct psr_cat_hw_info { + unsigned int cbm_len; + unsigned int cos_max; + }; + ``` + + - Member `cbm_len` + + `cbm_len` is one of the hardware info of CAT. It means the max number + of bits to set. + + - Member `cos_max` + + `cos_max` is one of the hardware info of CAT. It means the max number + of COS registers. + + 2. Feature node + + ``` + struct feat_node { + struct feat_ops { + unsigned int (*get_cos_max)(const struct feat_node *feat); + bool (*get_feat_info)(const struct feat_node *feat, + uint32_t data[], unsigned int array_len); + void (*get_val)(const struct feat_node *feat, unsigned int cos, + enum cbm_type type, uint32_t *val); + void (*get_old_val)(uint32_t val[], + const struct feat_node *feat, + unsigned int old_cos); + int (*set_new_val)(uint32_t val[], + const struct feat_node *feat, + enum cbm_type type, + uint32_t new_val); + int (*compare_val)(const uint32_t val[], const struct feat_node *feat, + unsigned int cos); + bool (*fits_cos_max)(const uint32_t val[], + const struct feat_node *feat, + unsigned int cos); + void (*write_msr)(unsigned int cos, uint32_t val, + enum cbm_type type, struct feat_node *feat); + } ops; + + union { + struct psr_cat_hw_info cat_info; + } info; + + uint32_t cos_reg_val[MAX_COS_REG_CNT]; + unsigned int cos_num; + }; + ``` + + When a PSR enforcement feature is enabled, it will be added into a + feature array. + + - Member `ops` + + `ops` maintains a callback function list of the feature. + + We abstract above callback functions to encapsulate the feature specific + behaviors into them. Then, it is easy to add a new feature: + 1) Implement such ops and callback functions for every feature. + 2) Register the ops into `struct feat_node`. + 3) Add the feature into feature array during CPU initialization. + + - Member `info` + + `info` maintains the feature HW information which is provided to + psr_hwinfo command. + + - Member `cos_reg_val` + + `cos_reg_val` is an array to maintain the value set in all COS registers + of the feature. The array is indexed by COS ID. + + - Member `cos_num` + + `cos_num` is the number of COS registers the feature uses, e.g. L3/L2 + CAT uses 1 register but CDP uses 2 registers. + + 3. Per-socket PSR features information structure + + ``` + struct psr_socket_info { + unsigned int feat_mask; + struct feat_node *features[PSR_SOCKET_MAX_FEAT]; + unsigned int cos_ref[MAX_COS_REG_NUM]; + spinlock_t ref_lock; + }; + ``` + + We collect all PSR allocation features information of a socket in this + `struct psr_socket_info`. + + - Member `feat_mask` + + `feat_mask` is a bitmap, to indicate which feature is enabled on current + socket. See values defined in `enum psr_feat_type`. E.g. + + bit 0: L3 CAT status. + bit 1: L3 CDP status. + bit 2: L2 CAT status. + + - Member `features` + + `features` is a pointer array to save all enabled features poniters + according to feature position defined in `enum psr_feat_type`. + + - Member `cos_ref` + + `cos_ref` is an array which maintains the reference of one COS. It maps + to cos_reg_val[MAX_COS_REG_NUM] in `struct feat_node`. If one COS is + used by one domain, the corresponding reference will increase by one. If + a domain releases the COS, the reference will decrease by one. The array + is indexed by COS ID. + +# Limitations + +CAT/CDP can only work on HW which enables it(check by CPUID). So far, there is +no HW which enables both L2 CAT and L3 CAT/CDP. But SW implementation has cons- +idered such scenario to enable both L2 CAT and L3 CAT/CDP. + +# Testing + +We can execute above xl commands to verify L2 CAT and L3 CAT/CDP on different +HWs support them. + +For example: + root@:~$ xl psr-hwinfo --cat + Cache Allocation Technology (CAT): L2 + Socket ID : 0 + Maximum COS : 3 + CBM length : 8 + Default CBM : 0xff + + root@:~$ xl psr-cat-cbm-set -l2 1 0x7f + + root@:~$ xl psr-cat-show -l2 1 + Socket ID : 0 + Default CBM : 0xff + ID NAME CBM + 1 ubuntu14 0x7f + +# Areas for improvement + +A hexadecimal number is used to set/show CBM for a domain now. Although this +is convenient to cover overlap/isolated bitmask requirement, it is not user- +friendly. + +To improve this, the libxl interfaces can be wrapped in libvirt to provide more +usr-friendly interfaces to user, e.g. a percentage number of the cache to set +and show. + +# Known issues + +N/A + +# References + +"INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES" [Intel® 64 and IA-32 Architectures Software Developer Manuals, vol3](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html) + +# History + +------------------------------------------------------------------------ +Date Revision Version Notes +---------- -------- -------- ------------------------------------------- +2016-08-12 1.0 Xen 4.9 Design document written +2017-02-13 1.7 Xen 4.9 Changes: + 1. Modify the design document to cover L3 + CAT/CDP and L2 CAT; + 2. Fix typos; + 3. Amend description of `feat_mask` to make + it clearer; + 4. Other minor changes. +2017-02-15 1.8 Xen 4.9 Changes: + 1. Add content in 'Areas for improvement'; + 2. Adjust revision number. +2017-03-16 1.9 Xen 4.9 Changes: + 1. Add 'CMT' in 'Terminology'; + 2. Change 'feature list' to 'feature array'. + 3. Modify data structure descriptions. + 4. Adjust revision number. +---------- -------- -------- -------------------------------------------