From patchwork Thu Jan 19 06:01:03 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Yi Sun X-Patchwork-Id: 9525235 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id BF02B60437 for ; Thu, 19 Jan 2017 06:05:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AAD992860C for ; Thu, 19 Jan 2017 06:05:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9F70E2862A; Thu, 19 Jan 2017 06:05:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id CF3302860C for ; Thu, 19 Jan 2017 06:05:39 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cU5oA-0006nT-FZ; Thu, 19 Jan 2017 06:02:50 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cU5o9-0006nC-1E for xen-devel@lists.xenproject.org; Thu, 19 Jan 2017 06:02:49 +0000 Received: from [85.158.137.68] by server-11.bemta-3.messagelabs.com id D5/03-01684-88650885; Thu, 19 Jan 2017 06:02:48 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrCIsWRWlGSWpSXmKPExsVywNykQrctrCH CYPUUcYvvWyYzOTB6HP5whSWAMYo1My8pvyKBNeP/s9nMBc+TKl6v0Whg/OTexcjJISRQKXHw wXtWEFtCgFfiyLIZULa/xLEd64FsLqCaBkaJSS+OMIEk2ATUJR5/7QGzRQSUJO6tmswEUsQs0 M0ksfLhVhaQhLBAssTMz1MZQWwWAVWJ+adesoPYvALuEg2rl0JtkJM4eWwymM0p4CFxZf4Fdo iL3CW2rLjEBFEvKHFy5hOgmRxAC9Ql1s8TAgkzC8hLNG+dzTyBUWAWkqpZCFWzkFQtYGRexah RnFpUllqka2ipl1SUmZ5RkpuYmaNraGCsl5taXJyYnpqTmFSsl5yfu4kRGJr1DAyMOxh/H/c7 xCjJwaQkymtv2RAhxJeUn1KZkVicEV9UmpNafIhRhoNDSYJ3TihQTrAoNT21Ii0zBxglMGkJD h4lEd4qkDRvcUFibnFmOkTqFKOilDhvFEhCACSRUZoH1waLzEuMslLCvIwMDAxCPAWpRbmZJa jyrxjFORiVhHlrQKbwZOaVwE1/BbSYCWixlXI9yOKSRISUVAOjp4TyrOCoydbyV+ucsu03bHe 6VT/znV7yd5X+Tfvn/l5+JZHnptcZIyYPZw3Lk16Rpyeev8LQ/p8r99PKH69aGvWna94+9yVp /eSrt3asfcBc13LH3iV6p9X2W9kcQbsYPmtWLnTpvaEppBJvdX1iyfdj6VrfvDqZX+Q/mdfHP EvPSKiJ2/SmEktxRqKhFnNRcSIAVUJooMcCAAA= X-Env-Sender: yi.y.sun@linux.intel.com X-Msg-Ref: server-3.tower-31.messagelabs.com!1484805763!81281826!2 X-Originating-IP: [192.55.52.120] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.1.1; banners=-,-,- X-VirusChecked: Checked Received: (qmail 12258 invoked from network); 19 Jan 2017 06:02:46 -0000 Received: from mga04.intel.com (HELO mga04.intel.com) (192.55.52.120) by server-3.tower-31.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 19 Jan 2017 06:02:46 -0000 Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga104.fm.intel.com with ESMTP; 18 Jan 2017 22:02:41 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.33,252,1477983600"; d="scan'208";a="215119432" Received: from vmmmba-s2600wft.bj.intel.com ([10.240.193.63]) by fmsmga004.fm.intel.com with ESMTP; 18 Jan 2017 22:02:38 -0800 From: Yi Sun To: xen-devel@lists.xenproject.org Date: Thu, 19 Jan 2017 14:01:03 +0800 Message-Id: <1484805686-7249-2-git-send-email-yi.y.sun@linux.intel.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1484805686-7249-1-git-send-email-yi.y.sun@linux.intel.com> References: <1484805686-7249-1-git-send-email-yi.y.sun@linux.intel.com> MIME-Version: 1.0 Cc: wei.liu2@citrix.com, konrad.wilk@oracle.com, andrew.cooper3@citrix.com, dario.faggioli@citrix.com, he.chen@linux.intel.com, ian.jackson@eu.citrix.com, Yi Sun , mengxu@cis.upenn.edu, jbeulich@suse.com, chao.p.peng@linux.intel.com Subject: [Xen-devel] [PATCH RESEND v5 01/24] docs: create L2 Cache Allocation Technology (CAT) feature document X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP This patch creates L2 CAT feature document in doc/features/. It describes details of L2 CAT. Signed-off-by: Yi Sun --- docs/features/intel_psr_l2_cat.pandoc | 347 ++++++++++++++++++++++++++++++++++ 1 file changed, 347 insertions(+) create mode 100644 docs/features/intel_psr_l2_cat.pandoc diff --git a/docs/features/intel_psr_l2_cat.pandoc b/docs/features/intel_psr_l2_cat.pandoc new file mode 100644 index 0000000..77bd61f --- /dev/null +++ b/docs/features/intel_psr_l2_cat.pandoc @@ -0,0 +1,347 @@ +% Intel L2 Cache Allocation Technology (L2 CAT) Feature +% Revision 1.0 + +\clearpage + +# Basics + +---------------- ---------------------------------------------------- + Status: **Tech Preview** + +Architecture(s): Intel x86 + + Component(s): Hypervisor, toolstack + + Hardware: Atom codename Goldmont and beyond CPUs +---------------- ---------------------------------------------------- + +# Overview + +L2 CAT allows an OS or Hypervisor/VMM to control allocation of a +CPU's shared L2 cache based on application priority or Class of Service +(COS). Each CLOS is configured using capacity bitmasks (CBM) which +represent cache capacity and indicate the degree of overlap and +isolation between classes. Once L2 CAT is configured, the processor +allows access to portions of L2 cache according to the established +class of service. + +## Terminology + +* CAT Cache Allocation Technology +* CBM Capacity BitMasks +* CDP Code and Data Prioritization +* COS/CLOS Class of Service +* MSRs Machine Specific Registers +* PSR Intel Platform Shared Resource +* VMM Virtual Machine Monitor + +# User details + +* Feature Enabling: + + Add "psr=cat" to boot line parameter to enable all supported level CAT + features. + +* xl interfaces: + + 1. `psr-cat-show [OPTIONS] domain-id`: + + Show domain L2 or L3 CAT CBM. + + New option `-l` is added. + `-l2`: Show cbm for L2 cache. + `-l3`: Show cbm for L3 cache. + + If neither `-l2` nor `-l3` is given, show both of them. If any one + is not supported, will print error info. + + 2. `psr-cat-cbm-set [OPTIONS] domain-id cbm`: + + Set domain L2 or L3 CBM. + + New option `-l` is added. + `-l2`: Specify cbm for L2 cache. + `-l3`: Specify cbm for L3 cache. + + If neither `-l2` nor `-l3` is given, level 3 is the default option. + + 3. `psr-hwinfo [OPTIONS]`: + + Show L2 & L3 CAT HW informations on every socket. + +# Technical details + +L2 CAT is a member of Intel PSR features and part of CAT, it shares +some base PSR infrastructure in Xen. + +## Hardware perspective + +L2 CAT defines a new range MSRs to assign different L2 cache access +patterns which are known as CBMs, each CBM is associated with a COS. + +``` + + +----------------------------+----------------+ + IA32_PQR_ASSOC | MSR (per socket) | Address | + +----+---+-------+ +----------------------------+----------------+ + | |COS| | | IA32_L2_QOS_MASK_0 | 0xD10 | + +----+---+-------+ +----------------------------+----------------+ + └-------------> | ... | ... | + +----------------------------+----------------+ + | IA32_L2_QOS_MASK_n | 0xD10+n (n<64) | + +----------------------------+----------------+ +``` + +When context switch happens, the COS of VCPU is written to per-thread +MSR `IA32_PQR_ASSOC`, and then hardware enforces L2 cache allocation +according to the corresponding CBM. + +## The relationship between L2 CAT and L3 CAT/CDP + +L2 CAT is independent of L3 CAT/CDP, which means L2 CAT would be enabled +while L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are all enabled. + +L2 CAT uses a new range CBMs from 0xD10 ~ 0xD10+n (n<64), following by +the L3 CAT/CDP CBMs, and supports setting different L2 cache accessing +patterns from L3 cache. Like L3 CAT/CDP requirement, the bits of CBM of +L2 CAT must be continuous too. + +N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same +associate register `IA32_PQR_ASSOC`, which means one COS associates to a +pair of L2 CBM and L3 CBM. + +Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or +other PSR features in future). In some cases, a VM is permitted to have a +COS that is beyond one (or more) of PSR features but within the others. +For instance, let's assume the max COS of L2 CAT is 8 but the max COS of +L3 CAT is 16, when a VM is assigned 9 as COS, the L3 CBM associated to +COS 9 would be enforced, but for L2 CAT, the behavior is fully open (no +limit) since COS 9 is beyond the max COS (8) of L2 CAT. + +## Design Overview + +* Core COS/CBM association + + When enforcing L2 CAT, all cores of domains have the same default + COS (COS0) which associated to the fully open CBM (all ones bitmask) + to access all L2 cache. The default COS is used only in hypervisor + and is transparent to tool stack and user. + + System administrator can change PSR allocation policy at runtime by + tool stack. Since L2 CAT share COS with L3 CAT/CDP, a COS corresponds + to a 2-tuple, like [L2 CBM, L3 CBM] with only-CAT enabled, when CDP + is enabled, one COS corresponds to a 3-tuple, like [L2 CBM, + L3 Code_CBM, L3 Data_CBM]. If neither L3 CAT nor L3 CDP is enabled, + things would be easier, one COS corresponds to one L2 CBM. + +* VCPU schedule + + This part reuses L3 CAT COS infrastructure. + +* Multi-sockets + + Different sockets may have different L2 CAT capability (e.g. max COS) + although it is consistent on the same socket. So the capability of + per-socket L2 CAT is specified. + +## Implementation Description + +* Hypervisor interfaces: + + 1. Boot line parameter "psr=cat" now will enable L2 CAT and L3 + CAT if hardware supported. + + 2. SYSCTL: + - XEN_SYSCTL_PSR_CAT_get_l2_info: Get L2 CAT information. + + 3. DOMCTL: + - XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM: Get L2 CBM for a domain. + - XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM: Set L2 CBM for a domain. + +* xl interfaces: + + 1. psr-cat-show -l2 domain-id + Show L2 cbm for a domain. + => XEN_SYSCTL_PSR_CAT_get_l2_info / + XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM + + 2. psr-mba-set -l2 domain-id cbm + Set L2 cbm for a domain. + => XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM + + 3. psr-hwinfo + Show PSR HW information, including L2 CAT + => XEN_SYSCTL_PSR_CAT_get_l2_info + +* Key data structure: + + 1. Feature HW info + + ``` + struct psr_cat_hw_info { + unsigned int cbm_len; + unsigned int cos_max; + }; + ``` + + - Member `cbm_len` + + `cbm_len` is one of the hardware info of CAT. + + - Member `cos_max` + + `cos_max` is one of the hardware info of CAT. + + 2. Feature list node + + ``` + struct feat_node { + enum psr_feat_type feature; + struct feat_ops ops; + struct psr_cat_hw_info info; + uint64_t cos_reg_val[MAX_COS_REG_NUM]; + struct list_head list; + }; + ``` + + When a PSR enforcement feature is enabled, it will be added into a + feature list. The head of the list is created in psr initialization. + + - Member `feature` + + `feature` is an integer number, to indicate which feature the list entry + corresponds to. + + - Member `ops` + + `ops` maintains a callback function list of the feature. It will be introduced + in details later. + + - Member `info` + + `info` maintains the feature HW information which can be got through + psr_hwinfo command. + + - Member `cos_reg_val` + + `cos_reg_val` is an array to maintain the value set in all COS registers of + the feature. + + 3. Per-socket PSR features information structure + + ``` + struct psr_cat_socket_info { + unsigned int feat_mask; + unsigned int nr_feat; + struct list_head feat_list; + unsigned int cos_ref[MAX_COS_REG_NUM]; + spinlock_t ref_lock; + }; + ``` + + We collect all PSR allocation features information of a socket in + this `struct psr_cat_socket_info`. + + - Member `feat_mask` + + `feat_mask` is a bitmap, to indicate which feature is enabled on + current socket. We define `feat_mask` bitmap as: + + bit 0~1: L3 CAT status, [01] stands for L3 CAT only and [10] + stands for L3 CDP is enalbed. + + bit 2: L2 CAT status. + + - Member `cos_ref` + + `cos_ref` is an array which maintains the reference of one COS. + If the COS is used by one domain, the reference will increase one. + If a domain releases the COS, the reference will decrease one. The + array is indexed by COS. + + 4. Feature operation functions structure + + ``` + struct feat_ops { + void (*init_feature)(unsigned int eax, unsigned int ebx, + unsigned int ecx, unsigned int edx, + struct feat_node *feat, + struct psr_cat_socket_info *info); + int (*get_feat_info)(const struct feat_node *feat, enum cbm_type type, + uint32_t dat[], uint32_t array_len); + int (*get_val)(const struct feat_node *feat, unsigned int cos, + enum cbm_type type, uint64_t *val); + unsigned int (*get_max_cos_max)(const struct feat_node *feat); + unsigned int (*get_cos_num)(const struct feat_node *feat); + int (*get_old_val)(uint64_t val[], + const struct feat_node *feat, + unsigned int old_cos); + int (*set_new_val)(uint64_t val[], + const struct feat_node *feat, + unsigned int old_cos, + enum cbm_type type, + uint64_t m); + int (*compare_val)(const uint64_t val[], const struct feat_node *feat, + unsigned int cos, bool *found); + unsigned int (*get_cos_max_from_type)(const struct feat_node *feat, + enum cbm_type type); + unsigned int (*exceeds_cos_max)(const uint64_t val[], + const struct feat_node *feat, + unsigned int cos); + int (*write_msr)(unsigned int cos, const uint64_t val[], + struct feat_node *feat); + }; + ``` + + We abstract above callback functions to encapsulate the feature specific + behaviors into them. Then, it is easy to add a new feature. We just need: + 1) Implement such ops and callback functions for every feature. + 2) Register the ops into `struct feat_node`. + 3) Add the feature into feature list during CPU initialization. + +# Limitations + +L2 CAT can only work on HW which enables it(check by CPUID). So far, there +is no HW enables both L2 CAT and L3 CAT/CDP. But SW implementation has considered +such scenario to enable both L2 CAT and L3 CAT/CDP. + +# Testing + +L2 CAT uses same xl interfaces as L3 CAT/CDP. So, we can execute these +commands to verify L2 CAT and L3 CAT/CDP on different HWs support them. + +For example: + root@:~$ xl psr-hwinfo --cat + Cache Allocation Technology (CAT): L2 + Socket ID : 0 + Maximum COS : 3 + CBM length : 8 + Default CBM : 0xff + + root@:~$ xl psr-cat-cbm-set -l2 1 0x7f + + root@:~$ xl psr-cat-show -l2 1 + Socket ID : 0 + Default CBM : 0xff + ID NAME CBM + 1 ubuntu14 0x7f + +# Areas for improvement + +N/A + +# Known issues + +N/A + +# References + +"INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES" [Intel® 64 and IA-32 Architectures Software Developer Manuals, vol3](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html) + +# History + +------------------------------------------------------------------------ +Date Revision Version Notes +---------- -------- -------- ------------------------------------------- +2016-08-12 1.0 Xen 4.9 Design document written +---------- -------- -------- -------------------------------------------