From patchwork Sat Apr 1 13:53:34 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Sun X-Patchwork-Id: 9657971 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 1A63C60349 for ; Sat, 1 Apr 2017 13:56:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1094F285C2 for ; Sat, 1 Apr 2017 13:56:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0539128616; Sat, 1 Apr 2017 13:56:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 48538285C2 for ; Sat, 1 Apr 2017 13:56:30 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cuJTM-0005T2-3f; Sat, 01 Apr 2017 13:53:44 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cuJTK-0005So-Tj for xen-devel@lists.xenproject.org; Sat, 01 Apr 2017 13:53:43 +0000 Received: from [85.158.137.68] by server-2.bemta-3.messagelabs.com id A7/46-01903-6E0BFD85; Sat, 01 Apr 2017 13:53:42 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpjkeJIrShJLcpLzFFi42Jpa+sQ0X264X6 EwacnQhbft0xmcmD0OPzhCksAYxRrZl5SfkUCa8aT1iXMBedDKjZMLmxgPGzTxcjFwSJwi0ni xeGZTCCOkMA0Romls9uZuxg5OSQEeCWOLJvBCmH7S9xqegxmCwnUS8zc3sUCYrMJqEs8/trDB GKLCChJ3Fs1GWwQs8BOJol1p7+DDRIW8JW4+vkCWDOLgKrE0n+fGLsYOTh4BTwkpn0wgZgvJ3 Hy2GSwEk4BT4krL/axQOzykFg16RLzBEa+BYwMqxg1ilOLylKLdA0t9JKKMtMzSnITM3N0DQ2 M9XJTi4sT01NzEpOK9ZLzczcxAsOknoGBcQfj79OehxglOZiURHm/F9+LEOJLyk+pzEgszogv Ks1JLT7EKMPBoSTBqwAMOyHBotT01Iq0zBxgwMKkJTh4lER4j60HSvMWFyTmFmemQ6ROMepy3 GrY84ZJiCUvPy9VSpz3GUiRAEhRRmke3AhY9FxilJUS5mVkYGAQ4ilILcrNLEGVf8UozsGoJM wrDzKFJzOvBG7TK6AjmICOsPh6F+SIkkSElFQDo5v+A4m/WRmHe/TOpz/x3xmlccF5PsdTOV3 T4lsvIxrPfnYXr2uv3m62V+zNigmbVrxw1jWrMV0jsuJXtv7F6dlf+gQd1jKcqp++6brCKq1j rm+9424VWZ70UKha//LkhZp7PjenzXny4HxZ/xHOU7LvU1bPCj8l0e+2d15N3QQeZpWtyoYJ7 5VYijMSDbWYi4oTAZVqDx2ZAgAA X-Env-Sender: yi.y.sun@linux.intel.com X-Msg-Ref: server-9.tower-31.messagelabs.com!1491054811!37933881!4 X-Originating-IP: [134.134.136.20] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTM0LjEzNC4xMzYuMjAgPT4gMzU1MzU4\n X-StarScan-Received: X-StarScan-Version: 9.2.3; banners=-,-,- X-VirusChecked: Checked Received: (qmail 49809 invoked from network); 1 Apr 2017 13:53:40 -0000 Received: from mga02.intel.com (HELO mga02.intel.com) (134.134.136.20) by server-9.tower-31.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 1 Apr 2017 13:53:40 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=intel.com; i=@intel.com; q=dns/txt; s=intel; t=1491054820; x=1522590820; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=vEv6V19zJuMcCAK2McrwQR0sTruIGrVt0vlgcshNiUY=; b=DHzzyX214f/Hfij1mWg9+4PFDdnNukHP3UH9T5F03gWSvQFgY+5OM4PV 6Ezh8B1ISKbn2vfCNOQYa906wB7qMQ==; Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 01 Apr 2017 06:53:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.36,258,1486454400"; d="scan'208";a="950544413" Received: from yisun1-ubuntu.bj.intel.com ([10.238.156.112]) by orsmga003.jf.intel.com with ESMTP; 01 Apr 2017 06:53:36 -0700 From: Yi Sun To: xen-devel@lists.xenproject.org Date: Sat, 1 Apr 2017 21:53:34 +0800 Message-Id: <1491054836-30488-4-git-send-email-yi.y.sun@linux.intel.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1491054836-30488-1-git-send-email-yi.y.sun@linux.intel.com> References: <1491054836-30488-1-git-send-email-yi.y.sun@linux.intel.com> Cc: kevin.tian@intel.com, wei.liu2@citrix.com, andrew.cooper3@citrix.com, dario.faggioli@citrix.com, he.chen@linux.intel.com, ian.jackson@eu.citrix.com, Yi Sun , mengxu@cis.upenn.edu, jbeulich@suse.com, chao.p.peng@linux.intel.com, roger.pau@citrix.com Subject: [Xen-devel] [PATCH v10 03/25] x86: refactor psr: implement main data structures. X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP To construct an extendible framework, we need analyze PSR features and abstract the common things and feature specific things. Then, encapsulate them into different data structures. By analyzing PSR features, we can get below map. +------+------+------+ --------->| Dom0 | Dom1 | ... | | +------+------+------+ | | |Dom ID | cos_id of domain | V | +-----------------------------------------------------------------------------+ User --------->| PSR | Socket ID | +--------------+---------------+---------------+ | | | Socket0 Info | Socket 1 Info | ... | | | +--------------+---------------+---------------+ | | | cos_id=0 cos_id=1 ... | | | +-----------------------+-----------------------+-----------+ | | |->Ref : | ref 0 | ref 1 | ... | | | | +-----------------------+-----------------------+-----------+ | | | +-----------------------+-----------------------+-----------+ | | |->L3 CAT: | cos 0 | cos 1 | ... | | | | +-----------------------+-----------------------+-----------+ | | | +-----------------------+-----------------------+-----------+ | | |->L2 CAT: | cos 0 | cos 1 | ... | | | | +-----------------------+-----------------------+-----------+ | | | +-----------+-----------+-----------+-----------+-----------+ | | |->CDP : | cos0 code | cos0 data | cos1 code | cos1 data | ... | | | +-----------+-----------+-----------+-----------+-----------+ | +-----------------------------------------------------------------------------+ So, we need define a socket info data structure, 'struct psr_socket_info' to manage information per socket. It contains a reference count array according to COS ID and a feature array to manage all features enabled. Every entry of the reference count array is used to record how many domains are using the COS registers according to the COS ID. For example, L3 CAT and L2 CAT are enabled, Dom1 uses COS_ID=1 registers of both features to save CBM values, like below. +-------+-------+-------+-----+ | COS 0 | COS 1 | COS 2 | ... | +-------+-------+-------+-----+ L3 CAT | 0x7ff | 0x1ff | ... | ... | +-------+-------+-------+-----+ L2 CAT | 0xff | 0xff | ... | ... | +-------+-------+-------+-----+ If Dom2 has same CBM values, it can reuse these registers which COS_ID=1. That means, both Dom1 and Dom2 use same COS registers(ID=1) to keep same L3/L2 values. So, the value of ref[1] is 2 which means 2 domains are using COS_ID 1. To manage a feature, we need define a feature node data structure, 'struct feat_node', to manage feature's specific HW info, its common properties (callback functions - all feature's specific behaviors are encapsulated into these callback functions, and generic values - e.g. the cos_max), the feature independent values, and an array of all COS registers values of this feature. CDP is a special feature which uses two entries of the array for one COS ID. So, the number of CDP COS registers is the half of L3 CAT. E.g. L3 CAT has 16 COS registers, then CDP has 8 COS registers if it is enabled. CDP uses the COS registers array as below. +-----------+-----------+-----------+-----------+-----------+ CDP cos_reg_val[] index: | 0 | 1 | 2 | 3 | ... | +-----------+-----------+-----------+-----------+-----------+ value: | cos0 code | cos0 data | cos1 code | cos1 data | ... | +-----------+-----------+-----------+-----------+-----------+ For more details, please refer SDM and patches to implement 'get value' and 'set value'. Signed-off-by: Yi Sun --- v10: - remove initialization for 'PSR_SOCKET_L3_CAT'. (suggested by Jan Beulich) - rename 'feat_ops' to 'feat_props'. (suggested by Jan Beulich) - move 'cbm_len' to 'feat_props' because it is feature independent so far. (suggested by Jan Beulich) - move 'cos_max' to 'feat_props' because it is feature independent. (suggested by Jan Beulich) - move 'cos_num' to 'feat_props' because it is feature independent. (suggested by Jan Beulich) - remove union 'info' and struct 'psr_cat_hw_info'. - remove 'get_cos_max' from 'feat_props'. (suggested by Jan Beulich) - remove 'feat_mask' from 'psr_socket_info' because we can use 'features[]' to check if any feature is initialized. (suggested by Jan Beulich) - move 'ref_lock' above 'cos_ref'. (suggested by Jan Beulich) - adjust comments and commit message according to above changes. v9: - replace feature list to a feature pointer array. (suggested by Roger Pau) - add 'PSR_SOCKET_MAX_FEAT' in 'enum psr_feat_type' to know features account. (suggested by Roger Pau) - move 'feat_ops' declaration into 'feat_node' structure. (suggested by Roger Pau) - directly use uninon for feature HW info and move its declaration into 'feat_node' structure. (suggested by Roger Pau) - remove 'enum psr_feat_type feature' declared in 'feat_ops' because it is not useful after using feature pointer array. (suggested by Roger Pau) - rename 'l3_cat_info' to 'cat_info' to be used by all CAT/CDP features. - remove 'nr_feat' which is only for a record. (suggested by Jan Beulich) - add 'cos_num' to record how many COS registers are used by a feature in one time access. (suggested by Jan Beulich) - replace 'uint64_t' to 'uint32_t' for cbm value because SDM specifies the max 32 bits for it. (suggested by Jan Beulich) v7: - sort inclusion files position. (suggested by Wei Liu) v6: - make commit message be clearer. (suggested by Konrad Rzeszutek Wilk) - fix wordings. (suggested by Konrad Rzeszutek Wilk) - add comments to explain relationship between 'feat_mask' and 'enum psr_feat_type'. (suggested by Konrad Rzeszutek Wilk) v5: - remove section number. (suggested by Jan Beulich) - remove double blank. (suggested by Jan Beulich) v4: - create this patch because of removing all old CAT/CDP codes to make implementation be more easily understood. (suggested by Jan Beulich) --- xen/arch/x86/psr.c | 86 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 85 insertions(+), 1 deletion(-) diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c index 96a8589..cf352d2 100644 --- a/xen/arch/x86/psr.c +++ b/xen/arch/x86/psr.c @@ -13,16 +13,100 @@ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for * more details. */ -#include #include #include +#include #include #include +/* + * Terminology: + * - CAT Cache Allocation Technology + * - CBM Capacity BitMasks + * - CDP Code and Data Prioritization + * - COS/CLOS Class of Service. Also mean COS registers. + * - COS_MAX Max number of COS for the feature (minus 1) + * - MSRs Machine Specific Registers + * - PSR Intel Platform Shared Resource + */ + #define PSR_CMT (1<<0) #define PSR_CAT (1<<1) #define PSR_CDP (1<<2) +/* + * Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration', + * the MSRs ranging from 0C90H through 0D0FH (inclusive), enables support for + * up to 128 L3 CAT Classes of Service. The COS_ID=[0,127]. + * + * The MSRs ranging from 0D10H through 0D4FH (inclusive), enables support for + * up to 64 L2 CAT COS. The COS_ID=[0,63]. + * + * So, the maximum COS register count of one feature is 128. + */ +#define MAX_COS_REG_CNT 128 + +enum psr_feat_type { + PSR_SOCKET_L3_CAT, + PSR_SOCKET_L3_CDP, + PSR_SOCKET_L2_CAT, + PSR_SOCKET_MAX_FEAT, +}; + +/* + * This structure represents one feature. + * feat_props - Feature properties, including operation callback functions + and feature common values. + * cos_reg_val - Array to store the values of COS registers. One entry stores + * the value of one COS register. + * For L3 CAT and L2 CAT, one entry corresponds to one COS_ID. + * For CDP, two entries correspond to one COS_ID. E.g. + * COS_ID=0 corresponds to cos_reg_val[0] (Data) and + * cos_reg_val[1] (Code). + */ +struct feat_node { + /* + * This structure defines feature operation callback functions. Every + * feature enabled MUST implement such callback functions and register + * them to props. + * + * Feature specific behaviors will be encapsulated into these callback + * functions. Then, the main flows will not be changed when introducing + * a new feature. + * + * Feature independent HW info and common values are also defined in it. + */ + const struct feat_props { + /* + * cos_num, cos_max and cbm_len are common values for all features + * so far. + * cos_num - COS registers number that feature uses for one COS ID. + * It is defined in SDM. + * cos_max - The max COS registers number got through CPUID. + * cbm_len - The length of CBM got through CPUID. + */ + unsigned int cos_num; + unsigned int cos_max; + unsigned int cbm_len; + } *props; + + uint32_t cos_reg_val[MAX_COS_REG_CNT]; +}; + +/* + * PSR features are managed per socket. Below structure defines the members + * used to manage these features. + * features - A feature node array used to manage all features enabled. + * ref_lock - A lock to protect cos_ref. + * cos_ref - A reference count array to record how many domains are using the + * COS ID. Every entry of cos_ref corresponds to one COS ID. + */ +struct psr_socket_info { + struct feat_node *features[PSR_SOCKET_MAX_FEAT]; + spinlock_t ref_lock; + unsigned int cos_ref[MAX_COS_REG_CNT]; +}; + struct psr_assoc { uint64_t val; uint64_t cos_mask;