From patchwork Wed Jan 18 02:02:31 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Sun X-Patchwork-Id: 9522461 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A03EA6043D for ; Wed, 18 Jan 2017 02:06:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 918802854A for ; Wed, 18 Jan 2017 02:06:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8631D28544; Wed, 18 Jan 2017 02:06:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id AB7E728544 for ; Wed, 18 Jan 2017 02:06:37 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cTfbn-0005bg-Sj; Wed, 18 Jan 2017 02:04:19 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cTfbm-0005ad-Gu for xen-devel@lists.xenproject.org; Wed, 18 Jan 2017 02:04:18 +0000 Received: from [85.158.137.68] by server-2.bemta-3.messagelabs.com id B0/50-09690-12DCE785; Wed, 18 Jan 2017 02:04:17 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrLLMWRWlGSWpSXmKPExsVywNykQlfhbF2 EQd9FE4vvWyYzOTB6HP5whSWAMYo1My8pvyKBNePZBfuC/tiKCyuusjQwPnHoYuTkEBKolNi4 cxYjiC0hwCtxZNkMVgjbT2JP31pmiJoGRol97wRBbDYBdYnHX3uYQGwRASWJe6smA9lcHMwC3 UwSKx9uZQFJCAv4SLx52w42iEVAVeL0129gDbwC7hKtd9dALZCTOHlsMpjNKeAh0Xt+NivEMn eJS6fXsk1g5F3AyLCKUaM4tagstUjXyEgvqSgzPaMkNzEzR9fQwFgvN7W4ODE9NScxqVgvOT9 3EyMwGOoZGBh3ME494XeIUZKDSUmUt+NxbYQQX1J+SmVGYnFGfFFpTmrxIUYZDg4lCd57p+si hASLUtNTK9Iyc4BhCZOW4OBREuG9AJLmLS5IzC3OTIdInWJUlBLnPQaSEABJZJTmwbXBYuESo 6yUMC8jAwODEE9BalFuZgmq/CtGcQ5GJWFeyTNAU3gy80rgpr8CWswEtPi6TjXI4pJEhJRUA2 PV25mzrz2vUd7fuFumP7mi4Nyjp/k79l/ke3zhN7PvvoUr9/BdWLZ3x+5pzueDRd9v/dkTvOD qi9mJ3B8nrpj08cubnjPCuTt+ivIy78vau+y6Yt28ui9XV3E3+vyKynj9/MaalfpOM5u/Oh+8 edf8ca6G3mNFp9Q/UysfP/e2M7r8PXp7zA3d3UosxRmJhlrMRcWJAA5qTY2AAgAA X-Env-Sender: yi.y.sun@linux.intel.com X-Msg-Ref: server-8.tower-31.messagelabs.com!1484705037!81141171!9 X-Originating-IP: [192.55.52.120] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.1.1; banners=-,-,- X-VirusChecked: Checked Received: (qmail 15566 invoked from network); 18 Jan 2017 02:04:16 -0000 Received: from mga04.intel.com (HELO mga04.intel.com) (192.55.52.120) by server-8.tower-31.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 18 Jan 2017 02:04:16 -0000 Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga104.fm.intel.com with ESMTP; 17 Jan 2017 18:04:15 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.33,247,1477983600"; d="scan'208";a="54503286" Received: from vmmmba-s2600wft.bj.intel.com ([10.240.193.63]) by fmsmga005.fm.intel.com with ESMTP; 17 Jan 2017 18:04:13 -0800 From: Yi Sun To: xen-devel@lists.xenproject.org Date: Wed, 18 Jan 2017 10:02:31 +0800 Message-Id: <1484704967-5609-9-git-send-email-yi.y.sun@linux.intel.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1484704967-5609-1-git-send-email-yi.y.sun@linux.intel.com> References: <1484704967-5609-1-git-send-email-yi.y.sun@linux.intel.com> Cc: wei.liu2@citrix.com, konrad.wilk@oracle.com, andrew.cooper3@citrix.com, dario.faggioli@citrix.com, he.chen@linux.intel.com, ian.jackson@eu.citrix.com, Yi Sun , mengxu@cis.upenn.edu, jbeulich@suse.com, chao.p.peng@linux.intel.com Subject: [Xen-devel] [PATCH v5 08/24] x86: refactor psr: set value: implement framework. X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP As set value flow is the most complicated one in psr, it will be divided to some patches to make things clearer. This patch implements the set value framework to show a whole picture firstly. It also changes domctl interface to make it more general. To make the set value flow be general and can support multiple features at same time, it includes below steps: 1. Get COS ID of current domain using. 2. Assemble a value array to store all features current value in it and replace the current value of the feature which is being set to the new input value. 3. Find if there is already a COS ID on which all features' values are same as the array. Then, we can reuse this COS ID. 4. If fail to find, we need pick an available COS ID. Only COS ID which ref is 0 or 1 can be picked. 5. Write all features MSRs according to the COS ID. 6. Update ref according to COS ID. 7. Save the COS ID into current domain's psr_cos_ids[socket] so that we can know which COS the domain is using on the socket. So, some functions are abstracted and the callback functions will be implemented in next patches. Here is an example to understand the process. The CPU supports two featuers, e.g. L3 CAT and L2 CAT. user wants to set L3 CAT of Dom1 to 0x1ff. 1. Get the old_cos of Dom1 which is 0. L3 CAT is the first element of feature list. The COS registers values are below at this time. ------------------------------- | COS 0 | COS 1 | COS 2 | ... | ------------------------------- L3 CAT | 0x7ff | ... | ... | ... | ------------------------------- L2 CAT | 0xff | ... | ... | ... | ------------------------------- 2. Assemble The value array to be: val[0]: 0x1ff val[1]: 0xff 3. It cannot find a matching COS. 4. Allocate COS 1 to store the value set. 5. Write the COS 1 registers. The COS registers values are changed to below now. ------------------------------- | COS 0 | COS 1 | COS 2 | ... | ------------------------------- L3 CAT | 0x7ff | 0x1ff | ... | ... | ------------------------------- L2 CAT | 0xff | 0xff | ... | ... | ------------------------------- 6. The ref[1] is increased to 1 because Dom1 is using it now. 7. Save 1 to Dom1's psr_cos_ids[socket]. Then, user wants to set L3 CAT of Dom2 to 0x1ff too. The old_cos of Dom2 is 0 too. Repeat above flow. The val array assembled is: val[0]: 0x1ff val[1]: 0xff So, it can find a matching COS, COS 1. Then, it can reuse COS 1 for Dom2. The ref[1] is increased to 2 now because both Dom1 and Dom2 are using this COS ID. Set 1 to Dom2's psr_cos_ids[socket]. Signed-off-by: Yi Sun --- xen/arch/x86/domctl.c | 18 ++--- xen/arch/x86/psr.c | 202 +++++++++++++++++++++++++++++++++++++++++++++- xen/include/asm-x86/psr.h | 4 +- 3 files changed, 210 insertions(+), 14 deletions(-) diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c index 11d2127..db56500 100644 --- a/xen/arch/x86/domctl.c +++ b/xen/arch/x86/domctl.c @@ -1365,21 +1365,21 @@ long arch_do_domctl( switch ( domctl->u.psr_cat_op.cmd ) { case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM: - ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target, - domctl->u.psr_cat_op.data, - PSR_CBM_TYPE_L3); + ret = psr_set_val(d, domctl->u.psr_cat_op.target, + domctl->u.psr_cat_op.data, + PSR_CBM_TYPE_L3); break; case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE: - ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target, - domctl->u.psr_cat_op.data, - PSR_CBM_TYPE_L3_CODE); + ret = psr_set_val(d, domctl->u.psr_cat_op.target, + domctl->u.psr_cat_op.data, + PSR_CBM_TYPE_L3_CODE); break; case XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA: - ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target, - domctl->u.psr_cat_op.data, - PSR_CBM_TYPE_L3_DATA); + ret = psr_set_val(d, domctl->u.psr_cat_op.target, + domctl->u.psr_cat_op.data, + PSR_CBM_TYPE_L3_DATA); break; case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM: diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c index 3cbb60c..050b0df 100644 --- a/xen/arch/x86/psr.c +++ b/xen/arch/x86/psr.c @@ -539,18 +539,214 @@ int psr_get_val(struct domain *d, unsigned int socket, return -ENOENT; } -int psr_set_l3_cbm(struct domain *d, unsigned int socket, - uint64_t cbm, enum cbm_type type) +/* Set value functions */ +static unsigned int get_cos_num(const struct psr_socket_info *info) { return 0; } +static int assemble_val_array(uint64_t *val, + uint32_t array_len, + const struct psr_socket_info *info, + unsigned int old_cos) +{ + return -EINVAL; +} + +static int set_new_val_to_array(uint64_t *val, + uint32_t array_len, + const struct psr_socket_info *info, + enum psr_feat_type feat_type, + enum cbm_type type, + uint64_t m) +{ + return -EINVAL; +} + +static int find_cos(const uint64_t *val, uint32_t array_len, + enum psr_feat_type feat_type, + const struct psr_socket_info *info) +{ + return -ENOENT; +} + +static int pick_avail_cos(const struct psr_socket_info *info, + const uint64_t *val, uint32_t array_len, + unsigned int old_cos, + enum psr_feat_type feat_type) +{ + return -ENOENT; +} + +static int write_psr_msr(unsigned int socket, unsigned int cos, + const uint64_t *val) +{ + return -ENOENT; +} + +int psr_set_val(struct domain *d, unsigned int socket, + uint64_t val, enum cbm_type type) +{ + unsigned int old_cos; + int cos, ret; + unsigned int *ref; + uint64_t *val_array; + struct psr_socket_info *info = get_socket_info(socket); + uint32_t array_len; + enum psr_feat_type feat_type; + + if ( IS_ERR(info) ) + return PTR_ERR(info); + + feat_type = psr_cbm_type_to_feat_type(type); + if ( !test_bit(feat_type, &info->feat_mask) ) + return -ENOENT; + + /* + * Step 0: + * old_cos means the COS ID current domain is using. By default, it is 0. + * + * For every COS ID, there is a reference count to record how many domains + * are using the COS register corresponding to this COS ID. + * - If ref[old_cos] is 0, that means this COS is not used by any domain. + * - If ref[old_cos] is 1, that means this COS is only used by current + * domain. + * - If ref[old_cos] is more than 1, that mean multiple domains are using + * this COS. + */ + old_cos = d->arch.psr_cos_ids[socket]; + if ( old_cos > MAX_COS_REG_CNT ) + return -EOVERFLOW; + + ref = info->cos_ref; + + /* + * Step 1: + * Assemle a value array to store all featues cos_reg_val[old_cos]. + * And, set the input val into array according to the feature's + * position in array. + */ + array_len = get_cos_num(info); + val_array = xzalloc_array(uint64_t, array_len); + if ( !val_array ) + return -ENOMEM; + + if ( (ret = assemble_val_array(val_array, array_len, info, old_cos)) != 0 ) + { + xfree(val_array); + return ret; + } + + if ( (ret = set_new_val_to_array(val_array, array_len, info, + feat_type, type, val)) != 0 ) + { + xfree(val_array); + return ret; + } + + /* + * Lock here to make sure the ref is not changed during find and + * write process. + */ + spin_lock(&info->ref_lock); + + /* + * Step 2: + * Try to find if there is already a COS ID on which all features' values + * are same as the array. Then, we can reuse this COS ID. + */ + cos = find_cos(val_array, array_len, feat_type, info); + if ( cos >= 0 ) + { + if ( cos == old_cos ) + { + spin_unlock(&info->ref_lock); + xfree(val_array); + return 0; + } + } + else + { + /* + * Step 3: + * If fail to find, we need allocate a new COS ID. + * If multiple domains are using same COS ID, its ref is more + * than 1. That means we cannot free this COS to make current domain + * use it. Because other domains are using the value saved in the COS. + * Unless the ref is changed to 1 (mean only current domain is using + * it), we cannot allocate the COS ID to current domain. + * So, only the COS ID which ref is 1 or 0 can be allocated. + */ + cos = pick_avail_cos(info, val_array, array_len, old_cos, feat_type); + if ( cos < 0 ) + { + spin_unlock(&info->ref_lock); + xfree(val_array); + return cos; + } + + /* + * Step 4: + * Write all features MSRs according to the COS ID. + */ + ret = write_psr_msr(socket, cos, val_array); + if ( ret ) + { + spin_unlock(&info->ref_lock); + xfree(val_array); + return ret; + } + } + + /* + * Step 5: + * Update ref according to COS ID. + */ + ref[cos]++; + ASSERT(ref[cos] || cos == 0); + ref[old_cos]--; + spin_unlock(&info->ref_lock); + + /* + * Step 6: + * Save the COS ID into current domain's psr_cos_ids[] so that we can know + * which COS the domain is using on the socket. One domain can only use + * one COS ID at same time on each socket. + */ + d->arch.psr_cos_ids[socket] = cos; + xfree(val_array); + + return 0; +} + /* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */ static void psr_free_cos(struct domain *d) { - if( !d->arch.psr_cos_ids ) + unsigned int socket, cos; + + if ( !d->arch.psr_cos_ids ) return; + /* Domain is free so its cos_ref should be decreased. */ + for ( socket = 0; socket < nr_sockets; socket++ ) + { + struct psr_socket_info *info; + + /* cos 0 is default one which does not need be handled. */ + if ( (cos = d->arch.psr_cos_ids[socket]) == 0 ) + continue; + + /* + * If domain uses other cos ids, all corresponding refs must have been + * increased 1 for this domain. So, we need decrease them. + */ + info = socket_info + socket; + ASSERT(info->cos_ref[cos] || cos == 0); + spin_lock(&info->ref_lock); + info->cos_ref[cos]--; + spin_unlock(&info->ref_lock); + } + xfree(d->arch.psr_cos_ids); d->arch.psr_cos_ids = NULL; } diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h index d50e359..97214fe 100644 --- a/xen/include/asm-x86/psr.h +++ b/xen/include/asm-x86/psr.h @@ -72,8 +72,8 @@ int psr_get_info(unsigned int socket, enum cbm_type type, uint32_t data[], unsigned int array_len); int psr_get_val(struct domain *d, unsigned int socket, uint64_t *val, enum cbm_type type); -int psr_set_l3_cbm(struct domain *d, unsigned int socket, - uint64_t cbm, enum cbm_type type); +int psr_set_val(struct domain *d, unsigned int socket, + uint64_t val, enum cbm_type type); int psr_domain_init(struct domain *d); void psr_domain_free(struct domain *d);