From patchwork Thu Jan 19 06:01:10 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Sun X-Patchwork-Id: 9525209 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 87AB76045A for ; Thu, 19 Jan 2017 06:05:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 73AB92860C for ; Thu, 19 Jan 2017 06:05:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 688FC2862A; Thu, 19 Jan 2017 06:05:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id A8F352860C for ; Thu, 19 Jan 2017 06:05:20 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cU5oO-0006sq-Ep; Thu, 19 Jan 2017 06:03:04 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cU5oN-0006sG-Ca for xen-devel@lists.xenproject.org; Thu, 19 Jan 2017 06:03:03 +0000 Received: from [85.158.137.68] by server-2.bemta-3.messagelabs.com id 99/EB-09690-69650885; Thu, 19 Jan 2017 06:03:02 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrALMWRWlGSWpSXmKPExsVywNykWHdqWEO EwcfD6hbft0xmcmD0OPzhCksAYxRrZl5SfkUCa8bDJf+ZCnoTKhpXvWNrYOxz7mLk4hASmM4o 8X7FWfYuRk4OCQFeiSPLZrBC2P4S7fvusUMUNTBKTFzxhREkwSagLvH4aw8TiC0ioCRxb9VkJ pAiZoFuJomVD7eygCSEBYIlVuzpBZvKIqAqsW7xaWYQm1fAXeLlwb9sEBvkJE4emwy2jVPAQ+ LK/Atg9UJANVtWXGKawMi7gJFhFaNGcWpRWWqRrqG5XlJRZnpGSW5iZo6uoYGxXm5qcXFiemp OYlKxXnJ+7iZGYEgwAMEOxpenPQ8xSnIwKYny2ls2RAjxJeWnVGYkFmfEF5XmpBYfYpTh4FCS 4FUNBcoJFqWmp1akZeYAgxMmLcHBoyTCywyS5i0uSMwtzkyHSJ1iVJQS530cApQQAElklObBt cEi4hKjrJQwLyPQIUI8BalFuZklqPKvGMU5GJWEeUVBxvNk5pXATX8FtJgJaLGVcj3I4pJEhJ RUA2Pbvr17zoSd/9X+5cP/+6rBuy5xyObNq67UmvG97hHD/HWvud/n1C5fvmriPYfmFbmqWRq BAmn9HrMvnWetF4lecuha4YMtKW8uz1j07nLgDKbUTA2LFO3j2h8TX6xrTWw8lrB2buqfW3Mb J6/j35js+dNtp4zml5hXM+pLBJy7THb0n/JRdnysxFKckWioxVxUnAgABxSaIoMCAAA= X-Env-Sender: yi.y.sun@linux.intel.com X-Msg-Ref: server-8.tower-31.messagelabs.com!1484805772!81354767!5 X-Originating-IP: [192.55.52.115] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.1.1; banners=-,-,- X-VirusChecked: Checked Received: (qmail 33160 invoked from network); 19 Jan 2017 06:03:00 -0000 Received: from mga14.intel.com (HELO mga14.intel.com) (192.55.52.115) by server-8.tower-31.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 19 Jan 2017 06:03:00 -0000 Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 18 Jan 2017 22:03:00 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.33,252,1477983600"; d="scan'208";a="215119488" Received: from vmmmba-s2600wft.bj.intel.com ([10.240.193.63]) by fmsmga004.fm.intel.com with ESMTP; 18 Jan 2017 22:02:58 -0800 From: Yi Sun To: xen-devel@lists.xenproject.org Date: Thu, 19 Jan 2017 14:01:10 +0800 Message-Id: <1484805686-7249-9-git-send-email-yi.y.sun@linux.intel.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1484805686-7249-1-git-send-email-yi.y.sun@linux.intel.com> References: <1484805686-7249-1-git-send-email-yi.y.sun@linux.intel.com> Cc: wei.liu2@citrix.com, konrad.wilk@oracle.com, andrew.cooper3@citrix.com, dario.faggioli@citrix.com, he.chen@linux.intel.com, ian.jackson@eu.citrix.com, Yi Sun , mengxu@cis.upenn.edu, jbeulich@suse.com, chao.p.peng@linux.intel.com Subject: [Xen-devel] [PATCH RESEND v5 08/24] x86: refactor psr: set value: implement framework. X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP As set value flow is the most complicated one in psr, it will be divided to some patches to make things clearer. This patch implements the set value framework to show a whole picture firstly. It also changes domctl interface to make it more general. To make the set value flow be general and can support multiple features at same time, it includes below steps: 1. Get COS ID of current domain using. 2. Assemble a value array to store all features current value in it and replace the current value of the feature which is being set to the new input value. 3. Find if there is already a COS ID on which all features' values are same as the array. Then, we can reuse this COS ID. 4. If fail to find, we need pick an available COS ID. Only COS ID which ref is 0 or 1 can be picked. 5. Write all features MSRs according to the COS ID. 6. Update ref according to COS ID. 7. Save the COS ID into current domain's psr_cos_ids[socket] so that we can know which COS the domain is using on the socket. So, some functions are abstracted and the callback functions will be implemented in next patches. Here is an example to understand the process. The CPU supports two featuers, e.g. L3 CAT and L2 CAT. user wants to set L3 CAT of Dom1 to 0x1ff. 1. Get the old_cos of Dom1 which is 0. L3 CAT is the first element of feature list. The COS registers values are below at this time. ------------------------------- | COS 0 | COS 1 | COS 2 | ... | ------------------------------- L3 CAT | 0x7ff | ... | ... | ... | ------------------------------- L2 CAT | 0xff | ... | ... | ... | ------------------------------- 2. Assemble The value array to be: val[0]: 0x1ff val[1]: 0xff 3. It cannot find a matching COS. 4. Allocate COS 1 to store the value set. 5. Write the COS 1 registers. The COS registers values are changed to below now. ------------------------------- | COS 0 | COS 1 | COS 2 | ... | ------------------------------- L3 CAT | 0x7ff | 0x1ff | ... | ... | ------------------------------- L2 CAT | 0xff | 0xff | ... | ... | ------------------------------- 6. The ref[1] is increased to 1 because Dom1 is using it now. 7. Save 1 to Dom1's psr_cos_ids[socket]. Then, user wants to set L3 CAT of Dom2 to 0x1ff too. The old_cos of Dom2 is 0 too. Repeat above flow. The val array assembled is: val[0]: 0x1ff val[1]: 0xff So, it can find a matching COS, COS 1. Then, it can reuse COS 1 for Dom2. The ref[1] is increased to 2 now because both Dom1 and Dom2 are using this COS ID. Set 1 to Dom2's psr_cos_ids[socket]. Signed-off-by: Yi Sun --- v5: - modify commit message because of function name change. - change 'alloc_new_cos' to 'pick_avail_cos' to make name accurate. - divide 'get_old_set_new' to two functions, 'assemble_val_array' and 'set_new_val_to_array'. - check feature type when entering 'psr_set_val'. - remove cast. - use ASSERT to check ref. - rename 'dat[]' to 'data[]' --- xen/arch/x86/domctl.c | 18 ++--- xen/arch/x86/psr.c | 202 +++++++++++++++++++++++++++++++++++++++++++++- xen/include/asm-x86/psr.h | 4 +- 3 files changed, 210 insertions(+), 14 deletions(-) diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c index 11d2127..db56500 100644 --- a/xen/arch/x86/domctl.c +++ b/xen/arch/x86/domctl.c @@ -1365,21 +1365,21 @@ long arch_do_domctl( switch ( domctl->u.psr_cat_op.cmd ) { case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM: - ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target, - domctl->u.psr_cat_op.data, - PSR_CBM_TYPE_L3); + ret = psr_set_val(d, domctl->u.psr_cat_op.target, + domctl->u.psr_cat_op.data, + PSR_CBM_TYPE_L3); break; case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE: - ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target, - domctl->u.psr_cat_op.data, - PSR_CBM_TYPE_L3_CODE); + ret = psr_set_val(d, domctl->u.psr_cat_op.target, + domctl->u.psr_cat_op.data, + PSR_CBM_TYPE_L3_CODE); break; case XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA: - ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target, - domctl->u.psr_cat_op.data, - PSR_CBM_TYPE_L3_DATA); + ret = psr_set_val(d, domctl->u.psr_cat_op.target, + domctl->u.psr_cat_op.data, + PSR_CBM_TYPE_L3_DATA); break; case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM: diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c index 3cbb60c..050b0df 100644 --- a/xen/arch/x86/psr.c +++ b/xen/arch/x86/psr.c @@ -539,18 +539,214 @@ int psr_get_val(struct domain *d, unsigned int socket, return -ENOENT; } -int psr_set_l3_cbm(struct domain *d, unsigned int socket, - uint64_t cbm, enum cbm_type type) +/* Set value functions */ +static unsigned int get_cos_num(const struct psr_socket_info *info) { return 0; } +static int assemble_val_array(uint64_t *val, + uint32_t array_len, + const struct psr_socket_info *info, + unsigned int old_cos) +{ + return -EINVAL; +} + +static int set_new_val_to_array(uint64_t *val, + uint32_t array_len, + const struct psr_socket_info *info, + enum psr_feat_type feat_type, + enum cbm_type type, + uint64_t m) +{ + return -EINVAL; +} + +static int find_cos(const uint64_t *val, uint32_t array_len, + enum psr_feat_type feat_type, + const struct psr_socket_info *info) +{ + return -ENOENT; +} + +static int pick_avail_cos(const struct psr_socket_info *info, + const uint64_t *val, uint32_t array_len, + unsigned int old_cos, + enum psr_feat_type feat_type) +{ + return -ENOENT; +} + +static int write_psr_msr(unsigned int socket, unsigned int cos, + const uint64_t *val) +{ + return -ENOENT; +} + +int psr_set_val(struct domain *d, unsigned int socket, + uint64_t val, enum cbm_type type) +{ + unsigned int old_cos; + int cos, ret; + unsigned int *ref; + uint64_t *val_array; + struct psr_socket_info *info = get_socket_info(socket); + uint32_t array_len; + enum psr_feat_type feat_type; + + if ( IS_ERR(info) ) + return PTR_ERR(info); + + feat_type = psr_cbm_type_to_feat_type(type); + if ( !test_bit(feat_type, &info->feat_mask) ) + return -ENOENT; + + /* + * Step 0: + * old_cos means the COS ID current domain is using. By default, it is 0. + * + * For every COS ID, there is a reference count to record how many domains + * are using the COS register corresponding to this COS ID. + * - If ref[old_cos] is 0, that means this COS is not used by any domain. + * - If ref[old_cos] is 1, that means this COS is only used by current + * domain. + * - If ref[old_cos] is more than 1, that mean multiple domains are using + * this COS. + */ + old_cos = d->arch.psr_cos_ids[socket]; + if ( old_cos > MAX_COS_REG_CNT ) + return -EOVERFLOW; + + ref = info->cos_ref; + + /* + * Step 1: + * Assemle a value array to store all featues cos_reg_val[old_cos]. + * And, set the input val into array according to the feature's + * position in array. + */ + array_len = get_cos_num(info); + val_array = xzalloc_array(uint64_t, array_len); + if ( !val_array ) + return -ENOMEM; + + if ( (ret = assemble_val_array(val_array, array_len, info, old_cos)) != 0 ) + { + xfree(val_array); + return ret; + } + + if ( (ret = set_new_val_to_array(val_array, array_len, info, + feat_type, type, val)) != 0 ) + { + xfree(val_array); + return ret; + } + + /* + * Lock here to make sure the ref is not changed during find and + * write process. + */ + spin_lock(&info->ref_lock); + + /* + * Step 2: + * Try to find if there is already a COS ID on which all features' values + * are same as the array. Then, we can reuse this COS ID. + */ + cos = find_cos(val_array, array_len, feat_type, info); + if ( cos >= 0 ) + { + if ( cos == old_cos ) + { + spin_unlock(&info->ref_lock); + xfree(val_array); + return 0; + } + } + else + { + /* + * Step 3: + * If fail to find, we need allocate a new COS ID. + * If multiple domains are using same COS ID, its ref is more + * than 1. That means we cannot free this COS to make current domain + * use it. Because other domains are using the value saved in the COS. + * Unless the ref is changed to 1 (mean only current domain is using + * it), we cannot allocate the COS ID to current domain. + * So, only the COS ID which ref is 1 or 0 can be allocated. + */ + cos = pick_avail_cos(info, val_array, array_len, old_cos, feat_type); + if ( cos < 0 ) + { + spin_unlock(&info->ref_lock); + xfree(val_array); + return cos; + } + + /* + * Step 4: + * Write all features MSRs according to the COS ID. + */ + ret = write_psr_msr(socket, cos, val_array); + if ( ret ) + { + spin_unlock(&info->ref_lock); + xfree(val_array); + return ret; + } + } + + /* + * Step 5: + * Update ref according to COS ID. + */ + ref[cos]++; + ASSERT(ref[cos] || cos == 0); + ref[old_cos]--; + spin_unlock(&info->ref_lock); + + /* + * Step 6: + * Save the COS ID into current domain's psr_cos_ids[] so that we can know + * which COS the domain is using on the socket. One domain can only use + * one COS ID at same time on each socket. + */ + d->arch.psr_cos_ids[socket] = cos; + xfree(val_array); + + return 0; +} + /* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */ static void psr_free_cos(struct domain *d) { - if( !d->arch.psr_cos_ids ) + unsigned int socket, cos; + + if ( !d->arch.psr_cos_ids ) return; + /* Domain is free so its cos_ref should be decreased. */ + for ( socket = 0; socket < nr_sockets; socket++ ) + { + struct psr_socket_info *info; + + /* cos 0 is default one which does not need be handled. */ + if ( (cos = d->arch.psr_cos_ids[socket]) == 0 ) + continue; + + /* + * If domain uses other cos ids, all corresponding refs must have been + * increased 1 for this domain. So, we need decrease them. + */ + info = socket_info + socket; + ASSERT(info->cos_ref[cos] || cos == 0); + spin_lock(&info->ref_lock); + info->cos_ref[cos]--; + spin_unlock(&info->ref_lock); + } + xfree(d->arch.psr_cos_ids); d->arch.psr_cos_ids = NULL; } diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h index d50e359..97214fe 100644 --- a/xen/include/asm-x86/psr.h +++ b/xen/include/asm-x86/psr.h @@ -72,8 +72,8 @@ int psr_get_info(unsigned int socket, enum cbm_type type, uint32_t data[], unsigned int array_len); int psr_get_val(struct domain *d, unsigned int socket, uint64_t *val, enum cbm_type type); -int psr_set_l3_cbm(struct domain *d, unsigned int socket, - uint64_t cbm, enum cbm_type type); +int psr_set_val(struct domain *d, unsigned int socket, + uint64_t val, enum cbm_type type); int psr_domain_init(struct domain *d); void psr_domain_free(struct domain *d);