From patchwork Wed Jun 30 06:02:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 12351259 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81FC2C11F68 for ; Wed, 30 Jun 2021 06:08:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 568ED61CA5 for ; Wed, 30 Jun 2021 06:08:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232278AbhF3GKq (ORCPT ); Wed, 30 Jun 2021 02:10:46 -0400 Received: from mga02.intel.com ([134.134.136.20]:46946 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231791AbhF3GKo (ORCPT ); Wed, 30 Jun 2021 02:10:44 -0400 X-IronPort-AV: E=McAfee;i="6200,9189,10030"; a="195448548" X-IronPort-AV: E=Sophos;i="5.83,311,1616482800"; d="scan'208";a="195448548" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Jun 2021 23:08:14 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.83,311,1616482800"; d="scan'208";a="455156480" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga008.jf.intel.com with ESMTP; 29 Jun 2021 23:08:14 -0700 From: "Chang S. Bae" To: bp@suse.de, luto@kernel.org, tglx@linutronix.de, mingo@kernel.org, x86@kernel.org Cc: len.brown@intel.com, dave.hansen@intel.com, jing2.liu@intel.com, ravi.v.shankar@intel.com, linux-kernel@vger.kernel.org, chang.seok.bae@intel.com, kvm@vger.kernel.org Subject: [PATCH v6 01/26] x86/fpu/xstate: Modify the initialization helper to handle both static and dynamic buffers Date: Tue, 29 Jun 2021 23:02:01 -0700 Message-Id: <20210630060226.24652-2-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210630060226.24652-1-chang.seok.bae@intel.com> References: <20210630060226.24652-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Have the function initializing the XSTATE buffer take a struct fpu * pointer in preparation for dynamic state buffer support. init_fpstate is a special case, which is indicated by a null pointer parameter to fpstate_init(). Also, fpstate_init_xstate() now accepts the state component bitmap to customize the compacted format. No functional change. Signed-off-by: Chang S. Bae Reviewed-by: Len Brown Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org Cc: kvm@vger.kernel.org --- Changes from v5: * Moved fpstate_init_xstate() back to the header (again). * Massaged the changelog. Changes from v4: * Added a proper function description. (Borislav Petkov) * Added the likely() statement as a null pointer is a special case. Changes from v3: * Updated the changelog. (Borislav Petkov) * Updated the function comment to use kernel-doc style. (Borislav Petkov) Changes from v2: * Updated the changelog with task->fpu removed. (Borislav Petkov) --- arch/x86/include/asm/fpu/internal.h | 11 ++++++++++- arch/x86/kernel/fpu/core.c | 30 ++++++++++++++++++----------- arch/x86/kernel/fpu/init.c | 2 +- arch/x86/kernel/fpu/xstate.c | 3 +-- arch/x86/kvm/x86.c | 2 +- 5 files changed, 32 insertions(+), 16 deletions(-) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index 5a18694a89b2..c7a64e2806a9 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -80,7 +80,7 @@ static __always_inline __pure bool use_fxsr(void) extern union fpregs_state init_fpstate; -extern void fpstate_init(union fpregs_state *state); +extern void fpstate_init(struct fpu *fpu); #ifdef CONFIG_MATH_EMULATION extern void fpstate_init_soft(struct swregs_state *soft); #else @@ -88,6 +88,15 @@ static inline void fpstate_init_soft(struct swregs_state *soft) {} #endif extern void save_fpregs_to_fpstate(struct fpu *fpu); +static inline void fpstate_init_xstate(struct xregs_state *xsave, u64 mask) +{ + /* + * XRSTORS requires these bits set in xcomp_bv, or it will + * trigger #GP: + */ + xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT | mask; +} + /* Returns 0 or the negated trap number, which results in -EFAULT for #PF */ #define user_insn(insn, output, input...) \ ({ \ diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c index 7ada7bd03a32..d0c16f5a9aeb 100644 --- a/arch/x86/kernel/fpu/core.c +++ b/arch/x86/kernel/fpu/core.c @@ -203,15 +203,6 @@ void fpu_sync_fpstate(struct fpu *fpu) fpregs_unlock(); } -static inline void fpstate_init_xstate(struct xregs_state *xsave) -{ - /* - * XRSTORS requires these bits set in xcomp_bv, or it will - * trigger #GP: - */ - xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT | xfeatures_mask_all; -} - static inline void fpstate_init_fxstate(struct fxregs_state *fx) { fx->cwd = 0x37f; @@ -229,8 +220,25 @@ static inline void fpstate_init_fstate(struct fregs_state *fp) fp->fos = 0xffff0000u; } -void fpstate_init(union fpregs_state *state) +/** + * + * fpstate_init() - initialize the xstate buffer + * + * If @fpu is NULL, initialize init_fpstate. + * + * @fpu: A struct fpu * pointer + * + * Returns nothing. + */ +void fpstate_init(struct fpu *fpu) { + union fpregs_state *state; + + if (likely(fpu)) + state = &fpu->state; + else + state = &init_fpstate; + if (!static_cpu_has(X86_FEATURE_FPU)) { fpstate_init_soft(&state->soft); return; @@ -239,7 +247,7 @@ void fpstate_init(union fpregs_state *state) memset(state, 0, fpu_kernel_xstate_size); if (static_cpu_has(X86_FEATURE_XSAVES)) - fpstate_init_xstate(&state->xsave); + fpstate_init_xstate(&state->xsave, xfeatures_mask_all); if (static_cpu_has(X86_FEATURE_FXSR)) fpstate_init_fxstate(&state->fxsave); else diff --git a/arch/x86/kernel/fpu/init.c b/arch/x86/kernel/fpu/init.c index 64e29927cc32..e14c72bc8706 100644 --- a/arch/x86/kernel/fpu/init.c +++ b/arch/x86/kernel/fpu/init.c @@ -124,7 +124,7 @@ static void __init fpu__init_system_generic(void) * Set up the legacy init FPU context. (xstate init might overwrite this * with a more modern format, if the CPU supports it.) */ - fpstate_init(&init_fpstate); + fpstate_init(NULL); fpu__init_system_mxcsr(); } diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index c8def1b7f8fb..d4fdceb9a309 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -395,8 +395,7 @@ static void __init setup_init_fpu_buf(void) print_xstate_features(); if (boot_cpu_has(X86_FEATURE_XSAVES)) - init_fpstate.xsave.header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT | - xfeatures_mask_all; + fpstate_init_xstate(&init_fpstate.xsave, xfeatures_mask_all); /* * Init all the features state with header.xfeatures being 0x0 diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 8ee7add0e763..3c3a52f6d490 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -10273,7 +10273,7 @@ static void fx_init(struct kvm_vcpu *vcpu) if (!vcpu->arch.guest_fpu) return; - fpstate_init(&vcpu->arch.guest_fpu->state); + fpstate_init(vcpu->arch.guest_fpu); if (boot_cpu_has(X86_FEATURE_XSAVES)) vcpu->arch.guest_fpu->state.xsave.header.xcomp_bv = host_xcr0 | XSTATE_COMPACTION_ENABLED; From patchwork Wed Jun 30 06:02:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 12351261 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 950C5C11F6A for ; Wed, 30 Jun 2021 06:08:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7C8A061D33 for ; Wed, 30 Jun 2021 06:08:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232343AbhF3GKq (ORCPT ); Wed, 30 Jun 2021 02:10:46 -0400 Received: from mga02.intel.com ([134.134.136.20]:46946 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232066AbhF3GKp (ORCPT ); Wed, 30 Jun 2021 02:10:45 -0400 X-IronPort-AV: E=McAfee;i="6200,9189,10030"; a="195448552" X-IronPort-AV: E=Sophos;i="5.83,311,1616482800"; d="scan'208";a="195448552" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Jun 2021 23:08:15 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.83,311,1616482800"; d="scan'208";a="455156489" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga008.jf.intel.com with ESMTP; 29 Jun 2021 23:08:15 -0700 From: "Chang S. Bae" To: bp@suse.de, luto@kernel.org, tglx@linutronix.de, mingo@kernel.org, x86@kernel.org Cc: len.brown@intel.com, dave.hansen@intel.com, jing2.liu@intel.com, ravi.v.shankar@intel.com, linux-kernel@vger.kernel.org, chang.seok.bae@intel.com, kvm@vger.kernel.org Subject: [PATCH v6 03/26] x86/fpu/xstate: Modify address finders to handle both static and dynamic buffers Date: Tue, 29 Jun 2021 23:02:03 -0700 Message-Id: <20210630060226.24652-4-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210630060226.24652-1-chang.seok.bae@intel.com> References: <20210630060226.24652-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Have all the functions finding XSTATE address take a struct fpu * pointer in preparation for dynamic state buffer support. init_fpstate is a special case, which is indicated by a null pointer parameter to get_xsave_addr() and __raw_xsave_addr(). No functional change. Signed-off-by: Chang S. Bae Reviewed-by: Len Brown Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org Cc: kvm@vger.kernel.org --- Changes from v5: * Adjusted some call sites for the new base. Changes from v3: * Updated the changelog. (Borislav Petkov) * Updated the function comment to use kernel-doc style. (Borislav Petkov) Changes from v2: * Updated the changelog with task->fpu removed. (Borislav Petkov) Changes from v1: * Rebased on the upstream kernel (5.10) --- arch/x86/include/asm/fpu/xstate.h | 2 +- arch/x86/kernel/fpu/xstate.c | 42 ++++++++++++++++++++++++------- arch/x86/kvm/x86.c | 10 +++----- 3 files changed, 38 insertions(+), 16 deletions(-) diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index ede166e9d3f2..2451bccc6cac 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -134,7 +134,7 @@ extern u64 xstate_fx_sw_bytes[USER_XSTATE_FX_SW_WORDS]; extern void __init update_regset_xstate_info(unsigned int size, u64 xstate_mask); -void *get_xsave_addr(struct xregs_state *xsave, int xfeature_nr); +void *get_xsave_addr(struct fpu *fpu, int xfeature_nr); int xfeature_size(int xfeature_nr); int copy_uabi_from_kernel_to_xstate(struct fpu *fpu, const void *kbuf); int copy_sigframe_from_user_to_xstate(struct fpu *fpu, const void __user *ubuf); diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index 59f08953201c..af2adc954578 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -841,19 +841,34 @@ void fpu__resume_cpu(void) } } -/* +/** + * __raw_xsave_addr() - Find the address where the feature state is saved. + * * Given an xstate feature nr, calculate where in the xsave * buffer the state is. Callers should ensure that the buffer * is valid. + * + * If @fpu is NULL, use init_fpstate. + * + * @fpu: A struct fpu * pointer + * + * Return: An address of the feature state in the buffer */ -static void *__raw_xsave_addr(struct xregs_state *xsave, int xfeature_nr) +static void *__raw_xsave_addr(struct fpu *fpu, int xfeature_nr) { + void *xsave; + if (!xfeature_enabled(xfeature_nr)) { WARN_ON_FPU(1); return NULL; } - return (void *)xsave + xstate_comp_offsets[xfeature_nr]; + if (fpu) + xsave = &fpu->state.xsave; + else + xsave = &init_fpstate.xsave; + + return xsave + xstate_comp_offsets[xfeature_nr]; } /* * Given the xsave area and a state inside, this function returns the @@ -866,15 +881,18 @@ static void *__raw_xsave_addr(struct xregs_state *xsave, int xfeature_nr) * this will return NULL. * * Inputs: - * xstate: the thread's storage area for all FPU data + * fpu: the thread's FPU data to reference xstate buffer(s). + * (A null pointer parameter indicates init_fpstate.) * xfeature_nr: state which is defined in xsave.h (e.g. XFEATURE_FP, * XFEATURE_SSE, etc...) * Output: * address of the state in the xsave area, or NULL if the * field is not present in the xsave buffer. */ -void *get_xsave_addr(struct xregs_state *xsave, int xfeature_nr) +void *get_xsave_addr(struct fpu *fpu, int xfeature_nr) { + struct xregs_state *xsave; + /* * Do we even *have* xsave state? */ @@ -887,6 +905,12 @@ void *get_xsave_addr(struct xregs_state *xsave, int xfeature_nr) */ WARN_ONCE(!(xfeatures_mask_all & BIT_ULL(xfeature_nr)), "get of unsupported state"); + + if (fpu) + xsave = &fpu->state.xsave; + else + xsave = &init_fpstate.xsave; + /* * This assumes the last 'xsave*' instruction to * have requested that 'xfeature_nr' be saved. @@ -901,7 +925,7 @@ void *get_xsave_addr(struct xregs_state *xsave, int xfeature_nr) if (!(xsave->header.xfeatures & BIT_ULL(xfeature_nr))) return NULL; - return __raw_xsave_addr(xsave, xfeature_nr); + return __raw_xsave_addr(fpu, xfeature_nr); } EXPORT_SYMBOL_GPL(get_xsave_addr); @@ -1061,8 +1085,8 @@ void copy_xstate_to_uabi_buf(struct membuf to, struct task_struct *tsk, membuf_write(&to, &pkru, sizeof(pkru)); } else { copy_feature(header.xfeatures & BIT_ULL(i), &to, - __raw_xsave_addr(xsave, i), - __raw_xsave_addr(xinit, i), + __raw_xsave_addr(&tsk->thread.fpu, i), + __raw_xsave_addr(NULL, i), xstate_sizes[i]); } /* @@ -1129,7 +1153,7 @@ static int copy_uabi_to_xstate(struct fpu *fpu, const void *kbuf, u64 mask = ((u64)1 << i); if (hdr.xfeatures & mask) { - void *dst = __raw_xsave_addr(xsave, i); + void *dst = __raw_xsave_addr(fpu, i); offset = xstate_offsets[i]; size = xstate_sizes[i]; diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 3c3a52f6d490..8c5a328c9631 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -4617,7 +4617,7 @@ static void fill_xsave(u8 *dest, struct kvm_vcpu *vcpu) memcpy(dest + offset, &vcpu->arch.pkru, sizeof(vcpu->arch.pkru)); } else { - src = get_xsave_addr(xsave, xfeature_nr); + src = get_xsave_addr(vcpu->arch.guest_fpu, xfeature_nr); if (src) memcpy(dest + offset, src, size); } @@ -4660,7 +4660,7 @@ static void load_xsave(struct kvm_vcpu *vcpu, u8 *src) memcpy(&vcpu->arch.pkru, src + offset, sizeof(vcpu->arch.pkru)); } else { - void *dest = get_xsave_addr(xsave, xfeature_nr); + void *dest = get_xsave_addr(vcpu->arch.guest_fpu, xfeature_nr); if (dest) memcpy(dest, src + offset, size); @@ -10498,12 +10498,10 @@ void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) */ if (init_event) kvm_put_guest_fpu(vcpu); - mpx_state_buffer = get_xsave_addr(&vcpu->arch.guest_fpu->state.xsave, - XFEATURE_BNDREGS); + mpx_state_buffer = get_xsave_addr(vcpu->arch.guest_fpu, XFEATURE_BNDREGS); if (mpx_state_buffer) memset(mpx_state_buffer, 0, sizeof(struct mpx_bndreg_state)); - mpx_state_buffer = get_xsave_addr(&vcpu->arch.guest_fpu->state.xsave, - XFEATURE_BNDCSR); + mpx_state_buffer = get_xsave_addr(vcpu->arch.guest_fpu, XFEATURE_BNDCSR); if (mpx_state_buffer) memset(mpx_state_buffer, 0, sizeof(struct mpx_bndcsr)); if (init_event) From patchwork Wed Jun 30 06:02:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 12351267 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB80BC11F69 for ; Wed, 30 Jun 2021 06:08:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A642661D49 for ; Wed, 30 Jun 2021 06:08:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232464AbhF3GKw (ORCPT ); Wed, 30 Jun 2021 02:10:52 -0400 Received: from mga02.intel.com ([134.134.136.20]:46942 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232254AbhF3GKq (ORCPT ); Wed, 30 Jun 2021 02:10:46 -0400 X-IronPort-AV: E=McAfee;i="6200,9189,10030"; a="195448555" X-IronPort-AV: E=Sophos;i="5.83,311,1616482800"; d="scan'208";a="195448555" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Jun 2021 23:08:16 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.83,311,1616482800"; d="scan'208";a="455156495" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga008.jf.intel.com with ESMTP; 29 Jun 2021 23:08:15 -0700 From: "Chang S. Bae" To: bp@suse.de, luto@kernel.org, tglx@linutronix.de, mingo@kernel.org, x86@kernel.org Cc: len.brown@intel.com, dave.hansen@intel.com, jing2.liu@intel.com, ravi.v.shankar@intel.com, linux-kernel@vger.kernel.org, chang.seok.bae@intel.com, kvm@vger.kernel.org Subject: [PATCH v6 05/26] x86/fpu/xstate: Add new variables to indicate dynamic XSTATE buffer size Date: Tue, 29 Jun 2021 23:02:05 -0700 Message-Id: <20210630060226.24652-6-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210630060226.24652-1-chang.seok.bae@intel.com> References: <20210630060226.24652-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The XSTATE per-task buffer is in preparation to be dynamic for user states. Introduce new size variables to indicate the minimum and maximum size of the buffer. The value is determined at boot-time. Instead of adding them as newly exported, introduce helper functions to access them as well as the user buffer size. No functional change. Those sizes have no difference, as the buffer is not dynamic yet. Signed-off-by: Chang S. Bae Reviewed-by: Len Brown Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org Cc: kvm@vger.kernel.org --- Changes from v5: * Made the new variables __ro_after_init for the new base code. * Fixed the init_fpstate size for memset(). Changes from v3: * Added as a new patch to add the variables along with new helpers. (Borislav Petkov) --- arch/x86/include/asm/fpu/xstate.h | 9 ++++ arch/x86/include/asm/processor.h | 10 +--- arch/x86/kernel/fpu/core.c | 26 +++++++--- arch/x86/kernel/fpu/init.c | 26 ++++------ arch/x86/kernel/fpu/regset.c | 2 +- arch/x86/kernel/fpu/signal.c | 26 +++++----- arch/x86/kernel/fpu/xstate.c | 82 +++++++++++++++++++++++++------ arch/x86/kernel/process.c | 7 +++ arch/x86/kvm/x86.c | 5 +- 9 files changed, 132 insertions(+), 61 deletions(-) diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index bc4cba62906b..d722e774a9f9 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -136,6 +136,15 @@ extern u64 xstate_fx_sw_bytes[USER_XSTATE_FX_SW_WORDS]; extern void __init update_regset_xstate_info(unsigned int size, u64 xstate_mask); +enum xstate_config { + XSTATE_MIN_SIZE, + XSTATE_MAX_SIZE, + XSTATE_USER_SIZE +}; + +extern unsigned int get_xstate_config(enum xstate_config cfg); +void set_xstate_config(enum xstate_config cfg, unsigned int value); + void *get_xsave_addr(struct fpu *fpu, int xfeature_nr); int xfeature_size(int xfeature_nr); int copy_uabi_from_kernel_to_xstate(struct fpu *fpu, const void *kbuf); diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index 91946fc3c006..3e50ddff417e 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -459,9 +459,6 @@ DECLARE_PER_CPU(struct irq_stack *, hardirq_stack_ptr); DECLARE_PER_CPU(struct irq_stack *, softirq_stack_ptr); #endif /* !X86_64 */ -extern unsigned int fpu_kernel_xstate_size; -extern unsigned int fpu_user_xstate_size; - struct perf_event; struct thread_struct { @@ -536,12 +533,7 @@ struct thread_struct { }; /* Whitelist the FPU state from the task_struct for hardened usercopy. */ -static inline void arch_thread_struct_whitelist(unsigned long *offset, - unsigned long *size) -{ - *offset = offsetof(struct thread_struct, fpu.state); - *size = fpu_kernel_xstate_size; -} +extern void arch_thread_struct_whitelist(unsigned long *offset, unsigned long *size); static inline void native_load_sp0(unsigned long sp0) diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c index d0c16f5a9aeb..461114a1460c 100644 --- a/arch/x86/kernel/fpu/core.c +++ b/arch/x86/kernel/fpu/core.c @@ -233,21 +233,30 @@ static inline void fpstate_init_fstate(struct fregs_state *fp) void fpstate_init(struct fpu *fpu) { union fpregs_state *state; + unsigned int size; + u64 mask; - if (likely(fpu)) + if (likely(fpu)) { state = &fpu->state; - else + /* The dynamic user states are not prepared yet. */ + mask = xfeatures_mask_all & ~xfeatures_mask_user_dynamic; + size = get_xstate_config(XSTATE_MIN_SIZE); + } else { state = &init_fpstate; + mask = xfeatures_mask_all; + size = sizeof(init_fpstate); + } if (!static_cpu_has(X86_FEATURE_FPU)) { fpstate_init_soft(&state->soft); return; } - memset(state, 0, fpu_kernel_xstate_size); + memset(state, 0, size); if (static_cpu_has(X86_FEATURE_XSAVES)) - fpstate_init_xstate(&state->xsave, xfeatures_mask_all); + fpstate_init_xstate(&state->xsave, mask); + if (static_cpu_has(X86_FEATURE_FXSR)) fpstate_init_fxstate(&state->fxsave); else @@ -270,8 +279,11 @@ int fpu_clone(struct task_struct *dst) /* * Don't let 'init optimized' areas of the XSAVE area * leak into the child task: + * + * The child does not inherit the dynamic states. So, + * the xstate buffer has the minimum size. */ - memset(&dst_fpu->state.xsave, 0, fpu_kernel_xstate_size); + memset(&dst_fpu->state.xsave, 0, get_xstate_config(XSTATE_MIN_SIZE)); /* * If the FPU registers are not owned by current just memcpy() the @@ -280,7 +292,7 @@ int fpu_clone(struct task_struct *dst) */ fpregs_lock(); if (test_thread_flag(TIF_NEED_FPU_LOAD)) - memcpy(&dst_fpu->state, &src_fpu->state, fpu_kernel_xstate_size); + memcpy(&dst_fpu->state, &src_fpu->state, get_xstate_config(XSTATE_MIN_SIZE)); else save_fpregs_to_fpstate(dst_fpu); @@ -339,7 +351,7 @@ static inline void restore_fpregs_from_init_fpstate(u64 features_mask) static inline unsigned int init_fpstate_copy_size(void) { if (!use_xsave()) - return fpu_kernel_xstate_size; + return get_xstate_config(XSTATE_MIN_SIZE); /* XSAVE(S) just needs the legacy and the xstate header part */ return sizeof(init_fpstate.xsave); diff --git a/arch/x86/kernel/fpu/init.c b/arch/x86/kernel/fpu/init.c index e14c72bc8706..10e2a95916aa 100644 --- a/arch/x86/kernel/fpu/init.c +++ b/arch/x86/kernel/fpu/init.c @@ -129,15 +129,6 @@ static void __init fpu__init_system_generic(void) fpu__init_system_mxcsr(); } -/* - * Size of the FPU context state. All tasks in the system use the - * same context size, regardless of what portion they use. - * This is inherent to the XSAVE architecture which puts all state - * components into a single, continuous memory block: - */ -unsigned int fpu_kernel_xstate_size __ro_after_init; -EXPORT_SYMBOL_GPL(fpu_kernel_xstate_size); - /* Get alignment of the TYPE. */ #define TYPE_ALIGN(TYPE) offsetof(struct { char x; TYPE test; }, test) @@ -167,8 +158,10 @@ static void __init fpu__init_task_struct_size(void) /* * Add back the dynamically-calculated register state * size. + * + * Use the minimum size as embedded to task_struct. */ - task_size += fpu_kernel_xstate_size; + task_size += get_xstate_config(XSTATE_MIN_SIZE); /* * We dynamically size 'struct fpu', so we require that @@ -193,6 +186,7 @@ static void __init fpu__init_task_struct_size(void) static void __init fpu__init_system_xstate_size_legacy(void) { static int on_boot_cpu __initdata = 1; + unsigned int xstate_size; WARN_ON_FPU(!on_boot_cpu); on_boot_cpu = 0; @@ -203,17 +197,17 @@ static void __init fpu__init_system_xstate_size_legacy(void) */ if (!boot_cpu_has(X86_FEATURE_FPU)) { - fpu_kernel_xstate_size = sizeof(struct swregs_state); + xstate_size = sizeof(struct swregs_state); } else { if (boot_cpu_has(X86_FEATURE_FXSR)) - fpu_kernel_xstate_size = - sizeof(struct fxregs_state); + xstate_size = sizeof(struct fxregs_state); else - fpu_kernel_xstate_size = - sizeof(struct fregs_state); + xstate_size = sizeof(struct fregs_state); } - fpu_user_xstate_size = fpu_kernel_xstate_size; + set_xstate_config(XSTATE_MIN_SIZE, xstate_size); + set_xstate_config(XSTATE_MAX_SIZE, xstate_size); + set_xstate_config(XSTATE_USER_SIZE, xstate_size); } /* Legacy code to initialize eager fpu mode. */ diff --git a/arch/x86/kernel/fpu/regset.c b/arch/x86/kernel/fpu/regset.c index 49dd307003ec..8dea3730620e 100644 --- a/arch/x86/kernel/fpu/regset.c +++ b/arch/x86/kernel/fpu/regset.c @@ -149,7 +149,7 @@ int xstateregs_set(struct task_struct *target, const struct user_regset *regset, /* * A whole standard-format XSAVE buffer is needed: */ - if (pos != 0 || count != fpu_user_xstate_size) + if (pos != 0 || count != get_xstate_config(XSTATE_USER_SIZE)) return -EFAULT; if (!kbuf) { diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index bec8c8046888..63f000988fa6 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -36,7 +36,7 @@ static inline int check_xstate_in_sigframe(struct fxregs_state __user *fxbuf, /* Check for the first magic field and other error scenarios. */ if (fx_sw->magic1 != FP_XSTATE_MAGIC1 || fx_sw->xstate_size < min_xstate_size || - fx_sw->xstate_size > fpu_user_xstate_size || + fx_sw->xstate_size > get_xstate_config(XSTATE_USER_SIZE) || fx_sw->xstate_size > fx_sw->extended_size) goto setfx; @@ -107,7 +107,7 @@ static inline int save_xstate_epilog(void __user *buf, int ia32_frame) return err; err |= __put_user(FP_XSTATE_MAGIC2, - (__u32 __user *)(buf + fpu_user_xstate_size)); + (__u32 __user *)(buf + get_xstate_config(XSTATE_USER_SIZE))); /* * Read the xfeatures which we copied (directly from the cpu or @@ -144,7 +144,7 @@ static inline int copy_fpregs_to_sigframe(struct xregs_state __user *buf) else err = fnsave_to_user_sigframe((struct fregs_state __user *) buf); - if (unlikely(err) && __clear_user(buf, fpu_user_xstate_size)) + if (unlikely(err) && __clear_user(buf, get_xstate_config(XSTATE_USER_SIZE))) err = -EFAULT; return err; } @@ -205,7 +205,7 @@ int copy_fpstate_to_sigframe(void __user *buf, void __user *buf_fx, int size) fpregs_unlock(); if (ret) { - if (!fault_in_pages_writeable(buf_fx, fpu_user_xstate_size)) + if (!fault_in_pages_writeable(buf_fx, get_xstate_config(XSTATE_USER_SIZE))) goto retry; return -EFAULT; } @@ -304,12 +304,12 @@ static int restore_fpregs_from_user(void __user *buf, u64 xrestore, static int __fpu_restore_sig(void __user *buf, void __user *buf_fx, bool ia32_fxstate) { - int state_size = fpu_kernel_xstate_size; struct task_struct *tsk = current; struct fpu *fpu = &tsk->thread.fpu; struct user_i387_ia32_struct env; u64 user_xfeatures = 0; bool fx_only = false; + int state_size; int ret; if (use_xsave()) { @@ -323,6 +323,8 @@ static int __fpu_restore_sig(void __user *buf, void __user *buf_fx, state_size = fx_sw_user.xstate_size; user_xfeatures = fx_sw_user.xfeatures; } else { + /* The buffer cannot be dynamic without using XSAVE. */ + state_size = get_xstate_config(XSTATE_MIN_SIZE); user_xfeatures = XFEATURE_MASK_FPSSE; } @@ -418,8 +420,9 @@ static int __fpu_restore_sig(void __user *buf, void __user *buf_fx, } static inline int xstate_sigframe_size(void) { - return use_xsave() ? fpu_user_xstate_size + FP_XSTATE_MAGIC2_SIZE : - fpu_user_xstate_size; + int xstate_size = get_xstate_config(XSTATE_USER_SIZE); + + return use_xsave() ? xstate_size + FP_XSTATE_MAGIC2_SIZE : xstate_size; } /* @@ -514,19 +517,20 @@ unsigned long fpu__get_fpstate_size(void) */ void fpu__init_prepare_fx_sw_frame(void) { - int size = fpu_user_xstate_size + FP_XSTATE_MAGIC2_SIZE; + int xstate_size = get_xstate_config(XSTATE_USER_SIZE); + int ext_size = xstate_size + FP_XSTATE_MAGIC2_SIZE; fx_sw_reserved.magic1 = FP_XSTATE_MAGIC1; - fx_sw_reserved.extended_size = size; + fx_sw_reserved.extended_size = ext_size; fx_sw_reserved.xfeatures = xfeatures_mask_uabi(); - fx_sw_reserved.xstate_size = fpu_user_xstate_size; + fx_sw_reserved.xstate_size = xstate_size; if (IS_ENABLED(CONFIG_IA32_EMULATION) || IS_ENABLED(CONFIG_X86_32)) { int fsave_header_size = sizeof(struct fregs_state); fx_sw_reserved_ia32 = fx_sw_reserved; - fx_sw_reserved_ia32.extended_size = size + fsave_header_size; + fx_sw_reserved_ia32.extended_size = ext_size + fsave_header_size; } } diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index 1d76afc29f19..e03853bb2603 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -77,12 +77,50 @@ static unsigned int xstate_comp_offsets[XFEATURE_MAX] __ro_after_init = static unsigned int xstate_supervisor_only_offsets[XFEATURE_MAX] __ro_after_init = { [ 0 ... XFEATURE_MAX - 1] = -1}; -/* - * The XSAVE area of kernel can be in standard or compacted format; - * it is always in standard format for user mode. This is the user - * mode standard format size used for signal and ptrace frames. +/** + * struct fpu_xstate_buffer_config - xstate per-task buffer configuration + * @min_size, @max_size: The size of the kernel buffer. It is variable with the dynamic user + * states. Every task has the minimum buffer by default. It can be + * expanded to the max size. The two sizes are the same when using the + * standard format. + * @user_size: The size of the userspace buffer. The buffer is always in the + * standard format. It is used for signal and ptrace frames. */ -unsigned int fpu_user_xstate_size __ro_after_init; +struct fpu_xstate_buffer_config { + unsigned int min_size, max_size; + unsigned int user_size; +}; + +static struct fpu_xstate_buffer_config buffer_config __ro_after_init; + +unsigned int get_xstate_config(enum xstate_config cfg) +{ + switch (cfg) { + case XSTATE_MIN_SIZE: + return buffer_config.min_size; + case XSTATE_MAX_SIZE: + return buffer_config.max_size; + case XSTATE_USER_SIZE: + return buffer_config.user_size; + default: + return 0; + } +} +EXPORT_SYMBOL_GPL(get_xstate_config); + +void set_xstate_config(enum xstate_config cfg, unsigned int value) +{ + switch (cfg) { + case XSTATE_MIN_SIZE: + buffer_config.min_size = value; + break; + case XSTATE_MAX_SIZE: + buffer_config.max_size = value; + break; + case XSTATE_USER_SIZE: + buffer_config.user_size = value; + } +} /* * Return whether the system supports a given xfeature. @@ -595,7 +633,11 @@ static void do_extra_xstate_size_checks(void) */ paranoid_xstate_size += xfeature_size(i); } - XSTATE_WARN_ON(paranoid_xstate_size != fpu_kernel_xstate_size); + /* + * The size accounts for all the possible states reserved in the + * per-task buffer. Check against the maximum size. + */ + XSTATE_WARN_ON(paranoid_xstate_size != get_xstate_config(XSTATE_MAX_SIZE)); } @@ -690,21 +732,29 @@ static int __init init_xstate_size(void) else possible_xstate_size = xsave_size; - /* Ensure we have the space to store all enabled: */ - if (!is_supported_xstate_size(possible_xstate_size)) - return -EINVAL; - /* - * The size is OK, we are definitely going to use xsave, - * make it known to the world that we need more space. + * The size accounts for all the possible states reserved in the + * per-task buffer. Set the maximum with this value. */ - fpu_kernel_xstate_size = possible_xstate_size; + set_xstate_config(XSTATE_MAX_SIZE, possible_xstate_size); + + /* Perform an extra check for the maximum size. */ do_extra_xstate_size_checks(); + /* + * Set the minimum to be the same as the maximum. The dynamic + * user states are not supported yet. + */ + set_xstate_config(XSTATE_MIN_SIZE, possible_xstate_size); + + /* Ensure the minimum size fits in the statically-allocated buffer: */ + if (!is_supported_xstate_size(get_xstate_config(XSTATE_MIN_SIZE))) + return -EINVAL; + /* * User space is always in standard format. */ - fpu_user_xstate_size = xsave_size; + set_xstate_config(XSTATE_USER_SIZE, xsave_size); return 0; } @@ -800,7 +850,7 @@ void __init fpu__init_system_xstate(void) * Update info used for ptrace frames; use standard-format size and no * supervisor xstates: */ - update_regset_xstate_info(fpu_user_xstate_size, xfeatures_mask_uabi()); + update_regset_xstate_info(get_xstate_config(XSTATE_USER_SIZE), xfeatures_mask_uabi()); fpu__init_prepare_fx_sw_frame(); setup_init_fpu_buf(); @@ -820,7 +870,7 @@ void __init fpu__init_system_xstate(void) print_xstate_offset_size(); pr_info("x86/fpu: Enabled xstate features 0x%llx, context size is %d bytes, using '%s' format.\n", xfeatures_mask_all, - fpu_kernel_xstate_size, + get_xstate_config(XSTATE_MAX_SIZE), boot_cpu_has(X86_FEATURE_XSAVES) ? "compacted" : "standard"); return; diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index fa6c8fa0f778..e415254c26d4 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -90,6 +90,13 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) return fpu_clone(dst); } +void arch_thread_struct_whitelist(unsigned long *offset, unsigned long *size) +{ + *offset = offsetof(struct thread_struct, fpu.state); + /* The buffer embedded in thread_struct has the minimum size. */ + *size = get_xstate_config(XSTATE_MIN_SIZE); +} + /* * Free thread data structures etc.. */ diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 8c5a328c9631..fc4fb289ac55 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -9633,10 +9633,13 @@ static void kvm_save_current_fpu(struct fpu *fpu) /* * If the target FPU state is not resident in the CPU registers, just * memcpy() from current, else save CPU state directly to the target. + * + * KVM does not support dynamic user states yet. Assume the buffer + * always has the minimum size. */ if (test_thread_flag(TIF_NEED_FPU_LOAD)) memcpy(&fpu->state, ¤t->thread.fpu.state, - fpu_kernel_xstate_size); + get_xstate_config(XSTATE_MIN_SIZE)); else save_fpregs_to_fpstate(fpu); } From patchwork Wed Jun 30 06:02:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 12351263 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4429C11F68 for ; Wed, 30 Jun 2021 06:08:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8699C61D2B for ; Wed, 30 Jun 2021 06:08:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232504AbhF3GKy (ORCPT ); Wed, 30 Jun 2021 02:10:54 -0400 Received: from mga02.intel.com ([134.134.136.20]:46949 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232241AbhF3GKq (ORCPT ); Wed, 30 Jun 2021 02:10:46 -0400 X-IronPort-AV: E=McAfee;i="6200,9189,10030"; a="195448559" X-IronPort-AV: E=Sophos;i="5.83,311,1616482800"; d="scan'208";a="195448559" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Jun 2021 23:08:16 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.83,311,1616482800"; d="scan'208";a="455156503" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga008.jf.intel.com with ESMTP; 29 Jun 2021 23:08:16 -0700 From: "Chang S. Bae" To: bp@suse.de, luto@kernel.org, tglx@linutronix.de, mingo@kernel.org, x86@kernel.org Cc: len.brown@intel.com, dave.hansen@intel.com, jing2.liu@intel.com, ravi.v.shankar@intel.com, linux-kernel@vger.kernel.org, chang.seok.bae@intel.com, kvm@vger.kernel.org Subject: [PATCH v6 07/26] x86/fpu/xstate: Convert the struct fpu 'state' field to a pointer Date: Tue, 29 Jun 2021 23:02:07 -0700 Message-Id: <20210630060226.24652-8-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210630060226.24652-1-chang.seok.bae@intel.com> References: <20210630060226.24652-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The XSTATE per-task buffer is embedded into struct fpu. The field 'state' represents the buffer. When the dynamic user state is in use, the buffer may be dynamically allocated. Convert the 'state' field to point either to the embedded buffer or to the dynamically-allocated buffer. Also, add a new field to represent the embedded buffer. The initial task sets it before dealing with soft FPU. Make sure that every FPU state has a valid pointer value on its creation. No functional change. Suggested-by: Borislav Petkov Signed-off-by: Chang S. Bae Reviewed-by: Len Brown Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org Cc: kvm@vger.kernel.org --- Changes from v5: * Tightened up task size calculation (previously, it could over-calculate) * Adjusted the changelog. Changes from v4: * Fixed KVM's user_fpu and guest_fpu to initialize the 'state' field correctly. * Massaged the changelog. Changes from v3: * Added as a new patch to simplify the buffer access. (Borislav Petkov) --- arch/x86/include/asm/fpu/internal.h | 2 +- arch/x86/include/asm/fpu/types.h | 27 ++++++++++++++++++------ arch/x86/include/asm/trace/fpu.h | 4 ++-- arch/x86/kernel/fpu/core.c | 32 +++++++++++++++-------------- arch/x86/kernel/fpu/init.c | 8 +++++--- arch/x86/kernel/fpu/regset.c | 24 +++++++++++----------- arch/x86/kernel/fpu/signal.c | 24 +++++++++++----------- arch/x86/kernel/fpu/xstate.c | 8 ++++---- arch/x86/kernel/process.c | 2 +- arch/x86/kvm/x86.c | 22 +++++++++++--------- arch/x86/math-emu/fpu_aux.c | 2 +- arch/x86/math-emu/fpu_entry.c | 4 ++-- arch/x86/math-emu/fpu_system.h | 2 +- 13 files changed, 91 insertions(+), 70 deletions(-) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index c7a64e2806a9..d2fc19c0e457 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -484,7 +484,7 @@ static inline void fpregs_restore_userregs(void) */ mask = xfeatures_mask_restore_user() | xfeatures_mask_supervisor(); - __restore_fpregs_from_fpstate(&fpu->state, mask); + __restore_fpregs_from_fpstate(fpu->state, mask); fpregs_activate(fpu); fpu->last_cpu = cpu; diff --git a/arch/x86/include/asm/fpu/types.h b/arch/x86/include/asm/fpu/types.h index f5a38a5f3ae1..dcd28a545377 100644 --- a/arch/x86/include/asm/fpu/types.h +++ b/arch/x86/include/asm/fpu/types.h @@ -339,13 +339,28 @@ struct fpu { /* * @state: * - * In-memory copy of all FPU registers that we save/restore - * over context switches. If the task is using the FPU then - * the registers in the FPU are more recent than this state - * copy. If the task context-switches away then they get - * saved here and represent the FPU state. + * A pointer to indicate the in-memory copy of all FPU registers that are + * saved/restored over context switches. + * + * Initially @state points to @__default_state. When dynamic states get + * used, a memory is allocated for the larger state copy and @state is + * updated to point to it. Then, the state in ->state supersedes and + * invalidates the state in @__default_state. + * + * In general, if the task is using the FPU then the registers in the FPU + * are more recent than the state copy. If the task context-switches away + * then they get saved in ->state and represent the FPU state. + */ + union fpregs_state *state; + + /* + * @__default_state: + * + * Initial in-memory copy of all FPU registers that saved/restored + * over context switches. When the task is switched to dynamic states, + * this copy is replaced with the new in-memory copy in ->state. */ - union fpregs_state state; + union fpregs_state __default_state; /* * WARNING: 'state' is dynamically-sized. Do not put * anything after it here. diff --git a/arch/x86/include/asm/trace/fpu.h b/arch/x86/include/asm/trace/fpu.h index 879b77792f94..ef82f4824ce7 100644 --- a/arch/x86/include/asm/trace/fpu.h +++ b/arch/x86/include/asm/trace/fpu.h @@ -22,8 +22,8 @@ DECLARE_EVENT_CLASS(x86_fpu, __entry->fpu = fpu; __entry->load_fpu = test_thread_flag(TIF_NEED_FPU_LOAD); if (boot_cpu_has(X86_FEATURE_OSXSAVE)) { - __entry->xfeatures = fpu->state.xsave.header.xfeatures; - __entry->xcomp_bv = fpu->state.xsave.header.xcomp_bv; + __entry->xfeatures = fpu->state->xsave.header.xfeatures; + __entry->xcomp_bv = fpu->state->xsave.header.xcomp_bv; } ), TP_printk("x86/fpu: %p load: %d xfeatures: %llx xcomp_bv: %llx", diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c index 461114a1460c..01eb3389d163 100644 --- a/arch/x86/kernel/fpu/core.c +++ b/arch/x86/kernel/fpu/core.c @@ -99,19 +99,19 @@ EXPORT_SYMBOL(irq_fpu_usable); void save_fpregs_to_fpstate(struct fpu *fpu) { if (likely(use_xsave())) { - os_xsave(&fpu->state.xsave); + os_xsave(&fpu->state->xsave); /* * AVX512 state is tracked here because its use is * known to slow the max clock speed of the core. */ - if (fpu->state.xsave.header.xfeatures & XFEATURE_MASK_AVX512) + if (fpu->state->xsave.header.xfeatures & XFEATURE_MASK_AVX512) fpu->avx512_timestamp = jiffies; return; } if (likely(use_fxsr())) { - fxsave(&fpu->state.fxsave); + fxsave(&fpu->state->fxsave); return; } @@ -119,8 +119,8 @@ void save_fpregs_to_fpstate(struct fpu *fpu) * Legacy FPU register saving, FNSAVE always clears FPU registers, * so we have to reload them from the memory state. */ - asm volatile("fnsave %[fp]; fwait" : [fp] "=m" (fpu->state.fsave)); - frstor(&fpu->state.fsave); + asm volatile("fnsave %[fp]; fwait" : [fp] "=m" (fpu->state->fsave)); + frstor(&fpu->state->fsave); } EXPORT_SYMBOL(save_fpregs_to_fpstate); @@ -237,7 +237,7 @@ void fpstate_init(struct fpu *fpu) u64 mask; if (likely(fpu)) { - state = &fpu->state; + state = fpu->state; /* The dynamic user states are not prepared yet. */ mask = xfeatures_mask_all & ~xfeatures_mask_user_dynamic; size = get_xstate_config(XSTATE_MIN_SIZE); @@ -276,6 +276,8 @@ int fpu_clone(struct task_struct *dst) if (!cpu_feature_enabled(X86_FEATURE_FPU)) return 0; + dst_fpu->state = &dst_fpu->__default_state; + /* * Don't let 'init optimized' areas of the XSAVE area * leak into the child task: @@ -283,7 +285,7 @@ int fpu_clone(struct task_struct *dst) * The child does not inherit the dynamic states. So, * the xstate buffer has the minimum size. */ - memset(&dst_fpu->state.xsave, 0, get_xstate_config(XSTATE_MIN_SIZE)); + memset(&dst_fpu->state->xsave, 0, get_xstate_config(XSTATE_MIN_SIZE)); /* * If the FPU registers are not owned by current just memcpy() the @@ -292,7 +294,7 @@ int fpu_clone(struct task_struct *dst) */ fpregs_lock(); if (test_thread_flag(TIF_NEED_FPU_LOAD)) - memcpy(&dst_fpu->state, &src_fpu->state, get_xstate_config(XSTATE_MIN_SIZE)); + memcpy(dst_fpu->state, src_fpu->state, get_xstate_config(XSTATE_MIN_SIZE)); else save_fpregs_to_fpstate(dst_fpu); @@ -379,7 +381,7 @@ static void fpu_reset_fpstate(void) * user space as PKRU is eagerly written in switch_to() and * flush_thread(). */ - memcpy(&fpu->state, &init_fpstate, init_fpstate_copy_size()); + memcpy(fpu->state, &init_fpstate, init_fpstate_copy_size()); set_thread_flag(TIF_NEED_FPU_LOAD); fpregs_unlock(); } @@ -406,7 +408,7 @@ void fpu__clear_user_states(struct fpu *fpu) */ if (xfeatures_mask_supervisor() && !fpregs_state_valid(fpu, smp_processor_id())) { - os_xrstor(&fpu->state.xsave, xfeatures_mask_supervisor()); + os_xrstor(&fpu->state->xsave, xfeatures_mask_supervisor()); } /* Reset user states in registers. */ @@ -488,11 +490,11 @@ int fpu__exception_code(struct fpu *fpu, int trap_nr) * fully reproduce the context of the exception. */ if (boot_cpu_has(X86_FEATURE_FXSR)) { - cwd = fpu->state.fxsave.cwd; - swd = fpu->state.fxsave.swd; + cwd = fpu->state->fxsave.cwd; + swd = fpu->state->fxsave.swd; } else { - cwd = (unsigned short)fpu->state.fsave.cwd; - swd = (unsigned short)fpu->state.fsave.swd; + cwd = (unsigned short)fpu->state->fsave.cwd; + swd = (unsigned short)fpu->state->fsave.swd; } err = swd & ~cwd; @@ -506,7 +508,7 @@ int fpu__exception_code(struct fpu *fpu, int trap_nr) unsigned short mxcsr = MXCSR_DEFAULT; if (boot_cpu_has(X86_FEATURE_XMM)) - mxcsr = fpu->state.fxsave.mxcsr; + mxcsr = fpu->state->fxsave.mxcsr; err = ~(mxcsr >> 7) & mxcsr; } diff --git a/arch/x86/kernel/fpu/init.c b/arch/x86/kernel/fpu/init.c index 10e2a95916aa..3e4e14ca723b 100644 --- a/arch/x86/kernel/fpu/init.c +++ b/arch/x86/kernel/fpu/init.c @@ -31,10 +31,12 @@ static void fpu__init_cpu_generic(void) cr0 |= X86_CR0_EM; write_cr0(cr0); + current->thread.fpu.state = ¤t->thread.fpu.__default_state; + /* Flush out any pending x87 state: */ #ifdef CONFIG_MATH_EMULATION if (!boot_cpu_has(X86_FEATURE_FPU)) - fpstate_init_soft(¤t->thread.fpu.state.soft); + fpstate_init_soft(¤t->thread.fpu.state->soft); else #endif asm volatile ("fninit"); @@ -153,7 +155,7 @@ static void __init fpu__init_task_struct_size(void) * Subtract off the static size of the register state. * It potentially has a bunch of padding. */ - task_size -= sizeof(((struct task_struct *)0)->thread.fpu.state); + task_size -= sizeof(((struct task_struct *)0)->thread.fpu.__default_state); /* * Add back the dynamically-calculated register state @@ -170,7 +172,7 @@ static void __init fpu__init_task_struct_size(void) * you hit a compile error here, check the structure to * see if something got added to the end. */ - CHECK_MEMBER_AT_END_OF(struct fpu, state); + CHECK_MEMBER_AT_END_OF(struct fpu, __default_state); CHECK_MEMBER_AT_END_OF(struct thread_struct, fpu); CHECK_MEMBER_AT_END_OF(struct task_struct, thread); diff --git a/arch/x86/kernel/fpu/regset.c b/arch/x86/kernel/fpu/regset.c index 8dea3730620e..73d7d7b489fe 100644 --- a/arch/x86/kernel/fpu/regset.c +++ b/arch/x86/kernel/fpu/regset.c @@ -74,8 +74,8 @@ int xfpregs_get(struct task_struct *target, const struct user_regset *regset, sync_fpstate(fpu); if (!use_xsave()) { - return membuf_write(&to, &fpu->state.fxsave, - sizeof(fpu->state.fxsave)); + return membuf_write(&to, &fpu->state->fxsave, + sizeof(fpu->state->fxsave)); } copy_xstate_to_uabi_buf(to, target, XSTATE_COPY_FX); @@ -110,15 +110,15 @@ int xfpregs_set(struct task_struct *target, const struct user_regset *regset, fpu_force_restore(fpu); /* Copy the state */ - memcpy(&fpu->state.fxsave, &newstate, sizeof(newstate)); + memcpy(&fpu->state->fxsave, &newstate, sizeof(newstate)); /* Clear xmm8..15 */ - BUILD_BUG_ON(sizeof(fpu->state.fxsave.xmm_space) != 16 * 16); - memset(&fpu->state.fxsave.xmm_space[8], 0, 8 * 16); + BUILD_BUG_ON(sizeof(fpu->state->fxsave.xmm_space) != 16 * 16); + memset(&fpu->state->fxsave.xmm_space[8], 0, 8 * 16); /* Mark FP and SSE as in use when XSAVE is enabled */ if (use_xsave()) - fpu->state.xsave.header.xfeatures |= XFEATURE_MASK_FPSSE; + fpu->state->xsave.header.xfeatures |= XFEATURE_MASK_FPSSE; return 0; } @@ -283,7 +283,7 @@ static void __convert_from_fxsr(struct user_i387_ia32_struct *env, void convert_from_fxsr(struct user_i387_ia32_struct *env, struct task_struct *tsk) { - __convert_from_fxsr(env, tsk, &tsk->thread.fpu.state.fxsave); + __convert_from_fxsr(env, tsk, &tsk->thread.fpu.state->fxsave); } void convert_to_fxsr(struct fxregs_state *fxsave, @@ -326,7 +326,7 @@ int fpregs_get(struct task_struct *target, const struct user_regset *regset, return fpregs_soft_get(target, regset, to); if (!cpu_feature_enabled(X86_FEATURE_FXSR)) { - return membuf_write(&to, &fpu->state.fsave, + return membuf_write(&to, &fpu->state->fsave, sizeof(struct fregs_state)); } @@ -337,7 +337,7 @@ int fpregs_get(struct task_struct *target, const struct user_regset *regset, copy_xstate_to_uabi_buf(mb, target, XSTATE_COPY_FP); fx = &fxsave; } else { - fx = &fpu->state.fxsave; + fx = &fpu->state->fxsave; } __convert_from_fxsr(&env, target, fx); @@ -366,16 +366,16 @@ int fpregs_set(struct task_struct *target, const struct user_regset *regset, fpu_force_restore(fpu); if (cpu_feature_enabled(X86_FEATURE_FXSR)) - convert_to_fxsr(&fpu->state.fxsave, &env); + convert_to_fxsr(&fpu->state->fxsave, &env); else - memcpy(&fpu->state.fsave, &env, sizeof(env)); + memcpy(&fpu->state->fsave, &env, sizeof(env)); /* * Update the header bit in the xsave header, indicating the * presence of FP. */ if (cpu_feature_enabled(X86_FEATURE_XSAVE)) - fpu->state.xsave.header.xfeatures |= XFEATURE_MASK_FP; + fpu->state->xsave.header.xfeatures |= XFEATURE_MASK_FP; return 0; } diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index 63f000988fa6..2f35aada2007 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -67,13 +67,13 @@ static inline int check_xstate_in_sigframe(struct fxregs_state __user *fxbuf, static inline int save_fsave_header(struct task_struct *tsk, void __user *buf) { if (use_fxsr()) { - struct xregs_state *xsave = &tsk->thread.fpu.state.xsave; + struct xregs_state *xsave = &tsk->thread.fpu.state->xsave; struct user_i387_ia32_struct env; struct _fpstate_32 __user *fp = buf; fpregs_lock(); if (!test_thread_flag(TIF_NEED_FPU_LOAD)) - fxsave(&tsk->thread.fpu.state.fxsave); + fxsave(&tsk->thread.fpu.state->fxsave); fpregs_unlock(); convert_from_fxsr(&env, tsk); @@ -294,7 +294,7 @@ static int restore_fpregs_from_user(void __user *buf, u64 xrestore, * been restored from a user buffer directly. */ if (test_thread_flag(TIF_NEED_FPU_LOAD) && xfeatures_mask_supervisor()) - os_xrstor(&fpu->state.xsave, xfeatures_mask_supervisor()); + os_xrstor(&fpu->state->xsave, xfeatures_mask_supervisor()); fpregs_mark_activate(); fpregs_unlock(); @@ -365,7 +365,7 @@ static int __fpu_restore_sig(void __user *buf, void __user *buf_fx, * the right place in memory. It's ia32 mode. Shrug. */ if (xfeatures_mask_supervisor()) - os_xsave(&fpu->state.xsave); + os_xsave(&fpu->state->xsave); set_thread_flag(TIF_NEED_FPU_LOAD); } __fpu_invalidate_fpregs_state(fpu); @@ -377,21 +377,21 @@ static int __fpu_restore_sig(void __user *buf, void __user *buf_fx, if (ret) return ret; } else { - if (__copy_from_user(&fpu->state.fxsave, buf_fx, - sizeof(fpu->state.fxsave))) + if (__copy_from_user(&fpu->state->fxsave, buf_fx, + sizeof(fpu->state->fxsave))) return -EFAULT; /* Reject invalid MXCSR values. */ - if (fpu->state.fxsave.mxcsr & ~mxcsr_feature_mask) + if (fpu->state->fxsave.mxcsr & ~mxcsr_feature_mask) return -EINVAL; /* Enforce XFEATURE_MASK_FPSSE when XSAVE is enabled */ if (use_xsave()) - fpu->state.xsave.header.xfeatures |= XFEATURE_MASK_FPSSE; + fpu->state->xsave.header.xfeatures |= XFEATURE_MASK_FPSSE; } /* Fold the legacy FP storage */ - convert_to_fxsr(&fpu->state.fxsave, &env); + convert_to_fxsr(&fpu->state->fxsave, &env); fpregs_lock(); if (use_xsave()) { @@ -406,10 +406,10 @@ static int __fpu_restore_sig(void __user *buf, void __user *buf_fx, */ u64 mask = user_xfeatures | xfeatures_mask_supervisor(); - fpu->state.xsave.header.xfeatures &= mask; - ret = os_xrstor_safe(&fpu->state.xsave, xfeatures_mask_all); + fpu->state->xsave.header.xfeatures &= mask; + ret = os_xrstor_safe(&fpu->state->xsave, xfeatures_mask_all); } else { - ret = fxrstor_safe(&fpu->state.fxsave); + ret = fxrstor_safe(&fpu->state->fxsave); } if (likely(!ret)) diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index 6a075852b2b5..c3e4a2e3ce10 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -944,7 +944,7 @@ static void *__raw_xsave_addr(struct fpu *fpu, int xfeature_nr) } if (fpu) - xsave = &fpu->state.xsave; + xsave = &fpu->state->xsave; else xsave = &init_fpstate.xsave; @@ -987,7 +987,7 @@ void *get_xsave_addr(struct fpu *fpu, int xfeature_nr) "get of unsupported state"); if (fpu) - xsave = &fpu->state.xsave; + xsave = &fpu->state->xsave; else xsave = &init_fpstate.xsave; @@ -1079,7 +1079,7 @@ void copy_xstate_to_uabi_buf(struct membuf to, struct task_struct *tsk, enum xstate_copy_mode copy_mode) { const unsigned int off_mxcsr = offsetof(struct fxregs_state, mxcsr); - struct xregs_state *xsave = &tsk->thread.fpu.state.xsave; + struct xregs_state *xsave = &tsk->thread.fpu.state->xsave; struct xregs_state *xinit = &init_fpstate.xsave; struct xstate_header header; unsigned int zerofrom; @@ -1196,7 +1196,7 @@ static int copy_from_buffer(void *dst, unsigned int offset, unsigned int size, static int copy_uabi_to_xstate(struct fpu *fpu, const void *kbuf, const void __user *ubuf) { - struct xregs_state *xsave = &fpu->state.xsave; + struct xregs_state *xsave = &fpu->state->xsave; unsigned int offset, size; struct xstate_header hdr; u64 mask; diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index e415254c26d4..0228557394f5 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -92,7 +92,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) void arch_thread_struct_whitelist(unsigned long *offset, unsigned long *size) { - *offset = offsetof(struct thread_struct, fpu.state); + *offset = offsetof(struct thread_struct, fpu.__default_state); /* The buffer embedded in thread_struct has the minimum size. */ *size = get_xstate_config(XSTATE_MIN_SIZE); } diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index fc4fb289ac55..3c8b6080a253 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -4585,7 +4585,7 @@ static int kvm_vcpu_ioctl_x86_set_debugregs(struct kvm_vcpu *vcpu, static void fill_xsave(u8 *dest, struct kvm_vcpu *vcpu) { - struct xregs_state *xsave = &vcpu->arch.guest_fpu->state.xsave; + struct xregs_state *xsave = &vcpu->arch.guest_fpu->state->xsave; u64 xstate_bv = xsave->header.xfeatures; u64 valid; @@ -4628,7 +4628,7 @@ static void fill_xsave(u8 *dest, struct kvm_vcpu *vcpu) static void load_xsave(struct kvm_vcpu *vcpu, u8 *src) { - struct xregs_state *xsave = &vcpu->arch.guest_fpu->state.xsave; + struct xregs_state *xsave = &vcpu->arch.guest_fpu->state->xsave; u64 xstate_bv = *(u64 *)(src + XSAVE_HDR_OFFSET); u64 valid; @@ -4681,7 +4681,7 @@ static void kvm_vcpu_ioctl_x86_get_xsave(struct kvm_vcpu *vcpu, fill_xsave((u8 *) guest_xsave->region, vcpu); } else { memcpy(guest_xsave->region, - &vcpu->arch.guest_fpu->state.fxsave, + &vcpu->arch.guest_fpu->state->fxsave, sizeof(struct fxregs_state)); *(u64 *)&guest_xsave->region[XSAVE_HDR_OFFSET / sizeof(u32)] = XFEATURE_MASK_FPSSE; @@ -4715,7 +4715,7 @@ static int kvm_vcpu_ioctl_x86_set_xsave(struct kvm_vcpu *vcpu, if (xstate_bv & ~XFEATURE_MASK_FPSSE || mxcsr & ~mxcsr_feature_mask) return -EINVAL; - memcpy(&vcpu->arch.guest_fpu->state.fxsave, + memcpy(&vcpu->arch.guest_fpu->state->fxsave, guest_xsave->region, sizeof(struct fxregs_state)); } return 0; @@ -9638,7 +9638,7 @@ static void kvm_save_current_fpu(struct fpu *fpu) * always has the minimum size. */ if (test_thread_flag(TIF_NEED_FPU_LOAD)) - memcpy(&fpu->state, ¤t->thread.fpu.state, + memcpy(fpu->state, current->thread.fpu.state, get_xstate_config(XSTATE_MIN_SIZE)); else save_fpregs_to_fpstate(fpu); @@ -9657,7 +9657,7 @@ static void kvm_load_guest_fpu(struct kvm_vcpu *vcpu) */ if (vcpu->arch.guest_fpu) /* PKRU is separately restored in kvm_x86_ops.run. */ - __restore_fpregs_from_fpstate(&vcpu->arch.guest_fpu->state, + __restore_fpregs_from_fpstate(vcpu->arch.guest_fpu->state, ~XFEATURE_MASK_PKRU); fpregs_mark_activate(); @@ -9678,7 +9678,7 @@ static void kvm_put_guest_fpu(struct kvm_vcpu *vcpu) if (vcpu->arch.guest_fpu) kvm_save_current_fpu(vcpu->arch.guest_fpu); - restore_fpregs_from_fpstate(&vcpu->arch.user_fpu->state); + restore_fpregs_from_fpstate(vcpu->arch.user_fpu->state); fpregs_mark_activate(); fpregs_unlock(); @@ -10194,7 +10194,7 @@ int kvm_arch_vcpu_ioctl_get_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu) vcpu_load(vcpu); - fxsave = &vcpu->arch.guest_fpu->state.fxsave; + fxsave = &vcpu->arch.guest_fpu->state->fxsave; memcpy(fpu->fpr, fxsave->st_space, 128); fpu->fcw = fxsave->cwd; fpu->fsw = fxsave->swd; @@ -10217,7 +10217,7 @@ int kvm_arch_vcpu_ioctl_set_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu) vcpu_load(vcpu); - fxsave = &vcpu->arch.guest_fpu->state.fxsave; + fxsave = &vcpu->arch.guest_fpu->state->fxsave; memcpy(fxsave->st_space, fpu->fpr, 128); fxsave->cwd = fpu->fcw; @@ -10278,7 +10278,7 @@ static void fx_init(struct kvm_vcpu *vcpu) fpstate_init(vcpu->arch.guest_fpu); if (boot_cpu_has(X86_FEATURE_XSAVES)) - vcpu->arch.guest_fpu->state.xsave.header.xcomp_bv = + vcpu->arch.guest_fpu->state->xsave.header.xcomp_bv = host_xcr0 | XSTATE_COMPACTION_ENABLED; /* @@ -10358,6 +10358,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) pr_err("kvm: failed to allocate userspace's fpu\n"); goto free_emulate_ctxt; } + vcpu->arch.user_fpu->state = &vcpu->arch.user_fpu->__default_state; vcpu->arch.guest_fpu = kmem_cache_zalloc(x86_fpu_cache, GFP_KERNEL_ACCOUNT); @@ -10365,6 +10366,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) pr_err("kvm: failed to allocate vcpu's fpu\n"); goto free_user_fpu; } + vcpu->arch.guest_fpu->state = &vcpu->arch.guest_fpu->__default_state; fx_init(vcpu); vcpu->arch.maxphyaddr = cpuid_query_maxphyaddr(vcpu); diff --git a/arch/x86/math-emu/fpu_aux.c b/arch/x86/math-emu/fpu_aux.c index 034748459482..51432a73024c 100644 --- a/arch/x86/math-emu/fpu_aux.c +++ b/arch/x86/math-emu/fpu_aux.c @@ -53,7 +53,7 @@ void fpstate_init_soft(struct swregs_state *soft) void finit(void) { - fpstate_init_soft(¤t->thread.fpu.state.soft); + fpstate_init_soft(¤t->thread.fpu.state->soft); } /* diff --git a/arch/x86/math-emu/fpu_entry.c b/arch/x86/math-emu/fpu_entry.c index 8679a9d6c47f..6ba56632170e 100644 --- a/arch/x86/math-emu/fpu_entry.c +++ b/arch/x86/math-emu/fpu_entry.c @@ -640,7 +640,7 @@ int fpregs_soft_set(struct task_struct *target, unsigned int pos, unsigned int count, const void *kbuf, const void __user *ubuf) { - struct swregs_state *s387 = &target->thread.fpu.state.soft; + struct swregs_state *s387 = &target->thread.fpu.state->soft; void *space = s387->st_space; int ret; int offset, other, i, tags, regnr, tag, newtop; @@ -691,7 +691,7 @@ int fpregs_soft_get(struct task_struct *target, const struct user_regset *regset, struct membuf to) { - struct swregs_state *s387 = &target->thread.fpu.state.soft; + struct swregs_state *s387 = &target->thread.fpu.state->soft; const void *space = s387->st_space; int offset = (S387->ftop & 7) * 10, other = 80 - offset; diff --git a/arch/x86/math-emu/fpu_system.h b/arch/x86/math-emu/fpu_system.h index 9b41391867dc..a6291ddfdda6 100644 --- a/arch/x86/math-emu/fpu_system.h +++ b/arch/x86/math-emu/fpu_system.h @@ -73,7 +73,7 @@ static inline bool seg_writable(struct desc_struct *d) return (d->type & SEG_TYPE_EXECUTE_MASK) == SEG_TYPE_WRITABLE; } -#define I387 (¤t->thread.fpu.state) +#define I387 (current->thread.fpu.state) #define FPU_info (I387->soft.info) #define FPU_CS (*(unsigned short *) &(FPU_info->regs->cs)) From patchwork Wed Jun 30 06:02:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 12351265 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D197CC11F6B for ; Wed, 30 Jun 2021 06:08:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B47A161D4A for ; Wed, 30 Jun 2021 06:08:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232520AbhF3GKy (ORCPT ); Wed, 30 Jun 2021 02:10:54 -0400 Received: from mga02.intel.com ([134.134.136.20]:46942 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232335AbhF3GKq (ORCPT ); Wed, 30 Jun 2021 02:10:46 -0400 X-IronPort-AV: E=McAfee;i="6200,9189,10030"; a="195448562" X-IronPort-AV: E=Sophos;i="5.83,311,1616482800"; d="scan'208";a="195448562" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Jun 2021 23:08:17 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.83,311,1616482800"; d="scan'208";a="455156515" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga008.jf.intel.com with ESMTP; 29 Jun 2021 23:08:17 -0700 From: "Chang S. Bae" To: bp@suse.de, luto@kernel.org, tglx@linutronix.de, mingo@kernel.org, x86@kernel.org Cc: len.brown@intel.com, dave.hansen@intel.com, jing2.liu@intel.com, ravi.v.shankar@intel.com, linux-kernel@vger.kernel.org, chang.seok.bae@intel.com, kvm@vger.kernel.org Subject: [PATCH v6 09/26] x86/fpu/xstate: Update the XSTATE save function to support dynamic states Date: Tue, 29 Jun 2021 23:02:09 -0700 Message-Id: <20210630060226.24652-10-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210630060226.24652-1-chang.seok.bae@intel.com> References: <20210630060226.24652-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Extend os_xsave() to receive a mask argument of which states to save, in preparation for dynamic user state handling. Update KVM to set a valid fpu->state_mask, so it can continue to share with the core code. Signed-off-by: Chang S. Bae Reviewed-by: Len Brown Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org Cc: kvm@vger.kernel.org --- Changes from v5: * Adjusted the changelog and code for the new base code. Changes from v3: * Updated the changelog. (Borislav Petkov) * Made the code change more reviewable. Changes from v2: * Updated the changelog to clarify the KVM code changes. --- arch/x86/include/asm/fpu/internal.h | 3 +-- arch/x86/kernel/fpu/core.c | 2 +- arch/x86/kernel/fpu/signal.c | 2 +- arch/x86/kvm/x86.c | 9 +++++++-- 4 files changed, 10 insertions(+), 6 deletions(-) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index d2fc19c0e457..263e349ff85a 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -298,9 +298,8 @@ static inline void os_xrstor_booting(struct xregs_state *xstate) * Uses either XSAVE or XSAVEOPT or XSAVES depending on the CPU features * and command line options. The choice is permanent until the next reboot. */ -static inline void os_xsave(struct xregs_state *xstate) +static inline void os_xsave(struct xregs_state *xstate, u64 mask) { - u64 mask = xfeatures_mask_all; u32 lmask = mask; u32 hmask = mask >> 32; int err; diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c index 0c28e3d389e5..5b50bcf9f4ff 100644 --- a/arch/x86/kernel/fpu/core.c +++ b/arch/x86/kernel/fpu/core.c @@ -99,7 +99,7 @@ EXPORT_SYMBOL(irq_fpu_usable); void save_fpregs_to_fpstate(struct fpu *fpu) { if (likely(use_xsave())) { - os_xsave(&fpu->state->xsave); + os_xsave(&fpu->state->xsave, fpu->state_mask); /* * AVX512 state is tracked here because its use is diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index 2f35aada2007..f70f84d53442 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -365,7 +365,7 @@ static int __fpu_restore_sig(void __user *buf, void __user *buf_fx, * the right place in memory. It's ia32 mode. Shrug. */ if (xfeatures_mask_supervisor()) - os_xsave(&fpu->state->xsave); + os_xsave(&fpu->state->xsave, fpu->state_mask); set_thread_flag(TIF_NEED_FPU_LOAD); } __fpu_invalidate_fpregs_state(fpu); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 3c8b6080a253..57c1fa628a20 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -9637,11 +9637,16 @@ static void kvm_save_current_fpu(struct fpu *fpu) * KVM does not support dynamic user states yet. Assume the buffer * always has the minimum size. */ - if (test_thread_flag(TIF_NEED_FPU_LOAD)) + if (test_thread_flag(TIF_NEED_FPU_LOAD)) { memcpy(fpu->state, current->thread.fpu.state, get_xstate_config(XSTATE_MIN_SIZE)); - else + } else { + struct fpu *src_fpu = ¤t->thread.fpu; + + if (fpu->state_mask != src_fpu->state_mask) + fpu->state_mask = src_fpu->state_mask; save_fpregs_to_fpstate(fpu); + } } /* Swap (qemu) user FPU context for the guest FPU context. */