From patchwork Thu Jan 19 21:22:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 13108784 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92C17C004D4 for ; Thu, 19 Jan 2023 21:23:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7741C6B0080; Thu, 19 Jan 2023 16:23:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6D6846B0081; Thu, 19 Jan 2023 16:23:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 418316B0088; Thu, 19 Jan 2023 16:23:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2C23A6B0080 for ; Thu, 19 Jan 2023 16:23:37 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id E7234AAAF5 for ; Thu, 19 Jan 2023 21:23:36 +0000 (UTC) X-FDA: 80372825232.30.F50B3B4 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by imf29.hostedemail.com (Postfix) with ESMTP id 0010512000F for ; Thu, 19 Jan 2023 21:23:34 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=XGv2svSQ; spf=pass (imf29.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674163415; a=rsa-sha256; cv=none; b=Rm3PmV5WLECHNsZImIBRF13BK2ITNQzwSGzNohboS8cGbev8HrG4rUQJX+feSSnPTdADh4 T864peQVxi/eGxHbi6qG7bykWDY+hVpRJ6EQy3T7H2y4i1RqUpKXJAci7j0v4HuCiB//U8 SuYqTnK0PsdaA/0RHAgtiCuOdFoQEZQ= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=XGv2svSQ; spf=pass (imf29.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674163415; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=/I2rA7g9gaeqAh7utVe0HX92Ho3+6Ko6l1A1i1JyDLw=; b=2f7k7K/drJPfCytRdBnzFm40vZYtbT5Ivi51VSATjaQ8SWwaa4SdtPa0hbX4i9L8ZMPSGd Bf5LobmpZo+1FcvX7MW/IyEKjeWowZ/q4EW93Vt6Ebsczquy681P2H0c8bNU/Y1hiTv7Hd EBjkLuWLXtMHGWfStgfXGJfNeoKd+bI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163415; x=1705699415; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=AD5tXA8hUee5SZ1Pd+Ll3tDzu/Wr8phq95vUnNoKkT8=; b=XGv2svSQ5sue7SbqSGaNPyyXdqh5nPX3klWE98bZ036cCkB/klMX/ZY1 NLzd6idsNWewSJAmvOQGlYDPIzW89pHt7STUnK+EHSbIwWxKcOO+zyO4j L4YJz1iktDYoVnz7m8dJgKuRLjpfu1qhVk0GUkSp0NP7dMa+u8ahbFib6 K4zThSQrldVI78cOBciJPCWjdF+r92HF1RbppetF6ndaIrUwFxJ2vlr8a NjwZJJkkIzgRpMvpg1+gqtkQQo/hvNDbPoyF0GUNThQT3XhYvXoa/ywLd cSpR2L56Maiw9pg6QPAOklJvLGtsfO6T6PWZxF4iSS0vb5wPHHz0lkizf A==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119295" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119295" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:34 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139011" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139011" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:32 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v5 06/39] x86/fpu: Add helper for modifying xstate Date: Thu, 19 Jan 2023 13:22:44 -0800 Message-Id: <20230119212317.8324-7-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Rspam-User: X-Rspamd-Queue-Id: 0010512000F X-Rspamd-Server: rspam01 X-Stat-Signature: 4wuf66ia56sa5a1nstg7t84a1mccmz4q X-HE-Tag: 1674163414-389894 X-HE-Meta: U2FsdGVkX181PWi47RDXV/MtHZp+4bVf0Gd8nLdSsBmkjvBIQoq2XkXKk5XKFlVBG7UHLxjsALG7ogxpDzEu1Cd8R4e3NXed3zSDqa3nh8dAPxkpDiq/PYNsRk9ENy7escPBpZ9stqCWQOnfeVBqtD9GMvQXzlhkppWhE8gVl2VZ4zT+V6A8mWJNDjKGJPSKHr5/jlfNQBWh4HofqTz/c2GiwPhGKr8uCPSBN5GhCCVWWOU9UxCU74gXk2ouB9Nm+fjGXNnr9EEjkKdCrRxZ0sS5DimMiXvo1knuscL0DY11Q8vEdCzh0AWJWi5tZ5b+rSVbkwut5LZxnnPAZMmzTLIKi6sxnbq39HoDzzJn+POAN9qxyLZZ5lbsU1/JI00vT5fxSC3XsituB/sWxIEwiM2Hq+8YmNlHTLo3BmS8M061v9/KLox30LybWgXlHZ2NOcFd8wit11GgGCzpH4ugJvx9qVWT1Ubzqor+ncQjBaJRBx1dl3WLjoCMbIwUPz92KrZZKh1hrfsVauZ81pln21temyr7lyCRxl33MuIv+NuMy/DJmiYAtGfY8msgyf5qYAtnJxTikkk5mQxgf0jV40oOUngmgvfzBNrEs0aluQh9pFZx3ZkF8FIFq5K4ozslUJShAe14mfJhB/RWBkh2+PNUrHD5LxhpLzb2+VSn9+J5FjQXM0EE4ZKUdPo7ipfeAyyBBgmcU7aERCA2kkeMpdgnoBBkH4I6URYfPI5Y2h74T7GLQAzx+djDLfIz8rtEv6t6le/U7eL2HUOGmStCVhEYH9jI1bKEyCTPR/tbuxalOctWS9Q76bDPY5CpcbWUkhAOeekbvaBEBT4gso0n+A+arixBP0c2jyWfYZMYlFNcGEDjE0WMaxvVEJhVtemqng9WxWug5AM/RJbdsp4Gb5IcvG5hbAlTnkshYESFQ2BYq6mbySWM4VzpTvvvgbf6ggvX3JOShWdoEE5kmvt 25UMx+ua sXVa16vYfC4YvLp9vAliXrpWf4fQmqBJPrBC56KHOmIi+2v30btm+XAjRrEEmB7bpqHPNvpZaDnAvgflZ90nopss9d4dJxDupxzSXZrvNY+YucHX+e8bn+LYnLjMK3W6PCD8TFmhAHawS+TwwxBrNt+K+nNj8bdGZwvZeO6ELHJag9tAcgsUkvfnssS10K7dZi+k2SOQ8jdzB/ECv40Ng8Hg4+NifCQq5cT9vNzCdIUbgACs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Just like user xfeatures, supervisor xfeatures can be active in the registers or present in the task FPU buffer. If the registers are active, the registers can be modified directly. If the registers are not active, the modification must be performed on the task FPU buffer. When the state is not active, the kernel could perform modifications directly to the buffer. But in order for it to do that, it needs to know where in the buffer the specific state it wants to modify is located. Doing this is not robust against optimizations that compact the FPU buffer, as each access would require computing where in the buffer it is. The easiest way to modify supervisor xfeature data is to force restore the registers and write directly to the MSRs. Often times this is just fine anyway as the registers need to be restored before returning to userspace. Do this for now, leaving buffer writing optimizations for the future. Add a new function fpregs_lock_and_load() that can simultaneously call fpregs_lock() and do this restore. Also perform some extra sanity checks in this function since this will be used in non-fpu focused code. Tested-by: Pengfei Xu Tested-by: John Allen Suggested-by: Thomas Gleixner Signed-off-by: Rick Edgecombe Reviewed-by: Kees Cook --- v5: - Fix spelling error (Boris) - Don't export fpregs_lock_and_load() (Boris) v3: - Rename to fpregs_lock_and_load() to match the unlocking fpregs_unlock(). (Kees) - Elaborate in comment about helper. (Dave) v2: - Drop optimization of writing directly the buffer, and change API accordingly. - fpregs_lock_and_load() suggested by tglx - Some commit log verbiage from dhansen v1: - New patch. arch/x86/include/asm/fpu/api.h | 9 +++++++++ arch/x86/kernel/fpu/core.c | 18 ++++++++++++++++++ 2 files changed, 27 insertions(+) diff --git a/arch/x86/include/asm/fpu/api.h b/arch/x86/include/asm/fpu/api.h index 503a577814b2..aadc6893dcaa 100644 --- a/arch/x86/include/asm/fpu/api.h +++ b/arch/x86/include/asm/fpu/api.h @@ -82,6 +82,15 @@ static inline void fpregs_unlock(void) preempt_enable(); } +/* + * FPU state gets lazily restored before returning to userspace. So when in the + * kernel, the valid FPU state may be kept in the buffer. This function will force + * restore all the fpu state to the registers early if needed, and lock them from + * being automatically saved/restored. Then FPU state can be modified safely in the + * registers, before unlocking with fpregs_unlock(). + */ +void fpregs_lock_and_load(void); + #ifdef CONFIG_X86_DEBUG_FPU extern void fpregs_assert_state_consistent(void); #else diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c index dccce58201b7..7317bfd5ea36 100644 --- a/arch/x86/kernel/fpu/core.c +++ b/arch/x86/kernel/fpu/core.c @@ -753,6 +753,24 @@ void switch_fpu_return(void) } EXPORT_SYMBOL_GPL(switch_fpu_return); +void fpregs_lock_and_load(void) +{ + /* + * fpregs_lock() only disables preemption (mostly). So modifying state + * in an interrupt could screw up some in progress fpregs operation, + * but appear to work. Warn about it. + */ + WARN_ON_ONCE(!irq_fpu_usable()); + WARN_ON_ONCE(current->flags & PF_KTHREAD); + + fpregs_lock(); + + fpregs_assert_state_consistent(); + + if (test_thread_flag(TIF_NEED_FPU_LOAD)) + fpregs_restore_userregs(); +} + #ifdef CONFIG_X86_DEBUG_FPU /* * If current FPU state according to its tracking (loaded FPU context on this