From patchwork Wed May 22 09:13:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13670625 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 91217C25B77 for ; Wed, 22 May 2024 09:14:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: Mime-Version:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=zQV5eUqaD4nuQSC9xoI51Iq4Gxwb6lRRz/fMso73Xh8=; b=AuR M/qW49BBoazX3Vl7+EzTNIH6bzy5gD/S3XlxrvuRf/J37YRM3rnKr32d1LSvHPfO+EryXZjCnBZ6Z MrIeNylT1uKHTZ+SRp5V0Y+uTKiqQ+UH5J6ywS6tvog7cTfKb7sCT4lCdbo6Gf8pUWV9N1BgVkxDu nNyi/DUtOmioEaSl6vJpIIpKXWEq1gxFL2sS+7a2wkTGIHRd3z6n9PTtfyR4P/VBtA3Pd+eDzUWlh Z7zUZUJZK9LVcvFs+Xa2YZXtftXfcCMHCBOmo8ncKI6ovxcDxMhCEmL7knnD6WqWQ4kTHjd6h4G4D amRq68YEyrxN+mHBiN05v6trjFTa+ZA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s9i30-00000002TC7-3xtg; Wed, 22 May 2024 09:14:10 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s9i2x-00000002T9R-3MNm for linux-arm-kernel@lists.infradead.org; Wed, 22 May 2024 09:14:09 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-df4dfceec00so1064894276.3 for ; Wed, 22 May 2024 02:14:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1716369243; x=1716974043; darn=lists.infradead.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=XuIA2lH6na2a94kxA/DTlJDooCvUPEJJ7XJAX9Gke70=; b=Iu34n7PyKuh4bhcVdkW7Z2IAWlegTKHBpOzpLU4pX6G12M7yzBmLU0ZdDyih3/Z/uG NxbwOC0MldeAhWRYyFvCiPQXx1J1zeuWRa3IgP8Rr22fg4B6MxkMGVrofyI0khN0q9jS 2q5uWwPhCpe50fhvnTncSwNglSgYUisLdYiuvVOcilIUXxTq+vajVhp2cGlVjJghg3IT XDMwTQZE5DggFKYWwad5EEZFvCyJrXe4HyyKQuK5Y83/4jgVZzbGtA6r2/R+O2cn5nK/ XQSTyv+pCob+k3BHuZPyGDnDubWGMu4yDjSN3ZxPSTwMulYQEoL7FVmpudNwSPQ3H4tX V+ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716369243; x=1716974043; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=XuIA2lH6na2a94kxA/DTlJDooCvUPEJJ7XJAX9Gke70=; b=HAnuiT4nocPf4g3+Sv9RvUNpZaIzIhB3jUZmjZKyJyO6fx8X1TU+0nANjhN4YyIYz7 0q8iGZ7xsHxNhRH7qv53vQAXAxuJnFb/lSnpWeubEqTIY2eD4BVqsYRt2u8huPyNBfo7 yO5o9j0SuBLaCzgX/iNuAg9SopeaAEgbb/NxpYlSc3kbFj6PRaYqvPAqtnXnRS5ht+I1 io3SSv3nStua+GCNDci6LLIvPWQNRO2AR2tR3fcyfDPieTA9TWXZyyWMl2YO3TyJLtwL vH3oOaR0zYAe37kEXgza5JyQEiFfDqcG7bGhO4mkPB3e7wL4RQ4w7zRYoO7fASbZ1uID Fibw== X-Gm-Message-State: AOJu0Yw3i1neCLwOGZpBb2wsN6n1q9Mua4NgZ9RtlDNfbQ+NkZZb/+N4 cHZ4l3MJznArjleOdSRjt9d9YCcKCUkq5HJVzdxHibO3ESqvb7CmMNN9FFrsINWRsKrBJmLm8Wi CeGzwszBg1I1gx8V1jFL8a8KOZGro802R/aq1dwXeiLondyO5d3v6cSGakvthoFApBxnV5WjaqA LzPE0GcdxicMpxOOBGPkJrRi3LT5Q9Z74Yv1w/yyvk X-Google-Smtp-Source: AGHT+IG7x9xP5IfsBJd2ycxebqXW66nf/YNJCkqNUQc76oX/whvQj5+O3eDOJwX87oV9OZ1mB3k2eXbG X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1022:b0:df4:9688:b28 with SMTP id 3f1490d57ef6-df4e0beb961mr144162276.3.1716369242949; Wed, 22 May 2024 02:14:02 -0700 (PDT) Date: Wed, 22 May 2024 11:13:36 +0200 Mime-Version: 1.0 X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=4147; i=ardb@kernel.org; h=from:subject; bh=VI8H3TWefNY52ZELPdqJfXtw/JEASZvzMgnjhW4wMTQ=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIc13u/1jry+n16z8++KXzpPG/Qf50nfY6E5NDGSYeSrW8 PXS2WfFOkpZGMQ4GGTFFFkEZv99t/P0RKla51myMHNYmUCGMHBxCsBEZOsZGRru5U2eb6z/5u6O mCm6ms3tzj5h5u7qRnFhH4Kj/Cc8nM/IMKtqTbC8sGuv9cST80OW3/aLX/JAa6YKD9MbXpafBr9 CWAE= X-Mailer: git-send-email 2.45.1.288.g0e0cd299f1-goog Message-ID: <20240522091335.335346-2-ardb+git@google.com> Subject: [PATCH] arm64/fpsimd: Avoid erroneous elide of user state reload From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: will@kernel.org, catalin.marinas@arm.com, maz@kernel.org, mark.rutland@arm.com, Ard Biesheuvel , Johannes Nixdorf , Mark Brown , Dave Martin , Janne Grunau X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240522_021407_891739_2C53067C X-CRM114-Status: GOOD ( 18.50 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel TIF_FOREIGN_FPSTATE is a 'convenience' flag that should reflect whether the current CPU holds the most recent user mode FP/SIMD state of the current task. It combines two conditions: - whether the current CPU's FP/SIMD state belongs to the task; - whether that state is the most recent associated with the task (as a task may have executed on other CPUs as well). When a task is scheduled in and TIF_KERNEL_FPSTATE is set, it means the task was in a kernel mode NEON section when it was scheduled out, and so the kernel mode FP/SIMD state is restored. Since this implies that the current CPU is *not* holding the most recent user mode FP/SIMD state of the current task, the TIF_FOREIGN_FPSTATE flag is set too, so that the user mode FP/SIMD state is reloaded from memory when returning to userland. However, the task may be scheduled out after completing the kernel mode NEON section, but before returning to userland. When this happens, the TIF_FOREIGN_FPSTATE flag will not be preserved, but will be set as usual the next time the task is scheduled in, and will be based on the above conditions. This means that, rather than setting TIF_FOREIGN_FPSTATE when scheduling in a task with TIF_KERNEL_FPSTATE set, the underlying state should be updated so that TIF_FOREIGN_FPSTATE will assume the expected value as a result. So instead, call fpsimd_flush_cpu_state(), which takes care of this. Closes: https://lore.kernel.org/all/cb8822182231850108fa43e0446a4c7f@kernel.org Reported-by: Johannes Nixdorf Fixes: aefbab8e77eb ("arm64: fpsimd: Preserve/restore kernel mode NEON at context switch") Cc: Mark Brown Cc: Dave Martin Cc: Janne Grunau Signed-off-by: Ard Biesheuvel Tested-by: Janne Grunau Tested-by: Johannes Nixdorf Reviewed-by: Mark Brown --- arch/arm64/kernel/fpsimd.c | 44 +++++++++++++++++++------------------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index ebb0158997ca..82e8a6017382 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -1535,6 +1535,27 @@ static void fpsimd_save_kernel_state(struct task_struct *task) task->thread.kernel_fpsimd_cpu = smp_processor_id(); } +/* + * Invalidate any task's FPSIMD state that is present on this cpu. + * The FPSIMD context should be acquired with get_cpu_fpsimd_context() + * before calling this function. + */ +static void fpsimd_flush_cpu_state(void) +{ + WARN_ON(!system_supports_fpsimd()); + __this_cpu_write(fpsimd_last_state.st, NULL); + + /* + * Leaving streaming mode enabled will cause issues for any kernel + * NEON and leaving streaming mode or ZA enabled may increase power + * consumption. + */ + if (system_supports_sme()) + sme_smstop(); + + set_thread_flag(TIF_FOREIGN_FPSTATE); +} + void fpsimd_thread_switch(struct task_struct *next) { bool wrong_task, wrong_cpu; @@ -1552,7 +1573,7 @@ void fpsimd_thread_switch(struct task_struct *next) if (test_tsk_thread_flag(next, TIF_KERNEL_FPSTATE)) { fpsimd_load_kernel_state(next); - set_tsk_thread_flag(next, TIF_FOREIGN_FPSTATE); + fpsimd_flush_cpu_state(); } else { /* * Fix up TIF_FOREIGN_FPSTATE to correctly describe next's @@ -1842,27 +1863,6 @@ void fpsimd_flush_task_state(struct task_struct *t) barrier(); } -/* - * Invalidate any task's FPSIMD state that is present on this cpu. - * The FPSIMD context should be acquired with get_cpu_fpsimd_context() - * before calling this function. - */ -static void fpsimd_flush_cpu_state(void) -{ - WARN_ON(!system_supports_fpsimd()); - __this_cpu_write(fpsimd_last_state.st, NULL); - - /* - * Leaving streaming mode enabled will cause issues for any kernel - * NEON and leaving streaming mode or ZA enabled may increase power - * consumption. - */ - if (system_supports_sme()) - sme_smstop(); - - set_thread_flag(TIF_FOREIGN_FPSTATE); -} - /* * Save the FPSIMD state to memory and invalidate cpu view. * This function must be called with preemption disabled.