From patchwork Thu Jul 1 03:14:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Collingbourne X-Patchwork-Id: 12353547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D35FC11F64 for ; Thu, 1 Jul 2021 03:16:30 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DAE436145E for ; Thu, 1 Jul 2021 03:16:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DAE436145E Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Mime-Version: Message-Id:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=qKIsZPM/hD8zCFqkPMXN9PMKbfqkUUQUVks3jifC7Pg=; b=Oal mFfvmgU+EchsI2bCHhEhJSxVUfrVwpJMvJhDV5vPp74fXMjY9+U6cLbNnJP7YnMMlnLkU4l8mv5hc m3HPRJIzJLtQJEki86b5KMNG15XBFpis4Yzb1bQjYKpt9q0CxHH1C+QocSDnnxceCUlVx7SSbYbIu +7J6EhpWGKxCE58zbX8s5P0IZs35vC4E3kHScCsJ00Yh3tbFDbranbfjTKT/9xBPfPa2CeSM56KJb QjYhFDBMXW3oiKhJWwKAJ3KVVqfZiixl6hfe0lk0RVD44FljWmIZUYYu044vsuC4YpmrafBM1+i/u JzSeWWRVZqNA3iujPvqIPUoanJLh5NQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lynAL-00Fy46-Dt; Thu, 01 Jul 2021 03:15:01 +0000 Received: from mail-qv1-xf4a.google.com ([2607:f8b0:4864:20::f4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lynAI-00Fy3S-3a for linux-arm-kernel@lists.infradead.org; Thu, 01 Jul 2021 03:14:59 +0000 Received: by mail-qv1-xf4a.google.com with SMTP id g14-20020ad4510e0000b02902890eb69b5aso2842541qvp.21 for ; Wed, 30 Jun 2021 20:14:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=aP/jaNBR/y9xWgp5Fx/+7Dt+ywDOHfsR4wwlSmUC80o=; b=FLazmHy+S9bmsquDzdFTidj6KQoFi2X5xt3sMWthQNqFk+r97DzeUg2rZJ7yVPbLhx ROYx0PWiFEJ/wNcmMi1yUmrVL6JmdmvWWij5zwstsK7qrlcUndshOEF+LpoyfIEXFu1R P4QXsvAlWsHX+Z0zFbWlZmEU927fS6ypjJvOVcuLK1DF94ssLHMmV4qdlW3SPz5mWYAK YawNox3EeJ9fSj7asMtmxuzUCeX4+o1tIzlRrj4IUqFxAS/ubsYqEU+WMd/lRPdlKMSO Fk3JLQrWemRFppQmgzGX27tNjAY78GGrja1TcKsyQ/m3bIV4CNXmpJGkdoOGEBvpowsX f7jQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=aP/jaNBR/y9xWgp5Fx/+7Dt+ywDOHfsR4wwlSmUC80o=; b=iISjJArVuRz5QoReNCpbjNBSSNJ0nXNOJTjkKwZDqEijEzgYimWN3e38RpbU3Z3iPc BrIUz7oPj87uJSD3SDnT9Fh6UMghHbDpsh18bM1n9wLMs0kzieSRoVW9wHAI3vGxyUP2 WsEpy1kl4Zdcnu/HkoK8FDA6TQEUlq+QRy6L25WuxMQHaeKok6gOb7MeNftClDl542MK xmXNnIUePDD4HtKuJEoJU8F3XU+Uop5u/QPiF4oL/TAEyr4ukqi+MlK0fh7C1wSvLiKw XhWz2bGR04LglJ0Gl7RXI5AH69/5zhRZveOy7rXEr75S3MeWEWLXMOh/xI0EN4wPAHcn LxSA== X-Gm-Message-State: AOAM533POtS/AI2AEWf1k3IYEO6epoooXj7JP43QfpfAxctTi4zPl4GO zaHabyBW3I67zECmWse+p4tArKc= X-Google-Smtp-Source: ABdhPJwqs76LZ+4tTXUqrHHQQiwA4Adj/vIZGnRdhrysAOXABdGumRTfHBUgI+E1rZhWA4oFKUKfZfs= X-Received: from pcc-desktop.svl.corp.google.com ([2620:15c:2ce:200:8b00:104d:c249:d343]) (user=pcc job=sendgmr) by 2002:a0c:b8aa:: with SMTP id y42mr40062484qvf.17.1625109296242; Wed, 30 Jun 2021 20:14:56 -0700 (PDT) Date: Wed, 30 Jun 2021 20:14:48 -0700 Message-Id: <20210701031448.2173-1-pcc@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.32.0.93.g670b81a890-goog Subject: [PATCH] arm64: mte: avoid TFSR related operations unless in async mode From: Peter Collingbourne To: Catalin Marinas , Vincenzo Frascino , Will Deacon Cc: Peter Collingbourne , Evgenii Stepanov , linux-arm-kernel@lists.infradead.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210630_201458_212721_9121917B X-CRM114-Status: GOOD ( 14.35 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org There is no reason to touch TFSR nor issue a DSB unless our task is in asynchronous mode. Since these operations (especially the DSB) may be expensive on certain microarchitectures, only perform them if necessary. Signed-off-by: Peter Collingbourne Link: https://linux-review.googlesource.com/id/Ib353a63e3d0abc2b0b008e96aa2d9692cfc1b815 --- arch/arm64/kernel/entry.S | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 863d44f73028..c2338414c558 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -133,12 +133,18 @@ alternative_cb_end .endm /* Check for MTE asynchronous tag check faults */ - .macro check_mte_async_tcf, tmp, ti_flags + .macro check_mte_async_tcf, tmp, ti_flags, thread_sctlr #ifdef CONFIG_ARM64_MTE .arch_extension lse alternative_if_not ARM64_MTE b 1f alternative_else_nop_endif + /* + * Asynchronous tag check faults are only possible in ASYNC (2) or + * ASYM (3) modes. In each of these modes bit 1 of SCTLR_EL1.TCF0 is + * set, so skip the check if it is unset. + */ + tbz \thread_sctlr, #(SCTLR_EL1_TCF0_SHIFT + 1), 1f mrs_s \tmp, SYS_TFSRE0_EL1 tbz \tmp, #SYS_TFSR_EL1_TF0_SHIFT, 1f /* Asynchronous TCF occurred for TTBR0 access, set the TI flag */ @@ -151,11 +157,14 @@ alternative_else_nop_endif .endm /* Clear the MTE asynchronous tag check faults */ - .macro clear_mte_async_tcf + .macro clear_mte_async_tcf thread_sctlr #ifdef CONFIG_ARM64_MTE alternative_if ARM64_MTE + /* See comment in check_mte_async_tcf above. */ + tbz \thread_sctlr, #(SCTLR_EL1_TCF0_SHIFT + 1), 1f dsb ish msr_s SYS_TFSRE0_EL1, xzr +1: alternative_else_nop_endif #endif .endm @@ -231,8 +240,8 @@ alternative_else_nop_endif disable_step_tsk x19, x20 /* Check for asynchronous tag check faults in user space */ - check_mte_async_tcf x22, x23 - apply_ssbd 1, x22, x23 + ldr x0, [tsk, THREAD_SCTLR_USER] + check_mte_async_tcf x22, x23, x0 #ifdef CONFIG_ARM64_PTR_AUTH alternative_if ARM64_HAS_ADDRESS_AUTH @@ -245,7 +254,6 @@ alternative_if ARM64_HAS_ADDRESS_AUTH * was disabled on kernel exit then we would have left the kernel IA * installed so there is no need to install it again. */ - ldr x0, [tsk, THREAD_SCTLR_USER] tbz x0, SCTLR_ELx_ENIA_SHIFT, 1f __ptrauth_keys_install_kernel_nosync tsk, x20, x22, x23 b 2f @@ -258,6 +266,8 @@ alternative_if ARM64_HAS_ADDRESS_AUTH alternative_else_nop_endif #endif + apply_ssbd 1, x22, x23 + mte_set_kernel_gcr x22, x23 scs_load tsk @@ -362,6 +372,10 @@ alternative_else_nop_endif 3: scs_save tsk + /* Ignore asynchronous tag check faults in the uaccess routines */ + ldr x0, [tsk, THREAD_SCTLR_USER] + clear_mte_async_tcf x0 + #ifdef CONFIG_ARM64_PTR_AUTH alternative_if ARM64_HAS_ADDRESS_AUTH /* @@ -371,7 +385,6 @@ alternative_if ARM64_HAS_ADDRESS_AUTH * * No kernel C function calls after this. */ - ldr x0, [tsk, THREAD_SCTLR_USER] tbz x0, SCTLR_ELx_ENIA_SHIFT, 1f __ptrauth_keys_install_user tsk, x0, x1, x2 b 2f @@ -599,8 +612,6 @@ SYM_CODE_START_LOCAL(ret_to_user) cbnz x2, work_pending finish_ret_to_user: user_enter_irqoff - /* Ignore asynchronous tag check faults in the uaccess routines */ - clear_mte_async_tcf enable_step_tsk x19, x2 #ifdef CONFIG_GCC_PLUGIN_STACKLEAK bl stackleak_erase