From patchwork Mon Jul 24 12:46:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 13324637 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6405BC0015E for ; Mon, 24 Jul 2023 12:48:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 075776B0085; Mon, 24 Jul 2023 08:48:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 024856B0087; Mon, 24 Jul 2023 08:48:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E2F518E0001; Mon, 24 Jul 2023 08:48:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D2E7B6B0085 for ; Mon, 24 Jul 2023 08:48:52 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A241A1A0A94 for ; Mon, 24 Jul 2023 12:48:52 +0000 (UTC) X-FDA: 81046484904.21.5B9DA5C Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf22.hostedemail.com (Postfix) with ESMTP id 7268AC000F for ; Mon, 24 Jul 2023 12:48:50 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=GA7d9sNT; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf22.hostedemail.com: domain of broonie@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=broonie@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690202930; a=rsa-sha256; cv=none; b=I/8I0udeuqbyg/mbdEFTMk5SSwfvvybklSxjm/aC3Dl8yGwAtNWbfK8YnviMR8wNQ73DO7 y07Puo9upReQ4WfGl4RLApL384IuLYvxc7iKif4wJmU/ljouawnjQAOd2bTFQR2QXCiatz e4TlmY7b4yrOL63bq4Kygrrq3FMilyg= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=GA7d9sNT; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf22.hostedemail.com: domain of broonie@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=broonie@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690202930; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DhriSxQor3DCGKL2zl7My1a/kfJpgsRjjKwA8agrULg=; b=Sx6QtbNLECO/x8JEuosDOcLh2R8+vDR98+cvUNkTYy2l4hJ+uBzVI6nYRwRonm5sj3mTZO uU9u9wkZfOTdVvIGeu4rfYABT5VLKgib/8y/D5FNlHB/1RvXY1cpcRIN/aAUYa3hPi4QSi e3Sx7HxxZA7cgvnpFk6gG1kQyjtuUi0= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7B96E61150; Mon, 24 Jul 2023 12:48:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 139E7C433C7; Mon, 24 Jul 2023 12:48:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690202928; bh=Qk1oAglcSxLTK1OL1yhAbWpcUwaL4dga6Y2wJZdhO9Q=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=GA7d9sNT/rJ0WJ+xEL1kedzmwFpOvzkvWpWRRSXDefPpF0h7Z3LoT3il5uE0/jaGL L6aLEJLHytNQA3WxKZLSH6zh8Ow1I0Sn6BDkVTKcQtowTsnxfbvl8xAWfXIBC43yNa gEuVty8LRHrHffr0mz/NHq9wuN5cxiRR+uZ0kCpgifaMxeY4L1cEPV1cDAt8Uw0LjM ZkAdqcq7fwA52dr5Nft0H2gFnqGTWJxohX1c7MFcual5CLTH0ItrzoHs2zGqDqt+32 qB+MJmtKIkbPeklyMwx3GGljV2KcajFETid9doKwrM3Xrz5jn8emTI7Aa/1hh1Wkzb cZC8+Lx9lXWJw== From: Mark Brown Date: Mon, 24 Jul 2023 13:46:04 +0100 Subject: [PATCH v2 17/35] arm64/mm: Handle GCS data aborts MIME-Version: 1.0 Message-Id: <20230724-arm64-gcs-v2-17-dc2c1d44c2eb@kernel.org> References: <20230724-arm64-gcs-v2-0-dc2c1d44c2eb@kernel.org> In-Reply-To: <20230724-arm64-gcs-v2-0-dc2c1d44c2eb@kernel.org> To: Catalin Marinas , Will Deacon , Jonathan Corbet , Andrew Morton , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Arnd Bergmann , Oleg Nesterov , Eric Biederman , Kees Cook , Shuah Khan , "Rick P. Edgecombe" , Deepak Gupta , Ard Biesheuvel , Szabolcs Nagy Cc: "H.J. Lu" , Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Mark Brown X-Mailer: b4 0.13-dev-099c9 X-Developer-Signature: v=1; a=openpgp-sha256; l=5613; i=broonie@kernel.org; h=from:subject:message-id; bh=Qk1oAglcSxLTK1OL1yhAbWpcUwaL4dga6Y2wJZdhO9Q=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBkvnKe3zPPjWT4b5WmOZ3LAhxzwLz4yNXLUgcttsYr ud3QVBKJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZL5yngAKCRAk1otyXVSH0CvEB/ 4jJiJu+LxDMxSGM2XECjRZURDqgeoOVcHxDefJs3PqcXqoU78dy0rTwv9nHdWMNe+DxcP2JfpG4vEb k9vfD4lKa5yBhhGMdAlXIkCXPuwQUuzai8rSnHIT/cdu+2ZVa6Hojaz9I5uQVc0pyJeOkZtFnCqtHB 77pPbY8SwUtKgnD5M77EcPmScDx3sKQBKkzWB3KLbTeMZDA8yfqMYK4CpIFEDwig1tpVD2vQfmE3NB q2o7Gyg5xJ2kaaV981T3GAAMlM9XFNkEzmfBPLjurRO3/0LXeRE0v0wNUaOdhK0nmUK36u9mja0sU1 6MZAvqkH9o6DodF4d0KyhqXs9hG9k8 X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 7268AC000F X-Stat-Signature: 7z9p4sk6hikcewoofymjfbg7un9hgud1 X-HE-Tag: 1690202930-882638 X-HE-Meta: U2FsdGVkX1/mIjckuF7AfBewRJ5diUcw+Nkx+QohWaUbeR3FUKbwvNA+mG5EAEGnsOH9YrfRhlYdFC6FtUi4wGt5xG32ine7ySO0XqH3sxWJao3elUb4z9XkGvW2PAsjVSw6LR1MWuPzYkWOMAqWcAIzXwKIgSqKIPmUNzjxH1VclbXSAs8qe8PdacmNP6roEIkrbEXvXBf/BUteMldrNA+SbY5Iq/BG6iAuH9Uy1cEhb8TQIttX0zrKQvYlmsgI5vmPaLcZ/oKbifiYvW9w1ds+2QOBl3KBVW47M6XNiQw+SPXODfFDYyle4GUW/i8L/FYVxfbgsHKjp+qKhLJ4FOjS2575DTFKgscEy+azxdPtQzGEK+LDq491tIW+rHbWrOQ+mrPjLUrO0H7inSWjL4EbiCGa+L3bo0883FrWXay9h4ND0PQVJjDxBbUgM1EDHe+YMBRT/bnsKLU1W2GcJQyz+cel7zinbUKBArsLx6tt1HoLTTyi9INErrAKNVkGmPKjz7qoBsem2euiGEoqDOlrsIE2QN9HuQNa5Gg7s5XCVHO36h44BYMEzgKefbkSy0Y2nBwuVUKkIRtEN1bdhjD+4YwZmBZY+Qlr3wBJaRkfLACh9drpmiIP84bVZvDVDR8nbKw8jLQ76d9E6THAR5IREwFXEndBw7vbXKEwMbOMO/w9sEjwADWDoImCnYdLrLyfyq0pxV19KlHU94vDxC7EZoWx1K7ZJFq6d+n3L77n22OQEfWhebR8kQQPXq620AWFR+DtqNTJsNXgaGSohTD9XukKo0dhgKg9ecGnga8cFJumpFsroyG+HpoABH1MoS6XhMlP5ijb0knlN+B7GPFVjt06hQBDcI/TdBRXDB1VRKErHX/BjGP0i3rw9MA4YQcRfE0xARQNPiBJ+Y887yPhkNFaT/+ljXcNQGCii0nFgD1hLkAbi7oLt+crZC/3JZAiWbkWct8mCTKneyf 6ckgjTX5 v8wuiW0W3hUWFNI7jqwEwtt2xBdgAxos66lVsDs5Gwe2Ket3nz/6OHG6XLFReavgfwI+l0Hg9kK5g5msTj1pb+N2svKCjWIq5aEGOX1riH31HQBpb6wSOYf71GO8rO9cpguZZaLTwrs+AYdQd8loSGJ9DouucJvhnyanKvLXS/GxEbwff0W2qKKZJZ8ovVC0K8VFhWTMf2bW2KmZixcZrylxW5+9OsmTEneI6BO53RB0/5bQ9RIa1/MSTciLFdkhLc9IwjKaAmiOl/1EbxNGlhhxtlqZqOQY/0TB07GGxgeV5i2dfHVTOE6OJCWjF9wuUBvg0wjale2vKmJGp1GWwEl8Gq9g8Qs1z0zVsLKBx1dswy4Iz0jDJaaYIzgg7FC88uGOApY6Z1Qh2lmSqpIl8oM2W5iOW+a2VlJZOfhDp8GwYTGQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All GCS operations at EL0 must happen on a page which is marked as having UnprivGCS access, including read operations. If a GCS operation attempts to access a page without this then it will generate a data abort with the GCS bit set in ESR_EL1.ISS2. EL0 may validly generate such faults, for example due to copy on write which will cause the GCS data to be stored in a read only page with no GCS permissions until the actual copy happens. Since UnprivGCS allows both reads and writes to the GCS (though only through GCS operations) we need to ensure that the memory management subsystem handles GCS accesses as writes at all times. Do this by adding FAULT_FLAG_WRITE to any GCS page faults, adding handling to ensure that invalid cases are identfied as such early so the memory management core does not think they will succeed. The core cannot distinguish between VMAs which are generally writeable and VMAs which are only writeable through GCS operations. EL1 may validly write to EL0 GCS for management purposes (eg, while initialising with cap tokens). We also report any GCS faults in VMAs not marked as part of a GCS as access violations, causing a fault to be delivered to userspace if it attempts to do GCS operations outside a GCS. Signed-off-by: Mark Brown --- arch/arm64/mm/fault.c | 78 +++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 70 insertions(+), 8 deletions(-) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 3fe516b32577..ec392207a475 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -494,13 +494,30 @@ static void do_bad_area(unsigned long far, unsigned long esr, } } +/* + * Note: not valid for EL1 DC IVAC, but we never use that such that it + * should fault. EL0 cannot issue DC IVAC (undef). + */ +static bool is_write_abort(unsigned long esr) +{ + return (esr & ESR_ELx_WNR) && !(esr & ESR_ELx_CM); +} + +static bool is_gcs_fault(unsigned long esr) +{ + if (!esr_is_data_abort(esr)) + return false; + + return ESR_ELx_ISS2(esr) & ESR_ELx_GCS; +} + #define VM_FAULT_BADMAP ((__force vm_fault_t)0x010000) #define VM_FAULT_BADACCESS ((__force vm_fault_t)0x020000) static vm_fault_t __do_page_fault(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, unsigned int mm_flags, unsigned long vm_flags, - struct pt_regs *regs) + unsigned long esr, struct pt_regs *regs) { /* * Ok, we have a good vm_area for this memory access, so we can handle @@ -510,6 +527,26 @@ static vm_fault_t __do_page_fault(struct mm_struct *mm, */ if (!(vma->vm_flags & vm_flags)) return VM_FAULT_BADACCESS; + + if (vma->vm_flags & VM_SHADOW_STACK) { + /* + * Writes to a GCS must either be generated by a GCS + * operation or be from EL1. + */ + if (is_write_abort(esr) && + !(is_gcs_fault(esr) || is_el1_data_abort(esr))) + return VM_FAULT_BADACCESS; + } else { + /* + * GCS faults should never happen for pages that are + * not part of a GCS and the operation being attempted + * can never succeed. + */ + if (is_gcs_fault(esr)) + return VM_FAULT_BADACCESS; + } + + return handle_mm_fault(vma, addr, mm_flags, regs); } @@ -518,13 +555,18 @@ static bool is_el0_instruction_abort(unsigned long esr) return ESR_ELx_EC(esr) == ESR_ELx_EC_IABT_LOW; } -/* - * Note: not valid for EL1 DC IVAC, but we never use that such that it - * should fault. EL0 cannot issue DC IVAC (undef). - */ -static bool is_write_abort(unsigned long esr) +static bool is_invalid_el0_gcs_access(struct vm_area_struct *vma, u64 esr) { - return (esr & ESR_ELx_WNR) && !(esr & ESR_ELx_CM); + if (!system_supports_gcs()) + return false; + if (likely(!(vma->vm_flags & VM_SHADOW_STACK))) { + if (is_gcs_fault(esr)) + return true; + return false; + } + if (is_gcs_fault(esr)) + return false; + return is_write_abort(esr); } static int __kprobes do_page_fault(unsigned long far, unsigned long esr, @@ -573,6 +615,13 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, /* If EPAN is absent then exec implies read */ if (!cpus_have_const_cap(ARM64_HAS_EPAN)) vm_flags |= VM_EXEC; + /* + * Upgrade read faults to write faults, GCS reads must + * occur on a page marked as GCS so we need to trigger + * copy on write always. + */ + if (is_gcs_fault(esr)) + mm_flags |= FAULT_FLAG_WRITE; } if (is_ttbr0_addr(addr) && is_el1_permission_fault(addr, esr, regs)) { @@ -595,6 +644,19 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, if (!vma) goto lock_mmap; + /* + * We get legitimate write faults for GCS pages from GCS + * operations and from EL1 writes to EL0 pages but just plain + * EL0 writes are invalid. Specifically check for this since + * as a result of upgrading read faults to write faults for + * CoW the mm core isn't able to distinguish these invalid + * writes. + */ + if (is_invalid_el0_gcs_access(vma, esr)) { + vma_end_read(vma); + goto lock_mmap; + } + if (!(vma->vm_flags & vm_flags)) { vma_end_read(vma); goto lock_mmap; @@ -624,7 +686,7 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, goto done; } - fault = __do_page_fault(mm, vma, addr, mm_flags, vm_flags, regs); + fault = __do_page_fault(mm, vma, addr, mm_flags, vm_flags, esr, regs); /* Quick path to respond to signals */ if (fault_signal_pending(fault, regs)) {