From patchwork Wed Feb 14 12:28:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556412 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 77799C48BC3 for ; Wed, 14 Feb 2024 12:30:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=bZcpm2vP19+bF8DNUkb7CXQYmmuBJPzS+29PTgDTl5Y=; b=q0GWdCXY72Aqn+6VW/wXTrOjKT xNzGmiHYHtg7WT2UOikmyIGWme79McHAU/LzCHV7DQKwKeRF6Z/xOkpCI8aHRf1xoIXanZkb1UTc0 HXv8nhXXM69kfx48+1AU871RxBw7l//+PzdLX1zYyC2yk8oIGDLuBXrXJELRVMhuiAmGASrIO/wYI KMOlEC8dDS0gLizDz+Zmdya209DvwZVrXld1tlwU8nVtg8Il3DVfkUbIjWdgXpoIW1SNF9jTq3pCg wZhomO83alvjA6AOltfBQxAPMvAAzycy56vyGv9O9mWyBaupQI/ySO295gCh8suj6vOi+ZbfP1im+ imp6XRCw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPE-0000000Cocs-0jNR; Wed, 14 Feb 2024 12:30:28 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEOp-0000000CoLF-0THl for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:09 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-33cee70ef73so195284f8f.1 for ; Wed, 14 Feb 2024 04:30:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913799; x=1708518599; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5T+JZlXorLJgdWJ3YLLbdFriHCmLYbGuJvbl7iiPg80=; b=u/4muvm/WJHlTkq0V14fUA4VPRz4+T2mZC7o1b8eI6/pfy3Jy17oSJ1GHhiWg+m6lr OI1azahohEelc0S6iFhnWfmXSCDefL8Ix0gWUnxKqW8AFxLiBJHhNl4yWLgOH9V8Q/tV GnGvD5Q0zYjuVkgiPdLWaJz6Zz4O8P5XB1UzDlrs3DpVPF9XIyds+P5657LC2yYX1Io6 +yLTTsm45ChaLYFLYyoiNkgCU2gvFpehk+52jM4OGoVUBBbg97zRYX5QOswctxeoIQ6U vJQi1SE4RTWzOq2aG0DBKyiahT2ZCiAiCe6wdlEUhzBgPsAAkjh3KOzFpU9Qgp2USzhb qxQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913799; x=1708518599; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5T+JZlXorLJgdWJ3YLLbdFriHCmLYbGuJvbl7iiPg80=; b=LNjhJc1V4vCb8huaOaEbYSOcfBPUQAwBbVHRpt6Ayz1TvaEr/OxKwiz/h0GoKwK7ar QZfwtljPuzEXUii+na7EYTCGbrHhswoWOLbY/bjpZYkez4oIBmZkmSxYuRVjGzM4IqFL QoXl1QxoUDRjbjEOA52FX8zjOqPoKkQGOClAKDaWNBU/HEFy+dmO/fSRdnneMwk+dEre K42sHbfNttnmrZ9OjnFkhkf3yNIGXxeXM0Z4e8DEF31YpjD2qHuEWDVfuPXxpTbFonX8 ZFWEoGsk2+IL0ZNaxX4ZWzs02HDeq196t2lPf0NZUDH6nnYZ9B3+FRvkotGQqWxJH5fm kaww== X-Gm-Message-State: AOJu0YxrhPr5wjs37IN08CUSNvT0IuSKp4ggpCw1x1m3XDq5rDuEM6Rt Hmvb35gvIQScL6YWDmzWxuFzuCxlNtA2owxyOsaHBuN6hruCA9Bvm0wSB05HRS+C3gpyY4PXjZ4 2Gmltjpyi4KhZZ4C4j9d+2ZxIm8vYo4f0DtH7kByFTTnUB54Uc/tuprYZkEwsDj0Q5IDXonv95R BfaKBwwPXHIGdlVylewO8g5KRcDBE2xv882pGDLady X-Google-Smtp-Source: AGHT+IF6+nN0KwInZ+gksSxO019cnqoyiiIyBKUSB5BtKzETRg3zo6CY75xpkNN3HSk6hR3KlvML/M/J X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:6e51:0:b0:33b:8867:9253 with SMTP id j17-20020a5d6e51000000b0033b88679253mr4592wrz.3.1707913799333; Wed, 14 Feb 2024 04:29:59 -0800 (PST) Date: Wed, 14 Feb 2024 13:28:55 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6671; i=ardb@kernel.org; h=from:subject; bh=IzdxyEQm3auU1iEc65FDEPHnvsNyp60IBKqPiNiKPyM=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJo7tO093/+evKHpb8NnPdXus6HGGSGEFF5srdZ/3V rdUSjt0lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgIl8Y2VkmMp4yLu0tpbbJqp/ xuVjoU3VNk8SdLQjtka7r+p9Ynz9OCPDg5q0H6EmemZPl06wc3p/fN9/3elvjm6ZsHbTyhVVnFW 3OQE= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-54-ardb+git@google.com> Subject: [PATCH v8 09/43] arm64: head: move dynamic shadow call stack patching into early C runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_043003_349034_541C6E10 X-CRM114-Status: GOOD ( 18.94 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Once we update the early kernel mapping code to only map the kernel once with the right permissions, we can no longer perform code patching via this mapping. So move this code to an earlier stage of the boot, right after applying the relocations. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/scs.h | 4 +-- arch/arm64/kernel/Makefile | 8 ------ arch/arm64/kernel/head.S | 8 +++--- arch/arm64/kernel/module.c | 2 +- arch/arm64/kernel/pi/Makefile | 10 +++++--- arch/arm64/kernel/{ => pi}/patch-scs.c | 26 ++++++++++---------- 6 files changed, 27 insertions(+), 31 deletions(-) diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index 3fdae5fe3142..eca2ba5a6276 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -72,8 +72,8 @@ static inline void dynamic_scs_init(void) static inline void dynamic_scs_init(void) {} #endif -int scs_patch(const u8 eh_frame[], int size); -asmlinkage void scs_patch_vmlinux(void); +int __pi_scs_patch(const u8 eh_frame[], int size); +asmlinkage void __pi_scs_patch_vmlinux(void); #endif /* __ASSEMBLY __ */ diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 4236f1e0fffa..14b4a179bad3 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -71,14 +71,6 @@ obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-y += vdso-wrap.o obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o -obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.o - -# We need to prevent the SCS patching code from patching itself. Using -# -mbranch-protection=none here to avoid the patchable PAC opcodes from being -# generated triggers an issue with full LTO on Clang, which stops emitting PAC -# instructions altogether. So disable LTO as well for the compilation unit. -CFLAGS_patch-scs.o += -mbranch-protection=none -CFLAGS_REMOVE_patch-scs.o += $(CC_FLAGS_LTO) # Force dependency (vdso*-wrap.S includes vdso.so through incbin) $(obj)/vdso-wrap.o: $(obj)/vdso/vdso.so diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 865ecc1f8255..b320702032a7 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -490,9 +490,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) #endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init -#endif -#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS - bl scs_patch_vmlinux #endif mov x0, x20 bl finalise_el2 // Prefer VHE if possible @@ -794,6 +791,11 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RELOCATABLE mov x0, x23 bl __pi_relocate_kernel +#endif +#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS + ldr x0, =__eh_frame_start + ldr x1, =__eh_frame_end + bl __pi_scs_patch_vmlinux #endif ldr x8, =__primary_switched adrp x0, KERNEL_START // __pa(KERNEL_START) diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c index dd851297596e..47e0be610bb6 100644 --- a/arch/arm64/kernel/module.c +++ b/arch/arm64/kernel/module.c @@ -595,7 +595,7 @@ int module_finalize(const Elf_Ehdr *hdr, if (scs_is_dynamic()) { s = find_section(hdr, sechdrs, ".init.eh_frame"); if (s) - scs_patch((void *)s->sh_addr, s->sh_size); + __pi_scs_patch((void *)s->sh_addr, s->sh_size); } return module_init_ftrace_plt(hdr, sechdrs, me); diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 7f6dfce893c3..a8b302245f15 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -38,7 +38,9 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := idreg-override.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o -obj-$(CONFIG_RELOCATABLE) += relocate.pi.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o -extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) +obj-y := idreg-override.pi.o \ + lib-fdt.pi.o lib-fdt_ro.pi.o +obj-$(CONFIG_RELOCATABLE) += relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o +obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.pi.o +extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/arm64/kernel/patch-scs.c b/arch/arm64/kernel/pi/patch-scs.c similarity index 91% rename from arch/arm64/kernel/patch-scs.c rename to arch/arm64/kernel/pi/patch-scs.c index a1fe4b4ff591..c65ef40d1e6b 100644 --- a/arch/arm64/kernel/patch-scs.c +++ b/arch/arm64/kernel/pi/patch-scs.c @@ -4,14 +4,11 @@ * Author: Ard Biesheuvel */ -#include #include #include #include -#include #include -#include #include // @@ -81,7 +78,11 @@ static void __always_inline scs_patch_loc(u64 loc) */ return; } - dcache_clean_pou(loc, loc + sizeof(u32)); + if (IS_ENABLED(CONFIG_ARM64_WORKAROUND_CLEAN_CACHE)) + asm("dc civac, %0" :: "r"(loc)); + else + asm(ALTERNATIVE("dc cvau, %0", "nop", ARM64_HAS_CACHE_IDC) + :: "r"(loc)); } /* @@ -128,10 +129,10 @@ struct eh_frame { }; }; -static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, - bool fde_has_augmentation_data, - int code_alignment_factor, - bool dry_run) +static int scs_handle_fde_frame(const struct eh_frame *frame, + bool fde_has_augmentation_data, + int code_alignment_factor, + bool dry_run) { int size = frame->size - offsetof(struct eh_frame, opcodes) + 4; u64 loc = (u64)offset_to_ptr(&frame->initial_loc); @@ -198,14 +199,13 @@ static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, break; default: - pr_err("unhandled opcode: %02x in FDE frame %lx\n", opcode[-1], (uintptr_t)frame); return -ENOEXEC; } } return 0; } -int noinstr scs_patch(const u8 eh_frame[], int size) +int scs_patch(const u8 eh_frame[], int size) { const u8 *p = eh_frame; @@ -251,12 +251,12 @@ int noinstr scs_patch(const u8 eh_frame[], int size) return 0; } -asmlinkage void __init scs_patch_vmlinux(void) +asmlinkage void __init scs_patch_vmlinux(const u8 start[], const u8 end[]) { if (!should_patch_pac_into_scs()) return; - WARN_ON(scs_patch(__eh_frame_start, __eh_frame_end - __eh_frame_start)); - icache_inval_all_pou(); + scs_patch(start, end - start); + asm("ic ialluis"); isb(); }