From patchwork Wed Jun 8 10:45:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12873235 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D005C43334 for ; Wed, 8 Jun 2022 10:45:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237221AbiFHKpa (ORCPT ); Wed, 8 Jun 2022 06:45:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237209AbiFHKp3 (ORCPT ); Wed, 8 Jun 2022 06:45:29 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D8E2DBBCC3; Wed, 8 Jun 2022 03:45:28 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B908E61336; Wed, 8 Jun 2022 10:45:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E6051C3411D; Wed, 8 Jun 2022 10:45:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1654685127; bh=ynyYyfkfJ6SlhYBng8HGwErHOBt9df5GswrHDhXe7mc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rZyVBfbNO/46P8aRTRJWgYHsaNKt85Lfx6yK2uR27gXHu31Hsn3u4GFCUnlDeWVFJ nZaIrl56r1KHtvyzYXtqNdP5aiUxIaP20hen8svrKQNH4fZsh1Vsmff3lkGwk3zICg /IeETw1talyz0JNE5ryOafk6k0AJpLHHDbZNRu/qj52H0c7JhdwyMUUJL0+aJHrLcr gqmslKQbarll87u7Djp4aeyWAAzj0gSNQe4RiCPygYX1GSwaCLToOIPV5WNjIjr93Q 9xdb4XDpllKG1JKwstPZV/0daWZhi7t1LCvvKB6v5oIQovx9b0ElUwb/kqlEnGA1+I C+2Vmd+e2khyA== From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Peter Zijlstra , Thomas Bogendoerfer , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , linux-mips@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 1/3] jump_label: s390: avoid pointless initial NOP patching Date: Wed, 8 Jun 2022 12:45:10 +0200 Message-Id: <20220608104512.1176209-2-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220608104512.1176209-1-ardb@kernel.org> References: <20220608104512.1176209-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3163; h=from:subject; bh=ynyYyfkfJ6SlhYBng8HGwErHOBt9df5GswrHDhXe7mc=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBioH2z8uiifCcodhcR3hXc8lJS1WnqKJGg+ZB6Jt0J yamDoA6JAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYqB9swAKCRDDTyI5ktmPJMkCDA CCL23njunOCQo0qjeRP1JcvaBTrPyHc2H6HENGAEBzzOHzFqRoVI4hgYHesRdEBIy1PDNlJBBcV/Ps Kwe2cplJuBN1U0Tqw1VfP9sXZ5Zi+OusuPFOw+XgNyUXGO/cr1YQxQ4YryJ0uV4f70911SpmDgLewO DLRt1BAKDj1kLju1sPtYfi8rMIivDdwOeLPg4UynXhsYmtjkjqe6ivUUK0AUK3D2hSrz7tIfLh1ECy QD7L77R1UmK2JycGjFL3n+rfHMDgY637t56gcVSuFb5sS+QdS2DueWLoE6Z1GWS9P/RS8zjfAmrLve 3DmwvGY7AxGcZQveq+j+OJTIqxGngjsEbgqRjxL8tOW0wxC2CwYxpQa1QiH9RKqbfxK8F3GyMl1S7e eryEQo+N8Y1UyTYP7icsLp0rqYLaWpoIePjZfiC52g6nDnh4XQkZllPwq8L/bsQeE5PW5TdcCBZwYV 4QG9A3HCYfxr9ZeJZ/F5t6PoWjqUMM76uWNVRBoZeyU0M= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-mips@vger.kernel.org Patching NOPs into other NOPs at boot time serves no purpose, so let's use the same NOP encodings at compile time and runtime. Signed-off-by: Ard Biesheuvel --- arch/s390/include/asm/jump_label.h | 5 ++--- arch/s390/kernel/jump_label.c | 23 +++++--------------- 2 files changed, 7 insertions(+), 21 deletions(-) diff --git a/arch/s390/include/asm/jump_label.h b/arch/s390/include/asm/jump_label.h index 916cfcb36d8a..895f774bbcc5 100644 --- a/arch/s390/include/asm/jump_label.h +++ b/arch/s390/include/asm/jump_label.h @@ -10,7 +10,6 @@ #include #define JUMP_LABEL_NOP_SIZE 6 -#define JUMP_LABEL_NOP_OFFSET 2 #ifdef CONFIG_CC_IS_CLANG #define JUMP_LABEL_STATIC_KEY_CONSTRAINT "i" @@ -21,12 +20,12 @@ #endif /* - * We use a brcl 0,2 instruction for jump labels at compile time so it + * We use a brcl 0, instruction for jump labels so it * can be easily distinguished from a hotpatch generated instruction. */ static __always_inline bool arch_static_branch(struct static_key *key, bool branch) { - asm_volatile_goto("0: brcl 0,"__stringify(JUMP_LABEL_NOP_OFFSET)"\n" + asm_volatile_goto("0: brcl 0,%l[label]\n" ".pushsection __jump_table,\"aw\"\n" ".balign 8\n" ".long 0b-.,%l[label]-.\n" diff --git a/arch/s390/kernel/jump_label.c b/arch/s390/kernel/jump_label.c index 6bec000c6c1c..d764f0d229ab 100644 --- a/arch/s390/kernel/jump_label.c +++ b/arch/s390/kernel/jump_label.c @@ -44,14 +44,8 @@ static void jump_label_bug(struct jump_entry *entry, struct insn *expected, panic("Corrupted kernel text"); } -static struct insn orignop = { - .opcode = 0xc004, - .offset = JUMP_LABEL_NOP_OFFSET >> 1, -}; - static void jump_label_transform(struct jump_entry *entry, - enum jump_label_type type, - int init) + enum jump_label_type type) { void *code = (void *)jump_entry_code(entry); struct insn old, new; @@ -63,27 +57,22 @@ static void jump_label_transform(struct jump_entry *entry, jump_label_make_branch(entry, &old); jump_label_make_nop(entry, &new); } - if (init) { - if (memcmp(code, &orignop, sizeof(orignop))) - jump_label_bug(entry, &orignop, &new); - } else { - if (memcmp(code, &old, sizeof(old))) - jump_label_bug(entry, &old, &new); - } + if (memcmp(code, &old, sizeof(old))) + jump_label_bug(entry, &old, &new); s390_kernel_write(code, &new, sizeof(new)); } void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type) { - jump_label_transform(entry, type, 0); + jump_label_transform(entry, type); text_poke_sync(); } bool arch_jump_label_transform_queue(struct jump_entry *entry, enum jump_label_type type) { - jump_label_transform(entry, type, 0); + jump_label_transform(entry, type); return true; } @@ -95,6 +84,4 @@ void arch_jump_label_transform_apply(void) void __init_or_module arch_jump_label_transform_static(struct jump_entry *entry, enum jump_label_type type) { - jump_label_transform(entry, type, 1); - text_poke_sync(); } From patchwork Wed Jun 8 10:45:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12873236 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0460CCCA47B for ; Wed, 8 Jun 2022 10:45:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237281AbiFHKpi (ORCPT ); Wed, 8 Jun 2022 06:45:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56486 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237233AbiFHKpe (ORCPT ); Wed, 8 Jun 2022 06:45:34 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9AE92D921C; Wed, 8 Jun 2022 03:45:32 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 25B67B82515; Wed, 8 Jun 2022 10:45:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 968E9C341C0; Wed, 8 Jun 2022 10:45:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1654685129; bh=c7CfPX19eVzkri+WoXphSC63kVD98IclMqfOQOsGVQU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Gp5kNjbpdizeodqHjqDwuOqmbpDqX/xM7zihgD8KK/CepnWCIDTuQvr6oZsMFMmEM x2SJ98AOT/2YFYGBGcb5GZ0DRp6//7O46Rvvruy9ffnUKV27XSUCRayD+6Qfn0zQ01 NmisZeppZtMmT9hHcJRf+qMM2Z62AWC2w7qYS84tKIYZRPbmExtCq0/jjcEAAUxDZ3 Gm+tDm7QOUW6HsORHyaOfAh+2iHP0MVchHp6wOPKcYGmyyEZHBYuIqUPs9WaKmePmt GEBSmjX/cEJVlZtOtk2nNcg28F+/UsDz45x3bl8ioSP8IPnkgyAT6wda4ktpqxZMJf kOEErzObRfYMA== From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Peter Zijlstra , Thomas Bogendoerfer , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , linux-mips@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 2/3] jump_label: mips: move module NOP patching into arch code Date: Wed, 8 Jun 2022 12:45:11 +0200 Message-Id: <20220608104512.1176209-3-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220608104512.1176209-1-ardb@kernel.org> References: <20220608104512.1176209-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6445; h=from:subject; bh=c7CfPX19eVzkri+WoXphSC63kVD98IclMqfOQOsGVQU=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBioH21qhfGEwZDqn2RBezVRa70NovaR4URqFttnuPf x+3Wx46JAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYqB9tQAKCRDDTyI5ktmPJGfIDA DCfwL+kpiWqH9Svu3esUbTPpqeLkIloRhP+PLurJECH+Y6CRoSnUT0BZTHjqXpyxWQ/y+C0NLqbYj7 xrJLjtRj2WDdr6+pn9CJVw7Wm9bHB3r+u89BMCwRNpoSJhPQfRM9Qx8UFHQ7IEb0PJ9ANE7U5BtExL Z6ffex8k7uqKCblp3NhuRgmts7toMQy2XXtX9He6mEMNeh0csNFvKjyHBo6F0KrG5FfB1mxzLpeny4 AsZY6moIlFp404CqkXHs4bIcUxWta8LtEfy1YbJsOCpdlGyOslyxleYvuV9ABlxlBFvf59DAVXsAiP GLbccLXhr00CBqT8QKAmXiaNg4NyK29fxtdg7IxWnhzCf/FE5naksCkCGtugnAXbcudQxsZBxPVNXm OP2vgjgy4BoNl90zeaVtN7OkPRnBCLHieDr81iUO89XZu+/JSJ0E5vCdOlZVHFnAGEEjHNaZlGWuCa LgW4kgRyaG3ELVL6zK0hgKsRjIgNihalTMD2fX7nJ4GqM= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-mips@vger.kernel.org MIPS is the only remaining architecture that needs to patch jump label NOP encodings to initialize them at load time. So let's move the module patching part of that from generic code into arch/mips, and drop it from the others. Signed-off-by: Ard Biesheuvel --- arch/mips/kernel/jump_label.c | 17 ++++++++++++ arch/mips/kernel/module.c | 5 ++-- arch/s390/kernel/module.c | 1 - arch/sparc/kernel/module.c | 3 --- arch/x86/kernel/module.c | 3 --- include/linux/jump_label.h | 7 +---- kernel/jump_label.c | 27 +------------------- 7 files changed, 22 insertions(+), 41 deletions(-) diff --git a/arch/mips/kernel/jump_label.c b/arch/mips/kernel/jump_label.c index 662c8db9f45b..0f98e7f1461f 100644 --- a/arch/mips/kernel/jump_label.c +++ b/arch/mips/kernel/jump_label.c @@ -88,3 +88,20 @@ void arch_jump_label_transform(struct jump_entry *e, mutex_unlock(&text_mutex); } + +void jump_label_apply_nops(struct module *mod) +{ + struct jump_entry *iter_start = mod->jump_entries; + struct jump_entry *iter_stop = iter_start + mod->num_jump_entries; + struct jump_entry *iter; + + /* if the module doesn't have jump label entries, just return */ + if (iter_start == iter_stop) + return; + + for (iter = iter_start; iter < iter_stop; iter++) { + /* Only write NOPs for arch_branch_static(). */ + if (jump_label_init_type(iter) == JUMP_LABEL_NOP) + arch_jump_label_transform(iter, JUMP_LABEL_NOP); + } +} diff --git a/arch/mips/kernel/module.c b/arch/mips/kernel/module.c index 14f46d17500a..0c936cbf20c5 100644 --- a/arch/mips/kernel/module.c +++ b/arch/mips/kernel/module.c @@ -21,6 +21,7 @@ #include #include +extern void jump_label_apply_nops(struct module *mod); struct mips_hi16 { struct mips_hi16 *next; @@ -428,8 +429,8 @@ int module_finalize(const Elf_Ehdr *hdr, const Elf_Shdr *s; char *secstrings = (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset; - /* Make jump label nops. */ - jump_label_apply_nops(me); + if (IS_ENABLED(CONFIG_JUMP_LABEL)) + jump_label_apply_nops(me); INIT_LIST_HEAD(&me->arch.dbe_list); for (s = sechdrs; s < sechdrs + hdr->e_shnum; s++) { diff --git a/arch/s390/kernel/module.c b/arch/s390/kernel/module.c index 26125a9c436d..2d159b32885b 100644 --- a/arch/s390/kernel/module.c +++ b/arch/s390/kernel/module.c @@ -548,6 +548,5 @@ int module_finalize(const Elf_Ehdr *hdr, #endif /* CONFIG_FUNCTION_TRACER */ } - jump_label_apply_nops(me); return 0; } diff --git a/arch/sparc/kernel/module.c b/arch/sparc/kernel/module.c index df39580f398d..66c45a2764bc 100644 --- a/arch/sparc/kernel/module.c +++ b/arch/sparc/kernel/module.c @@ -208,9 +208,6 @@ int module_finalize(const Elf_Ehdr *hdr, const Elf_Shdr *sechdrs, struct module *me) { - /* make jump label nops */ - jump_label_apply_nops(me); - do_patch_sections(hdr, sechdrs); /* Cheetah's I-cache is fully coherent. */ diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c index b98ffcf4d250..95b9cf25d4bd 100644 --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -304,9 +304,6 @@ int module_finalize(const Elf_Ehdr *hdr, tseg, tseg + text->sh_size); } - /* make jump label nops */ - jump_label_apply_nops(me); - if (orc && orc_ip) unwind_module_init(me, (void *)orc_ip->sh_addr, orc_ip->sh_size, (void *)orc->sh_addr, orc->sh_size); diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index bf1eef337a07..2003a0935478 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -230,12 +230,12 @@ extern void static_key_slow_inc(struct static_key *key); extern void static_key_slow_dec(struct static_key *key); extern void static_key_slow_inc_cpuslocked(struct static_key *key); extern void static_key_slow_dec_cpuslocked(struct static_key *key); -extern void jump_label_apply_nops(struct module *mod); extern int static_key_count(struct static_key *key); extern void static_key_enable(struct static_key *key); extern void static_key_disable(struct static_key *key); extern void static_key_enable_cpuslocked(struct static_key *key); extern void static_key_disable_cpuslocked(struct static_key *key); +extern enum jump_label_type jump_label_init_type(struct jump_entry *entry); /* * We should be using ATOMIC_INIT() for initializing .enabled, but @@ -303,11 +303,6 @@ static inline int jump_label_text_reserved(void *start, void *end) static inline void jump_label_lock(void) {} static inline void jump_label_unlock(void) {} -static inline int jump_label_apply_nops(struct module *mod) -{ - return 0; -} - static inline void static_key_enable(struct static_key *key) { STATIC_KEY_CHECK_USE(key); diff --git a/kernel/jump_label.c b/kernel/jump_label.c index b156e152d6b4..b1ac2948be79 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -508,7 +508,7 @@ void __init jump_label_init(void) #ifdef CONFIG_MODULES -static enum jump_label_type jump_label_init_type(struct jump_entry *entry) +enum jump_label_type jump_label_init_type(struct jump_entry *entry) { struct static_key *key = jump_entry_key(entry); bool type = static_key_type(key); @@ -596,31 +596,6 @@ static void __jump_label_mod_update(struct static_key *key) } } -/*** - * apply_jump_label_nops - patch module jump labels with arch_get_jump_label_nop() - * @mod: module to patch - * - * Allow for run-time selection of the optimal nops. Before the module - * loads patch these with arch_get_jump_label_nop(), which is specified by - * the arch specific jump label code. - */ -void jump_label_apply_nops(struct module *mod) -{ - struct jump_entry *iter_start = mod->jump_entries; - struct jump_entry *iter_stop = iter_start + mod->num_jump_entries; - struct jump_entry *iter; - - /* if the module doesn't have jump label entries, just return */ - if (iter_start == iter_stop) - return; - - for (iter = iter_start; iter < iter_stop; iter++) { - /* Only write NOPs for arch_branch_static(). */ - if (jump_label_init_type(iter) == JUMP_LABEL_NOP) - arch_jump_label_transform_static(iter, JUMP_LABEL_NOP); - } -} - static int jump_label_add_module(struct module *mod) { struct jump_entry *iter_start = mod->jump_entries; From patchwork Wed Jun 8 10:45:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12873237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52B1EC433EF for ; Wed, 8 Jun 2022 10:45:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237291AbiFHKpj (ORCPT ); Wed, 8 Jun 2022 06:45:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56656 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237266AbiFHKph (ORCPT ); Wed, 8 Jun 2022 06:45:37 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 534072AE11; Wed, 8 Jun 2022 03:45:35 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id B3C80B826B8; Wed, 8 Jun 2022 10:45:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 487FCC341C6; Wed, 8 Jun 2022 10:45:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1654685132; bh=IUkks1rKSJ/NdE5ptUYvV5ETNPKdfPh+xYmfyDN6j5A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Y3DH+BPm08KVSut05Senfp4vB0Ejiz90/0DzI4vHCNQF16R43bm9dX0GsLcK8wSpK plrR1TLzk5K/Qat9LG5AiN84lz9yBdURg1nRZ5wniBrPOMmDXCft6VEwf2ePW0UaO/ FVOeSu0qQBF1TU/ywvAWyjf2kUVucSdIMPerJSs75FMqBbnmcsEaKVPHtgYZf5QOWf cV/jySGcmUoNAsA2qd+ohhIyMDuAMABCr2rMK9SexxphliKJ4L+/URsW6eQ0qI7yLG DVqLtbIGZcSO2dOy3he82ehh6trVRaOjMN0KpC3PuYXvxSPp7rojL7Qb7P1bVGkRIi 4mr+zRf+Pg+uw== From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Peter Zijlstra , Thomas Bogendoerfer , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , linux-mips@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 3/3] jump_label: make initial NOP patching the special case Date: Wed, 8 Jun 2022 12:45:12 +0200 Message-Id: <20220608104512.1176209-4-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220608104512.1176209-1-ardb@kernel.org> References: <20220608104512.1176209-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=7711; h=from:subject; bh=IUkks1rKSJ/NdE5ptUYvV5ETNPKdfPh+xYmfyDN6j5A=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBioH23z/sItwB00nGMha5Ha+zYBS5GkKeFiVirkyRz Z6sfHB+JAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYqB9twAKCRDDTyI5ktmPJEPOC/ 90LIbIy+/gbgiX0AcTzY9J5qxBHb2tBJg57qdy0vZgq6YXW9lk1TbqhSB5oRSWenMgxFwgSV4pp/VS F0X8MC5aaIZMfaJb7AMP9o/n6MUHqaMI3Y/x9DbuK1LedWYrAU0pEgmA3Cp/Sh2zDl0lM3c7JJmxT5 iig2vO3AVMNxDkCpOIkvv49syUB4oNyRJdaHM/eBkkI4xuj4IFFEjrJ/wzHasavLpGBnIl97rpw/oN bEkHxz7TBWVXruTXcgcW4JwVFbrUGqVt5Zj5lKuuIr7YUf80r+6wz4xAj13Vxu94ierpwqVaze07vR rDcCaJo0SjwDVg/Sgu36X4sM8oSreMmHS/oT+CY3c6gtrIaVAto8wSwRBp/9goBEhst6To1wIkwwAy pjpVxGND5LfUI1EKLKlsAsogQ+mxaqTNn2Wn1tAT+9RGJZphb6BGs2wYj90u2tQECZlhnLUYjLrfh7 asC8ZlLoRWywY2bTprQg0YXwCam51umh4rvV5XP7SLUg4= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-mips@vger.kernel.org Instead of defaulting to patching NOP opcodes at init time, and leaving it to the architectures to override this if this is not needed, switch to a model where doing nothing is the default. This is the common case by far, as only MIPS requires NOP patching at init time. On all other architectures, the correct encodings are emitted by the compiler and so no initial patching is needed. Signed-off-by: Ard Biesheuvel Acked-by: Peter Zijlstra (Intel) --- Documentation/staging/static-keys.rst | 3 --- arch/arc/kernel/jump_label.c | 13 ------------- arch/arm/kernel/jump_label.c | 6 ------ arch/arm64/kernel/jump_label.c | 11 ----------- arch/mips/include/asm/jump_label.h | 2 ++ arch/parisc/kernel/jump_label.c | 11 ----------- arch/riscv/kernel/jump_label.c | 12 ------------ arch/s390/kernel/jump_label.c | 5 ----- arch/x86/kernel/jump_label.c | 13 ------------- kernel/jump_label.c | 14 +++----------- 10 files changed, 5 insertions(+), 85 deletions(-) diff --git a/Documentation/staging/static-keys.rst b/Documentation/staging/static-keys.rst index 38290b9f25eb..b0a519f456cf 100644 --- a/Documentation/staging/static-keys.rst +++ b/Documentation/staging/static-keys.rst @@ -201,9 +201,6 @@ static_key->entry field makes use of the two least significant bits. * ``void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type)``, see: arch/x86/kernel/jump_label.c -* ``__init_or_module void arch_jump_label_transform_static(struct jump_entry *entry, enum jump_label_type type)``, - see: arch/x86/kernel/jump_label.c - * ``struct jump_entry``, see: arch/x86/include/asm/jump_label.h diff --git a/arch/arc/kernel/jump_label.c b/arch/arc/kernel/jump_label.c index b8600dc325b5..70b74a5d047b 100644 --- a/arch/arc/kernel/jump_label.c +++ b/arch/arc/kernel/jump_label.c @@ -96,19 +96,6 @@ void arch_jump_label_transform(struct jump_entry *entry, flush_icache_range(entry->code, entry->code + JUMP_LABEL_NOP_SIZE); } -void arch_jump_label_transform_static(struct jump_entry *entry, - enum jump_label_type type) -{ - /* - * We use only one NOP type (1x, 4 byte) in arch_static_branch, so - * there's no need to patch an identical NOP over the top of it here. - * The generic code calls 'arch_jump_label_transform' if the NOP needs - * to be replaced by a branch, so 'arch_jump_label_transform_static' is - * never called with type other than JUMP_LABEL_NOP. - */ - BUG_ON(type != JUMP_LABEL_NOP); -} - #ifdef CONFIG_ARC_DBG_JUMP_LABEL #define SELFTEST_MSG "ARC: instruction generation self-test: " diff --git a/arch/arm/kernel/jump_label.c b/arch/arm/kernel/jump_label.c index 303b3ab87f7e..eb9c24b6e8e2 100644 --- a/arch/arm/kernel/jump_label.c +++ b/arch/arm/kernel/jump_label.c @@ -27,9 +27,3 @@ void arch_jump_label_transform(struct jump_entry *entry, { __arch_jump_label_transform(entry, type, false); } - -void arch_jump_label_transform_static(struct jump_entry *entry, - enum jump_label_type type) -{ - __arch_jump_label_transform(entry, type, true); -} diff --git a/arch/arm64/kernel/jump_label.c b/arch/arm64/kernel/jump_label.c index fc98037e1220..faf88ec9c48e 100644 --- a/arch/arm64/kernel/jump_label.c +++ b/arch/arm64/kernel/jump_label.c @@ -26,14 +26,3 @@ void arch_jump_label_transform(struct jump_entry *entry, aarch64_insn_patch_text_nosync(addr, insn); } - -void arch_jump_label_transform_static(struct jump_entry *entry, - enum jump_label_type type) -{ - /* - * We use the architected A64 NOP in arch_static_branch, so there's no - * need to patch an identical A64 NOP over the top of it here. The core - * will call arch_jump_label_transform from a module notifier if the - * NOP needs to be replaced by a branch. - */ -} diff --git a/arch/mips/include/asm/jump_label.h b/arch/mips/include/asm/jump_label.h index 3185fd3220ec..c5c6864e64bc 100644 --- a/arch/mips/include/asm/jump_label.h +++ b/arch/mips/include/asm/jump_label.h @@ -8,6 +8,8 @@ #ifndef _ASM_MIPS_JUMP_LABEL_H #define _ASM_MIPS_JUMP_LABEL_H +#define arch_jump_label_transform_static arch_jump_label_transform + #ifndef __ASSEMBLY__ #include diff --git a/arch/parisc/kernel/jump_label.c b/arch/parisc/kernel/jump_label.c index d2f3cb12e282..e253b134500d 100644 --- a/arch/parisc/kernel/jump_label.c +++ b/arch/parisc/kernel/jump_label.c @@ -42,14 +42,3 @@ void arch_jump_label_transform(struct jump_entry *entry, patch_text(addr, insn); } - -void arch_jump_label_transform_static(struct jump_entry *entry, - enum jump_label_type type) -{ - /* - * We use the architected NOP in arch_static_branch, so there's no - * need to patch an identical NOP over the top of it here. The core - * will call arch_jump_label_transform from a module notifier if the - * NOP needs to be replaced by a branch. - */ -} diff --git a/arch/riscv/kernel/jump_label.c b/arch/riscv/kernel/jump_label.c index 20e09056d141..e6694759dbd0 100644 --- a/arch/riscv/kernel/jump_label.c +++ b/arch/riscv/kernel/jump_label.c @@ -39,15 +39,3 @@ void arch_jump_label_transform(struct jump_entry *entry, patch_text_nosync(addr, &insn, sizeof(insn)); mutex_unlock(&text_mutex); } - -void arch_jump_label_transform_static(struct jump_entry *entry, - enum jump_label_type type) -{ - /* - * We use the same instructions in the arch_static_branch and - * arch_static_branch_jump inline functions, so there's no - * need to patch them up here. - * The core will call arch_jump_label_transform when those - * instructions need to be replaced. - */ -} diff --git a/arch/s390/kernel/jump_label.c b/arch/s390/kernel/jump_label.c index d764f0d229ab..e808bb8bc0da 100644 --- a/arch/s390/kernel/jump_label.c +++ b/arch/s390/kernel/jump_label.c @@ -80,8 +80,3 @@ void arch_jump_label_transform_apply(void) { text_poke_sync(); } - -void __init_or_module arch_jump_label_transform_static(struct jump_entry *entry, - enum jump_label_type type) -{ -} diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c index 68f091ba8443..f5b8ef02d172 100644 --- a/arch/x86/kernel/jump_label.c +++ b/arch/x86/kernel/jump_label.c @@ -146,16 +146,3 @@ void arch_jump_label_transform_apply(void) text_poke_finish(); mutex_unlock(&text_mutex); } - -static enum { - JL_STATE_START, - JL_STATE_NO_UPDATE, - JL_STATE_UPDATE, -} jlstate __initdata_or_module = JL_STATE_START; - -__init_or_module void arch_jump_label_transform_static(struct jump_entry *entry, - enum jump_label_type type) -{ - if (jlstate == JL_STATE_UPDATE) - jump_label_transform(entry, type, 1); -} diff --git a/kernel/jump_label.c b/kernel/jump_label.c index b1ac2948be79..ff8576c00893 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -332,17 +332,9 @@ static int __jump_label_text_reserved(struct jump_entry *iter_start, return 0; } -/* - * Update code which is definitely not currently executing. - * Architectures which need heavyweight synchronization to modify - * running code can override this to make the non-live update case - * cheaper. - */ -void __weak __init_or_module arch_jump_label_transform_static(struct jump_entry *entry, - enum jump_label_type type) -{ - arch_jump_label_transform(entry, type); -} +#ifndef arch_jump_label_transform_static +#define arch_jump_label_transform_static(entry, type) +#endif static inline struct jump_entry *static_key_entries(struct static_key *key) {