From patchwork Mon Jan 9 14:58:27 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luis Chamberlain X-Patchwork-Id: 9505035 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id E43C760710 for ; Mon, 9 Jan 2017 15:02:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D565827CEA for ; Mon, 9 Jan 2017 15:02:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C8CB42836D; Mon, 9 Jan 2017 15:02:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id B314C27CEA for ; Mon, 9 Jan 2017 15:02:46 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cQbQB-00081L-O8; Mon, 09 Jan 2017 14:59:39 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cQbQA-0007zl-9T for xen-devel@lists.xensource.com; Mon, 09 Jan 2017 14:59:38 +0000 Received: from [85.158.143.35] by server-10.bemta-6.messagelabs.com id DE/24-13192-955A3785; Mon, 09 Jan 2017 14:59:37 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFupgkeJIrShJLcpLzFFi42I5NlG2QzdiaXG EwaZGWYt7U96zOzB6bO/bxR7AGMWamZeUX5HAmnHu00XmguldjBULz31gb2CcXNLFyMUhJDCV UWLWo0nsEM4yJok1Dz8zdzFycLAJ6ErcvC0BEhcR+Mck8W5SAyuIwyzwhlvi7KobQB2cHMICH hILrj8Bs1kEVCWa7n0Hs3kFzCWurFvADGJLCMhL7Gq7yApicwLF519YC2YLCSRJnFt9gR1kmY RAgcT+w9EQ5V4Si25cYoWw1SSuntvEPIGRbwEjwypGjeLUorLUIl1DI72kosz0jJLcxMwcXUM DM73c1OLixPTUnMSkYr3k/NxNjMBQYQCCHYyXNwYcYpTkYFIS5XWMKogQ4kvKT6nMSCzOiC8q zUktPsQow8GhJME7dXFxhJBgUWp6akVaZg4waGHSEhw8SiK8p0DSvMUFibnFmekQqVOMuhwrb qx7yiTEkpeflyolznsEpEgApCijNA9uBCyCLjHKSgnzMgIdJcRTkFqUm1mCKv+KUZyDUUmY9y zIFJ7MvBK4Ta+AjmACOiLSDuyIkkSElFQDI+e6BImMdffsig/7zAqf9sS0leH5+mnVDjrbPTu vC5cZui3NKljXldwQoubuxtxz0T/D7iXH5M9Bc17++7L2smNj7K//tr6Va3nnt9SfNfgdezKp waJSZeufhy85LlpskXayORa288Evn5IEiR93PXytJnzuOcvEuOTWzIVrZxzpV+RZH7X7rxJLc UaioRZzUXEiAHrDJ/abAgAA X-Env-Sender: mcgrof@kernel.org X-Msg-Ref: server-4.tower-21.messagelabs.com!1483973974!45632945!1 X-Originating-IP: [198.145.29.136] X-SpamReason: No, hits=0.5 required=7.0 tests=BODY_RANDOM_LONG X-StarScan-Received: X-StarScan-Version: 9.1.1; banners=-,-,- X-VirusChecked: Checked Received: (qmail 52114 invoked from network); 9 Jan 2017 14:59:36 -0000 Received: from mail.kernel.org (HELO mail.kernel.org) (198.145.29.136) by server-4.tower-21.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 9 Jan 2017 14:59:36 -0000 Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id E5E012037F; Mon, 9 Jan 2017 14:59:24 +0000 (UTC) Received: from garbanzo.do-not-panic.com (c-73-15-241-2.hsd1.ca.comcast.net [73.15.241.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id BA7C5203A1; Mon, 9 Jan 2017 14:59:20 +0000 (UTC) From: "Luis R. Rodriguez" To: hpa@zytor.com, acme@redhat.com, tglx@linutronix.de, mingo@redhat.com, jpoimboe@redhat.com, bp@alien8.de, npiggin@gmail.com, mhiramat@kernel.org, masami.hiramatsu.pt@hitachi.com, jbaron@akamai.com, heiko.carstens@de.ibm.com, ananth@linux.vnet.ibm.com, anil.s.keshavamurthy@intel.com, davem@davemloft.net, realmz6@gmail.com, dalias@libc.org, linux@arm.linux.org.uk Date: Mon, 9 Jan 2017 06:58:27 -0800 Message-Id: <20170109145833.11502-10-mcgrof@kernel.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170109145833.11502-1-mcgrof@kernel.org> References: <20161222023811.21246-1-mcgrof@kernel.org> <20170109145833.11502-1-mcgrof@kernel.org> X-Virus-Scanned: ClamAV using ClamSMTP Cc: gnomes@lxorguk.ukuu.org.uk, linux-ia64@vger.kernel.org, jkosina@suse.cz, benh@kernel.crashing.org, ming.lei@canonical.com, linux@rasmusvillemoes.dk, platform-driver-x86@vger.kernel.org, James.Bottomley@HansenPartnership.com, paul.gortmaker@windriver.com, paulus@samba.org, sparclinux@vger.kernel.org, wangnan0@huawei.com, mchehab@osg.samsung.com, linux-arch@vger.kernel.org, markus.heiser@darmarit.de, xen-devel@lists.xensource.com, msalter@redhat.com, linux-sh@vger.kernel.org, mpe@ellerman.id.au, joro@8bytes.org, x86@kernel.org, fontana@sharpeleven.org, dsahern@gmail.com, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, dvhart@infradead.org, dwmw2@infradead.org, linux@roeck-us.net, linux-xtensa@linux-xtensa.org, pali.rohar@gmail.com, keescook@chromium.org, arnd@arndb.de, jani.nikula@intel.com, will.deacon@arm.com, rusty@rustcorp.com.au, rostedt@goodmis.org, ak@linux.intel.com, namhyung@kernel.org, andriy.shevchenko@linux.intel.com, mcb30@ipxe.org, linux-kbuild@vger.kernel.org, alan@linux.intel.com, jgross@suse.com, chris@zankel.net, pebolle@tiscali.nl, tony.luck@intel.com, jcmvbkbc@gmail.com, gregkh@linuxfoundation.org, dmitry.torokhov@gmail.com, adrian.hunter@intel.com, luto@amacapital.net, "Luis R. Rodriguez" , mmarek@suse.com, david.vrabel@citrix.com, jolsa@kernel.org, andrew.cooper3@citrix.com, akpm@linux-foundation.org, torvalds@linux-foundation.org, korea.drzix@gmail.com Subject: [Xen-devel] [PATCH v6 09/14] jump_label: port __jump_table to linker tables X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Move the __jump_table from the a custom section solution to a generic solution, this avoiding extra vmlinux.lds.h customizations. This also demos the use of the .data linker table and of the shared asm call push_section_tbl(). Built-in kernel functionality was tested with CONFIG_STATIC_KEYS_SELFTEST. Moduler kernel functionality was tested with CONFIG_TEST_STATIC_KEYS. Both work as expected. Since __jump_table sections are also supported per module this also required expanding module-common.lds.S to capture and fold all .data.tlb.__jump_table.* onto the the section __jump_table -- in this case for modules need to keep a reference in place, given the alternative is to use DEFINE_LINKTABLE(struct jump_entry, __jump_table) per module -- and later through macro hacks instantiate the jump entries per module upon init. This is doable but we'd loose out on the sorting of the table using the linker, to sort we'd always still need to expand the module common linker script. An alternative mechanism is possible which would make these custom module sections extensions dynamic without requiring manual changes, this however is best done later through a separate evolution once linker tables are in place. A careful reviewer may note that some architectures use "\n\t" to separate asm code, while others just use a new line. Upon review last time it was deemed reasonable to for all architectures to just use "\n", this is defined as ASM_CMD_SEP, and if an architecture needs to override they can do so on their architecture sections.h prior to including asm-generic/sections.h v6: rename table macro as suggested by Andy Shevchenko v5: o Use ..tbl instead of .tbl as suggested by Nicholas Piggin. This is the typical way to avoid clash with compiler generated section. o Replace section macros with section names o Use LINKTABLE_START() and LINKTABLE_END() o fixed tile jump label port -- tile got jump label support as of commit 65a792e84f25d1 ("tile/jump_label: add jump label support for TILE-Gx"), as such we just needed to adjust the asm to account for the new linker table API use. This commit was merged as of v4.5-rc1. v4: o Some architectures allow linker scripts to follow including header files, some others do not, so if you need a helper on a linker script you need to explicitly include it. So for instance although scripts/module-common.lds.S includes and this file includes , you still need to explicitly include it on the linker script. This issue is present on ARM. o as per Josh Poimboeuf open code the section table name instead of including the kernel section headers, the simplicity and independence from the kernel is preferred. v3: o More elaborate tests performed o first modular support use case, module tested was CONFIG_TEST_STATIC_KEYS (lib/test_static_keys.ko), this required us to extend module-common.lds.S o use generic push_section_tbl_any() for all architectures o Makes use of ASM_CMD_SEP to enable architectures to override later if needed o guard tables.h inclusion and table definition with __KERNEL__ v2: introduced in this series Signed-off-by: Luis R. Rodriguez --- arch/arm/include/asm/jump_label.h | 6 ++++-- arch/arm64/include/asm/jump_label.h | 6 ++++-- arch/mips/include/asm/jump_label.h | 6 ++++-- arch/powerpc/include/asm/jump_label.h | 8 +++++--- arch/s390/include/asm/jump_label.h | 6 ++++-- arch/sparc/include/asm/jump_label.h | 6 ++++-- arch/tile/include/asm/jump_label.h | 5 +++-- arch/x86/include/asm/jump_label.h | 10 ++++++---- include/asm-generic/vmlinux.lds.h | 5 ----- include/linux/jump_label.h | 4 ++-- kernel/jump_label.c | 17 ++++++++++------- scripts/module-common.lds | 1 + tools/objtool/special.c | 2 +- 13 files changed, 48 insertions(+), 34 deletions(-) diff --git a/arch/arm/include/asm/jump_label.h b/arch/arm/include/asm/jump_label.h index 34f7b6980d21..609e0592a942 100644 --- a/arch/arm/include/asm/jump_label.h +++ b/arch/arm/include/asm/jump_label.h @@ -1,6 +1,8 @@ #ifndef _ASM_ARM_JUMP_LABEL_H #define _ASM_ARM_JUMP_LABEL_H +#include + #ifndef __ASSEMBLY__ #include @@ -12,7 +14,7 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran { asm_volatile_goto("1:\n\t" WASM(nop) "\n\t" - ".pushsection __jump_table, \"aw\"\n\t" + push_section_tbl_any(.data, __jump_table, aw) ".word 1b, %l[l_yes], %c0\n\t" ".popsection\n\t" : : "i" (&((char *)key)[branch]) : : l_yes); @@ -26,7 +28,7 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool { asm_volatile_goto("1:\n\t" WASM(b) " %l[l_yes]\n\t" - ".pushsection __jump_table, \"aw\"\n\t" + push_section_tbl_any(.data, __jump_table, aw) ".word 1b, %l[l_yes], %c0\n\t" ".popsection\n\t" : : "i" (&((char *)key)[branch]) : : l_yes); diff --git a/arch/arm64/include/asm/jump_label.h b/arch/arm64/include/asm/jump_label.h index 1b5e0e843c3a..bb56ac4fafb5 100644 --- a/arch/arm64/include/asm/jump_label.h +++ b/arch/arm64/include/asm/jump_label.h @@ -19,6 +19,8 @@ #ifndef __ASM_JUMP_LABEL_H #define __ASM_JUMP_LABEL_H +#include + #ifndef __ASSEMBLY__ #include @@ -29,7 +31,7 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool branch) { asm goto("1: nop\n\t" - ".pushsection __jump_table, \"aw\"\n\t" + push_section_tbl_any(.data, __jump_table, aw) ".align 3\n\t" ".quad 1b, %l[l_yes], %c0\n\t" ".popsection\n\t" @@ -43,7 +45,7 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran static __always_inline bool arch_static_branch_jump(struct static_key *key, bool branch) { asm goto("1: b %l[l_yes]\n\t" - ".pushsection __jump_table, \"aw\"\n\t" + push_section_tbl_any(.data, __jump_table, aw) ".align 3\n\t" ".quad 1b, %l[l_yes], %c0\n\t" ".popsection\n\t" diff --git a/arch/mips/include/asm/jump_label.h b/arch/mips/include/asm/jump_label.h index e77672539e8e..07289a1b1bcd 100644 --- a/arch/mips/include/asm/jump_label.h +++ b/arch/mips/include/asm/jump_label.h @@ -8,6 +8,8 @@ #ifndef _ASM_MIPS_JUMP_LABEL_H #define _ASM_MIPS_JUMP_LABEL_H +#include + #ifndef __ASSEMBLY__ #include @@ -30,7 +32,7 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran { asm_volatile_goto("1:\t" NOP_INSN "\n\t" "nop\n\t" - ".pushsection __jump_table, \"aw\"\n\t" + push_section_tbl_any(.data, __jump_table, aw) WORD_INSN " 1b, %l[l_yes], %0\n\t" ".popsection\n\t" : : "i" (&((char *)key)[branch]) : : l_yes); @@ -44,7 +46,7 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool { asm_volatile_goto("1:\tj %l[l_yes]\n\t" "nop\n\t" - ".pushsection __jump_table, \"aw\"\n\t" + push_section_tbl_any(.data, __jump_table, aw) WORD_INSN " 1b, %l[l_yes], %0\n\t" ".popsection\n\t" : : "i" (&((char *)key)[branch]) : : l_yes); diff --git a/arch/powerpc/include/asm/jump_label.h b/arch/powerpc/include/asm/jump_label.h index 9a287e0ac8b1..7a75623740d0 100644 --- a/arch/powerpc/include/asm/jump_label.h +++ b/arch/powerpc/include/asm/jump_label.h @@ -10,6 +10,8 @@ * 2 of the License, or (at your option) any later version. */ +#include + #ifndef __ASSEMBLY__ #include @@ -23,7 +25,7 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran { asm_volatile_goto("1:\n\t" "nop # arch_static_branch\n\t" - ".pushsection __jump_table, \"aw\"\n\t" + push_section_tbl_any(.data, __jump_table, aw) JUMP_ENTRY_TYPE "1b, %l[l_yes], %c0\n\t" ".popsection \n\t" : : "i" (&((char *)key)[branch]) : : l_yes); @@ -37,7 +39,7 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool { asm_volatile_goto("1:\n\t" "b %l[l_yes] # arch_static_branch_jump\n\t" - ".pushsection __jump_table, \"aw\"\n\t" + push_section_tbl_any(.data, __jump_table, aw) JUMP_ENTRY_TYPE "1b, %l[l_yes], %c0\n\t" ".popsection \n\t" : : "i" (&((char *)key)[branch]) : : l_yes); @@ -62,7 +64,7 @@ struct jump_entry { #else #define ARCH_STATIC_BRANCH(LABEL, KEY) \ 1098: nop; \ - .pushsection __jump_table, "aw"; \ + push_section_tbl_any(.data, __jump_table, aw); \ FTR_ENTRY_LONG 1098b, LABEL, KEY; \ .popsection #endif diff --git a/arch/s390/include/asm/jump_label.h b/arch/s390/include/asm/jump_label.h index 9be198f5ee79..58e53af49e7e 100644 --- a/arch/s390/include/asm/jump_label.h +++ b/arch/s390/include/asm/jump_label.h @@ -1,6 +1,8 @@ #ifndef _ASM_S390_JUMP_LABEL_H #define _ASM_S390_JUMP_LABEL_H +#include + #ifndef __ASSEMBLY__ #include @@ -16,7 +18,7 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool branch) { asm_volatile_goto("0: brcl 0,"__stringify(JUMP_LABEL_NOP_OFFSET)"\n" - ".pushsection __jump_table, \"aw\"\n" + push_section_tbl_any(.data, __jump_table, aw) ".balign 8\n" ".quad 0b, %l[label], %0\n" ".popsection\n" @@ -30,7 +32,7 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran static __always_inline bool arch_static_branch_jump(struct static_key *key, bool branch) { asm_volatile_goto("0: brcl 15, %l[label]\n" - ".pushsection __jump_table, \"aw\"\n" + push_section_tbl_any(.data, __jump_table, aw) ".balign 8\n" ".quad 0b, %l[label], %0\n" ".popsection\n" diff --git a/arch/sparc/include/asm/jump_label.h b/arch/sparc/include/asm/jump_label.h index 62d0354d1727..35d3e5d31821 100644 --- a/arch/sparc/include/asm/jump_label.h +++ b/arch/sparc/include/asm/jump_label.h @@ -1,6 +1,8 @@ #ifndef _ASM_SPARC_JUMP_LABEL_H #define _ASM_SPARC_JUMP_LABEL_H +#include + #ifndef __ASSEMBLY__ #include @@ -12,7 +14,7 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran asm_volatile_goto("1:\n\t" "nop\n\t" "nop\n\t" - ".pushsection __jump_table, \"aw\"\n\t" + push_section_tbl_any(.data, __jump_table, aw) ".align 4\n\t" ".word 1b, %l[l_yes], %c0\n\t" ".popsection \n\t" @@ -28,7 +30,7 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool asm_volatile_goto("1:\n\t" "b %l[l_yes]\n\t" "nop\n\t" - ".pushsection __jump_table, \"aw\"\n\t" + push_section_tbl_any(.data, __jump_table, aw) ".align 4\n\t" ".word 1b, %l[l_yes], %c0\n\t" ".popsection \n\t" diff --git a/arch/tile/include/asm/jump_label.h b/arch/tile/include/asm/jump_label.h index cde7573f397b..9bd989daad4a 100644 --- a/arch/tile/include/asm/jump_label.h +++ b/arch/tile/include/asm/jump_label.h @@ -16,6 +16,7 @@ #define _ASM_TILE_JUMP_LABEL_H #include +#include #define JUMP_LABEL_NOP_SIZE TILE_BUNDLE_SIZE_IN_BYTES @@ -24,7 +25,7 @@ static __always_inline bool arch_static_branch(struct static_key *key, { asm_volatile_goto("1:\n\t" "nop" "\n\t" - ".pushsection __jump_table, \"aw\"\n\t" + push_section_tbl_any(.data, __jump_table, aw) ".quad 1b, %l[l_yes], %0 + %1 \n\t" ".popsection\n\t" : : "i" (key), "i" (branch) : : l_yes); @@ -38,7 +39,7 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, { asm_volatile_goto("1:\n\t" "j %l[l_yes]" "\n\t" - ".pushsection __jump_table, \"aw\"\n\t" + push_section_tbl_any(.data, __jump_table, aw) ".quad 1b, %l[l_yes], %0 + %1 \n\t" ".popsection\n\t" : : "i" (key), "i" (branch) : : l_yes); diff --git a/arch/x86/include/asm/jump_label.h b/arch/x86/include/asm/jump_label.h index adc54c12cbd1..627df4752db0 100644 --- a/arch/x86/include/asm/jump_label.h +++ b/arch/x86/include/asm/jump_label.h @@ -1,6 +1,8 @@ #ifndef _ASM_X86_JUMP_LABEL_H #define _ASM_X86_JUMP_LABEL_H +#include + #ifndef HAVE_JUMP_LABEL /* * For better or for worse, if jump labels (the gcc extension) are missing, @@ -34,7 +36,7 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran { asm_volatile_goto("1:" ".byte " __stringify(STATIC_KEY_INIT_NOP) "\n\t" - ".pushsection __jump_table, \"aw\" \n\t" + push_section_tbl_any(.data, __jump_table, aw) _ASM_ALIGN "\n\t" _ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t" ".popsection \n\t" @@ -50,7 +52,7 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool asm_volatile_goto("1:" ".byte 0xe9\n\t .long %l[l_yes] - 2f\n\t" "2:\n\t" - ".pushsection __jump_table, \"aw\" \n\t" + push_section_tbl_any(.data, __jump_table, aw) _ASM_ALIGN "\n\t" _ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t" ".popsection \n\t" @@ -85,7 +87,7 @@ struct jump_entry { .else .byte STATIC_KEY_INIT_NOP .endif - .pushsection __jump_table, "aw" + push_section_tbl_any(.data, __jump_table, aw) _ASM_ALIGN _ASM_PTR .Lstatic_jump_\@, \target, \key .popsection @@ -101,7 +103,7 @@ struct jump_entry { .long \target - .Lstatic_jump_after_\@ .Lstatic_jump_after_\@: .endif - .pushsection __jump_table, "aw" + push_section_tbl_any(.data, __jump_table, aw) _ASM_ALIGN _ASM_PTR .Lstatic_jump_\@, \target, \key + 1 .popsection diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index 8a5325d75932..887d844f0406 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -216,11 +216,6 @@ *(.data.unlikely) \ STRUCT_ALIGN(); \ *(__tracepoints) \ - /* implement dynamic printk debug */ \ - . = ALIGN(8); \ - VMLINUX_SYMBOL(__start___jump_table) = .; \ - KEEP(*(__jump_table)) \ - VMLINUX_SYMBOL(__stop___jump_table) = .; \ . = ALIGN(8); \ VMLINUX_SYMBOL(__start___verbose) = .; \ KEEP(*(__verbose)) \ diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index b63d6b7b0db0..e6277888b558 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -117,6 +117,7 @@ enum jump_label_type { struct module; #ifdef HAVE_JUMP_LABEL +#include #define JUMP_TYPE_FALSE 0UL #define JUMP_TYPE_TRUE 1UL @@ -132,8 +133,7 @@ static __always_inline bool static_key_true(struct static_key *key) return !arch_static_branch(key, true); } -extern struct jump_entry __start___jump_table[]; -extern struct jump_entry __stop___jump_table[]; +DECLARE_LINKTABLE(struct jump_entry, __jump_table); extern void jump_label_init(void); extern void jump_label_lock(void); diff --git a/kernel/jump_label.c b/kernel/jump_label.c index 93ad6c1fb9b6..4252f17eaeb5 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -15,9 +15,12 @@ #include #include #include +#include #ifdef HAVE_JUMP_LABEL +DEFINE_LINKTABLE(struct jump_entry, __jump_table); + /* mutex to protect coming/going of the the jump_label table */ static DEFINE_MUTEX(jump_label_mutex); @@ -274,8 +277,6 @@ static void __jump_label_update(struct static_key *key, void __init jump_label_init(void) { - struct jump_entry *iter_start = __start___jump_table; - struct jump_entry *iter_stop = __stop___jump_table; struct static_key *key = NULL; struct jump_entry *iter; @@ -292,9 +293,10 @@ void __init jump_label_init(void) return; jump_label_lock(); - jump_label_sort_entries(iter_start, iter_stop); + jump_label_sort_entries(LINKTABLE_START(__jump_table), + LINKTABLE_END(__jump_table)); - for (iter = iter_start; iter < iter_stop; iter++) { + linktable_for_each(iter, __jump_table) { struct static_key *iterk; /* rewrite NOPs */ @@ -539,8 +541,9 @@ early_initcall(jump_label_init_module); */ int jump_label_text_reserved(void *start, void *end) { - int ret = __jump_label_text_reserved(__start___jump_table, - __stop___jump_table, start, end); + int ret = __jump_label_text_reserved(LINKTABLE_START(__jump_table), + LINKTABLE_END(__jump_table), + start, end); if (ret) return ret; @@ -553,7 +556,7 @@ int jump_label_text_reserved(void *start, void *end) static void jump_label_update(struct static_key *key) { - struct jump_entry *stop = __stop___jump_table; + struct jump_entry *stop = LINKTABLE_END(__jump_table); struct jump_entry *entry = static_key_entries(key); #ifdef CONFIG_MODULES struct module *mod; diff --git a/scripts/module-common.lds b/scripts/module-common.lds index 73a2c7da0e55..db0e724556c5 100644 --- a/scripts/module-common.lds +++ b/scripts/module-common.lds @@ -16,6 +16,7 @@ SECTIONS { __kcrctab_unused 0 : { *(SORT(___kcrctab_unused+*)) } __kcrctab_unused_gpl 0 : { *(SORT(___kcrctab_unused_gpl+*)) } __kcrctab_gpl_future 0 : { *(SORT(___kcrctab_gpl_future+*)) } + __jump_table 0 : { *(SORT(.data..tbl.__jump_table.*)) } . = ALIGN(8); .init_array 0 : { *(SORT(.init_array.*)) *(.init_array) } diff --git a/tools/objtool/special.c b/tools/objtool/special.c index bff8abb3a4aa..79968e118294 100644 --- a/tools/objtool/special.c +++ b/tools/objtool/special.c @@ -63,7 +63,7 @@ struct special_entry entries[] = { .feature = ALT_FEATURE_OFFSET, }, { - .sec = "__jump_table", + .sec = ".data..tbl.__jump_table.any", .jump_or_nop = true, .size = JUMP_ENTRY_SIZE, .orig = JUMP_ORIG_OFFSET,