From patchwork Tue Jul 9 15:52:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Torvalds X-Patchwork-Id: 13728273 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3ABFCC2BD09 for ; Tue, 9 Jul 2024 15:59:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=iR+380rJgO+02OvTpZE23TZZ7C2iGxr0+gL81RUTwPc=; b=QEm3PclcAEF95uidUAS0gzg8ZI 7LG91X5mISvNcPmjU+OQrfQysegFa8fPz4d1ye7xcqzu4xqXigyURp5wcKTphEjB5xYXY1veRrKhD Y2QBMA4bw7lkTl4O6N2u9ThGcYzbYubjIWSwA0TyCQI9u/f3YZo7s0ThNErbKf8R7BnGB8XIpX6Yy LlIroweblnm5meAJBWLoZ87G5/Bhhcm7+YuVCnRMN9JnUuayO1cQuhgA5UTYGDLE4odku3S+A+qnx FpNvcmEOxmVG+PbDsN9uOK4t4akP41dukM5TkTf6PEDPeeYlofKCQFT+tdIJ/XUjFuqrBXshuS2Kd IR1wlzzA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sRDFW-00000007pY4-2rKE; Tue, 09 Jul 2024 15:59:26 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sRDFA-00000007pSz-1FBr for linux-arm-kernel@lists.infradead.org; Tue, 09 Jul 2024 15:59:05 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id CACA561495; Tue, 9 Jul 2024 15:59:03 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 71F1AC3277B; Tue, 9 Jul 2024 15:59:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1720540743; bh=AxeTD70Xdnmf73ygmURvb2qqMwfuW1qpYQt8rNtFhzM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ceH+F4MBb6DamtsVEjLGY5AVU0BR9zwrJ4frY4h1Zt/2+DEj9/itKuUuKaA0mi61u 1Cjhn9Iv+q083M83UpmNH5xu0HAURGNIP8tGJ9k/AQn/BoSnAr0lND76ElmNhfsPt8 lqgV8rMtDoGdbY9ocfQFcP98nnRLf7FvbyYbq190= From: Linus Torvalds To: Mark Rutland Cc: Linux ARM , Linus Torvalds Subject: [PATCH 1/4] vfs: dcache: move hashlen_hash() from callers into d_hash() Date: Tue, 9 Jul 2024 08:52:35 -0700 Message-ID: <20240709155850.1034904-2-torvalds@linux-foundation.org> X-Mailer: git-send-email 2.45.1.209.gc6f12300df In-Reply-To: <20240709155850.1034904-1-torvalds@linux-foundation.org> References: <20240709155850.1034904-1-torvalds@linux-foundation.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240709_085904_396655_BDCCB118 X-CRM114-Status: GOOD ( 16.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Both __d_lookup_rcu() and __d_lookup_rcu_op_compare() have the full 'name_hash' value of the qstr that they want to look up, and mask it off to just the low 32-bit hash before calling down to d_hash(). Other callers just load the 32-bit hash and pass it as the argument. If we move the masking into d_hash() itself, it simplifies the two callers that currently do the masking, and is a no-op for the other cases. It doesn't actually change the generated code since the compiler will inline d_hash() and see that the end result is the same. [ Technically, since the parse tree changes, the code generation may not be 100% the same, and for me on x86-64, this does result in gcc switching the operands around for one 'cmpl' instruction. So not necessarily the exact same code generation, but equivalent ] However, this does encapsulate the 'd_hash()' operation more, and makes the shift operation in particular be a "shift 32 bits right, return full word". Which matches the instruction semantics on both x86-64 and arm64 better, since a 32-bit shift will clear the upper bits. That makes the next step of introducing a "shift by runtime constant" more obvious and generates the shift with no extraneous type masking. Signed-off-by: Linus Torvalds --- fs/dcache.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/fs/dcache.c b/fs/dcache.c index 407095188f83..8b4382f5c99d 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -100,9 +100,9 @@ static unsigned int d_hash_shift __ro_after_init; static struct hlist_bl_head *dentry_hashtable __ro_after_init; -static inline struct hlist_bl_head *d_hash(unsigned int hash) +static inline struct hlist_bl_head *d_hash(unsigned long hashlen) { - return dentry_hashtable + (hash >> d_hash_shift); + return dentry_hashtable + ((u32)hashlen >> d_hash_shift); } #define IN_LOOKUP_SHIFT 10 @@ -2104,7 +2104,7 @@ static noinline struct dentry *__d_lookup_rcu_op_compare( unsigned *seqp) { u64 hashlen = name->hash_len; - struct hlist_bl_head *b = d_hash(hashlen_hash(hashlen)); + struct hlist_bl_head *b = d_hash(hashlen); struct hlist_bl_node *node; struct dentry *dentry; @@ -2171,7 +2171,7 @@ struct dentry *__d_lookup_rcu(const struct dentry *parent, { u64 hashlen = name->hash_len; const unsigned char *str = name->name; - struct hlist_bl_head *b = d_hash(hashlen_hash(hashlen)); + struct hlist_bl_head *b = d_hash(hashlen); struct hlist_bl_node *node; struct dentry *dentry; From patchwork Tue Jul 9 15:52:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Torvalds X-Patchwork-Id: 13728274 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 38E8EC3DA41 for ; Tue, 9 Jul 2024 15:59:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=KavYbSsMmZaJbJkJaH08wZcBYTvA6P16zb5vRnf11nw=; b=NFFE41bmtEm+xT5qmgm92kx01H umvCxF4nG4Fa1OJSI3J/5xtAS92+zIxHo5lVpHUVLsQeD9b2JGYS3aIqJ+RORfJSIlckQc/Q1WKZL iv9C2qpxMzPdhimifUc+AHi90H44hXe0spM3VwY0Eb7HQ0ZOwJL0ecbyl3eTyWrECts0JDEmUfMOL Wkj7GG6eYSLxI3SnSmkJh8mNm9dptb1H3VnMW7ZsJw0XzVHYq16O7iA8MLHXZZ5zzEcR8E+yOideZ jj95L2g55NUXdo6hiJxoT5nf3PsKpV5W1VZzn/M2wp75GRB2a+tIHT4hzD9haBgFMJrEoSFCpAwF2 25NOvB6Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sRDFk-00000007pcP-3XX7; Tue, 09 Jul 2024 15:59:40 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sRDFH-00000007pTx-2uJK for linux-arm-kernel@lists.infradead.org; Tue, 09 Jul 2024 15:59:13 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 79F5E61363; Tue, 9 Jul 2024 15:59:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 26501C3277B; Tue, 9 Jul 2024 15:59:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1720540750; bh=dZtZTWzs2dqoSlS9RVX7l1MeMI0JlUEGMgfGbHX6Cso=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tsNFgy8M84zo13yJyBATI1DrPuvGs4s6LnThEyG3ynHuR6sICpYNew4VI58Jj9+5k QSmBrnMBbejD7ycbRcL4uqUeJZ9ivVwuFdMuawNyOJHHdJHh4GAnzdagbvs82GRBox e8IjKorNOVVZWtrGc2U6GsmhVXSy8/3MV9ZknDiA= From: Linus Torvalds To: Mark Rutland Cc: Linux ARM , Linus Torvalds Subject: [PATCH 2/4] runtime constants: add default dummy infrastructure Date: Tue, 9 Jul 2024 08:52:36 -0700 Message-ID: <20240709155850.1034904-3-torvalds@linux-foundation.org> X-Mailer: git-send-email 2.45.1.209.gc6f12300df In-Reply-To: <20240709155850.1034904-1-torvalds@linux-foundation.org> References: <20240709155850.1034904-1-torvalds@linux-foundation.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240709_085911_881830_1476D789 X-CRM114-Status: GOOD ( 20.05 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This adds the initial dummy support for 'runtime constants' for when an architecture doesn't actually support an implementation of fixing up said runtime constants. This ends up being the fallback to just using the variables as regular __ro_after_init variables, and changes the dcache d_hash() function to use this model. Signed-off-by: Linus Torvalds --- fs/dcache.c | 11 ++++++++++- include/asm-generic/Kbuild | 1 + include/asm-generic/runtime-const.h | 15 +++++++++++++++ include/asm-generic/vmlinux.lds.h | 8 ++++++++ 4 files changed, 34 insertions(+), 1 deletion(-) create mode 100644 include/asm-generic/runtime-const.h diff --git a/fs/dcache.c b/fs/dcache.c index 8b4382f5c99d..5d3a5b315692 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -35,6 +35,8 @@ #include "internal.h" #include "mount.h" +#include + /* * Usage: * dcache->d_inode->i_lock protects: @@ -102,7 +104,8 @@ static struct hlist_bl_head *dentry_hashtable __ro_after_init; static inline struct hlist_bl_head *d_hash(unsigned long hashlen) { - return dentry_hashtable + ((u32)hashlen >> d_hash_shift); + return runtime_const_ptr(dentry_hashtable) + + runtime_const_shift_right_32(hashlen, d_hash_shift); } #define IN_LOOKUP_SHIFT 10 @@ -3129,6 +3132,9 @@ static void __init dcache_init_early(void) 0, 0); d_hash_shift = 32 - d_hash_shift; + + runtime_const_init(shift, d_hash_shift); + runtime_const_init(ptr, dentry_hashtable); } static void __init dcache_init(void) @@ -3157,6 +3163,9 @@ static void __init dcache_init(void) 0, 0); d_hash_shift = 32 - d_hash_shift; + + runtime_const_init(shift, d_hash_shift); + runtime_const_init(ptr, dentry_hashtable); } /* SLAB cache for __getname() consumers */ diff --git a/include/asm-generic/Kbuild b/include/asm-generic/Kbuild index b20fa25a7e8d..052e5c98c105 100644 --- a/include/asm-generic/Kbuild +++ b/include/asm-generic/Kbuild @@ -46,6 +46,7 @@ mandatory-y += pci.h mandatory-y += percpu.h mandatory-y += pgalloc.h mandatory-y += preempt.h +mandatory-y += runtime-const.h mandatory-y += rwonce.h mandatory-y += sections.h mandatory-y += serial.h diff --git a/include/asm-generic/runtime-const.h b/include/asm-generic/runtime-const.h new file mode 100644 index 000000000000..670499459514 --- /dev/null +++ b/include/asm-generic/runtime-const.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_RUNTIME_CONST_H +#define _ASM_RUNTIME_CONST_H + +/* + * This is the fallback for when the architecture doesn't + * support the runtime const operations. + * + * We just use the actual symbols as-is. + */ +#define runtime_const_ptr(sym) (sym) +#define runtime_const_shift_right_32(val, sym) ((u32)(val)>>(sym)) +#define runtime_const_init(type,sym) do { } while (0) + +#endif diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index 5703526d6ebf..389a78415b9b 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -944,6 +944,14 @@ #define CON_INITCALL \ BOUNDED_SECTION_POST_LABEL(.con_initcall.init, __con_initcall, _start, _end) +#define RUNTIME_NAME(t,x) runtime_##t##_##x + +#define RUNTIME_CONST(t,x) \ + . = ALIGN(8); \ + RUNTIME_NAME(t,x) : AT(ADDR(RUNTIME_NAME(t,x)) - LOAD_OFFSET) { \ + *(RUNTIME_NAME(t,x)); \ + } + /* Alignment must be consistent with (kunit_suite *) in include/kunit/test.h */ #define KUNIT_TABLE() \ . = ALIGN(8); \ From patchwork Tue Jul 9 15:52:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Torvalds X-Patchwork-Id: 13728275 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5EA7FC3DA41 for ; Tue, 9 Jul 2024 16:00:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=0iU5zmVbhgxT4dsRxaiKfwBHkM4BLg/GO4Fb9WoHbvk=; b=Nh6ZOG7FmjAs6ywJUMnufIzbYK +x1wbxjAIg+d39rcYr/1BIy0PARDpPls9kociFuKquZRtx3OlCxT7OiCnKe2Jg9CNvVAj7GOfx1X8 AXhb+KcaEUPe8tz3YE3mXSJ6z9TF2peDtyZniavnxyj+5Zq5gpgWIF+hzwNR599BhYvA27wDtSV+G GS3Mo9q5cRkMLTxpV5ZFr3A6Xns3OGMSierYNSiUcULCWjmQLIWsBjte4Xx2mejG7I2IFTE+lQ3Yh 6GEDwlRSFU/QKlWAsRuM9h9+REoZYSJ2S+7hCeA6KmazWefCBiA3BAxhtgcATJsQFr+sH6yQ4CtD+ brwSxKPw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sRDFx-00000007pfW-3Vms; Tue, 09 Jul 2024 15:59:53 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sRDFK-00000007pUo-3DGO for linux-arm-kernel@lists.infradead.org; Tue, 09 Jul 2024 15:59:16 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 1F56861495; Tue, 9 Jul 2024 15:59:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BDFF2C3277B; Tue, 9 Jul 2024 15:59:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1720540753; bh=qEWhUwOtY4dP2oKEOdCg3gFZW6ZZXNIFIaLC2/7lZ38=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tu9CxVdlLl71f35UGAw1yqfOHP/IzhWMHe5o9gjUGh2wriwEPwA9r0fprr1gHHJzq R1GbrQqe3Ijmancms62GDovv+I2sPhfrm+VgG8+RsHWIUPuHjbEtsnr6n0sax4d+Ab 5dTGwu/I0Qx0rczI1G5OM9GKMsLijqPVP8KsAP3w= From: Linus Torvalds To: Mark Rutland Cc: Linux ARM , Linus Torvalds Subject: [PATCH 3/4] runtime constants: add x86 architecture support Date: Tue, 9 Jul 2024 08:52:37 -0700 Message-ID: <20240709155850.1034904-4-torvalds@linux-foundation.org> X-Mailer: git-send-email 2.45.1.209.gc6f12300df In-Reply-To: <20240709155850.1034904-1-torvalds@linux-foundation.org> References: <20240709155850.1034904-1-torvalds@linux-foundation.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240709_085914_956832_12468A79 X-CRM114-Status: GOOD ( 15.72 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This implements the runtime constant infrastructure for x86, allowing the dcache d_hash() function to be generated using as a constant for hash table address followed by shift by a constant of the hash index. Signed-off-by: Linus Torvalds --- arch/x86/include/asm/runtime-const.h | 61 ++++++++++++++++++++++++++++ arch/x86/kernel/vmlinux.lds.S | 3 ++ 2 files changed, 64 insertions(+) create mode 100644 arch/x86/include/asm/runtime-const.h diff --git a/arch/x86/include/asm/runtime-const.h b/arch/x86/include/asm/runtime-const.h new file mode 100644 index 000000000000..24e3a53ca255 --- /dev/null +++ b/arch/x86/include/asm/runtime-const.h @@ -0,0 +1,61 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_RUNTIME_CONST_H +#define _ASM_RUNTIME_CONST_H + +#define runtime_const_ptr(sym) ({ \ + typeof(sym) __ret; \ + asm_inline("mov %1,%0\n1:\n" \ + ".pushsection runtime_ptr_" #sym ",\"a\"\n\t" \ + ".long 1b - %c2 - .\n\t" \ + ".popsection" \ + :"=r" (__ret) \ + :"i" ((unsigned long)0x0123456789abcdefull), \ + "i" (sizeof(long))); \ + __ret; }) + +// The 'typeof' will create at _least_ a 32-bit type, but +// will happily also take a bigger type and the 'shrl' will +// clear the upper bits +#define runtime_const_shift_right_32(val, sym) ({ \ + typeof(0u+(val)) __ret = (val); \ + asm_inline("shrl $12,%k0\n1:\n" \ + ".pushsection runtime_shift_" #sym ",\"a\"\n\t" \ + ".long 1b - 1 - .\n\t" \ + ".popsection" \ + :"+r" (__ret)); \ + __ret; }) + +#define runtime_const_init(type, sym) do { \ + extern s32 __start_runtime_##type##_##sym[]; \ + extern s32 __stop_runtime_##type##_##sym[]; \ + runtime_const_fixup(__runtime_fixup_##type, \ + (unsigned long)(sym), \ + __start_runtime_##type##_##sym, \ + __stop_runtime_##type##_##sym); \ +} while (0) + +/* + * The text patching is trivial - you can only do this at init time, + * when the text section hasn't been marked RO, and before the text + * has ever been executed. + */ +static inline void __runtime_fixup_ptr(void *where, unsigned long val) +{ + *(unsigned long *)where = val; +} + +static inline void __runtime_fixup_shift(void *where, unsigned long val) +{ + *(unsigned char *)where = val; +} + +static inline void runtime_const_fixup(void (*fn)(void *, unsigned long), + unsigned long val, s32 *start, s32 *end) +{ + while (start < end) { + fn(*start + (void *)start, val); + start++; + } +} + +#endif diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index 3509afc6a672..6e73403e874f 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -357,6 +357,9 @@ SECTIONS PERCPU_SECTION(INTERNODE_CACHE_BYTES) #endif + RUNTIME_CONST(shift, d_hash_shift) + RUNTIME_CONST(ptr, dentry_hashtable) + . = ALIGN(PAGE_SIZE); /* freed after init ends here */ From patchwork Tue Jul 9 15:52:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Torvalds X-Patchwork-Id: 13728276 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7B170C3DA41 for ; Tue, 9 Jul 2024 16:00:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=AdOC+RvXkWTWmc5GjpiQMrAQpJEdoBJ1e1hrRCQ0LSE=; b=Q5xplzCrR2a9LEk9hSUugak+Dt f1ats0x2c1wKMAzbWok8dMy0K0sGbzTCStXDZ+fplL7fKNAIFBv8sOYhYTtUcehwLemtBc5B2mnvi gEn+HQEEtlZsz7rXnhoQmmCotsnrIy1a/+OLRTlXz8FbK8LFrmUYE8rEzbYeWtVMEh+G51mf6l5EK jpfZr6VP3MHruK/QzYi27z0X6t4wGnxon0BJry00d4fwhciRk5FpNF6NWCYToPgK6m+4Ch7V8FGW8 e2vzcHydi/zkWC6PaP2v4tVtEd/Vkd3Xq3vY3RDvOfqt6icPSumK1kceHdRNmFggURYu4QWEZxA/t 9ag+vRNQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sRDGA-00000007pja-3sZ7; Tue, 09 Jul 2024 16:00:06 +0000 Received: from sin.source.kernel.org ([145.40.73.55]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sRDFP-00000007pVk-3Fj8 for linux-arm-kernel@lists.infradead.org; Tue, 09 Jul 2024 15:59:21 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 17779CE127A; Tue, 9 Jul 2024 15:59:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 40AE9C3277B; Tue, 9 Jul 2024 15:59:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1720540756; bh=0Lk/31VRhF6JzV8dtgK7Hkxt17ELFPo4TeVcxEuEAI4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=J/fjcYIDC2lDv47XvUDrlAgYju907+K6K6mKUIy8IyF+77vUkEE2Py4kSZXTSMHpT ljq1348LoCOD6z6WF6vTC1GhscnbZn3gKS1lnfbhqL600EAyOPj9GT+3yrQesJmlVj CJd+03Tfj02/CfQ39bcf8TJT8o6/5IVoY8BifPeU= From: Linus Torvalds To: Mark Rutland Cc: Linux ARM , Linus Torvalds Subject: [PATCH 4/4] arm64: add 'runtime constant' support Date: Tue, 9 Jul 2024 08:52:38 -0700 Message-ID: <20240709155850.1034904-5-torvalds@linux-foundation.org> X-Mailer: git-send-email 2.45.1.209.gc6f12300df In-Reply-To: <20240709155850.1034904-1-torvalds@linux-foundation.org> References: <20240709155850.1034904-1-torvalds@linux-foundation.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240709_085920_214167_6F25B18B X-CRM114-Status: GOOD ( 15.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This implements the runtime constant infrastructure for arm64, allowing the dcache d_hash() function to be generated using as a constant for hash table address followed by shift by a constant of the hash index. [ Fixed up to deal with the big-endian case as per Mark Rutland ] Signed-off-by: Linus Torvalds --- arch/arm64/include/asm/runtime-const.h | 88 ++++++++++++++++++++++++++ arch/arm64/kernel/vmlinux.lds.S | 3 + 2 files changed, 91 insertions(+) create mode 100644 arch/arm64/include/asm/runtime-const.h diff --git a/arch/arm64/include/asm/runtime-const.h b/arch/arm64/include/asm/runtime-const.h new file mode 100644 index 000000000000..be5915669d23 --- /dev/null +++ b/arch/arm64/include/asm/runtime-const.h @@ -0,0 +1,88 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_RUNTIME_CONST_H +#define _ASM_RUNTIME_CONST_H + +#include + +/* Sigh. You can still run arm64 in BE mode */ +#include + +#define runtime_const_ptr(sym) ({ \ + typeof(sym) __ret; \ + asm_inline("1:\t" \ + "movz %0, #0xcdef\n\t" \ + "movk %0, #0x89ab, lsl #16\n\t" \ + "movk %0, #0x4567, lsl #32\n\t" \ + "movk %0, #0x0123, lsl #48\n\t" \ + ".pushsection runtime_ptr_" #sym ",\"a\"\n\t" \ + ".long 1b - .\n\t" \ + ".popsection" \ + :"=r" (__ret)); \ + __ret; }) + +#define runtime_const_shift_right_32(val, sym) ({ \ + unsigned long __ret; \ + asm_inline("1:\t" \ + "lsr %w0,%w1,#12\n\t" \ + ".pushsection runtime_shift_" #sym ",\"a\"\n\t" \ + ".long 1b - .\n\t" \ + ".popsection" \ + :"=r" (__ret) \ + :"r" (0u+(val))); \ + __ret; }) + +#define runtime_const_init(type, sym) do { \ + extern s32 __start_runtime_##type##_##sym[]; \ + extern s32 __stop_runtime_##type##_##sym[]; \ + runtime_const_fixup(__runtime_fixup_##type, \ + (unsigned long)(sym), \ + __start_runtime_##type##_##sym, \ + __stop_runtime_##type##_##sym); \ +} while (0) + +/* 16-bit immediate for wide move (movz and movk) in bits 5..20 */ +static inline void __runtime_fixup_16(__le32 *p, unsigned int val) +{ + u32 insn = le32_to_cpu(*p); + insn &= 0xffe0001f; + insn |= (val & 0xffff) << 5; + *p = cpu_to_le32(insn); +} + +static inline void __runtime_fixup_caches(void *where, unsigned int insns) +{ + unsigned long va = (unsigned long)where; + caches_clean_inval_pou(va, va + 4*insns); +} + +static inline void __runtime_fixup_ptr(void *where, unsigned long val) +{ + __le32 *p = lm_alias(where); + __runtime_fixup_16(p, val); + __runtime_fixup_16(p+1, val >> 16); + __runtime_fixup_16(p+2, val >> 32); + __runtime_fixup_16(p+3, val >> 48); + __runtime_fixup_caches(where, 4); +} + +/* Immediate value is 6 bits starting at bit #16 */ +static inline void __runtime_fixup_shift(void *where, unsigned long val) +{ + __le32 *p = lm_alias(where); + u32 insn = le32_to_cpu(*p); + insn &= 0xffc0ffff; + insn |= (val & 63) << 16; + *p = cpu_to_le32(insn); + __runtime_fixup_caches(where, 1); +} + +static inline void runtime_const_fixup(void (*fn)(void *, unsigned long), + unsigned long val, s32 *start, s32 *end) +{ + while (start < end) { + fn(*start + (void *)start, val); + start++; + } +} + +#endif diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 755a22d4f840..55a8e310ea12 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -264,6 +264,9 @@ SECTIONS EXIT_DATA } + RUNTIME_CONST(shift, d_hash_shift) + RUNTIME_CONST(ptr, dentry_hashtable) + PERCPU_SECTION(L1_CACHE_BYTES) HYPERVISOR_PERCPU_SECTION