From patchwork Mon Sep 18 07:29:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13388973 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63DA7CD37B0 for ; Mon, 18 Sep 2023 07:30:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F06D86B027F; Mon, 18 Sep 2023 03:30:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EB6E56B0281; Mon, 18 Sep 2023 03:30:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D09656B0282; Mon, 18 Sep 2023 03:30:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id BB0DA6B027F for ; Mon, 18 Sep 2023 03:30:57 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 7AA32C0C23 for ; Mon, 18 Sep 2023 07:30:57 +0000 (UTC) X-FDA: 81248896554.29.C09ADE9 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf08.hostedemail.com (Postfix) with ESMTP id BB44E160017 for ; Mon, 18 Sep 2023 07:30:55 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=B3MpqZd8; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf08.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695022255; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pR/DurLeWvhG1FKfSYFkfT2vHvRwZ8tEPH/eAXER1RU=; b=qQp0hOtK3xcaG6vM+38LBgc2CwrHbzyU26GimzNMp981bXUFqd2naK4HlTQRx6a4agPJWK GgJ5gt7VJzyla7M0OcFkT375Y19QtoB+zG9f27bVWOVpI2b0mMj/zLP2wdZanTXeypWLxn SwSSU+NWZsnCveEfqwumzftjZO4Afmw= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=B3MpqZd8; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf08.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695022255; a=rsa-sha256; cv=none; b=IzLV10BO8RXETJGJ9D1Iux4ekzJjZtHqBP7AxQmlsUBTvpTMXQvMAKZVQSjyFzsaJ4s6EC IV5ALmppgpd47T6bU0tvLJ9YdUOaUUukZoEgx36D3TewtZRTJLygG2mFmlCO5D4EsQs5eV 4c8gYazlbc7iY7R1U858GG5SAW0I4rs= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id AF9D260F83; Mon, 18 Sep 2023 07:30:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AFDC9C433B9; Mon, 18 Sep 2023 07:30:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695022253; bh=Ab3JxL0vUtd+Dgg9Evr/24yil9x7cJIlVNJ+mIHZ+Fw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=B3MpqZd8kSnshGSIZk32sAvbfxOLIcvRzx/L6e6twO1GyIkCPbyDJ8YFZK7YV3DYj XGhAnNQsgAWy/xSCUlo5IqNl0WBTUqkmBh0vCS1bt48EfgT1+dVwPuJNvN1ivOBOMn gvH0utW4rdMOUN0nD6KFxNUfq8geoRPWIJ6VKmk1aH2z09LsV+3CM1A5GN8hUlJfXJ ASdeswxXtYTc0LaHjCD2b/Jj+5rTLeYIx1gwiLh7ZEjRZ1XJfx7lSWfp5ovjLk7FgH t/b+tjGlJaBG8XvWXgVzie1/pkYA3m8DAD1W+ztdCKD9/UtdKJD1oiqOVjKdWg5YrB ItW+yxqIqOhzA== From: Mike Rapoport To: linux-kernel@vger.kernel.org Cc: Andrew Morton , =?utf-8?b?QmrDtnJuIFTDtnBl?= =?utf-8?b?bA==?= , Catalin Marinas , Christophe Leroy , "David S. Miller" , Dinh Nguyen , Heiko Carstens , Helge Deller , Huacai Chen , Kent Overstreet , Luis Chamberlain , Mark Rutland , Michael Ellerman , Mike Rapoport , Nadav Amit , "Naveen N. Rao" , Palmer Dabbelt , Puranjay Mohan , Rick Edgecombe , Russell King , Song Liu , Steven Rostedt , Thomas Bogendoerfer , Thomas Gleixner , Will Deacon , bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, linux-parisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, loongarch@lists.linux.dev, netdev@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: [PATCH v3 04/13] mm/execmem, arch: convert remaining overrides of module_alloc to execmem Date: Mon, 18 Sep 2023 10:29:46 +0300 Message-Id: <20230918072955.2507221-5-rppt@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230918072955.2507221-1-rppt@kernel.org> References: <20230918072955.2507221-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: BB44E160017 X-Stat-Signature: 4bxdo73f5xenoqtzektghixuus13ag6x X-Rspam-User: X-HE-Tag: 1695022255-87998 X-HE-Meta: U2FsdGVkX1+kJcv/kZtNw8dgzKeRHeZog8oci+Y2nx0Gk45FePZ3CM8ig4jfxTDpxn/rL55JFCZZTLMKb4/edjxPXeF//0Do9Tn/qQIaVQ3rh6UzqxozQ/TSnTLDx2S455hsQy4Poqxe5yrNAtQmUZXEWbHhBGZwT3iGvZm6Ncnrj9ez81JUFzwdHsNkb1ihPt3ozJOEYsQk0JV4EvvisVqBk2yx88hpHWUCHf8av3IH527ZRLJSEFEFn//tOhJDBTBw2I7ZiKHy+5NQlwDKbG85TCOl/xtf6tU/1Yioc3U6VAYnxM2yL370bQOQpnj8w6KSMlwRJa52pw91HA7wBL54C38KjrduK/or9bpwZOO1Hhg/V9GEwMTDJWoR0t4KTNV8vNg9Q3avSSCNHgb1hSf0eUmkq02yFhEyXgDlb6QJQizWv7jNaBHrjg9jGOtmpgOyg/W0pv34YtnvyLQK7PQWsRN1w90YaaVK/hAM6Dm0V5JKsSDXRfKR1rg9Cw7MNtvJJsEo/ghrpNYl22C35FZz05NvsWebSyMy7O8vlSlcigntBXMTuOxg7Bx9cwjQ0rN7xTPuTh2JgBuM5t+6gx6oSUPUiE2TUragVM3xsTkzMQCH0FlaMLHugmZaF4P5/HDLHyLNP9ANC2zzPgSn+lVNIBDoixFl7c63+jDvuj1oUBw9eHEonEzPZQ+Ms4hfGzoC0pdppjXs8tReLPncmAQt1prTD5bKfFkoZG+IcueZF+iIrsQmHH2FFGYpjDcvMIqoVYD1gg5jsC2DuPGMLTEHcZYeDiF7A/TL9IeD9vkAF3HxWTGfxfxfeVg7Zi+lpAW9z5BWFIQrdezTgce+4xR3Kr80TSO4+zJDhui9CQ+VVgg0G0Btwe66ypCcB4bn/bbY/jLWXELNac2J0dKJZNTqEd3Qib6x71uzOnyt7hSZ4uTmW0rMjk387dRG1urvVfljIS0KP2UPuwi+V8l 4k1Em/mA G6PxeR94ydjatFsPrp5wiAQkEOd6jPEDTY+/RIldy8UaCoYPkOc+adY7g4/BP7uJMCbrVyJbMJrGnzD68MM8ih8Iz/Y6ViDcpjCRu2xB0If9icK1FPRYCK64TXPq1VPUtYwm8SDoLsBMiyQtw0QbdXE6X9ISIKhBebXoQrpZioJA0uncgteFcglLGVi6N+pZsojKa2DSCgOkv/c4wi3jVkQil9LLWNK3adgk7oDaoOVFymUGnXI3GtzyXl1A1NP93QwY3u2lF5gd+z3Mw48WsPc7BeQzmEnlnvpYObEDCscrs72rb9KT4RMCOGeOn7gjhHvNYYFKauu49lJ1+Li0zEVFKYO9/sqZ3eP9lBn4WpVuOM40j9HulrdEXQ4MDwhdSqZZQPeiGb/PDXQXMAd0v8REbtH3AxrRXAq9mkM6l4Uq5v/AOqtNyN5TdgB5a5XHtNHYUxLX0pdVHl8I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" Extend execmem parameters to accommodate more complex overrides of module_alloc() by architectures. This includes specification of a fallback range required by arm, arm64 and powerpc and support for allocation of KASAN shadow required by arm64, s390 and x86. The core implementation of execmem_alloc() takes care of suppressing warnings when the initial allocation fails but there is a fallback range defined. Signed-off-by: Mike Rapoport (IBM) --- arch/arm/kernel/module.c | 38 ++++++++++++--------- arch/arm64/kernel/module.c | 57 ++++++++++++++------------------ arch/powerpc/kernel/module.c | 52 ++++++++++++++--------------- arch/s390/kernel/module.c | 52 +++++++++++------------------ arch/x86/kernel/module.c | 64 +++++++++++------------------------- include/linux/execmem.h | 14 ++++++++ mm/execmem.c | 43 ++++++++++++++++++++++-- 7 files changed, 167 insertions(+), 153 deletions(-) diff --git a/arch/arm/kernel/module.c b/arch/arm/kernel/module.c index e74d84f58b77..2c7651a2d84c 100644 --- a/arch/arm/kernel/module.c +++ b/arch/arm/kernel/module.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include @@ -34,23 +35,28 @@ #endif #ifdef CONFIG_MMU -void *module_alloc(unsigned long size) +static struct execmem_params execmem_params __ro_after_init = { + .ranges = { + [EXECMEM_DEFAULT] = { + .start = MODULES_VADDR, + .end = MODULES_END, + .alignment = 1, + }, + }, +}; + +struct execmem_params __init *execmem_arch_params(void) { - gfp_t gfp_mask = GFP_KERNEL; - void *p; - - /* Silence the initial allocation */ - if (IS_ENABLED(CONFIG_ARM_MODULE_PLTS)) - gfp_mask |= __GFP_NOWARN; - - p = __vmalloc_node_range(size, 1, MODULES_VADDR, MODULES_END, - gfp_mask, PAGE_KERNEL_EXEC, 0, NUMA_NO_NODE, - __builtin_return_address(0)); - if (!IS_ENABLED(CONFIG_ARM_MODULE_PLTS) || p) - return p; - return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, - GFP_KERNEL, PAGE_KERNEL_EXEC, 0, NUMA_NO_NODE, - __builtin_return_address(0)); + struct execmem_range *r = &execmem_params.ranges[EXECMEM_DEFAULT]; + + r->pgprot = PAGE_KERNEL_EXEC; + + if (IS_ENABLED(CONFIG_ARM_MODULE_PLTS)) { + r->fallback_start = VMALLOC_START; + r->fallback_end = VMALLOC_END; + } + + return &execmem_params; } #endif diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c index dd851297596e..cd6320de1c54 100644 --- a/arch/arm64/kernel/module.c +++ b/arch/arm64/kernel/module.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include @@ -108,46 +109,38 @@ static int __init module_init_limits(void) return 0; } -subsys_initcall(module_init_limits); -void *module_alloc(unsigned long size) +static struct execmem_params execmem_params __ro_after_init = { + .ranges = { + [EXECMEM_DEFAULT] = { + .flags = EXECMEM_KASAN_SHADOW, + .alignment = MODULE_ALIGN, + }, + }, +}; + +struct execmem_params __init *execmem_arch_params(void) { - void *p = NULL; + struct execmem_range *r = &execmem_params.ranges[EXECMEM_DEFAULT]; - /* - * Where possible, prefer to allocate within direct branch range of the - * kernel such that no PLTs are necessary. - */ - if (module_direct_base) { - p = __vmalloc_node_range(size, MODULE_ALIGN, - module_direct_base, - module_direct_base + SZ_128M, - GFP_KERNEL | __GFP_NOWARN, - PAGE_KERNEL, 0, NUMA_NO_NODE, - __builtin_return_address(0)); - } + module_init_limits(); - if (!p && module_plt_base) { - p = __vmalloc_node_range(size, MODULE_ALIGN, - module_plt_base, - module_plt_base + SZ_2G, - GFP_KERNEL | __GFP_NOWARN, - PAGE_KERNEL, 0, NUMA_NO_NODE, - __builtin_return_address(0)); - } + r->pgprot = PAGE_KERNEL; - if (!p) { - pr_warn_ratelimited("%s: unable to allocate memory\n", - __func__); - } + if (module_direct_base) { + r->start = module_direct_base; + r->end = module_direct_base + SZ_128M; - if (p && (kasan_alloc_module_shadow(p, size, GFP_KERNEL) < 0)) { - vfree(p); - return NULL; + if (module_plt_base) { + r->fallback_start = module_plt_base; + r->fallback_end = module_plt_base + SZ_2G; + } + } else if (module_plt_base) { + r->start = module_plt_base; + r->end = module_plt_base + SZ_2G; } - /* Memory is intended to be executable, reset the pointer tag. */ - return kasan_reset_tag(p); + return &execmem_params; } enum aarch64_reloc_op { diff --git a/arch/powerpc/kernel/module.c b/arch/powerpc/kernel/module.c index f6d6ae0a1692..f4dd26f693a3 100644 --- a/arch/powerpc/kernel/module.c +++ b/arch/powerpc/kernel/module.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -89,39 +90,38 @@ int module_finalize(const Elf_Ehdr *hdr, return 0; } -static __always_inline void * -__module_alloc(unsigned long size, unsigned long start, unsigned long end, bool nowarn) +static struct execmem_params execmem_params __ro_after_init = { + .ranges = { + [EXECMEM_DEFAULT] = { + .alignment = 1, + }, + }, +}; + +struct execmem_params __init *execmem_arch_params(void) { pgprot_t prot = strict_module_rwx_enabled() ? PAGE_KERNEL : PAGE_KERNEL_EXEC; - gfp_t gfp = GFP_KERNEL | (nowarn ? __GFP_NOWARN : 0); - - /* - * Don't do huge page allocations for modules yet until more testing - * is done. STRICT_MODULE_RWX may require extra work to support this - * too. - */ - return __vmalloc_node_range(size, 1, start, end, gfp, prot, - VM_FLUSH_RESET_PERMS, - NUMA_NO_NODE, __builtin_return_address(0)); -} + struct execmem_range *range = &execmem_params.ranges[EXECMEM_DEFAULT]; -void *module_alloc(unsigned long size) -{ #ifdef MODULES_VADDR unsigned long limit = (unsigned long)_etext - SZ_32M; - void *ptr = NULL; - - BUILD_BUG_ON(TASK_SIZE > MODULES_VADDR); /* First try within 32M limit from _etext to avoid branch trampolines */ - if (MODULES_VADDR < PAGE_OFFSET && MODULES_END > limit) - ptr = __module_alloc(size, limit, MODULES_END, true); - - if (!ptr) - ptr = __module_alloc(size, MODULES_VADDR, MODULES_END, false); - - return ptr; + if (MODULES_VADDR < PAGE_OFFSET && MODULES_END > limit) { + range->start = limit; + range->end = MODULES_END; + range->fallback_start = MODULES_VADDR; + range->fallback_end = MODULES_END; + } else { + range->start = MODULES_VADDR; + range->end = MODULES_END; + } #else - return __module_alloc(size, VMALLOC_START, VMALLOC_END, false); + range->start = VMALLOC_START; + range->end = VMALLOC_END; #endif + + range->pgprot = prot; + + return &execmem_params; } diff --git a/arch/s390/kernel/module.c b/arch/s390/kernel/module.c index db5561d0c233..538d5f24af66 100644 --- a/arch/s390/kernel/module.c +++ b/arch/s390/kernel/module.c @@ -37,41 +37,29 @@ #define PLT_ENTRY_SIZE 22 -static unsigned long get_module_load_offset(void) +static struct execmem_params execmem_params __ro_after_init = { + .ranges = { + [EXECMEM_DEFAULT] = { + .flags = EXECMEM_KASAN_SHADOW, + .alignment = MODULE_ALIGN, + .pgprot = PAGE_KERNEL, + }, + }, +}; + +struct execmem_params __init *execmem_arch_params(void) { - static DEFINE_MUTEX(module_kaslr_mutex); - static unsigned long module_load_offset; - - if (!kaslr_enabled()) - return 0; - /* - * Calculate the module_load_offset the first time this code - * is called. Once calculated it stays the same until reboot. - */ - mutex_lock(&module_kaslr_mutex); - if (!module_load_offset) + unsigned long module_load_offset = 0; + unsigned long start; + + if (kaslr_enabled()) module_load_offset = get_random_u32_inclusive(1, 1024) * PAGE_SIZE; - mutex_unlock(&module_kaslr_mutex); - return module_load_offset; -} -void *module_alloc(unsigned long size) -{ - gfp_t gfp_mask = GFP_KERNEL; - void *p; - - if (PAGE_ALIGN(size) > MODULES_LEN) - return NULL; - p = __vmalloc_node_range(size, MODULE_ALIGN, - MODULES_VADDR + get_module_load_offset(), - MODULES_END, gfp_mask, PAGE_KERNEL, - VM_FLUSH_RESET_PERMS | VM_DEFER_KMEMLEAK, - NUMA_NO_NODE, __builtin_return_address(0)); - if (p && (kasan_alloc_module_shadow(p, size, gfp_mask) < 0)) { - vfree(p); - return NULL; - } - return p; + start = MODULES_VADDR + module_load_offset; + execmem_params.ranges[EXECMEM_DEFAULT].start = start; + execmem_params.ranges[EXECMEM_DEFAULT].end = MODULES_END; + + return &execmem_params; } #ifdef CONFIG_FUNCTION_TRACER diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c index 5f71a0cf4399..9d37375e2f05 100644 --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -19,6 +19,7 @@ #include #include #include +#include #include #include @@ -36,55 +37,30 @@ do { \ } while (0) #endif -#ifdef CONFIG_RANDOMIZE_BASE -static unsigned long module_load_offset; +static struct execmem_params execmem_params __ro_after_init = { + .ranges = { + [EXECMEM_DEFAULT] = { + .flags = EXECMEM_KASAN_SHADOW, + .alignment = MODULE_ALIGN, + }, + }, +}; -/* Mutex protects the module_load_offset. */ -static DEFINE_MUTEX(module_kaslr_mutex); - -static unsigned long int get_module_load_offset(void) -{ - if (kaslr_enabled()) { - mutex_lock(&module_kaslr_mutex); - /* - * Calculate the module_load_offset the first time this - * code is called. Once calculated it stays the same until - * reboot. - */ - if (module_load_offset == 0) - module_load_offset = - get_random_u32_inclusive(1, 1024) * PAGE_SIZE; - mutex_unlock(&module_kaslr_mutex); - } - return module_load_offset; -} -#else -static unsigned long int get_module_load_offset(void) -{ - return 0; -} -#endif - -void *module_alloc(unsigned long size) +struct execmem_params __init *execmem_arch_params(void) { - gfp_t gfp_mask = GFP_KERNEL; - void *p; - - if (PAGE_ALIGN(size) > MODULES_LEN) - return NULL; + unsigned long module_load_offset = 0; + unsigned long start; - p = __vmalloc_node_range(size, MODULE_ALIGN, - MODULES_VADDR + get_module_load_offset(), - MODULES_END, gfp_mask, PAGE_KERNEL, - VM_FLUSH_RESET_PERMS | VM_DEFER_KMEMLEAK, - NUMA_NO_NODE, __builtin_return_address(0)); + if (IS_ENABLED(CONFIG_RANDOMIZE_BASE) && kaslr_enabled()) + module_load_offset = + get_random_u32_inclusive(1, 1024) * PAGE_SIZE; - if (p && (kasan_alloc_module_shadow(p, size, gfp_mask) < 0)) { - vfree(p); - return NULL; - } + start = MODULES_VADDR + module_load_offset; + execmem_params.ranges[EXECMEM_DEFAULT].start = start; + execmem_params.ranges[EXECMEM_DEFAULT].end = MODULES_END; + execmem_params.ranges[EXECMEM_DEFAULT].pgprot = PAGE_KERNEL; - return p; + return &execmem_params; } #ifdef CONFIG_X86_32 diff --git a/include/linux/execmem.h b/include/linux/execmem.h index 44e213625053..806ad1a0088d 100644 --- a/include/linux/execmem.h +++ b/include/linux/execmem.h @@ -32,19 +32,33 @@ enum execmem_type { EXECMEM_TYPE_MAX, }; +/** + * enum execmem_module_flags - options for executable memory allocations + * @EXECMEM_KASAN_SHADOW: allocate kasan shadow + */ +enum execmem_range_flags { + EXECMEM_KASAN_SHADOW = (1 << 0), +}; + /** * struct execmem_range - definition of a memory range suitable for code and * related data allocations * @start: address space start * @end: address space end (inclusive) + * @fallback_start: start of the range for fallback allocations + * @fallback_end: end of the range for fallback allocations (inclusive) * @pgprot: permissions for memory in this address space * @alignment: alignment required for text allocations + * @flags: options for memory allocations for this range */ struct execmem_range { unsigned long start; unsigned long end; + unsigned long fallback_start; + unsigned long fallback_end; pgprot_t pgprot; unsigned int alignment; + enum execmem_range_flags flags; }; /** diff --git a/mm/execmem.c b/mm/execmem.c index f25a5e064886..a8c2f44d0133 100644 --- a/mm/execmem.c +++ b/mm/execmem.c @@ -11,12 +11,46 @@ static void *execmem_alloc(size_t size, struct execmem_range *range) { unsigned long start = range->start; unsigned long end = range->end; + unsigned long fallback_start = range->fallback_start; + unsigned long fallback_end = range->fallback_end; unsigned int align = range->alignment; pgprot_t pgprot = range->pgprot; + bool kasan = range->flags & EXECMEM_KASAN_SHADOW; + unsigned long vm_flags = VM_FLUSH_RESET_PERMS; + bool fallback = !!fallback_start; + gfp_t gfp_flags = GFP_KERNEL; + void *p; - return __vmalloc_node_range(size, align, start, end, - GFP_KERNEL, pgprot, VM_FLUSH_RESET_PERMS, - NUMA_NO_NODE, __builtin_return_address(0)); + if (PAGE_ALIGN(size) > (end - start)) + return NULL; + + if (kasan) + vm_flags |= VM_DEFER_KMEMLEAK; + + if (fallback) + gfp_flags |= __GFP_NOWARN; + + p = __vmalloc_node_range(size, align, start, end, gfp_flags, + pgprot, vm_flags, NUMA_NO_NODE, + __builtin_return_address(0)); + + if (!p && fallback) { + start = fallback_start; + end = fallback_end; + gfp_flags = GFP_KERNEL; + + p = __vmalloc_node_range(size, align, start, end, gfp_flags, + pgprot, vm_flags, NUMA_NO_NODE, + __builtin_return_address(0)); + } + + if (p && kasan && + (kasan_alloc_module_shadow(p, size, GFP_KERNEL) < 0)) { + vfree(p); + return NULL; + } + + return kasan_reset_tag(p); } void *execmem_text_alloc(enum execmem_type type, size_t size) @@ -66,6 +100,9 @@ static void execmem_init_missing(struct execmem_params *p) r->alignment = default_range->alignment; r->start = default_range->start; r->end = default_range->end; + r->flags = default_range->flags; + r->fallback_start = default_range->fallback_start; + r->fallback_end = default_range->fallback_end; } } }