From patchwork Mon Oct 14 20:36:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13835447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECD3FD18146 for ; Mon, 14 Oct 2024 20:37:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 34A3F6B0092; Mon, 14 Oct 2024 16:37:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2D4E36B0093; Mon, 14 Oct 2024 16:37:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 14E386B0095; Mon, 14 Oct 2024 16:37:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DF0286B0092 for ; Mon, 14 Oct 2024 16:36:59 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 365951A0C10 for ; Mon, 14 Oct 2024 20:36:45 +0000 (UTC) X-FDA: 82673366706.19.C943D63 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf16.hostedemail.com (Postfix) with ESMTP id 64404180009 for ; Mon, 14 Oct 2024 20:36:51 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=VOQdMesi; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 36IANZwYKCF8PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=36IANZwYKCF8PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1728938061; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bOvroPog9pmluzu4zs2xrkfEGIpV84j+wuh+gRgw5PA=; b=gEVF64mYeaD0Kd2XvNLbnML5q24SxaysClUpCBR/Ff7L9M8J4dlFL4r9cLT5+J/lIR8Vqg df7pOoTOBwa/UxG07dvvpIAmiCmgEaHKPkS2H4OxqoApKwva0F5Tm8NnawqAjVQxdP1z7s GlVOQwMa8aCeGQOlNfjsofruza/3Cps= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1728938061; a=rsa-sha256; cv=none; b=Ff29KacgnilNzc8DnZD0TvZZj6D+cTEnJVY5Bc0RC18iZIDZ2gIO1swN9N4mNVJOpHoz7v QpjkwtGenL2dSbIiBK9U+GIgz2E1vPEhicN/ajpC9n4oJBRedghvBSqDbvMBPbZ0Eg+ySi oOiM5uWgh0gg5P/G2IXA3aq9MKS9eHY= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=VOQdMesi; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 36IANZwYKCF8PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=36IANZwYKCF8PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6e3705b2883so26817317b3.3 for ; Mon, 14 Oct 2024 13:36:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728938217; x=1729543017; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=bOvroPog9pmluzu4zs2xrkfEGIpV84j+wuh+gRgw5PA=; b=VOQdMesizs/YNQ521EcAB6zYeX7RpnLe11BvJ6gTGeHX3MvXw4WYiHczCG6GIx8yfr ZxDspMv/B1+Zq1MQYQMtKtXIQzW8cqnlWeXqKNi0O0oly936NzNP3kKe/5JDMDwNZ57V klMqxXG2FoiZPryFxzuXu5NQCbBLrGjtv4bG4n9a/QZZNodKE/MaSwkqionzusp4PDEk u8m096KVGVHluCx7GCIw+b+Aigu5aQAWCMs5Xc5llAPYco8vAN4MLSwEv0948Vim/Gjq vhF1Z2Tf7CgUpf8+MnCVw0go4vCZ4RBy5lxslXBfAk+ZhZRPksMaaUME8aQt/J/VDeAW wJWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728938217; x=1729543017; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bOvroPog9pmluzu4zs2xrkfEGIpV84j+wuh+gRgw5PA=; b=a7R/PwQgbUvjor08VBpD6ClIo684YH8BZW7UYjtAX6egvCKhC9lKDGdf9LPasz4T2Z LWFuhg+l5+4EzebU03T0n5YT0CF0i62glJqSZqLh9JQzZ8moTH4vu1Y/PrtcpxpIDIS0 DSqoe5e9tRT+hlVaLdUijrW5XT9fDGNqxEJfZJl5oQNArTeZdzVzGJWzmqCNUUn/IEYI 6BiHur6neZu7fml1jHDqRs6MPA1+SKZxI+xfCWBJgw/4zMMO+dUw2D+/1pMdPFKE5q7u aAvVi4fFCSgXgAVxX9xaciBPuUtvch+D9/pH39T2BHjLhhuLEPFjgpzFtKjrjcymH77G Zu/g== X-Forwarded-Encrypted: i=1; AJvYcCW0+h2JYwrMAAFEXC7y4wkw4nxLJA0gSL7+K8dATt8f+ol1hRTyGxh1yGaPUbhNcxHvn+KwacQTow==@kvack.org X-Gm-Message-State: AOJu0Yys2WvgAF8djuGrfKyg5sy470T5vHuB+OhQedOE2jcw8JGsH+r+ 4ZkfV1QSHOoNu8OhbjEnabGtUoc/zmsmF9ZREGAK+owd0QYGB7FCDaDYteujgt6N0Bg2Z8j1/dA 7HA== X-Google-Smtp-Source: AGHT+IHN+F4HHrAed80eKaPYhTH0JCc5Mv3vEPFTteI9kO52+1hB5IgMv4halrrXePZu5DiEH+LMAHeUbpw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:915:bdd7:e08a:7997]) (user=surenb job=sendgmr) by 2002:a05:690c:c09:b0:6e3:1702:b3e6 with SMTP id 00721157ae682-6e347b368d2mr1991517b3.4.1728938216731; Mon, 14 Oct 2024 13:36:56 -0700 (PDT) Date: Mon, 14 Oct 2024 13:36:44 -0700 In-Reply-To: <20241014203646.1952505-1-surenb@google.com> Mime-Version: 1.0 References: <20241014203646.1952505-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241014203646.1952505-4-surenb@google.com> Subject: [PATCH v3 3/5] alloc_tag: populate memory for module tags as needed From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 64404180009 X-Stat-Signature: bmjjddqdaqy39i6axcodqgueneoeghtn X-Rspam-User: X-HE-Tag: 1728938211-157190 X-HE-Meta: U2FsdGVkX1/RwzR8LBZBsutH10DF+1zVG039fsjBD1uWDGI7i8zyNHKCrmnwQKMjkLKJv8JYSZqZ5o9YAcdwYOA3u5pwsOddwu0LHEf6LVPD7hDA4tRa1G3+aezlWKBhso3vt8sQLAHLu0z4Zunacw16XTqPHBxC4hjfwHCz4bTyIK3AE5/4aWWcQ+/YOeu3f/2Avrg51kLHOKqR/0IBqKV/DjLja04KXmjAL5bpPi6Qs5oFWXdQUqsHpWwbYJiLJNPu973Ja5EICc/xy9/Gn5nPk4A2JxtYNUCRbBlFny9dvAqaj5lQbsI6pMA5va6JPTgOnrBNRRYYuyn4XAA/KOlYRe3UxAsNd764xi5Q0J6iXqZWYjP7n7+tuPH8GI4tWaM8rN0ruB1E/Q51HT64Gbg+lgfdFAii4gS8fCJv3z6cNgHQRbKLcUl7e2RHjJYl7CxR1+gAmMzOiT9BpSxPLHQBWE9BArTuw/T2lgtx8uWIhXgMB37dQN9CkkzFF1lfgYYCHT5Ah+GKx5xzsUiaAPCOVwlM9dNN48mTg5Rlzn/zhz3JGKYgSYiT8Sehvw//sdyG4yS23bqXwuzAVvmpVTWiCcrjZbPCYD8ZgOdkjXYPOUx2RXdg8R3f0psJJySmLJ/CE57cTlnyeym3d8CI7lry0DUSHpom9FSRL+htGgy6Rikw1m+UHRc3hVHslZvGbGb7qcUX7XSbu8QQS93ZX42oFvlSYOInrJPOFRJxUwfl/SZBonGcbe89BAmXG62QQ7XYuXtoyQXbrl+PgTAoKTgLG1uCcjfh41Vtq2aejKNNeHQ1nB1PYrKkcRBhxjVYJyY2lyA7+/aDPnq3rM6/7vuvmwCIf1jf0w1ADryQmvi7v3m6uRDF3PpaLyftcaGMxHse9PcUoQplPXQQOX2dviwOMhzDycUqYPTY9vx2Xcdc4MzHc7QQWcQ/r2ZQi20E+fO81j2hgp33e5Dx5lP jh/lQEl8 REKVLIa5lx4s0i/Z7lmy4O7Cgty6/s9Q0CfgILBaqTkKBxzIyaS+UAGq0NhUI3cTGVogxT6ebbMua7VzntXdrWwH1Fa95JhqkDoEAw3ptRw35r9D4aGQmH3b+B4u/jsiESHWMxm2GFqhpXdALv++IWrsSaSPZ0YKPreLvldLx7KgOlEtBbrr97kA3PBgDakAJjpjg0Mi4CkM3TyF2+C1VmkwpD6019jPmeiZG4aEQtDkPBY28yU3yqBWbXBwH4Ykg4lx+drOiSbvOdw/tPXJhReKlZCqGn5ghX/03JopKeF6FlQGj7z043LGNpnMiyA9eV86XPLoaiLJoMGga1SedhRG0eEOBvIN00BEsWITN6SlIQy75Sgnra1XZV/X8nrGAqgqncg4A0iVdoQ7wwmsmVvmFxWdOFg6UF+ScdL+ID5e0EEhnBX1k7YXUg2iGJCE2IjL436CCJTZgs8Vh4mnPZJbMGr+kpxanXAzo2dkGz6SKrA04KgbZvE9q8crd3g/UQTdsPbOxbn3QZI3FFhFY9xw05EFsVzNTDUv4gd8Jr1DYOuCQjqxG3qYRMosyftfEcGTv/LOdnUKqxXcqCh5DKbkXfw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The memory reserved for module tags does not need to be backed by physical pages until there are tags to store there. Change the way we reserve this memory to allocate only virtual area for the tags and populate it with physical pages as needed when we load a module. Signed-off-by: Suren Baghdasaryan --- include/linux/execmem.h | 11 ++++++ include/linux/vmalloc.h | 9 +++++ lib/alloc_tag.c | 84 +++++++++++++++++++++++++++++++++-------- mm/execmem.c | 16 ++++++++ mm/vmalloc.c | 4 +- 5 files changed, 106 insertions(+), 18 deletions(-) diff --git a/include/linux/execmem.h b/include/linux/execmem.h index 7436aa547818..a159a073270a 100644 --- a/include/linux/execmem.h +++ b/include/linux/execmem.h @@ -127,6 +127,17 @@ void *execmem_alloc(enum execmem_type type, size_t size); */ void execmem_free(void *ptr); +/** + * execmem_vmap - create virtual mapping for executable memory + * @type: type of the allocation + * @size: size of the virtual mapping in bytes + * + * Maps virtually contiguous area that can be populated with executable code. + * + * Return: the area descriptor on success or %NULL on failure. + */ +struct vm_struct *execmem_vmap(enum execmem_type type, size_t size); + /** * execmem_update_copy - copy an update to executable memory * @dst: destination address to update diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 9a012cd4fad2..9d64cc6f24d1 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -202,6 +202,9 @@ extern int remap_vmalloc_range_partial(struct vm_area_struct *vma, extern int remap_vmalloc_range(struct vm_area_struct *vma, void *addr, unsigned long pgoff); +int vmap_pages_range(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift); + /* * Architectures can set this mask to a combination of PGTBL_P?D_MODIFIED values * and let generic vmalloc and ioremap code know when arch_sync_kernel_mappings() @@ -239,6 +242,12 @@ extern struct vm_struct *__get_vm_area_caller(unsigned long size, unsigned long flags, unsigned long start, unsigned long end, const void *caller); +struct vm_struct *__get_vm_area_node(unsigned long size, + unsigned long align, unsigned long shift, + unsigned long flags, unsigned long start, + unsigned long end, int node, gfp_t gfp_mask, + const void *caller); + void free_vm_area(struct vm_struct *area); extern struct vm_struct *remove_vm_area(const void *addr); extern struct vm_struct *find_vm_area(const void *addr); diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index b10e7f17eeda..648f32d52b8d 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -8,6 +8,7 @@ #include #include #include +#include static struct codetag_type *alloc_tag_cttype; @@ -153,6 +154,7 @@ static void __init procfs_init(void) #ifdef CONFIG_MODULES static struct maple_tree mod_area_mt = MTREE_INIT(mod_area_mt, MT_FLAGS_ALLOC_RANGE); +static struct vm_struct *vm_module_tags; /* A dummy object used to indicate an unloaded module */ static struct module unloaded_mod; /* A dummy object used to indicate a module prepended area */ @@ -195,6 +197,25 @@ static void clean_unused_module_areas_locked(void) } } +static int vm_module_tags_grow(unsigned long addr, unsigned long bytes) +{ + struct page **next_page = vm_module_tags->pages + vm_module_tags->nr_pages; + unsigned long more_pages = ALIGN(bytes, PAGE_SIZE) >> PAGE_SHIFT; + unsigned long nr; + + nr = alloc_pages_bulk_array_node(GFP_KERNEL | __GFP_NOWARN, + NUMA_NO_NODE, more_pages, next_page); + if (nr != more_pages) + return -ENOMEM; + + vm_module_tags->nr_pages += nr; + if (vmap_pages_range(addr, addr + (nr << PAGE_SHIFT), + PAGE_KERNEL, next_page, PAGE_SHIFT) < 0) + return -ENOMEM; + + return 0; +} + static void *reserve_module_tags(struct module *mod, unsigned long size, unsigned int prepend, unsigned long align) { @@ -202,7 +223,7 @@ static void *reserve_module_tags(struct module *mod, unsigned long size, MA_STATE(mas, &mod_area_mt, 0, section_size - 1); bool cleanup_done = false; unsigned long offset; - void *ret; + void *ret = NULL; /* If no tags return NULL */ if (size < sizeof(struct alloc_tag)) @@ -239,7 +260,7 @@ static void *reserve_module_tags(struct module *mod, unsigned long size, goto repeat; } else { ret = ERR_PTR(-ENOMEM); - goto out; + goto unlock; } found: @@ -254,7 +275,7 @@ static void *reserve_module_tags(struct module *mod, unsigned long size, mas_store(&mas, &prepend_mod); if (mas_is_err(&mas)) { ret = ERR_PTR(xa_err(mas.node)); - goto out; + goto unlock; } mas.index = offset; mas.last = offset + size - 1; @@ -263,7 +284,7 @@ static void *reserve_module_tags(struct module *mod, unsigned long size, ret = ERR_PTR(xa_err(mas.node)); mas.index = pad_start; mas_erase(&mas); - goto out; + goto unlock; } } else { @@ -271,18 +292,33 @@ static void *reserve_module_tags(struct module *mod, unsigned long size, mas_store(&mas, mod); if (mas_is_err(&mas)) { ret = ERR_PTR(xa_err(mas.node)); - goto out; + goto unlock; } } +unlock: + mas_unlock(&mas); + if (IS_ERR(ret)) + return ret; - if (module_tags.size < offset + size) - module_tags.size = offset + size; + if (module_tags.size < offset + size) { + unsigned long phys_size = vm_module_tags->nr_pages << PAGE_SHIFT; - ret = (struct alloc_tag *)(module_tags.start_addr + offset); -out: - mas_unlock(&mas); + module_tags.size = offset + size; + if (phys_size < module_tags.size) { + int grow_res; + + grow_res = vm_module_tags_grow(module_tags.start_addr + phys_size, + module_tags.size - phys_size); + if (grow_res) { + static_branch_disable(&mem_alloc_profiling_key); + pr_warn("Failed to allocate tags memory for module %s. Memory profiling is disabled!\n", + mod->name); + return ERR_PTR(grow_res); + } + } + } - return ret; + return (struct alloc_tag *)(module_tags.start_addr + offset); } static void release_module_tags(struct module *mod, bool unused) @@ -351,12 +387,23 @@ static void replace_module(struct module *mod, struct module *new_mod) static int __init alloc_mod_tags_mem(void) { - /* Allocate space to copy allocation tags */ - module_tags.start_addr = (unsigned long)execmem_alloc(EXECMEM_MODULE_DATA, - MODULE_ALLOC_TAG_VMAP_SIZE); - if (!module_tags.start_addr) + /* Map space to copy allocation tags */ + vm_module_tags = execmem_vmap(EXECMEM_MODULE_DATA, MODULE_ALLOC_TAG_VMAP_SIZE); + if (!vm_module_tags) { + pr_err("Failed to map %lu bytes for module allocation tags\n", + MODULE_ALLOC_TAG_VMAP_SIZE); + module_tags.start_addr = 0; return -ENOMEM; + } + vm_module_tags->pages = kmalloc_array(get_vm_area_size(vm_module_tags) >> PAGE_SHIFT, + sizeof(struct page *), GFP_KERNEL | __GFP_ZERO); + if (!vm_module_tags->pages) { + free_vm_area(vm_module_tags); + return -ENOMEM; + } + + module_tags.start_addr = (unsigned long)vm_module_tags->addr; module_tags.end_addr = module_tags.start_addr + MODULE_ALLOC_TAG_VMAP_SIZE; return 0; @@ -364,8 +411,13 @@ static int __init alloc_mod_tags_mem(void) static void __init free_mod_tags_mem(void) { - execmem_free((void *)module_tags.start_addr); + int i; + module_tags.start_addr = 0; + for (i = 0; i < vm_module_tags->nr_pages; i++) + __free_page(vm_module_tags->pages[i]); + kfree(vm_module_tags->pages); + free_vm_area(vm_module_tags); } #else /* CONFIG_MODULES */ diff --git a/mm/execmem.c b/mm/execmem.c index 97706d8ed720..eb346f4eaaff 100644 --- a/mm/execmem.c +++ b/mm/execmem.c @@ -366,6 +366,22 @@ void execmem_free(void *ptr) vfree(ptr); } +struct vm_struct *execmem_vmap(enum execmem_type type, size_t size) +{ + struct execmem_range *range = &execmem_info->ranges[type]; + struct vm_struct *area; + + area = __get_vm_area_node(size, range->alignment, PAGE_SHIFT, VM_ALLOC, + range->start, range->end, NUMA_NO_NODE, + GFP_KERNEL, __builtin_return_address(0)); + if (!area && range->fallback_start) + area = __get_vm_area_node(size, range->alignment, PAGE_SHIFT, VM_ALLOC, + range->fallback_start, range->fallback_end, + NUMA_NO_NODE, GFP_KERNEL, __builtin_return_address(0)); + + return area; +} + void *execmem_update_copy(void *dst, const void *src, size_t size) { return text_poke_copy(dst, src, size); diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 74c0a5eae210..7ed39d104201 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -653,7 +653,7 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end, * RETURNS: * 0 on success, -errno on failure. */ -static int vmap_pages_range(unsigned long addr, unsigned long end, +int vmap_pages_range(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { int err; @@ -3106,7 +3106,7 @@ static void clear_vm_uninitialized_flag(struct vm_struct *vm) vm->flags &= ~VM_UNINITIALIZED; } -static struct vm_struct *__get_vm_area_node(unsigned long size, +struct vm_struct *__get_vm_area_node(unsigned long size, unsigned long align, unsigned long shift, unsigned long flags, unsigned long start, unsigned long end, int node, gfp_t gfp_mask, const void *caller)