From patchwork Thu Aug 18 22:42:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12948142 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4344CC00140 for ; Thu, 18 Aug 2022 23:10:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 96E518D0003; Thu, 18 Aug 2022 19:10:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8F5D18D0002; Thu, 18 Aug 2022 19:10:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 796F28D0003; Thu, 18 Aug 2022 19:10:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 635F28D0002 for ; Thu, 18 Aug 2022 19:10:43 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 411051C2E04 for ; Thu, 18 Aug 2022 23:10:43 +0000 (UTC) X-FDA: 79814259966.10.823C933 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf05.hostedemail.com (Postfix) with ESMTP id E6BB81014F0 for ; Thu, 18 Aug 2022 22:54:19 +0000 (UTC) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 27IKhkT1015187 for ; Thu, 18 Aug 2022 15:54:17 -0700 Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3j1k0p51jt-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 18 Aug 2022 15:54:16 -0700 Received: from twshared7570.37.frc1.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Thu, 18 Aug 2022 15:54:15 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id 246D5BBF66DE; Thu, 18 Aug 2022 15:42:44 -0700 (PDT) From: Song Liu To: , CC: , , , , , , , , Song Liu Subject: [RFC 4/5] vmalloc_exec: share a huge page with kernel text Date: Thu, 18 Aug 2022 15:42:17 -0700 Message-ID: <20220818224218.2399791-5-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220818224218.2399791-1-song@kernel.org> References: <20220818224218.2399791-1-song@kernel.org> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: HdmwlR4yktwATB65AtTVFefP7Czhrdpu X-Proofpoint-GUID: HdmwlR4yktwATB65AtTVFefP7Czhrdpu X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-08-18_16,2022-08-18_01,2022-06-22_01 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660863260; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RieGaWIGYAqVDoeh3TBoetqbaJ4MfEhZj2zMhc2yBZk=; b=wcUdRz8fX8vcjclCRUU0IVT12eqmWi4cZGkv+b9KACeyHPC/tUx6mJrON157xsHrhTI/V4 HaTZFl7LHKEJtYlBUj/TJUpddjOZF+PC5SEeu3pSLx/uZ540+WYQ54dIesJp+qJAj9Mg0S dn3S3wxS0zkRCeTC4bSvuuxPtG+lCok= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=kernel.org (policy=none); spf=pass (imf05.hostedemail.com: domain of "prvs=9229e92ca3=songliubraving@fb.com" designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=9229e92ca3=songliubraving@fb.com" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660863260; a=rsa-sha256; cv=none; b=xnwRnbb9GfZF8IivoAjvziiYDQv2MDxdtzLq71nvdW7Qlj6mvTVvuXgA/zdkOvAG+wis/r pAV8n50BM8XJHI2iAQDAJEBVssI0kwcGq+/45K4tMX7P0cASJ1VZ1gXxA1/ozfghoQcOAh 9rQex0v0z+e+wudmql5d9lKVXHr17nU= X-Stat-Signature: 71rd7z8j5tcxrq1mj9yeeu3jugrkiwg3 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: E6BB81014F0 Authentication-Results: imf05.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=kernel.org (policy=none); spf=pass (imf05.hostedemail.com: domain of "prvs=9229e92ca3=songliubraving@fb.com" designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=9229e92ca3=songliubraving@fb.com" X-Rspam-User: X-HE-Tag: 1660863259-702225 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On x86 kernel, we allocate 2MB pages for kernel text up to round_down(_etext, 2MB). Therefore, some of the kernel text is still on 4kB pages. With vmalloc_exec, we can allocate 2MB pages up to round_up(_etext, 2MB), and use the rest of the page for modules and BPF programs. Here is an example: [root@eth50-1 ~]# grep _etext /proc/kallsyms ffffffff82202a08 T _etext [root@eth50-1 ~]# grep bpf_prog_ /proc/kallsyms | tail -n 3 ffffffff8220f920 t bpf_prog_cc61a5364ac11d93_handle__sched_wakeup [bpf] ffffffff8220fa28 t bpf_prog_cc61a5364ac11d93_handle__sched_wakeup_new [bpf] ffffffff8220fad4 t bpf_prog_3bf73fa16f5e3d92_handle__sched_switch [bpf] [root@eth50-1 ~]# grep 0xffffffff82200000 /sys/kernel/debug/page_tables/kernel 0xffffffff82200000-0xffffffff82400000 2M ro PSE x pmd [root@eth50-1 ~]# grep xfs_flush_inodes /proc/kallsyms ffffffff822ba910 t xfs_flush_inodes_worker [xfs] ffffffff822bc580 t xfs_flush_inodes [xfs] ffffffff82200000-ffffffff82400000 is a 2MB page, serving kernel text, xfs module, and bpf programs. --- arch/x86/mm/init_64.c | 3 ++- mm/vmalloc.c | 27 +++++++++++++++++++++++++++ 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 39c5246964a9..d27d0af5beb5 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -1367,12 +1367,13 @@ int __init deferred_page_init_max_threads(const struct cpumask *node_cpumask) int kernel_set_to_readonly; +#define PMD_ALIGN(x) (((unsigned long)(x) + (PMD_SIZE - 1)) & PMD_MASK) void mark_rodata_ro(void) { unsigned long start = PFN_ALIGN(_text); unsigned long rodata_start = PFN_ALIGN(__start_rodata); unsigned long end = (unsigned long)__end_rodata_hpage_align; - unsigned long text_end = PFN_ALIGN(_etext); + unsigned long text_end = PMD_ALIGN(_etext); unsigned long rodata_end = PFN_ALIGN(__end_rodata); unsigned long all_end; diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 472287e71bf1..5f3b5df9313f 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -72,6 +72,11 @@ early_param("nohugevmalloc", set_nohugevmalloc); static const bool vmap_allow_huge = false; #endif /* CONFIG_HAVE_ARCH_HUGE_VMALLOC */ +#define PMD_ALIGN(x) (((unsigned long)(x) + (PMD_SIZE - 1)) & PMD_MASK) + +static struct vm_struct text_tail_vm; +static struct vmap_area text_tail_va; + bool is_vmalloc_addr(const void *x) { unsigned long addr = (unsigned long)kasan_reset_tag(x); @@ -634,6 +639,8 @@ int is_vmalloc_or_module_addr(const void *x) unsigned long addr = (unsigned long)kasan_reset_tag(x); if (addr >= MODULES_VADDR && addr < MODULES_END) return 1; + if (addr >= text_tail_va.va_start && addr < text_tail_va.va_end) + return 1; #endif return is_vmalloc_addr(x); } @@ -2371,6 +2378,25 @@ static void vmap_init_free_space(void) } } +static void register_text_tail_vm(void) +{ + unsigned long start = PFN_ALIGN(_etext); + unsigned long end = PMD_ALIGN(_etext); + struct vmap_area *va; + + va = kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT); + if (WARN_ON_ONCE(!va)) + return; + text_tail_vm.addr = (void *)start; + text_tail_vm.size = end - start; + text_tail_vm.flags = VM_KERNEL_EXEC; + text_tail_va.va_start = start; + text_tail_va.va_end = end; + text_tail_va.vm = &text_tail_vm; + memcpy(va, &text_tail_va, sizeof(*va)); + insert_vmap_area(va, &free_text_area_root, &free_text_area_list); +} + void __init vmalloc_init(void) { struct vmap_area *va; @@ -2381,6 +2407,7 @@ void __init vmalloc_init(void) * Create the cache for vmap_area objects. */ vmap_area_cachep = KMEM_CACHE(vmap_area, SLAB_PANIC); + register_text_tail_vm(); for_each_possible_cpu(i) { struct vmap_block_queue *vbq;