From patchwork Fri Apr 22 05:18:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12822748 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3293AC433EF for ; Fri, 22 Apr 2022 05:24:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C9FA86B0075; Fri, 22 Apr 2022 01:24:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C4F906B0078; Fri, 22 Apr 2022 01:24:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B65726B007B; Fri, 22 Apr 2022 01:24:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.26]) by kanga.kvack.org (Postfix) with ESMTP id A6B0A6B0075 for ; Fri, 22 Apr 2022 01:24:37 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 522A6923 for ; Fri, 22 Apr 2022 05:24:33 +0000 (UTC) X-FDA: 79383374826.08.D581308 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf04.hostedemail.com (Postfix) with ESMTP id 9FF794001A for ; Fri, 22 Apr 2022 05:24:30 +0000 (UTC) Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 23LNWiuG017164 for ; Thu, 21 Apr 2022 22:24:31 -0700 Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3fjtd41jad-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 21 Apr 2022 22:24:31 -0700 Received: from twshared13345.18.frc3.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:11d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Thu, 21 Apr 2022 22:24:30 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id D979360E40E5; Thu, 21 Apr 2022 22:18:18 -0700 (PDT) From: Song Liu To: , , CC: , , , , , , , , Song Liu Subject: [PATCH bpf 1/4] bpf: invalidate unused part of bpf_prog_pack Date: Thu, 21 Apr 2022 22:18:10 -0700 Message-ID: <20220422051813.1989257-2-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220422051813.1989257-1-song@kernel.org> References: <20220422051813.1989257-1-song@kernel.org> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: hkpjtgRcHCoxcKeu5X_bEAAwSVyYfbdb X-Proofpoint-GUID: hkpjtgRcHCoxcKeu5X_bEAAwSVyYfbdb X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.858,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-04-22_01,2022-04-21_01,2022-02-23_01 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 9FF794001A X-Rspam-User: Authentication-Results: imf04.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none); spf=none (imf04.hostedemail.com: domain of "prvs=5111fe83e2=songliubraving@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=5111fe83e2=songliubraving@fb.com" X-Stat-Signature: bfyke8bbsyrpdjorjw9fmsf4iqsm4h94 X-HE-Tag: 1650605070-968616 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: bpf_prog_pack enables sharing huge pages among multiple BPF programs. These pages are marked as executable, but some part of these huge page may not contain proper BPF programs. To make these pages safe, fill such unused part of these pages with invalid instructions. This is done when a pack is first allocated, and when a bpf program is freed.. Fixes: 57631054fae6 ("bpf: Introduce bpf_prog_pack allocator") Fixes: 33c9805860e5 ("bpf: Introduce bpf_jit_binary_pack_[alloc|finalize|free]") Signed-off-by: Song Liu --- arch/x86/net/bpf_jit_comp.c | 22 ++++++++++++++++++++++ include/linux/bpf.h | 2 ++ kernel/bpf/core.c | 21 +++++++++++++++++---- 3 files changed, 41 insertions(+), 4 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 16b6efacf7c6..ff3f0811c9a1 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -228,6 +228,28 @@ static void jit_fill_hole(void *area, unsigned int size) memset(area, 0xcc, size); } +#define INVALID_BUF_SIZE PAGE_SIZE +static char invalid_insn_buf[INVALID_BUF_SIZE]; + +static int __init bpf_init_invalid_insn_buf(void) +{ + jit_fill_hole(invalid_insn_buf, INVALID_BUF_SIZE); + return 0; +} +pure_initcall(bpf_init_invalid_insn_buf); + +void bpf_arch_invalidate_text(void *dst, size_t len) +{ + size_t i = 0; + + while (i < len) { + size_t s = min_t(size_t, len - i, INVALID_BUF_SIZE); + + bpf_arch_text_copy(dst + i, invalid_insn_buf, s); + i += s; + } +} + struct jit_context { int cleanup_addr; /* Epilogue code offset */ diff --git a/include/linux/bpf.h b/include/linux/bpf.h index bdb5298735ce..2157392a94d1 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -2382,6 +2382,8 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, void *bpf_arch_text_copy(void *dst, void *src, size_t len); +void bpf_arch_invalidate_text(void *dst, size_t len); + struct btf_id_set; bool btf_id_set_contains(const struct btf_id_set *set, u32 id); diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 13e9dbeeedf3..2be2d73fb6a8 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -873,7 +873,7 @@ static size_t select_bpf_prog_pack_size(void) return size; } -static struct bpf_prog_pack *alloc_new_pack(void) +static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_insns) { struct bpf_prog_pack *pack; @@ -886,6 +886,7 @@ static struct bpf_prog_pack *alloc_new_pack(void) kfree(pack); return NULL; } + bpf_fill_ill_insns(pack->ptr, bpf_prog_pack_size); bitmap_zero(pack->bitmap, bpf_prog_pack_size / BPF_PROG_CHUNK_SIZE); list_add_tail(&pack->list, &pack_list); @@ -895,7 +896,7 @@ static struct bpf_prog_pack *alloc_new_pack(void) return pack; } -static void *bpf_prog_pack_alloc(u32 size) +static void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns) { unsigned int nbits = BPF_PROG_SIZE_TO_NBITS(size); struct bpf_prog_pack *pack; @@ -910,6 +911,7 @@ static void *bpf_prog_pack_alloc(u32 size) size = round_up(size, PAGE_SIZE); ptr = module_alloc(size); if (ptr) { + bpf_fill_ill_insns(ptr, size); set_vm_flush_reset_perms(ptr); set_memory_ro((unsigned long)ptr, size / PAGE_SIZE); set_memory_x((unsigned long)ptr, size / PAGE_SIZE); @@ -923,7 +925,7 @@ static void *bpf_prog_pack_alloc(u32 size) goto found_free_area; } - pack = alloc_new_pack(); + pack = alloc_new_pack(bpf_fill_ill_insns); if (!pack) goto out; @@ -966,6 +968,7 @@ static void bpf_prog_pack_free(struct bpf_binary_header *hdr) nbits = BPF_PROG_SIZE_TO_NBITS(hdr->size); pos = ((unsigned long)hdr - (unsigned long)pack_ptr) >> BPF_PROG_CHUNK_SHIFT; + bpf_arch_invalidate_text(hdr, hdr->size); bitmap_clear(pack->bitmap, pos, nbits); if (bitmap_find_next_zero_area(pack->bitmap, bpf_prog_chunk_count(), 0, bpf_prog_chunk_count(), 0) == 0) { @@ -1102,7 +1105,7 @@ bpf_jit_binary_pack_alloc(unsigned int proglen, u8 **image_ptr, if (bpf_jit_charge_modmem(size)) return NULL; - ro_header = bpf_prog_pack_alloc(size); + ro_header = bpf_prog_pack_alloc(size, bpf_fill_ill_insns); if (!ro_header) { bpf_jit_uncharge_modmem(size); return NULL; @@ -1203,6 +1206,16 @@ void __weak bpf_jit_free(struct bpf_prog *fp) bpf_prog_unlock_free(fp); } +void __weak bpf_arch_invalidate_text(void *dst, size_t len) +{ + char buf[1] = {}; + int i; + + WARN_ONCE(1, "Please override %s for bpf_prog_pack\n", __func__); + for (i = 0; i < len; i++) + bpf_arch_text_copy(dst + i, buf, 1); +} + int bpf_jit_get_func_addr(const struct bpf_prog *prog, const struct bpf_insn *insn, bool extra_pass, u64 *func_addr, bool *func_addr_fixed) From patchwork Fri Apr 22 05:18:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12822754 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64F68C433F5 for ; Fri, 22 Apr 2022 05:35:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C1E586B0072; Fri, 22 Apr 2022 01:35:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BA7F16B0073; Fri, 22 Apr 2022 01:35:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A6E696B0074; Fri, 22 Apr 2022 01:35:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.25]) by kanga.kvack.org (Postfix) with ESMTP id 935356B0072 for ; Fri, 22 Apr 2022 01:35:23 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 61EE924357 for ; Fri, 22 Apr 2022 05:35:23 +0000 (UTC) X-FDA: 79383402126.09.98DA2FB Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by imf04.hostedemail.com (Postfix) with ESMTP id D8C084001A for ; Fri, 22 Apr 2022 05:35:20 +0000 (UTC) Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.16.1.2/8.16.1.2) with ESMTP id 23LNWhv2025707 for ; Thu, 21 Apr 2022 22:35:22 -0700 Received: from maileast.thefacebook.com ([163.114.130.16]) by m0089730.ppops.net (PPS) with ESMTPS id 3fjvsw0jhf-7 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 21 Apr 2022 22:35:22 -0700 Received: from twshared13345.18.frc3.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Thu, 21 Apr 2022 22:35:19 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id 9235C60E40EE; Thu, 21 Apr 2022 22:18:21 -0700 (PDT) From: Song Liu To: , , CC: , , , , , , , , Song Liu , Christoph Hellwig Subject: [PATCH bpf 2/4] page_alloc: use vmalloc_huge for large system hash Date: Thu, 21 Apr 2022 22:18:11 -0700 Message-ID: <20220422051813.1989257-3-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220422051813.1989257-1-song@kernel.org> References: <20220422051813.1989257-1-song@kernel.org> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: FA8v_BmaTJpVW-C_TPFMEknrrAd1zAyN X-Proofpoint-ORIG-GUID: FA8v_BmaTJpVW-C_TPFMEknrrAd1zAyN X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.858,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-04-22_01,2022-04-21_01,2022-02-23_01 X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: D8C084001A X-Stat-Signature: dbscorse7k14qzkyd491s8qtc333i11e Authentication-Results: imf04.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none); spf=none (imf04.hostedemail.com: domain of "prvs=5111fe83e2=songliubraving@fb.com" has no SPF policy when checking 67.231.153.30) smtp.mailfrom="prvs=5111fe83e2=songliubraving@fb.com" X-HE-Tag: 1650605720-558523 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use vmalloc_huge() in alloc_large_system_hash() so that large system hash (>= PMD_SIZE) could benefit from huge pages. Note that vmalloc_huge only allocates huge pages for systems with HAVE_ARCH_HUGE_VMALLOC. Reviewed-by: Christoph Hellwig Signed-off-by: Song Liu Reported-by: kernel test robot --- mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 33ca8cab21e6..0e42038382c1 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -8919,7 +8919,7 @@ void *__init alloc_large_system_hash(const char *tablename, table = memblock_alloc_raw(size, SMP_CACHE_BYTES); } else if (get_order(size) >= MAX_ORDER || hashdist) { - table = __vmalloc(size, gfp_flags); + table = vmalloc_huge(size, gfp_flags); virt = true; if (table) huge = is_vm_area_hugepages(table); From patchwork Fri Apr 22 05:18:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12822747 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8ABA1C433F5 for ; Fri, 22 Apr 2022 05:24:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 06F916B0074; Fri, 22 Apr 2022 01:24:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EC0AD6B0075; Fri, 22 Apr 2022 01:24:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D61E56B0078; Fri, 22 Apr 2022 01:24:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.a.hostedemail.com [64.99.140.24]) by kanga.kvack.org (Postfix) with ESMTP id C2DA96B0074 for ; Fri, 22 Apr 2022 01:24:24 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 8F02C2097E for ; Fri, 22 Apr 2022 05:24:24 +0000 (UTC) X-FDA: 79383374448.15.29D35D4 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by imf29.hostedemail.com (Postfix) with ESMTP id D245F12001D for ; Fri, 22 Apr 2022 05:24:22 +0000 (UTC) Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 23LNWm7a004479 for ; Thu, 21 Apr 2022 22:24:23 -0700 Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3fjsrk1snn-7 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 21 Apr 2022 22:24:23 -0700 Received: from twshared19572.14.frc2.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Thu, 21 Apr 2022 22:24:21 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id 4B0A260E40F8; Thu, 21 Apr 2022 22:18:23 -0700 (PDT) From: Song Liu To: , , CC: , , , , , , , , Song Liu Subject: [PATCH bpf 3/4] module: introduce module_alloc_huge Date: Thu, 21 Apr 2022 22:18:12 -0700 Message-ID: <20220422051813.1989257-4-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220422051813.1989257-1-song@kernel.org> References: <20220422051813.1989257-1-song@kernel.org> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: VgQqH2N316QSsEgTWMMhk226ee3VPkLP X-Proofpoint-GUID: VgQqH2N316QSsEgTWMMhk226ee3VPkLP X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.858,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-04-22_01,2022-04-21_01,2022-02-23_01 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: D245F12001D X-Rspam-User: Authentication-Results: imf29.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none); spf=none (imf29.hostedemail.com: domain of "prvs=5111fe83e2=songliubraving@fb.com" has no SPF policy when checking 67.231.153.30) smtp.mailfrom="prvs=5111fe83e2=songliubraving@fb.com" X-Stat-Signature: zy5c6g8fbboddca54j5k7mjb4idwsmna X-HE-Tag: 1650605062-573870 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce module_alloc_huge, which allocates huge page backed memory in module memory space. The primary user of this memory is bpf_prog_pack (multiple BPF programs sharing a huge page). Signed-off-by: Song Liu Reported-by: kernel test robot --- arch/x86/kernel/module.c | 21 +++++++++++++++++++++ include/linux/moduleloader.h | 5 +++++ kernel/module.c | 8 ++++++++ 3 files changed, 34 insertions(+) diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c index b98ffcf4d250..63f6a16c70dc 100644 --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -86,6 +86,27 @@ void *module_alloc(unsigned long size) return p; } +void *module_alloc_huge(unsigned long size) +{ + gfp_t gfp_mask = GFP_KERNEL; + void *p; + + if (PAGE_ALIGN(size) > MODULES_LEN) + return NULL; + + p = __vmalloc_node_range(size, MODULE_ALIGN, + MODULES_VADDR + get_module_load_offset(), + MODULES_END, gfp_mask, PAGE_KERNEL, + VM_DEFER_KMEMLEAK | VM_ALLOW_HUGE_VMAP, + NUMA_NO_NODE, __builtin_return_address(0)); + if (p && (kasan_alloc_module_shadow(p, size, gfp_mask) < 0)) { + vfree(p); + return NULL; + } + + return p; +} + #ifdef CONFIG_X86_32 int apply_relocate(Elf32_Shdr *sechdrs, const char *strtab, diff --git a/include/linux/moduleloader.h b/include/linux/moduleloader.h index 9e09d11ffe5b..d34743a88938 100644 --- a/include/linux/moduleloader.h +++ b/include/linux/moduleloader.h @@ -26,6 +26,11 @@ unsigned int arch_mod_section_prepend(struct module *mod, unsigned int section); sections. Returns NULL on failure. */ void *module_alloc(unsigned long size); +/* Allocator used for allocating memory in module memory space. If size is + * greater than PMD_SIZE, allow using huge pages. Returns NULL on failure. + */ +void *module_alloc_huge(unsigned long size); + /* Free memory returned from module_alloc. */ void module_memfree(void *module_region); diff --git a/kernel/module.c b/kernel/module.c index 6cea788fd965..2af20ac3209c 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -2839,6 +2839,14 @@ void * __weak module_alloc(unsigned long size) NUMA_NO_NODE, __builtin_return_address(0)); } +void * __weak module_alloc_huge(unsigned long size) +{ + return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, + GFP_KERNEL, PAGE_KERNEL_EXEC, + VM_FLUSH_RESET_PERMS | VM_ALLOW_HUGE_VMAP, + NUMA_NO_NODE, __builtin_return_address(0)); +} + bool __weak module_init_section(const char *name) { return strstarts(name, ".init"); From patchwork Fri Apr 22 05:18:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12822746 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53071C433EF for ; Fri, 22 Apr 2022 05:24:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BE1EF6B0073; Fri, 22 Apr 2022 01:24:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B8C356B0074; Fri, 22 Apr 2022 01:24:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A7C7D6B0075; Fri, 22 Apr 2022 01:24:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.a.hostedemail.com [64.99.140.24]) by kanga.kvack.org (Postfix) with ESMTP id 9A1F86B0073 for ; Fri, 22 Apr 2022 01:24:23 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 71C2C209DA for ; Fri, 22 Apr 2022 05:24:23 +0000 (UTC) X-FDA: 79383374406.01.9A335EC Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf04.hostedemail.com (Postfix) with ESMTP id B298F40018 for ; Fri, 22 Apr 2022 05:24:20 +0000 (UTC) Received: from pps.filterd (m0109333.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 23M0Zwa7009310 for ; Thu, 21 Apr 2022 22:24:21 -0700 Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3fkhy711r5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 21 Apr 2022 22:24:21 -0700 Received: from twshared10896.25.frc3.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:11d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Thu, 21 Apr 2022 22:24:20 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id 6169560E4108; Thu, 21 Apr 2022 22:18:24 -0700 (PDT) From: Song Liu To: , , CC: , , , , , , , , Song Liu Subject: [PATCH bpf 4/4] bpf: use module_alloc_huge for bpf_prog_pack Date: Thu, 21 Apr 2022 22:18:13 -0700 Message-ID: <20220422051813.1989257-5-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220422051813.1989257-1-song@kernel.org> References: <20220422051813.1989257-1-song@kernel.org> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: _3EQWLkj5WSwtMV_amc41Bm7oDg2iQG7 X-Proofpoint-ORIG-GUID: _3EQWLkj5WSwtMV_amc41Bm7oDg2iQG7 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.858,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-04-22_01,2022-04-21_01,2022-02-23_01 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: B298F40018 X-Rspam-User: Authentication-Results: imf04.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none); spf=none (imf04.hostedemail.com: domain of "prvs=5111fe83e2=songliubraving@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=5111fe83e2=songliubraving@fb.com" X-Stat-Signature: meujf1fnygg7sddie91xce8jo1onchn7 X-HE-Tag: 1650605060-341405 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use module_alloc_huge for bpf_prog_pack so that BPF programs sit on PMD_SIZE pages. This benefits system performance by reducing iTLB miss rate. Also, remove set_vm_flush_reset_perms() from alloc_new_pack() and use set_memory_[nx|rw] in bpf_prog_pack_free(). This is because VM_FLUSH_RESET_PERMS does not work with huge pages yet. Signed-off-by: Song Liu --- kernel/bpf/core.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 2be2d73fb6a8..9acfa8e21fda 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -857,7 +857,7 @@ static size_t select_bpf_prog_pack_size(void) void *ptr; size = BPF_HPAGE_SIZE * num_online_nodes(); - ptr = module_alloc(size); + ptr = module_alloc_huge(size); /* Test whether we can get huge pages. If not just use PAGE_SIZE * packs. @@ -881,7 +881,7 @@ static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_ins GFP_KERNEL); if (!pack) return NULL; - pack->ptr = module_alloc(bpf_prog_pack_size); + pack->ptr = module_alloc_huge(bpf_prog_pack_size); if (!pack->ptr) { kfree(pack); return NULL; @@ -890,7 +890,6 @@ static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_ins bitmap_zero(pack->bitmap, bpf_prog_pack_size / BPF_PROG_CHUNK_SIZE); list_add_tail(&pack->list, &pack_list); - set_vm_flush_reset_perms(pack->ptr); set_memory_ro((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE_SIZE); set_memory_x((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE_SIZE); return pack; @@ -973,6 +972,8 @@ static void bpf_prog_pack_free(struct bpf_binary_header *hdr) if (bitmap_find_next_zero_area(pack->bitmap, bpf_prog_chunk_count(), 0, bpf_prog_chunk_count(), 0) == 0) { list_del(&pack->list); + set_memory_nx((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE_SIZE); + set_memory_rw((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE_SIZE); module_memfree(pack->ptr); kfree(pack); }