From patchwork Thu Jul 7 22:35:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12910361 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 133D1C43334 for ; Thu, 7 Jul 2022 22:36:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E1F236B0074; Thu, 7 Jul 2022 18:36:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DB4936B0075; Thu, 7 Jul 2022 18:36:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C7CA16B0078; Thu, 7 Jul 2022 18:36:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id BA3196B0074 for ; Thu, 7 Jul 2022 18:36:28 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 8907E606EB for ; Thu, 7 Jul 2022 22:36:28 +0000 (UTC) X-FDA: 79661764056.10.883D5F7 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf08.hostedemail.com (Postfix) with ESMTP id 0D35D160003 for ; Thu, 7 Jul 2022 22:36:27 +0000 (UTC) Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 267KPopV010715 for ; Thu, 7 Jul 2022 15:36:27 -0700 Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3h5ashmbgn-11 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 07 Jul 2022 15:36:26 -0700 Received: from twshared0725.22.frc3.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Thu, 7 Jul 2022 15:36:20 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id 573139D349C0; Thu, 7 Jul 2022 15:36:02 -0700 (PDT) From: Song Liu To: , , CC: , , , , , , Song Liu Subject: [PATCH v6 bpf-next 1/5] module: introduce module_alloc_huge Date: Thu, 7 Jul 2022 15:35:42 -0700 Message-ID: <20220707223546.4124919-2-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220707223546.4124919-1-song@kernel.org> References: <20220707223546.4124919-1-song@kernel.org> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: zf6sHWhklkGLAEBUtwbpQS6pRU_8HOy1 X-Proofpoint-GUID: zf6sHWhklkGLAEBUtwbpQS6pRU_8HOy1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-07-07_17,2022-06-28_01,2022-06-22_01 ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; spf=none (imf08.hostedemail.com: domain of "prvs=8187d1962c=songliubraving@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=8187d1962c=songliubraving@fb.com"; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657233388; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ly7Z8J8zibKW8V3s2/I5pWiQQJ1AX0TbNnXeWxU2ZUs=; b=KMXs4UA6hhSMGaHrQET3PLTywCHGQ2WO1lpSEf7ILG/fzjYdVy0sc3bRhVbPeDba+VITl9 JUD7uXWTMmrts27F2a8accpH2o5A3lCHrxDjCtUTRK63rMAYLM1/ne8gNmgxaNRz2I1+aY 9cB+LiIYfpttbCqWrVYsY9MH3iBMgh4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657233388; a=rsa-sha256; cv=none; b=tTUTOmEkdIkH3wELud6ZUdenGhY+B67KO3uA9I6OOfkuPN+cD0FzbcdLodBd6sSMWcrDAD lEGJPFQxAM0XUH40w+cD6woXRJJvQ7ZYztO448gJHue8SX7uT+lWiI0xFOZtBZV+nWmsXb HB7ncaAFkJ1+sBL0q0GFnSfdy5pVHvE= X-Rspam-User: X-Stat-Signature: 74quz81zwgs7yd6whdr999gchfw1e6y6 X-Rspamd-Queue-Id: 0D35D160003 Authentication-Results: imf08.hostedemail.com; dkim=none; spf=none (imf08.hostedemail.com: domain of "prvs=8187d1962c=songliubraving@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=8187d1962c=songliubraving@fb.com"; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none) X-Rspamd-Server: rspam11 X-HE-Tag: 1657233387-280208 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce module_alloc_huge, which allocates huge page backed memory in module memory space. The primary user of this memory is bpf_prog_pack (multiple BPF programs sharing a huge page). Signed-off-by: Song Liu --- arch/x86/kernel/module.c | 21 +++++++++++++++++++++ include/linux/moduleloader.h | 5 +++++ kernel/module/main.c | 8 ++++++++ 3 files changed, 34 insertions(+) diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c index b98ffcf4d250..63f6a16c70dc 100644 --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -86,6 +86,27 @@ void *module_alloc(unsigned long size) return p; } +void *module_alloc_huge(unsigned long size) +{ + gfp_t gfp_mask = GFP_KERNEL; + void *p; + + if (PAGE_ALIGN(size) > MODULES_LEN) + return NULL; + + p = __vmalloc_node_range(size, MODULE_ALIGN, + MODULES_VADDR + get_module_load_offset(), + MODULES_END, gfp_mask, PAGE_KERNEL, + VM_DEFER_KMEMLEAK | VM_ALLOW_HUGE_VMAP, + NUMA_NO_NODE, __builtin_return_address(0)); + if (p && (kasan_alloc_module_shadow(p, size, gfp_mask) < 0)) { + vfree(p); + return NULL; + } + + return p; +} + #ifdef CONFIG_X86_32 int apply_relocate(Elf32_Shdr *sechdrs, const char *strtab, diff --git a/include/linux/moduleloader.h b/include/linux/moduleloader.h index 9e09d11ffe5b..d34743a88938 100644 --- a/include/linux/moduleloader.h +++ b/include/linux/moduleloader.h @@ -26,6 +26,11 @@ unsigned int arch_mod_section_prepend(struct module *mod, unsigned int section); sections. Returns NULL on failure. */ void *module_alloc(unsigned long size); +/* Allocator used for allocating memory in module memory space. If size is + * greater than PMD_SIZE, allow using huge pages. Returns NULL on failure. + */ +void *module_alloc_huge(unsigned long size); + /* Free memory returned from module_alloc. */ void module_memfree(void *module_region); diff --git a/kernel/module/main.c b/kernel/module/main.c index fed58d30725d..349b2a8bd20f 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -1613,6 +1613,14 @@ void * __weak module_alloc(unsigned long size) NUMA_NO_NODE, __builtin_return_address(0)); } +void * __weak module_alloc_huge(unsigned long size) +{ + return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, + GFP_KERNEL, PAGE_KERNEL_EXEC, + VM_FLUSH_RESET_PERMS | VM_ALLOW_HUGE_VMAP, + NUMA_NO_NODE, __builtin_return_address(0)); +} + bool __weak module_init_section(const char *name) { return strstarts(name, ".init"); From patchwork Thu Jul 7 22:35:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12910360 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9D05CCA479 for ; Thu, 7 Jul 2022 22:36:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 701A56B0073; Thu, 7 Jul 2022 18:36:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6B0926B0074; Thu, 7 Jul 2022 18:36:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 577226B0075; Thu, 7 Jul 2022 18:36:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 458706B0073 for ; Thu, 7 Jul 2022 18:36:28 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 074C620472 for ; Thu, 7 Jul 2022 22:36:28 +0000 (UTC) X-FDA: 79661764056.24.E409F2C Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf19.hostedemail.com (Postfix) with ESMTP id 8237E1A0041 for ; Thu, 7 Jul 2022 22:36:27 +0000 (UTC) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 267KPkcO013753 for ; Thu, 7 Jul 2022 15:36:26 -0700 Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3h4uaqsjpa-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 07 Jul 2022 15:36:26 -0700 Received: from twshared30313.14.frc2.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Thu, 7 Jul 2022 15:36:24 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id 79C5B9D349C1; Thu, 7 Jul 2022 15:36:04 -0700 (PDT) From: Song Liu To: , , CC: , , , , , , Song Liu Subject: [PATCH v6 bpf-next 2/5] bpf: use module_alloc_huge for bpf_prog_pack Date: Thu, 7 Jul 2022 15:35:43 -0700 Message-ID: <20220707223546.4124919-3-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220707223546.4124919-1-song@kernel.org> References: <20220707223546.4124919-1-song@kernel.org> X-FB-Internal: Safe X-Proofpoint-GUID: g2IJoHYICHXqzHrFFtVlCQFDMghKWC1- X-Proofpoint-ORIG-GUID: g2IJoHYICHXqzHrFFtVlCQFDMghKWC1- X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-07-07_17,2022-06-28_01,2022-06-22_01 ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none); spf=none (imf19.hostedemail.com: domain of "prvs=8187d1962c=songliubraving@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=8187d1962c=songliubraving@fb.com" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657233387; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UhEi7K6fIUDHVEzwFO1fv+/2M4mJZASmumFOxQzfp8E=; b=sG6b+uKUpRLJHsTFhsYmcY7RpFh+u96Tf+NtkF2PBY0O1NzaFiyw9vgDJyeVf10Yy8QZN0 jU7kUKhdsrinFT7p4FQI3p1FDv94oB0U3mWR6XKQi0PhUMi9CBjpJzxGdhPMqnWxM/ZVvb x9AA/BCKh1f5Z45FUgFk4nrHyndlmvI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657233387; a=rsa-sha256; cv=none; b=KL0rMH9Opo5bDDwmysuxn42h//6akBI5EtsCckz+pim/+iOnmtlSW6cg3tyGsKvmz5cDsJ /f8+mP5ZeByX51vwaLPv5Nh+7bgyXDI+iFoH7EyRFVDNRu4g8W1yAEx1bTBxYJ+vFL0pDf 0XIxzGpoJHoaUtMMXj075XumI/dkSg0= X-Stat-Signature: gr85w6xh7z7mdn6qqq9owmz44z9p33eo X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 8237E1A0041 Authentication-Results: imf19.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none); spf=none (imf19.hostedemail.com: domain of "prvs=8187d1962c=songliubraving@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=8187d1962c=songliubraving@fb.com" X-Rspam-User: X-HE-Tag: 1657233387-129063 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use module_alloc_huge for bpf_prog_pack so that BPF programs sit on PMD_SIZE pages. This benefits system performance by reducing iTLB miss rate. Benchmark of a real web service workload shows this change gives another ~0.2% performance boost on top of PAGE_SIZE bpf_prog_pack (which improve system throughput by ~0.5%). Also, remove set_vm_flush_reset_perms() from alloc_new_pack() and use set_memory_[nx|rw] in bpf_prog_pack_free(). This is because VM_FLUSH_RESET_PERMS does not work with huge pages yet. [1] [1] https://lore.kernel.org/bpf/aeeeaf0b7ec63fdba55d4834d2f524d8bf05b71b.camel@intel.com/ Suggested-by: Rick Edgecombe Signed-off-by: Song Liu --- kernel/bpf/core.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 805c2ad5c793..d1f32ac354d3 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -860,7 +860,7 @@ static size_t select_bpf_prog_pack_size(void) void *ptr; size = BPF_HPAGE_SIZE * num_online_nodes(); - ptr = module_alloc(size); + ptr = module_alloc_huge(size); /* Test whether we can get huge pages. If not just use PAGE_SIZE * packs. @@ -884,7 +884,7 @@ static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_ins GFP_KERNEL); if (!pack) return NULL; - pack->ptr = module_alloc(bpf_prog_pack_size); + pack->ptr = module_alloc_huge(bpf_prog_pack_size); if (!pack->ptr) { kfree(pack); return NULL; @@ -893,7 +893,6 @@ static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_ins bitmap_zero(pack->bitmap, bpf_prog_pack_size / BPF_PROG_CHUNK_SIZE); list_add_tail(&pack->list, &pack_list); - set_vm_flush_reset_perms(pack->ptr); set_memory_ro((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE_SIZE); set_memory_x((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE_SIZE); return pack; @@ -912,10 +911,9 @@ static void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insn if (size > bpf_prog_pack_size) { size = round_up(size, PAGE_SIZE); - ptr = module_alloc(size); + ptr = module_alloc_huge(size); if (ptr) { bpf_fill_ill_insns(ptr, size); - set_vm_flush_reset_perms(ptr); set_memory_ro((unsigned long)ptr, size / PAGE_SIZE); set_memory_x((unsigned long)ptr, size / PAGE_SIZE); } @@ -952,6 +950,8 @@ static void bpf_prog_pack_free(struct bpf_binary_header *hdr) mutex_lock(&pack_mutex); if (hdr->size > bpf_prog_pack_size) { + set_memory_nx((unsigned long)hdr, hdr->size / PAGE_SIZE); + set_memory_rw((unsigned long)hdr, hdr->size / PAGE_SIZE); module_memfree(hdr); goto out; } @@ -978,6 +978,8 @@ static void bpf_prog_pack_free(struct bpf_binary_header *hdr) if (bitmap_find_next_zero_area(pack->bitmap, bpf_prog_chunk_count(), 0, bpf_prog_chunk_count(), 0) == 0) { list_del(&pack->list); + set_memory_nx((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE_SIZE); + set_memory_rw((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE_SIZE); module_memfree(pack->ptr); kfree(pack); } From patchwork Thu Jul 7 22:35:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12910362 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A802DCCA479 for ; Thu, 7 Jul 2022 22:36:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 48F9C6B0075; Thu, 7 Jul 2022 18:36:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 43F186B0078; Thu, 7 Jul 2022 18:36:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 21C916B007B; Thu, 7 Jul 2022 18:36:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1503D6B0078 for ; Thu, 7 Jul 2022 18:36:29 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id DB9C634E87 for ; Thu, 7 Jul 2022 22:36:28 +0000 (UTC) X-FDA: 79661764056.15.B4DB8F1 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by imf15.hostedemail.com (Postfix) with ESMTP id 70801A0056 for ; Thu, 7 Jul 2022 22:36:28 +0000 (UTC) Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 267KPhSB030988 for ; Thu, 7 Jul 2022 15:36:27 -0700 Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3h5nw2f4te-7 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 07 Jul 2022 15:36:27 -0700 Received: from twshared0725.22.frc3.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::f) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Thu, 7 Jul 2022 15:36:24 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id 8C1649D349C2; Thu, 7 Jul 2022 15:36:06 -0700 (PDT) From: Song Liu To: , , CC: , , , , , , Song Liu Subject: [PATCH v6 bpf-next 3/5] vmalloc: WARN for set_vm_flush_reset_perms() on huge pages Date: Thu, 7 Jul 2022 15:35:44 -0700 Message-ID: <20220707223546.4124919-4-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220707223546.4124919-1-song@kernel.org> References: <20220707223546.4124919-1-song@kernel.org> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: WJvjTnquoat1m-feBYyqX0S8Gt6_JmtT X-Proofpoint-ORIG-GUID: WJvjTnquoat1m-feBYyqX0S8Gt6_JmtT X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-07-07_17,2022-06-28_01,2022-06-22_01 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657233388; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7SpS0eTSk+dIwsOkOvbwE7M6VhiTubOfZqa+CZotE4k=; b=Pv4FyV9Y8ZvrCLetgaRqIRry1b7PRR3HT4bKMUwTDaJpq+XlM+svRAcPnN47G6J6ptXU2x EkjwBs6XuFEBWGZibIi3EHqV2Lv022BmdQqi+8hThD/AhpLuHWvuSt7fscs+yQzx2fAX65 N+0xikOR1kdelfvl7pu/oUJnWeY2zco= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657233388; a=rsa-sha256; cv=none; b=2DAiWRC3+WS0tH4QpC83QygstRJyrHGaTMUxPrK83pFRf8k5etiCbMNwD6I0zE4birWAgG WsIAWOByffsyuOKy9peu1ptGwZ/0AsXBnCxkfFXyXCbMfF4CkADYL19o/Syhwp/S1LPQ19 z6U+q6nduAFxvXaYboqjLRro3LseWxA= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none); spf=none (imf15.hostedemail.com: domain of "prvs=8187d1962c=songliubraving@fb.com" has no SPF policy when checking 67.231.153.30) smtp.mailfrom="prvs=8187d1962c=songliubraving@fb.com" Authentication-Results: imf15.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none); spf=none (imf15.hostedemail.com: domain of "prvs=8187d1962c=songliubraving@fb.com" has no SPF policy when checking 67.231.153.30) smtp.mailfrom="prvs=8187d1962c=songliubraving@fb.com" X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 70801A0056 X-Stat-Signature: g9gbkxszfnb4rnemdxx5feq7ibu1wm7h X-Rspam-User: X-HE-Tag: 1657233388-446528 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: VM_FLUSH_RESET_PERMS is not yet ready for huge pages, add a WARN to catch misuse soon. Suggested-by: Rick Edgecombe Signed-off-by: Song Liu --- include/linux/vmalloc.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 096d48aa3437..59d3e1f3e108 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -239,6 +239,7 @@ static inline void set_vm_flush_reset_perms(void *addr) { struct vm_struct *vm = find_vm_area(addr); + WARN_ON_ONCE(is_vm_area_hugepages(addr)); if (vm) vm->flags |= VM_FLUSH_RESET_PERMS; } From patchwork Thu Jul 7 22:35:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12910363 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79093CCA483 for ; Thu, 7 Jul 2022 22:36:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 52C846B0078; Thu, 7 Jul 2022 18:36:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 468476B007B; Thu, 7 Jul 2022 18:36:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 020886B007D; Thu, 7 Jul 2022 18:36:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D7D8F6B0078 for ; Thu, 7 Jul 2022 18:36:31 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B876334E5F for ; Thu, 7 Jul 2022 22:36:31 +0000 (UTC) X-FDA: 79661764182.18.9878127 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf12.hostedemail.com (Postfix) with ESMTP id 588614002C for ; Thu, 7 Jul 2022 22:36:31 +0000 (UTC) Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 267KPhtP018282 for ; Thu, 7 Jul 2022 15:36:30 -0700 Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3h5y1dccdr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 07 Jul 2022 15:36:30 -0700 Received: from twshared34609.14.frc2.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Thu, 7 Jul 2022 15:36:28 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id 40CF59D349C4; Thu, 7 Jul 2022 15:36:08 -0700 (PDT) From: Song Liu To: , , CC: , , , , , , Song Liu Subject: [PATCH v6 bpf-next 4/5] vmalloc: introduce huge_vmalloc_supported Date: Thu, 7 Jul 2022 15:35:45 -0700 Message-ID: <20220707223546.4124919-5-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220707223546.4124919-1-song@kernel.org> References: <20220707223546.4124919-1-song@kernel.org> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: ZQpGZu8-j98xsKQII3WbdBaVO45vYFGM X-Proofpoint-ORIG-GUID: ZQpGZu8-j98xsKQII3WbdBaVO45vYFGM X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-07-07_17,2022-06-28_01,2022-06-22_01 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657233391; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SjqbaNnaefnBWQiBhAxgL4uD+qvHCngWIt/mtMQVOlg=; b=HV3z+Fct+LNkdgnFUhLFRd//APHzaVdclRT1jC1wlTIWk+06Hfts/67VmWs0Z8TERaFFe2 fMaHsJZcBtfxmDl1y1OTRdTUg3hrobcUfQTJXjqQomf8ov2/tN6O2UkIhhLJvNhuEEzXE6 FdDDYNO1XUxB2Fv+RXzJZ4qn2hqoh84= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657233391; a=rsa-sha256; cv=none; b=ck5BJOCEZmypboalIUlGF37dFgyHJ/+oyP5BQ6/tzxUGa2/m8h9tZvUzXzr3IEDjdx5FVe EgXr80YfNcZLEeYyPfW5MDNCjtOyesHja4w5h2A+GxjRdlkwFG8E/0sJHMw0FlsbbyFPW7 hBLwzO1zVYictsLbIWA0sorPX9C4LMs= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=none; spf=none (imf12.hostedemail.com: domain of "prvs=8187d1962c=songliubraving@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=8187d1962c=songliubraving@fb.com"; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none) X-Rspamd-Queue-Id: 588614002C X-Rspam-User: Authentication-Results: imf12.hostedemail.com; dkim=none; spf=none (imf12.hostedemail.com: domain of "prvs=8187d1962c=songliubraving@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=8187d1962c=songliubraving@fb.com"; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none) X-Rspamd-Server: rspam03 X-Stat-Signature: 9aewwj4hywydx4otcxt8q9dgepo831dn X-HE-Tag: 1657233391-644834 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: huge_vmalloc_supported() exposes vmap_allow_huge so that users of vmalloc APIs could know whether vmalloc will return huge pages. Suggested-by: Rick Edgecombe Signed-off-by: Song Liu --- include/linux/vmalloc.h | 6 ++++++ mm/vmalloc.c | 5 +++++ 2 files changed, 11 insertions(+) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 59d3e1f3e108..aa2182959fc5 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -243,11 +243,17 @@ static inline void set_vm_flush_reset_perms(void *addr) if (vm) vm->flags |= VM_FLUSH_RESET_PERMS; } +bool huge_vmalloc_supported(void); #else static inline void set_vm_flush_reset_perms(void *addr) { } + +static inline bool huge_vmalloc_supported(void) +{ + return false; +} #endif /* for /proc/kcore */ diff --git a/mm/vmalloc.c b/mm/vmalloc.c index effd1ff6a4b4..0a5add4b5b2d 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -72,6 +72,11 @@ early_param("nohugevmalloc", set_nohugevmalloc); static const bool vmap_allow_huge = false; #endif /* CONFIG_HAVE_ARCH_HUGE_VMALLOC */ +bool huge_vmalloc_supported(void) +{ + return vmap_allow_huge; +} + bool is_vmalloc_addr(const void *x) { unsigned long addr = (unsigned long)kasan_reset_tag(x); From patchwork Thu Jul 7 22:35:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12910364 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56819C433EF for ; Thu, 7 Jul 2022 22:36:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7EB546B007B; Thu, 7 Jul 2022 18:36:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 725CC6B007D; Thu, 7 Jul 2022 18:36:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4DC036B007E; Thu, 7 Jul 2022 18:36:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 363B06B007B for ; Thu, 7 Jul 2022 18:36:34 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 18F9E606FB for ; Thu, 7 Jul 2022 22:36:34 +0000 (UTC) X-FDA: 79661764308.23.FBAA1D9 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf08.hostedemail.com (Postfix) with ESMTP id 9C792160015 for ; Thu, 7 Jul 2022 22:36:33 +0000 (UTC) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 267KPxCE000753 for ; Thu, 7 Jul 2022 15:36:32 -0700 Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3h5kgpqxk0-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 07 Jul 2022 15:36:32 -0700 Received: from twshared0725.22.frc3.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::c) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Thu, 7 Jul 2022 15:36:30 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id 4AFEB9D349C5; Thu, 7 Jul 2022 15:36:10 -0700 (PDT) From: Song Liu To: , , CC: , , , , , , Song Liu Subject: [PATCH v6 bpf-next 5/5] bpf: simplify select_bpf_prog_pack_size Date: Thu, 7 Jul 2022 15:35:46 -0700 Message-ID: <20220707223546.4124919-6-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220707223546.4124919-1-song@kernel.org> References: <20220707223546.4124919-1-song@kernel.org> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: ouni0F4mbSbavV6OFrwxstpkI4j4E0UX X-Proofpoint-GUID: ouni0F4mbSbavV6OFrwxstpkI4j4E0UX X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-07-07_17,2022-06-28_01,2022-06-22_01 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657233393; a=rsa-sha256; cv=none; b=oXCKMqK0Z65ze8Ez9zsFW+mecFgKL6Y1mU1z65XbLlUTS3ixBAnBtVKecccTPJ4tSWu78k CP+Wp/+yEQZQn1F6QtPVPcalV0RhjvenJbKyhPUwNg8ThB0/GGLxGXoDa7H+NvlRVqZQHc 2c4MCfkEtASFiea73QbUX+35V6KqS80= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; spf=none (imf08.hostedemail.com: domain of "prvs=8187d1962c=songliubraving@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=8187d1962c=songliubraving@fb.com"; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657233393; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=arf+ZE5uvAQHwO/XYDInL6YozP4LMI1hCPKixw7SNiQ=; b=54QNOXIjXLGqKx46UR7tRIllmK7cyVGgdwchwdYmAZHsYlMEvoDWQJWoSgy/Zq2F1fGmCg dI+Xd/PNZw4H0lAIif+I1JJ2jlZZv+yne5M3yfNn5dBBA/H4m+PKxITkVr/biGguo1IlSN je6RHrR6mhxQi2EWo9Bh2M2ko21OQAE= X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 9C792160015 Authentication-Results: imf08.hostedemail.com; dkim=none; spf=none (imf08.hostedemail.com: domain of "prvs=8187d1962c=songliubraving@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=8187d1962c=songliubraving@fb.com"; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none) X-Stat-Signature: b6hpys7wzmnhh94cuh13kwgzrofw7dp5 X-Rspam-User: X-HE-Tag: 1657233393-499750 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use huge_vmalloc_supported to simplify select_bpf_prog_pack_size, so that we don't allocate some huge pages and free them immediately. Suggested-by: Rick Edgecombe Signed-off-by: Song Liu --- kernel/bpf/core.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index d1f32ac354d3..e1f8d36fb95c 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -857,22 +857,15 @@ static LIST_HEAD(pack_list); static size_t select_bpf_prog_pack_size(void) { size_t size; - void *ptr; - - size = BPF_HPAGE_SIZE * num_online_nodes(); - ptr = module_alloc_huge(size); - /* Test whether we can get huge pages. If not just use PAGE_SIZE - * packs. - */ - if (!ptr || !is_vm_area_hugepages(ptr)) { + if (huge_vmalloc_supported()) { + size = BPF_HPAGE_SIZE * num_online_nodes(); + bpf_prog_pack_mask = BPF_HPAGE_MASK; + } else { size = PAGE_SIZE; bpf_prog_pack_mask = PAGE_MASK; - } else { - bpf_prog_pack_mask = BPF_HPAGE_MASK; } - vfree(ptr); return size; }