From patchwork Wed Mar 30 22:56:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12796448 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F253C433EF for ; Wed, 30 Mar 2022 23:02:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CD1B96B0074; Wed, 30 Mar 2022 19:02:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C7FC38D0002; Wed, 30 Mar 2022 19:02:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B48548D0001; Wed, 30 Mar 2022 19:02:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0037.hostedemail.com [216.40.44.37]) by kanga.kvack.org (Postfix) with ESMTP id A66966B0074 for ; Wed, 30 Mar 2022 19:02:34 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 6D307A5BCC for ; Wed, 30 Mar 2022 23:02:34 +0000 (UTC) X-FDA: 79302578628.30.8148A0D Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf16.hostedemail.com (Postfix) with ESMTP id C4618180009 for ; Wed, 30 Mar 2022 23:02:33 +0000 (UTC) Received: from pps.filterd (m0109333.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 22UHKY4x006003 for ; Wed, 30 Mar 2022 16:02:32 -0700 Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3f4r98454c-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 30 Mar 2022 16:02:32 -0700 Received: from twshared10896.25.frc3.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:11d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Wed, 30 Mar 2022 16:02:31 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id 668A34764A9D; Wed, 30 Mar 2022 15:56:48 -0700 (PDT) From: Song Liu To: , , , CC: , , , , , , , Song Liu Subject: [PATCH bpf 1/4] x86: disable HAVE_ARCH_HUGE_VMALLOC Date: Wed, 30 Mar 2022 15:56:39 -0700 Message-ID: <20220330225642.1163897-2-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220330225642.1163897-1-song@kernel.org> References: <20220330225642.1163897-1-song@kernel.org> X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: nYd_LmjH4_SJMNlia4vjW4r7GFsu6ptd X-Proofpoint-GUID: nYd_LmjH4_SJMNlia4vjW4r7GFsu6ptd X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.850,Hydra:6.0.425,FMLib:17.11.64.514 definitions=2022-03-30_06,2022-03-30_01,2022-02-23_01 X-Stat-Signature: suinoqfhuyjg4fsna684uxy5bp4fapk3 Authentication-Results: imf16.hostedemail.com; dkim=none; spf=none (imf16.hostedemail.com: domain of "prvs=50888507eb=songliubraving@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=50888507eb=songliubraving@fb.com"; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none) X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: C4618180009 X-HE-Tag: 1648681353-461618 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We cannot savely enable HAVE_ARCH_HUGE_VMALLOC for X86_64 yet. See [1] and [2] for more details. Disable it for now. [1] https://lore.kernel.org/lkml/5bd16e2c06a2df357400556c6ae01bb5d3c5c32a.camel@intel.com/ [2] https://lore.kernel.org/netdev/14444103-d51b-0fb3-ee63-c3f182f0b546@molgen.mpg.de/ Fixes: fac54e2bfb5b ("x86/Kconfig: Select HAVE_ARCH_HUGE_VMALLOC with HAVE_ARCH_HUGE_VMAP") Signed-off-by: Song Liu --- arch/x86/Kconfig | 1 - 1 file changed, 1 deletion(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index ff45a27fc29c..4691d3aef681 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -161,7 +161,6 @@ config X86 select HAVE_ALIGNED_STRUCT_PAGE if SLUB select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMAP if X86_64 || X86_PAE - select HAVE_ARCH_HUGE_VMALLOC if X86_64 select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if X86_64 From patchwork Wed Mar 30 22:56:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12796447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A28BC433F5 for ; Wed, 30 Mar 2022 23:02:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 35A9C6B0073; Wed, 30 Mar 2022 19:02:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2E2F88D0002; Wed, 30 Mar 2022 19:02:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 183F18D0001; Wed, 30 Mar 2022 19:02:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0118.hostedemail.com [216.40.44.118]) by kanga.kvack.org (Postfix) with ESMTP id 0C61D6B0073 for ; Wed, 30 Mar 2022 19:02:27 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id C09F0182EE046 for ; Wed, 30 Mar 2022 23:02:26 +0000 (UTC) X-FDA: 79302578292.30.1EFBE38 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by imf09.hostedemail.com (Postfix) with ESMTP id 3E059140016 for ; Wed, 30 Mar 2022 23:02:26 +0000 (UTC) Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 22UHUoVI022848 for ; Wed, 30 Mar 2022 16:02:25 -0700 Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3f4unuaa65-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 30 Mar 2022 16:02:25 -0700 Received: from twshared29473.14.frc2.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:21d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Wed, 30 Mar 2022 16:02:23 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id CE4F34764AAC; Wed, 30 Mar 2022 15:56:51 -0700 (PDT) From: Song Liu To: , , , CC: , , , , , , , Song Liu Subject: [PATCH bpf 2/4] vmalloc: introduce HAVE_ARCH_HUGE_VMALLOC_FLAG Date: Wed, 30 Mar 2022 15:56:40 -0700 Message-ID: <20220330225642.1163897-3-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220330225642.1163897-1-song@kernel.org> References: <20220330225642.1163897-1-song@kernel.org> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: jM08Mzf0VNFuEJVFOinvMhqkabcaqHA3 X-Proofpoint-GUID: jM08Mzf0VNFuEJVFOinvMhqkabcaqHA3 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.850,Hydra:6.0.425,FMLib:17.11.64.514 definitions=2022-03-30_06,2022-03-30_01,2022-02-23_01 X-Rspam-User: Authentication-Results: imf09.hostedemail.com; dkim=none; spf=none (imf09.hostedemail.com: domain of "prvs=50888507eb=songliubraving@fb.com" has no SPF policy when checking 67.231.153.30) smtp.mailfrom="prvs=50888507eb=songliubraving@fb.com"; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none) X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 3E059140016 X-Stat-Signature: n7698puo3erc6ug18md396smqcf39xrm X-HE-Tag: 1648681346-655815 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With HAVE_ARCH_HUGE_VMALLOC_FLAG, users of __vmalloc_node_range() could use VM_TRY_HUGE_VMAP to (try to) allocate PMD_SIZE pages for size >= PMD_SIZE cases. Similar to HAVE_ARCH_HUGE_VMALLOC, the use can disable huge page by specifying nohugeiomap in kernel command line. The first user of VM_TRY_HUGE_VMAP will be bpf_prog_pack. Signed-off-by: Song Liu --- arch/Kconfig | 9 +++++++++ include/linux/vmalloc.h | 9 +++++++-- mm/vmalloc.c | 28 +++++++++++++++++++--------- 3 files changed, 35 insertions(+), 11 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 33e06966f248..23b6e92aebaa 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -864,6 +864,15 @@ config HAVE_ARCH_HUGE_VMALLOC depends on HAVE_ARCH_HUGE_VMAP bool +# +# HAVE_ARCH_HUGE_VMALLOC_FLAG allows users of __vmalloc_node_range to allocate +# huge page without HAVE_ARCH_HUGE_VMALLOC. To allocate huge pages, the user +# need to call __vmalloc_node_range with VM_TRY_HUGE_VMAP. +# +config HAVE_ARCH_HUGE_VMALLOC_FLAG + depends on HAVE_ARCH_HUGE_VMAP + bool + config ARCH_WANT_HUGE_PMD_SHARE bool diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 3b1df7da402d..a48d0690b66f 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -35,6 +35,11 @@ struct notifier_block; /* in notifier.h */ #define VM_DEFER_KMEMLEAK 0 #endif +#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC_FLAG +#define VM_TRY_HUGE_VMAP 0x00001000 /* Allow for huge pages on HAVE_ARCH_HUGE_VMALLOC_FLAG arch's */ +#else +#define VM_TRY_HUGE_VMAP 0 +#endif /* bits [20..32] reserved for arch specific ioremap internals */ /* @@ -51,7 +56,7 @@ struct vm_struct { unsigned long size; unsigned long flags; struct page **pages; -#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC +#if (defined(CONFIG_HAVE_ARCH_HUGE_VMALLOC) || defined(CONFIG_HAVE_ARCH_HUGE_VMALLOC_FLAG)) unsigned int page_order; #endif unsigned int nr_pages; @@ -225,7 +230,7 @@ static inline bool is_vm_area_hugepages(const void *addr) * prevents that. This only indicates the size of the physical page * allocated in the vmalloc layer. */ -#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC +#if (defined(CONFIG_HAVE_ARCH_HUGE_VMALLOC) || defined(CONFIG_HAVE_ARCH_HUGE_VMALLOC_FLAG)) return find_vm_area(addr)->page_order > 0; #else return false; diff --git a/mm/vmalloc.c b/mm/vmalloc.c index e163372d3967..179200bce285 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -46,7 +46,7 @@ #include "internal.h" #include "pgalloc-track.h" -#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +#if (defined(CONFIG_HAVE_ARCH_HUGE_VMALLOC) || defined(CONFIG_HAVE_ARCH_HUGE_VMALLOC_FLAG)) static unsigned int __ro_after_init ioremap_max_page_shift = BITS_PER_LONG - 1; static int __init set_nohugeiomap(char *str) @@ -55,11 +55,11 @@ static int __init set_nohugeiomap(char *str) return 0; } early_param("nohugeiomap", set_nohugeiomap); -#else /* CONFIG_HAVE_ARCH_HUGE_VMAP */ +#else /* CONFIG_HAVE_ARCH_HUGE_VMAP || CONFIG_HAVE_ARCH_HUGE_VMALLOC_FLAG */ static const unsigned int ioremap_max_page_shift = PAGE_SHIFT; -#endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ +#endif /* CONFIG_HAVE_ARCH_HUGE_VMAP || CONFIG_HAVE_ARCH_HUGE_VMALLOC_FLAG*/ -#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC +#if (defined(CONFIG_HAVE_ARCH_HUGE_VMALLOC) || defined(CONFIG_HAVE_ARCH_HUGE_VMALLOC_FLAG)) static bool __ro_after_init vmap_allow_huge = true; static int __init set_nohugevmalloc(char *str) @@ -582,8 +582,9 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end, WARN_ON(page_shift < PAGE_SHIFT); - if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) || - page_shift == PAGE_SHIFT) + if ((!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) && + !IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC_FLAG)) || + (page_shift == PAGE_SHIFT)) return vmap_small_pages_range_noflush(addr, end, prot, pages); for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) { @@ -2252,7 +2253,7 @@ static struct vm_struct *vmlist __initdata; static inline unsigned int vm_area_page_order(struct vm_struct *vm) { -#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC +#if (defined(CONFIG_HAVE_ARCH_HUGE_VMALLOC) || defined(CONFIG_HAVE_ARCH_HUGE_VMALLOC_FLAG)) return vm->page_order; #else return 0; @@ -2261,7 +2262,7 @@ static inline unsigned int vm_area_page_order(struct vm_struct *vm) static inline void set_vm_area_page_order(struct vm_struct *vm, unsigned int order) { -#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC +#if (defined(CONFIG_HAVE_ARCH_HUGE_VMALLOC) || defined(CONFIG_HAVE_ARCH_HUGE_VMALLOC_FLAG)) vm->page_order = order; #else BUG_ON(order != 0); @@ -3056,6 +3057,15 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, return NULL; } +static bool vmalloc_try_huge_page(unsigned long vm_flags) +{ + if (!vmap_allow_huge || (vm_flags & VM_NO_HUGE_VMAP)) + return false; + + /* VM_TRY_HUGE_VMAP only works for CONFIG_HAVE_ARCH_HUGE_VMALLOC_FLAG */ + return vm_flags & VM_TRY_HUGE_VMAP; +} + /** * __vmalloc_node_range - allocate virtually contiguous memory * @size: allocation size @@ -3106,7 +3116,7 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, return NULL; } - if (vmap_allow_huge && !(vm_flags & VM_NO_HUGE_VMAP)) { + if (vmalloc_try_huge_page(vm_flags)) { unsigned long size_per_node; /* From patchwork Wed Mar 30 22:56:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12796449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8BA5C433FE for ; Wed, 30 Mar 2022 23:02:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 690C68D0001; Wed, 30 Mar 2022 19:02:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 63E536B0078; Wed, 30 Mar 2022 19:02:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 554318D0001; Wed, 30 Mar 2022 19:02:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.a.hostedemail.com [64.99.140.24]) by kanga.kvack.org (Postfix) with ESMTP id 48EB06B0075 for ; Wed, 30 Mar 2022 19:02:36 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 0DDDB23F26 for ; Wed, 30 Mar 2022 23:02:36 +0000 (UTC) X-FDA: 79302578712.07.9D8F2FE Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf30.hostedemail.com (Postfix) with ESMTP id 55D128001A for ; Wed, 30 Mar 2022 23:02:35 +0000 (UTC) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 22UHKbwD009153 for ; Wed, 30 Mar 2022 16:02:34 -0700 Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3f4f1txph1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 30 Mar 2022 16:02:33 -0700 Received: from twshared13345.18.frc3.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Wed, 30 Mar 2022 16:02:32 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id 7A5AD4764AB2; Wed, 30 Mar 2022 15:56:53 -0700 (PDT) From: Song Liu To: , , , CC: , , , , , , , Song Liu Subject: [PATCH bpf 3/4] x86: select HAVE_ARCH_HUGE_VMALLOC_FLAG for X86_64 Date: Wed, 30 Mar 2022 15:56:41 -0700 Message-ID: <20220330225642.1163897-4-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220330225642.1163897-1-song@kernel.org> References: <20220330225642.1163897-1-song@kernel.org> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: r8O81vwpjMRurWujGXis8wETUolkfjGo X-Proofpoint-GUID: r8O81vwpjMRurWujGXis8wETUolkfjGo X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.850,Hydra:6.0.425,FMLib:17.11.64.514 definitions=2022-03-30_06,2022-03-30_01,2022-02-23_01 X-Stat-Signature: 1zt986nfhdwchddjkg3q5nb13hbtkgpg Authentication-Results: imf30.hostedemail.com; dkim=none; spf=none (imf30.hostedemail.com: domain of "prvs=50888507eb=songliubraving@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=50888507eb=songliubraving@fb.com"; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none) X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 55D128001A X-HE-Tag: 1648681355-150353 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000022, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As HAVE_ARCH_HUGE_VMALLOC is not ready for X86_64, enable HAVE_ARCH_HUGE_VMALLOC_FLAG to allow bpf_prog_pack to allocate huge pages. Signed-off-by: Song Liu --- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 4691d3aef681..2195120c8ebb 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -161,6 +161,7 @@ config X86 select HAVE_ALIGNED_STRUCT_PAGE if SLUB select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMAP if X86_64 || X86_PAE + select HAVE_ARCH_HUGE_VMALLOC_FLAG if X86_64 select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if X86_64 From patchwork Wed Mar 30 22:56:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12796451 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38366C433EF for ; Wed, 30 Mar 2022 23:08:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB77C6B0072; Wed, 30 Mar 2022 19:08:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B711E6B0073; Wed, 30 Mar 2022 19:08:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A2F138D0001; Wed, 30 Mar 2022 19:08:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0142.hostedemail.com [216.40.44.142]) by kanga.kvack.org (Postfix) with ESMTP id 936806B0072 for ; Wed, 30 Mar 2022 19:08:29 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 4ADE218298B00 for ; Wed, 30 Mar 2022 23:08:29 +0000 (UTC) X-FDA: 79302593538.29.1818858 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf03.hostedemail.com (Postfix) with ESMTP id AFE3720006 for ; Wed, 30 Mar 2022 23:08:28 +0000 (UTC) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 22UHKauS009109 for ; Wed, 30 Mar 2022 16:08:27 -0700 Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3f4f1txqh7-7 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 30 Mar 2022 16:08:27 -0700 Received: from twshared5730.23.frc3.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::f) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Wed, 30 Mar 2022 16:08:23 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id 45F3B4764ABA; Wed, 30 Mar 2022 15:56:55 -0700 (PDT) From: Song Liu To: , , , CC: , , , , , , , Song Liu Subject: [PATCH bpf 4/4] bpf: use __vmalloc_node_range() with VM_TRY_HUGE_VMAP for bpf_prog_pack Date: Wed, 30 Mar 2022 15:56:42 -0700 Message-ID: <20220330225642.1163897-5-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220330225642.1163897-1-song@kernel.org> References: <20220330225642.1163897-1-song@kernel.org> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: k9mTj7C2HY7KiwPqYgblpHsoyBZ7QbhV X-Proofpoint-GUID: k9mTj7C2HY7KiwPqYgblpHsoyBZ7QbhV X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.850,Hydra:6.0.425,FMLib:17.11.64.514 definitions=2022-03-30_06,2022-03-30_01,2022-02-23_01 X-Stat-Signature: p7dnfg8jx7tbccm6yemy5obrduazgxpr Authentication-Results: imf03.hostedemail.com; dkim=none; spf=none (imf03.hostedemail.com: domain of "prvs=50888507eb=songliubraving@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=50888507eb=songliubraving@fb.com"; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none) X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: AFE3720006 X-HE-Tag: 1648681708-786176 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We cannot yet savely enable HAVE_ARCH_HUGE_VMAP for all vmalloc in X86_64. Let bpf_prog_pack to call __vmalloc_node_range() with VM_TRY_HUGE_VMAP directly. Signed-off-by: Song Liu --- kernel/bpf/core.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 13e9dbeeedf3..257c6457f256 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -851,13 +851,28 @@ static LIST_HEAD(pack_list); #define BPF_HPAGE_MASK PAGE_MASK #endif +static void *bpf_prog_pack_vmalloc(unsigned long size) +{ +#if defined(MODULES_VADDR) + unsigned long start = MODULES_VADDR; + unsigned long end = MODULES_END; +#else + unsigned long start = VMALLOC_START; + unsigned long end = VMALLOC_END; +#endif + + return __vmalloc_node_range(size, PAGE_SIZE, start, end, GFP_KERNEL, PAGE_KERNEL, + VM_DEFER_KMEMLEAK | VM_TRY_HUGE_VMAP, + NUMA_NO_NODE, __builtin_return_address(0)); +} + static size_t select_bpf_prog_pack_size(void) { size_t size; void *ptr; size = BPF_HPAGE_SIZE * num_online_nodes(); - ptr = module_alloc(size); + ptr = bpf_prog_pack_vmalloc(size); /* Test whether we can get huge pages. If not just use PAGE_SIZE * packs. @@ -881,7 +896,7 @@ static struct bpf_prog_pack *alloc_new_pack(void) GFP_KERNEL); if (!pack) return NULL; - pack->ptr = module_alloc(bpf_prog_pack_size); + pack->ptr = bpf_prog_pack_vmalloc(bpf_prog_pack_size); if (!pack->ptr) { kfree(pack); return NULL; @@ -970,7 +985,7 @@ static void bpf_prog_pack_free(struct bpf_binary_header *hdr) if (bitmap_find_next_zero_area(pack->bitmap, bpf_prog_chunk_count(), 0, bpf_prog_chunk_count(), 0) == 0) { list_del(&pack->list); - module_memfree(pack->ptr); + vfree(pack->ptr); kfree(pack); } out: