From patchwork Fri May 20 23:57:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12857583 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9731C433F5 for ; Sat, 21 May 2022 00:00:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9CE4E6B007D; Fri, 20 May 2022 20:00:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9558E6B007E; Fri, 20 May 2022 20:00:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7A79E6B0080; Fri, 20 May 2022 20:00:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 5B52B6B007E for ; Fri, 20 May 2022 20:00:44 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 36D8A32103 for ; Sat, 21 May 2022 00:00:44 +0000 (UTC) X-FDA: 79487794008.28.A6D71D8 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf11.hostedemail.com (Postfix) with ESMTP id DAA53400DD for ; Sat, 21 May 2022 00:00:36 +0000 (UTC) Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 24KMrt7Y023419 for ; Fri, 20 May 2022 17:00:42 -0700 Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3g604bq6qy-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 20 May 2022 17:00:42 -0700 Received: from twshared11660.23.frc3.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:21d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Fri, 20 May 2022 17:00:39 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id 8B5107E22184; Fri, 20 May 2022 16:58:10 -0700 (PDT) From: Song Liu To: , , CC: , , , , , , , Song Liu Subject: [PATCH v4 bpf-next 1/8] bpf: fill new bpf_prog_pack with illegal instructions Date: Fri, 20 May 2022 16:57:51 -0700 Message-ID: <20220520235758.1858153-2-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220520235758.1858153-1-song@kernel.org> References: <20220520235758.1858153-1-song@kernel.org> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: dyueRrmNqqrvfJyNw1ZJBGy0IGZAdivc X-Proofpoint-GUID: dyueRrmNqqrvfJyNw1ZJBGy0IGZAdivc X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-05-20_08,2022-05-20_02,2022-02-23_01 X-Rspam-User: X-Rspamd-Queue-Id: DAA53400DD X-Stat-Signature: kn4yjosapded8ry7y946u11cepobk54d Authentication-Results: imf11.hostedemail.com; dkim=none; spf=none (imf11.hostedemail.com: domain of "prvs=6140678c23=songliubraving@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=6140678c23=songliubraving@fb.com"; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none) X-Rspamd-Server: rspam04 X-HE-Tag: 1653091236-925862 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: bpf_prog_pack enables sharing huge pages among multiple BPF programs. These pages are marked as executable before the JIT engine fill it with BPF programs. To make these pages safe, fill the hole bpf_prog_pack with illegal instructions before making it executable. Fixes: 57631054fae6 ("bpf: Introduce bpf_prog_pack allocator") Fixes: 33c9805860e5 ("bpf: Introduce bpf_jit_binary_pack_[alloc|finalize|free]") Signed-off-by: Song Liu --- kernel/bpf/core.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 9cc91f0f3115..2d0c9d4696ad 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -873,7 +873,7 @@ static size_t select_bpf_prog_pack_size(void) return size; } -static struct bpf_prog_pack *alloc_new_pack(void) +static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_insns) { struct bpf_prog_pack *pack; @@ -886,6 +886,7 @@ static struct bpf_prog_pack *alloc_new_pack(void) kfree(pack); return NULL; } + bpf_fill_ill_insns(pack->ptr, bpf_prog_pack_size); bitmap_zero(pack->bitmap, bpf_prog_pack_size / BPF_PROG_CHUNK_SIZE); list_add_tail(&pack->list, &pack_list); @@ -895,7 +896,7 @@ static struct bpf_prog_pack *alloc_new_pack(void) return pack; } -static void *bpf_prog_pack_alloc(u32 size) +static void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns) { unsigned int nbits = BPF_PROG_SIZE_TO_NBITS(size); struct bpf_prog_pack *pack; @@ -910,6 +911,7 @@ static void *bpf_prog_pack_alloc(u32 size) size = round_up(size, PAGE_SIZE); ptr = module_alloc(size); if (ptr) { + bpf_fill_ill_insns(ptr, size); set_vm_flush_reset_perms(ptr); set_memory_ro((unsigned long)ptr, size / PAGE_SIZE); set_memory_x((unsigned long)ptr, size / PAGE_SIZE); @@ -923,7 +925,7 @@ static void *bpf_prog_pack_alloc(u32 size) goto found_free_area; } - pack = alloc_new_pack(); + pack = alloc_new_pack(bpf_fill_ill_insns); if (!pack) goto out; @@ -1102,7 +1104,7 @@ bpf_jit_binary_pack_alloc(unsigned int proglen, u8 **image_ptr, if (bpf_jit_charge_modmem(size)) return NULL; - ro_header = bpf_prog_pack_alloc(size); + ro_header = bpf_prog_pack_alloc(size, bpf_fill_ill_insns); if (!ro_header) { bpf_jit_uncharge_modmem(size); return NULL; From patchwork Fri May 20 23:57:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12857577 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF5DEC433F5 for ; Sat, 21 May 2022 00:00:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 031796B0071; Fri, 20 May 2022 20:00:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EFCA56B0072; Fri, 20 May 2022 20:00:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D4DDE6B0073; Fri, 20 May 2022 20:00:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id BEBAB6B0071 for ; Fri, 20 May 2022 20:00:27 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 91BE632EC4 for ; Sat, 21 May 2022 00:00:27 +0000 (UTC) X-FDA: 79487793294.15.CEB53E7 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by imf05.hostedemail.com (Postfix) with ESMTP id 8B26210001E for ; Sat, 21 May 2022 00:00:01 +0000 (UTC) Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 24KMs7K0027860 for ; Fri, 20 May 2022 17:00:26 -0700 Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3g5rgjaqtw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 20 May 2022 17:00:26 -0700 Received: from snc-exhub201.TheFacebook.com (2620:10d:c085:21d::7) by snc-exhub103.TheFacebook.com (2620:10d:c085:11d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Fri, 20 May 2022 17:00:24 -0700 Received: from twshared19572.14.frc2.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:21d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Fri, 20 May 2022 17:00:24 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id A1B4E7E221DD; Fri, 20 May 2022 16:58:12 -0700 (PDT) From: Song Liu To: , , CC: , , , , , , , Song Liu Subject: [PATCH v4 bpf-next 2/8] x86/alternative: introduce text_poke_set Date: Fri, 20 May 2022 16:57:52 -0700 Message-ID: <20220520235758.1858153-3-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220520235758.1858153-1-song@kernel.org> References: <20220520235758.1858153-1-song@kernel.org> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: VNKIrTTULkZrkldzljJ4_w06SI-P4qU9 X-Proofpoint-ORIG-GUID: VNKIrTTULkZrkldzljJ4_w06SI-P4qU9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-05-20_08,2022-05-20_02,2022-02-23_01 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 8B26210001E X-Stat-Signature: 1njs5szei5ehyhoqtg3x9u3jn161u99f Authentication-Results: imf05.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none); spf=none (imf05.hostedemail.com: domain of "prvs=6140678c23=songliubraving@fb.com" has no SPF policy when checking 67.231.153.30) smtp.mailfrom="prvs=6140678c23=songliubraving@fb.com" X-Rspam-User: X-HE-Tag: 1653091201-755124 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce a memset like API for text_poke. This will be used to fill the unused RX memory with illegal instructions. Suggested-by: Peter Zijlstra (Intel) Acked-by: Peter Zijlstra (Intel) Signed-off-by: Song Liu --- arch/x86/include/asm/text-patching.h | 1 + arch/x86/kernel/alternative.c | 67 +++++++++++++++++++++++----- 2 files changed, 58 insertions(+), 10 deletions(-) diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h index d20ab0921480..1cc15528ce29 100644 --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -45,6 +45,7 @@ extern void *text_poke(void *addr, const void *opcode, size_t len); extern void text_poke_sync(void); extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len); extern void *text_poke_copy(void *addr, const void *opcode, size_t len); +extern void *text_poke_set(void *addr, int c, size_t len); extern int poke_int3_handler(struct pt_regs *regs); extern void text_poke_bp(void *addr, const void *opcode, size_t len, const void *emulate); diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index d374cb3cf024..7563b5bc8328 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -994,7 +994,21 @@ static inline void unuse_temporary_mm(temp_mm_state_t prev_state) __ro_after_init struct mm_struct *poking_mm; __ro_after_init unsigned long poking_addr; -static void *__text_poke(void *addr, const void *opcode, size_t len) +static void text_poke_memcpy(void *dst, const void *src, size_t len) +{ + memcpy(dst, src, len); +} + +static void text_poke_memset(void *dst, const void *src, size_t len) +{ + int c = *(const int *)src; + + memset(dst, c, len); +} + +typedef void text_poke_f(void *dst, const void *src, size_t len); + +static void *__text_poke(text_poke_f func, void *addr, const void *src, size_t len) { bool cross_page_boundary = offset_in_page(addr) + len > PAGE_SIZE; struct page *pages[2] = {NULL}; @@ -1059,7 +1073,7 @@ static void *__text_poke(void *addr, const void *opcode, size_t len) prev = use_temporary_mm(poking_mm); kasan_disable_current(); - memcpy((u8 *)poking_addr + offset_in_page(addr), opcode, len); + func((u8 *)poking_addr + offset_in_page(addr), src, len); kasan_enable_current(); /* @@ -1087,11 +1101,13 @@ static void *__text_poke(void *addr, const void *opcode, size_t len) (cross_page_boundary ? 2 : 1) * PAGE_SIZE, PAGE_SHIFT, false); - /* - * If the text does not match what we just wrote then something is - * fundamentally screwy; there's nothing we can really do about that. - */ - BUG_ON(memcmp(addr, opcode, len)); + if (func == text_poke_memcpy) { + /* + * If the text does not match what we just wrote then something is + * fundamentally screwy; there's nothing we can really do about that. + */ + BUG_ON(memcmp(addr, src, len)); + } local_irq_restore(flags); pte_unmap_unlock(ptep, ptl); @@ -1118,7 +1134,7 @@ void *text_poke(void *addr, const void *opcode, size_t len) { lockdep_assert_held(&text_mutex); - return __text_poke(addr, opcode, len); + return __text_poke(text_poke_memcpy, addr, opcode, len); } /** @@ -1137,7 +1153,7 @@ void *text_poke(void *addr, const void *opcode, size_t len) */ void *text_poke_kgdb(void *addr, const void *opcode, size_t len) { - return __text_poke(addr, opcode, len); + return __text_poke(text_poke_memcpy, addr, opcode, len); } /** @@ -1167,7 +1183,38 @@ void *text_poke_copy(void *addr, const void *opcode, size_t len) s = min_t(size_t, PAGE_SIZE * 2 - offset_in_page(ptr), len - patched); - __text_poke((void *)ptr, opcode + patched, s); + __text_poke(text_poke_memcpy, (void *)ptr, opcode + patched, s); + patched += s; + } + mutex_unlock(&text_mutex); + return addr; +} + +/** + * text_poke_set - memset into (an unused part of) RX memory + * @addr: address to modify + * @c: the byte to fill the area with + * @len: length to copy, could be more than 2x PAGE_SIZE + * + * This is useful to overwrite unused regions of RX memory with illegal + * instructions. + */ +void *text_poke_set(void *addr, int c, size_t len) +{ + unsigned long start = (unsigned long)addr; + size_t patched = 0; + + if (WARN_ON_ONCE(core_kernel_text(start))) + return NULL; + + mutex_lock(&text_mutex); + while (patched < len) { + unsigned long ptr = start + patched; + size_t s; + + s = min_t(size_t, PAGE_SIZE * 2 - offset_in_page(ptr), len - patched); + + __text_poke(text_poke_memset, (void *)ptr, (void *)&c, s); patched += s; } mutex_unlock(&text_mutex); From patchwork Fri May 20 23:57:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12857580 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B83C7C433F5 for ; Sat, 21 May 2022 00:00:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 64B2C6B0074; Fri, 20 May 2022 20:00:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5D4BC6B0075; Fri, 20 May 2022 20:00:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 425076B0078; Fri, 20 May 2022 20:00:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1B3066B0075 for ; Fri, 20 May 2022 20:00:37 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id E680132E3B for ; Sat, 21 May 2022 00:00:36 +0000 (UTC) X-FDA: 79487793672.03.53578A4 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf26.hostedemail.com (Postfix) with ESMTP id E1CF114000C for ; Sat, 21 May 2022 00:00:32 +0000 (UTC) Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 24KMsIZu029543 for ; Fri, 20 May 2022 17:00:35 -0700 Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3g5xexfqxf-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 20 May 2022 17:00:35 -0700 Received: from twshared24024.25.frc3.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:11d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Fri, 20 May 2022 17:00:33 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id A9C727E221E7; Fri, 20 May 2022 16:58:14 -0700 (PDT) From: Song Liu To: , , CC: , , , , , , , Song Liu Subject: [PATCH v4 bpf-next 3/8] bpf: introduce bpf_arch_text_invalidate for bpf_prog_pack Date: Fri, 20 May 2022 16:57:53 -0700 Message-ID: <20220520235758.1858153-4-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220520235758.1858153-1-song@kernel.org> References: <20220520235758.1858153-1-song@kernel.org> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: VBgU7iln5p8oD2OIIZACvTZaycHBLTJ9 X-Proofpoint-GUID: VBgU7iln5p8oD2OIIZACvTZaycHBLTJ9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-05-20_08,2022-05-20_02,2022-02-23_01 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: E1CF114000C X-Rspam-User: Authentication-Results: imf26.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none); spf=none (imf26.hostedemail.com: domain of "prvs=6140678c23=songliubraving@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=6140678c23=songliubraving@fb.com" X-Stat-Signature: tfcruancnfxpab8e7f84eoigi4dyskhb X-HE-Tag: 1653091232-760415 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce bpf_arch_text_invalidate and use it to fill unused part of the bpf_prog_pack with illegal instructions when a BPF program is freed. Signed-off-by: Song Liu --- arch/x86/net/bpf_jit_comp.c | 5 +++++ include/linux/bpf.h | 1 + kernel/bpf/core.c | 8 ++++++++ 3 files changed, 14 insertions(+) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index a2b6d197c226..f298b18a9a3d 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -228,6 +228,11 @@ static void jit_fill_hole(void *area, unsigned int size) memset(area, 0xcc, size); } +int bpf_arch_text_invalidate(void *dst, size_t len) +{ + return IS_ERR_OR_NULL(text_poke_set(dst, 0xcc, len)); +} + struct jit_context { int cleanup_addr; /* Epilogue code offset */ diff --git a/include/linux/bpf.h b/include/linux/bpf.h index cc4d5e394031..a9b1875212f6 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -2365,6 +2365,7 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, void *addr1, void *addr2); void *bpf_arch_text_copy(void *dst, void *src, size_t len); +int bpf_arch_text_invalidate(void *dst, size_t len); struct btf_id_set; bool btf_id_set_contains(const struct btf_id_set *set, u32 id); diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 2d0c9d4696ad..cacd8684c3c4 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -968,6 +968,9 @@ static void bpf_prog_pack_free(struct bpf_binary_header *hdr) nbits = BPF_PROG_SIZE_TO_NBITS(hdr->size); pos = ((unsigned long)hdr - (unsigned long)pack_ptr) >> BPF_PROG_CHUNK_SHIFT; + WARN_ONCE(bpf_arch_text_invalidate(hdr, hdr->size), + "bpf_prog_pack bug: missing bpf_arch_text_invalidate?\n"); + bitmap_clear(pack->bitmap, pos, nbits); if (bitmap_find_next_zero_area(pack->bitmap, bpf_prog_chunk_count(), 0, bpf_prog_chunk_count(), 0) == 0) { @@ -2740,6 +2743,11 @@ void * __weak bpf_arch_text_copy(void *dst, void *src, size_t len) return ERR_PTR(-ENOTSUPP); } +int __weak bpf_arch_text_invalidate(void *dst, size_t len) +{ + return -ENOTSUPP; +} + DEFINE_STATIC_KEY_FALSE(bpf_stats_enabled_key); EXPORT_SYMBOL(bpf_stats_enabled_key); From patchwork Fri May 20 23:57:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12857579 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E5DAC4332F for ; Sat, 21 May 2022 00:00:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F240B6B0073; Fri, 20 May 2022 20:00:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EAEB66B0074; Fri, 20 May 2022 20:00:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CDC606B0075; Fri, 20 May 2022 20:00:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B95F96B0073 for ; Fri, 20 May 2022 20:00:36 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 8FAB2604F4 for ; Sat, 21 May 2022 00:00:36 +0000 (UTC) X-FDA: 79487793672.07.65D8D57 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by imf22.hostedemail.com (Postfix) with ESMTP id 5D5D4C00E8 for ; Sat, 21 May 2022 00:00:33 +0000 (UTC) Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.17.1.5/8.17.1.5) with ESMTP id 24KMsBtV005347 for ; Fri, 20 May 2022 17:00:35 -0700 Received: from mail.thefacebook.com ([163.114.132.120]) by m0089730.ppops.net (PPS) with ESMTPS id 3g6bja46uj-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 20 May 2022 17:00:35 -0700 Received: from twshared24024.25.frc3.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:21d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Fri, 20 May 2022 17:00:34 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id C84FB7E221EC; Fri, 20 May 2022 16:58:16 -0700 (PDT) From: Song Liu To: , , CC: , , , , , , , Song Liu Subject: [PATCH v4 bpf-next 4/8] module: introduce module_alloc_huge Date: Fri, 20 May 2022 16:57:54 -0700 Message-ID: <20220520235758.1858153-5-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220520235758.1858153-1-song@kernel.org> References: <20220520235758.1858153-1-song@kernel.org> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: VSM6BjH1CwCVGVfMX6z2E89fvRbvdnIt X-Proofpoint-ORIG-GUID: VSM6BjH1CwCVGVfMX6z2E89fvRbvdnIt X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-05-20_08,2022-05-20_02,2022-02-23_01 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 5D5D4C00E8 X-Stat-Signature: ahfs85bxxpkuc1smc97yo1thsesxr9w3 X-Rspam-User: Authentication-Results: imf22.hostedemail.com; dkim=none; spf=none (imf22.hostedemail.com: domain of "prvs=6140678c23=songliubraving@fb.com" has no SPF policy when checking 67.231.153.30) smtp.mailfrom="prvs=6140678c23=songliubraving@fb.com"; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none) X-HE-Tag: 1653091233-684905 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce module_alloc_huge, which allocates huge page backed memory in module memory space. The primary user of this memory is bpf_prog_pack (multiple BPF programs sharing a huge page). Signed-off-by: Song Liu --- Note: This conflicts with the module.c => module/ split in modules-next. Current patch is for module.c in the bpf-next tree. After the split, __weak module_alloc_huge() should be added to kernel/module/main.c. --- arch/x86/kernel/module.c | 21 +++++++++++++++++++++ include/linux/moduleloader.h | 5 +++++ kernel/module.c | 8 ++++++++ 3 files changed, 34 insertions(+) diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c index b98ffcf4d250..63f6a16c70dc 100644 --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -86,6 +86,27 @@ void *module_alloc(unsigned long size) return p; } +void *module_alloc_huge(unsigned long size) +{ + gfp_t gfp_mask = GFP_KERNEL; + void *p; + + if (PAGE_ALIGN(size) > MODULES_LEN) + return NULL; + + p = __vmalloc_node_range(size, MODULE_ALIGN, + MODULES_VADDR + get_module_load_offset(), + MODULES_END, gfp_mask, PAGE_KERNEL, + VM_DEFER_KMEMLEAK | VM_ALLOW_HUGE_VMAP, + NUMA_NO_NODE, __builtin_return_address(0)); + if (p && (kasan_alloc_module_shadow(p, size, gfp_mask) < 0)) { + vfree(p); + return NULL; + } + + return p; +} + #ifdef CONFIG_X86_32 int apply_relocate(Elf32_Shdr *sechdrs, const char *strtab, diff --git a/include/linux/moduleloader.h b/include/linux/moduleloader.h index 9e09d11ffe5b..d34743a88938 100644 --- a/include/linux/moduleloader.h +++ b/include/linux/moduleloader.h @@ -26,6 +26,11 @@ unsigned int arch_mod_section_prepend(struct module *mod, unsigned int section); sections. Returns NULL on failure. */ void *module_alloc(unsigned long size); +/* Allocator used for allocating memory in module memory space. If size is + * greater than PMD_SIZE, allow using huge pages. Returns NULL on failure. + */ +void *module_alloc_huge(unsigned long size); + /* Free memory returned from module_alloc. */ void module_memfree(void *module_region); diff --git a/kernel/module.c b/kernel/module.c index 6cea788fd965..2af20ac3209c 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -2839,6 +2839,14 @@ void * __weak module_alloc(unsigned long size) NUMA_NO_NODE, __builtin_return_address(0)); } +void * __weak module_alloc_huge(unsigned long size) +{ + return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, + GFP_KERNEL, PAGE_KERNEL_EXEC, + VM_FLUSH_RESET_PERMS | VM_ALLOW_HUGE_VMAP, + NUMA_NO_NODE, __builtin_return_address(0)); +} + bool __weak module_init_section(const char *name) { return strstarts(name, ".init"); From patchwork Fri May 20 23:57:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12857582 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC160C433FE for ; Sat, 21 May 2022 00:00:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 33F956B0078; Fri, 20 May 2022 20:00:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2C6556B007D; Fri, 20 May 2022 20:00:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 11A186B007E; Fri, 20 May 2022 20:00:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id F0B7F6B0078 for ; Fri, 20 May 2022 20:00:43 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id C25CF32F2D for ; Sat, 21 May 2022 00:00:43 +0000 (UTC) X-FDA: 79487793966.09.1486515 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf04.hostedemail.com (Postfix) with ESMTP id E0200400EB for ; Sat, 21 May 2022 00:00:26 +0000 (UTC) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 24KMsDZc016795 for ; Fri, 20 May 2022 17:00:40 -0700 Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3g5pj53new-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 20 May 2022 17:00:39 -0700 Received: from twshared0725.22.frc3.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:11d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Fri, 20 May 2022 17:00:38 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id F0FCA7E221FE; Fri, 20 May 2022 16:58:18 -0700 (PDT) From: Song Liu To: , , CC: , , , , , , , Song Liu Subject: [PATCH v4 bpf-next 5/8] bpf: use module_alloc_huge for bpf_prog_pack Date: Fri, 20 May 2022 16:57:55 -0700 Message-ID: <20220520235758.1858153-6-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220520235758.1858153-1-song@kernel.org> References: <20220520235758.1858153-1-song@kernel.org> X-FB-Internal: Safe X-Proofpoint-GUID: qP7UMzIwPg7MPOSGlkf8z2Z6E92ojy9N X-Proofpoint-ORIG-GUID: qP7UMzIwPg7MPOSGlkf8z2Z6E92ojy9N X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-05-20_08,2022-05-20_02,2022-02-23_01 Authentication-Results: imf04.hostedemail.com; dkim=none; spf=none (imf04.hostedemail.com: domain of "prvs=6140678c23=songliubraving@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=6140678c23=songliubraving@fb.com"; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none) X-Rspam-User: X-Stat-Signature: e5h7erx4gigfysksw3syi1o74uwu7iw1 X-Rspamd-Queue-Id: E0200400EB X-Rspamd-Server: rspam01 X-HE-Tag: 1653091226-33003 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use module_alloc_huge for bpf_prog_pack so that BPF programs sit on PMD_SIZE pages. This benefits system performance by reducing iTLB miss rate. Benchmark of a real web service workload shows this change gives another ~0.2% performance boost on top of PAGE_SIZE bpf_prog_pack (which improve system throughput by ~0.5%). Also, remove set_vm_flush_reset_perms() from alloc_new_pack() and use set_memory_[nx|rw] in bpf_prog_pack_free(). This is because VM_FLUSH_RESET_PERMS does not work with huge pages yet. [1] [1] https://lore.kernel.org/bpf/aeeeaf0b7ec63fdba55d4834d2f524d8bf05b71b.camel@intel.com/ Suggested-by: Rick Edgecombe Signed-off-by: Song Liu --- kernel/bpf/core.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index cacd8684c3c4..b64d91fcb0ba 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -857,7 +857,7 @@ static size_t select_bpf_prog_pack_size(void) void *ptr; size = BPF_HPAGE_SIZE * num_online_nodes(); - ptr = module_alloc(size); + ptr = module_alloc_huge(size); /* Test whether we can get huge pages. If not just use PAGE_SIZE * packs. @@ -881,7 +881,7 @@ static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_ins GFP_KERNEL); if (!pack) return NULL; - pack->ptr = module_alloc(bpf_prog_pack_size); + pack->ptr = module_alloc_huge(bpf_prog_pack_size); if (!pack->ptr) { kfree(pack); return NULL; @@ -890,7 +890,6 @@ static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_ins bitmap_zero(pack->bitmap, bpf_prog_pack_size / BPF_PROG_CHUNK_SIZE); list_add_tail(&pack->list, &pack_list); - set_vm_flush_reset_perms(pack->ptr); set_memory_ro((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE_SIZE); set_memory_x((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE_SIZE); return pack; @@ -909,10 +908,9 @@ static void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insn if (size > bpf_prog_pack_size) { size = round_up(size, PAGE_SIZE); - ptr = module_alloc(size); + ptr = module_alloc_huge(size); if (ptr) { bpf_fill_ill_insns(ptr, size); - set_vm_flush_reset_perms(ptr); set_memory_ro((unsigned long)ptr, size / PAGE_SIZE); set_memory_x((unsigned long)ptr, size / PAGE_SIZE); } @@ -949,6 +947,8 @@ static void bpf_prog_pack_free(struct bpf_binary_header *hdr) mutex_lock(&pack_mutex); if (hdr->size > bpf_prog_pack_size) { + set_memory_nx((unsigned long)hdr, hdr->size / PAGE_SIZE); + set_memory_rw((unsigned long)hdr, hdr->size / PAGE_SIZE); module_memfree(hdr); goto out; } @@ -975,6 +975,8 @@ static void bpf_prog_pack_free(struct bpf_binary_header *hdr) if (bitmap_find_next_zero_area(pack->bitmap, bpf_prog_chunk_count(), 0, bpf_prog_chunk_count(), 0) == 0) { list_del(&pack->list); + set_memory_nx((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE_SIZE); + set_memory_rw((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE_SIZE); module_memfree(pack->ptr); kfree(pack); } From patchwork Fri May 20 23:57:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12857578 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FF36C433F5 for ; Sat, 21 May 2022 00:00:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EFEB06B0072; Fri, 20 May 2022 20:00:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EB8006B0073; Fri, 20 May 2022 20:00:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D27986B0074; Fri, 20 May 2022 20:00:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C4B8A6B0072 for ; Fri, 20 May 2022 20:00:35 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A011A32EBC for ; Sat, 21 May 2022 00:00:35 +0000 (UTC) X-FDA: 79487793630.04.1A0BF3D Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by imf20.hostedemail.com (Postfix) with ESMTP id BA4E01C0018 for ; Sat, 21 May 2022 00:00:21 +0000 (UTC) Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.17.1.5/8.17.1.5) with ESMTP id 24KMsAm5021874 for ; Fri, 20 May 2022 17:00:34 -0700 Received: from mail.thefacebook.com ([163.114.132.120]) by m0001303.ppops.net (PPS) with ESMTPS id 3g5wkrg2ru-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 20 May 2022 17:00:34 -0700 Received: from twshared10560.18.frc3.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:21d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Fri, 20 May 2022 17:00:32 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id 0B1EE7E22204; Fri, 20 May 2022 16:58:21 -0700 (PDT) From: Song Liu To: , , CC: , , , , , , , Song Liu Subject: [PATCH v4 bpf-next 6/8] vmalloc: WARN for set_vm_flush_reset_perms() on huge pages Date: Fri, 20 May 2022 16:57:56 -0700 Message-ID: <20220520235758.1858153-7-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220520235758.1858153-1-song@kernel.org> References: <20220520235758.1858153-1-song@kernel.org> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: hoFza5jLnX94KWey5tNtha5fDmRDy_M7 X-Proofpoint-ORIG-GUID: hoFza5jLnX94KWey5tNtha5fDmRDy_M7 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-05-20_08,2022-05-20_02,2022-02-23_01 X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: BA4E01C0018 X-Stat-Signature: tu85516cc5epd37awbbdpahnsijdma7n X-Rspam-User: Authentication-Results: imf20.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none); spf=none (imf20.hostedemail.com: domain of "prvs=6140678c23=songliubraving@fb.com" has no SPF policy when checking 67.231.153.30) smtp.mailfrom="prvs=6140678c23=songliubraving@fb.com" X-HE-Tag: 1653091221-72371 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: VM_FLUSH_RESET_PERMS is not yet ready for huge pages, add a WARN to catch misuse soon. Suggested-by: Rick Edgecombe Signed-off-by: Song Liu --- include/linux/vmalloc.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index b159c2789961..5e0d0a60d9d5 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -238,6 +238,7 @@ static inline void set_vm_flush_reset_perms(void *addr) { struct vm_struct *vm = find_vm_area(addr); + WARN_ON_ONCE(is_vm_area_hugepages(addr)); if (vm) vm->flags |= VM_FLUSH_RESET_PERMS; } From patchwork Fri May 20 23:57:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12857585 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CF8BC433EF for ; Sat, 21 May 2022 00:01:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C15956B0072; Fri, 20 May 2022 20:01:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BA0086B0073; Fri, 20 May 2022 20:01:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A66006B0074; Fri, 20 May 2022 20:01:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 94F266B0072 for ; Fri, 20 May 2022 20:01:33 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 6F6F7205EB for ; Sat, 21 May 2022 00:01:33 +0000 (UTC) X-FDA: 79487796066.13.B2334C8 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by imf07.hostedemail.com (Postfix) with ESMTP id F379A40016 for ; Sat, 21 May 2022 00:01:23 +0000 (UTC) Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 24KMsDbU018349 for ; Fri, 20 May 2022 17:01:32 -0700 Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3g59tby96s-7 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 20 May 2022 17:01:32 -0700 Received: from twshared14818.18.frc3.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::d) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Fri, 20 May 2022 17:01:29 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id 84DCB7E2220E; Fri, 20 May 2022 16:58:23 -0700 (PDT) From: Song Liu To: , , CC: , , , , , , , Song Liu Subject: [PATCH v4 bpf-next 7/8] vmalloc: introduce huge_vmalloc_supported Date: Fri, 20 May 2022 16:57:57 -0700 Message-ID: <20220520235758.1858153-8-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220520235758.1858153-1-song@kernel.org> References: <20220520235758.1858153-1-song@kernel.org> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: _uWtTYdMcIX32PodIS7iDeOtG3_b1wBE X-Proofpoint-GUID: _uWtTYdMcIX32PodIS7iDeOtG3_b1wBE X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-05-20_08,2022-05-20_02,2022-02-23_01 X-Stat-Signature: wfbsytpw67uhkbt9crazcabi1bbxmnnf X-Rspam-User: Authentication-Results: imf07.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none); spf=none (imf07.hostedemail.com: domain of "prvs=6140678c23=songliubraving@fb.com" has no SPF policy when checking 67.231.153.30) smtp.mailfrom="prvs=6140678c23=songliubraving@fb.com" X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: F379A40016 X-HE-Tag: 1653091283-600580 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: huge_vmalloc_supported() exposes vmap_allow_huge so that users of vmalloc APIs could know whether vmalloc will return huge pages. Suggested-by: Rick Edgecombe Signed-off-by: Song Liu --- include/linux/vmalloc.h | 6 ++++++ mm/vmalloc.c | 5 +++++ 2 files changed, 11 insertions(+) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 5e0d0a60d9d5..22e81c1813bd 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -242,11 +242,17 @@ static inline void set_vm_flush_reset_perms(void *addr) if (vm) vm->flags |= VM_FLUSH_RESET_PERMS; } +bool huge_vmalloc_supported(void); #else static inline void set_vm_flush_reset_perms(void *addr) { } + +static inline bool huge_vmalloc_supported(void) +{ + return false; +} #endif /* for /proc/kcore */ diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 07da85ae825b..d3b11317b025 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -72,6 +72,11 @@ early_param("nohugevmalloc", set_nohugevmalloc); static const bool vmap_allow_huge = false; #endif /* CONFIG_HAVE_ARCH_HUGE_VMALLOC */ +bool huge_vmalloc_supported(void) +{ + return vmap_allow_huge; +} + bool is_vmalloc_addr(const void *x) { unsigned long addr = (unsigned long)kasan_reset_tag(x); From patchwork Fri May 20 23:57:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12857584 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE665C433EF for ; Sat, 21 May 2022 00:01:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5D6F56B0071; Fri, 20 May 2022 20:01:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 560596B0072; Fri, 20 May 2022 20:01:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3FE226B0073; Fri, 20 May 2022 20:01:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 274466B0071 for ; Fri, 20 May 2022 20:01:28 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id F224B60780 for ; Sat, 21 May 2022 00:01:27 +0000 (UTC) X-FDA: 79487795856.27.6E6FC40 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by imf08.hostedemail.com (Postfix) with ESMTP id 80A1F1600D6 for ; Sat, 21 May 2022 00:01:08 +0000 (UTC) Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.17.1.5/8.17.1.5) with ESMTP id 24KMs7Q1005136 for ; Fri, 20 May 2022 17:01:27 -0700 Received: from mail.thefacebook.com ([163.114.132.120]) by m0089730.ppops.net (PPS) with ESMTPS id 3g6bja470w-8 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 20 May 2022 17:01:26 -0700 Received: from twshared24024.25.frc3.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:11d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Fri, 20 May 2022 17:01:23 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id 7B01C7E22215; Fri, 20 May 2022 16:58:26 -0700 (PDT) From: Song Liu To: , , CC: , , , , , , , Song Liu Subject: [PATCH v4 bpf-next 8/8] bpf: simplify select_bpf_prog_pack_size Date: Fri, 20 May 2022 16:57:58 -0700 Message-ID: <20220520235758.1858153-9-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220520235758.1858153-1-song@kernel.org> References: <20220520235758.1858153-1-song@kernel.org> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: fCX_svnTMSl0d1nU6sfROC4GMJtDKmce X-Proofpoint-ORIG-GUID: fCX_svnTMSl0d1nU6sfROC4GMJtDKmce X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-05-20_08,2022-05-20_02,2022-02-23_01 Authentication-Results: imf08.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none); spf=none (imf08.hostedemail.com: domain of "prvs=6140678c23=songliubraving@fb.com" has no SPF policy when checking 67.231.153.30) smtp.mailfrom="prvs=6140678c23=songliubraving@fb.com" X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 80A1F1600D6 X-Stat-Signature: bkk9wfjobudmfj6jspf7rnaigwawrc1s X-HE-Tag: 1653091268-445562 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use huge_vmalloc_supported to simplify select_bpf_prog_pack_size, so that we don't allocate some huge pages and free them immediately. Suggested-by: Rick Edgecombe Signed-off-by: Song Liu --- kernel/bpf/core.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index b64d91fcb0ba..b5dcc8f182b3 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -854,22 +854,15 @@ static LIST_HEAD(pack_list); static size_t select_bpf_prog_pack_size(void) { size_t size; - void *ptr; - - size = BPF_HPAGE_SIZE * num_online_nodes(); - ptr = module_alloc_huge(size); - /* Test whether we can get huge pages. If not just use PAGE_SIZE - * packs. - */ - if (!ptr || !is_vm_area_hugepages(ptr)) { + if (huge_vmalloc_supported()) { + size = BPF_HPAGE_SIZE * num_online_nodes(); + bpf_prog_pack_mask = BPF_HPAGE_MASK; + } else { size = PAGE_SIZE; bpf_prog_pack_mask = PAGE_MASK; - } else { - bpf_prog_pack_mask = BPF_HPAGE_MASK; } - vfree(ptr); return size; }