From patchwork Thu Apr 21 07:22:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12821202 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2A44C433EF for ; Thu, 21 Apr 2022 07:27:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 20B086B0071; Thu, 21 Apr 2022 03:27:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1BB376B0073; Thu, 21 Apr 2022 03:27:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 083BD6B0074; Thu, 21 Apr 2022 03:27:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.25]) by kanga.kvack.org (Postfix) with ESMTP id E9B236B0071 for ; Thu, 21 Apr 2022 03:27:41 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id AA5D180DC4 for ; Thu, 21 Apr 2022 07:27:41 +0000 (UTC) X-FDA: 79380056322.18.A6F6CA0 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf06.hostedemail.com (Postfix) with ESMTP id 4830D180024 for ; Thu, 21 Apr 2022 07:27:40 +0000 (UTC) Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 23L7MCZJ005876 for ; Thu, 21 Apr 2022 00:27:39 -0700 Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3fj7s40ucx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 21 Apr 2022 00:27:39 -0700 Received: from twshared19572.14.frc2.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:11d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Thu, 21 Apr 2022 00:27:38 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id C15966014D3B; Thu, 21 Apr 2022 00:22:26 -0700 (PDT) From: Song Liu To: , , CC: , , , , , , , , Song Liu Subject: [PATCH bpf] bpf: invalidate unused part of bpf_prog_pack Date: Thu, 21 Apr 2022 00:22:12 -0700 Message-ID: <20220421072212.608884-1-song@kernel.org> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: Lkm2xPDRsssxJCOzWQvwhuNo-gFhBj1i X-Proofpoint-GUID: Lkm2xPDRsssxJCOzWQvwhuNo-gFhBj1i X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.858,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-04-20_06,2022-04-20_01,2022-02-23_01 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 4830D180024 X-Stat-Signature: yqe5ccb79wd47fnajjbtrjs7w8wu5fuq Authentication-Results: imf06.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none); spf=none (imf06.hostedemail.com: domain of "prvs=5110c58c6a=songliubraving@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=5110c58c6a=songliubraving@fb.com" X-Rspam-User: X-HE-Tag: 1650526060-523953 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: bpf_prog_pack enables sharing huge pages among multiple BPF programs. These pages are marked as executable, but some part of these huge page may not contain proper BPF programs. To make these pages safe, fill such unused part of these pages with invalid instructions. This is done when a pack is first allocated, and when a bpf program is freed.. Fixes: 57631054fae6 ("bpf: Introduce bpf_prog_pack allocator") Fixes: 33c9805860e5 ("bpf: Introduce bpf_jit_binary_pack_[alloc|finalize|free]") Signed-off-by: Song Liu --- arch/x86/net/bpf_jit_comp.c | 22 ++++++++++++++++++++++ include/linux/bpf.h | 2 ++ kernel/bpf/core.c | 20 ++++++++++++++++---- 3 files changed, 40 insertions(+), 4 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 8fe35ed11fd6..57099d89f2bf 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -228,6 +228,28 @@ static void jit_fill_hole(void *area, unsigned int size) memset(area, 0xcc, size); } +#define INVALID_BUF_SIZE PAGE_SIZE +static char invalid_insn_buf[INVALID_BUF_SIZE]; + +static int __init bpf_init_invalid_insn_buf(void) +{ + jit_fill_hole(invalid_insn_buf, INVALID_BUF_SIZE); + return 0; +} +pure_initcall(bpf_init_invalid_insn_buf); + +void bpf_arch_invalidate_text(void *dst, size_t len) +{ + size_t i = 0; + + while (i < len) { + size_t s = min_t(size_t, len - i, INVALID_BUF_SIZE); + + bpf_arch_text_copy(dst + i, invalid_insn_buf, s); + i += s; + } +} + struct jit_context { int cleanup_addr; /* Epilogue code offset */ diff --git a/include/linux/bpf.h b/include/linux/bpf.h index bdb5298735ce..2157392a94d1 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -2382,6 +2382,8 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, void *bpf_arch_text_copy(void *dst, void *src, size_t len); +void bpf_arch_invalidate_text(void *dst, size_t len); + struct btf_id_set; bool btf_id_set_contains(const struct btf_id_set *set, u32 id); diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index b2a634d0f842..3f8bdbc0e994 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -873,7 +873,7 @@ static size_t select_bpf_prog_pack_size(void) return size; } -static struct bpf_prog_pack *alloc_new_pack(void) +static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_insns) { struct bpf_prog_pack *pack; @@ -886,6 +886,7 @@ static struct bpf_prog_pack *alloc_new_pack(void) kfree(pack); return NULL; } + bpf_fill_ill_insns(pack->ptr, bpf_prog_pack_size); bitmap_zero(pack->bitmap, bpf_prog_pack_size / BPF_PROG_CHUNK_SIZE); list_add_tail(&pack->list, &pack_list); @@ -894,7 +895,7 @@ static struct bpf_prog_pack *alloc_new_pack(void) return pack; } -static void *bpf_prog_pack_alloc(u32 size) +static void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns) { unsigned int nbits = BPF_PROG_SIZE_TO_NBITS(size); struct bpf_prog_pack *pack; @@ -922,7 +923,7 @@ static void *bpf_prog_pack_alloc(u32 size) goto found_free_area; } - pack = alloc_new_pack(); + pack = alloc_new_pack(bpf_fill_ill_insns); if (!pack) goto out; @@ -965,6 +966,7 @@ static void bpf_prog_pack_free(struct bpf_binary_header *hdr) nbits = BPF_PROG_SIZE_TO_NBITS(hdr->size); pos = ((unsigned long)hdr - (unsigned long)pack_ptr) >> BPF_PROG_CHUNK_SHIFT; + bpf_arch_invalidate_text(hdr, hdr->size); bitmap_clear(pack->bitmap, pos, nbits); if (bitmap_find_next_zero_area(pack->bitmap, bpf_prog_chunk_count(), 0, bpf_prog_chunk_count(), 0) == 0) { @@ -1103,7 +1105,7 @@ bpf_jit_binary_pack_alloc(unsigned int proglen, u8 **image_ptr, if (bpf_jit_charge_modmem(size)) return NULL; - ro_header = bpf_prog_pack_alloc(size); + ro_header = bpf_prog_pack_alloc(size, bpf_fill_ill_insns); if (!ro_header) { bpf_jit_uncharge_modmem(size); return NULL; @@ -1204,6 +1206,16 @@ void __weak bpf_jit_free(struct bpf_prog *fp) bpf_prog_unlock_free(fp); } +void __weak bpf_arch_invalidate_text(void *dst, size_t len) +{ + char buf[1] = {}; + int i; + + WARN_ONCE(1, "Please override bpf_arch_invalidate_text for bpf_prog_pack\n"); + for (i = 0; i < len; i++) + bpf_arch_text_copy(dst + i, buf, 1); +} + int bpf_jit_get_func_addr(const struct bpf_prog *prog, const struct bpf_insn *insn, bool extra_pass, u64 *func_addr, bool *func_addr_fixed)