From patchwork Thu Jul 7 22:35:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12910359 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D468C433EF for ; Thu, 7 Jul 2022 22:36:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 358BC6B0072; Thu, 7 Jul 2022 18:36:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 306766B0073; Thu, 7 Jul 2022 18:36:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F4156B0074; Thu, 7 Jul 2022 18:36:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 131F56B0072 for ; Thu, 7 Jul 2022 18:36:26 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E15C761038 for ; Thu, 7 Jul 2022 22:36:25 +0000 (UTC) X-FDA: 79661763930.10.C2683CE Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf29.hostedemail.com (Postfix) with ESMTP id 5C41C120058 for ; Thu, 7 Jul 2022 22:36:25 +0000 (UTC) Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 267KPopQ010715 for ; Thu, 7 Jul 2022 15:36:24 -0700 Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3h5ashmbgn-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 07 Jul 2022 15:36:23 -0700 Received: from twshared0725.22.frc3.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Thu, 7 Jul 2022 15:36:19 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id 49BC09D349BD; Thu, 7 Jul 2022 15:36:00 -0700 (PDT) From: Song Liu To: , , CC: , , , , , , Song Liu Subject: [PATCH v6 bpf-next 0/5] bpf_prog_pack followup Date: Thu, 7 Jul 2022 15:35:41 -0700 Message-ID: <20220707223546.4124919-1-song@kernel.org> X-Mailer: git-send-email 2.30.2 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: zzDlYado5-T_BeGxrRBV8NvdQCYloIdI X-Proofpoint-GUID: zzDlYado5-T_BeGxrRBV8NvdQCYloIdI X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-07-07_17,2022-06-28_01,2022-06-22_01 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657233385; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references; bh=3MtMax0n2l37thkIcGV0NuGx3e374+5gMuuhnB72YgU=; b=DgSdY6G6cKL/NIkoWadQKWYRpfMxwq9Ave6aYWMqBNvNsFVPWqHCG1yEzaKUg+mhgYK5uS bY7zGXf7gI7D0P2Za41Db8AQ3gnmsaS3Wr5jeYPZ1SPM6uLJOA/BvrdyNoMIwn729eePdu qjOyrmZVOuIAc2WKUUFgqsln+qvs8Nc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657233385; a=rsa-sha256; cv=none; b=KB8hdiLrygMdSEcntfbpFzVAmcKW9XyQ/t547XUS0mj/THjksIe4Pu1eZmDhxBdLf/WJ2Q fxyizBt4n+QCt8yfahZj0EpXerpKFZt10VS1rKUYl4F/QR11bAy+ReV03jtomlCOqeyMmD ITPB54zqeufiJ4II390qtgoLiixjf8I= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none); spf=none (imf29.hostedemail.com: domain of "prvs=8187d1962c=songliubraving@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=8187d1962c=songliubraving@fb.com" Authentication-Results: imf29.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none); spf=none (imf29.hostedemail.com: domain of "prvs=8187d1962c=songliubraving@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=8187d1962c=songliubraving@fb.com" X-Stat-Signature: r8egudf7tmfrbq4m9gefep685d1oqqzw X-Rspamd-Queue-Id: 5C41C120058 X-Rspamd-Server: rspam07 X-Rspam-User: X-HE-Tag: 1657233385-918087 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This set is the second half of v4 [1]. Changes v5 => v6: 1. Rebase and extend CC list. Changes v4 => v5: 1. Rebase and resolve conflicts due to module.c split. 2. Update experiment results (below). For our web service production benchmark, bpf_prog_pack on 4kB pages gives 0.5% to 0.7% more throughput than not using bpf_prog_pack. bpf_prog_pack on 2MB pages 0.6% to 0.9% more throughput than not using bpf_prog_pack. Note that 0.5% is a huge improvement for our fleet. I believe this is also significant for other companies with many thousand servers. Update: Further experiments (suggested by Rick Edgecombe) showed that most of benefit on the web service benchmark came from less direct map fragmentation. The experiment is as follows: Side A: 2MB bpf prog pack on a single 2MB page; Side B: 2MB bpf prog pack on 512x 4kB pages; The system only uses about 200kB for BPF programs, but 2MB is allocated for bpf_prog_pack (for both A and B). Therefore, direct map fragmentation caused by BPF programs is elminated, and we are only measuring the performance difference of 1x 2MB page vs. ~50 4kB pages (we only use about 50 out of the 512 pages). For these two sides, the difference in system throughput is within the noise. I also measured iTLB-load-misses caused by bpf programs, which is ~300/s for case A, and ~1600/s for case B. The overall iTLB-load-misses is about 1.5M/s on these hosts. Therefore, we can clearly see 2MB page reduces iTLB misses, but the difference is not enough to have visible impact on system throughput. Of course, the impact of iTLB miss will be more significant for systems with more BPF programs loaded. [1] https://lore.kernel.org/bpf/20220520235758.1858153-1-song@kernel.org/ Song Liu (5): module: introduce module_alloc_huge bpf: use module_alloc_huge for bpf_prog_pack vmalloc: WARN for set_vm_flush_reset_perms() on huge pages vmalloc: introduce huge_vmalloc_supported bpf: simplify select_bpf_prog_pack_size arch/x86/kernel/module.c | 21 +++++++++++++++++++++ include/linux/moduleloader.h | 5 +++++ include/linux/vmalloc.h | 7 +++++++ kernel/bpf/core.c | 25 ++++++++++--------------- kernel/module/main.c | 8 ++++++++ mm/vmalloc.c | 5 +++++ 6 files changed, 56 insertions(+), 15 deletions(-) --- 2.30.2