From patchwork Wed Aug 17 21:04:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 12946470 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76AC9C25B08 for ; Wed, 17 Aug 2022 21:04:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ACE528D0002; Wed, 17 Aug 2022 17:04:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A7C8E6B0074; Wed, 17 Aug 2022 17:04:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 96BA08D0002; Wed, 17 Aug 2022 17:04:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 889BB6B0073 for ; Wed, 17 Aug 2022 17:04:24 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 57E09817C1 for ; Wed, 17 Aug 2022 21:04:24 +0000 (UTC) X-FDA: 79810312806.26.DA47EFB Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by imf22.hostedemail.com (Postfix) with ESMTP id A70B9C0181 for ; Wed, 17 Aug 2022 21:04:23 +0000 (UTC) Received: by mail-pj1-f41.google.com with SMTP id x63-20020a17090a6c4500b001fabbf8debfso2185540pjj.4 for ; Wed, 17 Aug 2022 14:04:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc; bh=Cs6ZaYSOdLy0uVNOcAEYuyrIAH7fMhdm+14tTuynm/s=; b=pN5d0gKlMk3faOVL8p163QIYP49ZssaKhgF8GmlcN+hEvSr9FoDbJzKvfw7ECLuZSg ErR5J16cSFuLNIx9jrn84ZavQHz/6gkvpfu3pnHXBa3weYxco6tMchKRq5I5qLb9gWHf wbtFfVDeABbQRer860VZJN98Cunid31YJKMNrhgIgml8JA9EF4H8SxgEO0r5RXr5Ud8l lsLCShuG3gkwvzYk5HSVu33CiIQq9uZIH1GCCJG3zsi4EQPBv8tGqyK5vhBRv74QtuHP saWkCkh9o/3SjI6iZX/6K2eE9VgsuxnfR2H3kfilukuOxDIeGTzF0U8Tnruorvd0FTB1 DhbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc; bh=Cs6ZaYSOdLy0uVNOcAEYuyrIAH7fMhdm+14tTuynm/s=; b=HYewRnQARDBfUU9Sk5yjBHq5xw6X6svKEISzabsf77YmaBOlMcR5HSrXIkvRMC1Ttb ntRMxoc+oLSFa1N2WycKwY6l+nkaxvwMwaarCk3zofFSBtEH2CUYSenFHGFqR9dpOmYs lPxbjaaGp/2CtjBCDUD9xnyspbUEp+lSiFsmU9xZft4cQLQxaQUiPA9j5feW1EmCCEia FmPUCN2TiINuL2vbRPSdjXTs2wEmX0vlJoqEJ8YS0tCKTa3JdNF5i4Z4jcDAvpSHz6E8 jmJqqw17sMFzJaSnimwB+/10Dx4g3qeLm2BHgE2vRQWoS3nU299Tc8BPEzfCb7VoUrn6 WmBg== X-Gm-Message-State: ACgBeo0pz5OgKbneD+91h7AjUVYphQcPD84Yg/3D0e0aHsjhLYg+U22E WmT8BIY35FJLoMYa1i/Mh8Y= X-Google-Smtp-Source: AA6agR64XKxcpMiyQcC7Y5kPr+MTWmGE1PdbZe+2g7xU3EQPACQlIh3X8rfgqZlqcp1cprEbXCMIWA== X-Received: by 2002:a17:90b:4c12:b0:1f5:958:c313 with SMTP id na18-20020a17090b4c1200b001f50958c313mr5614957pjb.6.1660770262462; Wed, 17 Aug 2022 14:04:22 -0700 (PDT) Received: from localhost.localdomain ([2620:10d:c090:500::1:ccd6]) by smtp.gmail.com with ESMTPSA id u15-20020a170903124f00b0016c46ff9741sm375198plh.67.2022.08.17.14.04.21 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 17 Aug 2022 14:04:22 -0700 (PDT) From: Alexei Starovoitov To: davem@davemloft.net Cc: daniel@iogearbox.net, andrii@kernel.org, tj@kernel.org, memxor@gmail.com, delyank@fb.com, linux-mm@kvack.org, bpf@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 00/12] bpf: BPF specific memory allocator. Date: Wed, 17 Aug 2022 14:04:07 -0700 Message-Id: <20220817210419.95560-1-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.36.1 MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=pN5d0gKl; spf=pass (imf22.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660770262; a=rsa-sha256; cv=none; b=8EXFfRZnBIC2AsxcXUdL2CiFJQSEfLgDqta8vxDaROr/IGIlNcH2Gd3AJG9iycgz5w1W5r tCGFJblt6cEeRIg2aZBwjajQZqAxWlNEvK3AH5CxYjokr7QzVEgnOOk9ZBcLVKWGgXeGop jy9WQ4r1nTGOFiQfHd9l4geTKoETEtg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660770262; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=Cs6ZaYSOdLy0uVNOcAEYuyrIAH7fMhdm+14tTuynm/s=; b=Ftar3ahjH7pOvZhWJqtaJ1/rMnm9AA2D3O4SG9fn8kvZyI6GPqBSPC1KFehxn2YGfWPTtZ 19BsdkkNOU7vnj5EHnMdEeyeX7wEoKgCAPostc06AfVs2E2wyQYwV4oQR0m+oLnqVsfAEj +XErZYvDvedcO+dxBMXsIkSKCQdRKys= X-Rspam-User: X-Rspamd-Queue-Id: A70B9C0181 X-Stat-Signature: yrfstyt76rn7q9e18c4a456wxxjbd3i1 Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=pN5d0gKl; spf=pass (imf22.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam11 X-HE-Tag: 1660770263-350544 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alexei Starovoitov Introduce any context BPF specific memory allocator. Tracing BPF programs can attach to kprobe and fentry. Hence they run in unknown context where calling plain kmalloc() might not be safe. Front-end kmalloc() with per-cpu cache of free elements. Refill this cache asynchronously from irq_work. v1->v2: - Moved unsafe direct call_rcu() from hash map into safe place inside bpf_mem_alloc. Patches 7 and 9. - Optimized atomic_inc/dec in hash map with percpu_counter. Patch 6. - Tuned watermarks per allocation size. Patch 8 - Adopted this approach to per-cpu allocation. Patch 10. - Fully converted hash map to bpf_mem_alloc. Patch 11. - Removed tracing prog restriction on map types. Combination of all patches and final patch 12. v1 thread: https://lore.kernel.org/bpf/20220623003230.37497-1-alexei.starovoitov@gmail.com/ LWN article: https://lwn.net/Articles/899274/ There is a lot more work ahead, but this set is useful base. Future work: - allow sleepable bpf progs use dynamically allocated maps - expose bpf_mem_alloc as uapi FD to be used in dynptr_alloc, kptr_alloc - add sysctl to force bpf_mem_alloc in hash map when safe even if pre-alloc requested to reduce memory consumption - convert lru map to bpf_mem_alloc Alexei Starovoitov (12): bpf: Introduce any context BPF specific memory allocator. bpf: Convert hash map to bpf_mem_alloc. selftests/bpf: Improve test coverage of test_maps samples/bpf: Reduce syscall overhead in map_perf_test. bpf: Relax the requirement to use preallocated hash maps in tracing progs. bpf: Optimize element count in non-preallocated hash map. bpf: Optimize call_rcu in non-preallocated hash map. bpf: Adjust low/high watermarks in bpf_mem_cache bpf: Batch call_rcu callbacks instead of SLAB_TYPESAFE_BY_RCU. bpf: Add percpu allocation support to bpf_mem_alloc. bpf: Convert percpu hash map to per-cpu bpf_mem_alloc. bpf: Remove tracing program restriction on map types include/linux/bpf_mem_alloc.h | 26 + kernel/bpf/Makefile | 2 +- kernel/bpf/hashtab.c | 120 ++-- kernel/bpf/memalloc.c | 638 ++++++++++++++++++++++ kernel/bpf/syscall.c | 5 +- kernel/bpf/verifier.c | 29 - samples/bpf/map_perf_test_kern.c | 44 +- samples/bpf/map_perf_test_user.c | 2 +- tools/testing/selftests/bpf/progs/timer.c | 11 - tools/testing/selftests/bpf/test_maps.c | 38 +- 10 files changed, 811 insertions(+), 104 deletions(-) create mode 100644 include/linux/bpf_mem_alloc.h create mode 100644 kernel/bpf/memalloc.c