From patchwork Sat Feb 22 02:44:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13986518 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0C35C021B3 for ; Sat, 22 Feb 2025 02:45:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C5B2280006; Fri, 21 Feb 2025 21:45:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 75100280002; Fri, 21 Feb 2025 21:45:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C80A280006; Fri, 21 Feb 2025 21:45:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 399A2280002 for ; Fri, 21 Feb 2025 21:45:03 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B425150A82 for ; Sat, 22 Feb 2025 02:45:02 +0000 (UTC) X-FDA: 83146038444.11.E26C48E Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf30.hostedemail.com (Postfix) with ESMTP id DF9E680008 for ; Sat, 22 Feb 2025 02:45:00 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fFlXZiOG; spf=pass (imf30.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740192300; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SPs3yc+PiAaWT0mDird4KgOQtxAcHODpsiThplYmhyc=; b=iatWt9wQ8yWwtiquCv2JP/cipmSAmiD/Y1rnSmavSUCaU36mjezbQ7W7yfkTxFvLz2EDOi pcutxWxIzT38nX2jc0rRJuzt9UIMgoaEyO+jEUsYd//xtMICGavM3J0GdXjK2QqrkL/p2t 4Boz8UxtqBMfKlheKHYUV24sR0NhNE8= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fFlXZiOG; spf=pass (imf30.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740192300; a=rsa-sha256; cv=none; b=lSh8dp/ocRaFRxxrII/xZHpL9dgz6bl+OjVskTzyt//rwVfj5JmBMY3Nuj+FZQPvoi4yXN /7tLg5umkyDKRR7/WOFJBoHHMk/hzr5PYJI1ps6Oj8eVR6qxT0e6k3trqOBDTdgXk3CzwG PeBh+FsKWX4LpuxFZDgOHg/YYlPImOs= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-220d28c215eso43583885ad.1 for ; Fri, 21 Feb 2025 18:45:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740192300; x=1740797100; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SPs3yc+PiAaWT0mDird4KgOQtxAcHODpsiThplYmhyc=; b=fFlXZiOGdukoaj5Nn1V+AnDlvQxEF+OAeoY+us1Ypgm9NUUkRydW+gNQWNcD5I9hey iVHN6sJgBA4WUrA95ARnnnNtF01kYvub8FgHpe7qkxDRHEAFwOUDvxq9ff32zuQqHdnl g5BO4OYitH4zCAPZOQI4vBkSv3VWye3bIvcR8RyGabzSU84ga03z/N9juQENaIqV1vmr s1w9dijl15RGyy5sqhWBBRUOK+pwWY8haEkEX1PLcXZl/ejRuGb5SqVj4JTpV9PelAi6 00xSdLwjSNhxtWkWU7zeSAFv3MSyKTmL733HmZfRwMqNfQcPhhxHvV2uPq/HiobHZG93 95KA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740192300; x=1740797100; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SPs3yc+PiAaWT0mDird4KgOQtxAcHODpsiThplYmhyc=; b=nO/N7VZIJ2ptAQYx9rIW2nmlTELjzrsmP71ypMVlWb7jdkeJCzqQfm3h12D3s9K0Gj DLk/BV/HF2x7eMUD/k9mWY1mMBUny8Z1dtXiywa/BwTwQXwLWNUbEKX5PFBMzNbiZU9S dv4sOxVT4L7SwQ+yNnsFDcIQE+tBfxJi9AVovbNJYZ8+g++xNO87oexMy9yMKlkOq/Lm BWfS3FA2v3NBHqa2mhAJGWzrI7LFKSOuXEAnJTNn0qTX/LU0lp5aJpYuQl76tnw6oRLw jbZ9HU7xmMiO6Vvi85Sue65OZZt8cvNBUsI4pVXaGxCMSMw3IFbnabL+irwZaELcFpIY NSkQ== X-Forwarded-Encrypted: i=1; AJvYcCVptSjHjDQoOaLWpi5r+p21GaxXEXK8tgke4p2lLu4SVTcpsx5MLM7AnSMbB7VJoFX3CRq3H+d+/w==@kvack.org X-Gm-Message-State: AOJu0YxS2K7nyUO8xjBq3cMUd77Etd5K1TeE/5GZkASiGqhrElDci/FC gIfqWBcMIGs+fzjT2uWupbWJoKiziDvDD3yJ8teM/wu55UlTbyUO X-Gm-Gg: ASbGncveRhn4BZOyZPy6JiA7i3jIXXQWBHWwIuXwx8F3r1GpvRQDIS4CdZDcKfU9BBH etbD1JA0aDbB7ykzo8xEu0+SbgUEJ8ovoTkU22lOA9hLG0nt2FeLHk2xxqis5Tt3epR0frL4nQp bZqhKL+aYqz2WUECGln+lXNRgCzLuk2WjAOXMxKVKxv8XhB56pZwtyqEnTSXRRlldzYUkwFY45e 94hAjPUZiPfaNXdUlr41DDnJBxNMtR2vKv4O3rParh4q4cMVHQDqBy25h9FWW9untfCjJ2LG0eg 3rVxyX6+Hh+Eikj3BktZczSHY+LuMj/i3Ox65MLACi9UcOeUV1LTjg== X-Google-Smtp-Source: AGHT+IGe5ruubfI6Kv2JX5V72ftGr8+QNktHSqGRav+UGawI2sLc8KeipECS2WW3ixuo2162gZmKxQ== X-Received: by 2002:a17:903:8c5:b0:21f:4c8b:c514 with SMTP id d9443c01a7336-2219fffa9f3mr82089215ad.45.1740192299832; Fri, 21 Feb 2025 18:44:59 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:fd1b]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d545d43dsm144772515ad.158.2025.02.21.18.44.57 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 21 Feb 2025 18:44:59 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v9 6/6] bpf: Use try_alloc_pages() to allocate pages for bpf needs. Date: Fri, 21 Feb 2025 18:44:27 -0800 Message-Id: <20250222024427.30294-7-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250222024427.30294-1-alexei.starovoitov@gmail.com> References: <20250222024427.30294-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: DF9E680008 X-Stat-Signature: ukwi56ibrqd3ikk5uu7wo16fq8eyyhrf X-Rspam-User: X-HE-Tag: 1740192300-792729 X-HE-Meta: U2FsdGVkX1+Z8f23zzdBCSQH5l3Dzo9DYvM+1kZsvSHQLoUYtPlWR9aJdB/FqklYgPIL5EyTCH7HPVdvuc92/jurVIxHwwp5EkMUN014roKyPyRSJa7t0C4YdQlsouOELDhAGSnDtWMs2jwZ4ISYMJTMOe/ywN+FN0BDLiIJjzg98oeTWjShlPiejXImHqw4kfbP2b8jBuNkJ2eT6S2II2VC/8Ad3CIs5tnJI1JVCVNSSujqPY6f15qJCaIuQ3K3niv6FdogVbO9uAk5OfodbfR5PPAeZO7Xmv7GZ1OcMC3+08PX9rVCSIsQ10z8CtTOWW0IBi2/QX39DjqOWUM0P0mIWOxqnd8/XXIVDjvCUfBX9nlocqOultrpUXFP6Fz6OFzUdzHvm8JSgf0G/rViJ8wQnpomb51V4I7068OQBLvqXbjdXoVW0PtmZ17OgCxB23z1Nj3tgcVxdVnFdcnnh7NtZSU86GJoCfo7yN0EKNCH7b/3Q3QSLRT1qMLwGFP+0AYW6USna6JrkAstaZhaXADqlTmuvNWNBEHHPYI7icBuNyA2VMBNUhKoQnKavc4y8E6E3kNBs4MpiPCNQl3qQi3/j2i6W6fsJ2uTgQI84597p0kE/1R+ItyXH68UciGBnKTKdvoSCrlOKSQKs1ZrrROIgPUbsz6eIOcHmQr6Qddh/VbtEVAkhQRWR2UqaOJaH7JudJF2ty88G7vm10SFp1i36w1lbNqioFK20kgwteOPgjD6OhW231oTQVlQDXXiqqlz55Ko0K/Qfr5Tyq+hHTmtq9qeTewFe2Xoc24YSvBE89XhuTQ9c1/evj1yI4xZZvoYs2CybiIIaPUpYfZuN0wFv46hXCupdszjbcmqTpXh7YgpBciTUSl6rFb38ISagTejT06/UcBXb5tsrHz+myw9A6NXZWv9ZjLpmIBgIwiKC6YivWF9HRW6czWvyaPeqJ5NTrDLQ6NgtH3V2yC WKXQJBbX F2caaV+PJVLYrvBP8CmBB+GPQeEYnkHmCCMzVHrHjPWN+K8X4o9wfGMhfis6e07q2rCEhujUE2hraFL0kJ1LV2+BjsbjLgAg5kGp9TbwLfoiwwLDxk9554rtgS6w5/SNBoQ+yRILqwZku16lxbm14edSx0AJ+MgPr+z6Mo+EJ7eQa78qEv++aYdEmg7jAzKVyE5A3SZ12US0+e1Jxa6yi9J20iYjVLmd8zvFkZIFTGvdgLKQD+/ZCpYMM3xdKSQG+MOV1wWTj7a8e9oNmDt07Y+dq1E3l4+f6Xn6vidwf0Hus3W4mBkBVYx6mPnn2X2d2kPf0NsmWOqwFplouRNj1KTvRNAkYFnf2XG435mkRV8g748hU+LhcSAFXLeWDYjvM3Q4EvcZCXtkjXqLywF+zixBOMwVgqsF8VIAebRvypJrqYVDn1akkya/bnQqlZ66/FDJ+MrNpAUmodn/snNk3ROeEod55Ij2nPFpwQk1CKsuzKOdlX+BBaTqoGg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Use try_alloc_pages() and free_pages_nolock() for BPF needs when context doesn't allow using normal alloc_pages. This is a prerequisite for further work. Signed-off-by: Alexei Starovoitov --- include/linux/bpf.h | 2 +- kernel/bpf/arena.c | 5 ++--- kernel/bpf/syscall.c | 23 ++++++++++++++++++++--- 3 files changed, 23 insertions(+), 7 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 15164787ce7f..aec102868b93 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -2354,7 +2354,7 @@ int generic_map_delete_batch(struct bpf_map *map, struct bpf_map *bpf_map_get_curr_or_next(u32 *id); struct bpf_prog *bpf_prog_get_curr_or_next(u32 *id); -int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, +int bpf_map_alloc_pages(const struct bpf_map *map, int nid, unsigned long nr_pages, struct page **page_array); #ifdef CONFIG_MEMCG void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c index 647b709d7d77..0d56cea71602 100644 --- a/kernel/bpf/arena.c +++ b/kernel/bpf/arena.c @@ -287,7 +287,7 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf) return VM_FAULT_SIGSEGV; /* Account into memcg of the process that created bpf_arena */ - ret = bpf_map_alloc_pages(map, GFP_KERNEL | __GFP_ZERO, NUMA_NO_NODE, 1, &page); + ret = bpf_map_alloc_pages(map, NUMA_NO_NODE, 1, &page); if (ret) { range_tree_set(&arena->rt, vmf->pgoff, 1); return VM_FAULT_SIGSEGV; @@ -465,8 +465,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt if (ret) goto out_free_pages; - ret = bpf_map_alloc_pages(&arena->map, GFP_KERNEL | __GFP_ZERO, - node_id, page_cnt, pages); + ret = bpf_map_alloc_pages(&arena->map, node_id, page_cnt, pages); if (ret) goto out; diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index dbd89c13dd32..28680896c6a4 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -569,7 +569,24 @@ static void bpf_map_release_memcg(struct bpf_map *map) } #endif -int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, +static bool can_alloc_pages(void) +{ + return preempt_count() == 0 && !irqs_disabled() && + !IS_ENABLED(CONFIG_PREEMPT_RT); +} + +static struct page *__bpf_alloc_page(int nid) +{ + if (!can_alloc_pages()) + return try_alloc_pages(nid, 0); + + return alloc_pages_node(nid, + GFP_KERNEL | __GFP_ZERO | __GFP_ACCOUNT + | __GFP_NOWARN, + 0); +} + +int bpf_map_alloc_pages(const struct bpf_map *map, int nid, unsigned long nr_pages, struct page **pages) { unsigned long i, j; @@ -582,14 +599,14 @@ int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, old_memcg = set_active_memcg(memcg); #endif for (i = 0; i < nr_pages; i++) { - pg = alloc_pages_node(nid, gfp | __GFP_ACCOUNT, 0); + pg = __bpf_alloc_page(nid); if (pg) { pages[i] = pg; continue; } for (j = 0; j < i; j++) - __free_page(pages[j]); + free_pages_nolock(pages[j], 0); ret = -ENOMEM; break; }