From patchwork Tue Dec 10 02:39:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13900669 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BE77070816 for ; Tue, 10 Dec 2024 02:39:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733798388; cv=none; b=gbLZHQusdszrRrq/xufPVaa7zM6aUyUp00y+CljCOWDi8u6QhTLJoF8YsOvejczm8QU13uzPJEZN9NtURFiT28NE+/Ob8yguS9NLcwB8Zrw9jzfk3rQbpAM/3FhPI/xdCuw5+m8Q61y11ANZnCxmE5wnnsFlRsjz1sykcKQrK04= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733798388; c=relaxed/simple; bh=2ahhnq6krYaIwI87BpkWONxMpUh4JoYZwltPnleHSbc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rqyHgOZFJq1WZyUu3uhoqPUJNxXDXAJ9j4O8mx3UHFWkinjvM7+SuSvyKrzHcOnkpfC+QeEqpnxcEWGN5MydxVLS74Xc+3T/6wXZm7hd0X8kPnx8KuVNHw4Ar6zM09fKyx1hFWOUuiVcY0/DCoh9opw56gYWIY2NQVxRhgU8ftE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=eKIIOrVA; arc=none smtp.client-ip=209.85.215.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="eKIIOrVA" Received: by mail-pg1-f181.google.com with SMTP id 41be03b00d2f7-7feffe7cdb7so335294a12.1 for ; Mon, 09 Dec 2024 18:39:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733798385; x=1734403185; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8e35BB82VxfWbGWqr1nA/p1/ofzs8olLXSCBkO0zstU=; b=eKIIOrVAhn22n0Y/ZLJVtI2ucrpDg2hDuipv3JsCzdJ4d1NDtDmloOoRD2UPiVZ4RO YtuPynYYY/z/Q/wQ/xO3uhDRN3Mfu/lThVaKhpRnsvoEZmKYDzwF89b13adsG00ymmao yhLyH0qDcn6rscDvwAp1+4ByZn6XU+P/NXWjOkkkUR69hMzYXJghQ1gE2huunLU48Vro Zv7jvXYRasqghTlhOtdSh3+v8Y31OOsXrGpL2K6imE92+aHQt5fvHqPEDwswa2yf/jgQ b8Bk2oK52sitYa64jAP5Ct2kAXGMEfSzk8Y+XQkNtBxTu6zMEOnaSwmTLQizk9dW7UQ5 d0pg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733798385; x=1734403185; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8e35BB82VxfWbGWqr1nA/p1/ofzs8olLXSCBkO0zstU=; b=Jl3Ak8HcwhgY4iaryV5XMC+CK2OYQJGZnGorAQEsbr/ne4DlSlxstvTEak5bp4qje+ tL6IFy3gH2h6FYNiNZcyYOvbHpFfYr6k/1V130INJHFTeDvrKbnWR0yaktw2PaIimwzM UL6OpRS9aPZn/jqHIlvsS7AWCLM/3tgWXW5tEZM8yFm/bPbMTsic967YNKSnDxARjMbq 9ED3sNdzxRWQGcuedO+22vX3JwKy0jc99Xca/75y49V/H0aQwgftqz8cRH6oAW+92GGa H3Qi49WBuXeY9U2R5i0OHDLA9xwBbtxW1f+WsDCZWFITO3uikyp8+3UKdvVDnd1PHlDa BV1g== X-Gm-Message-State: AOJu0YymyPCSkqn35UzEU5nwxAxxP3CQwCPP0kXa37ZtqFzfUjWPU6qk 4ltXZLP4eemEue4n5rnJhICYQX1jGh2X53kvV9yspbmN8vJyVNBE8q70rg== X-Gm-Gg: ASbGncuNqfbwafHvKo7lGNiMeVqWaFrugVaosFZ+rAmrGFOM/86ly5MLf5qjtlCyIWn 05vH/ADtEHiIm5X8OdLlS34Q7SMCN4jKbGp1esh+xAjTsURCEuPk0g3Zx3KBhkUT3C7cELemVJt 1L7U+uLe10T2rWa/sHtnchYqnzZE2Zx4z7c2jn5Gx4xsT0dYTuPkfn9mzX2tYfBffJIvfaIvLi+ 9HeXAlGEVQjeNSvaa98Z5yCmyqlca4E6J96RBVfGh0CLH8C6DOP1xrw8Bx9hIxD+dEnxIx34lyL 8UsBEA== X-Google-Smtp-Source: AGHT+IETlDQVakNHy9XhiE9Y3JSlfzGxx7wfgdM3m/HG+ucSQQHFTGfnOzVwILgjB0Zho8MlhPqG4w== X-Received: by 2002:a17:90b:2e43:b0:2ee:acb4:fecd with SMTP id 98e67ed59e1d1-2ef69e16bd4mr21878450a91.9.1733798385246; Mon, 09 Dec 2024 18:39:45 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:83b0]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2ef4600d2c4sm9544043a91.47.2024.12.09.18.39.43 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 09 Dec 2024 18:39:44 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v2 1/6] mm, bpf: Introduce __GFP_TRYLOCK for opportunistic page allocation Date: Mon, 9 Dec 2024 18:39:31 -0800 Message-Id: <20241210023936.46871-2-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241210023936.46871-1-alexei.starovoitov@gmail.com> References: <20241210023936.46871-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Tracing BPF programs execute from tracepoints and kprobes where running context is unknown, but they need to request additional memory. The prior workarounds were using pre-allocated memory and BPF specific freelists to satisfy such allocation requests. Instead, introduce __GFP_TRYLOCK flag that makes page allocator accessible from any context. It relies on percpu free list of pages that rmqueue_pcplist() should be able to pop the page from. If it fails (due to IRQ re-entrancy or list being empty) then try_alloc_pages() attempts to spin_trylock zone->lock and refill percpu freelist as normal. BPF program may execute with IRQs disabled and zone->lock is sleeping in RT, so trylock is the only option. In theory we can introduce percpu reentrance counter and increment it every time spin_lock_irqsave(&zone->lock, flags) is used, but we cannot rely on it. Even if this cpu is not in page_alloc path the spin_lock_irqsave() is not safe, since BPF prog might be called from tracepoint where preemption is disabled. So trylock only. Note, free_page and memcg are not taught about __GFP_TRYLOCK yet. The support comes in the next patches. This is a first step towards supporting BPF requirements in SLUB and getting rid of bpf_mem_alloc. That goal was discussed at LSFMM: https://lwn.net/Articles/974138/ Signed-off-by: Alexei Starovoitov --- include/linux/gfp.h | 25 +++++++++++++++++++++++++ include/linux/gfp_types.h | 3 +++ include/trace/events/mmflags.h | 1 + mm/fail_page_alloc.c | 6 ++++++ mm/internal.h | 1 + mm/page_alloc.c | 17 ++++++++++++++--- tools/perf/builtin-kmem.c | 1 + 7 files changed, 51 insertions(+), 3 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index b0fe9f62d15b..f68daa9c997b 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -347,6 +347,31 @@ static inline struct page *alloc_page_vma_noprof(gfp_t gfp, } #define alloc_page_vma(...) alloc_hooks(alloc_page_vma_noprof(__VA_ARGS__)) +static inline struct page *try_alloc_pages_noprof(int nid, unsigned int order) +{ + /* + * If spin_locks are not held and interrupts are enabled, use normal + * path. BPF progs run under rcu_read_lock(), so in PREEMPT_RT + * rcu_preempt_depth() will be >= 1 and will use trylock path. + */ + if (preemptible() && !rcu_preempt_depth()) + return alloc_pages_node_noprof(nid, + GFP_NOWAIT | __GFP_ZERO, + order); + /* + * Best effort allocation from percpu free list. + * If it's empty attempt to spin_trylock zone->lock. + * Do not specify __GFP_KSWAPD_RECLAIM to avoid wakeup_kswapd + * that may need to grab a lock. + * Do not specify __GFP_ACCOUNT to avoid local_lock. + * Do not warn either. + */ + return alloc_pages_node_noprof(nid, + __GFP_TRYLOCK | __GFP_NOWARN | __GFP_ZERO, + order); +} +#define try_alloc_pages(...) alloc_hooks(try_alloc_pages_noprof(__VA_ARGS__)) + extern unsigned long get_free_pages_noprof(gfp_t gfp_mask, unsigned int order); #define __get_free_pages(...) alloc_hooks(get_free_pages_noprof(__VA_ARGS__)) diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index 65db9349f905..72b385a7888d 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -48,6 +48,7 @@ enum { ___GFP_THISNODE_BIT, ___GFP_ACCOUNT_BIT, ___GFP_ZEROTAGS_BIT, + ___GFP_TRYLOCK_BIT, #ifdef CONFIG_KASAN_HW_TAGS ___GFP_SKIP_ZERO_BIT, ___GFP_SKIP_KASAN_BIT, @@ -86,6 +87,7 @@ enum { #define ___GFP_THISNODE BIT(___GFP_THISNODE_BIT) #define ___GFP_ACCOUNT BIT(___GFP_ACCOUNT_BIT) #define ___GFP_ZEROTAGS BIT(___GFP_ZEROTAGS_BIT) +#define ___GFP_TRYLOCK BIT(___GFP_TRYLOCK_BIT) #ifdef CONFIG_KASAN_HW_TAGS #define ___GFP_SKIP_ZERO BIT(___GFP_SKIP_ZERO_BIT) #define ___GFP_SKIP_KASAN BIT(___GFP_SKIP_KASAN_BIT) @@ -293,6 +295,7 @@ enum { #define __GFP_COMP ((__force gfp_t)___GFP_COMP) #define __GFP_ZERO ((__force gfp_t)___GFP_ZERO) #define __GFP_ZEROTAGS ((__force gfp_t)___GFP_ZEROTAGS) +#define __GFP_TRYLOCK ((__force gfp_t)___GFP_TRYLOCK) #define __GFP_SKIP_ZERO ((__force gfp_t)___GFP_SKIP_ZERO) #define __GFP_SKIP_KASAN ((__force gfp_t)___GFP_SKIP_KASAN) diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index bb8a59c6caa2..592c93ee5f35 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -50,6 +50,7 @@ gfpflag_string(__GFP_RECLAIM), \ gfpflag_string(__GFP_DIRECT_RECLAIM), \ gfpflag_string(__GFP_KSWAPD_RECLAIM), \ + gfpflag_string(__GFP_TRYLOCK), \ gfpflag_string(__GFP_ZEROTAGS) #ifdef CONFIG_KASAN_HW_TAGS diff --git a/mm/fail_page_alloc.c b/mm/fail_page_alloc.c index 7647096170e9..b3b297d67909 100644 --- a/mm/fail_page_alloc.c +++ b/mm/fail_page_alloc.c @@ -31,6 +31,12 @@ bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order) return false; if (gfp_mask & __GFP_NOFAIL) return false; + if (gfp_mask & __GFP_TRYLOCK) + /* + * Internals of should_fail_ex() are not compatible + * with trylock concept. + */ + return false; if (fail_page_alloc.ignore_gfp_highmem && (gfp_mask & __GFP_HIGHMEM)) return false; if (fail_page_alloc.ignore_gfp_reclaim && diff --git a/mm/internal.h b/mm/internal.h index cb8d8e8e3ffa..c082b8fa1d71 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1175,6 +1175,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, #endif #define ALLOC_HIGHATOMIC 0x200 /* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_KSWAPD 0x800 /* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ +#define ALLOC_TRYLOCK 0x1000000 /* Only use spin_trylock in allocation path */ /* Flags that allow allocations below the min watermark. */ #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 1cb4b8c8886d..d511e68903c6 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2304,7 +2304,11 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, unsigned long flags; int i; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(alloc_flags & ALLOC_TRYLOCK)) + return 0; + spin_lock_irqsave(&zone->lock, flags); + } for (i = 0; i < count; ++i) { struct page *page = __rmqueue(zone, order, migratetype, alloc_flags); @@ -2904,7 +2908,11 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, do { page = NULL; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)){ + if (unlikely(alloc_flags & ALLOC_TRYLOCK)) + return NULL; + spin_lock_irqsave(&zone->lock, flags); + } if (alloc_flags & ALLOC_HIGHATOMIC) page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC); if (!page) { @@ -4001,6 +4009,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) */ BUILD_BUG_ON(__GFP_HIGH != (__force gfp_t) ALLOC_MIN_RESERVE); BUILD_BUG_ON(__GFP_KSWAPD_RECLAIM != (__force gfp_t) ALLOC_KSWAPD); + BUILD_BUG_ON(__GFP_TRYLOCK != (__force gfp_t) ALLOC_TRYLOCK); /* * The caller may dip into page reserves a bit more if the caller @@ -4009,7 +4018,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) * set both ALLOC_NON_BLOCK and ALLOC_MIN_RESERVE(__GFP_HIGH). */ alloc_flags |= (__force int) - (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM)); + (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM | __GFP_TRYLOCK)); if (!(gfp_mask & __GFP_DIRECT_RECLAIM)) { /* @@ -4509,6 +4518,8 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, might_alloc(gfp_mask); + *alloc_flags |= (__force int) (gfp_mask & __GFP_TRYLOCK); + if (should_fail_alloc_page(gfp_mask, order)) return false; diff --git a/tools/perf/builtin-kmem.c b/tools/perf/builtin-kmem.c index 4d8d94146f8d..1f7f4269fa10 100644 --- a/tools/perf/builtin-kmem.c +++ b/tools/perf/builtin-kmem.c @@ -682,6 +682,7 @@ static const struct { { "__GFP_RECLAIM", "R" }, { "__GFP_DIRECT_RECLAIM", "DR" }, { "__GFP_KSWAPD_RECLAIM", "KR" }, + { "__GFP_TRYLOCK", "TL" }, }; static size_t max_gfp_len; From patchwork Tue Dec 10 02:39:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13900670 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DAFFA27456 for ; Tue, 10 Dec 2024 02:39:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733798393; cv=none; b=EOyTRKiTLEqLsvYOBH6ctSFpj7PjV5YJ8ZeQznbeE39XM6JMO2J/POGNbwkUM1TCVlJcXq4Iqz3seeQSRt86oFLzRgvrITHqKIu5TuacpIhXUhPNqFPuE2m9J1seUUVLrDxQmhG6W3gcAl35lMFebPRETD8uU5M1oVF8aru9It8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733798393; c=relaxed/simple; bh=9h5GtWWC1L/C/cYZyIAto5Swc6ITBZJ7lAo/dVkICu4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=i/Hf55jTUhSiAkPu8gIhLVrn8Ni3UxWqZXtIYk0w0SAYZ2Qu2V2bVSbuz4SAY5SpVFwK/we6g5rc2FYcpGCOWDUelS7dw+lK8pF+BoYpOEttKXDSYNeE3HQSJWPqYMZDGfQ+eLSjp8ySvMM2t1f7iV7kyEL83S8KyF9U72Hq8Kc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Md5FQ4H0; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Md5FQ4H0" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2164b662090so15925575ad.1 for ; Mon, 09 Dec 2024 18:39:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733798390; x=1734403190; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XSOnooig0JqcXQ1Y/KpafF9RbsF0QDm5Q4FOwHWSe4I=; b=Md5FQ4H0CrVeTkiiGw/H2GDxw+lfBENIr+0jpuWCk93gPia3Z+WsLEvnUdghX42owg Iprb4bO0Hr5rJ7nXr/F1FtYH+qrWEDLrkwL6Ur5Y3/qZH1ZabWBpOrtGtYapD4GG9cFC xLJi7eGR99CQjcM8LDwuIKgxeXl2WlsHkVdZ1CuclOUdLwc0NWrLps2mOESTJKxpR4Ux ydT0XZttx2epLarmj5o/uzistzjSGfBSvP7SqguQQQtCrS24N9uu/dXfav8PK+yd3oH8 Qmndp+8aCcJDE/+IYCjD/GNCFsV9xEn9wHgHFJdtoRWysBFIF3lm50F34megX7HPnPIS IowQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733798390; x=1734403190; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XSOnooig0JqcXQ1Y/KpafF9RbsF0QDm5Q4FOwHWSe4I=; b=ZVCGcJ7XIf1q7uEAtQoc6Yy7Ri+i78jGays/XCb5znGJb58MUdwxgmnfdmGKInLv+9 cdiL/U84j5GngGfmANYKjvV2FGpQZmPEFS7L4g6ibZ00Vr+Obrn9hWQYIdFtgR4A80ov f+gbn0jcvKCR0d0dyY+mGH6v5bfC91v8tdCezjxiVtI3cKmjZKgHL5/LldZRQBq/U/Xz /aJTstjJKxFR+9hK9QPLpcrOFN4Z8R5blAQVtVw+E2Tgs1ypwzeJUBzepmr6wW1TOMHG mJeQybRlRP/GgAnsDuE3cK4KbMb9OFHcR7RuNHGTMc1dpVlwcMVTWXoMuCWFTQFtLnir 8MbQ== X-Gm-Message-State: AOJu0YxSiVlE1zOnaOOwxH1lQBAM/gYRdWU0yHiJkjomYIYxvLtON+xe ZiXFy2cCirH3qFRfk7IXieAoKTSvz9efVbP1c3Z4zp2kp0ixTfdrilqqug== X-Gm-Gg: ASbGncs9fSBJ9xTpf5NkJFEXSA6kI4M8PtkROAGjC5y9LvvE6n1xSU9N4JbhhmRGUVq WsWTXp3gTYcIRipMe1Uh4kGSKwaawJR2c0f8LUxesuOfeOdLIxMclpYrHPeCUalG4IYFI7DSl8C LkQzH6ahVM66gX5D/QW7LhM2k6a644uRrYL5JwkXdAK/wtCisfeoP+dJ1HFUJftfWqgXKXikEwK fAx7DACmMbRgH/ke6IbHmxM/PIpb8vj2Kuiwtnh9DTo4R+GIlBrwv2vvplfOKsqCk65NarDaOT/ yLkmCA== X-Google-Smtp-Source: AGHT+IFs9AQy6lVySBTzfSVqJTRJXQoQU6RtOKdvZs51As+Ukwzy42oozyaRiTjoeSpSVQBsyUhvdA== X-Received: by 2002:a17:903:110f:b0:212:996:3536 with SMTP id d9443c01a7336-21614d2e719mr238769345ad.10.1733798389764; Mon, 09 Dec 2024 18:39:49 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:83b0]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd45b78c7fsm2837312a12.15.2024.12.09.18.39.47 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 09 Dec 2024 18:39:49 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v2 2/6] mm, bpf: Introduce free_pages_nolock() Date: Mon, 9 Dec 2024 18:39:32 -0800 Message-Id: <20241210023936.46871-3-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241210023936.46871-1-alexei.starovoitov@gmail.com> References: <20241210023936.46871-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Introduce free_pages_nolock() that can free a page without taking locks. It relies on trylock only and can be called from any context. Signed-off-by: Alexei Starovoitov --- include/linux/gfp.h | 1 + include/linux/mm_types.h | 4 +++ include/linux/mmzone.h | 3 ++ mm/page_alloc.c | 72 +++++++++++++++++++++++++++++++++++----- 4 files changed, 72 insertions(+), 8 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index f68daa9c997b..dcae733ed006 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -394,6 +394,7 @@ __meminit void *alloc_pages_exact_nid_noprof(int nid, size_t size, gfp_t gfp_mas __get_free_pages((gfp_mask) | GFP_DMA, (order)) extern void __free_pages(struct page *page, unsigned int order); +extern void free_pages_nolock(struct page *page, unsigned int order); extern void free_pages(unsigned long addr, unsigned int order); #define __free_page(page) __free_pages((page), 0) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 7361a8f3ab68..52547b3e5fd8 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -99,6 +99,10 @@ struct page { /* Or, free page */ struct list_head buddy_list; struct list_head pcp_list; + struct { + struct llist_node pcp_llist; + unsigned int order; + }; }; /* See page-flags.h for PAGE_MAPPING_FLAGS */ struct address_space *mapping; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index b36124145a16..1a854e0a9e3b 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -953,6 +953,9 @@ struct zone { /* Primarily protects free_area */ spinlock_t lock; + /* Pages to be freed when next trylock succeeds */ + struct llist_head trylock_free_pages; + /* Write-intensive fields used by compaction and vmstats. */ CACHELINE_PADDING(_pad2_); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d511e68903c6..a969a62ec0c3 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -88,6 +88,9 @@ typedef int __bitwise fpi_t; */ #define FPI_TO_TAIL ((__force fpi_t)BIT(1)) +/* Free the page without taking locks. Rely on trylock only. */ +#define FPI_TRYLOCK ((__force fpi_t)BIT(2)) + /* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ static DEFINE_MUTEX(pcp_batch_high_lock); #define MIN_PERCPU_PAGELIST_HIGH_FRACTION (8) @@ -1251,9 +1254,33 @@ static void free_one_page(struct zone *zone, struct page *page, unsigned long pfn, unsigned int order, fpi_t fpi_flags) { + struct llist_head *llhead; unsigned long flags; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(fpi_flags & FPI_TRYLOCK)) { + /* Remember the order */ + page->order = order; + /* Add the page to the free list */ + llist_add(&page->pcp_llist, &zone->trylock_free_pages); + return; + } + spin_lock_irqsave(&zone->lock, flags); + } + + /* The lock succeeded. Process deferred pages. */ + llhead = &zone->trylock_free_pages; + if (unlikely(!llist_empty(llhead))) { + struct llist_node *llnode; + struct page *p, *tmp; + + llnode = llist_del_all(llhead); + llist_for_each_entry_safe(p, tmp, llnode, pcp_llist) { + unsigned int p_order = p->order; + split_large_buddy(zone, p, page_to_pfn(p), p_order, fpi_flags); + __count_vm_events(PGFREE, 1 << p_order); + } + } split_large_buddy(zone, page, pfn, order, fpi_flags); spin_unlock_irqrestore(&zone->lock, flags); @@ -2596,7 +2623,7 @@ static int nr_pcp_high(struct per_cpu_pages *pcp, struct zone *zone, static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, struct page *page, int migratetype, - unsigned int order) + unsigned int order, fpi_t fpi_flags) { int high, batch; int pindex; @@ -2631,6 +2658,14 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, } if (pcp->free_count < (batch << CONFIG_PCP_BATCH_SCALE_MAX)) pcp->free_count += (1 << order); + + if (unlikely(fpi_flags & FPI_TRYLOCK)) { + /* + * Do not attempt to take a zone lock. Let pcp->count get + * over high mark temporarily. + */ + return; + } high = nr_pcp_high(pcp, zone, batch, free_high); if (pcp->count >= high) { free_pcppages_bulk(zone, nr_pcp_free(pcp, batch, high, free_high), @@ -2645,7 +2680,8 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, /* * Free a pcp page */ -void free_unref_page(struct page *page, unsigned int order) +static void __free_unref_page(struct page *page, unsigned int order, + fpi_t fpi_flags) { unsigned long __maybe_unused UP_flags; struct per_cpu_pages *pcp; @@ -2654,7 +2690,7 @@ void free_unref_page(struct page *page, unsigned int order) int migratetype; if (!pcp_allowed_order(order)) { - __free_pages_ok(page, order, FPI_NONE); + __free_pages_ok(page, order, fpi_flags); return; } @@ -2671,7 +2707,7 @@ void free_unref_page(struct page *page, unsigned int order) migratetype = get_pfnblock_migratetype(page, pfn); if (unlikely(migratetype >= MIGRATE_PCPTYPES)) { if (unlikely(is_migrate_isolate(migratetype))) { - free_one_page(page_zone(page), page, pfn, order, FPI_NONE); + free_one_page(page_zone(page), page, pfn, order, fpi_flags); return; } migratetype = MIGRATE_MOVABLE; @@ -2681,14 +2717,19 @@ void free_unref_page(struct page *page, unsigned int order) pcp_trylock_prepare(UP_flags); pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (pcp) { - free_unref_page_commit(zone, pcp, page, migratetype, order); + free_unref_page_commit(zone, pcp, page, migratetype, order, fpi_flags); pcp_spin_unlock(pcp); } else { - free_one_page(zone, page, pfn, order, FPI_NONE); + free_one_page(zone, page, pfn, order, fpi_flags); } pcp_trylock_finish(UP_flags); } +void free_unref_page(struct page *page, unsigned int order) +{ + __free_unref_page(page, order, FPI_NONE); +} + /* * Free a batch of folios */ @@ -2777,7 +2818,7 @@ void free_unref_folios(struct folio_batch *folios) trace_mm_page_free_batched(&folio->page); free_unref_page_commit(zone, pcp, &folio->page, migratetype, - order); + order, FPI_NONE); } if (pcp) { @@ -4855,6 +4896,21 @@ void __free_pages(struct page *page, unsigned int order) } EXPORT_SYMBOL(__free_pages); +/* Can be called while holding raw_spin_lock or from IRQ. RCU must be watching. */ +void free_pages_nolock(struct page *page, unsigned int order) +{ + int head = PageHead(page); + struct alloc_tag *tag = pgalloc_tag_get(page); + + if (put_page_testzero(page)) { + __free_unref_page(page, order, FPI_TRYLOCK); + } else if (!head) { + pgalloc_tag_sub_pages(tag, (1 << order) - 1); + while (order-- > 0) + __free_unref_page(page + (1 << order), order, FPI_TRYLOCK); + } +} + void free_pages(unsigned long addr, unsigned int order) { if (addr != 0) { From patchwork Tue Dec 10 02:39:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13900671 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0A73B27456 for ; Tue, 10 Dec 2024 02:39:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733798397; cv=none; b=aRrt7Z21C0mfb7aWhFakkfmQpMTqZEkzGF6UvM/aAoPYCXb39aGlJQC/oJ0U/uqxjXLHzUm/rlxiPt7MNBXydjnCjrECQ+jvRnuv6octyHLovCKoIlDb1VjumUGeX1VO2tKXFGrSZ6QbKP0RNQXV1lAeYvCtWakoCg30bszHWc8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733798397; c=relaxed/simple; bh=KYyuDO6Ahs01hjYzBDC8IjaST0acLjXnJda3FVGwIv4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=DTGbL+ORl3dSYxCvCdhH8TFpgMA1p+yHaRsW4eWP+DUSeDOjwyoOolxblueZCMMNhsMvTrGUgYIFr56We4IUex9YKTa9r3LHi04gtC+Mgiw1BDC+nonPFAJ6C5d+30m2y1L6IAjNZ5r63qVNCz18YcyWpAc0MJRsqredKxXVuDc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=QJv1fdgK; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="QJv1fdgK" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-216426b0865so16316905ad.0 for ; Mon, 09 Dec 2024 18:39:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733798394; x=1734403194; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MT/DPo97yF9oIvPtZgU314ePao/rByOm5QF7hTFkWYs=; b=QJv1fdgKXn/2RhJSh1x8ajX87Hn2Na/ZZS31Rbg6VchynMN/sSbhIaOOedFR/xpsgm I7S1mHrDPS8V5S+/wa+r5VLQmMn5dvBjkrxPpckqDgsk7EdyGLCcqAPQoCWUsn6Z/WTQ DeWpRPN1mx5i8lJ8n0PcmNI1ATKiqPpGYJndu319edyB7of4Vhh9niRlvwOpMB4AE8kE B1QbkfMLu6/ObifX99iu72aUlB3dU45zNwv0JxaPUymJj2p69fa+9jbIZg/66wea2Si2 qjr/tmRdo27WhgM5l5J7qD9weFe5ljfJG4anr9xlDIh94Hh+8XjhMJx9l9DbraBdYJsh rlkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733798394; x=1734403194; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MT/DPo97yF9oIvPtZgU314ePao/rByOm5QF7hTFkWYs=; b=PZt7da0RWUuO4/kDmpCR8mU2kkq8PTc6/uOcoP+iWMzROuOao70sa1pBBikIPcHKwv h75jb4fAFeQropx1oEuXTappEgyV1PtGPrQr/4KKNb65SiWqr/BBufwWLR0PtjKL0FBw PLBYYOXrCE5ZJECRDzd2GmWt18xud/3pI1y4w59TWrddL5Lm2vNsOc1ENvX12pK5AGQW lIilhiW5D6jIwXPTRNaW0eC8ayieDcsFheKRr7lx9J672wmEgQP4HBwpGmT4jFGnLQxZ yemYZDEPoYSqvqtj20eGgeMs7/aMxiuj+hlsTn73qjkGu7oBBlleQzqQZLsFV9WtRm1/ ITAg== X-Gm-Message-State: AOJu0YyrVjplVK5YchI4cNBe0Mi9Cey4hb3w6cd/oXHJS4E+0/uMJ3bk /+MY2mVa/Va8+D427poYHgss2D1x8Sb7WnYzuUvAbxIfzCmc+uK71VgTgg== X-Gm-Gg: ASbGncuA2JJZWsk3survv4tr5IXEVmfHRYAwO4eSsV43JZuGkot+ul0rvk1s/vcbngy +oOoIfoCuYp4+2Fegy4ay6f6KbOh7lPSuuZ6iUTM6DdUAg4C7ixCeLggx64lBd32SexU9dze2uC 0H29sfmWBvr5NJy4az7b5tg6OG7ASi8/0wRpjos6Xx6oeuzcFI3JQ5JOrMvO6WijJZucrSRYkxl oOf4tIHlowNy67Qa+PjFut08/BnenV6odt5VNL8GaW99/eBHIihJIzoEsKG1pf6IIWTAuhiLI6b q6tBZA== X-Google-Smtp-Source: AGHT+IGKLmePoGcXe+HAKRjlAgf3YSGz9J3cPaKQKdSbicAELPOmItuHpzzqliawMg0yEsYXj3ExxQ== X-Received: by 2002:a17:902:ecc5:b0:215:4fbf:11da with SMTP id d9443c01a7336-21669fc28d6mr43610845ad.21.1733798394490; Mon, 09 Dec 2024 18:39:54 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:83b0]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-215f8e5f031sm79221425ad.82.2024.12.09.18.39.52 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 09 Dec 2024 18:39:54 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v2 3/6] locking/local_lock: Introduce local_trylock_irqsave() Date: Mon, 9 Dec 2024 18:39:33 -0800 Message-Id: <20241210023936.46871-4-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241210023936.46871-1-alexei.starovoitov@gmail.com> References: <20241210023936.46871-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Similar to local_lock_irqsave() introduce local_trylock_irqsave(). It uses spin_trylock in PREEMPT_RT and always succeeds when !RT. Signed-off-by: Alexei Starovoitov --- include/linux/local_lock.h | 9 +++++++++ include/linux/local_lock_internal.h | 23 +++++++++++++++++++++++ 2 files changed, 32 insertions(+) diff --git a/include/linux/local_lock.h b/include/linux/local_lock.h index 091dc0b6bdfb..6880c29b8b98 100644 --- a/include/linux/local_lock.h +++ b/include/linux/local_lock.h @@ -30,6 +30,15 @@ #define local_lock_irqsave(lock, flags) \ __local_lock_irqsave(lock, flags) +/** + * local_trylock_irqsave - Try to acquire a per CPU local lock, save and disable + * interrupts. Always succeeds in !PREEMPT_RT. + * @lock: The lock variable + * @flags: Storage for interrupt flags + */ +#define local_trylock_irqsave(lock, flags) \ + __local_trylock_irqsave(lock, flags) + /** * local_unlock - Release a per CPU local lock * @lock: The lock variable diff --git a/include/linux/local_lock_internal.h b/include/linux/local_lock_internal.h index 8dd71fbbb6d2..2c0f8a49c2d0 100644 --- a/include/linux/local_lock_internal.h +++ b/include/linux/local_lock_internal.h @@ -31,6 +31,13 @@ static inline void local_lock_acquire(local_lock_t *l) l->owner = current; } +static inline void local_trylock_acquire(local_lock_t *l) +{ + lock_map_acquire_try(&l->dep_map); + DEBUG_LOCKS_WARN_ON(l->owner); + l->owner = current; +} + static inline void local_lock_release(local_lock_t *l) { DEBUG_LOCKS_WARN_ON(l->owner != current); @@ -45,6 +52,7 @@ static inline void local_lock_debug_init(local_lock_t *l) #else /* CONFIG_DEBUG_LOCK_ALLOC */ # define LOCAL_LOCK_DEBUG_INIT(lockname) static inline void local_lock_acquire(local_lock_t *l) { } +static inline void local_trylock_acquire(local_lock_t *l) { } static inline void local_lock_release(local_lock_t *l) { } static inline void local_lock_debug_init(local_lock_t *l) { } #endif /* !CONFIG_DEBUG_LOCK_ALLOC */ @@ -91,6 +99,13 @@ do { \ local_lock_acquire(this_cpu_ptr(lock)); \ } while (0) +#define __local_trylock_irqsave(lock, flags) \ + ({ \ + local_irq_save(flags); \ + local_trylock_acquire(this_cpu_ptr(lock)); \ + 1; \ + }) + #define __local_unlock(lock) \ do { \ local_lock_release(this_cpu_ptr(lock)); \ @@ -148,6 +163,14 @@ typedef spinlock_t local_lock_t; __local_lock(lock); \ } while (0) +#define __local_trylock_irqsave(lock, flags) \ + ({ \ + typecheck(unsigned long, flags); \ + flags = 0; \ + migrate_disable(); \ + spin_trylock(this_cpu_ptr((__lock))); \ + }) + #define __local_unlock(__lock) \ do { \ spin_unlock(this_cpu_ptr((__lock))); \ From patchwork Tue Dec 10 02:39:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13900672 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pg1-f172.google.com (mail-pg1-f172.google.com [209.85.215.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5D5D670816 for ; Tue, 10 Dec 2024 02:40:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733798402; cv=none; b=L6dfmUzcSyx9COcsr7Qr2/0DsxRv3ne4x0KuUOoAtS+JF89P/JxPFywvpGYAJXhG5nE8DCPtqEbIRdxA92cjeT+FWZipJyKf4q+f+1dHooUzLc38mfw8KDpctOU5aNtQfvJuYTSQxYjWU2007ECe+looNfnj4ExvX3Xi2ilmsSs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733798402; c=relaxed/simple; bh=eeArVuRQjLMkKnmW3cNtegLJbfRg67rqk55sMUQzZ/8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=PeD2AbuaNh/9618bB0YSAC98IXRR3PMFrbiHhl7kcXVJ4wv0HRI+o9euwiHuSw47AfYXBeZbxFThR2ZWiR75XL8ZkAMw6UXKgQKY3FLodKZo3UK0PpIFN0bEXhtihuul5YO+aVdcQxRubexbqbUiCTDz3A5HI+FtW2R8zpdXwGE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=S2YGs4Im; arc=none smtp.client-ip=209.85.215.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="S2YGs4Im" Received: by mail-pg1-f172.google.com with SMTP id 41be03b00d2f7-7ee51d9ae30so3335275a12.1 for ; Mon, 09 Dec 2024 18:40:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733798399; x=1734403199; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wo7sUpYgRQrXUd48YB6NzK0xhu80+vJLM/DolMWSp9w=; b=S2YGs4ImZobTOFiXGnByqwV+eDWhJ8AInKpMkPQyfBJYgFd6iUTcqkN6gIP6rXXkqJ R4wiJ3oAqszUBmvSxNQ6pbhazFEVq6bzFlFkUAz0RbFVkvQk1eZWZciyu45LDlj4mh/O qwGiDKzDQrevQNVlAxUHABtIDWoBPudvEPkwhXPxEmyCXwBEVFcIDnW3dQS4JRhcolJD DKGl3X+gGCQKCqzh+fLiGtlWZF+tuH7O7bzi09NLWMKW1s0KItCjgowlgPPZ3MLIgFY9 aRVYSgAZG0kCifZ7gchg+hBT5RfgKXA/VMF6sQls/NhWFCNu1hD3PHmyYraCZwww1luk hC3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733798399; x=1734403199; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wo7sUpYgRQrXUd48YB6NzK0xhu80+vJLM/DolMWSp9w=; b=G190Xp0jYu0zJbx43/8rY8dfY2QYj2P2ebYLEZdRV7w0X4a6ZMwyecG7pU+EGKaE7z x9ogmI96HBDCx9Nq4Iikki/zccDvwUMCL5F5xtdrJRBJh6baOrNGLGPBmiQ/dZUaBD9X ATpnGbVqfJGX42O8N6ZREwMtJEL7N+VF1Tqh5DPJVdM+JuNhE9ZHHjNbqV95Dev9Mx00 xuvdz4CUjDYjzylgMYViLn4yoD8Udo1MeoZAXPk9d0Tp+Usm5Dw3dXL6H0lAHg193Ros i1+3UlaszM1d0s8QV5piEsHVHYV39EX2BdRv2b4TWukMxCFmhMtvi+qcutmUKEPqKfKW WPJQ== X-Gm-Message-State: AOJu0YwYakJMhtSB1xT2F4bY0MNFfHw9oJlF4nxGsg67Ib+8sQNhu8Nz WO8YY9F5UTLFK00MO/IdNxq9+DB6zEjh9u66GqmNT3cjkmdha4NidiDP5A== X-Gm-Gg: ASbGnctgjin+Vgw4fGpEGa7RhWMk1Z6ggU4m085+Cqz3A61F2TC8WvDAaND4eH9vwPf X5oOUCE9a69GTmwstSRYG2bEzTa5T4WL4AEihWD9Xj/umVXbZ0Koc63r7npddxibEZhsRgT2Ofi tYXvyRtAng8agGEkwKHNvo/1nOhUkbmp5TEZHLGaOis7YDWB5acgn+T6nTLhBxqZKnw62LyvCaB uebRUWklwmVtuDJHQ0pocnJJoC0YpY5krjIaQexNbO5516eELEgPIAxfTJCLlePoK2vZ7jugvZI 9JVKIg== X-Google-Smtp-Source: AGHT+IFtiAsGhIVFD64KUeuc/HqOnxB2YRGY24DKyXhrRpv8mSEuSX8pUEmuRY3vKJ8avlYCY8VlLQ== X-Received: by 2002:a05:6a21:890d:b0:1e1:9fef:e96a with SMTP id adf61e73a8af0-1e1b1a79c40mr3694721637.6.1733798399161; Mon, 09 Dec 2024 18:39:59 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:83b0]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd560985e0sm1471732a12.79.2024.12.09.18.39.57 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 09 Dec 2024 18:39:58 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v2 4/6] memcg: Add __GFP_TRYLOCK support. Date: Mon, 9 Dec 2024 18:39:34 -0800 Message-Id: <20241210023936.46871-5-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241210023936.46871-1-alexei.starovoitov@gmail.com> References: <20241210023936.46871-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Teach memcg to operate under __GFP_TRYLOCK conditions when spinning locks cannot be used. The end result is __memcg_kmem_charge_page() and __memcg_kmem_uncharge_page() become lockless. Signed-off-by: Alexei Starovoitov --- mm/memcontrol.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7b3503d12aaf..459f35f15819 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1756,7 +1756,8 @@ static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, * * returns true if successful, false otherwise. */ -static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) +static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages, + gfp_t gfp_mask) { struct memcg_stock_pcp *stock; unsigned int stock_pages; @@ -1766,7 +1767,11 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) if (nr_pages > MEMCG_CHARGE_BATCH) return ret; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!local_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + if (gfp_mask & __GFP_TRYLOCK) + return ret; + local_lock_irqsave(&memcg_stock.stock_lock, flags); + } stock = this_cpu_ptr(&memcg_stock); stock_pages = READ_ONCE(stock->nr_pages); @@ -1851,7 +1856,15 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) { unsigned long flags; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!local_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + /* + * In !RT local_trylock_irqsave() always succeeds. + * In case of unlikely failure to lock percpu stock_lock in RT + * uncharge memcg directly. + */ + mem_cgroup_cancel_charge(memcg, nr_pages); + return; + } __refill_stock(memcg, nr_pages); local_unlock_irqrestore(&memcg_stock.stock_lock, flags); } @@ -2196,7 +2209,7 @@ int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, unsigned long pflags; retry: - if (consume_stock(memcg, nr_pages)) + if (consume_stock(memcg, nr_pages, gfp_mask)) return 0; if (!do_memsw_account() || From patchwork Tue Dec 10 02:39:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13900673 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3074B199949 for ; Tue, 10 Dec 2024 02:40:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733798406; cv=none; b=ii1mXTnmWBkSyuUulXhnzIMD3WOSHb4tq81LX3WTF4w0DUrtjQHRKYN5Gg6FGYbtjYyqsXtcY/39gA3bg5L897twF+gcZ/4llXehPm2uqfzWp+sjQx4X6PR5E2I6aD/4Y3Ueu0zLaVim3uXQ/3uiZSXEMpbew3c8ywVqwkw17a4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733798406; c=relaxed/simple; bh=Yf10ydUs8cwUl/6z5MH4k9pW7IZNRoVK3iTxZDFOqGg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=WyTiOp5mdVqaW9vT8f4b24PGa/Ca5h7/9goYbUvdJdsY7IPxntZHfg/pqVFlPoRFXFLBlqtiEMAeb/FLYjfsqlfV0WyD3j7yblt/8K9kTOgr2ew9L1n8C2uWRmej1XSBQ4W17RGKMhBgP5snVgjwAN67W1MYfZxYEBeQcV74klg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Uu1iho7c; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Uu1iho7c" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-20cf3e36a76so43693405ad.0 for ; Mon, 09 Dec 2024 18:40:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733798404; x=1734403204; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dZcki3SirXCmN69CdcGKxt31rfH5khiUYpTKaefczMs=; b=Uu1iho7c6APy9wfXzGh0DnPfJQskzKUo013sQ56ncDCctCXgZwZsZt56V2WwroMmU7 qcvIzsUXgo2jrdNDIr/5i+nw3Id6cFVt+ogvYr4Z709h6crOY0/hVWRlXlw10bbXOH1D hmaz5sofappQVSQ1QPLdrWTa2WA3wup+VZVccA60GLqxU9gP30QCH1Zngkh4gr2wuK1b +PZYBhmNefgfyDnJ6OTC1G97HICgdRyna0sCF2fIci5bxmgLpUJOtcsfb/6AV3wdpmy6 gXoEC1uEbg/v7K8OhOeIpQBazW7JNl3mfP8XUZ/uZ5OwcxlQoQY6TTgiFIcxtcndL+x7 GXWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733798404; x=1734403204; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dZcki3SirXCmN69CdcGKxt31rfH5khiUYpTKaefczMs=; b=KOMP5PfRBTL4tqTSwq5zjBJe7qUIWM7AXdA/T5pkNIGXluzlyxQZExan7yt+VRkW3j LeoNmKfR1rvff7h/x982sjTce+hXWntpnWG7o2rQ6RhRLR39VwKnc1hG+7mgqHe17E88 +TrPFhawSDcxgz2+ADv/xZ/lGdEsdpFPjXK1v63LZfoSGxreVuyNiTnxxuk/7uhrtEN2 BSdTo8OoGcsEOJ+rfX0cgiw+3CKjNHMjyGYBGjPJqdrsFXNOC88/E1qNyrUcZfCAHUIZ hF3AinD83WVPFheDQUwVS5lUlHIYFcZRaAhhLIsNPSXDqVzZ9WKDHC7cdwAVDe1GevD2 ei7A== X-Gm-Message-State: AOJu0YyNj61BXyCI/GQZ+mUVSncqqOkmcN5fOSjaOc1bGaoCfjmNjaFd ogfxMgWpvGaCvP1pBVNnqsyv1kStQ6smMSGMXdO0ffmz1edA0VcWnCqieg== X-Gm-Gg: ASbGnctkzlUZpXuX6p+mjQETju7oIk25RxGghrb2OfAjF8LRtEqAsQ40QZUjyK50AY9 hRlHtudLHpx+xweLN3NfP3PkITjEPUlB+RNYmpTjBJY/geSvr69zHkDD2N6fOs+tzWbLmwAw1Ys tcRVkJqveuI/Rzb+lyscMTaeX7HvuM0lzK2Dla4WiGNl4UjKHQy5RVkTL0LnGwv7alyQjPFG5bv udrSBSySSlNSkXaBdXarvLJ98n/xBTXyGfOBNi26z8bzfGoVllYfRIo17u+E0UogfKsTKehF8Tu 6C7l4g== X-Google-Smtp-Source: AGHT+IGCrCmcyV05haNAd+ohuQ5mWaS3nvBsc6i7L/ZiIyD+Czd5Of6TvkBrVGS6hN/453APVBLTQg== X-Received: by 2002:a17:902:cec4:b0:216:501e:e314 with SMTP id d9443c01a7336-21669fb7abcmr31154365ad.20.1733798403643; Mon, 09 Dec 2024 18:40:03 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:83b0]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-216150886e7sm61609365ad.282.2024.12.09.18.40.01 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 09 Dec 2024 18:40:03 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v2 5/6] mm, bpf: Use __GFP_ACCOUNT in try_alloc_pages(). Date: Mon, 9 Dec 2024 18:39:35 -0800 Message-Id: <20241210023936.46871-6-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241210023936.46871-1-alexei.starovoitov@gmail.com> References: <20241210023936.46871-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Unconditionally use __GFP_ACCOUNT in try_alloc_pages(). The caller is responsible to setup memcg correctly. All BPF memory accounting is memcg based. Signed-off-by: Alexei Starovoitov --- include/linux/gfp.h | 5 ++--- mm/page_alloc.c | 5 ++++- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index dcae733ed006..820c4938c9cd 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -356,18 +356,17 @@ static inline struct page *try_alloc_pages_noprof(int nid, unsigned int order) */ if (preemptible() && !rcu_preempt_depth()) return alloc_pages_node_noprof(nid, - GFP_NOWAIT | __GFP_ZERO, + GFP_NOWAIT | __GFP_ZERO | __GFP_ACCOUNT, order); /* * Best effort allocation from percpu free list. * If it's empty attempt to spin_trylock zone->lock. * Do not specify __GFP_KSWAPD_RECLAIM to avoid wakeup_kswapd * that may need to grab a lock. - * Do not specify __GFP_ACCOUNT to avoid local_lock. * Do not warn either. */ return alloc_pages_node_noprof(nid, - __GFP_TRYLOCK | __GFP_NOWARN | __GFP_ZERO, + __GFP_TRYLOCK | __GFP_NOWARN | __GFP_ZERO | __GFP_ACCOUNT, order); } #define try_alloc_pages(...) alloc_hooks(try_alloc_pages_noprof(__VA_ARGS__)) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a969a62ec0c3..1fada16b8a14 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4818,7 +4818,10 @@ struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, out: if (memcg_kmem_online() && (gfp & __GFP_ACCOUNT) && page && unlikely(__memcg_kmem_charge_page(page, gfp, order) != 0)) { - __free_pages(page, order); + if (unlikely(gfp & __GFP_TRYLOCK)) + free_pages_nolock(page, order); + else + __free_pages(page, order); page = NULL; } From patchwork Tue Dec 10 02:39:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13900674 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BFFE127456 for ; Tue, 10 Dec 2024 02:40:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733798412; cv=none; b=Q/VQQOQVV8ULyF4e7Ms0is3g1pTIG9yYXVHyhFhi5tZCKhh16+aEybFnN1UqzgR0gf2sXoc8mWUOuTtvVtEvXnOYXoDJTSE7SnNAFvctvw8AF6wB4rDkoXNo18PXD2KGsSRMXNYUvbEDr/1D+86xdrrYdcr7ebmrkV+UDpqhiSc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733798412; c=relaxed/simple; bh=JR613i1llUMzESxxaqycZ9MjvkFCwoZDaJ+9bxjM9vk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=YrwrVeHM1SV78BnGrpHmOROnWIh6sEe1aitqoXZcj2cNLTdCbii6MXdhdS2d1SWYuZXnxQT6tTioS0jCGOJ1PczBZFi9dEy9K8xBvO13TQoe3JMZtUSp2vExaJ45mtcXtlKBEFxRPuU8JBcXmJY5FFLsvtpV7rIIngSMOGu3iko= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=fSltBbeG; arc=none smtp.client-ip=209.85.215.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fSltBbeG" Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-7feb6871730so308291a12.2 for ; Mon, 09 Dec 2024 18:40:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733798409; x=1734403209; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uiTKIgIWEw4pjiXWRJupFp29ffG8tRf5Pk0epbA95Ok=; b=fSltBbeGvVjaiUwSeXJgCVW+e/QifyesUDhVGYu+wGA0E/gKR7vb00BKIzZg3CL/hY 2Zg2llFP24OgFB1gXYNNssAXfS/7N34qKla09XCmfsC8vPFuzCsghluJ3gxNfH4mFjmE UlxFRLL4dR/Lp/R0AzUFGcO8MhmYxn40lszgx2tcKPAL07bIsN2aCJBz/5fS7UrOVs/f K9EMhcD6aL/FdlIghQR+boAtNFjfZF+Z6rcxWtOgICUf0ewgi1GDCx+iQecxFs7noGYv ml0p3VgVjx+Ms2pV9JXtV7cEVRy9cm2pBIgHu4Z0uzzs/X9TYTB6GQMWtLMIv8lf2k0r uvZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733798409; x=1734403209; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uiTKIgIWEw4pjiXWRJupFp29ffG8tRf5Pk0epbA95Ok=; b=xE1LWmw6go1XmV1/ZEu7ohIcOs3IgnW4evAS5/sXgFVisFOVw49IQNnDVvFq+ZQ+d8 5524naTPmihPCjJiA6oUzC1YmnTqWj1IQqwoOj+KZd6RuwaIv8Eigps0+va1sy8Ax/pN cVvhqUqMQ9XMVQFocTJ23KMSqKyo3vwrZJP3r+B7fSop7U3678DJ61C2I90qhXrQgYnb gukYpqSOuQe6Y7xdyMV/vLkkefJ+3Epi09M+LjH93o4UNLeecZ0RluQ0FOapzDeRpppH Y48TM9dN8YyZCnDrTbb97CxhqtLLwCmIzuH3o5RaGrs3x07jMiN6hXLbixkXTB9PcRRq AMHw== X-Gm-Message-State: AOJu0Yz0PZiABbm8LdlHSyaKVdUF9tehO89gR6biPrbLx81pKG0M8+3z aFi/4oR5wHCMwiPpiD04WTEo1r8UTWWkl6tughUwpPzSQ7a0VcX9LDkYbA== X-Gm-Gg: ASbGncu0ogAtTUpA4EYuneCoWdAwKo4juvl+Nj+rQ4UfKO/9iCxqVCjunmr+kXh3t4A bn4WF+5IIa2umcPaY00ql5om377czXUznmwIWrbFCdgs988Hu22dGQWSQI2l2dg3T0rjYaAwabb QF43HtsLRP7hwK6nfh6amH0wsjzheLNIHjgpr711JUdl2F43hJGWuutybzTnBUyCMh0ahY0wE0T FrixCTdDMhAJ/vQBxQbHSDy1kGzojZiQEGvBdzr7tmhHkvxmUfBtJEDJEydWcDVVt7Aj0KKmzdu D3f55w== X-Google-Smtp-Source: AGHT+IEyaw7MLXSrpLiOPJxXq3UM5b+L21nSTK6Jj0nYt9CgSbssrgPTsovcJ8e9A4v907+avThj1Q== X-Received: by 2002:a17:90b:3847:b0:2ee:c4f2:a77d with SMTP id 98e67ed59e1d1-2efcf16f547mr3957084a91.21.1733798409404; Mon, 09 Dec 2024 18:40:09 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:83b0]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2ef1e920587sm8088928a91.1.2024.12.09.18.40.06 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 09 Dec 2024 18:40:09 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v2 6/6] bpf: Use try_alloc_pages() to allocate pages for bpf needs. Date: Mon, 9 Dec 2024 18:39:36 -0800 Message-Id: <20241210023936.46871-7-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241210023936.46871-1-alexei.starovoitov@gmail.com> References: <20241210023936.46871-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Use try_alloc_pages() and free_pages_nolock() Signed-off-by: Alexei Starovoitov --- kernel/bpf/syscall.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 5684e8ce132d..70589208b545 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -582,14 +582,14 @@ int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, old_memcg = set_active_memcg(memcg); #endif for (i = 0; i < nr_pages; i++) { - pg = alloc_pages_node(nid, gfp | __GFP_ACCOUNT, 0); + pg = try_alloc_pages(nid, 0); if (pg) { pages[i] = pg; continue; } for (j = 0; j < i; j++) - __free_page(pages[j]); + free_pages_nolock(pages[j], 0); ret = -ENOMEM; break; }