From patchwork Wed Feb 12 17:47:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13972249 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A2E0C20F09A for ; Wed, 12 Feb 2025 17:47:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739382436; cv=none; b=OXxF4RgN6fZLcqt+BMHMn/kt0JfYOl3nz76gGe8S1peWh36NML2cJhjVGlxSO/pqxPmJ0N2TLTE6I3P84K8fzAOUwzUEm3VliM5mCjrtDuJH9K+GkhIjOtjqZ1EvVPzHqM071cKJ1UUjFljVmWhMJRuCzVZ2GnvW2gq/Imdt9CU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739382436; c=relaxed/simple; bh=OAkIZrImYMNxJ6LZVhxCz5HGhS23V3BZnYycvYLu++g=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=E91SylsMQB4B99sCV/5kymVY12IArodpYUXB7P38LjEDfRanvey6YfTe4lUAovmDUAA2gJBPlBxsxMMgEmYbEDFpBhyUgtHkY+EaCY0bTAxFOHgdDkwpwu0Tj3oge/+xvrQvj0LqteMBBNp1JuLF1sY2Yu7dF8bdVwhZlbpKBsg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=IvhI3w7G; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="IvhI3w7G" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-220c92c857aso9919745ad.0 for ; Wed, 12 Feb 2025 09:47:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739382433; x=1739987233; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZcDqEp0sdulY57w8+974+V84oF4S+MilCZmcAMnl5T8=; b=IvhI3w7GID2lNgdRbve3sE5mYhh4ouwwo03mna2TI7rkkrSeuZ4Em8ZyQhibWzQAvR bQsxjZNSZyjjTeO5YQFXYbAcUJ9Q046QqR8Il+8vqDHbVFL+zsXwJUUymiaW1ctmV/SX BkEEycfWxgbRhNJBJBrdsuvZo1JAnAd7HwXFo1evyH09BvCqBcoxB2CMS1SK10JerXSl 9mQlDx2EEAJYRDfwliRQdqQ6bdJoidWoY0aPle0rvn3qG2FMfOc9lXBL/jL1jYju54Ib YfWIFM1345bbudGPZt/TBp0WIlHUGoVLOsporOXWW96fTrMnB4r6+zU/0BlLJURNqpK2 OmQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739382433; x=1739987233; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZcDqEp0sdulY57w8+974+V84oF4S+MilCZmcAMnl5T8=; b=Qq9DE5LK/wTPvqYdzxtc4oBgnyjxFNqmrNfPUITE0K90U3DV+3TLB4ZGFBQwuMZgCT luP1FnMl362BgT7DH+fHg0MtJvJWPOxOBV7tsaW53b5p+ar/QUZPWDnG6IHPNKTsNziZ U3bX732HO03Dhs1gZTPhGLREKO7DyW2oxJl2eh6D9Gtqi7K3B3zW3MeRC3hCWN9np2fN 3kR2cJBQ1ujBYxhlhnH6bwx20qJ7gT/l3bq8K0EKZ/PYKMSCTFL4QlMHsApGFVnx7U+k 7dLkVywyoQuyBJQHE9f+oYfjRVJDKRKO2DN33UB9eJNYQUq/Np7WCO+HbBQhwhnKCFVa KcOw== X-Gm-Message-State: AOJu0YxCgoI/lhHUf4S2ijfql6cOWySsMxVWIj3j+VOUuo5vzuoM+iko J5ahvvO82CHtkt1efafkhTz9kRm/2n577upqxu78U8XercjqPb5QulHV1g== X-Gm-Gg: ASbGncs1plKOOErgP0LWVEXTHC0Bb2b6bdYWVwwLDiEDs3cpT99r9AOoh0POcaH1RO7 rKU1bXx0z0N+hg0O2vCA+55Y+wDM8kknrZiRGMycGgT/8WSc0JVFNNle68CmBxKccoZnzcjKrfz 0SW3NaCl1pAGe5cfF4nRu5d5+jYNeUFLw7adnfo1kCT9NdjQuwNzHlQmeZ0oT1Z2MzQW2KU6/wk S1sf9DzrHgfkps03ap5I5FzIwexf/tPQws5SdIGGGGEkpyGVc/0+hf84elkZQMPNf/CKjbKj2C/ QXBDbpQo14HVBdqdVdDrxiZHovwKhONgwWiBdV+76KXTHQ== X-Google-Smtp-Source: AGHT+IGiz7lPxFiFSSCtFNzPA8nthkoVk2uvweVCB4Zudmmc+mzIhBfhSZ/Vst38614VBFH4Hr8CXg== X-Received: by 2002:a17:903:3d04:b0:21f:5e6f:a406 with SMTP id d9443c01a7336-220d2112aa1mr3048445ad.20.1739382433389; Wed, 12 Feb 2025 09:47:13 -0800 (PST) Received: from ast-mac.thefacebook.com ([2620:10d:c090:500::4:c330]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2fbf9ab0233sm1792313a91.44.2025.02.12.09.47.11 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 12 Feb 2025 09:47:12 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v7 1/6] mm, bpf: Introduce try_alloc_pages() for opportunistic page allocation Date: Wed, 12 Feb 2025 09:47:00 -0800 Message-Id: <20250212174705.44492-2-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250212174705.44492-1-alexei.starovoitov@gmail.com> References: <20250212174705.44492-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Tracing BPF programs execute from tracepoints and kprobes where running context is unknown, but they need to request additional memory. The prior workarounds were using pre-allocated memory and BPF specific freelists to satisfy such allocation requests. Instead, introduce gfpflags_allow_spinning() condition that signals to the allocator that running context is unknown. Then rely on percpu free list of pages to allocate a page. try_alloc_pages() -> get_page_from_freelist() -> rmqueue() -> rmqueue_pcplist() will spin_trylock to grab the page from percpu free list. If it fails (due to re-entrancy or list being empty) then rmqueue_bulk()/rmqueue_buddy() will attempt to spin_trylock zone->lock and grab the page from there. spin_trylock() is not safe in PREEMPT_RT when in NMI or in hard IRQ. Bailout early in such case. The support for gfpflags_allow_spinning() mode for free_page and memcg comes in the next patches. This is a first step towards supporting BPF requirements in SLUB and getting rid of bpf_mem_alloc. That goal was discussed at LSFMM: https://lwn.net/Articles/974138/ Acked-by: Michal Hocko Acked-by: Vlastimil Babka Acked-by: Sebastian Andrzej Siewior Reviewed-by: Shakeel Butt Signed-off-by: Alexei Starovoitov --- include/linux/gfp.h | 22 ++++++++++ lib/stackdepot.c | 5 ++- mm/internal.h | 1 + mm/page_alloc.c | 104 ++++++++++++++++++++++++++++++++++++++++++-- 4 files changed, 127 insertions(+), 5 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index b0fe9f62d15b..82bfb65b8d15 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -39,6 +39,25 @@ static inline bool gfpflags_allow_blocking(const gfp_t gfp_flags) return !!(gfp_flags & __GFP_DIRECT_RECLAIM); } +static inline bool gfpflags_allow_spinning(const gfp_t gfp_flags) +{ + /* + * !__GFP_DIRECT_RECLAIM -> direct claim is not allowed. + * !__GFP_KSWAPD_RECLAIM -> it's not safe to wake up kswapd. + * All GFP_* flags including GFP_NOWAIT use one or both flags. + * try_alloc_pages() is the only API that doesn't specify either flag. + * + * This is stronger than GFP_NOWAIT or GFP_ATOMIC because + * those are guaranteed to never block on a sleeping lock. + * Here we are enforcing that the allocation doesn't ever spin + * on any locks (i.e. only trylocks). There is no high level + * GFP_$FOO flag for this use in try_alloc_pages() as the + * regular page allocator doesn't fully support this + * allocation mode. + */ + return !(gfp_flags & __GFP_RECLAIM); +} + #ifdef CONFIG_HIGHMEM #define OPT_ZONE_HIGHMEM ZONE_HIGHMEM #else @@ -347,6 +366,9 @@ static inline struct page *alloc_page_vma_noprof(gfp_t gfp, } #define alloc_page_vma(...) alloc_hooks(alloc_page_vma_noprof(__VA_ARGS__)) +struct page *try_alloc_pages_noprof(int nid, unsigned int order); +#define try_alloc_pages(...) alloc_hooks(try_alloc_pages_noprof(__VA_ARGS__)) + extern unsigned long get_free_pages_noprof(gfp_t gfp_mask, unsigned int order); #define __get_free_pages(...) alloc_hooks(get_free_pages_noprof(__VA_ARGS__)) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 245d5b416699..377194969e61 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -591,7 +591,8 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries, depot_stack_handle_t handle = 0; struct page *page = NULL; void *prealloc = NULL; - bool can_alloc = depot_flags & STACK_DEPOT_FLAG_CAN_ALLOC; + bool allow_spin = gfpflags_allow_spinning(alloc_flags); + bool can_alloc = (depot_flags & STACK_DEPOT_FLAG_CAN_ALLOC) && allow_spin; unsigned long flags; u32 hash; @@ -630,7 +631,7 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries, prealloc = page_address(page); } - if (in_nmi()) { + if (in_nmi() || !allow_spin) { /* We can never allocate in NMI context. */ WARN_ON_ONCE(can_alloc); /* Best effort; bail if we fail to take the lock. */ diff --git a/mm/internal.h b/mm/internal.h index 9826f7dce607..6c3c664aa346 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1174,6 +1174,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, #define ALLOC_NOFRAGMENT 0x0 #endif #define ALLOC_HIGHATOMIC 0x200 /* Allows access to MIGRATE_HIGHATOMIC */ +#define ALLOC_TRYLOCK 0x400 /* Only use spin_trylock in allocation path */ #define ALLOC_KSWAPD 0x800 /* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ /* Flags that allow allocations below the min watermark. */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 01eab25edf89..a82bc67abbdb 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2306,7 +2306,11 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, unsigned long flags; int i; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(alloc_flags & ALLOC_TRYLOCK)) + return 0; + spin_lock_irqsave(&zone->lock, flags); + } for (i = 0; i < count; ++i) { struct page *page = __rmqueue(zone, order, migratetype, alloc_flags); @@ -2906,7 +2910,11 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, do { page = NULL; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(alloc_flags & ALLOC_TRYLOCK)) + return NULL; + spin_lock_irqsave(&zone->lock, flags); + } if (alloc_flags & ALLOC_HIGHATOMIC) page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC); if (!page) { @@ -4511,7 +4519,12 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, might_alloc(gfp_mask); - if (should_fail_alloc_page(gfp_mask, order)) + /* + * Don't invoke should_fail logic, since it may call + * get_random_u32() and printk() which need to spin_lock. + */ + if (!(*alloc_flags & ALLOC_TRYLOCK) && + should_fail_alloc_page(gfp_mask, order)) return false; *alloc_flags = gfp_to_alloc_flags_cma(gfp_mask, *alloc_flags); @@ -7028,3 +7041,88 @@ static bool __free_unaccepted(struct page *page) } #endif /* CONFIG_UNACCEPTED_MEMORY */ + +/** + * try_alloc_pages_noprof - opportunistic reentrant allocation from any context + * @nid - node to allocate from + * @order - allocation order size + * + * Allocates pages of a given order from the given node. This is safe to + * call from any context (from atomic, NMI, and also reentrant + * allocator -> tracepoint -> try_alloc_pages_noprof). + * Allocation is best effort and to be expected to fail easily so nobody should + * rely on the success. Failures are not reported via warn_alloc(). + * See always fail conditions below. + * + * Return: allocated page or NULL on failure. + */ +struct page *try_alloc_pages_noprof(int nid, unsigned int order) +{ + /* + * Do not specify __GFP_DIRECT_RECLAIM, since direct claim is not allowed. + * Do not specify __GFP_KSWAPD_RECLAIM either, since wake up of kswapd + * is not safe in arbitrary context. + * + * These two are the conditions for gfpflags_allow_spinning() being true. + * + * Specify __GFP_NOWARN since failing try_alloc_pages() is not a reason + * to warn. Also warn would trigger printk() which is unsafe from + * various contexts. We cannot use printk_deferred_enter() to mitigate, + * since the running context is unknown. + * + * Specify __GFP_ZERO to make sure that call to kmsan_alloc_page() below + * is safe in any context. Also zeroing the page is mandatory for + * BPF use cases. + * + * Though __GFP_NOMEMALLOC is not checked in the code path below, + * specify it here to highlight that try_alloc_pages() + * doesn't want to deplete reserves. + */ + gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC; + unsigned int alloc_flags = ALLOC_TRYLOCK; + struct alloc_context ac = { }; + struct page *page; + + /* + * In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is + * unsafe in NMI. If spin_trylock() is called from hard IRQ the current + * task may be waiting for one rt_spin_lock, but rt_spin_trylock() will + * mark the task as the owner of another rt_spin_lock which will + * confuse PI logic, so return immediately if called form hard IRQ or + * NMI. + * + * Note, irqs_disabled() case is ok. This function can be called + * from raw_spin_lock_irqsave region. + */ + if (IS_ENABLED(CONFIG_PREEMPT_RT) && (in_nmi() || in_hardirq())) + return NULL; + if (!pcp_allowed_order(order)) + return NULL; + +#ifdef CONFIG_UNACCEPTED_MEMORY + /* Bailout, since try_to_accept_memory_one() needs to take a lock */ + if (has_unaccepted_memory()) + return NULL; +#endif + /* Bailout, since _deferred_grow_zone() needs to take a lock */ + if (deferred_pages_enabled()) + return NULL; + + if (nid == NUMA_NO_NODE) + nid = numa_node_id(); + + prepare_alloc_pages(alloc_gfp, order, nid, NULL, &ac, + &alloc_gfp, &alloc_flags); + + /* + * Best effort allocation from percpu free list. + * If it's empty attempt to spin_trylock zone->lock. + */ + page = get_page_from_freelist(alloc_gfp, order, alloc_flags, &ac); + + /* Unlike regular alloc_pages() there is no __alloc_pages_slowpath(). */ + + trace_mm_page_alloc(page, order, alloc_gfp, ac.migratetype); + kmsan_alloc_page(page, order, alloc_gfp); + return page; +} From patchwork Wed Feb 12 17:47:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13972250 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 415EA20F09A for ; Wed, 12 Feb 2025 17:47:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739382442; cv=none; b=hU1K9cPlNZKWD5V0xTIemTBl7FpNzJmv4ltlkWADselZUh/lRWryKfC6m5wSblU8mGuC69Y4uG18Szm1rVcjNWTrca2nFaMj03N5+F+RQxuzQd/XjzeB8vz3nSPwU4ySdIY3fi94p4liD9NI8e91dnjAGvRe/cyPGHhQeHv3RFI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739382442; c=relaxed/simple; bh=zh4CBilixCd1RG8FS2CSA2f+NFm/6QC0FOcf1W8wAP4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FY1QCTez6GLM34Dl9sxUVSznGxJHeSpLALBB0rWdGh14z7mzPSW8u270Twljp3B2b48mP5LYqUJW0x7pl6Xgd4KbmMmvG8r+V4rfCJW4MAZ2ltCfeCvDAqR45cExhXO1xQEcXjgNL+N2WlSbzWARU886wOiRlgIhfWcCklAmZGk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=iz7cF5mo; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="iz7cF5mo" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-220c2a87378so13936455ad.1 for ; Wed, 12 Feb 2025 09:47:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739382438; x=1739987238; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bfOI9MX/hHEauaA9ofUhjkEEOuqobZaUZXdU+efBBTw=; b=iz7cF5movGUM3dn2JiNgJphmMQW9QhOsBT+xeENufJtj7kpMT86RYZtf75hL3IRcOV GDkEsFhBbobPKb66EgsPtWjsTMD096djJwReQB+SvufW/VdSWSKHKlPF7yxZj2V1vRIz +SXM9MkZMe/zX6aXneAcK6dgBmrUflL1QDmHf05vpDOtJwscjFMxQ+wACCAA6w4zkik8 djO4aoXUp1/G2+wQuxIT62pfirWlbJRf+KRgrFxoqS2mA8jUMpPv5vlKrxQSYGRMvLO5 +5k6euZeWLbxf0kx2CBGAGiC4IzLQorh2uoF8lzoOKA1PbNsVQvWLBqyIoEDiBrYaLRg ATXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739382438; x=1739987238; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bfOI9MX/hHEauaA9ofUhjkEEOuqobZaUZXdU+efBBTw=; b=dl3eMKhR0eon+NzPZKRZ83JGwKXRgkiqdJB4UN+SgsY3hVV+jp83iwJ0EGCIMxdvor iuqWsVXUHIoGoI/ZS1NThu3ib0fliVbW4LzKU6cNYWra03iGJx6MiobP5bRypeUYlC3z OUsz9b7nxdW7g4XTqn8HCUUk/DjZ1M50soWBwSndEwityRSwvkNf2NXIKV+H33e1SR5U uKsaVJ3PFWkO7jUdFdmBgAkCdak0nsat7IvJDH8g0yGIX0Fgmy8NV+3sOzEdKtJ17WX8 n3RB3sXiPqOuDoxAgGjGiyL8SThbftkaXx+EisKFNJyZfwKmyywPskAdw3OEduKobiaY 3uHw== X-Gm-Message-State: AOJu0YwhopqpoyVc5vAgZE7S9PGvSX+T4+ZPeTJloBIIA2nSTcbVShs1 qofZ3YXrBFpSeshw+d0C13XTcQ1MFcvXfHNk+rqKs4RwiSI5WyoLjU67Ww== X-Gm-Gg: ASbGnctZtHXTTZzrqbcGfC7+ojZXfDmAQ+iVApGNvKHlRuT+9IQP+fYU5C9HeZLN1Gb 3sxL+0ig66gGQBDUkuCUnWsHbsetvOPEjF14iG+EfJ3wSdUdtonjaL19DQv5Rc733ql4300eeQO ZJKVCI5a46nbaJ52PKoGxyC2SleZz80/oTyhoMp77aWgM31vgseNGyuxmwpeHwK7xcgTkxwNuA5 gO6m66LdUP7NQJ3lUXQiduKp5s5RlveLPYdB0HOdq0q8RsWizCXs+uFyQ/gd9aBMdAl+ARndjuy arZKimAN4SvgBDmHrfLXizlJF9IEoS/+IwtekFDqYbe+gw== X-Google-Smtp-Source: AGHT+IHkhCXV1pRFdGKufwsizq7brzMYRyNrfIELMSWMRiL7LCkFISysHio/g0WXe8rLxsku0EUl5g== X-Received: by 2002:a17:902:ce12:b0:21f:3e2d:7d4a with SMTP id d9443c01a7336-220bbb08e1cmr69447445ad.26.1739382437693; Wed, 12 Feb 2025 09:47:17 -0800 (PST) Received: from ast-mac.thefacebook.com ([2620:10d:c090:500::4:c330]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21f3653bfb2sm116573095ad.67.2025.02.12.09.47.15 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 12 Feb 2025 09:47:17 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v7 2/6] mm, bpf: Introduce free_pages_nolock() Date: Wed, 12 Feb 2025 09:47:01 -0800 Message-Id: <20250212174705.44492-3-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250212174705.44492-1-alexei.starovoitov@gmail.com> References: <20250212174705.44492-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Introduce free_pages_nolock() that can free pages without taking locks. It relies on trylock and can be called from any context. Since spin_trylock() cannot be used in PREEMPT_RT from hard IRQ or NMI it uses lockless link list to stash the pages which will be freed by subsequent free_pages() from good context. Do not use llist unconditionally. BPF maps continuously allocate/free, so we cannot unconditionally delay the freeing to llist. When the memory becomes free make it available to the kernel and BPF users right away if possible, and fallback to llist as the last resort. Acked-by: Vlastimil Babka Acked-by: Sebastian Andrzej Siewior Reviewed-by: Shakeel Butt Signed-off-by: Alexei Starovoitov --- include/linux/gfp.h | 1 + include/linux/mm_types.h | 4 ++ include/linux/mmzone.h | 3 ++ lib/stackdepot.c | 5 ++- mm/page_alloc.c | 90 +++++++++++++++++++++++++++++++++++----- mm/page_owner.c | 8 +++- 6 files changed, 98 insertions(+), 13 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 82bfb65b8d15..a8233d09acfa 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -391,6 +391,7 @@ __meminit void *alloc_pages_exact_nid_noprof(int nid, size_t size, gfp_t gfp_mas __get_free_pages((gfp_mask) | GFP_DMA, (order)) extern void __free_pages(struct page *page, unsigned int order); +extern void free_pages_nolock(struct page *page, unsigned int order); extern void free_pages(unsigned long addr, unsigned int order); #define __free_page(page) __free_pages((page), 0) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 825c04b56403..583bf59e2627 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -99,6 +99,10 @@ struct page { /* Or, free page */ struct list_head buddy_list; struct list_head pcp_list; + struct { + struct llist_node pcp_llist; + unsigned int order; + }; }; /* See page-flags.h for PAGE_MAPPING_FLAGS */ struct address_space *mapping; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index b36124145a16..1a854e0a9e3b 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -953,6 +953,9 @@ struct zone { /* Primarily protects free_area */ spinlock_t lock; + /* Pages to be freed when next trylock succeeds */ + struct llist_head trylock_free_pages; + /* Write-intensive fields used by compaction and vmstats. */ CACHELINE_PADDING(_pad2_); diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 377194969e61..73d7b50924ef 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -672,7 +672,10 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries, exit: if (prealloc) { /* Stack depot didn't use this memory, free it. */ - free_pages((unsigned long)prealloc, DEPOT_POOL_ORDER); + if (!allow_spin) + free_pages_nolock(virt_to_page(prealloc), DEPOT_POOL_ORDER); + else + free_pages((unsigned long)prealloc, DEPOT_POOL_ORDER); } if (found) handle = found->handle.handle; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a82bc67abbdb..fa750c46e0fc 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -88,6 +88,9 @@ typedef int __bitwise fpi_t; */ #define FPI_TO_TAIL ((__force fpi_t)BIT(1)) +/* Free the page without taking locks. Rely on trylock only. */ +#define FPI_TRYLOCK ((__force fpi_t)BIT(2)) + /* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ static DEFINE_MUTEX(pcp_batch_high_lock); #define MIN_PERCPU_PAGELIST_HIGH_FRACTION (8) @@ -1249,13 +1252,44 @@ static void split_large_buddy(struct zone *zone, struct page *page, } while (1); } +static void add_page_to_zone_llist(struct zone *zone, struct page *page, + unsigned int order) +{ + /* Remember the order */ + page->order = order; + /* Add the page to the free list */ + llist_add(&page->pcp_llist, &zone->trylock_free_pages); +} + static void free_one_page(struct zone *zone, struct page *page, unsigned long pfn, unsigned int order, fpi_t fpi_flags) { + struct llist_head *llhead; unsigned long flags; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(fpi_flags & FPI_TRYLOCK)) { + add_page_to_zone_llist(zone, page, order); + return; + } + spin_lock_irqsave(&zone->lock, flags); + } + + /* The lock succeeded. Process deferred pages. */ + llhead = &zone->trylock_free_pages; + if (unlikely(!llist_empty(llhead) && !(fpi_flags & FPI_TRYLOCK))) { + struct llist_node *llnode; + struct page *p, *tmp; + + llnode = llist_del_all(llhead); + llist_for_each_entry_safe(p, tmp, llnode, pcp_llist) { + unsigned int p_order = p->order; + + split_large_buddy(zone, p, page_to_pfn(p), p_order, fpi_flags); + __count_vm_events(PGFREE, 1 << p_order); + } + } split_large_buddy(zone, page, pfn, order, fpi_flags); spin_unlock_irqrestore(&zone->lock, flags); @@ -2598,7 +2632,7 @@ static int nr_pcp_high(struct per_cpu_pages *pcp, struct zone *zone, static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, struct page *page, int migratetype, - unsigned int order) + unsigned int order, fpi_t fpi_flags) { int high, batch; int pindex; @@ -2633,6 +2667,14 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, } if (pcp->free_count < (batch << CONFIG_PCP_BATCH_SCALE_MAX)) pcp->free_count += (1 << order); + + if (unlikely(fpi_flags & FPI_TRYLOCK)) { + /* + * Do not attempt to take a zone lock. Let pcp->count get + * over high mark temporarily. + */ + return; + } high = nr_pcp_high(pcp, zone, batch, free_high); if (pcp->count >= high) { free_pcppages_bulk(zone, nr_pcp_free(pcp, batch, high, free_high), @@ -2647,7 +2689,8 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, /* * Free a pcp page */ -void free_unref_page(struct page *page, unsigned int order) +static void __free_unref_page(struct page *page, unsigned int order, + fpi_t fpi_flags) { unsigned long __maybe_unused UP_flags; struct per_cpu_pages *pcp; @@ -2656,7 +2699,7 @@ void free_unref_page(struct page *page, unsigned int order) int migratetype; if (!pcp_allowed_order(order)) { - __free_pages_ok(page, order, FPI_NONE); + __free_pages_ok(page, order, fpi_flags); return; } @@ -2673,24 +2716,34 @@ void free_unref_page(struct page *page, unsigned int order) migratetype = get_pfnblock_migratetype(page, pfn); if (unlikely(migratetype >= MIGRATE_PCPTYPES)) { if (unlikely(is_migrate_isolate(migratetype))) { - free_one_page(page_zone(page), page, pfn, order, FPI_NONE); + free_one_page(page_zone(page), page, pfn, order, fpi_flags); return; } migratetype = MIGRATE_MOVABLE; } zone = page_zone(page); + if (unlikely((fpi_flags & FPI_TRYLOCK) && IS_ENABLED(CONFIG_PREEMPT_RT) + && (in_nmi() || in_hardirq()))) { + add_page_to_zone_llist(zone, page, order); + return; + } pcp_trylock_prepare(UP_flags); pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (pcp) { - free_unref_page_commit(zone, pcp, page, migratetype, order); + free_unref_page_commit(zone, pcp, page, migratetype, order, fpi_flags); pcp_spin_unlock(pcp); } else { - free_one_page(zone, page, pfn, order, FPI_NONE); + free_one_page(zone, page, pfn, order, fpi_flags); } pcp_trylock_finish(UP_flags); } +void free_unref_page(struct page *page, unsigned int order) +{ + __free_unref_page(page, order, FPI_NONE); +} + /* * Free a batch of folios */ @@ -2779,7 +2832,7 @@ void free_unref_folios(struct folio_batch *folios) trace_mm_page_free_batched(&folio->page); free_unref_page_commit(zone, pcp, &folio->page, migratetype, - order); + order, FPI_NONE); } if (pcp) { @@ -4843,22 +4896,37 @@ EXPORT_SYMBOL(get_zeroed_page_noprof); * Context: May be called in interrupt context or while holding a normal * spinlock, but not in NMI context or while holding a raw spinlock. */ -void __free_pages(struct page *page, unsigned int order) +static void ___free_pages(struct page *page, unsigned int order, + fpi_t fpi_flags) { /* get PageHead before we drop reference */ int head = PageHead(page); struct alloc_tag *tag = pgalloc_tag_get(page); if (put_page_testzero(page)) - free_unref_page(page, order); + __free_unref_page(page, order, fpi_flags); else if (!head) { pgalloc_tag_sub_pages(tag, (1 << order) - 1); while (order-- > 0) - free_unref_page(page + (1 << order), order); + __free_unref_page(page + (1 << order), order, + fpi_flags); } } +void __free_pages(struct page *page, unsigned int order) +{ + ___free_pages(page, order, FPI_NONE); +} EXPORT_SYMBOL(__free_pages); +/* + * Can be called while holding raw_spin_lock or from IRQ and NMI for any + * page type (not only those that came from try_alloc_pages) + */ +void free_pages_nolock(struct page *page, unsigned int order) +{ + ___free_pages(page, order, FPI_TRYLOCK); +} + void free_pages(unsigned long addr, unsigned int order) { if (addr != 0) { diff --git a/mm/page_owner.c b/mm/page_owner.c index 2d6360eaccbb..90e31d0e3ed7 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -294,7 +294,13 @@ void __reset_page_owner(struct page *page, unsigned short order) page_owner = get_page_owner(page_ext); alloc_handle = page_owner->handle; - handle = save_stack(GFP_NOWAIT | __GFP_NOWARN); + /* + * Do not specify GFP_NOWAIT to make gfpflags_allow_spinning() == false + * to prevent issues in stack_depot_save(). + * This is similar to try_alloc_pages() gfp flags, but only used + * to signal stack_depot to avoid spin_locks. + */ + handle = save_stack(__GFP_NOWARN); __update_page_owner_free_handle(page_ext, handle, order, current->pid, current->tgid, free_ts_nsec); page_ext_put(page_ext); From patchwork Wed Feb 12 17:47:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13972251 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F52D20F09A for ; Wed, 12 Feb 2025 17:47:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739382444; cv=none; b=nOPZvYdqjT//KwGh7eGDgF61mgqGji3+tM+7OtA+N3QRFpxjZH1Eptz3OMq2ZmuTmlDaXBg9Aahr71oXxVgQm9xLSc7AkkyABP50iO2v3WN8baJ9FRcIARJEii3xsjUL3yj+YZyG58p8w0IMqiCHeNk0dwzNNPMDC2/Bid91qkc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739382444; c=relaxed/simple; bh=U90v5Pk93VeiqRlGaWd3w3TfoItyJk0Pr1aSpT1G4VI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=u73ZhNn7XHB+zXq7fz7kdwKfwes06zp1OHObGM8AQyeQh73Z2ZQTV/g0e9/mdU83nhzGWycghO3L6u3pRzU4m432FW7nQrRQPYJrXSRg9EQv/FnjlkgpLa2DBrh4q7oAqkoeKwGOSKzbetCVUVVKp5G14gF4z3s7S3qVOS/+1vU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=NUfWEvd9; arc=none smtp.client-ip=209.85.216.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="NUfWEvd9" Received: by mail-pj1-f51.google.com with SMTP id 98e67ed59e1d1-2f833af7a09so62913a91.2 for ; Wed, 12 Feb 2025 09:47:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739382442; x=1739987242; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Dxu1Rc7/sI72JB4NYxxk9elaSjOiJ9Td89bVV/zMYyc=; b=NUfWEvd9I/BwfrNrsNRizS1dd6sYvY3R/PAumgzHPgUWZfCRqgj1uq3Xdz5j3gIYBo DjVl2a3yn4v2o97TW/qC5HxHcExDdXID00O7qBNVodI+lJObsJsqjnnITseN0IQDK4d4 UOSKpl5nZpu4QkGHsKm3jgS6R7mvaEV8ykMdyjYAokzfVEhek28c4A8t7lZjIKpPLitR zEzXLOpIvX2ejGPNjPX3XWQbiyHza/Z200auOPCYKlLglbi1BgxneewZZcr4A87xh4DB 5fOS9/AQ7yeaI3i3VrCVh1JyQm4zhI8MMyBki6zKEo3MF9lWEkjfh2XVBE3GhNHE600r 3UKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739382442; x=1739987242; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Dxu1Rc7/sI72JB4NYxxk9elaSjOiJ9Td89bVV/zMYyc=; b=Fam220VK4yd+pSM7iU7hRQwyEq+t1b23idF6mO9ACyyToD2RxA6+hPx2HpEjyLp33K G/EmeCxwJ7fWNCDd7sHpH/2OcmgO4Jf/2DwR+s7RVxXx1aPYawivRXdtZbYu/dt9Ur7U 3ZEd2Nv8eSIXu52c81z6Ho5dDAepHFrBAV0xqQwL+6OWlnuISXqnP7uVNjiZBGIH8Mbq l7CQKRc2OOA9t7O0lyarDWAgRdqAUQVzL5OBSHLIBIjuPpNfjdmKP6AfToqZqA9QnoVh oJIyzLk6cCU+ESUncaJalfOL70/FHQR1LofwNK9O9rQLmVGqaATncpln9D0z8a7wZ4wT CKcQ== X-Gm-Message-State: AOJu0YwIMSRLMiiuixEFnLUiY2WLsgDuvDqZuFCBWNwLOUKB4DR9tAqZ 5A4/aUffkf4NZLhuPWfx3E97YwXPyGjOU8D7YAZHsH7eCYEeaS3ddWb40Q== X-Gm-Gg: ASbGncscWAhcdl5M4mT3cWToysWrobqgIAi3u/4syrxM+Z1LkvnpPIooope0YDxLUdw kvs9nTDVejDERTae+Q5oNzR8xJyrpXbFdgDyVT5dUgjgwJ1b2cwha2UEukfFscgdPNoVn+UL8O1 0TTzaYc6jYJ7qV3qSKynXLJ2gf5Y7JBGaWIPEQ/p+9U4aM51ZJgEnYMqD32hWdNO4d8xhokTN9D q2u6V5tK0sWw6SzXqjXudT6ikzJJvjcEqfzAgtqzIaop7tUJLdJ1mcpfduhSt8uMuopz3wQ4wYV zQkyt8xcUxEDIEJajJoozXSEnvab+qmEhelEER7XHN1kWg== X-Google-Smtp-Source: AGHT+IFdzeREzMu6Nvpg1XZZgAM871rh2ju2mt7o2mzuvJjED9/Nt9+adPMlY6ggnat49bQIcDjxPQ== X-Received: by 2002:a17:90a:d00e:b0:2ee:d433:7c50 with SMTP id 98e67ed59e1d1-2fbf5c5a06dmr4937863a91.23.1739382441983; Wed, 12 Feb 2025 09:47:21 -0800 (PST) Received: from ast-mac.thefacebook.com ([2620:10d:c090:500::4:c330]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2fbf9908835sm1805194a91.42.2025.02.12.09.47.20 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 12 Feb 2025 09:47:21 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v7 3/6] locking/local_lock: Introduce localtry_lock_t Date: Wed, 12 Feb 2025 09:47:02 -0800 Message-Id: <20250212174705.44492-4-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250212174705.44492-1-alexei.starovoitov@gmail.com> References: <20250212174705.44492-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Sebastian Andrzej Siewior In !PREEMPT_RT local_lock_irqsave() disables interrupts to protect critical section, but it doesn't prevent NMI, so the fully reentrant code cannot use local_lock_irqsave() for exclusive access. Introduce localtry_lock_t and localtry_lock_irqsave() that disables interrupts and sets acquired=1, so localtry_lock_irqsave() from NMI attempting to acquire the same lock will return false. In PREEMPT_RT local_lock_irqsave() maps to preemptible spin_lock(). Map localtry_lock_irqsave() to preemptible spin_trylock(). When in hard IRQ or NMI return false right away, since spin_trylock() is not safe due to PI issues. Note there is no need to use local_inc for acquired variable, since it's a percpu variable with strict nesting scopes. Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Alexei Starovoitov --- include/linux/local_lock.h | 59 +++++++++++++ include/linux/local_lock_internal.h | 123 ++++++++++++++++++++++++++++ 2 files changed, 182 insertions(+) diff --git a/include/linux/local_lock.h b/include/linux/local_lock.h index 091dc0b6bdfb..05c254a5d7d3 100644 --- a/include/linux/local_lock.h +++ b/include/linux/local_lock.h @@ -51,6 +51,65 @@ #define local_unlock_irqrestore(lock, flags) \ __local_unlock_irqrestore(lock, flags) +/** + * localtry_lock_init - Runtime initialize a lock instance + */ +#define localtry_lock_init(lock) __localtry_lock_init(lock) + +/** + * localtry_lock - Acquire a per CPU local lock + * @lock: The lock variable + */ +#define localtry_lock(lock) __localtry_lock(lock) + +/** + * localtry_lock_irq - Acquire a per CPU local lock and disable interrupts + * @lock: The lock variable + */ +#define localtry_lock_irq(lock) __localtry_lock_irq(lock) + +/** + * localtry_lock_irqsave - Acquire a per CPU local lock, save and disable + * interrupts + * @lock: The lock variable + * @flags: Storage for interrupt flags + */ +#define localtry_lock_irqsave(lock, flags) \ + __localtry_lock_irqsave(lock, flags) + +/** + * localtry_trylock_irqsave - Try to acquire a per CPU local lock, save and disable + * interrupts if acquired + * @lock: The lock variable + * @flags: Storage for interrupt flags + * + * The function can be used in any context such as NMI or HARDIRQ. Due to + * locking constrains it will _always_ fail to acquire the lock on PREEMPT_RT. + */ +#define localtry_trylock_irqsave(lock, flags) \ + __localtry_trylock_irqsave(lock, flags) + +/** + * local_unlock - Release a per CPU local lock + * @lock: The lock variable + */ +#define localtry_unlock(lock) __localtry_unlock(lock) + +/** + * local_unlock_irq - Release a per CPU local lock and enable interrupts + * @lock: The lock variable + */ +#define localtry_unlock_irq(lock) __localtry_unlock_irq(lock) + +/** + * localtry_unlock_irqrestore - Release a per CPU local lock and restore + * interrupt flags + * @lock: The lock variable + * @flags: Interrupt flags to restore + */ +#define localtry_unlock_irqrestore(lock, flags) \ + __localtry_unlock_irqrestore(lock, flags) + DEFINE_GUARD(local_lock, local_lock_t __percpu*, local_lock(_T), local_unlock(_T)) diff --git a/include/linux/local_lock_internal.h b/include/linux/local_lock_internal.h index 8dd71fbbb6d2..c1369b300777 100644 --- a/include/linux/local_lock_internal.h +++ b/include/linux/local_lock_internal.h @@ -15,6 +15,11 @@ typedef struct { #endif } local_lock_t; +typedef struct { + local_lock_t llock; + unsigned int acquired; +} localtry_lock_t; + #ifdef CONFIG_DEBUG_LOCK_ALLOC # define LOCAL_LOCK_DEBUG_INIT(lockname) \ .dep_map = { \ @@ -31,6 +36,13 @@ static inline void local_lock_acquire(local_lock_t *l) l->owner = current; } +static inline void local_trylock_acquire(local_lock_t *l) +{ + lock_map_acquire_try(&l->dep_map); + DEBUG_LOCKS_WARN_ON(l->owner); + l->owner = current; +} + static inline void local_lock_release(local_lock_t *l) { DEBUG_LOCKS_WARN_ON(l->owner != current); @@ -45,11 +57,13 @@ static inline void local_lock_debug_init(local_lock_t *l) #else /* CONFIG_DEBUG_LOCK_ALLOC */ # define LOCAL_LOCK_DEBUG_INIT(lockname) static inline void local_lock_acquire(local_lock_t *l) { } +static inline void local_trylock_acquire(local_lock_t *l) { } static inline void local_lock_release(local_lock_t *l) { } static inline void local_lock_debug_init(local_lock_t *l) { } #endif /* !CONFIG_DEBUG_LOCK_ALLOC */ #define INIT_LOCAL_LOCK(lockname) { LOCAL_LOCK_DEBUG_INIT(lockname) } +#define INIT_LOCALTRY_LOCK(lockname) { .llock = { LOCAL_LOCK_DEBUG_INIT(lockname.llock) }} #define __local_lock_init(lock) \ do { \ @@ -118,6 +132,86 @@ do { \ #define __local_unlock_nested_bh(lock) \ local_lock_release(this_cpu_ptr(lock)) +/* localtry_lock_t variants */ + +#define __localtry_lock_init(lock) \ +do { \ + __local_lock_init(&(lock)->llock); \ + WRITE_ONCE(&(lock)->acquired, 0); \ +} while (0) + +#define __localtry_lock(lock) \ + do { \ + localtry_lock_t *lt; \ + preempt_disable(); \ + lt = this_cpu_ptr(lock); \ + local_lock_acquire(<->llock); \ + WRITE_ONCE(lt->acquired, 1); \ + } while (0) + +#define __localtry_lock_irq(lock) \ + do { \ + localtry_lock_t *lt; \ + local_irq_disable(); \ + lt = this_cpu_ptr(lock); \ + local_lock_acquire(<->llock); \ + WRITE_ONCE(lt->acquired, 1); \ + } while (0) + +#define __localtry_lock_irqsave(lock, flags) \ + do { \ + localtry_lock_t *lt; \ + local_irq_save(flags); \ + lt = this_cpu_ptr(lock); \ + local_lock_acquire(<->llock); \ + WRITE_ONCE(lt->acquired, 1); \ + } while (0) + +#define __localtry_trylock_irqsave(lock, flags) \ + ({ \ + localtry_lock_t *lt; \ + bool _ret; \ + \ + local_irq_save(flags); \ + lt = this_cpu_ptr(lock); \ + if (!READ_ONCE(lt->acquired)) { \ + WRITE_ONCE(lt->acquired, 1); \ + local_trylock_acquire(<->llock); \ + _ret = true; \ + } else { \ + _ret = false; \ + local_irq_restore(flags); \ + } \ + _ret; \ + }) + +#define __localtry_unlock(lock) \ + do { \ + localtry_lock_t *lt; \ + lt = this_cpu_ptr(lock); \ + WRITE_ONCE(lt->acquired, 0); \ + local_lock_release(<->llock); \ + preempt_enable(); \ + } while (0) + +#define __localtry_unlock_irq(lock) \ + do { \ + localtry_lock_t *lt; \ + lt = this_cpu_ptr(lock); \ + WRITE_ONCE(lt->acquired, 0); \ + local_lock_release(<->llock); \ + local_irq_enable(); \ + } while (0) + +#define __localtry_unlock_irqrestore(lock, flags) \ + do { \ + localtry_lock_t *lt; \ + lt = this_cpu_ptr(lock); \ + WRITE_ONCE(lt->acquired, 0); \ + local_lock_release(<->llock); \ + local_irq_restore(flags); \ + } while (0) + #else /* !CONFIG_PREEMPT_RT */ /* @@ -125,8 +219,10 @@ do { \ * critical section while staying preemptible. */ typedef spinlock_t local_lock_t; +typedef spinlock_t localtry_lock_t; #define INIT_LOCAL_LOCK(lockname) __LOCAL_SPIN_LOCK_UNLOCKED((lockname)) +#define INIT_LOCALTRY_LOCK(lockname) INIT_LOCAL_LOCK(lockname) #define __local_lock_init(l) \ do { \ @@ -169,4 +265,31 @@ do { \ spin_unlock(this_cpu_ptr((lock))); \ } while (0) +/* localtry_lock_t variants */ + +#define __localtry_lock_init(lock) __local_lock_init(lock) +#define __localtry_lock(lock) __local_lock(lock) +#define __localtry_lock_irq(lock) __local_lock(lock) +#define __localtry_lock_irqsave(lock, flags) __local_lock_irqsave(lock, flags) +#define __localtry_unlock(lock) __local_unlock(lock) +#define __localtry_unlock_irq(lock) __local_unlock(lock) +#define __localtry_unlock_irqrestore(lock, flags) __local_unlock_irqrestore(lock, flags) + +#define __localtry_trylock_irqsave(lock, flags) \ + ({ \ + int __locked; \ + \ + typecheck(unsigned long, flags); \ + flags = 0; \ + if (in_nmi() | in_hardirq()) { \ + __locked = 0; \ + } else { \ + migrate_disable(); \ + __locked = spin_trylock(this_cpu_ptr((lock))); \ + if (!__locked) \ + migrate_enable(); \ + } \ + __locked; \ + }) + #endif /* CONFIG_PREEMPT_RT */ From patchwork Wed Feb 12 17:47:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13972252 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A522325C6F1 for ; Wed, 12 Feb 2025 17:47:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739382449; cv=none; b=CSOvCuCOimO7JDGXqsbPm5iM2QW6YC5B0d7Ec20rWGddvqnnbLCEtoNj8gbdDlsCojoZIKukz7WOA6J/ire036KoJ1OtHLhemXbFPyDV2YRyiLRe5LwIOKFdg4yJHOxHSfbFN+KMKNY59+5+kkiNOVi3LDzWyeM2VUoO00rpGxc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739382449; c=relaxed/simple; bh=rvkD0voG5AFWPbxH/cFvKKuqBd+a8nG/qfAcky5jsKI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Rcm0/2DumftLCK73IQSH1wEdtr6yGlTpGeodCcRkPB6ThjnRPzKxc+QQAAcg3Vq1hn1SYMUe0HGuSrJgJ7JZmCKV3Na5n3CsWoO097AxiMf2mFczNEUMqkzE9epkywf0kv4k1TXVt27D+e/nzHnWv2rptH4E6QeimFszxlLN5sA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Px6cIex+; arc=none smtp.client-ip=209.85.216.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Px6cIex+" Received: by mail-pj1-f45.google.com with SMTP id 98e67ed59e1d1-2f9bac7699aso78869a91.1 for ; Wed, 12 Feb 2025 09:47:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739382447; x=1739987247; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lgL2WdPUSp9IlhXwRVgFpZY7gPxe7SOzqv/p6kt86b8=; b=Px6cIex+7mP7z7X6xdOOXQdxAFn5OKR84+RXhOv9wooAwgKrsmEM7HV+J/WJFFk+IS 1dk0Idq8KnTs/2QW7/28kkjMB4BottzGr/Bk3ifNayRuFznrPuXcjYBU1cDKPmx5l8J2 HzDcmi19DrZlzhk5n18n/RHL1PyhGzEuqVpmWh/AjJnXuDH0/q2JYUpKE5+3zaSZTQpK MjYOoWJqj5iVpAzO4rxoeFCMf5ufshoOoUclfDWNeP+iowgnzUCa11VpTmX4x1K69Vke G6XqYRuzvSt1ZPJ7wc2RGnWbnEGRNgkeBcWN8QTQhekOTzpFlIfXTACNvImoSWzVicRA c/Kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739382447; x=1739987247; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lgL2WdPUSp9IlhXwRVgFpZY7gPxe7SOzqv/p6kt86b8=; b=ZPz2ise4TOj2rOvKhd9DYHjxlWgXO5w5lXrmFVR8Mv/mzZKhJxHY2i7J7+94QzJAxU K5g7hRDMFKBXDK8qUAOe1G8uCH8BWma5leTVKrebAEEQJYLIxDxinlutW9clK96Xwl9m gXuMkNoJLDLese9XD/HjbxM4q6+AU5xMaSO2v5wb3O+KZXanlk5fbcAP8WEtECS2v4WQ WGWR0YegAXXmMovb8YIXQ3wDFKEYlTu9HS0zRUrtaKm8eUiC3JzjqKVBczlNEho16XJj be3B2ngi1MedidblKoO3BsO3CmddSiQvahUsfof0c7sqwhLBmaUm0zRQmi2g1EII34Y6 2tMA== X-Gm-Message-State: AOJu0YzZSY2Tlzo/aVnGeciKa+i2bjBHInLxawx2b71aQ+03lGM8gjGb UrVY2pAMGO/Vs8NIBinGKJmHMo1s7gup0JFq0FP7t7p3fxNkLhTFY9CqBQ== X-Gm-Gg: ASbGncuPgNmhqBkrOjP0obUuvAlueVb6GkJlob0MDVkCfc+QVIqG4yUQBWOcAkJxnrx QsONyMCCOwQjBeDKRYagRZ90zEBEHVNtPVHKHt9pZonX3uagIeavIQESu05XIvgF+mXKWf+v1z+ wsMf3Gaxf6d4XB683L7CpwG2TR+UvND1XDp8StXBJuJgvwpiC7yZ6MkNiFrTx0cpPyb3pR41bmT mSA9MDjFfNumab4ppseRWaVb+uL6I/t0c0d1THb9tZ8olHm+B0F8cx1HS/b9vvc/RZpD9uuiY7W wEZ5WUC715P451dzV8+1VDzpTd6mQGSaXTxatfy3MuBu6Q== X-Google-Smtp-Source: AGHT+IGEj3NYQQrGwrnzg5FtQkvq3P2X1QMPIINMUYPk4XwQqFltxvoCKJbMwmmlCOqX7njlFq8C2Q== X-Received: by 2002:a17:90b:1dcb:b0:2ee:bbd8:2b9d with SMTP id 98e67ed59e1d1-2fbf91358eamr4369795a91.34.1739382446220; Wed, 12 Feb 2025 09:47:26 -0800 (PST) Received: from ast-mac.thefacebook.com ([2620:10d:c090:500::4:c330]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21f36896b86sm116452455ad.212.2025.02.12.09.47.24 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 12 Feb 2025 09:47:25 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v7 4/6] memcg: Use trylock to access memcg stock_lock. Date: Wed, 12 Feb 2025 09:47:03 -0800 Message-Id: <20250212174705.44492-5-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250212174705.44492-1-alexei.starovoitov@gmail.com> References: <20250212174705.44492-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Teach memcg to operate under trylock conditions when spinning locks cannot be used. localtry_trylock might fail and this would lead to charge cache bypass if the calling context doesn't allow spinning (gfpflags_allow_spinning). In those cases charge the memcg counter directly and fail early if that is not possible. This might cause a pre-mature charge failing but it will allow an opportunistic charging that is safe from try_alloc_pages path. Acked-by: Michal Hocko Acked-by: Vlastimil Babka Acked-by: Shakeel Butt Signed-off-by: Alexei Starovoitov --- mm/memcontrol.c | 52 ++++++++++++++++++++++++++++++++++--------------- 1 file changed, 36 insertions(+), 16 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7b3503d12aaf..f3af615f727c 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1722,7 +1722,7 @@ void mem_cgroup_print_oom_group(struct mem_cgroup *memcg) } struct memcg_stock_pcp { - local_lock_t stock_lock; + localtry_lock_t stock_lock; struct mem_cgroup *cached; /* this never be root cgroup */ unsigned int nr_pages; @@ -1737,7 +1737,7 @@ struct memcg_stock_pcp { #define FLUSHING_CACHED_CHARGE 0 }; static DEFINE_PER_CPU(struct memcg_stock_pcp, memcg_stock) = { - .stock_lock = INIT_LOCAL_LOCK(stock_lock), + .stock_lock = INIT_LOCALTRY_LOCK(stock_lock), }; static DEFINE_MUTEX(percpu_charge_mutex); @@ -1756,7 +1756,8 @@ static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, * * returns true if successful, false otherwise. */ -static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) +static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages, + gfp_t gfp_mask) { struct memcg_stock_pcp *stock; unsigned int stock_pages; @@ -1766,7 +1767,11 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) if (nr_pages > MEMCG_CHARGE_BATCH) return ret; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!localtry_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + if (!gfpflags_allow_spinning(gfp_mask)) + return ret; + localtry_lock_irqsave(&memcg_stock.stock_lock, flags); + } stock = this_cpu_ptr(&memcg_stock); stock_pages = READ_ONCE(stock->nr_pages); @@ -1775,7 +1780,7 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) ret = true; } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags); return ret; } @@ -1814,14 +1819,14 @@ static void drain_local_stock(struct work_struct *dummy) * drain_stock races is that we always operate on local CPU stock * here with IRQ disabled */ - local_lock_irqsave(&memcg_stock.stock_lock, flags); + localtry_lock_irqsave(&memcg_stock.stock_lock, flags); stock = this_cpu_ptr(&memcg_stock); old = drain_obj_stock(stock); drain_stock(stock); clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags); obj_cgroup_put(old); } @@ -1851,9 +1856,20 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) { unsigned long flags; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!localtry_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + /* + * In case of unlikely failure to lock percpu stock_lock + * uncharge memcg directly. + */ + if (mem_cgroup_is_root(memcg)) + return; + page_counter_uncharge(&memcg->memory, nr_pages); + if (do_memsw_account()) + page_counter_uncharge(&memcg->memsw, nr_pages); + return; + } __refill_stock(memcg, nr_pages); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags); } /* @@ -2196,9 +2212,13 @@ int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, unsigned long pflags; retry: - if (consume_stock(memcg, nr_pages)) + if (consume_stock(memcg, nr_pages, gfp_mask)) return 0; + if (!gfpflags_allow_spinning(gfp_mask)) + /* Avoid the refill and flush of the older stock */ + batch = nr_pages; + if (!do_memsw_account() || page_counter_try_charge(&memcg->memsw, batch, &counter)) { if (page_counter_try_charge(&memcg->memory, batch, &counter)) @@ -2709,7 +2729,7 @@ static void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, unsigned long flags; int *bytes; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + localtry_lock_irqsave(&memcg_stock.stock_lock, flags); stock = this_cpu_ptr(&memcg_stock); /* @@ -2762,7 +2782,7 @@ static void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, if (nr) __mod_objcg_mlstate(objcg, pgdat, idx, nr); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags); obj_cgroup_put(old); } @@ -2772,7 +2792,7 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) unsigned long flags; bool ret = false; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + localtry_lock_irqsave(&memcg_stock.stock_lock, flags); stock = this_cpu_ptr(&memcg_stock); if (objcg == READ_ONCE(stock->cached_objcg) && stock->nr_bytes >= nr_bytes) { @@ -2780,7 +2800,7 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) ret = true; } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags); return ret; } @@ -2872,7 +2892,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, unsigned long flags; unsigned int nr_pages = 0; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + localtry_lock_irqsave(&memcg_stock.stock_lock, flags); stock = this_cpu_ptr(&memcg_stock); if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */ @@ -2890,7 +2910,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, stock->nr_bytes &= (PAGE_SIZE - 1); } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags); obj_cgroup_put(old); if (nr_pages) From patchwork Wed Feb 12 17:47:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13972253 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 27D202586E6 for ; Wed, 12 Feb 2025 17:47:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739382453; cv=none; b=mpjakLTP1P2pGA557EBwPfBmWkadkenAuaOJ1G321/XQqV/o5TrlTikvlP4GNU66VQjWpjx8GLKm4SAbFtHPWvPotv13xrh+aLXfpNGeg/ibL2xj+VKvx+6KqR6aGZSFyAEO4D76fXCGbRxR850Di8VK0Q2MivF7EfaZ18iaKK0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739382453; c=relaxed/simple; bh=PkbsL7XC8j5LBWMTJb1ZtearOWIvOtlSfq++Sl18MI8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=iPM/voIXPgwfHL0uKFPERwqJPjma5dSz5964BsP9GwvnV4TzEpnGkk2IRTENquldKT/n2rOrSwSoBy//RqTCvq0SLy5MmdoL82ySG1WZNgEWTkH6XXw6LsfCBMZhUhZ648uqY+IaRfAq0zNVPKWtLD1oc+oldrbXHq07UCZNk+o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=UVN3LoKu; arc=none smtp.client-ip=209.85.216.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UVN3LoKu" Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-2fa2eb7eb45so1941478a91.1 for ; Wed, 12 Feb 2025 09:47:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739382451; x=1739987251; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iXCJ0gdG5EF9yWrbtRQyNOOIbBLtByDZRqzMTlbBC2I=; b=UVN3LoKuhI1MBlV33+dmZswMaxUufID29CAGnbMJQe2we1GUm3ZtEoi0f1Kl8xEvFc lC70bzlWNLtLXOJi0ahnWbRUgCo/v6yUAOyA6nTGpLiWA7isyCdCdjyPrVBbdHg1S4dp Jf4MVS8DJHwS5w3Wb3dMfkgb1JMMkZDqQQEgBTRhe39HJ4Hy7VPet1vbBuDx9dkj88kh 6ZBYGVfZD1/JIoIPwJKS7VS3mooh2n/ae2kHpJPri4j4XLenY5IgPkorPnenlVtbmCZB XFftsraelJ189luxA205AinrYuKKV/FxlhB4shKZr226QEwORYg/yr9kDDqVLSeVEYmB 9Cjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739382451; x=1739987251; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iXCJ0gdG5EF9yWrbtRQyNOOIbBLtByDZRqzMTlbBC2I=; b=Raf1ansOymjVQLSiTH7fnN4KnFF6YMdt9dHoPLpeQ+QreCBDcpATUwVBj85Ua5Zz5W v5feI5xaIaAaSgzCB/JukJJtk7qbvPdDv3PdbxzrlWgWAM76I4RxWTOIvuiHPLYmvqK/ fsFACq2/ii+g0aq0dwv3ZhqtAwDi17noRgXnUK39BTQHFbWXXyVKLOik5ow2YHacHP8k mlXezVESQnD17sDUNLRcM9Hpwsiz1kWFfmuq6trGhVLVPd8cLdqRaBam78MJoiu2M3Zr TUiunci0RP7bOymiLHJyfCw82D1F05mdRahirA0Ta6QzJsw/uCPJA2sZjEHY9t6DsThE Flag== X-Gm-Message-State: AOJu0Yy06z2epsWOyFrJ21gBRCG0fS0owuakrs6MDP+d0R8CKqJ6IY2B wctDD+oS4VbLx8aCiilsYhXnHgR2TUJLvf5qF+S+RxJQCWxqCVhcOXglcw== X-Gm-Gg: ASbGnctXA74FvwnrOyB1xGFjccTpwpFmgaa1ym3axYtmZb8zQDQlmgS5BqkLqi44iN+ c7fDx3pFumr/z12DR9acYaw8DcGKMJ9Pa8nr4mFvinAW+OSJGPf7H9FE6HpB/XyKQLwwMXa+G3U MsUiqMd/N6pkQTM3UmmdzmD+mMnW0VeAPFrYbAK9k++iULj98/2Ep9F/6inxB8pFpKR64b/Tgem NYFc+YP5RdmjCBmNLs3m9Jtt9CDwDCQCxX7XCLZerG5nYaA2K+D4zs/Sg+GfBVoHyz1vrOQNppP o8fLSoKsJz2HQR+ToKigMQKrHkOnha21c2XCzZ3cc6sO3w== X-Google-Smtp-Source: AGHT+IGQQe2Ltprl1s0EVAQgrtpuMzau10NCDgeIlNPMnvs7s+Zh8a9Qe2IaEzLCOBtyjvHoHPP7WA== X-Received: by 2002:a17:90b:38c6:b0:2ea:8aac:6ac1 with SMTP id 98e67ed59e1d1-2fc0db48d32mr293200a91.15.1739382450483; Wed, 12 Feb 2025 09:47:30 -0800 (PST) Received: from ast-mac.thefacebook.com ([2620:10d:c090:500::4:c330]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2fbf9908835sm1805334a91.42.2025.02.12.09.47.28 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 12 Feb 2025 09:47:30 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v7 5/6] mm, bpf: Use memcg in try_alloc_pages(). Date: Wed, 12 Feb 2025 09:47:04 -0800 Message-Id: <20250212174705.44492-6-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250212174705.44492-1-alexei.starovoitov@gmail.com> References: <20250212174705.44492-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Unconditionally use __GFP_ACCOUNT in try_alloc_pages(). The caller is responsible to setup memcg correctly. All BPF memory accounting is memcg based. Acked-by: Vlastimil Babka Acked-by: Shakeel Butt Signed-off-by: Alexei Starovoitov --- mm/page_alloc.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index fa750c46e0fc..931cedcda788 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7146,7 +7146,8 @@ struct page *try_alloc_pages_noprof(int nid, unsigned int order) * specify it here to highlight that try_alloc_pages() * doesn't want to deplete reserves. */ - gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC; + gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC + | __GFP_ACCOUNT; unsigned int alloc_flags = ALLOC_TRYLOCK; struct alloc_context ac = { }; struct page *page; @@ -7190,6 +7191,11 @@ struct page *try_alloc_pages_noprof(int nid, unsigned int order) /* Unlike regular alloc_pages() there is no __alloc_pages_slowpath(). */ + if (memcg_kmem_online() && page && + unlikely(__memcg_kmem_charge_page(page, alloc_gfp, order) != 0)) { + free_pages_nolock(page, order); + page = NULL; + } trace_mm_page_alloc(page, order, alloc_gfp, ac.migratetype); kmsan_alloc_page(page, order, alloc_gfp); return page; From patchwork Wed Feb 12 17:47:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13972254 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E46012586E6 for ; Wed, 12 Feb 2025 17:47:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739382457; cv=none; b=UtP0c+PqDSive73ta4dZTAJDx7lOv7qAu1QLlNdf3HOZd69RXsYtVSZ1T7Ljf3evkJ5fpfGeOKOYlZrIJUgHYMy17CPowOxcqRnx0ok6XFhM99ser9DvLWiZi3HhadGunrURN7QPsS9HcHid5w+pApm8Y+5PCJYyCZr/vsyt6GQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739382457; c=relaxed/simple; bh=vfzkPtuMn7THzMDejWzHSa5NwJvatkmrRDhM3Ln6UxU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ZG78ZxOpEZyqdgt3bFExuqrQ1Mc4NlugYYoIlBsNCVaIFPtX1NFHE2daCs2lDydJdm3htQFkkMYNx3D8M3lFkVtUNEQIIdEsKP6b2SRuMawDWLTa8UapFAXlY3eqSn9ZbGNThjIVj7hnUeaBfX1B+xul5Fs2Rsn0ivvg8kohOtQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ahLzWmb4; arc=none smtp.client-ip=209.85.216.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ahLzWmb4" Received: by mail-pj1-f51.google.com with SMTP id 98e67ed59e1d1-2fa3fe04dd2so79055a91.0 for ; Wed, 12 Feb 2025 09:47:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739382455; x=1739987255; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bDuPZRtqsVh7bFH3yxNIFanE739bKkzWZ/UfDyq/wdw=; b=ahLzWmb4SjK/vznhv2SZA0rnH3N94FRz2G+jo6geuWVQrm3sSsboy8WLH2IkoMs5XL +Wtk8iVIPdq1ZUwKvrK3ycqo2ELh/xngtN6J/ogYKv5Ma2t5JGAlCcg/TKpTg2/2Vl5g 5ZlQQJJdRJiz3AcXDLSq5uzx1+bamY5qCaOieKRyeAzpv5VKwyv2exoxNy1CtQK5LIuq JYQHsZXMoP+pFBg+7ipkajRMbRozYSQZh8RaW7iIP/NCj6ag2+gD8zg6Ex0UZYsG5cd7 5RoYI2/qMwmfMdad4sRt4YyX3ncS9WPuBz3KoOlhQOtC+g1TzlCk2rEf0NwlYpRndXfD e9bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739382455; x=1739987255; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bDuPZRtqsVh7bFH3yxNIFanE739bKkzWZ/UfDyq/wdw=; b=bcMgGjf+rzMMhvsYVErthRJstZldOBJCt9XLuRZ5ka1GxjVNeGFzFjkMxmdwvAOrYq KPb4U0DXpZS7ShyryL3IVmsO4rz/2P1gxhuID2OMDLwMGV6j+CiDsdZ+R8uPBAcX/sjI KDhQgSfDUfm6K9kF4HNbL/VzIyO+JT/ZXM8U4JeoOB2kkyh2k3NCngcUcqQZ3XWhktzY 3nLUqZuU8mw2QrwK0PQapfhwBAs66ef/PK4/2/XKvUyXPf5FHakXCOz+OY3aRxW8PXBM JGPyzhrEor9QyPvbnhv2SRvWESREq20iqZ011JeGw/pY5/7jGXyqG9ZMNfvSNNHu5W1S IFqw== X-Gm-Message-State: AOJu0Yxw8dM53D+XMKvlrcB+UJYDoZ/qRr1GUy1jqdSAfZqmfx2xlvN1 i6Ppg45ydtSTnvZlWGWHx4YtMdK9rcdkOJyfphGmqoSMMZ8IrPZnphuGqA== X-Gm-Gg: ASbGncvKSIjzxgEREgU7iPEBCKtq76dAwBka41OhoC8eb3Mdz/kdFuw2tO1cFx8rU5h fk1XD+US3mWuUiSDd6qx6glA/h95s6VA6WGrImQ5KK1zlScVyswvhk8pcFkalZq2bee+2VO3wy1 Zt1ifQ7NbvX+tZHqg+lAut+uwPBpA9MZXdwEiHmdFMXX+vT/7bQfF18MIIZFaT4ZmOhwB2oq+fu m9l+5mVxPX8bfOr7Mp0n4xgKgEYWc2Q/32F7nm0lPLJQp/tZxhKi1TdI6t1qyQoEuj68Ccf6y/C r3n/xv9zu6Tk3LPf+djqCshlGX2d4zeljU3krnpksZ1RcA== X-Google-Smtp-Source: AGHT+IE9puw91tznIalD6EtLkH8J2zj9zmB6kCfaMb8CIT8gL4HusyDm8J9VmRaGQdeer0qVub9Dvw== X-Received: by 2002:a17:90b:280b:b0:2f8:2c47:fb36 with SMTP id 98e67ed59e1d1-2fbf5c6ea65mr7171319a91.33.1739382454794; Wed, 12 Feb 2025 09:47:34 -0800 (PST) Received: from ast-mac.thefacebook.com ([2620:10d:c090:500::4:c330]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21f36882dd7sm115611505ad.199.2025.02.12.09.47.32 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 12 Feb 2025 09:47:34 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v7 6/6] bpf: Use try_alloc_pages() to allocate pages for bpf needs. Date: Wed, 12 Feb 2025 09:47:05 -0800 Message-Id: <20250212174705.44492-7-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250212174705.44492-1-alexei.starovoitov@gmail.com> References: <20250212174705.44492-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Use try_alloc_pages() and free_pages_nolock() for BPF needs when context doesn't allow using normal alloc_pages. This is a prerequisite for further work. Signed-off-by: Alexei Starovoitov --- include/linux/bpf.h | 2 +- kernel/bpf/arena.c | 5 ++--- kernel/bpf/syscall.c | 23 ++++++++++++++++++++--- 3 files changed, 23 insertions(+), 7 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index f3f50e29d639..e1838a341817 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -2348,7 +2348,7 @@ int generic_map_delete_batch(struct bpf_map *map, struct bpf_map *bpf_map_get_curr_or_next(u32 *id); struct bpf_prog *bpf_prog_get_curr_or_next(u32 *id); -int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, +int bpf_map_alloc_pages(const struct bpf_map *map, int nid, unsigned long nr_pages, struct page **page_array); #ifdef CONFIG_MEMCG void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c index 4b22a651b5d5..642399a5fd9f 100644 --- a/kernel/bpf/arena.c +++ b/kernel/bpf/arena.c @@ -287,7 +287,7 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf) return VM_FAULT_SIGSEGV; /* Account into memcg of the process that created bpf_arena */ - ret = bpf_map_alloc_pages(map, GFP_KERNEL | __GFP_ZERO, NUMA_NO_NODE, 1, &page); + ret = bpf_map_alloc_pages(map, NUMA_NO_NODE, 1, &page); if (ret) { range_tree_set(&arena->rt, vmf->pgoff, 1); return VM_FAULT_SIGSEGV; @@ -465,8 +465,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt if (ret) goto out_free_pages; - ret = bpf_map_alloc_pages(&arena->map, GFP_KERNEL | __GFP_ZERO, - node_id, page_cnt, pages); + ret = bpf_map_alloc_pages(&arena->map, node_id, page_cnt, pages); if (ret) goto out; diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 0daf098e3207..55588dbd2fce 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -569,7 +569,24 @@ static void bpf_map_release_memcg(struct bpf_map *map) } #endif -int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, +static bool can_alloc_pages(void) +{ + return preempt_count() == 0 && !irqs_disabled() && + !IS_ENABLED(CONFIG_PREEMPT_RT); +} + +static struct page *__bpf_alloc_page(int nid) +{ + if (!can_alloc_pages()) + return try_alloc_pages(nid, 0); + + return alloc_pages_node(nid, + GFP_KERNEL | __GFP_ZERO | __GFP_ACCOUNT + | __GFP_NOWARN, + 0); +} + +int bpf_map_alloc_pages(const struct bpf_map *map, int nid, unsigned long nr_pages, struct page **pages) { unsigned long i, j; @@ -582,14 +599,14 @@ int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, old_memcg = set_active_memcg(memcg); #endif for (i = 0; i < nr_pages; i++) { - pg = alloc_pages_node(nid, gfp | __GFP_ACCOUNT, 0); + pg = __bpf_alloc_page(nid); if (pg) { pages[i] = pg; continue; } for (j = 0; j < i; j++) - __free_page(pages[j]); + free_pages_nolock(pages[j], 0); ret = -ENOMEM; break; }