From patchwork Wed Dec 18 03:07:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13912974 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F7E7E77187 for ; Wed, 18 Dec 2024 03:07:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7D9486B0085; Tue, 17 Dec 2024 22:07:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 78AD16B0088; Tue, 17 Dec 2024 22:07:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 62A146B0089; Tue, 17 Dec 2024 22:07:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 446916B0085 for ; Tue, 17 Dec 2024 22:07:33 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id CA4BE120AFB for ; Wed, 18 Dec 2024 03:07:32 +0000 (UTC) X-FDA: 82906593336.21.91B9FAC Received: from mail-ot1-f43.google.com (mail-ot1-f43.google.com [209.85.210.43]) by imf18.hostedemail.com (Postfix) with ESMTP id BB9311C0014 for ; Wed, 18 Dec 2024 03:07:16 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=NIEshg0A; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf18.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.43 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734491236; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OGzoMP4pz94E3iZ2ZVAY+FjoyH+9uJ8+ltaU5mTC/cc=; b=7i5VDMoy7kQDhm6uZ9qkaYgOok5NICPazYVIYbAb1cADylX1jZrCkLEgr4M1sAvm12hoiv NyDVXaBJ6nk9GafdUz0fVtrR7cVjMuIAp+NWD3F3QXYzdADb9xwHtGsEbSxqS9X68TbDfc zsJjxeymnn5UG6KaWmqx7aI4CXfco74= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734491236; a=rsa-sha256; cv=none; b=fcRfGoylR2T4ipW0ADaKkIJ527ufQmA3ixX8TZ2EqDoF3vscfG5IUmfCCmeodJ06vAScYN ccHLiNsNRVigxB+PaGBzMlRP6FEo/z6e+T+n3F2fma3nh3AqNYWjs7zIV7FkfofnPVklVy H+h2I8n3afcuvFFuGXGicsC9+m42Zz0= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=NIEshg0A; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf18.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.43 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com Received: by mail-ot1-f43.google.com with SMTP id 46e09a7af769-71e2aa8d5e3so3072766a34.2 for ; Tue, 17 Dec 2024 19:07:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734491250; x=1735096050; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OGzoMP4pz94E3iZ2ZVAY+FjoyH+9uJ8+ltaU5mTC/cc=; b=NIEshg0AIj6q6XKHDpgFfCECDQ10hk/XZ0hbzfdxMOl6gCvAXDde6Z8JHvyBC8uJjR FUU0zCRU3OnrIG1udFUsMemDSBfMwKXeDZ0CAN/3mUA5CVeeuYhLbVyRQkI+8Q0qriu9 8s56sjOu6d2IP+YIoYj3fMzLePB6DAx94xzGQ+QqTpZCemDaKHyqfEAATe54d0OAH9W3 dbnFTpy2mK1XmX76NeVGVFdb3MtPnPHz9P9yBBDpykMmW492RU3yo4GCGIZ+k99gt7eI 0yCnXitFoVPsO4mWgt41V+VR9/jUHZBzmX7Vste7wRl5aeT+wpClpzzHsBbZ2UbA+RV6 atCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734491250; x=1735096050; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OGzoMP4pz94E3iZ2ZVAY+FjoyH+9uJ8+ltaU5mTC/cc=; b=AAbcwEV9JUaWFOrAHb1lDUkG87hpzkfUXuU5PzphSUvD7g1BzA5IquW05ElWZD65uO hj6O5r8ccJrgcAFI5fKGV6TiiwAvnj0r9/qEIYC/h7heCOVjJxMvUykaxtfCbwKemJ2K LcRQejub0cxw4kpQPTFS3PhXnB6DD78kWOelL7yBnI23Kt/TRBFdeQaPpos7xBYjhEaI smo/TAJWVoeGhPbouGwY+dDZurwVk+gJj5MCTrN0WdRv918PfkZ3GHUVGYAFm3ujQ9aI is1nF/ixjkp9UwFtGJPIZU4oRCA3ZeAls7DlbebFvODkYa2mPXALRh9eJe2UiiT0t4D0 wKGw== X-Forwarded-Encrypted: i=1; AJvYcCX08ykVl9Oz4Ky6NdMvevIco4cS9wEXX9/XZ7qhrX5pBKlh6iCF8acE85RelEGryN4qUdVOUTqmlQ==@kvack.org X-Gm-Message-State: AOJu0YxfmxBTZ+NTPvB5mAYOY8aFDfP4tFcrjEcZuC9+pH0gIN0/6JmQ IPk4PMYJqM/Zz/CTAtk36OAvsZbMFV8pFZS6g9Z3s4YJSP1aCJuE X-Gm-Gg: ASbGnct44PnMZ04TVvSgatW62rM94cJzbLgFybs3X3e8awQGf9INkQUuKbUuG67T5Pt Be4ONioyuVpdAw51QR6huzxgLYp8K/SKGGGW+WAnntsRaNfNKHol84FvSRvSxYaeAmVvDwJWKtr fBQD8ZJMxUkRTlF1oe9o3nKPDmgAfTglrGZaNRHxjsaiyXLdLLiVKU3LezI8zvesYM710+qdWLh fCvbrfKIixzSbbOBYx8vlW4E2DN/A4C/43CrdDDL5zEx9ZmvjuRrnDjnK648w== X-Google-Smtp-Source: AGHT+IHS1I5BEwtWaQWxY8nR9/3Gb5s3FBDzligOvFHMM+1wBgrv8C56AV9pKBLPsNZs5Vq+R7EbDg== X-Received: by 2002:a05:6830:4391:b0:71d:eee3:fd26 with SMTP id 46e09a7af769-71fb75611a4mr755835a34.4.1734491249808; Tue, 17 Dec 2024 19:07:29 -0800 (PST) Received: from localhost ([2a03:2880:12ff:4::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-71e48307e07sm2482972a34.1.2024.12.17.19.07.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Dec 2024 19:07:28 -0800 (PST) From: alexei.starovoitov@gmail.com To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v3 1/6] mm, bpf: Introduce try_alloc_pages() for opportunistic page allocation Date: Tue, 17 Dec 2024 19:07:14 -0800 Message-ID: <20241218030720.1602449-2-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241218030720.1602449-1-alexei.starovoitov@gmail.com> References: <20241218030720.1602449-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Stat-Signature: p76a3ygt4dp8hunumbacaopt3iy5kxnn X-Rspamd-Queue-Id: BB9311C0014 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1734491236-222264 X-HE-Meta: U2FsdGVkX19PH33FuuaoGZyKJid3y1YXpKTlMHJ/zKi88+aXHI3JkqfAEgPmFqjagdtFhJ5C4UDSYKOHgCJvS6bMLgbOHpa6GPQKhFSTQKZOGs6Fgdb7G4pOQN3ELbZnyp+IU+WFzazyROg8f9JhEOvtUgGoiwzE6HqU/x6syK9/w1dE/5ykRRsi8YM8jAkFUuHsiHDeIgeC2E8c18vQBQDrtktPgDL41Gv3C9SZODWyVVcg0x7+kdjzZUZgdhWOkE7llazMOB3cg1BoVez8oDP3aYPFlHJrjFWRbA5LQcbJtpxInzTMvV2uvNYFx5uajF9Q4U0zT8iF5nWmfVlJUW2zWRM/uHrF+Z8bOZOUjUUS/W2qOiyU0+S/boq+P88o9HJCv/r+0gj3xRA3jxrmMxo5RB+xGQIKTSAxerhxVXZp0XDNh9Fue1U+FVITlqxZDIvpZhqKLJHXM9CTEdPXS8lnBuPDB3Pry0Nb2zM7Fn5xmJE/c0UKChTYl/YmJqkhCUgJ+8bBlOvV8V2L7mS/sdqKwOxa1guwmo9DfHRS0Sub6XHSd4RsBbgkwcbEfbwFtSqUMkGsPD6Sk2Sap0tXtx3ytQqQYeAhLH1njYZdWA6K56UqUszZR5PjoB8Qoomn+BTQDUSTwdv88cmlJPdzR0eUTLacbjAJ0+gycJFiJzQIQ1b5Ccq9snMKkm/0aS6UVVi6p8G+mAUDaOCSmYWMC4uZk0L418ahg0/IEyFZTKI+79yJn96v4LgmvAkxI8POtSODgOnO3hwNqw1SJMsVkFJOREzl3J3tNcu861et7/Zlql327tfPfhmhhMXhm1XBU2y6nzyBsG1AZZJv89DHrcsk9gNMLJIT5rAQJ9vCs0jBIcXnpujayHKU4Mku5ghLBWACU/ISjV0mv9koChMFgY/0GKpr89v+GS08fSV+tZPTGOQfhW7rJFKBAnE9dCrx//VY4ox115i/sRALFsa VVFgCWTs Ht3L1/5EJKRNpVSujM3ZGxdSZKwcm1qrbBGG4174d+CM9iQ2+90bmlkTu5eCnYFwlp8In885EfkdRW9yCvxgFCFE2wIxvdLMz359zpYxTIecJBfOOQwOUov7+qKsT+8WoFyv+S24rYaKbc3046tXI8Urh7ukkQP9OIpGr3HX3WdIPwh+9ejc1Y55ZZquVSdrcA8hV1wOrpUz6Asalvr0FWXYeeZMu4d7PZH8yMPJPz1nYMY+op3g1gTzqWfXBQQFd3zE/lZ0zL+rqQtFfOzGOcdi/l0Ay9FBHyzCzL4BmyHel2qrmMMYwWebb+4xVt4CiAwJ7IvKiFkSDpC18qR3T9ykgFPHdRVIdFqnTC3h50ildEllf12NGZRFjNts3/46/84k8C5N721FAoLcP9NPYwCePGivq2sfvlY7hCbjd19O2u5cl332GF/ojBO1GLXDufSYRV8+UISui7YD1F+rh1MoyO/bHrqEXPq118oEdg1SQWpiAs2WRFT4FV6mIaGrunOArFsq3+thEDe9Ks0ZHPmXXlxZxmgQuhGeaE+AMN3s7YpYemsS56ejoPQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Tracing BPF programs execute from tracepoints and kprobes where running context is unknown, but they need to request additional memory. The prior workarounds were using pre-allocated memory and BPF specific freelists to satisfy such allocation requests. Instead, introduce internal __GFP_TRYLOCK flag that makes page allocator accessible from any context. It relies on percpu free list of pages that rmqueue_pcplist() should be able to pop the page from. If it fails (due to IRQ re-entrancy or list being empty) then try_alloc_pages() attempts to spin_trylock zone->lock and refill percpu freelist as normal. BPF program may execute with IRQs disabled and zone->lock is sleeping in RT, so trylock is the only option. In theory we can introduce percpu reentrance counter and increment it every time spin_lock_irqsave(&zone->lock, flags) is used, but we cannot rely on it. Even if this cpu is not in page_alloc path the spin_lock_irqsave() is not safe, since BPF prog might be called from tracepoint where preemption is disabled. So trylock only. Note, free_page and memcg are not taught about __GFP_TRYLOCK yet. The support comes in the next patches. This is a first step towards supporting BPF requirements in SLUB and getting rid of bpf_mem_alloc. That goal was discussed at LSFMM: https://lwn.net/Articles/974138/ Signed-off-by: Alexei Starovoitov --- include/linux/gfp.h | 3 ++ include/linux/gfp_types.h | 1 + mm/internal.h | 2 ++ mm/page_alloc.c | 69 ++++++++++++++++++++++++++++++++++++--- 4 files changed, 71 insertions(+), 4 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index b0fe9f62d15b..65b8df1db26a 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -347,6 +347,9 @@ static inline struct page *alloc_page_vma_noprof(gfp_t gfp, } #define alloc_page_vma(...) alloc_hooks(alloc_page_vma_noprof(__VA_ARGS__)) +struct page *try_alloc_pages_noprof(int nid, unsigned int order); +#define try_alloc_pages(...) alloc_hooks(try_alloc_pages_noprof(__VA_ARGS__)) + extern unsigned long get_free_pages_noprof(gfp_t gfp_mask, unsigned int order); #define __get_free_pages(...) alloc_hooks(get_free_pages_noprof(__VA_ARGS__)) diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index 65db9349f905..65b148ec86eb 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -48,6 +48,7 @@ enum { ___GFP_THISNODE_BIT, ___GFP_ACCOUNT_BIT, ___GFP_ZEROTAGS_BIT, + ___GFP_TRYLOCK_BIT, #ifdef CONFIG_KASAN_HW_TAGS ___GFP_SKIP_ZERO_BIT, ___GFP_SKIP_KASAN_BIT, diff --git a/mm/internal.h b/mm/internal.h index cb8d8e8e3ffa..122fce7e1a9e 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1175,6 +1175,8 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, #endif #define ALLOC_HIGHATOMIC 0x200 /* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_KSWAPD 0x800 /* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ +#define __GFP_TRYLOCK ((__force gfp_t)BIT(___GFP_TRYLOCK_BIT)) +#define ALLOC_TRYLOCK 0x1000000 /* Only use spin_trylock in allocation path */ /* Flags that allow allocations below the min watermark. */ #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 1cb4b8c8886d..d23545057b6e 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2304,7 +2304,11 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, unsigned long flags; int i; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(alloc_flags & ALLOC_TRYLOCK)) + return 0; + spin_lock_irqsave(&zone->lock, flags); + } for (i = 0; i < count; ++i) { struct page *page = __rmqueue(zone, order, migratetype, alloc_flags); @@ -2904,7 +2908,11 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, do { page = NULL; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(alloc_flags & ALLOC_TRYLOCK)) + return NULL; + spin_lock_irqsave(&zone->lock, flags); + } if (alloc_flags & ALLOC_HIGHATOMIC) page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC); if (!page) { @@ -4001,6 +4009,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) */ BUILD_BUG_ON(__GFP_HIGH != (__force gfp_t) ALLOC_MIN_RESERVE); BUILD_BUG_ON(__GFP_KSWAPD_RECLAIM != (__force gfp_t) ALLOC_KSWAPD); + BUILD_BUG_ON(__GFP_TRYLOCK != (__force gfp_t) ALLOC_TRYLOCK); /* * The caller may dip into page reserves a bit more if the caller @@ -4009,7 +4018,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) * set both ALLOC_NON_BLOCK and ALLOC_MIN_RESERVE(__GFP_HIGH). */ alloc_flags |= (__force int) - (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM)); + (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM | __GFP_TRYLOCK)); if (!(gfp_mask & __GFP_DIRECT_RECLAIM)) { /* @@ -4509,7 +4518,8 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, might_alloc(gfp_mask); - if (should_fail_alloc_page(gfp_mask, order)) + if (!(*alloc_flags & ALLOC_TRYLOCK) && + should_fail_alloc_page(gfp_mask, order)) return false; *alloc_flags = gfp_to_alloc_flags_cma(gfp_mask, *alloc_flags); @@ -7023,3 +7033,54 @@ static bool __free_unaccepted(struct page *page) } #endif /* CONFIG_UNACCEPTED_MEMORY */ + +struct page *try_alloc_pages_noprof(int nid, unsigned int order) +{ + gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | + __GFP_NOMEMALLOC | __GFP_TRYLOCK; + unsigned int alloc_flags = ALLOC_TRYLOCK; + struct alloc_context ac = { }; + struct page *page; + + /* + * In RT spin_trylock() may call raw_spin_lock() which is unsafe in NMI. + * If spin_trylock() is called from hard IRQ the current task may be + * waiting for one rt_spin_lock, but rt_spin_trylock() will mark the + * task as the owner of another rt_spin_lock which will confuse PI + * logic, so return immediately if called form hard IRQ or NMI. + * + * Note, irqs_disabled() case is ok. This function can be called + * from raw_spin_lock_irqsave region. + */ + if (IS_ENABLED(CONFIG_PREEMPT_RT) && (in_nmi() || in_hardirq())) + return NULL; + if (!pcp_allowed_order(order)) + return NULL; + +#ifdef CONFIG_UNACCEPTED_MEMORY + if (has_unaccepted_memory() && !list_empty(&zone->unaccepted_pages)) + return NULL; +#endif + + if (nid == NUMA_NO_NODE) + nid = numa_node_id(); + + prepare_alloc_pages(alloc_gfp, order, nid, NULL, &ac, + &alloc_gfp, &alloc_flags); + + /* + * Best effort allocation from percpu free list. + * If it's empty attempt to spin_trylock zone->lock. + * Do not specify __GFP_KSWAPD_RECLAIM to avoid wakeup_kswapd + * that may need to grab a lock. + * Do not specify __GFP_ACCOUNT to avoid local_lock. + * Do not warn either. + */ + page = get_page_from_freelist(alloc_gfp, order, alloc_flags, &ac); + + /* Unlike regular alloc_pages() there is no __alloc_pages_slowpath(). */ + + trace_mm_page_alloc(page, order, alloc_gfp & ~__GFP_TRYLOCK, ac.migratetype); + kmsan_alloc_page(page, order, alloc_gfp); + return page; +} From patchwork Wed Dec 18 03:07:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13912975 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1186E77183 for ; Wed, 18 Dec 2024 03:07:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 851616B0089; Tue, 17 Dec 2024 22:07:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7FE496B008A; Tue, 17 Dec 2024 22:07:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 678AE6B008C; Tue, 17 Dec 2024 22:07:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 478286B0089 for ; Tue, 17 Dec 2024 22:07:38 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B6843160B5C for ; Wed, 18 Dec 2024 03:07:37 +0000 (UTC) X-FDA: 82906593966.04.AE00BF6 Received: from mail-oa1-f50.google.com (mail-oa1-f50.google.com [209.85.160.50]) by imf30.hostedemail.com (Postfix) with ESMTP id 1D5D28000C for ; Wed, 18 Dec 2024 03:06:35 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=lny7k7mK; spf=pass (imf30.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.160.50 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734491221; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tgdU2JbWztEEEhB3TtDMib8w6jV6+vZNMAAIDtQqMFA=; b=Zv6eEX3XnSdmtWj9IxLftTBICPdkOp1d9bUKnXESu+618E6JoQa9wOa5nqjy3wp8ULE5kG sae9KAlKk2E/XQi/XTaMr8RkLn/JUUMsCFyt0gj4GpsRoyqFXWXFw4iOD54smtv6U3PG+O POMu6cHt78xABcdkROXcB5wvpHHCGRs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734491221; a=rsa-sha256; cv=none; b=YAKJ7jT01AZm2Te0SIN9klyZBmxkgvwY+luAvYGZuvkh6Ce5MY1Ke2WLs3xoH/gfijcSIw ztVNH7WmCRFsSwG/Ksn35+WeZoXuewul8PslihNQMrU+w9LWsRlfMtkUT7nB4abHnCGeFm RdBUFEjoXspNwUBsO7LpH30jpnih9xQ= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=lny7k7mK; spf=pass (imf30.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.160.50 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oa1-f50.google.com with SMTP id 586e51a60fabf-29e61fcc3d2so4697138fac.2 for ; Tue, 17 Dec 2024 19:07:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734491255; x=1735096055; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tgdU2JbWztEEEhB3TtDMib8w6jV6+vZNMAAIDtQqMFA=; b=lny7k7mKpAqpQCGbs/s5SacAP4QyR6JhNbHj41ZCIIGm9zaI5xfq0jW8lXSWL2G0J9 JPiZXh40Gsdeu4JnhPQF1xG2NIfNHUJQa0ztg/zWJYioKi//nlyNsAp3W7mR8/XmGuh9 5uwHLb9vjmCeBCZEyAByxbF90Mggt9le+VNZ0Z5vIpLLwOeP0MEByh5vhfh0N2FVcdQV RjQVUvPlH7EaGxnKSwrvB0nvED84j5b/6nEhHN+vgR45nxVgQdUz4wq26cETUirwRILd JBxPOv0763ftopRscTkJFcC921sfmjldPxnPeQwE3lkO1DOfFdiszddcQTEQ7zw/nN83 Bg4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734491255; x=1735096055; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tgdU2JbWztEEEhB3TtDMib8w6jV6+vZNMAAIDtQqMFA=; b=HYsPXwbd8dlmFsxIH1+32I5MDbVl8Pay8p2FfuWYdCxIvxd+XEWyMN5PsP4MgGR4te suDQ8dafsQfDqMiE6kkMenan6RcHS9HgS7bu7ZABAvSjifAaXx+EtQ8q3RCpKIL09HTi f78FooGpvR4oBIO9SQz2CyoW7FwdEXlgxoa5zS5YSZfPQ3QVsMM2IYEdn0OxZYf3pmNy 0ir1FBOyi3WxdhdJ5eY+ABi9ztklqmsnfdh7sx6lLGALrkv4fpNuEcTwJB/DN1f+diCG 381+6AKZV+jNqCBgPo9gRhT112hgByGHYJ3VHDz/VeFwTHWSVfsLXfSs2HhcvwfKwOow XdkA== X-Forwarded-Encrypted: i=1; AJvYcCU9Ed2K5A2rzpLr3ocSpe0uIiWJOyIIEeTB8lnvzU2fEV9qsZQ643b6LIjen01W8IZuW41tjwS8Ug==@kvack.org X-Gm-Message-State: AOJu0YwboLdQyQiycia6ZoSLFsTDBiurRDSgdQoKwX+4OFT4b8sNa8tI dDrlM/LpDWK5nyi7+6CJjJ95cGqOywrQBYS5kHc8tR9vzORsTmi6 X-Gm-Gg: ASbGnctzOUK/D8maLM1uTbhY2yFy5/5avOfXMmFor/HzpOnZN4JVUms3Xz4ejxgYjsj 4VztHvK50+R3JvUWzbNqHMTnKgwMXRlzowG/ycyy8YcDg5oKdNuhlrQOlEcq39eblamLMFgroPT ZFtDokft6I/Bl4TmyGGeH7GD6yDMGLlaU5WkbdRBAeYXKGyhn/fEZTcoKMkl+L+JCMslGRMZxjn qdjadf3HY88uz75V0Ms3QkqNegipyhe/MJC8GlCzDDdQ+hBcqEOubEy+XPT3oU= X-Google-Smtp-Source: AGHT+IFncZAcAnWxKKV09ycgYCqm0lYNyvdm4XJIXodyQo0lsEwlel1FnkF3rC4Igm8LoVETXeQkKQ== X-Received: by 2002:a05:6871:650:b0:288:2906:6882 with SMTP id 586e51a60fabf-2a7b32cc1b1mr802178fac.29.1734491254754; Tue, 17 Dec 2024 19:07:34 -0800 (PST) Received: from localhost ([2a03:2880:12ff:72::]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-2a3d25473acsm3311761fac.13.2024.12.17.19.07.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Dec 2024 19:07:33 -0800 (PST) From: alexei.starovoitov@gmail.com To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v3 2/6] mm, bpf: Introduce free_pages_nolock() Date: Tue, 17 Dec 2024 19:07:15 -0800 Message-ID: <20241218030720.1602449-3-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241218030720.1602449-1-alexei.starovoitov@gmail.com> References: <20241218030720.1602449-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1D5D28000C X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: br899hskx7cuqd5s646e8b38hiw6pp15 X-HE-Tag: 1734491195-624861 X-HE-Meta: U2FsdGVkX1/FrKTqB3JeXmEI34SSOc509myTKMSwUJwWHZ23wFJFVKB/h0pIDWZXzvKRkBo64OKeLPjHjB8fXFzY7G8HWh9dEUJCSUmpJwGAbOb/DHQ0qnHGXJ6zmn/650DGf4yZ6ScPVsXuylUK+IYCDScb4nwlbvZNm8GrILJ7vUL8nJn+hdrf5DMFlHNZ9WIuCpuBFb0I8cig9gx2xvldbY/L3CGMDscVvVErOaRNyln1dE0M8O+8SQUn9VvO6VDUbPZHuN6QzVI7jq9PozSDypx4otuOUTM5OY4rAAgXZe+FDXW7rJYO6O7Yg3QviyIcV15/pABIBNt6Cw0N0POKNGAyfBZx9aN934cv/3LgcTuVYtdBa8HunOvTOGGhEw9H36JknOm+w8z1cBwb+4oblB8oDEUiX8EQG13Z71oJ5QrO1nRSs4rHQX5GPijbQ1WhouD7/EOBaPmGutSPcUBVrQizBA9QBxRBf5BQyfm+wgLDKUfYPrjiIZFWVGAAMw5WmJT+iD+XGZM2OlIb1gxNFzp2BTXUPPuADtnmoG7PXqTzussiXYUEvXVoQDTG1ljR1jcnQySPv/7zwk2kZyHvy8mLq95GGTXxzERNd4Or99BZkK4wMWF+d75t8azRHN/NDyFEpKPdPKFpvaqep1yA5GRYh785NPiG/bZ+GeIDhNNnT1sj3cqPcKByjb9o3dL4OfVUoKXG90f7m5lPTw8aAy6KfwEd+HtbMXD205e6x7vrwIKEDC2lunzyqub6guIgUQUz8Ukrl8eVyIyTtuLUgruJ/1deFUImrUAgUqAgL++3ivhCfVXeNsUiDV0J/0sRLSXWrfihJscXsklx9p0iaWOzlcyh84PuK1BXaUq9DyTm1bsTV/+u3adUyEOY8/0Bq0Sw7ahcZ0btu6DhtEqmi2cQ58qTAYtqL+cPc5KKYqFNn27X/d1AAYizLDkiKzkZG9anhO0GsT5P0bv f/eALmjE QesGFQZ5czWWnfOVGgXFzNzZIbBoSrofcwRP6c2dyz8SxNlMm5RRLKXKqvTskf/07l2AsXJduryiEzSFH4UEy2xOqcW4DaZxmDcJEdSnixqWd8Dv3nLFIo6Txq7p0bYs7tHPVJuRRSqU4bgYeMeRA98Lzhw22b7So5w07pa6uoFg1mABOA70IKFC8kToDL0DFbeE0D/6cTNgfqgGZg24omXiltUE/gKZz6KnHg4RK+XWXaLlDx/H6nPasWlzt9cWCG6xviOA0RpGedyHx5byt2oRzSF0nTERZRIPiYYecLusA2/mCRrsRwZN5U+zxHI/ILQs7u3UO+5iZpTCWQMZ8QIG+/OjIv0fX+UUN0oavWh3X76C0CwnKXwyve2vrIL29DYqvLi8S47dUwgoVttL/qJnM8Xg8xxu5qdKerYl8dVUqVULwO9zX8pR9Aw88CMlrn5SLvjApcpnOfBZH/yydkBdGNJ50XnqDYFnfZIYI2Rl4QijkNIC+D4DtUQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Introduce free_pages_nolock() that can free pages without taking locks. It relies on trylock and can be called from any context. Since spin_trylock() cannot be used in RT from hard IRQ or NMI it uses lockless link list to stash the pages which will be freed by subsequent free_pages() from good context. Signed-off-by: Alexei Starovoitov --- include/linux/gfp.h | 1 + include/linux/mm_types.h | 4 ++ include/linux/mmzone.h | 3 ++ mm/page_alloc.c | 79 ++++++++++++++++++++++++++++++++++++---- 4 files changed, 79 insertions(+), 8 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 65b8df1db26a..ff9060af6295 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -372,6 +372,7 @@ __meminit void *alloc_pages_exact_nid_noprof(int nid, size_t size, gfp_t gfp_mas __get_free_pages((gfp_mask) | GFP_DMA, (order)) extern void __free_pages(struct page *page, unsigned int order); +extern void free_pages_nolock(struct page *page, unsigned int order); extern void free_pages(unsigned long addr, unsigned int order); #define __free_page(page) __free_pages((page), 0) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 7361a8f3ab68..52547b3e5fd8 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -99,6 +99,10 @@ struct page { /* Or, free page */ struct list_head buddy_list; struct list_head pcp_list; + struct { + struct llist_node pcp_llist; + unsigned int order; + }; }; /* See page-flags.h for PAGE_MAPPING_FLAGS */ struct address_space *mapping; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index b36124145a16..1a854e0a9e3b 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -953,6 +953,9 @@ struct zone { /* Primarily protects free_area */ spinlock_t lock; + /* Pages to be freed when next trylock succeeds */ + struct llist_head trylock_free_pages; + /* Write-intensive fields used by compaction and vmstats. */ CACHELINE_PADDING(_pad2_); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d23545057b6e..10918bfc6734 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -88,6 +88,9 @@ typedef int __bitwise fpi_t; */ #define FPI_TO_TAIL ((__force fpi_t)BIT(1)) +/* Free the page without taking locks. Rely on trylock only. */ +#define FPI_TRYLOCK ((__force fpi_t)BIT(2)) + /* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ static DEFINE_MUTEX(pcp_batch_high_lock); #define MIN_PERCPU_PAGELIST_HIGH_FRACTION (8) @@ -1247,13 +1250,44 @@ static void split_large_buddy(struct zone *zone, struct page *page, } } +static void add_page_to_zone_llist(struct zone *zone, struct page *page, + unsigned int order) +{ + /* Remember the order */ + page->order = order; + /* Add the page to the free list */ + llist_add(&page->pcp_llist, &zone->trylock_free_pages); +} + static void free_one_page(struct zone *zone, struct page *page, unsigned long pfn, unsigned int order, fpi_t fpi_flags) { + struct llist_head *llhead; unsigned long flags; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(fpi_flags & FPI_TRYLOCK)) { + add_page_to_zone_llist(zone, page, order); + return; + } + spin_lock_irqsave(&zone->lock, flags); + } + + /* The lock succeeded. Process deferred pages. */ + llhead = &zone->trylock_free_pages; + if (unlikely(!llist_empty(llhead) && !(fpi_flags & FPI_TRYLOCK))) { + struct llist_node *llnode; + struct page *p, *tmp; + + llnode = llist_del_all(llhead); + llist_for_each_entry_safe(p, tmp, llnode, pcp_llist) { + unsigned int p_order = p->order; + + split_large_buddy(zone, p, page_to_pfn(p), p_order, fpi_flags); + __count_vm_events(PGFREE, 1 << p_order); + } + } split_large_buddy(zone, page, pfn, order, fpi_flags); spin_unlock_irqrestore(&zone->lock, flags); @@ -2596,7 +2630,7 @@ static int nr_pcp_high(struct per_cpu_pages *pcp, struct zone *zone, static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, struct page *page, int migratetype, - unsigned int order) + unsigned int order, fpi_t fpi_flags) { int high, batch; int pindex; @@ -2631,6 +2665,14 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, } if (pcp->free_count < (batch << CONFIG_PCP_BATCH_SCALE_MAX)) pcp->free_count += (1 << order); + + if (unlikely(fpi_flags & FPI_TRYLOCK)) { + /* + * Do not attempt to take a zone lock. Let pcp->count get + * over high mark temporarily. + */ + return; + } high = nr_pcp_high(pcp, zone, batch, free_high); if (pcp->count >= high) { free_pcppages_bulk(zone, nr_pcp_free(pcp, batch, high, free_high), @@ -2645,7 +2687,8 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, /* * Free a pcp page */ -void free_unref_page(struct page *page, unsigned int order) +static void __free_unref_page(struct page *page, unsigned int order, + fpi_t fpi_flags) { unsigned long __maybe_unused UP_flags; struct per_cpu_pages *pcp; @@ -2654,7 +2697,7 @@ void free_unref_page(struct page *page, unsigned int order) int migratetype; if (!pcp_allowed_order(order)) { - __free_pages_ok(page, order, FPI_NONE); + __free_pages_ok(page, order, fpi_flags); return; } @@ -2671,24 +2714,33 @@ void free_unref_page(struct page *page, unsigned int order) migratetype = get_pfnblock_migratetype(page, pfn); if (unlikely(migratetype >= MIGRATE_PCPTYPES)) { if (unlikely(is_migrate_isolate(migratetype))) { - free_one_page(page_zone(page), page, pfn, order, FPI_NONE); + free_one_page(page_zone(page), page, pfn, order, fpi_flags); return; } migratetype = MIGRATE_MOVABLE; } zone = page_zone(page); + if (IS_ENABLED(CONFIG_PREEMPT_RT) && (in_nmi() || in_hardirq())) { + add_page_to_zone_llist(zone, page, order); + return; + } pcp_trylock_prepare(UP_flags); pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (pcp) { - free_unref_page_commit(zone, pcp, page, migratetype, order); + free_unref_page_commit(zone, pcp, page, migratetype, order, fpi_flags); pcp_spin_unlock(pcp); } else { - free_one_page(zone, page, pfn, order, FPI_NONE); + free_one_page(zone, page, pfn, order, fpi_flags); } pcp_trylock_finish(UP_flags); } +void free_unref_page(struct page *page, unsigned int order) +{ + __free_unref_page(page, order, FPI_NONE); +} + /* * Free a batch of folios */ @@ -2777,7 +2829,7 @@ void free_unref_folios(struct folio_batch *folios) trace_mm_page_free_batched(&folio->page); free_unref_page_commit(zone, pcp, &folio->page, migratetype, - order); + order, FPI_NONE); } if (pcp) { @@ -4854,6 +4906,17 @@ void __free_pages(struct page *page, unsigned int order) } EXPORT_SYMBOL(__free_pages); +/* + * Can be called while holding raw_spin_lock or from IRQ and NMI, + * but only for pages that came from try_alloc_pages(): + * order <= 3, !folio, etc + */ +void free_pages_nolock(struct page *page, unsigned int order) +{ + if (put_page_testzero(page)) + __free_unref_page(page, order, FPI_TRYLOCK); +} + void free_pages(unsigned long addr, unsigned int order) { if (addr != 0) { From patchwork Wed Dec 18 03:07:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13912976 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C0DEE77183 for ; Wed, 18 Dec 2024 03:07:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 197B36B008C; Tue, 17 Dec 2024 22:07:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1484C6B0092; Tue, 17 Dec 2024 22:07:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F29FF6B0093; Tue, 17 Dec 2024 22:07:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D48E96B008C for ; Tue, 17 Dec 2024 22:07:42 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 92882120AC2 for ; Wed, 18 Dec 2024 03:07:42 +0000 (UTC) X-FDA: 82906594050.05.195CD33 Received: from mail-oo1-f41.google.com (mail-oo1-f41.google.com [209.85.161.41]) by imf06.hostedemail.com (Postfix) with ESMTP id B476118000E for ; Wed, 18 Dec 2024 03:07:18 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=PRbomrfq; spf=pass (imf06.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.161.41 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734491239; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fcHIP1+8Pwyc2xkUuGVXiolOQkisEg3F+R5j1Yt+uzY=; b=PhuRiMMCGaYIhnXmRsWj6/NPmO8CEqc0Ha24Mm16eBUuOwv1VqN883dmy8ThOIVLBHPWXD qsU18MC8Pv4+e5bGel4qS9vsBK5pDvHB55gct10UTdGcloyNLfO91DihNeS7QF1nyxle3F 0E4xR5fAWzKWyW9o+0mKP16/BENUk+k= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734491239; a=rsa-sha256; cv=none; b=VT+PsvHZFwM5oGOppbAaJ30a6LJcWfv+B4RTYVZRUgsTxsx0AeEov+jmAmTvE/JwH/QwJz sr8KJ/LKNfaqRed46Ho1+LzHGAzU9Nfti0H8UA/DkC+lgjYVftWxLF/IyjW+2U8wL19Lzt G/16C9sGsZ7tL6dW020l0iWTPGJYupo= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=PRbomrfq; spf=pass (imf06.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.161.41 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oo1-f41.google.com with SMTP id 006d021491bc7-5f4c111991bso185328eaf.0 for ; Tue, 17 Dec 2024 19:07:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734491260; x=1735096060; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fcHIP1+8Pwyc2xkUuGVXiolOQkisEg3F+R5j1Yt+uzY=; b=PRbomrfqMw9wkMkgVgNx9CIIXX7m2D+Pf6dcJsTIjVMhreYIM19UbrZXixyn1assfb am238bacuqDGJS7uTQzJ6qxmUttk55FZKIMEHWUM8bNH1DkZYXhVTLIBzsathHsOOfwF P4FPTtGxeO+4sHQGx2JfOxR+2CkgO7BFY8af2qZAdmvG1sopSVLn0wXPCrRcPtMLayXx GwsrAl8sCAPI+lNVuuEq7C33KaBFVzk9FCgpnydjfBHvLe2/SzbZriTyOnf2sEt9msaR z88zQbnhzGR4OPYO8RYpCfgwcgUgp8Gee4fDacTBom+V95W2yFJr769Mr6H7sBT9wxC5 rXaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734491260; x=1735096060; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fcHIP1+8Pwyc2xkUuGVXiolOQkisEg3F+R5j1Yt+uzY=; b=J3819hoLrpWqqId3yu8NYQHHhR6baAKBOfhw1gsNYnOZIaD+O20LH734+pkjF2A7wg bgFpm+WoP7rNsYlw7hjFQ3+wm4LDOqVnoRfsAGL3vDFMU35ogSxYZ6CxE2aIR3UICuJh wz8Fc0Ea+KlEKSJYmPz2Q76mX95MD2ctsLEzeVDGHIeBtj9PPMRN2p2lroLxKO2KzSzt Mfo8NiKGvsbaCl/4HeQzT/APE8vkdPM1xWMv9I/BSePvldIay0/USY/o4d/POzPMhE3J Jjk41EohdjN+4UqGa4Bp55MJPUEv3Oo0iYHCnYdLk2KEoNSLxO8rmjppllFGuVqi/C4B lUJA== X-Forwarded-Encrypted: i=1; AJvYcCUUkAWwUDmDx30U5DNIfy5J7FiL0gGKUxMgoEGcFofhTIFLyhqJMZwaoId5S+YEgHwJPnrXSj37lQ==@kvack.org X-Gm-Message-State: AOJu0YzxAel234V9CnQUjwCE6GKmhhtlD4X7GPt4GmzmDeAh4IeyKeJl GuvB2Io9+RY2DxatE1XDjh6CTz1VWTv3RtdegBZ79kI5bjnIA0/D X-Gm-Gg: ASbGncuIQkjzno7xcceoWGXLenKvGJzjkfoUnyx8b6ep46Mx7EIY/p/Yv3qgIToH6oQ AHnXR7KcLHJpBeN+qvmY/KELAlWr+3oloP3SWoj25CoYKJth4wS3SuWh7N9eIk8EJiuwRx3Bq6s 3u66fuhWNrxOKuNuXfB78hAbY8YkX7I+NL84s4dp+5mkNUxcAGKmLjUQ7Kg/kZ/N3gFC86BFbqE nEC7QQ5AnGYV1QhsKcBK+sGf8ESC9bIxXAau7juik9T5PgPUF3q0B7RTsleEQ== X-Google-Smtp-Source: AGHT+IGAOWU8WTtpajWGghl6QstJosNCQ5UcWaUN0/NUcMXEsAsN5z1Ap2Hz23QuLwZvMvbxMXfsRw== X-Received: by 2002:a4a:d74c:0:b0:5f2:b6d5:bf53 with SMTP id 006d021491bc7-5f45c71a68cmr2978615eaf.1.1734491259593; Tue, 17 Dec 2024 19:07:39 -0800 (PST) Received: from localhost ([2a03:2880:12ff:4::]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5f33a98a3f7sm2526797eaf.36.2024.12.17.19.07.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Dec 2024 19:07:38 -0800 (PST) From: alexei.starovoitov@gmail.com To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v3 3/6] locking/local_lock: Introduce local_trylock_irqsave() Date: Tue, 17 Dec 2024 19:07:16 -0800 Message-ID: <20241218030720.1602449-4-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241218030720.1602449-1-alexei.starovoitov@gmail.com> References: <20241218030720.1602449-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: B476118000E X-Stat-Signature: b3gynxygkj5gcdxsaicf97c7j9mah3ck X-Rspam-User: X-HE-Tag: 1734491238-569817 X-HE-Meta: U2FsdGVkX1/0l4WTHdoLBTdZBmbtJ0EWOgDzsFt4OEFCds3+x5+MpLbuTSIpXQqioR4LogZuKUvt+eKepARPTzOKUsCgtpHdVfjaFeeQOFWshxU+OUopFS3amL2bka+6oZntthBFAxKCxuAX2v6q5RNV+io/bTL8flzrzpXhWcZgvhsH2skcrN4ZhbQzffeH8I1JKCngtj1Vr4gnW71S1n4ClXZdMnY3X8iApwGEjK8UgEQfYrQAoK9+BkTOTggMlXG1Dbi/kMIBVQAU7VJiARh3ch1kxyHlMEK8ZKkmYqJUVyflyuBb0CQgPllr3TLmBN4Azd7gdWmpy9VE22wm0yywAbIAP7XYHxb6lnqwwlku3YkLy/O8nhmO2VIa8aNk2MglqpMQulOLgh2fEDapdDTbbu8agACzcQ9eZbybqHF9SbJueEso5a7XHjck+4ETHItmeFVAyf47cQAqIURQdcaGo8LG+ygoDABG+nLl953FvtYOBRSXd9LL8q2txSqX9HY32QUGyh80P11lrwyLUC5qjHGqgV6AeAMfYoyGpvV4IGbH7biFnwWB2Q/sZ2rkhz4/4LISwQ0j89hdj+vUmgUfyDo9ZAwJPIx25Zvc634DojM8CRTzUIxGKJ63jtZ5+D8QSzWluX3B7VvOdDAHRt4WvY02lMeebyvc1QxCm2oZj6pD5oL/v/VnMrSNuJzd1ZZxNY+P76mAsxNphHVUMq6lkkXt233HxyZwXfEaPpFXdNnk+EZg1XK0RkO9SPU1k+daCwAfYTLX4GkqX1Raa0yIRfiq1ky29jGJ54WFtHyokxqbTCkaB8i97Dip2j6gM22i/QiqXGVfXxTq4QIzt9h8SAbB9lDAJ6xUIviQ3IIGQkkx4P1hs5URAwMvjQqJql0ALgG7KCg3Jxi6SWJzMrKkikat9WdEjkFIDY7NswJMyOwkUokSjlt3QiVU5HZHTM8INHMVNwlZdKv3Kay AMUVjnOK zCa5yNnEVqI+8CxsEtEYkhT6n29W4ZPEgB6sjuaQF+d4t51RR94jq3kHwA94agOfzgACNpd8+kdvS8vjuMyte10lbtMCym5v+xi+rCu3aGSt/xAo7C76dyVn8jBwj3NDhNgfGueon+FiTc6IDhjM6K0nSg+dCWuwAhM3KnjclKJJ7zGZ8ButEZta8FRiWOZxgMSPq0wmYd0pE0gNLc+ljJDYofaArrDAZF2/OoU1Aor34epwHB92Cnwf3Pkd0Z38wJWADJ0FvZ0DTgKFNofZyIDzhq8t8xqSB/Tuh9hbmxgjkoYBlaJrcJPjvpsA3kCTMvRfOR62ioKIGRE/Svmj9Jsoo/U49jihg/qKVJ1cceaYzEc/iAwbuUURT0QElESD+j59niJoXsA54W03YARD1aPzDEtj5tWyqYy8uO7lBaCSNPNUCdHp0f2slyudcCFUWOfWtAdNbB/WeR0qIEbaZqFecEg0ew8se8NIjYtN292ze3EyNNdI+JR7OKQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov This is inspired by 'struct local_tryirq_lock' in: https://lore.kernel.org/all/20241112-slub-percpu-caches-v1-5-ddc0bdc27e05@suse.cz/ Similar to local_lock_irqsave() introduce local_trylock_irqsave(). It uses spin_trylock in PREEMPT_RT when not in hard IRQ and not in NMI and instantly fails otherwise. In !RT it uses simple active flag that prevents IRQs or NMIs reentering locked region. Note there is no need to use local_inc for active flag. If IRQ handler grabs the same local_lock after READ_ONCE(lock->active) already completed it has to unlock it before returning. Similar with NMI handler. So there is a strict nesting of scopes. It's a per cpu lock, so multiple cpus do not access it in parallel. Signed-off-by: Alexei Starovoitov --- include/linux/local_lock.h | 9 ++++ include/linux/local_lock_internal.h | 76 ++++++++++++++++++++++++++--- 2 files changed, 78 insertions(+), 7 deletions(-) diff --git a/include/linux/local_lock.h b/include/linux/local_lock.h index 091dc0b6bdfb..84ee560c4f51 100644 --- a/include/linux/local_lock.h +++ b/include/linux/local_lock.h @@ -30,6 +30,15 @@ #define local_lock_irqsave(lock, flags) \ __local_lock_irqsave(lock, flags) +/** + * local_trylock_irqsave - Try to acquire a per CPU local lock, save and disable + * interrupts. Always fails in RT when in_hardirq or NMI. + * @lock: The lock variable + * @flags: Storage for interrupt flags + */ +#define local_trylock_irqsave(lock, flags) \ + __local_trylock_irqsave(lock, flags) + /** * local_unlock - Release a per CPU local lock * @lock: The lock variable diff --git a/include/linux/local_lock_internal.h b/include/linux/local_lock_internal.h index 8dd71fbbb6d2..93672127c73d 100644 --- a/include/linux/local_lock_internal.h +++ b/include/linux/local_lock_internal.h @@ -9,6 +9,7 @@ #ifndef CONFIG_PREEMPT_RT typedef struct { + int active; #ifdef CONFIG_DEBUG_LOCK_ALLOC struct lockdep_map dep_map; struct task_struct *owner; @@ -22,7 +23,7 @@ typedef struct { .wait_type_inner = LD_WAIT_CONFIG, \ .lock_type = LD_LOCK_PERCPU, \ }, \ - .owner = NULL, + .owner = NULL, .active = 0 static inline void local_lock_acquire(local_lock_t *l) { @@ -31,6 +32,13 @@ static inline void local_lock_acquire(local_lock_t *l) l->owner = current; } +static inline void local_trylock_acquire(local_lock_t *l) +{ + lock_map_acquire_try(&l->dep_map); + DEBUG_LOCKS_WARN_ON(l->owner); + l->owner = current; +} + static inline void local_lock_release(local_lock_t *l) { DEBUG_LOCKS_WARN_ON(l->owner != current); @@ -45,6 +53,7 @@ static inline void local_lock_debug_init(local_lock_t *l) #else /* CONFIG_DEBUG_LOCK_ALLOC */ # define LOCAL_LOCK_DEBUG_INIT(lockname) static inline void local_lock_acquire(local_lock_t *l) { } +static inline void local_trylock_acquire(local_lock_t *l) { } static inline void local_lock_release(local_lock_t *l) { } static inline void local_lock_debug_init(local_lock_t *l) { } #endif /* !CONFIG_DEBUG_LOCK_ALLOC */ @@ -60,6 +69,7 @@ do { \ 0, LD_WAIT_CONFIG, LD_WAIT_INV, \ LD_LOCK_PERCPU); \ local_lock_debug_init(lock); \ + (lock)->active = 0; \ } while (0) #define __spinlock_nested_bh_init(lock) \ @@ -75,37 +85,73 @@ do { \ #define __local_lock(lock) \ do { \ + local_lock_t *l; \ preempt_disable(); \ - local_lock_acquire(this_cpu_ptr(lock)); \ + l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 0); \ + WRITE_ONCE(l->active, 1); \ + local_lock_acquire(l); \ } while (0) #define __local_lock_irq(lock) \ do { \ + local_lock_t *l; \ local_irq_disable(); \ - local_lock_acquire(this_cpu_ptr(lock)); \ + l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 0); \ + WRITE_ONCE(l->active, 1); \ + local_lock_acquire(l); \ } while (0) #define __local_lock_irqsave(lock, flags) \ do { \ + local_lock_t *l; \ local_irq_save(flags); \ - local_lock_acquire(this_cpu_ptr(lock)); \ + l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 0); \ + WRITE_ONCE(l->active, 1); \ + local_lock_acquire(l); \ } while (0) +#define __local_trylock_irqsave(lock, flags) \ + ({ \ + local_lock_t *l; \ + local_irq_save(flags); \ + l = this_cpu_ptr(lock); \ + if (READ_ONCE(l->active) == 1) { \ + local_irq_restore(flags); \ + l = NULL; \ + } else { \ + WRITE_ONCE(l->active, 1); \ + local_trylock_acquire(l); \ + } \ + !!l; \ + }) + #define __local_unlock(lock) \ do { \ - local_lock_release(this_cpu_ptr(lock)); \ + local_lock_t *l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 1); \ + WRITE_ONCE(l->active, 0); \ + local_lock_release(l); \ preempt_enable(); \ } while (0) #define __local_unlock_irq(lock) \ do { \ - local_lock_release(this_cpu_ptr(lock)); \ + local_lock_t *l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 1); \ + WRITE_ONCE(l->active, 0); \ + local_lock_release(l); \ local_irq_enable(); \ } while (0) #define __local_unlock_irqrestore(lock, flags) \ do { \ - local_lock_release(this_cpu_ptr(lock)); \ + local_lock_t *l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 1); \ + WRITE_ONCE(l->active, 0); \ + local_lock_release(l); \ local_irq_restore(flags); \ } while (0) @@ -148,6 +194,22 @@ typedef spinlock_t local_lock_t; __local_lock(lock); \ } while (0) +#define __local_trylock_irqsave(lock, flags) \ + ({ \ + __label__ out; \ + int ret = 0; \ + typecheck(unsigned long, flags); \ + flags = 0; \ + if (in_nmi() || in_hardirq()) \ + goto out; \ + migrate_disable(); \ + ret = spin_trylock(this_cpu_ptr((lock))); \ + if (!ret) \ + migrate_enable(); \ + out: \ + ret; \ + }) + #define __local_unlock(__lock) \ do { \ spin_unlock(this_cpu_ptr((__lock))); \ From patchwork Wed Dec 18 03:07:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13912977 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C8E0E77187 for ; Wed, 18 Dec 2024 03:07:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 345996B0093; Tue, 17 Dec 2024 22:07:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F5976B0095; Tue, 17 Dec 2024 22:07:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 16F546B0096; Tue, 17 Dec 2024 22:07:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id EBB6F6B0093 for ; Tue, 17 Dec 2024 22:07:47 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6C4DD42E24 for ; Wed, 18 Dec 2024 03:07:47 +0000 (UTC) X-FDA: 82906593126.09.8A1BA7E Received: from mail-ot1-f45.google.com (mail-ot1-f45.google.com [209.85.210.45]) by imf17.hostedemail.com (Postfix) with ESMTP id C5D0740004 for ; Wed, 18 Dec 2024 03:07:21 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=lUHKGoGY; spf=pass (imf17.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.45 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734491236; a=rsa-sha256; cv=none; b=txTLCBa/lsOAgA0mrjL+iBNQ0rBO00qBuMYCD37f9g3vS5Qt9RvcUMT2wXOXnhr4D9X4XX db+CtL9pyTC0tB4mY+C6TmTNEB/EeJ4C13lYMcbrFD0FV/lbTc6XOS5INCpbCsquZ6RgWS nhNg0tMnyP1LO9PIJsjld/AG/KwAO9w= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=lUHKGoGY; spf=pass (imf17.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.45 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734491236; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/7W5s6nR1eKMiZGmAkj0lhGxTA+TFwBoEtTYrZWxIj4=; b=8ajmAgA4mEMuZVoBuDVEuDKIGGHqbhRgLIlVog0WIAjTkuNfpJkpVEjpq0/wmaxNDQmZbI BKnpyqxc2neQYNBXKsrADwEFJ1a0j4al9Cfi0dclPOy2BWn56UzduupCPHV+gs0dPtRl83 sLsY+cwE2S6h/entoWqzmVq67wIybIc= Received: by mail-ot1-f45.google.com with SMTP id 46e09a7af769-71e2764aa46so3714448a34.2 for ; Tue, 17 Dec 2024 19:07:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734491264; x=1735096064; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/7W5s6nR1eKMiZGmAkj0lhGxTA+TFwBoEtTYrZWxIj4=; b=lUHKGoGYphXDqiOfXdoPh5vg46kqg5eUrkKZ5wl9mX/D8ZYM2lPkuQuQgqpF2goR67 69WTxx4Nv//TFBmXuFzTKehfiiAqgmCwOdQTQl9hiVx/qWKjKoCEopjxI1KyuHvsfMrl Zi/25m9P6cjB9dcsagTtga9d+adHPH9zzgyqchWMdXFtAkP6Mvs+eZZ0DrN1YFXVUqlO LrnbqNPs/WI5hkmmOyJ/39yCYTGDH5IYqZf/+IPooCLq7yTcT+Z3HjSY/ulajQ0TG3b/ G8efpG8Qgg4dh5mYsMgP5LLPflsdXdahDbo8I6up3ed5L2MLRk1lkpmiBVyjl6c5uHRW xevw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734491264; x=1735096064; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/7W5s6nR1eKMiZGmAkj0lhGxTA+TFwBoEtTYrZWxIj4=; b=sQnSgI6L97ldBz7ML/2v1OEzQUu72d1e5pjnL/XBqQbQlbnkTSlKXRhpT0HhrVs97r f0LGwHHsSTwqR6XsM7gljKUpUWtkFLWoiZhcS1ZgHuqUsRXSaLplKrCg0R+xSe9QC6ro hmEWl26EBcVnr/XoWn6Nznt5kWXstR2CjwAXyYJI+paoEeF0xuOyi/JDwcHMwUefqXG3 11shij+9hRbH100X9S3ZBuitF5J8KJTLLZ0ajFRTCE1XcpxKutzvgnYEJDfUOgeRvn0o BXPxbJQZED1F7acawqp4o7VoZ5qW4u5XwpKtfQp7ozz7/yQ04eKEXC+ar2U1uO9d5zsO uaTA== X-Forwarded-Encrypted: i=1; AJvYcCUv2HASiZK3Ip9D8bZO/nVgpUL96if0bCTmOsw9WMLMWudHDbbfausjME8KSQzq691u5caJ6q/yVw==@kvack.org X-Gm-Message-State: AOJu0Yx9rYci6/63h1IYzCb+iDUzZPsauBTT555jL3NZ9lvAuHhg5BAu nMaRW7FuLLfi+hLYxThyALZNhOyovNe8h4iDWX9nmpI+M5i778mh X-Gm-Gg: ASbGncvOe53ijRnfF6km3h7mquOH26O+IM7g5m4jQpVUJjInQQ0eoPlR10wsi1nvJj3 POOK4+BX/xNzTp0F0s2uGG1aWTMMOP+MScLa+SJ4zn8qWHsSrJZYC5OCSxym7MZpL8JzlXbCEgO JiszNcdbG1jsKqbCuLJgveSQ7EM+oeHzOeujlV/DMDJVOGl2NrDIczFjGctACjZX6LBGB63U1oa o2H+CrVEwta4+9VwXtBuckCMu7JxQ9FVOhkwzEcSdgy/a7vrDYfM08RuPnWUw== X-Google-Smtp-Source: AGHT+IH2tfKmT4WhT188FuyTuGvwz4iinrO9XQia8eG+MuqgHEUEQ+qNAXFpX68m9XBO2n0bhRDyxw== X-Received: by 2002:a05:6830:7308:b0:71e:904:6aed with SMTP id 46e09a7af769-71fb757a1afmr792576a34.10.1734491264502; Tue, 17 Dec 2024 19:07:44 -0800 (PST) Received: from localhost ([2a03:2880:12ff:2::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-71e4835630esm2481271a34.27.2024.12.17.19.07.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Dec 2024 19:07:43 -0800 (PST) From: alexei.starovoitov@gmail.com To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v3 4/6] memcg: Use trylock to access memcg stock_lock. Date: Tue, 17 Dec 2024 19:07:17 -0800 Message-ID: <20241218030720.1602449-5-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241218030720.1602449-1-alexei.starovoitov@gmail.com> References: <20241218030720.1602449-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: C5D0740004 X-Stat-Signature: og6kb5p4umn1m9z66stxddbgwihf9ccb X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1734491241-931576 X-HE-Meta: U2FsdGVkX1+FG3boiuMhi0yhiNTScS566hRHXOdx6MAqGIAUykQe+P/HUz6mTXyZFXBCQha/sufvfZM0xBFTiu0Jf8fu11IjilRuWiBQjqCtbOgdFBDVDaReAo0ibbDTzicBHBKO4PdCgEhxO0/mvn1LKwRQLNW3vGLxPfcB9wqFcisyMiQKjjbE2j69qVskwHYcO29Qp257zrHqZjJfdTSrkXT8uJS3NMD0vJ9v1DY4aFQqyGD306wSbNbhGn8RiGSoNdTb+LSTyQdvMEYXPu65Dt8lfuoD26IzBhZ08PDVsLKNURdM+gFnmht6ydWs2Ug9OA/ADQ6wcCKEjmXdkHhzHMuD3SLi+wEvPAWZ12VCGKBMLjfrRisVVEPfRTqU/gRTqUrmDPRpXu739lwOrgpT3UCIeYXrZ2xK46rwiJmZvQEewu3Is4bz5yf45GRJLy0t/fZStMgmm6JlNzk8p2i6K3staYm/gycrUGmrDI8/4xMzfNeBSvi6Tcssc5Ty1kzljQM+oBz3A2jNTxrmNxqgx8VqwoNJqk6kuPntJXNdqHnBaxB5Gxp2Rs57bf0XiIZppJ4HPFEMy0Y++lNdGO7V/gcpzk3ths5zwuEmHjXOLs+aSNCbVCDAR0aV148of5QTMPpoiN7WLd0BzXdyMz6qG1RY7h8lEC0EncCImIA6UDeruC6uClMm5JP+FnMuDqxON0luuZM0/KU8eam+2EN+peHnG3V4PH9kLHlUKBrvtLrptmXV1kzGpPFdDj/s5zCfm6q47OqebtSirXzrm1xOAT7CqGF5EtEdS+aWkar37n7x8ihlORT1ZN4T4SeUyE5NxaU4pKcdp7wjg/J56Qr5DFDTOC0TXh0p3BFfeGgbHwG4XMO2+pp+/t+ABwC9jogysLsxhttG31S8fEG2SjkGJAL54jOmQX6yXiHF3K2dIpQ2dwVJulVoA/pFKZRnI7lRXye//e+QwRg7Fgg 4PklYmBg VDTEKUdxj+ZS8qDc8KCHx5AVSYI1RjE4g/bNuwndvxVmmjMg19emr/Ac3+WFy6hocidDzRPqa+aZyqTFx3lTF0v/J2khNCrshsF5wUKg6Nhys0+b5wWsB5LeJzqA6eLTkaxAdAYC6dGYMbOgZdegPPVKsrmUZmuSzowqW7+UvT6k5Wy99CPIDS4s1LPdd34mS6yYZ6xnO0URdS5JATMr3NN+QaHBriozrlkom7Qvq3qNUOGTk8m44mdwY0Uz1M3VXIEKmkJnTIXP37zI357az4n/IgUwB539BD83KqeRQlmpkfx7Xu7JWct74DsCs2J/9QZU4A8BjMFiBtjKdp/igz15U8tOn/q2FC4szuHRD9+ecoebbJ9bIdSCBkevrTjeIwgNjZYmujsVdBLw1it9ZxmFJ6F7BtZx8KhPEHQhYfF/TRd1E0VhzMwc2oez+FFtzzwowsS6A0x2094hJvMo0s6yPsQS94RlNIhQGAB862b0WvCet7xVuOOjcpg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.010465, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Teach memcg to operate under trylock conditions when spinning locks cannot be used. The end result is __memcg_kmem_charge_page() and __memcg_kmem_uncharge_page() are safe to use from any context in RT and !RT. In !RT the NMI handler may fail to trylock stock_lock. In RT hard IRQ and NMI handlers will not attempt to trylock. Signed-off-by: Alexei Starovoitov --- mm/memcontrol.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7b3503d12aaf..f168d223375f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1756,7 +1756,8 @@ static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, * * returns true if successful, false otherwise. */ -static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) +static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages, + gfp_t gfp_mask) { struct memcg_stock_pcp *stock; unsigned int stock_pages; @@ -1766,7 +1767,11 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) if (nr_pages > MEMCG_CHARGE_BATCH) return ret; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!local_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + if (gfp_mask & __GFP_TRYLOCK) + return ret; + local_lock_irqsave(&memcg_stock.stock_lock, flags); + } stock = this_cpu_ptr(&memcg_stock); stock_pages = READ_ONCE(stock->nr_pages); @@ -1851,7 +1856,14 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) { unsigned long flags; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!local_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + /* + * In case of unlikely failure to lock percpu stock_lock + * uncharge memcg directly. + */ + mem_cgroup_cancel_charge(memcg, nr_pages); + return; + } __refill_stock(memcg, nr_pages); local_unlock_irqrestore(&memcg_stock.stock_lock, flags); } @@ -2196,7 +2208,7 @@ int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, unsigned long pflags; retry: - if (consume_stock(memcg, nr_pages)) + if (consume_stock(memcg, nr_pages, gfp_mask)) return 0; if (!do_memsw_account() || From patchwork Wed Dec 18 03:07:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13912978 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 332C5E77187 for ; Wed, 18 Dec 2024 03:07:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BAA946B0096; Tue, 17 Dec 2024 22:07:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B59B76B0098; Tue, 17 Dec 2024 22:07:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9D3706B0099; Tue, 17 Dec 2024 22:07:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 80BC26B0096 for ; Tue, 17 Dec 2024 22:07:52 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 3A64780A80 for ; Wed, 18 Dec 2024 03:07:52 +0000 (UTC) X-FDA: 82906594806.02.8BE5438 Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177]) by imf09.hostedemail.com (Postfix) with ESMTP id 016E7140004 for ; Wed, 18 Dec 2024 03:07:28 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Jv5mlybk; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf09.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.167.177 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734491256; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=we3p1MJO0mH53GLx5S6kuhiFdcHuR7bEtawg6CO6LzQ=; b=7OQ55ye1lkYn2ygr4zkKuuHI1RVsjttnzo0cjw8vh/a+exGr0akz1vjecpcDZNtWqUfW5T h4PTV2atoUHc2pMC9nnRCRgLdz4aU+POo5pyr6DZ/4eY5RiGoEftMC+s9fCxjiAMseEX7R NPnkoevo2VJ0ZW/E3ycwULpsQZTTt3U= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734491256; a=rsa-sha256; cv=none; b=t3FhMZs/0u2iybiomXLPLxvq7sxq4YHQAiPVa4XHh2rPqe0c1tpsziTawtCzn1PTsRVDRq +DaOvxHR5WJRb9G4sBR63qUgFabwU3p+gjqxs+8NZVqAEZVC7aOTrDbOztHUfcOc5x0EkR CvmJoPUxmS9YZKkQtcSADHs4GUE32N4= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Jv5mlybk; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf09.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.167.177 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com Received: by mail-oi1-f177.google.com with SMTP id 5614622812f47-3eb7f3b1342so2557493b6e.1 for ; Tue, 17 Dec 2024 19:07:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734491269; x=1735096069; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=we3p1MJO0mH53GLx5S6kuhiFdcHuR7bEtawg6CO6LzQ=; b=Jv5mlybkGrGUzQRPF7VwJBFNpE0nInqJAkqhVakQK81qt10OfOtoBl2Nc1m8T1CyJz WIkaMZzcsLDoQpGU5Vlq7JJFuhqLt/ts3MOJI/QWyZI5d86oWZGlEhRJ6urtOClYdGOD D+Smo6pPvch/zGiulEeLullsObUC2jztStCJ1qnDvuS4+nJNNfGsa4LRwmeB5spt129t TsFqPnwgTlL7ACqG3MdKTDxJNLXjwi30uROzricY5EHRzv52VyKN3dUpDQvp848h7DK4 7nOXytzVgd07oxJnN5xrMeg25gdFrPKyrR9ZUGaik1R2h1nxBC6eaWXw+rz1ZwTpBqA0 vcsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734491269; x=1735096069; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=we3p1MJO0mH53GLx5S6kuhiFdcHuR7bEtawg6CO6LzQ=; b=EMB7cPVpzPJ3cYXaiWqagAhWurB29OpJY/geGYbf7vSFPmMUWj0/xXbkpQEjNChmQM XHS6UitbpNzhdfgyL4vNSWUu5t9RhLFiR/ncX1YooxeqaoXeo770BF4J1/AMCX7o+xPu ugit0hs4NuyrX0PA1wJUHcUZLhySjUCUbWEADnBnOlm3fMK8onTYb7bmztPP/g14QmYl Bg2bJBmOV6wskOGaqlIeVqo5/jNAAZkKsmLdnDHUcZ60rBmVzql+jwqawLjToRnSulK2 pXrI8yIPPcbu1WV9c9xbpujeF4fHHimJeewcR3suwaQHkzx4Rm96epTa/BXn43Zf1UG2 G2CA== X-Forwarded-Encrypted: i=1; AJvYcCVRDXl2nwJmMYSp5+y4F5q9fA7SXv0YPp47keE9tNGklqS/btwg0smM9k12y131k0Dn0xc1sbedSg==@kvack.org X-Gm-Message-State: AOJu0Yx9qKNrLJZW8m5UyOEkt8PBSg3VjgARN88CWm0eFCaBTJlPoBcg bYGDh4hPESe1gRrrMZxszvUI+Qr0vv6GnXllQE6mE0UbKRra/mdH X-Gm-Gg: ASbGncs+lkQuvT0uTcuuCUNg9OIg3e1WylPPxcnGav2MVXjEWG0KwHW9V4/H7qThKPV U+dArNs7wrGrb2NaDhwq6rPx8ID6lRaFHLLaPrNzAMo6UwndIyavdumPxfBHv/TJ4veVhGhtina PH30kH5iBVjYMkABYwpE8omdVbk8OADg1Gog1x3mDuZm89qUe0YYOOTpSvqr1pzYRVboYQQJAb2 ZNHcapAq5Q6KJbcPpqIweyJmUYbKgPw1HolFx4JO6MrgbWbenRcP53zbbPRHg== X-Google-Smtp-Source: AGHT+IGL10nUsHo3mK7ebRhA2Sibr6OXQ514T2ZZGOY0T/DlM7tZqSomQ9DnyLhRUkqKMnmcfJnnaQ== X-Received: by 2002:a05:6808:bc4:b0:3eb:5d3a:5b1e with SMTP id 5614622812f47-3eccbf091c1mr839843b6e.3.1734491269205; Tue, 17 Dec 2024 19:07:49 -0800 (PST) Received: from localhost ([2a03:2880:12ff:3::]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ebb479702asm2651497b6e.13.2024.12.17.19.07.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Dec 2024 19:07:48 -0800 (PST) From: alexei.starovoitov@gmail.com To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v3 5/6] mm, bpf: Use memcg in try_alloc_pages(). Date: Tue, 17 Dec 2024 19:07:18 -0800 Message-ID: <20241218030720.1602449-6-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241218030720.1602449-1-alexei.starovoitov@gmail.com> References: <20241218030720.1602449-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Stat-Signature: y1cpjugfizm4oe94z5d8i8j47jkp35nx X-Rspamd-Queue-Id: 016E7140004 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1734491248-874318 X-HE-Meta: U2FsdGVkX1+QxCoS/28hNkncxHW+yjE8I5+vqo8wSsCfuktlgRoqc1SXSAM0oUWY+Rw7L6vXToObD2FaLL14QDtV93aVUDEv6t+RZSdsujtuZVgYeSdWu4hUynuaL1b7cj3svI+n3SoX/Prx8gfMUv+p/PTPyceIqkThAvXH5DIY/gqbD60o3u7GDpUv4rMRkJ/H2d0xshl8aI8sYLSsmtNq+3C9ENcze5OKvq/iZqB8lTKYV9muuD833/PziWeKlp/lFUhX4pHw4eVnQsnpnT8Fu45NzOi62LQvPGSJK3EOcpTSGslSl3otiPqDrePj6aNzLrFpi3aIo6ImJxndORzC5kXcq/W79oBSt24v4IOEd2vFXum62nduHxIjVamUeIfTGyQ0ZzkwzMPIFgos0lv7dqlIvrzbWwUM4ZHR49Aog9qVWb2k/yoIYgAj6YQQNseT8zIwA+c5Q/+EkWBXHdk5f3unf29nXnq4gKYbXp+cGGJMFRBV5DBVgzglV/AXG20LAycyiFPsessxSHT5dCo67cTGRcQWBE2uzDWGoWU2ocTQ8vkGFm3PWyFOfeBS3/gqH9j18r1hEx+7iKELham/ombWMwkbIzMWvMReqREbIfksIqBlwS+EZzhIrglwbfB1oYe2gBeLHDdxGbCW4e4kmPU/f0U6teRMFIcHaUoZcueP4BaAM2yEdEnkY3D693sHHKqJgF2rLLNSxzCDN18B/JYB7g7SvIhQ2mD+etTodLewLv6S2XTryHilweXBR14PMr0XDGp5rrpLDpKxNQrH1qLn9JRk95uKGQINZKzy2osdPUTOaUtqzMwhsUZ+qHfVD55pCUQkMfJhJv7Vm2EfxZX4cB3JxkCBIawC5RtbGnm34Y1MzaAqWho2U09P568bMt6dFgfp8583T02LodpcDaMWpoB1+yVX94/XwdcfeGb6JPCH3Qxs9MyGTjq+z/f77pN7GHah0vQRMgC XDaKZ58n 4H4OClK+dRLekN9XWqmh81lIf2h1TRlS9Y8Snhvb/ZWFvfyyMfrZWzH53QEIkIN9JBJaauIMTcskies1f1bEGyIvagqg8OLfivhZRAYtgUoD/lRStezvjU0/QPMQq+rcfPdAAaa9cJivQA8SgdMwts5XbTgFHQ7Zb7oZrPopdn26t4wNFyokAbcYe/McLAXRY3ZWAsaz8w/4cysUxY3rifkAyl6PoXDeCkpAXlHH/rCnUmsvQzjZvYy+/3aoLshYBWOn4Bcgp40ezBxwt6TDpYJxjZjOsXSRUBBQBn4Vre8733yTs8viXw49CvN6l55C4YqJOimKZw0m3t4OgUIJZmuQUd0LMHikyvCPE1hnN8sB5lYp00cbb5+pkEpySQXsDIzjta4esz33C6Q4QMuqtG3A/Jd4NI/KVg35g2t1tbu/OagZ4Jw37gSBXQdndB1YhuvEW84kZdOew0sfjIqva/+cCqQNE6IhH2png3+4oX9pb5R51enMJQKubPg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.020475, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Unconditionally use __GFP_ACCOUNT in try_alloc_pages(). The caller is responsible to setup memcg correctly. All BPF memory accounting is memcg based. Signed-off-by: Alexei Starovoitov --- mm/page_alloc.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 10918bfc6734..5d0e56fbb65b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7100,7 +7100,7 @@ static bool __free_unaccepted(struct page *page) struct page *try_alloc_pages_noprof(int nid, unsigned int order) { gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | - __GFP_NOMEMALLOC | __GFP_TRYLOCK; + __GFP_NOMEMALLOC | __GFP_TRYLOCK | __GFP_ACCOUNT; unsigned int alloc_flags = ALLOC_TRYLOCK; struct alloc_context ac = { }; struct page *page; @@ -7136,13 +7136,17 @@ struct page *try_alloc_pages_noprof(int nid, unsigned int order) * If it's empty attempt to spin_trylock zone->lock. * Do not specify __GFP_KSWAPD_RECLAIM to avoid wakeup_kswapd * that may need to grab a lock. - * Do not specify __GFP_ACCOUNT to avoid local_lock. * Do not warn either. */ page = get_page_from_freelist(alloc_gfp, order, alloc_flags, &ac); /* Unlike regular alloc_pages() there is no __alloc_pages_slowpath(). */ + if (memcg_kmem_online() && page && + unlikely(__memcg_kmem_charge_page(page, alloc_gfp, order) != 0)) { + free_pages_nolock(page, order); + page = NULL; + } trace_mm_page_alloc(page, order, alloc_gfp & ~__GFP_TRYLOCK, ac.migratetype); kmsan_alloc_page(page, order, alloc_gfp); return page; From patchwork Wed Dec 18 03:07:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13912979 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 705D8E77187 for ; Wed, 18 Dec 2024 03:07:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F2D626B0099; Tue, 17 Dec 2024 22:07:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EDCA76B009A; Tue, 17 Dec 2024 22:07:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D567B6B009B; Tue, 17 Dec 2024 22:07:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B7AE66B0099 for ; Tue, 17 Dec 2024 22:07:56 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 40F59A0B2E for ; Wed, 18 Dec 2024 03:07:56 +0000 (UTC) X-FDA: 82906594428.12.14F06CF Received: from mail-ot1-f53.google.com (mail-ot1-f53.google.com [209.85.210.53]) by imf10.hostedemail.com (Postfix) with ESMTP id B1B1CC0010 for ; Wed, 18 Dec 2024 03:07:40 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=k7Ne29aV; spf=pass (imf10.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.53 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734491259; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=e77ZoQKvlnjPoOZm7mn9NDRKoW7hQ+ydIa8Wb5hZHHY=; b=XB00fmPUelBNUkVaCdgoAbqrB3h4K8dTvigKR8v1VeoStqY3xSAObKhK42UGhKbcIMi13c 9Jmb+npC0p/9rnjaYe2twUn1Lx3R4aunq3jDOmkeE9N3s+PHMwISlIH7o+1r9W9gvJDGfv XzhjAHSlFuse0Ox4GML6axAE3yxdt0c= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734491259; a=rsa-sha256; cv=none; b=oNjFZ3NUW7v4Yt3SfZxJsFmfiqsJ0CqeU+XodQNDEPe4iGLSgkojMZS15vkOJv1poR3MhR K7LZctUGuKYk9agLeUmCBBB8yK0FYQHJ4noljY9674wJBdSzBLIIcWqx+0IDl7adL9Tgod Fc9ZMTj/sH9WLNnjB+SpiGYblUur1Ow= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=k7Ne29aV; spf=pass (imf10.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.53 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ot1-f53.google.com with SMTP id 46e09a7af769-71e3005916aso1066771a34.2 for ; Tue, 17 Dec 2024 19:07:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734491273; x=1735096073; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=e77ZoQKvlnjPoOZm7mn9NDRKoW7hQ+ydIa8Wb5hZHHY=; b=k7Ne29aVi/8yNEdBb18uW8IEMcpvm3b4+GLXqluVveb+nrg1MZgsiWKKiMQ1LI2yM4 8ErEWMPCwpMCws3CQPN2ZL5SbmgxRjNazjxHgNGdgbwake+L1NJ9BhNyQCC4YjsSdPhn tPjGqzO1SpL3SokoVfTeIY/n8sfWhjKSupOOdnHw7UNYExyUjkzyGhMOYhl1b8sSOEPs OIzYx6WbG8FMnoE3ImFpzkciXIW3kiax1YxpEbjTorbmNRnrDrQbUajbi3S+p0ge9BOU koMvAZb4m2hVH4/LuZzPnNoKnENNjUyMFtsxj7E+6r4OpHqGaPsdOYwLOR1Ci48hGgET 63rA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734491273; x=1735096073; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=e77ZoQKvlnjPoOZm7mn9NDRKoW7hQ+ydIa8Wb5hZHHY=; b=KVTRjOqwA/7r2zxw+iGvnkVLUYcwWqgFTTgmd5pDeldL4vUuhTjRANndlYPOsRlH1Q n9SqugfJlqb3SHMdeHuljVqM90YUTga7mPSjjBjLkoo/5Yo7IyElOE+sennSokT+ulQ4 OhE9bzlPGeYltP3ab1kfeJDg0bqcnwpC4a316GfBrABN4ukICplCAmve1tUxNupd4Ibc 43rJw4f3mtjW0GGg9Zdj7cF3UXyH0UxpCfJzcxBlUhL4teV6DdWy/VUVz9X/kQaC6pNC rO+iS/7/9aZ3p7WL9+SAJMHt5aLPx82R7IaWc2GGjvAtjp2RYU2NOeJ3dSUeia44MDWJ mPwg== X-Forwarded-Encrypted: i=1; AJvYcCU4dEasZ4VhQ1R9+mbWb2TUfNfDBCt7Gy6U10rnBOttOhOPj0Mltdf2ngRm7e17sY3swjIbli2OQA==@kvack.org X-Gm-Message-State: AOJu0YxFWiyXUaY/BSDQO6n4r3B/lDst0UnqIn7/Fm/IjOXAqljsl0Jl DwYx7KDMlCu29G3SDfj1XIshEu5r9L0g4u7qJI4lqahJoGq3KgxK X-Gm-Gg: ASbGnct+eP4I7YTiplv/0PS4jR8rkKE8702T//Z80MW+NLwSfPHcy+/OhuWLutVPvDN rA3LFDhswOXi+E/M3422WTXrAu+UVbEobL1EOClszEC0cilzWUqierAnRVdA5mXkxUveufsNWxU GVfcxyF1oO2gD2TD2ta0jHwTFVVgwN33tR+a/v0Kz09yHfmVFm0XQnHdjLFJtM0tUAzpAFqI0yO dGiP04X21zaWG90vQI2/4TtYzTZNcepRSUircNYo5ooWFJ2IV1GNDRD58FsyA== X-Google-Smtp-Source: AGHT+IHi/o8gjZtYzcre/yxStX7zDbYufL6oPcWXItTfKnkU83y7CXItnrrjbp8U2xtfjDPf2FnqzQ== X-Received: by 2002:a05:6870:ff06:b0:29e:684d:2739 with SMTP id 586e51a60fabf-2a7b35c8056mr580007fac.32.1734491273379; Tue, 17 Dec 2024 19:07:53 -0800 (PST) Received: from localhost ([2a03:2880:12ff:7::]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-2a3d263b839sm3295077fac.14.2024.12.17.19.07.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Dec 2024 19:07:52 -0800 (PST) From: alexei.starovoitov@gmail.com To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v3 6/6] bpf: Use try_alloc_pages() to allocate pages for bpf needs. Date: Tue, 17 Dec 2024 19:07:19 -0800 Message-ID: <20241218030720.1602449-7-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241218030720.1602449-1-alexei.starovoitov@gmail.com> References: <20241218030720.1602449-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: B1B1CC0010 X-Stat-Signature: 1te77omx4owutyn43fst8jf39uhr59ft X-Rspam-User: X-HE-Tag: 1734491260-459435 X-HE-Meta: U2FsdGVkX19yM3dIqZCTsGr8EOIG3Mq47Mz2agsAMEfZiK91jBMlFqq8qC8lYgfFuRP7KICWDnYOn6h2+Ovbrex7gqJmLzekeZKBoLvpNWw1l97hZLAYKbhOTlS7Jzr5ExYqoDANSS2PLVWDVHq9Kj3oOB9grhBJ128aPqqz9Z+GweCZGM4exGwPAhiE0dFrqFqvC1wCLpF0ZE3XDjmzFvZdlhEVNWQPGUB/0iJg69gLpBibqPHFrmNG/kelDn0LUSVuNLPNhHtF+gs/6BhvFichi5UP3levqOyYW5tKUAeaDQWO/6n5wLm7qVg1FOPv2MJVd4pxou4qY5mYMcJXAP6pneDz4Pti3MynPdp0Q7jv/ASk4kR+WCfoTsNnA1S/vXRdOixnnFG9ZwqV00yz3/c4gmkMgfyqsvYyfWZfOYBxAd+EuwA0RfzcvdhwBLwOszqDxp6qM4Sjbsv9TJJ7YktjpJDHyYRdNWJKsHqBybYKmmUsdWsEw9FgNOQGHhhYQKjWEYkJSdbXYXcDOYtThK7mM/QFDux3dwF6iKemcmqn41JdDP+tthxRXHZSqWJ0sjRL16F22nu5yq2a6wZ+sYk+ocR0ZyYKPnar2+CYCyhqX6QR/eViz2wZjUOeXDpWwKxNcnH+sBUJVN5mdp+gN4+4evXr5ZqW7vTbdq1FhmJ4KLR61hZHzDNH55p4rpOn43Ab7+W3SMX7K+nbJG/pawBl3OhDasSlx+miyd283j1cqQsQP6JeNtb+fw7JuPgbcg5whPasfiIs4QztrHDzA4OmdDz2FyIEiW501W4K99/lprKEOj1ik97F70bpbA0RluAAhvwqaGi0JdVVcCGsKv/6f4RLQB2jJIAtC0y3ANmNudJlvhBMDJyOwzF7Ea1oHgiG4sebwEEMSv0LKcgJWmXqisiR2Bxp1/9WOkep7sMy6wsR8K3Z86jq9UptJ0+JNeyaAU8P6GEmhWvRTiG FSxk9i6I Ul10nStOzH+ChXpBud3iMCunSzY2d/08szch5rww6dUta6/hmdpzY/FPw6FkKj2dNvjyeveGhAYZFwi74cjES/B8Fmm0QeeKUhTng4TelW3lqS+MGqHUWkFQCsezW7lNCMCach2UbFzA+TWty1+Ax74CoE36GTKXSMtpc7BoC21W7bYllrjHHGJ27GSpt2zGh5lupPSonS1n2WgjxcLnnZrUOJyelOX3WSxau5YNXoyyW0hu9Qr40hjLEnhToXe3vYz8WSMdz4aXr86tzcCPT+9mxFJIHkQlljVijthJoBHG89etHWNclKjyNv4qjCiHjuv337U/WDN4MXjIqRu2EDpap715CE2nHpEY3+NkyEGoR1xBY+RgA4JeB9g772md31fdTZxvHIXi4hqPaGzIDD1akNPiWR3m06+adc7cgkNZOhzHsxnuD19ZDWHdv9daVggNyPAsPsw0/h8EKW1avkHH2RCKw4cVHfphA5s/kQuC18ArumCiIJsy6YQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.134461, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Use try_alloc_pages() and free_pages_nolock() Signed-off-by: Alexei Starovoitov --- kernel/bpf/syscall.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 4e88797fdbeb..45099d24909c 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -582,14 +582,14 @@ int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, old_memcg = set_active_memcg(memcg); #endif for (i = 0; i < nr_pages; i++) { - pg = alloc_pages_node(nid, gfp | __GFP_ACCOUNT, 0); + pg = try_alloc_pages(nid, 0); if (pg) { pages[i] = pg; continue; } for (j = 0; j < i; j++) - __free_page(pages[j]); + free_pages_nolock(pages[j], 0); ret = -ENOMEM; break; }