From patchwork Tue Oct 15 12:51:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Fleming X-Patchwork-Id: 13836340 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 809ADD21690 for ; Tue, 15 Oct 2024 12:52:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ECCB86B0085; Tue, 15 Oct 2024 08:52:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E7C936B0088; Tue, 15 Oct 2024 08:52:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D6AEB6B0089; Tue, 15 Oct 2024 08:52:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B8BCC6B0085 for ; Tue, 15 Oct 2024 08:52:06 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A1EE28163B for ; Tue, 15 Oct 2024 12:51:58 +0000 (UTC) X-FDA: 82675823790.17.E057A62 Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) by imf30.hostedemail.com (Postfix) with ESMTP id 7B33680006 for ; Tue, 15 Oct 2024 12:51:50 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=readmodwrite-com.20230601.gappssmtp.com header.s=20230601 header.b=Jb3t3sRs; spf=none (imf30.hostedemail.com: domain of matt@readmodwrite.com has no SPF policy when checking 209.85.128.47) smtp.mailfrom=matt@readmodwrite.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1728996534; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=qamiraWdxMoUu029itAshQt6SKv15LdwoZC2cdLFctQ=; b=EK4yqPa/B7INsLYZ6+S8xruIcJ/igxpXi3tZO9YP4oUyW/UMsuXNM+mUZNW1D+dWiFUvU9 FUjraP9qAAset3cNmDv1xlNNeC2mBHksNlEo1oxnw576mCi8H98/Yxyh2u7p4yWDX4dD+q WRqizk/AyF+KWpx3H1xMccSzMs6ZRBU= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=readmodwrite-com.20230601.gappssmtp.com header.s=20230601 header.b=Jb3t3sRs; spf=none (imf30.hostedemail.com: domain of matt@readmodwrite.com has no SPF policy when checking 209.85.128.47) smtp.mailfrom=matt@readmodwrite.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1728996534; a=rsa-sha256; cv=none; b=af9Vmn6aQRkO0UmE/i9Uia3O3yZaRHBHi9z6p7GFbUOYEXxdhUoHxpSF3GJy/gYJ6kQMtl M0CBu3d4BPihyc02vnNaHQlsNj53SjxjspsGa+au/yYWVbJoYBp3R6BkLlV0HUIsuza0Vz 2HoUoJjftIUdzIW4D0gI56OBsWZh3o8= Received: by mail-wm1-f47.google.com with SMTP id 5b1f17b1804b1-431137d12a5so50302775e9.1 for ; Tue, 15 Oct 2024 05:52:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=readmodwrite-com.20230601.gappssmtp.com; s=20230601; t=1728996723; x=1729601523; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=qamiraWdxMoUu029itAshQt6SKv15LdwoZC2cdLFctQ=; b=Jb3t3sRswdOcZ3XQFWfYYfg1Ivn8MxjeFsXydhMQWImqQOH1sp9BU4Qq7yONCfmCia RC+X1CrFEX0I2LZgM8ZoFlwKtRJSpFE/KSCEI0ehNf9lCf69cxWX+TKPpyPI6hsZOjBC wOzXN4xjhpBi0UPNS/temwfzmoIWTDCuKzGrObax6mx4WPJpBMWHzi2Xe4C+MlZHAdWx CnL1Uj0iRjHBajoDYLquo5hbz970x04yI+64Hc0SsDDvjTjPNJexJDJSuJxmY0KtZW+V KJ9keiZU9Ah1yBD8JfdFw3qAxKKKQzRzvWoo9KT4rDjc1wzYVmQe2ZQpzjoQg0IWNQ1N lfYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728996723; x=1729601523; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=qamiraWdxMoUu029itAshQt6SKv15LdwoZC2cdLFctQ=; b=Pt1SgPHnzGmdQLAw5pBzaaQ18y6Nh2yTsfskZJjzK/VMH6KS7I3LZG4SaAWk/A+QDn sZOMUwNVODUW1frmai664tDIV8URXnHNjGrBIYRpaZSZL97aOLUifGMiyyzCepu+ivP3 JNmldCmIGPC+CgsxJhtGIcViTnW1xFvPQDpPnp8q9LFYMZzJpxW8K0e2Pc1oFcP89UXl ZlwewpJeLc5Q1OipH6dZP5Kn+WV3C2/w88DpylQ5lMu1F336kp8TG63oOb3dzmwxKlU0 G72gAsb0d5v2rDhH/bUTs1Vk8FjEMGYYQwHvv32x9D5E7F5v0eHmo5SmvJ+uJJgENKkY vDtg== X-Forwarded-Encrypted: i=1; AJvYcCWgLJ78m7SggKTWU4R2vz1Etjpz9i/mrQWQ9xXGLoY6AbAJL0pT/Qb1HR0NMqZRO2Jq6G3F36uG4Q==@kvack.org X-Gm-Message-State: AOJu0Yx1X6hYLXuO8zlau2KJWjFZBtZ/JgphG2nfqe4ScGiYRZUfQFmk Iut/7nSkaBOcdr4VKnY5U69KWhj5joNGEE0NIQg+ZrGGAIuyqLeWoBiV76OhT9Q= X-Google-Smtp-Source: AGHT+IGlcfxtmIQrnyIIKngfm0k3VUAbv5d96errbARi7ueQs0o9UH2OdW8TpdwQkwZ8/lcbTm4l1Q== X-Received: by 2002:a05:600c:1d29:b0:42c:a574:6360 with SMTP id 5b1f17b1804b1-4311df5bda1mr134372545e9.29.1728996722337; Tue, 15 Oct 2024 05:52:02 -0700 (PDT) Received: from matt-Precision-5490.. ([2a09:bac1:2880:f0::179:225]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37d7fa89c55sm1508730f8f.46.2024.10.15.05.52.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Oct 2024 05:52:01 -0700 (PDT) From: Matt Fleming To: Vlastimil Babka Cc: linux-kernel@vger.kernel.org, kernel-team@cloudflare.com, Matt Fleming , Andrew Morton , Mel Gorman , Joonsoo Kim , linux-mm@kvack.org, stable@vger.kernel.org Subject: [PATCH v2] mm/page_alloc: Let GFP_ATOMIC order-0 allocs access highatomic reserves Date: Tue, 15 Oct 2024 13:51:58 +0100 Message-Id: <20241015125158.3597702-1-matt@readmodwrite.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 7B33680006 X-Stat-Signature: tdgt5f4mx6316a861yjbfoz985591upx X-Rspam-User: X-HE-Tag: 1728996710-903972 X-HE-Meta: U2FsdGVkX18oxmQ7/ujOVm/4T3wBMCbW3yfTmpxGOMYRFeUzD9IZJ7KR4rGLapSRnUTfCFDeXaGvpng7P3LTUiS7iCS5f+hxQYbBfoE3VuQc49DvzUR7mxgAAuuMm9+D8WoYDgUgsCMPsezh1L7ygz4JRQZaWCwkrcIhVgjegFWoRfCPkX+tE4QwyFCImsqhEEks+MVXiB0v0agqf2wx9xSXNmI0llAp/tY+SIOlEucFeylYz1rRwRtC+wiYWlsCmZYAT8Xrr0LdghKJqqeOJc76jb0pBM4UM2Se3paMF+Bp0WZmuh5F6hS7pg+HQlI4Vu3qJAloRDl2kE3ycoUIzNb+2EQg1wE4fwIY8vQUh1cWJf6cAL0RIevojOkj7ao7VksMAg8xcV7+0Cvk1fKbMTy0EyiUc3buxFwL0BmszOZp37nm1qUeT3+vcSajgwBDgx8vz0shAa1qbxZVtXnfaoZjQpAQbqGSJDLyL6EtBKSkDg184BHhWHqjUVOFZaEDp7JA8nl/i/EL1NVw3pkOTDYy4GZ0Jq4fZ6hIPm8G51KANzccn1yQtfWMmrgmhgu7hS7qToe5HZHCThms5vDsYqGhwH2jrAxSV9GfI391sVMSOBHveDaH2Hi8YMLDccbB9XNpjv3P0m6Wi5nL+9iQcofkmihjUPM+iTh+xqfkaemtAsKA6Ou0hh0FysoviXHya/VBZiTT2ONkURqT9xFP1p8ndCuhq+caA2WBW+ApXN9IxjGRegJyQEtEQ9D3XQMYf3w3xWzTF4U09eiu/Kf8H3wivRwnuLVLkywdLIU5gM8kZqAsB1xZ58RE31r4+xDswgFy8ortD1TSw5GWTarYrHiCsXAcyPju2Wn3tRqN57PATpURAONYEGnn9gNnm8u3uMed/OOEffn2L07ZHgjD4BhdBVNSBFR/3J3OyVdz+lsYL93kYAEnJDQFIklrUdCgLMFcEPakGjgxG7Z7psO Kd8wXpeW VKYVcaYgo1xwQNnul3pdmMmwyhKM7Ze18EVugkdOMwLS7XDU4WhGBq99n8xyShF1WVMMYmWSHj1l2Cop221mfoCdfiX82knXhvYRWXhSlXzfFLjToO3WjQsOaIFLnwCMi1fi7mdBXILx4sUWqxh6QBfk+Ug7JWvOo1aXRfl1B5wVgDXywsqaWux1tnR6lHcrR5ar3ajRhizOtUl4tq3/CcZYhWXlDwyy/33UxyqZxfGZTGcihVJyLNt+v/X/CCEUP7Vn4GNlmE7nzREH1hJsVF8wVLjSPl9mI36x/PKyQLb2vYyzxIIa/s1W6+lI0tDGSqVbm0JT/uhvFX1tGA18b5b8yEfrCcqvFj7GIM7k3lRRzew81zp9IsTrS9Z18jabXDGbMe6RdLDYBmcC2zJkE8kvxRMZ5g0iDXRugMe3vkh9cmqd+E7Dld+m3OkMKvxGbBlr4dSvQ9ionNiP0ziKzMCoJhlmAuFbQYV8gNOaZETdC3OtvQoWgT20qLELC7nBgEQgl1ldaW7DgXvHzVduCbLFic/6QbkQO18Y4L6iTAs2BeJoOyZR5AOZOm+3oIZgpG0jAIkdE/ofuGw1w3Dw4oBQ0JnXC253QQvSeRULd4Ukb8KpWPAYDj+h14+60a3MGloAjLMYC9hbkTjOYHGGeLMXA/FxrNKe9C/r3JBevx3rqCBSUUNcu45TreTbFc4G4ApV02PaeH+b+pxzt3FKKXoonig== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Matt Fleming Under memory pressure it's possible for GFP_ATOMIC order-0 allocations to fail even though free pages are available in the highatomic reserves. GFP_ATOMIC allocations cannot trigger unreserve_highatomic_pageblock() since it's only run from reclaim. Given that such allocations will pass the watermarks in __zone_watermark_unusable_free(), it makes sense to fallback to highatomic reserves the same way that ALLOC_OOM can. This fixes order-0 page allocation failures observed on Cloudflare's fleet when handling network packets: kswapd1: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0-7 CPU: 10 PID: 696 Comm: kswapd1 Kdump: loaded Tainted: G O 6.6.43-CUSTOM #1 Hardware name: MACHINE Call Trace: dump_stack_lvl+0x3c/0x50 warn_alloc+0x13a/0x1c0 __alloc_pages_slowpath.constprop.0+0xc9d/0xd10 __alloc_pages+0x327/0x340 __napi_alloc_skb+0x16d/0x1f0 bnxt_rx_page_skb+0x96/0x1b0 [bnxt_en] bnxt_rx_pkt+0x201/0x15e0 [bnxt_en] __bnxt_poll_work+0x156/0x2b0 [bnxt_en] bnxt_poll+0xd9/0x1c0 [bnxt_en] __napi_poll+0x2b/0x1b0 bpf_trampoline_6442524138+0x7d/0x1000 __napi_poll+0x5/0x1b0 net_rx_action+0x342/0x740 handle_softirqs+0xcf/0x2b0 irq_exit_rcu+0x6c/0x90 sysvec_apic_timer_interrupt+0x72/0x90 Fixes: 1d91df85f399 ("mm/page_alloc: handle a missing case for memalloc_nocma_{save/restore} APIs") Suggested-by: Vlastimil Babka Reviewed-by: Vlastimil Babka Cc: Andrew Morton Cc: Mel Gorman Cc: Joonsoo Kim Cc: linux-mm@kvack.org Cc: stable@vger.kernel.org # v5.9+ Link: https://lore.kernel.org/all/CAGis_TWzSu=P7QJmjD58WWiu3zjMTVKSzdOwWE8ORaGytzWJwQ@mail.gmail.com/ Signed-off-by: Matt Fleming --- mm/page_alloc.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) v2: Update comment and add Fixes, Reviewed-by, and Cc: stable tags. diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 8afab64814dc..94a2ffe28008 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2893,12 +2893,12 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, page = __rmqueue(zone, order, migratetype, alloc_flags); /* - * If the allocation fails, allow OOM handling access - * to HIGHATOMIC reserves as failing now is worse than - * failing a high-order atomic allocation in the - * future. + * If the allocation fails, allow OOM handling and + * order-0 (atomic) allocs access to HIGHATOMIC + * reserves as failing now is worse than failing a + * high-order atomic allocation in the future. */ - if (!page && (alloc_flags & ALLOC_OOM)) + if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK))) page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC); if (!page) {