From patchwork Sat Oct 26 03:36:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13852010 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEA37D10BE4 for ; Sat, 26 Oct 2024 03:36:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1E5356B0098; Fri, 25 Oct 2024 23:36:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 194206B0099; Fri, 25 Oct 2024 23:36:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 033F66B009B; Fri, 25 Oct 2024 23:36:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D436A6B0098 for ; Fri, 25 Oct 2024 23:36:32 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C7A47A14AD for ; Sat, 26 Oct 2024 03:35:55 +0000 (UTC) X-FDA: 82714340772.02.21FA9B1 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf21.hostedemail.com (Postfix) with ESMTP id B966C1C000F for ; Sat, 26 Oct 2024 03:35:51 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=mk7i1Inx; spf=pass (imf21.hostedemail.com: domain of 3vWMcZwYKCDYqmrZSgYggYdW.Ugedafmp-eecnSUc.gjY@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3vWMcZwYKCDYqmrZSgYggYdW.Ugedafmp-eecnSUc.gjY@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729913584; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=zJvo8rQelVugAeNPiMO8n98xYiD6Va85kwZA/R+E8fA=; b=TcU2d3R2T2f/AvxZKvACAB4+E0Z+MANABfOwd1L0r/ev3NC/c3IU3+go+qn2HhbKU0Zz3B 5KA5VhXBVW5jJK5QKuQ85AOLwg71/QerX7ByOTgRi0yx+jQ8ITOzhgDqZ4l+gcZ4wSftzF A113MXGxA0dALt+C3eLnk/yBFeDAXVw= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=mk7i1Inx; spf=pass (imf21.hostedemail.com: domain of 3vWMcZwYKCDYqmrZSgYggYdW.Ugedafmp-eecnSUc.gjY@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3vWMcZwYKCDYqmrZSgYggYdW.Ugedafmp-eecnSUc.gjY@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729913584; a=rsa-sha256; cv=none; b=wsr+icoJuWXKjprFh5Sq+9AWPVuc8O794+GFwknGc91DkPT8yUhONDBeGn9CFoqr06Tcel 6BwHU/b43OkDehkR1H3Q0vXnHfiHvWyQLoJKk6gkMDuAZNghjevzQ5H4ZBDEiryve2ERAb zRO50W5s3zhU+0KEehNvThwRdfso180= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e24a31ad88aso4055320276.1 for ; Fri, 25 Oct 2024 20:36:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729913790; x=1730518590; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=zJvo8rQelVugAeNPiMO8n98xYiD6Va85kwZA/R+E8fA=; b=mk7i1Inxy9lGflkrWwQkWeFdGVlOCjNggwl+37EA9q2RhHXQ5ZrgEJY2xPc843DFjm RaUXVonmiWu+sUNe6CJhtUEiO1czXOwPntcdb5i5u5RPjJmUsdsm8Lxt30gxz0Q4xpV6 6KtxpFYAfhaFeKwat42M+knIa3gkGQmjuPFIZBpjKbRzDZToCKUKX3zotJn+JJUrAqkB 3gXXGG9CpLmL5LNcM5DZnfqs7S+CmgxHNBw8ffnA3Xd1hnGWM5JUiAi+nKfzz3E7Vp5A GnBk/MTbc/aAhciDT5MRfHNYxt7hrU4RbRpy0bZ7NXqyP5avQ06Yr4bw1uJ/xOSmi2b6 Xf8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729913790; x=1730518590; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=zJvo8rQelVugAeNPiMO8n98xYiD6Va85kwZA/R+E8fA=; b=W/CIGkqzily3S/o3sKrB5LN3kLRDco5mih36WDJgOjLarH3RPVgPb8azf6niVnbPpn Md6UD9zyyKiiTlw1QBLvkPfY9Qp5XKZKl6HJi8QMX+72o4acy2cAY7+fD5yerrHK1iD5 N/kD90AMrkZpJr+xIiQERCKfbtG0gxDKHZo5KWy+BADRtMHxdoA63jXT6OaKma8F6VDa E0OxgGqOQ3lthD4VPHNhrcKzttbgzN4XvpUWvb63ZhUb4KZq14VjElcZnnt+VcysIWMD sz5DrUkimFW+Fg3vr7mB/nJDf4hXq766Xy1E3wx7S/kTGoxO/dEXxQZhrcb+RKLTx0+M Mt5A== X-Forwarded-Encrypted: i=1; AJvYcCWTojEE5zJarsecyn7f/Pg2Gu6dThkE83lJVCtQwNPnTz705HSM3KVPL30/YCepJzlpPXh9FPHWcw==@kvack.org X-Gm-Message-State: AOJu0YydNJezS2urfaVKnTvuW9jY1UK4iJNsbg1jdVtuzWkudSSqH10/ se8NGkLTpuq/oxePjs2PhLA0F3ChvKlF8CADzeI6IpCGLhzurZTDlB1/XP9YrivHvoJa/qjFH31 k8Q== X-Google-Smtp-Source: AGHT+IFjF2xbY+QdTNq0yXUWw8dhcTHCSRZ/2q1qhW0aC06XSt+a7yPkzdJRVZQVGegp4OdYmAxx4N97DDE= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:94d3:b09:4b32:5539]) (user=yuzhao job=sendgmr) by 2002:a05:6902:1822:b0:e1c:ed3d:7bb7 with SMTP id 3f1490d57ef6-e308784a6eamr1084276.1.1729913789367; Fri, 25 Oct 2024 20:36:29 -0700 (PDT) Date: Fri, 25 Oct 2024 21:36:25 -0600 Mime-Version: 1.0 X-Mailer: git-send-email 2.47.0.163.g1226f6d8fa-goog Message-ID: <20241026033625.2237102-1-yuzhao@google.com> Subject: [PATCH mm-unstable v2] mm/page_alloc: keep track of free highatomic From: Yu Zhao To: Andrew Morton Cc: David Rientjes , Vlastimil Babka , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , Link Lin X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: B966C1C000F X-Stat-Signature: n994ae6x185yaa7ikzbtkq7gi8tipbzc X-Rspam-User: X-HE-Tag: 1729913751-413819 X-HE-Meta: U2FsdGVkX1+rWDo4x/4f3Q5sXC11qMu1bEf8D+6DcduSDx1h2w9yEFArZFAlIZBk0dtXmwzDcr2kKfDTIOKRoTlu6jB9GZQU6hU6TtSw9KuVz99cXlLkflAsAQ4rKHNWdHTAdxOgjLcbEeTUSGsKCGArkzT2LAknT1nHfHT3gOdDNRWxVogrGJ/vTjOnS6K83qauWKHeXCA8UAuKCpup9pNmB/m0ou5yXjymjAcbT7Vf0aFCAKU+TaRR34WKLZVI+buVVqiiCLY9T0JRToog036sAQE50EjreCfX2zGoeiskkA1WpIA4r8nCYZxR6pO2ESuSAh2DTLs7YQhb71yrTxrKeWkLgPKN1Bq921f9Rx3Do3QlCYr59ChK6wd5UHeA1uGm/RmDESRRzOMzDCob1orxjGLNpkRGV9LDIG9ooA3Xw3QqmBHUj5V1OJOZ+JzBtg9qK2fTZsUjahIEuVtQHAxDy4k+H/SMcfD3aO5ImnUdG1DFt6KU9EoaGi2cQfFYX2wJMxQj9LBBZQ9uC08XZuA3CX79clmcxJ/bHr+NhtyngfOzdDeRby00sEFa6zNMKGrXX+dme4a/DhbvkVl5yYgvp0voiDE/3yNZm9kdySaV7u2A90HRSvEPfFkF66MtE0NpdFY/TPsanmTwFu8dpQr3EzWaA8QNxULtN3GD/6jtCj4EUG9DKu0O5Y/79Jo12HO+57TcrokIpVIvD7MqC1G8qN7VZKNNAHTJAwZzSYbdHqpshikb+q4PnsJ4il6pfW0Rhdnmuv4kn0EROLRCmGuK4tjC6HlFnI18ZeOHn1/JANeTgK1oC93YC77fMkfxtw0AXlTR+pp2Mr/SRWS9ajcAo1d38DFbw0uk3IR+AnUZZ+V1cvxLqhSt3aIB89uxv9YhqTq6SRfUf184M5TW+TXJMn3WabXA1EVBBRQFzvOV9eRi69lfP/KJeDchW3jWWHmJaAH+BYYjRoNd24U syX9sLpa b8+v5I6weQPvHFsJgECLX99QhktC+4hiaTewBvxTLQKLse6kYlapsYoc/3EwopGVU9T1PBZA9AZR9aMjUPiXjRxPvyzjnpWFR/tlt+bJj/BkD9wDE5ubokKl87nYUzcGYIzRMh2xxvndRIk0jw/irKI++o6wZLAYnV7snLPPh8HcnSIPcFWbEZQczCmkHbxUCWIPHruH/b/SFPQMVQM//dA2JGYOYm14V3UPTEt9HYTUkzb4FsjDo3GeuCkqZLH6Ms3BTP7dVjFFJwy/qmSRhWaRoJvN6ZgIyAaEb3kFCAWVqr36zAZRf9MydedQyN5gcT1hntUVIbqbfb7BBLlrDJ3Y3lsGx+5mBeM1T4SWQ5xy/w4on5ZtYgYFVrWLZpl6MnhGwWG+YMCrHwoGtILdVEyfoy5vGVB7/iukAgOVJ8aEAgeX93cgCdkD/33FdM7FxFZve7SNyp/5d2SuWAa6zLzN7svlpEyruSYkXDACcXAxV1ENDnnQwMKicJ7RWKlVZ585zxjo9hBFg+iniKQB+Gr+ERlOqZZ6Ce0IZdJTsPq3RZNt8cNqpiqWBqXYKhfHfTWGGEVpY9gDLHuI1jWzj9gRcEA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: OOM kills due to vastly overestimated free highatomic reserves were observed: ... invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0 ... Node 0 Normal free:1482936kB boost:0kB min:410416kB low:739404kB high:1068392kB reserved_highatomic:1073152KB ... Node 0 Normal: 1292*4kB (ME) 1920*8kB (E) 383*16kB (UE) 220*32kB (ME) 340*64kB (E) 2155*128kB (UE) 3243*256kB (UE) 615*512kB (U) 1*1024kB (M) 0*2048kB 0*4096kB = 1477408kB The second line above shows that the OOM kill was due to the following condition: free (1482936kB) - reserved_highatomic (1073152kB) = 409784KB < min (410416kB) And the third line shows there were no free pages in any MIGRATE_HIGHATOMIC pageblocks, which otherwise would show up as type 'H'. Therefore __zone_watermark_unusable_free() underestimated the usable free memory by over 1GB, which resulted in the unnecessary OOM kill above. The comments in __zone_watermark_unusable_free() warns about the potential risk, i.e., If the caller does not have rights to reserves below the min watermark then subtract the high-atomic reserves. This will over-estimate the size of the atomic reserve but it avoids a search. However, it is possible to keep track of free pages in reserved highatomic pageblocks with a new per-zone counter nr_free_highatomic protected by the zone lock, to avoid a search when calculating the usable free memory. And the cost would be minimal, i.e., simple arithmetics in the highatomic alloc/free/move paths. Reported-by: Link Lin Signed-off-by: Yu Zhao Acked-by: David Rientjes --- include/linux/mmzone.h | 1 + mm/page_alloc.c | 23 ++++++++++++++++++++--- 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 2e8c4307c728..5e8f567753bd 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -825,6 +825,7 @@ struct zone { unsigned long watermark_boost; unsigned long nr_reserved_highatomic; + unsigned long nr_free_highatomic; /* * We don't know if the memory that we're going to allocate will be diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c81762c49b00..43ecc7d75a2a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -644,6 +644,18 @@ static inline void account_freepages(struct zone *zone, int nr_pages, __mod_zone_page_state(zone, NR_FREE_CMA_PAGES, nr_pages); } +static void account_highatomic_freepages(struct zone *zone, unsigned int order, + int old_mt, int new_mt) +{ + int delta = 1 << order; + + if (is_migrate_highatomic(old_mt)) + WRITE_ONCE(zone->nr_free_highatomic, zone->nr_free_highatomic - delta); + + if (is_migrate_highatomic(new_mt)) + WRITE_ONCE(zone->nr_free_highatomic, zone->nr_free_highatomic + delta); +} + /* Used for pages not on another list */ static inline void __add_to_free_list(struct page *page, struct zone *zone, unsigned int order, int migratetype, @@ -660,6 +672,8 @@ static inline void __add_to_free_list(struct page *page, struct zone *zone, else list_add(&page->buddy_list, &area->free_list[migratetype]); area->nr_free++; + + account_highatomic_freepages(zone, order, -1, migratetype); } /* @@ -681,6 +695,8 @@ static inline void move_to_free_list(struct page *page, struct zone *zone, account_freepages(zone, -(1 << order), old_mt); account_freepages(zone, 1 << order, new_mt); + + account_highatomic_freepages(zone, order, old_mt, new_mt); } static inline void __del_page_from_free_list(struct page *page, struct zone *zone, @@ -698,6 +714,8 @@ static inline void __del_page_from_free_list(struct page *page, struct zone *zon __ClearPageBuddy(page); set_page_private(page, 0); zone->free_area[order].nr_free--; + + account_highatomic_freepages(zone, order, migratetype, -1); } static inline void del_page_from_free_list(struct page *page, struct zone *zone, @@ -3119,11 +3137,10 @@ static inline long __zone_watermark_unusable_free(struct zone *z, /* * If the caller does not have rights to reserves below the min - * watermark then subtract the high-atomic reserves. This will - * over-estimate the size of the atomic reserve but it avoids a search. + * watermark then subtract the free pages reserved for highatomic. */ if (likely(!(alloc_flags & ALLOC_RESERVES))) - unusable_free += z->nr_reserved_highatomic; + unusable_free += READ_ONCE(z->nr_free_highatomic); #ifdef CONFIG_CMA /* If allocation can't use CMA areas don't use free CMA pages */