From patchwork Tue Apr 18 19:13:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216104 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52F4CC77B78 for ; Tue, 18 Apr 2023 19:13:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EDC2D8E0001; Tue, 18 Apr 2023 15:13:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E8BA1900009; Tue, 18 Apr 2023 15:13:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CDDBE8E0003; Tue, 18 Apr 2023 15:13:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B87AC8E0001 for ; Tue, 18 Apr 2023 15:13:40 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 8DC7A40450 for ; Tue, 18 Apr 2023 19:13:40 +0000 (UTC) X-FDA: 80695461000.20.F52B6BB Received: from mail-qt1-f182.google.com (mail-qt1-f182.google.com [209.85.160.182]) by imf06.hostedemail.com (Postfix) with ESMTP id B74DC180010 for ; Tue, 18 Apr 2023 19:13:38 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=ROm0ArlP; spf=pass (imf06.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.182 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845218; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=G1QAyddYnWJUKuKAIak/itfaS7Ph3sBa4ofaAGmayWk=; b=N6bi+bnm3ooprNj/Bp6FmNZbaBjgsN47cWMV7RvFp5C63Mfj1LA00S/tbCJEIXJ1dhG5RX a8502Qqej5QwXjCibiW3OOtgrjAV0RSCEgq4b/CtMJ1kLLpQYHqNMoUTkaLvknf7feHZku sv0wueYpWcK6uELI9iLTJ1gegdKvhIM= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=ROm0ArlP; spf=pass (imf06.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.182 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845218; a=rsa-sha256; cv=none; b=BDMTrWWz/GexrKA95ge1TgfE1ctdRlcRQ34/DC4GFyFrkpGbDy/JmqB/eGY25zwh/ommnt LHMEHr35S8a1jq/phz5L1dQwNPVVCs1yxFul/VxSdCOO3+XYU5nHD4E/KJQJm8M+hXSXVZ GUYimjBPYTUQma3AVMiorWuSph02ixE= Received: by mail-qt1-f182.google.com with SMTP id br19so1860695qtb.7 for ; Tue, 18 Apr 2023 12:13:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845217; x=1684437217; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=G1QAyddYnWJUKuKAIak/itfaS7Ph3sBa4ofaAGmayWk=; b=ROm0ArlPuXBY1EzWc/8PRgqaTVBI+lb7icLJ0Q57BP9ugdMTZmrUmryBS96SSVH9Nh eeLrq3q4kINpOOeYwSiqOxAJrdmOgMg80zVTThvjx9VkzE5k0WCPil9tESj3Hku5cLuy SPJk1gyPDWl9cU4d7hzMlhCvpJn6xdGkNXplbkrtzq0OiV6LSdAYuNrP+KJa/f3MP5dx 4wKlDRUsNPqCre2MCxjIRsO9xKT8/coTjpyJXWIkKBFyHmv7zdnJQKHUNs+zIEe6gm+0 cTwYRc8KLsJOFaKN9aFOQI6B99BvcPA5pw+NCRvbGraHEF3iyw5uVssKuGhFRoPeJkSZ mtEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845217; x=1684437217; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G1QAyddYnWJUKuKAIak/itfaS7Ph3sBa4ofaAGmayWk=; b=Nqfond4DSoj/7OE9RXcS7HkPaimoYf+3u9bEBV7M6ngwob3ZpVlhOZtcpoTgZPw2jI O/pBaOYMRqWNf9COHHwhaJpuSmyYiYsx9vITRViFse+XWl63HB4qy54Se3mU/08C2VEQ DlgWdwAzRXrnL5H/ko3nG47tiADS7Mvc6AHu6/FJY76GBocNmUWgkhS3T8u/nxuMLBT8 WLKVYM9a1mBF7NM8UCvMyy2r23Fw9d4gTS+2mcjdizHHtyZJrCEi+pNHUvKtqnXvTXjT p41MM6HdaII4szatCazFsnSn9zls4E7MZbB9JfxIwUFE/tgfvEQhoCIhpDnB2wjoiO0S E/Ww== X-Gm-Message-State: AAQBX9cni7NtchRd56BT2lH8eMioE+AnBLbiQitJtCxg3zmyXmiD8lNP NFZzDYIg8ev1BxxBwtjRe0vructnZUorXwShuJQ= X-Google-Smtp-Source: AKy350ZUW/3fdavMXIT1x5ul2i8OAycLqVKqq8/q+G7rUbGTG7TjkJBhgSj/nJexX1DU7fRe/5Ryig== X-Received: by 2002:a05:622a:1703:b0:3e4:deff:4634 with SMTP id h3-20020a05622a170300b003e4deff4634mr1499645qtk.24.1681845217590; Tue, 18 Apr 2023 12:13:37 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id w4-20020ac86b04000000b003e64d076256sm4281417qts.51.2023.04.18.12.13.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:37 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 16/26] mm: compaction: improve compaction_suitable() accuracy Date: Tue, 18 Apr 2023 15:13:03 -0400 Message-Id: <20230418191313.268131-17-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: sugw8bc6eghqiqhkxsxemwwufswujdbc X-Rspamd-Queue-Id: B74DC180010 X-HE-Tag: 1681845218-273923 X-HE-Meta: U2FsdGVkX19tlf2gx/mVZRd1IcWYGO0m4Vv732YV7OL32ezsj/KAWymrWSlRKlz+yVbBvCeV93zQbM0b+O1brEFzjeuhmmlNkwZONsYLkQ7kDugf+pYFhT1RYQEFr13ooDIbRfJ8uxWrCgVQB5tbUDvWqXfEKk5zNWaaMlkkwPZgQwx3jHhPLoTHb7DYvawU/E+1LlKtIOeyVoaAYxNr/5DQj3Y6CFCBkniOHr/6OJ3vlrWsIcdlgttPMLUHnZSr4RK5i8Sm7r+t+9J1NHqXeNp27uvTfrTIEjUHsUkSY7396S7pjBpb/5ViN4NZ2OsOQ9px7aGlwePd20xYBqtCRifM2NUPAd27XNj67X+bMWhbfTjslYPs7N5aGkImMqz1rfYZw9KN+27YkyxHxZ4XuvwpC8ua5CNe/O4uxYogCYyBWRwJo8wOx3Kdl5k90X0MB3VvEgws66jW0R72N66fuPhWhNGukBOrvTvqQ054JIi8ui4xK6owPvL86vvp6PDFEqVO+cMT6zeqKWW3kI5RxL7Up57EbsnY2jHP2YKCKHT/qKgv3GVvevFE4Fk6axdmLIx7uSRYe3+eDVAqouO0pwTos4p/9zAdqhJrxlZlxF4E8bbmlzt8pxtqbWYDXyJH4a0slofm/YYdiMOviJ5uyHZ0OnrLuiCSaZzVnadtNWS1e7lSLeETbRVy+9weU75iJz6/hmFYQVuOkiBfdStxG7JOBsb2nivrWTtAyrS5AFfKLsuVb52Gkrwdkp5+7xnwI0fTeri6Km5EnZpQczpTU4pv3tGfKiXe2hmq36+ziVe4wulRI6JpqHjrKe6k7F+D83i3vhXEu7HgvZ80QLWtlBg7axahmLhXPusReTMb580HOOTHvZZEVXYAPImq9sdReJRYzawLoFcGywwzA2EpJoc7w1sOUxfZvrJoBBrFbRKG0+XR/oWg6hm09X98ef1SdMlD2LqfbisQhAxX+Ex WjFub14F IHLW+mHVXh5cozXKsz1VAnTib4A5pgG0mq53UpT+wLe7TO+5+vioPhc2VOBTxwDwJkJZflAwnTbKKCef5GHNFS5/qBO/3Gzmrs6gxJHxB5nLTb+CaNhkqa8+QWsyG8yIS2w5UJcHlyTCEYoCOlepo/BR5NOfHL+j8RTgsJhnTqOs/K5WHT80NWbAUusxVt4XDMo+ODMf1LIIeq+rwzec1Mm3WlgvIvpKo3WzEyRJDiDYNr8jBJ3ZfT+4w6LoPDfONbeb+XSF4l8qAYpwDShEPtkaxOg7q4Z4POeLuy5B4s++VLTmGpnlDsCStXlAgCiwfJd36UERVF3hpZ4v1P8aQAt+4uO6N2Ph6IwbyyTwhMiFfiJK8DXieEb/ncfNoeIMGpf70tBXvlFvOFKbJ1xmxqaI7VuEDAXG4/5V5TiRIj6SMfbtSuwjx6eztQlVdFT392DDCF3royzdNH1e9BK2NBn+Ql9zE1uni2J+WcJE+iiirLgfIb3NS2JeWINn3Z1Imx6lH0Nd3aXXuYoGamv96YyJsV6rRO/ZR5yeJ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With the new per-mt free counts, compaction can check the watermarks specifically against suitable migration targets. This ensures reclaim keeps going when the free pages are in blocks that aren't actually suitable migration targets: MIGRATE_FREE, UNMOVABLE, RECLAIMABLE. Signed-off-by: Johannes Weiner --- mm/compaction.c | 23 +++++++++++++++-------- mm/vmscan.c | 7 +++++-- 2 files changed, 20 insertions(+), 10 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index b9eed0d43403..f637b4ed7f3c 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2226,11 +2226,17 @@ enum compact_result compaction_suitable(struct zone *zone, int order, unsigned int alloc_flags, int highest_zoneidx) { + unsigned long free_pages; enum compact_result ret; int fragindex; - ret = __compaction_suitable(zone, order, alloc_flags, highest_zoneidx, - zone_page_state(zone, NR_FREE_PAGES)); + /* Suitable migration targets */ + free_pages = zone_page_state(zone, NR_FREE_MOVABLE); + free_pages += zone_page_state(zone, NR_FREE_CMA_PAGES); + + ret = __compaction_suitable(zone, order, alloc_flags, + highest_zoneidx, free_pages); + /* * fragmentation index determines if allocation failures are due to * low memory or external fragmentation @@ -2273,19 +2279,20 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, for_each_zone_zonelist_nodemask(zone, z, ac->zonelist, ac->highest_zoneidx, ac->nodemask) { unsigned long available; - enum compact_result compact_result; + available = zone_page_state_snapshot(zone, NR_FREE_MOVABLE); + available += zone_page_state_snapshot(zone, NR_FREE_CMA_PAGES); /* * Do not consider all the reclaimable memory because we do not * want to trash just for a single high order allocation which * is even not guaranteed to appear even if __compaction_suitable * is happy about the watermark check. */ - available = zone_reclaimable_pages(zone) / order; - available += zone_page_state_snapshot(zone, NR_FREE_PAGES); - compact_result = __compaction_suitable(zone, order, alloc_flags, - ac->highest_zoneidx, available); - if (compact_result == COMPACT_CONTINUE) + available += zone_reclaimable_pages(zone) / order; + + if (__compaction_suitable(zone, order, alloc_flags, + ac->highest_zoneidx, + available) == COMPACT_CONTINUE) return true; } diff --git a/mm/vmscan.c b/mm/vmscan.c index 5b7b8d4f5297..9ecf29f4dab8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -6270,6 +6270,7 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc) static inline bool compaction_ready(struct zone *zone, struct scan_control *sc) { unsigned long watermark; + unsigned long free_pages; enum compact_result suitable; suitable = compaction_suitable(zone, sc->order, 0, sc->reclaim_idx); @@ -6290,8 +6291,10 @@ static inline bool compaction_ready(struct zone *zone, struct scan_control *sc) * we are already above the high+gap watermark, don't reclaim at all. */ watermark = high_wmark_pages(zone) + compact_gap(sc->order); - - return zone_watermark_ok_safe(zone, 0, watermark, sc->reclaim_idx); + free_pages = zone_page_state_snapshot(zone, NR_FREE_MOVABLE); + free_pages += zone_page_state_snapshot(zone, NR_FREE_CMA_PAGES); + return __zone_watermark_ok(zone, 0, watermark, sc->reclaim_idx, + ALLOC_CMA, free_pages); } static void consider_reclaim_throttle(pg_data_t *pgdat, struct scan_control *sc)