From patchwork Fri Jul 7 08:51:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13304601 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8E33EB64D9 for ; Fri, 7 Jul 2023 08:51:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4C7AD8D0002; Fri, 7 Jul 2023 04:51:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 477408D0001; Fri, 7 Jul 2023 04:51:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 365CE8D0002; Fri, 7 Jul 2023 04:51:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 2280D8D0001 for ; Fri, 7 Jul 2023 04:51:57 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id EAF1D160656 for ; Fri, 7 Jul 2023 08:51:56 +0000 (UTC) X-FDA: 80984198232.05.E8DB924 Received: from out30-110.freemail.mail.aliyun.com (out30-110.freemail.mail.aliyun.com [115.124.30.110]) by imf13.hostedemail.com (Postfix) with ESMTP id 6696B20005 for ; Fri, 7 Jul 2023 08:51:53 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=alibaba.com; spf=pass (imf13.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.110 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688719915; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references; bh=AOxPq48FtdM+DmsA8kY++8wzQAkMRTzYKQYl15WQD/E=; b=2tLAGHVM8effcBUENq6eEYctMiTdisunoY+Yxr4W4rKyP1LdnTRoxR+/dURrKVfWeyl+eh uR9VMZBjEwx3vRqZlJ10vfAqgw+7tBm108dhxEucuapfnPsguFdjl7YhYKuafkbjCDvbNE pfnApdYeV5quZgiQbd6GbHHuTb33sic= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=alibaba.com; spf=pass (imf13.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.110 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688719915; a=rsa-sha256; cv=none; b=f6NTL54usUzcZmcgToNL3EvGRGdrgd2OUydl8KRkCUusj3xGFOak6Pz/QwSELvLyZLmLFS co4Z+KzWBQ8p1SNqeTtwKHaxH7hUSH0q0kysIUnKZcnwBowtWCZpaEvO2SfAaLcAlFH7yR cf+3z2q+WQQrRWa5g2936gLTMAmaXZs= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R431e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046051;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=8;SR=0;TI=SMTPD_---0VmnvSa9_1688719909; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VmnvSa9_1688719909) by smtp.aliyun-inc.com; Fri, 07 Jul 2023 16:51:50 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: mgorman@techsingularity.net, vbabka@suse.cz, david@redhat.com, ying.huang@intel.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/2] mm: compaction: use the correct type of list for free pages Date: Fri, 7 Jul 2023 16:51:46 +0800 Message-Id: X-Mailer: git-send-email 2.39.3 MIME-Version: 1.0 X-Rspamd-Queue-Id: 6696B20005 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 698jsgfjj8obxm5jj4swp7bh8i6pcimh X-HE-Tag: 1688719913-255702 X-HE-Meta: U2FsdGVkX1+1GVRp/PO2G5Islh6QnFjsr/F0BUlgHGjafznwviVl+O53igz8iPHktznQpbliJwvU1UPIT+u8JfnPbn9AAPAWNPZZT22RPz4ARzIjj2sbXEeK/LcpuEa2W+jIGasEwmr9vSn3ta6fKx53wKj3Oz83PNNn2fLtkHyDOSv0Xi5iosEeyIuUoVDT1OmK+W2gWAMW0WwYxU9n+Lg7or/S1QZynVKs+GpUmZUlvCIzc8pLgMCRassX3n1EUVTTXxmBsoFbTeO3DQXipPh5xiCWqCnoaqkYh4QVojo6bqRCEVAeSVhCxQm/YYu1tZTFpeo7HNBOk5ICqHXHWlUd/oT3Ax0d77uVap7O8nRJJ/TTmpvU6e4/9Y+kxKQGdkj9pt8dhwTPBx+8lM5qWVNjGXTm0/C4c9hWhSy0T0xBWno5QnUaizjAJVsA3ekoXP+SeC998IgDiI50wskuQwSywIEQXXPbOdHVyVNP/kaZ8pVEABfWanbddm7LJBG9TIxITfO/GlmbEjA404WUjCrNhSU2Xxf6Wa2Waihi+qX7k65H6+vMoPjRs/OLGCfiDzVwGh4rxqRoqg4vtBpB+GNrzvcpscWLEmzX3A9PskUaVpqLES8PP3/aZLCod7P8dEmCPAfwPCK0bgMhdJoI/wUgDITDUMNuFUwrqhONp4tqPTI/Rc8//P1GGY3eWLm38+o6Vy+LQxdtETXQ96YT45oYpMkJURkgWUoxvghWi4lmyX31h/m4H/665bQw2vwiVGMfhopBEZJv/6VmPp5wcaWvsdWOZDLrapKy42SOnbop9Ssajeu2gOvAwlwFoXMu7XPqLVJwnv579ReruqRXc32Bob+nvu4M6BMKLhE3ExxMtqlTwngVppiTvy3zaPmT73da2WCXifoBDXhlBlCKL9fhYO0OwsR1rUFQboA0elTwaP6L6UW9Aojk+9/T4p8/U3saFUQPBLpXJV5psFp uSU0DWzq g+0XRtfGCQ9sie/csBdtFu6eTDzDcdHyJmVgWpRkjBgJNvzu5tiwzUth1r0WhupDG3x7T+bXnvhlT5ypZPf2a4KZxbTVh3Uj/RkbUJQIkoJuHrK2BPpODN0UBQ9gxk9AHuZOb X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use the page->buddy_list instead of page->lru to clarify the correct type of list for free pages. Signed-off-by: Baolin Wang Reviewed-by: David Hildenbrand --- mm/compaction.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index dbc9f86b1934..43358efdbdc2 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1500,7 +1500,7 @@ static void fast_isolate_freepages(struct compact_control *cc) spin_lock_irqsave(&cc->zone->lock, flags); freelist = &area->free_list[MIGRATE_MOVABLE]; - list_for_each_entry_reverse(freepage, freelist, lru) { + list_for_each_entry_reverse(freepage, freelist, buddy_list) { unsigned long pfn; order_scanned++; @@ -1883,7 +1883,7 @@ static unsigned long fast_find_migrateblock(struct compact_control *cc) spin_lock_irqsave(&cc->zone->lock, flags); freelist = &area->free_list[MIGRATE_MOVABLE]; - list_for_each_entry(freepage, freelist, lru) { + list_for_each_entry(freepage, freelist, buddy_list) { unsigned long free_pfn; if (nr_scanned++ >= limit) { From patchwork Fri Jul 7 08:51:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13304602 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE4BCC001DD for ; Fri, 7 Jul 2023 08:51:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8E5848D0001; Fri, 7 Jul 2023 04:51:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 895028D0003; Fri, 7 Jul 2023 04:51:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7844D8D0001; Fri, 7 Jul 2023 04:51:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 62B688D0003 for ; Fri, 7 Jul 2023 04:51:57 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 37DD8160656 for ; Fri, 7 Jul 2023 08:51:57 +0000 (UTC) X-FDA: 80984198274.14.5755B13 Received: from out30-118.freemail.mail.aliyun.com (out30-118.freemail.mail.aliyun.com [115.124.30.118]) by imf16.hostedemail.com (Postfix) with ESMTP id 018F8180016 for ; Fri, 7 Jul 2023 08:51:54 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=none; spf=pass (imf16.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.118 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688719915; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QuEz8VtQYPea+3TAX/L5Nki2qQd83grwnRUVMVkXeeE=; b=FB1OeNVEgkSWA8IX7GxAx5q8ckBLIIeykQ86eljLXscACN9lrKfiOLmqXD8UqGW07Hk3ue HB263mH5oQxUb4Ka7CM7qHOU/ABGCTpb//NK/RIO7pSLjk9guwPPMv+XkelwH/yTL9U2S8 Q1ckwCxdz4lHa+98bq7a8xnB4+/SCBw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688719915; a=rsa-sha256; cv=none; b=Md5LZb/LHJeDZAyGeafqVzhuKB1K9OTcm5yMmaTvTKrW32UeGR6DgQ7q42l9GrkzChlNn9 GWs51fUKUAMaGI2TVyeuOXulgGm3wVx0K+Is4J5933I8tHIMbbZgBVsrPXSMKxVw+uehj4 cwaaHCK48aegKpGHi7J4BD6s6nmWrz4= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=none; spf=pass (imf16.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.118 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R151e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045192;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=8;SR=0;TI=SMTPD_---0Vmo.NB4_1688719910; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0Vmo.NB4_1688719910) by smtp.aliyun-inc.com; Fri, 07 Jul 2023 16:51:51 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: mgorman@techsingularity.net, vbabka@suse.cz, david@redhat.com, ying.huang@intel.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/2] mm: compaction: skip the memory hole rapidly when isolating free pages Date: Fri, 7 Jul 2023 16:51:47 +0800 Message-Id: X-Mailer: git-send-email 2.39.3 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 018F8180016 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: egggzq8jebw5h8cd7mxujepuf8hrw3nc X-HE-Tag: 1688719914-403927 X-HE-Meta: U2FsdGVkX18QzjeevEAM8TF8z4GFLMz7Ud0C02rke3cK34WrEUVXvqorp6f1ljS06hnWJftHqR3J/9oCM1nnILQ+X3SnHCMLKuMxGVe2SQnS4Odie5kATo0FRqVeSfshuL4JfaoQKrokYihLe6/6HQCeDxx63wAEZIDmpbCVR9cK1Hg1coFbSroMj6pKlYNHoNoAsI263eFPQPx+64YkZuvf2lF7tt/IYEYz7fR43MJPXhojUPnldez6/NBN7iXvc+Oc6A35ImqvInX4hbTmOOeXAkOFz8JKCdUDSNEQW7ckaiIuMy63QNLLXoLrC9fd5Fpk+omA+Po/08veEjafpAoMnLkYJIrdcytaY0uzS8kEz94Ntmo2+a1R74zh7aAy/wEiwi5PLdYW0616YJf3AytdxBFrxyZCd3ujcOElynb9xOOz+blrvXIMCTMzne2heqa4lq1KckSVlltzmWW1gB7l7rS9XRpSOWBiBVqrj0RhV9VCY4SbWzu4FQA3zbq0ap+ih+Ew/tk6za7zh10pnZUg5viWKedcKjFo9RZnY8y4PmJwzk7cty01xlCF9YLdVc5MwwH5ehTr7lD94r3iRZxEjeYHANpWI8niNiriXqfyG7BA86AXtZbq/wRRZ/MVnMN/LWoF+2aix8iqBdRiFCDHQcAqSGZ+f26QEEJQbpVj/T1HtfVIHXNG5fEQDlD73W9IeaWnakQtAq3Xezuq7LICpJpCTvEFyWYmQDihoHOKtH7PwFiI9bVlscx8FFATsICr/zuNmz+XCbPEYnpjtCe4uyYgBEmzIHLjOezbnnimrkoWjlVxPCxP8qroWJydYebaL8E5DItpgTCWFBTM3cFQxZ7eQtKJ7Ueil8nODfRlziGy6gLW+lItbqBdbOV08IR5J91IGSYevPQOa/QPTLP7RjI+6aUKm+RWgiuyO+nsW0hdmW0zUN3DlmBoyK0zRd7w6JOx7CPbENNFDjW OCBeJ7Ad X96jrrAyftdQNn2ZQ3P4vhiGO+A2Ak0hxYO5+FLvkw80nWnH2dH3Fv8pVkQ5e2vYOpo3gfgNb9Bo4JNQDrDp1xeCXqXwjOaosrqRc X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On my machine with below memory layout, and I can see it will take more time to skip the larger memory hole (range: 0x100000000 - 0x1800000000) when isolating free pages. So adding a new helper to skip the memory hole rapidly, which can reduce the time consumed from about 70us to less than 1us. [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000040000000-0x00000000ffffffff] [ 0.000000] DMA32 empty [ 0.000000] Normal [mem 0x0000000100000000-0x0000001fa7ffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000040000000-0x0000000fffffffff] [ 0.000000] node 0: [mem 0x0000001800000000-0x0000001fa3c7ffff] [ 0.000000] node 0: [mem 0x0000001fa3c80000-0x0000001fa3ffffff] [ 0.000000] node 0: [mem 0x0000001fa4000000-0x0000001fa402ffff] [ 0.000000] node 0: [mem 0x0000001fa4030000-0x0000001fa40effff] [ 0.000000] node 0: [mem 0x0000001fa40f0000-0x0000001fa73cffff] [ 0.000000] node 0: [mem 0x0000001fa73d0000-0x0000001fa745ffff] [ 0.000000] node 0: [mem 0x0000001fa7460000-0x0000001fa746ffff] [ 0.000000] node 0: [mem 0x0000001fa7470000-0x0000001fa758ffff] [ 0.000000] node 0: [mem 0x0000001fa7590000-0x0000001fa7ffffff] Signed-off-by: Baolin Wang Acked-by: David Hildenbrand Reviewed-by: "Huang, Ying" --- mm/compaction.c | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/mm/compaction.c b/mm/compaction.c index 43358efdbdc2..9641e2131901 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -249,11 +249,31 @@ static unsigned long skip_offline_sections(unsigned long start_pfn) return 0; } + +static unsigned long skip_offline_sections_reverse(unsigned long start_pfn) +{ + unsigned long start_nr = pfn_to_section_nr(start_pfn); + + if (!start_nr || online_section_nr(start_nr)) + return 0; + + while (start_nr-- > 0) { + if (online_section_nr(start_nr)) + return section_nr_to_pfn(start_nr) + PAGES_PER_SECTION - 1; + } + + return 0; +} #else static unsigned long skip_offline_sections(unsigned long start_pfn) { return 0; } + +static unsigned long skip_offline_sections_reverse(unsigned long start_pfn) +{ + return 0; +} #endif /* @@ -1668,8 +1688,16 @@ static void isolate_freepages(struct compact_control *cc) page = pageblock_pfn_to_page(block_start_pfn, block_end_pfn, zone); - if (!page) + if (!page) { + unsigned long next_pfn; + + next_pfn = skip_offline_sections_reverse(block_start_pfn); + if (next_pfn) + block_start_pfn = max(pageblock_start_pfn(next_pfn), + low_pfn); + continue; + } /* Check the block is suitable for migration */ if (!suitable_migration_target(cc, page))