From patchwork Wed Jun 14 08:40:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13279724 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73666EB64DA for ; Wed, 14 Jun 2023 08:40:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AE4576B0074; Wed, 14 Jun 2023 04:40:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A95A56B0075; Wed, 14 Jun 2023 04:40:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 983678E0002; Wed, 14 Jun 2023 04:40:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 857576B0074 for ; Wed, 14 Jun 2023 04:40:36 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 32A711C8861 for ; Wed, 14 Jun 2023 08:40:36 +0000 (UTC) X-FDA: 80900707272.21.3AFBFE2 Received: from out30-119.freemail.mail.aliyun.com (out30-119.freemail.mail.aliyun.com [115.124.30.119]) by imf26.hostedemail.com (Postfix) with ESMTP id 29ABC14000C for ; Wed, 14 Jun 2023 08:40:32 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=none; spf=pass (imf26.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.119 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686732034; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references; bh=anwy2nJenaZMEVPT17e8ThGgvVIZ680A3/ngu4eVbmA=; b=f9LPfN6iDhPVFSPUPdPpW8DMtf3AgqEw57gsqLIbevb5KHmdZp4IeNXanRK5EaKqzZrcoa DJumRA2s4vgXE2JHGHGxCUB3S6F21ZuRk7JFk+wuTCaTjNlhWiwpkg6YiKX88QeG90wYAW ZmBfQ3eTVobPgQcFPtiyILqold2FxAA= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=none; spf=pass (imf26.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.119 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686732034; a=rsa-sha256; cv=none; b=TXoHpQMwnH/Py0kCEpF7afUA+8NtPbKqfreyGTfajqsgKlbkOaKIp/gRL8CXLdVVNb+OqH z4jLW6bMQkUpsm4x9lZCnUqcoa8Hh/MP1J8m4DCJPxCvxQZFO6LbZbvyod83DGvzukyWmV sw7x0h9B04GNGUs3Q8kU0Z2SFzyoSxo= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R121e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046049;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=8;SR=0;TI=SMTPD_---0Vl5okOP_1686732027; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0Vl5okOP_1686732027) by smtp.aliyun-inc.com; Wed, 14 Jun 2023 16:40:28 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: mgorman@techsingularity.net, vbabka@suse.cz, david@redhat.com, ying.huang@intel.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3] mm: compaction: skip memory hole rapidly when isolating migratable pages Date: Wed, 14 Jun 2023 16:40:20 +0800 Message-Id: <75b4c8ca36bf44ad8c42bf0685ac19d272e426ec.1686705221.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 X-Rspamd-Queue-Id: 29ABC14000C X-Rspam-User: X-Stat-Signature: egpc7odk4nxqdene71ko8o768gp3djxu X-Rspamd-Server: rspam01 X-HE-Tag: 1686732032-623980 X-HE-Meta: U2FsdGVkX1+o8Aj2tDyDMGYT0wjMbjzKs4S71sGXaH/JpwJ5hLKzoAsRKPwTy7BQdO9X8b+30m0J22RhAYcaX8b803MD/Cqhp0w9D53sEYnQRXxMcsByMvbrUhbMRJM1tZitahms2YjDfDWLmSF0RORVvcwStBWAvjiD0Xa+WV+aXOYRyDz63zRblK9MPP9T3VrBSOaHisQeIONRY2XYmJgG9Th377SunTJ7SZ1URBIv4naZz9UlHY1I83MBdI5RBJE59jL8qbJVbr6FVDEx36S3u6gksStDyAsjN1iAJvQ6mL8Vh3KEvJ/2TEeWV+FmQ6RWwqAo1aqK8eEH4UyD6HNgeIWVQdywtcThJToMyDvUVBkxuID6JEMpZ9vSpgBjgxiHpF+mwSk2KNDBPw96X8XlsbTNNbxg9/tSUXzrIHB5PwZe4FQXyNQGDA6PeG2cXe0joP/HaqpPld6ku3DrPh2yiaLQRRX9YpTC+fNk61RLu9YUglWBXVjiMIiBOQpKUMI7z7K0kXaUi0MYlXD7trw0uv4Kf7Gf2LpzLiRf39r77zRZD02ywnO5wdqveKLMFvx2LUcReM7a/Mr1z4SBLIos3v42Noo00wiQLMy9k6rM0iIDaoKFQf85ubMwEp7k54AClz/wcsi0WNnLtiNKWIPtLsQE03o8ROIW68n3k/pn51WnxNDc5MAzi7/e1M2bByvJ6UaFR0qbrhSgCP9LBEIhlAsBQsIrIqscWH18Yzq560zIaO6Wfnqi34SUjKkMF8wPNe9VAlX2doK2rRPGEi9CZIrevrdSVi3MxI+Sv1OR7J7+7PemeMIKGaGTMutLbYhtJy2vvqwhWNcObh+WgxVx2XT9PM3Z/3q4aVfnE1Ny/MWfg8J4Yv5bGfEJg5tqHGS8YDYC8wmuAc9WoKO0EKJFQQzval6xEDi0wsKThSmP9a/j2Ue463zJoDdyuvqb8ycU56Wel/e3nQ+8ws6 aTAIwFe0 dQlVD06TLfr9asjiygi1qgBcsq2XC9jkCqZicUbHb4jOPAw7W/YMAPOqwXjo0U9moY7bVc5Cq2vBDyW6dDAPYdwVRoVCSDjTwTvCbRqrr5PbXJN+x1FkeWaBcvdk2GLXwqpaQh6I3TlhhvNUfXl1bafp6dBmed9Yy9GWc+I+GdwwqOQfHciGiiTvSBw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On some machines, the normal zone can have a large memory hole like below memory layout, and we can see the range from 0x100000000 to 0x1800000000 is a hole. So when isolating some migratable pages, the scanner can meet the hole and it will take more time to skip the large hole. From my measurement, I can see the isolation scanner will take 80us ~ 100us to skip the large hole [0x100000000 - 0x1800000000]. So adding a new helper to fast search next online memory section to skip the large hole can help to find next suitable pageblock efficiently. With this patch, I can see the large hole scanning only takes < 1us. [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000040000000-0x00000000ffffffff] [ 0.000000] DMA32 empty [ 0.000000] Normal [mem 0x0000000100000000-0x0000001fa7ffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000040000000-0x0000000fffffffff] [ 0.000000] node 0: [mem 0x0000001800000000-0x0000001fa3c7ffff] [ 0.000000] node 0: [mem 0x0000001fa3c80000-0x0000001fa3ffffff] [ 0.000000] node 0: [mem 0x0000001fa4000000-0x0000001fa402ffff] [ 0.000000] node 0: [mem 0x0000001fa4030000-0x0000001fa40effff] [ 0.000000] node 0: [mem 0x0000001fa40f0000-0x0000001fa73cffff] [ 0.000000] node 0: [mem 0x0000001fa73d0000-0x0000001fa745ffff] [ 0.000000] node 0: [mem 0x0000001fa7460000-0x0000001fa746ffff] [ 0.000000] node 0: [mem 0x0000001fa7470000-0x0000001fa758ffff] [ 0.000000] node 0: [mem 0x0000001fa7590000-0x0000001fa7ffffff] Suggested-by: David Hildenbrand Signed-off-by: Baolin Wang Acked-by: David Hildenbrand Acked-by: "Huang, Ying" --- Changes from v2: - Simplify the function suggested by David. - Collect acked tag from David and Ying. Thanks. Changes from v1: - Fix building errors if CONFIG_SPARSEMEM is not selected. - Use NR_MEM_SECTIONS instead of '-1' per Huang Ying. --- mm/compaction.c | 35 ++++++++++++++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/mm/compaction.c b/mm/compaction.c index 3398ef3a55fe..0b2680b075fd 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -229,6 +229,33 @@ static void reset_cached_positions(struct zone *zone) pageblock_start_pfn(zone_end_pfn(zone) - 1); } +#ifdef CONFIG_SPARSEMEM +/* + * If the PFN falls into an offline section, return the start PFN of the + * next online section. If the PFN falls into an online section or if + * there is no next online section, return 0. + */ +static unsigned long skip_offline_sections(unsigned long start_pfn) +{ + unsigned long start_nr = pfn_to_section_nr(start_pfn); + + if (online_section_nr(start_nr)) + return 0; + + while (++start_nr <= __highest_present_section_nr) { + if (online_section_nr(start_nr)) + return section_nr_to_pfn(start_nr); + } + + return 0; +} +#else +static unsigned long skip_offline_sections(unsigned long start_pfn) +{ + return 0; +} +#endif + /* * Compound pages of >= pageblock_order should consistently be skipped until * released. It is always pointless to compact pages of such order (if they are @@ -1991,8 +2018,14 @@ static isolate_migrate_t isolate_migratepages(struct compact_control *cc) page = pageblock_pfn_to_page(block_start_pfn, block_end_pfn, cc->zone); - if (!page) + if (!page) { + unsigned long next_pfn; + + next_pfn = skip_offline_sections(block_start_pfn); + if (next_pfn) + block_end_pfn = next_pfn; continue; + } /* * If isolation recently failed, do not retry. Only check the