From patchwork Fri Jan 18 17:51:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771393 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DEA6A91E for ; Fri, 18 Jan 2019 17:55:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CE4E42FF95 for ; Fri, 18 Jan 2019 17:55:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C26AE2FFCC; Fri, 18 Jan 2019 17:55:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1E1832FF95 for ; Fri, 18 Jan 2019 17:55:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B8658E0024; Fri, 18 Jan 2019 12:55:23 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 167E28E0002; Fri, 18 Jan 2019 12:55:23 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 009FA8E0024; Fri, 18 Jan 2019 12:55:22 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 96A398E0002 for ; Fri, 18 Jan 2019 12:55:22 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id f17so5240942edm.20 for ; Fri, 18 Jan 2019 09:55:22 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=17tWphiIoHiPa1vvYb3H3FZPlbycDfBU7EmArSPipfw=; b=nKPQzbB8Y3LpnZk3SSxQxbHAPG3DYw0osTZ5d2no+IyJZYwd6gqgBcPsBekFdnQYGN SPy7VxDOeKY6FHZmPQOALbc18x4izyEi5NIwq6f3d3sSph+rbCjQssIolOsaO+RRGR+A Dtt9DasA/HJWKaFVgtHsU+6gRURl/N6pajNw7MI75C33eHYXT/wJM5Q9DwW5M76ROUO9 PFsOBqi1bAsXVkGS+s89XqLOiGKlsP+DN5pdFsHxg1AQD09UWIWawtVLcACr2Bq/Zqz0 eQoiBFNqlEyAvtm5+2O+MWut+f9+oUfKASKyVGFCdFKfTlVWzfdKSvVXqCjjhaT0otE4 48ww== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukcU+1+PquNwvzaSQLukzErfzUHBZLKD5V260OGNT3dqJV1hBK4f oNxkA0ms6dPMPcixho0Y/30j8Tdui1Jt5k9bfkjl6l9QUKlBXfvwgO+3XP/FodOb/tgfcFapQ3w 6bEZh8cce588ArUOtJc32MQGC7CvLu+maKMvZ9qTe/IOe5Xj9UpN8PaM104qps1VTAQ== X-Received: by 2002:aa7:ccc8:: with SMTP id y8mr15949278edt.118.1547834122143; Fri, 18 Jan 2019 09:55:22 -0800 (PST) X-Google-Smtp-Source: ALg8bN4mabAnFjPn9BrlrhyrcRrqmtJJ9VY6aPqEPzqyEJP9u8GDT5QA2e0GPuSLhOfJXyglUPBc X-Received: by 2002:aa7:ccc8:: with SMTP id y8mr15949220edt.118.1547834120922; Fri, 18 Jan 2019 09:55:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547834120; cv=none; d=google.com; s=arc-20160816; b=wjlV1qv/f4qZaX4R9gAZcBqNhGPU9nUp4AJdXc5ehCQAushpz0pId5RopVK0PkEBwd 8M2fzRH8ZNpG0hMZbs4YVjJ0CLDf2uBLhcpko595WmhaEbpNz2vEezAlzNFD6XdPPlp/ vjLvBcYrWPXyJFBnv03RZiqLVP0R/gNNU92CDrtO1bSXd+l/+Bnd2fWZb8NQY4DIOD4+ fdlr6FXZw15GPb1puxf7HSG01jzaKKCcysuoKF5FxElnJRxArYs+Edc2+hT7vaYiiCdV nYq3AK+gmmFJv+uY+RUhxdrL0cEdOGDU5GA6kAIIzdT3bQNxolT43tV5yXeyxbbp9CLV fhAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=17tWphiIoHiPa1vvYb3H3FZPlbycDfBU7EmArSPipfw=; b=r+52+erMclaGxY72Om5YisXDSTon5fU3Eg+R7qWY2y4Jl1vuzNnIrhM7zVeXpRRsXc zio9pbggWBmGez5/7YX1i5qkfNwRZfOunB6SjOq5SBGdK3teJ7a7S/FOWtJQLN/Yh9CO XG1+omRNZ7kcm94UmAcFM0LQx2dZQDAlpn6Gnk4+64YCI30AJcInP5u8lved7Sux/Cg8 BAkMs76hUn3UxJQNK0lRtvMJVD93UXC9Ha6+ma0wx1AAvbybDhzM1XDwCbeMBv7Ss4wx 4Oul4J9fELNlK2VQ1ckpfbTF6XBBYMbVC/I1e/l7cCaPBN2UrMv7fZSKG2O2ca8YC470 UYfQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp26.blacknight.com (outbound-smtp26.blacknight.com. [81.17.249.194]) by mx.google.com with ESMTPS id w58si5438430edc.162.2019.01.18.09.55.20 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:55:20 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) client-ip=81.17.249.194; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp26.blacknight.com (Postfix) with ESMTPS id 97DFDB8A73 for ; Fri, 18 Jan 2019 17:55:20 +0000 (GMT) Received: (qmail 8547 invoked from network); 18 Jan 2019 17:55:20 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:55:20 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 21/22] mm, compaction: Be selective about what pageblocks to clear skip hints Date: Fri, 18 Jan 2019 17:51:35 +0000 Message-Id: <20190118175136.31341-22-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Pageblock hints are cleared when compaction restarts or kswapd makes enough progress that it can sleep but it's over-eager in that the bit is cleared for migration sources with no LRU pages and migration targets with no free pages. As pageblock skip hint flushes are relatively rare and out-of-band with respect to kswapd, this patch makes a few more expensive checks to see if it's appropriate to even clear the bit. Every pageblock that is not cleared will avoid 512 pages being scanned unnecessarily on x86-64. The impact is variable with different workloads showing small differences in latency, success rates and scan rates. This is expected as clearing the hints is not that common but doing a small amount of work out-of-band to avoid a large amount of work in-band later is generally a good thing. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- include/linux/mmzone.h | 2 + mm/compaction.c | 125 ++++++++++++++++++++++++++++++++++++++++++------- 2 files changed, 109 insertions(+), 18 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 842f9189537b..90c13cdeefb5 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -480,6 +480,8 @@ struct zone { unsigned long compact_cached_free_pfn; /* pfn where async and sync compaction migration scanner should start */ unsigned long compact_cached_migrate_pfn[2]; + unsigned long compact_init_migrate_pfn; + unsigned long compact_init_free_pfn; #endif #ifdef CONFIG_COMPACTION diff --git a/mm/compaction.c b/mm/compaction.c index 74bf620e3dcd..de558f110319 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -237,6 +237,71 @@ static bool pageblock_skip_persistent(struct page *page) return false; } +static bool +__reset_isolation_pfn(struct zone *zone, unsigned long pfn, bool check_source, + bool check_target) +{ + struct page *page = pfn_to_online_page(pfn); + struct page *end_page; + unsigned long block_pfn; + + if (!page) + return false; + if (zone != page_zone(page)) + return false; + if (pageblock_skip_persistent(page)) + return false; + + /* + * If skip is already cleared do no further checking once the + * restart points have been set. + */ + if (check_source && check_target && !get_pageblock_skip(page)) + return true; + + /* + * If clearing skip for the target scanner, do not select a + * non-movable pageblock as the starting point. + */ + if (!check_source && check_target && + get_pageblock_migratetype(page) != MIGRATE_MOVABLE) + return false; + + /* + * Only clear the hint if a sample indicates there is either a + * free page or an LRU page in the block. One or other condition + * is necessary for the block to be a migration source/target. + */ + block_pfn = pageblock_start_pfn(pfn); + pfn = max(block_pfn, zone->zone_start_pfn); + page = pfn_to_page(pfn); + if (zone != page_zone(page)) + return false; + pfn = block_pfn + pageblock_nr_pages; + pfn = min(pfn, zone_end_pfn(zone)); + end_page = pfn_to_page(pfn); + + do { + if (!pfn_valid_within(pfn)) + continue; + + if (check_source && PageLRU(page)) { + clear_pageblock_skip(page); + return true; + } + + if (check_target && PageBuddy(page)) { + clear_pageblock_skip(page); + return true; + } + + page += (1 << PAGE_ALLOC_COSTLY_ORDER); + pfn += (1 << PAGE_ALLOC_COSTLY_ORDER); + } while (page < end_page); + + return false; +} + /* * This function is called to clear all cached information on pageblocks that * should be skipped for page isolation when the migrate and free page scanner @@ -244,30 +309,54 @@ static bool pageblock_skip_persistent(struct page *page) */ static void __reset_isolation_suitable(struct zone *zone) { - unsigned long start_pfn = zone->zone_start_pfn; - unsigned long end_pfn = zone_end_pfn(zone); - unsigned long pfn; + unsigned long migrate_pfn = zone->zone_start_pfn; + unsigned long free_pfn = zone_end_pfn(zone); + unsigned long reset_migrate = free_pfn; + unsigned long reset_free = migrate_pfn; + bool source_set = false; + bool free_set = false; - zone->compact_blockskip_flush = false; + if (!zone->compact_blockskip_flush) + return; - /* Walk the zone and mark every pageblock as suitable for isolation */ - for (pfn = start_pfn; pfn < end_pfn; pfn += pageblock_nr_pages) { - struct page *page; + zone->compact_blockskip_flush = false; + /* + * Walk the zone and update pageblock skip information. Source looks + * for PageLRU while target looks for PageBuddy. When the scanner + * is found, both PageBuddy and PageLRU are checked as the pageblock + * is suitable as both source and target. + */ + for (; migrate_pfn < free_pfn; migrate_pfn += pageblock_nr_pages, + free_pfn -= pageblock_nr_pages) { cond_resched(); - page = pfn_to_online_page(pfn); - if (!page) - continue; - if (zone != page_zone(page)) - continue; - if (pageblock_skip_persistent(page)) - continue; + /* Update the migrate PFN */ + if (__reset_isolation_pfn(zone, migrate_pfn, true, source_set) && + migrate_pfn < reset_migrate) { + source_set = true; + reset_migrate = migrate_pfn; + zone->compact_init_migrate_pfn = reset_migrate; + zone->compact_cached_migrate_pfn[0] = reset_migrate; + zone->compact_cached_migrate_pfn[1] = reset_migrate; + } - clear_pageblock_skip(page); + /* Update the free PFN */ + if (__reset_isolation_pfn(zone, free_pfn, free_set, true) && + free_pfn > reset_free) { + free_set = true; + reset_free = free_pfn; + zone->compact_init_free_pfn = reset_free; + zone->compact_cached_free_pfn = reset_free; + } } - reset_cached_positions(zone); + /* Leave no distance if no suitable block was reset */ + if (reset_migrate >= reset_free) { + zone->compact_cached_migrate_pfn[0] = migrate_pfn; + zone->compact_cached_migrate_pfn[1] = migrate_pfn; + zone->compact_cached_free_pfn = free_pfn; + } } void reset_isolation_suitable(pg_data_t *pgdat) @@ -1190,7 +1279,7 @@ fast_isolate_freepages(struct compact_control *cc) * If starting the scan, use a deeper search and use the highest * PFN found if a suitable one is not found. */ - if (cc->free_pfn == pageblock_start_pfn(zone_end_pfn(cc->zone) - 1)) { + if (cc->free_pfn >= cc->zone->compact_init_free_pfn) { limit = pageblock_nr_pages >> 1; scan_start = true; } @@ -2015,7 +2104,7 @@ static enum compact_result compact_zone(struct compact_control *cc) cc->zone->compact_cached_migrate_pfn[1] = cc->migrate_pfn; } - if (cc->migrate_pfn == start_pfn) + if (cc->migrate_pfn <= cc->zone->compact_init_migrate_pfn) cc->whole_zone = true; }