From patchwork Fri Jan 18 17:51:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771387 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 57FFF91E for ; Fri, 18 Jan 2019 17:54:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 46D532FF95 for ; Fri, 18 Jan 2019 17:54:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 39F682FFCC; Fri, 18 Jan 2019 17:54:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 883A02FF95 for ; Fri, 18 Jan 2019 17:54:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7AECE8E0021; Fri, 18 Jan 2019 12:54:52 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 75B128E0002; Fri, 18 Jan 2019 12:54:52 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 64ABA8E0021; Fri, 18 Jan 2019 12:54:52 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 018108E0002 for ; Fri, 18 Jan 2019 12:54:52 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id z10so5254853edz.15 for ; Fri, 18 Jan 2019 09:54:51 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=ocNhlG4rMc1hcPy/Atf76NV7+izKch3aoerr5YAXBbA=; b=BPZzxsa6EAMXwwoDxZ4s7LhqT/UdHlug+xjJwzDE4orMoMiKIXnu9Q0kBPGG3VqObE bEObz4Fm4RpVzaku1jSjOQx0AXIOYS0kjKzMDQkr56teNdFgOmzBXfanFDugn7MyVelm oHA6MnVElcAGcN0f2kcNYEpLf03U5Up+ya6xufhL9Ud+qMISmyPqYPWgywOQw2Gvxu0T 0/Zzz/CnFXVIVx9s1DduaPt01BVHu21idkZXKhGRWhhfsKVwkCg2ElMyTMcq+xClXN1S pcNY+93i3hoCPOBujIzIzm/YV16AIu00rBNOjFQZIWLlqjqApPjrx5RbbXlvRgooNTnf ZboQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukf6Gi/iv00NwF4/aO1ffDBYlIvWudp8OZqTKeFajIHD1YhMn6Fj hQo0x8P0v/Pet0R0gYYJK31RtXAQoEY0IzAIFiWniaccoeBexxUqsQcXoKCT/4MsDRikOF8h7jz L8MBc5jtSMLTGrbKK5teaFKfvoj0YD1clsmClS1nXMRFIhM8jm+7NoGU2nF/shY4P3w== X-Received: by 2002:a50:983a:: with SMTP id g55mr16903764edb.143.1547834091476; Fri, 18 Jan 2019 09:54:51 -0800 (PST) X-Google-Smtp-Source: ALg8bN5nSRtQ2GKwBLajYV4LJVfJkdBEemUrYmscI+keZhrNXo6M9W6/asagEiooC6+uwB+BOuWN X-Received: by 2002:a50:983a:: with SMTP id g55mr16903703edb.143.1547834090412; Fri, 18 Jan 2019 09:54:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547834090; cv=none; d=google.com; s=arc-20160816; b=o7f+NOf2tkMsQdYWxuAQczHnuCUGT89/EyKulG5moyhtTagh/37YymnY2WPhbDKA6V HDS60NQQRBiBQdV0rlTJkLoEaRswtvEeR8XPWFCkvUlcsEiz6C+UKht+C6WuWR5Yoc1W of7rBRs9NdgHm6VqB/i0moisZ0D8hkg0LKVOd4RNt42no/UfMXCyfy0H26xcKbHorJ/t KrrrxXy5a/Dmach6Fns5/VlZPtfuT8OkQe0+kNtjl5QuEAAB7srYi5jfwOQ37MLvrYmQ djgtDszxPC//lc95kZZyT65Nb2vfnEaJ1MbbXqGrMZYc5jTjSJN5DiDOe9K8jfdgZjyZ EPOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=ocNhlG4rMc1hcPy/Atf76NV7+izKch3aoerr5YAXBbA=; b=VIo6Cy/zH5j8Goq6T7suQlZkdj5shArtE9IIpFpf86W2FbdmssGGb/ri7nl53jWb8B OGVKmlXZsy5UrfCKs8kle+X4jvfIfTy4e38y3fvKARLNryt78evG4z1jGuJ460vweKWl 3uAAPpBT/vMuXZjdBDKzEwiqj/Zi2kniJO3AbnijD8JAO9vnNN+hU+xkThfrlAQ2vnYF 4Z5ev0SiDGEC4qSgoL1NR5+k3au1XfYnl0QUMnYN9np10ZUvXOvTNkB1i2HR3+7sL6r+ RWgvVzn+3f/oFNJ/vVSkX7nN7SzEIsaJ+5657jYlWBTub/ny0kxlD5KLYuNLAtde0y9u DZ4A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp02.blacknight.com (outbound-smtp02.blacknight.com. [81.17.249.8]) by mx.google.com with ESMTPS id w11si2420117edl.223.2019.01.18.09.54.50 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:54:50 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) client-ip=81.17.249.8; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp02.blacknight.com (Postfix) with ESMTPS id 1BE1D98BD2 for ; Fri, 18 Jan 2019 17:54:50 +0000 (UTC) Received: (qmail 7288 invoked from network); 18 Jan 2019 17:54:50 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:54:50 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 18/22] mm, compaction: Reduce premature advancement of the migration target scanner Date: Fri, 18 Jan 2019 17:51:32 +0000 Message-Id: <20190118175136.31341-19-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The fast isolation of free pages allows the cached PFN of the free scanner to advance faster than necessary depending on the contents of the free list. The key is that fast_isolate_freepages() can update zone->compact_cached_free_pfn via isolate_freepages_block(). When the fast search fails, the linear scan can start from a point that has skipped valid migration targets, particularly pageblocks with just low-order free pages. This can cause the migration source/target scanners to meet prematurely causing a reset. This patch starts by avoiding an update of the pageblock skip information and cached PFN from isolate_freepages_block() and puts the responsibility of updating that information in the callers. The fast scanner will update the cached PFN if and only if it finds a block that is higher than the existing cached PFN and sets the skip if the pageblock is full or nearly full. The linear scanner will update skipped information and the cached PFN only when a block is completely scanned. The total impact is that the free scanner advances more slowly as it is primarily driven by the linear scanner instead of the fast search. 5.0.0-rc1 5.0.0-rc1 noresched-v3r17 slowfree-v3r17 Amean fault-both-3 2965.68 ( 0.00%) 3036.75 ( -2.40%) Amean fault-both-5 3995.90 ( 0.00%) 4522.24 * -13.17%* Amean fault-both-7 5842.12 ( 0.00%) 6365.35 ( -8.96%) Amean fault-both-12 9550.87 ( 0.00%) 10340.93 ( -8.27%) Amean fault-both-18 13304.72 ( 0.00%) 14732.46 ( -10.73%) Amean fault-both-24 14618.59 ( 0.00%) 16288.96 ( -11.43%) Amean fault-both-30 16650.96 ( 0.00%) 16346.21 ( 1.83%) Amean fault-both-32 17145.15 ( 0.00%) 19317.49 ( -12.67%) The impact to latency is higher than the last version but it appears to be due to a slight increase in the free scan rates which is a potential side-effect of the patch. However, this is necessary for later patches that are more careful about how pageblocks are treated as earlier iterations of those patches hit corner cases where the restarts were punishing and very visible. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 27 ++++++++++----------------- 1 file changed, 10 insertions(+), 17 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 293d9a9e6f00..04ec7d4da719 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -330,10 +330,9 @@ static void update_cached_migrate(struct compact_control *cc, unsigned long pfn) * future. The information is later cleared by __reset_isolation_suitable(). */ static void update_pageblock_skip(struct compact_control *cc, - struct page *page, unsigned long nr_isolated) + struct page *page, unsigned long pfn) { struct zone *zone = cc->zone; - unsigned long pfn; if (cc->no_set_skip_hint) return; @@ -341,13 +340,8 @@ static void update_pageblock_skip(struct compact_control *cc, if (!page) return; - if (nr_isolated) - return; - set_pageblock_skip(page); - pfn = page_to_pfn(page); - /* Update where async and sync compaction should restart */ if (pfn < zone->compact_cached_free_pfn) zone->compact_cached_free_pfn = pfn; @@ -365,7 +359,7 @@ static inline bool pageblock_skip_persistent(struct page *page) } static inline void update_pageblock_skip(struct compact_control *cc, - struct page *page, unsigned long nr_isolated) + struct page *page, unsigned long pfn) { } @@ -449,7 +443,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, bool strict) { int nr_scanned = 0, total_isolated = 0; - struct page *cursor, *valid_page = NULL; + struct page *cursor; unsigned long flags = 0; bool locked = false; unsigned long blockpfn = *start_pfn; @@ -476,9 +470,6 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, if (!pfn_valid_within(blockpfn)) goto isolate_fail; - if (!valid_page) - valid_page = page; - /* * For compound pages such as THP and hugetlbfs, we can save * potentially a lot of iterations if we skip them at once. @@ -566,10 +557,6 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, if (strict && blockpfn < end_pfn) total_isolated = 0; - /* Update the pageblock-skip if the whole pageblock was scanned */ - if (blockpfn == end_pfn) - update_pageblock_skip(cc, valid_page, total_isolated); - cc->total_free_scanned += nr_scanned; if (total_isolated) count_compact_events(COMPACTISOLATED, total_isolated); @@ -1293,8 +1280,10 @@ fast_isolate_freepages(struct compact_control *cc) } } - if (highest && highest > cc->zone->compact_cached_free_pfn) + if (highest && highest >= cc->zone->compact_cached_free_pfn) { + highest -= pageblock_nr_pages; cc->zone->compact_cached_free_pfn = highest; + } cc->total_free_scanned += nr_scanned; if (!page) @@ -1374,6 +1363,10 @@ static void isolate_freepages(struct compact_control *cc) isolate_freepages_block(cc, &isolate_start_pfn, block_end_pfn, freelist, false); + /* Update the skip hint if the full pageblock was scanned */ + if (isolate_start_pfn == block_end_pfn) + update_pageblock_skip(cc, page, block_start_pfn); + /* Are enough freepages isolated? */ if (cc->nr_freepages >= cc->nr_migratepages) { if (isolate_start_pfn >= block_end_pfn) {