From patchwork Fri Jan 4 12:50:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748341 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3F22814E5 for ; Fri, 4 Jan 2019 12:54:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2E3A92810E for ; Fri, 4 Jan 2019 12:54:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1E7A62836F; Fri, 4 Jan 2019 12:54:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 909E62810E for ; Fri, 4 Jan 2019 12:53:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9ED1D8E00E7; Fri, 4 Jan 2019 07:53:58 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 99BEA8E00AE; Fri, 4 Jan 2019 07:53:58 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 88B718E00E7; Fri, 4 Jan 2019 07:53:58 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id 2B0078E00AE for ; Fri, 4 Jan 2019 07:53:58 -0500 (EST) Received: by mail-ed1-f69.google.com with SMTP id c3so35320440eda.3 for ; Fri, 04 Jan 2019 04:53:58 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=7NMy9m+YgJH0m/HLzWMXEowcaGcqNh8354CNSDiq+Cs=; b=WWfJP/uWwYSVcu/kPCr1ByHiS/ezCds/YJ0Q8Ggc1Hc80YP0FDTJux3xIwgQIkaLpC m750wxjiTfHEF6hwiij6+bN7JlTdMa5dX8cRS3Xl+aY/E58JkkUuVc9X74Clzux5GXJU zEyVJTmsQIqH0135lFFoC7l5llyQnkHBW1vlr2KksC1deOzcKIlSx4PP3xXFsPrYFa9m H5X/K+r6oLDDVmJnPV+ducHi1nYNIE1N2dNhaP6tTMROMqVi3aVYiTfWeYra53uiEbWB 3pjPDjcY6shYlP2QNXuR30QHZXJrOi5jW6sRBv5aWTMJ+bjj3xW8SKqu13NuYNz/Ybe1 4T5A== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWYYrlpUzIln/FA1akLq14g61LLQYh375E/0ZZfpUObOAuEaB69Z Iy91xb2LkRl3PIy2KZ9tWgL9vXtwjSOr9AkTfvT7WiRmNXJ1kJ3kvqv8rbEsbYuDIdks8TPpP4A 8JtpX80hOjzLKHusOkxPuF7jMVhcS6rT6ucnEXZHbf0Dh+ptzhSnkD4CStjAZtNrqKA== X-Received: by 2002:aa7:d29a:: with SMTP id w26mr46997385edq.30.1546606437624; Fri, 04 Jan 2019 04:53:57 -0800 (PST) X-Google-Smtp-Source: AFSGD/VJ7ZWgNWpR20YpWk8kqhedBmE7scs6rp+N3SHfsDvJxoMOALeV7ODANrSrwoO4OJAp52tY X-Received: by 2002:aa7:d29a:: with SMTP id w26mr46997347edq.30.1546606436655; Fri, 04 Jan 2019 04:53:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606436; cv=none; d=google.com; s=arc-20160816; b=g9kGyyMsO98zcs8kG8dEV9wqQxdF0FW3lxDNvEM+nZcZbkjILwHmYzGHksutMQF79+ 0FTUwU5obU+Kz+5eYPSoaFqT4GIRVxhM+SHpb2RP/gvoAgyk3ZXuEDWPQ9RyVfuocFNE bOfrZD8HgD/WMa5PxNvu2Akkpwlux+xcMiv1WbF2ZOkv7M2QfqnkJe9HKB7G/J1l7flq G8XCP2WxywfJbQA8OPmyIQbPHpgc7jigZJQ1MoL7MvVifdKidSEgW1FoQJKRB9SDUnjD DPIOTca0Rnf+JDht8mAIyZFTb+ZGLNWeltI+6Xi44BhD0kgndybiq4rQ7EE5SgGb36Aw 19Hw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=7NMy9m+YgJH0m/HLzWMXEowcaGcqNh8354CNSDiq+Cs=; b=u82VPZSgZhpqUGAOXr5b2C+bNhvMjhSDwtOFP8QovgbumqhZke1RvQDOunLWmQPkjT HXJRf7f1cQrR0ISWUSdPunMxYEy2TO5Q/k0/8EzbBY1lC0oK3F7CoOV6k2FefS0cNQT+ O8mRxQmp+fQpSrcppP9vSSssx0T5iQjmVWphQNeK/jTUadaXi9O99/ilUd545DZyVZF0 xVUB4B+c075NX5QVfRjfADe/znK+iNiIYTQiGkqO1Md6ljyybX3ikSWiac0GrbF8vaTc W1NT+5URsqqpXUU3gfgkCmCiYNVKA0m4CVNpAYpm9DG/2vvRmUjCDez1a69AvNt7wLNe Rq9A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp08.blacknight.com (outbound-smtp08.blacknight.com. [46.22.139.13]) by mx.google.com with ESMTPS id l16-v6si1298091ejq.174.2019.01.04.04.53.56 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:53:56 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) client-ip=46.22.139.13; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp08.blacknight.com (Postfix) with ESMTPS id 499061C1BE5 for ; Fri, 4 Jan 2019 12:53:56 +0000 (GMT) Received: (qmail 10079 invoked from network); 4 Jan 2019 12:53:56 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:53:56 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 21/25] mm, compaction: Round-robin the order while searching the free lists for a target Date: Fri, 4 Jan 2019 12:50:07 +0000 Message-Id: <20190104125011.16071-22-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP As compaction proceeds and creates high-order blocks, the free list search gets less efficient as the larger blocks are used as compaction targets. Eventually, the larger blocks will be behind the migration scanner for partially migrated pageblocks and the search fails. This patch round-robins what orders are searched so that larger blocks can be ignored and find smaller blocks that can be used as migration targets. The overall impact was small on 1-socket but it avoids corner cases where the migration/free scanners meet prematurely or situations where many of the pageblocks encountered by the free scanner are almost full instead of being properly packed. Previous testing had indicated that without this patch there were occasional large spikes in the free scanner without this patch. By co-incidence, the 2-socket results showed a 54% reduction in the free scanner but will not be universally true. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 33 ++++++++++++++++++++++++++++++--- mm/internal.h | 3 ++- 2 files changed, 32 insertions(+), 4 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 6c5552c6d8f9..652e249168b1 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1154,6 +1154,24 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn, unsigned long set_pageblock_skip(page); } +/* Search orders in round-robin fashion */ +static int next_search_order(struct compact_control *cc, int order) +{ + order--; + if (order < 0) + order = cc->order - 1; + + /* Search wrapped around? */ + if (order == cc->search_order) { + cc->search_order--; + if (cc->search_order < 0) + cc->search_order = cc->order - 1; + return -1; + } + + return order; +} + static unsigned long fast_isolate_freepages(struct compact_control *cc) { @@ -1186,9 +1204,15 @@ fast_isolate_freepages(struct compact_control *cc) if (WARN_ON_ONCE(min_pfn > low_pfn)) low_pfn = min_pfn; - for (order = cc->order - 1; - order >= 0 && !page; - order--) { + /* + * Search starts from the last successful isolation order or the next + * order to search after a previous failure + */ + cc->search_order = min_t(unsigned int, cc->order - 1, cc->search_order); + + for (order = cc->search_order; + !page && order >= 0; + order = next_search_order(cc, order)) { struct free_area *area = &cc->zone->free_area[order]; struct list_head *freelist; struct page *freepage; @@ -1211,6 +1235,7 @@ fast_isolate_freepages(struct compact_control *cc) if (pfn >= low_pfn) { cc->fast_search_fail = 0; + cc->search_order = order; page = freepage; break; } @@ -2146,6 +2171,7 @@ static enum compact_result compact_zone_order(struct zone *zone, int order, .total_migrate_scanned = 0, .total_free_scanned = 0, .order = order, + .search_order = order, .gfp_mask = gfp_mask, .zone = zone, .mode = (prio == COMPACT_PRIO_ASYNC) ? @@ -2385,6 +2411,7 @@ static void kcompactd_do_work(pg_data_t *pgdat) struct zone *zone; struct compact_control cc = { .order = pgdat->kcompactd_max_order, + .search_order = pgdat->kcompactd_max_order, .total_migrate_scanned = 0, .total_free_scanned = 0, .classzone_idx = pgdat->kcompactd_classzone_idx, diff --git a/mm/internal.h b/mm/internal.h index e5ca2a10b8ad..d028abd8a8f3 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -191,7 +191,8 @@ struct compact_control { struct zone *zone; unsigned long total_migrate_scanned; unsigned long total_free_scanned; - unsigned int fast_search_fail; /* failures to use free list searches */ + unsigned short fast_search_fail;/* failures to use free list searches */ + unsigned short search_order; /* order to start a fast search at */ const gfp_t gfp_mask; /* gfp mask of a direct compactor */ int order; /* order a direct compactor needs */ int migratetype; /* migratetype of direct compactor */