From patchwork Fri Jan 4 12:50:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748343 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8346213B5 for ; Fri, 4 Jan 2019 12:54:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 742EF2810E for ; Fri, 4 Jan 2019 12:54:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 682622836F; Fri, 4 Jan 2019 12:54:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CEAA42810E for ; Fri, 4 Jan 2019 12:54:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C5E748E00E8; Fri, 4 Jan 2019 07:54:08 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C333C8E00AE; Fri, 4 Jan 2019 07:54:08 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD9098E00E8; Fri, 4 Jan 2019 07:54:08 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id 4EFEB8E00AE for ; Fri, 4 Jan 2019 07:54:08 -0500 (EST) Received: by mail-ed1-f71.google.com with SMTP id c34so34720212edb.8 for ; Fri, 04 Jan 2019 04:54:08 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=nBi3kpmLXtB+/L42OygzYTjrh03AFHn26ukynBWzS6c=; b=Pj8zjsLxfH+eu9NpEA16wN5Fg7E8Bu2JV3wjbd7qNA311GRyey425bvn8b31CEQqze N9posbLxsvJWAYH56JyEmn9H0c7Nr/PHflMfvhzjqWEYDcWiohoIbUg25FqeZ09LXdJt peqHi558WxSsEXXijaxqDOSJ5V6na5Zx+RqDA+IJAPL4VOrTgd0oYI58o5M5WQntSFTw 6qwReX+EsDu7WZaL34Wpz92hD3irLsSLXnM1wknf2B8BobSJdmsGriUZdAzDtrId3v1O WRg8yvFAEI5On1ZvM656XKzpJjxF5F2WOKQhUMGVud5/BFgGRxnvOJEcbvFSnWpBOdGS Q0rQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWaxpgVQVFvBhIgfhQA9gDio57bcbdf+CV4QRmpXY5bM9r+i/OyR 6+634V2tHhKmWU19S5ZTdXCaf9drdZeW3tJoqebrBkNVrBUdIjoRisFG1uRFFQJ49foc64ZTjf4 yinnnUeVVimvY4GVxwDw27im4BAiiLCTbeiaFkTmSh8936lzRsDQhU4s5+fzR+YDB8A== X-Received: by 2002:a17:906:753:: with SMTP id z19-v6mr38800139ejb.175.1546606447804; Fri, 04 Jan 2019 04:54:07 -0800 (PST) X-Google-Smtp-Source: AFSGD/WWeqYmykrh1Rwxnoh61OwFm/OESBGfmBt0clagtE0lZUqR7Ho6NE+OYx58mLtwkSCrXlno X-Received: by 2002:a17:906:753:: with SMTP id z19-v6mr38800103ejb.175.1546606446817; Fri, 04 Jan 2019 04:54:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606446; cv=none; d=google.com; s=arc-20160816; b=lw/xE6Mful1UGo4naJ03VOeIQ/iXipMvvHl38nYeQ3M2+qfRURg0FdQa4D/SUuCViX 8VMmDKePvjA2xKw6hnTCKufp2BbvMxheXNzoupz6+QeP+jf/DrGb2Fj1DU47wdKrg5DC SbQBBYOe9+zEEcQzM2N/wgtN1nhRImDSIlRp39kpGnDhqbTjZM/+pxEhhShg5S6E8Z2y QKQ2v79CacWuwj/3Jcne6vNiAWEfamLchsgQ0KpLNtWzmQwO6hGZBVN9rE9GdA4bqAhb w/5vaqc5IvbIniWJhbSPdpnT4Rp+AxIMX3VjG1vb7WLJJzeC+k++uvtkwAGXgd0GHIx4 IFSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=nBi3kpmLXtB+/L42OygzYTjrh03AFHn26ukynBWzS6c=; b=w9bKwaBelO0v+9BjyKsyTj43J3snIPCIWFuAwWFFBCqgDDrMcgpSu7LxYiHUl4bzfo 1lQvByRIZSshBswTjjB6GLgU1rU54tm5mw51vPG1wYnEwnMhge29hscg+9gD/mOKn95t iqN1Fkja4Qento/bmDnhOVB5tqPhvZogi4smmX6WlAM1hm/LKij/sQNL5Y1b1Jnfa8CN KV7RqLfhUct87VHnqwWxQpdX/faSbbjeNnxDw+EspYSjmmgFvV6I2i4BaFSGg86LlFmA 42ytYhtBmBBuwdF34FINLtlmTYgKYIIpxYhtZCEkIi+b0UNIa6SNYGPPOAlvLH20dSBS NIHA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp13.blacknight.com (outbound-smtp13.blacknight.com. [46.22.139.230]) by mx.google.com with ESMTPS id p18si6154495edi.197.2019.01.04.04.54.06 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:54:06 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) client-ip=46.22.139.230; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (unknown [81.17.254.16]) by outbound-smtp13.blacknight.com (Postfix) with ESMTPS id 7509D1C1CE7 for ; Fri, 4 Jan 2019 12:54:06 +0000 (GMT) Received: (qmail 10824 invoked from network); 4 Jan 2019 12:54:06 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:54:06 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 22/25] mm, compaction: Sample pageblocks for free pages Date: Fri, 4 Jan 2019 12:50:08 +0000 Message-Id: <20190104125011.16071-23-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Once fast searching finishes, there is a possibility that the linear scanner is scanning full blocks found by the fast scanner earlier. This patch uses an adaptive stride to sample pageblocks for free pages. The more consecutive full pageblocks encountered, the larger the stride until a pageblock with free pages is found. The scanners might meet slightly sooner but it is an acceptable risk given that the search of the free lists may still encounter the pages and adjust the cached PFN of the free scanner accordingly. In terms of latency and success rates, the impact is not obvious but the free scan rate is reduced by 87% on a 1-socket machine and 92% on a 2-socket machine. It's also the first time in the series where the number of pages scanned by the migration scanner is greater than the free scanner due to the increased search efficiency. Signed-off-by: Mel Gorman --- mm/compaction.c | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 652e249168b1..cc532e81a7b7 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -441,6 +441,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, unsigned long *start_pfn, unsigned long end_pfn, struct list_head *freelist, + unsigned int stride, bool strict) { int nr_scanned = 0, total_isolated = 0; @@ -450,10 +451,14 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, unsigned long blockpfn = *start_pfn; unsigned int order; + /* Strict mode is for isolation, speed is secondary */ + if (strict) + stride = 1; + cursor = pfn_to_page(blockpfn); /* Isolate free pages. */ - for (; blockpfn < end_pfn; blockpfn++, cursor++) { + for (; blockpfn < end_pfn; blockpfn += stride, cursor += stride) { int isolated; struct page *page = cursor; @@ -624,7 +629,7 @@ isolate_freepages_range(struct compact_control *cc, break; isolated = isolate_freepages_block(cc, &isolate_start_pfn, - block_end_pfn, &freelist, true); + block_end_pfn, &freelist, 0, true); /* * In strict mode, isolate_freepages_block() returns 0 if @@ -1139,7 +1144,7 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn, unsigned long /* Scan before */ if (start_pfn != pfn) { - isolate_freepages_block(cc, &start_pfn, pfn, &cc->freepages, false); + isolate_freepages_block(cc, &start_pfn, pfn, &cc->freepages, 1, false); if (cc->nr_freepages >= cc->nr_migratepages) return; } @@ -1147,7 +1152,7 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn, unsigned long /* Scan after */ start_pfn = pfn + nr_isolated; if (start_pfn != end_pfn) - isolate_freepages_block(cc, &start_pfn, end_pfn, &cc->freepages, false); + isolate_freepages_block(cc, &start_pfn, end_pfn, &cc->freepages, 1, false); /* Skip this pageblock in the future as it's full or nearly full */ if (cc->nr_freepages < cc->nr_migratepages) @@ -1333,7 +1338,9 @@ static void isolate_freepages(struct compact_control *cc) unsigned long isolate_start_pfn; /* exact pfn we start at */ unsigned long block_end_pfn; /* end of current pageblock */ unsigned long low_pfn; /* lowest pfn scanner is able to scan */ + unsigned long nr_isolated; struct list_head *freelist = &cc->freepages; + unsigned int stride; /* Try a small search of the free lists for a candidate */ isolate_start_pfn = fast_isolate_freepages(cc); @@ -1356,6 +1363,7 @@ static void isolate_freepages(struct compact_control *cc) block_end_pfn = min(block_start_pfn + pageblock_nr_pages, zone_end_pfn(zone)); low_pfn = pageblock_end_pfn(cc->migrate_pfn); + stride = cc->mode == MIGRATE_ASYNC ? COMPACT_CLUSTER_MAX : 1; /* * Isolate free pages until enough are available to migrate the @@ -1387,8 +1395,8 @@ static void isolate_freepages(struct compact_control *cc) continue; /* Found a block suitable for isolating free pages from. */ - isolate_freepages_block(cc, &isolate_start_pfn, block_end_pfn, - freelist, false); + nr_isolated = isolate_freepages_block(cc, &isolate_start_pfn, + block_end_pfn, freelist, stride, false); /* Update the skip hint if the full pageblock was scanned */ if (isolate_start_pfn == block_end_pfn) @@ -1412,6 +1420,13 @@ static void isolate_freepages(struct compact_control *cc) */ break; } + + /* Adjust stride depending on isolation */ + if (nr_isolated) { + stride = 1; + continue; + } + stride = min_t(unsigned int, COMPACT_CLUSTER_MAX, stride << 1); } /*