From patchwork Fri Jan 4 12:50:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748321 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1CF2214E5 for ; Fri, 4 Jan 2019 12:52:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0DEE9283C9 for ; Fri, 4 Jan 2019 12:52:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 01E012844C; Fri, 4 Jan 2019 12:52:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 50C50283C9 for ; Fri, 4 Jan 2019 12:52:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 691DF8E00E0; Fri, 4 Jan 2019 07:52:47 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 61A4E8E00AE; Fri, 4 Jan 2019 07:52:47 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 509908E00E0; Fri, 4 Jan 2019 07:52:47 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by kanga.kvack.org (Postfix) with ESMTP id E5C0D8E00AE for ; Fri, 4 Jan 2019 07:52:46 -0500 (EST) Received: by mail-ed1-f70.google.com with SMTP id b3so35181023edi.0 for ; Fri, 04 Jan 2019 04:52:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=/hdAGFXBogCDzBnIyWLlp9WDxwYv3s4mVWOn70vkZJ0=; b=oPZ7b+WAcwu1l9Mom+qavfTwK7T24rGH15hGpsu9//Cy6f8nwjfNAlAeLUXft0CMKP NJMC41siHz/OCV/bqqawIE1jiU+QQVI0IlzVgvSBpaYZdguJ6SloaI2Fx6ytZCo4WIHo hHOBdVrLGZMVbMRYpKEAZc8o/Pv6ZieNHHJ2t/adkMGhessv/HJmR9VzEZQ/S7/IBPbU zqqgfhVCFsWT3VxkS7S+r9lvoNX3CDNub7KSi+vMe9VzrPpd1MhT8M1ar8fKP/u2H3C9 jquzbrj7H68D/5a8BxdclGuBNlPG7Uxf4+ljV0zL8SuwMphyHsxD3B6KKgpRXZf4IHED JErg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWZ0gkBmXHcvTC3I0g+gDCaAKlH4TwD83jj+BMTkWUN/AfFHywsX BJRBI6HKXAr6TxG7VkrogbYxlE0x0ilt1CCRMcqjEaRfNQoPUkn+ybij8RQE3BcHopNitBx2X9B BD0xL+/CNa0fGzuNqyISUGNFIgkT6MckD1SaQVmuQTf4rEhyVqbSbjPDlA7qJXR+lEQ== X-Received: by 2002:a50:98a6:: with SMTP id j35mr45802620edb.141.1546606366414; Fri, 04 Jan 2019 04:52:46 -0800 (PST) X-Google-Smtp-Source: AFSGD/WDVAlvoS+CGMG4iMlGpPECxfOjAiZxXQdnzueQT6gxB0zXswBsA7zEcHF2Lanz0n+QrmTy X-Received: by 2002:a50:98a6:: with SMTP id j35mr45802576edb.141.1546606365315; Fri, 04 Jan 2019 04:52:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606365; cv=none; d=google.com; s=arc-20160816; b=SoxajhyN+5acztqaSXU5Ojns7lExyHuiAP/nD56zmK3kWlPX+BdpHPXYokumEEjF1C ko90Q7r9lJkmsDwrHypympv6XGJNCA+Nw3xX3DkPQaj7/ikmerDBu9nfPXUTXiOQPCzC A3lVz1RNmNagtlXG3NJEmItFidUXKla6SiLVbvItHKkpLB+yhptf+zubA9dvr2/g+kAp xr+dtzmFze+anVKqPaZjUu5atczv4upaOXJ2To1sNrBk62s8YE/QIn/A1Af/m2QQpdms GP8/w5yBMs2SRtpkltkByi+B+KsOfi26PGqLPaUoyLCncbtpd5KS5l5xVrWSm/SBWqsf nYBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=/hdAGFXBogCDzBnIyWLlp9WDxwYv3s4mVWOn70vkZJ0=; b=PybZ2NQFnphGUaPm5++IomNckUkl6THvA5IT08N68WiOZa5kiJ80aD8Ge+vYFcTYlz FX69D4CrLpPEmQ4HkNJGwtEEhgzI+vEmI59rrsWvOGzdxLdEG5z23TqqXTZeRBA6Hf0g hqFMRmXAUKCZ0jaxn/L2fm+hPWEmB/qtf56cQ77dIOOAnFB6ymra9BAAm/ZEal1GZH9W 3rCC9AvGJf2YSuyWtGvWrxbznH9fO2U5moqiPuf326ZHCp5GMdIIjMoNUI9YHosYNT8E IVk1PHqaMg6dCpTZFw89gdLyS/Will0zdUZKEKDMwDpgr+1TW56riO3QXQEvcSVUBLFH TAZQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp10.blacknight.com (outbound-smtp10.blacknight.com. [46.22.139.15]) by mx.google.com with ESMTPS id u22-v6si2809239ejb.233.2019.01.04.04.52.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:52:45 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) client-ip=46.22.139.15; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp10.blacknight.com (Postfix) with ESMTPS id EA6291C1CDA for ; Fri, 4 Jan 2019 12:52:44 +0000 (GMT) Received: (qmail 4922 invoked from network); 4 Jan 2019 12:52:44 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:52:44 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 14/25] mm, compaction: Avoid rescanning the same pageblock multiple times Date: Fri, 4 Jan 2019 12:50:00 +0000 Message-Id: <20190104125011.16071-15-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Pageblocks are marked for skip when no pages are isolated after a scan. However, it's possible to hit corner cases where the migration scanner gets stuck near the boundary between the source and target scanner. Due to pages being migrated in blocks of COMPACT_CLUSTER_MAX, pages that are migrated can be reallocated before the pageblock is complete. The pageblock is not necessarily skipped so it can be rescanned multiple times. Similarly, a pageblock with some dirty/writeback pages may fail to isolate and be rescanned until writeback completes which is wasteful. This patch tracks if a pageblock is being rescanned. If so, then the entire pageblock will be migrated as one operation. This narrows the race window during which pages can be reallocated during migration. Secondly, if there are pages that cannot be isolated then the pageblock will still be fully scanned and marked for skipping. On the second rescan, the pageblock skip is set and the migration scanner makes progress. 4.20.0 4.20.0 finishscan-v2r15 norescan-v2r15 Amean fault-both-3 3729.80 ( 0.00%) 2872.13 * 23.00%* Amean fault-both-5 5148.49 ( 0.00%) 4330.56 * 15.89%* Amean fault-both-7 7393.24 ( 0.00%) 6496.63 ( 12.13%) Amean fault-both-12 11709.32 ( 0.00%) 10280.59 ( 12.20%) Amean fault-both-18 16626.82 ( 0.00%) 11079.19 * 33.37%* Amean fault-both-24 19944.34 ( 0.00%) 17207.80 * 13.72%* Amean fault-both-30 23435.53 ( 0.00%) 17736.13 * 24.32%* Amean fault-both-32 23948.70 ( 0.00%) 18509.41 * 22.71%* 4.20.0 4.20.0 finishscan-v2r15 norescan-v2r15 Percentage huge-1 0.00 ( 0.00%) 0.00 ( 0.00%) Percentage huge-3 88.39 ( 0.00%) 96.87 ( 9.60%) Percentage huge-5 92.07 ( 0.00%) 94.63 ( 2.77%) Percentage huge-7 91.96 ( 0.00%) 93.83 ( 2.03%) Percentage huge-12 93.38 ( 0.00%) 92.65 ( -0.78%) Percentage huge-18 91.89 ( 0.00%) 93.66 ( 1.94%) Percentage huge-24 91.37 ( 0.00%) 93.15 ( 1.95%) Percentage huge-30 92.77 ( 0.00%) 93.16 ( 0.42%) Percentage huge-32 87.97 ( 0.00%) 92.58 ( 5.24%) The fault latency reduction is large and while the THP allocation success rate is only slightly higher, it's already high at this point of the series. Compaction migrate scanned 60718343.00 31772603.00 Compaction free scanned 933061894.00 63267928.00 Migration scan rates are reduced by 48% and free scan rates are also reduced as the same migration source block is not being selected multiple times. The corner case where migration scan rates go through the roof due to a dirty/writeback pageblock located at the boundary of the migration/free scanner did not happen in this case. When it does happen, the scan rates multiple by factors measured in the hundreds and would be misleading to present. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 32 ++++++++++++++++++++++++++------ mm/internal.h | 1 + 2 files changed, 27 insertions(+), 6 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 9438f0564ed5..9c2cc7955446 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -959,8 +959,11 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, cc->nr_migratepages++; nr_isolated++; - /* Avoid isolating too much */ - if (cc->nr_migratepages == COMPACT_CLUSTER_MAX) { + /* + * Avoid isolating too much unless this block is being + * rescanned (e.g. dirty/writeback pages, parallel allocation). + */ + if (cc->nr_migratepages == COMPACT_CLUSTER_MAX && !cc->rescan) { ++low_pfn; break; } @@ -1007,11 +1010,14 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, spin_unlock_irqrestore(zone_lru_lock(zone), flags); /* - * Updated the cached scanner pfn if the pageblock was scanned - * without isolating a page. The pageblock may not be marked - * skipped already if there were no LRU pages in the block. + * Updated the cached scanner pfn once the pageblock has been scanned + * Pages will either be migrated in which case there is no point + * scanning in the near future or migration failed in which case the + * failure reason may persist. The block is marked for skipping if + * there were no pages isolated in the block or if the block is + * rescanned twice in a row. */ - if (low_pfn == end_pfn && !nr_isolated) { + if (low_pfn == end_pfn && (!nr_isolated || cc->rescan)) { if (valid_page && !skip_updated) set_pageblock_skip(valid_page); update_cached_migrate(cc, low_pfn); @@ -2031,6 +2037,20 @@ static enum compact_result compact_zone(struct compact_control *cc) int err; unsigned long start_pfn = cc->migrate_pfn; + /* + * Avoid multiple rescans which can happen if a page cannot be + * isolated (dirty/writeback in async mode) or if the migrated + * pages are being allocated before the pageblock is cleared. + * The first rescan will capture the entire pageblock for + * migration. If it fails, it'll be marked skip and scanning + * will proceed as normal. + */ + cc->rescan = false; + if (pageblock_start_pfn(last_migrated_pfn) == + pageblock_start_pfn(start_pfn)) { + cc->rescan = true; + } + switch (isolate_migratepages(cc->zone, cc)) { case ISOLATE_ABORT: ret = COMPACT_CONTENDED; diff --git a/mm/internal.h b/mm/internal.h index b25b33c5dd80..e5ca2a10b8ad 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -204,6 +204,7 @@ struct compact_control { bool direct_compaction; /* False from kcompactd or /proc/... */ bool whole_zone; /* Whole zone should/has been scanned */ bool contended; /* Signal lock or sched contention */ + bool rescan; /* Rescanning the same pageblock */ }; unsigned long