From patchwork Fri Jan 18 17:51:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771355 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3494A6C5 for ; Fri, 18 Jan 2019 17:52:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2422F2FC0E for ; Fri, 18 Jan 2019 17:52:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 17D612FF1B; Fri, 18 Jan 2019 17:52:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AC5952FC0E for ; Fri, 18 Jan 2019 17:52:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D53108E0010; Fri, 18 Jan 2019 12:51:59 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D01B88E0002; Fri, 18 Jan 2019 12:51:59 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF0F38E0010; Fri, 18 Jan 2019 12:51:59 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by kanga.kvack.org (Postfix) with ESMTP id 655338E0002 for ; Fri, 18 Jan 2019 12:51:59 -0500 (EST) Received: by mail-ed1-f70.google.com with SMTP id s50so5189401edd.11 for ; Fri, 18 Jan 2019 09:51:59 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=8mTqu6XmTXueq68p+xTpjDNtIf6Fpr2hYaUsjecSpjE=; b=bWviq+58sZIh2na2Wt2Q1FdOsCCA8E98PaQMIbhkjw9mekTT4FJsXCSAg/Bjbms+5Q Ig3z4R19kuZxcHSCvqh9GA3pJQJKFY6f9qoYwsDCgnlovG8r3rU2koPeLr1d3EPK4lQy 49cHpMpk54PmaFW4Nf/4NyBcxS1pET6xxY9GD1G2GBrDL7sVTVYoU/bzmZLMj/e8a0Pm nkqamO0V4gaGSTgPNMv4vZzN/TbhXUjE6IMO5RSQOd4GKierY02rjextYy9vFxrx/P+L 53kYmADvrSI7Xhg0imzAVK75pHjDybtMj2bDRG7LirjOuJRBZuDcYJbUMP67q6Zaqv5Z n6FQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukcRtgfDLmx3yF3BTjpmOaLiD70OyQ+3ZiTEXWI2XJfAmbDj4vH8 ocZx9UaAaeUidcnd1Qv7swHTicnZJrUxeOW79MjcxI03r2Le/Op9jkRIgJubvpKWSpYXexfBZLO Fl0eAtEdsTZCOkJyr9H/kpPulkQzzboPQ7QuIUJNl8oc77Xc4L7azGVbRdeo4/XOX4A== X-Received: by 2002:a17:906:258f:: with SMTP id m15-v6mr15521943ejb.191.1547833918880; Fri, 18 Jan 2019 09:51:58 -0800 (PST) X-Google-Smtp-Source: ALg8bN4fgOPvLoy1jbq/m6yU2HZ04rvssPwBFKyflsolDLp5le3Ln6k9IgrEe2fdCy+UyCPwr2GU X-Received: by 2002:a17:906:258f:: with SMTP id m15-v6mr15521890ejb.191.1547833917809; Fri, 18 Jan 2019 09:51:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547833917; cv=none; d=google.com; s=arc-20160816; b=kGxAerj/ju7fYSm7RO4X9OqVSJlS6zgeY3d55BXnhUzUK3azfjn8AIqIPRZFgdp0u0 wH6if0QEMJOW58jMBcsBbP12VXAqmMxDMnQFXbNKK79JYAR9lPOaUzsCmTqsgPyjfCUP 1wmK7HkK7xS6Zfx0xc3XoZato6FChJRyRc3g8yto69NdrREB4HfTFnrbdfusJPtGebWp l5CVA19BPHERA3O8k9LWmTEjK6MaqmNXgC3nipUxJHxCi742hvP3yNjcGnEnUFgR9hww 7cEdd96k4N5MFOxkpzju3div7A02o2nQB2uB9K7VmPqeSTkLDwmaw392P/EBeMsqor72 H+0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=8mTqu6XmTXueq68p+xTpjDNtIf6Fpr2hYaUsjecSpjE=; b=H9RuZz7naIn7pNkwciP5Tdacd+vWtDGGwLkVtJvJKVT5SCPNE6ygLijEgzyOurX1oO 3MUbWSE6RVVbHNiFmP1WPwXfJWSXzuVtYAYqHekBJvyJDkW23eb9MPSz0392f7JLrish 32Y0badSeYPPSXMsr9j3IO+1Eln9cAjexUG2hElDXTAc/erMjEN4LiTBbkGDsCn69iS9 jMUwpsOOVeJbVn2SgmP5n0ux+7migZ4wRkpIGx1Fxqd5UcQ85yKYDECnmODH5QZKLv2Y m0kJ89h6fjBNzwNUR3Nvbz09nwboQtWzN/sGpFZd87WIfD9ZuTr6UfNuAIoNuKfSI4k1 AcWg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp16.blacknight.com (outbound-smtp16.blacknight.com. [46.22.139.233]) by mx.google.com with ESMTPS id m2-v6si4454730eje.191.2019.01.18.09.51.57 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:51:57 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) client-ip=46.22.139.233; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp16.blacknight.com (Postfix) with ESMTPS id 58BB31C3589 for ; Fri, 18 Jan 2019 17:51:57 +0000 (GMT) Received: (qmail 30679 invoked from network); 18 Jan 2019 17:51:57 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:51:57 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 01/22] mm, compaction: Shrink compact_control Date: Fri, 18 Jan 2019 17:51:15 +0000 Message-Id: <20190118175136.31341-2-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The isolate and migrate scanners should never isolate more than a pageblock of pages so unsigned int is sufficient saving 8 bytes on a 64-bit build. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/internal.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 536bc2a839b9..5564841fce36 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -185,8 +185,8 @@ struct compact_control { struct list_head freepages; /* List of free pages to migrate to */ struct list_head migratepages; /* List of pages being migrated */ struct zone *zone; - unsigned long nr_freepages; /* Number of isolated free pages */ - unsigned long nr_migratepages; /* Number of pages to migrate */ + unsigned int nr_freepages; /* Number of isolated free pages */ + unsigned int nr_migratepages; /* Number of pages to migrate */ unsigned long total_migrate_scanned; unsigned long total_free_scanned; unsigned long free_pfn; /* isolate_freepages search base */ From patchwork Fri Jan 18 17:51:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771357 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 33A1A6C5 for ; Fri, 18 Jan 2019 17:52:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 236D82FB87 for ; Fri, 18 Jan 2019 17:52:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 16D7A2FC0E; Fri, 18 Jan 2019 17:52:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ACB202FB87 for ; Fri, 18 Jan 2019 17:52:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D05DA8E0011; Fri, 18 Jan 2019 12:52:09 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CB3258E0002; Fri, 18 Jan 2019 12:52:09 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B7D5D8E0011; Fri, 18 Jan 2019 12:52:09 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id 618468E0002 for ; Fri, 18 Jan 2019 12:52:09 -0500 (EST) Received: by mail-ed1-f69.google.com with SMTP id e17so5196841edr.7 for ; Fri, 18 Jan 2019 09:52:09 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=oJAP/potus7kSDKt4pUHIFnn7W7cakTIxSUPolk2DRg=; b=M8aZO/f+zoaVlbbrOVCESDeyBDOTV5TVNYPpsnFrFWrZjo9gF6cTS8rVBP5/KCe4HL KBvXSYvtKq4Rd+iQgrd43Bwb7VafjHt+hZymVwsBMJ2Lisivw/0jRJSqhCjjd48LbQEh M7loyAvubCATJw1WlajBvdgPXCsuqZwa2LyBoZNy1ON8jB1WUO+jtSOWn9n43Wtyt1aS 4dqcDB2Fd5I1/Y/tD/7GxffTk/RUu3a6pPwC95yXn5pTGtXKh4DBfuvsqn6BqTdfzjic qsRqtrWMQmffCu+MY47AfF309hPooIj2Df8drG6shqJv8OZCrNP0fDGFRP+nEgqrTbGX hI4Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukfQ2+WqUFaV+tFaNCIR5npEWMjtEm1Ftd8eczeCGjK+0LV5iNZz vCouB1QXsnQMdRi4v88hPRMom0jqRUFLzVWZv18xNZ1TgXcfX06KsNxtTV1RZS+8Tjms1/RWaNy 3Tuj5YQF95hlZba/zfTkEobyFPHtVMCouyIUaYiRVQAcDW7tVHUFblvOO9kJVGX9wXQ== X-Received: by 2002:a50:86c2:: with SMTP id 2mr16825868edu.34.1547833928872; Fri, 18 Jan 2019 09:52:08 -0800 (PST) X-Google-Smtp-Source: ALg8bN4Dm12wdvC5cm9/TE339RczVJwsM8QmlbnGl4KTPiNNPXnhKC+WspqhheURB7zWMiOMLd7X X-Received: by 2002:a50:86c2:: with SMTP id 2mr16825814edu.34.1547833927795; Fri, 18 Jan 2019 09:52:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547833927; cv=none; d=google.com; s=arc-20160816; b=cP2IYaKlcpVdJ1xS9bJP65t5YtW4qISlarcDt94yWMUVYx8HVy1+2mgIacGgfCMnTO 5peh5BFHT5KS3Ptyww5bxi4iMid+x1krlq1NLfzTlgh1GuqTJBclAoqNyQLo/wRrcASk tNGdeSJ7tqfDoWaj72KCLDun2KJ+r1EjRpF2gzBIF4se6fordpn9fBrPy9xZ6+8NSvAj haM3Cx9zvo5kWz9TgthiTwgyzmfm8Bo8P6qI+EaA9viUGxG4gtlM6NBYr+fIIn+3JcYc /RuvnfJGqhhVi/6dHLX6y4qFGY7NUf+VFZSPGBiIYAqxZEETsLMjX8HBcD9br89cAfei L9RA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=oJAP/potus7kSDKt4pUHIFnn7W7cakTIxSUPolk2DRg=; b=mHR5d6npYnyKvXHYKJg7+5qBlEfnS50j5a6EJS5n9bjC7HembyAP+VnAx0/Y6dF3sz NApD2ckoiPp1aveEx4nb4u3jfgQs7D0WdFA4EN1+/k1o6AwnIZ4az6t1nybboLjj66qP aYY0fSa6y+FuSZszDGSsXfBZ6c2kqoXTqtSUUT33A0qBfamx6SXih3EDvqSQI+wPK/wW hRGxT+NqhXPIM06Mjf7+wi9qDEQFekRevz2rBLm7ve7R+rEl2Z3liReRlkYY1SRFQBok jdvuFGhSL5j9VcIhILOTF5IXMgAYBlGJwbX4gEuvLDvcIbu6+uRjWUjr9qb9b/sgJamq cFKw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp04.blacknight.com (outbound-smtp04.blacknight.com. [81.17.249.35]) by mx.google.com with ESMTPS id x17-v6si7755173eji.266.2019.01.18.09.52.07 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:52:07 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) client-ip=81.17.249.35; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp04.blacknight.com (Postfix) with ESMTPS id 7CB4A98C35 for ; Fri, 18 Jan 2019 17:52:07 +0000 (UTC) Received: (qmail 31214 invoked from network); 18 Jan 2019 17:52:07 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:52:07 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 02/22] mm, compaction: Rearrange compact_control Date: Fri, 18 Jan 2019 17:51:16 +0000 Message-Id: <20190118175136.31341-3-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP compact_control spans two cache lines with write-intensive lines on both. Rearrange so the most write-intensive fields are in the same cache line. This has a negligible impact on the overall performance of compaction and is more a tidying exercise than anything. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/internal.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 5564841fce36..867af5425432 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -184,14 +184,14 @@ extern int user_min_free_kbytes; struct compact_control { struct list_head freepages; /* List of free pages to migrate to */ struct list_head migratepages; /* List of pages being migrated */ - struct zone *zone; unsigned int nr_freepages; /* Number of isolated free pages */ unsigned int nr_migratepages; /* Number of pages to migrate */ - unsigned long total_migrate_scanned; - unsigned long total_free_scanned; unsigned long free_pfn; /* isolate_freepages search base */ unsigned long migrate_pfn; /* isolate_migratepages search base */ unsigned long last_migrated_pfn;/* Not yet flushed page being freed */ + struct zone *zone; + unsigned long total_migrate_scanned; + unsigned long total_free_scanned; const gfp_t gfp_mask; /* gfp mask of a direct compactor */ int order; /* order a direct compactor needs */ int migratetype; /* migratetype of direct compactor */ From patchwork Fri Jan 18 17:51:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771359 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9FD0C6C5 for ; Fri, 18 Jan 2019 17:52:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8E60A2FC07 for ; Fri, 18 Jan 2019 17:52:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7D97E2FEF7; Fri, 18 Jan 2019 17:52:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E537F2FC07 for ; Fri, 18 Jan 2019 17:52:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 16B878E0012; Fri, 18 Jan 2019 12:52:20 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 11D5D8E0002; Fri, 18 Jan 2019 12:52:20 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0316D8E0012; Fri, 18 Jan 2019 12:52:19 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by kanga.kvack.org (Postfix) with ESMTP id 9E3588E0002 for ; Fri, 18 Jan 2019 12:52:19 -0500 (EST) Received: by mail-ed1-f70.google.com with SMTP id c3so5330279eda.3 for ; Fri, 18 Jan 2019 09:52:19 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=KxB+olxFRz15ByQGQdOPEkv73792TGdbwQgnAht0mak=; b=Qyy4mDNx9uEyrEM5mD5vthWqTvc7bbzFzAOmhWtiRusCoYzBLXiF14tDSn/iDQPhGp GVC2cmrmrx9sm54H5OJfnPmKfzr4ae4JWP2MtYt3wnMZtwiTv7XgIa29wOgXMN297Pqi bgcjglFm/mN2Gx+4G5+OtRpmRFQYCXdzKXhViUe6uQCeCwG4joRxMbJYO21oKsgTmm+Q s9Qx4fUimh5iK6XZCXb5kj8ez8UMc0qMNpJfKh+Z1bBQQ7sIdJF/adsnjc4EpOBCbKoe LwjBYu8YjEbvFcIEUjDyZln9/hdzIlk4DLRFrH0ctynCjv1lyIf2TzgwRrHX43oRHVHC ULhg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukcr0kXKvNWcJvoQUjdDMHFF38kGCwf2NxR0+EQB7AMiQM3kA5G5 3o5a2LiDTloOlps9Lmo7ABefOgAKP8+TMnV2NGcYY/zRHQWbDHzoxmrg7Ayb5y6i6NBG27ytqgK GH00L0h3XR5pTr+3dorH+KeTWN3cxwHcUJ5FUuzhjz0oyh5gdYXn8sJXWImM7KWZE9Q== X-Received: by 2002:a50:f4c3:: with SMTP id v3mr16714278edm.196.1547833939131; Fri, 18 Jan 2019 09:52:19 -0800 (PST) X-Google-Smtp-Source: ALg8bN4wfGrcH2o2wH3W3AZluoil44ZkmKQNCdxW+idW+HpH2x1xU0NnuJ0uh8wCdVT+it/YYJMJ X-Received: by 2002:a50:f4c3:: with SMTP id v3mr16714217edm.196.1547833937966; Fri, 18 Jan 2019 09:52:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547833937; cv=none; d=google.com; s=arc-20160816; b=gm/T67fgq6IW0mc1h0Cxn/xGJn60+OpyPuGdKOVB3uQtt6yu1WigiA+ktwdRU3xFQY 3Q+MaVTgbVmHoDSFVitnKucdfjwFmyajHG+JYAHla5rJ6LXF2JosVg5NMCwfg9W+wZ5q jzDAdoSNCFAYjCa35oVO3yfj0UH1DUKV3JGRX9IzxjHlMRJ9DrLpstaGsJdrJiIEEr2P MGmdg1ZNMntr1X9Kb0/teGN3F3548d+XeqnyyBCpM9aqLna8/jc13f8SMrW8g9KdAGof mktRLwYtTBc+F3xyq0+VJBE5Z8PnImehw02Ci6Ugw5BAcsl/27MG6ZJZSIKke+Z3U5Tn oacw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=KxB+olxFRz15ByQGQdOPEkv73792TGdbwQgnAht0mak=; b=YM2uIujRTOXO0Rd58xINFBulvXKwbJsuGQTMXjC5BdzkMIoQnPTwTGjqna5/W9rLi5 N0FtrFJBmCbSa+XKpU/+0rsIUF462Fb+YsLkIG23Bg0iIA+rHpoRmSGJgj5uxGaqezwE zwXXqHmCg3VB+PmkWDdzAE0uaIyQF4KwDNvDxP5h6O0AMSSoF91ARRa0APoy1iQUGpJE DB0ZlUoYVsNfHrF4H/bNFg7bX/wQoFoRGD+RGjK8AVR21//GElMNjIWYpEYtxz7/Kvk8 b3a/StMsDogZ8ag5DjC8iyQ9U3kpmF9KTzYSu9+5cwczyjc8Jzph2OcYiyzjBcFtDW3V Ph4g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp04.blacknight.com (outbound-smtp04.blacknight.com. [81.17.249.35]) by mx.google.com with ESMTPS id x1-v6si2649794eju.324.2019.01.18.09.52.17 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:52:17 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) client-ip=81.17.249.35; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp04.blacknight.com (Postfix) with ESMTPS id A2D9D98C47 for ; Fri, 18 Jan 2019 17:52:17 +0000 (UTC) Received: (qmail 31737 invoked from network); 18 Jan 2019 17:52:17 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:52:17 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 03/22] mm, compaction: Remove last_migrated_pfn from compact_control Date: Fri, 18 Jan 2019 17:51:17 +0000 Message-Id: <20190118175136.31341-4-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The last_migrated_pfn field is a bit dubious as to whether it really helps but either way, the information from it can be inferred without increasing the size of compact_control so remove the field. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 25 +++++++++---------------- mm/internal.h | 1 - 2 files changed, 9 insertions(+), 17 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index c15b4bbc9e9e..e59dd7a7564c 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -886,15 +886,6 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, cc->nr_migratepages++; nr_isolated++; - /* - * Record where we could have freed pages by migration and not - * yet flushed them to buddy allocator. - * - this is the lowest page that was isolated and likely be - * then freed by migration. - */ - if (!cc->last_migrated_pfn) - cc->last_migrated_pfn = low_pfn; - /* Avoid isolating too much */ if (cc->nr_migratepages == COMPACT_CLUSTER_MAX) { ++low_pfn; @@ -918,7 +909,6 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, } putback_movable_pages(&cc->migratepages); cc->nr_migratepages = 0; - cc->last_migrated_pfn = 0; nr_isolated = 0; } @@ -1539,6 +1529,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro enum compact_result ret; unsigned long start_pfn = zone->zone_start_pfn; unsigned long end_pfn = zone_end_pfn(zone); + unsigned long last_migrated_pfn; const bool sync = cc->mode != MIGRATE_ASYNC; cc->migratetype = gfpflags_to_migratetype(cc->gfp_mask); @@ -1584,7 +1575,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro cc->whole_zone = true; } - cc->last_migrated_pfn = 0; + last_migrated_pfn = 0; trace_mm_compaction_begin(start_pfn, cc->migrate_pfn, cc->free_pfn, end_pfn, sync); @@ -1593,12 +1584,14 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro while ((ret = compact_finished(zone, cc)) == COMPACT_CONTINUE) { int err; + unsigned long start_pfn = cc->migrate_pfn; switch (isolate_migratepages(zone, cc)) { case ISOLATE_ABORT: ret = COMPACT_CONTENDED; putback_movable_pages(&cc->migratepages); cc->nr_migratepages = 0; + last_migrated_pfn = 0; goto out; case ISOLATE_NONE: /* @@ -1608,6 +1601,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro */ goto check_drain; case ISOLATE_SUCCESS: + last_migrated_pfn = start_pfn; ; } @@ -1639,8 +1633,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro cc->migrate_pfn = block_end_pfn( cc->migrate_pfn - 1, cc->order); /* Draining pcplists is useless in this case */ - cc->last_migrated_pfn = 0; - + last_migrated_pfn = 0; } } @@ -1652,18 +1645,18 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro * compact_finished() can detect immediately if allocation * would succeed. */ - if (cc->order > 0 && cc->last_migrated_pfn) { + if (cc->order > 0 && last_migrated_pfn) { int cpu; unsigned long current_block_start = block_start_pfn(cc->migrate_pfn, cc->order); - if (cc->last_migrated_pfn < current_block_start) { + if (last_migrated_pfn < current_block_start) { cpu = get_cpu(); lru_add_drain_cpu(cpu); drain_local_pages(zone); put_cpu(); /* No more flushing until we migrate again */ - cc->last_migrated_pfn = 0; + last_migrated_pfn = 0; } } diff --git a/mm/internal.h b/mm/internal.h index 867af5425432..f40d06d70683 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -188,7 +188,6 @@ struct compact_control { unsigned int nr_migratepages; /* Number of pages to migrate */ unsigned long free_pfn; /* isolate_freepages search base */ unsigned long migrate_pfn; /* isolate_migratepages search base */ - unsigned long last_migrated_pfn;/* Not yet flushed page being freed */ struct zone *zone; unsigned long total_migrate_scanned; unsigned long total_free_scanned; From patchwork Fri Jan 18 17:51:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771361 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2F6DB91E for ; Fri, 18 Jan 2019 17:52:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1FA5A2FC07 for ; Fri, 18 Jan 2019 17:52:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 13DED2FEF7; Fri, 18 Jan 2019 17:52:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5EFD42FC07 for ; Fri, 18 Jan 2019 17:52:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 75E848E0013; Fri, 18 Jan 2019 12:52:30 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 70E498E0002; Fri, 18 Jan 2019 12:52:30 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6243E8E0013; Fri, 18 Jan 2019 12:52:30 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 09E558E0002 for ; Fri, 18 Jan 2019 12:52:30 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id t7so5235930edr.21 for ; Fri, 18 Jan 2019 09:52:29 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=6HcLSuUw2FKRerxYAlk1yerYWKKn9v5ftxhoxvARnXs=; b=rwGz5tlKLH6X7J0IrH3JJSDQqiEhSo7C5XDtEe3yz+n03uNkLFvBcWnLQi4QA4s9JJ UcPlq8eGdBojp90HM5BGwk3xkwqabUbMaMyqBFVok4xf+R0m1lr+KesE3mhRnwwSigyO bM33pNgI4gAcjAqZXG8eXEIMeiOR9IRD5/yNHLhMfX3ieRQpdENhu8NGQqZRASbarJfx x8PwrnDVhT8B+rJCyqccY18S20MZ53e8dLnPG1dNREjWZXtuSS+vvzBG1dqnAzbyx0eJ FoJPLfEB//IhHuPIYW0YbpWf/k8YkvDk4iPVa4gZd3IL82p2TaTmkM+Tq9i69VlSIZeP 07Lg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukfU4eTEK/fapb0STt5D9/vcULxz7LLFzvxGGmB9tbYvYAz/8jGd HB2xyHp+3uFw/7WqUund4InoHachqh02DA9XpTflH86ijWTRK0/+DV5nPTjpJ9tIFKDVzuYOFDq X+J39svnuEuX6ZHrHjrV2U2FPobjTnrJJRRUgpPr5RUGPw6Mz0Y8l9jNCYv7rs8ETYg== X-Received: by 2002:aa7:c981:: with SMTP id c1mr16794958edt.54.1547833949479; Fri, 18 Jan 2019 09:52:29 -0800 (PST) X-Google-Smtp-Source: ALg8bN70TklU6nChrpbD4Msn+a8EzHXxuMtxAqpAFfoA0uZ45PuHI9bN8ld84HoPdwLZOJOFo8d8 X-Received: by 2002:aa7:c981:: with SMTP id c1mr16794884edt.54.1547833948126; Fri, 18 Jan 2019 09:52:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547833948; cv=none; d=google.com; s=arc-20160816; b=YneD3DokCN59DiAKYX9SzlPz/B7Tdgw3PRHb3qFymAzdaRXGSH2gqHS4zBCqvcd46/ BB+1Ys5TDPp0OdnguukenoaL0TZfhaFAkCW+34RjDiiQQ1F2wI7vOPYN8P5Vgbi4Pu69 QMy+s4DJyAIKebNVGA1WiGZ7cJbRruaE5Gz2TlFK3ARzA1oyT9ZPxbrVKHWTNHHgZ7eS 6255nQMjXcrKfDVn1lqVUppLbBTetZnu39bj41gM3CcoHeMiyPYP2GZV8VdhQR07kOXM LXUJAdX7sJLnDD4TB+wqvSvC69R21+U6nA0WYPBHJLISM8WVVggboUnAMOJ2dApR7DTu vsUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=6HcLSuUw2FKRerxYAlk1yerYWKKn9v5ftxhoxvARnXs=; b=oAqZ5ijr/wOyxYFoeEYnh2XDDWNokFkIpvxddzzsqPbUXdgJyy20LXnhVRKfSFEuF4 jHHwC/aT1FQSt62ufVZH9MflyyUtmxcMlgp96D6+S2KGCinuX9ZS2uhUJcbPBrp2lhh6 OtEIa3Yrn+k8GayQCU6490kXlLHSVKwtQ1skHyv9hzE1UMrkn1wy3h3euIRQ3S8QUQnh 2nggDDT6Xn6P2jmKlrUlk8KtrStcYRBFjwwLb61kv+bPRnYZwLgqd1o54P6Z3I7v1Tla ibzaAFX6ujRMZ1KnXSDEGA1c9h8lxzTPmMrwCCjEZS4mpm6iLLRyItxODMvoyP8kOkFq WkPQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp26.blacknight.com (outbound-smtp26.blacknight.com. [81.17.249.194]) by mx.google.com with ESMTPS id f27-v6si137708ejh.100.2019.01.18.09.52.27 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:52:28 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) client-ip=81.17.249.194; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp26.blacknight.com (Postfix) with ESMTPS id CA52FB8AA6 for ; Fri, 18 Jan 2019 17:52:27 +0000 (GMT) Received: (qmail 32350 invoked from network); 18 Jan 2019 17:52:27 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:52:27 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 04/22] mm, compaction: Remove unnecessary zone parameter in some instances Date: Fri, 18 Jan 2019 17:51:18 +0000 Message-Id: <20190118175136.31341-5-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP A zone parameter is passed into a number of top-level compaction functions despite the fact that it's already in compact_control. This is harmless but it did need an audit to check if zone actually ever changes meaningfully. This patches removes the parameter in a number of top-level functions. The change could be much deeper but this was enough to briefly clarify the flow. No functional change. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 54 ++++++++++++++++++++++++++---------------------------- 1 file changed, 26 insertions(+), 28 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index e59dd7a7564c..163841e1b167 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1300,8 +1300,7 @@ static inline bool is_via_compact_memory(int order) return order == -1; } -static enum compact_result __compact_finished(struct zone *zone, - struct compact_control *cc) +static enum compact_result __compact_finished(struct compact_control *cc) { unsigned int order; const int migratetype = cc->migratetype; @@ -1312,7 +1311,7 @@ static enum compact_result __compact_finished(struct zone *zone, /* Compaction run completes if the migrate and free scanner meet */ if (compact_scanners_met(cc)) { /* Let the next compaction start anew. */ - reset_cached_positions(zone); + reset_cached_positions(cc->zone); /* * Mark that the PG_migrate_skip information should be cleared @@ -1321,7 +1320,7 @@ static enum compact_result __compact_finished(struct zone *zone, * based on an allocation request. */ if (cc->direct_compaction) - zone->compact_blockskip_flush = true; + cc->zone->compact_blockskip_flush = true; if (cc->whole_zone) return COMPACT_COMPLETE; @@ -1345,7 +1344,7 @@ static enum compact_result __compact_finished(struct zone *zone, /* Direct compactor: Is a suitable page free? */ for (order = cc->order; order < MAX_ORDER; order++) { - struct free_area *area = &zone->free_area[order]; + struct free_area *area = &cc->zone->free_area[order]; bool can_steal; /* Job done if page is free of the right migratetype */ @@ -1391,13 +1390,12 @@ static enum compact_result __compact_finished(struct zone *zone, return COMPACT_NO_SUITABLE_PAGE; } -static enum compact_result compact_finished(struct zone *zone, - struct compact_control *cc) +static enum compact_result compact_finished(struct compact_control *cc) { int ret; - ret = __compact_finished(zone, cc); - trace_mm_compaction_finished(zone, cc->order, ret); + ret = __compact_finished(cc); + trace_mm_compaction_finished(cc->zone, cc->order, ret); if (ret == COMPACT_NO_SUITABLE_PAGE) ret = COMPACT_CONTINUE; @@ -1524,16 +1522,16 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, return false; } -static enum compact_result compact_zone(struct zone *zone, struct compact_control *cc) +static enum compact_result compact_zone(struct compact_control *cc) { enum compact_result ret; - unsigned long start_pfn = zone->zone_start_pfn; - unsigned long end_pfn = zone_end_pfn(zone); + unsigned long start_pfn = cc->zone->zone_start_pfn; + unsigned long end_pfn = zone_end_pfn(cc->zone); unsigned long last_migrated_pfn; const bool sync = cc->mode != MIGRATE_ASYNC; cc->migratetype = gfpflags_to_migratetype(cc->gfp_mask); - ret = compaction_suitable(zone, cc->order, cc->alloc_flags, + ret = compaction_suitable(cc->zone, cc->order, cc->alloc_flags, cc->classzone_idx); /* Compaction is likely to fail */ if (ret == COMPACT_SUCCESS || ret == COMPACT_SKIPPED) @@ -1546,8 +1544,8 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro * Clear pageblock skip if there were failures recently and compaction * is about to be retried after being deferred. */ - if (compaction_restarting(zone, cc->order)) - __reset_isolation_suitable(zone); + if (compaction_restarting(cc->zone, cc->order)) + __reset_isolation_suitable(cc->zone); /* * Setup to move all movable pages to the end of the zone. Used cached @@ -1559,16 +1557,16 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro cc->migrate_pfn = start_pfn; cc->free_pfn = pageblock_start_pfn(end_pfn - 1); } else { - cc->migrate_pfn = zone->compact_cached_migrate_pfn[sync]; - cc->free_pfn = zone->compact_cached_free_pfn; + cc->migrate_pfn = cc->zone->compact_cached_migrate_pfn[sync]; + cc->free_pfn = cc->zone->compact_cached_free_pfn; if (cc->free_pfn < start_pfn || cc->free_pfn >= end_pfn) { cc->free_pfn = pageblock_start_pfn(end_pfn - 1); - zone->compact_cached_free_pfn = cc->free_pfn; + cc->zone->compact_cached_free_pfn = cc->free_pfn; } if (cc->migrate_pfn < start_pfn || cc->migrate_pfn >= end_pfn) { cc->migrate_pfn = start_pfn; - zone->compact_cached_migrate_pfn[0] = cc->migrate_pfn; - zone->compact_cached_migrate_pfn[1] = cc->migrate_pfn; + cc->zone->compact_cached_migrate_pfn[0] = cc->migrate_pfn; + cc->zone->compact_cached_migrate_pfn[1] = cc->migrate_pfn; } if (cc->migrate_pfn == start_pfn) @@ -1582,11 +1580,11 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro migrate_prep_local(); - while ((ret = compact_finished(zone, cc)) == COMPACT_CONTINUE) { + while ((ret = compact_finished(cc)) == COMPACT_CONTINUE) { int err; unsigned long start_pfn = cc->migrate_pfn; - switch (isolate_migratepages(zone, cc)) { + switch (isolate_migratepages(cc->zone, cc)) { case ISOLATE_ABORT: ret = COMPACT_CONTENDED; putback_movable_pages(&cc->migratepages); @@ -1653,7 +1651,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro if (last_migrated_pfn < current_block_start) { cpu = get_cpu(); lru_add_drain_cpu(cpu); - drain_local_pages(zone); + drain_local_pages(cc->zone); put_cpu(); /* No more flushing until we migrate again */ last_migrated_pfn = 0; @@ -1678,8 +1676,8 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro * Only go back, not forward. The cached pfn might have been * already reset to zone end in compact_finished() */ - if (free_pfn > zone->compact_cached_free_pfn) - zone->compact_cached_free_pfn = free_pfn; + if (free_pfn > cc->zone->compact_cached_free_pfn) + cc->zone->compact_cached_free_pfn = free_pfn; } count_compact_events(COMPACTMIGRATE_SCANNED, cc->total_migrate_scanned); @@ -1716,7 +1714,7 @@ static enum compact_result compact_zone_order(struct zone *zone, int order, INIT_LIST_HEAD(&cc.freepages); INIT_LIST_HEAD(&cc.migratepages); - ret = compact_zone(zone, &cc); + ret = compact_zone(&cc); VM_BUG_ON(!list_empty(&cc.freepages)); VM_BUG_ON(!list_empty(&cc.migratepages)); @@ -1834,7 +1832,7 @@ static void compact_node(int nid) INIT_LIST_HEAD(&cc.freepages); INIT_LIST_HEAD(&cc.migratepages); - compact_zone(zone, &cc); + compact_zone(&cc); VM_BUG_ON(!list_empty(&cc.freepages)); VM_BUG_ON(!list_empty(&cc.migratepages)); @@ -1968,7 +1966,7 @@ static void kcompactd_do_work(pg_data_t *pgdat) if (kthread_should_stop()) return; - status = compact_zone(zone, &cc); + status = compact_zone(&cc); if (status == COMPACT_SUCCESS) { compaction_defer_reset(zone, cc.order, false); From patchwork Fri Jan 18 17:51:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771363 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CB36291E for ; Fri, 18 Jan 2019 17:52:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B845D2FC07 for ; Fri, 18 Jan 2019 17:52:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id ABE102FEF7; Fri, 18 Jan 2019 17:52:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6318A2FC07 for ; Fri, 18 Jan 2019 17:52:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 90D048E0014; Fri, 18 Jan 2019 12:52:50 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8BB548E0002; Fri, 18 Jan 2019 12:52:50 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D1248E0014; Fri, 18 Jan 2019 12:52:50 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id 26D288E0002 for ; Fri, 18 Jan 2019 12:52:50 -0500 (EST) Received: by mail-ed1-f71.google.com with SMTP id e29so5312915ede.19 for ; Fri, 18 Jan 2019 09:52:50 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=C0isV1ZJDgqFU3QoOVAtWQlnGfgiq1t7rqJXM+ekiIs=; b=fVXp9op+ocm/5jCnIKhmTBKDUHvrLApNI4WFw0ntLCgMIqWOyQ2f7cXTFK6nFY68Tn 2Tdt0NjlxODzfTLiDWmzZEdzb4EPn5P15XtVYD9lnmnEJ+14UyyXcNTylElW4Lgg4zqv b5QXei2CbcHzlNkHAZXNnu6d6yLhhD1XCmkzP/EUM04YsiqYB/kXGoTuNllzgiTg3Leb 2MOJOFZf+0obP4MWrd1XBlsoT1VfaXkdvZuuuYNm1A/tIQpGonIF8oSc4qiYoeKGB7Kc JHc94FupIqZ8GGtvV9aguPf4qBE3ow0tErybSGqumBeQWBLS3+XGN+Dc1o6G1B6H3SQz DMvg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukfDF199oEwweX68/5hPMre3ZFFPLaMGINbSF31Z8JJ8yt4E0GOR dtDkwabSScLx/efRtkxQMHaKPkxh7xY3fwHADJj5AQPPDM5hRdwfImLoKqEy0VJlyL57vICqmnY OrVK+wymUhbR5efhIFYaZI6xt0DnwmDntDFam7IKVFcuabO3j4R6+hbsBy8aKJxcG7Q== X-Received: by 2002:a50:95ce:: with SMTP id x14mr16683941eda.204.1547833969638; Fri, 18 Jan 2019 09:52:49 -0800 (PST) X-Google-Smtp-Source: ALg8bN4Cb4zotVFM8CU96m20gtFKkX8rPKrT8ytuixYJPYZF1CltEdtstBfEzR1BQw4ueAvD2Wo6 X-Received: by 2002:a50:95ce:: with SMTP id x14mr16683887eda.204.1547833968647; Fri, 18 Jan 2019 09:52:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547833968; cv=none; d=google.com; s=arc-20160816; b=v3MXnIhnf1JgE9+r3rC+MyWH6A+ZaMfFh+gAPILkbQ6v7JhdO8mWX4AREZZDw1C0R5 v2rnXze60hzfQJf84Zq0YSKW6UevV09VfzFswfu4H42C8HMFH1bwJ773Ik57MRtr7hhP HMr6Jo//5bPKLIiCO4ne3pxCuxll3L3t51ZjapT2fDR1UHmSvhGQjJSWAZAtbpAzaXWK qOGcxVUMoUV1fFyrVv/+ygI1aGhzENjnKZTc8T2ZXSUuJ1EK+NrANcxGLtxpZZiJLK9T /o9rzpmrY+YJj0DMt1Lqz/dGuU2wOBnr8b4THLouo1NrF1l7p0lnECkK6YUUN+gDxsZ2 +JCw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=C0isV1ZJDgqFU3QoOVAtWQlnGfgiq1t7rqJXM+ekiIs=; b=A9VQW+U+/A1NPMtPkKUmY2Fw2kGzGXsDEljqWaLHg3sAULVtm+QJOpQJWZN0IQDZwd rVKjiwD30CRjlTjX/Y/9zyOa50oUTszLIpa2F6dhwECT1mbNvZzzan54/1uqbOztZtSJ fpI8R6WIW9dkLkKs7cYOC2uTzwbSCdJCHS5IE6tHWBxyYUPS5bqg4igRSULlqr3qymK4 6dh927Li6Q29LRD07KR26dhorig59mAzBpmzr3dNxcl1D7gEjcU2Cms4V9olbyzogSXR gCREx2t+0kEZHq27oNLDTXIwC0XghiRfvKvO2lyouZfMOsd1CGYp2pnaK0ofqaOgUazG 3Idw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp16.blacknight.com (outbound-smtp16.blacknight.com. [46.22.139.233]) by mx.google.com with ESMTPS id m3-v6si1788975ejb.316.2019.01.18.09.52.48 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:52:48 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) client-ip=46.22.139.233; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp16.blacknight.com (Postfix) with ESMTPS id 341701C35B5 for ; Fri, 18 Jan 2019 17:52:48 +0000 (GMT) Received: (qmail 750 invoked from network); 18 Jan 2019 17:52:48 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:52:48 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 06/22] mm, migrate: Immediately fail migration of a page with no migration handler Date: Fri, 18 Jan 2019 17:51:20 +0000 Message-Id: <20190118175136.31341-7-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Pages with no migration handler use a fallback handler which sometimes works and sometimes persistently retries. A historical example was blockdev pages but there are others such as odd refcounting when page->private is used. These are retried multiple times which is wasteful during compaction so this patch will fail migration faster unless the caller specifies MIGRATE_SYNC. This is not expected to help THP allocation success rates but it did reduce latencies very slightly in some cases. 1-socket thpfioscale 4.20.0 4.20.0 noreserved-v2r15 failfast-v2r15 Amean fault-both-1 0.00 ( 0.00%) 0.00 * 0.00%* Amean fault-both-3 3839.67 ( 0.00%) 3833.72 ( 0.15%) Amean fault-both-5 5177.47 ( 0.00%) 4967.15 ( 4.06%) Amean fault-both-7 7245.03 ( 0.00%) 7139.19 ( 1.46%) Amean fault-both-12 11534.89 ( 0.00%) 11326.30 ( 1.81%) Amean fault-both-18 16241.10 ( 0.00%) 16270.70 ( -0.18%) Amean fault-both-24 19075.91 ( 0.00%) 19839.65 ( -4.00%) Amean fault-both-30 22712.11 ( 0.00%) 21707.05 ( 4.43%) Amean fault-both-32 21692.92 ( 0.00%) 21968.16 ( -1.27%) The 2-socket results are not materially different. Scan rates are similar as expected. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/migrate.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/migrate.c b/mm/migrate.c index 52b04c217e30..4512afab46ac 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -916,7 +916,7 @@ static int fallback_migrate_page(struct address_space *mapping, */ if (page_has_private(page) && !try_to_release_page(page, GFP_KERNEL)) - return -EAGAIN; + return mode == MIGRATE_SYNC ? -EAGAIN : -EBUSY; return migrate_page(mapping, newpage, page, mode); } From patchwork Fri Jan 18 17:51:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771365 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 618B06C5 for ; Fri, 18 Jan 2019 17:53:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4F9A32FC07 for ; Fri, 18 Jan 2019 17:53:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 429072FEF7; Fri, 18 Jan 2019 17:53:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B50D12FC07 for ; Fri, 18 Jan 2019 17:53:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D53448E0015; Fri, 18 Jan 2019 12:53:00 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D01E38E0002; Fri, 18 Jan 2019 12:53:00 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C177D8E0015; Fri, 18 Jan 2019 12:53:00 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id 6AFBC8E0002 for ; Fri, 18 Jan 2019 12:53:00 -0500 (EST) Received: by mail-ed1-f71.google.com with SMTP id d41so5132626eda.12 for ; Fri, 18 Jan 2019 09:53:00 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=GIODWge4DE70mVQvwggzR7h97SS/9XUfczXOJuwvTWs=; b=sMP5jvgwV/17VSUm8Ivluof22pLODaKvLCt94AX47ViWyEjrOtZiRHtwvAAetH7Sci DVh2BPxEJF/jOMC0IQS9wX6By7RaZjvn8UAE7f+dLXZ8Zq05YHHznhgoN9knnSb104Wt SBDDz84ztY2l5orsVHADqzIjuxcoJVGB5t3kFofBi8qxgJe1ODvNbR9bAF38Ce4sKF73 MDufaD3ekeOw+/faQZFnEKw/y5itHb9Z2rwuyNpCgIogaq+qNF4WnqVRyYBzJpQ3wMJj E4E6OyyV8hrzVMCx3TH8tvtIzPoc6bU7hV0YNX0nbVFWVXTonVB+oEKhJkpFVyt8gZLJ O88A== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukd0MCo5JqsIX2CmhYaKcvQugRuw7RDCBHG1lWamjfnwI9GwwAea aqmJXuQGjpWomxgHaaDd6eqp0twllOO3TH7jg6/w0dZYJcVVAQ25jUmBQ6zYIquNB2bCDkmVYSA dwPS0Fg6XhuoUBI8o9Ym3b0crUDWnYhU5jawFrPAd4RDDkcuLV6Hxcjr38aWQFpdnoA== X-Received: by 2002:a17:906:2615:: with SMTP id h21-v6mr15507519ejc.21.1547833979906; Fri, 18 Jan 2019 09:52:59 -0800 (PST) X-Google-Smtp-Source: ALg8bN4+iLQcUg+CQlSKuQ4I1+05aWfA25XDPmw+en6ANXX5zqy4sqFNBGGwaCLzEe2n0/HzE/JY X-Received: by 2002:a17:906:2615:: with SMTP id h21-v6mr15507447ejc.21.1547833978644; Fri, 18 Jan 2019 09:52:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547833978; cv=none; d=google.com; s=arc-20160816; b=b5EmYFLg+iazEly6z/DuGVhf6PIJrrA4WG48pWQtLXck+z3eNPEHIIuIf/HmJ6PWN8 eHsBhuK68yQ8UsXlcBroD3BRTFnoafPVqiI+sPHgC7whbj8GYgBh0aJikMYH4parBVfT poqPzDfsssnio5qJsbVuoZqPprQTshZpH2oh4brOk1EdC2G6zqk4wQzyRuZvGbe4e4aZ ICajHu6pm/0ivMCnX0nO272RIos/2rzoN9vKgWmjTyifg4nIMwW+54tHimJBQ+AsNWnN SufFJr0xdFpcYoev+3w9rTA+5EFoNZlBWNQizgI+R7oUN60vO56aaycEG8m/ClFFK8KZ UUzg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=GIODWge4DE70mVQvwggzR7h97SS/9XUfczXOJuwvTWs=; b=Px1yJhVyaWwBnMYPTN2QhPdxBDzf9LywYvjOvnsAy0P1wlcHAm5RFYkjifIs/8OVl+ /d6AKnYBSIqKRWg8V+49cUNSDWMiijlmmmy0fRhmykn5BYCJb70kcyqziT4vMfycBqH2 dt8LwKHrUnQTYp0G4QKQE7sjHwhcVTdXsvpyOEbPnNUfmR2AijTB2HvY3Hnu5uY/rVL/ 4k1z63a1m/YQe7qiw2J6Tew7hrVHDXAj0MFTvYRdiqlQ6CVxlTP4XK3QLAvDLJdK0rEL WcB7QpU8iEhxMJqmP4vE4/nsy+TDjFsvzbkdk97Pstk6dLSXaBHa7Q/JREemsnaIq2cm BkeA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp08.blacknight.com (outbound-smtp08.blacknight.com. [46.22.139.13]) by mx.google.com with ESMTPS id h13si7559633edf.24.2019.01.18.09.52.58 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:52:58 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) client-ip=46.22.139.13; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp08.blacknight.com (Postfix) with ESMTPS id 579DF1C35A9 for ; Fri, 18 Jan 2019 17:52:58 +0000 (GMT) Received: (qmail 1326 invoked from network); 18 Jan 2019 17:52:58 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:52:58 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 07/22] mm, compaction: Always finish scanning of a full pageblock Date: Fri, 18 Jan 2019 17:51:21 +0000 Message-Id: <20190118175136.31341-8-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When compaction is finishing, it uses a flag to ensure the pageblock is complete but it makes sense to always complete migration of a pageblock. Minimally, skip information is based on a pageblock and partially scanned pageblocks may incur more scanning in the future. The pageblock skip handling also becomes more strict later in the series and the hint is more useful if a complete pageblock was always scanned. The potentially impacts latency as more scanning is done but it's not a consistent win or loss as the scanning is not always a high percentage of the pageblock and sometimes it is offset by future reductions in scanning. Hence, the results are not presented this time due to a misleading mix of gains/losses without any clear pattern. However, full scanning of the pageblock is important for later patches. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 19 ++++++++----------- mm/internal.h | 1 - 2 files changed, 8 insertions(+), 12 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 32a88b49f973..3d11c209614a 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1331,16 +1331,14 @@ static enum compact_result __compact_finished(struct compact_control *cc) if (is_via_compact_memory(cc->order)) return COMPACT_CONTINUE; - if (cc->finishing_block) { - /* - * We have finished the pageblock, but better check again that - * we really succeeded. - */ - if (IS_ALIGNED(cc->migrate_pfn, pageblock_nr_pages)) - cc->finishing_block = false; - else - return COMPACT_CONTINUE; - } + /* + * Always finish scanning a pageblock to reduce the possibility of + * fallbacks in the future. This is particularly important when + * migration source is unmovable/reclaimable but it's not worth + * special casing. + */ + if (!IS_ALIGNED(cc->migrate_pfn, pageblock_nr_pages)) + return COMPACT_CONTINUE; /* Direct compactor: Is a suitable page free? */ for (order = cc->order; order < MAX_ORDER; order++) { @@ -1382,7 +1380,6 @@ static enum compact_result __compact_finished(struct compact_control *cc) return COMPACT_SUCCESS; } - cc->finishing_block = true; return COMPACT_CONTINUE; } } diff --git a/mm/internal.h b/mm/internal.h index f40d06d70683..9b32f4cab0ae 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -203,7 +203,6 @@ struct compact_control { bool direct_compaction; /* False from kcompactd or /proc/... */ bool whole_zone; /* Whole zone should/has been scanned */ bool contended; /* Signal lock or sched contention */ - bool finishing_block; /* Finishing current pageblock */ }; unsigned long From patchwork Fri Jan 18 17:51:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771367 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 221206C5 for ; Fri, 18 Jan 2019 17:53:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 119B12FC07 for ; Fri, 18 Jan 2019 17:53:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0606E2FEF7; Fri, 18 Jan 2019 17:53:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D54662FC07 for ; Fri, 18 Jan 2019 17:53:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E8A2E8E0016; Fri, 18 Jan 2019 12:53:10 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E3A808E0002; Fri, 18 Jan 2019 12:53:10 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D4F3F8E0016; Fri, 18 Jan 2019 12:53:10 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id 7DC468E0002 for ; Fri, 18 Jan 2019 12:53:10 -0500 (EST) Received: by mail-ed1-f69.google.com with SMTP id v4so5124829edm.18 for ; Fri, 18 Jan 2019 09:53:10 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=XY602cqHy6JklbpqaWogx0es27AK1EZmJD68GbOsrjc=; b=cC3u8ML+id9i6LUjth1jRCIxCRGfDgpiBeMBzRrmV0bumMDEpGK6t9uHoonp3vE/rW Vgk+RJn2RVfVjRexCjtWJd8VvfAwbkQa9hGpHGyKKadjFIBmmWt5Mlw928Mx7K4vlDpS rpopZpcNW47tv72BQr4HXaxzOlmFX2ZFB4g0LkSKJADDuMmsjXH4vHw0aHSlWDnQVOJJ Vq6Ww5DjJUVaXPkVTD6rbugeLAX2jnGZOspgIKtMpD6FFJtUkqLu/PWq92NdY2YKwlJc i4zAzciWS9BO25mROSlWm3quuZwqLSUXc5HyqZzjHW6Z9ED/axCwqjvWsg7AU4Aq5WLa maSw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukccbmFs97OqHS5QrNBzyYVE2Yee9j4mvLOusyQGovNK/RA2YCa0 LdvFin6auJeflaSOwTJEUfFxapHdKvXeABOVshOH8EWxWet0jBYO8GsP7O9BGGdAzeSkY8Ug+H/ K7LThV1QYEBsFoRyvtsZN4fyFni/Qvr77ya5PoXjRoElyJ90weLN7E7CP/X04+x429Q== X-Received: by 2002:a50:b103:: with SMTP id k3mr16145672edd.247.1547833989999; Fri, 18 Jan 2019 09:53:09 -0800 (PST) X-Google-Smtp-Source: ALg8bN47zsbkS8VwshLUMoO1Qp/4ql+orBEJ14D6VOl708vBlccvtxMlAJC2lXniUPomInZsulLg X-Received: by 2002:a50:b103:: with SMTP id k3mr16145603edd.247.1547833988839; Fri, 18 Jan 2019 09:53:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547833988; cv=none; d=google.com; s=arc-20160816; b=x//nFE2NAKfMo6C6YpUk+w8H/7i5y6E8CPHJiriZGEnTZLst/JfnbTwrc327TJk0j4 +pTYMtIC2eJ9490tPzMEiz1vxz1iijOYbVB0MSVjEsK5Zw3zFMEr1rTk51FiTvzTGqeV DLqzfE9AOuJQQCBn5OlH8QJ2/0qPjz9KUYmqYxguayNx4/2UCzk4cT+mSf0YCvAWS7Ze PbYUccaJOhLGSLPFgBJn4j8m4mdgKr+lC7RyN7uXoEyaZYbMzZQRPPtrXJPCO8VW51W4 dqCjeRKv46Da9HvMMsg3esv7mFjorbowqHK8TtDg5MmtPftw6kvkIz/GWUXOi9ENitnO fHqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=XY602cqHy6JklbpqaWogx0es27AK1EZmJD68GbOsrjc=; b=AyMCikrUEeqjSK3HSRuDpXg0gMTohsvPWPDDcdITm3GWrhHJpi8RbkXBDF/gEaKQPs ZzDe3ogI1EsSpdywDnZeHrv8RMLHJ8QXQTWPOsH9ovIUvztBgbU1gtiE38GEVqcMn5hC MSjmFVbiINlOciVBMGQKjOS5Ox8+yfF0l+K/4q0Gp6s2bnZlmTLk6Eujp8j14EIRXNaC Iposlp1ufOHDIM+N6JIeYbPk5uczYhTgngQ4Z1ynwrQajtNnAZtko46B0sAsApDxagSh F6Ym5UmgXwo3DHmN07ANyiuAy2Rhxv7obP1m+WvxwImuJTTt/Au/neDIrE68BU3Q+MSI Ji2A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp16.blacknight.com (outbound-smtp16.blacknight.com. [46.22.139.233]) by mx.google.com with ESMTPS id b11si2180785edj.393.2019.01.18.09.53.08 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:53:08 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) client-ip=46.22.139.233; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp16.blacknight.com (Postfix) with ESMTPS id 803011C35AE for ; Fri, 18 Jan 2019 17:53:08 +0000 (GMT) Received: (qmail 2023 invoked from network); 18 Jan 2019 17:53:08 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:53:08 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 08/22] mm, compaction: Ignore the fragmentation avoidance boost for isolation and compaction Date: Fri, 18 Jan 2019 17:51:22 +0000 Message-Id: <20190118175136.31341-9-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When pageblocks get fragmented, watermarks are artifically boosted to reclaim pages to avoid further fragmentation events. However, compaction is often either fragmentation-neutral or moving movable pages away from unmovable/reclaimable pages. As the true watermarks are preserved, allow compaction to ignore the boost factor. The expected impact is very slight as the main benefit is that compaction is slightly more likely to succeed when the system has been fragmented very recently. On both 1-socket and 2-socket machines for THP-intensive allocation during fragmentation the success rate was increased by less than 1% which is marginal. However, detailed tracing indicated that failure of migration due to a premature ENOMEM triggered by watermark checks were eliminated. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index fc769ff4fb2c..6607cb7131b0 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3071,7 +3071,7 @@ int __isolate_free_page(struct page *page, unsigned int order) * watermark, because we already know our high-order page * exists. */ - watermark = min_wmark_pages(zone) + (1UL << order); + watermark = zone->_watermark[WMARK_MIN] + (1UL << order); if (!zone_watermark_ok(zone, 0, watermark, 0, ALLOC_CMA)) return 0; From patchwork Fri Jan 18 17:51:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771369 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A5D1F91E for ; Fri, 18 Jan 2019 17:53:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 93F1D2FF1B for ; Fri, 18 Jan 2019 17:53:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 881722FF7F; Fri, 18 Jan 2019 17:53:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 836CB2FF1B for ; Fri, 18 Jan 2019 17:53:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 70EAA8E0017; Fri, 18 Jan 2019 12:53:21 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6BF018E0002; Fri, 18 Jan 2019 12:53:21 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5D83C8E0017; Fri, 18 Jan 2019 12:53:21 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id 07E258E0002 for ; Fri, 18 Jan 2019 12:53:21 -0500 (EST) Received: by mail-ed1-f69.google.com with SMTP id i55so5263617ede.14 for ; Fri, 18 Jan 2019 09:53:20 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=AqJLYoE7Zbb/Ide/2MXl1VQOSujIbLcZu3ZyxAJmez8=; b=Ql4GO+oW4CeaUXDuCj/5rnySmghUivnkkFwCEmCx1pwxxaTABUHBkvH0dk4WKTrv0S tm4PvKS64LTEau1dAL3pX/oeGJxFrGXnE0CQAw7ZYcbhBM5Q102Ed5kQT74cpPj8mx5v ir0o0Vacs2yUHbcXOlCLJhwUTKM59jJNpes6xtkCPh59A9dxQ6pxR7TNfhIHVEbpTNus roCNAGfeYpUbuA2Y+dxVcku9YKmOYCJ5td7um9CIRNXOxY6dykwP5s2Wfzm0+8LW7W4P AL/dRbESpuLYCaTTmiKRMAMTyfHrdgeT/6FI55XSaQr3xLGmy1ScxLjjYfTypcm5QyPt hbxg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukfOVi4A1+mK9W+HQ5U4NKP06/ZxYesHlYcyYbfYWggOj7BoWHQz uDxwWqMocejBRIeji6ReeLIWZIn0jikEXKftl2lmeD+HxlQEWVHKBLouSAgAG+Y8Avu3La/mpVW vZcMrVeg0S8B7GnEXrFg8FJuAMcpQVuoQA7iFu7VQKLBNhkQ04LxcFASAg+75qXIqEA== X-Received: by 2002:a50:ca86:: with SMTP id x6mr16239719edh.287.1547834000483; Fri, 18 Jan 2019 09:53:20 -0800 (PST) X-Google-Smtp-Source: ALg8bN7mC4IfdGESIqB3ConJd+ywa8DIlQ4KnTeEnarP4XfMI8z9u7xaHnKYvf0bajvAj2zqTBOD X-Received: by 2002:a50:ca86:: with SMTP id x6mr16239636edh.287.1547833999021; Fri, 18 Jan 2019 09:53:19 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547833999; cv=none; d=google.com; s=arc-20160816; b=g//5exKtK8NDnx4yBSmVXNPZMrEr/EiV+KxvIZ+n2kneRDM/Esz8YuIe9P8zhUxZZ0 88b4d+peFYB7USV6J1AhVk26qEWd6T+/hx9+bGJqtqEN5GjdLzJDa/Uh8wWo2dcLOFGJ IzEV4b3JusKIpAPYtSLcTWw2C48VVJ+UyCKlmKvQ3JHZWruCv0IQScrgARjDAGsM2iHX /mL9tRC2Zg9tzhZvxqwTHvzo0409wch1UhC3vtxzhSSj7jZqCaPkU4+0IAcCTDeBoOEs yNDP9L3fccXMq2sCP0996J17PS10p9c9bHZmxsbIL7s1u1mDOeKeX6rBEUXIRXtJsPZS R6Hg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=AqJLYoE7Zbb/Ide/2MXl1VQOSujIbLcZu3ZyxAJmez8=; b=LfL+ONS2b7xwqMja7qNZVPt36JiF+7dyPBDvf+jgAYkrAtmlQsDyTTb65bDHR17+VA PB+sMS67GhEbwEmi0dtNIISZkzrzFh3k3L+NBFp+54/+5PzQgjZDkuGTSKUd3fBFUgRa JGLAtBGKh+7CUNp/Mo/wjELL59HyrIuC9fs9FKlD+PwsIGj/ORnprPtoJ/todachIxFu MRt9ZRQb3cXlt5vQYoobyR8f07k7bc/NujcdYH5wYo3xaGu6uWFG0vFSHFT4wVNF3CGu LjvOptHXIrpJ61GJf/Cb2sSZDBFFUiKcnzSyoxSmLa7egPcIZBKDn3jM0JCmpHAjK/E1 zm5w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp11.blacknight.com (outbound-smtp11.blacknight.com. [46.22.139.106]) by mx.google.com with ESMTPS id r18-v6si2086662ejf.218.2019.01.18.09.53.18 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:53:19 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) client-ip=46.22.139.106; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp11.blacknight.com (Postfix) with ESMTPS id AA68F1C35D1 for ; Fri, 18 Jan 2019 17:53:18 +0000 (GMT) Received: (qmail 2691 invoked from network); 18 Jan 2019 17:53:18 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:53:18 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 09/22] mm, compaction: Use free lists to quickly locate a migration source Date: Fri, 18 Jan 2019 17:51:23 +0000 Message-Id: <20190118175136.31341-10-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The migration scanner is a linear scan of a zone with a potentiall large search space. Furthermore, many pageblocks are unusable such as those filled with reserved pages or partially filled with pages that cannot migrate. These still get scanned in the common case of allocating a THP and the cost accumulates. The patch uses a partial search of the free lists to locate a migration source candidate that is marked as MOVABLE when allocating a THP. It prefers picking a block with a larger number of free pages already on the basis that there are fewer pages to migrate to free the entire block. The lowest PFN found during searches is tracked as the basis of the start for the linear search after the first search of the free list fails. After the search, the free list is shuffled so that the next search will not encounter the same page. If the search fails then the subsequent searches will be shorter and the linear scanner is used. If this search fails, or if the request is for a small or unmovable/reclaimable allocation then the linear scanner is still used. It is somewhat pointless to use the list search in those cases. Small free pages must be used for the search and there is no guarantee that movable pages are located within that block that are contiguous. 5.0.0-rc1 5.0.0-rc1 noboost-v3r10 findmig-v3r15 Amean fault-both-3 3771.41 ( 0.00%) 3390.40 ( 10.10%) Amean fault-both-5 5409.05 ( 0.00%) 5082.28 ( 6.04%) Amean fault-both-7 7040.74 ( 0.00%) 7012.51 ( 0.40%) Amean fault-both-12 11887.35 ( 0.00%) 11346.63 ( 4.55%) Amean fault-both-18 16718.19 ( 0.00%) 15324.19 ( 8.34%) Amean fault-both-24 21157.19 ( 0.00%) 16088.50 * 23.96%* Amean fault-both-30 21175.92 ( 0.00%) 18723.42 * 11.58%* Amean fault-both-32 21339.03 ( 0.00%) 18612.01 * 12.78%* 5.0.0-rc1 5.0.0-rc1 noboost-v3r10 findmig-v3r15 Percentage huge-3 86.50 ( 0.00%) 89.83 ( 3.85%) Percentage huge-5 92.52 ( 0.00%) 91.96 ( -0.61%) Percentage huge-7 92.44 ( 0.00%) 92.85 ( 0.44%) Percentage huge-12 92.98 ( 0.00%) 92.74 ( -0.25%) Percentage huge-18 91.70 ( 0.00%) 91.71 ( 0.02%) Percentage huge-24 91.59 ( 0.00%) 92.13 ( 0.60%) Percentage huge-30 90.14 ( 0.00%) 93.79 ( 4.04%) Percentage huge-32 90.03 ( 0.00%) 91.27 ( 1.37%) This shows an improvement in allocation latencies with similar allocation success rates. While not presented, there was a 31% reduction in migration scanning and a 8% reduction on system CPU usage. A 2-socket machine showed similar benefits. [vbabka@suse.cz: Migrate block that was found-fast, some optimisations] Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 176 +++++++++++++++++++++++++++++++++++++++++++++++++++++++- mm/internal.h | 2 + 2 files changed, 175 insertions(+), 3 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 3d11c209614a..92d10eb3d1c7 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1040,6 +1040,12 @@ static bool suitable_migration_target(struct compact_control *cc, return false; } +static inline unsigned int +freelist_scan_limit(struct compact_control *cc) +{ + return (COMPACT_CLUSTER_MAX >> cc->fast_search_fail) + 1; +} + /* * Test whether the free scanner has reached the same or lower pageblock than * the migration scanner, and compaction should thus terminate. @@ -1050,6 +1056,19 @@ static inline bool compact_scanners_met(struct compact_control *cc) <= (cc->migrate_pfn >> pageblock_order); } +/* Reorder the free list to reduce repeated future searches */ +static void +move_freelist_tail(struct list_head *freelist, struct page *freepage) +{ + LIST_HEAD(sublist); + + if (!list_is_last(freelist, &freepage->lru)) { + list_cut_position(&sublist, freelist, &freepage->lru); + if (!list_empty(&sublist)) + list_splice_tail(&sublist, freelist); + } +} + /* * Based on information in the current compact_control, find blocks * suitable for isolating free pages from and then isolate them. @@ -1207,6 +1226,146 @@ typedef enum { */ int sysctl_compact_unevictable_allowed __read_mostly = 1; +static inline void +update_fast_start_pfn(struct compact_control *cc, unsigned long pfn) +{ + if (cc->fast_start_pfn == ULONG_MAX) + return; + + if (!cc->fast_start_pfn) + cc->fast_start_pfn = pfn; + + cc->fast_start_pfn = min(cc->fast_start_pfn, pfn); +} + +static inline void +reinit_migrate_pfn(struct compact_control *cc) +{ + if (!cc->fast_start_pfn || cc->fast_start_pfn == ULONG_MAX) + return; + + cc->migrate_pfn = cc->fast_start_pfn; + cc->fast_start_pfn = ULONG_MAX; +} + +/* + * Briefly search the free lists for a migration source that already has + * some free pages to reduce the number of pages that need migration + * before a pageblock is free. + */ +static unsigned long fast_find_migrateblock(struct compact_control *cc) +{ + unsigned int limit = freelist_scan_limit(cc); + unsigned int nr_scanned = 0; + unsigned long distance; + unsigned long pfn = cc->migrate_pfn; + unsigned long high_pfn; + int order; + + /* Skip hints are relied on to avoid repeats on the fast search */ + if (cc->ignore_skip_hint) + return pfn; + + /* + * If the migrate_pfn is not at the start of a zone or the start + * of a pageblock then assume this is a continuation of a previous + * scan restarted due to COMPACT_CLUSTER_MAX. + */ + if (pfn != cc->zone->zone_start_pfn && pfn != pageblock_start_pfn(pfn)) + return pfn; + + /* + * For smaller orders, just linearly scan as the number of pages + * to migrate should be relatively small and does not necessarily + * justify freeing up a large block for a small allocation. + */ + if (cc->order <= PAGE_ALLOC_COSTLY_ORDER) + return pfn; + + /* + * Only allow kcompactd and direct requests for movable pages to + * quickly clear out a MOVABLE pageblock for allocation. This + * reduces the risk that a large movable pageblock is freed for + * an unmovable/reclaimable small allocation. + */ + if (cc->direct_compaction && cc->migratetype != MIGRATE_MOVABLE) + return pfn; + + /* + * When starting the migration scanner, pick any pageblock within the + * first half of the search space. Otherwise try and pick a pageblock + * within the first eighth to reduce the chances that a migration + * target later becomes a source. + */ + distance = (cc->free_pfn - cc->migrate_pfn) >> 1; + if (cc->migrate_pfn != cc->zone->zone_start_pfn) + distance >>= 2; + high_pfn = pageblock_start_pfn(cc->migrate_pfn + distance); + + for (order = cc->order - 1; + order >= PAGE_ALLOC_COSTLY_ORDER && pfn == cc->migrate_pfn && nr_scanned < limit; + order--) { + struct free_area *area = &cc->zone->free_area[order]; + struct list_head *freelist; + unsigned long flags; + struct page *freepage; + + if (!area->nr_free) + continue; + + spin_lock_irqsave(&cc->zone->lock, flags); + freelist = &area->free_list[MIGRATE_MOVABLE]; + list_for_each_entry(freepage, freelist, lru) { + unsigned long free_pfn; + + nr_scanned++; + free_pfn = page_to_pfn(freepage); + if (free_pfn < high_pfn) { + update_fast_start_pfn(cc, free_pfn); + + /* + * Avoid if skipped recently. Ideally it would + * move to the tail but even safe iteration of + * the list assumes an entry is deleted, not + * reordered. + */ + if (get_pageblock_skip(freepage)) { + if (list_is_last(freelist, &freepage->lru)) + break; + + continue; + } + + /* Reorder to so a future search skips recent pages */ + move_freelist_tail(freelist, freepage); + + pfn = pageblock_start_pfn(free_pfn); + cc->fast_search_fail = 0; + set_pageblock_skip(freepage); + break; + } + + if (nr_scanned >= limit) { + cc->fast_search_fail++; + move_freelist_tail(freelist, freepage); + break; + } + } + spin_unlock_irqrestore(&cc->zone->lock, flags); + } + + cc->total_migrate_scanned += nr_scanned; + + /* + * If fast scanning failed then use a cached entry for a page block + * that had free pages as the basis for starting a linear scan. + */ + if (pfn == cc->migrate_pfn) + reinit_migrate_pfn(cc); + + return pfn; +} + /* * Isolate all pages that can be migrated from the first suitable block, * starting at the block pointed to by the migrate scanner pfn within @@ -1222,16 +1381,25 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone, const isolate_mode_t isolate_mode = (sysctl_compact_unevictable_allowed ? ISOLATE_UNEVICTABLE : 0) | (cc->mode != MIGRATE_SYNC ? ISOLATE_ASYNC_MIGRATE : 0); + bool fast_find_block; /* * Start at where we last stopped, or beginning of the zone as - * initialized by compact_zone() + * initialized by compact_zone(). The first failure will use + * the lowest PFN as the starting point for linear scanning. */ - low_pfn = cc->migrate_pfn; + low_pfn = fast_find_migrateblock(cc); block_start_pfn = pageblock_start_pfn(low_pfn); if (block_start_pfn < zone->zone_start_pfn) block_start_pfn = zone->zone_start_pfn; + /* + * fast_find_migrateblock marks a pageblock skipped so to avoid + * the isolation_suitable check below, check whether the fast + * search was successful. + */ + fast_find_block = low_pfn != cc->migrate_pfn && !cc->fast_search_fail; + /* Only scan within a pageblock boundary */ block_end_pfn = pageblock_end_pfn(low_pfn); @@ -1240,6 +1408,7 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone, * Do not cross the free scanner. */ for (; block_end_pfn <= cc->free_pfn; + fast_find_block = false, low_pfn = block_end_pfn, block_start_pfn = block_end_pfn, block_end_pfn += pageblock_nr_pages) { @@ -1259,7 +1428,7 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone, continue; /* If isolation recently failed, do not retry */ - if (!isolation_suitable(cc, page)) + if (!isolation_suitable(cc, page) && !fast_find_block) continue; /* @@ -1550,6 +1719,7 @@ static enum compact_result compact_zone(struct compact_control *cc) * want to compact the whole zone), but check that it is initialised * by ensuring the values are within zone boundaries. */ + cc->fast_start_pfn = 0; if (cc->whole_zone) { cc->migrate_pfn = start_pfn; cc->free_pfn = pageblock_start_pfn(end_pfn - 1); diff --git a/mm/internal.h b/mm/internal.h index 9b32f4cab0ae..983cb975545f 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -188,9 +188,11 @@ struct compact_control { unsigned int nr_migratepages; /* Number of pages to migrate */ unsigned long free_pfn; /* isolate_freepages search base */ unsigned long migrate_pfn; /* isolate_migratepages search base */ + unsigned long fast_start_pfn; /* a pfn to start linear scan from */ struct zone *zone; unsigned long total_migrate_scanned; unsigned long total_free_scanned; + unsigned int fast_search_fail; /* failures to use free list searches */ const gfp_t gfp_mask; /* gfp mask of a direct compactor */ int order; /* order a direct compactor needs */ int migratetype; /* migratetype of direct compactor */ From patchwork Fri Jan 18 17:51:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771371 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D40E591E for ; Fri, 18 Jan 2019 17:53:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C0E1A2FF1B for ; Fri, 18 Jan 2019 17:53:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B3AF12FF7F; Fri, 18 Jan 2019 17:53:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DCD902FF1B for ; Fri, 18 Jan 2019 17:53:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D2BCB8E0018; Fri, 18 Jan 2019 12:53:31 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CFFFB8E0002; Fri, 18 Jan 2019 12:53:31 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF0308E0018; Fri, 18 Jan 2019 12:53:31 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by kanga.kvack.org (Postfix) with ESMTP id 640978E0002 for ; Fri, 18 Jan 2019 12:53:31 -0500 (EST) Received: by mail-ed1-f70.google.com with SMTP id e29so5313619ede.19 for ; Fri, 18 Jan 2019 09:53:31 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=WlXZdPxbW4Slf9YZxH/vq+9M4F2fJaOayxvPx5Uo+ag=; b=JcdIY0qmblKRFbUFVHpF8BZ7PbSlJn6bjQ0danKHP+hUbXSiUIau0M0wqJ8/iqEsby Nj2pC9Uoe7fgLpkPrufO/9sbuwnXlK5etVeBhV03E5e1jpr7Bz0roXfbcLF+Qh88vvBT u303elKXOr3iG7fNadDmeksq1PkGNggUNnkXKpoFl7fAphvSgz2Mal2LFuYD1H/ocepy YV3NbymH2wz8+KZGN4g/YxuqsfxbAYBa0d+AQCeWpgMgEhE/5Fsdy6ywWccjcskSkPp7 CzBoY3sPzMOCA4QGgLNJ8UY8s8Y0OkXJwWwTD+z9p+asne6+RpTsZxPF3UI7QhqvqI/o /AbQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukfoU36aMb3bBPU0kXmXq6z80ihE3y+yK8/hLjGxaqI9JtLQi0Hr X8CwqsORihzhmUNBC3SzV/1zr9cV40rcI93SzAD2hsY4Q0h0t8Y0h1FNa3GBhy0tSBqFcZ90iUa nqEUs/b4MRxIE9VbouG9nm+ygT83955E5MwXuSn6eWdEnvBH29EGrjBBrq21lmEED/g== X-Received: by 2002:a17:906:2f88:: with SMTP id w8-v6mr15171904eji.114.1547834010824; Fri, 18 Jan 2019 09:53:30 -0800 (PST) X-Google-Smtp-Source: ALg8bN4OCoKcAHh6DXl7L/zkX/Z2yShxmKhl3GvK+tMhfDqAz2UpH6GnJbjU3Fny0pKxJM7UDzWK X-Received: by 2002:a17:906:2f88:: with SMTP id w8-v6mr15171825eji.114.1547834009123; Fri, 18 Jan 2019 09:53:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547834009; cv=none; d=google.com; s=arc-20160816; b=IAcEYBhfHSkEaGxerUEVHrmpxuoPRGbeuYxsLOAu2uYff/TuqU61BrRvAT7VVVLMBA JJBYQR29RW62KIYh+YOwBCsHg2dLrLJRClSeJh325DfszbbEXTBU9KZTF7nPTTRo2p8Z H+b/TcD5tul2DMxxb8JsJt+10MeWrPWiVknD7sKoyJT9dJf1KpqR46Mww1V27+e+wK+3 XN2RGYxJdLAY4lKEmK5K0LrNSz2LU9IofwV8Gfk238wrgIEoA6NX0IzniqvqqmAFS5H2 AbEsiO6RFCVnqEe8z2Ilo08vKCmOxwtHAjpx9/0rSMKtMtnzpqda0TCiqYoQNjDppBtY EX/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=WlXZdPxbW4Slf9YZxH/vq+9M4F2fJaOayxvPx5Uo+ag=; b=Z7jizAz0KsbM5X4u4q3Ap4HHh/iRrE90Xli/stifSEmn5LgHN2eUnGGRD7dp6vp0Zd kDbF0chK1QK1VT+bPZt+UNjEd38yeKiTMlsfHvH47fAggg2OUUuBW7UzDuYH/Wnqmsjh RUlCe3gQYyIz8bh/5HkTO+D9Mh+Sj7mkKM6uz/qji/5+tbjxXEOzgpKV4ZHk9iywDGNG 3lLIjL3//HZwEeMzP+npG/yoxXH9ArKhVme6sWRmq2q1LALzw44JHQvw+3JtlO19KuTJ wntKA5h5mbHjgjTYWkBiisV+w14j7Jn5HN6kjygnfFERaq4VE7IjFV0taafEIy9STHSH 3D7w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp26.blacknight.com (outbound-smtp26.blacknight.com. [81.17.249.194]) by mx.google.com with ESMTPS id a16-v6si230705ejx.192.2019.01.18.09.53.28 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:53:29 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) client-ip=81.17.249.194; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp26.blacknight.com (Postfix) with ESMTPS id C9604B87ED for ; Fri, 18 Jan 2019 17:53:28 +0000 (GMT) Received: (qmail 3552 invoked from network); 18 Jan 2019 17:53:28 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:53:28 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 10/22] mm, compaction: Keep migration source private to a single compaction instance Date: Fri, 18 Jan 2019 17:51:24 +0000 Message-Id: <20190118175136.31341-11-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Due to either a fast search of the free list or a linear scan, it is possible for multiple compaction instances to pick the same pageblock for migration. This is lucky for one scanner and increased scanning for all the others. It also allows a race between requests on which first allocates the resulting free block. This patch tests and updates the pageblock skip for the migration scanner carefully. When isolating a block, it will check and skip if the block is already in use. Once the zone lock is acquired, it will be rechecked so that only one scanner can set the pageblock skip for exclusive use. Any scanner contending will continue with a linear scan. The skip bit is still set if no pages can be isolated in a range. While this may result in redundant scanning, it avoids unnecessarily acquiring the zone lock when there are no suitable migration sources. 1-socket thpscale Amean fault-both-1 0.00 ( 0.00%) 0.00 * 0.00%* Amean fault-both-3 3390.40 ( 0.00%) 3024.41 ( 10.80%) Amean fault-both-5 5082.28 ( 0.00%) 4749.30 ( 6.55%) Amean fault-both-7 7012.51 ( 0.00%) 6454.95 ( 7.95%) Amean fault-both-12 11346.63 ( 0.00%) 10324.83 ( 9.01%) Amean fault-both-18 15324.19 ( 0.00%) 12896.82 * 15.84%* Amean fault-both-24 16088.50 ( 0.00%) 13470.60 * 16.27%* Amean fault-both-30 18723.42 ( 0.00%) 17143.99 ( 8.44%) Amean fault-both-32 18612.01 ( 0.00%) 17743.91 ( 4.66%) 5.0.0-rc1 5.0.0-rc1 findmig-v3r15 isolmig-v3r15 Percentage huge-3 89.83 ( 0.00%) 92.96 ( 3.48%) Percentage huge-5 91.96 ( 0.00%) 93.26 ( 1.41%) Percentage huge-7 92.85 ( 0.00%) 93.63 ( 0.84%) Percentage huge-12 92.74 ( 0.00%) 92.80 ( 0.07%) Percentage huge-18 91.71 ( 0.00%) 91.62 ( -0.10%) Percentage huge-24 92.13 ( 0.00%) 91.50 ( -0.69%) Percentage huge-30 93.79 ( 0.00%) 92.73 ( -1.13%) Percentage huge-32 91.27 ( 0.00%) 91.94 ( 0.74%) This shows a reasonable reduction in latency as multiple compaction scanners do not operate on the same blocks with a similar allocation success rate. Compaction migrate scanned 41093126 25646769 Migration scan rates are reduced by 38%. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 124 ++++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 99 insertions(+), 25 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 92d10eb3d1c7..7c4c9cce7907 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -285,13 +285,52 @@ void reset_isolation_suitable(pg_data_t *pgdat) } } +/* + * Sets the pageblock skip bit if it was clear. Note that this is a hint as + * locks are not required for read/writers. Returns true if it was already set. + */ +static bool test_and_set_skip(struct compact_control *cc, struct page *page, + unsigned long pfn) +{ + bool skip; + + /* Do no update if skip hint is being ignored */ + if (cc->ignore_skip_hint) + return false; + + if (!IS_ALIGNED(pfn, pageblock_nr_pages)) + return false; + + skip = get_pageblock_skip(page); + if (!skip && !cc->no_set_skip_hint) + set_pageblock_skip(page); + + return skip; +} + +static void update_cached_migrate(struct compact_control *cc, unsigned long pfn) +{ + struct zone *zone = cc->zone; + + pfn = pageblock_end_pfn(pfn); + + /* Set for isolation rather than compaction */ + if (cc->no_set_skip_hint) + return; + + if (pfn > zone->compact_cached_migrate_pfn[0]) + zone->compact_cached_migrate_pfn[0] = pfn; + if (cc->mode != MIGRATE_ASYNC && + pfn > zone->compact_cached_migrate_pfn[1]) + zone->compact_cached_migrate_pfn[1] = pfn; +} + /* * If no pages were isolated then mark this pageblock to be skipped in the * future. The information is later cleared by __reset_isolation_suitable(). */ static void update_pageblock_skip(struct compact_control *cc, - struct page *page, unsigned long nr_isolated, - bool migrate_scanner) + struct page *page, unsigned long nr_isolated) { struct zone *zone = cc->zone; unsigned long pfn; @@ -310,16 +349,8 @@ static void update_pageblock_skip(struct compact_control *cc, pfn = page_to_pfn(page); /* Update where async and sync compaction should restart */ - if (migrate_scanner) { - if (pfn > zone->compact_cached_migrate_pfn[0]) - zone->compact_cached_migrate_pfn[0] = pfn; - if (cc->mode != MIGRATE_ASYNC && - pfn > zone->compact_cached_migrate_pfn[1]) - zone->compact_cached_migrate_pfn[1] = pfn; - } else { - if (pfn < zone->compact_cached_free_pfn) - zone->compact_cached_free_pfn = pfn; - } + if (pfn < zone->compact_cached_free_pfn) + zone->compact_cached_free_pfn = pfn; } #else static inline bool isolation_suitable(struct compact_control *cc, @@ -334,10 +365,19 @@ static inline bool pageblock_skip_persistent(struct page *page) } static inline void update_pageblock_skip(struct compact_control *cc, - struct page *page, unsigned long nr_isolated, - bool migrate_scanner) + struct page *page, unsigned long nr_isolated) { } + +static void update_cached_migrate(struct compact_control *cc, unsigned long pfn) +{ +} + +static bool test_and_set_skip(struct compact_control *cc, struct page *page, + unsigned long pfn) +{ + return false; +} #endif /* CONFIG_COMPACTION */ /* @@ -567,7 +607,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, /* Update the pageblock-skip if the whole pageblock was scanned */ if (blockpfn == end_pfn) - update_pageblock_skip(cc, valid_page, total_isolated, false); + update_pageblock_skip(cc, valid_page, total_isolated); cc->total_free_scanned += nr_scanned; if (total_isolated) @@ -702,6 +742,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, unsigned long start_pfn = low_pfn; bool skip_on_failure = false; unsigned long next_skip_pfn = 0; + bool skip_updated = false; /* * Ensure that there are not too many pages isolated from the LRU @@ -768,8 +809,19 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, page = pfn_to_page(low_pfn); - if (!valid_page) + /* + * Check if the pageblock has already been marked skipped. + * Only the aligned PFN is checked as the caller isolates + * COMPACT_CLUSTER_MAX at a time so the second call must + * not falsely conclude that the block should be skipped. + */ + if (!valid_page && IS_ALIGNED(low_pfn, pageblock_nr_pages)) { + if (!cc->ignore_skip_hint && get_pageblock_skip(page)) { + low_pfn = end_pfn; + goto isolate_abort; + } valid_page = page; + } /* * Skip if free. We read page order here without zone lock @@ -850,8 +902,19 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, if (!locked) { locked = compact_trylock_irqsave(zone_lru_lock(zone), &flags, cc); - if (!locked) + + /* Allow future scanning if the lock is contended */ + if (!locked) { + clear_pageblock_skip(page); break; + } + + /* Try get exclusive access under lock */ + if (!skip_updated) { + skip_updated = true; + if (test_and_set_skip(cc, page, low_pfn)) + goto isolate_abort; + } /* Recheck PageLRU and PageCompound under lock */ if (!PageLRU(page)) @@ -929,15 +992,20 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, if (unlikely(low_pfn > end_pfn)) low_pfn = end_pfn; +isolate_abort: if (locked) spin_unlock_irqrestore(zone_lru_lock(zone), flags); /* - * Update the pageblock-skip information and cached scanner pfn, - * if the whole pageblock was scanned without isolating any page. + * Updated the cached scanner pfn if the pageblock was scanned + * without isolating a page. The pageblock may not be marked + * skipped already if there were no LRU pages in the block. */ - if (low_pfn == end_pfn) - update_pageblock_skip(cc, valid_page, nr_isolated, true); + if (low_pfn == end_pfn && !nr_isolated) { + if (valid_page && !skip_updated) + set_pageblock_skip(valid_page); + update_cached_migrate(cc, low_pfn); + } trace_mm_compaction_isolate_migratepages(start_pfn, low_pfn, nr_scanned, nr_isolated); @@ -1321,8 +1389,6 @@ static unsigned long fast_find_migrateblock(struct compact_control *cc) nr_scanned++; free_pfn = page_to_pfn(freepage); if (free_pfn < high_pfn) { - update_fast_start_pfn(cc, free_pfn); - /* * Avoid if skipped recently. Ideally it would * move to the tail but even safe iteration of @@ -1339,6 +1405,7 @@ static unsigned long fast_find_migrateblock(struct compact_control *cc) /* Reorder to so a future search skips recent pages */ move_freelist_tail(freelist, freepage); + update_fast_start_pfn(cc, free_pfn); pfn = pageblock_start_pfn(free_pfn); cc->fast_search_fail = 0; set_pageblock_skip(freepage); @@ -1427,8 +1494,15 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone, if (!page) continue; - /* If isolation recently failed, do not retry */ - if (!isolation_suitable(cc, page) && !fast_find_block) + /* + * If isolation recently failed, do not retry. Only check the + * pageblock once. COMPACT_CLUSTER_MAX causes a pageblock + * to be visited multiple times. Assume skip was checked + * before making it "skip" so other compaction instances do + * not scan the same block. + */ + if (IS_ALIGNED(low_pfn, pageblock_nr_pages) && + !fast_find_block && !isolation_suitable(cc, page)) continue; /* From patchwork Fri Jan 18 17:51:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771373 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3946D91E for ; Fri, 18 Jan 2019 17:53:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 281212FF1B for ; Fri, 18 Jan 2019 17:53:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1CA822FF7F; Fri, 18 Jan 2019 17:53:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4755D2FF1B for ; Fri, 18 Jan 2019 17:53:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 191838E001A; Fri, 18 Jan 2019 12:53:42 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 140FB8E0002; Fri, 18 Jan 2019 12:53:42 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0080A8E001A; Fri, 18 Jan 2019 12:53:41 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 9D0C28E0002 for ; Fri, 18 Jan 2019 12:53:41 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id b7so5259311eda.10 for ; Fri, 18 Jan 2019 09:53:41 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=i9X01A5ItcQcVL37AQu/chNb8qW0PJ1cJD/GV5eny6Y=; b=KIZEhb3cOKm9wGrwFBhBLXSrJ3izhu8ldauJ5Af6X6FKYdhdJcJUkCqfoHLJV9IN0P EoMyZM71p/zoY3Ty8M2B/22v6kaC0Tr3exIpjH+U/rKdaHXzynAOAEcVs7FZ7CJ5TPxz gtME7nlxE0foqYcuwr49FY/S7Eu6YaLiTxECX7UI8w9rJqkMY7iSVm7fgk36MoNToTTq jRRZeTypffJtpXfrv1u91sM/ONFY/0E6R5XRyCdVkfXKs33mG5qzGsD3A0TZ93XuxVrj AV3t+P/pf75xCJEAMpScblRxUfKRjQYGlibzsFywNqkLAtEsW8TImGVQ1NaXl5JA+4gM 6c6g== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukeNlpWAPQDU7+60Ck5LlYCV5Alp+A+E8Ik4P+8/fqpv6ynWGRqO NxBLnofvfzPzHXpsBp1tg59L5ayngcNgT5St1b+uTmnKQXFvhNuQkHXrJe/AXf9bgdHNgon4dfH d/G/Xm5spA8j+T0KXTyPB5iVVUx89u2+u5v0R8iFalBSNQeHm7S2x3CqJJ+vm4OeIOg== X-Received: by 2002:a17:906:a455:: with SMTP id cb21-v6mr15292151ejb.185.1547834021034; Fri, 18 Jan 2019 09:53:41 -0800 (PST) X-Google-Smtp-Source: ALg8bN6Rex5VwdmQNib0LYn1/EU51kiMHeoMQkU6i0D1xLCk0ic5QDdpfSt5+aisa5Gq4tKF/d3p X-Received: by 2002:a17:906:a455:: with SMTP id cb21-v6mr15292065ejb.185.1547834019325; Fri, 18 Jan 2019 09:53:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547834019; cv=none; d=google.com; s=arc-20160816; b=ue5sMBcPDPuNmcgqNNwNMFKm4aACB3b6gM08BMPAY0TKMappokpfETZPFCsv5wAO7C ySu38vpskvyQ8qJnq1nCtQcLHGdEW6vQZyHx3AttX9cl316tB5zPuLjXU10ldwsvhpM6 wyl6d6x/Eq/vKlDU5G7WO9iVGadNdoJ51YrViQpk4KJOERFo3DcBgw9+FXpeRXBg7PG6 5j6qBfQ05sMi6inImbCYD3u6RMKGepcfUujvSt5xSGiAbVN5B9295oDCmGH4xytg+bja pqAug1keFa4BTBZMhaTBdgT4T/Q+UlwfOYMgt2Fx+xsod8+8VH/iUAYnz3LnsiqtoJ9C yHhw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=i9X01A5ItcQcVL37AQu/chNb8qW0PJ1cJD/GV5eny6Y=; b=i9DJ68D0e15wcwqe3NR2oojKoRiQw1gOJ+hYdohcDvieI6bD3dnmX1Od8BLsY2Ko3h JC+9bjtCvU08z+4NKZyGN0P+Qy04Jo3ddrS2+2LLtlSsv9wcXRD2R2MSyxrDLB4nkLWu FNKchyRO4vRnyd/dbUSSGqUe8sJlEaqpZeMVmJ4ySTUBJJff1XKEPzAEfbaI09kAvjYh ZO/fPkjkocI2POw5MtbfecwUhekzJ5GPYurU4qrkqnnL7d39QCUcqb86B4f5HOMoPFdE aT9SvaGXdG3yHKY87Ox1N0Fir922/DqRbNS773sHnNp49ZXGflrhgUYrlD58PPa4iIjW QKEg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp16.blacknight.com (outbound-smtp16.blacknight.com. [46.22.139.233]) by mx.google.com with ESMTPS id h13-v6si4521001eja.107.2019.01.18.09.53.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:53:39 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) client-ip=46.22.139.233; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp16.blacknight.com (Postfix) with ESMTPS id F3E431C35B5 for ; Fri, 18 Jan 2019 17:53:38 +0000 (GMT) Received: (qmail 3977 invoked from network); 18 Jan 2019 17:53:38 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:53:38 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 11/22] mm, compaction: Use free lists to quickly locate a migration target Date: Fri, 18 Jan 2019 17:51:25 +0000 Message-Id: <20190118175136.31341-12-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Similar to the migration scanner, this patch uses the free lists to quickly locate a migration target. The search is different in that lower orders will be searched for a suitable high PFN if necessary but the search is still bound. This is justified on the grounds that the free scanner typically scans linearly much more than the migration scanner. If a free page is found, it is isolated and compaction continues if enough pages were isolated. For SYNC* scanning, the full pageblock is scanned for any remaining free pages so that is can be marked for skipping in the near future. 1-socket thpfioscale 5.0.0-rc1 5.0.0-rc1 isolmig-v3r15 findfree-v3r16 Amean fault-both-3 3024.41 ( 0.00%) 3200.68 ( -5.83%) Amean fault-both-5 4749.30 ( 0.00%) 4847.75 ( -2.07%) Amean fault-both-7 6454.95 ( 0.00%) 6658.92 ( -3.16%) Amean fault-both-12 10324.83 ( 0.00%) 11077.62 ( -7.29%) Amean fault-both-18 12896.82 ( 0.00%) 12403.97 ( 3.82%) Amean fault-both-24 13470.60 ( 0.00%) 15607.10 * -15.86%* Amean fault-both-30 17143.99 ( 0.00%) 18752.27 ( -9.38%) Amean fault-both-32 17743.91 ( 0.00%) 21207.54 * -19.52%* The impact on latency is variable but the search is optimistic and sensitive to the exact system state. Success rates are similar but the major impact is to the rate of scanning 5.0.0-rc1 5.0.0-rc1 isolmig-v3r15 findfree-v3r16 Compaction migrate scanned 25646769 29507205 Compaction free scanned 201558184 100359571 The free scan rates are reduced by 50%. The 2-socket reductions for the free scanner are more dramatic which is a likely reflection that the machine has more memory. [dan.carpenter@oracle.com: Fix static checker warning] [vbabka@suse.cz: Correct number of pages scanned for lower orders] Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 218 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 213 insertions(+), 5 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 7c4c9cce7907..19fea4a7b3f4 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1124,7 +1124,29 @@ static inline bool compact_scanners_met(struct compact_control *cc) <= (cc->migrate_pfn >> pageblock_order); } -/* Reorder the free list to reduce repeated future searches */ +/* + * Used when scanning for a suitable migration target which scans freelists + * in reverse. Reorders the list such as the unscanned pages are scanned + * first on the next iteration of the free scanner + */ +static void +move_freelist_head(struct list_head *freelist, struct page *freepage) +{ + LIST_HEAD(sublist); + + if (!list_is_last(freelist, &freepage->lru)) { + list_cut_before(&sublist, freelist, &freepage->lru); + if (!list_empty(&sublist)) + list_splice_tail(&sublist, freelist); + } +} + +/* + * Similar to move_freelist_head except used by the migration scanner + * when scanning forward. It's possible for these list operations to + * move against each other if they search the free list exactly in + * lockstep. + */ static void move_freelist_tail(struct list_head *freelist, struct page *freepage) { @@ -1137,6 +1159,186 @@ move_freelist_tail(struct list_head *freelist, struct page *freepage) } } +static void +fast_isolate_around(struct compact_control *cc, unsigned long pfn, unsigned long nr_isolated) +{ + unsigned long start_pfn, end_pfn; + struct page *page = pfn_to_page(pfn); + + /* Do not search around if there are enough pages already */ + if (cc->nr_freepages >= cc->nr_migratepages) + return; + + /* Minimise scanning during async compaction */ + if (cc->direct_compaction && cc->mode == MIGRATE_ASYNC) + return; + + /* Pageblock boundaries */ + start_pfn = pageblock_start_pfn(pfn); + end_pfn = min(start_pfn + pageblock_nr_pages, zone_end_pfn(cc->zone)); + + /* Scan before */ + if (start_pfn != pfn) { + isolate_freepages_block(cc, &start_pfn, pfn, &cc->freepages, false); + if (cc->nr_freepages >= cc->nr_migratepages) + return; + } + + /* Scan after */ + start_pfn = pfn + nr_isolated; + if (start_pfn != end_pfn) + isolate_freepages_block(cc, &start_pfn, end_pfn, &cc->freepages, false); + + /* Skip this pageblock in the future as it's full or nearly full */ + if (cc->nr_freepages < cc->nr_migratepages) + set_pageblock_skip(page); +} + +static unsigned long +fast_isolate_freepages(struct compact_control *cc) +{ + unsigned int limit = min(1U, freelist_scan_limit(cc) >> 1); + unsigned int nr_scanned = 0; + unsigned long low_pfn, min_pfn, high_pfn = 0, highest = 0; + unsigned long nr_isolated = 0; + unsigned long distance; + struct page *page = NULL; + bool scan_start = false; + int order; + + /* Full compaction passes in a negative order */ + if (cc->order <= 0) + return cc->free_pfn; + + /* + * If starting the scan, use a deeper search and use the highest + * PFN found if a suitable one is not found. + */ + if (cc->free_pfn == pageblock_start_pfn(zone_end_pfn(cc->zone) - 1)) { + limit = pageblock_nr_pages >> 1; + scan_start = true; + } + + /* + * Preferred point is in the top quarter of the scan space but take + * a pfn from the top half if the search is problematic. + */ + distance = (cc->free_pfn - cc->migrate_pfn); + low_pfn = pageblock_start_pfn(cc->free_pfn - (distance >> 2)); + min_pfn = pageblock_start_pfn(cc->free_pfn - (distance >> 1)); + + if (WARN_ON_ONCE(min_pfn > low_pfn)) + low_pfn = min_pfn; + + for (order = cc->order - 1; + order >= 0 && !page; + order--) { + struct free_area *area = &cc->zone->free_area[order]; + struct list_head *freelist; + struct page *freepage; + unsigned long flags; + unsigned int order_scanned = 0; + + if (!area->nr_free) + continue; + + spin_lock_irqsave(&cc->zone->lock, flags); + freelist = &area->free_list[MIGRATE_MOVABLE]; + list_for_each_entry_reverse(freepage, freelist, lru) { + unsigned long pfn; + + order_scanned++; + nr_scanned++; + pfn = page_to_pfn(freepage); + + if (pfn >= highest) + highest = pageblock_start_pfn(pfn); + + if (pfn >= low_pfn) { + cc->fast_search_fail = 0; + page = freepage; + break; + } + + if (pfn >= min_pfn && pfn > high_pfn) { + high_pfn = pfn; + + /* Shorten the scan if a candidate is found */ + limit >>= 1; + } + + if (order_scanned >= limit) + break; + } + + /* Use a minimum pfn if a preferred one was not found */ + if (!page && high_pfn) { + page = pfn_to_page(high_pfn); + + /* Update freepage for the list reorder below */ + freepage = page; + } + + /* Reorder to so a future search skips recent pages */ + move_freelist_head(freelist, freepage); + + /* Isolate the page if available */ + if (page) { + if (__isolate_free_page(page, order)) { + set_page_private(page, order); + nr_isolated = 1 << order; + cc->nr_freepages += nr_isolated; + list_add_tail(&page->lru, &cc->freepages); + count_compact_events(COMPACTISOLATED, nr_isolated); + } else { + /* If isolation fails, abort the search */ + order = -1; + page = NULL; + } + } + + spin_unlock_irqrestore(&cc->zone->lock, flags); + + /* + * Smaller scan on next order so the total scan ig related + * to freelist_scan_limit. + */ + if (order_scanned >= limit) + limit = min(1U, limit >> 1); + } + + if (!page) { + cc->fast_search_fail++; + if (scan_start) { + /* + * Use the highest PFN found above min. If one was + * not found, be pessemistic for direct compaction + * and use the min mark. + */ + if (highest) { + page = pfn_to_page(highest); + cc->free_pfn = highest; + } else { + if (cc->direct_compaction) { + page = pfn_to_page(min_pfn); + cc->free_pfn = min_pfn; + } + } + } + } + + if (highest && highest > cc->zone->compact_cached_free_pfn) + cc->zone->compact_cached_free_pfn = highest; + + cc->total_free_scanned += nr_scanned; + if (!page) + return cc->free_pfn; + + low_pfn = page_to_pfn(page); + fast_isolate_around(cc, low_pfn, nr_isolated); + return low_pfn; +} + /* * Based on information in the current compact_control, find blocks * suitable for isolating free pages from and then isolate them. @@ -1151,6 +1353,11 @@ static void isolate_freepages(struct compact_control *cc) unsigned long low_pfn; /* lowest pfn scanner is able to scan */ struct list_head *freelist = &cc->freepages; + /* Try a small search of the free lists for a candidate */ + isolate_start_pfn = fast_isolate_freepages(cc); + if (cc->nr_freepages) + goto splitmap; + /* * Initialise the free scanner. The starting point is where we last * successfully isolated from, zone-cached value, or the end of the @@ -1163,7 +1370,7 @@ static void isolate_freepages(struct compact_control *cc) * is using. */ isolate_start_pfn = cc->free_pfn; - block_start_pfn = pageblock_start_pfn(cc->free_pfn); + block_start_pfn = pageblock_start_pfn(isolate_start_pfn); block_end_pfn = min(block_start_pfn + pageblock_nr_pages, zone_end_pfn(zone)); low_pfn = pageblock_end_pfn(cc->migrate_pfn); @@ -1227,9 +1434,6 @@ static void isolate_freepages(struct compact_control *cc) } } - /* __isolate_free_page() does not map the pages */ - split_map_pages(freelist); - /* * Record where the free scanner will restart next time. Either we * broke from the loop and set isolate_start_pfn based on the last @@ -1237,6 +1441,10 @@ static void isolate_freepages(struct compact_control *cc) * and the loop terminated due to isolate_start_pfn < low_pfn */ cc->free_pfn = isolate_start_pfn; + +splitmap: + /* __isolate_free_page() does not map the pages */ + split_map_pages(freelist); } /* From patchwork Fri Jan 18 17:51:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771375 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8EDFA6C5 for ; Fri, 18 Jan 2019 17:53:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7F043285CC for ; Fri, 18 Jan 2019 17:53:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 725D1286B2; Fri, 18 Jan 2019 17:53:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BA9BB285CC for ; Fri, 18 Jan 2019 17:53:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A92CB8E001B; Fri, 18 Jan 2019 12:53:51 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A42558E0002; Fri, 18 Jan 2019 12:53:51 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 909878E001B; Fri, 18 Jan 2019 12:53:51 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 3A8ED8E0002 for ; Fri, 18 Jan 2019 12:53:51 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id i55so5264060ede.14 for ; Fri, 18 Jan 2019 09:53:51 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=q4XbqljV4csmVEUb4HH/NDfBhps7WKZmhbq0+dowWqs=; b=oxdWuSeJG29y24d6om8HbrKdDEv0Y+PRmnh05tEBG9tYIBp6XzaT8Fp3RwLBh650fg JfzdoJkEPA/5CmQLs8GYwqGr2V3fZLHBdFmAKy8AsxDj2QHc7hqkf5zyOgJ6q8uVtKmw yI5uUFmzFnYfgxPRmtBZ+PZmNE7i5zQ3cuH4Bz1k8wzVsG9JvhNKnzTh6iwNaZMSMHpU LncQt2WAmBdLUghdVWKmLGoZjhHH0bSr4RXgWnh1Y2tyThOHlD8xD6acC8D1E9zok8Wu uiztI/5mZuh53PM3JWEic64WKn3LpC2rU3RANbQEWBuIcnf3CLgaQuTn4BUVxNmGE049 FFOg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukeH1VdZhDoioQ/ELPqtnc+EJnbAdY/gkL3p+RBKqJUpH8gmviNJ drRVrz1+a6EeRzSs6IH3G/9QVzEU9FUNWbcnu1YnZRTbZvpvvIw12fQswMfI+2xqTDFUEnIqTw2 e+9EYtk72UOW3idJaTxNS3MLSB7e7HEvqs+F9ZTsqf7IjyyTS9DzafB2Pj4jEIATgTg== X-Received: by 2002:a17:906:a44:: with SMTP id x4-v6mr15294936ejf.177.1547834030691; Fri, 18 Jan 2019 09:53:50 -0800 (PST) X-Google-Smtp-Source: ALg8bN4KTDx7d0NfnJ1NFFP8jeU+ZWBxIVxlivjWau5hnzVKfDzBRl6+g9saF0oPMoWvq+q8LX/d X-Received: by 2002:a17:906:a44:: with SMTP id x4-v6mr15294878ejf.177.1547834029482; Fri, 18 Jan 2019 09:53:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547834029; cv=none; d=google.com; s=arc-20160816; b=cbrMOdb4dhFemysUtIR5Y6BTUC1JR4Z/zWd1DggvI/4s9/Bm4nR61skgB0hYyHfjBv uBYzbwXktsebF0WzPnV7sHLcahrudnj5V9/bmDvdTGlP2nU1sYGxS7TXs+Xw/WlLVZQE NISo7wLFML4yrTxJOQbJVH4GzHXxyZGreUdmvtaFM9Du4ERinFlQnCvJKoxe6b5BYHhP 1DFUCDawuvEaaBKBy/WmK2Hr2VLH4dxNI8GNqmcs/6Nz7zE1S9305RDVmsFdKXScUjA1 LxQrb1bQmFbgg2hzPKeYu/l6hUwFLQ/Yi+gXvyDIN2xZKRSSvz1Nl2CwOXuCj3t2Re4E zGqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=q4XbqljV4csmVEUb4HH/NDfBhps7WKZmhbq0+dowWqs=; b=YXtJcvaOl5+fGYYlcVQtAMx9ANHdhvMrYG5ekF2eKcejaF0nvHDlCM+RmMXLQULBAU e2IAYnGENylO/8iAHXjsvBJ8Pt5cMbamKc1bgtWNpibb0T1BFolnwgCsHCQZkf+lEmkE 13KB/nVoAsglPGkqS9UzXGAezKUp7CqF2Ww5B0eTf3jOQ5fsYh1lbpCHvcqNyw0OHCLQ rxserqIVuULjBI2Lre2+6vy05aaRe8EjmdZo4emVTc6kLtjQOgJwTnxNXhV1IYy6GBZz hpu0Q66lypcOzvhAdvIxGT8ICFKT39i+FseMsZ+lQC8K+xXp5dIQbPwBgiU36E3xEynU viOg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp13.blacknight.com (outbound-smtp13.blacknight.com. [46.22.139.230]) by mx.google.com with ESMTPS id k13si3535974edl.377.2019.01.18.09.53.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:53:49 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) client-ip=46.22.139.230; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (unknown [81.17.254.16]) by outbound-smtp13.blacknight.com (Postfix) with ESMTPS id 2B06B1C35C8 for ; Fri, 18 Jan 2019 17:53:49 +0000 (GMT) Received: (qmail 4431 invoked from network); 18 Jan 2019 17:53:49 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:53:49 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 12/22] mm, compaction: Avoid rescanning the same pageblock multiple times Date: Fri, 18 Jan 2019 17:51:26 +0000 Message-Id: <20190118175136.31341-13-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Pageblocks are marked for skip when no pages are isolated after a scan. However, it's possible to hit corner cases where the migration scanner gets stuck near the boundary between the source and target scanner. Due to pages being migrated in blocks of COMPACT_CLUSTER_MAX, pages that are migrated can be reallocated before the pageblock is complete. The pageblock is not necessarily skipped so it can be rescanned multiple times. Similarly, a pageblock with some dirty/writeback pages may fail to migrate and be rescanned until writeback completes which is wasteful. This patch tracks if a pageblock is being rescanned. If so, then the entire pageblock will be migrated as one operation. This narrows the race window during which pages can be reallocated during migration. Secondly, if there are pages that cannot be isolated then the pageblock will still be fully scanned and marked for skipping. On the second rescan, the pageblock skip is set and the migration scanner makes progress. 5.0.0-rc1 5.0.0-rc1 findfree-v3r16 norescan-v3r16 Amean fault-both-1 0.00 ( 0.00%) 0.00 * 0.00%* Amean fault-both-3 3200.68 ( 0.00%) 3002.07 ( 6.21%) Amean fault-both-5 4847.75 ( 0.00%) 4684.47 ( 3.37%) Amean fault-both-7 6658.92 ( 0.00%) 6815.54 ( -2.35%) Amean fault-both-12 11077.62 ( 0.00%) 10864.02 ( 1.93%) Amean fault-both-18 12403.97 ( 0.00%) 12247.52 ( 1.26%) Amean fault-both-24 15607.10 ( 0.00%) 15683.99 ( -0.49%) Amean fault-both-30 18752.27 ( 0.00%) 18620.02 ( 0.71%) Amean fault-both-32 21207.54 ( 0.00%) 19250.28 * 9.23%* 5.0.0-rc1 5.0.0-rc1 findfree-v3r16 norescan-v3r16 Percentage huge-3 96.86 ( 0.00%) 95.00 ( -1.91%) Percentage huge-5 93.72 ( 0.00%) 94.22 ( 0.53%) Percentage huge-7 94.31 ( 0.00%) 92.35 ( -2.08%) Percentage huge-12 92.66 ( 0.00%) 91.90 ( -0.82%) Percentage huge-18 91.51 ( 0.00%) 89.58 ( -2.11%) Percentage huge-24 90.50 ( 0.00%) 90.03 ( -0.52%) Percentage huge-30 91.57 ( 0.00%) 89.14 ( -2.65%) Percentage huge-32 91.00 ( 0.00%) 90.58 ( -0.46%) Negligible difference but this was likely a case when the specific corner case was not hit. A previous run of the same patch based on an earlier iteration of the series showed large differences where migration rates could be halved when the corner case was hit. The specific corner case where migration scan rates go through the roof was due to a dirty/writeback pageblock located at the boundary of the migration/free scanner did not happen in this case. When it does happen, the scan rates multipled by massive margins. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 32 ++++++++++++++++++++++++++------ mm/internal.h | 1 + 2 files changed, 27 insertions(+), 6 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 19fea4a7b3f4..a31fea7b3f96 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -949,8 +949,11 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, cc->nr_migratepages++; nr_isolated++; - /* Avoid isolating too much */ - if (cc->nr_migratepages == COMPACT_CLUSTER_MAX) { + /* + * Avoid isolating too much unless this block is being + * rescanned (e.g. dirty/writeback pages, parallel allocation). + */ + if (cc->nr_migratepages == COMPACT_CLUSTER_MAX && !cc->rescan) { ++low_pfn; break; } @@ -997,11 +1000,14 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, spin_unlock_irqrestore(zone_lru_lock(zone), flags); /* - * Updated the cached scanner pfn if the pageblock was scanned - * without isolating a page. The pageblock may not be marked - * skipped already if there were no LRU pages in the block. + * Updated the cached scanner pfn once the pageblock has been scanned + * Pages will either be migrated in which case there is no point + * scanning in the near future or migration failed in which case the + * failure reason may persist. The block is marked for skipping if + * there were no pages isolated in the block or if the block is + * rescanned twice in a row. */ - if (low_pfn == end_pfn && !nr_isolated) { + if (low_pfn == end_pfn && (!nr_isolated || cc->rescan)) { if (valid_page && !skip_updated) set_pageblock_skip(valid_page); update_cached_migrate(cc, low_pfn); @@ -2033,6 +2039,20 @@ static enum compact_result compact_zone(struct compact_control *cc) int err; unsigned long start_pfn = cc->migrate_pfn; + /* + * Avoid multiple rescans which can happen if a page cannot be + * isolated (dirty/writeback in async mode) or if the migrated + * pages are being allocated before the pageblock is cleared. + * The first rescan will capture the entire pageblock for + * migration. If it fails, it'll be marked skip and scanning + * will proceed as normal. + */ + cc->rescan = false; + if (pageblock_start_pfn(last_migrated_pfn) == + pageblock_start_pfn(start_pfn)) { + cc->rescan = true; + } + switch (isolate_migratepages(cc->zone, cc)) { case ISOLATE_ABORT: ret = COMPACT_CONTENDED; diff --git a/mm/internal.h b/mm/internal.h index 983cb975545f..d5b999e5eb5f 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -205,6 +205,7 @@ struct compact_control { bool direct_compaction; /* False from kcompactd or /proc/... */ bool whole_zone; /* Whole zone should/has been scanned */ bool contended; /* Signal lock or sched contention */ + bool rescan; /* Rescanning the same pageblock */ }; unsigned long From patchwork Fri Jan 18 17:51:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771377 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 232746C5 for ; Fri, 18 Jan 2019 17:54:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0FA4C285CC for ; Fri, 18 Jan 2019 17:54:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 02655286B2; Fri, 18 Jan 2019 17:54:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2CCBD285CC for ; Fri, 18 Jan 2019 17:54:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 22F4E8E001C; Fri, 18 Jan 2019 12:54:02 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 205B38E0002; Fri, 18 Jan 2019 12:54:02 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0CB928E001C; Fri, 18 Jan 2019 12:54:02 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id A798C8E0002 for ; Fri, 18 Jan 2019 12:54:01 -0500 (EST) Received: by mail-ed1-f69.google.com with SMTP id o21so5274435edq.4 for ; Fri, 18 Jan 2019 09:54:01 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=rUemIHUE6CuZBG2nXkHvzVOmBOge0v2A1/ryev+BUHY=; b=Gp+ha6qBbYBLscec7zFMddClag2Um8yNcGyKvCLJ1f3CIV8cZtQyxVq7rxemR+Ypgt FNiF7mnkj0+9VpQCRmZlkEdYcWRAqcIjIovU4vZI9bVn+MB9KQLUi1PBKgmXCZqCjJtg MVmGrk78fPU0lWLkNyMGeA82ThZnQrXWmI86OsoeEG7LdKzno3oVcNgOmx5stMWCJ4Dw yk0QZYMLBgDPn0wI1q8dBEseZxO+Yewtx5bYj36i7/5S1r+wodmlicTZYvYY60sNh6i/ 1L09X/zrawx4lMjsHE2TJPlx/ANHsWNVfvVvbsccHuBSUE60yGlv/rCfhkJEiGhS4YjP 9rKQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.193 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukd144YUyljLKIEz0QvB1yrE2va0gn4gy74u6+MlOz+KcaWCMz7m GMek3EDq22IZgWTRO9d31ytMg87lPikyeKdb6apLOWX7vmCLPjSwoh59m8g57ADc/BL+8qTXFOq fGwMqQz0ziUov9guqGbtEiqMcWkU4Dd8YVyXXJmrLCcjNtjW4rr3skKS7iZCY9LpzuQ== X-Received: by 2002:aa7:d88c:: with SMTP id u12mr16015376edq.237.1547834041130; Fri, 18 Jan 2019 09:54:01 -0800 (PST) X-Google-Smtp-Source: ALg8bN4cCMnIOZf86c1BYenad0agCMji2KIej3t45962as27LtcD4NlRNGUD7U5TvNU6xIpW+N+s X-Received: by 2002:aa7:d88c:: with SMTP id u12mr16015296edq.237.1547834039604; Fri, 18 Jan 2019 09:53:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547834039; cv=none; d=google.com; s=arc-20160816; b=rpc1yagzlxkhpq6R6Mb4gbZfjkDubxR2Fp9RTjRy83LbblPVk4fOsqDuADmZQSDj9W F+DAbBlz6hhfugKsFReUO0AbpYwWhO/SqyR5PJLTKkzMw5SgCR3M/Gb62nfztYFsGNSJ +wwrhYdW293i1mEO2Ffrdl6rm2pVmhqS4pMAZR98WZJnRGyJVIJGvYkXKc0U+5qGJWsp vO8/bktvleBzKuEB8lh9S5loUaVYPA1JtwqXmAsm+aulqDbixBTTaYkV8k9UEc9hG751 u9PwsV8f/EVGtUkf5nrZAYnPgZfT81ENCIot6VFmEirCp2HtzoHPRCUiorPvqvV911lg r6yg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=rUemIHUE6CuZBG2nXkHvzVOmBOge0v2A1/ryev+BUHY=; b=nhuWNrKYkwA+5cD2Ac0nfmeQ9EmtEthpjaHsZda/Uj5IIEYayxC2OuFfB3l7pmtHL1 IdKjtlztrtS3454aZzpH4z2FNWGeDDgGIR3bFBLCl4S8lyi57SvA1n3rx77IN60gg2Az ikQYg6KOqndiCj25MkwfvFiIduoKwVDbyZVKH3INcuZpOi/J56EAoRJ+hfpeXx7RmN70 lpQvGhSffX9XV25HkbMyx0mChn5gPrxZH5NG1/wQa1qjOWWjvIgcP+2isl3QP3xyxtqx YD05UmQYlngy3N9VMjEdIJ2aY7d79cu3qwl2NdEDURUJrRCrUh584kY4elxufPxaGRO9 DFkQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.193 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp25.blacknight.com (outbound-smtp25.blacknight.com. [81.17.249.193]) by mx.google.com with ESMTPS id 25si2894796edv.63.2019.01.18.09.53.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:53:59 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.193 as permitted sender) client-ip=81.17.249.193; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.193 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp25.blacknight.com (Postfix) with ESMTPS id 49873B87ED for ; Fri, 18 Jan 2019 17:53:59 +0000 (GMT) Received: (qmail 4915 invoked from network); 18 Jan 2019 17:53:59 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:53:59 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 13/22] mm, compaction: Finish pageblock scanning on contention Date: Fri, 18 Jan 2019 17:51:27 +0000 Message-Id: <20190118175136.31341-14-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Async migration aborts on spinlock contention but contention can be high when there are multiple compaction attempts and kswapd is active. The consequence is that the migration scanners move forward uselessly while still contending on locks for longer while leaving suitable migration sources behind. This patch will acquire the lock but track when contention occurs. When it does, the current pageblock will finish as compaction may succeed for that block and then abort. This will have a variable impact on latency as in some cases useless scanning is avoided (reduces latency) but a lock will be contended (increase latency) or a single contended pageblock is scanned that would otherwise have been skipped (increase latency). 5.0.0-rc1 5.0.0-rc1 norescan-v3r16 finishcontend-v3r16 Amean fault-both-1 0.00 ( 0.00%) 0.00 * 0.00%* Amean fault-both-3 3002.07 ( 0.00%) 3153.17 ( -5.03%) Amean fault-both-5 4684.47 ( 0.00%) 4280.52 ( 8.62%) Amean fault-both-7 6815.54 ( 0.00%) 5811.50 * 14.73%* Amean fault-both-12 10864.02 ( 0.00%) 9276.85 ( 14.61%) Amean fault-both-18 12247.52 ( 0.00%) 11032.67 ( 9.92%) Amean fault-both-24 15683.99 ( 0.00%) 14285.70 ( 8.92%) Amean fault-both-30 18620.02 ( 0.00%) 16293.76 * 12.49%* Amean fault-both-32 19250.28 ( 0.00%) 16721.02 * 13.14%* 5.0.0-rc1 5.0.0-rc1 norescan-v3r16 finishcontend-v3r16 Percentage huge-1 0.00 ( 0.00%) 0.00 ( 0.00%) Percentage huge-3 95.00 ( 0.00%) 96.82 ( 1.92%) Percentage huge-5 94.22 ( 0.00%) 95.40 ( 1.26%) Percentage huge-7 92.35 ( 0.00%) 95.92 ( 3.86%) Percentage huge-12 91.90 ( 0.00%) 96.73 ( 5.25%) Percentage huge-18 89.58 ( 0.00%) 96.77 ( 8.03%) Percentage huge-24 90.03 ( 0.00%) 96.05 ( 6.69%) Percentage huge-30 89.14 ( 0.00%) 96.81 ( 8.60%) Percentage huge-32 90.58 ( 0.00%) 97.41 ( 7.54%) There is a variable impact that is mostly good on latency while allocation success rates are slightly higher. System CPU usage is reduced by about 10% but scan rate impact is mixed Compaction migrate scanned 27997659.00 20148867 Compaction free scanned 120782791.00 118324914 Migration scan rates are reduced 28% which is expected as a pageblock is used by the async scanner instead of skipped. The impact on the free scanner is known to be variable. Overall the primary justification for this patch is that completing scanning of a pageblock is very important for later patches. [yuehaibing@huawei.com: Fix unused variable warning] Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 90 ++++++++++++++++++++++----------------------------------- 1 file changed, 34 insertions(+), 56 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index a31fea7b3f96..b261c0bfac24 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -382,24 +382,25 @@ static bool test_and_set_skip(struct compact_control *cc, struct page *page, /* * Compaction requires the taking of some coarse locks that are potentially - * very heavily contended. For async compaction, back out if the lock cannot - * be taken immediately. For sync compaction, spin on the lock if needed. + * very heavily contended. For async compaction, trylock and record if the + * lock is contended. The lock will still be acquired but compaction will + * abort when the current block is finished regardless of success rate. + * Sync compaction acquires the lock. * - * Returns true if the lock is held - * Returns false if the lock is not held and compaction should abort + * Always returns true which makes it easier to track lock state in callers. */ -static bool compact_trylock_irqsave(spinlock_t *lock, unsigned long *flags, +static bool compact_lock_irqsave(spinlock_t *lock, unsigned long *flags, struct compact_control *cc) { - if (cc->mode == MIGRATE_ASYNC) { - if (!spin_trylock_irqsave(lock, *flags)) { - cc->contended = true; - return false; - } - } else { - spin_lock_irqsave(lock, *flags); + /* Track if the lock is contended in async mode */ + if (cc->mode == MIGRATE_ASYNC && !cc->contended) { + if (spin_trylock_irqsave(lock, *flags)) + return true; + + cc->contended = true; } + spin_lock_irqsave(lock, *flags); return true; } @@ -432,10 +433,8 @@ static bool compact_unlock_should_abort(spinlock_t *lock, } if (need_resched()) { - if (cc->mode == MIGRATE_ASYNC) { + if (cc->mode == MIGRATE_ASYNC) cc->contended = true; - return true; - } cond_resched(); } @@ -455,10 +454,8 @@ static inline bool compact_should_abort(struct compact_control *cc) { /* async compaction aborts if contended */ if (need_resched()) { - if (cc->mode == MIGRATE_ASYNC) { + if (cc->mode == MIGRATE_ASYNC) cc->contended = true; - return true; - } cond_resched(); } @@ -535,18 +532,8 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, * recheck as well. */ if (!locked) { - /* - * The zone lock must be held to isolate freepages. - * Unfortunately this is a very coarse lock and can be - * heavily contended if there are parallel allocations - * or parallel compactions. For async compaction do not - * spin on the lock and we acquire the lock as late as - * possible. - */ - locked = compact_trylock_irqsave(&cc->zone->lock, + locked = compact_lock_irqsave(&cc->zone->lock, &flags, cc); - if (!locked) - break; /* Recheck this is a buddy page under lock */ if (!PageBuddy(page)) @@ -900,15 +887,9 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, /* If we already hold the lock, we can skip some rechecking */ if (!locked) { - locked = compact_trylock_irqsave(zone_lru_lock(zone), + locked = compact_lock_irqsave(zone_lru_lock(zone), &flags, cc); - /* Allow future scanning if the lock is contended */ - if (!locked) { - clear_pageblock_skip(page); - break; - } - /* Try get exclusive access under lock */ if (!skip_updated) { skip_updated = true; @@ -951,9 +932,12 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, /* * Avoid isolating too much unless this block is being - * rescanned (e.g. dirty/writeback pages, parallel allocation). + * rescanned (e.g. dirty/writeback pages, parallel allocation) + * or a lock is contended. For contention, isolate quickly to + * potentially remove one source of contention. */ - if (cc->nr_migratepages == COMPACT_CLUSTER_MAX && !cc->rescan) { + if (cc->nr_migratepages == COMPACT_CLUSTER_MAX && + !cc->rescan && !cc->contended) { ++low_pfn; break; } @@ -1416,12 +1400,8 @@ static void isolate_freepages(struct compact_control *cc) isolate_freepages_block(cc, &isolate_start_pfn, block_end_pfn, freelist, false); - /* - * If we isolated enough freepages, or aborted due to lock - * contention, terminate. - */ - if ((cc->nr_freepages >= cc->nr_migratepages) - || cc->contended) { + /* Are enough freepages isolated? */ + if (cc->nr_freepages >= cc->nr_migratepages) { if (isolate_start_pfn >= block_end_pfn) { /* * Restart at previous pageblock if more @@ -1463,13 +1443,8 @@ static struct page *compaction_alloc(struct page *migratepage, struct compact_control *cc = (struct compact_control *)data; struct page *freepage; - /* - * Isolate free pages if necessary, and if we are not aborting due to - * contention. - */ if (list_empty(&cc->freepages)) { - if (!cc->contended) - isolate_freepages(cc); + isolate_freepages(cc); if (list_empty(&cc->freepages)) return NULL; @@ -1731,7 +1706,7 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone, low_pfn = isolate_migratepages_block(cc, low_pfn, block_end_pfn, isolate_mode); - if (!low_pfn || cc->contended) + if (!low_pfn) return ISOLATE_ABORT; /* @@ -1761,9 +1736,7 @@ static enum compact_result __compact_finished(struct compact_control *cc) { unsigned int order; const int migratetype = cc->migratetype; - - if (cc->contended || fatal_signal_pending(current)) - return COMPACT_CONTENDED; + int ret; /* Compaction run completes if the migrate and free scanner meet */ if (compact_scanners_met(cc)) { @@ -1798,6 +1771,7 @@ static enum compact_result __compact_finished(struct compact_control *cc) return COMPACT_CONTINUE; /* Direct compactor: Is a suitable page free? */ + ret = COMPACT_NO_SUITABLE_PAGE; for (order = cc->order; order < MAX_ORDER; order++) { struct free_area *area = &cc->zone->free_area[order]; bool can_steal; @@ -1837,11 +1811,15 @@ static enum compact_result __compact_finished(struct compact_control *cc) return COMPACT_SUCCESS; } - return COMPACT_CONTINUE; + ret = COMPACT_CONTINUE; + break; } } - return COMPACT_NO_SUITABLE_PAGE; + if (cc->contended || fatal_signal_pending(current)) + ret = COMPACT_CONTENDED; + + return ret; } static enum compact_result compact_finished(struct compact_control *cc) From patchwork Fri Jan 18 17:51:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771379 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 922E313B4 for ; Fri, 18 Jan 2019 17:54:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 80357285CC for ; Fri, 18 Jan 2019 17:54:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 73E23286B2; Fri, 18 Jan 2019 17:54:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E2293285CC for ; Fri, 18 Jan 2019 17:54:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DEDC38E001D; Fri, 18 Jan 2019 12:54:11 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D9DE58E0002; Fri, 18 Jan 2019 12:54:11 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C646C8E001D; Fri, 18 Jan 2019 12:54:11 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id 69E558E0002 for ; Fri, 18 Jan 2019 12:54:11 -0500 (EST) Received: by mail-ed1-f71.google.com with SMTP id c18so5125896edt.23 for ; Fri, 18 Jan 2019 09:54:11 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=ob00ivw8MGpNGXusnKzOvKzjGg/MG1s7CTxxOGMVEys=; b=a05KyptPkFGUQ5mzRCv2OdB6eiLfUksMJePfVSe2hLSmz+WcTYGC58qXyBUft2Yi6j ZnpdTTiFNZPDB7MrPMkCiAAZV4aoyY4+n6a1ZiFDIWaTHCJK77E0bbmfEUUR2N1WNPcK Zr9n2kE385tAIRygBTWE167JGlO3xT0FuViz7dPb+dRtLmDqG007JAzj33yVV3UxFJQi o/+OXWoUKMNcppVDBFTPWP2Kq8NeTAt/sHmCRuY8IdGll1BwMGYjtWTzqEMlWhkoO68h +aljcRCc57Ku9vApdB1JvcbKfp3jw9zDKpqH1iE3VdMtsX+ahvqa0u4Esc2juKmbXEzE BSfw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukfEdi+nQHOcfjemBfAyNeQD3ZbzaHRwn0T4IynuPZHpmMFQYaIl E7KVAzT886USUQrTkRE8lT6r3ayPB5tf3aYGYVIdaLUiAj+b7/EFVP//LS5p2wl+Tn9z7GVh/c6 E0AXXqsCtTrLgnHcNE/7svjbxukpNW9lXbGRbXDbELikBVf0kVvSgZJE+J5i4zH8UEQ== X-Received: by 2002:aa7:d597:: with SMTP id r23mr17141630edq.51.1547834050811; Fri, 18 Jan 2019 09:54:10 -0800 (PST) X-Google-Smtp-Source: ALg8bN5cDvYNBim+QEMM2D0HKMTALMj9rZeHONbH9VoeEE/NJdWxZFBbWxC1J1kbEYkszzO2fhGl X-Received: by 2002:aa7:d597:: with SMTP id r23mr17141572edq.51.1547834049764; Fri, 18 Jan 2019 09:54:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547834049; cv=none; d=google.com; s=arc-20160816; b=yYeWbDqHORJBVYC1KhKytfq378OSJQhjRSNS6RxbmkCQuWt5E69uvEMH9ditseSUpX 4wOgP1jAsaui953c5nBjBsy02675UdFeIwewmHoCDk4+DcONMAtT7JE/F3VvZyjgeSU8 yvbAP6TF+NWOxXAG/zEdIl+rYTsj/nhhr1wbbKb/c9PbxH/kBZvk56n3btCSMflEiCmb oGLmYTJuRWpqir4XI2oduHByS+pchaheTeHLq+dnneZaQ8mfJJcAncqEqbYcfxwT8NGg /CdQWiQrAYyhp8gRgNYP9tzFXUI5/Fiyn+bsEN8M7U/dJIrJy+OP3/p27Q+n+AHouW9Y MAJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=ob00ivw8MGpNGXusnKzOvKzjGg/MG1s7CTxxOGMVEys=; b=YCX9KU9fT2PzoXpsUi4HC3V341zxOm/uG49umgxz5jRmI6xl0WdU3rCWZVrjbi9ITo Vz4yjIlVW7BYU6xFUCFWde7gYQxTHJsTerxb5HIjetrXsZp8FSNi8JE7efvnjQdPC+vR L3X6gRDLSEoWia+YtTEOTUT1y+1lfBFN7vkh4HIGB5ExIeu+HQ7hl5BytDjWCt2yFVga C0Sd+s/sTHTizerSHP02OZb57L4fxvB0UGOa2QvqEPwthseWbCvf8lCa0R68xQOM63yq bPfw5le394JrlktPwgTd7CYjen8Fu/tsASHOzc9lAEu5eCftV9F3EzyD/bfckWONyobc srXA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp02.blacknight.com (outbound-smtp02.blacknight.com. [81.17.249.8]) by mx.google.com with ESMTPS id b12si1459128edb.125.2019.01.18.09.54.09 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:54:09 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) client-ip=81.17.249.8; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp02.blacknight.com (Postfix) with ESMTPS id 744DE98C38 for ; Fri, 18 Jan 2019 17:54:09 +0000 (UTC) Received: (qmail 5490 invoked from network); 18 Jan 2019 17:54:09 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:54:09 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 14/22] mm, compaction: Check early for huge pages encountered by the migration scanner Date: Fri, 18 Jan 2019 17:51:28 +0000 Message-Id: <20190118175136.31341-15-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When scanning for sources or targets, PageCompound is checked for huge pages as they can be skipped quickly but it happens relatively late after a lot of setup and checking. This patch short-cuts the check to make it earlier. It might still change when the lock is acquired but this has less overhead overall. The free scanner advances but the migration scanner does not. Typically the free scanner encounters more movable blocks that change state over the lifetime of the system and also tends to scan more aggressively as it's actively filling its portion of the physical address space with data. This could change in the future but for the moment, this worked better in practice and incurred fewer scan restarts. The impact on latency and allocation success rates is marginal but the free scan rates are reduced by 15% and system CPU usage is reduced by 3.3%. The 2-socket results are not materially different. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index b261c0bfac24..14bb66d48392 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1061,6 +1061,9 @@ static bool suitable_migration_source(struct compact_control *cc, { int block_mt; + if (pageblock_skip_persistent(page)) + return false; + if ((cc->mode != MIGRATE_ASYNC) || !cc->direct_compaction) return true; @@ -1695,12 +1698,17 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone, continue; /* - * For async compaction, also only scan in MOVABLE blocks. - * Async compaction is optimistic to see if the minimum amount - * of work satisfies the allocation. + * For async compaction, also only scan in MOVABLE blocks + * without huge pages. Async compaction is optimistic to see + * if the minimum amount of work satisfies the allocation. + * The cached PFN is updated as it's possible that all + * remaining blocks between source and target are unsuitable + * and the compaction scanners fail to meet. */ - if (!suitable_migration_source(cc, page)) + if (!suitable_migration_source(cc, page)) { + update_cached_migrate(cc, block_end_pfn); continue; + } /* Perform the isolation */ low_pfn = isolate_migratepages_block(cc, low_pfn, From patchwork Fri Jan 18 17:51:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771381 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B9A5B6C5 for ; Fri, 18 Jan 2019 17:54:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A9B402FF7F for ; Fri, 18 Jan 2019 17:54:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9E1342FFB8; Fri, 18 Jan 2019 17:54:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 276092FF7F for ; Fri, 18 Jan 2019 17:54:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1E0BC8E001E; Fri, 18 Jan 2019 12:54:22 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 168858E0002; Fri, 18 Jan 2019 12:54:22 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0315F8E001E; Fri, 18 Jan 2019 12:54:21 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id 984568E0002 for ; Fri, 18 Jan 2019 12:54:21 -0500 (EST) Received: by mail-ed1-f71.google.com with SMTP id o21so5274782edq.4 for ; Fri, 18 Jan 2019 09:54:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=1umYCuCqPEt8hgwHsF5GWN8/VBehoTF7sexCPQhQrac=; b=oKA760+P/+WfvXHe081SLHiLhvb9rTEW2x7CQZy/H00A3yWy33bNp2Hrp9nX9MF+U4 rn5pfqxX7XL0Fo4ziNprbhPh9YxQcB28mqb+ilBblL8yF+42db0KrjndjM+kiQj0Yxyz EKtJgp+/yINSIWmjh/Ndy7Pr6+VzAeN+GANWpNNbmcvocIhNXA70A7cEsK1gwDstr0z8 yw7WKgGlf47ShLX4WcPOPFDl2jrI2cFOQiHM5Jm1AqCnZgOJeXH9VNbCRLJC8l8wGbyV pRdD2LI6xPkdtQNC7+fdaWfIIfeaWM1T4WvZdr90nNWJYRen+riRkhKE8mhU945MzwlN B78Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukdjPd8Hp/N7Ek+2ubrd4qhLJWYQO3B+CiWFm8MUbI0u1jNAQsfj ZsEzMP4lcYb8HgE2PJEut5LrfZju5+eKAtV5XFC0WWSqF4J0RoRVtaoGnIBz4XFkRBICEpKYBgl IAbNdaN6f7SvygkbIPf7UypEXiE+2tweNnWGW08nK38vhq8qCI8Zuym+gZIHbKrkW3A== X-Received: by 2002:a50:c089:: with SMTP id k9mr13045446edf.89.1547834061083; Fri, 18 Jan 2019 09:54:21 -0800 (PST) X-Google-Smtp-Source: ALg8bN6Se6YROrqAEokcSGwmeQF0kAuPDm05P007BQf0MnhXAxxrPgbU45TAUkgWZD6rI1ymcHzp X-Received: by 2002:a50:c089:: with SMTP id k9mr13045380edf.89.1547834060003; Fri, 18 Jan 2019 09:54:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547834060; cv=none; d=google.com; s=arc-20160816; b=lYdorYpvO/M4iXQ3tVMJ9pvNm1hpFGD7Lg7FibGCIPyiCCEb9zOYUFL9Rd7F4O+hi2 FY07imjpjI7qcuxJbCpZaMbwFWsrxuuC7BM9opRbiGMfjfqsqmPmC1kpw3B3aIvUNPY/ Me/REVTgL0ZZjK0BlqWbp3qZKY2cmBm8VETDkfyj9+Uzx88Fp5pvn7Dubi4lmGkZ6VLd FbfMwDvOk+oc4Ww8Rs/m5DG1iHoAlyiipI044+xwU91GXaiEqwzyRnZFVfmvtn/k/ckc fmdPhKBOhBkicBQ1JHPEWaKcSbSM46NAMHKdBWxi+sfw6xiAGhrmsySYlFkbSA3/nWcL weAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=1umYCuCqPEt8hgwHsF5GWN8/VBehoTF7sexCPQhQrac=; b=xtXV2zC7eHwy/5MgmsMpNZY/TtvOfm3cjTvSqYKzaR2MWCucl0v9yfQSoYFjTGlsoI i0tqUkIWlIfijHQkQ8RU7TlvbRbqqJGljCEwB4vXN5W2oZrBWLEGxqP7FEwwbyM4qbkk udyL9R4+zNk5moWMFtJnI7Dvy5M8wjqH3nv72uGgyO9TnjK7tou3jLPNLoP5uML0S9TU RDrLdoaOd/cV6A5XcZTeHNOiNQqVip3V165RfNEik2cdngPV1P+oOxEgjckU2bErp7bu B+mylbBQJpoaMuY3TRls/xcoULsJjP4Kfcyi9TcKp/2H1CS2nhRlsEpTiekTQibH4pce /iKw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp11.blacknight.com (outbound-smtp11.blacknight.com. [46.22.139.106]) by mx.google.com with ESMTPS id b11si2182035edj.393.2019.01.18.09.54.19 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:54:19 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) client-ip=46.22.139.106; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp11.blacknight.com (Postfix) with ESMTPS id A150C1C3579 for ; Fri, 18 Jan 2019 17:54:19 +0000 (GMT) Received: (qmail 5918 invoked from network); 18 Jan 2019 17:54:19 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:54:19 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 15/22] mm, compaction: Keep cached migration PFNs synced for unusable pageblocks Date: Fri, 18 Jan 2019 17:51:29 +0000 Message-Id: <20190118175136.31341-16-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Migrate has separate cached PFNs for ASYNC and SYNC* migration on the basis that some migrations will fail in ASYNC mode. However, if the cached PFNs match at the start of scanning and pageblocks are skipped due to having no isolation candidates, then the sync state does not matter. This patch keeps matching cached PFNs in sync until a pageblock with isolation candidates is found. The actual benefit is marginal given that the sync scanner following the async scanner will often skip a number of pageblocks but it's useless work. Any benefit depends heavily on whether the scanners restarted recently. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/mm/compaction.c b/mm/compaction.c index 14bb66d48392..829540f6f3da 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1969,6 +1969,7 @@ static enum compact_result compact_zone(struct compact_control *cc) unsigned long end_pfn = zone_end_pfn(cc->zone); unsigned long last_migrated_pfn; const bool sync = cc->mode != MIGRATE_ASYNC; + bool update_cached; cc->migratetype = gfpflags_to_migratetype(cc->gfp_mask); ret = compaction_suitable(cc->zone, cc->order, cc->alloc_flags, @@ -2016,6 +2017,17 @@ static enum compact_result compact_zone(struct compact_control *cc) last_migrated_pfn = 0; + /* + * Migrate has separate cached PFNs for ASYNC and SYNC* migration on + * the basis that some migrations will fail in ASYNC mode. However, + * if the cached PFNs match and pageblocks are skipped due to having + * no isolation candidates, then the sync state does not matter. + * Until a pageblock with isolation candidates is found, keep the + * cached PFNs in sync to avoid revisiting the same blocks. + */ + update_cached = !sync && + cc->zone->compact_cached_migrate_pfn[0] == cc->zone->compact_cached_migrate_pfn[1]; + trace_mm_compaction_begin(start_pfn, cc->migrate_pfn, cc->free_pfn, end_pfn, sync); @@ -2047,6 +2059,11 @@ static enum compact_result compact_zone(struct compact_control *cc) last_migrated_pfn = 0; goto out; case ISOLATE_NONE: + if (update_cached) { + cc->zone->compact_cached_migrate_pfn[1] = + cc->zone->compact_cached_migrate_pfn[0]; + } + /* * We haven't isolated and migrated anything, but * there might still be unflushed migrations from @@ -2054,6 +2071,7 @@ static enum compact_result compact_zone(struct compact_control *cc) */ goto check_drain; case ISOLATE_SUCCESS: + update_cached = false; last_migrated_pfn = start_pfn; ; } From patchwork Fri Jan 18 17:51:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771383 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C8E9B6C5 for ; Fri, 18 Jan 2019 17:54:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B469F2FF7F for ; Fri, 18 Jan 2019 17:54:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A6F722FFB8; Fri, 18 Jan 2019 17:54:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2EA182FF7F for ; Fri, 18 Jan 2019 17:54:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2CAA48E001F; Fri, 18 Jan 2019 12:54:32 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 278758E0002; Fri, 18 Jan 2019 12:54:32 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 168578E001F; Fri, 18 Jan 2019 12:54:32 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id AD57D8E0002 for ; Fri, 18 Jan 2019 12:54:31 -0500 (EST) Received: by mail-ed1-f71.google.com with SMTP id v4so5126000edm.18 for ; Fri, 18 Jan 2019 09:54:31 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=svAeuwrNxudtwt6CtwTN2Hw03wYxbsBKu/JBqRy4INM=; b=kb4R/N3b+BA107lI4gILFokG4aA94pltPmaoE7zvA0Uut7PSilqTwRQPkrJfgbHw9m MUMD1IGk/Lakw2iMmtrUf55fgf2RNe2otJgCsm9gC5MTgXJC0fputFOlfK/FlgHLo/rb 2pi+0qCtaTlpKN0Q2IDaszKS/Y61qapCeIzNu79MK1grP0HBvNiMnXrR04lRnZ4w3Mv/ yC7WS7TOjmKnvwnSeTipM9tYlM50uUtpGAJdgk5xb9sJzjZfqrCJHVVvkwJAIt7jwIMe SynOMaZAFjaJ1jGEN9sqLFDsbqkN6lIP9M6x+sBK1MSCCp0GMs8ilVJhNNG/OnUj0MI9 gqLg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukcrdBn4FQzdocnk2Rt7HCbxx75cq3j+DzLT9X3mECByFplgJxyn exL6kBAS8y1t+vWE1SkJ7ZDRlrkhfQmtpaOiFxPUsO1lNxWjES4hijhe0MGA0i6RiwQJxfEv0LT l8OQztvXHlsNKSUu3j7Nz8kzf2dciohzWnkhD75Ey1AWg58STBEjBuN3HVcZgDy2V8Q== X-Received: by 2002:a17:906:734c:: with SMTP id h12-v6mr15115804ejl.50.1547834071177; Fri, 18 Jan 2019 09:54:31 -0800 (PST) X-Google-Smtp-Source: ALg8bN6nN5+xM6IsZ03G+4OWgnMjJxzJsdLwVxKUsS7r0fnD+XE2x3k7MnjJO9WQxamPKuwDZpZ9 X-Received: by 2002:a17:906:734c:: with SMTP id h12-v6mr15115755ejl.50.1547834070057; Fri, 18 Jan 2019 09:54:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547834070; cv=none; d=google.com; s=arc-20160816; b=CEHbLtazmwMZ0kKta25/6xVFK9bbjgPpZ4GAbkNUCo+/miu9ajVIAeghmTVksy5ebj K+OGLGm0qCQ18mufwBYILogwcmURp6C/3jPQGjWZLNVIFqai0DK8H+ZUQ1VvUuAuOOaK jj9sM9t5cYZko42zrJ9WczUgkHzS67FKtcO+xH+1o/fr4UiCS3fcMcn8wTy+/BKBZj5z ZYgKuaaN5Umftw2IOkuqloGBF05+erVABzBGP9xCVx4ikqRG5921ANGuoidJTnsgGirr 8MBzX2JYfgHD2uuuiSIqH8y9NlojCL2PQD8h990QnJq9nkprDFnZKGJXpkWE2Im5CXhQ OowQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=svAeuwrNxudtwt6CtwTN2Hw03wYxbsBKu/JBqRy4INM=; b=HtBxjNIg9h+C0mQgTmZxjCkhikW6ri329FmBUV8XEpJXEleQptO2D0C4SV7XRMew4e Kmv40mDKfz6IQVHfQRvqgh3/jJUJRTTIhqH/rZLtSf8oONx4QMUMA3XBva/BJgBobIjB hTvhOdUP5EWI5iZY/eG7wzRaRRM3Hea5+0e+vyWXyoZHbJjULHy6FgfmwyYaxSmNzRVw XmCsb5T0ErcxE3BDaEZbC32En5TvoM+96sX7hEvqYxS8ymejUcTBSxPTmUwmlYxj1Adi 0nYqHp4yPSRf3I7J37dBGWadG/ZClstYtTZkyvJXYVC8XrlRyFQ2YDW1LehEyjMTVZ3N xHEA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp26.blacknight.com (outbound-smtp26.blacknight.com. [81.17.249.194]) by mx.google.com with ESMTPS id 18-v6si236742ejo.149.2019.01.18.09.54.29 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:54:30 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) client-ip=81.17.249.194; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp26.blacknight.com (Postfix) with ESMTPS id C0B4BB879A for ; Fri, 18 Jan 2019 17:54:29 +0000 (GMT) Received: (qmail 6551 invoked from network); 18 Jan 2019 17:54:29 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:54:29 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 16/22] mm, compaction: Rework compact_should_abort as compact_check_resched Date: Fri, 18 Jan 2019 17:51:30 +0000 Message-Id: <20190118175136.31341-17-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP With incremental changes, compact_should_abort no longer makes any documented sense. Rename to compact_check_resched and update the associated comments. There is no benefit other than reducing redundant code and making the intent slightly clearer. It could potentially be merged with earlier patches but it just makes the review slightly harder. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 61 ++++++++++++++++++++++----------------------------------- 1 file changed, 23 insertions(+), 38 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 829540f6f3da..9aa71945255d 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -404,6 +404,21 @@ static bool compact_lock_irqsave(spinlock_t *lock, unsigned long *flags, return true; } +/* + * Aside from avoiding lock contention, compaction also periodically checks + * need_resched() and records async compaction as contended if necessary. + */ +static inline void compact_check_resched(struct compact_control *cc) +{ + /* async compaction aborts if contended */ + if (need_resched()) { + if (cc->mode == MIGRATE_ASYNC) + cc->contended = true; + + cond_resched(); + } +} + /* * Compaction requires the taking of some coarse locks that are potentially * very heavily contended. The lock should be periodically unlocked to avoid @@ -432,33 +447,7 @@ static bool compact_unlock_should_abort(spinlock_t *lock, return true; } - if (need_resched()) { - if (cc->mode == MIGRATE_ASYNC) - cc->contended = true; - cond_resched(); - } - - return false; -} - -/* - * Aside from avoiding lock contention, compaction also periodically checks - * need_resched() and either schedules in sync compaction or aborts async - * compaction. This is similar to what compact_unlock_should_abort() does, but - * is used where no lock is concerned. - * - * Returns false when no scheduling was needed, or sync compaction scheduled. - * Returns true when async compaction should abort. - */ -static inline bool compact_should_abort(struct compact_control *cc) -{ - /* async compaction aborts if contended */ - if (need_resched()) { - if (cc->mode == MIGRATE_ASYNC) - cc->contended = true; - - cond_resched(); - } + compact_check_resched(cc); return false; } @@ -747,8 +736,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, return 0; } - if (compact_should_abort(cc)) - return 0; + compact_check_resched(cc); if (cc->direct_compaction && (cc->mode == MIGRATE_ASYNC)) { skip_on_failure = true; @@ -1379,12 +1367,10 @@ static void isolate_freepages(struct compact_control *cc) isolate_start_pfn = block_start_pfn) { /* * This can iterate a massively long zone without finding any - * suitable migration targets, so periodically check if we need - * to schedule, or even abort async compaction. + * suitable migration targets, so periodically check resched. */ - if (!(block_start_pfn % (SWAP_CLUSTER_MAX * pageblock_nr_pages)) - && compact_should_abort(cc)) - break; + if (!(block_start_pfn % (SWAP_CLUSTER_MAX * pageblock_nr_pages))) + compact_check_resched(cc); page = pageblock_pfn_to_page(block_start_pfn, block_end_pfn, zone); @@ -1675,11 +1661,10 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone, /* * This can potentially iterate a massively long zone with * many pageblocks unsuitable, so periodically check if we - * need to schedule, or even abort async compaction. + * need to schedule. */ - if (!(low_pfn % (SWAP_CLUSTER_MAX * pageblock_nr_pages)) - && compact_should_abort(cc)) - break; + if (!(low_pfn % (SWAP_CLUSTER_MAX * pageblock_nr_pages))) + compact_check_resched(cc); page = pageblock_pfn_to_page(block_start_pfn, block_end_pfn, zone); From patchwork Fri Jan 18 17:51:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771385 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2DFAA91E for ; Fri, 18 Jan 2019 17:54:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1C58D2FF7F for ; Fri, 18 Jan 2019 17:54:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 10A482FFB8; Fri, 18 Jan 2019 17:54:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8483B2FF7F for ; Fri, 18 Jan 2019 17:54:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7FB008E0020; Fri, 18 Jan 2019 12:54:42 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 781D48E0002; Fri, 18 Jan 2019 12:54:42 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 623158E0020; Fri, 18 Jan 2019 12:54:42 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id 04E898E0002 for ; Fri, 18 Jan 2019 12:54:42 -0500 (EST) Received: by mail-ed1-f71.google.com with SMTP id l45so5281158edb.1 for ; Fri, 18 Jan 2019 09:54:41 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=zQG6rQG/frHRZQkm9OpALx7BvtKeG7jt6cS8TBK6YUc=; b=klLurSjjjSVFiaMjA+F0Ssgv/IBezp9mcgheqMW5DqKXQBPxOttVII/eTdStg117xt 9zywyBlw48iAHXempxuGCWsqgLlBvT5ZkDcImIqoQf9MuWEwBoEX8Q9WJ7My9uFNCdeC NFIyKN+4k7cANnMIQq4027/pECqEa7sDWKTElu7z4tiUhKlYgSAHnaVnFl+JcoLMBEaF Qb+yYP+UnBgFOhpp5iNaHwNeAg7HI4hcboeDVqqUzdts3P1zN1xEJJyJXOYPczggtE+X of3rLS6w9QpB+gXtJ1gmbfEiZTMQgEAG3NAsdxXbqq3iDxsjaYbmZc6+4Eku4Z4xBMvP +8Uw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.193 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukcePWr7YdoxS8tqydY8sQVcxhBH27q1wnwvtfw5jgkp5Khcfnbk gum2Kkx12NL6+qIwWupIDkdCunVdou+ZuRyPJZE1ViD/DuNxvfTkM4cO/n/qHW8lldoRUOUgwF8 VX91lbfivjdpARsweSUGJKwEZJiA7rJF05m1Yn3h+STXav1XoLZVDPdSQkRYWk8rjAw== X-Received: by 2002:a50:d797:: with SMTP id w23mr17389065edi.19.1547834081475; Fri, 18 Jan 2019 09:54:41 -0800 (PST) X-Google-Smtp-Source: ALg8bN7+cSYcxi/2JBLCtBAjJEvPtqFhrSl8GBG4VCSy9rBVARjnNBWdB5Cg6q0BInZwWkN3aFSA X-Received: by 2002:a50:d797:: with SMTP id w23mr17389002edi.19.1547834080299; Fri, 18 Jan 2019 09:54:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547834080; cv=none; d=google.com; s=arc-20160816; b=pgkjL2pxTfnj4PcnJ9mLQgoi6aDz/s1mFP80kEyQp1YeckRki45u3vf1Mpp3jXu9k2 HX/8cjqSrjxgsVGti/nb3/9/x5fD+m4IU+rU/1PJYloRj1Eb+SqnR562xxyIKKHLLefL sdEbSTXm1aZxXf5BLRoyDyKX+Ihrmektwz77R2Zf9hdLQ/ZrpzroxYTqzxieNo3YhFjz PIRBZJ+/J0+eAo8vV2kixRLczoSdWVfv+2T1+v4vFj3Atfh9ivw6OT5Mbl2J2v/2eTI4 rqOzeAMMQ+kMO8t/iPMRZtJzh0aAxYgbtdmCfnoTkMLa+e9bWe/b6X/FN5nab5OSbR96 IjJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=zQG6rQG/frHRZQkm9OpALx7BvtKeG7jt6cS8TBK6YUc=; b=xGHNY+7NzNIlEIrIhCyz/cMPeKtLp7MriVDv1nZ1PBM7QEF9g7Md4WxkJzCzFMoyI9 cWMLS4ZG6oQs/OT2mbrtFy/6D4LGYG+FxgJMLNjYfUjKT3FZJIy6HqTWuzTP3tL4uvlJ kBT1xK0LBrvsvcRXJMGDUssEgXX9YhtJFCcAf90mjOyWUqcJdn7dZR5sr+L/PcXY7rvH ua6GoLicesGMCFgH1GG07Ls7ImtrrOcw+MpCCSrq+ljxpZ1BqWI1jDYhh6m2RoJoiUd5 LLXPrMnlBsLbSrdHI08ilXp4oLq7Mtm5R/LaZOIggENfaSRyAmae8yYno3i25ODdDsLp dRTA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.193 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp25.blacknight.com (outbound-smtp25.blacknight.com. [81.17.249.193]) by mx.google.com with ESMTPS id p33si9098726eda.412.2019.01.18.09.54.40 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:54:40 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.193 as permitted sender) client-ip=81.17.249.193; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.193 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp25.blacknight.com (Postfix) with ESMTPS id EE285B879A for ; Fri, 18 Jan 2019 17:54:39 +0000 (GMT) Received: (qmail 6988 invoked from network); 18 Jan 2019 17:54:39 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:54:39 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 17/22] mm, compaction: Do not consider a need to reschedule as contention Date: Fri, 18 Jan 2019 17:51:31 +0000 Message-Id: <20190118175136.31341-18-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Scanning on large machines can take a considerable length of time and eventually need to be rescheduled. This is treated as an abort event but that's not appropriate as the attempt is likely to be retried after making numerous checks and taking another cycle through the page allocator. This patch will check the need to reschedule if necessary but continue the scanning. The main benefit is reduced scanning when compaction is taking a long time or the machine is over-saturated. It also avoids an unnecessary exit of compaction that ends up being retried by the page allocator in the outer loop. 5.0.0-rc1 5.0.0-rc1 synccached-v3r16 noresched-v3r17 Amean fault-both-1 0.00 ( 0.00%) 0.00 * 0.00%* Amean fault-both-3 2958.27 ( 0.00%) 2965.68 ( -0.25%) Amean fault-both-5 4091.90 ( 0.00%) 3995.90 ( 2.35%) Amean fault-both-7 5803.05 ( 0.00%) 5842.12 ( -0.67%) Amean fault-both-12 9481.06 ( 0.00%) 9550.87 ( -0.74%) Amean fault-both-18 14141.51 ( 0.00%) 13304.72 ( 5.92%) Amean fault-both-24 16438.00 ( 0.00%) 14618.59 ( 11.07%) Amean fault-both-30 17531.72 ( 0.00%) 16650.96 ( 5.02%) Amean fault-both-32 17101.96 ( 0.00%) 17145.15 ( -0.25%) Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 23 ++++------------------- 1 file changed, 4 insertions(+), 19 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 9aa71945255d..293d9a9e6f00 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -404,21 +404,6 @@ static bool compact_lock_irqsave(spinlock_t *lock, unsigned long *flags, return true; } -/* - * Aside from avoiding lock contention, compaction also periodically checks - * need_resched() and records async compaction as contended if necessary. - */ -static inline void compact_check_resched(struct compact_control *cc) -{ - /* async compaction aborts if contended */ - if (need_resched()) { - if (cc->mode == MIGRATE_ASYNC) - cc->contended = true; - - cond_resched(); - } -} - /* * Compaction requires the taking of some coarse locks that are potentially * very heavily contended. The lock should be periodically unlocked to avoid @@ -447,7 +432,7 @@ static bool compact_unlock_should_abort(spinlock_t *lock, return true; } - compact_check_resched(cc); + cond_resched(); return false; } @@ -736,7 +721,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, return 0; } - compact_check_resched(cc); + cond_resched(); if (cc->direct_compaction && (cc->mode == MIGRATE_ASYNC)) { skip_on_failure = true; @@ -1370,7 +1355,7 @@ static void isolate_freepages(struct compact_control *cc) * suitable migration targets, so periodically check resched. */ if (!(block_start_pfn % (SWAP_CLUSTER_MAX * pageblock_nr_pages))) - compact_check_resched(cc); + cond_resched(); page = pageblock_pfn_to_page(block_start_pfn, block_end_pfn, zone); @@ -1664,7 +1649,7 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone, * need to schedule. */ if (!(low_pfn % (SWAP_CLUSTER_MAX * pageblock_nr_pages))) - compact_check_resched(cc); + cond_resched(); page = pageblock_pfn_to_page(block_start_pfn, block_end_pfn, zone); From patchwork Fri Jan 18 17:51:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771387 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 57FFF91E for ; Fri, 18 Jan 2019 17:54:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 46D532FF95 for ; Fri, 18 Jan 2019 17:54:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 39F682FFCC; Fri, 18 Jan 2019 17:54:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 883A02FF95 for ; Fri, 18 Jan 2019 17:54:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7AECE8E0021; Fri, 18 Jan 2019 12:54:52 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 75B128E0002; Fri, 18 Jan 2019 12:54:52 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 64ABA8E0021; Fri, 18 Jan 2019 12:54:52 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 018108E0002 for ; Fri, 18 Jan 2019 12:54:52 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id z10so5254853edz.15 for ; Fri, 18 Jan 2019 09:54:51 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=ocNhlG4rMc1hcPy/Atf76NV7+izKch3aoerr5YAXBbA=; b=BPZzxsa6EAMXwwoDxZ4s7LhqT/UdHlug+xjJwzDE4orMoMiKIXnu9Q0kBPGG3VqObE bEObz4Fm4RpVzaku1jSjOQx0AXIOYS0kjKzMDQkr56teNdFgOmzBXfanFDugn7MyVelm oHA6MnVElcAGcN0f2kcNYEpLf03U5Up+ya6xufhL9Ud+qMISmyPqYPWgywOQw2Gvxu0T 0/Zzz/CnFXVIVx9s1DduaPt01BVHu21idkZXKhGRWhhfsKVwkCg2ElMyTMcq+xClXN1S pcNY+93i3hoCPOBujIzIzm/YV16AIu00rBNOjFQZIWLlqjqApPjrx5RbbXlvRgooNTnf ZboQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukf6Gi/iv00NwF4/aO1ffDBYlIvWudp8OZqTKeFajIHD1YhMn6Fj hQo0x8P0v/Pet0R0gYYJK31RtXAQoEY0IzAIFiWniaccoeBexxUqsQcXoKCT/4MsDRikOF8h7jz L8MBc5jtSMLTGrbKK5teaFKfvoj0YD1clsmClS1nXMRFIhM8jm+7NoGU2nF/shY4P3w== X-Received: by 2002:a50:983a:: with SMTP id g55mr16903764edb.143.1547834091476; Fri, 18 Jan 2019 09:54:51 -0800 (PST) X-Google-Smtp-Source: ALg8bN5nSRtQ2GKwBLajYV4LJVfJkdBEemUrYmscI+keZhrNXo6M9W6/asagEiooC6+uwB+BOuWN X-Received: by 2002:a50:983a:: with SMTP id g55mr16903703edb.143.1547834090412; Fri, 18 Jan 2019 09:54:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547834090; cv=none; d=google.com; s=arc-20160816; b=o7f+NOf2tkMsQdYWxuAQczHnuCUGT89/EyKulG5moyhtTagh/37YymnY2WPhbDKA6V HDS60NQQRBiBQdV0rlTJkLoEaRswtvEeR8XPWFCkvUlcsEiz6C+UKht+C6WuWR5Yoc1W of7rBRs9NdgHm6VqB/i0moisZ0D8hkg0LKVOd4RNt42no/UfMXCyfy0H26xcKbHorJ/t KrrrxXy5a/Dmach6Fns5/VlZPtfuT8OkQe0+kNtjl5QuEAAB7srYi5jfwOQ37MLvrYmQ djgtDszxPC//lc95kZZyT65Nb2vfnEaJ1MbbXqGrMZYc5jTjSJN5DiDOe9K8jfdgZjyZ EPOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=ocNhlG4rMc1hcPy/Atf76NV7+izKch3aoerr5YAXBbA=; b=VIo6Cy/zH5j8Goq6T7suQlZkdj5shArtE9IIpFpf86W2FbdmssGGb/ri7nl53jWb8B OGVKmlXZsy5UrfCKs8kle+X4jvfIfTy4e38y3fvKARLNryt78evG4z1jGuJ460vweKWl 3uAAPpBT/vMuXZjdBDKzEwiqj/Zi2kniJO3AbnijD8JAO9vnNN+hU+xkThfrlAQ2vnYF 4Z5ev0SiDGEC4qSgoL1NR5+k3au1XfYnl0QUMnYN9np10ZUvXOvTNkB1i2HR3+7sL6r+ RWgvVzn+3f/oFNJ/vVSkX7nN7SzEIsaJ+5657jYlWBTub/ny0kxlD5KLYuNLAtde0y9u DZ4A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp02.blacknight.com (outbound-smtp02.blacknight.com. [81.17.249.8]) by mx.google.com with ESMTPS id w11si2420117edl.223.2019.01.18.09.54.50 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:54:50 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) client-ip=81.17.249.8; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp02.blacknight.com (Postfix) with ESMTPS id 1BE1D98BD2 for ; Fri, 18 Jan 2019 17:54:50 +0000 (UTC) Received: (qmail 7288 invoked from network); 18 Jan 2019 17:54:50 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:54:50 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 18/22] mm, compaction: Reduce premature advancement of the migration target scanner Date: Fri, 18 Jan 2019 17:51:32 +0000 Message-Id: <20190118175136.31341-19-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The fast isolation of free pages allows the cached PFN of the free scanner to advance faster than necessary depending on the contents of the free list. The key is that fast_isolate_freepages() can update zone->compact_cached_free_pfn via isolate_freepages_block(). When the fast search fails, the linear scan can start from a point that has skipped valid migration targets, particularly pageblocks with just low-order free pages. This can cause the migration source/target scanners to meet prematurely causing a reset. This patch starts by avoiding an update of the pageblock skip information and cached PFN from isolate_freepages_block() and puts the responsibility of updating that information in the callers. The fast scanner will update the cached PFN if and only if it finds a block that is higher than the existing cached PFN and sets the skip if the pageblock is full or nearly full. The linear scanner will update skipped information and the cached PFN only when a block is completely scanned. The total impact is that the free scanner advances more slowly as it is primarily driven by the linear scanner instead of the fast search. 5.0.0-rc1 5.0.0-rc1 noresched-v3r17 slowfree-v3r17 Amean fault-both-3 2965.68 ( 0.00%) 3036.75 ( -2.40%) Amean fault-both-5 3995.90 ( 0.00%) 4522.24 * -13.17%* Amean fault-both-7 5842.12 ( 0.00%) 6365.35 ( -8.96%) Amean fault-both-12 9550.87 ( 0.00%) 10340.93 ( -8.27%) Amean fault-both-18 13304.72 ( 0.00%) 14732.46 ( -10.73%) Amean fault-both-24 14618.59 ( 0.00%) 16288.96 ( -11.43%) Amean fault-both-30 16650.96 ( 0.00%) 16346.21 ( 1.83%) Amean fault-both-32 17145.15 ( 0.00%) 19317.49 ( -12.67%) The impact to latency is higher than the last version but it appears to be due to a slight increase in the free scan rates which is a potential side-effect of the patch. However, this is necessary for later patches that are more careful about how pageblocks are treated as earlier iterations of those patches hit corner cases where the restarts were punishing and very visible. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 27 ++++++++++----------------- 1 file changed, 10 insertions(+), 17 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 293d9a9e6f00..04ec7d4da719 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -330,10 +330,9 @@ static void update_cached_migrate(struct compact_control *cc, unsigned long pfn) * future. The information is later cleared by __reset_isolation_suitable(). */ static void update_pageblock_skip(struct compact_control *cc, - struct page *page, unsigned long nr_isolated) + struct page *page, unsigned long pfn) { struct zone *zone = cc->zone; - unsigned long pfn; if (cc->no_set_skip_hint) return; @@ -341,13 +340,8 @@ static void update_pageblock_skip(struct compact_control *cc, if (!page) return; - if (nr_isolated) - return; - set_pageblock_skip(page); - pfn = page_to_pfn(page); - /* Update where async and sync compaction should restart */ if (pfn < zone->compact_cached_free_pfn) zone->compact_cached_free_pfn = pfn; @@ -365,7 +359,7 @@ static inline bool pageblock_skip_persistent(struct page *page) } static inline void update_pageblock_skip(struct compact_control *cc, - struct page *page, unsigned long nr_isolated) + struct page *page, unsigned long pfn) { } @@ -449,7 +443,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, bool strict) { int nr_scanned = 0, total_isolated = 0; - struct page *cursor, *valid_page = NULL; + struct page *cursor; unsigned long flags = 0; bool locked = false; unsigned long blockpfn = *start_pfn; @@ -476,9 +470,6 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, if (!pfn_valid_within(blockpfn)) goto isolate_fail; - if (!valid_page) - valid_page = page; - /* * For compound pages such as THP and hugetlbfs, we can save * potentially a lot of iterations if we skip them at once. @@ -566,10 +557,6 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, if (strict && blockpfn < end_pfn) total_isolated = 0; - /* Update the pageblock-skip if the whole pageblock was scanned */ - if (blockpfn == end_pfn) - update_pageblock_skip(cc, valid_page, total_isolated); - cc->total_free_scanned += nr_scanned; if (total_isolated) count_compact_events(COMPACTISOLATED, total_isolated); @@ -1293,8 +1280,10 @@ fast_isolate_freepages(struct compact_control *cc) } } - if (highest && highest > cc->zone->compact_cached_free_pfn) + if (highest && highest >= cc->zone->compact_cached_free_pfn) { + highest -= pageblock_nr_pages; cc->zone->compact_cached_free_pfn = highest; + } cc->total_free_scanned += nr_scanned; if (!page) @@ -1374,6 +1363,10 @@ static void isolate_freepages(struct compact_control *cc) isolate_freepages_block(cc, &isolate_start_pfn, block_end_pfn, freelist, false); + /* Update the skip hint if the full pageblock was scanned */ + if (isolate_start_pfn == block_end_pfn) + update_pageblock_skip(cc, page, block_start_pfn); + /* Are enough freepages isolated? */ if (cc->nr_freepages >= cc->nr_migratepages) { if (isolate_start_pfn >= block_end_pfn) { From patchwork Fri Jan 18 17:51:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771389 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7F2A46C5 for ; Fri, 18 Jan 2019 17:55:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6A0372FF95 for ; Fri, 18 Jan 2019 17:55:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 584182FFCC; Fri, 18 Jan 2019 17:55:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CC47D2FF95 for ; Fri, 18 Jan 2019 17:55:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C8E858E0022; Fri, 18 Jan 2019 12:55:02 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C3E468E0002; Fri, 18 Jan 2019 12:55:02 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B03918E0022; Fri, 18 Jan 2019 12:55:02 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 52F598E0002 for ; Fri, 18 Jan 2019 12:55:02 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id i55so5265197ede.14 for ; Fri, 18 Jan 2019 09:55:02 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=r+ZeJpz1lZS5QW4P1aAtJ+FjrlN8m7Grj1Z7u/uZEQM=; b=gfQMVY9z32e19yJ6xChdD9og1rUt7No1j/WikEaYLKqeDXQAMRKLNQmOrrNSzUhXCY liRjxD+S1Sa+XSGUI0Dy6vxCyaPfyGEj0a+XvjCIQM5ba/LWiaB+5V1kDWyS1tzb9hms qRwaNUGen0IcyfLkv2Zl7PsKGA4VGlLKOLajaExBQjhcExGoLkaquLJQXkREzCzCzZJz sLMD2Ds3VZG85kAYpmpfWkpL2XKiYMp4kNsSqFKTbBxWi2dTCcVq3jZ/2Cu31YAvUvXH uvYFsPPMihj7QuhlQC6heHhz9yM/QjFp5XwGPieE79Y6KtFQwk0iuyv15ye8h6560/MQ +QyA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukcCIKzLQADKHd2CgBYCqk/SEKXMiLfu40pC7pqIuGQu/0cpC2pG RGiQgXBy+Xmk600QS86hWifhxk+IzdhcvJgghT3b43/jiJrYfU54COhWDQub6IgOA1OCRyAmTIW JYjD7Der4L4Tjzx7gbr97KKPmGZrRY/2RNDgtzQWN6D8KWkPxDMUE0XicfQDOKZhAXg== X-Received: by 2002:a50:a517:: with SMTP id y23mr17051937edb.219.1547834101812; Fri, 18 Jan 2019 09:55:01 -0800 (PST) X-Google-Smtp-Source: ALg8bN5YI1UQ7CLCwxCVp5m50sqeGLUr7nKCEzksn2jLHSJp5WQq+ojIbuPAJCMeHuBxcMD1rTqD X-Received: by 2002:a50:a517:: with SMTP id y23mr17051876edb.219.1547834100645; Fri, 18 Jan 2019 09:55:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547834100; cv=none; d=google.com; s=arc-20160816; b=XX3XugCoivF3J8qQQGiiTz8BsBLgXtAaUcHZg9nN/loqq7BFahbvKkXvd2BozV2zs9 OJhXsFR1Jy8XUvpt9pwJs0Q+L7zdVgj1PGphOc/aFT+m75cnr/qz4rACqg3l/IPViIGO ooYRwwTI0Oc430h+8zWW9+YNkbSNA+MLoW6XU2KtVL/9gLRGwmx7CXBesmgtk0Tp6t7F cNPy54fdfi9aiY1U4IzhqIJcIlFg74reoOB5kcu3Zb0W4TRSGEef/4RnWgA9Txw/zjlN 6scjGoAOmIqDrZrR8NGUGh/u+n9IMpaFTgWUlgzgWKW18X+MXiHdIEWFbf/BRRaOgGVF x1nA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=r+ZeJpz1lZS5QW4P1aAtJ+FjrlN8m7Grj1Z7u/uZEQM=; b=TGDxcmT2gIN8BLKjr9Uj0lKUQMYlDO1Qc7KpUzb4pJK/S9NqdBv9oTGuow8EqZ8Hwa iBKrXu+pBndryBiMfGntYqKIL3OIZsFC8vsV9wXYEfsYShVdaGM+BOHwpUlFLFcy/1MD ClmLmnSWY2ZYUknsTNwPAse5htMIWTRcvm5vC2OodLL7sjn1v7uHZyw/JSyWqQU0/qXg 2lKZgZSQUvb4waYTy7mvTewFMWayrhpzMn7+tAigLYr/0dX2yxzeHp3xXdBoIl0ZPhPU Wm3yc9AjPeSO8+f/yEar57/i8wO6qnhWmBLVWItKDOHtXer4rooac6zW3fHAlEa1pF5Y kzmA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp13.blacknight.com (outbound-smtp13.blacknight.com. [46.22.139.230]) by mx.google.com with ESMTPS id l17si5873171edq.20.2019.01.18.09.55.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:55:00 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) client-ip=46.22.139.230; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (unknown [81.17.254.16]) by outbound-smtp13.blacknight.com (Postfix) with ESMTPS id 4B9D31C3064 for ; Fri, 18 Jan 2019 17:55:00 +0000 (GMT) Received: (qmail 7671 invoked from network); 18 Jan 2019 17:55:00 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:55:00 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 19/22] mm, compaction: Round-robin the order while searching the free lists for a target Date: Fri, 18 Jan 2019 17:51:33 +0000 Message-Id: <20190118175136.31341-20-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP As compaction proceeds and creates high-order blocks, the free list search gets less efficient as the larger blocks are used as compaction targets. Eventually, the larger blocks will be behind the migration scanner for partially migrated pageblocks and the search fails. This patch round-robins what orders are searched so that larger blocks can be ignored and find smaller blocks that can be used as migration targets. The overall impact was small on 1-socket but it avoids corner cases where the migration/free scanners meet prematurely or situations where many of the pageblocks encountered by the free scanner are almost full instead of being properly packed. Previous testing had indicated that without this patch there were occasional large spikes in the free scanner without this patch. [dan.carpenter@oracle.com: Fix static checker warning] Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 33 ++++++++++++++++++++++++++++++--- mm/internal.h | 3 ++- 2 files changed, 32 insertions(+), 4 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 04ec7d4da719..7d39ff08269a 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1147,6 +1147,24 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn, unsigned long set_pageblock_skip(page); } +/* Search orders in round-robin fashion */ +static int next_search_order(struct compact_control *cc, int order) +{ + order--; + if (order < 0) + order = cc->order - 1; + + /* Search wrapped around? */ + if (order == cc->search_order) { + cc->search_order--; + if (cc->search_order < 0) + cc->search_order = cc->order - 1; + return -1; + } + + return order; +} + static unsigned long fast_isolate_freepages(struct compact_control *cc) { @@ -1183,9 +1201,15 @@ fast_isolate_freepages(struct compact_control *cc) if (WARN_ON_ONCE(min_pfn > low_pfn)) low_pfn = min_pfn; - for (order = cc->order - 1; - order >= 0 && !page; - order--) { + /* + * Search starts from the last successful isolation order or the next + * order to search after a previous failure + */ + cc->search_order = min_t(unsigned int, cc->order - 1, cc->search_order); + + for (order = cc->search_order; + !page && order >= 0; + order = next_search_order(cc, order)) { struct free_area *area = &cc->zone->free_area[order]; struct list_head *freelist; struct page *freepage; @@ -1209,6 +1233,7 @@ fast_isolate_freepages(struct compact_control *cc) if (pfn >= low_pfn) { cc->fast_search_fail = 0; + cc->search_order = order; page = freepage; break; } @@ -2136,6 +2161,7 @@ static enum compact_result compact_zone_order(struct zone *zone, int order, .total_migrate_scanned = 0, .total_free_scanned = 0, .order = order, + .search_order = order, .gfp_mask = gfp_mask, .zone = zone, .mode = (prio == COMPACT_PRIO_ASYNC) ? @@ -2367,6 +2393,7 @@ static void kcompactd_do_work(pg_data_t *pgdat) struct zone *zone; struct compact_control cc = { .order = pgdat->kcompactd_max_order, + .search_order = pgdat->kcompactd_max_order, .total_migrate_scanned = 0, .total_free_scanned = 0, .classzone_idx = pgdat->kcompactd_classzone_idx, diff --git a/mm/internal.h b/mm/internal.h index d5b999e5eb5f..31bb0be6fd52 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -192,7 +192,8 @@ struct compact_control { struct zone *zone; unsigned long total_migrate_scanned; unsigned long total_free_scanned; - unsigned int fast_search_fail; /* failures to use free list searches */ + unsigned short fast_search_fail;/* failures to use free list searches */ + short search_order; /* order to start a fast search at */ const gfp_t gfp_mask; /* gfp mask of a direct compactor */ int order; /* order a direct compactor needs */ int migratetype; /* migratetype of direct compactor */ From patchwork Fri Jan 18 17:51:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771391 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C7C5191E for ; Fri, 18 Jan 2019 17:55:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B3B592FF95 for ; Fri, 18 Jan 2019 17:55:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A79602FFCC; Fri, 18 Jan 2019 17:55:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0A5952FF95 for ; Fri, 18 Jan 2019 17:55:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A95C8E0023; Fri, 18 Jan 2019 12:55:13 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 062688E0002; Fri, 18 Jan 2019 12:55:13 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E3B748E0023; Fri, 18 Jan 2019 12:55:12 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id 858AE8E0002 for ; Fri, 18 Jan 2019 12:55:12 -0500 (EST) Received: by mail-ed1-f71.google.com with SMTP id m19so5262586edc.6 for ; Fri, 18 Jan 2019 09:55:12 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=/tOz8Xi3Yig60kJs65JVB2MYQvvX6oCbfn8y9xUdCPg=; b=bdIeHIhP0C/9uhNCs/a17Uq7EHoBt2/U2L6YzWT2CTuoTA+EzhrFKgg9Dhrz7zoFrC q/r/viIytHdjF293uvB+MOFusA9NfOAIan6IP6z8mVeYecZgd5nc94Mv4l/4eT+xMw3x BF1ijfI9KMozlQu6Gjsewg5iijRZ1pKN279vssdpbgxVBEvPJ+hcRSAggAvlVdeogi+a JwieS4cFLZ97dOscunfTMK8baHJ9fmDEU8wOznz9ugHgG+lHBZkMvquXSqmrsvSApv7k vDEbOj1f5rhv131ElnnWU1euHHKjCKmUgdXu5WZEWRW7dBWEY7djM4u65DI02b+IN0Mq jBkg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukf2xlcoJMp4vVkBOxCfxl09WnWfE5Y3Jbjv7FA2V3VRo7lKIMuq n8LhA2ClWGAATChAeAQa7kAWFDvEZ4dbVCVq4ibVIZ094+DO/mVZtMIMhvPwXvvutqNvTONNBVd DoccnHW/YE8Y3gDKCHds+i1FtgiDntvqSzKqfBSZuGHphu6L/A79cAvnFKJIDQskQUw== X-Received: by 2002:a05:6402:643:: with SMTP id u3mr16677495edx.276.1547834112083; Fri, 18 Jan 2019 09:55:12 -0800 (PST) X-Google-Smtp-Source: ALg8bN6yeugYPhoimoasRiItUREB5EV/v9V+dAg+RkDxWHUccfKHyss0kjjsyw9TBrYhcq5FAb8/ X-Received: by 2002:a05:6402:643:: with SMTP id u3mr16677424edx.276.1547834110799; Fri, 18 Jan 2019 09:55:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547834110; cv=none; d=google.com; s=arc-20160816; b=uMdo8aaVTISGBtLk9iEXOLTa4nrL1SoCGFpAz43jWXOdxP4cr1KxnxWodzCtNgYZkV 8ucw85oxk+cTDSWF6N5RyPMXlboeWzSMv31uPh/Sr/BeY2tT4ljElRkkkFbvz6SJi8qu vpecYKkiBzgmjZzpTdYIcvlO/pQ1dWVMxLCwBjftPTsYP3XOHXI9ykkWQrwga0tbdg4a V6G2hao7wXDDcbssTaam469LqzYpkpdTPQU9+/Venic93bbfxAwXOdZLgEgsc+12PKN5 hsPXRIzCnRp1ei3tDESkLukstrWr4m08VYjgnxdypBCs5U3XOo7vNMafoo/lVCYlBDO8 ugfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=/tOz8Xi3Yig60kJs65JVB2MYQvvX6oCbfn8y9xUdCPg=; b=OPlp54mVJWkNbJ3RJueVyefg5XZyBvyzKjjVimLOK7JLIMyBvYXyZv9ynvO/30xtow 1PPDK05Fir3KWbLs4RxO8DBsInqzxeg0XwG4LUxcsB/v5rTOLgisATz/9ryHElFf1guf ZpWcrYHKsju19w2Q4b/WEuwmm45Xnv0f3pasvayq1TncRYav3C3OQ8a+c82eiUXc+hSd jZHz5Ms11Do8Bg1l7hMOqW1r82JceoLiTk7JwxEu4u1Qdvlmm8YC7KfQJpiHlFQLEep/ lt3lG/rruEizPsfZAk19ZHip9o/X6wLgEf2CO9vC5W9dpgc3q9YluRkXq9iGHE9eYoO/ pcMA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp11.blacknight.com (outbound-smtp11.blacknight.com. [46.22.139.106]) by mx.google.com with ESMTPS id v14-v6si3876451ejw.273.2019.01.18.09.55.10 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:55:10 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) client-ip=46.22.139.106; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp11.blacknight.com (Postfix) with ESMTPS id 729001C30FC for ; Fri, 18 Jan 2019 17:55:10 +0000 (GMT) Received: (qmail 8177 invoked from network); 18 Jan 2019 17:55:10 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:55:10 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 20/22] mm, compaction: Sample pageblocks for free pages Date: Fri, 18 Jan 2019 17:51:34 +0000 Message-Id: <20190118175136.31341-21-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Once fast searching finishes, there is a possibility that the linear scanner is scanning full blocks found by the fast scanner earlier. This patch uses an adaptive stride to sample pageblocks for free pages. The more consecutive full pageblocks encountered, the larger the stride until a pageblock with free pages is found. The scanners might meet slightly sooner but it is an acceptable risk given that the search of the free lists may still encounter the pages and adjust the cached PFN of the free scanner accordingly. 5.0.0-rc1 5.0.0-rc1 roundrobin-v3r17 samplefree-v3r17 Amean fault-both-1 0.00 ( 0.00%) 0.00 * 0.00%* Amean fault-both-3 2752.37 ( 0.00%) 2729.95 ( 0.81%) Amean fault-both-5 4341.69 ( 0.00%) 4397.80 ( -1.29%) Amean fault-both-7 6308.75 ( 0.00%) 6097.61 ( 3.35%) Amean fault-both-12 10241.81 ( 0.00%) 9407.15 ( 8.15%) Amean fault-both-18 13736.09 ( 0.00%) 10857.63 * 20.96%* Amean fault-both-24 16853.95 ( 0.00%) 13323.24 * 20.95%* Amean fault-both-30 15862.61 ( 0.00%) 17345.44 ( -9.35%) Amean fault-both-32 18450.85 ( 0.00%) 16892.00 ( 8.45%) The latency is mildly improved offseting some overhead from earlier patches that are prerequisites for the rest of the series. However, a major impact is on the free scan rate with an 82% reduction. 5.0.0-rc1 5.0.0-rc1 roundrobin-v3r17 samplefree-v3r17 Compaction migrate scanned 21607271 20116887 Compaction free scanned 95336406 16668703 It's also the first time in the series where the number of pages scanned by the migration scanner is greater than the free scanner due to the increased search efficiency. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 7d39ff08269a..74bf620e3dcd 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -440,6 +440,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, unsigned long *start_pfn, unsigned long end_pfn, struct list_head *freelist, + unsigned int stride, bool strict) { int nr_scanned = 0, total_isolated = 0; @@ -449,10 +450,14 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, unsigned long blockpfn = *start_pfn; unsigned int order; + /* Strict mode is for isolation, speed is secondary */ + if (strict) + stride = 1; + cursor = pfn_to_page(blockpfn); /* Isolate free pages. */ - for (; blockpfn < end_pfn; blockpfn++, cursor++) { + for (; blockpfn < end_pfn; blockpfn += stride, cursor += stride) { int isolated; struct page *page = cursor; @@ -614,7 +619,7 @@ isolate_freepages_range(struct compact_control *cc, break; isolated = isolate_freepages_block(cc, &isolate_start_pfn, - block_end_pfn, &freelist, true); + block_end_pfn, &freelist, 0, true); /* * In strict mode, isolate_freepages_block() returns 0 if @@ -1132,7 +1137,7 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn, unsigned long /* Scan before */ if (start_pfn != pfn) { - isolate_freepages_block(cc, &start_pfn, pfn, &cc->freepages, false); + isolate_freepages_block(cc, &start_pfn, pfn, &cc->freepages, 1, false); if (cc->nr_freepages >= cc->nr_migratepages) return; } @@ -1140,7 +1145,7 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn, unsigned long /* Scan after */ start_pfn = pfn + nr_isolated; if (start_pfn != end_pfn) - isolate_freepages_block(cc, &start_pfn, end_pfn, &cc->freepages, false); + isolate_freepages_block(cc, &start_pfn, end_pfn, &cc->freepages, 1, false); /* Skip this pageblock in the future as it's full or nearly full */ if (cc->nr_freepages < cc->nr_migratepages) @@ -1332,6 +1337,7 @@ static void isolate_freepages(struct compact_control *cc) unsigned long block_end_pfn; /* end of current pageblock */ unsigned long low_pfn; /* lowest pfn scanner is able to scan */ struct list_head *freelist = &cc->freepages; + unsigned int stride; /* Try a small search of the free lists for a candidate */ isolate_start_pfn = fast_isolate_freepages(cc); @@ -1354,6 +1360,7 @@ static void isolate_freepages(struct compact_control *cc) block_end_pfn = min(block_start_pfn + pageblock_nr_pages, zone_end_pfn(zone)); low_pfn = pageblock_end_pfn(cc->migrate_pfn); + stride = cc->mode == MIGRATE_ASYNC ? COMPACT_CLUSTER_MAX : 1; /* * Isolate free pages until enough are available to migrate the @@ -1364,6 +1371,8 @@ static void isolate_freepages(struct compact_control *cc) block_end_pfn = block_start_pfn, block_start_pfn -= pageblock_nr_pages, isolate_start_pfn = block_start_pfn) { + unsigned long nr_isolated; + /* * This can iterate a massively long zone without finding any * suitable migration targets, so periodically check resched. @@ -1385,8 +1394,8 @@ static void isolate_freepages(struct compact_control *cc) continue; /* Found a block suitable for isolating free pages from. */ - isolate_freepages_block(cc, &isolate_start_pfn, block_end_pfn, - freelist, false); + nr_isolated = isolate_freepages_block(cc, &isolate_start_pfn, + block_end_pfn, freelist, stride, false); /* Update the skip hint if the full pageblock was scanned */ if (isolate_start_pfn == block_end_pfn) @@ -1410,6 +1419,13 @@ static void isolate_freepages(struct compact_control *cc) */ break; } + + /* Adjust stride depending on isolation */ + if (nr_isolated) { + stride = 1; + continue; + } + stride = min_t(unsigned int, COMPACT_CLUSTER_MAX, stride << 1); } /* From patchwork Fri Jan 18 17:51:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771393 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DEA6A91E for ; Fri, 18 Jan 2019 17:55:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CE4E42FF95 for ; Fri, 18 Jan 2019 17:55:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C26AE2FFCC; Fri, 18 Jan 2019 17:55:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1E1832FF95 for ; Fri, 18 Jan 2019 17:55:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B8658E0024; Fri, 18 Jan 2019 12:55:23 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 167E28E0002; Fri, 18 Jan 2019 12:55:23 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 009FA8E0024; Fri, 18 Jan 2019 12:55:22 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 96A398E0002 for ; Fri, 18 Jan 2019 12:55:22 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id f17so5240942edm.20 for ; Fri, 18 Jan 2019 09:55:22 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=17tWphiIoHiPa1vvYb3H3FZPlbycDfBU7EmArSPipfw=; b=nKPQzbB8Y3LpnZk3SSxQxbHAPG3DYw0osTZ5d2no+IyJZYwd6gqgBcPsBekFdnQYGN SPy7VxDOeKY6FHZmPQOALbc18x4izyEi5NIwq6f3d3sSph+rbCjQssIolOsaO+RRGR+A Dtt9DasA/HJWKaFVgtHsU+6gRURl/N6pajNw7MI75C33eHYXT/wJM5Q9DwW5M76ROUO9 PFsOBqi1bAsXVkGS+s89XqLOiGKlsP+DN5pdFsHxg1AQD09UWIWawtVLcACr2Bq/Zqz0 eQoiBFNqlEyAvtm5+2O+MWut+f9+oUfKASKyVGFCdFKfTlVWzfdKSvVXqCjjhaT0otE4 48ww== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukcU+1+PquNwvzaSQLukzErfzUHBZLKD5V260OGNT3dqJV1hBK4f oNxkA0ms6dPMPcixho0Y/30j8Tdui1Jt5k9bfkjl6l9QUKlBXfvwgO+3XP/FodOb/tgfcFapQ3w 6bEZh8cce588ArUOtJc32MQGC7CvLu+maKMvZ9qTe/IOe5Xj9UpN8PaM104qps1VTAQ== X-Received: by 2002:aa7:ccc8:: with SMTP id y8mr15949278edt.118.1547834122143; Fri, 18 Jan 2019 09:55:22 -0800 (PST) X-Google-Smtp-Source: ALg8bN4mabAnFjPn9BrlrhyrcRrqmtJJ9VY6aPqEPzqyEJP9u8GDT5QA2e0GPuSLhOfJXyglUPBc X-Received: by 2002:aa7:ccc8:: with SMTP id y8mr15949220edt.118.1547834120922; Fri, 18 Jan 2019 09:55:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547834120; cv=none; d=google.com; s=arc-20160816; b=wjlV1qv/f4qZaX4R9gAZcBqNhGPU9nUp4AJdXc5ehCQAushpz0pId5RopVK0PkEBwd 8M2fzRH8ZNpG0hMZbs4YVjJ0CLDf2uBLhcpko595WmhaEbpNz2vEezAlzNFD6XdPPlp/ vjLvBcYrWPXyJFBnv03RZiqLVP0R/gNNU92CDrtO1bSXd+l/+Bnd2fWZb8NQY4DIOD4+ fdlr6FXZw15GPb1puxf7HSG01jzaKKCcysuoKF5FxElnJRxArYs+Edc2+hT7vaYiiCdV nYq3AK+gmmFJv+uY+RUhxdrL0cEdOGDU5GA6kAIIzdT3bQNxolT43tV5yXeyxbbp9CLV fhAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=17tWphiIoHiPa1vvYb3H3FZPlbycDfBU7EmArSPipfw=; b=r+52+erMclaGxY72Om5YisXDSTon5fU3Eg+R7qWY2y4Jl1vuzNnIrhM7zVeXpRRsXc zio9pbggWBmGez5/7YX1i5qkfNwRZfOunB6SjOq5SBGdK3teJ7a7S/FOWtJQLN/Yh9CO XG1+omRNZ7kcm94UmAcFM0LQx2dZQDAlpn6Gnk4+64YCI30AJcInP5u8lved7Sux/Cg8 BAkMs76hUn3UxJQNK0lRtvMJVD93UXC9Ha6+ma0wx1AAvbybDhzM1XDwCbeMBv7Ss4wx 4Oul4J9fELNlK2VQ1ckpfbTF6XBBYMbVC/I1e/l7cCaPBN2UrMv7fZSKG2O2ca8YC470 UYfQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp26.blacknight.com (outbound-smtp26.blacknight.com. [81.17.249.194]) by mx.google.com with ESMTPS id w58si5438430edc.162.2019.01.18.09.55.20 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:55:20 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) client-ip=81.17.249.194; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp26.blacknight.com (Postfix) with ESMTPS id 97DFDB8A73 for ; Fri, 18 Jan 2019 17:55:20 +0000 (GMT) Received: (qmail 8547 invoked from network); 18 Jan 2019 17:55:20 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:55:20 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 21/22] mm, compaction: Be selective about what pageblocks to clear skip hints Date: Fri, 18 Jan 2019 17:51:35 +0000 Message-Id: <20190118175136.31341-22-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Pageblock hints are cleared when compaction restarts or kswapd makes enough progress that it can sleep but it's over-eager in that the bit is cleared for migration sources with no LRU pages and migration targets with no free pages. As pageblock skip hint flushes are relatively rare and out-of-band with respect to kswapd, this patch makes a few more expensive checks to see if it's appropriate to even clear the bit. Every pageblock that is not cleared will avoid 512 pages being scanned unnecessarily on x86-64. The impact is variable with different workloads showing small differences in latency, success rates and scan rates. This is expected as clearing the hints is not that common but doing a small amount of work out-of-band to avoid a large amount of work in-band later is generally a good thing. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- include/linux/mmzone.h | 2 + mm/compaction.c | 125 ++++++++++++++++++++++++++++++++++++++++++------- 2 files changed, 109 insertions(+), 18 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 842f9189537b..90c13cdeefb5 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -480,6 +480,8 @@ struct zone { unsigned long compact_cached_free_pfn; /* pfn where async and sync compaction migration scanner should start */ unsigned long compact_cached_migrate_pfn[2]; + unsigned long compact_init_migrate_pfn; + unsigned long compact_init_free_pfn; #endif #ifdef CONFIG_COMPACTION diff --git a/mm/compaction.c b/mm/compaction.c index 74bf620e3dcd..de558f110319 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -237,6 +237,71 @@ static bool pageblock_skip_persistent(struct page *page) return false; } +static bool +__reset_isolation_pfn(struct zone *zone, unsigned long pfn, bool check_source, + bool check_target) +{ + struct page *page = pfn_to_online_page(pfn); + struct page *end_page; + unsigned long block_pfn; + + if (!page) + return false; + if (zone != page_zone(page)) + return false; + if (pageblock_skip_persistent(page)) + return false; + + /* + * If skip is already cleared do no further checking once the + * restart points have been set. + */ + if (check_source && check_target && !get_pageblock_skip(page)) + return true; + + /* + * If clearing skip for the target scanner, do not select a + * non-movable pageblock as the starting point. + */ + if (!check_source && check_target && + get_pageblock_migratetype(page) != MIGRATE_MOVABLE) + return false; + + /* + * Only clear the hint if a sample indicates there is either a + * free page or an LRU page in the block. One or other condition + * is necessary for the block to be a migration source/target. + */ + block_pfn = pageblock_start_pfn(pfn); + pfn = max(block_pfn, zone->zone_start_pfn); + page = pfn_to_page(pfn); + if (zone != page_zone(page)) + return false; + pfn = block_pfn + pageblock_nr_pages; + pfn = min(pfn, zone_end_pfn(zone)); + end_page = pfn_to_page(pfn); + + do { + if (!pfn_valid_within(pfn)) + continue; + + if (check_source && PageLRU(page)) { + clear_pageblock_skip(page); + return true; + } + + if (check_target && PageBuddy(page)) { + clear_pageblock_skip(page); + return true; + } + + page += (1 << PAGE_ALLOC_COSTLY_ORDER); + pfn += (1 << PAGE_ALLOC_COSTLY_ORDER); + } while (page < end_page); + + return false; +} + /* * This function is called to clear all cached information on pageblocks that * should be skipped for page isolation when the migrate and free page scanner @@ -244,30 +309,54 @@ static bool pageblock_skip_persistent(struct page *page) */ static void __reset_isolation_suitable(struct zone *zone) { - unsigned long start_pfn = zone->zone_start_pfn; - unsigned long end_pfn = zone_end_pfn(zone); - unsigned long pfn; + unsigned long migrate_pfn = zone->zone_start_pfn; + unsigned long free_pfn = zone_end_pfn(zone); + unsigned long reset_migrate = free_pfn; + unsigned long reset_free = migrate_pfn; + bool source_set = false; + bool free_set = false; - zone->compact_blockskip_flush = false; + if (!zone->compact_blockskip_flush) + return; - /* Walk the zone and mark every pageblock as suitable for isolation */ - for (pfn = start_pfn; pfn < end_pfn; pfn += pageblock_nr_pages) { - struct page *page; + zone->compact_blockskip_flush = false; + /* + * Walk the zone and update pageblock skip information. Source looks + * for PageLRU while target looks for PageBuddy. When the scanner + * is found, both PageBuddy and PageLRU are checked as the pageblock + * is suitable as both source and target. + */ + for (; migrate_pfn < free_pfn; migrate_pfn += pageblock_nr_pages, + free_pfn -= pageblock_nr_pages) { cond_resched(); - page = pfn_to_online_page(pfn); - if (!page) - continue; - if (zone != page_zone(page)) - continue; - if (pageblock_skip_persistent(page)) - continue; + /* Update the migrate PFN */ + if (__reset_isolation_pfn(zone, migrate_pfn, true, source_set) && + migrate_pfn < reset_migrate) { + source_set = true; + reset_migrate = migrate_pfn; + zone->compact_init_migrate_pfn = reset_migrate; + zone->compact_cached_migrate_pfn[0] = reset_migrate; + zone->compact_cached_migrate_pfn[1] = reset_migrate; + } - clear_pageblock_skip(page); + /* Update the free PFN */ + if (__reset_isolation_pfn(zone, free_pfn, free_set, true) && + free_pfn > reset_free) { + free_set = true; + reset_free = free_pfn; + zone->compact_init_free_pfn = reset_free; + zone->compact_cached_free_pfn = reset_free; + } } - reset_cached_positions(zone); + /* Leave no distance if no suitable block was reset */ + if (reset_migrate >= reset_free) { + zone->compact_cached_migrate_pfn[0] = migrate_pfn; + zone->compact_cached_migrate_pfn[1] = migrate_pfn; + zone->compact_cached_free_pfn = free_pfn; + } } void reset_isolation_suitable(pg_data_t *pgdat) @@ -1190,7 +1279,7 @@ fast_isolate_freepages(struct compact_control *cc) * If starting the scan, use a deeper search and use the highest * PFN found if a suitable one is not found. */ - if (cc->free_pfn == pageblock_start_pfn(zone_end_pfn(cc->zone) - 1)) { + if (cc->free_pfn >= cc->zone->compact_init_free_pfn) { limit = pageblock_nr_pages >> 1; scan_start = true; } @@ -2015,7 +2104,7 @@ static enum compact_result compact_zone(struct compact_control *cc) cc->zone->compact_cached_migrate_pfn[1] = cc->migrate_pfn; } - if (cc->migrate_pfn == start_pfn) + if (cc->migrate_pfn <= cc->zone->compact_init_migrate_pfn) cc->whole_zone = true; } From patchwork Fri Jan 18 17:51:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771395 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AC6776C5 for ; Fri, 18 Jan 2019 17:55:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9BC542FF95 for ; Fri, 18 Jan 2019 17:55:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8FB242FFCC; Fri, 18 Jan 2019 17:55:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 955302FF95 for ; Fri, 18 Jan 2019 17:55:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 823298E0025; Fri, 18 Jan 2019 12:55:33 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7A8558E0002; Fri, 18 Jan 2019 12:55:33 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6718A8E0025; Fri, 18 Jan 2019 12:55:33 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id 06A058E0002 for ; Fri, 18 Jan 2019 12:55:33 -0500 (EST) Received: by mail-ed1-f69.google.com with SMTP id x15so5288296edd.2 for ; Fri, 18 Jan 2019 09:55:32 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=0bubIcgLmW8d8/ip6gRvUPCZZQFmOE9zr88ZVPqK+lE=; b=dG063MN5at+ywxK0TyRlZwmZ+f5aUfjC9sylZHUIE2/M+p9qlYKMdGV1s3BOQoIaqm HT1UPP27Qlbmh68KPcLTSL61wFEPzeJYgATQAQ05Mr70J+yDI6pXYlCpY6VDfT+OKqHM hg1URhA50tDjW4ga8owfe3rRn0+FvqhCTFDi9nufChg0G9teIPOK5vcbtI1/FaxDwPUK kajv0ZCNv7rtUk7pDG25Z5X+e2ybvP/jgW8TqZNrx+t+z5e6nbT2CvfStWEyE0tUUuOY KmvZYv6kI/8HVBK+NTMuLH7jHTouQKheFpfI2AUZtYsQ99LsCsAVw0cNS89EiXogQmmx Hhyg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukeh+RJQtIUALGI72xHzPMvArxtYfddeV+sFIrGrPMAtjr08D4uf O1t5K5i8s6AVKSZ2+t7hngoGinuvUJj8QfPZ464pwAD8rYKVgscliKRwbEpNrAzSdjaXtqbXIhJ sAE3oJXN85hXFcXrXBRo40lb+2l5kVMwpxtm5OeS56FXm8/wMe0T5K7SB5d3dWtYpHw== X-Received: by 2002:a17:906:20c2:: with SMTP id c2-v6mr15431424ejc.241.1547834132446; Fri, 18 Jan 2019 09:55:32 -0800 (PST) X-Google-Smtp-Source: ALg8bN5a3bkg2VSYPYoI+vOCOb8pbdhhCN8jYKG2FtLFdAuqn9RM+em3DplxZZt7D/aviwv6wAAt X-Received: by 2002:a17:906:20c2:: with SMTP id c2-v6mr15431357ejc.241.1547834131067; Fri, 18 Jan 2019 09:55:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547834131; cv=none; d=google.com; s=arc-20160816; b=DBOpQ6xkEDDeWAzzg98dqbYZ3ExqvMmD/GpIFjwkQ1LFzaX/tyv3E5QynqmmQ3YkeH 4/h3NrrrLxXFTKHLdZSTYX8AqvnIuqo54m9aMOHaWixk6AXcIuYT9IMzQRA5cMjJBtpm Z1dRIf1DhiRME5kTzmoXGmjPuvNit2P7yEMV+/T6vCEjRYJ2Kg11nFFrMun78Uezbd8n kaevKX98GWR3VUJbrQtERAnTRehAQ503B69KbCL38r7S03pH0Id8XL9qWzqPQeFrXyI9 9hn5Qp3RmXFLhHXo1OWNQANjckGm8n8gPTUhkdDmn4pd5z9OPcxeAKsJN+AqJmnQupOW 81Mw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=0bubIcgLmW8d8/ip6gRvUPCZZQFmOE9zr88ZVPqK+lE=; b=wOUoZU/pyMKo1AYpOFGp8Yl/+rvnhAj+VGRdlk6XHUEsET/ojbXwHzxzbVisr0TzVl 72SbN7+8fnkD8mnVGov3Mt2rtnZLmMNoY2oekIHXNEjDT99m0i/lZLefTDFoXmrMszQ2 iAA1pRT8gTh6WU7MiEnAQdGoEIu0whrZ+VcGZUq8CbGQQMU8QT2zt2bhy6sFquNiX8J5 mEYdaReqaNh3bKwqGReIKGNs15m5NLcCTdVABlCMbP3QWd1XbLMt0DCbJz6a3oGHDH6j yM7FBZGKn00CJr0Q5KC6Abr06s04TjeGFT7+F0LGWCqwWR/I15HCzu4TiMafaMQpmZ9z cVxA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp04.blacknight.com (outbound-smtp04.blacknight.com. [81.17.249.35]) by mx.google.com with ESMTPS id hh8-v6si1326332ejb.41.2019.01.18.09.55.30 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:55:31 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) client-ip=81.17.249.35; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp04.blacknight.com (Postfix) with ESMTPS id BC19198C63 for ; Fri, 18 Jan 2019 17:55:30 +0000 (UTC) Received: (qmail 9316 invoked from network); 18 Jan 2019 17:55:30 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:55:30 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 22/22] mm, compaction: Capture a page under direct compaction Date: Fri, 18 Jan 2019 17:51:36 +0000 Message-Id: <20190118175136.31341-23-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Compaction is inherently race-prone as a suitable page freed during compaction can be allocated by any parallel task. This patch uses a capture_control structure to isolate a page immediately when it is freed by a direct compactor in the slow path of the page allocator. The intent is to avoid redundant scanning. 5.0.0-rc1 5.0.0-rc1 selective-v3r17 capture-v3r19 Amean fault-both-1 0.00 ( 0.00%) 0.00 * 0.00%* Amean fault-both-3 2582.11 ( 0.00%) 2563.68 ( 0.71%) Amean fault-both-5 4500.26 ( 0.00%) 4233.52 ( 5.93%) Amean fault-both-7 5819.53 ( 0.00%) 6333.65 ( -8.83%) Amean fault-both-12 9321.18 ( 0.00%) 9759.38 ( -4.70%) Amean fault-both-18 9782.76 ( 0.00%) 10338.76 ( -5.68%) Amean fault-both-24 15272.81 ( 0.00%) 13379.55 * 12.40%* Amean fault-both-30 15121.34 ( 0.00%) 16158.25 ( -6.86%) Amean fault-both-32 18466.67 ( 0.00%) 18971.21 ( -2.73%) Latency is only moderately affected but the devil is in the details. A closer examination indicates that base page fault latency is reduced but latency of huge pages is increased as it takes creater care to succeed. Part of the "problem" is that allocation success rates are close to 100% even when under pressure and compaction gets harder 5.0.0-rc1 5.0.0-rc1 selective-v3r17 capture-v3r19 Percentage huge-3 96.70 ( 0.00%) 98.23 ( 1.58%) Percentage huge-5 96.99 ( 0.00%) 95.30 ( -1.75%) Percentage huge-7 94.19 ( 0.00%) 97.24 ( 3.24%) Percentage huge-12 94.95 ( 0.00%) 97.35 ( 2.53%) Percentage huge-18 96.74 ( 0.00%) 97.30 ( 0.58%) Percentage huge-24 97.07 ( 0.00%) 97.55 ( 0.50%) Percentage huge-30 95.69 ( 0.00%) 98.50 ( 2.95%) Percentage huge-32 96.70 ( 0.00%) 99.27 ( 2.65%) And scan rates are reduced as expected by 6% for the migration scanner and 29% for the free scanner indicating that there is less redundant work. Compaction migrate scanned 20815362 19573286 Compaction free scanned 16352612 11510663 Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- include/linux/compaction.h | 3 +- include/linux/sched.h | 4 +++ kernel/sched/core.c | 3 ++ mm/compaction.c | 31 ++++++++++++++----- mm/internal.h | 9 ++++++ mm/page_alloc.c | 74 +++++++++++++++++++++++++++++++++++++++++++--- 6 files changed, 112 insertions(+), 12 deletions(-) diff --git a/include/linux/compaction.h b/include/linux/compaction.h index 70d0256edd31..c960923d9ec2 100644 --- a/include/linux/compaction.h +++ b/include/linux/compaction.h @@ -93,7 +93,8 @@ extern int sysctl_compact_unevictable_allowed; extern int fragmentation_index(struct zone *zone, unsigned int order); extern enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, unsigned int alloc_flags, - const struct alloc_context *ac, enum compact_priority prio); + const struct alloc_context *ac, enum compact_priority prio, + struct page **page); extern void reset_isolation_suitable(pg_data_t *pgdat); extern enum compact_result compaction_suitable(struct zone *zone, int order, unsigned int alloc_flags, int classzone_idx); diff --git a/include/linux/sched.h b/include/linux/sched.h index 9a46243e6585..5e6690042497 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -47,6 +47,7 @@ struct pid_namespace; struct pipe_inode_info; struct rcu_node; struct reclaim_state; +struct capture_control; struct robust_list_head; struct sched_attr; struct sched_param; @@ -964,6 +965,9 @@ struct task_struct { struct io_context *io_context; +#ifdef CONFIG_COMPACTION + struct capture_control *capture_control; +#endif /* Ptrace state: */ unsigned long ptrace_message; kernel_siginfo_t *last_siginfo; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index a674c7db2f29..ae5beb3ed09e 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2177,6 +2177,9 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) INIT_HLIST_HEAD(&p->preempt_notifiers); #endif +#ifdef CONFIG_COMPACTION + p->capture_control = NULL; +#endif init_numa_balancing(clone_flags, p); } diff --git a/mm/compaction.c b/mm/compaction.c index de558f110319..2a6240d940e9 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2055,7 +2055,8 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, return false; } -static enum compact_result compact_zone(struct compact_control *cc) +static enum compact_result +compact_zone(struct compact_control *cc, struct capture_control *capc) { enum compact_result ret; unsigned long start_pfn = cc->zone->zone_start_pfn; @@ -2224,6 +2225,11 @@ static enum compact_result compact_zone(struct compact_control *cc) } } + /* Stop if a page has been captured */ + if (capc && capc->page) { + ret = COMPACT_SUCCESS; + break; + } } out: @@ -2257,7 +2263,8 @@ static enum compact_result compact_zone(struct compact_control *cc) static enum compact_result compact_zone_order(struct zone *zone, int order, gfp_t gfp_mask, enum compact_priority prio, - unsigned int alloc_flags, int classzone_idx) + unsigned int alloc_flags, int classzone_idx, + struct page **capture) { enum compact_result ret; struct compact_control cc = { @@ -2278,14 +2285,24 @@ static enum compact_result compact_zone_order(struct zone *zone, int order, .ignore_skip_hint = (prio == MIN_COMPACT_PRIORITY), .ignore_block_suitable = (prio == MIN_COMPACT_PRIORITY) }; + struct capture_control capc = { + .cc = &cc, + .page = NULL, + }; + + if (capture) + current->capture_control = &capc; INIT_LIST_HEAD(&cc.freepages); INIT_LIST_HEAD(&cc.migratepages); - ret = compact_zone(&cc); + ret = compact_zone(&cc, &capc); VM_BUG_ON(!list_empty(&cc.freepages)); VM_BUG_ON(!list_empty(&cc.migratepages)); + *capture = capc.page; + current->capture_control = NULL; + return ret; } @@ -2303,7 +2320,7 @@ int sysctl_extfrag_threshold = 500; */ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, unsigned int alloc_flags, const struct alloc_context *ac, - enum compact_priority prio) + enum compact_priority prio, struct page **capture) { int may_perform_io = gfp_mask & __GFP_IO; struct zoneref *z; @@ -2331,7 +2348,7 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, } status = compact_zone_order(zone, order, gfp_mask, prio, - alloc_flags, ac_classzone_idx(ac)); + alloc_flags, ac_classzone_idx(ac), capture); rc = max(status, rc); /* The allocation should succeed, stop compacting */ @@ -2399,7 +2416,7 @@ static void compact_node(int nid) INIT_LIST_HEAD(&cc.freepages); INIT_LIST_HEAD(&cc.migratepages); - compact_zone(&cc); + compact_zone(&cc, NULL); VM_BUG_ON(!list_empty(&cc.freepages)); VM_BUG_ON(!list_empty(&cc.migratepages)); @@ -2534,7 +2551,7 @@ static void kcompactd_do_work(pg_data_t *pgdat) if (kthread_should_stop()) return; - status = compact_zone(&cc); + status = compact_zone(&cc, NULL); if (status == COMPACT_SUCCESS) { compaction_defer_reset(zone, cc.order, false); diff --git a/mm/internal.h b/mm/internal.h index 31bb0be6fd52..9eeaf2b95166 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -209,6 +209,15 @@ struct compact_control { bool rescan; /* Rescanning the same pageblock */ }; +/* + * Used in direct compaction when a page should be taken from the freelists + * immediately when one is created during the free path. + */ +struct capture_control { + struct compact_control *cc; + struct page *page; +}; + unsigned long isolate_freepages_range(struct compact_control *cc, unsigned long start_pfn, unsigned long end_pfn); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 6607cb7131b0..d61174bb0333 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -786,6 +786,57 @@ static inline int page_is_buddy(struct page *page, struct page *buddy, return 0; } +#ifdef CONFIG_COMPACTION +static inline struct capture_control *task_capc(struct zone *zone) +{ + struct capture_control *capc = current->capture_control; + + return capc && + !(current->flags & PF_KTHREAD) && + !capc->page && + capc->cc->zone == zone && + capc->cc->direct_compaction ? capc : NULL; +} + +static inline bool +compaction_capture(struct capture_control *capc, struct page *page, + int order, int migratetype) +{ + if (!capc || order != capc->cc->order) + return false; + + /* Do not accidentally pollute CMA or isolated regions*/ + if (is_migrate_cma(migratetype) || + is_migrate_isolate(migratetype)) + return false; + + /* + * Do not let lower order allocations polluate a movable pageblock. + * This might let an unmovable request use a reclaimable pageblock + * and vice-versa but no more than normal fallback logic which can + * have trouble finding a high-order free page. + */ + if (order < pageblock_order && migratetype == MIGRATE_MOVABLE) + return false; + + capc->page = page; + return true; +} + +#else +static inline struct capture_control *task_capc(struct zone *zone) +{ + return NULL; +} + +static inline bool +compaction_capture(struct capture_control *capc, struct page *page, + int order, int migratetype) +{ + return false; +} +#endif /* CONFIG_COMPACTION */ + /* * Freeing function for a buddy system allocator. * @@ -819,6 +870,7 @@ static inline void __free_one_page(struct page *page, unsigned long uninitialized_var(buddy_pfn); struct page *buddy; unsigned int max_order; + struct capture_control *capc = task_capc(zone); max_order = min_t(unsigned int, MAX_ORDER, pageblock_order + 1); @@ -834,6 +886,12 @@ static inline void __free_one_page(struct page *page, continue_merging: while (order < max_order - 1) { + if (compaction_capture(capc, page, order, migratetype)) { + if (likely(!is_migrate_isolate(migratetype))) + __mod_zone_freepage_state(zone, -(1 << order), + migratetype); + return; + } buddy_pfn = __find_buddy_pfn(pfn, order); buddy = page + (buddy_pfn - pfn); @@ -3819,7 +3877,7 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, unsigned int alloc_flags, const struct alloc_context *ac, enum compact_priority prio, enum compact_result *compact_result) { - struct page *page; + struct page *page = NULL; unsigned long pflags; unsigned int noreclaim_flag; @@ -3830,13 +3888,15 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, noreclaim_flag = memalloc_noreclaim_save(); *compact_result = try_to_compact_pages(gfp_mask, order, alloc_flags, ac, - prio); + prio, &page); memalloc_noreclaim_restore(noreclaim_flag); psi_memstall_leave(&pflags); - if (*compact_result <= COMPACT_INACTIVE) + if (*compact_result <= COMPACT_INACTIVE) { + WARN_ON_ONCE(page); return NULL; + } /* * At least in one zone compaction wasn't deferred or skipped, so let's @@ -3844,7 +3904,13 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, */ count_vm_event(COMPACTSTALL); - page = get_page_from_freelist(gfp_mask, order, alloc_flags, ac); + /* Prep a captured page if available */ + if (page) + prep_new_page(page, order, gfp_mask, alloc_flags); + + /* Try get a page from the freelist if available */ + if (!page) + page = get_page_from_freelist(gfp_mask, order, alloc_flags, ac); if (page) { struct zone *zone = page_zone(page);