From patchwork Fri Dec 14 23:02:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10731775 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 067541399 for ; Fri, 14 Dec 2018 23:03:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AD5772D639 for ; Fri, 14 Dec 2018 23:03:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A1A3F2D63D; Fri, 14 Dec 2018 23:03:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0BB162D639 for ; Fri, 14 Dec 2018 23:03:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DDCB88E0219; Fri, 14 Dec 2018 18:03:13 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D45B68E021A; Fri, 14 Dec 2018 18:03:13 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8FCCF8E0219; Fri, 14 Dec 2018 18:03:13 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id 2752D8E0217 for ; Fri, 14 Dec 2018 18:03:13 -0500 (EST) Received: by mail-ed1-f69.google.com with SMTP id d41so3361101eda.12 for ; Fri, 14 Dec 2018 15:03:13 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=O/ocCy4XM3f5RRpCxQDpjepYrV6d4WKm5yIjsPMqPpc=; b=eSi+VKaN9W3WNPBxw8bGL095Z1q3WzJ8oAS22IqripkmpeQKPhffPK83vzT1DwO9SM U3djGlcPYMI0ek3l8U50Mp2uBo1ZYtZOo+4mLcUskCthpcDCSZiLG2PPbGT6Cap5MXIN kokoGYX1F7rBvWp7qbOX42kflS7NILrd0k28fBgO9+jHKptOEsHS3UUHb2RGuUXi3Wb/ zijnqp0x6U2HjLP7LZfaFojFe9radlcCquIc3nUopMFW9G/LQAB4auakaeQB3f0VYcQL UqkHm0eJqiU1XncsPYwrPkUQCbpJ7O9h6ogI0bUdP3FInNf5m+gxB2M7H3XGFv8NIiQ/ WB4Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWbnz2gqg2+g6RvM7iCSnAQKu19mxDC+jSvXirmgbP161ogN4fxM vNnhh1QFJsLMuPuvuDbOcyEq5ZUXzgFFoyPTExEeoBOS4Y0mpOeToB2X8nuEhhtPjlB0X7kx6R3 AGhPacEH91PC6euOR475EkriGU2CxqGFRp7LfGuV3jgj3Yec9RMVqul/QmrtWP3GdpA== X-Received: by 2002:a17:906:3548:: with SMTP id s8-v6mr3533330eja.161.1544828592657; Fri, 14 Dec 2018 15:03:12 -0800 (PST) X-Google-Smtp-Source: AFSGD/Xk8gIRDAKloMUK/qzkfBoPUzZgPPTabr1AcdLlZYKHEulMslNaZ+t6WbjJXcGpi67PJz/u X-Received: by 2002:a17:906:3548:: with SMTP id s8-v6mr3533307eja.161.1544828591609; Fri, 14 Dec 2018 15:03:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544828591; cv=none; d=google.com; s=arc-20160816; b=adMAPX7Gq1V808KqzTL0TZWAgJTKhUpE2Updw5GAnbaDIRS6xn5iHO+TFSE41RQO/+ 6MvgXcAxQUBtcB3pvXDksbkdQzBWC8fIWhW/Z+T2S2CNroiUaNS+6VU1ZG2E52OAVRjL eAbalU4Sxum+erK6rCZLNWePgNSg9nuRwoPEBKKroh7NkMyNFgKJyhNft1tS7gs7Y8wN cDwbl5p+Ct104cwWC81o7gjiiyvFYsaD40GoyTg8ihtr29zDZrYrVKOsTexnmKr69Rgn s1tLHI/hz3YXjBXVc0ZjzCTkddhl0nnFetaUGXhVCJ8lmuQ+u8SlxlBzfcoS7ByqClpQ 9SAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=O/ocCy4XM3f5RRpCxQDpjepYrV6d4WKm5yIjsPMqPpc=; b=0l8RU9+jmEMm1gXW4+Y1S09o/PlK70+oCLFchWRhY3ZHn0ddPIYBfO5KoZp2+KaGo9 pqaxYhzsZwvN2FisfAkhBadL7WprLyd1XZGx1ESWsF+9YJBwvNOv7p3ikPoDKJqBoh64 ieDkWkJQSlKaegZ6jOwwfBTZ+BPL8TCv6S+Or3PYOsEM+hJwRhniomQQg21SU7RrEHAv E+fV/dxNj8mciD/GehiGDGU6jmhd9eNmtxVJ1ITsh+qI87On4kL8UcTfdZ9aqNO3GmtI MOSHYVHd6EoT7sEkTV1TYfp+HwNI7dT/vYA5cyVD/d+U40bVFBbzmDt/o54qTSgC3QEN YMnQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp13.blacknight.com (outbound-smtp13.blacknight.com. [46.22.139.230]) by mx.google.com with ESMTPS id w44si1092158edb.74.2018.12.14.15.03.11 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 14 Dec 2018 15:03:11 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) client-ip=46.22.139.230; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (unknown [81.17.254.10]) by outbound-smtp13.blacknight.com (Postfix) with ESMTPS id 1C37B1C1FBE for ; Fri, 14 Dec 2018 23:03:11 +0000 (GMT) Received: (qmail 31163 invoked from network); 14 Dec 2018 23:03:11 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.245.71]) by 81.17.254.9 with ESMTPA; 14 Dec 2018 23:03:10 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Linus Torvalds , Michal Hocko , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 01/14] mm, compaction: Shrink compact_control Date: Fri, 14 Dec 2018 23:02:57 +0000 Message-Id: <20181214230310.572-2-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20181214230310.572-1-mgorman@techsingularity.net> References: <20181214230310.572-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The isolate and migrate scanners should never isolate more than a pageblock of pages so unsigned int is sufficient saving 8 bytes on a 64-bit build. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/internal.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 536bc2a839b9..5564841fce36 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -185,8 +185,8 @@ struct compact_control { struct list_head freepages; /* List of free pages to migrate to */ struct list_head migratepages; /* List of pages being migrated */ struct zone *zone; - unsigned long nr_freepages; /* Number of isolated free pages */ - unsigned long nr_migratepages; /* Number of pages to migrate */ + unsigned int nr_freepages; /* Number of isolated free pages */ + unsigned int nr_migratepages; /* Number of pages to migrate */ unsigned long total_migrate_scanned; unsigned long total_free_scanned; unsigned long free_pfn; /* isolate_freepages search base */ From patchwork Fri Dec 14 23:02:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10731771 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E4A0313AD for ; Fri, 14 Dec 2018 23:03:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 890422D639 for ; Fri, 14 Dec 2018 23:03:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 77A2C2D63D; Fri, 14 Dec 2018 23:03:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E41B52D639 for ; Fri, 14 Dec 2018 23:03:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9169B8E0217; Fri, 14 Dec 2018 18:03:13 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 887738E01DC; Fri, 14 Dec 2018 18:03:13 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 777D58E0219; Fri, 14 Dec 2018 18:03:13 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 1B0D58E0216 for ; Fri, 14 Dec 2018 18:03:13 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id e17so3418500edr.7 for ; Fri, 14 Dec 2018 15:03:13 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=8ylXBwUHUp/AHX4BZPSE6g0fwobKPJEsFH/Fxi+Stfo=; b=IBMJLZ1RNJFHmrJY2rSFiaxQgYYbbkoBqhtCQY45apt5sxW9zX0opImDiXDBg0l+w7 E09LV9XVuFsJmMXkWpxXLzqcO6x1ZxfCgAjNFaSZUC3FrHKs3wYbm8qvigtufnLDRIfF 15bIIaL8GlMIMH7eh0pviuWUwjkh/WEwLSWpjaNaF35PoAo2fwjXTGqCWMYZgX+3wxd2 ZVr7YqcmmShyQsykNTmA46Hl5vj47qp1g8J8ItqjLtuO35gwVnV4enFS4Q0Ih977cvM/ S8nLEz0v93EsXuROf7MJWKJC9ib4LM8GrGhReqDCxSJst/ddFzuGJ3DPz3Alpte/kMW2 mJxg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWanKJQCHea/r1YPrKtQM0yg7bpMIGO5vhmNkCszqYUx9h0dHpaS M8/1ez+/GMFHLPTwzzauE2Y/Gm6OEawXfjaDEsbfz9hpCZzBJDChNhUt8dobky6X/5v10aUf3TE HNIZrqtIktrpUfxgjnuj7Hajf4Vqgno0fl/IrmQpaQfQsiffMfrVIJNrHgxS6OOtvJw== X-Received: by 2002:aa7:c703:: with SMTP id i3mr176278edq.170.1544828592608; Fri, 14 Dec 2018 15:03:12 -0800 (PST) X-Google-Smtp-Source: AFSGD/VwQNxgSniiaw7jyLs0SHL69yNTkLToq+JEdkwlqPXxu6az5WsyvXu/qALF+g7InLD3vx5C X-Received: by 2002:aa7:c703:: with SMTP id i3mr176251edq.170.1544828591657; Fri, 14 Dec 2018 15:03:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544828591; cv=none; d=google.com; s=arc-20160816; b=btJQ+62gD7z5Hj5iVpp4X0J0Xu4/FYKIg7YvtodtfrPlEScQzx5Gwm4IryVkjigf/k /GxUTt1MaF+CM8HD2kaF6573zWbvcyO7V/rXRsmYXr0D0EJzCp26+SXG9l+9CElwK3C6 ExS8yLHuRKHG1v1WbOSNbdnnR0bAjCeE2T5D7M+aY8kIhEGHwfntvCUpJR3464dPOMJ+ fKcxr+ch8Z04QKCntOluXLUiWh9hFGsmmXx8U1/CiY+J6kHIAsuc5tLjvmnNaKbV4pph T2JAQq90SNnnjAtKkVez9jbfW/5peoIv3w57So2p2mgrY0iRKyApovca0+pr67Uhcj5U bItQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=8ylXBwUHUp/AHX4BZPSE6g0fwobKPJEsFH/Fxi+Stfo=; b=oAs+sTyT505AsQKPJrZaiGsiMG0RtMzC+73uNikRpjVwWWrTF2n8P0H6OSjchvNzFD 1z7/5UQB6bnH4Wn76Pa1MiVv/kzSDaoEQpW61Klh38JlRLQdMYeXM5olGNwVSaSFQvEw dUJGTGRPj05Sq3aVOCS5Cm7KuyP9uI3v/UPIe6XNFHnEOh8JBHqOr/T653tbzi/M+J+j eBs2dIOrzi4TfWpw4jE6dMQeYkzkcGyxV/FQCN8Grkys1mJ7CZ/zSOu77gWFkpRUHt5Z zazeGSVPzT+UNR4lvHM6Dz72IGpyChiAu/2+Hzj8rByV+SIvTXnsQNu7sG+jVdhDefXK Im3Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp10.blacknight.com (outbound-smtp10.blacknight.com. [46.22.139.15]) by mx.google.com with ESMTPS id hk19-v6si195815ejb.253.2018.12.14.15.03.11 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 14 Dec 2018 15:03:11 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) client-ip=46.22.139.15; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail01.blacknight.ie [81.17.254.10]) by outbound-smtp10.blacknight.com (Postfix) with ESMTPS id 54D231C1D19 for ; Fri, 14 Dec 2018 23:03:11 +0000 (GMT) Received: (qmail 31182 invoked from network); 14 Dec 2018 23:03:11 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.245.71]) by 81.17.254.9 with ESMTPA; 14 Dec 2018 23:03:11 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Linus Torvalds , Michal Hocko , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 02/14] mm, compaction: Rearrange compact_control Date: Fri, 14 Dec 2018 23:02:58 +0000 Message-Id: <20181214230310.572-3-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20181214230310.572-1-mgorman@techsingularity.net> References: <20181214230310.572-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP compact_control spans two cache lines with write-intensive lines on both. Rearrange so the most write-intensive fields are in the same cache line. This has a negligible impact on the overall performance of compaction and is more a tidying exercise than anything. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/internal.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 5564841fce36..867af5425432 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -184,14 +184,14 @@ extern int user_min_free_kbytes; struct compact_control { struct list_head freepages; /* List of free pages to migrate to */ struct list_head migratepages; /* List of pages being migrated */ - struct zone *zone; unsigned int nr_freepages; /* Number of isolated free pages */ unsigned int nr_migratepages; /* Number of pages to migrate */ - unsigned long total_migrate_scanned; - unsigned long total_free_scanned; unsigned long free_pfn; /* isolate_freepages search base */ unsigned long migrate_pfn; /* isolate_migratepages search base */ unsigned long last_migrated_pfn;/* Not yet flushed page being freed */ + struct zone *zone; + unsigned long total_migrate_scanned; + unsigned long total_free_scanned; const gfp_t gfp_mask; /* gfp mask of a direct compactor */ int order; /* order a direct compactor needs */ int migratetype; /* migratetype of direct compactor */ From patchwork Fri Dec 14 23:02:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10731777 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A67E913AD for ; Fri, 14 Dec 2018 23:03:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 599732D639 for ; Fri, 14 Dec 2018 23:03:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4E1F62D63D; Fri, 14 Dec 2018 23:03:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C206D2D639 for ; Fri, 14 Dec 2018 23:03:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2ECF08E021A; Fri, 14 Dec 2018 18:03:14 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 221C28E0218; Fri, 14 Dec 2018 18:03:14 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A6088E021B; Fri, 14 Dec 2018 18:03:14 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id 7085D8E0218 for ; Fri, 14 Dec 2018 18:03:13 -0500 (EST) Received: by mail-ed1-f69.google.com with SMTP id l45so3463095edb.1 for ; Fri, 14 Dec 2018 15:03:13 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=vDYTnelWXCZf8RMPZjSheUW9//OOrmryct6bxpECWz4=; b=ono/rst/dxIJiDSGNCtbR2R58RD2pbmG4Z0cWM2E0FXDFkAznBAweCRRGjj+tWy68k faaspaUhd/KIi7DG7VYhlqrcpcxU8kCdBF0Eu28QmkETcIpfXZs7fw5z/Vqj6H/eXWtW nLyjoq5Etk/6l1epd+eNjBpv8CQizDGmXgWLX7WEiihCFOabUq8OuETR7u+KJ8imJ+S/ ham3hEdZ9crH+SMgspl2pMsKfqSKUut13WwZ93zUK4ynIP8QURSs6IHfItjpwzUMUUw8 qjuxUkyhMcNCCfcpI7OiwmYDNJNJPZ6QeyFVr6Frpz9KZ8uroaXa7/Q0hC5knEveMJvE PKiA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWb7eGDs0JXbKTxlft9M33asVW/Tt/VjsSx/IjZvvU1ggHSxEknw qAwQ30pGuuji2YDR44m6Lm5rg8Lp79j0uTcOCmCQqMLPtGZB5mn2HdQ36lIzeEMMkU7R64t2adt 7IFvlyd9aqiko18u0xWlrfU4+3jwUiokV0Kq2ZGttnqCZnImAsC6YufUD7g/xx0b3TQ== X-Received: by 2002:a50:b487:: with SMTP id w7mr4646737edd.72.1544828592950; Fri, 14 Dec 2018 15:03:12 -0800 (PST) X-Google-Smtp-Source: AFSGD/V7XTNe3rRGdRD4qF8uAKJb0aVcA2oGaWrBdifeUHvtQyPNLWuNN0V9prrdLt4HtEwvPyNb X-Received: by 2002:a50:b487:: with SMTP id w7mr4646703edd.72.1544828592006; Fri, 14 Dec 2018 15:03:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544828591; cv=none; d=google.com; s=arc-20160816; b=Vl+bhf8gjiVlq1xaZK9POjb+TGr+spI5xljBaGKMDRnR0nWfy/bVliurxzq3WfEisf XvujrGWNaEScKw7WZJE/LMt3MLtuHYq3J6v5khFydYD9+skBVKWVRLb3OtZjk3Lk7d9e bSYtqMTLrn983Pb8gnTOdAjCywHP7EKinwJYQGU3wt4ARKnxEocRAPSxNueV8IbckZdj 8Iht0WBeZXnniJRdyL/gy446O3qTDaBtB3pAu8WDAB8J7Yt2ZOvarlQ4FtilBMYB5OEH YWxCAqJLgucIFAc7lcNx65pkjGeDQb7plVbCpYY/jc19QRFSWHxMHkE73tpcuasTmOI8 xydw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=vDYTnelWXCZf8RMPZjSheUW9//OOrmryct6bxpECWz4=; b=axkNOXZinR76dZedBSP+RhfOJroHnQL5u40umDN5lleo3x3zKjQPxxAJXToGCK+ln3 byw1CDj5+lgTbluEhXXpMJ9wnMx4N4oN/6E3nEhognVfCWlrUHFPPeMgWs+6npZySocY ck67a6prg33rlVl6k3zs7f/V8lZIAI+MhQAfUwxU5Xz92K24IWxUmLtZvvuLA4Bjtf4/ x0dI4YqCrSbfPw2UORuzxhPhREl/udLC5xsljz4FOe7RqEbJVk2BkED7rMMj1bkhtIOq HKPp3vhublKtFssomq/jKXPrGzXveS8B3zWnDhGJdPXt2pHOqphCkR8ylQVeKUyffWj9 2DOw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp11.blacknight.com (outbound-smtp11.blacknight.com. [46.22.139.106]) by mx.google.com with ESMTPS id p8-v6si2062488ejm.303.2018.12.14.15.03.11 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 14 Dec 2018 15:03:11 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) client-ip=46.22.139.106; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail01.blacknight.ie [81.17.254.10]) by outbound-smtp11.blacknight.com (Postfix) with ESMTPS id 8F9121C205F for ; Fri, 14 Dec 2018 23:03:11 +0000 (GMT) Received: (qmail 31209 invoked from network); 14 Dec 2018 23:03:11 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.245.71]) by 81.17.254.9 with ESMTPA; 14 Dec 2018 23:03:11 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Linus Torvalds , Michal Hocko , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 03/14] mm, compaction: Remove last_migrated_pfn from compact_control Date: Fri, 14 Dec 2018 23:02:59 +0000 Message-Id: <20181214230310.572-4-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20181214230310.572-1-mgorman@techsingularity.net> References: <20181214230310.572-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The last_migrated_pfn field is a bit dubious as to whether it really helps but either way, the information from it can be inferred without increasing the size of compact_control so remove the field. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 25 +++++++++---------------- mm/internal.h | 1 - 2 files changed, 9 insertions(+), 17 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index ef29490b0f46..fb4d9f52ed56 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -886,15 +886,6 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, cc->nr_migratepages++; nr_isolated++; - /* - * Record where we could have freed pages by migration and not - * yet flushed them to buddy allocator. - * - this is the lowest page that was isolated and likely be - * then freed by migration. - */ - if (!cc->last_migrated_pfn) - cc->last_migrated_pfn = low_pfn; - /* Avoid isolating too much */ if (cc->nr_migratepages == COMPACT_CLUSTER_MAX) { ++low_pfn; @@ -918,7 +909,6 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, } putback_movable_pages(&cc->migratepages); cc->nr_migratepages = 0; - cc->last_migrated_pfn = 0; nr_isolated = 0; } @@ -1539,6 +1529,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro enum compact_result ret; unsigned long start_pfn = zone->zone_start_pfn; unsigned long end_pfn = zone_end_pfn(zone); + unsigned long last_migrated_pfn; const bool sync = cc->mode != MIGRATE_ASYNC; cc->migratetype = gfpflags_to_migratetype(cc->gfp_mask); @@ -1584,7 +1575,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro cc->whole_zone = true; } - cc->last_migrated_pfn = 0; + last_migrated_pfn = 0; trace_mm_compaction_begin(start_pfn, cc->migrate_pfn, cc->free_pfn, end_pfn, sync); @@ -1593,12 +1584,14 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro while ((ret = compact_finished(zone, cc)) == COMPACT_CONTINUE) { int err; + unsigned long start_pfn = cc->migrate_pfn; switch (isolate_migratepages(zone, cc)) { case ISOLATE_ABORT: ret = COMPACT_CONTENDED; putback_movable_pages(&cc->migratepages); cc->nr_migratepages = 0; + last_migrated_pfn = 0; goto out; case ISOLATE_NONE: /* @@ -1608,6 +1601,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro */ goto check_drain; case ISOLATE_SUCCESS: + last_migrated_pfn = start_pfn; ; } @@ -1639,8 +1633,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro cc->migrate_pfn = block_end_pfn( cc->migrate_pfn - 1, cc->order); /* Draining pcplists is useless in this case */ - cc->last_migrated_pfn = 0; - + last_migrated_pfn = 0; } } @@ -1652,18 +1645,18 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro * compact_finished() can detect immediately if allocation * would succeed. */ - if (cc->order > 0 && cc->last_migrated_pfn) { + if (cc->order > 0 && last_migrated_pfn) { int cpu; unsigned long current_block_start = block_start_pfn(cc->migrate_pfn, cc->order); - if (cc->last_migrated_pfn < current_block_start) { + if (last_migrated_pfn < current_block_start) { cpu = get_cpu(); lru_add_drain_cpu(cpu); drain_local_pages(zone); put_cpu(); /* No more flushing until we migrate again */ - cc->last_migrated_pfn = 0; + last_migrated_pfn = 0; } } diff --git a/mm/internal.h b/mm/internal.h index 867af5425432..f40d06d70683 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -188,7 +188,6 @@ struct compact_control { unsigned int nr_migratepages; /* Number of pages to migrate */ unsigned long free_pfn; /* isolate_freepages search base */ unsigned long migrate_pfn; /* isolate_migratepages search base */ - unsigned long last_migrated_pfn;/* Not yet flushed page being freed */ struct zone *zone; unsigned long total_migrate_scanned; unsigned long total_free_scanned; From patchwork Fri Dec 14 23:03:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10731779 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C813A13AD for ; Fri, 14 Dec 2018 23:03:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7A5372D639 for ; Fri, 14 Dec 2018 23:03:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6E9BD2D63D; Fri, 14 Dec 2018 23:03:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AA9C22D639 for ; Fri, 14 Dec 2018 23:03:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6675C8E0218; Fri, 14 Dec 2018 18:03:14 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 640D68E0216; Fri, 14 Dec 2018 18:03:14 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 49A328E021E; Fri, 14 Dec 2018 18:03:14 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id CFD1D8E0216 for ; Fri, 14 Dec 2018 18:03:13 -0500 (EST) Received: by mail-ed1-f69.google.com with SMTP id w15so3435552edl.21 for ; Fri, 14 Dec 2018 15:03:13 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=LANCfU6JkeojD8p0hJc/ua7hJ3reHx8NxLuFWTot2E4=; b=cGrcNUwROPOXIMpKnneH/o7O312oblmPlR7rheBrvLqY3BaQ+GuE4VkaMbehQuN6zA F1z0ZxTPUgHpa7OGYWEyNjj8ABYNh9wsVJ8uIDWHcLKiWzXOd/eFDJqKApMhF4/ZbKoI 9Ve+g872SqPCdZ6UMUJozSTN26IfC31tuDGcXbjoan/iFjXCAyxcmZP82MmiRUJvjHte 6oQeVu8CyPAwsQ4Pg05V2SnVzzweYRCAn23ChJgCKcS/LR7Qn927zL5ekou5ZpX+al4n Ytj0Evgdh0pYs8p7hEeM63Z6F8QLGXd/FCd5EWuAqvfShNx59HCyyPJbqecHiBwQyY15 MBKg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWaiTfTKa/H48ibRAtCvAeGFcABFew/mNnU7Y93awR3NhbI1p9F/ zXAfPsjl2dCE23HBzX2wg1E4uQJcDUAghZvPqhd+hxM8LOq8OymzZIVpOdiOTD/156K58fZ5Pl0 VuuKsZLq1Rw8Z8JBzzEM0ndtD/q6jVJzbgXclMsjxYd0RLXoJXC6Kn2sASudyQF3WqA== X-Received: by 2002:a50:aed5:: with SMTP id f21mr4619075edd.120.1544828593327; Fri, 14 Dec 2018 15:03:13 -0800 (PST) X-Google-Smtp-Source: AFSGD/XBfRrZe4ngokEtQu2lKow2qEROLYdbEr4pMmauHJDeecmFGkMi0vCU3vFVzy6rrRWoj17g X-Received: by 2002:a50:aed5:: with SMTP id f21mr4619043edd.120.1544828592264; Fri, 14 Dec 2018 15:03:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544828592; cv=none; d=google.com; s=arc-20160816; b=H+LfHSdYBV9GdjfZ4mc/ueODpwM81d0VPsiyl1BYu9sMCPq7/KGZu4C6oodg8L5att NjXkxTyaEwW2iXyjtYEoEpUZzdUGKbC4A4bfWcn6YdB4WNKWMEVhVdLRsD1gxWVdtEMF 2wvGJ5EYMzd4rrkMNXEi3G0z4lHbVDNteaGQcKjeaCXiZWvsG0nCdU03p+SGCYhXbsz0 CeHRpxSzOqQ3mtUsam14UHhie2AnGpWl0o6wAwcJhE3wQyafx8a4E5ZDz6JWXUSo5Tax gQP7duml5dRjqj0F/j4eO5IHSfD5o8y2YxeQK0lF09ZmDE9Lh6AUwyRGH2tE9cOP1E0Q 47IA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=LANCfU6JkeojD8p0hJc/ua7hJ3reHx8NxLuFWTot2E4=; b=mAQX/ALj6+MGoRd+OHyzIRVCxE3PKQgFhKBd+a3RlmHSUTkGa3blqhZ+ofym1+fxtV lm+/GJhVgWrcTg+offnaY5QW7keJtyBy9VZ1s4/yTSCqSXBeq3wKZPanciM55JR84tti r7ZITGIw9w2yjXi8F+iEjC/cefyvC6uKfaibZvo0Wc2InkLwXGDjva2YyseJWpxi5Rp3 GbRWavn4GX6Rh/NbxhI4Vk3Qagx1x1J/hU4Ve25pKG/cI9t4i2tJHN5NG2PwdGKCYPjG sT0orW4bV6ReYAC2PWxWtkul8/SCetz7MXWn8MzvXRxtJvOQ9C9mCrEOQX9o8UaTy+9k 5N8w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp12.blacknight.com (outbound-smtp12.blacknight.com. [46.22.139.17]) by mx.google.com with ESMTPS id r4si2360597edh.133.2018.12.14.15.03.11 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 14 Dec 2018 15:03:12 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) client-ip=46.22.139.17; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail01.blacknight.ie [81.17.254.10]) by outbound-smtp12.blacknight.com (Postfix) with ESMTPS id C63C81C1D19 for ; Fri, 14 Dec 2018 23:03:11 +0000 (GMT) Received: (qmail 31225 invoked from network); 14 Dec 2018 23:03:11 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.245.71]) by 81.17.254.9 with ESMTPA; 14 Dec 2018 23:03:11 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Linus Torvalds , Michal Hocko , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 04/14] mm, compaction: Rename map_pages to split_map_pages Date: Fri, 14 Dec 2018 23:03:00 +0000 Message-Id: <20181214230310.572-5-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20181214230310.572-1-mgorman@techsingularity.net> References: <20181214230310.572-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP It's non-obvious that high-order free pages are split into order-0 pages from the function name. Fix it. Signed-off-by: Mel Gorman --- mm/compaction.c | 60 ++++++++++++++++++++++++++++----------------------------- 1 file changed, 29 insertions(+), 31 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index fb4d9f52ed56..3afa4e9188b6 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -66,7 +66,7 @@ static unsigned long release_freepages(struct list_head *freelist) return high_pfn; } -static void map_pages(struct list_head *list) +static void split_map_pages(struct list_head *list) { unsigned int i, order, nr_pages; struct page *page, *next; @@ -644,7 +644,7 @@ isolate_freepages_range(struct compact_control *cc, } /* __isolate_free_page() does not map the pages */ - map_pages(&freelist); + split_map_pages(&freelist); if (pfn < end_pfn) { /* Loop terminated early, cleanup. */ @@ -1141,7 +1141,7 @@ static void isolate_freepages(struct compact_control *cc) } /* __isolate_free_page() does not map the pages */ - map_pages(freelist); + split_map_pages(freelist); /* * Record where the free scanner will restart next time. Either we @@ -1300,8 +1300,7 @@ static inline bool is_via_compact_memory(int order) return order == -1; } -static enum compact_result __compact_finished(struct zone *zone, - struct compact_control *cc) +static enum compact_result __compact_finished(struct compact_control *cc) { unsigned int order; const int migratetype = cc->migratetype; @@ -1312,7 +1311,7 @@ static enum compact_result __compact_finished(struct zone *zone, /* Compaction run completes if the migrate and free scanner meet */ if (compact_scanners_met(cc)) { /* Let the next compaction start anew. */ - reset_cached_positions(zone); + reset_cached_positions(cc->zone); /* * Mark that the PG_migrate_skip information should be cleared @@ -1321,7 +1320,7 @@ static enum compact_result __compact_finished(struct zone *zone, * based on an allocation request. */ if (cc->direct_compaction) - zone->compact_blockskip_flush = true; + cc->zone->compact_blockskip_flush = true; if (cc->whole_zone) return COMPACT_COMPLETE; @@ -1345,7 +1344,7 @@ static enum compact_result __compact_finished(struct zone *zone, /* Direct compactor: Is a suitable page free? */ for (order = cc->order; order < MAX_ORDER; order++) { - struct free_area *area = &zone->free_area[order]; + struct free_area *area = &cc->zone->free_area[order]; bool can_steal; /* Job done if page is free of the right migratetype */ @@ -1391,13 +1390,12 @@ static enum compact_result __compact_finished(struct zone *zone, return COMPACT_NO_SUITABLE_PAGE; } -static enum compact_result compact_finished(struct zone *zone, - struct compact_control *cc) +static enum compact_result compact_finished(struct compact_control *cc) { int ret; - ret = __compact_finished(zone, cc); - trace_mm_compaction_finished(zone, cc->order, ret); + ret = __compact_finished(cc); + trace_mm_compaction_finished(cc->zone, cc->order, ret); if (ret == COMPACT_NO_SUITABLE_PAGE) ret = COMPACT_CONTINUE; @@ -1524,16 +1522,16 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, return false; } -static enum compact_result compact_zone(struct zone *zone, struct compact_control *cc) +static enum compact_result compact_zone(struct compact_control *cc) { enum compact_result ret; - unsigned long start_pfn = zone->zone_start_pfn; - unsigned long end_pfn = zone_end_pfn(zone); + unsigned long start_pfn = cc->zone->zone_start_pfn; + unsigned long end_pfn = zone_end_pfn(cc->zone); unsigned long last_migrated_pfn; const bool sync = cc->mode != MIGRATE_ASYNC; cc->migratetype = gfpflags_to_migratetype(cc->gfp_mask); - ret = compaction_suitable(zone, cc->order, cc->alloc_flags, + ret = compaction_suitable(cc->zone, cc->order, cc->alloc_flags, cc->classzone_idx); /* Compaction is likely to fail */ if (ret == COMPACT_SUCCESS || ret == COMPACT_SKIPPED) @@ -1546,8 +1544,8 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro * Clear pageblock skip if there were failures recently and compaction * is about to be retried after being deferred. */ - if (compaction_restarting(zone, cc->order)) - __reset_isolation_suitable(zone); + if (compaction_restarting(cc->zone, cc->order)) + __reset_isolation_suitable(cc->zone); /* * Setup to move all movable pages to the end of the zone. Used cached @@ -1559,16 +1557,16 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro cc->migrate_pfn = start_pfn; cc->free_pfn = pageblock_start_pfn(end_pfn - 1); } else { - cc->migrate_pfn = zone->compact_cached_migrate_pfn[sync]; - cc->free_pfn = zone->compact_cached_free_pfn; + cc->migrate_pfn = cc->zone->compact_cached_migrate_pfn[sync]; + cc->free_pfn = cc->zone->compact_cached_free_pfn; if (cc->free_pfn < start_pfn || cc->free_pfn >= end_pfn) { cc->free_pfn = pageblock_start_pfn(end_pfn - 1); - zone->compact_cached_free_pfn = cc->free_pfn; + cc->zone->compact_cached_free_pfn = cc->free_pfn; } if (cc->migrate_pfn < start_pfn || cc->migrate_pfn >= end_pfn) { cc->migrate_pfn = start_pfn; - zone->compact_cached_migrate_pfn[0] = cc->migrate_pfn; - zone->compact_cached_migrate_pfn[1] = cc->migrate_pfn; + cc->zone->compact_cached_migrate_pfn[0] = cc->migrate_pfn; + cc->zone->compact_cached_migrate_pfn[1] = cc->migrate_pfn; } if (cc->migrate_pfn == start_pfn) @@ -1582,11 +1580,11 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro migrate_prep_local(); - while ((ret = compact_finished(zone, cc)) == COMPACT_CONTINUE) { + while ((ret = compact_finished(cc)) == COMPACT_CONTINUE) { int err; unsigned long start_pfn = cc->migrate_pfn; - switch (isolate_migratepages(zone, cc)) { + switch (isolate_migratepages(cc->zone, cc)) { case ISOLATE_ABORT: ret = COMPACT_CONTENDED; putback_movable_pages(&cc->migratepages); @@ -1653,7 +1651,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro if (last_migrated_pfn < current_block_start) { cpu = get_cpu(); lru_add_drain_cpu(cpu); - drain_local_pages(zone); + drain_local_pages(cc->zone); put_cpu(); /* No more flushing until we migrate again */ last_migrated_pfn = 0; @@ -1678,8 +1676,8 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro * Only go back, not forward. The cached pfn might have been * already reset to zone end in compact_finished() */ - if (free_pfn > zone->compact_cached_free_pfn) - zone->compact_cached_free_pfn = free_pfn; + if (free_pfn > cc->zone->compact_cached_free_pfn) + cc->zone->compact_cached_free_pfn = free_pfn; } count_compact_events(COMPACTMIGRATE_SCANNED, cc->total_migrate_scanned); @@ -1716,7 +1714,7 @@ static enum compact_result compact_zone_order(struct zone *zone, int order, INIT_LIST_HEAD(&cc.freepages); INIT_LIST_HEAD(&cc.migratepages); - ret = compact_zone(zone, &cc); + ret = compact_zone(&cc); VM_BUG_ON(!list_empty(&cc.freepages)); VM_BUG_ON(!list_empty(&cc.migratepages)); @@ -1834,7 +1832,7 @@ static void compact_node(int nid) INIT_LIST_HEAD(&cc.freepages); INIT_LIST_HEAD(&cc.migratepages); - compact_zone(zone, &cc); + compact_zone(&cc); VM_BUG_ON(!list_empty(&cc.freepages)); VM_BUG_ON(!list_empty(&cc.migratepages)); @@ -1976,7 +1974,7 @@ static void kcompactd_do_work(pg_data_t *pgdat) if (kthread_should_stop()) return; - status = compact_zone(zone, &cc); + status = compact_zone(&cc); if (status == COMPACT_SUCCESS) { compaction_defer_reset(zone, cc.order, false); From patchwork Fri Dec 14 23:03:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10731781 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8E2BD1399 for ; Fri, 14 Dec 2018 23:03:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 41E9F2D639 for ; Fri, 14 Dec 2018 23:03:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 360502D63D; Fri, 14 Dec 2018 23:03:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C898B2D639 for ; Fri, 14 Dec 2018 23:03:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9DE0C8E021C; Fri, 14 Dec 2018 18:03:14 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9993F8E0216; Fri, 14 Dec 2018 18:03:14 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 701F88E021E; Fri, 14 Dec 2018 18:03:14 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 0E6888E021D for ; Fri, 14 Dec 2018 18:03:14 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id c18so3349587edt.23 for ; Fri, 14 Dec 2018 15:03:14 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=Dw0UV1TCQFA4d8gJKrA3/p7ohO2soOuPiIEMi/M3pF4=; b=c5jol0+sDgyZuOs/VfEPsPuO/Z2HCnx0EvDd9tMzBNyJ2ZprQ2OHOtyq9BUtCwb2HM xIQgkFi48m4kVAgbn8/1JAyPL796xKLM3D982iRxNqKTDAsZhomcA3FtE/g5Oir/jpLZ VNtKRgf0Wnu6JXPksRh8J5hdJITNubI8VcmaA6s8SrPcZsFvpBqwI1zcpU3Dm5Uh657m rTnmuZ2A4PTkEbY5u03JrX7AibeHLWxSCTGTY4H/Kxzr/NTchbRxaEHs8PI2IgEXjSFm IPmh5jr8uifRirI+FFoJ0imzt1Zu1TaVE9OxrikTCVJqaz5L1mztnZcSX1b9MBY9thLd 7xNw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWbfIrBSKpZX7N+ZHww2XNA9yh22h1mh3F0plCbDcqUWgwRiYEpV 3Ak4YpVxjZrXHCARUX6sY6tfq8m9v7avkTKjmAYK56jECucP3B8SMrt4Ov6Eou/lY5/zJM71HHz 4U4i+ijdy7mFN1FRvPp8eIZ0EeO2KtHrwUQ7GEXOgYWh3Cye9F6PYLNy/QvkSCzMHRA== X-Received: by 2002:aa7:d394:: with SMTP id x20mr4424999edq.193.1544828593563; Fri, 14 Dec 2018 15:03:13 -0800 (PST) X-Google-Smtp-Source: AFSGD/X0pbTJkjudu8UG7CHlb5eCafExrLKKsz92/kEqJxK0MSfkq6tllYmM1LFR0mYTEiILyysI X-Received: by 2002:aa7:d394:: with SMTP id x20mr4424966edq.193.1544828592462; Fri, 14 Dec 2018 15:03:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544828592; cv=none; d=google.com; s=arc-20160816; b=uiJXbmtzFNwzSvwPM7/olbZoS5P915pNxzoAQGQoBGNnYxtEG45oRt67uUGyWiSUR7 dqx5efaQ7z9tfU/2TKAG7onEXSh6dTTgxEis0hzaDnvosu/Y+DhDw3jvRNvlUWLICDnO XWaI2QggySwzm2uPvLAA51+wAd4UYklSRcpotQHr6Ryt+hppSSOCxDrN3JZSyuRDiJzY 2H/SFk0Kc/qXgO4TM5mwtki3dQNd5xRXE3DjUhLUE7buFLyzkMKVyA5lDsjvif1bP+Vd QFwBy0sS+IXR1uTxFpRb7z3MvewCRx6sr8E/cSeOqthkHL+TO24SVvtaRxtPeQrgnm7u FYBQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=Dw0UV1TCQFA4d8gJKrA3/p7ohO2soOuPiIEMi/M3pF4=; b=EpPH0xzYfEUxGAlpJOvYz87VsWvh0dVFsZn4n/EfXMpurG7/zy1pepDuFFuyhQt2M4 PdWFi0dGEavwKN31ttnUuoeFw4IrJ5fYTryw9ggJYf/uEo76jmuVMkqel70GYkvXpn4O /Jeu7Sin4QLZ30JXEBqrrTAoKay+FMFkp7MFfIfffeZZyzxhpJYeu35bzAh/fieVeZ9J 9uzbcm79L6r0br27tkvZaU/0OgdLURvXfA0H796MOm/z3ZIQIlQA458EqTr39ypvOZGA DPpldhVCbDk6OrWjha/K6qhiCQukNx8p12Ycb8mwl6DS4AuF79vA+zZcVRpV5zR4940E 6d2Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp02.blacknight.com (outbound-smtp02.blacknight.com. [81.17.249.8]) by mx.google.com with ESMTPS id x67si3305ede.100.2018.12.14.15.03.12 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 14 Dec 2018 15:03:12 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) client-ip=81.17.249.8; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail01.blacknight.ie [81.17.254.10]) by outbound-smtp02.blacknight.com (Postfix) with ESMTPS id 09AAE987AC for ; Fri, 14 Dec 2018 23:03:12 +0000 (UTC) Received: (qmail 31244 invoked from network); 14 Dec 2018 23:03:11 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.245.71]) by 81.17.254.9 with ESMTPA; 14 Dec 2018 23:03:11 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Linus Torvalds , Michal Hocko , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 05/14] mm, compaction: Skip pageblocks with reserved pages Date: Fri, 14 Dec 2018 23:03:01 +0000 Message-Id: <20181214230310.572-6-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20181214230310.572-1-mgorman@techsingularity.net> References: <20181214230310.572-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Reserved pages are set at boot time, tend to be clustered and almost never become unreserved. When isolating pages for migrating, skip the entire pageblock is one PageReserved page is encountered on the grounds that it is highly probable the entire pageblock is reserved. The impact depends on the machine and timing but both thpscale and thpfioscale when using MADV_HUGEPAGE show a reduction of scanning and fault latency on a 1-socket machine. The 2-socket results were too noisy to draw any meaningful conclusion but it's safe to assume less scanning is useful. 1-socket thpfioscale 4.20.0-rc6 4.20.0-rc6 mmotm-20181210 noreserved-v1r4 Amean fault-base-1 1481.32 ( 0.00%) 1443.63 ( 2.54%) Amean fault-huge-1 1118.17 ( 0.00%) 981.30 * 12.24%* Amean fault-both-1 1176.43 ( 0.00%) 1052.64 * 10.52%* Compaction migrate scanned 3860713 3294284 Compaction free scanned 613786341 433423502 Kcompactd migrate scanned 408711 291915 Kcompactd free scanned 242509759 217164988 Signed-off-by: Mel Gorman --- mm/compaction.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/mm/compaction.c b/mm/compaction.c index 3afa4e9188b6..8134dba47584 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -827,6 +827,13 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, goto isolate_success; } + /* + * A reserved page is never freed and tend to be + * clustered in the same pageblocks. Skip the block. + */ + if (PageReserved(page)) + low_pfn = end_pfn; + goto isolate_fail; } From patchwork Fri Dec 14 23:03:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10731783 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7012113AD for ; Fri, 14 Dec 2018 23:03:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 236F82D639 for ; Fri, 14 Dec 2018 23:03:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 175822D63D; Fri, 14 Dec 2018 23:03:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9FA3A2D639 for ; Fri, 14 Dec 2018 23:03:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CA2E88E0216; Fri, 14 Dec 2018 18:03:14 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BE75F8E021B; Fri, 14 Dec 2018 18:03:14 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 83D638E021F; Fri, 14 Dec 2018 18:03:14 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id 0CF008E021C for ; Fri, 14 Dec 2018 18:03:14 -0500 (EST) Received: by mail-ed1-f71.google.com with SMTP id f31so281587edf.17 for ; Fri, 14 Dec 2018 15:03:14 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=tHWZXRJo9uzt9m49/dCLzcy4vshcjBPRtaI+9Pk2x5Y=; b=J/4gHY7gw0vNq0FoRh2nWKqCbwyeoe1utvjNCzstlO2wDV5bfGcStgWnHIa4YrT4ZW /s5BFWEkGmjQVllKK+XZAJ98irrvxN7X99rCE/5CX/vTFTzZ6GWUh8ZA9BoUJ57axJyO GfJpvLeAQByuAA/O0uB7eD1pf0dXRHxBVnxsoXzRewoQ9S/UbZUUr5St3WJfh7VP/LM/ bQOlEYGp5FiYz5vYw+UAYD75mxBms5YUqiZ4vfh/dLeH6k+uTQcjrsKDbKGuSZ3z19Va pfp4CicCkj5QULUlKfex3F57k2tCCN4wbia4G5u29Fyi6ony8aiJTbRxfwAJkIFpM493 kpCw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWaQFqmm2w9Z5wgc1FC6nkvuqrataqCST1Vv6M2u8IVAAOTdzvEn x7ckpyAmMwrQIkjGI86krVvggGMc7tgEKGsv7T4gePC0SqG7ZIvsuQRmxcAKe+iaNrL5Eo6iqLK 8WzWcva5oVWdsdQvbaogAB7/1amHwvHIYMy8y8fObiBSqEMVAocXyTGMvCDxcY141lA== X-Received: by 2002:a17:906:77c1:: with SMTP id m1-v6mr3549408ejn.233.1544828593540; Fri, 14 Dec 2018 15:03:13 -0800 (PST) X-Google-Smtp-Source: AFSGD/XiZiUaD9t14W82uRoW6UGNFxx9lMJSe0yPdS0ER7EU7T8/cNZpFTe9OvjgikLPSIahk0jx X-Received: by 2002:a17:906:77c1:: with SMTP id m1-v6mr3549386ejn.233.1544828592613; Fri, 14 Dec 2018 15:03:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544828592; cv=none; d=google.com; s=arc-20160816; b=lCoq2Ksg8WcfRoxsN79Kp9Mjlmu9w7FX+BRMJsMwXmyhTts+HI5XGB5BB7Qk4rr7Ub 3NowWBRul5elu3m5FIcXJBGo6BDH80i5fVUHPM5t/EeN4hvvREJuuhfI6qweSVF54WHw BPzx9d1Su43OSGr4s9bq0X0uy05v2BijIM8dCIZTaXy+dtyg+GiEYUQSIN+WRgkOVXxy rztzW+SbVn6FGPyBI4qwAac+VFyvRm+q2fH+PwK7cqZY/OQDC8uN1kkfQPmegsuC9Gaw SKMm8dWTsXlrpXjkLDzFBXjyuBKY4tNN4m855SAdB7LC1/QFna9EbCXOAQGLiAa4X75E V0hA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=tHWZXRJo9uzt9m49/dCLzcy4vshcjBPRtaI+9Pk2x5Y=; b=JsH3v/3yqV7WQ6QpBQ6zJi3nXegPIFIHfH5YH7mERNbZ2O3Bylm/sVMMxZ7+G8DrMp IZhoxWV9c3ueE8dB5KvMKLjKKBBoht9dpUr7LUV8HwnENkI+QMvRfUBpCiC4Qd+cJQ2v 4Do1E+NeLLYgy26mzSunmLLSfCSQS7ARwQvbkJHnBGtGOOfoCNu+b6dhikIFCKqZOOQM jR7mSmUsu5wyqX27xEHD+gtQ3Y0Dix6UlYT/MdbMS1hmbrzhBn/zX1Gb/lP88JFGufko c4iDN9hIn9vmCfd6+0xNlHeFcd7vmwWIMQc2taOYFrES+5XVp8vnOmauUf8xZ8qobBNq cnYw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp13.blacknight.com (outbound-smtp13.blacknight.com. [46.22.139.230]) by mx.google.com with ESMTPS id k13si643824edl.377.2018.12.14.15.03.12 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 14 Dec 2018 15:03:12 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) client-ip=46.22.139.230; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (unknown [81.17.254.10]) by outbound-smtp13.blacknight.com (Postfix) with ESMTPS id 4AFED1C1D19 for ; Fri, 14 Dec 2018 23:03:12 +0000 (GMT) Received: (qmail 31262 invoked from network); 14 Dec 2018 23:03:12 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.245.71]) by 81.17.254.9 with ESMTPA; 14 Dec 2018 23:03:12 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Linus Torvalds , Michal Hocko , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 06/14] mm, migrate: Immediately fail migration of a page with no migration handler Date: Fri, 14 Dec 2018 23:03:02 +0000 Message-Id: <20181214230310.572-7-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20181214230310.572-1-mgorman@techsingularity.net> References: <20181214230310.572-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Pages with no migration handler use a fallback hander which sometimes works and sometimes persistently fails such as blockdev pages. Migration will retry a number of times on these persistent pages which is wasteful during compaction. This patch will fail migration immediately unless the caller is in MIGRATE_SYNC mode which indicates the caller is willing to wait while being persistent. This is not expected to help THP allocation success rates but it does reduce latencies slightly. 1-socket thpfioscale 4.20.0-rc6 4.20.0-rc6 noreserved-v1r4 failfast-v1r4 Amean fault-both-1 0.00 ( 0.00%) 0.00 * 0.00%* Amean fault-both-3 2276.15 ( 0.00%) 3867.54 * -69.92%* Amean fault-both-5 4992.20 ( 0.00%) 5313.20 ( -6.43%) Amean fault-both-7 7373.30 ( 0.00%) 7039.11 ( 4.53%) Amean fault-both-12 11911.52 ( 0.00%) 11328.29 ( 4.90%) Amean fault-both-18 17209.42 ( 0.00%) 16455.34 ( 4.38%) Amean fault-both-24 20943.71 ( 0.00%) 20448.94 ( 2.36%) Amean fault-both-30 22703.00 ( 0.00%) 21655.07 ( 4.62%) Amean fault-both-32 22461.41 ( 0.00%) 21415.35 ( 4.66%) The 2-socket results are not materially different. Scan rates are similar as expected. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/migrate.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/migrate.c b/mm/migrate.c index df17a710e2c7..0e27a10429e2 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -885,7 +885,7 @@ static int fallback_migrate_page(struct address_space *mapping, */ if (page_has_private(page) && !try_to_release_page(page, GFP_KERNEL)) - return -EAGAIN; + return mode == MIGRATE_SYNC ? -EAGAIN : -EBUSY; return migrate_page(mapping, newpage, page, mode); } From patchwork Fri Dec 14 23:03:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10731785 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 611E41399 for ; Fri, 14 Dec 2018 23:03:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 13C772D639 for ; Fri, 14 Dec 2018 23:03:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0811E2D63D; Fri, 14 Dec 2018 23:03:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 88C9B2D639 for ; Fri, 14 Dec 2018 23:03:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 02AF88E021B; Fri, 14 Dec 2018 18:03:15 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E0B018E021F; Fri, 14 Dec 2018 18:03:14 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A0E528E021E; Fri, 14 Dec 2018 18:03:14 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 4425A8E021B for ; Fri, 14 Dec 2018 18:03:14 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id l45so3463112edb.1 for ; Fri, 14 Dec 2018 15:03:14 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=W8Lfqfpa4DlGkxutCqLVK1CDgi6G7m2LALKtQM047S4=; b=m8nlNIjIt7C+scohbZkzlxrxMJuF4QOHYR8jo8zntGC4kOmMQyxoId/B59PR70SPoV VCOZaBYh5HNNrNDNFTqPIylJ4P8+i8j9owQWGolvLGknF27+gFosCud/so8hUF9PLEiL It14PZoAlJ/2+frjjeqw7mQPbMODKfzarkUf0o9OlfAhntQm13j4KTu3IGKk46xXyE1S RwhUaKT/sCR2HJ8g+WsT0A98XLaeJujS/fEbY1CxsoqS6KC3zED5vVbdujda03TO7ZqC 7miewi+iLUfui21H5RS26FtC8BevkSAwzA0HEOunX8SgpqQMBxSWdgfY8MJOt+vtTM0C uEZw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWayFcgS/BeK1CwoiTx0NPVxkL1e9M4iyDugi02/2Lc04bS3qbtE 8Hfz60SMnmJXBcGJ/EhIBaZFo72pmj3DTTO64hS8GQEtp5WU9gsh99KhgU1QY7xZFXDFpUjhJKv bwCecYEPH8fECT25VznMsmDlpBngHSzlXsyZ9m+E9WFMQ/XZPJ7USOrQMTG0VHlwN1Q== X-Received: by 2002:a50:b721:: with SMTP id g30mr4404911ede.155.1544828593773; Fri, 14 Dec 2018 15:03:13 -0800 (PST) X-Google-Smtp-Source: AFSGD/Xe5fm1ZEap2k8/b17wBTufGltuq3mar9sE+KeuGPxWo+uwEnx76JdLAN3SC8Sq92Wk76vD X-Received: by 2002:a50:b721:: with SMTP id g30mr4404891ede.155.1544828592839; Fri, 14 Dec 2018 15:03:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544828592; cv=none; d=google.com; s=arc-20160816; b=F3kNi4s7oOzp8M66zCvLFj/WMBGcqTLwqCVh6KZmbDhHhENYEgHEv7nNH8VDtYYKSW ZHFVRHxfkrjbDn+6aEoNpXwGjvN475FN+iSuyLtSnh8mlHoHyS8nKMYMevoUs9DQen0q O1vbMwHv8rsZjGLOJUelLhSurCjwwMRYumhTQ7/FlPc+QFydmynkdYc863vym4xe6Hs2 OpOQZbRexxxC/pCDrpioMR0D6bY0LWQzbSye9wxi7UpwGefQ2wTEi/9DwptAt8bDJi3i NKrKgvTLPEcM3rMKg+o79qMwKVFq2BXIL4Z8BIGjf0BZvgXsVMUSy+5/VrzTUP7tjm82 v/Jg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=W8Lfqfpa4DlGkxutCqLVK1CDgi6G7m2LALKtQM047S4=; b=rHu/0vlX75OwGBb+TjtwqiJdiLZSsUITvYO8klciBaWKcnldGHEKMjZDlhZHcHpyMj 5LvUlQLCtdHQn6lxiE3Vi+31wEBakCsOsbJ1ro7oFvsoOBkDCm16AoDABCwR/qIR7y2b nodj+1tmQTB+uCMSUt4Vmh8fUN/abpVCReRWBRdcWfxJmRYWu5qa94iMZSuBWqeezVwt wi9gYpag9JKGTuACTp/ijZrczJwApll5JoytNrxbL8ioGvfS3n0JIpN8LXRYg4GaucmC 0s6dP/n6+QembDG53S3ElTUJjMp7hCTqXEOSsmvF6mTQOPwIyFGzRbjXCR12r0/p5oJ3 AzMA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp08.blacknight.com (outbound-smtp08.blacknight.com. [46.22.139.13]) by mx.google.com with ESMTPS id e18si316800eds.58.2018.12.14.15.03.12 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 14 Dec 2018 15:03:12 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) client-ip=46.22.139.13; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail01.blacknight.ie [81.17.254.10]) by outbound-smtp08.blacknight.com (Postfix) with ESMTPS id 82EC91C208A for ; Fri, 14 Dec 2018 23:03:12 +0000 (GMT) Received: (qmail 31280 invoked from network); 14 Dec 2018 23:03:12 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.245.71]) by 81.17.254.9 with ESMTPA; 14 Dec 2018 23:03:12 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Linus Torvalds , Michal Hocko , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 07/14] mm, compaction: Always finish scanning of a full pageblock Date: Fri, 14 Dec 2018 23:03:03 +0000 Message-Id: <20181214230310.572-8-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20181214230310.572-1-mgorman@techsingularity.net> References: <20181214230310.572-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When compaction is finishing, it uses a flag to ensure the pageblock is complete. However, in general it makes sense to always complete migration of a pageblock. Minimally, skip information is based on a pageblock and partially scanned pageblocks may incur more scanning in the future. The pageblock skip handling also becomes more strict later in the series and the hint is more useful if a complete pageblock was always scanned. The impact here is potentially on latencies as more scanning is done but it's not a consistent win or loss as the scanning is not always a high percentage of the pageblock and sometimes it is offset by future reductions in scanning. Hence, the results are not presented this time as it's a mix of gains/losses without any clear pattern. However, completing scanning of the pageblock is important for later patches. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 19 ++++++++----------- mm/internal.h | 1 - 2 files changed, 8 insertions(+), 12 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 8134dba47584..4f51435c645a 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1338,16 +1338,14 @@ static enum compact_result __compact_finished(struct compact_control *cc) if (is_via_compact_memory(cc->order)) return COMPACT_CONTINUE; - if (cc->finishing_block) { - /* - * We have finished the pageblock, but better check again that - * we really succeeded. - */ - if (IS_ALIGNED(cc->migrate_pfn, pageblock_nr_pages)) - cc->finishing_block = false; - else - return COMPACT_CONTINUE; - } + /* + * Always finish scanning a pageblock to reduce the possibility of + * fallbacks in the future. This is particularly important when + * migration source is unmovable/reclaimable but it's not worth + * special casing. + */ + if (!IS_ALIGNED(cc->migrate_pfn, pageblock_nr_pages)) + return COMPACT_CONTINUE; /* Direct compactor: Is a suitable page free? */ for (order = cc->order; order < MAX_ORDER; order++) { @@ -1389,7 +1387,6 @@ static enum compact_result __compact_finished(struct compact_control *cc) return COMPACT_SUCCESS; } - cc->finishing_block = true; return COMPACT_CONTINUE; } } diff --git a/mm/internal.h b/mm/internal.h index f40d06d70683..9b32f4cab0ae 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -203,7 +203,6 @@ struct compact_control { bool direct_compaction; /* False from kcompactd or /proc/... */ bool whole_zone; /* Whole zone should/has been scanned */ bool contended; /* Signal lock or sched contention */ - bool finishing_block; /* Finishing current pageblock */ }; unsigned long From patchwork Fri Dec 14 23:03:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10731787 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C915F1399 for ; Fri, 14 Dec 2018 23:03:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6D46528767 for ; Fri, 14 Dec 2018 23:03:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5D6472878F; Fri, 14 Dec 2018 23:03:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E424428767 for ; Fri, 14 Dec 2018 23:03:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3289F8E021E; Fri, 14 Dec 2018 18:03:15 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0E9A48E021D; Fri, 14 Dec 2018 18:03:15 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ED1228E021E; Fri, 14 Dec 2018 18:03:14 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 763588E021D for ; Fri, 14 Dec 2018 18:03:14 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id x15so3461414edd.2 for ; Fri, 14 Dec 2018 15:03:14 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=DimBv5q7GksENALkOeOYb6i7ChULbIUvx43bviqjEa8=; b=raMEnosvfw/tRO7le3zIrgiegReKcpMf15thou0CLcDfrRYl5OLzYWJdPmMxNixUeq 0UtxLg+Pmyem33LOkctL596jChv7odFO3LV9plJ39cbvzmm4cF9FTmiMeA3mExILtn18 hayawnypdfoLCPspld3cT//Qgoc6aZOEdjvG0fthJtepNjavHoT57xWaSxdnvB+DT0Or Dd7HYNta1qf1g9jWvrRbMx33FEO+laAAk+HSeeRvXv++QoyE2ws0LAuCEMbVLBDfLmTh gmxUbV9WGgOvWnQ8M/pgkt35+GQoAq6GaypJujCC2wzS7JQ8FEiQ6wJtaAe1NU54z7a8 NUUg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWZPsniXo4ENi6TbrPP6zVTaV2eao9s2O4igT5190hvC8PvSiFuU JX49SzA37iWZzCjmaLVZ6n3VQgbk0IxGXX1eG3s4/d0xAVyj34KBs7A5zDyIYPykmlTcC+NAg7X rAiZJGfrFKaObdl8iy/DJXICKJYs1T6KXmnh6oOkP4SsNctsSa/kwwGwldDZtj+tJFA== X-Received: by 2002:a50:8bb5:: with SMTP id m50mr4362790edm.211.1544828593981; Fri, 14 Dec 2018 15:03:13 -0800 (PST) X-Google-Smtp-Source: AFSGD/XoKkBhDRNHIyEXXB/5ltMsb+0xbKsAQPKVHQE9Ka/eai1n+uBjl4I/POPDWiVOTaoODigT X-Received: by 2002:a50:8bb5:: with SMTP id m50mr4362767edm.211.1544828593097; Fri, 14 Dec 2018 15:03:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544828593; cv=none; d=google.com; s=arc-20160816; b=Ax8UanoBEz9+LpF7xsiX3c9QMhYFoZbUdx98SlnMrY43vTrzPtARfLz8cmmEWtqnT3 wO0rDcgU+of8GuZwUzqiPI78hDF6jXynyJ6T2O6qdtFevSV5miOOR/pPWL6bnQlLsNZa q5kgGGTECnMX+kiFxRKgqsoqJFjBl74zdy9AdOxQR9ZirHij3k3iaAZpCKX54MIoM87t TFI7kMKMrq0xgveA5XSVLe+aX6z79JmJYuUdifEp6deOxChrqZxQm3UqVZXU5RMbYsYf lEBsVVTsNFz96WYAfR7JUmWxD8orqEtEcRGw08kTAAi41jlgRHpb7+7IfJE5qXNp84c9 K1iA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=DimBv5q7GksENALkOeOYb6i7ChULbIUvx43bviqjEa8=; b=oS/oZPxVdMQcKAW0pXkjpyHvMtxdWszsjw16W8MJUKMoTpIiPN6Xou2wTxSw+efPT/ UU3FIDW5fTka4auUY7f5sTR+ouuS1W2wvGCvuAfAWMmFeJaTD684KO+2tRy+47wCEPnj 1F1lhQu0Dx949MhQOatA6XV3cwMQAypBj0vWoUpmr9pgQelDf/yLHKJbJ8wreOOSDkT+ QtKuUMuCQHqU8m7ICO0qNYfW1KKqpyvR/CpFVHE9gerfdf0GykjJY/tXfHBZsDDAPc4/ 44VeEfDOZAmxexOKh97lLslp7SvXozc/TSgRcJjN0PkRrADzQyEpefVCvRXWvUm9BCHR O0Hw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp16.blacknight.com (outbound-smtp16.blacknight.com. [46.22.139.233]) by mx.google.com with ESMTPS id 89si2196684edr.235.2018.12.14.15.03.12 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 14 Dec 2018 15:03:13 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) client-ip=46.22.139.233; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail01.blacknight.ie [81.17.254.10]) by outbound-smtp16.blacknight.com (Postfix) with ESMTPS id BF8111C1D19 for ; Fri, 14 Dec 2018 23:03:12 +0000 (GMT) Received: (qmail 31294 invoked from network); 14 Dec 2018 23:03:12 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.245.71]) by 81.17.254.9 with ESMTPA; 14 Dec 2018 23:03:12 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Linus Torvalds , Michal Hocko , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 08/14] mm, compaction: Use the page allocator bulk-free helper for lists of pages Date: Fri, 14 Dec 2018 23:03:04 +0000 Message-Id: <20181214230310.572-9-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20181214230310.572-1-mgorman@techsingularity.net> References: <20181214230310.572-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP release_pages() is a simpler version of free_unref_page_list() but it tracks the highest PFN for caching the restart point of the compaction free scanner. This patch optionally tracks the highest PFN in the core helper and converts compaction to use it. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- include/linux/gfp.h | 7 ++++++- mm/compaction.c | 12 +++--------- mm/page_alloc.c | 10 +++++++++- 3 files changed, 18 insertions(+), 11 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 0705164f928c..ed9a95b63374 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -543,7 +543,12 @@ void * __meminit alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask); extern void __free_pages(struct page *page, unsigned int order); extern void free_pages(unsigned long addr, unsigned int order); extern void free_unref_page(struct page *page); -extern void free_unref_page_list(struct list_head *list); +extern void __free_page_list(struct list_head *list, bool dropref, unsigned long *highest_pfn); + +static inline void free_unref_page_list(struct list_head *list) +{ + return __free_page_list(list, false, NULL); +} struct page_frag_cache; extern void __page_frag_cache_drain(struct page *page, unsigned int count); diff --git a/mm/compaction.c b/mm/compaction.c index 4f51435c645a..8ba9b3b479e3 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -52,16 +52,10 @@ static inline void count_compact_events(enum vm_event_item item, long delta) static unsigned long release_freepages(struct list_head *freelist) { - struct page *page, *next; - unsigned long high_pfn = 0; + unsigned long high_pfn; - list_for_each_entry_safe(page, next, freelist, lru) { - unsigned long pfn = page_to_pfn(page); - list_del(&page->lru); - __free_page(page); - if (pfn > high_pfn) - high_pfn = pfn; - } + __free_page_list(freelist, true, &high_pfn); + INIT_LIST_HEAD(freelist); return high_pfn; } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a6e7bfd18cde..80535cd55a92 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2961,18 +2961,26 @@ void free_unref_page(struct page *page) /* * Free a list of 0-order pages */ -void free_unref_page_list(struct list_head *list) +void __free_page_list(struct list_head *list, bool dropref, + unsigned long *highest_pfn) { struct page *page, *next; unsigned long flags, pfn; int batch_count = 0; + if (highest_pfn) + *highest_pfn = 0; + /* Prepare pages for freeing */ list_for_each_entry_safe(page, next, list, lru) { + if (dropref) + WARN_ON_ONCE(!put_page_testzero(page)); pfn = page_to_pfn(page); if (!free_unref_page_prepare(page, pfn)) list_del(&page->lru); set_page_private(page, pfn); + if (highest_pfn && pfn > *highest_pfn) + *highest_pfn = pfn; } local_irq_save(flags); From patchwork Fri Dec 14 23:03:05 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10731789 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 38EE11399 for ; Fri, 14 Dec 2018 23:03:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E06A228767 for ; Fri, 14 Dec 2018 23:03:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D34912878F; Fri, 14 Dec 2018 23:03:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6B7FB28767 for ; Fri, 14 Dec 2018 23:03:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 62A528E021F; Fri, 14 Dec 2018 18:03:15 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 506CF8E0220; Fri, 14 Dec 2018 18:03:15 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3CF318E021F; Fri, 14 Dec 2018 18:03:15 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id C19308E0220 for ; Fri, 14 Dec 2018 18:03:14 -0500 (EST) Received: by mail-ed1-f69.google.com with SMTP id o21so3462957edq.4 for ; Fri, 14 Dec 2018 15:03:14 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=PBG8hjESQWWVTGMrklYE9aIZO34XECn3ThLlLFNY6h0=; b=KbLzSlB4XLaQua7d3wdXO474sGNX+jNSgkJHILbMKixK5/RayGkBTp+G9/1qc7e2eQ 2HxFOHHIvNv2oBK5qD+51aCHlmdJIwRnFRWsMW1xHxIPy3yqoRprhzNfbvZQW8s2Noi5 TX1r6JP1KSnQz5P75PqVUL2McgaOTno0ff4DYEgU/dOrRYsXp51dR+VZ2GQOExsazNCh 1qVCtCLc5JzwRb+0A+8PWSxi+/NWseYrbtPdz3Ar2GofcytaRQsBVlvv2bQ/q9O3hQV/ 3epNz8K2mRx7mFTJRo/EgcZB/oSv/SkxLQq7lz/258AZp8yjD3fUh3ikCYE+A8S/fZMt /1MA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWbQeZNYAZXOuvY/Mg9u4GCoDhuUvH4hRUASZAXOx73rwXq07L/u 5blodaHEaItmsm/PfBgsrDuuk1atLYWGjy+Q/A9P6TARuh9xKIFLbrRLjTcOThrLxK0n/jb3xtu 4uZvotratf3YYjVaar8x7TF3VvyhHpSUTxS0LLJ3F/th6OJaF3yt/4J59vwB+Qc/3GQ== X-Received: by 2002:a17:906:c288:: with SMTP id r8-v6mr3638633ejz.9.1544828594281; Fri, 14 Dec 2018 15:03:14 -0800 (PST) X-Google-Smtp-Source: AFSGD/WwOg8/zvIBU7Kr1zDzUK5Z/Y3vyQI+CuzH2T+6dY3ez82Iplilnaj6OzML/Lu6AFfTrXFn X-Received: by 2002:a17:906:c288:: with SMTP id r8-v6mr3638614ejz.9.1544828593345; Fri, 14 Dec 2018 15:03:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544828593; cv=none; d=google.com; s=arc-20160816; b=tm26IrNNSPzR1Kmn/9iGd2coku5widXydelO6TZuqN3Ds9xmLKbHwmJHuYtHAwIG5R O92W2z4g8bSxBbMQqzugSqw796qg/KU9VaBlzhhp3h218qUgdcsEFq5A4SnBwyo1fn1L sxjJGWpdb0B65ffv54UDRlHS2aFk0fYb+krBAZX7z4/cJWpQ8DnmiIsGcTAC6mbzKobG 8un7SLat5mD2sYOUdNLdAC3tb6IxohkTSWMnYmJcGIKz2FACV1wpP9A41AjVvp00XRBB dmr/Tb6mk5DD7DnasAFs5Lr3oc2AaACf9EpHd7UtomLAR446gnox/CHHoh2Hk7V2jajx sdew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=PBG8hjESQWWVTGMrklYE9aIZO34XECn3ThLlLFNY6h0=; b=MOmgZxWo1CO5KcGuhgTKMgl94IuOxxPiLSC9p47oMSaGl+Oj5afrvk7RXWaznTVdTc v2njdI+zcYYqRH/4pE5h5xEOzPUN3/JYND4Tc60TT2ltClbHuTvJk5Pu/70/MMGPj7ji sGH4DTj7rK+v1mzacrbo5lj3GcfQ1PkwnhoyitsqXySh/S2iMDBpprBaNCgUax+Hz67I uL61PDaNlOINvm5nIpSDXIQQ4ZROCZX09c4/vTvr8Qp1AVoZHwwox2hl4Pk5az7N4lI4 5ZLYUJzpgsFHPxipW7AUoRYBVsGp6ufPtrwbzrZzTNM+qYv2g8+NXdUzvvGOfKhAW8FA u/qA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp12.blacknight.com (outbound-smtp12.blacknight.com. [46.22.139.17]) by mx.google.com with ESMTPS id p21-v6si2083183ejx.67.2018.12.14.15.03.13 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 14 Dec 2018 15:03:13 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) client-ip=46.22.139.17; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail01.blacknight.ie [81.17.254.10]) by outbound-smtp12.blacknight.com (Postfix) with ESMTPS id 0911D1C1D97 for ; Fri, 14 Dec 2018 23:03:13 +0000 (GMT) Received: (qmail 31313 invoked from network); 14 Dec 2018 23:03:12 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.245.71]) by 81.17.254.9 with ESMTPA; 14 Dec 2018 23:03:12 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Linus Torvalds , Michal Hocko , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 09/14] mm, compaction: Ignore the fragmentation avoidance boost for isolation and compaction Date: Fri, 14 Dec 2018 23:03:05 +0000 Message-Id: <20181214230310.572-10-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20181214230310.572-1-mgorman@techsingularity.net> References: <20181214230310.572-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When pageblocks get fragmented, watermarks are artifically boosted to pages are reclaimed to avoid further fragmentation events. However, compaction is often either fragmentation-neutral or moving movable pages away from unmovable/reclaimable pages. As the actual watermarks are preserved, allow compaction to ignore the boost factor. 1-socket thpscale 4.20.0-rc6 4.20.0-rc6 finishscan-v1r4 noboost-v1r4 Amean fault-both-1 0.00 ( 0.00%) 0.00 * 0.00%* Amean fault-both-3 3849.90 ( 0.00%) 3753.53 ( 2.50%) Amean fault-both-5 5054.13 ( 0.00%) 5396.32 ( -6.77%) Amean fault-both-7 7061.77 ( 0.00%) 7393.46 ( -4.70%) Amean fault-both-12 11560.59 ( 0.00%) 12155.50 ( -5.15%) Amean fault-both-18 16120.15 ( 0.00%) 16445.96 ( -2.02%) Amean fault-both-24 19804.31 ( 0.00%) 20465.03 ( -3.34%) Amean fault-both-30 25018.73 ( 0.00%) 20813.54 * 16.81%* Amean fault-both-32 24380.19 ( 0.00%) 22384.02 ( 8.19%) The impact on the scan rates is a mixed bag because this patch is very sensitive to timing and whether the boost was active or not. However, detailed tracing indicated that failure of migration due to a premature ENOMEM triggered by watermark checks were eliminated. Signed-off-by: Mel Gorman --- mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 80535cd55a92..c7b80e62bfd9 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3043,7 +3043,7 @@ int __isolate_free_page(struct page *page, unsigned int order) * watermark, because we already know our high-order page * exists. */ - watermark = min_wmark_pages(zone) + (1UL << order); + watermark = zone->_watermark[WMARK_MIN] + (1UL << order); if (!zone_watermark_ok(zone, 0, watermark, 0, ALLOC_CMA)) return 0; From patchwork Fri Dec 14 23:04:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10731791 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A050A746 for ; Fri, 14 Dec 2018 23:04:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4BB652D595 for ; Fri, 14 Dec 2018 23:04:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3C3A52D639; Fri, 14 Dec 2018 23:04:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8BA0B2D595 for ; Fri, 14 Dec 2018 23:04:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9725B8E0220; Fri, 14 Dec 2018 18:04:56 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 923558E021D; Fri, 14 Dec 2018 18:04:56 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8122D8E0220; Fri, 14 Dec 2018 18:04:56 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by kanga.kvack.org (Postfix) with ESMTP id 29F748E021D for ; Fri, 14 Dec 2018 18:04:56 -0500 (EST) Received: by mail-ed1-f70.google.com with SMTP id c18so3351131edt.23 for ; Fri, 14 Dec 2018 15:04:56 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:date:from:to :cc:subject:message-id:references:mime-version:content-disposition :in-reply-to:user-agent; bh=rFUIgtfuwGneSdnRCoG2SM+ftmHlORctQAz5O6b5lNs=; b=N6bzgY6YcmjVtU8lKl1GDDDlaJwgzWar0ivoHccKSb8XLoKStnrUpVGQYuAJWbnJx7 He6apgoneJg/B1QGqVdNtr4JERYq9pQ3Ajc+PyVw3827e85ZHdo1MVaWVHkRmmhxpn/g PYXo2iuS56VIOjrOBQwtTl4xzoTkIqMns5beWaTUnrt6MGEGHP7uNLIqSMrtRKq5JFEz V9/jn38qKJsDQ8XSih7Jof8pcs1pJUWZQ1nb8h+fj2H9WrkzCIL8DZ2NwdMELvLfEZF8 ckx4uP9Bj0bXN95JBh+YOJxzDaJoQsqylCfTnOLG2+CCWsmyNHfU8IbtyJo3FrYyJzCu MkjQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWbqCjkbOaTQQEIU6qo2O9nZfXHcW7T3TS7SfjGRoJOkFqJaDexs Ekk32ZJew5CcfDOOmMw23SZtNm0pBLcuZoMRRLeeDiwuF6pYsGaPj0LxGCXKn5ieTRCl85Fvfkj SVyCLvKxCZ6Iju7LdPNR3TnqkZUpF9hE156UaZKemn3hs2rcLe/4lJ5AzE0o5U8ZCSQ== X-Received: by 2002:aa7:d416:: with SMTP id z22mr4442435edq.3.1544828695634; Fri, 14 Dec 2018 15:04:55 -0800 (PST) X-Google-Smtp-Source: AFSGD/UiqkH43eU1JQS4JEHzMWOZEyxIaLI2o5l40MO44yBcOP8iKBGPofB7BnnAjl4sK6GKZHea X-Received: by 2002:aa7:d416:: with SMTP id z22mr4442403edq.3.1544828694244; Fri, 14 Dec 2018 15:04:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544828694; cv=none; d=google.com; s=arc-20160816; b=oyym0VYhDGQ0iaUANGiPMNtxJxyIgGZBK8TO+30dYvB21j6gV75p3uk/K95L/w6uLY q05m/4O8/yDihpb6/n95RGOWUircJ/+jWgWEoB18490mW7TMIrs14nusRDbbiN8Yi8Cp K0psh3lIMcSxK8UN1c5sL9VqF6AElZt0aXJtY3MDqCuzgiTUaQNNvBz3MZZGN0Q7jZEC 4xtp0eY/VjGjxlD4173Jqv2ttFP15zwMkiTf68EJSimGPmE4w9R8yDWMk+2n/5Rp6jkd 11FbW22oPwAxtR28XltGijPXk52Q5DMTkPIZOUoh/Bh7nozJAFv61hLkL15b41SUa5eg UA4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=user-agent:in-reply-to:content-disposition:mime-version:references :message-id:subject:cc:to:from:date; bh=rFUIgtfuwGneSdnRCoG2SM+ftmHlORctQAz5O6b5lNs=; b=D4q07O5No2y3x9sWGhrOzaynaFxSVh5/H2EIqz9IgcSZEdTQuyK/j4NhGRDHoYWoMN hytkbzhODjaxCxac2Fc3jKpjiT4gZ81hisf2H+mSlvKAsHZmJ5+sKnDOdmJq4qhluz2W n3eIjg+oNAceC5eAfjRjvq+aOw0TPG6IFeAAzojnpiDMESqBKXWJiuJZKJHbgoJhezWn z1HVU6G2psASoo97GRSwLqYYB/AH/XqXqcLEU5xiaktywnlqCRxIDLQC78lunTG60QOt rc3mQVl+H4riFKbbjBD8TVEjA7SXKHfRBLe6olfdg8sSoYCWs1JjrEV7AvgbAmIegwJy u9HA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp04.blacknight.com (outbound-smtp04.blacknight.com. [81.17.249.35]) by mx.google.com with ESMTPS id i3si1022936edk.411.2018.12.14.15.04.54 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 14 Dec 2018 15:04:54 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) client-ip=81.17.249.35; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail04.blacknight.ie [81.17.254.17]) by outbound-smtp04.blacknight.com (Postfix) with ESMTPS id E9E3398473 for ; Fri, 14 Dec 2018 23:04:53 +0000 (UTC) Received: (qmail 32511 invoked from network); 14 Dec 2018 23:04:53 -0000 Received: from unknown (HELO techsingularity.net) (mgorman@techsingularity.net@[37.228.245.71]) by 81.17.254.9 with ESMTPSA (AES256-SHA encrypted, authenticated); 14 Dec 2018 23:04:53 -0000 Date: Fri, 14 Dec 2018 23:04:49 +0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Linus Torvalds , Michal Hocko , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing Subject: [PATCH 10/14] mm, compaction: Use free lists to quickly locate a migration source Message-ID: <20181214230449.GA29005@techsingularity.net> References: <20181214230310.572-1-mgorman@techsingularity.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20181214230310.572-1-mgorman@techsingularity.net> User-Agent: Mutt/1.10.1 (2018-07-13) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The migration scanner is a linear scan of a zone which is a potentially very large search space. Furthermore, many pageblocks are unusable such as those filled with reserved pages or partially filled with pages that cannot migrate. These still get scanned in the common case of allocating a THP and the cost accumulates. The patch uses a partial search of the free lists to locate a migration source candidate that is marked as MOVABLE when allocating a THP. It prefers picking a block with a larger number of free pages already on the basis that there are fewer pages to migrate to free the entire block. The lowest PFN found during searches is tracked as the basis of the start for the linear search after the first search of the free list fails. After the search, the free list is shuffled so that the next search will not encounter the same page. If the search fails then the subsequent searches will be shorter and the linear scanner is used. If this search fails, or if the request is for a small or unmovable/reclaimable allocation then the linear scanner is still used. It is somewhat pointless to use the list search in these cases. Small free pages must be used for the search and there is no guarantee that movable pages are located within that block that are contiguous. 4.20.0-rc6 4.20.0-rc6 noboost-v1r4 findmig-v1r4 Amean fault-both-3 3753.53 ( 0.00%) 3545.40 ( 5.54%) Amean fault-both-5 5396.32 ( 0.00%) 5431.98 ( -0.66%) Amean fault-both-7 7393.46 ( 0.00%) 7185.11 ( 2.82%) Amean fault-both-12 12155.50 ( 0.00%) 11424.68 ( 6.01%) Amean fault-both-18 16445.96 ( 0.00%) 14170.10 * 13.84%* Amean fault-both-24 20465.03 ( 0.00%) 16143.57 * 21.12%* Amean fault-both-30 20813.54 ( 0.00%) 19207.96 ( 7.71%) Amean fault-both-32 22384.02 ( 0.00%) 20051.01 * 10.42%* Compaction migrate scanned 60836989 51005450 Compaction free scanned 890084421 780359464 This is showing a 16% reduction in migration scanning with some mild improvements on latency. A 2-socket machine showed similar reductions of scan rates in percentage terms. Signed-off-by: Mel Gorman --- mm/compaction.c | 179 +++++++++++++++++++++++++++++++++++++++++++++++++++++++- mm/internal.h | 2 + 2 files changed, 179 insertions(+), 2 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 8ba9b3b479e3..65c7ab1847a0 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1041,6 +1041,12 @@ static bool suitable_migration_target(struct compact_control *cc, return false; } +static inline unsigned int +freelist_scan_limit(struct compact_control *cc) +{ + return (COMPACT_CLUSTER_MAX >> cc->fast_search_fail) + 1; +} + /* * Test whether the free scanner has reached the same or lower pageblock than * the migration scanner, and compaction should thus terminate. @@ -1051,6 +1057,19 @@ static inline bool compact_scanners_met(struct compact_control *cc) <= (cc->migrate_pfn >> pageblock_order); } +/* Reorder the free list to reduce repeated future searches */ +static void +move_freelist_tail(struct list_head *freelist, struct page *freepage) +{ + LIST_HEAD(sublist); + + if (!list_is_last(freelist, &freepage->lru)) { + list_cut_position(&sublist, freelist, &freepage->lru); + if (!list_empty(&sublist)) + list_splice_tail(&sublist, freelist); + } +} + /* * Based on information in the current compact_control, find blocks * suitable for isolating free pages from and then isolate them. @@ -1208,6 +1227,160 @@ typedef enum { */ int sysctl_compact_unevictable_allowed __read_mostly = 1; +static inline void +update_fast_start_pfn(struct compact_control *cc, unsigned long pfn) +{ + if (cc->fast_start_pfn == ULONG_MAX) + return; + + if (!cc->fast_start_pfn) + cc->fast_start_pfn = pfn; + + cc->fast_start_pfn = min(cc->fast_start_pfn, pfn); +} + +static inline void +reinit_migrate_pfn(struct compact_control *cc) +{ + if (!cc->fast_start_pfn || cc->fast_start_pfn == ULONG_MAX) + return; + + cc->migrate_pfn = cc->fast_start_pfn; + cc->fast_start_pfn = ULONG_MAX; +} + +/* + * Briefly search the free lists for a migration source that already has + * some free pages to reduce the number of pages that need migration + * before a pageblock is free. + */ +static unsigned long fast_find_migrateblock(struct compact_control *cc) +{ + unsigned int limit = freelist_scan_limit(cc); + unsigned int nr_scanned = 0; + unsigned long distance; + unsigned long pfn = cc->migrate_pfn; + unsigned long high_pfn; + int order; + + /* Skip hints are relied on to avoid repeats on the fast search */ + if (cc->ignore_skip_hint) + return pfn; + + /* + * If the migrate_pfn is not at the start of a zone or the start + * of a pageblock then assume this is a continuation of a previous + * scan restarted due to COMPACT_CLUSTER_MAX. + */ + if (pfn != cc->zone->zone_start_pfn && pfn != pageblock_start_pfn(pfn)) + return pfn; + + /* + * For smaller orders, just linearly scan as the number of pages + * to migrate should be relatively small and does not necessarily + * justify freeing up a large block for a small allocation. + */ + if (cc->order <= PAGE_ALLOC_COSTLY_ORDER) + return pfn; + + /* + * Only allow kcompactd and direct requests for movable pages to + * quickly clear out a MOVABLE pageblock for allocation. This + * reduces the risk that a large movable pageblock is freed for + * an unmovable/reclaimable small allocation. + */ + if (cc->direct_compaction && cc->migratetype != MIGRATE_MOVABLE) + return pfn; + + /* + * When starting the migration scanner, pick any pageblock within the + * first half of the search space. Otherwise try and pick a pageblock + * within the first eighth to reduce the chances that a migration + * target later becomes a source. + */ + distance = (cc->free_pfn - cc->migrate_pfn) >> 1; + if (cc->migrate_pfn != cc->zone->zone_start_pfn) + distance >>= 2; + high_pfn = pageblock_start_pfn(cc->migrate_pfn + distance); + + for (order = cc->order - 1; + order >= PAGE_ALLOC_COSTLY_ORDER && pfn == cc->migrate_pfn && nr_scanned < limit; + order--) { + struct free_area *area = &cc->zone->free_area[order]; + struct list_head *freelist; + unsigned long nr_skipped = 0; + unsigned long flags; + struct page *freepage; + + if (!area->nr_free) + continue; + + spin_lock_irqsave(&cc->zone->lock, flags); + freelist = &area->free_list[MIGRATE_MOVABLE]; + list_for_each_entry(freepage, freelist, lru) { + unsigned long free_pfn; + + nr_scanned++; + free_pfn = page_to_pfn(freepage); + if (free_pfn < high_pfn) { + update_fast_start_pfn(cc, free_pfn); + + /* + * Avoid if skipped recently. Move to the tail + * of the list so it will not be found again + * soon + */ + if (get_pageblock_skip(freepage)) { + + if (list_is_last(freelist, &freepage->lru)) + break; + + nr_skipped++; + list_del(&freepage->lru); + list_add_tail(&freepage->lru, freelist); + if (nr_skipped > 2) + break; + continue; + } + + /* Reorder to so a future search skips recent pages */ + move_freelist_tail(freelist, freepage); + + pfn = pageblock_start_pfn(free_pfn); + cc->fast_search_fail = 0; + set_pageblock_skip(freepage); + break; + } + + /* + * If low PFNs are being found and discarded then + * limit the scan as fast searching is finding + * poor candidates. + */ + if (free_pfn < cc->migrate_pfn) + limit >>= 1; + + if (nr_scanned >= limit) { + cc->fast_search_fail++; + move_freelist_tail(freelist, freepage); + break; + } + } + spin_unlock_irqrestore(&cc->zone->lock, flags); + } + + cc->total_migrate_scanned += nr_scanned; + + /* + * If fast scanning failed then use a cached entry for a page block + * that had free pages as the basis for starting a linear scan. + */ + if (pfn == cc->migrate_pfn) + reinit_migrate_pfn(cc); + + return pfn; +} + /* * Isolate all pages that can be migrated from the first suitable block, * starting at the block pointed to by the migrate scanner pfn within @@ -1226,9 +1399,10 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone, /* * Start at where we last stopped, or beginning of the zone as - * initialized by compact_zone() + * initialized by compact_zone(). The first failure will use + * the lowest PFN as the starting point for linear scanning. */ - low_pfn = cc->migrate_pfn; + low_pfn = fast_find_migrateblock(cc); block_start_pfn = pageblock_start_pfn(low_pfn); if (block_start_pfn < zone->zone_start_pfn) block_start_pfn = zone->zone_start_pfn; @@ -1551,6 +1725,7 @@ static enum compact_result compact_zone(struct compact_control *cc) * want to compact the whole zone), but check that it is initialised * by ensuring the values are within zone boundaries. */ + cc->fast_start_pfn = 0; if (cc->whole_zone) { cc->migrate_pfn = start_pfn; cc->free_pfn = pageblock_start_pfn(end_pfn - 1); diff --git a/mm/internal.h b/mm/internal.h index 9b32f4cab0ae..983cb975545f 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -188,9 +188,11 @@ struct compact_control { unsigned int nr_migratepages; /* Number of pages to migrate */ unsigned long free_pfn; /* isolate_freepages search base */ unsigned long migrate_pfn; /* isolate_migratepages search base */ + unsigned long fast_start_pfn; /* a pfn to start linear scan from */ struct zone *zone; unsigned long total_migrate_scanned; unsigned long total_free_scanned; + unsigned int fast_search_fail; /* failures to use free list searches */ const gfp_t gfp_mask; /* gfp mask of a direct compactor */ int order; /* order a direct compactor needs */ int migratetype; /* migratetype of direct compactor */ From patchwork Fri Dec 14 23:05:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10731793 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C3F45746 for ; Fri, 14 Dec 2018 23:05:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 720772D653 for ; Fri, 14 Dec 2018 23:05:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 65E0C2D666; Fri, 14 Dec 2018 23:05:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 923172D653 for ; Fri, 14 Dec 2018 23:05:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B6FF58E0221; Fri, 14 Dec 2018 18:05:17 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B1E358E021D; Fri, 14 Dec 2018 18:05:17 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9E69E8E0221; Fri, 14 Dec 2018 18:05:17 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by kanga.kvack.org (Postfix) with ESMTP id 46D738E021D for ; Fri, 14 Dec 2018 18:05:17 -0500 (EST) Received: by mail-ed1-f70.google.com with SMTP id x15so3463515edd.2 for ; Fri, 14 Dec 2018 15:05:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:date:from:to :cc:subject:message-id:references:mime-version:content-disposition :in-reply-to:user-agent; bh=XYOui0TBhQHSSxBVXCACJXmiHYr0r2sm2VSVO/EdJtI=; b=JfnAAUGy5xbu9CQhMXem+HsUtzckA4k5YFk0gn3RrdVJZb2ikfgVOUDh8QIxYdVZ8e gtiK93ooXPErD6cYC+evKHex5cvuoYuWlP2MuXxs0FZ+xC4FcyJmcF+MqS8N/Z1rDu9w v4TQXySLuN/ddx11S3H4jbyaiAirHPM+eqtQGem4szXCgx+l+Fup3sImT+MqOlWLcsX2 nU2Af9FvJFGQjy8ZkloUOHhXB2VN6ztZNcVtMyMfS3LmfK8ciZ4Jg2uvPVRH2XhdMy9b MVwuamfZoJrD+iSt4LgouKVFeP74yqJBq4QXiK9SqU14AyaGnueKYkT/B4e9ZgaX1u9h T2ug== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWa+0YXeQRH80BxxWkrhqgm1gPrN1BSjB8oHhKXDZr2z5ZWVoVRm 5BF7GqFC7v3LF1C7jhpVJHqPnyQE2cQvu5WVKBu6nzlggmnWeqTTbMwFoa7ArjOeFYFC40oqC1h Xf2xUD2PtujqH4iC2wKtVem9lKZaFKhwBw7wrRgUKoeJmnyyzfjofDlzI4A32ItQvXA== X-Received: by 2002:a50:fe15:: with SMTP id f21mr4286751edt.116.1544828716761; Fri, 14 Dec 2018 15:05:16 -0800 (PST) X-Google-Smtp-Source: AFSGD/W+t2Jrs9RHodFG4GhFV7mFOuqzXPyDcKIS7XTqbVQLDVi1NiySqn2Bg+wHt5bwXsJH3g6W X-Received: by 2002:a50:fe15:: with SMTP id f21mr4286709edt.116.1544828715543; Fri, 14 Dec 2018 15:05:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544828715; cv=none; d=google.com; s=arc-20160816; b=feTQxKKbGYwCp8Y4YYwSwAtNmYTUDd0VMCEFUMVk1jM7EvFcWznewLa2f/ue/kUduS I8ynFeguN7tXNvM2kTL9RB0ubRaSC2LsnVcR180b30G091HiclWVQLQYXXbPj8AS2/Uh cgvRPXSYsUugCqkLx27mU75CZO1MFbYZZl3OV2V5zrZru6AOQ+6NbQSdzqv4xIaI5woo jyoADEFxpiz2XATYRcBFUDf/I2lK19zaZH7FAJW6JvaBSk5Q5t92qLw10PFSq5XaANvH HnWhHEBMjCSPTnhl/s4771x+PcBDiEri7PyXD0jIyqxx1QE3QB6Oqzt2M1TJg5sWS0Mf kgDA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=user-agent:in-reply-to:content-disposition:mime-version:references :message-id:subject:cc:to:from:date; bh=XYOui0TBhQHSSxBVXCACJXmiHYr0r2sm2VSVO/EdJtI=; b=DOR/4feRpEbfVZd6q8xWm5RWn4vTaIE/b+GTYPGsBDnq0D5hQKgzxfpNK8t9IFopH5 0uDYtNF89KhAn87vu8THa3jE5ez6OJM9rtP1k39Dz0RhLgzwy/xE0uKfMDIb35haTS/w bnd9j6SMuvzE3LipC8BBxNBcJmymK+TEszlNYKo0Nhj4LyHt2HdRbhi/b+RkrKMtCXpB 9bjmmXY8tKOhdGOLqytqppyz3Gi1GBaEJOeB3ifyd7YhWeLAPqpxWs2Trk1qLzamGbu6 RAKMroWYyF+0/qr0JfwKHUAyUH4Ox8j02wLq7iy4krxPywMcSEPedIAQ1yx8HsSzNJ2O PXMw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp04.blacknight.com (outbound-smtp04.blacknight.com. [81.17.249.35]) by mx.google.com with ESMTPS id 26-v6si682808ejl.106.2018.12.14.15.05.15 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 14 Dec 2018 15:05:15 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) client-ip=81.17.249.35; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail06.blacknight.ie [81.17.255.152]) by outbound-smtp04.blacknight.com (Postfix) with ESMTPS id 32DC4987B6 for ; Fri, 14 Dec 2018 23:05:15 +0000 (UTC) Received: (qmail 2238 invoked from network); 14 Dec 2018 23:05:15 -0000 Received: from unknown (HELO techsingularity.net) (mgorman@techsingularity.net@[37.228.245.71]) by 81.17.254.9 with ESMTPSA (AES256-SHA encrypted, authenticated); 14 Dec 2018 23:05:15 -0000 Date: Fri, 14 Dec 2018 23:05:13 +0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Linus Torvalds , Michal Hocko , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing Subject: [PATCH 11/14] mm, compaction: Keep migration source private to a single compaction instance Message-ID: <20181214230513.GB29005@techsingularity.net> References: <20181214230310.572-1-mgorman@techsingularity.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20181214230310.572-1-mgorman@techsingularity.net> User-Agent: Mutt/1.10.1 (2018-07-13) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Due to either a fast search of the free list or a linear scan, it's possible for multiple compaction instances to pick the same pageblock for migration. This is lucky for one scanner and increased scanning for all the others. It also opens a race to allocate the resulting free block. This patch tests and updates the pageblock skip for the migration scanner more carefully. When isolating a block, it will check and skip if the block is already in use. Once the zone lock is acquired, it will be rechecked so that only one scanner can set the pageblock skip for exclusive use. Any scanner contending will continue with a linear scan. The skip bit is still set if no pages can be isolated in a range. While this may result in redundant scanning, it avoids unnecessarily acquiring the zone lock when there are no suitable migration sources. 1-socket thpscale 4.20.0-rc6 4.20.0-rc6 findmig-v1r4 isolmig-v1r4 Amean fault-both-3 3545.40 ( 0.00%) 2980.25 * 15.94%* Amean fault-both-5 5431.98 ( 0.00%) 4393.04 * 19.13%* Amean fault-both-7 7185.11 ( 0.00%) 5797.16 * 19.32%* Amean fault-both-12 11424.68 ( 0.00%) 9849.61 ( 13.79%) Amean fault-both-18 14170.10 ( 0.00%) 13816.96 ( 2.49%) Amean fault-both-24 16143.57 ( 0.00%) 16255.20 ( -0.69%) Amean fault-both-30 19207.96 ( 0.00%) 15741.25 * 18.05%* Amean fault-both-32 20051.01 ( 0.00%) 16624.73 * 17.09%* This is the first patch that shows a significant reduction in latency as multiple compaction scanners do not operate on the same blocks. There is a small increase in the success rate 4.20.0-rc6 4.20.0-rc6 findmig-v1r4 isolmig-v1r4 Percentage huge-3 91.95 ( 0.00%) 95.97 ( 4.37%) Percentage huge-5 91.40 ( 0.00%) 94.78 ( 3.70%) Percentage huge-7 92.94 ( 0.00%) 93.94 ( 1.07%) Percentage huge-12 92.13 ( 0.00%) 93.77 ( 1.78%) Percentage huge-18 91.01 ( 0.00%) 94.57 ( 3.91%) Percentage huge-24 89.56 ( 0.00%) 93.71 ( 4.63%) Percentage huge-30 90.26 ( 0.00%) 94.14 ( 4.30%) Percentage huge-32 90.70 ( 0.00%) 94.44 ( 4.12%) Compaction migrate scanned 51005450 25587453 Compaction free scanned 780359464 87735894 Migration scan rates are reduced by 49%. At the time of writing, the 2-socket results are not yet available. Signed-off-by: Mel Gorman --- mm/compaction.c | 112 +++++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 87 insertions(+), 25 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 65c7ab1847a0..b0309bf409b3 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -279,13 +279,51 @@ void reset_isolation_suitable(pg_data_t *pgdat) } } +/* + * Sets the pageblock skip bit if it was clear. Note that this is a hint as + * locks are not required for read/writers. Returns true if it was already set. + */ +static bool test_and_set_skip(struct compact_control *cc, struct page *page, + unsigned long pfn) +{ + bool skip; + + /* Do no update if skip hint is being ignored */ + if (cc->ignore_skip_hint) + return false; + + if (!IS_ALIGNED(pfn, pageblock_nr_pages)) + return false; + + skip = get_pageblock_skip(page); + if (!skip && !cc->no_set_skip_hint) + set_pageblock_skip(page); + + return skip; +} + +static void update_cached_migrate(struct compact_control *cc, unsigned long pfn) +{ + struct zone *zone = cc->zone; + pfn = pageblock_end_pfn(pfn); + + /* Set for isolation rather than compaction */ + if (cc->no_set_skip_hint) + return; + + if (pfn > zone->compact_cached_migrate_pfn[0]) + zone->compact_cached_migrate_pfn[0] = pfn; + if (cc->mode != MIGRATE_ASYNC && + pfn > zone->compact_cached_migrate_pfn[1]) + zone->compact_cached_migrate_pfn[1] = pfn; +} + /* * If no pages were isolated then mark this pageblock to be skipped in the * future. The information is later cleared by __reset_isolation_suitable(). */ static void update_pageblock_skip(struct compact_control *cc, - struct page *page, unsigned long nr_isolated, - bool migrate_scanner) + struct page *page, unsigned long nr_isolated) { struct zone *zone = cc->zone; unsigned long pfn; @@ -304,16 +342,8 @@ static void update_pageblock_skip(struct compact_control *cc, pfn = page_to_pfn(page); /* Update where async and sync compaction should restart */ - if (migrate_scanner) { - if (pfn > zone->compact_cached_migrate_pfn[0]) - zone->compact_cached_migrate_pfn[0] = pfn; - if (cc->mode != MIGRATE_ASYNC && - pfn > zone->compact_cached_migrate_pfn[1]) - zone->compact_cached_migrate_pfn[1] = pfn; - } else { - if (pfn < zone->compact_cached_free_pfn) - zone->compact_cached_free_pfn = pfn; - } + if (pfn < zone->compact_cached_free_pfn) + zone->compact_cached_free_pfn = pfn; } #else static inline bool isolation_suitable(struct compact_control *cc, @@ -561,7 +591,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, /* Update the pageblock-skip if the whole pageblock was scanned */ if (blockpfn == end_pfn) - update_pageblock_skip(cc, valid_page, total_isolated, false); + update_pageblock_skip(cc, valid_page, total_isolated); cc->total_free_scanned += nr_scanned; if (total_isolated) @@ -696,6 +726,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, unsigned long start_pfn = low_pfn; bool skip_on_failure = false; unsigned long next_skip_pfn = 0; + bool skip_updated = false; /* * Ensure that there are not too many pages isolated from the LRU @@ -762,8 +793,19 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, page = pfn_to_page(low_pfn); - if (!valid_page) + /* + * Check if the pageblock has already been marked skipped. + * Only the aligned PFN is checked as the caller isolates + * COMPACT_CLUSTER_MAX at a time so the second call must + * not falsely conclude that the block should be skipped. + */ + if (!valid_page && IS_ALIGNED(low_pfn, pageblock_nr_pages)) { + if (!cc->ignore_skip_hint && get_pageblock_skip(page)) { + low_pfn = end_pfn; + goto isolate_abort; + } valid_page = page; + } /* * Skip if free. We read page order here without zone lock @@ -851,8 +893,19 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, if (!locked) { locked = compact_trylock_irqsave(zone_lru_lock(zone), &flags, cc); - if (!locked) + + /* Allow future scanning if the lock is contended */ + if (!locked) { + clear_pageblock_skip(page); break; + } + + /* Try get exclusive access under lock */ + if (!skip_updated) { + skip_updated = true; + if (test_and_set_skip(cc, page, low_pfn)) + goto isolate_abort; + } /* Recheck PageLRU and PageCompound under lock */ if (!PageLRU(page)) @@ -930,15 +983,20 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, if (unlikely(low_pfn > end_pfn)) low_pfn = end_pfn; +isolate_abort: if (locked) spin_unlock_irqrestore(zone_lru_lock(zone), flags); /* - * Update the pageblock-skip information and cached scanner pfn, - * if the whole pageblock was scanned without isolating any page. + * Updated the cached scanner pfn if the pageblock was scanned + * without isolating a page. The pageblock may not be marked + * skipped already if there were no LRU pages in the block. */ - if (low_pfn == end_pfn) - update_pageblock_skip(cc, valid_page, nr_isolated, true); + if (low_pfn == end_pfn && !nr_isolated) { + if (valid_page && !skip_updated) + set_pageblock_skip(valid_page); + update_cached_migrate(cc, low_pfn); + } trace_mm_compaction_isolate_migratepages(start_pfn, low_pfn, nr_scanned, nr_isolated); @@ -1323,8 +1381,6 @@ static unsigned long fast_find_migrateblock(struct compact_control *cc) nr_scanned++; free_pfn = page_to_pfn(freepage); if (free_pfn < high_pfn) { - update_fast_start_pfn(cc, free_pfn); - /* * Avoid if skipped recently. Move to the tail * of the list so it will not be found again @@ -1346,9 +1402,9 @@ static unsigned long fast_find_migrateblock(struct compact_control *cc) /* Reorder to so a future search skips recent pages */ move_freelist_tail(freelist, freepage); + update_fast_start_pfn(cc, free_pfn); pfn = pageblock_start_pfn(free_pfn); cc->fast_search_fail = 0; - set_pageblock_skip(freepage); break; } @@ -1418,7 +1474,6 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone, low_pfn = block_end_pfn, block_start_pfn = block_end_pfn, block_end_pfn += pageblock_nr_pages) { - /* * This can potentially iterate a massively long zone with * many pageblocks unsuitable, so periodically check if we @@ -1433,8 +1488,15 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone, if (!page) continue; - /* If isolation recently failed, do not retry */ - if (!isolation_suitable(cc, page)) + /* + * If isolation recently failed, do not retry. Only check the + * pageblock once. COMPACT_CLUSTER_MAX causes a pageblock + * to be visited multiple times. Assume skip was checked + * before making it "skip" so other compaction instances do + * not scan the same block. + */ + if (IS_ALIGNED(low_pfn, pageblock_nr_pages) && + !isolation_suitable(cc, page)) continue; /* From patchwork Fri Dec 14 23:05:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10731795 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DE5EC746 for ; Fri, 14 Dec 2018 23:05:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 897EF2D653 for ; Fri, 14 Dec 2018 23:05:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7C9BC2D666; Fri, 14 Dec 2018 23:05:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AF2702D653 for ; Fri, 14 Dec 2018 23:05:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EF3068E0223; Fri, 14 Dec 2018 18:05:35 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id ED2118E021D; Fri, 14 Dec 2018 18:05:35 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CCF238E0223; Fri, 14 Dec 2018 18:05:35 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id 751F28E021D for ; Fri, 14 Dec 2018 18:05:35 -0500 (EST) Received: by mail-ed1-f69.google.com with SMTP id c53so3488015edc.9 for ; Fri, 14 Dec 2018 15:05:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:date:from:to :cc:subject:message-id:references:mime-version:content-disposition :in-reply-to:user-agent; bh=qDTJd7Ss/7rGQZW0ySEEhEcnkOUrvh6IS8g085XCK0U=; b=lmqiEfyg+oQx32EfVjNldXJO7QLcfl+JGVRlIzAWsF1/n2pTxCYnXj5tapsDqxIBt0 IeraiXfch+8javFxFwTXeuDIPuuirUgYrSiD/JsbqENKl0xXXMDKM7VwQMLkQgNgyHNI QTFaGgrW3d56hWudkNR2qI8oFz/FQqd6bZiVnaRV1KDGQXwbakyeBxhbuUU9mIiljPoM c/aaXNsj+8wbt8DpyAsNigvhXjChnY/EIwv7zKWZnnCdVx6aP7y/3ir3+k56TtuZgCT+ DB2YWk2Q64rQr8tQQLZQdPnHDEB2vPNsRnnsZHE4c1/U65G/gqAJvhtAT2yOOOMLt2tJ PI/A== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWbKe+abvpxXSTVCsUDwyoBr5tvEfF6kUWQ8ZF3DTDRAp/C8Rrt4 45XGgVDaT3TJq2dFZqYRUD6VON64bphA53TLJzhQ159UZhIG1QZ8e27JfvqsAy7++Gn2XwRRa54 Dx7PmGcaf5NnhJWceCWPfYrnIK11SN2+l59u0ceE10y0lY/iC3tie8pPO6xTYM9rUsA== X-Received: by 2002:a50:8a45:: with SMTP id i63mr4585990edi.262.1544828734962; Fri, 14 Dec 2018 15:05:34 -0800 (PST) X-Google-Smtp-Source: AFSGD/Xto0G6IP6sySBzg1cyYPYcUeAvxjRpY6RiVgO+DLmFaGQyCZgXYeHrMk+Km5vWXgCLghHL X-Received: by 2002:a50:8a45:: with SMTP id i63mr4585945edi.262.1544828733784; Fri, 14 Dec 2018 15:05:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544828733; cv=none; d=google.com; s=arc-20160816; b=UtGbPa5dWfLYQFX/DbsgcUtkil+8i80/5aXyISaFoVC0LIo4dm5v6kUUXSJrIdJB7P 2EFpi74kJunac0oDRCUJ8CyW9MAR0q0PWXNCki9E8ypfj3icmvhWqPXlf5x9OyF7W8yt ivfqrPgnwtgvbc3YfoeUVFPsPWGJJzEx5bTlyZwMSCdp8W5ZCmyBevRMt2yS0QNq9U1b TfLEIXmWaubMJiGyKA54MQE9x7D5EuPa3TLYvjWeeki4gSoMKG+Zex5yWMr5MPlBSptK HrnB089+8AmSLLZCUMSNc69I2Mf0rp/zN0xRDFMmWbocDkqEgqj+jRIMgYAU1eLSGGkd h0fQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=user-agent:in-reply-to:content-disposition:mime-version:references :message-id:subject:cc:to:from:date; bh=qDTJd7Ss/7rGQZW0ySEEhEcnkOUrvh6IS8g085XCK0U=; b=uiwCSWb7amRE2u6kpi9dw5WtXjvSWZYFdvjJW9xCooDdF28pQBYuv+f4rc+y2eZ3TL TdQdfDFbBlKbahm2kVkIM95FfmCgBGImLBauUdOnCJfKVkkw9e6+Fk7rF9s78YoGeLNw jrtAWeeoSpM91e1Ks+NuhQtjHe/ECA2Aj+sxoiOagG2JoGtep1Z6NU8lcpTdq3qNyy60 6u7KxW6yvPu/tiQGoBz7/OKLY8qHwQZ6zyogEumeU/3JvQym2xGnBS4DQta2yv27D/nV b3yoqKCD3uAKhj50pgqG7FX8Xr/iOmiHi6gC4W69oaI7BN/dT824ovI964pjfDweXi6p 3DBA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp12.blacknight.com (outbound-smtp12.blacknight.com. [46.22.139.17]) by mx.google.com with ESMTPS id v3-v6si981560eji.60.2018.12.14.15.05.33 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 14 Dec 2018 15:05:33 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) client-ip=46.22.139.17; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail06.blacknight.ie [81.17.255.152]) by outbound-smtp12.blacknight.com (Postfix) with ESMTPS id 66E911C1DF6 for ; Fri, 14 Dec 2018 23:05:33 +0000 (GMT) Received: (qmail 2609 invoked from network); 14 Dec 2018 23:05:33 -0000 Received: from unknown (HELO techsingularity.net) (mgorman@techsingularity.net@[37.228.245.71]) by 81.17.254.9 with ESMTPSA (AES256-SHA encrypted, authenticated); 14 Dec 2018 23:05:33 -0000 Date: Fri, 14 Dec 2018 23:05:31 +0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Linus Torvalds , Michal Hocko , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing Subject: [PATCH 12/14] mm, compaction: Use free lists to quickly locate a migration target Message-ID: <20181214230531.GC29005@techsingularity.net> References: <20181214230310.572-1-mgorman@techsingularity.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20181214230310.572-1-mgorman@techsingularity.net> User-Agent: Mutt/1.10.1 (2018-07-13) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Similar to the migration scanner, this uses the free lists to quickly locate a migration target. The search is different in that lower orders will be searched for a suitable high PFN if necessary but the search is still bound. This is justified on the grounds that the free scanner typically scans linearly much more than the migration scanner. If a free page is found, it is isolated and the full pageblock is scanned for any remaining free pages. This is done so that it's possible to mark the pageblock for skipping in the near future. 1-socket thpfioscale 4.20.0-rc6 4.20.0-rc6 isolmig-v1r4 findfree-v1r8 Amean fault-both-3 2980.25 ( 0.00%) 2911.07 ( 2.32%) Amean fault-both-5 4393.04 ( 0.00%) 4692.96 ( -6.83%) Amean fault-both-7 5797.16 ( 0.00%) 6449.17 ( -11.25%) Amean fault-both-12 9849.61 ( 0.00%) 9778.40 ( 0.72%) Amean fault-both-18 13816.96 ( 0.00%) 11756.92 ( 14.91%) Amean fault-both-24 16255.20 ( 0.00%) 13675.93 * 15.87%* Amean fault-both-30 15741.25 ( 0.00%) 17195.41 ( -9.24%) Amean fault-both-32 16624.73 ( 0.00%) 18150.08 ( -9.18%) The impact on latency is variable but the search is optimistic and sensitive to the exact system state. Success rates are similar but the major impact is to the rate of scanning 4.20.0-rc6 4.20.0-rc6 isolmig-v1r4findfree-v1r8 Compaction migrate scanned 25587453 27634284 Compaction free scanned 87735894 55279519 The free scan rates are reduced by 37%. Signed-off-by: Mel Gorman --- mm/compaction.c | 201 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 197 insertions(+), 4 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index b0309bf409b3..ba3035dcc548 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1117,7 +1117,7 @@ static inline bool compact_scanners_met(struct compact_control *cc) /* Reorder the free list to reduce repeated future searches */ static void -move_freelist_tail(struct list_head *freelist, struct page *freepage) +move_freelist_head(struct list_head *freelist, struct page *freepage) { LIST_HEAD(sublist); @@ -1128,6 +1128,193 @@ move_freelist_tail(struct list_head *freelist, struct page *freepage) } } +static void +move_freelist_tail(struct list_head *freelist, struct page *freepage) +{ + LIST_HEAD(sublist); + + if (!list_is_last(freelist, &freepage->lru)) { + list_cut_before(&sublist, freelist, &freepage->lru); + if (!list_empty(&sublist)) + list_splice_tail(&sublist, freelist); + } +} + +static void +fast_isolate_around(struct compact_control *cc, unsigned long pfn, unsigned long nr_isolated) +{ + unsigned long start_pfn, end_pfn; + struct page *page = pfn_to_page(pfn); + + /* Do not search around if there are enough pages already */ + if (cc->nr_freepages >= cc->nr_migratepages) + return; + + /* Minimise scanning during async compaction */ + if (cc->direct_compaction && cc->mode == MIGRATE_ASYNC) + return; + + /* Pageblock boundaries */ + start_pfn = pageblock_start_pfn(pfn); + end_pfn = min(start_pfn + pageblock_nr_pages, zone_end_pfn(cc->zone)); + + /* Scan before */ + if (start_pfn != pfn) { + isolate_freepages_block(cc, &start_pfn, pfn, &cc->freepages, false); + if (cc->nr_freepages >= cc->nr_migratepages) + return; + } + + /* Scan after */ + start_pfn = pfn + nr_isolated; + if (start_pfn != end_pfn) + isolate_freepages_block(cc, &start_pfn, end_pfn, &cc->freepages, false); + + /* Skip this pageblock in the future as it's full or nearly full */ + if (cc->nr_freepages < cc->nr_migratepages) + set_pageblock_skip(page); +} + +static unsigned long +fast_isolate_freepages(struct compact_control *cc) +{ + unsigned int limit = min(1U, freelist_scan_limit(cc) >> 1); + unsigned int order_scanned = 0, nr_scanned = 0; + unsigned long low_pfn, min_pfn, high_pfn = 0, highest = 0; + unsigned long nr_isolated = 0; + unsigned long distance; + struct page *page = NULL; + bool scan_start = false; + int order; + + /* + * If starting the scan, use a deeper search and use the highest + * PFN found if a suitable one is not found. + */ + if (cc->free_pfn == pageblock_start_pfn(zone_end_pfn(cc->zone) - 1)) { + limit = pageblock_nr_pages >> 1; + scan_start = true; + } + + /* + * Preferred point is in the top quarter of the scan space but take + * a pfn from the top half if the search is problematic. + */ + distance = (cc->free_pfn - cc->migrate_pfn); + low_pfn = pageblock_start_pfn(cc->free_pfn - (distance >> 2)); + min_pfn = pageblock_start_pfn(cc->free_pfn - (distance >> 1)); + + if (WARN_ON_ONCE(min_pfn > low_pfn)) + low_pfn = min_pfn; + + for (order = cc->order - 1; + order >= 0 && !page; + order--) { + struct free_area *area = &cc->zone->free_area[order]; + struct list_head *freelist; + struct page *freepage; + unsigned long flags; + + if (!area->nr_free) + continue; + + spin_lock_irqsave(&cc->zone->lock, flags); + freelist = &area->free_list[MIGRATE_MOVABLE]; + list_for_each_entry_reverse(freepage, freelist, lru) { + unsigned long pfn; + + order_scanned++; + nr_scanned++; + pfn = page_to_pfn(freepage); + + if (pfn >= highest) + highest = pageblock_start_pfn(pfn); + + if (pfn >= low_pfn) { + cc->fast_search_fail = 0; + page = freepage; + break; + } + + if (pfn >= min_pfn && pfn > high_pfn) { + high_pfn = pfn; + + /* Shorten the scan if a candidate is found */ + limit >>= 1; + } + + if (order_scanned >= limit) + break; + } + + /* Use a minimum pfn if a preferred one was not found */ + if (!page && high_pfn) { + page = pfn_to_page(high_pfn); + + /* Update freepage for the list reorder below */ + freepage = page; + } + + /* Reorder to so a future search skips recent pages */ + move_freelist_head(freelist, freepage); + + /* Isolate the page if available */ + if (page) { + if (__isolate_free_page(page, order)) { + set_page_private(page, order); + nr_isolated = 1 << order; + cc->nr_freepages += nr_isolated; + list_add_tail(&page->lru, &cc->freepages); + count_compact_events(COMPACTISOLATED, nr_isolated); + } else { + /* If isolation fails, abort the search */ + order = -1; + page = NULL; + } + } + + spin_unlock_irqrestore(&cc->zone->lock, flags); + + /* + * Smaller scan on next order so the total scan ig related + * to freelist_scan_limit. + */ + if (order_scanned >= limit) + limit = min(1U, limit >> 1); + } + + if (!page) { + cc->fast_search_fail++; + if (scan_start) { + /* + * Use the highest PFN found above min. If one was + * not found, be pessemistic for direct compaction + * and use the min mark. + */ + if (highest) { + page = pfn_to_page(highest); + cc->free_pfn = highest; + } else { + if (cc->direct_compaction) { + page = pfn_to_page(min_pfn); + cc->free_pfn = min_pfn; + } + } + } + } + + if (highest && highest > cc->zone->compact_cached_free_pfn) + cc->zone->compact_cached_free_pfn = highest; + + cc->total_free_scanned += nr_scanned; + if (!page) + return 0; + + low_pfn = page_to_pfn(page); + fast_isolate_around(cc, low_pfn, nr_isolated); + return low_pfn; +} + /* * Based on information in the current compact_control, find blocks * suitable for isolating free pages from and then isolate them. @@ -1142,6 +1329,11 @@ static void isolate_freepages(struct compact_control *cc) unsigned long low_pfn; /* lowest pfn scanner is able to scan */ struct list_head *freelist = &cc->freepages; + /* Try a small search of the free lists for a candidate */ + isolate_start_pfn = fast_isolate_freepages(cc); + if (isolate_start_pfn) + goto splitmap; + /* * Initialise the free scanner. The starting point is where we last * successfully isolated from, zone-cached value, or the end of the @@ -1218,9 +1410,6 @@ static void isolate_freepages(struct compact_control *cc) } } - /* __isolate_free_page() does not map the pages */ - split_map_pages(freelist); - /* * Record where the free scanner will restart next time. Either we * broke from the loop and set isolate_start_pfn based on the last @@ -1228,6 +1417,10 @@ static void isolate_freepages(struct compact_control *cc) * and the loop terminated due to isolate_start_pfn < low_pfn */ cc->free_pfn = isolate_start_pfn; + +splitmap: + /* __isolate_free_page() does not map the pages */ + split_map_pages(freelist); } /* From patchwork Fri Dec 14 23:06:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10731797 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 927FF1399 for ; Fri, 14 Dec 2018 23:06:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3F74E2D653 for ; Fri, 14 Dec 2018 23:06:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 31A832D666; Fri, 14 Dec 2018 23:06:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B9E3D2D653 for ; Fri, 14 Dec 2018 23:06:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ECA1E8E0224; Fri, 14 Dec 2018 18:06:04 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E78068E021D; Fri, 14 Dec 2018 18:06:04 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D41C58E0224; Fri, 14 Dec 2018 18:06:04 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 7F1F38E021D for ; Fri, 14 Dec 2018 18:06:04 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id c18so3352390edt.23 for ; Fri, 14 Dec 2018 15:06:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:date:from:to :cc:subject:message-id:references:mime-version:content-disposition :in-reply-to:user-agent; bh=trAgTToOUXQrMMJDw26L4N1jLTSPWeCf0bP1DLO9Oh0=; b=DNJkZ9jb6FgjfSeqYQR197KfbQRSXWLfvhDHHRYu6BNRdBNe4b+1GYA0bXaW0M5Ksg TeSPc5B10+WT4QwJyuGa18I71VuiEONm7XPoYDMRwU5K1R+xjzUd7jC94JsfNmvp1DHs +aWhlwtAUamLjCJCGZkikfbqY+fgeOmag4DCqVcuQ1cee75692zIUUJk1PSyqwvhy2jo lmlNfOYyk6oV8G28LuGcBefIrER2x2tC+sGiT9pTHKKZKGhwu+YNxykrRSTqBj2CKNw5 yQC2pt7GUAG1MDAJmcCeKeOQKmZWc7DgcLPCF8BTGBOw+vTNM74aGiPK3PAJ97qJtcFA CdMg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWbB6JSppw2uRUjAq97/bF1RxOQXsN+yGY0Vownho8sceTPYz2yQ XmWBp7GWZfg0z7AdGYufcpDTOV+/59nmChvdTyzFgvfbqfsaFy6rG94bBJG8KcCuV4jjpu4Pyyx AsTBZDG9h0VB8BMwFqor3hMaZ8odslvuSXvPWZ0FIxP7ymbR5v/mVuXE6mWJ3lWuG0w== X-Received: by 2002:a50:a826:: with SMTP id j35mr4668216edc.230.1544828764034; Fri, 14 Dec 2018 15:06:04 -0800 (PST) X-Google-Smtp-Source: AFSGD/XgyXtbML6xN/afw1f4VaHjGlDB79AYR6Or3VIWO/e9E3lE9/Zpvmhx/Rv2jhSK2gD/y75X X-Received: by 2002:a50:a826:: with SMTP id j35mr4668197edc.230.1544828763195; Fri, 14 Dec 2018 15:06:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544828763; cv=none; d=google.com; s=arc-20160816; b=SGIw2HqkyqkFJRKaRSawDAIUCAMO+2jfNOKejkMpqk9VcL71St3l9JyswRv3CowZwS 12bRkmE2Mg+BdgRTLcjueFNPuPE5MNJ931rLemEEc+8Qhsq0wZpOJZ1O/AYnGglVtC58 LAU0ArY4KVGeVAWLDhvuGxN2A3lp0rsrnJTqd+3DslRsEwQJ7F3QcB8OWPbHpUGL4F5J +UoV183o2YeWYVObsMSYWxOsEx5KIAXRIIrSy0OHJ/2G/g5w0WHgUiQ9GX1rH8uvnJff ctRnQPgbS4SAix//1WrDvcBsmijEDrSkMSG6WF5CJ6g8GIBarLrbi/0taIL19XGAxAjJ xrpg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=user-agent:in-reply-to:content-disposition:mime-version:references :message-id:subject:cc:to:from:date; bh=trAgTToOUXQrMMJDw26L4N1jLTSPWeCf0bP1DLO9Oh0=; b=PjTncBHzoHg4sWq/e5fW60qYoS4K/1uAi0BVU/ffuXpOjUlQzJvoqmkSM8M59CgyBm CQH8NHq3+eNLAefWzYy8DWc2n1vUtXzq43QoFXwvixggPd+iOcyLMdwBUEMpWGVvwwZ4 f/x7T+C/W64rChuG5QnZqdrxUEolqHAKEZvUiViEg9Al+Y50mDCfW+5DhR405N5GgzyS 8G8xSf9FhKBxpoeC8+oUat6dwZTXmuZwTkBhCLEDgDeUKkyzJ1cU5gDyopIonjKholpF 9Eeacxvf4NkV+XuEBGLUra7eb3gySAFn5l3cWZUWKQWBdIvHoa2PDdCGds0RrgPq4vrc 5pdw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp10.blacknight.com (outbound-smtp10.blacknight.com. [46.22.139.15]) by mx.google.com with ESMTPS id l22si1221132edj.93.2018.12.14.15.06.03 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 14 Dec 2018 15:06:03 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) client-ip=46.22.139.15; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail01.blacknight.ie [81.17.254.10]) by outbound-smtp10.blacknight.com (Postfix) with ESMTPS id D19B71C1DFE for ; Fri, 14 Dec 2018 23:06:02 +0000 (GMT) Received: (qmail 3420 invoked from network); 14 Dec 2018 23:06:02 -0000 Received: from unknown (HELO techsingularity.net) (mgorman@techsingularity.net@[37.228.245.71]) by 81.17.254.9 with ESMTPSA (AES256-SHA encrypted, authenticated); 14 Dec 2018 23:06:02 -0000 Date: Fri, 14 Dec 2018 23:06:01 +0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Linus Torvalds , Michal Hocko , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing Subject: [PATCH 14/14] mm, compaction: Do not direct compact remote memory Message-ID: <20181214230601.GE29005@techsingularity.net> References: <20181214230310.572-1-mgorman@techsingularity.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20181214230310.572-1-mgorman@techsingularity.net> User-Agent: Mutt/1.10.1 (2018-07-13) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Remote compaction is expensive and possibly counter-productive. Locality is expected to often have better performance characteristics than remote high-order pages. For small allocations, it's expected that locality is generally required or fallbacks are possible. For larger allocations such as THP, they are forbidden at the time of writing but if __GFP_THISNODE is ever removed, then it would still be preferable to fallback to small local base pages over remote THP in the general case. kcompactd is still woken via kswapd so compaction happens eventually. Signed-off-by: Mel Gorman --- mm/compaction.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/mm/compaction.c b/mm/compaction.c index 39d33b6d1172..05fecd7227e4 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2208,6 +2208,16 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, continue; } + /* + * Do not compact remote memory. It's expensive and high-order + * small allocations are expected to prefer or require local + * memory. Similarly, larger requests such as THP can fallback + * to base pages in preference to remote huge pages if + * __GFP_THISNODE is not specified + */ + if (zone_to_nid(zone) != zone_to_nid(ac->preferred_zoneref->zone)) + continue; + status = compact_zone_order(zone, order, gfp_mask, prio, alloc_flags, ac_classzone_idx(ac), capture); rc = max(status, rc);