From patchwork Mon Oct 1 10:05:25 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10621771 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2D09015A6 for ; Mon, 1 Oct 2018 10:05:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0F1AB28911 for ; Mon, 1 Oct 2018 10:05:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0385D29495; Mon, 1 Oct 2018 10:05:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 89A4D28911 for ; Mon, 1 Oct 2018 10:05:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C5E9B6B0266; Mon, 1 Oct 2018 06:05:29 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BE4086B0269; Mon, 1 Oct 2018 06:05:29 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A867F6B026A; Mon, 1 Oct 2018 06:05:29 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by kanga.kvack.org (Postfix) with ESMTP id 457AC6B0266 for ; Mon, 1 Oct 2018 06:05:29 -0400 (EDT) Received: by mail-ed1-f70.google.com with SMTP id w44-v6so9943113edb.16 for ; Mon, 01 Oct 2018 03:05:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=Kp/+LsFBT1b8hPAW2TSnSE48fvnqhLyqpNyfD0ym8M0=; b=LurMyrQKr0q9wBAbf+p2eUve0sY191JIcCCvjTYOglLZQg0ERrDlqBHJLr/2ABF+Qi hIDo2905xNSLRB1I/kFfClAhpuHGzCAHDQfkPaZCnIVz6rKK6Lztlux/Rc3XT2Ty4p4J vaBqRwFxa0i8OF2UYl7iA7RxNhu2x5bdLPfc48ZPHD4vcT9ZicC5vgpmMAOm9D25nUXj HVSexYh3MSRCNyubS1bx0MNY3DH5hNsfJuE0hsoG4XCH/Tuvu9A1CvO6zy9LXnbYlsTG HuKwXfBW9G6PF+crk0ioa0VZn7vQ/13sCV45QrZNudepXJW0dE/pFmhWHJilDor1YbkP RrZA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: ABuFfohFcXNrPbdMMaIeUAB+UuhUaWaJXK03BoDGAp5oCwVzf7Zctl5C xRbs13aYN/m7QDXowBX2XUUyUyf8kv+hjYHyXMfOqaN7vyaaI0y5hZ44JgmHt6GD4Hl7pifx5OB 0DG03oxfQvOxiboIyiWuvh7s2eLkOPeGZ/k5Li8DHTrqjkGR/JseG1qINdGxLWoRtBg== X-Received: by 2002:a50:fa8b:: with SMTP id w11-v6mr16639831edr.59.1538388328769; Mon, 01 Oct 2018 03:05:28 -0700 (PDT) X-Google-Smtp-Source: ACcGV63gEcr4a8PMmTLIpnTeve8Fvis7QRn8nB/XQwxlm7HbPJwWVmApZubK7irLnwqjzlcK5hZp X-Received: by 2002:a50:fa8b:: with SMTP id w11-v6mr16639669edr.59.1538388327001; Mon, 01 Oct 2018 03:05:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1538388326; cv=none; d=google.com; s=arc-20160816; b=fO2IzeGiwmNzIgRQpEHOLrKhVTLtncwtcQMz26YSqmU39rVNcJ2GI1TBcmx7uaGvFz g+nNnGzXqMKt1SL8TrEQmoJnKI4unCZ6gqHrEID87I0WoBot+jNwjsttIehe3piFy8/t hSI/RYPHQ2n3EtIZ/u3vp58SHk7kQAV/5zRLjAvrPkZUx6L8NFaVAq5PCm9pjhw0T4ld gh+xre8xNktd8/sJAQ5lu0eauvjpR2mH9eJLPncFy/a5NnhsrZKm1ad5/yJO4ZNmspyn brUnaMYkKs0u7F+4nk4G0oIcQSmt0KSQyGxXH69BngLX+sv/S1cth0ed9/6oDyaTQqH7 wraw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=Kp/+LsFBT1b8hPAW2TSnSE48fvnqhLyqpNyfD0ym8M0=; b=eAhzi74IMfPFrxL+Fc3Q2sXON44ETR6G/dGxZWkgaHnft5mTSM7ic679rdc5l0ZNmO grLWJQPk4eMZLk0AlIzSBSNhkAxgNGZKJcvfHrGbRw5gQNwTy+UXYOP6DqHsfff6HzK4 bRqxA7m0QLsqSsvPszn19VFtqFkGWXeOG0UJcys+EQh7PGJPBA3udONxazzeGRERrw5u 3pELx1T1LLHnl+AseMoQiGW0WJCgBnrt3babToYrp8W2l61CkecS+MbkwoLd9DRBNrZL MTNbuT3l2iSUs9b26KPLBePYtkhdDCQnBhPupeG+LTUBSlWy4wCcIsXHjPSNBdNOGqa/ RuLg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp08.blacknight.com (outbound-smtp08.blacknight.com. [46.22.139.13]) by mx.google.com with ESMTPS id q48-v6si1379674eda.413.2018.10.01.03.05.26 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 01 Oct 2018 03:05:26 -0700 (PDT) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) client-ip=46.22.139.13; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail04.blacknight.ie [81.17.254.17]) by outbound-smtp08.blacknight.com (Postfix) with ESMTPS id 8F2931C289E for ; Mon, 1 Oct 2018 11:05:26 +0100 (IST) Received: (qmail 4703 invoked from network); 1 Oct 2018 10:05:26 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.88]) by 81.17.254.9 with ESMTPA; 1 Oct 2018 10:05:26 -0000 From: Mel Gorman To: Peter Zijlstra Cc: Ingo Molnar , Srikar Dronamraju , Jirka Hladky , Rik van Riel , LKML , Linux-MM , Mel Gorman Subject: [PATCH 2/2] mm, numa: Migrate pages to local nodes quicker early in the lifetime of a task Date: Mon, 1 Oct 2018 11:05:25 +0100 Message-Id: <20181001100525.29789-3-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20181001100525.29789-1-mgorman@techsingularity.net> References: <20181001100525.29789-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Automatic NUMA Balancing uses a multi-stage pass to decide whether a page should migrate to a local node. This filter avoids excessive ping-ponging if a page is shared or used by threads that migrate cross-node frequently. Threads inherit both page tables and the preferred node ID from the parent. This means that threads can trigger hinting faults earlier than a new task which delays scanning for a number of seconds. As it can be load balanced very early in its lifetime there can be an unnecessary delay before it starts migrating thread-local data. This patch migrates private pages faster early in the lifetime of a thread using the sequence counter as an identifier of new tasks. With this patch applied, STREAM performance is the same as 4.17 even though processes are not spread cross-node prematurely. Other workloads showed a mix of minor gains and losses. This is somewhat expected most workloads are not very sensitive to the starting conditions of a process. 4.19.0-rc5 4.19.0-rc5 4.17.0 numab-v1r1 fastmigrate-v1r1 vanilla MB/sec copy 43298.52 ( 0.00%) 47335.46 ( 9.32%) 47219.24 ( 9.06%) MB/sec scale 30115.06 ( 0.00%) 32568.12 ( 8.15%) 32527.56 ( 8.01%) MB/sec add 32825.12 ( 0.00%) 36078.94 ( 9.91%) 35928.02 ( 9.45%) MB/sec triad 32549.52 ( 0.00%) 35935.94 ( 10.40%) 35969.88 ( 10.51%) Signed-off-by: Mel Gorman Reviewed-by: Rik van Riel --- kernel/sched/fair.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 25c7c7e09cbd..7fc4a371bdd2 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1392,6 +1392,17 @@ bool should_numa_migrate_memory(struct task_struct *p, struct page * page, int last_cpupid, this_cpupid; this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid); + last_cpupid = page_cpupid_xchg_last(page, this_cpupid); + + /* + * Allow first faults or private faults to migrate immediately early in + * the lifetime of a task. The magic number 4 is based on waiting for + * two full passes of the "multi-stage node selection" test that is + * executed below. + */ + if ((p->numa_preferred_nid == -1 || p->numa_scan_seq <= 4) && + (cpupid_pid_unset(last_cpupid) || cpupid_match_pid(p, last_cpupid))) + return true; /* * Multi-stage node selection is used in conjunction with a periodic @@ -1410,7 +1421,6 @@ bool should_numa_migrate_memory(struct task_struct *p, struct page * page, * This quadric squishes small probabilities, making it less likely we * act on an unlikely task<->page relation. */ - last_cpupid = page_cpupid_xchg_last(page, this_cpupid); if (!cpupid_pid_unset(last_cpupid) && cpupid_to_nid(last_cpupid) != dst_nid) return false;