From patchwork Fri Apr 8 07:12:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 12806182 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CAEAC433EF for ; Fri, 8 Apr 2022 07:13:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B79396B0075; Fri, 8 Apr 2022 03:12:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B51438D0002; Fri, 8 Apr 2022 03:12:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A188C8D0001; Fri, 8 Apr 2022 03:12:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.28]) by kanga.kvack.org (Postfix) with ESMTP id 93A586B0075 for ; Fri, 8 Apr 2022 03:12:59 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 6A34E94D for ; Fri, 8 Apr 2022 07:12:59 +0000 (UTC) X-FDA: 79332844878.07.73A4D4D Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by imf24.hostedemail.com (Postfix) with ESMTP id 61E3C180002 for ; Fri, 8 Apr 2022 07:12:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1649401978; x=1680937978; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=AE7SAJ6AeNcsbMBtxxvdlJ0UXLadL45KpsV0PlrSAtg=; b=DgUR2UI+O/t5eU8U6JrMuTgd5t4FFcR2aKjMB95k2edEeshJbkOjK37r tnyY4UXO2wmjjh2q2KxMqokDLHrJwpgCFvK7TQM3Q/iq8ke3cWD/+aImt kKJbWYZipAJwPVJc6qE3EyNs2AlC+K2B0i5fN66RhA6qJ/O9IYzexe1KH z7JxVsgG1GOHvUwU4Y69goCGsdDCmyiXA16EFvtVpJzHOGDMC4z4cHnae gzOZgsjAnMR8DP3Mq+sCOqSb/zULysp9LeOnbnnl7PI4MB0vEj8ZYb2mJ GrfEhasXUya2FOk/p40P0u1MNZoRuEGTl1T9tnTFgurq2MC6E2YErexNj w==; X-IronPort-AV: E=McAfee;i="6400,9594,10310"; a="259129796" X-IronPort-AV: E=Sophos;i="5.90,244,1643702400"; d="scan'208";a="259129796" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Apr 2022 00:12:57 -0700 X-IronPort-AV: E=Sophos;i="5.90,244,1643702400"; d="scan'208";a="550403980" Received: from fangyaxu-mobl.ccr.corp.intel.com (HELO yhuang6-mobl1.ccr.corp.intel.com) ([10.254.214.217]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Apr 2022 00:12:54 -0700 From: Huang Ying To: Peter Zijlstra , Mel Gorman , Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Michal Hocko , Rik van Riel , Dave Hansen , Yang Shi , Zi Yan , Wei Xu , osalvador , Shakeel Butt Subject: [PATCH 3/3] memory tiering: adjust hot threshold automatically Date: Fri, 8 Apr 2022 15:12:22 +0800 Message-Id: <20220408071222.219689-4-ying.huang@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220408071222.219689-1-ying.huang@intel.com> References: <20220408071222.219689-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 61E3C180002 X-Stat-Signature: fadu6nnqwb88sthm574guiwdno5exyuc Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=DgUR2UI+; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf24.hostedemail.com: domain of ying.huang@intel.com has no SPF policy when checking 192.55.52.93) smtp.mailfrom=ying.huang@intel.com X-Rspam-User: X-HE-Tag: 1649401978-144930 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The promotion hot threshold may be workload dependent. So in this patch, a method to adjust the hot threshold automatically is implemented. The basic idea is to control the number of the candidate promotion pages to match the promotion rate limit. If the hint page fault latency of a page is less than the hot threshold, we will try to promote the page, and the page is called the candidate promotion page. If the number of the candidate promotion pages in the statistics interval is much more than the promotion rate limit, the hot threshold will be decreased to reduce the number of the candidate promotion pages. Otherwise, the hot threshold will be increased to increase the number of the candidate promotion pages. To make the above method works, in each statistics interval, the total number of the pages to check (on which the hint page faults occur) and the hot/cold distribution need to be stable. Because the page tables are scanned linearly in NUMA balancing, but the hot/cold distribution isn't uniform along the address usually, the statistics interval should be larger than the NUMA balancing scan period. So in the patch, the max scan period is used as statistics interval and it works well in our tests. Signed-off-by: "Huang, Ying" Cc: Andrew Morton Cc: Michal Hocko Cc: Rik van Riel Cc: Mel Gorman Cc: Peter Zijlstra Cc: Dave Hansen Cc: Yang Shi Cc: Zi Yan Cc: Wei Xu Cc: osalvador Cc: Shakeel Butt Cc: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org --- include/linux/mmzone.h | 3 ++ include/linux/sched/sysctl.h | 2 ++ kernel/sched/core.c | 15 +++++++++ kernel/sched/fair.c | 62 +++++++++++++++++++++++++++++++++--- 4 files changed, 78 insertions(+), 4 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index e9b4767619bc..4abf1af20372 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -919,6 +919,9 @@ typedef struct pglist_data { #ifdef CONFIG_NUMA_BALANCING unsigned long numa_ts; unsigned long numa_nr_candidate; + unsigned long numa_threshold_ts; + unsigned long numa_threshold_nr_candidate; + unsigned long numa_threshold; #endif /* Fields commonly accessed by the page reclaim scanner */ diff --git a/include/linux/sched/sysctl.h b/include/linux/sched/sysctl.h index b25ce13e646a..58c1937ba09f 100644 --- a/include/linux/sched/sysctl.h +++ b/include/linux/sched/sysctl.h @@ -67,6 +67,8 @@ int sysctl_sched_uclamp_handler(struct ctl_table *table, int write, void *buffer, size_t *lenp, loff_t *ppos); int sysctl_numa_balancing(struct ctl_table *table, int write, void *buffer, size_t *lenp, loff_t *ppos); +int sysctl_numa_balancing_threshold(struct ctl_table *table, int write, void *buffer, + size_t *lenp, loff_t *ppos); int sysctl_schedstats(struct ctl_table *table, int write, void *buffer, size_t *lenp, loff_t *ppos); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index d575b4914925..179cc4c16e24 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4364,6 +4364,18 @@ void set_numabalancing_state(bool enabled) } #ifdef CONFIG_PROC_SYSCTL +static void reset_memory_tiering(void) +{ + struct pglist_data *pgdat; + + for_each_online_pgdat(pgdat) { + pgdat->numa_threshold = 0; + pgdat->numa_threshold_nr_candidate = + node_page_state(pgdat, PGPROMOTE_CANDIDATE); + pgdat->numa_threshold_ts = jiffies; + } +} + int sysctl_numa_balancing(struct ctl_table *table, int write, void *buffer, size_t *lenp, loff_t *ppos) { @@ -4380,6 +4392,9 @@ int sysctl_numa_balancing(struct ctl_table *table, int write, if (err < 0) return err; if (write) { + if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) && + (state & NUMA_BALANCING_MEMORY_TIERING)) + reset_memory_tiering(); sysctl_numa_balancing_mode = state; __set_numabalancing_state(state); } diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index a1e7cbfb2647..a01216d95bd7 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1456,6 +1456,54 @@ static bool numa_migration_check_rate_limit(struct pglist_data *pgdat, return true; } +int sysctl_numa_balancing_threshold(struct ctl_table *table, int write, void *buffer, + size_t *lenp, loff_t *ppos) +{ + int err; + struct pglist_data *pgdat; + + if (write && !capable(CAP_SYS_ADMIN)) + return -EPERM; + + err = proc_dointvec_minmax(table, write, buffer, lenp, ppos); + if (err < 0 || !write) + return err; + + for_each_online_pgdat(pgdat) + pgdat->numa_threshold = 0; + + return err; +} + +#define NUMA_MIGRATION_ADJUST_STEPS 16 + +static void numa_migration_adjust_threshold(struct pglist_data *pgdat, + unsigned long rate_limit, + unsigned long ref_th) +{ + unsigned long now = jiffies, last_th_ts, th_period; + unsigned long unit_th, th; + unsigned long nr_cand, ref_cand, diff_cand; + + th_period = msecs_to_jiffies(sysctl_numa_balancing_scan_period_max); + last_th_ts = pgdat->numa_threshold_ts; + if (now > last_th_ts + th_period && + cmpxchg(&pgdat->numa_threshold_ts, last_th_ts, now) == last_th_ts) { + ref_cand = rate_limit * + sysctl_numa_balancing_scan_period_max / 1000; + nr_cand = node_page_state(pgdat, PGPROMOTE_CANDIDATE); + diff_cand = nr_cand - pgdat->numa_threshold_nr_candidate; + unit_th = ref_th / NUMA_MIGRATION_ADJUST_STEPS; + th = pgdat->numa_threshold ? : ref_th; + if (diff_cand > ref_cand * 11 / 10) + th = max(th - unit_th, unit_th); + else if (diff_cand < ref_cand * 9 / 10) + th = min(th + unit_th, ref_th); + pgdat->numa_threshold_nr_candidate = nr_cand; + pgdat->numa_threshold = th; + } +} + bool should_numa_migrate_memory(struct task_struct *p, struct page * page, int src_nid, int dst_cpu) { @@ -1470,18 +1518,24 @@ bool should_numa_migrate_memory(struct task_struct *p, struct page * page, if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && !node_is_toptier(src_nid)) { struct pglist_data *pgdat; - unsigned long rate_limit, latency, th; + unsigned long rate_limit, latency, th, def_th; pgdat = NODE_DATA(dst_nid); - if (pgdat_free_space_enough(pgdat)) + if (pgdat_free_space_enough(pgdat)) { + /* workload changed, reset hot threshold */ + pgdat->numa_threshold = 0; return true; + } + + def_th = sysctl_numa_balancing_hot_threshold; + rate_limit = sysctl_numa_balancing_rate_limit << (20 - PAGE_SHIFT); + numa_migration_adjust_threshold(pgdat, rate_limit, def_th); - th = sysctl_numa_balancing_hot_threshold; + th = pgdat->numa_threshold ? : def_th; latency = numa_hint_fault_latency(page); if (latency >= th) return false; - rate_limit = sysctl_numa_balancing_rate_limit << (20 - PAGE_SHIFT); return numa_migration_check_rate_limit(pgdat, rate_limit, thp_nr_pages(page)); }