From patchwork Wed Jun 22 08:35:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 12890310 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 452A6CCA47E for ; Wed, 22 Jun 2022 08:35:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9866B8E0096; Wed, 22 Jun 2022 04:35:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 936148E008A; Wed, 22 Jun 2022 04:35:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D7268E0096; Wed, 22 Jun 2022 04:35:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6CF678E008A for ; Wed, 22 Jun 2022 04:35:53 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 48858613B1 for ; Wed, 22 Jun 2022 08:35:53 +0000 (UTC) X-FDA: 79605213786.12.43C8B81 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf16.hostedemail.com (Postfix) with ESMTP id B16B5180003 for ; Wed, 22 Jun 2022 08:35:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655886950; x=1687422950; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=fGU5qVYnZ44PhRhKnwGKDPYjJ6o5iYYEzvHLjp7lw+k=; b=EF7Av4xKP0hZ8vmku+wTY8tZQNT0sZaO79OpGlAGY3SnwDGH8YrS/uks bdJgH7E60wmVuvCG5FkOMYZ5lT/PDmnxqm2A7TyMGGr1Vw2Shr2VLgpWt NF0VHvgjqPzhtg1Zb3EWajvTW2Vd+6tkPSv8+ogfSH1BxKGT3QYW8Jpih Fus1Zl2J0iWJpq1x46NAIUHvABD6w65/qCaECHvd4xrBa1F7/OXuP4AsN 5so+xvS7Q/nCo8erZFbATx/7DjyHbnVr0bmnrraxSBaqKe2cwm8x8r8P/ SyogcKPPU4YY9xwcSIkOyrlpkG7h40DfVONw68m4WVBsii8c9e8bRUM/a Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10385"; a="342039898" X-IronPort-AV: E=Sophos;i="5.92,212,1650956400"; d="scan'208";a="342039898" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Jun 2022 01:35:42 -0700 X-IronPort-AV: E=Sophos;i="5.92,212,1650956400"; d="scan'208";a="644077005" Received: from lzha111-mobl.ccr.corp.intel.com (HELO yhuang6-mobl1.ccr.corp.intel.com) ([10.254.215.232]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Jun 2022 01:35:38 -0700 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Baolin Wang , Johannes Weiner , Michal Hocko , Rik van Riel , Mel Gorman , Peter Zijlstra , Dave Hansen , Yang Shi , Zi Yan , Wei Xu , osalvador , Shakeel Butt , Zhong Jiang Subject: [PATCH -V4 3/3] memory tiering: adjust hot threshold automatically Date: Wed, 22 Jun 2022 16:35:19 +0800 Message-Id: <20220622083519.708236-4-ying.huang@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220622083519.708236-1-ying.huang@intel.com> References: <20220622083519.708236-1-ying.huang@intel.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655886951; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3DfLjYPhDCx8rrwACiKz1LTd7tUqhIK6flv+MIAarp0=; b=NIZWPGkYzWDd4cCkDI7woBMMTuuwkQyxT0sqLphT2a2CIHO+gilMd3YCvuDGh1pDdDPt6v dOBUKxttV6yMatGoIAPXIwdsBZUS96+bQj8RwZlyS3uo1byBq/n9IWTpIamT7B9cFPb+YQ xcwo5+dvyg1GL6sScUMbb420oKHHHUA= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=EF7Av4xK; spf=none (imf16.hostedemail.com: domain of ying.huang@intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655886951; a=rsa-sha256; cv=none; b=UGGJG6GG60Ks4JEqsBh/B14oIvZx0H2n/1XAzYh05ZzPgOyIiGOHcQ8zKQIQqRSYBzTm5j IonPvhm6QcdmeflVnqofPuCnD75CYcmeCfDGZB0h353Hl0OC6XIUECu3LW3Hf0U+aY7eo/ 6T12BVQGxe83Taf4xf6nPYFnMOpyV34= X-Rspamd-Queue-Id: B16B5180003 Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=EF7Av4xK; spf=none (imf16.hostedemail.com: domain of ying.huang@intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com X-Rspam-User: X-Stat-Signature: b7iseas613t6cufeh98eyjz1bo6dhs9r X-Rspamd-Server: rspam04 X-HE-Tag: 1655886950-180005 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The promotion hot threshold is workload and system configuration dependent. So in this patch, a method to adjust the hot threshold automatically is implemented. The basic idea is to control the number of the candidate promotion pages to match the promotion rate limit. If the hint page fault latency of a page is less than the hot threshold, we will try to promote the page, and the page is called the candidate promotion page. If the number of the candidate promotion pages in the statistics interval is much more than the promotion rate limit, the hot threshold will be decreased to reduce the number of the candidate promotion pages. Otherwise, the hot threshold will be increased to increase the number of the candidate promotion pages. To make the above method works, in each statistics interval, the total number of the pages to check (on which the hint page faults occur) and the hot/cold distribution need to be stable. Because the page tables are scanned linearly in NUMA balancing, but the hot/cold distribution isn't uniform along the address usually, the statistics interval should be larger than the NUMA balancing scan period. So in the patch, the max scan period is used as statistics interval and it works well in our tests. Signed-off-by: "Huang, Ying" Reviewed-by: Baolin Wang Tested-by: Baolin Wang Cc: Andrew Morton Cc: Johannes Weiner Cc: Michal Hocko Cc: Rik van Riel Cc: Mel Gorman Cc: Peter Zijlstra Cc: Dave Hansen Cc: Yang Shi Cc: Zi Yan Cc: Wei Xu Cc: osalvador Cc: Shakeel Butt Cc: Zhong Jiang Cc: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org --- include/linux/mmzone.h | 9 +++++++++ kernel/sched/core.c | 14 +++++++++++++ kernel/sched/fair.c | 46 +++++++++++++++++++++++++++++++++++++----- 3 files changed, 64 insertions(+), 5 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 994a0cd39595..33d875d23e9a 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -918,6 +918,15 @@ typedef struct pglist_data { unsigned int nbp_rl_start; /* number of promote candidate pages at start time of current rate limit period */ unsigned long nbp_rl_nr_cand; + /* promote threshold in ms */ + unsigned int nbp_threshold; + /* start time in ms of current promote threshold adjustment period */ + unsigned int nbp_th_start; + /* + * number of promote candidate pages at stat time of current promote + * threshold adjustment period + */ + unsigned long nbp_th_nr_cand; #endif /* Fields commonly accessed by the page reclaim scanner */ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index da0bf6fe9ecd..2183a368d4b0 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4361,6 +4361,17 @@ void set_numabalancing_state(bool enabled) } #ifdef CONFIG_PROC_SYSCTL +static void reset_memory_tiering(void) +{ + struct pglist_data *pgdat; + + for_each_online_pgdat(pgdat) { + pgdat->nbp_threshold = 0; + pgdat->nbp_th_nr_cand = node_page_state(pgdat, PGPROMOTE_CANDIDATE); + pgdat->nbp_th_start = jiffies_to_msecs(jiffies); + } +} + int sysctl_numa_balancing(struct ctl_table *table, int write, void *buffer, size_t *lenp, loff_t *ppos) { @@ -4377,6 +4388,9 @@ int sysctl_numa_balancing(struct ctl_table *table, int write, if (err < 0) return err; if (write) { + if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) && + (state & NUMA_BALANCING_MEMORY_TIERING)) + reset_memory_tiering(); sysctl_numa_balancing_mode = state; __set_numabalancing_state(state); } diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d779a91a8ca0..cc5b26fefae8 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1503,6 +1503,35 @@ static bool numa_promotion_rate_limit(struct pglist_data *pgdat, return false; } +#define NUMA_MIGRATION_ADJUST_STEPS 16 + +static void numa_promotion_adjust_threshold(struct pglist_data *pgdat, + unsigned long rate_limit, + unsigned int ref_th) +{ + unsigned int now, start, th_period, unit_th, th; + unsigned long nr_cand, ref_cand, diff_cand; + + now = jiffies_to_msecs(jiffies); + th_period = sysctl_numa_balancing_scan_period_max; + start = pgdat->nbp_th_start; + if (now - start > th_period && + cmpxchg(&pgdat->nbp_th_start, start, now) == start) { + ref_cand = rate_limit * + sysctl_numa_balancing_scan_period_max / MSEC_PER_SEC; + nr_cand = node_page_state(pgdat, PGPROMOTE_CANDIDATE); + diff_cand = nr_cand - pgdat->nbp_th_nr_cand; + unit_th = ref_th * 2 / NUMA_MIGRATION_ADJUST_STEPS; + th = pgdat->nbp_threshold ? : ref_th; + if (diff_cand > ref_cand * 11 / 10) + th = max(th - unit_th, unit_th); + else if (diff_cand < ref_cand * 9 / 10) + th = min(th + unit_th, ref_th * 2); + pgdat->nbp_th_nr_cand = nr_cand; + pgdat->nbp_threshold = th; + } +} + bool should_numa_migrate_memory(struct task_struct *p, struct page * page, int src_nid, int dst_cpu) { @@ -1517,19 +1546,26 @@ bool should_numa_migrate_memory(struct task_struct *p, struct page * page, if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && !node_is_toptier(src_nid)) { struct pglist_data *pgdat; - unsigned long rate_limit, latency, th; + unsigned long rate_limit; + unsigned int latency, th, def_th; pgdat = NODE_DATA(dst_nid); - if (pgdat_free_space_enough(pgdat)) + if (pgdat_free_space_enough(pgdat)) { + /* workload changed, reset hot threshold */ + pgdat->nbp_threshold = 0; return true; + } + + def_th = sysctl_numa_balancing_hot_threshold; + rate_limit = sysctl_numa_balancing_promote_rate_limit << \ + (20 - PAGE_SHIFT); + numa_promotion_adjust_threshold(pgdat, rate_limit, def_th); - th = sysctl_numa_balancing_hot_threshold; + th = pgdat->nbp_threshold ? : def_th; latency = numa_hint_fault_latency(page); if (latency >= th) return false; - rate_limit = sysctl_numa_balancing_promote_rate_limit << \ - (20 - PAGE_SHIFT); return !numa_promotion_rate_limit(pgdat, rate_limit, thp_nr_pages(page)); }