From patchwork Tue Jun 14 08:16:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 12880648 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6632FCCA47A for ; Tue, 14 Jun 2022 08:17:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 01BDA8D022D; Tue, 14 Jun 2022 04:17:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F0E418D0229; Tue, 14 Jun 2022 04:17:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD5BD8D022D; Tue, 14 Jun 2022 04:17:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CE4588D0229 for ; Tue, 14 Jun 2022 04:17:21 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 9BF7360DDF for ; Tue, 14 Jun 2022 08:17:21 +0000 (UTC) X-FDA: 79576136682.26.190ABE6 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf31.hostedemail.com (Postfix) with ESMTP id 7AC30200AE for ; Tue, 14 Jun 2022 08:17:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655194640; x=1686730640; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=4QG2XiVbJqz/xC9LjuotKkfae+D+n4VDh6QA+EdAYC0=; b=kr7teayzS0v+SIySHkca5n8/GyeVS6JD9pAXmdVZ/GMb65v3zl9VqWlA KHiNzph14H050iQXYq0RWCuQ56zq5ojjLXh5womc8N/we29i5PE6oEW4o m2Ig4FQwH2QbjTOTfsCGbWxKqBdlV1hTBhse1oQYY2/W+yDo7HSU/U1VX e93R4bkPhKjaXHTZJWztI2Ej4XcUCnGwfiNA1QHbRivzrjgAJK2OnXXoK gOVMWFIG8+yHBBRLs7uMhOh39Puzj4J0abdzkaqSnWbjRH8mmWOcAiyJ5 UMkm7/kwtf4kRh0DHuSn80AwX1B4UPBk7rsYhKWTmzy3iaNe7cZEvKRBP g==; X-IronPort-AV: E=McAfee;i="6400,9594,10377"; a="303954712" X-IronPort-AV: E=Sophos;i="5.91,299,1647327600"; d="scan'208";a="303954712" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2022 01:17:20 -0700 X-IronPort-AV: E=Sophos;i="5.91,299,1647327600"; d="scan'208";a="830274985" Received: from unknown (HELO yhuang6-mobl1.ccr.corp.intel.com) ([10.254.215.153]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2022 01:17:16 -0700 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Johannes Weiner , Michal Hocko , Rik van Riel , Mel Gorman , Peter Zijlstra , Dave Hansen , Yang Shi , Zi Yan , Wei Xu , osalvador , Shakeel Butt Subject: [PATCH -V3 3/3] memory tiering: adjust hot threshold automatically Date: Tue, 14 Jun 2022 16:16:35 +0800 Message-Id: <20220614081635.194014-4-ying.huang@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220614081635.194014-1-ying.huang@intel.com> References: <20220614081635.194014-1-ying.huang@intel.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655194640; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BDS2QS5Mse3BtUPgAKBhOcUBW6Jr9Ks0lz1PZU0nlnk=; b=WQ/ryEfDSyNLZUYFbx41kK+rC6djCBTcRTee3Q2ItCijZUZyNo7IVcCERcDh9PFCLoet0o XboRkiWQlrxAeusd32bFYtuG+hs24A5Q/iC6xAZgph5LwZHZNctCkHhkcWMXKs3cWKCQz0 XNhibSngoGpeVEerUbxs8NAiLlhxGPM= ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=kr7teayz; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf31.hostedemail.com: domain of ying.huang@intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=ying.huang@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655194640; a=rsa-sha256; cv=none; b=4doLKTzqFlFsefwpyr+fYXV5pw2JxJzC/Un3PL2gAZUPFVTQSB3yX+E23NgdJ5vHyfqik1 hkbSQM8832GHY4PUlmxU9s3GfUpwiQtv+2+OIw2iDsnJs9YKP80vVfu5Xfcz1332cKQZWs uG1w1dvp0YxX0b1TaTupmMKjOFYej/8= X-Stat-Signature: j1sq9whhori5hypotewf1pbsswbkmhf8 X-Rspamd-Queue-Id: 7AC30200AE X-Rspam-User: X-Rspamd-Server: rspam05 Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=kr7teayz; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf31.hostedemail.com: domain of ying.huang@intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=ying.huang@intel.com X-HE-Tag: 1655194640-31364 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The promotion hot threshold is workload and system configuration dependent. So in this patch, a method to adjust the hot threshold automatically is implemented. The basic idea is to control the number of the candidate promotion pages to match the promotion rate limit. If the hint page fault latency of a page is less than the hot threshold, we will try to promote the page, and the page is called the candidate promotion page. If the number of the candidate promotion pages in the statistics interval is much more than the promotion rate limit, the hot threshold will be decreased to reduce the number of the candidate promotion pages. Otherwise, the hot threshold will be increased to increase the number of the candidate promotion pages. To make the above method works, in each statistics interval, the total number of the pages to check (on which the hint page faults occur) and the hot/cold distribution need to be stable. Because the page tables are scanned linearly in NUMA balancing, but the hot/cold distribution isn't uniform along the address usually, the statistics interval should be larger than the NUMA balancing scan period. So in the patch, the max scan period is used as statistics interval and it works well in our tests. Signed-off-by: "Huang, Ying" Cc: Andrew Morton Cc: Johannes Weiner Cc: Michal Hocko Cc: Rik van Riel Cc: Mel Gorman Cc: Peter Zijlstra Cc: Dave Hansen Cc: Yang Shi Cc: Zi Yan Cc: Wei Xu Cc: osalvador Cc: Shakeel Butt Cc: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org --- include/linux/mmzone.h | 9 +++++++++ kernel/sched/core.c | 14 +++++++++++++ kernel/sched/fair.c | 46 +++++++++++++++++++++++++++++++++++++----- 3 files changed, 64 insertions(+), 5 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 994a0cd39595..33d875d23e9a 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -918,6 +918,15 @@ typedef struct pglist_data { unsigned int nbp_rl_start; /* number of promote candidate pages at start time of current rate limit period */ unsigned long nbp_rl_nr_cand; + /* promote threshold in ms */ + unsigned int nbp_threshold; + /* start time in ms of current promote threshold adjustment period */ + unsigned int nbp_th_start; + /* + * number of promote candidate pages at stat time of current promote + * threshold adjustment period + */ + unsigned long nbp_th_nr_cand; #endif /* Fields commonly accessed by the page reclaim scanner */ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index bfa7452ca92e..6f9c7a4f647f 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4361,6 +4361,17 @@ void set_numabalancing_state(bool enabled) } #ifdef CONFIG_PROC_SYSCTL +static void reset_memory_tiering(void) +{ + struct pglist_data *pgdat; + + for_each_online_pgdat(pgdat) { + pgdat->nbp_threshold = 0; + pgdat->nbp_th_nr_cand = node_page_state(pgdat, PGPROMOTE_CANDIDATE); + pgdat->nbp_th_start = jiffies_to_msecs(jiffies); + } +} + int sysctl_numa_balancing(struct ctl_table *table, int write, void *buffer, size_t *lenp, loff_t *ppos) { @@ -4377,6 +4388,9 @@ int sysctl_numa_balancing(struct ctl_table *table, int write, if (err < 0) return err; if (write) { + if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) && + (state & NUMA_BALANCING_MEMORY_TIERING)) + reset_memory_tiering(); sysctl_numa_balancing_mode = state; __set_numabalancing_state(state); } diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d779a91a8ca0..cc5b26fefae8 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1503,6 +1503,35 @@ static bool numa_promotion_rate_limit(struct pglist_data *pgdat, return false; } +#define NUMA_MIGRATION_ADJUST_STEPS 16 + +static void numa_promotion_adjust_threshold(struct pglist_data *pgdat, + unsigned long rate_limit, + unsigned int ref_th) +{ + unsigned int now, start, th_period, unit_th, th; + unsigned long nr_cand, ref_cand, diff_cand; + + now = jiffies_to_msecs(jiffies); + th_period = sysctl_numa_balancing_scan_period_max; + start = pgdat->nbp_th_start; + if (now - start > th_period && + cmpxchg(&pgdat->nbp_th_start, start, now) == start) { + ref_cand = rate_limit * + sysctl_numa_balancing_scan_period_max / MSEC_PER_SEC; + nr_cand = node_page_state(pgdat, PGPROMOTE_CANDIDATE); + diff_cand = nr_cand - pgdat->nbp_th_nr_cand; + unit_th = ref_th * 2 / NUMA_MIGRATION_ADJUST_STEPS; + th = pgdat->nbp_threshold ? : ref_th; + if (diff_cand > ref_cand * 11 / 10) + th = max(th - unit_th, unit_th); + else if (diff_cand < ref_cand * 9 / 10) + th = min(th + unit_th, ref_th * 2); + pgdat->nbp_th_nr_cand = nr_cand; + pgdat->nbp_threshold = th; + } +} + bool should_numa_migrate_memory(struct task_struct *p, struct page * page, int src_nid, int dst_cpu) { @@ -1517,19 +1546,26 @@ bool should_numa_migrate_memory(struct task_struct *p, struct page * page, if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && !node_is_toptier(src_nid)) { struct pglist_data *pgdat; - unsigned long rate_limit, latency, th; + unsigned long rate_limit; + unsigned int latency, th, def_th; pgdat = NODE_DATA(dst_nid); - if (pgdat_free_space_enough(pgdat)) + if (pgdat_free_space_enough(pgdat)) { + /* workload changed, reset hot threshold */ + pgdat->nbp_threshold = 0; return true; + } + + def_th = sysctl_numa_balancing_hot_threshold; + rate_limit = sysctl_numa_balancing_promote_rate_limit << \ + (20 - PAGE_SHIFT); + numa_promotion_adjust_threshold(pgdat, rate_limit, def_th); - th = sysctl_numa_balancing_hot_threshold; + th = pgdat->nbp_threshold ? : def_th; latency = numa_hint_fault_latency(page); if (latency >= th) return false; - rate_limit = sysctl_numa_balancing_promote_rate_limit << \ - (20 - PAGE_SHIFT); return !numa_promotion_rate_limit(pgdat, rate_limit, thp_nr_pages(page)); }