From patchwork Sat Sep 24 15:22:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aaron Tomlin X-Patchwork-Id: 12987726 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07DFAC07E9D for ; Sat, 24 Sep 2022 15:22:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 554C08E0009; Sat, 24 Sep 2022 11:22:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 502CB8E0007; Sat, 24 Sep 2022 11:22:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3567A8E0009; Sat, 24 Sep 2022 11:22:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 219A48E0007 for ; Sat, 24 Sep 2022 11:22:40 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id F20CD160EE8 for ; Sat, 24 Sep 2022 15:22:39 +0000 (UTC) X-FDA: 79947346038.20.1078604 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 6E52F2000E for ; Sat, 24 Sep 2022 15:22:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1664032958; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TJy6J71aJA3IuazPVpZvOJ+o0HTfRS9wHinNTI5//yw=; b=LKhyTp7KeZbz1zbj1VPQo27peVVbEbBMgUUxhtdEatQm0oEMt+ooaQ1maomwp5xOewi5i0 z5g8cmmOhPvupqS9iwiOS/mV/BHrF9HRJETqX3joIeyB8KKyxo4QovU4Eu8/cFvfVrdT/1 i89aCGVJIfYRycI6PFroh7+O0AYuEIw= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-282-eYzHn1zlOcKITVqMQXxLQQ-1; Sat, 24 Sep 2022 11:22:36 -0400 X-MC-Unique: eYzHn1zlOcKITVqMQXxLQQ-1 Received: by mail-wm1-f69.google.com with SMTP id p36-20020a05600c1da400b003b4faefa2b9so1629249wms.6 for ; Sat, 24 Sep 2022 08:22:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=TJy6J71aJA3IuazPVpZvOJ+o0HTfRS9wHinNTI5//yw=; b=B6Q+dreccql9k7pHINyoND27sMqAZ5EeXkdtKTW9WrVlUTijtPBEoV2kkmxe/7w7BH GdyKE1j+nQbTPAyYqewrDHz5fOzGzbEJTJ1PrlT2RE1lgaNsdgz6BGMVBMfJWwFF4GUQ DRmqyu902NZ0f2KNA9QJL+Bx2+Rxo+ZysjgIDWfzht2sY6hWSh7s8qLGt4/PrH5sMxXH 2iaeMWcpEOQOEHNk1sB9VjkmdVnIBikRyWsIYVfOO8tBLZnCo/myeONaQDrZqOn1b4M+ ofUAO7uog3qVHelwTBZrcaprdskYgnDf/GSLOZSCdoHK1RUoxDPXgsnRoCnlStGUIvb2 3xPw== X-Gm-Message-State: ACrzQf0xNPh8fjYZf794LXsYdI6PP4F5z/HIHJYoMAORcQlyyW+MNVcn Y1CXk39XKCm+ykw1yveICiMvMBXyWYq/dZBNwkQThGnSNaS+dmq1FX+wDl43MR43STgqZamCYjd 5rMQb6lLVkw== X-Received: by 2002:adf:fd47:0:b0:228:dbab:8f5d with SMTP id h7-20020adffd47000000b00228dbab8f5dmr8180137wrs.524.1664032954966; Sat, 24 Sep 2022 08:22:34 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6sU8DZqkudooT7dcV2k2CxDwD6Eg+x8I5ijbDfegC963F4VJd1EaU/hhsShwvAdgEZ3nK3Sg== X-Received: by 2002:adf:fd47:0:b0:228:dbab:8f5d with SMTP id h7-20020adffd47000000b00228dbab8f5dmr8180125wrs.524.1664032954760; Sat, 24 Sep 2022 08:22:34 -0700 (PDT) Received: from localhost (cpc111743-lutn13-2-0-cust979.9-3.cable.virginm.net. [82.17.115.212]) by smtp.gmail.com with ESMTPSA id p21-20020a1c5455000000b003b27f644488sm5396182wmi.29.2022.09.24.08.22.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 24 Sep 2022 08:22:29 -0700 (PDT) From: Aaron Tomlin To: frederic@kernel.org, mtosatti@redhat.com Cc: cl@linux.com, tglx@linutronix.de, mingo@kernel.org, peterz@infradead.org, pauld@redhat.com, neelx@redhat.com, oleksandr@natalenko.name, atomlin@atomlin.com, akpm@linux-foundation.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v8 1/5] mm/vmstat: Add CPU-specific variable to track a vmstat discrepancy Date: Sat, 24 Sep 2022 16:22:23 +0100 Message-Id: <20220924152227.819815-2-atomlin@redhat.com> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220924152227.819815-1-atomlin@redhat.com> References: <20220924152227.819815-1-atomlin@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1664032959; a=rsa-sha256; cv=none; b=rAOsk7AfwSreMo3ys/0gpHYZV6hwEQWLJKFMjqoBtcHAv2aE1VFq5q0CWrg/2eoLhf5Wgk mC3HShbwIjQytSvHjlO/uYKWHA6Fu4/aZPeWxBlJ+eUXODWbAxzqC0oZzpGKQ8rQh9hqkQ OowrWDo3jWwrQKMbDyVwcloCbZO1oKA= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LKhyTp7K; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of atomlin@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=atomlin@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1664032959; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TJy6J71aJA3IuazPVpZvOJ+o0HTfRS9wHinNTI5//yw=; b=PxJ+35+Rav5iZVMeuxNC5UsvslWbM/ZCCyqD/12kcb5OJEu2MPFiPHsCp2GhJvqK7pgLWP kt8KxABB8EyuHkUEtZNiAnIS5dlXx1eBPtI2IqLGDGLcFPkOE9lDhFXPeKWuMaF+uH2aKT 8YKu6h4ZvIQCek4S/nL/XnjisaF7Je4= X-Rspam-User: X-Rspamd-Queue-Id: 6E52F2000E X-Rspamd-Server: rspam08 X-Stat-Signature: gisogsfd6ykzw4p7y6x6ne8sw7mgcsck Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LKhyTp7K; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of atomlin@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=atomlin@redhat.com X-HE-Tag: 1664032959-477343 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce a CPU-specific variable namely vmstat_dirty to indicate if a vmstat imbalance is present for a given CPU. Therefore, at the appropriate time, we can fold all the remaining differentials. This patch also provides trivial helpers for modification and testing. Signed-off-by: Aaron Tomlin --- mm/vmstat.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/mm/vmstat.c b/mm/vmstat.c index 90af9a8572f5..24c67b2e58fd 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -195,6 +195,22 @@ void fold_vm_numa_events(void) #endif #ifdef CONFIG_SMP +static DEFINE_PER_CPU_ALIGNED(bool, vmstat_dirty); + +static inline void vmstat_mark_dirty(void) +{ + this_cpu_write(vmstat_dirty, true); +} + +static inline void vmstat_clear_dirty(void) +{ + this_cpu_write(vmstat_dirty, false); +} + +static inline bool is_vmstat_dirty(void) +{ + return this_cpu_read(vmstat_dirty); +} int calculate_pressure_threshold(struct zone *zone) { From patchwork Sat Sep 24 15:22:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aaron Tomlin X-Patchwork-Id: 12987727 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B76F3C6FA86 for ; Sat, 24 Sep 2022 15:22:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 111CC8E000A; Sat, 24 Sep 2022 11:22:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0734C8E0007; Sat, 24 Sep 2022 11:22:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB6138E000A; Sat, 24 Sep 2022 11:22:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C34A18E0007 for ; Sat, 24 Sep 2022 11:22:40 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 5D265A0E81 for ; Sat, 24 Sep 2022 15:22:40 +0000 (UTC) X-FDA: 79947346080.12.07062DD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 0D3CC180026 for ; Sat, 24 Sep 2022 15:22:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1664032959; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vx+xKpue7wTXyWig6qL4Y2Lw1zImsNhejs2t+QDspOc=; b=iwFL8+gY+xjbmdhR/Rha5jWi6/n4icTwnLan/h8bu+YzhFsZiVlmiULkpyr5NtcxnifK3T +h3XVAXsdkLxnMtyLz1gSMJmKc1M7gTCRU8o+twDHSn8Zs4arj9duVzwzMnZIItFmftmyQ TAeajTlMXizvOuJnKRDo3t/b9h8Jo5w= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-583-eMxxyPVSNXOwbp4KuN0vCA-1; Sat, 24 Sep 2022 11:22:38 -0400 X-MC-Unique: eMxxyPVSNXOwbp4KuN0vCA-1 Received: by mail-wm1-f70.google.com with SMTP id l15-20020a05600c4f0f00b003b4bec80edbso1624025wmq.9 for ; Sat, 24 Sep 2022 08:22:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=vx+xKpue7wTXyWig6qL4Y2Lw1zImsNhejs2t+QDspOc=; b=fRFYTpTQFdXwsOPC4o976UQBPK1WwruxKIVArVi1TP4CGPpvlHLTrBHwbSzgNnxmMk QHzuvXx7s47xxOKqR31bUyZdoL93HUQG/zlHZI2zcrMze1SCcqvaoMsTEegmUvKgt0rF WtTDDHj1INrclg08+jkeKRBQiUrGQxNH9qgRBr+7uoL35qXn+lblzS+XSeGcEJx029yP JRoiPT66Dind/WB+IfT3RynUClIckJMG8Gcld54rQA+Sghp5rTB3+/it2ZTFr/vVKpl9 dFlIrA82+YfAWxBJfzoglX2XEbpwy6l9SGtILccxYbynZhpNPc7hC2HQ+9oC7/DM32HB 5IMA== X-Gm-Message-State: ACrzQf0CQFRTtugH4mXLp9A3fCENlFsJmPQ+ciJX7Mto1c7+3JTxYBVp Mp4bDCyksV0jNkaQ5lYmA0usFp7AVVfPVdOKm5CAm24tT+02EN2ylgZzfUUBc832xh80L/F9pbC xoBiixh5/dQ== X-Received: by 2002:adf:ea86:0:b0:22b:9f9f:a3b with SMTP id s6-20020adfea86000000b0022b9f9f0a3bmr5684016wrm.130.1664032957441; Sat, 24 Sep 2022 08:22:37 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4XlRA/1nNXyboTHo5AUkSzsW77JkBgIWvz5OJ8eCCBl08evcHlFngS43fDNqZ1Sk/si/OyYQ== X-Received: by 2002:adf:ea86:0:b0:22b:9f9f:a3b with SMTP id s6-20020adfea86000000b0022b9f9f0a3bmr5683998wrm.130.1664032957196; Sat, 24 Sep 2022 08:22:37 -0700 (PDT) Received: from localhost (cpc111743-lutn13-2-0-cust979.9-3.cable.virginm.net. [82.17.115.212]) by smtp.gmail.com with ESMTPSA id f11-20020a5d664b000000b0022af63bb6f2sm9701817wrw.113.2022.09.24.08.22.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 24 Sep 2022 08:22:36 -0700 (PDT) From: Aaron Tomlin To: frederic@kernel.org, mtosatti@redhat.com Cc: cl@linux.com, tglx@linutronix.de, mingo@kernel.org, peterz@infradead.org, pauld@redhat.com, neelx@redhat.com, oleksandr@natalenko.name, atomlin@atomlin.com, akpm@linux-foundation.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v8 2/5] mm/vmstat: Use vmstat_dirty to track CPU-specific vmstat discrepancies Date: Sat, 24 Sep 2022 16:22:24 +0100 Message-Id: <20220924152227.819815-3-atomlin@redhat.com> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220924152227.819815-1-atomlin@redhat.com> References: <20220924152227.819815-1-atomlin@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1664032960; a=rsa-sha256; cv=none; b=pfiWJ2n1jAFwo/QY/6vIOBi9mQ6uEEcBURBsjoh/OKWSFdXJx2fuEqw3VXk2ymPap22g4e quEvVXoc9xGqOhwUCvfFe3J+LBbUXgRDZx7M24X/WDZJBLZDxAISbajLTuxmRj2pmA7Tiy +kyxyNceMq7HVXS0rEzJq8+BuxJGRyQ= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=iwFL8+gY; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf06.hostedemail.com: domain of atomlin@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=atomlin@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1664032960; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vx+xKpue7wTXyWig6qL4Y2Lw1zImsNhejs2t+QDspOc=; b=nvrG4pALlZzKY2Ux3a4SP5nJSNEb6RHSUDU6Y2ZmIPBekPueZFC6ybqtB+LnpdrbXtJUOw CGqLOp9R3wbykkOprDzxx+p7XbwiRbBHyokln4Bof6Ezh8enQQsEsipmi5fZ/SHF3gDR+M 3EWes3pTi9AXzaZ56OOfAHXyVD5+iE8= Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=iwFL8+gY; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf06.hostedemail.com: domain of atomlin@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=atomlin@redhat.com X-Rspam-User: X-Stat-Signature: fa3pdfztbih6w84rj97kna41fm13ykjt X-Rspamd-Queue-Id: 0D3CC180026 X-Rspamd-Server: rspam06 X-HE-Tag: 1664032959-692497 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch will now use the previously introduced CPU-specific variable namely vmstat_dirty to indicate if a vmstat differential/or imbalance is present for a given CPU. So, at the appropriate time, vmstat processing can be initiated. The hope is that this particular approach is "cheaper" when compared to need_update(). The idea is based on Marcelo's patch [1]. [1]: https://lore.kernel.org/lkml/20220204173554.763888172@fedora.localdomain/ Signed-off-by: Aaron Tomlin --- mm/vmstat.c | 48 ++++++++++++++---------------------------------- 1 file changed, 14 insertions(+), 34 deletions(-) diff --git a/mm/vmstat.c b/mm/vmstat.c index 24c67b2e58fd..472175642bd9 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -383,6 +383,7 @@ void __mod_zone_page_state(struct zone *zone, enum zone_stat_item item, x = 0; } __this_cpu_write(*p, x); + vmstat_mark_dirty(); if (IS_ENABLED(CONFIG_PREEMPT_RT)) preempt_enable(); @@ -421,6 +422,7 @@ void __mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, x = 0; } __this_cpu_write(*p, x); + vmstat_mark_dirty(); if (IS_ENABLED(CONFIG_PREEMPT_RT)) preempt_enable(); @@ -619,6 +621,7 @@ static inline void mod_zone_state(struct zone *zone, if (z) zone_page_state_add(z, zone, item); + vmstat_mark_dirty(); } void mod_zone_page_state(struct zone *zone, enum zone_stat_item item, @@ -687,6 +690,7 @@ static inline void mod_node_state(struct pglist_data *pgdat, if (z) node_page_state_add(z, pgdat, item); + vmstat_mark_dirty(); } void mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, @@ -841,6 +845,14 @@ static int refresh_cpu_vm_stats(bool do_pagesets) int global_node_diff[NR_VM_NODE_STAT_ITEMS] = { 0, }; int changes = 0; + /* + * Clear vmstat_dirty before clearing the percpu vmstats. + * If interrupts are enabled, it is possible that an interrupt + * or another task modifies a percpu vmstat, which will + * set vmstat_dirty to true. + */ + vmstat_clear_dirty(); + for_each_populated_zone(zone) { struct per_cpu_zonestat __percpu *pzstats = zone->per_cpu_zonestats; #ifdef CONFIG_NUMA @@ -1971,35 +1983,6 @@ static void vmstat_update(struct work_struct *w) } } -/* - * Check if the diffs for a certain cpu indicate that - * an update is needed. - */ -static bool need_update(int cpu) -{ - pg_data_t *last_pgdat = NULL; - struct zone *zone; - - for_each_populated_zone(zone) { - struct per_cpu_zonestat *pzstats = per_cpu_ptr(zone->per_cpu_zonestats, cpu); - struct per_cpu_nodestat *n; - - /* - * The fast way of checking if there are any vmstat diffs. - */ - if (memchr_inv(pzstats->vm_stat_diff, 0, sizeof(pzstats->vm_stat_diff))) - return true; - - if (last_pgdat == zone->zone_pgdat) - continue; - last_pgdat = zone->zone_pgdat; - n = per_cpu_ptr(zone->zone_pgdat->per_cpu_nodestats, cpu); - if (memchr_inv(n->vm_node_stat_diff, 0, sizeof(n->vm_node_stat_diff))) - return true; - } - return false; -} - /* * Switch off vmstat processing and then fold all the remaining differentials * until the diffs stay at zero. The function is used by NOHZ and can only be @@ -2010,10 +1993,7 @@ void quiet_vmstat(void) if (system_state != SYSTEM_RUNNING) return; - if (!delayed_work_pending(this_cpu_ptr(&vmstat_work))) - return; - - if (!need_update(smp_processor_id())) + if (!is_vmstat_dirty()) return; /* @@ -2044,7 +2024,7 @@ static void vmstat_shepherd(struct work_struct *w) for_each_online_cpu(cpu) { struct delayed_work *dw = &per_cpu(vmstat_work, cpu); - if (!delayed_work_pending(dw) && need_update(cpu)) + if (!delayed_work_pending(dw) && per_cpu(vmstat_dirty, cpu)) queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0); cond_resched(); From patchwork Sat Sep 24 15:22:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aaron Tomlin X-Patchwork-Id: 12987729 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CBEAC07E9D for ; Sat, 24 Sep 2022 15:22:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D447E8E000C; Sat, 24 Sep 2022 11:22:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CCC668E0007; Sat, 24 Sep 2022 11:22:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B44D98E000C; Sat, 24 Sep 2022 11:22:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A1FF78E0007 for ; Sat, 24 Sep 2022 11:22:46 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 8ADAE1C6267 for ; Sat, 24 Sep 2022 15:22:46 +0000 (UTC) X-FDA: 79947346332.09.3F22BE3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 343DB18000B for ; Sat, 24 Sep 2022 15:22:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1664032965; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TUeQREelS5Rh9IXzWSL5LmYfttFnlORFlEDP5NM1JJE=; b=gf9zEOe+osUNy6CMBwWvzDDvZD6am4Ek0i+vIBXD7tPFz8Ulxqau4VGvc/XRkpCB5xG764 HgNNucqBS265c2IA2ryPM1vOh8W42oDb1istMleCf0k07a8m0k2USfmsQFh33ihEqt6fTv +nnngur+RecBT2jteC/lZzmZM8rpSIQ= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-649-vIiwMX-GMaGICWZ8dpSCBQ-1; Sat, 24 Sep 2022 11:22:41 -0400 X-MC-Unique: vIiwMX-GMaGICWZ8dpSCBQ-1 Received: by mail-wm1-f72.google.com with SMTP id i129-20020a1c3b87000000b003b33e6160bdso4334818wma.7 for ; Sat, 24 Sep 2022 08:22:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=TUeQREelS5Rh9IXzWSL5LmYfttFnlORFlEDP5NM1JJE=; b=2NPKilgwtUardzFO9Epu+ogAw0S0Gvm7TJem59RQtBxX7RKelCtxYznkuS3nFaG3Nr XrOEx9cD52f0d3UYH6STJUG71GWmZPjV05uV55+gtbc77v+OvWj/21N4IRDmjP0WfRvT SwLnU7yNYdUYQ3xeR2BMD2w5a6wdugZFcbErBmVaf2pvgQ/uUgzJpYC3BHcC1ClaM/2K Ca0qFyHIjcflCe5EmBTcoZERF5gY5K1asBLbiyQgvUjC8da2emMr96z3+DkoLImhEeIw gHbCAJcOQKTn2psvsf8f213Y0TAKWk0T5GLowlViHQe+vSxsoPc4dQwobO20OTZFi2km MkPA== X-Gm-Message-State: ACrzQf1gQ2t20cwF6S0bpmO63xZGGzvWJr2MJJZU9gRc2WnrUf0rj+bv yCc09/UHEtePIwfsvQFvNij4S7AAN3aqXbJ7gfiTvZZ2V1RyyhKrWks9xrbDvZ/89WcLS4Na8/y ZqaysCY6Ecg== X-Received: by 2002:a05:600c:3781:b0:3a6:804a:afc with SMTP id o1-20020a05600c378100b003a6804a0afcmr16215760wmr.27.1664032960277; Sat, 24 Sep 2022 08:22:40 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5m/4w+5hkvjp8DxE5I0zgzzGcCHocqRX8TyR/j15JVMnw/b2kAHfK1Vm5eShxot1oDE/8fJQ== X-Received: by 2002:a05:600c:3781:b0:3a6:804a:afc with SMTP id o1-20020a05600c378100b003a6804a0afcmr16215737wmr.27.1664032960043; Sat, 24 Sep 2022 08:22:40 -0700 (PDT) Received: from localhost (cpc111743-lutn13-2-0-cust979.9-3.cable.virginm.net. [82.17.115.212]) by smtp.gmail.com with ESMTPSA id e19-20020a5d5953000000b00228c792aaaasm9941182wri.100.2022.09.24.08.22.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 24 Sep 2022 08:22:39 -0700 (PDT) From: Aaron Tomlin To: frederic@kernel.org, mtosatti@redhat.com Cc: cl@linux.com, tglx@linutronix.de, mingo@kernel.org, peterz@infradead.org, pauld@redhat.com, neelx@redhat.com, oleksandr@natalenko.name, atomlin@atomlin.com, akpm@linux-foundation.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v8 3/5] mm/vmstat: Do not queue vmstat_update if tick is stopped Date: Sat, 24 Sep 2022 16:22:25 +0100 Message-Id: <20220924152227.819815-4-atomlin@redhat.com> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220924152227.819815-1-atomlin@redhat.com> References: <20220924152227.819815-1-atomlin@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gf9zEOe+; spf=pass (imf06.hostedemail.com: domain of atomlin@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=atomlin@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1664032966; a=rsa-sha256; cv=none; b=cC5EPzkBNyRFWZ8A0pMPCUCurU9wxrBYMkAM6pUqEDqQ//+HaPN8J/yDJEIWOi1UGtOAdE 3qFNfpAoI81PEXJpS4EuwNNsSyr2XUobiAFERGqKRx2rzbCu7sB2Y21uOJxDC9aXB6oW6a 8IO7AHco6+sT+IVa/U9taFfKJuw2zcg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1664032966; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TUeQREelS5Rh9IXzWSL5LmYfttFnlORFlEDP5NM1JJE=; b=HaSJEMbTlWwjCQruZaqfKI5UpAqcGmtoW5uGafU6RhXX06/khZNBnyyeZ57E4wnxUKEKDH f3vVAOLogw8tSUTkBQaMnNcXGXZSXi9WODV7wkpJHMQXQeKykhua4vcaWzapek0pDUFQ0p HN5e2f7x+iRkr+Ueb/eKqkpRk1qvh/c= X-Stat-Signature: fktycxq1pezbcp8bo3djo8i9qmjsw3py X-Rspamd-Queue-Id: 343DB18000B Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gf9zEOe+; spf=pass (imf06.hostedemail.com: domain of atomlin@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=atomlin@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspam-User: X-Rspamd-Server: rspam03 X-HE-Tag: 1664032966-589050 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Marcelo Tosatti From the vmstat shepherd, for CPUs that have the tick stopped, do not queue local work to flush the per-CPU vmstats, since in that case the flush is performed on return to userspace or when entering idle. Also cancel any delayed work on the local CPU, when entering idle on nohz full CPUs. Per-CPU pages can be freed remotely from housekeeping CPUs. Signed-off-by: Marcelo Tosatti --- mm/vmstat.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/mm/vmstat.c b/mm/vmstat.c index 472175642bd9..3b9a497965b4 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -29,6 +29,7 @@ #include #include #include +#include #include "internal.h" @@ -1990,19 +1991,23 @@ static void vmstat_update(struct work_struct *w) */ void quiet_vmstat(void) { + struct delayed_work *dw; + if (system_state != SYSTEM_RUNNING) return; if (!is_vmstat_dirty()) return; + refresh_cpu_vm_stats(false); + /* - * Just refresh counters and do not care about the pending delayed - * vmstat_update. It doesn't fire that often to matter and canceling - * it would be too expensive from this path. - * vmstat_shepherd will take care about that for us. + * If the tick is stopped, cancel any delayed work to avoid + * interruptions to this CPU in the future. */ - refresh_cpu_vm_stats(false); + dw = &per_cpu(vmstat_work, smp_processor_id()); + if (delayed_work_pending(dw) && tick_nohz_tick_stopped()) + cancel_delayed_work(dw); } /* @@ -2024,6 +2029,9 @@ static void vmstat_shepherd(struct work_struct *w) for_each_online_cpu(cpu) { struct delayed_work *dw = &per_cpu(vmstat_work, cpu); + if (tick_nohz_tick_stopped_cpu(cpu)) + continue; + if (!delayed_work_pending(dw) && per_cpu(vmstat_dirty, cpu)) queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0); From patchwork Sat Sep 24 15:22:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aaron Tomlin X-Patchwork-Id: 12987728 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34FEBC32771 for ; Sat, 24 Sep 2022 15:22:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7EBF58E000B; Sat, 24 Sep 2022 11:22:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 79B0D8E0007; Sat, 24 Sep 2022 11:22:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 63C178E000B; Sat, 24 Sep 2022 11:22:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 517358E0007 for ; Sat, 24 Sep 2022 11:22:46 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 244A340854 for ; Sat, 24 Sep 2022 15:22:46 +0000 (UTC) X-FDA: 79947346332.15.338A156 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf17.hostedemail.com (Postfix) with ESMTP id A5AE84000A for ; Sat, 24 Sep 2022 15:22:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1664032965; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FfsmTvmGXeM1/5qFA1Qh7h37ne6tkdVL6kPv6MNEep4=; b=P4rzTqCevCxROlEU4wsc+c5YT7ZRCcHzIOsQr+8sg4soEKmWiceSjzqjoDlD+C/c8j50ut xUQb8H8Gn7BE4D2JM29dDnPL1RQf9Bm+Zy1PJiPrnyGHR3t8ivg1/NKXtkCmeQ69ZVDfuP 2698CSW6xFdIdxzYWo5ehW7sTbmxW9o= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-12-9MQ6R1VyMXqtYhZWxmZoag-1; Sat, 24 Sep 2022 11:22:44 -0400 X-MC-Unique: 9MQ6R1VyMXqtYhZWxmZoag-1 Received: by mail-wr1-f70.google.com with SMTP id i26-20020adfa51a000000b0022c8fe3f26cso374840wrb.15 for ; Sat, 24 Sep 2022 08:22:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=FfsmTvmGXeM1/5qFA1Qh7h37ne6tkdVL6kPv6MNEep4=; b=A+gRX8ooR6UeGN3mnVoKV81leTqVeejTHfEc7dQ3wlVbg4hA2aK4z75O6ruxy+tCeD IajIGJOSd4Dai4cJVVBvOaMBmYnYZ8y6jaeWw5lHr43F0paCTCxjwEa73DVlSzzbzhIl gzSgk1VexB4z5dF4MgU3pQpdfTqUNDoAn1fHKK7GkwYLS8ZOKzq5WoGxeF5FODENKJUu Zz5jwLPFKa0NcTf3qyyyb8WideBdSKMfKVXzNB5FuyN0D7W8zvWmmyBJ3paIFgmDnXoV 0+rF+BwIkRgtjSPCpTGND6M/RzBDCx8fhlik6PlGSLeuts9qhYZbyj965zRxDf8P+TZd CbBQ== X-Gm-Message-State: ACrzQf1OXeEssGW1a5SdUQLLPNpej03kpw+nUuq39rgYKYcRDi4WTF+e xiG+Ei1y6MV1EdTKn5bAM1iW34p4kr6bOIjnu0CJ1Cem2tmWwCxuRxgDSK/A2N1ethtkg4I6agR J1RqxhB65eg== X-Received: by 2002:a05:6000:18aa:b0:22a:b61d:877a with SMTP id b10-20020a05600018aa00b0022ab61d877amr8412526wri.512.1664032961876; Sat, 24 Sep 2022 08:22:41 -0700 (PDT) X-Google-Smtp-Source: AMsMyM56wiVnnOEt4nyShWrMqWg2SekETLv7jY1m2hDmQ/noBoVfgCeZVDYazjwtyeDkFVI65YEN6g== X-Received: by 2002:a05:6000:18aa:b0:22a:b61d:877a with SMTP id b10-20020a05600018aa00b0022ab61d877amr8412517wri.512.1664032961709; Sat, 24 Sep 2022 08:22:41 -0700 (PDT) Received: from localhost (cpc111743-lutn13-2-0-cust979.9-3.cable.virginm.net. [82.17.115.212]) by smtp.gmail.com with ESMTPSA id o3-20020a05600c378300b003b48dac344esm5421645wmr.43.2022.09.24.08.22.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 24 Sep 2022 08:22:41 -0700 (PDT) From: Aaron Tomlin To: frederic@kernel.org, mtosatti@redhat.com Cc: cl@linux.com, tglx@linutronix.de, mingo@kernel.org, peterz@infradead.org, pauld@redhat.com, neelx@redhat.com, oleksandr@natalenko.name, atomlin@atomlin.com, akpm@linux-foundation.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v8 4/5] tick/nohz_full: Ensure quiet_vmstat() is called on exit to user-mode when the idle tick is stopped Date: Sat, 24 Sep 2022 16:22:26 +0100 Message-Id: <20220924152227.819815-5-atomlin@redhat.com> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220924152227.819815-1-atomlin@redhat.com> References: <20220924152227.819815-1-atomlin@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1664032965; a=rsa-sha256; cv=none; b=zCx8uW+QShuCdNSoUaF6yMCAZI0MpNERAJoOcr7JRQs+EK/DQU0NrvRkUPENIbBtFn71Vx sXINmLiWbExjrqQjt1x23rSCfwknU6+95Payt5DgXI9vAbyx9bnF4OyLAqLSzaKTtKkVGk ho/qmRS/n8KP0dmMQH8h1BFBow51Blw= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=P4rzTqCe; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of atomlin@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=atomlin@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1664032965; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FfsmTvmGXeM1/5qFA1Qh7h37ne6tkdVL6kPv6MNEep4=; b=UP+eucSZq0q+QDa3CKmtdDlFTsS/HXLT10Ara2UCnhYePDuRMWgYe9eHfzFhLOSjR0cw6h PLYyYzLlJSEl2G3k8qfkdlueIXDlgB5Fp796vwGlCbvpR/05U5aHjUKOUFBJQzZPabvoVz JOimyeChKUihBVYnIIuf+5jHtygYtqY= Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=P4rzTqCe; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of atomlin@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=atomlin@redhat.com X-Rspam-User: X-Stat-Signature: xg4xmeofq5td9aycekhdjhzfcj38zami X-Rspamd-Queue-Id: A5AE84000A X-Rspamd-Server: rspam06 X-HE-Tag: 1664032965-12713 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch ensures CPU-specific vmstat differentials do not remain when the scheduling-tick is stopped and before exiting to user-mode in the context of nohz_full only. A trivial test program was used to determine the impact of the proposed changes and under vanilla. The mlock(2) and munlock(2) system calls was used solely to modify vmstat item 'NR_MLOCK'. The following is an average count of CPU-cycles across the aforementioned system calls: Vanilla Modified Cycles per syscall 8461 8690 (+2.6%) Signed-off-by: Aaron Tomlin --- include/linux/tick.h | 5 +++-- kernel/time/tick-sched.c | 15 +++++++++++++++ 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/include/linux/tick.h b/include/linux/tick.h index bfd571f18cfd..a2bbd6d32e33 100644 --- a/include/linux/tick.h +++ b/include/linux/tick.h @@ -11,7 +11,6 @@ #include #include #include -#include #ifdef CONFIG_GENERIC_CLOCKEVENTS extern void __init tick_init(void); @@ -272,6 +271,7 @@ static inline void tick_dep_clear_signal(struct signal_struct *signal, extern void tick_nohz_full_kick_cpu(int cpu); extern void __tick_nohz_task_switch(void); +void __tick_nohz_user_enter_prepare(void); extern void __init tick_nohz_full_setup(cpumask_var_t cpumask); #else static inline bool tick_nohz_full_enabled(void) { return false; } @@ -296,6 +296,7 @@ static inline void tick_dep_clear_signal(struct signal_struct *signal, static inline void tick_nohz_full_kick_cpu(int cpu) { } static inline void __tick_nohz_task_switch(void) { } +static inline void __tick_nohz_user_enter_prepare(void) { } static inline void tick_nohz_full_setup(cpumask_var_t cpumask) { } #endif @@ -308,7 +309,7 @@ static inline void tick_nohz_task_switch(void) static inline void tick_nohz_user_enter_prepare(void) { if (tick_nohz_full_cpu(smp_processor_id())) - rcu_nocb_flush_deferred_wakeup(); + __tick_nohz_user_enter_prepare(); } #endif diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index b0e3c9205946..634cd0fac267 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -26,6 +26,7 @@ #include #include #include +#include #include @@ -519,6 +520,20 @@ void __tick_nohz_task_switch(void) } } +void __tick_nohz_user_enter_prepare(void) +{ + struct tick_sched *ts; + + if (tick_nohz_full_cpu(smp_processor_id())) { + ts = this_cpu_ptr(&tick_cpu_sched); + + if (ts->tick_stopped) + quiet_vmstat(); + rcu_nocb_flush_deferred_wakeup(); + } +} +EXPORT_SYMBOL_GPL(__tick_nohz_user_enter_prepare); + /* Get the boot-time nohz CPU list from the kernel parameters. */ void __init tick_nohz_full_setup(cpumask_var_t cpumask) { From patchwork Sat Sep 24 15:24:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aaron Tomlin X-Patchwork-Id: 12987730 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54B34C07E9D for ; Sat, 24 Sep 2022 15:24:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E2FF88E000D; Sat, 24 Sep 2022 11:24:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E04428E0007; Sat, 24 Sep 2022 11:24:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA5108E000D; Sat, 24 Sep 2022 11:24:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B72398E0007 for ; Sat, 24 Sep 2022 11:24:46 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 893F1160EFB for ; Sat, 24 Sep 2022 15:24:46 +0000 (UTC) X-FDA: 79947351372.06.5B15559 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf15.hostedemail.com (Postfix) with ESMTP id 1114DA0006 for ; Sat, 24 Sep 2022 15:24:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1664033085; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Bph3AMipjhSnF4iEVnBvZMNWDbHORj0Zn37njSUExec=; b=NrprgsmAFaJcWCj+3U9ZfOr5AJKcj4YPFwqfyo0dCcAHbK2JAMBTJO5PUYwRARnFrNFnT4 jxglDRXRBjSacZTIeFzM8WWswXzHkpGXM44lim7dw6vouAhJVudoQYUeOyK6Fk22PHI9Kr fEsY2ajYAGGsoLsawB5eYJ+grOpFldQ= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-449-KbMNXBfIO_Su4r0ZC4FbzA-1; Sat, 24 Sep 2022 11:24:44 -0400 X-MC-Unique: KbMNXBfIO_Su4r0ZC4FbzA-1 Received: by mail-wm1-f72.google.com with SMTP id 84-20020a1c0257000000b003b4be28d7e3so4365732wmc.0 for ; Sat, 24 Sep 2022 08:24:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=Bph3AMipjhSnF4iEVnBvZMNWDbHORj0Zn37njSUExec=; b=41Y/dfBLvYX0OVEnAkIvm42pPXkVbTr+vlflZcypDH1/TD/VvxViVsK24RumxjiKxr Ew2azK5xwEe3OuPksSuctim+7AWtvufHtQHmqPZe4YCSlQKM/8NDJwksG1gZ/vrFWiUN GJciGoXO2SNJ9lnvTSx9QqRhfrrNA6Gp39jGbEiv9jUDMUSoGNPcbYqheSnLEmjXdW7O RgvLBOCBN5sjc3yePULQjkzPEuearLkTA+VBiUL2xeoG7lU/D65qnt3KAY0CoYi2tbJv ZiY+fO9fpylYPB97uzBPtRE38XOxXZNwfb/a9zY624vQaSfZhtmGeEikAUfGMpt+azuq oF+A== X-Gm-Message-State: ACrzQf3L+ryGGnORPU8gm7bLHWubdqfH2hEkQydL54MhALm4is8yBBZn RXsQQqc+/LUFKapfEW3RcAzQwmq3R3FpD/AkMsXoLuL6KFUbP/08EOYxGPdH0KgAliPwdbtEfhc hHQHn6VhrWw== X-Received: by 2002:a5d:5846:0:b0:22a:f7d2:9045 with SMTP id i6-20020a5d5846000000b0022af7d29045mr8819000wrf.250.1664033083083; Sat, 24 Sep 2022 08:24:43 -0700 (PDT) X-Google-Smtp-Source: AMsMyM70EhfgjAjHBzizhVeJg3h6UFmOlUDSwdahLb7n7DZk1+e242Hq1wNDo3XO+Xfaa3AFFa0HHg== X-Received: by 2002:a5d:5846:0:b0:22a:f7d2:9045 with SMTP id i6-20020a5d5846000000b0022af7d29045mr8818987wrf.250.1664033082825; Sat, 24 Sep 2022 08:24:42 -0700 (PDT) Received: from localhost (cpc111743-lutn13-2-0-cust979.9-3.cable.virginm.net. [82.17.115.212]) by smtp.gmail.com with ESMTPSA id h6-20020a05600c350600b003b491f99a25sm6157684wmq.22.2022.09.24.08.24.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 24 Sep 2022 08:24:41 -0700 (PDT) From: Aaron Tomlin To: frederic@kernel.org, mtosatti@redhat.com Cc: cl@linux.com, tglx@linutronix.de, mingo@kernel.org, peterz@infradead.org, pauld@redhat.com, neelx@redhat.com, oleksandr@natalenko.name, atomlin@atomlin.com, akpm@linux-foundation.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v8 5/5] tick/sched: Ensure quiet_vmstat() is called when the idle tick was stopped too Date: Sat, 24 Sep 2022 16:24:41 +0100 Message-Id: <20220924152441.822460-1-atomlin@redhat.com> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220924152227.819815-1-atomlin@redhat.com> References: <20220924152227.819815-1-atomlin@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1664033086; a=rsa-sha256; cv=none; b=RRe/hWnyBlXe6s0YE6rhMLyprYi4SofBVoP+KpkWqRWijv5jUaJwSfjJJm1fO9/5yCdxI6 1plERSrRqE47NbAgSlqldGN9m9EkOeEDqrXa34fxQR7M+z8eTmBIVdAiDoIwBIJV6LfxKT LIwKatbn1FTJlSWwRRvyr4pyhSVr/AY= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NrprgsmA; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf15.hostedemail.com: domain of atomlin@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=atomlin@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1664033086; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Bph3AMipjhSnF4iEVnBvZMNWDbHORj0Zn37njSUExec=; b=5vTeDdN7E7AFkf0lqbtgET/OoIYWzOcv6O+GSODWulyOiDdcWD8VHrDFTzBp6zYQKdC578 bKonRCQUwZk9wdVzGjewWIotA9JWBLEmZ5nRym7inqwRogkIe91XZeh/5oVWh07hgpLjkz lZeZRDnfxmyEdKM56GBsCg7qPMcSes8= X-Rspamd-Server: rspam11 X-Rspam-User: X-Rspamd-Queue-Id: 1114DA0006 Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NrprgsmA; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf15.hostedemail.com: domain of atomlin@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=atomlin@redhat.com X-Stat-Signature: xxazhsch4ch9ujhg56ecof8bkrz9ah65 X-HE-Tag: 1664033085-301817 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In the context of the idle task and an adaptive-tick mode/or a nohz_full CPU, quiet_vmstat() can be called: before stopping the idle tick, entering an idle state and on exit. In particular, for the latter case, when the idle task is required to reschedule, the idle tick can remain stopped and the timer expiration time endless i.e., KTIME_MAX. Now, indeed before a nohz_full CPU enters an idle state, CPU-specific vmstat counters should be processed to ensure the respective values have been reset and folded into the zone specific 'vm_stat[]'. That being said, it can only occur when: the idle tick was previously stopped, and reprogramming of the timer is not required. A customer provided some evidence which indicates that the idle tick was stopped; albeit, CPU-specific vmstat counters still remained populated. Thus one can only assume quiet_vmstat() was not invoked on return to the idle loop. If I understand correctly, I suspect this divergence might erroneously prevent a reclaim attempt by kswapd. If the number of zone specific free pages are below their per-cpu drift value then zone_page_state_snapshot() is used to compute a more accurate view of the aforementioned statistic. Thus any task blocked on the NUMA node specific pfmemalloc_wait queue will be unable to make significant progress via direct reclaim unless it is killed after being woken up by kswapd (see throttle_direct_reclaim()). Consider the following theoretical scenario: - Note: CPU X is part of 'tick_nohz_full_mask' 1. CPU Y migrated running task A to CPU X that was in an idle state i.e. waiting for an IRQ; marked the current task on CPU X to need/or require a reschedule i.e., set TIF_NEED_RESCHED and invoked a reschedule IPI to CPU X (see sched_move_task()) 2. CPU X acknowledged the reschedule IPI. Generic idle loop code noticed the TIF_NEED_RESCHED flag against the idle task and attempts to exit of the loop and calls the main scheduler function i.e. __schedule(). Since the idle tick was previously stopped no scheduling-clock tick would occur. So, no deferred timers would be handled 3. Post transition to kernel execution Task A running on CPU X, indirectly released a few pages (e.g. see __free_one_page()); CPU X's 'vm_stat_diff[NR_FREE_PAGES]' was updated and zone specific 'vm_stat[]' update was deferred as per the CPU-specific stat threshold 4. Task A does invoke exit(2) and the kernel does remove the task from the run-queue; the idle task was selected to execute next since there are no other runnable tasks assigned to the given CPU (see pick_next_task() and pick_next_task_idle()) 5. On return to the idle loop since the idle tick was already stopped and can remain so (see [1] below) e.g. no pending soft IRQs, no attempt is made to zero and fold CPU X's vmstat counters since reprogramming of the scheduling-clock tick is not required/or needed (see [2]) ... do_idle { __current_set_polling() tick_nohz_idle_enter() while (!need_resched()) { local_irq_disable() ... /* No polling or broadcast event */ cpuidle_idle_call() { if (cpuidle_not_available(drv, dev)) { tick_nohz_idle_stop_tick() __tick_nohz_idle_stop_tick(this_cpu_ptr(&tick_cpu_sched)) { int cpu = smp_processor_id() if (ts->timer_expires_base) expires = ts->timer_expires else if (can_stop_idle_tick(cpu, ts)) (1) -------> expires = tick_nohz_next_event(ts, cpu) else return ts->idle_calls++ if (expires > 0LL) { tick_nohz_stop_tick(ts, cpu) { if (ts->tick_stopped && (expires == ts->next_tick)) { (2) -------> if (tick == KTIME_MAX || ts->next_tick == hrtimer_get_expires(&ts->sched_timer)) return } ... } So, the idea of this patch is to ensure refresh_cpu_vm_stats(false) is called, when it is appropriate, on return to the idle loop if the idle tick was previously stopped too. A trivial test program was used to determine the impact of the proposed changes and under vanilla. The nanosleep(2) system call was used several times to suspend execution for a period of time to approximately compute the number of CPU-cycles in the idle code path. The following is an average count of CPU-cycles: Vanilla Modified Cycles per idle loop 151858 153258 (+1.0%) Signed-off-by: Aaron Tomlin --- kernel/time/tick-sched.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 634cd0fac267..88a3e9fc3824 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -926,13 +926,14 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu) */ if (!ts->tick_stopped) { calc_load_nohz_start(); - quiet_vmstat(); ts->last_tick = hrtimer_get_expires(&ts->sched_timer); ts->tick_stopped = 1; trace_tick_stop(1, TICK_DEP_MASK_NONE); } + /* Attempt to fold when the idle tick is stopped or not */ + quiet_vmstat(); ts->next_tick = tick; /*