From patchwork Fri Jun 16 18:07:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Michal_Koutn=C3=BD?= X-Patchwork-Id: 13283096 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12E2AEB64D7 for ; Fri, 16 Jun 2023 18:07:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 47AAA8E0003; Fri, 16 Jun 2023 14:07:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 429398E0001; Fri, 16 Jun 2023 14:07:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2CB028E0003; Fri, 16 Jun 2023 14:07:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 180288E0001 for ; Fri, 16 Jun 2023 14:07:27 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D942840D0C for ; Fri, 16 Jun 2023 18:07:26 +0000 (UTC) X-FDA: 80909393292.24.A97118C Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf17.hostedemail.com (Postfix) with ESMTP id E909E40009 for ; Fri, 16 Jun 2023 18:07:23 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=QvJrVkKi; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf17.hostedemail.com: domain of mkoutny@suse.com designates 195.135.220.28 as permitted sender) smtp.mailfrom=mkoutny@suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686938844; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=Bm9Vw5H6yGrwn4tTzKc5q8OJHix81Wz2+IxzfBcuFEw=; b=mYF6Qw1LG5Rh5TiMbTWfsTD32lbUdqgh5/Pdlls6uP8S8bYxHhRqU4we0uDQg6VLWVqiak JM69YSBFMlqXgXBk5gcQMyKLrHf6XIKwrI71a67QBvo1tUa/T4yM99koN1PuF5zFUnS6bu j4t085nnBX+pRG5LxIjxvwf4wZIn+rA= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=QvJrVkKi; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf17.hostedemail.com: domain of mkoutny@suse.com designates 195.135.220.28 as permitted sender) smtp.mailfrom=mkoutny@suse.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686938844; a=rsa-sha256; cv=none; b=aku4pRwupKDUtR+Bc091cdSPy17PX1/EQiW7NvP5ZkC3L5lM+ZzslHpdxOEo1Tl+FrEUw0 OzRShq13INtqBwL9NOLRkYslqJqSUgSMtzXHxOWIhLT+UhRDCKePpAJOeXic5BeWCjPDPA ReqrWr8/AZuXpUo84lFCKqY0KvfsQYc= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 40FE321C3A; Fri, 16 Jun 2023 18:07:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1686938842; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Bm9Vw5H6yGrwn4tTzKc5q8OJHix81Wz2+IxzfBcuFEw=; b=QvJrVkKiH+XeJGuNdMS5Su60VmPHay4I0XeSzYlltz06ro5xEuwX9jSff4IG+wmDRNut+D OivGqOuBREkZic7P0vGnuwg5cTFXMI3L5WNfJPwPmJG17y4vT90o2NHyiw865vSOinYWt1 w4OyzF3V7B1D4MiwQ+P/KfBIbxvm8xc= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id EA487138E8; Fri, 16 Jun 2023 18:07:21 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 1gwSONmkjGSTPwAAMHmgww (envelope-from ); Fri, 16 Jun 2023 18:07:21 +0000 From: =?utf-8?q?Michal_Koutn=C3=BD?= To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , Christian Brauner , Suren Baghdasaryan , "Liam R . Howlett" , Mike Christie , Mathieu Desnoyers , Peter Zijlstra , Nicholas Piggin , Andrei Vagin , Shakeel Butt , Adam Majer , Jan Kara , Michal Hocko Subject: [PATCH] mm: Sync percpu mm RSS counters before querying Date: Fri, 16 Jun 2023 20:07:18 +0200 Message-ID: <20230616180718.17725-1-mkoutny@suse.com> X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 X-Rspamd-Queue-Id: E909E40009 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: xrhj53oib8knfti8f981xj8awh7u3xmp X-HE-Tag: 1686938843-762919 X-HE-Meta: U2FsdGVkX19MXaUfaBPWg/vAueQLrjZ4wwezgLRs33mioLiEeZ7VLX2FFzIo/FxZeCUK6GHWSVT3+8LijHlsFWdsDwjM0hiYmwh+Z0IYJXpa1zOiKn5c/UNh2WvrqsZMTauB87Jp5YccZw8GF5A4pgvZDfXPjOpWVrC6p9zPD3f0zixMqlsN9akGh6cXlUvaKhphstkWBVBhgg5boS0Je+GQibujSfxa9Unc3FewcjIVnAQDNVQIPy8p0Zz/UwkJ9UAvwX1UjoWImJk8JB9iYsqEhPAUQVUEE/xP05aFEygfpDobH1pG7GW+/hnU68lmIpakayvR1MQmhkt9AJLBfFYizG2zS13JESOhdsLoQIaY+V6ojs0xDObrRc6TUtHV06z+dW0H8nn2fHLtjqTYd+yRIVdPFV5j3xbEI7ByZL8O1ldrDSStXrkTjcdlzZXaeob4Fk8JMwJumaQxfUY9sg7+4EGg6vQVWn38xMrutvciUtEAXG4C0WinFSIvGGH5Pwuc0xxsyP/Zhcfnze7oRRxrThSz7jBevqoZbBwlJ47ZPnmS54OrLPBGynCqTXD+9Y7Rn1NDuyJoevCbWOzqwbV6AKPHwlBh7G4htaJNPW7XmVyC0nHbkHUmwcd6fpIt7JyPXiUCjGyi8PnV32l6gtycesfXUgKPfmLq1lMGCk/0sGqyuU0Ih1wbO3jWUWbdpwTaDLyJLXAhhoKaO/OV0V62KykPA6yH0angmuxWv5VCO6KVfYxCk3DExlms8fyzvP0X5O57FDJVgsyxSJI7OWdc+kEbp9kjsHw3+bxX5YuZJK2/LvA5ygN1NbDNGQjVdparFhk6sELKBwAZkgg8x5mK1ut0UamiIV2bSOx+bU+hAUUtVMqkA2bpwhcS3R1fBXAgTuyGtuXZKjJbbbRRybL2VhX18BMYi95pFubj9juAi57JZozOW1Pb9GwW+8Hi+bABHaY8iZOFhQquFTv Y5Oc8Zp0 fAhJgDN2+jCv+CGJbuJZosj4tGP7Ej6cLOUCIkbRu4gBP3mDEIvVOXbw8yNbonGvyLbRcdqPBO4zknTREGBya+wZwT8Od0/YcoGThpphc2MGHqeRAmp7ICjQTFkFLxMZNhuzK0OMsWzqmkhY0un9oGyUJYIa+TeDDgz3Hm51Tyba17lfrX0NsZFuhu21LIm4LEgpOcv/E6itr4ZUMJ/KWGV0e8Gzy7hPmZss0439IqO5rHFwd27D27/LvK/lwQLj3AmB1tptUASXYcZSbmPrjAM1d37yKpX2HBOlYghgjnNiB17PH6zMbrCe9waFbPRSvNJBj9kvsnfcd/sg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: An issue was observed with stats collected in struct rusage on ppc64le with 64kB pages. The percpu counters use batching with percpu_counter_batch = max(32, nr*2) # in PAGE_SIZE i.e. with larger pages but similar RSS consumption (bytes), there'll be less flushes and error more noticeable. In this given case (getting consumption of exited child), we can request percpu counter's flush without worrying about contention with updaters. Fortunately, the commit f1a7941243c1 ("mm: convert mm's rss stats into percpu_counter") didn't eradicate all traces of SPLIT_RSS_COUNTING and this mechanism already provided some synchronization points before reading stats. Therefore, use sync_mm_rss as carrier for percpu counters refreshes and forget SPLIT_RSS_COUNTING macro for good. Impact of summing on a 8 CPU machine: Benchmark 1: taskset -c 1 ./shell-bench.sh Before Time (mean ± σ): 9.950 s ± 0.052 s [User: 7.773 s, System: 2.023 s] After Time (mean ± σ): 9.990 s ± 0.070 s [User: 7.825 s, System: 2.011 s] cat >shell-bench.sh < Signed-off-by: Michal Koutný --- Notes: - dummy approach to fix correctness, performance should be treated separately - add percpu_counter_set to really update the value - RFC https://lore.kernel.org/r/20230608171256.17827-1-mkoutny@suse.com/ include/linux/mm.h | 7 +++---- kernel/fork.c | 4 ---- 2 files changed, 3 insertions(+), 8 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 27ce77080c79..c7ac1cbc6fb3 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2547,13 +2547,12 @@ static inline void setmax_mm_hiwater_rss(unsigned long *maxrss, *maxrss = hiwater_rss; } -#if defined(SPLIT_RSS_COUNTING) -void sync_mm_rss(struct mm_struct *mm); -#else static inline void sync_mm_rss(struct mm_struct *mm) { + for (int i = 0; i < NR_MM_COUNTERS; ++i) + percpu_counter_set(&mm->rss_stat[i], + percpu_counter_sum(&mm->rss_stat[i])); } -#endif #ifndef CONFIG_ARCH_HAS_PTE_SPECIAL static inline int pte_special(pte_t pte) diff --git a/kernel/fork.c b/kernel/fork.c index 81cba91f30bb..e030eb902e4b 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -2412,10 +2412,6 @@ __latent_entropy struct task_struct *copy_process( p->io_uring = NULL; #endif -#if defined(SPLIT_RSS_COUNTING) - memset(&p->rss_stat, 0, sizeof(p->rss_stat)); -#endif - p->default_timer_slack_ns = current->timer_slack_ns; #ifdef CONFIG_PSI