From patchwork Thu Apr 18 14:20:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peng Zhang X-Patchwork-Id: 13634846 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0C13C4345F for ; Thu, 18 Apr 2024 14:20:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6741F6B007B; Thu, 18 Apr 2024 10:20:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5FD0B6B0089; Thu, 18 Apr 2024 10:20:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 49EBC6B008C; Thu, 18 Apr 2024 10:20:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2969D6B0089 for ; Thu, 18 Apr 2024 10:20:25 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D3578A1E82 for ; Thu, 18 Apr 2024 14:20:24 +0000 (UTC) X-FDA: 82022862768.25.F9A588C Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by imf14.hostedemail.com (Postfix) with ESMTP id E51A4100017 for ; Thu, 18 Apr 2024 14:20:21 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf14.hostedemail.com: domain of zhangpeng362@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=zhangpeng362@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713450023; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references; bh=m4RG0H4rIGMZQlEyfzt8y0Vv87T4UILB5We9+7p5cMs=; b=t7YvEnsAIEQqM1k5Lx7IRPK5AjfmJbhAbF7B5RWffzGRsoF568YWRHsLSkNy4v3wifqNTG Crj7g5JheyjC5jKqHbzcloVe3VBNFWeLfhQ9JJv5ruV1hxeW7hK7TTlHOVY0SCcIQpwgWZ qs4zcyyUnDbKtyiLKa9cSfEUF5ebqA4= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf14.hostedemail.com: domain of zhangpeng362@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=zhangpeng362@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713450023; a=rsa-sha256; cv=none; b=3Wtw0zdWxl5TVWQYkNmdrUsj6Wj39dD/y+I3TJ9W/+kPghgu+CssFXLhNlffBGlecWaTez EOU+LNZHs/US8oOgGqeNEDyBOpn5wDXqDX2npwijKUbo4VKlTHxR4NzlEBPoP5E1RNWhu+ fSqkThP0Ea86kGwjWQvsdAY5bQ1qPls= Received: from mail.maildlp.com (unknown [172.19.163.48]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4VL0GK16f1zwSgx; Thu, 18 Apr 2024 22:17:13 +0800 (CST) Received: from kwepemm600020.china.huawei.com (unknown [7.193.23.147]) by mail.maildlp.com (Postfix) with ESMTPS id 7F5B6180073; Thu, 18 Apr 2024 22:20:17 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by kwepemm600020.china.huawei.com (7.193.23.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Thu, 18 Apr 2024 22:20:16 +0800 From: Peng Zhang To: , CC: , , , , , , , , , , , , Subject: [RFC PATCH v2 0/2] mm: convert mm's rss stats to use atomic mode Date: Thu, 18 Apr 2024 22:20:06 +0800 Message-ID: <20240418142008.2775308-1-zhangpeng362@huawei.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To kwepemm600020.china.huawei.com (7.193.23.147) X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: E51A4100017 X-Stat-Signature: kna3hrcishbnw6u1a5kzr6jc4wms7ofx X-HE-Tag: 1713450021-996151 X-HE-Meta: U2FsdGVkX1+Bc52viOMc6+G3IBdcPO1DGZlihT6/+xKeK/MNj6enVSiVRt4bSpG1TRgPcite77f0niQ+Oyn80hj9/Xi+lroGGUivoUid+Yul8jpQIloTkpTqGeYTqp9dn+1NIcwrdgpQrXNatk2vvl4Wqev+MWwICFxin23fBgrKNVKeppju9zvZ6K5siYZGTLKaIgtAK5QdNK9Mv2F8z2WFbY8lNfI/rTmYlJiqCvo1k2Z2r6u45I37dy2Jd9RzY+dQvLdp6qKxdt3krh6DGWIbWkESQbETgylKrQdA7r+FtcnJKPDGR8yzgeQU/wLUdppdimwLg5koHxFNqwPwAyfpEdezTlswz5CZvD1nuD8j2yzuMNmXjHFGeCgjMDlw/PXroxGiR3sfM18R+C4HeZBzlQ6IuLfCFFDksw7rUeTZvCYT3XCJG8IHPD+pxBVhhvoqL+dSMw9+vC1t/AC9AQ1KjDch8+uWbQX2vOK19nW5YbSIcoNimiB895aFlRayUsPXcah10TR8pGB5wufNAbJfdCsMYD4ZwbCGP1Ws77uD2nB7FYKu0doz8vdS+kBJS9VFTceom7/GO1F3aOK2roz1/vSlG9vKAvJSQCdm8Tbw2uHDF5FfpDTUY1hDNozzWtQDTozZ0blm3XjGtjWZUYnqoBiSkHh750h71JYX+9QCz9cm/0l7Dw4Z8qSqsUGko0IEQAs8eefPYkhRB/ZCTUaMKnCGtd5MlQXs+FH7GE7hxl5+CeaoeyszG2x7OL4D5gtDxMUc36Q7vjA1cnYN8KTgHuTPctvb6/xuARnzxbbPNLpS3K1iZMcGl5W+g2d2x7VdPQ9oV7iq2fqmV0n5Ly1n932PKkNNK1BO3OjLCdxU2qECsHVoLXJ6SxN3EnRGzAuwv8iEAMLvWsc6Pw5qZwXsOzBq0danto4Qs5/UOD663nu6aeoOSXUxcToVk2NEVcDW5b/0wRP7Jvy2qIH GyFvfHrs 4/KaHFQ07iFNihxTrXJkDxLh9q1zJIx76iDj+FKydZlY+CSmsk4O4t3fzuIYAJXAVFz8gxfSd/5nbj1+xIGrbqafQh2DvjOZZcsF6F7Q9JoM1Phoo/ym3k6XSLeDSmAtQRyt6GAIcK9pEqnfEjK7ROUJTbMkY3V+7apB4Kq94o0c11UoeW2uSewONXnK8+P7QZYB+bcigXIU7JsI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: ZhangPeng Since commit f1a7941243c1 ("mm: convert mm's rss stats into percpu_counter"), the rss_stats have converted into percpu_counter, which convert the error margin from (nr_threads * 64) to approximately (nr_cpus ^ 2). However, the new percpu allocation in mm_init() causes a performance regression on fork/exec/shell. Even after commit 14ef95be6f55 ("kernel/fork: group allocation/free of per-cpu counters for mm struct"), the performance of fork/exec/shell is still poor compared to previous kernel versions. To mitigate performance regression, we delay the allocation of percpu memory for rss_stats. Therefore, we convert mm's rss stats to use percpu_counter atomic mode. For single-thread processes, rss_stat is in atomic mode, which reduces the memory consumption and performance regression caused by using percpu. For multiple-thread processes, rss_stat is switched to the percpu mode to reduce the error margin. We convert rss_stats from atomic mode to percpu mode only when the second thread is created. After lmbench test, we can get 2% ~ 4% performance improvement for lmbench fork_proc/exec_proc/shell_proc and 6.7% performance improvement for lmbench page_fault (before batch mode[1]). The test results are as follows: base base+revert base+this patch fork_proc 416.3ms 400.0ms (3.9%) 398.6ms (4.2%) exec_proc 2095.9ms 2061.1ms (1.7%) 2047.7ms (2.3%) shell_proc 3028.2ms 2954.7ms (2.4%) 2961.2ms (2.2%) page_fault 0.3603ms 0.3358ms (6.8%) 0.3361ms (6.7%) [1] https://lore.kernel.org/all/20240412064751.119015-1-wangkefeng.wang@huawei.com/ ChangeLog: v2->v1: - Convert rss_stats from atomic mode to percpu mode only when the second thread is created per Jan Kara. - Compared with v1, the performance data may be different due to different test machines. ZhangPeng (2): percpu_counter: introduce atomic mode for percpu_counter mm: convert mm's rss stats to use atomic mode include/linux/mm.h | 50 +++++++++++++++++++++++++++++----- include/linux/percpu_counter.h | 43 +++++++++++++++++++++++++++-- include/trace/events/kmem.h | 4 +-- kernel/fork.c | 18 +++++++----- lib/percpu_counter.c | 31 +++++++++++++++++++-- 5 files changed, 125 insertions(+), 21 deletions(-)