From patchwork Mon Oct 16 05:29:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13422487 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3599FC41513 for ; Mon, 16 Oct 2023 05:30:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A8E138D002F; Mon, 16 Oct 2023 01:30:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A3EB38D0001; Mon, 16 Oct 2023 01:30:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8DED08D002F; Mon, 16 Oct 2023 01:30:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7DF378D0001 for ; Mon, 16 Oct 2023 01:30:24 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 3EE731A0A8B for ; Mon, 16 Oct 2023 05:30:24 +0000 (UTC) X-FDA: 81350199168.03.30800DE Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.65]) by imf03.hostedemail.com (Postfix) with ESMTP id 39F9720004 for ; Mon, 16 Oct 2023 05:30:21 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=h6JmN7JV; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf03.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697434222; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=+Fushq8irou3NCVFZiyRg8uGMmGRLe3yRCymTSlf6C0=; b=sz8uukreKE6IvdBhAY97z2cRN+Kiissvw3tdyYhsOvMccrb9z8oxD7DMC9RrzrbTUCbFQx D9iWzMRd4PyalTTv5pzI+NwgnWgcrzF8P4/ybgFdgH/Ot5EOkoHYFtoTj03NaG5duaheyp ipA408SW6z0IrybW8JaYDGJFcKLP7g8= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=h6JmN7JV; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf03.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697434222; a=rsa-sha256; cv=none; b=h5H6xqKdlA1ciTjEWMMCWUY52IgpwzKfBlkLJeDF78GdAU/2ISI0xNS+9xhIFZBtGA2QB4 umq4goACzW9lSdFsCDwUMMscYy99yd/FvgqKdbnmi90s0ds1+92BhZiT7mbRzMbGQzMmRK PKS8hExlGJiIztEZB2R5gTIh5O1kHQ4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697434222; x=1728970222; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=xJgWFzff+uqaalunn6Rh1CxrexTGRP3XuKZ2zgaIiRI=; b=h6JmN7JVBFoAMqXnzpesOtej6frE/shMBMGxZtVrENm36IUAHfhW8JpJ oH3oYcL3yTTz1OXk84bqM0xeq0MBaGqdzqaXyGBnwDoldMuP3JdgdvRUV 2rnVkz54wydtK1rwFlSDBFRgR7eAovYHh5JFKxPBe+N8zjk5sVKAQvDbx 5hgP2xliWlDjsr7fge0kra5vOjhAENd4HWuf65DOU6NwX/jw/P34MZkBU FLptKTle8ZR41O8+Pt/Nt2nGocUYElpF/bmNtC2OLcCmI/bVxXjKISYet uABXLN7AzHdU41zDEhWzKHzMUD6GteYLR9AQSJ3a7kJv0JKVAjP7n/+4B w==; X-IronPort-AV: E=McAfee;i="6600,9927,10863"; a="389307913" X-IronPort-AV: E=Sophos;i="6.03,228,1694761200"; d="scan'208";a="389307913" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2023 22:30:20 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10863"; a="899356623" X-IronPort-AV: E=Sophos;i="6.03,228,1694761200"; d="scan'208";a="899356623" Received: from yhuang6-mobl2.sh.intel.com ([10.238.6.133]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2023 22:28:18 -0700 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Arjan Van De Ven , Huang Ying , Mel Gorman , Vlastimil Babka , David Hildenbrand , Johannes Weiner , Dave Hansen , Michal Hocko , Pavel Tatashin , Matthew Wilcox , Christoph Lameter Subject: [PATCH -V3 0/9] mm: PCP high auto-tuning Date: Mon, 16 Oct 2023 13:29:53 +0800 Message-Id: <20231016053002.756205-1-ying.huang@intel.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 X-Rspamd-Queue-Id: 39F9720004 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 5pyg518uia5q1tt5zddpz4a8wti1cjhe X-HE-Tag: 1697434221-294817 X-HE-Meta: U2FsdGVkX1+lefo951r6a7vvGEAsl3wlYaT7KWSo+z2gFg3ryHPwaGPBbdt4R75pxoPRS9XvwAUsK/WQsU8C5UXIuVxz3Lnf6MNnrWbHyNf3n7cSsjHG9oCROfmqyvo70Iglwxey/GLOOv0dglNzAx6LY0liK1z4wu2iTFulqdz73/QnKiuYfBk72rPqnH9Th8zTD28vekmsi3u5eB1WxjoQ4uIvplFoZLLg67Xlh07xRiAfjVXiVfIVuYa2Kev9eYP1VUWmVpTk0a+doEaxuKMY+4T8JpQasLalMhwbc/5ou/YEFFGRnyD8rKbn6pH2GaFw43ju6cWyP1/qdZq4AM+ltq182FYaWA1fE6exAjNeo2WDKtaHzQKG3XHaqeCazC5Q9teT6VHUG77ZPvBmasXZiSe1pW+QaQ5XWf4bFm2Ej0qeccGT8CROHu7swpq6TKLZvBG4TbhuCiTKMsvhpUFX53kUNm4ZIo81zVl9odxyd+pRt0zbI4hwaqhqXQVD1pmqC2xVjOgP4Je50CdGD0J3pwG8GVzS48pt9TNrNTIXeh4qrV63dbDu2Et2b9xvrXriB7MsoK2QMV0hCgunQVPHldB/dKlAq/5/2FPTp676fR4KRShnTtbJvI8s0Oc/2BvOvWQaII0+XrnA+XIXv2Quf5HGVGs5cm/JpMH43a6MpxOTdQ41j7mEudATCXgm+wbuwJkl3hCs4hceC0fVlQlq0wWwCg2bXXS3EZYRF+7SNUHm9QiWVkFWVnRbFq8Wd0y84HAuHOtL3AfrK2IHj2nnycxtdIGChcieK1nfDjrG6/oYmWczYZqT7IFrbdPLkvulwngmf6r4ctHlk/5Hqq4UQmxkad7OlJQI3VdrghpVGnqAUdAP+EvCdI0wG5N26GmGQSi2H67VPAXm3NqjhSMz8M7uHN5qhUs3cEzerbR7rEZSEGBM8W4QASzzj4cnagpw6Es9xNbOxJU53Ls Eawnc2I5 F6VnIKtPcTMG+HfVEmF90opkEbVISCSx17iKaLrACK/akBOGzEgF2lMgKzlWX8dHSbJNTh4VHvF6wuERGFqlmur2Zo4FJ3rkltRBF76AXOvaTgZe/kpzfU5L/XNj8aAPSGqMxmyyt/WNhHh4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The page allocation performance requirements of different workloads are often different. So, we need to tune the PCP (Per-CPU Pageset) high on each CPU automatically to optimize the page allocation performance. The list of patches in series is as follows, [1/9] mm, pcp: avoid to drain PCP when process exit [2/9] cacheinfo: calculate per-CPU data cache size [3/9] mm, pcp: reduce lock contention for draining high-order pages [4/9] mm: restrict the pcp batch scale factor to avoid too long latency [5/9] mm, page_alloc: scale the number of pages that are batch allocated [6/9] mm: add framework for PCP high auto-tuning [7/9] mm: tune PCP high automatically [8/9] mm, pcp: decrease PCP high if free pages < high watermark [9/9] mm, pcp: reduce detecting time of consecutive high order page freeing Patch [1/9], [2/9], [3/9] optimize the PCP draining for consecutive high-order pages freeing. Patch [4/9], [5/9] optimize batch freeing and allocating. Patch [6/9], [7/9], [8/9] implement and optimize a PCP high auto-tuning method. Patch [9/9] optimize the PCP draining for consecutive high order page freeing based on PCP high auto-tuning. The test results for patches with performance impact are as follows, kbuild ====== On a 2-socket Intel server with 224 logical CPU, we run 8 kbuild instances in parallel (each with `make -j 28`) in 8 cgroup. This simulates the kbuild server that is used by 0-Day kbuild service. build time lock contend% free_high alloc_zone ---------- ---------- --------- ---------- base 100.0 14.0 100.0 100.0 patch1 99.5 12.8 19.5 95.6 patch3 99.4 12.6 7.1 95.6 patch5 98.6 11.0 8.1 97.1 patch7 95.1 0.5 2.8 15.6 patch9 95.0 1.0 8.8 20.0 The PCP draining optimization (patch [1/9], [3/9]) and PCP batch allocation optimization (patch [5/9]) reduces zone lock contention a little. The PCP high auto-tuning (patch [7/9], [9/9]) reduces build time visibly. Where the tuning target: the number of pages allocated from zone reduces greatly. So, the zone contention cycles% reduces greatly. With PCP tuning patches (patch [7/9], [9/9]), the average used memory during test increases up to 18.4% because more pages are cached in PCP. But at the end of the test, the number of the used memory decreases to the same level as that of the base patch. That is, the pages cached in PCP will be released to zone after not being used actively. netperf SCTP_STREAM_MANY ======================== On a 2-socket Intel server with 128 logical CPU, we tested SCTP_STREAM_MANY test case of netperf test suite with 64-pair processes. score lock contend% free_high alloc_zone cache miss rate% ----- ---------- --------- ---------- ---------------- base 100.0 2.1 100.0 100.0 1.3 patch1 99.4 2.1 99.4 99.4 1.3 patch3 106.4 1.3 13.3 106.3 1.3 patch5 106.0 1.2 13.2 105.9 1.3 patch7 103.4 1.9 6.7 90.3 7.6 patch9 108.6 1.3 13.7 108.6 1.3 The PCP draining optimization (patch [1/9]+[3/9]) improves performance. The PCP high auto-tuning (patch [7/9]) reduces performance a little because PCP draining cannot be triggered in time sometimes. So, the cache miss rate% increases. The further PCP draining optimization (patch [9/9]) based on PCP tuning restore the performance. lmbench3 UNIX (AF_UNIX) ======================= On a 2-socket Intel server with 128 logical CPU, we tested UNIX (AF_UNIX socket) test case of lmbench3 test suite with 16-pair processes. score lock contend% free_high alloc_zone cache miss rate% ----- ---------- --------- ---------- ---------------- base 100.0 51.4 100.0 100.0 0.2 patch1 116.8 46.1 69.5 104.3 0.2 patch3 199.1 21.3 7.0 104.9 0.2 patch5 200.0 20.8 7.1 106.9 0.3 patch7 191.6 19.9 6.8 103.8 2.8 patch9 193.4 21.7 7.0 104.7 2.1 The PCP draining optimization (patch [1/9], [3/9]) improves performance much. The PCP tuning (patch [7/9]) reduces performance a little because PCP draining cannot be triggered in time sometimes. The further PCP draining optimization (patch [9/9]) based on PCP tuning restores the performance partly. The patchset adds several fields in struct per_cpu_pages. The struct layout before/after the patchset is as follows, base ==== struct per_cpu_pages { spinlock_t lock; /* 0 4 */ int count; /* 4 4 */ int high; /* 8 4 */ int batch; /* 12 4 */ short int free_factor; /* 16 2 */ short int expire; /* 18 2 */ /* XXX 4 bytes hole, try to pack */ struct list_head lists[13]; /* 24 208 */ /* size: 256, cachelines: 4, members: 7 */ /* sum members: 228, holes: 1, sum holes: 4 */ /* padding: 24 */ } __attribute__((__aligned__(64))); patched ======= struct per_cpu_pages { spinlock_t lock; /* 0 4 */ int count; /* 4 4 */ int high; /* 8 4 */ int high_min; /* 12 4 */ int high_max; /* 16 4 */ int batch; /* 20 4 */ u8 flags; /* 24 1 */ u8 alloc_factor; /* 25 1 */ u8 expire; /* 26 1 */ /* XXX 1 byte hole, try to pack */ short int free_count; /* 28 2 */ /* XXX 2 bytes hole, try to pack */ struct list_head lists[13]; /* 32 208 */ /* size: 256, cachelines: 4, members: 11 */ /* sum members: 237, holes: 2, sum holes: 3 */ /* padding: 16 */ } __attribute__((__aligned__(64))); The size of the struct doesn't changed with the patchset. Changelog: v3: - Fix the per-CPU data cache slice size calculation in [3/9]. Thanks Mel for pointing this out! - Fix a PCP->high decrement amount issue when free pages become low. - Dropped original "[9/10] mm, pcp: avoid to reduce PCP high unnecessarily", because of no enough performance data to support it. - Add some comments per Mel's comments. Thanks! - Add a Kconfig option for max batch scale factor in [4/9] per Mel's comments. Thanks! - Collected Acked-by. Thanks Mel! v2: - Fix the kbuild test configuration and results. Thanks Andrew for reminding on test results! - Add document for sysctl behavior extension in [06/10] per Andrew's comments. Best Regards, Huang, Ying