From patchwork Wed Sep 20 06:18:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13392108 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23A02CE79AD for ; Wed, 20 Sep 2023 06:19:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B3976B010B; Wed, 20 Sep 2023 02:19:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 957E76B010D; Wed, 20 Sep 2023 02:19:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D1956B010C; Wed, 20 Sep 2023 02:19:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 69B206B010A for ; Wed, 20 Sep 2023 02:19:41 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 3BD8C140EA9 for ; Wed, 20 Sep 2023 06:19:41 +0000 (UTC) X-FDA: 81255974562.28.C9DDA30 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.126]) by imf29.hostedemail.com (Postfix) with ESMTP id 16ACE120015 for ; Wed, 20 Sep 2023 06:19:37 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="H+InEKo/"; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf29.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695190778; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GPl4E/VNt1Dv9MdbE8wBSTxkzdvu+TPDhPll3vaYXgs=; b=l/XE4B0zDVxNGVOIxzsNa3fsmSArP5UcVecy90lRgAUkAtuUbOfY4gvEt4yqsCGjlXVJWX wkWVRjRCBjcF2a09M0oBmiQfjkT8sskjYsikZ2RevIWHHENpukI7eUMGItdC8eRhYIC+sp Qu03FDfu+eHaB9rwazR2SEbQh8Sdkm4= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="H+InEKo/"; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf29.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695190778; a=rsa-sha256; cv=none; b=POH2MFhA/ubULsISdjwBfYKPQjpHU4ieAG6rHWBsC09TroZrGqR7G7y6xGWmCxVqe/3mgR JV0tZYIRabcYIeBuZ8azxd1pyAPa5jfiFAr7mXGnPFsPiscdNmk9eXYxrpDjouTfAmJ3G0 Ak4jG+w5moBQ5/4benIER68rJr+5QFU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1695190778; x=1726726778; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=y5KqaVwntn1O5nTc80IjgYebQMrM7BtNCmGfR5qYkwg=; b=H+InEKo/foeVkfovEPk9LPTEI1LusxUaeqweCswNPaWYE6m5JlGWAyou eKsnKsJ8sRSOTDGiVvWEh282gmTRX60n6+hnQrVDQHG5m6eqMsTa4bUxA L1f6yoRNRQBsrYd3TGn71dG7Cp+PzZ+ce4xZfbgU3tOUB3y8CQKDUEhRc YWTT95qCeb38SCxvMSY17Vjbo5noRmHjeiQJ0jvddSTqS4/rdKVOCymX7 wKPS9drvmTlIoG0CoQCFV/1tg4S3B5vAdmri+KLJYh9DlqVS6nU/KdLTi KUkBzkSUa89BhqKXSDrI0TzWCt7JwDtVsm+JnlNRGWV02cyt56F4fH8M1 Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10838"; a="365187582" X-IronPort-AV: E=Sophos;i="6.02,161,1688454000"; d="scan'208";a="365187582" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Sep 2023 23:19:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10838"; a="740060503" X-IronPort-AV: E=Sophos;i="6.02,161,1688454000"; d="scan'208";a="740060503" Received: from yhuang6-mobl2.sh.intel.com ([10.238.6.133]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Sep 2023 23:19:33 -0700 From: Huang Ying To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Arjan Van De Ven , Huang Ying , Andrew Morton , Mel Gorman , Vlastimil Babka , David Hildenbrand , Johannes Weiner , Dave Hansen , Michal Hocko , Pavel Tatashin , Matthew Wilcox , Christoph Lameter Subject: [PATCH 01/10] mm, pcp: avoid to drain PCP when process exit Date: Wed, 20 Sep 2023 14:18:47 +0800 Message-Id: <20230920061856.257597-2-ying.huang@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230920061856.257597-1-ying.huang@intel.com> References: <20230920061856.257597-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 16ACE120015 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: aji8hr66mixy4ieq6764k3m917asceqe X-HE-Tag: 1695190777-651895 X-HE-Meta: U2FsdGVkX19RL7c44hbK7hXff7S+oLkVMogBnVq+3WE7syxcqh1asnNxGLax2jz8bhVc+EH7fOuj5YYRTeah2Fr3cUEzNieSP6bdrLvx/zQywyXbBEyTHYrR73ryKSc00V5XazI6Q+ogYEVBtfmL0SnRYiLQQ1Ntct+vQnD8c76znCJnLzm4tU1h8ezyp5OOGHmgvLSh71dh8kFx30IHNqvN0Qtm0hHTdnbbsTsGeksSmUCe3T+na7y43mrbsuCpC5iPZ9juxPCOnH4HnFIW8gxX33DFOP0dcb3NK5P6NXgo9ZRB9agO95/yj19Q97XpasA5+fWiqPNIzoT9Iq3rdJd49KqZqNZyEtwF2TK3Zqo3sucQ2oNktiQLl+Gdl1YOhAZQ+ZhkUKi5/HvvKcIjrNWBndGXb4AE1KL8+yR7aHos6r1EFPMX4zpJBTMwMdDc6dUaO7nx9u+CLOhkDRrXJns08sEp1WYSnGa8oHTLOXojgSf9zH9FsrvD4hI/+kp9ZWyHSfigvA6HQohqjaHyZNJlb9PyE5Rx+pry4dMqSmphPLRSbYyfNJPb0AmFvwera8qOT2xbKD1xzMrDWXZDObTT0Fn6l+A1FKPVpnxg2Y78hgAexNRJzPmDE5Cmo5mj/KMFGo3ZMQcpTvbwmMtfO+tWAmonMci4oZS9QZ4uVxFNqDpLi1htcbd4JMpXyph3ueWJ8kYiaJaoFkFzkPwC4q1Pwt/Uj3OMJVYQQorsqUbtvLdve7ALgxQ7OGqsp11LRclCYZQKusPHLOe/hGKWPxJKWqofmKODz/xdHX5kc2PhznnUMcg9UQkW4LtWMsNVce/WZn/Fh0lXPo6ZzaTa9e8A8lzBNbblk5eCUJ7QhDg0uOJ1tvT1NA+GuCjQjL3xl+Sa1uvYg/z/BhYVzog9T7Dak+kFEC9xARELPGWvkvwqk0qufLeJYaRrpLAmLk0u1jG25LSzecpsirZoELw azA/Dnai 7+acvapvckzJU6xWU81TRs4L+Ql8GOZs64z8X76dB/8otENE7J/ONawulAr7ltZCQ7S6zw6H3vG1Lgez6XZkrMlXHsUZzhCp6DBJZZzv9Bb4C8Kws24NFpogOeDVS2PoFlMyZaLB3desJIBu95GdOmky7Jccuz9/FR1PwxsP7F0gytOgje89O/beyosYea3UcysICzs3vIFf0qBrgTACsE54uzOvuw4P1W4afAo2knQpz4CwbK8rIEDpxiNto5ktjIVclDn2MWWvXjqaGOfUH0j250NQF2QpP5iaa4szUoRsfg0Fp2/UqgGT1RP+ygwTHApKYQmtdvZ6kFSfOAp39mKchjMkfK8yYGzVayBndinaivqlrCCPYY3bWy3eKeY1hR4pKIQfVqHWL1AQbVay9zdRkAtDAxB2/Jazsa/9uLTtijlIwxAwMMx+Y679DNsuTd1utsVX7COEiza8g/l6fJfCZpvF6JV3uolBK6CP3XuN7M5tMxHa8IRkOpinhWTD6cheaCS/n503k4xLlSMVXPiBoZcICMW2eacq0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In commit f26b3fa04611 ("mm/page_alloc: limit number of high-order pages on PCP during bulk free"), the PCP (Per-CPU Pageset) will be drained when PCP is mostly used for high-order pages freeing to improve the cache-hot pages reusing between page allocation and freeing CPUs. But, the PCP draining mechanism may be triggered unexpectedly when process exits. With some customized trace point, it was found that PCP draining (free_high == true) was triggered with the order-1 page freeing with the following call stack, => free_unref_page_commit => free_unref_page => __mmdrop => exit_mm => do_exit => do_group_exit => __x64_sys_exit_group => do_syscall_64 Checking the source code, this is the page table PGD freeing (mm_free_pgd()). It's a order-1 page freeing if CONFIG_PAGE_TABLE_ISOLATION=y. Which is a common configuration for security. Just before that, page freeing with the following call stack was found, => free_unref_page_commit => free_unref_page_list => release_pages => tlb_batch_pages_flush => tlb_finish_mmu => exit_mmap => __mmput => exit_mm => do_exit => do_group_exit => __x64_sys_exit_group => do_syscall_64 So, when a process exits, - a large number of user pages of the process will be freed without page allocation, it's highly possible that pcp->free_factor becomes > 0. - after freeing all user pages, the PGD will be freed, which is a order-1 page freeing, PCP will be drained. All in all, when a process exits, it's high possible that the PCP will be drained. This is an unexpected behavior. To avoid this, in the patch, the PCP draining will only be triggered for 2 consecutive high-order page freeing. On a 2-socket Intel server with 224 logical CPU, we tested kbuild on one socket with `make -j 112`. With the patch, the build time decreases 3.4% (from 206s to 199s). The cycles% of the spinlock contention (mostly for zone lock) decreases from 43.6% to 40.3% (with PCP size == 361). The number of PCP draining for high order pages freeing (free_high) decreases 50.8%. This helps network workload too for reduced zone lock contention. On a 2-socket Intel server with 128 logical CPU, with the patch, the network bandwidth of the UNIX (AF_UNIX) test case of lmbench test suite with 16-pair processes increase 17.1%. The cycles% of the spinlock contention (mostly for zone lock) decreases from 50.0% to 45.8%. The number of PCP draining for high order pages freeing (free_high) decreases 27.4%. The cache miss rate keeps 0.3%. Signed-off-by: "Huang, Ying" Cc: Andrew Morton Cc: Mel Gorman Cc: Vlastimil Babka Cc: David Hildenbrand Cc: Johannes Weiner Cc: Dave Hansen Cc: Michal Hocko Cc: Pavel Tatashin Cc: Matthew Wilcox Cc: Christoph Lameter Acked-by: Mel Gorman --- include/linux/mmzone.h | 5 ++++- mm/page_alloc.c | 11 ++++++++--- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 4106fbc5b4b3..64d5ed2bb724 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -676,12 +676,15 @@ enum zone_watermarks { #define high_wmark_pages(z) (z->_watermark[WMARK_HIGH] + z->watermark_boost) #define wmark_pages(z, i) (z->_watermark[i] + z->watermark_boost) +#define PCPF_PREV_FREE_HIGH_ORDER 0x01 + struct per_cpu_pages { spinlock_t lock; /* Protects lists field */ int count; /* number of pages in the list */ int high; /* high watermark, emptying needed */ int batch; /* chunk size for buddy add/remove */ - short free_factor; /* batch scaling factor during free */ + u8 flags; /* protected by pcp->lock */ + u8 free_factor; /* batch scaling factor during free */ #ifdef CONFIG_NUMA short expire; /* When 0, remote pagesets are drained */ #endif diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0c5be12f9336..828dcc24b030 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2370,7 +2370,7 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, { int high; int pindex; - bool free_high; + bool free_high = false; __count_vm_events(PGFREE, 1 << order); pindex = order_to_pindex(migratetype, order); @@ -2383,8 +2383,13 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, * freeing without allocation. The remainder after bulk freeing * stops will be drained from vmstat refresh context. */ - free_high = (pcp->free_factor && order && order <= PAGE_ALLOC_COSTLY_ORDER); - + if (order && order <= PAGE_ALLOC_COSTLY_ORDER) { + free_high = (pcp->free_factor && + (pcp->flags & PCPF_PREV_FREE_HIGH_ORDER)); + pcp->flags |= PCPF_PREV_FREE_HIGH_ORDER; + } else if (pcp->flags & PCPF_PREV_FREE_HIGH_ORDER) { + pcp->flags &= ~PCPF_PREV_FREE_HIGH_ORDER; + } high = nr_pcp_high(pcp, zone, free_high); if (pcp->count >= high) { free_pcppages_bulk(zone, nr_pcp_free(pcp, high, free_high), pcp, pindex);