From patchwork Tue Sep 26 06:09:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13398719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FDBDE8181F for ; Tue, 26 Sep 2023 06:10:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2BEF78D006F; Tue, 26 Sep 2023 02:10:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 26FA98D0005; Tue, 26 Sep 2023 02:10:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 136BD8D006F; Tue, 26 Sep 2023 02:10:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E02A08D0005 for ; Tue, 26 Sep 2023 02:10:14 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 93905140F84 for ; Tue, 26 Sep 2023 06:10:14 +0000 (UTC) X-FDA: 81277723548.22.D4A7927 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.100]) by imf30.hostedemail.com (Postfix) with ESMTP id 9FA1F8000D for ; Tue, 26 Sep 2023 06:10:12 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=aSES3Wjb; spf=pass (imf30.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695708612; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5wUpIHiH+nJzKfqxLHQ/6d31aYDaI0+xwS2iId27j5U=; b=DqkBISYEDRi4etj6t0QSAW8dFsBAi3No2nEPIWlFvOJxL203mUfw048Lrjd1T6a44dKtgI +9oAwAVkh7t4M5HHqiqGivpzhbOAPYgPRHGaNZ85yL/w9V3x1R5TjITpvVvxKROd6gJLYA 4QfQRpGU8ymGWhLIJJtUCPUTylB2wA4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695708612; a=rsa-sha256; cv=none; b=o693y7gPGuO91sZhpPOvOj0kMgH1kdpBoec2d3RSU8/KQ8MGK6fOkcutY2XWV4ydqAvN1+ bpVNO7ZdZwlPeh6+fjjewFIoMfDDfJOK5ef265Bw5cbTw8ulrjc0JyqWZ1B2M4I/dd2Ejr EEbPa6ZU5yYE0EYQTRSocLVEr0tnAt8= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=aSES3Wjb; spf=pass (imf30.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1695708612; x=1727244612; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=H9bLA8f1e2FoPWWn6dF2nLkxOgqJitjO66ZJfB8995Y=; b=aSES3WjboghDLhqoIerj+3QmREj9bfRb6Kqa1Fnes2eTp4wWNsYytSjG DuxyDWu5V4v/6hobw58U2IWYm1+ESq9KMR3guqXU24gR5+8vXpQmEJvVy A3T4ELvnHO+a4Oww6SY4U1rXIAppLvxPDUA8D+kv4byItHA5HwQ5JFcLk vVM/R9X/hTJc3MKo9n85pLubgUdU1W26i6uHJoON5T9ZxdwJarF8Gv7br 8hwZZHIY++NMAinXm6lnQsZNnFi1RJH2xDoiwzv0qOPiIpPSzaYFpOpdN M1YZPIZr1PVr9AgFgHaTxdVp2DSoLVfRvKRolEALS6/5g2sFw6aySu6a7 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10843"; a="447991514" X-IronPort-AV: E=Sophos;i="6.03,177,1694761200"; d="scan'208";a="447991514" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Sep 2023 23:10:11 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10843"; a="892076144" X-IronPort-AV: E=Sophos;i="6.03,177,1694761200"; d="scan'208";a="892076144" Received: from aozhu-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.255.31.94]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Sep 2023 23:09:03 -0700 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Arjan Van De Ven , Huang Ying , Mel Gorman , Vlastimil Babka , David Hildenbrand , Johannes Weiner , Dave Hansen , Michal Hocko , Pavel Tatashin , Matthew Wilcox , Christoph Lameter Subject: [PATCH -V2 10/10] mm, pcp: reduce detecting time of consecutive high order page freeing Date: Tue, 26 Sep 2023 14:09:11 +0800 Message-Id: <20230926060911.266511-11-ying.huang@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230926060911.266511-1-ying.huang@intel.com> References: <20230926060911.266511-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9FA1F8000D X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: h4fi8mfene6b9gnnytixmfwokfs3j19y X-HE-Tag: 1695708612-270262 X-HE-Meta: U2FsdGVkX1+3YS4wTUyei2MquQrmU+mBZVZVKS8ysx5WV+M6+VxiAw/cjGcNlGfQ0DjutPgONRJ/BQHTLUidesJTcd8/wZwWvOxn7oLy30BfEDg4oSJQR2Q9jKiSiFpSCmYmagKQjenRv+k56ZG0LlcrXiWkNrT6eDvtJGJkVV6yV+Moc7GxDWyF3sVU7SPOT+Ub6Ch0hW8vkXlsuKOz57EzF7Fm2MdNAQoAkcgGa+lU871hNRmZF8nQxvzHjF6goLQkiJ5JFiKc3uPBb4Uy6o9ExJXRcmTRDDMRPlPrxMKFCeQm8Ws38Hu1Q1aKFNTQHKyLEEeirtbpjV1aXhSUPI22PiHEiIAE0TYjlN//2R2xM8tSbFD0m6jiUf574egUUf7j9IzCB91T9p90Ur+fQJfeBT544syhX87jtVbC9tSSpzKXvHlb+2wLtinCRwDdtcyJZHUoFc4NTiTi7spJV9TVYTVRQcrqWBA5cQdi8bGAr9GqPtzxjyjqq/S8OKtQD2ah1SnlkfaI5WOIyZQM1h/HFnijnu3wiH5UIyh3W3eGgHA1FotQD6N40VpOOUDwaxJ6W967es4kS969v4LIb7c1IQccYRIJp9+Ta3oLW811flifFSiEuK41qm4rk4UdoldZiWwowZzNBvYDDJ6r3XLGRKj27TT78xyMXk0hhxHZI7EESEv3tXzu7nZaFHNGSXm/vp9VklnVhE1XpoWYF25GwGO3NMZon7fQbNjDGBM6QpvDe7htzNFXklmfYxxBqoOtH53hYtZiPn1w9qPjr5ezMxQNOm9nAvqFLp8XDsnUi+dKxaAtfmw7w5oX5SKnKXS/cdSg5NT7HGLeNYAidSnFkXG5K0znBGGBTXuaebwcepziDQJBQsQ3ayyvUpHWXW5/vgSkZcaolcWnn6ZM3HPhxPfhuo8LIN4eCobrGPX+Crv+ouEUqmR1v5iz3+iWMeqYoXwrboAHpkSqTS2 47aUbdge +Nr5Ye6FEbkWclosU/4nt1tnUrvyoPHXi1emujg+IZxgpwGxDJRFZbqa6MfRz2Mn5OIayPRQEGmrRkJwlKK8gCvzU1ADL5hfZu2FzF6N5yuek75aO6OFsVVHlkvOfJHMpMYH9fbBrABiO9QEjD0yCSGawmkP0TJn3Kq4dqP1oR1IcF86XQw+x8HnEBeKdV17HfnkiYEtJ5UKB2z4JQGOCWKG4rdEdx3BFVpiOrPX9VJgv9k6Ke93FtWw9A/eipAXEKSo5eVyPqbFJ9nLF1fBkpwByO/6wmbgcZvrPsO7uiJgZsHADTtsveEoiIbTaRT2o/Vp7/qqgnQwsc1yTCzWFZrhaF3gFdAUtH8RtuHKItFJs6Jj8xV3fU03gofscOr7KBcGwcHqfRsVO7/hRG2m2+oOVwkg6uWeOpPVGdJCl2lc8+MzjAmPoFXmbhukqKTGomdcqzpz8tg1vjiNj9nLiN28LAgRFk1Z4sZZnvUg+PjU1FOyri6COP2L9kuRrN5Yun4SXP8fwwlE45Q9Zy4Hos3r3AeZR+Iet5/UA X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In current PCP auto-tuning design, if the number of pages allocated is much more than that of pages freed on a CPU, the PCP high may become the maximal value even if the allocating/freeing depth is small, for example, in the sender of network workloads. If a CPU was used as sender originally, then it is used as receiver after context switching, we need to fill the whole PCP with maximal high before triggering PCP draining for consecutive high order freeing. This will hurt the performance of some network workloads. To solve the issue, in this patch, we will track the consecutive page freeing with a counter in stead of relying on PCP draining. So, we can detect consecutive page freeing much earlier. On a 2-socket Intel server with 128 logical CPU, we tested SCTP_STREAM_MANY test case of netperf test suite with 64-pair processes. With the patch, the network bandwidth improves 3.1%. This restores the performance drop caused by PCP auto-tuning. Signed-off-by: "Huang, Ying" Cc: Andrew Morton Cc: Mel Gorman Cc: Vlastimil Babka Cc: David Hildenbrand Cc: Johannes Weiner Cc: Dave Hansen Cc: Michal Hocko Cc: Pavel Tatashin Cc: Matthew Wilcox Cc: Christoph Lameter --- include/linux/mmzone.h | 2 +- mm/page_alloc.c | 23 +++++++++++------------ 2 files changed, 12 insertions(+), 13 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 35b78c7522a7..44f6dc3cdeeb 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -689,10 +689,10 @@ struct per_cpu_pages { int batch; /* chunk size for buddy add/remove */ u8 flags; /* protected by pcp->lock */ u8 alloc_factor; /* batch scaling factor during allocate */ - u8 free_factor; /* batch scaling factor during free */ #ifdef CONFIG_NUMA u8 expire; /* When 0, remote pagesets are drained */ #endif + short free_count; /* consecutive free count */ /* Lists of pages, one per migrate type stored on the pcp-lists */ struct list_head lists[NR_PCP_LISTS]; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d7b602822ab3..206ab768ec23 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2375,13 +2375,10 @@ static int nr_pcp_free(struct per_cpu_pages *pcp, int batch, int high, bool free max_nr_free = high - batch; /* - * Double the number of pages freed each time there is subsequent - * freeing of pages without any allocation. + * Increase the batch number to the number of the consecutive + * freed pages to reduce zone lock contention. */ - batch <<= pcp->free_factor; - if (batch <= max_nr_free && pcp->free_factor < PCP_BATCH_SCALE_MAX) - pcp->free_factor++; - batch = clamp(batch, min_nr_free, max_nr_free); + batch = clamp_t(int, pcp->free_count, min_nr_free, max_nr_free); return batch; } @@ -2408,7 +2405,7 @@ static int nr_pcp_high(struct per_cpu_pages *pcp, struct zone *zone, * stored on pcp lists */ if (test_bit(ZONE_RECLAIM_ACTIVE, &zone->flags)) { - pcp->high = max(high - (batch << pcp->free_factor), high_min); + pcp->high = max(high - pcp->free_count, high_min); return min(batch << 2, pcp->high); } @@ -2416,10 +2413,10 @@ static int nr_pcp_high(struct per_cpu_pages *pcp, struct zone *zone, return high; if (test_bit(ZONE_BELOW_HIGH, &zone->flags)) { - pcp->high = max(high - (batch << pcp->free_factor), high_min); + pcp->high = max(high - pcp->free_count, high_min); high = max(pcp->count, high_min); } else if (pcp->count >= high) { - int need_high = (batch << pcp->free_factor) + batch; + int need_high = pcp->free_count + batch; /* pcp->high should be large enough to hold batch freed pages */ if (pcp->high < need_high) @@ -2456,7 +2453,7 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, * stops will be drained from vmstat refresh context. */ if (order && order <= PAGE_ALLOC_COSTLY_ORDER) { - free_high = (pcp->free_factor && + free_high = (pcp->free_count >= batch && (pcp->flags & PCPF_PREV_FREE_HIGH_ORDER) && (!(pcp->flags & PCPF_FREE_HIGH_BATCH) || pcp->count >= READ_ONCE(batch))); @@ -2464,6 +2461,8 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, } else if (pcp->flags & PCPF_PREV_FREE_HIGH_ORDER) { pcp->flags &= ~PCPF_PREV_FREE_HIGH_ORDER; } + if (pcp->free_count < (batch << PCP_BATCH_SCALE_MAX)) + pcp->free_count += (1 << order); high = nr_pcp_high(pcp, zone, batch, free_high); if (pcp->count >= high) { free_pcppages_bulk(zone, nr_pcp_free(pcp, batch, high, free_high), @@ -2861,7 +2860,7 @@ static struct page *rmqueue_pcplist(struct zone *preferred_zone, * See nr_pcp_free() where free_factor is increased for subsequent * frees. */ - pcp->free_factor >>= 1; + pcp->free_count >>= 1; list = &pcp->lists[order_to_pindex(migratetype, order)]; page = __rmqueue_pcplist(zone, order, migratetype, alloc_flags, pcp, list); pcp_spin_unlock(pcp); @@ -5483,7 +5482,7 @@ static void per_cpu_pages_init(struct per_cpu_pages *pcp, struct per_cpu_zonesta pcp->high_min = BOOT_PAGESET_HIGH; pcp->high_max = BOOT_PAGESET_HIGH; pcp->batch = BOOT_PAGESET_BATCH; - pcp->free_factor = 0; + pcp->free_count = 0; } static void __zone_set_pageset_high_and_batch(struct zone *zone, unsigned long high_min,