From patchwork Mon Mar 18 20:07:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lucas Stach X-Patchwork-Id: 13595787 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CD0EC54E58 for ; Mon, 18 Mar 2024 20:07:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 049846B0092; Mon, 18 Mar 2024 16:07:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F15746B0095; Mon, 18 Mar 2024 16:07:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB5DF6B0099; Mon, 18 Mar 2024 16:07:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C68D16B0092 for ; Mon, 18 Mar 2024 16:07:42 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A27E61A0734 for ; Mon, 18 Mar 2024 20:07:42 +0000 (UTC) X-FDA: 81911245164.28.3631A40 Received: from metis.whiteo.stw.pengutronix.de (metis.whiteo.stw.pengutronix.de [185.203.201.7]) by imf16.hostedemail.com (Postfix) with ESMTP id E276018000A for ; Mon, 18 Mar 2024 20:07:39 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf16.hostedemail.com: domain of l.stach@pengutronix.de designates 185.203.201.7 as permitted sender) smtp.mailfrom=l.stach@pengutronix.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710792460; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references; bh=/h0P/ttKzo6iOOuZQfdHV/NizfP3LN6T/mv/U8hmp9s=; b=jQWxDt6O4g86Ce6opg2DxVnH88WWXsPJosNk11N8YxFL50sIfcyFZwFV+Gd7iterWeaZ5o GOjE3pZC6t4nuz4F/cnGgbEfzM3BkgFi07I0RuXDWGzbcADgXeNuOIL4yGpUvVjBdQdTB7 QT27RU85LupIq9D6b9T2JtBcLvn7AsA= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf16.hostedemail.com: domain of l.stach@pengutronix.de designates 185.203.201.7 as permitted sender) smtp.mailfrom=l.stach@pengutronix.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710792460; a=rsa-sha256; cv=none; b=K/70kAnfxJIR9pSh2998boSlGgs3kTMtcI+fCG3x+kH6+NJwEbs71ncegAedcvHyve6EC/ sK+DNFiSqNTiiMhtHl1I4D4qAtYNP3Pmsr2PsMn2elLI03aB2ZJKnfuXYgX0HsWOAkZAKA 96j1m6CnqfFuNUGMGNhQ/bC7Mh7cwCM= Received: from drehscheibe.grey.stw.pengutronix.de ([2a0a:edc0:0:c01:1d::a2]) by metis.whiteo.stw.pengutronix.de with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rmJGj-0000Ny-MS; Mon, 18 Mar 2024 21:07:37 +0100 Received: from [2a0a:edc0:0:1101:1d::28] (helo=dude02.red.stw.pengutronix.de) by drehscheibe.grey.stw.pengutronix.de with esmtp (Exim 4.94.2) (envelope-from ) id 1rmJGi-0078sd-Be; Mon, 18 Mar 2024 21:07:36 +0100 From: Lucas Stach To: Andrew Morton Cc: linux-mm@kvack.org, kernel@pengutronix.de, patchwork-lst@pengutronix.de, Mel Gorman Subject: [PATCH] mm: page_alloc: control latency caused by zone PCP draining Date: Mon, 18 Mar 2024 21:07:36 +0100 Message-Id: <20240318200736.2835502-1-l.stach@pengutronix.de> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 X-SA-Exim-Connect-IP: 2a0a:edc0:0:c01:1d::a2 X-SA-Exim-Mail-From: l.stach@pengutronix.de X-SA-Exim-Scanned: No (on metis.whiteo.stw.pengutronix.de); SAEximRunCond expanded to false X-PTX-Original-Recipient: linux-mm@kvack.org X-Rspamd-Queue-Id: E276018000A X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: y66c1pn7b4f53kf9qqaen8hfzpuhgts1 X-HE-Tag: 1710792459-739621 X-HE-Meta: U2FsdGVkX1/grSmRiuW16/lVeN+5pmV2M3RyEnRZNFy7pOKWK8atjTFOVOY6LBONI5/F660EitvQ7qEUWlWSHfkQM3Al330zk3QkfLBTvevf0JGO1bA1CCMoKaPcYUCElf52MBVqgfwkIQi4kP3EOsQKj2xc+8vKdw7rcL8nii070uiyhfvKoGZ5syHHxlE2whS+Njymlm9LVje5bNWRwEK5Gto7Ququ6n5DNLmOffl0w01jm3YT4eBWBXTWnwfDOEaxKDbZiZq0orw1EWeU0lmd/zurMngygeX/t/ValpJ6GJENEGgF+WJ3jnVj050Z6A6PBUfVKGXd/F7aurADiTUiDHTteuBB2WV60tP52vf3GC917bWp5gUVeyyVVPmvDda1qKxF8kqiV8M+Aqs+/jYdzdqiYZKa2BAJcGNvmzKPbrAsHhKVVhB0PEogJ/kCHuyK1xyXyrXrGERVb9JvdBkeYeh6sKQGFGvRt+8TwQk1b5SJ3CXgNsL+NNcjSXy8Eonc89DMp3R33sAXYrX9fKREMpFGpcGAiE7h9n36/atuIu0GfPjcnZFZ/uH/R0iZvTptNS7ajMTgKwPrx4emnHmWN/wEPe76XIB/sCGVCWXs4m34c37ayfyulXS/v1wspEcBbmI0lpsbKt8Owi2yv943mb/ovA+qlG5d+9xf/V2SRSFGzIGTdkF6y57pAt7evJLhItCTQcN9UFmNHwnftJCXbZA9MpT72sR6cR6ViOXZa5zGPe40tgHgqpZt+bNzgO3K2P9DKp9z2QxuN1SdtJf+gCirlySY52D8YcQq0dklSlozCvp81UG7gopoh2X7jZ3YxZEr4cbjl/py54qUIf1X2KynSJ8oU9PHJ8iapC/Uhus3vUE3IVhR/tvzMdmGDoDtiywYscXjS30iIQoBy8OPNfzWX+Un84mgKk0XQqoQJuT1kcVlOcSENB+mGd7zKoGTGR4r+SIrBBdckt0 ZWZZgjEo rFzmgHHnl9N/ckH30HYy8Ze9Dr0ofnSNEuHmJFCwkUi6fWWZP0T8Lp21G1LLNp6jVruXDkAj/WH95DEcTk7JsYrIEYQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When the complete PCP is drained a much larger number of pages than the usual batch size might be freed at once, causing large IRQ and preemption latency spikes, as they are all freed while holding the pcp and zone spinlocks. To avoid those latency spikes, limit the number of pages freed in a single bulk operation to common batch limits. Signed-off-by: Lucas Stach --- mm/page_alloc.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a663202045dc..64a6f9823c8c 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2215,12 +2215,15 @@ void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp) */ static void drain_pages_zone(unsigned int cpu, struct zone *zone) { - struct per_cpu_pages *pcp; + struct per_cpu_pages *pcp = per_cpu_ptr(zone->per_cpu_pageset, cpu); + int count = READ_ONCE(pcp->count); + + while (count) { + int to_drain = min(count, pcp->batch << CONFIG_PCP_BATCH_SCALE_MAX); + count -= to_drain; - pcp = per_cpu_ptr(zone->per_cpu_pageset, cpu); - if (pcp->count) { spin_lock(&pcp->lock); - free_pcppages_bulk(zone, pcp->count, pcp, 0); + free_pcppages_bulk(zone, to_drain, pcp, 0); spin_unlock(&pcp->lock); } }