From patchwork Sun Jul 7 09:49:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13725990 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA4E1C30653 for ; Sun, 7 Jul 2024 09:50:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E78656B0089; Sun, 7 Jul 2024 05:50:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E27116B008A; Sun, 7 Jul 2024 05:50:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CEEC16B008C; Sun, 7 Jul 2024 05:50:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id AEC7E6B0089 for ; Sun, 7 Jul 2024 05:50:38 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2C77380D17 for ; Sun, 7 Jul 2024 09:50:38 +0000 (UTC) X-FDA: 82312486956.04.7B0EA91 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf06.hostedemail.com (Postfix) with ESMTP id 5C325180007 for ; Sun, 7 Jul 2024 09:50:36 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LoXgOXCq; spf=pass (imf06.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720345822; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eHY9t9f2nXb+wivtFSZFhOpolrGiFvEp4lIjiX9vV38=; b=rd7YKAt5ktNF7yfdDT/Ji6zXlijhUVFx1nIL08qci4AdXM50ajkD7DeaH1bMwoVLoxtCYk FAxwvwLcY0hgfUnYVJnxKbKkn/qEZceawpiOkyIa84k/P31kQA8UfXnGBZW0H3OLwJKU9u qgigC2Dj4VCqcnBToaFopZ3BC9aDVOg= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LoXgOXCq; spf=pass (imf06.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720345822; a=rsa-sha256; cv=none; b=3vrHtTPW7Jt19j1n5kMPqNiS96PNTxIacEsMEfiJY40H4UTSVWFP1Ne8R0ir9MuU3dGq2p JAYEu0QPrA3pcBe6T+0m2wCa5z783TvP34B/rBy4+0QnrX2c9pIgBWLSCBX/O8BPAaufty nmxGlAdd+bvwN0WrN51AmYR3iTB5Wis= Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-70b31272a04so11337b3a.1 for ; Sun, 07 Jul 2024 02:50:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1720345835; x=1720950635; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eHY9t9f2nXb+wivtFSZFhOpolrGiFvEp4lIjiX9vV38=; b=LoXgOXCq51JWrJdKQgc7MnsIIRj1MJ6Bt7iyrNB5tI7jO0YjTT2DPrq2Bwpv5GEbF0 NuR5vW7UEjMG77uybH2nFVEh8aGU+Xy9Y8sKuPq+NBdi4LqhvS1zEVx8c0+Oci+8xcZj OuwtVUq7zd96a6Vels2STD1D6J/9z9RywaCspIMJq0FZL1uvmoCSspG5ikpfysWPTmQ4 auk58o/tuAU3qHlnf18Ks5HTt8lftekAIIWrfv3fmPIp4qfhNP4VUA/JEhhqiEs3M9aC pWNG/IgWedRoXpPnsO9S4K0eqW50OQRPHH2exyLYkLAy+OvepaDyTEA/ltbl0QAxq6K+ wjIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720345835; x=1720950635; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eHY9t9f2nXb+wivtFSZFhOpolrGiFvEp4lIjiX9vV38=; b=qn6HxXXcrioSILXKP2jrW5Dkb7Axl6nxRvnBEJ03lX4JjVq/WjtgTWx74d7y+IOkFp DExirrAVAX/Ets+WnyMusm1XYieAW+yRpsKOdbRHQeGnNu8EomrjoodOAnluBATq6Pe9 3cgndrhWO2+vSNKWubkIhK+AUnw02pRIOub1a5MNkTF/gREps5S8tQLtER8VoU8JMLvu WrcTAH62++Yp19UQEUdcgqFjrvMMl05NOj6gCDoFs0qokxqXCXSCTF6wNsZCE7SRZ8of TX/+3arACKGogwV6MW4jKBw3TezPfzKMhXE0OwCkPGjG33c3T4yRasF80G4d0IIKyR5W ByqQ== X-Forwarded-Encrypted: i=1; AJvYcCWVDvF9BcthDfl0X8OX08wNx9cEM1bhbnI0WJ9Yv9j6vQh7lFDztwHsYQsOA12KB4VWUO5TZa3CN/qmW4ub95L1hZc= X-Gm-Message-State: AOJu0Ywur+Wu9ETBiJqf1qA5qhh6nk56OaEs2Wj3QicFp5w73lXI8t8b DwtpAfY8KtzmhSdnQaEWlQb+CFCK1jz7yU/W8EXBUAFnbEJOfUw+ X-Google-Smtp-Source: AGHT+IFD4VCZPhcPm2Rk63mPUix3jDHrVISF+2jKDO75i/IpB19bfGUMK55l/v+7yOpi8L7by6qmNg== X-Received: by 2002:a05:6a20:3d89:b0:1c0:e673:736c with SMTP id adf61e73a8af0-1c0e67375famr5151048637.3.1720345834895; Sun, 07 Jul 2024 02:50:34 -0700 (PDT) Received: from localhost.localdomain ([39.144.43.178]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70b1fdc335csm2184601b3a.22.2024.07.07.02.50.28 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 07 Jul 2024 02:50:34 -0700 (PDT) From: Yafang Shao To: akpm@linux-foundation.org Cc: ying.huang@intel.com, mgorman@techsingularity.net, linux-mm@kvack.org, Yafang Shao Subject: [PATCH 1/3] mm/page_alloc: A minor fix to the calculation of pcp->free_count Date: Sun, 7 Jul 2024 17:49:54 +0800 Message-Id: <20240707094956.94654-2-laoar.shao@gmail.com> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20240707094956.94654-1-laoar.shao@gmail.com> References: <20240707094956.94654-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 5C325180007 X-Stat-Signature: md61anomr5bt8ztjx1nsxgia3okbda4x X-HE-Tag: 1720345836-502057 X-HE-Meta: U2FsdGVkX1++3/5HbFrDjHlV+bJPVMmeAC6puoOk41S1VyR0iMe5fPrTe+Pgt+jwxbo6hby33Lctx0tXSZACtL6AROcYCWYRRFuipkTkVxG4d25SJEyYu+U9e2sMypdMk9OqVnzbkQbLY6TglQVMtJRM/3Ns5lyQAfab3SnzQqMA60R2lNMRs0mSyuPupwtf997kQmOoXmhKOe0vlwnPMaj2VXGzWx2t7xDiwsW4ciYiH/ukhvbJ3Tn0ewwjCEuPT4NzXHk+7VUwsM+pl6I4czckjaXL2f8jyBvXLYbEKi6jmy5Qtc7ixtBb+5vHJHh/Z1PKKWfQUgfWFHoMoZNaXiDl2VBZofeBOToe9C8FFmMUqNIKD4+UBuhVvp6JdEZdD4AAdAb2tCX44uqQzYRXdiOu0RpY/LxKFEmF10W7OlD81yb/DbdyAiFDgogxrJg4zXObjHWHsooeIzjLGJqEywpkVlUaqmTHgrndE6HQjrSiLH2t9zwM5aLM2THkcpeQvHwLYgEnEYL1kx4LADccC/XsLansOA4GrQcBnDPCe+bu4MxiR0kTy/d/uS1l696EC58Tns57X+bOftyJm9zNkiMv5k4ZYAqRrg3pS3sAWdgZDPo3u+tA4pRUWHhCTGzgxfVamBS9txVoWsFRMJuQ97wjAcfORnPtLNUdeZ6gxF/2+7c++OeaRFNd5dgASjOJbfmiIbb4ccrTTTAclIwKy8IW8ONzU8Iq4xa7JbyDtxHaabc5rnY3R+TO0e0AdZqlGwr9aKaSzkB5dOpz5qnpEuxXCntLHZhNgmZF3Kt1iMh4o0EhyNt9ywEaK5H5c+uuhhaxYft4PDn2fBm2J7+T70GbKLVVD120azJS5KeNf4jflNxuKNuZzrIGz9ro/9CNG2XExPG6t3CkrkiqfbFBuJAW4HHAOQqHHcqV7kTN/xNVR6bUidKZhlqe1yjrKNK74c5o2VTPOjC73AXLxX3 f1ERaXyV KbnePfbWbFqzVUKHAhWR9PGBzMeTt27lOsES09kb5j3Znmeme7/dh3b/7kznCDb86uNumsZfPIHP6eRho223kagtVrAa9iFw9lcubGt2XonH+uWl1ZUwk2XUdVKMcjcByej6ubXY86WQQ/PWYVZ/0MSBjlBSkryjDIA2IBCVHamwO1uuISiacjaPUBybtWgbKcnItwxTGBza6QgBHY2iDVIKUOWhiUDa5SjfKgw6TSvt4LrOXU8eFwxUvGotPioum0/gR3mTXmVjedJMYspMOG1pbPtwd4TileCFE9ViYDgCBsGJpWNR27ctyVBKYWGeBNidPYjezQFxihDBKO3IFcKW9tlTV/QlWZ9At41yblSvepn2iDKaULZjXFgZRIl3IF8Kwp+ZRyvpMHEhmW0TyKVVPiF/ewdLfmBbC7RZT3KsH/h344VStq3/sxPKtpwVFZ1cvOF1XTTr5buF4FOwlXD/Nddrbjw8oUHosUBB8XWDHz2CqEjF1pBErsL7ZAwQZMKpi1JRYTgc67c4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.006057, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently, At worst, the pcp->free_count can be (batch - 1 + (1 << MAX_ORDER)), which may exceed the expected max value of (batch << CONFIG_PCP_BATCH_SCALE_MAX). This issue was identified through code review, and no real problems have been observed. Signed-off-by: Yafang Shao Cc: "Huang, Ying" Reviewed-by: "Huang, Ying" --- mm/page_alloc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 2e22ce5675ca..8e2f4e1ab4f2 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2534,7 +2534,8 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, pcp->flags &= ~PCPF_PREV_FREE_HIGH_ORDER; } if (pcp->free_count < (batch << CONFIG_PCP_BATCH_SCALE_MAX)) - pcp->free_count += (1 << order); + pcp->free_count = min(pcp->free_count + (1 << order), + batch << CONFIG_PCP_BATCH_SCALE_MAX); high = nr_pcp_high(pcp, zone, batch, free_high); if (pcp->count >= high) { free_pcppages_bulk(zone, nr_pcp_free(pcp, batch, high, free_high), From patchwork Sun Jul 7 09:49:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13725991 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01FF0C30653 for ; Sun, 7 Jul 2024 09:50:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 85D7B6B008C; Sun, 7 Jul 2024 05:50:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 80CC26B0092; Sun, 7 Jul 2024 05:50:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6AD986B0093; Sun, 7 Jul 2024 05:50:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4AB216B008C for ; Sun, 7 Jul 2024 05:50:45 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B812AA3791 for ; Sun, 7 Jul 2024 09:50:44 +0000 (UTC) X-FDA: 82312487208.25.978ABF6 Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177]) by imf16.hostedemail.com (Postfix) with ESMTP id EF2F018000D for ; Sun, 7 Jul 2024 09:50:42 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=nAbjWl9a; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf16.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.167.177 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720345811; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8ypI2csRyBiyBJMCDlnFfjGoyjtJt9UTQgB5QzLtvc8=; b=pAMxky4zhoxoA1cebEkz/4rbH/7H0jpFTiijeQT8AQZ/H2thP6hwpjlnEP3ZRIZgpzoPsj NcUDmoGoxYCFj7v7P+PVVgvWjjAJ+pKwROS1yga+exdoqkjhbtb8QWu6Bm0s3bTHAbqfDt OgtV7Ks794A+W1pIL5NNSr97sv3H3sM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720345811; a=rsa-sha256; cv=none; b=QyTEk7gJOAf7XiA2x57YboI80+BOuEpkIqLsnj9I3QfsKTxAMLv19GVnclst6CvcQkN7Wv J0CLivifhqswe3FDmiPfRDeW0Wgu+El6liTaqCHFmpeOmo1AsRq8rTrdfS7RiTE7i+uTjn /pQSTS3Xeif1SSPlsnjpgdNFXLPqTZU= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=nAbjWl9a; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf16.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.167.177 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com Received: by mail-oi1-f177.google.com with SMTP id 5614622812f47-3d91a2c95b2so1267872b6e.3 for ; Sun, 07 Jul 2024 02:50:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1720345842; x=1720950642; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8ypI2csRyBiyBJMCDlnFfjGoyjtJt9UTQgB5QzLtvc8=; b=nAbjWl9aUb17UVohpbkMIpWzp1KxuJTIJKZsU9VDlis9JwdzsZLozFjMuCLXF71oX3 1ummTq61pNNzaGksjaxy5Y8sPo/zZ68IlBbhfobxEOgYIe/60OHRyIqDbtHwHJJcnieR CRmNxZkdv86HQrNK6APeY1oeqaGcN6H+0/p8mbPGMBsED+hDkOyYZhrA9Jzpq3pNzf2z 7B4W6bYjEPCMUMoXN8dHpui6lJdEDAjqGYjuyz22jWOrWMlVj5efYSrS+ad9viD1l+gi FuH/KQPIgRxO6OY+GA7WxEXWjhg9Kev7yR8rxPFD7b/rl0vLWKcotekUZHCqSeLE7pg5 w2EA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720345842; x=1720950642; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8ypI2csRyBiyBJMCDlnFfjGoyjtJt9UTQgB5QzLtvc8=; b=hFyTD29eGDblV1eM6oMdV9L9cCX4zvmz3Ps3Vd9RPyHO2Xg+WhxIPT5wNoadkkS9Nk L4fydGdwXWLfPwWgGHAgh5sSvNy1fogs6d5Rn3poY7/qNJ+HtcEPhImu9xjEVwLHdJOE lKLOsSI1W64MoaDbMyt7Y91ZKMacgrr28nweK/Zc1FUlJhB5q3Hp/LN/Ae0dH5yuK3Qv 7zgLoqPxx53QcXdaOkno9jWIVm8CO2otRqMUanjuWZL3bcbDYw/gpw9qhW432Rnhf6GH uburk+bAj/sM2wEVhz1PhgZcCoZHaTe19vNFcgwM9aEbqfXTjzzOYWYXVez35yb5FnDB Sq3g== X-Forwarded-Encrypted: i=1; AJvYcCXEd/4Fa41+t95r64P3UkkGpjVdAMZ11hIOEFZtxLDrjtm2cfjwoapFH2wAWnkqFuzuidr6V0rjpMi/BUVptZbixr8= X-Gm-Message-State: AOJu0Yw9Kxttzpx7Spgg9AC7rCF6fgyc6rqLyHyBLQPPx1u30rH0ApdJ zqSZvYkRPSIdndpMei+UQloew942qkohbZ9SuBBoR5IlTg+AiPHm X-Google-Smtp-Source: AGHT+IFscFmOzTVxQJ1Mzado8UnTN7rHKEWouraXKJGeb8dCg90qotgQ/t1CjGuIii0r2G41HYilZw== X-Received: by 2002:a05:6808:1646:b0:3d6:32a5:66f1 with SMTP id 5614622812f47-3d914eaa843mr11195044b6e.52.1720345840459; Sun, 07 Jul 2024 02:50:40 -0700 (PDT) Received: from localhost.localdomain ([39.144.43.178]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70b1fdc335csm2184601b3a.22.2024.07.07.02.50.35 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 07 Jul 2024 02:50:39 -0700 (PDT) From: Yafang Shao To: akpm@linux-foundation.org Cc: ying.huang@intel.com, mgorman@techsingularity.net, linux-mm@kvack.org, Yafang Shao Subject: [PATCH 2/3] mm/page_alloc: Avoid changing pcp->high decaying when adjusting CONFIG_PCP_BATCH_SCALE_MAX Date: Sun, 7 Jul 2024 17:49:55 +0800 Message-Id: <20240707094956.94654-3-laoar.shao@gmail.com> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20240707094956.94654-1-laoar.shao@gmail.com> References: <20240707094956.94654-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: EF2F018000D X-Stat-Signature: ky11osmhc586ai6qh6u3x38qrowfters X-Rspam-User: X-HE-Tag: 1720345842-22256 X-HE-Meta: U2FsdGVkX1/pqbSIum9+Vy6eAtIOcDnxFMbw5sUNDkGdyilrDOq1ui2Wxk9VsEqA0dHvktxnCfvrIpogFh6I2WA3hQ5xeMm4OHVk8IEYzoraCbPtQUhQ9Mb342++/wTQjLir1NuZYftWCW1wVgVLRwj0LvhxhaI3H7Dc2ycn+k14vlprB0GKYpemhI0lfmkYfwtRVGVl7NDFjkA4FlotyoL45UjSpSBYnKQMCnmaMERP5J0bdqJ1Hda4iU07sMqCkHhlXOfYIjsZfaplXAd/UEzaH9DdPp9LBYVpEE9puEZpsOCoAxbdzWEdiI9uedaQqOW5QhGYThGehSWMkCujynPqVJKeWhXz9VnUm7Zjo0+QeA2FE+lxYJ+32w+EGm7doi+rrcWO70nZjIfFjD596jO134Iahfyp9QV0Jhaknm2HnSDXSi058t9FQQ/HcKgeHvEUnSkLy9IdFvK7nFtJDf/FtQe48DTm90pCdKmVfizyAoHA03ZwEhv+9+A+X2JinXp94w3UAyQ8QMNBob1iYmsSCkylDe9+rrezsrNMsOJ0wS5Idlvh/yhIYjMI0nqmMR69KBwK9seHbSXxH78Qqf/TCZ7i81kzc4yF7QQAqbA+ksKDpQwNJkAVNJjr8RKK3mD9e00Z9A4gK6rnMInq7RFNpZX93xEAa0N5I5JzSmtcQo0dt7m/5e3o1wN8D78+lAXeKHt/B5nA/NO7XqNGkv/PPN06ja/gnRK9qZp7/bCGHFp9u1XGUvjq1EY36ztZc7FoU+/0Z2ZNmf7rEmtI59WkMLdEiIxipJZs/vVuRri5GhjpEVaa+g0wOC2K0Qkf0+KInyO0ASJA3sbv6oQhu7LlTw97OqC4508H9vUbzCgJQ0x7Edxgx5jmptubMR4KhR3H0plUPwwFMDZnIOzJG6zQeJIzDDN8uh98CZvqSshvlf5fVb4rx3NOd2neWInszOiItKReA+5XJVN6bT9 fpTc7MRV QPbRvn5DaNyQtGd//+bdn3uj/DbbkntvrjSBq4yorNpQ0QEMa9D05vSmJz0ugOdIuIng+Qa3w0r8IDg8T8IWB4b4Y0cTYHzJ7XJXS31YZDtloMlVkPmiluvOpgs6qSgc0LxA75uUg20Uh7NAKLQx58diNHQNXbPLxjR39ulAdIf6FwEUV2eSylbAMikKjjmlP9vZ6XweIca43FiOUV2UYDJimBrxZTIJdI0oCZmQ5IEMhKpAcI+zEFA8keHN/5T9Ol6ywDZhztEM4Il6nLrcftROyQeOivptW7zVah4iR4dq7hOcSQtoFolMG7S4DNlxt5/nJs/3j/PFAjDY93oWAwXk+ET3DUgFbVpxwrb5NmFRhmyMUf7L2ERltTl/82Ib367jO3S5ehl/jz5d5MXd7xLQfj7Retv4B4EiDyaaiQKCC3R6I2lqlsx/1BH4vLS1k7CGUyt+jGtlIqN0k8bL9oYJVX7RTGs8Vgf7mPxalT5jhgYqcgvlImROtjssVij3dWSYlDa42Ymzh1Ws= X-Bogosity: Ham, tests=bogofilter, spamicity=0.028773, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When adjusting the CONFIG_PCP_BATCH_SCALE_MAX configuration from its default value of 5 to a lower value, such as 0, it's important to ensure that the pcp->high decaying is not inadvertently slowed down. Similarly, when increasing CONFIG_PCP_BATCH_SCALE_MAX to a larger value, like 6, we must avoid inadvertently increasing the number of pages freed in free_pcppages_bulk() as a result of this change. So below improvements are made: - hardcode the default value of 5 to avoiding modifying the pcp->high - refactore free_pcppages_bulk() into multiple steps, with each step processing a fixed batch size of pages Suggested-by: "Huang, Ying" Signed-off-by: Yafang Shao --- mm/page_alloc.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 8e2f4e1ab4f2..2b76754a48e0 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2247,7 +2247,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, int decay_pcp_high(struct zone *zone, struct per_cpu_pages *pcp) { int high_min, to_drain, batch; - int todo = 0; + int todo = 0, count = 0; high_min = READ_ONCE(pcp->high_min); batch = READ_ONCE(pcp->batch); @@ -2257,18 +2257,25 @@ int decay_pcp_high(struct zone *zone, struct per_cpu_pages *pcp) * control latency. This caps pcp->high decrement too. */ if (pcp->high > high_min) { - pcp->high = max3(pcp->count - (batch << CONFIG_PCP_BATCH_SCALE_MAX), + /* When tuning the pcp batch scale value, we want to ensure that + * the pcp->high decay rate is not slowed down. Therefore, we + * hard-code the historical default scale value of 5 here to + * prevent any unintended effects. + */ + pcp->high = max3(pcp->count - (batch << 5), pcp->high - (pcp->high >> 3), high_min); if (pcp->high > high_min) todo++; } to_drain = pcp->count - pcp->high; - if (to_drain > 0) { + while (count < to_drain) { spin_lock(&pcp->lock); - free_pcppages_bulk(zone, to_drain, pcp, 0); + free_pcppages_bulk(zone, batch, pcp, 0); spin_unlock(&pcp->lock); + count += batch; todo++; + cond_resched(); } return todo; From patchwork Sun Jul 7 09:49:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13725992 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BB13C30653 for ; Sun, 7 Jul 2024 09:50:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9BD876B0093; Sun, 7 Jul 2024 05:50:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 96D0B6B0095; Sun, 7 Jul 2024 05:50:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 80E8A6B0096; Sun, 7 Jul 2024 05:50:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 61DD46B0093 for ; Sun, 7 Jul 2024 05:50:50 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 15C251C1A02 for ; Sun, 7 Jul 2024 09:50:50 +0000 (UTC) X-FDA: 82312487460.18.7243B6C Received: from mail-yw1-f173.google.com (mail-yw1-f173.google.com [209.85.128.173]) by imf16.hostedemail.com (Postfix) with ESMTP id 38B2A180008 for ; Sun, 7 Jul 2024 09:50:48 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=L8sozZGe; spf=pass (imf16.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.128.173 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720345834; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sD7NfWgTVs0F8RAzMXcGiyzg6zCjJB986hliTgQKeuM=; b=LK7mGV+YVyhakXxrqzzNf7bQC59rAONdMJlQ6bR04qgTlMhcyjZPRs+WhKReK7SqP4fPnx b/Eichk1JcMtkl15HwZ7I8ci7KBDq0htoSva4jGv1R3x3CjgkKp67vcXKHmmCMZgmZAPYg 1RmuXnwx9puZryiDlhH4RGyJSYrqE3U= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=L8sozZGe; spf=pass (imf16.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.128.173 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720345834; a=rsa-sha256; cv=none; b=D5t7DZqDbJ6E88r629FdO+zPk0odR20q+HaRGD3JT1aTYni2AA8tv3LOXDiOTbBQrfqpfh +M79hyrrMZ6FZlTr4GKIkrgf1TrlYSUqOb3wM7Is3QSm3MT+sAD401lQA2Jt/+K7oL6NET TJNT//iTn7BBY+jGK9IWnRYRGxzxXUE= Received: by mail-yw1-f173.google.com with SMTP id 00721157ae682-64b29539d87so26223477b3.0 for ; Sun, 07 Jul 2024 02:50:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1720345847; x=1720950647; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sD7NfWgTVs0F8RAzMXcGiyzg6zCjJB986hliTgQKeuM=; b=L8sozZGeI7xEU44kRP2vx4/gtunUdb3ZsbElXMaAhD/J6If4bZR8O76N0O0SLwpEYE w/LAym21s1DImAmS0NkXdqHw8ql4p4jUnIvuKYDKMnxIX+Kp0wDUCguzRl/EFvI2i2E3 5jeKV/DhBFAff6Yk6HozYW9J75OkfWfs41ta3ccDWWughRwR/Vllr7CiiNXhe61rxp+8 vusN4rf2CJv1/IaU+AsIa2oBATPQiHusnCq/KA78rfkqR1sEr82BVcx3SpiAQ+M/S/y3 mzBp7FmAoxQlNSwgpgVVjrBNsMxXcxZtW8+Ii0ljMMsLwivlH0fsjIVjzj43swNFc4mS hVow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720345847; x=1720950647; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sD7NfWgTVs0F8RAzMXcGiyzg6zCjJB986hliTgQKeuM=; b=tdPRfaKNTbFrDKML5Wtoq79TZo/OAWhP94kIwB3DN+F73wdeQ7w/Gsp/7maRMQ06rA IpdqJouwwXZsDYoHB7EOTzjuHCZvFQ+r6xhO4YFQA1RbTT3E5QkzXipTUkFSuiAqzBcw bFwZQn925F2/b7F+OiIHHGaMIlSgmtEjOwTE+kX1nXqGAg9Sur5YMPBhgCOKWTXgDAXs SjAVeIk/Cl7C8+AjS6p6eIq8MDAjr+zrQScq1e6ZaZ8bElePlDZGTDfnRz/iGFCd3WAO 70rg4ngP/mUkpjgVk8d9z7maNrp0RgmaZRL8WlZVxbFNk23om5v4oLndDlTa0ASHpSom Y0Yg== X-Forwarded-Encrypted: i=1; AJvYcCWmVQw06aJJY2rX0K/oaz7g1m+GxBTx/Zn377DhWCSJIe2V1FSOafnkScfjrcbTZCej1B/0jwz+T3kM6RJcR0k1nU0= X-Gm-Message-State: AOJu0YznnO2CxAE4H79WWpP1UmTeKNFUpp7KN3v0fdBcGUULvr343flf jlfLg6bw4YUPnblzbE6Rwa568vXZyoy8g1AQOh39/hUaw4IX5HKm X-Google-Smtp-Source: AGHT+IGXOdiPbTJIJ5f4sdMD5U4qP3UzLYv6wZ9lkRKsq7bdrwD6nYfjB3zi9Ul4pQTIJE0D0OayXQ== X-Received: by 2002:a81:b649:0:b0:627:a757:cdfa with SMTP id 00721157ae682-652d7b5cf79mr92659127b3.38.1720345847252; Sun, 07 Jul 2024 02:50:47 -0700 (PDT) Received: from localhost.localdomain ([39.144.43.178]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70b1fdc335csm2184601b3a.22.2024.07.07.02.50.41 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 07 Jul 2024 02:50:46 -0700 (PDT) From: Yafang Shao To: akpm@linux-foundation.org Cc: ying.huang@intel.com, mgorman@techsingularity.net, linux-mm@kvack.org, Yafang Shao , Matthew Wilcox , David Rientjes Subject: [PATCH 3/3] mm/page_alloc: Introduce a new sysctl knob vm.pcp_batch_scale_max Date: Sun, 7 Jul 2024 17:49:56 +0800 Message-Id: <20240707094956.94654-4-laoar.shao@gmail.com> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20240707094956.94654-1-laoar.shao@gmail.com> References: <20240707094956.94654-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 38B2A180008 X-Stat-Signature: t711iuwftsjyq94no5edxhsmy8u3jq9u X-HE-Tag: 1720345848-126003 X-HE-Meta: U2FsdGVkX19JmZXEhZJVDPwVKIUe52idaaMlHAdvyyjUncrB7QK0m7vEn3r+nG26GWc7Rdkju3E3BmOzxFwr9LZkayZP6oym9tXqbRVh8/6Vs5XvKnQyxQNt84x+pp/WjuBZlpWzUwCaPyOmzsYqwh8Csa4H5cL8n2QFAgB4PNsyFROod/vq/2KDAuKkOUIAVN5x8dCm10ISUslDNBKnddAI/f26BpL3NNFalF4lSUSL3pHY118mQC5oF3fzxL4vxVKsyXPd774lVdU3A3hQ+L1P0vkQVBniY03SGNhRB5NcaoxogrZzASQjb0SpKEDVlhOgSGA1+eIBNBOWMfFza2j+i8egNgKl4/CmhBFI5+pKpUyCChld+yRv9YIIyWZDtisYZVTblT79WrRPKFDkkF/C6Mk+rq7SQ6nl21+bpoLY12QaEOp/eWyPs1u1mwq0CAQkAzqcQU6AGAtxGE0h1KI1I8RSTavRbxPj5tOPYj+ZU4cg58zi9FUtkXypRKDDBqwYrt/i9xYmEiiuAgmquPJ4LPdbcZ3EAD7F5iKsgeCZPNZE3eDbplBotIjRwWE7UHCpikcFgd0eNK3tVTP9rm2vDy72kxrlgFAFqtDYrSmgCdfPRBj+7WsAbDNmavnB4TeGM/n/OGc8CdqFPbSqd07aa5HU+ADtsRflxT94lXNtwkVZFVzybf3UmKpDPIUdD0yjh+s+auDxfWLn0RJlgEkXgMFJvgGe6uCj8+YZutwcxJg70a+Oke5eagJTyMlJVjiRbf5xkNZ1QquXNFxgt3/eh1y2kcw/JEBGd7tCts3IEfrFl/aoWIq3kYOAMf4WNsJV+9/IvHGWtFqLXHqTiTQg1vL2L8nVpoACWiyl6WAb2mvcOZW1O0rWNmIYAxEhhms6nCV3T2Z1LLyBgN/ORwqlFqeeNhXFlhIa4xvr05u/QoauIXQFHdxpe5Cq5gfBkzHfiDuHRaivmX8SACJ o8iWekNo 9zZNgBl8CzjXL1jZQ0GyBEx2ci17tRKUJdgZBxLXjBUKW/BXLUozY7vGEZgCUWPNC60/ybKIxaeagzwUzuOMNpbQqjwe/xmEI1JHydrAtHqZcmTBn6PSRV6YTkexsCmLtAsrNXMDn9Cwy5MxFX6c4VOx7p6qs5TTDW2I98gH1HvrVxCrCh0omm0mewCiHH67kaRN1rfjikhoikXGXQMUNz1LnrYDUl/3wgncqs3JPjvrpKrheMRHDZPVzKtymbtfIYOO6HMrMmekDjY5UNW4LAon7t4jnr8h9cLOBtm8gJ83PbCM08YkeV1FF4D0IkbvS8VbBIfi7ME6qUxUuIfO9uJvmv88ro/CRLfF4Rf4pLqNkuckiQCT9P15INR05z6QbJyxpvSABdAVXGfyQH3VwTCWrSDx/gAQ26ghz5gZsel8SzVUSxC+G7QIn0juIyfEHkJlhcYZs6Jcro6wiIuwsnDHvBenCpYpDVcIHY47NdhrdvNafJZizv4/kFH29GzjFwGZtcl9dBYbNfrhVlFuLC5/nW+Bs+NLA49MRXQobwy92qIm8bbn9hTZY5fRWAcXOZ7wNcxEHKbJNVX8BcibU1zzOa1BfKELTM9AEye55EIDdSDHeCA4AZ89YuQck9CfpZLjqEMLmZtzpfaksk+cUh8Is4BN8Etz+z5iOV1hRADJZpHBtx79xVCgdYA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The configuration parameter PCP_BATCH_SCALE_MAX poses challenges for quickly experimenting with specific workloads in a production environment, particularly when monitoring latency spikes caused by contention on the zone->lock. To address this, a new sysctl parameter vm.pcp_batch_scale_max is introduced as a more practical alternative. To ultimately mitigate the zone->lock contention issue, several suggestions have been proposed. One approach involves dividing large zones into multi smaller zones, as suggested by Matthew[0], while another entails splitting the zone->lock using a mechanism similar to memory arenas and shifting away from relying solely on zone_id to identify the range of free lists a particular page belongs to[1]. However, implementing these solutions is likely to necessitate a more extended development effort. Link: https://lore.kernel.org/linux-mm/ZnTrZ9mcAIRodnjx@casper.infradead.org/ [0] Link: https://lore.kernel.org/linux-mm/20240705130943.htsyhhhzbcptnkcu@techsingularity.net/ [1] Signed-off-by: Yafang Shao Cc: "Huang, Ying" Cc: Mel Gorman Cc: Matthew Wilcox Cc: David Rientjes --- Documentation/admin-guide/sysctl/vm.rst | 15 +++++++++++++++ include/linux/sysctl.h | 1 + kernel/sysctl.c | 2 +- mm/Kconfig | 11 ----------- mm/page_alloc.c | 22 ++++++++++++++++------ 5 files changed, 33 insertions(+), 18 deletions(-) diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst index e86c968a7a0e..eb9e5216eefe 100644 --- a/Documentation/admin-guide/sysctl/vm.rst +++ b/Documentation/admin-guide/sysctl/vm.rst @@ -66,6 +66,7 @@ Currently, these files are in /proc/sys/vm: - page_lock_unfairness - panic_on_oom - percpu_pagelist_high_fraction +- pcp_batch_scale_max - stat_interval - stat_refresh - numa_stat @@ -864,6 +865,20 @@ mark based on the low watermark for the zone and the number of local online CPUs. If the user writes '0' to this sysctl, it will revert to this default behavior. +pcp_batch_scale_max +=================== + +In page allocator, PCP (Per-CPU pageset) is refilled and drained in +batches. The batch number is scaled automatically to improve page +allocation/free throughput. But too large scale factor may hurt +latency. This option sets the upper limit of scale factor to limit +the maximum latency. + +The range for this parameter spans from 0 to 6, with a default value of 5. +The value assigned to 'N' signifies that during each refilling or draining +process, a maximum of (batch << N) pages will be involved, where "batch" +represents the default batch size automatically computed by the kernel for +each zone. stat_interval ============= diff --git a/include/linux/sysctl.h b/include/linux/sysctl.h index 09db2f2e6488..fb797f1c0ef7 100644 --- a/include/linux/sysctl.h +++ b/include/linux/sysctl.h @@ -52,6 +52,7 @@ struct ctl_dir; /* this is needed for the proc_dointvec_minmax for [fs_]overflow UID and GID */ #define SYSCTL_MAXOLDUID ((void *)&sysctl_vals[10]) #define SYSCTL_NEG_ONE ((void *)&sysctl_vals[11]) +#define SYSCTL_SIX ((void *)&sysctl_vals[12]) extern const int sysctl_vals[]; diff --git a/kernel/sysctl.c b/kernel/sysctl.c index e0b917328cf9..430ac4f58eb7 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -82,7 +82,7 @@ #endif /* shared constants to be used in various sysctls */ -const int sysctl_vals[] = { 0, 1, 2, 3, 4, 100, 200, 1000, 3000, INT_MAX, 65535, -1 }; +const int sysctl_vals[] = { 0, 1, 2, 3, 4, 100, 200, 1000, 3000, INT_MAX, 65535, -1, 6 }; EXPORT_SYMBOL(sysctl_vals); const unsigned long sysctl_long_vals[] = { 0, 1, LONG_MAX }; diff --git a/mm/Kconfig b/mm/Kconfig index b4cb45255a54..41fe4c13b7ac 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -663,17 +663,6 @@ config HUGETLB_PAGE_SIZE_VARIABLE config CONTIG_ALLOC def_bool (MEMORY_ISOLATION && COMPACTION) || CMA -config PCP_BATCH_SCALE_MAX - int "Maximum scale factor of PCP (Per-CPU pageset) batch allocate/free" - default 5 - range 0 6 - help - In page allocator, PCP (Per-CPU pageset) is refilled and drained in - batches. The batch number is scaled automatically to improve page - allocation/free throughput. But too large scale factor may hurt - latency. This option sets the upper limit of scale factor to limit - the maximum latency. - config PHYS_ADDR_T_64BIT def_bool 64BIT diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 2b76754a48e0..703eec22a997 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -273,6 +273,7 @@ int min_free_kbytes = 1024; int user_min_free_kbytes = -1; static int watermark_boost_factor __read_mostly = 15000; static int watermark_scale_factor = 10; +static int pcp_batch_scale_max = 5; /* movable_zone is the "real" zone pages in ZONE_MOVABLE are taken from */ int movable_zone; @@ -2310,7 +2311,7 @@ static void drain_pages_zone(unsigned int cpu, struct zone *zone) int count = READ_ONCE(pcp->count); while (count) { - int to_drain = min(count, pcp->batch << CONFIG_PCP_BATCH_SCALE_MAX); + int to_drain = min(count, pcp->batch << pcp_batch_scale_max); count -= to_drain; spin_lock(&pcp->lock); @@ -2438,7 +2439,7 @@ static int nr_pcp_free(struct per_cpu_pages *pcp, int batch, int high, bool free /* Free as much as possible if batch freeing high-order pages. */ if (unlikely(free_high)) - return min(pcp->count, batch << CONFIG_PCP_BATCH_SCALE_MAX); + return min(pcp->count, batch << pcp_batch_scale_max); /* Check for PCP disabled or boot pageset */ if (unlikely(high < batch)) @@ -2470,7 +2471,7 @@ static int nr_pcp_high(struct per_cpu_pages *pcp, struct zone *zone, return 0; if (unlikely(free_high)) { - pcp->high = max(high - (batch << CONFIG_PCP_BATCH_SCALE_MAX), + pcp->high = max(high - (batch << pcp_batch_scale_max), high_min); return 0; } @@ -2540,9 +2541,9 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, } else if (pcp->flags & PCPF_PREV_FREE_HIGH_ORDER) { pcp->flags &= ~PCPF_PREV_FREE_HIGH_ORDER; } - if (pcp->free_count < (batch << CONFIG_PCP_BATCH_SCALE_MAX)) + if (pcp->free_count < (batch << pcp_batch_scale_max)) pcp->free_count = min(pcp->free_count + (1 << order), - batch << CONFIG_PCP_BATCH_SCALE_MAX); + batch << pcp_batch_scale_max); high = nr_pcp_high(pcp, zone, batch, free_high); if (pcp->count >= high) { free_pcppages_bulk(zone, nr_pcp_free(pcp, batch, high, free_high), @@ -2884,7 +2885,7 @@ static int nr_pcp_alloc(struct per_cpu_pages *pcp, struct zone *zone, int order) * subsequent allocation of order-0 pages without any freeing. */ if (batch <= max_nr_alloc && - pcp->alloc_factor < CONFIG_PCP_BATCH_SCALE_MAX) + pcp->alloc_factor < pcp_batch_scale_max) pcp->alloc_factor++; batch = min(batch, max_nr_alloc); } @@ -6251,6 +6252,15 @@ static struct ctl_table page_alloc_sysctl_table[] = { .proc_handler = percpu_pagelist_high_fraction_sysctl_handler, .extra1 = SYSCTL_ZERO, }, + { + .procname = "pcp_batch_scale_max", + .data = &pcp_batch_scale_max, + .maxlen = sizeof(pcp_batch_scale_max), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = SYSCTL_SIX, + }, { .procname = "lowmem_reserve_ratio", .data = &sysctl_lowmem_reserve_ratio,