From patchwork Wed Sep 4 20:54:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13791433 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14E7ACD4F32 for ; Wed, 4 Sep 2024 20:54:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 839676B020F; Wed, 4 Sep 2024 16:54:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7E7B16B0213; Wed, 4 Sep 2024 16:54:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6AFB96B0248; Wed, 4 Sep 2024 16:54:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 46E906B020F for ; Wed, 4 Sep 2024 16:54:25 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id BA4C4A0157 for ; Wed, 4 Sep 2024 20:54:24 +0000 (UTC) X-FDA: 82528258848.18.A2B7E37 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf19.hostedemail.com (Postfix) with ESMTP id 151261A000C for ; Wed, 4 Sep 2024 20:54:22 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=hg7g788V; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3_sjYZgoKCDElbfelNUZRQTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3_sjYZgoKCDElbfelNUZRQTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725483155; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=8wW8R+MHjMe6Movf5KpIkdjuSKAp7NEYib8XCqV/wxI=; b=IPZoJMdE1UVVj8XwiH37xWnkG2JsIaz/5leJkfwFTP6m1Xcb5W/vK3wrorZxhnF6c9iJFl Z69VXrCOsqRy0Dta66aVkvxvXZgpa2FKlxb06AjCszGOWaACRanOs13ZRkrIp86oZFOp9A QbxFtFPLbu4k69uAA5TrrVJ/SD7N/8A= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725483155; a=rsa-sha256; cv=none; b=mv2lRwAWFaaBOW1IM1FhdVg35lkP34wYRmnD9nf8rMtGpUCOJZDeTHVNvYvx20CwWnfeLl FKDMVBsfFvyW3/FenaeWhrgt3fDO7zir6sczICG5mbx1LmSyDYK4tvOSgJMV1LyJ5Miuyd QaPIi8s47iAU30aVU3YV4d6YW0scLeU= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=hg7g788V; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3_sjYZgoKCDElbfelNUZRQTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3_sjYZgoKCDElbfelNUZRQTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--yosryahmed.bounces.google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e165fc5d94fso210754276.2 for ; Wed, 04 Sep 2024 13:54:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725483262; x=1726088062; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=8wW8R+MHjMe6Movf5KpIkdjuSKAp7NEYib8XCqV/wxI=; b=hg7g788VE6crLeTVxZNrRv89B+dsEczSjFKL2z7ygjeUhYAcl8/QucuLtBrwf8qG15 gixM+crlOa72LXSzrvkrc7ZalOKY2pUXPK843ZBOHYWmoSwPmW4z1Z+mVlsaR24CkcGm uogIvY9PmDsL8y2ufvjlX9Z19MMm2vZhcLttcPGIE4X1sVSrrxGA3I/jCYEHYGSXB93j qlojJoWYfWTUt8BA2C4jUn4Jog2U2d6DV7j/JnuEUF+sotixqz5CbCKHCBoHVPxkGUHT P8G4oFzOeNDbDh4pgziq5WWDwvuTMiLdrhNBvFNHhZgdRZ61GAKm2ndedDMd9GVTMwlr HiSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725483262; x=1726088062; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=8wW8R+MHjMe6Movf5KpIkdjuSKAp7NEYib8XCqV/wxI=; b=T4IjpDjYO4i9CyEupklCV2e0oe1P0wo2sov9uUrCrpx8ApOKEzPOKB2AqHNdTph4ED NmUZHrsQ0ClND6RlUDHOw6GlD5PZ9Q6aqwuDa2rXg+amlI4+pD9YqPGCAIbnuCsbTwdy 5X+EyXqRs+aJMcDCzHIHRygFJo1RGaw2w/6sbcBMG2gVkGbOKejuIN1sTHCAbQhEsQTm 9fgJzZdA6EAF3qRio0ZeevNViqHO99vUh1OnALREpMidKNde+YErcgGdupUQZNDZLLei HbB0BJdqsKkBvk9o0/c81ABweTcj9G43bQATvxe8m7+P8k9ATWgwpuYuEVDRGTk7wxK1 +TTA== X-Forwarded-Encrypted: i=1; AJvYcCW4GfHrCpBNMXFwZBEf9/FJi7J3YXLIfqbLsrOoZZkMEhrH4Rw9W7oDsjH4RYI8SNSwluoesVCTOw==@kvack.org X-Gm-Message-State: AOJu0Yyf6it1XkXQCkgRg5pLK+o/rhkd6sJtsak6PT+meAaFOrkUv8ML 2bZTWSp9U6lJJl9SlY+z6Zz6HKG7qhEWfpDEwUGkPgXsWDOSKC3eO9SOjRL4/Xd3GFoEEOEtw49 vLb4xh+api8E2YVCyvw== X-Google-Smtp-Source: AGHT+IFB3LFvRGR28dsEVnjNSxmCKIJPF4I9vUm7XHFxsYvTLrDOZ9dN4Ac49G8pX6q1R5j6LIG9hqwVRHFDcGcP X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a25:df8b:0:b0:e1d:7ce:4844 with SMTP id 3f1490d57ef6-e1d07ce4908mr16614276.4.1725483262002; Wed, 04 Sep 2024 13:54:22 -0700 (PDT) Date: Wed, 4 Sep 2024 20:54:19 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240904205419.821776-1-yosryahmed@google.com> Subject: [PATCH] mm: page_alloc: fix missed updates of PGFREE in free_unref_{page/folios} From: Yosry Ahmed To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , "Peter Zijlstra (Intel)" , Brendan Jackman , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 151261A000C X-Stat-Signature: xqwmc8qihgrsfm3ajsxu4jjb5dssbfeo X-Rspam-User: X-HE-Tag: 1725483262-880007 X-HE-Meta: U2FsdGVkX1806hIOzzulfMeRw3/71VPOBsjLi/AMTx7FYMkBx+gNF1J2qnCIhZGBHlniOuVK7d3BscdZl4eIZybLs3rb9ATcCGQjMtG1tMKsP58CAIJM9WUNmzyZECRgUPqyeyT5uNXLD1pkfSVJ0i2VSDPT1rzA518S/HVtK6pO+C5dPDfh1FUSEisLC+Cl96FqAsDhrPMgjItz2PCE/NiR7AeU+l3m4HpekMRBskXFQ8gSmJd4pR5IMzxo0By0D8bHlN0Sga54oV0adRyoE02rgKJi7v8OuwkPWD1IGec26RYSS5St1vlZ5C9exsGrCwaiS18YqwlBJ20ZV2DHgBrB9TlAQyfwZZWHrUhoHRh8EOOpEnj4NWa+BivoHneO7lI95CoQ7YkHcOgOTutpcY61ZGuQL4bXPTcWT5bNPTOKaAf1NfeuYSkrzM1BBjzt3U061ETpAkc8VzfggCYv/01L7dbxC3W+acK+l2rEb4VieejA60wjJdCRWy+VaHsampI9+rGUjnwebhFAlrRGzByMggD1onpHUr8Ulh5ZDIyoKPJfqzpAdvLYSmEWpADmJL2hPtvNBBgxeOxP9lHsYwlLfrqLJJQlrllMsb/yhj+/H163HQhygx5KaTwDPR0e81LpmRUqoXJq09P9v4jxlREJ5iNCsIYYv+iLTIjKlxRqdgqkucJEIMffy3hqasdjTCQHw80if/2txVpsfAjxOWQsVK45eaL0dr7eAAxXsV6VF7rYnxMJb+OFTrBAXImZ+onJbWsqkg6I6BTEobtt46ZMwpShzKOZjWWo24SccXPrY36cPaRhoRPWYvWDXnUvM0SRCVhNyP+1JicfzE6qVHRj5fw99p9oZJtvaNKaSossbHQyDJoMWrwFbUsLFwBQzKHgex8+5pKGtmgLREwqtBozT7h9KmvkbI6PX6grF813YmoGSGWlkf2JD75BLUF48kEFq6+byxG71XGmdyT zlkHx19z 4LBVFUlj662cEjcXDxHgdQJpdLJ7C5M25A2qROZ3TasnNcnUwmz36xMorcjoMy/m+VLyyxU0AEe+KuNv0hJXB3lOcyyhn+WGBNHOIGLIGq1m+PK8bm9HeDoK1boaGkgWwJYCV0OUulQEhAfrZhTtPWtQ7mziRD6Js2CxFFqSPeCHkU2ItjZJrsQcWmnyh7EshekGrE3Us6ATl8ZJIBmet3PQno31cRYGDIf1dfgKKJ5gB+g/CeuOF8a4r+R2XvZFdYcvphq0QdUj3c2pU436+TNjYvj/GdEKTFI6DxfXUi2w0jEW0B3R6/+LThgU0+SnGRj5ySyEuhircN42/eSXb9xSi+zz0dv/F3W9+Z4ScQpADEO7jOPHuqB3Hdnat2jLd013LV1TaeYUZKguAT+2rBHEWvAzPvLUsSGJ2fOcC9lJpU7nvZt4HHOpWxtCW+ISF+DR0DWudDTycaz+unv0VjQBDTjxbSIkJqt4j+u8fb+MrgmQFn/ylC6H+urtemvvGhCJ7GQJXKACA5y//NTsr4mKWgFuTNTww43spyvkMeBCvyuUwpyLj6DISrq1TEOViAtYNs1ZYvhAgeDBsENGuSCsIWKr6fZ+S5Jl43f2MR8rHLcUg9soK4Yr85g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: PGFREE is currently updated in two code paths: - __free_pages_ok(): for pages freed to the buddy allocator. - free_unref_page_commit(): for pages freed to the pcplists. Before commit df1acc856923 ("mm/page_alloc: avoid conflating IRQs disabled with zone->lock"), free_unref_page_commit() used to fallback to freeing isolated pages directly to the buddy allocator through free_one_page(). This was done _after_ updating PGFREE, so the counter was correctly updated. However, that commit moved the fallback logic to its callers (now called free_unref_page() and free_unref_folios()), so PGFREE was no longer updated in this fallback case. Now that the code has developed, there are more cases in free_unref_page() and free_unref_folios() where we fallback to calling free_one_page() (e.g. !pcp_allowed_order(), pcp_spin_trylock() fails). These cases also miss updating PGFREE. To make sure PGFREE is updated in all cases where pages are freed to the buddy allocator, move the update down the stack to free_one_page(). This was noticed through code inspection, although it should be noticeable at runtime (at least with some workloads). Fixes: df1acc856923 ("mm/page_alloc: avoid conflating IRQs disabled with zone->lock") Signed-off-by: Yosry Ahmed --- I am not really sure if not updating PGFREE for isolated pages was an intentional choice when it was made, but it seems like we are missing updating PGFREE for more cases now, which looks wrong. --- mm/page_alloc.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c242d61fc4fd8..57872af9c897c 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1238,6 +1238,8 @@ static void free_one_page(struct zone *zone, struct page *page, spin_lock_irqsave(&zone->lock, flags); split_large_buddy(zone, page, pfn, order, fpi_flags); spin_unlock_irqrestore(&zone->lock, flags); + + __count_vm_events(PGFREE, 1 << order); } static void __free_pages_ok(struct page *page, unsigned int order, @@ -1246,12 +1248,8 @@ static void __free_pages_ok(struct page *page, unsigned int order, unsigned long pfn = page_to_pfn(page); struct zone *zone = page_zone(page); - if (!free_pages_prepare(page, order)) - return; - - free_one_page(zone, page, pfn, order, fpi_flags); - - __count_vm_events(PGFREE, 1 << order); + if (free_pages_prepare(page, order)) + free_one_page(zone, page, pfn, order, fpi_flags); } void __meminit __free_pages_core(struct page *page, unsigned int order,