From patchwork Wed Mar 27 19:01:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13607295 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FD51C47DD9 for ; Wed, 27 Mar 2024 19:01:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA3BF6B0098; Wed, 27 Mar 2024 15:01:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D54456B0099; Wed, 27 Mar 2024 15:01:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C1C096B009A; Wed, 27 Mar 2024 15:01:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A3B836B0098 for ; Wed, 27 Mar 2024 15:01:16 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 5CC4B120756 for ; Wed, 27 Mar 2024 19:01:16 +0000 (UTC) X-FDA: 81943736952.14.9060128 Received: from mail-qv1-f54.google.com (mail-qv1-f54.google.com [209.85.219.54]) by imf21.hostedemail.com (Postfix) with ESMTP id 2D2361C0022 for ; Wed, 27 Mar 2024 19:01:13 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=dvov4tee; spf=pass (imf21.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.54 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711566074; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8GxQ/m2i2KHb6JEyfIRJnzrg7z0UGo5xSQekII3ANKE=; b=B3l2v+0Z7qx9YfKRO/vHOvJSN5Y6yiB9MOwWhzt84hIvWGKwJhjuqTg8vHSmE1YmojlKMT rtEkmkQZRPWPzX9ljp0QeOndj0p4zpTXITVMx1UPpEdpoeKFqjblmix+uEKxDvauQlstsv GNe1KXDi1w6701HmkLZP7Fd5kQqe6W0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711566074; a=rsa-sha256; cv=none; b=6nqj2LYWrEU1/3uivd2H567NkzkzgeEMxdBm6wB6YttAfnBditbOGTFcqUBpvSDVRhJKLg WOI5yUpvEk2S3t9xbw9/r/f1du2m4dGDLF+0SGCOQ94J8i5Fw0Npbt63AsC7xItFsC5yjJ KpbXNG9J5Y7Cv73RyEKb2fib4T5hPrs= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=dvov4tee; spf=pass (imf21.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.54 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org Received: by mail-qv1-f54.google.com with SMTP id 6a1803df08f44-69625f89aa2so1479236d6.3 for ; Wed, 27 Mar 2024 12:01:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1711566073; x=1712170873; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=8GxQ/m2i2KHb6JEyfIRJnzrg7z0UGo5xSQekII3ANKE=; b=dvov4teeaC0DFW1ykntLV1zK+SHQh8WwCNjb/EHeVxlZQCG8QwYFZR0rKHUCCfJr6z xDZ77+lY6DJmmvyZ1cJDGjcrCZbOCeURQcc+8YB0U+YOiXcxn88yHk/wivLGOQUMfkUa HW0zcwY2P6iArlmfLHpPaBPWo0cuEbWtoyARkjDNe1rIAsMkMINC8iF6HG4jVz/AKQDg B+sgYIJv9uafZe4JXIpWag8Xf6ycBw/fT7J9HEx807z+slk0xi5+xAf2hFp3feYLBUqA fQeP6i/55rAUz/jj9mwUseH6uSfGRNnaI/bkiZFE5d+Ubtbr55Ptw7vI/d7Dx9iVS2DM iDAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711566073; x=1712170873; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=8GxQ/m2i2KHb6JEyfIRJnzrg7z0UGo5xSQekII3ANKE=; b=Q3yTrA48P/wu509NT+uvrWmX6Qne1O/I3SqLxpIYfURmnEMNqXgJ0qsklLtA9nwtoz CI9Id2Dy6Yk6taZ3YEY2q53sHoWxHzKE/y6vUcJ8jAWw3Lce9H9aoSgO74KUVrYhXJl6 LfYl/Z+zmV7DS75fANUuKgrchS0jFqS5pxFJVchSsj93+Qmx0D+rY1uDfvURhtZvWG7V 6TC8w6sYvVzr4YrgdD8uoXYbKocCUjc4NR+pK0k1QomUMzY6wXb2Ts6dlUg1fiEh8mWZ ikdQ1Yk27hfWPJbVtPaFiXMlDXC4bAcscQi2bEUiiW9MeOIIrllg6bF+SnGTffLg0IiY YCcA== X-Forwarded-Encrypted: i=1; AJvYcCX9Le/B+oBbgOU1MY335hlNxXERROFTwJ2J+yq6VJFSUhCB5bto6EuHDUEwoae/eVCTXS2Hw1YA4p67jd2QQo3/GBw= X-Gm-Message-State: AOJu0YzWdzYZJMSUOnDkS08766piNrVkh26oC3v76AGtj805D5uZPlBU ON4M1OkQqCJYPQ0CYU81dXUPtEzjy1LUM+uCsO+hhPtxF7b50GAqrKWhKdhsv7A= X-Google-Smtp-Source: AGHT+IEMWc0z3NWlQR8No9ANDoskQvZ+0Bj532IQLCTNjDCCjXJTlv4aoJ+acHAsX68VVMDqZgNiCg== X-Received: by 2002:a0c:f085:0:b0:690:734e:50c9 with SMTP id g5-20020a0cf085000000b00690734e50c9mr540594qvk.24.1711566073038; Wed, 27 Mar 2024 12:01:13 -0700 (PDT) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id 2-20020a0562140d6200b0069778761fb4sm154033qvs.73.2024.03.27.12.01.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Mar 2024 12:01:12 -0700 (PDT) Date: Wed, 27 Mar 2024 15:01:11 -0400 From: Johannes Weiner To: Vlastimil Babka Cc: Andrew Morton , Mel Gorman , Zi Yan , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/3] mm: page_alloc: batch vmstat updates in expand() Message-ID: <20240327190111.GC7597@cmpxchg.org> References: <20240320180429.678181-1-hannes@cmpxchg.org> <20240320180429.678181-11-hannes@cmpxchg.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 2D2361C0022 X-Rspam-User: X-Stat-Signature: kc3jnip3w9ydcq37ip6pior7e5mhzaq9 X-Rspamd-Server: rspam03 X-HE-Tag: 1711566073-998249 X-HE-Meta: U2FsdGVkX188c44Ie0oj+A/jvcbjiy0GcoydD/dF2o9TCdYYAB8pWUcyrQImsgGULZErgZ1RYWl4JmmPwnBH+L91AONLosFo5NUPJOzbDJPhgmGYwwjbYiSAnL6J3h3rgXor/eOGVSSwQBjT3CdkvxrUvN4awzQHhyH+3HErmGvjeSjqVTRXoJq7/M6liVWtwX8epXn08QDIYeVkOlyG5c3e4EwhZAdu3+uzk8qfwnPm7suJ1JwNuZakW9RSbfc0j9l1lqbi6voS9vg4eYW8S6GJkS8SrDrdJy+RwxlPurdnnWjaXbRcXqI9IA9PusLFlQQ34LvJmIDCcWLcZUYiLl8gi6rRvMbCv5ODZjau+TrM9Y95KTW3md7AO08hVEBHf6uU/DvUXK/8ukwInGXbRHV9tuRU6TjHCIAFQ+pYa8DNNGV9EipB4M5mkt3dMhOqON9R3KXfVCfLlL/Z5zoqpNgve4N7fSVBlFKJ5Wq/QFgPs9wCm6/RhCNdi1oxLmGFvhzUQeVAHqr3/gZTtZ7oz3SdoutbM3a8lZ1VMEY3/ffHgIhE7Ik5acRJsW1Iob6vAQH77hbfDRRFluy5irBrTXLj6MGp5owTiGUhHD7ZIceTsbiyHrKFDjZYlX+ijWgQUTxw1WxEKlF0DGS7m4rzrSsVlgHfACclI3ka2NGuovRYta38lK+lVSZd/YHfxTw67Lb8DgAY2xTfuJ+Phw1iuWtOd5drNH4mS70d/XX4X/1l6WpHCJeV8D08BTUwJi9F3hsxZ4mteDT9nAe6g0jhmL/c+tm6ogEA6j79cYittRNrxfxOSEZkaffVjxn5remSw+AR+hb6lMqpAkaxDHXRElz3XULHbpcma4N81Zrf88HZVYlLjWUfB5dcs2MG5SQRr3xilMvcbf2S3Q2o9hdIyXNJnLc1PWIneMm3JnLU2NQdQmSWUyz3x+R/tx4o03FIn0/opOoPnjEkQcibEuI leJkzl4Z N5OcT8fE8YN8vJVGXlBDT+o07cNV/whgbqeGQBDuUy9KmP1FvBJYFzOwOYOtWSkY6/xBEZkn3VaezGvqk57n1GtBG+XOS2BaJnZmvEiLEfu8xaIeZ3cg+2mICkOvfhsGjQRMVmuqQVsxJSb4Bt/H/ORdsaXV4DXnX3vN75sHjBzxMZEHhyazvKdckUJijel9ySsgery/zc45rNbHKWld2liEwA1Ldx2Tjm5U44mUIh5qlkTTtV93BjYp0YtaMlgmrzlq5zhmn6LZ+zbGdmCvT3Y10q+SqbbZCJV7XQTSF/VPghSyW+txiDUoDUHf/y06tlQVg/3XV3lKXBoY8orGyuZVCtLHDo4tf3D9gMibHwzdjrSiTGPDi4WSeMvfBEwSO2ARRnqCW3Pd6q55OY/LLBipXkUQr0IpOGmK+IR8v8VDcnRQiIlmYv6AWBIwy95E8p1WJ2olIv1+OIRJkf5xG7Aab2/KeBYjvPPKca4Lmdycw0zjXvH7xbhHH5w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Mar 27, 2024 at 09:54:01AM +0100, Vlastimil Babka wrote: > > @@ -1314,10 +1349,10 @@ static inline void expand(struct zone *zone, struct page *page, > > * Corresponding page table entries will not be touched, > > * pages will stay not present in virtual address space > > */ > > - if (set_page_guard(zone, &page[size], high, migratetype)) > > + if (set_page_guard(zone, &page[size], high)) > > continue; > > > > - add_to_free_list(&page[size], zone, high, migratetype); > > + add_to_free_list(&page[size], zone, high, migratetype, false); > > This is account_freepages() in the hot loop, what if we instead used > __add_to_free_list(), sum up nr_pages and called account_freepages() once > outside of the loop? How about this on top of the series? Could be folded into mm-page_alloc-consolidate-free-page-accounting, but for credit and bisectability (just in case) I think stand-alone makes sense. Reviewed-by: Vlastimil Babka --- From 361f5df28183d85c7718fe0b579438d3d58777be Mon Sep 17 00:00:00 2001 From: Johannes Weiner Date: Wed, 27 Mar 2024 12:29:25 -0400 Subject: [PATCH 3/3] mm: page_alloc: batch vmstat updates in expand() expand() currently updates vmstat for every subpage. This is unnecessary, since they're all of the same zone and migratetype. Count added pages locally, then do a single vmstat update. Suggested-by: Vlastimil Babka Signed-off-by: Johannes Weiner --- mm/page_alloc.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 8987e8869f6d..13fe5c612fbe 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1341,6 +1341,7 @@ static inline void expand(struct zone *zone, struct page *page, int low, int high, int migratetype) { unsigned long size = 1 << high; + unsigned long nr_added = 0; while (high > low) { high--; @@ -1356,9 +1357,11 @@ static inline void expand(struct zone *zone, struct page *page, if (set_page_guard(zone, &page[size], high)) continue; - add_to_free_list(&page[size], zone, high, migratetype, false); + __add_to_free_list(&page[size], zone, high, migratetype, false); set_buddy_order(&page[size], high); + nr_added += size; } + account_freepages(zone, nr_added, migratetype); } static void check_new_page_bad(struct page *page)