From patchwork Wed Jun 5 14:48:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10977133 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0D90113AD for ; Wed, 5 Jun 2019 14:49:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F2E472899E for ; Wed, 5 Jun 2019 14:49:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E6C2D2888D; Wed, 5 Jun 2019 14:49:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6E1E1289A3 for ; Wed, 5 Jun 2019 14:49:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 363866B0006; Wed, 5 Jun 2019 10:49:22 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3141F6B0007; Wed, 5 Jun 2019 10:49:22 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2035F6B000A; Wed, 5 Jun 2019 10:49:22 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id E00026B0006 for ; Wed, 5 Jun 2019 10:49:21 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id a21so14931077pgh.11 for ; Wed, 05 Jun 2019 07:49:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:mime-version:content-transfer-encoding; bh=n/tZofbBogKZ5HTBUbSV6rWcxAAFfPOtXDpX6liTabU=; b=KMBUvTpBDrjnFZAPVN7kp7Ma7fiUl54TfUAAod5fRyGZHyG7roHNc01u7ZDv9GqpdQ 5z2kl2ZseHWV3Fhem3w3qYSFV3Yu+x1EQ0b/BfDxdKo3WMzqo7hCek4d6l0krj0VMTPp Df0whEuBcB0Wz9pTiRCF5FlA1LplcpF0AQY/Ke6oD9gM4hMa3KSkLhZkK1BKkSJ3+e/Q Sgc9I+IYw5dSwm+aHlcsCQAy6yjWA+cL0aJ2cfa2Aene6X3M5hpi5YaKmb+wM4YjbLJg tK+1G0BZH/c4zMMEczuL9SOiWsrcqfbJxn6a+mJ3OsV3TkozFcf5pNYx+ENmzZYNQwci hcrg== X-Gm-Message-State: APjAAAVfh+uI8p3XE+cSPtnfFtbZ1ol9im6wAb6c8KlUbQZizm9eEtOo xchl615/Rq+S+TkWIPq7x9Y863vae/AqJsY4MfkYe9tDOT0TM+6r68V2dPAkhh5UxApskAbtHfX ZOtFlJH9sKCc4rzFm5ttCfD3qad/kXqoSyEwOFh1AuUjga/GIqc4izKjpdJI/T7xxgg== X-Received: by 2002:a63:6e48:: with SMTP id j69mr4928484pgc.34.1559746161267; Wed, 05 Jun 2019 07:49:21 -0700 (PDT) X-Received: by 2002:a63:6e48:: with SMTP id j69mr4928322pgc.34.1559746159737; Wed, 05 Jun 2019 07:49:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559746159; cv=none; d=google.com; s=arc-20160816; b=TX56CDykETvSfuLAVli1vQUy5qS7Nck8T0H70YUKPE1JQIsQyQzgMIjbVcqiQooTIs cwWfp7dnX5EV3TI58lyPiXRRi4Mj1VugvOGkZwsNEQUvVuqCx5Na7XwB933cLHkQ7XlY a9i9QtntG8cR7+VoulXjTlniQ8qXr2IbwW2VoILCVlkAuZuUbeEYqjv2fF8lwDTRn2SL ygul2vqWKItrIQaGmPwiEuGSwaoFn3+SxWuElBsjmPXcM1Cef9aG9Cfa5rt1VRqy0f5r gMToeofihsPbOYKh8khyPX/QxXbMr0a3/gGuIvsBP5j4XY1cfY/lM6cHw271Evmjoh8h zT/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dkim-signature; bh=n/tZofbBogKZ5HTBUbSV6rWcxAAFfPOtXDpX6liTabU=; b=MJ+HiIMYYY/4/WsO2XdqRVOFWAbfoS2hoZ8ZWBPKLNIAktF59pQzL4hcCCxZZuk+73 oaueLy5I5N5m4zDhVusZFE6VZm85aVltMNAQ0P8Z9r8oTHiEeCZJkxFktlLQq/eJx+6g r5Z0BQ+8AIegktnWo5OwXguMmWwXI5J1Ipu0rFNf1qAR9WZlZd3/Mv3rsqUnesqkzy1d foWwMEQOPJwhrc4MzK6+AjZtbogXKqWIkZQpK8dP15o1kH2zwqRVz5HQ1UUKaDre1ado KVUu+93Bbdo/1WPpSOVrpFSSugnvE+Fqm+XR9cd5YehhqGWzdDuoJh67Tf0Rd9YORn// WW5g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=Xt1p4DdY; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id j19sor14396498pfr.25.2019.06.05.07.49.19 for (Google Transport Security); Wed, 05 Jun 2019 07:49:19 -0700 (PDT) Received-SPF: pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=Xt1p4DdY; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=n/tZofbBogKZ5HTBUbSV6rWcxAAFfPOtXDpX6liTabU=; b=Xt1p4DdYpo2iTbRqldHe1m4pbYWezS+2NvUE8s9WGrvQ32mQm/SvJ95a+v5FCwvhdU uqsVALe56+yi0vvM5IqsD2+B0U3wt3mi4jPtG98X84/K4ZoJscczxTB2FTr2kalU7QK8 QSodgzdiM1cHC8jLZ18alaSGJ3nAp3XoBqOFU5A8CEy3WV2rlKZx9RxnQOHgpV4zo4oJ CppFN9hDpmIhW3P0WfQKj5WDkYmLQKU1W+qF8L5Vjn33pvdVB8asrbfEEzdHdBvTGSPo N6+kKFpe8QK6I1vGkt9zZfHzCU4hbgxOeDyHQC0m4Z1/n87MhLnjpbJkVj+rPlVtc+pG tveQ== X-Google-Smtp-Source: APXvYqyBbhev4RKfSBMh3SGEtpfz2O+6DtVwVokbLlv0pD8pFHlxodfBt6wDsYpKmza793o7PvgMMA== X-Received: by 2002:a62:2643:: with SMTP id m64mr45312432pfm.46.1559746159262; Wed, 05 Jun 2019 07:49:19 -0700 (PDT) Received: from bobo.local0.net ([203.220.89.252]) by smtp.gmail.com with ESMTPSA id m19sm13375840pff.153.2019.06.05.07.49.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Jun 2019 07:49:17 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, Andrew Morton , Linus Torvalds Subject: [PATCH 1/2] mm/large system hash: use vmalloc for size > MAX_ORDER when !hashdist Date: Thu, 6 Jun 2019 00:48:13 +1000 Message-Id: <20190605144814.29319-1-npiggin@gmail.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The kernel currently clamps large system hashes to MAX_ORDER when hashdist is not set, which is rather arbitrary. vmalloc space is limited on 32-bit machines, but this shouldn't result in much more used because of small physical memory limiting system hash sizes. Include "vmalloc" or "linear" in the kernel log message. Signed-off-by: Nicholas Piggin --- This is a better solution than the previous one for the case of !NUMA systems running on CONFIG_NUMA kernels, we can clear the default hashdist early and have everything allocated out of the linear map. The hugepage vmap series I will post later, but it's quite independent from this improvement. mm/page_alloc.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d66bc8abe0af..15f46be7d210 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7966,6 +7966,7 @@ void *__init alloc_large_system_hash(const char *tablename, unsigned long log2qty, size; void *table = NULL; gfp_t gfp_flags; + bool virt; /* allow the kernel cmdline to have a say */ if (!numentries) { @@ -8022,6 +8023,7 @@ void *__init alloc_large_system_hash(const char *tablename, gfp_flags = (flags & HASH_ZERO) ? GFP_ATOMIC | __GFP_ZERO : GFP_ATOMIC; do { + virt = false; size = bucketsize << log2qty; if (flags & HASH_EARLY) { if (flags & HASH_ZERO) @@ -8029,26 +8031,26 @@ void *__init alloc_large_system_hash(const char *tablename, else table = memblock_alloc_raw(size, SMP_CACHE_BYTES); - } else if (hashdist) { + } else if (get_order(size) >= MAX_ORDER || hashdist) { table = __vmalloc(size, gfp_flags, PAGE_KERNEL); + virt = true; } else { /* * If bucketsize is not a power-of-two, we may free * some pages at the end of hash table which * alloc_pages_exact() automatically does */ - if (get_order(size) < MAX_ORDER) { - table = alloc_pages_exact(size, gfp_flags); - kmemleak_alloc(table, size, 1, gfp_flags); - } + table = alloc_pages_exact(size, gfp_flags); + kmemleak_alloc(table, size, 1, gfp_flags); } } while (!table && size > PAGE_SIZE && --log2qty); if (!table) panic("Failed to allocate %s hash table\n", tablename); - pr_info("%s hash table entries: %ld (order: %d, %lu bytes)\n", - tablename, 1UL << log2qty, ilog2(size) - PAGE_SHIFT, size); + pr_info("%s hash table entries: %ld (order: %d, %lu bytes, %s)\n", + tablename, 1UL << log2qty, ilog2(size) - PAGE_SHIFT, size, + virt ? "vmalloc" : "linear"); if (_hash_shift) *_hash_shift = log2qty; From patchwork Wed Jun 5 14:48:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10977135 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C80F614E5 for ; Wed, 5 Jun 2019 14:49:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B9D6B289E1 for ; Wed, 5 Jun 2019 14:49:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id ADD6C289A3; Wed, 5 Jun 2019 14:49:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 965DD289A3 for ; Wed, 5 Jun 2019 14:49:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F0646B0007; Wed, 5 Jun 2019 10:49:25 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 29F396B000A; Wed, 5 Jun 2019 10:49:25 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 18EEE6B000D; Wed, 5 Jun 2019 10:49:25 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id D6C3F6B0007 for ; Wed, 5 Jun 2019 10:49:24 -0400 (EDT) Received: by mail-pl1-f197.google.com with SMTP id a5so16247099pla.3 for ; Wed, 05 Jun 2019 07:49:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=uokpXhUGAbY6p5XmmfFzDRpYAOfHcWN+Kl+0vKMAotA=; b=WmXmu1DusMhn+t/r6aO7AEnBN42Ag/w33+OvG2CV9l+R2dw+VHQXhxMyR7Ec2MmsyV lw5qwRaZJJlEvsD5m1FzPpcgBNJWj0dovO1YZV415eQX9zmzYrDJ9jWAD1Z6PGO7ETYC qUSgByaOSLBxTOk3eHQV3O/nsJqVpEOXlEdsPMRUlvmLqdOT5MGFSGVszqJ5BDnaOt93 LjqbNXP9Sqbu05hh91CKTg5SukTXg1Npor8AGViXlHwlqe6GhM6BkUBKddPg1DWKeMn0 1O6hLGfzTaBAVRxIiPPiEaeyUzPVt3v75gfylbFLhRLMGxgha8blDm+AzrHFQxdx/KHw c+BQ== X-Gm-Message-State: APjAAAVbCtdY7ZDAfBUR+mmgLeaV7AUMkJzSBdL4GogNzlK5Jx+zm38E 2tRkw7rZ0I5SpK2OPQL5sbFlI4+aLk2xpbYiV60ZPmN1o6BRhKXcmBGl4BGWXiEx4GN3p2McsI9 HqJReEkvWf1ogdwoqhlCfzIzFAA+UI3iAFrbTV1kd74Fpj5MIh70+2Dm+whmiEhEUpA== X-Received: by 2002:a17:902:42d:: with SMTP id 42mr42463485ple.228.1559746164355; Wed, 05 Jun 2019 07:49:24 -0700 (PDT) X-Received: by 2002:a17:902:42d:: with SMTP id 42mr42463328ple.228.1559746163009; Wed, 05 Jun 2019 07:49:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559746163; cv=none; d=google.com; s=arc-20160816; b=a5KMQ379CU/Wv/MqVQPN5zn7RInSVMoZ6Z+XG4mChHFHodvtl/1KV8kdlmbrd9dBBv lz5QheovgOzAlRWJTNJydoyPTgJ/pf7eInG3Ao04bjlD4esZfBsSJYUf4bnqzTN0zknf EPNmwmPVFkctkdb9SwsyC/AekpE41JNkq4z69c55QsQDlPq2bRNST8YfBjl2xV+DgMJA LFTblGJuZxOrwuzAZHvWnJqKg/vCaKrKVD5/e++SC3zyLKOyAPgDuK+yG5a1EtKL15fr ALdUNr54IPwCKHYjwt5MCWQauvB6aQmuCaFyp8maEpJvfFKZYhE2sBxClwDbYNRiFBfP T3gg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=uokpXhUGAbY6p5XmmfFzDRpYAOfHcWN+Kl+0vKMAotA=; b=RKha9UCEqbgM7TzB1T49nF/xaFU9BzKk3UyWQecyokn6wfW+U+R3VO4r2oonSEdqaH QTHZHI8grlgiksn6g26PI4H8m5yuIpgjkJqJdhJLV8H+uumWu0Hi8hvXxlejQO5kLMQn oasTNF+bl+yM1erweX0aRnMCaIZ5weypsv72cKqxzMOQ+KDeKId+UtuhCO913dhIcEPy 0jp6cWwjvQIJrjbCESIXUTEYvcIiEcQBZ0j26o7Jhr0J20XyvbspXQ0oHW1TYnHgTiRK NicG5puFkpwqrz5O9Qk/hMHQ/d4RbI8QCDGmOu/CMtF95SF4dHq1suXSqxgaVwc/awgQ UI0w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=J2IZunxC; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id q16sor24358175pls.29.2019.06.05.07.49.22 for (Google Transport Security); Wed, 05 Jun 2019 07:49:23 -0700 (PDT) Received-SPF: pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=J2IZunxC; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uokpXhUGAbY6p5XmmfFzDRpYAOfHcWN+Kl+0vKMAotA=; b=J2IZunxCvI+d08sLrBzSc3RRPxO2OwlM39JK3Wfz7NHVngOBAtWqR8LmRGyHdMML4H xJ2memtPxZu00hLPKDBQq3Kxz797fYuSHyz5MjFeuGvOGgdFli2RaOk5+fCl8K4RUByH zTyfexL/KNSXqkjKSBxsg1gZsKpQTyVhwUrXQjXGO4pG0TSYNyDeCjnb/skVPGUmy0mf ILTmQLnxJ+gq9qzMJv6iRgnIiHoZZNuXwF0++xu1MmCaYxdA47zGacG7Tnhqr/6kHJ0Y 8L7tgd4Zfm/GQ1ySDh3QfXdINL81pLmMoedb74dlImuilFm4D8it+aDUxKka7xcx2ezR kYeQ== X-Google-Smtp-Source: APXvYqzclwzYKdfqEnZ6+MDN2VJ6wxc2CPFyVlmBhB2yRZPhVC1P4kxcpKFPMHk9z1/c/77J1YvRaQ== X-Received: by 2002:a17:902:e306:: with SMTP id cg6mr15878349plb.341.1559746162576; Wed, 05 Jun 2019 07:49:22 -0700 (PDT) Received: from bobo.local0.net ([203.220.89.252]) by smtp.gmail.com with ESMTPSA id m19sm13375840pff.153.2019.06.05.07.49.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Jun 2019 07:49:22 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, Andrew Morton , Linus Torvalds Subject: [PATCH 2/2] mm/large system hash: clear hashdist when only one node with memory is booted Date: Thu, 6 Jun 2019 00:48:14 +1000 Message-Id: <20190605144814.29319-2-npiggin@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190605144814.29319-1-npiggin@gmail.com> References: <20190605144814.29319-1-npiggin@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP CONFIG_NUMA on 64-bit CPUs currently enables hashdist unconditionally even when booting on single node machines. This causes the large system hashes to be allocated with vmalloc, and mapped with small pages. This change clears hashdist if only one node has come up with memory. This results in the important large inode and dentry hashes using memblock allocations. All others are within 4MB size up to about 128GB of RAM, which allows them to be allocated from the linear map on most non-NUMA images. Other big hashes like futex and TCP should eventually be moved over to the same style of allocation as those vfs caches that use HASH_EARLY if !hashdist, so they don't exceed MAX_ORDER on very large non-NUMA images. This brings dTLB misses for linux kernel tree `git diff` from ~45,000 to ~8,000 on a Kaby Lake KVM guest with 8MB dentry hash and mitigations=off (performance is in the noise, under 1% difference, page tables are likely to be well cached for this workload). Signed-off-by: Nicholas Piggin --- mm/page_alloc.c | 31 ++++++++++++++++++------------- 1 file changed, 18 insertions(+), 13 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 15f46be7d210..cd944f48be9a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7519,10 +7519,28 @@ static int page_alloc_cpu_dead(unsigned int cpu) return 0; } +#ifdef CONFIG_NUMA +int hashdist = HASHDIST_DEFAULT; + +static int __init set_hashdist(char *str) +{ + if (!str) + return 0; + hashdist = simple_strtoul(str, &str, 0); + return 1; +} +__setup("hashdist=", set_hashdist); +#endif + void __init page_alloc_init(void) { int ret; +#ifdef CONFIG_NUMA + if (num_node_state(N_MEMORY) == 1) + hashdist = 0; +#endif + ret = cpuhp_setup_state_nocalls(CPUHP_PAGE_ALLOC_DEAD, "mm/page_alloc:dead", NULL, page_alloc_cpu_dead); @@ -7907,19 +7925,6 @@ int percpu_pagelist_fraction_sysctl_handler(struct ctl_table *table, int write, return ret; } -#ifdef CONFIG_NUMA -int hashdist = HASHDIST_DEFAULT; - -static int __init set_hashdist(char *str) -{ - if (!str) - return 0; - hashdist = simple_strtoul(str, &str, 0); - return 1; -} -__setup("hashdist=", set_hashdist); -#endif - #ifndef __HAVE_ARCH_RESERVED_KERNEL_PAGES /* * Returns the number of pages that arch has reserved but