From patchwork Mon Sep 18 23:01:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13390562 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29740CD3424 for ; Mon, 18 Sep 2023 23:03:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 875FB6B0461; Mon, 18 Sep 2023 19:03:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7FF796B0462; Mon, 18 Sep 2023 19:03:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B6856B0464; Mon, 18 Sep 2023 19:03:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 318D76B0462 for ; Mon, 18 Sep 2023 19:03:04 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id F172A140B36 for ; Mon, 18 Sep 2023 23:03:03 +0000 (UTC) X-FDA: 81251245446.20.839B0F4 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf08.hostedemail.com (Postfix) with ESMTP id AA618160002 for ; Mon, 18 Sep 2023 23:03:00 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b="H/2MiC8t"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=ro9iDMxE; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf08.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695078180; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aGyfKWmz6L5+jK4Oa5XR9qAiw4qVE4/FVGtZT6ufHEc=; b=w4FOBeuSQtAJgrwyWIOCHqkQ1TsHfW4HtPACboGDfuyfGyo68zsCA+SSiEnPcRZX3ouB4Y xG0M1Vs1r76q1f0DCaKe+t+8QQGXmBAJdvSAyNqVFIz++EyhjoDH2rQA6QgUnhE9jFX/eh 9bcEbjDaFayR7m077gEAPtFbw7kpt8w= ARC-Authentication-Results: i=2; imf08.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b="H/2MiC8t"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=ro9iDMxE; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf08.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1695078180; a=rsa-sha256; cv=pass; b=1iXxj7DS6VPWZWiW4Em8HpsNe6Vi6hLEeR6dE4a8XRdu03sof7Qc0nS4ptwJceSYG6VLwx yASiIwS7kkuf1C/8uQm4ohCmh+JpmbpT4nFIfBVQZAX5Ppg/os84dPUrEQnOOsT7nni6e5 1CT8051BrNoGp3rHVnXTm8nUcmGPI7k= Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38IK3xs7031273; Mon, 18 Sep 2023 23:02:20 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=aGyfKWmz6L5+jK4Oa5XR9qAiw4qVE4/FVGtZT6ufHEc=; b=H/2MiC8t2yAXPqKgu5r5jjRUgDy/WJLA0V1y0TzCx++s8Ef1crtwNY2PzKpedYr8CXH0 BeACQkZM5lle23qp7JLwBTvu2osgdAu/uQ5M0EOh1OuA3FDSxD7aZCfhgezpXnlaU0uY /7tC/8v7sM829+wS1KfJLtPzzbRzPWpDyIK/xbq3lvmxVBKVLCkiGCuvOVx+S66x3i4q N5EqSBC2h5+jehEZdhRGRHq/jz0l1sB14vzSF9lhWZJDHPbtjvZNOkjK6UPEZxu4B+Rk tG2ajwER+aD5zWPLdV4TfpdbwV8Ja14yK3scHKxRaqydcX5/VUmMX5CHo3nV4MKpDStL 8w== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t5352upry-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 18 Sep 2023 23:02:20 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38ILV2Hj002042; Mon, 18 Sep 2023 23:02:19 GMT Received: from nam02-sn1-obe.outbound.protection.outlook.com (mail-sn1nam02lp2040.outbound.protection.outlook.com [104.47.57.40]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3t52t4nvs2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 18 Sep 2023 23:02:19 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=l2eVagItC1t6taDapo+wBiH/fJOOgFc9ijxF4DbZ0F0nrtjDiqY3iw4YWWuuxYdtODW6izMQ4Lm43ooCcDOHzMq6dZq0qr9F7OTqOJb8S//vA5zZZTdcn93EjK1oka4mgcW1ik9W3Q/aES4pfO5kKZH8hjY3hMgyQ6qClh7j/b/6oOHMiy7ebza1416zKh/w4QiZMwVmddrVJQJG1Ntc8viX7/ocz2pfAOeb2Pj0LLUSIaAOnh851tYutRV1R+95rfoJJJZ4HU+2BQLJfx+TJ6nSiUsQpKLF6cEfViktAnuFZptRYYI4IC+Y9ji/DzQ3IdsYgex2qBZyT8EgyY3GFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=aGyfKWmz6L5+jK4Oa5XR9qAiw4qVE4/FVGtZT6ufHEc=; b=fo7AffVd0B69gwRti465K2Ls7BQVoEL0eybIciERbDHQ5cDMRa3YhuwXwFXZewyXFbOZ+j0Vx0zxTj0JKEq+Xv8NTNt1VHl7eCa217JuxyIm96dn+nivHbyNbXXogzZl5KmNsa3EXDtfIYwKkdIubOUxFYns3A3WivfB2+zj9Gi+6ZrlQyRKFxMkDWdzEVLoHzQw7gueC8MDp/6k/sU59lyjK54SFJbAcVKOuizeODrDuZhaAvQqm7cTChbY68Jn1KxpVRG0SxbLiZ4THUYguQ/xpRuWfx6ULxdqZ3nadGypIHUQX3g4+Nvm/K6MD9+pIbafi/J722a2+f6xfgPycQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=aGyfKWmz6L5+jK4Oa5XR9qAiw4qVE4/FVGtZT6ufHEc=; b=ro9iDMxEqtmzqV2tG+f3REUu2cHIPmQjdJYVQFIkBMfXH4DSZK5mKIKp2Nkbetui4Huu6N3xmg4wKyydkR4GOjjS4K1XogWIomLR0lFdBYtYl+QziwvY88l69gtqFdaOeHOCVlpugptCj+HEEy8/OQvtzNzLZSB1nOkKzYdZsXo= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by DS0PR10MB7204.namprd10.prod.outlook.com (2603:10b6:8:f3::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.24; Mon, 18 Sep 2023 23:02:14 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.026; Mon, 18 Sep 2023 23:02:14 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v4 3/8] hugetlb: perform vmemmap optimization on a list of pages Date: Mon, 18 Sep 2023 16:01:55 -0700 Message-ID: <20230918230202.254631-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230918230202.254631-1-mike.kravetz@oracle.com> References: <20230918230202.254631-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW3PR05CA0018.namprd05.prod.outlook.com (2603:10b6:303:2b::23) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|DS0PR10MB7204:EE_ X-MS-Office365-Filtering-Correlation-Id: cf4c3a7d-1c99-46e9-dc3c-08dbb89b4c8f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: FuKXB8Zs31XdTsAKVrof9dhZl/6e4NvWEaAK/badTca5Q6q0BYnBtq+QdLlQYaunDMOnIl8W432ctcm7FDVp8LanDbO3BaMvXuh90KNpCMtx8DAmCSS8csI09THh5Bs7uEltO38uMLoVFBeRePhAa4pJN6nyP3+KIMQ+LlxgoOkhUR8TFzxnrmyXQhtxMeywhEIOO+/8lLHEl3au21xgUZ7o9fz4gq33t3HNFnVoLFdrr/mLY2rM/r6Nfy+rBteiftiCIuMBuZCDKGwYc7nfYfd8KdEjEfR4ZPYdLQ0fepJN9cqg/EPIV+eRtIt4qEhG3GDjTyRJ4vm5sNQkgpfuEOcuMWgm4o/EyhWQ/xzzux6o7Ekj1nXCHjetVIf8JD4p0GgydHpVin/zX4hqjR/7gLyhsKm1v1GcIGu+TDyALuN8b8JwBVL77jea/7WTyIE0rgU/gkXiCZxt+NA9tPyGN6UmIuyIHptRB23xieoCKniRPObLR/FibXk3xnCK4lGujNsP+zvpSJ+qTtt7qXCyluYUiFRKkwTU8zFmdKqLZsolyvAUkYWKqqI9WB3C9w2X X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(39860400002)(376002)(136003)(366004)(396003)(451199024)(1800799009)(186009)(6486002)(5660300002)(44832011)(6506007)(86362001)(6512007)(54906003)(316002)(38100700002)(66946007)(66556008)(41300700001)(66476007)(6666004)(478600001)(8936002)(2616005)(26005)(2906002)(36756003)(107886003)(8676002)(1076003)(4326008)(83380400001)(7416002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: eSKkpDq7NJZ4bvnqsFWMSIO7S4oL2wMHn6qIyxcpJfBHiIj0Z1991Yffpy/R6bLOOi109vK+5oWUnHcRS9QqWQQ77G4lxdYtdfCz6PWCkX580fVQ7LiVrQoq7Fpo2Ke16SdrV74DCoEvAUDVYemzpXOaG7yXqMGldFSpEMvRgWMHzABpG5bOief45E/GY9fa/+83Eoa3V3eEHCExq0Q5biS90rm301dCW0S6W3zTTJ8FXv/Q0vZWKSSqPD/btmRRd3gdYxgBIj0+hhWH/Bq2WKV/VAR2M6C+Ky3B3NH9XUVcJiXY3HY5x1nC4EH+r77wZTi4SWQp4F1MADXiJbe2K4gZKsrxrg0Pp50yq1ytYz3O0R82Gspn0usK9p36A0j1WSYPVGmWw2DSyfwzd3GjXM43R7wFlfB9wULF0aNNk1bk+d0XehlFXryBQPPCJSE5qe8fzyH67mUybpMVvXWoo3IT1IsWMB04GJqQ4S2tG2Em2sGK3gr27jG1zsuHz50q/7uLY7Zn/4W2UJlviCUR2XQucYJzLKWhIHIyUC3Kh1uSRW+rQxQKRRQ5BOkoNlbko8YgEQsSKXRGDQvfHoYnSRCkH1ZHr+d16/pL3sQv2eSArXRnKGsGxtKjZKgVg3CP4O56C7OjQy1jA1tjbewUbaEYkzvKMZJYk2Zbu+tkRYz2q5U6Wuq3L0x809+D3uAqJ95eyt7BRWTXB63DGRAcTQdODoQ7KG0PMiwrgHRPwV+QqLGUmMp7qOVZ56J55bKwaTmdvBth2YYzNydRlXDUuMKjbAhyEcNBu5Gq8Mv1FMGKbZxkuNxJQ5zw+pZLh3bCRoM+nsvkOaI2U0zu0R950RW/yTLurQ4nlub6E/0cJGaefOBf6YSr8momZ1lo7VfWfjvqtyKeCmRiMXh1pLgsiTrbcOAjhHbvKfDs83DKkvLrKoTSwX3M1DBQoSybs7bpHvWS7iMKLXNo+4rmlDfAsHkty+OQgdXY+Rf69EhkVBdFgmnsNeOcnToi8YzFl9kf3wftMlSf9pyn7gyEoZLk70Wi6FAs0rd+CJ40CWZTIH/JLZ7wsalDu4MBEE5INlmCpH1EMG4Zc9gKjvHDot1sN62bC45J9fH8MkLroWCNdNQV44BDXtQDsJHsxl+RpArssaDROO1Ai2JImRcGx7GYG4be2EWLkvSeVsGWORD6bY3luz8V/5t84UMVwfwH0xCMuwB0WH4D9N2NIqzNAghq6AHU4AAFnjiPoDd8BXU0Dg4sgnCeTTCeSaLYFUOQLAhgB/5fep/MWXkYnEglBD9+5KetpVtSsEP4F7e3i3BQL85M0we4dlSBSUqZYQjp6JFCvqoI9vy3Owy8hLGm5EC2Ks6z0qk9BGQdmBcuAuBz3mAiNJ8c6IKA7kOHV3t8yPu4SmIYYpE/kQwur5DVzFRQDhrQIkOTlYiUPNIArB5u4ZZ1sEQdkc7jGu1TfK0rxD6iO4CGslD/sO7Yh8hEP9/19ziiStSAlXFNe72yNcSunPq/etfjmhlw0xDb8V7xnxmGiwuc0LPMj4lRIXIn8Cu0efoyPFwCB0+SPNMgtbawwDUs2iJGHKuMWOPJMvGL+tsM X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: /OXf8B4pm+LE1lDf+QICMnVvhZIiKBqdO4AHwo52mys4yqpEOSKRtX5Hvj0pGk/pbeQqzISpIMpwk/c+o9IpEo5fUVAdqIx/o+frDchsRFIoBehtpepqNcGuM9RbXJraKU3FYdxjO9DGSxZkMViNLezBDmcEKcPz5i7nr2Q8t7AlMWw5T4OeHJrQDgkgKxLSfXFAtD8VRYmRALZ5i6Hl1/xxG5pnyMmP20rhKdp2VTviuqU0sPKg3OfI8USM83Sc8IM7TdCXnuARwaZg591AnsC+Pwaoay2mfHNnILVo4k1YYyj7gWG2sz/mnTeUygSVvmjLbbKg5VgfUARHoiCNFjMcHrBio7L1Rmyhix8felomBY5FB1jg7Dkfw/zyPA2yOTIZGmY5i0qc3+TRxIeY+KlCYFjDYKM6+Uuz5ANtLDDqTrWMGH/hj6uLthpMagXDIum/oyL3ylamzgM+drOC3ilsaOVBRZVELsrdV+gPaW+xiYR8QDzWKMCxnRedWKQfvYY3CBbK8Xm7BS3IY1VIoARvIoDskyPmwvibShU46cUZ/mZ6Tt6EBPD5oisk2OrjIc0iLwp17t8RQi4j5tAvSA7BR7gqOSshbjxqfZEzk9IKFTn7blHsJ0IeaW2EjGwipCUPcP3EM2Q2R+MSVhWT/d47iJe7YuF5vEkJ6SkdxJYN9j6U/ZOKRQfLI8luZQgCCm9tPLZXB0/8JkrpHlXQqc9R6BqXGJ+3wA6KhMi6bnhLbATRMQo+PY5FPzHIhcSHpNAY5EKzQEyAYRLZoqckPDXopStZ3eL8e9L0I+wUgrdYjqrMiX5JIWQw65FhYPfUpMkIDnbb/0dtsX10OJTWTRqhB/hhwF0Y0jOextURHdPF8faxh/qXvsM/H7Aw/IhXHuoxvZm4i8qLfZr89IBc/bbHK4q7W9EtMNbovI9kwjLWRu9HUwK218jDabjDa18de+MlNGs5mAu6syFjgWk86G4xTFLWNUKtFqKkbfwLdD8eUesozOlarxwQFKTklfIL/PlWsmUD1YKWQQHSdvU2RBLd0Trgs3zMY/ySLuye35LqQ0NtOwpri6bZfweO+v6exakdXSuirQi8VnjpmdWORlnW+cAnRlO6rCYGQrzBI6vNqYa4YbyEa8gjnu7w+5a1 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: cf4c3a7d-1c99-46e9-dc3c-08dbb89b4c8f X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Sep 2023 23:02:14.8375 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: AqMO9WmiE9jboqjbZET893o76n1IfVpoe0NRkFgmbfNYt+APNYll5eSe+7dSPS5yfSUyw/KkSNzYIfrn5ygNUA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB7204 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-18_11,2023-09-18_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 adultscore=0 bulkscore=0 suspectscore=0 phishscore=0 malwarescore=0 mlxscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309180199 X-Proofpoint-ORIG-GUID: I1HVvo5hbKKAlkoc8CyrG-Rl11FwEtQr X-Proofpoint-GUID: I1HVvo5hbKKAlkoc8CyrG-Rl11FwEtQr X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: AA618160002 X-Stat-Signature: 7g6m5p4ki44b4qb31jm3bxpmbq8fe96d X-HE-Tag: 1695078180-262058 X-HE-Meta: U2FsdGVkX190ZEdB8F1Kyl3ZqTQRitL7bf1EcqUqhg+OkrMQjnvutbRfefoeDpJEUU9IPbTH16vM6UNIVtw+58Z7s7/nDHxdutsdjgvruwPBRta5B+6G1fc6pj+iIOEge1v9XPyO9X7fUEbCHiwnJgBOM0AxaVNX9THoUCIX8tAlhWiEJj2KAwMCwOE4mgWXWTUPE+/jAZF2p8fXWERyhEXg0LRHNzXxOuei3svqAP5ux457Db/oIFxoWCq5gpz583wcshtX+dOrlCkGZ54huKIr1AcCTp/CM3P+ae0IZzWIUDNDDbv3y3QsvA0/IjCRkLtEfikrizkro6/Iz82OTLtCMjSY0iGuBs6I1HiqKXQCqZAk50f7rax4n/7S7MEtSzd/NxbwmJ8RKC0MZZYlKAiT8UOSKF1KNDgWlM1daj5TYXJDnA4+DroDF2z7S+W+tsYfgBfnIzGe9e0TznmuocbHDkGqf/qmAPnXOmagPBVdE84fT5N963XSdmUONVo5EXVtUBO9YcVRQ9QmGLNSYBcaBDRnL0q7kWd/ubDdutiQIbknTvl3kMtrfURGQyPQ0jqSPSQ/XTQinL/EccakWktENvi0mVFvnUO5p6QCGYftNML3RR26BaBV0pha7Kh4qKetchXgHtlHFGkbTKTmOXanTf5cGNDSA1cqmABeqJdt4kv74HCHSaNQQ8noZb2ZS5Q0s8bxbjzD8IbNxWAPFYrtbLaLKNsGaZ2YOzggU5xOAdk8YlQaPFghBOlMQVkhaMQlX3u9WLY29arAH5YqHZzPfab/MFHNA7xuFq1y9Ylegd6O2JGCyQoAe5rCnjgdEdylhjqaFRz6SiNaTe2IDorUluG9T7vmOm87pYxVD8RDeOyD5QU+mamwPJpJMlklkfjAWGCpxdDlDkuWgsuyvaGWtOYygTSFYyq/61FuhPadlO9HqY+2p3i/FY4pMf9KkP76xa6tw9fhm1OWu83 pfJlpyqu nmgpe4mRiw0Oi7Ln7yzubgyij6TQ4xxFnTBPudtptH2t2rcS2VkW4eo8E68702261KstR8nf3qJT3/5ISbGBz2Akz6xqZKa8Pg1IuyPEfF++KY9yMmW+iYKG8OOuO2G8cKDhRrS0PKCPVqur7SCA0veyCmaR9rPq5iF95qBnONPWfqgZ53wTHE526tcXb2nX4/swM6uLEZGWW1gZVQvwSQgjB+MESUn4ikuYlH0HdirUrIhg2cuiM4gYdbDYjzryrBwzU2UnviCyADBGYkWyJueup6sNQJRdCvO5Ke1Xp40QhSTl9aWnM7J1q8ksBH2XxYCdl2XhBDF3MNPFOuU6Bxj1nsLI7Au+8I6e/q6QdD6pwrcYRpknTr1nkr6hhIN3T4wXkt3BIu8PKgN66DzCleEY5abyQCNektZkEBWMtc+MQNmCE3sE1hJ5UVMD5omSmgJQPhT26E0r4D9DieF3pW7kGtAP2BtCyyfP87Io7Q3DHuBENNpv9Uy9jHYG0mxAIIYzg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When adding hugetlb pages to the pool, we first create a list of the allocated pages before adding to the pool. Pass this list of pages to a new routine hugetlb_vmemmap_optimize_folios() for vmemmap optimization. Due to significant differences in vmemmmap initialization for bootmem allocated hugetlb pages, a new routine prep_and_add_bootmem_folios is created. We also modify the routine vmemmap_should_optimize() to check for pages that are already optimized. There are code paths that might request vmemmap optimization twice and we want to make sure this is not attempted. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 50 +++++++++++++++++++++++++++++++++++++------- mm/hugetlb_vmemmap.c | 11 ++++++++++ mm/hugetlb_vmemmap.h | 5 +++++ 3 files changed, 58 insertions(+), 8 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 8624286be273..d6f3db3c1313 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2269,6 +2269,11 @@ static void prep_and_add_allocated_folios(struct hstate *h, { struct folio *folio, *tmp_f; + /* + * Send list for bulk vmemmap optimization processing + */ + hugetlb_vmemmap_optimize_folios(h, folio_list); + /* * Add all new pool pages to free lists in one lock cycle */ @@ -3309,6 +3314,40 @@ static void __init hugetlb_folio_init_vmemmap(struct folio *folio, prep_compound_head((struct page *)folio, huge_page_order(h)); } +static void __init prep_and_add_bootmem_folios(struct hstate *h, + struct list_head *folio_list) +{ + struct folio *folio, *tmp_f; + + /* + * Send list for bulk vmemmap optimization processing + */ + hugetlb_vmemmap_optimize_folios(h, folio_list); + + /* + * Add all new pool pages to free lists in one lock cycle + */ + spin_lock_irq(&hugetlb_lock); + list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { + if (!folio_test_hugetlb_vmemmap_optimized(folio)) { + /* + * If HVO fails, initialize all tail struct pages + * We do not worry about potential long lock hold + * time as this is early in boot and there should + * be no contention. + */ + hugetlb_folio_init_tail_vmemmap(folio, + HUGETLB_VMEMMAP_RESERVE_PAGES, + pages_per_huge_page(h)); + } + __prep_account_new_huge_page(h, folio_nid(folio)); + enqueue_hugetlb_folio(h, folio); + } + spin_unlock_irq(&hugetlb_lock); + + INIT_LIST_HEAD(folio_list); +} + /* * Put bootmem huge pages into the standard lists after mem_map is up. * Note: This only applies to gigantic (order > MAX_ORDER) pages. @@ -3329,7 +3368,7 @@ static void __init gather_bootmem_prealloc(void) * in this list. If so, process each size separately. */ if (h != prev_h && prev_h != NULL) - prep_and_add_allocated_folios(prev_h, &folio_list); + prep_and_add_bootmem_folios(prev_h, &folio_list); prev_h = h; VM_BUG_ON(!hstate_is_gigantic(h)); @@ -3337,12 +3376,7 @@ static void __init gather_bootmem_prealloc(void) hugetlb_folio_init_vmemmap(folio, h, HUGETLB_VMEMMAP_RESERVE_PAGES); - __prep_new_hugetlb_folio(h, folio); - /* If HVO fails, initialize all tail struct pages */ - if (!HPageVmemmapOptimized(&folio->page)) - hugetlb_folio_init_tail_vmemmap(folio, - HUGETLB_VMEMMAP_RESERVE_PAGES, - pages_per_huge_page(h)); + init_new_hugetlb_folio(h, folio); list_add(&folio->lru, &folio_list); /* @@ -3354,7 +3388,7 @@ static void __init gather_bootmem_prealloc(void) cond_resched(); } - prep_and_add_allocated_folios(h, &folio_list); + prep_and_add_bootmem_folios(h, &folio_list); } static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 76682d1d79a7..4558b814ffab 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -483,6 +483,9 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) { + if (HPageVmemmapOptimized((struct page *)head)) + return false; + if (!READ_ONCE(vmemmap_optimize_enabled)) return false; @@ -572,6 +575,14 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) SetHPageVmemmapOptimized(head); } +void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) +{ + struct folio *folio; + + list_for_each_entry(folio, folio_list, lru) + hugetlb_vmemmap_optimize(h, &folio->page); +} + static struct ctl_table hugetlb_vmemmap_sysctls[] = { { .procname = "hugetlb_optimize_vmemmap", diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 4573899855d7..c512e388dbb4 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -20,6 +20,7 @@ #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); +void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list); static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) { @@ -48,6 +49,10 @@ static inline void hugetlb_vmemmap_optimize(const struct hstate *h, struct page { } +static inline void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) +{ +} + static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate *h) { return 0;