From patchwork Mon Sep 25 23:48:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13398573 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1B48E7D0C5 for ; Mon, 25 Sep 2023 23:49:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 15A9E8D004E; Mon, 25 Sep 2023 19:49:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 109ED8D0007; Mon, 25 Sep 2023 19:49:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC5A28D004E; Mon, 25 Sep 2023 19:49:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D99B08D0007 for ; Mon, 25 Sep 2023 19:49:20 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id ABD3A14075F for ; Mon, 25 Sep 2023 23:49:20 +0000 (UTC) X-FDA: 81276763680.18.549D278 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf03.hostedemail.com (Postfix) with ESMTP id 608EF20004 for ; Mon, 25 Sep 2023 23:49:17 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=VkXTZyRG; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=TJC8KxsQ; dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf03.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:dns request to arcselector9901._domainkey.microsoft.com failed: no records with this name}") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695685757; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=E/x7mTYtq0AK43XWU8sgezCXfY3I6f53AD910fUr/7c=; b=SMkgyzInlXfS9PEKKbcKVOkfq7ZIiaIy7AHajVZ8f3MhCtsZrc5wkP32ocBytc/carX3Ms /IXOxwRoH1psW2h/mjG8rQl0j/SoLe0D2wRuzBMCaJGdwlk8VbZmjw355KpMG8RNvmmmH2 HmfWTfyStd3WU4W8JvS1kg7AaO32llU= ARC-Authentication-Results: i=2; imf03.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=VkXTZyRG; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=TJC8KxsQ; dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf03.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:dns request to arcselector9901._domainkey.microsoft.com failed: no records with this name}") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1695685757; a=rsa-sha256; cv=fail; b=ptB/4dT1dWjKyVmPHcfBJ7l2QULYOjVZeF/sAFP7yHSK1DAt6sxf6oYwc3IwKxgmcZjang AeGJ29MqOdFnZkQaS6YrH6RDOPOPhXkcGw6APzjxSjCt3eMZv4zHJovm9LTZu4cSt5/lzf tByjP2PsBXUs0uaZBoACV732b5X4PTg= Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38PNGK1E001276; Mon, 25 Sep 2023 23:48:54 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=E/x7mTYtq0AK43XWU8sgezCXfY3I6f53AD910fUr/7c=; b=VkXTZyRGhaWD+cqPKuqRdxfPZITI176yi9ODgFpjq8/YGqPlkHd0J72QRe/1xBHUhk2c eUTnmhyKSf7F01RzEPaaVea4IzFSNDKN4jlB4AOHl1+efpbrMQxd170yq/Mzz1Hrr0Ox 8DuYdqwex7ysNxd/H8RA7mKdWa5HgF0MiXHV7FM5Gxw8OLlbz0hXvxxcp4tp8EMAjES9 WrMuLIcFc52KjbUsxZe+w02u8Q2RO+ZO8f47t+rD6yoegkmg/YTm/FIl7B0FwGFjekpS edhhAyO02drUMKbPIDXJh/0IT2lnsiO3OGWW3qjyzutwnFVXsz8yxSs17unKzfhpwnPM iA== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t9pt3n62t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Sep 2023 23:48:54 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38PMe1QR021229; Mon, 25 Sep 2023 23:48:51 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2046.outbound.protection.outlook.com [104.47.66.46]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3t9pf5f8n6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Sep 2023 23:48:51 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iFzVaHiXfwHUpolBY0R+SVsgTRg9V8Y9fRJm6e86nK34zSnvpvnFjY2iuDhwUUgaq2OBxS7Zp1tZ97DUDT6xyWhJPjpf748fOD8YjqQnKRWnV6O78/9n5nnWY6m9FOTrk47g9K461T/7+32dp6sl30rNUTRE61DVLoHJCZ7Dx7d+VBt2gqYMKaSP1dpEq/k090IEcKNMZEHXFCz0GaRKK4nsU5BMC9Qw7sn0THDylxHA5TAigb0BuCHjmtmj017JuBB2yQD0rL3EhhcLAc9HK/MN28lJ0VXz9nqfxsZ8gOv7YZ8xvzoqu3CknWSOg1qKXLzQSuIz3FgE78FScbq2Gg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=E/x7mTYtq0AK43XWU8sgezCXfY3I6f53AD910fUr/7c=; b=PdP2xUyAwbps+acZK7YbBeRLN9Sx02Flw6F68kyDkND28tOsoYmMny9pAtyIYbfYdiMr35vLwhcxm/uT8xsOon/mxldwCPA5MMQJKI2tHPDsL7MQTWqeRPfcPvOHLnifxWZh6YyR2utNc+F3KKz2nyjtXI95f2zQXYO8/ppgByt7IKw9iak7Vc6MxtJNk7K8u6rHojtErzZ7KTyYGF+pGQEMX4OcaSdU5uckXw1f/aOQl9Zj2perOHzuKFRs/XuLlL6otMxnTz+K9ZfXRC8Xe/s/EHmiKc6z6nIf519dxLa/BqzFJ2tFHCGT+ndNX4zwtJx7cE72UrzTOQsxU0QOXA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=E/x7mTYtq0AK43XWU8sgezCXfY3I6f53AD910fUr/7c=; b=TJC8KxsQvaCQKC0b0Tfx9D9Hkr469Kbq9vJkk5MLWjiR7BoeAsVrtho130W+Ton1NHVvFHyj6dIGPAuJPJFbhEJWRv82RUojTyyywWe8qcEZAdpF1jguFUW1YlXH+WSqLtN6Qe3cNb/mtEGPd7KSUN93UpTO7rhl7U/f3o9aXfI= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by SA2PR10MB4523.namprd10.prod.outlook.com (2603:10b6:806:113::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6813.23; Mon, 25 Sep 2023 23:48:47 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6813.027; Mon, 25 Sep 2023 23:48:47 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz , James Houghton Subject: [PATCH v6 1/8] hugetlb: optimize update_and_free_pages_bulk to avoid lock cycles Date: Mon, 25 Sep 2023 16:48:29 -0700 Message-ID: <20230925234837.86786-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230925234837.86786-1-mike.kravetz@oracle.com> References: <20230925234837.86786-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0348.namprd04.prod.outlook.com (2603:10b6:303:8a::23) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|SA2PR10MB4523:EE_ X-MS-Office365-Filtering-Correlation-Id: 24bd1c18-adc4-4215-35d4-08dbbe21f5c1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: dY2q7nngGuQN/OyaC/XPBPR1JTwa9z39ncmR6J++35agu+Xn37qRIkswY9rUic/huPf3JQaLKle88ovqicVlu9cq/mLVEfrGwbQMh+9I5/tXOI7fdokiTtSIx58u2fOKwlQgAJxeTqryZ/Xs2tjwEXPe1DsQ6q2WbGq1Z1C8fPYHzr8A9SPwdvvBZFrYB1jaJyYFMHX6V5kTL8KEHqjo3HTDlG+1On8TCl1WhXeG0u3U+/B3K4VliNAjrEZ67YuPZkpFt1uijZd7FOQ4+6iksRepul6QQYnUUZOrpg+8Xulpr5PNDGttUDGEZvU/a6XoUeoHChCVkNEZKGUTH/n1nkLv8kLAxZcvWIHzhQrj/jecNaqvgcOaliAewFRbUhEnAtq7wy8OPB143FKPWKIH0PaaLcSubrRSRCVBoDhiOhjND/PN8rfsUp/LCWixsxWHzjpIVovMI0pXiocgxNrKRd8fIvouUctZYXl9SRUKj+BO4QY6qt83NNbqYcUxcloihaZuCCayrlVtVHv+1E21yUJ+Kti9SodGWFjnBZQ4J/oht7ryK5FKWF1IWV2BXTF8 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(396003)(366004)(346002)(136003)(376002)(230922051799003)(451199024)(186009)(1800799009)(2906002)(7416002)(1076003)(6666004)(6506007)(66476007)(6486002)(5660300002)(316002)(44832011)(15650500001)(36756003)(8676002)(4326008)(66556008)(41300700001)(54906003)(8936002)(2616005)(66946007)(86362001)(6512007)(26005)(83380400001)(38100700002)(478600001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: bdyTOjSqjSRUF3X4KcAkyBSVqrHd+mwl6Q+B/+KbRhPopm1SZ7CoTorcWwjpklUbdU3H9NBeVR50Hrx9BjA3ObEpaqVwzt67El+u7A27JcPUYBj5mdGIjotTja0GLRgL9E36zhAPnYQSqkYk1bDEGIFze+VIExzAEgZ6xNdwA7imK0iu5uUnXpk/59tjLJRuxX2VcXuodNQtckDcuVQW6q5R801ndBi6va9pHkc8et0vzCh0IhCfql8v87dU+6g1mRUnMVHDnqcFBE5ETzlcZR8byhF2LOh/9Y3wFx0gjMn820tf8qjH9LlqeHQz+qF0JjnyOg/j/aeUdJj55iYE9IkjbrR4mjKX4PNzA3zarsFOb7EdGZ58llrh/4IW2dE0+/aogwLwDmJGhhlU7hh++x2k4SHYj11GXJehuml2khSnP481Gac7hbi0+4rSfzzgWub7lsfZFyFY5b7baIYbpcmA7kK143JJjllwpwHqCS8EYv1p8iXncj2SYoWXqciqwBtpV8oy0YNmkHbbC6vmtj3urebpPydPDPZ1iaAXYlegwpYu/rKQCwxopWW2dKFlZ6j/wiSiySS5+YxhbyBXN8xxbcFUYVPDeQU6ErUgbr7a9Ts7gonYBiDAu5azZaBciHpVPCHqiyE45AP5wK8RzvsdtTzNm9NVzGP9sf+o3LmRlP+De1TFViArQFQEnTsW/1FR7WJgK3aXLbTfcYybh08bB5Z3cH8fTGN+dj2tsvCobehHlZPRztapNF2uLKX9KGh1PR9ZjTRTwmjkqMXgg/EuXAD3jYBW1O1jZg3oO3K0HR+3hOKpwFtl4FyEGthMbz4HvdxEZO1OMwqTDcL8q9lf2UEXVuBXnYBXGcvKqS4IpRkIhSt1vHsald0vxS5H/reGKAYVLQAt9uIyhfa8fLyQCdPxFzHWDeGng+FEyqIvLU1tXna9Vk4N1lzw+I1rNf5huJ2OsHpKW7rIIxS8elsm7KZtirQwkkfs9E8J8bhRDAZVdMHGjPc7HPIaFDntXDwaOS/+ppVng4hK03M1CpfzN8wgJHMH/Pd8G71PC0OiqQek8ti2um+87VcIAF6uG40mgenaWUK2FIQEekeJGmEM7xNniXN1MFBO8ObgVJBVv7KfCBS4qx1oCZxvEzBFCH8x7HjDX/fIfoK0im+iqzzuRPAOw/FTgRfCSNd4vWVy3qB5CAurSanAFuuDxnwHGNpiNQZcT6giUOxpqaEcvHCMUrbaYgeTepNm70TUd1ILy2f7VX0Y4JPWCp+ZXVv+ZFehH5dH6CFfQFib5Ww1OcztUVaJEnXbPzgvjUDsx5Y43sRIJI/DQjSJCVMV5DjQmWwI+mRWNYY6+1mYjEirPd+d7hC2q61QJZ80f9oOT/SDaDdVWNt5XbJlwn6JjnUo+lK1hfi715FTlQa5yo3m+Y5bFrhf7pGlxeAtsuuF8fRfFoSluJA1TZkm5DMhCORd1NR7xfnFbVCUzV196x2u0gUElANTpHappgTJ/j5Umm9a6HLW8tbWp4kI94w3O0qNHcHPu9km2l9DtMWdtAb4Gpi8yFYCfOPXm36wsFjxAMXo0hM9rEazc/w8p1n3B9r7u0mVYYtWOvimLuPzxrs2dA== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: bItTguq9BnuIX5yDdDvPmkcBM8UPfU9ZGs1xZx7GKShaBvKFsgNCz0rwFNI0jubKubfzdCsmXtbEeo8V1xxHOfEMjqZPwjJ93ZRLnWLGoRAuRie+H7M+QM6XOl0duNXPnn1Oyaj7oLk4ds8O5+a/Iww/+x47IaMHL9mDIRSCNP8iSfdAVvoTF7A9Ci3oC2vpy1Js1/FEdmfGkdbE4Y2d+ppxZxRh4v769IEAmYP0MWoO4XF/BlVmPCVN05lYUnQcHtL5a7Hv1+uLjKrrzgrSi29gbF+GaYr8GcDfCDQNkhwSzKvSa2N3dJTfv15mHwkJuNmPCzBK9p0S9S7efugyVho/Q4Y4lfrzmZy40zkoWuaa9jutNqbw7NbhhTrvoqSA46p7Hw6cDDmJhhZUKrbcGHIJF00Pk29V42uWIMx9iXIgc48+mv2iJR4MZCyTuWdjdzDS/RNP0TZ02Y3/Q/Ik4KRsb5PKZTkIU96aJrr+a5Ab6TYu07Ixh0IPA532uZgcciMr8glL9leWDUAD5dqRkEH37/SoMWgsbf87f0igpCmvDE3NG3WsQwCf/TI8fsDLQgpr5GSew9Ij9HTdHWSeG5gRKiGO9HhBAxqTQGguNjUyK0d/f0lNQoFNYKtqGZ4/sLbZXmu/8ujrYlkNx+dphyBTuCX1IkOxQg/GbbPiZ4mvWBkfcB6gBpGBFzfTb2UpxhsUnIayUMje/HTVxOYLCVngbzDH5pSbCsBls2N/WNgAo8+A5drXs/E1Mv9cco3YmXHcKjZ5BC019C24mI+LVTN1zgpTOVvqPJFR+dios26Bk2sI8nWCRwD6C3tPtn7Vbko2+cXThiZDuZRkZ30bg2zuCHPCLBm3cPh7WTzsglFI8eWPXDyx/tOqX7gtUETlJvnZEn6kDiaqzp+xEBQA6KXGywjriWdrXJSlOt7cIbw6ah0ZykECetH+1zRLHDYcSuNskXmnwXhL28YmHh3zbTswyyLadhR5CuIJZxyKrB+vScEltSbMG6RAW+JrgUk7GP2NR0IEZeGnnmmq85DAccEcJ/UCsotW16RlVdoOkZakfzj5E7MOVVHIMmE3ePUuZWuOsblI6joUtsFcD2id9WJ3lRDWxJ3q0JKs8RV96tOHai1GKAbZWNDKY1Jf9jhW X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 24bd1c18-adc4-4215-35d4-08dbbe21f5c1 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Sep 2023 23:48:47.0513 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7bFZi8Z+n1XiV3zFRDAKYyzk5aBk6YgM+0aMkktTNUb/EG35qr5lwmbPu8tk0VH7mW3v32gQwjzrAtSK+LZMrQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR10MB4523 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-09-25_18,2023-09-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 mlxscore=0 adultscore=0 mlxlogscore=999 suspectscore=0 malwarescore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000 definitions=main-2309250185 X-Proofpoint-ORIG-GUID: C89YP67kEzTSF99YpOTpKviXctZOhN0Q X-Proofpoint-GUID: C89YP67kEzTSF99YpOTpKviXctZOhN0Q X-Rspamd-Queue-Id: 608EF20004 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: mu1rqxggmoh3px1zj4o1ba3n5herhkny X-HE-Tag: 1695685757-995403 X-HE-Meta: U2FsdGVkX1/CpQyPzshAugKnLP6gnvLe7PUdup6QhOrgf50X1DunDLQU7VDFyZtURbu0KHL0087JmqxKVp59JRVr1bT8mVGRdxtmkFxCe4Cgq1/sDJ7FfysremP/y5ZxhTz0e41//DbWF7g7svExl0WWpOFU+ezJB5WgS/EqRtCwGKpJg+wBX4jgNUoFkfHW38lVmbJ7qPixvOqpv9Bo2u3dH0cZE0SLepM5U752+Z/JqjllZaPk6fotwcmFGbgtmRgcXM3YrKTdueU4aZlDyUyesVEXTnSz4Ce06e9fcn8Fjd5wM1Dt31z89h9FoRVl9DyGJzsGyP7BIaGyZlhl/SoRz4YGTDqiebGDJSUauDY+zjubHe5jBKaPEf6yB032bqB5z5xqUnEEqebIPna3z/6qVucxybDL4hpvV9yT82kNWGw1JNRYa+eKTuTTA1WYFDvfOato2D9uB0ottOkpMSJ1IToQlOe4sqXKxnLAOo+iDZyW3wF9oudPQ4XOHDJQ99Y1VlgZUTQ8w7uG6nhgnQ5HHpawtspdJ/gyZKsb2NeaO0Jv5OL9YU12ff4+i+Z2jWlGVebds4eBlJ1fgBsx7AX/lpElyB0pmumUur0ZQJsMzm+udESD7bGqBRwqdYTG8Qac5WuvBqzU4WkrcC3G7O8dnUw2QKLxzSpApOudBqGg6JtqCdN2ll1fajLBhgNFQiScoW/hKUx5o/CwRD0x/AzADa3uaEmIxsoQqkZ0JHUH9F4kYwx9yNpjTki3U6nYHArIMWS2rgtPJB2AbrAVokEb6L3BtCq0SodgZxlIip54hSQ7V+7Ra7UP6Xn/xSQG42E9Lq3g8TjgLAnA42qOv/2oIBosLoLKpnV5pPYe21nvm0Lr80m53WWboUzf4Cq8yA9Og2ZoRTesGgEIMT2BJ4WwBDboF0TxajW4WD0xhuSIvFJV5s8e8cvz47VprgeOUPtuyuJt6hnx3JsUZt9 TElTW1TY Gni65c25VRAn2v+Pvvmq9Uqt/29Qe/Nv5rRS7vOdXZmN+Z2elJU8tjSFYyd1BG2TrCEMotN+ojjSuR4gYlN9ivVFeZgibW03iiyJLjg+MPJPNPpPu2Nc02EjW8Nv0CtPcefFNBc47s5+9seB3kPqnW5BIQC4+ijZAgqHczzeJB86IqAWMpgQ3EQb1foFHikSuDna20hfq2eiynln1G2ApA2gp0A4vUMt1WK40kKHI00GwjfVjvr7x34zNxdKdpWUKGa2GIvl5CfhHXhjcMK/AktCpR2J8Z6ByOsViMNLawE/LibxLDmAK8Ky04bYd4qByrLaFLU4ua2bChO2DDaCPbbUGGAp8NP6eQBoT6tugNL/AhSPx+8qU6OrhqUqfjBflBUXuROrlEPIZyzhFnAuPLJto0FZq3ozcKIJJd/IEiMsMVeNqQDZfFgMkJH3dwS5tqDHVbew6LnsZatGEoyFQKiO9Fx/V7gmjEvLAiDHaRdbadQl2pYCPWL2rC+jtjT4oTfWUeeu7Ms72HBMqUgfNhpFmGLHnbPH7vnN6qKTBLJPzl8x8xOCDyx3nXZToAEoe7Z7SOumF9ztDrgQGjupHbEiykmHSRVvohj84FY5dUB3viaLEy2ZV0bthz3bFKyyEm/oP5a8WMzvpgM9qYMLR62rdPQc5BmbAmca5F7+A5/L3LtdYo+Lb7mocig== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: update_and_free_pages_bulk is designed to free a list of hugetlb pages back to their associated lower level allocators. This may require allocating vmemmmap pages associated with each hugetlb page. The hugetlb page destructor must be changed before pages are freed to lower level allocators. However, the destructor must be changed under the hugetlb lock. This means there is potentially one lock cycle per page. Minimize the number of lock cycles in update_and_free_pages_bulk by: 1) allocating necessary vmemmap for all hugetlb pages on the list 2) take hugetlb lock and clear destructor for all pages on the list 3) free all pages on list back to low level allocators Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song Acked-by: James Houghton --- mm/hugetlb.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index de220e3ff8be..47159b9de633 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1837,7 +1837,46 @@ static void update_and_free_hugetlb_folio(struct hstate *h, struct folio *folio, static void update_and_free_pages_bulk(struct hstate *h, struct list_head *list) { struct folio *folio, *t_folio; + bool clear_dtor = false; + /* + * First allocate required vmemmmap (if necessary) for all folios on + * list. If vmemmap can not be allocated, we can not free folio to + * lower level allocator, so add back as hugetlb surplus page. + * add_hugetlb_folio() removes the page from THIS list. + * Use clear_dtor to note if vmemmap was successfully allocated for + * ANY page on the list. + */ + list_for_each_entry_safe(folio, t_folio, list, lru) { + if (folio_test_hugetlb_vmemmap_optimized(folio)) { + if (hugetlb_vmemmap_restore(h, &folio->page)) { + spin_lock_irq(&hugetlb_lock); + add_hugetlb_folio(h, folio, true); + spin_unlock_irq(&hugetlb_lock); + } else + clear_dtor = true; + } + } + + /* + * If vmemmmap allocation was performed on any folio above, take lock + * to clear destructor of all folios on list. This avoids the need to + * lock/unlock for each individual folio. + * The assumption is vmemmap allocation was performed on all or none + * of the folios on the list. This is true expect in VERY rare cases. + */ + if (clear_dtor) { + spin_lock_irq(&hugetlb_lock); + list_for_each_entry(folio, list, lru) + __clear_hugetlb_destructor(h, folio); + spin_unlock_irq(&hugetlb_lock); + } + + /* + * Free folios back to low level allocators. vmemmap and destructors + * were taken care of above, so update_and_free_hugetlb_folio will + * not need to take hugetlb lock. + */ list_for_each_entry_safe(folio, t_folio, list, lru) { update_and_free_hugetlb_folio(h, folio, false); cond_resched(); From patchwork Mon Sep 25 23:48:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13398575 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB649E81806 for ; Mon, 25 Sep 2023 23:49:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 702D78D0050; Mon, 25 Sep 2023 19:49:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6B1998D0007; Mon, 25 Sep 2023 19:49:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3F0A48D0050; Mon, 25 Sep 2023 19:49:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2E5328D0007 for ; Mon, 25 Sep 2023 19:49:23 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 61B4912083E for ; Mon, 25 Sep 2023 23:49:22 +0000 (UTC) X-FDA: 81276763764.03.DAF7FDD Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf13.hostedemail.com (Postfix) with ESMTP id 1B3E320020 for ; Mon, 25 Sep 2023 23:49:18 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=xAPfYYE0; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=VLbGU6nW; dmarc=pass (policy=none) header.from=oracle.com; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:dns request to arcselector9901._domainkey.microsoft.com failed: no records with this name}"); spf=pass (imf13.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695685759; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/NHM7VaeEfDia6EH7j8Eb5iVUqEnbrMv6N5cLhl9+Pw=; b=o0KroIS/wwd1VDjWECDWOlH39rtPAXC5mXlexG5BE6xJ8rJ+Uqz/8iAoN8po2rzVr9PMv0 9q5Ev3ip61wOclpBua21D05verrwdqsNMcnYwTOFIa7v64AYSusYh4eb82boMfLe4HuwWF 5rBV/VqBM5gSUhwm61BL3cqt6pqIyGM= ARC-Authentication-Results: i=2; imf13.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=xAPfYYE0; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=VLbGU6nW; dmarc=pass (policy=none) header.from=oracle.com; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:dns request to arcselector9901._domainkey.microsoft.com failed: no records with this name}"); spf=pass (imf13.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1695685759; a=rsa-sha256; cv=fail; b=oPV0d126Dv2Pr/ldFmISZU3k9Tl6WawMXgTvC0JqClZw/dkol+7t0xEilMBAjEEVmpuu4N 9TkX0LQPbYdhzVD3U2SpQMcBENBXSoAzS4RqxVGpvOVMrc3f3XOiYclLkvGsnCYyTRBSnJ bJRFWC42py+YzTUaBy3x5agdanKRZpc= Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38PNDtbL021928; Mon, 25 Sep 2023 23:48:54 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=/NHM7VaeEfDia6EH7j8Eb5iVUqEnbrMv6N5cLhl9+Pw=; b=xAPfYYE00r4rCaYekqqAULdNWuCuncbrI00jFywUSOr7Y1fdTpO2WvbIDx6S6HjLi8fx KpRiQGSSqieVH5K6aE87x6cRKOEfUm0pBTIvAXWOS29fOiB5e5avOMhKt4pGzeZJIxBZ 4D1CzCQl7rgjY2Ow/ON4LwYltLq0y4b9F0t2+B5dgyI39JGyTiM/npDDXc6Alr1Ml8QS V1yZnCBD0erL3CRd3Vf8VVaav6MX10kyPUDatlwqcUU6tH7gftTRaAO7SlvMa5oR3PZ2 xxyMeV0DIrBzDXK0l/FIeUBLcmrpQugYCikWb5YhtDPCNAYzRUfWj97eJ72+bMxEkc4t rw== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t9qmud5pt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Sep 2023 23:48:53 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38PN0WIq039366; Mon, 25 Sep 2023 23:48:53 GMT Received: from nam02-sn1-obe.outbound.protection.outlook.com (mail-sn1nam02lp2044.outbound.protection.outlook.com [104.47.57.44]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3t9pfbbda5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Sep 2023 23:48:53 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lSJ1ZOK5202HkesphdzaDSMBa1YutXu+Rk/xGQV7tua4cpn3w+3WPWeD9K4hXa5UOBdh7kCIwnI5RPYKyeKJJuPmPaDR/wHjAi0FbufPYnTgwv4G9UhGl0OcfHZG81ZC8nOq9KRfVDkwMfWGkkdPkhLlt9h9C+odKS7H4K6TR6x9cZ7kx66aydnAM+/STDtpJtNSBJXD2VWItRTNyyfA460NJYjStcbChLgryqXB99hUftGFZSWEx3xAhLNgp2Cfyg2X2/zbEcuYb5bh82QmesA5m0cGWElNJI9c78VOUmSc3BQ48g4CRgwuOMl+h2borWRpv59BW2CE3j/AcZ8n/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/NHM7VaeEfDia6EH7j8Eb5iVUqEnbrMv6N5cLhl9+Pw=; b=asJet6LOSePIDIOUgN8i030unH140m3C1PGB6W6hWX4aQHL5aXohfdVVp+1AI/FYYRraNdvuBKOEoK6+XRFBShZKN3667ae/fWtOX2fcESNePK9B2y+ajrtAeEDTUm471mjV8U7jZJzAxNrwDuKGZKCyoKQbkIoqp8u2hc8KRyvur/EKf6UBNSjSr4PZPUOAh1ItDQqUw7eMiZTIVUqw13uaSlt51p7XfxeBLLQQoq3ruTTxtEX8KiTErq01Ra5Jln10dHlfFVhDLhbU4Hgdn97jrqbfQP2Nnriq2nPm65tiKv09gt2HVDQFmGzZxO4TMBr4yinH/tcpwCR5CbcPkg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/NHM7VaeEfDia6EH7j8Eb5iVUqEnbrMv6N5cLhl9+Pw=; b=VLbGU6nWnov2i7knhLNQpKtLdOTC0wwza5UdQUbu7cpgNoXx/RvMv0rH7kizyf7ZkkORP1F3CPyo8gWd7G6Wdj6ZJ4KJqGb3o6V0NPDseIXXtLswKtBFtFbwWuAaNB2rhBpPgqfzL/wblkX9a7qFBqlsAyfsVjRo50RLEOz/sVM= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by SN4PR10MB5653.namprd10.prod.outlook.com (2603:10b6:806:20c::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.30; Mon, 25 Sep 2023 23:48:50 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6813.027; Mon, 25 Sep 2023 23:48:50 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v6 2/8] hugetlb: restructure pool allocations Date: Mon, 25 Sep 2023 16:48:30 -0700 Message-ID: <20230925234837.86786-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230925234837.86786-1-mike.kravetz@oracle.com> References: <20230925234837.86786-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0170.namprd03.prod.outlook.com (2603:10b6:303:8d::25) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|SN4PR10MB5653:EE_ X-MS-Office365-Filtering-Correlation-Id: 42372051-d28d-4f04-e0a5-08dbbe21f78a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: b4BHOX2j42xs9dyNUUfN0Bcyvhmo0HIWCHWB/WNMSVag7NcFTFk++/ClKGhFapOq83liKTe0Qym81Oyp/mIr/zZq7b7FHxVAIfPu6r12jTVvgk4Fw9+3b2xNy1nGyT2O2NxTVPmtPyysJDYGO3EVHMkFb67DqDDUCHspSjpp5TA+BGRZxT0bikwUtmn7rzMWRTb8JgLn1DkMWZweryOoc2o6IDuXBR23uzaD0IBSKpkwblDLW7A1+DNuE933MMSxVO8e+LOpjjqbdQ79dyKjNoILTOqXVtUzyIUpcJcC10uO12X6jSvvDAW6pv/QQirjrgkUw46NrlgqKQylp+vtiI7J5WWrFg+HcXQENme4gj+F5hG28FP8bIhUlCVDXCUHKxGO8lZ0182DbvnF71PuNPoqTIGDlOIt9nvcKN8pOt8KFmv454uLtqoyubE42/F/DuOl4d0Xx4D309zpOoQCB0LoYE5J2pCsCrj8v8yY2gA6fSC8ah+vcCqyRk28hHOc0LFWPdDoqmpPfQYtY5aHKr8hD0VURPOPnwZ59BiLSxQ0QZVxlGqEsV4EYXJGNsU+ X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(39860400002)(396003)(376002)(366004)(136003)(230922051799003)(451199024)(186009)(1800799009)(86362001)(2616005)(2906002)(30864003)(7416002)(36756003)(26005)(107886003)(44832011)(1076003)(5660300002)(83380400001)(4326008)(6512007)(6486002)(6506007)(8936002)(6666004)(8676002)(478600001)(38100700002)(41300700001)(316002)(54906003)(66556008)(66476007)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: NN7MG4Du/x5LGF3hnAYTLETFlUaoEU8yZxQiaS1vGIATZ+Wnt667jpu0r5WJDbbUNHpjLwjdlLmYQyI+dHgJ1R5neidzbHYZFicRr5G/pQUVDsyDEOSwgJZvuqVWwWEYbru87RT8G9zQ+X7yPbbv7jeLMbWxHuSKeH1Vf49hvk51fxZ+5JomWBiyaaQxdWUHp73DuLaZF4AFJ4+KB2kBKGPHjTwRYy0u4s9CSAac37Ib2vsqoJah/Vos72E0T/O3fXvqNdyeD0mg3G2hwa/1QwI/T/Wfzuro9MBvArGC4C1QTAF+jPCudzkLzvOUJUYMl+zP3n0KD/BRCnQp/Uc1e9634PMMZdN/7kxaY8fnhizW8LipK+Zx3a+cJxxjvn18vF6BIsT0+hPEAeHiEQhmo6Z+mcAkx1320u2msHGyM6U1kijFDTkc4XnVdcvOL9FNBHD5AU3oehix2duRlZ2pMpbTw78+smjJ1Suq0wJNg2KeEbOyRnnXnZ2RHDdQ9lKTh3NNdZoHmYdAvTY0QdKJI2H6/KnMbjUDnhQigJryZ2xRDxHKCasAK2IJ6aXLxgTdl2uHUjlUFMfKe3MfKahIAC5SKeTs/VEt1OalzAxKsUPEpHffccoiq++29wBVbxN9xelwKhO30hhoCYtI8j/N0r3K4ixHFgi7QoLckGvFyqYpE2LI+WoWvELSVfHrisQm8Q7UwdkSWGBCnVGlkGf4KtJm5AaZcH26FMjkHkD582r5vSsT2TLXAnQcGU50ZHL7miVLL4chyYlSUtX/Emk5JORnbLBJMpFRsGlnkKcLinleNEj1ArowgFyQZxyhdzNGiArkEzG8SothzJgcJhUnsVdxQRhzX0KV61f8qCO97ddTeheqCwdIqNBOzx8uQJwpbh6Wjcq6a3c1QCcBdwCgrUVNfCi3ZNK3o2058XEfM25TGynTCmseJdrynyJmrwsfDAtYhySBufcBZ8nDesYp6cCO0U8PSD8IgTmwtcSFU4EuywMx8IFyaK/H/OsWwwQpCxUuO5Xn8NKXUservdoHkJAen5WtAytzjAdCTKLRhhjqqvBMfX1owWe9qeACXn6OmmKQOufvpoh4YYPGfwHDryKMpQtdqijNfUmRRdoXkcqnaKb2TyC8eTCjTGgS7xPp5Q7Px4Q/q9URUKzHTIaeehn/rhvbtFUG5xo+Ght5ljsePLMeK1FTRiNojyQ35QvmO8YaYwnjMl+GWpYl2Lv/7Id6abwc78LxBXN7FsqKwWg7pYfbA7pPZaEoAfzGWLm6YGXQYRNwPsOTNIqC8ioipVcX1djLApxcJLveHHXSbXK1FEs++9WPoUPfAemRQ8k6O7C56X/M+rHyr+i4OKbiTVojjie80D5AvByMyPFfmBi7O53mEEwA9tdeSuhoZNRfSB4isaQjucZBEjgrXAvkq4nHz1ICSFeVfhQbfMdLyeiGTYmPcgBJ8DM76bTbBJ2caerilglSVumBgfBWNJ9JLomy2xPt7KLqw/cPQCHL9tQ5EqfnMWdH9IrA2L7Ib9336wyXuVLSbkjFF8KbLzmS2KTHeNjHPZLCJVM7pEIybur9cK5qlQawx06mqPpU/FsKGyCvX+T2r6SRnmBPyt8EGQ== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: J4zBtftiUWee2MpjCGDIDSWtp0ryXYinkIOXDS1h/tRkHir8cN7E5mlnREAGypoZN7pNEsgzSzvNx0tAdJlNLtSnvHQtsHsSKoaESis06clOUmcCg/ZrJqGru4+ozhxvByOTZjEUJcUhwYDmzaGOGctP4BhfGN5Tb+jh+IHC4uuZnWPTEV6NvVQBbw3Vw2sUQKECFK1SIg2jVP8kwacbQOYp5clBdfhX3z4q5dIfPCBxreiHl+H7F33lv6z4GuEbcubS5emKwUm2FR9ldT+wWDbTdLapm/cPX8fR0JlyQRe9apEt9dOjnttanm86YcZTbrijX5gNkJ38QqJM/jplvdQ0y7nEndMl2VzAAx6QDfIbVw2fQCSQegxELf4n6e6FJDmtd7QdAPN7QoAl6yJPme6XXRwCoMOqCFpDIChwfcIPkJ98ztUAmjkRt1aKEybxgWoP+bzFoS1/TIO1fNIdRmHcxkZGeoV049Nkk5mtxJKl1ZLJwBe4hqvtuUm5ouNepZkjIIElQPWHqRcHP1LhOWTZH/kFhje7JG8W1hNk5owkOqWSunYLczbN5bHf2+5xVSMWdHr6aaiXwOM2pv9o5vnHQ+2xUJ6IKSzyf8O6/6UkRzhp0QcTJ21KllrBNJ+6aGhRjVGOQRcHnXAGod8OsWw1Cw9OBUW/6gDJT/zbzNXf7OMtI+FL/fJk2j0yOlyPgZ6HSyPZ9N87gf7m7Q0lKI8CfoW+gS+nRJX0+Ed86ei78DeRP8llmobCSYAI9hJsCZK9d8W6rW6UQ+894c8zvJJ3Z3mryQOb/XHl5nMg02CWAOJ5vZqqCZxVvdtiRZoRm4Vg1vC9V4jyVw85FSaRoXmhUFKbo8nmUXdlij1zR4fAbKL/+6ERP3qEuy5DvVijQUj3r7K0W8nLPg93P4lSDPhFfuwgoOp5gJBdain88ERzD01zinrXoZs1p/sielUQ6V/Ps3iRAyqRiXbdRI7k2Kjui4ZcxWKjL66X5ZrDowmisuqeVUN26IndgWnInS5GKq55/CJrZdY2Q6d54fNn/Cg39IkpFVidgSfpGav0LcJOg6JUTUp9JPcOM2J9ILGMCKoBTx5mfmcQOmNSkyaXb5zy6LlX48G0pvGwv63X9jskm3/nJ1yMw+NwIkbUI1ru X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 42372051-d28d-4f04-e0a5-08dbbe21f78a X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Sep 2023 23:48:50.0379 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: b/8KoQzpAwDS/iCKc8FhinE0n5OGXy6bxIfADs1CqGopF/FCGD+uS8/eniCMX+QDa4CY5dEJHw2rPBXx6YTabA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN4PR10MB5653 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-09-25_18,2023-09-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0 suspectscore=0 adultscore=0 phishscore=0 mlxscore=0 spamscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000 definitions=main-2309250185 X-Proofpoint-GUID: EU_Vg7tC1sN9L_Gf4iYJuU3zZEyHGk9h X-Proofpoint-ORIG-GUID: EU_Vg7tC1sN9L_Gf4iYJuU3zZEyHGk9h X-Rspamd-Queue-Id: 1B3E320020 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: gknpkqiqi5p3udrd4asuiiymj6inekty X-HE-Tag: 1695685758-564442 X-HE-Meta: U2FsdGVkX1+i2/oHQX4VGzsc6M/R3yzM7ASHCuC5UNVetWAexkWXK1EO2ZLiX39Df4tvazgfFVTC16ZMLE3fMteI2TLmqhHPcX0tW7V9WuL5sAg0m3llVFmAzw3i4BnVOjMvjepNea2eF4MH2yhn0aLs1oaOvARt1YxH2bKeQLwAMYk/AaVi5e69xe5KaGhfWDPjy+CjuWZKYmy27oWzUZLGM/IzIbfxB/uL/O73Qrcw+UB72gSHnM3Y1dn2RxnKaL2FzYyOsbSISB2Mg+BNre/KErDo6n0rwluCCGDa0r11VSBhp6Rr9bBx20ZA/I7/mOKtrNqPD1CYtfI3dsNsXIxhWftghbBB7aVlwS3krsAXE4FlLWqJEJsbYm/P9L2Q+bx7oVkLSw79ZkPYk8WUo4SNcTgQpvOBtBgTsViHa9gKxXagil4vRrlVKjfSmSfwz2QjD11pGW6viYxZftS0K9xsuyvQsI4OCDwJFvGW1fgmML6FwqvmvLhaT5oBBGcyvy/gkOfdODhJgB/VFGwGazrggw/R1mxpR4Z3+U/YoeBw74s+dG0COKqZPIbGj8usjAT09ObPt9BZsB7D1ZubZy+7+IxNeWbkLVKNzliDN+IpZNCEvvyrSO15lJu4Ffhr9ytkdCnjJ2xw9Hw+cJdT0IuutJemutVWbsp+C7k+/c+HH7pexJSESX0jIaU5rGpSRaeR0rwDbHQGlZQqwWzL1Xxay3+89uiUdwbnyR1h/YsrnWvihSTu+kdxKLFsXrm+zX4PTBkQLJmNNziGRNpzW1AXx3YgpwOhIvP9CnMhl3zGB5gndBygVvzuLSYvzacaSL2pGgRxCX3XpPWluaX9vk5k5T5I6YwcY/EeoGC1o2SQorl3UCvlWPTt2sB0SGIDjxXUWr+VATY+K8zmmD1K2FgkVaIC9XQwXeXLRhrkJ0Ow3ll1jPOJB1khcqIHpWl6bFMKJZDXOOFF1DjE2vN 2FAOT7Gr jxJg10ZZH4SqL1CsAh2Z1raNMQoGbPaa0QybEFMM6vD/DHKIQORnTV7J97Mi1Is65DxGU4oXp9iDgUK69jr/nisjO02yJMNabkqnCC+t7wgYT5k/9FlOBYBk+R6R7Y6i6WoW+ooGrW3vCFnSJ75yY4F5gjJ58mGc6D5eClzyB1p7MyiawsuhpfSrjg9E9t3mOVCfUWdP1H4UFQ+3CmmYIlUQFYSVPmetl/i5UwhX6KBMTOk16OXIZC+pax7usYN28PTCyq27BxKOEwsG9Oqoaqh04DQtgii2kOpHpJTER5r1YNXDXg7OqtAwD61IZAJQJllb7sFELbJM5xsYdOY6awhpdFhyo10hxOQCQn6e2sW+IDrcraSAP+OH0QitXsDSAuclMpfuAzGVDbFdhkHLooCQnaX/GZjnppgxFt2xN73SJx078ykkiR2cVRM6eYrxFhCI49QXKJU5xDq/m9ZdI9BbIVy5tE5rcar/Nx00Q+1jrUY9Ncbh002vBiHMghb1kt9QnMe2DrbJHjhL5BrYgqwjJHdg4J7zIgFurwodrjC5sO2xHlkUHTOVK+ioSmDbTaxgVbqSpkuSaRGE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allocation of a hugetlb page for the hugetlb pool is done by the routine alloc_pool_huge_page. This routine will allocate contiguous pages from a low level allocator, prep the pages for usage as a hugetlb page and then add the resulting hugetlb page to the pool. In the 'prep' stage, optional vmemmap optimization is done. For performance reasons we want to perform vmemmap optimization on multiple hugetlb pages at once. To do this, restructure the hugetlb pool allocation code such that vmemmap optimization can be isolated and later batched. The code to allocate hugetlb pages from bootmem was also modified to allow batching. No functional changes, only code restructure. Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song Signed-off-by: Usama Arif Signed-off-by: Andrew Morton --- mm/hugetlb.c | 179 ++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 140 insertions(+), 39 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 47159b9de633..64f50f3844fc 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1970,16 +1970,21 @@ static void __prep_account_new_huge_page(struct hstate *h, int nid) h->nr_huge_pages_node[nid]++; } -static void __prep_new_hugetlb_folio(struct hstate *h, struct folio *folio) +static void init_new_hugetlb_folio(struct hstate *h, struct folio *folio) { folio_set_hugetlb(folio); - hugetlb_vmemmap_optimize(h, &folio->page); INIT_LIST_HEAD(&folio->lru); hugetlb_set_folio_subpool(folio, NULL); set_hugetlb_cgroup(folio, NULL); set_hugetlb_cgroup_rsvd(folio, NULL); } +static void __prep_new_hugetlb_folio(struct hstate *h, struct folio *folio) +{ + init_new_hugetlb_folio(h, folio); + hugetlb_vmemmap_optimize(h, &folio->page); +} + static void prep_new_hugetlb_folio(struct hstate *h, struct folio *folio, int nid) { __prep_new_hugetlb_folio(h, folio); @@ -2190,16 +2195,9 @@ static struct folio *alloc_buddy_hugetlb_folio(struct hstate *h, return page_folio(page); } -/* - * Common helper to allocate a fresh hugetlb page. All specific allocators - * should use this function to get new hugetlb pages - * - * Note that returned page is 'frozen': ref count of head page and all tail - * pages is zero. - */ -static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, - gfp_t gfp_mask, int nid, nodemask_t *nmask, - nodemask_t *node_alloc_noretry) +static struct folio *__alloc_fresh_hugetlb_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nmask, + nodemask_t *node_alloc_noretry) { struct folio *folio; bool retry = false; @@ -2212,6 +2210,7 @@ static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, nid, nmask, node_alloc_noretry); if (!folio) return NULL; + if (hstate_is_gigantic(h)) { if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) { /* @@ -2226,32 +2225,80 @@ static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, return NULL; } } - prep_new_hugetlb_folio(h, folio, folio_nid(folio)); return folio; } +static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nmask, + nodemask_t *node_alloc_noretry) +{ + struct folio *folio; + + folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, + node_alloc_noretry); + if (folio) + init_new_hugetlb_folio(h, folio); + return folio; +} + /* - * Allocates a fresh page to the hugetlb allocator pool in the node interleaved - * manner. + * Common helper to allocate a fresh hugetlb page. All specific allocators + * should use this function to get new hugetlb pages + * + * Note that returned page is 'frozen': ref count of head page and all tail + * pages is zero. */ -static int alloc_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, - nodemask_t *node_alloc_noretry) +static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nmask, + nodemask_t *node_alloc_noretry) { struct folio *folio; - int nr_nodes, node; + + folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, + node_alloc_noretry); + if (!folio) + return NULL; + + prep_new_hugetlb_folio(h, folio, folio_nid(folio)); + return folio; +} + +static void prep_and_add_allocated_folios(struct hstate *h, + struct list_head *folio_list) +{ + struct folio *folio, *tmp_f; + + /* Add all new pool pages to free lists in one lock cycle */ + spin_lock_irq(&hugetlb_lock); + list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { + __prep_account_new_huge_page(h, folio_nid(folio)); + enqueue_hugetlb_folio(h, folio); + } + spin_unlock_irq(&hugetlb_lock); +} + +/* + * Allocates a fresh hugetlb page in a node interleaved manner. The page + * will later be added to the appropriate hugetlb pool. + */ +static struct folio *alloc_pool_huge_folio(struct hstate *h, + nodemask_t *nodes_allowed, + nodemask_t *node_alloc_noretry) +{ gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE; + int nr_nodes, node; for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) { - folio = alloc_fresh_hugetlb_folio(h, gfp_mask, node, + struct folio *folio; + + folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, node, nodes_allowed, node_alloc_noretry); - if (folio) { - free_huge_folio(folio); /* free it into the hugepage allocator */ - return 1; - } + if (folio) + return folio; } - return 0; + return NULL; } /* @@ -3264,25 +3311,35 @@ static void __init hugetlb_folio_init_vmemmap(struct folio *folio, */ static void __init gather_bootmem_prealloc(void) { + LIST_HEAD(folio_list); struct huge_bootmem_page *m; + struct hstate *h, *prev_h = NULL; list_for_each_entry(m, &huge_boot_pages, list) { struct page *page = virt_to_page(m); struct folio *folio = (void *)page; - struct hstate *h = m->hstate; + + h = m->hstate; + /* + * It is possible to have multiple huge page sizes (hstates) + * in this list. If so, process each size separately. + */ + if (h != prev_h && prev_h != NULL) + prep_and_add_allocated_folios(prev_h, &folio_list); + prev_h = h; VM_BUG_ON(!hstate_is_gigantic(h)); WARN_ON(folio_ref_count(folio) != 1); hugetlb_folio_init_vmemmap(folio, h, HUGETLB_VMEMMAP_RESERVE_PAGES); - prep_new_hugetlb_folio(h, folio, folio_nid(folio)); + __prep_new_hugetlb_folio(h, folio); /* If HVO fails, initialize all tail struct pages */ if (!HPageVmemmapOptimized(&folio->page)) hugetlb_folio_init_tail_vmemmap(folio, HUGETLB_VMEMMAP_RESERVE_PAGES, pages_per_huge_page(h)); - free_huge_folio(folio); /* add to the hugepage allocator */ + list_add(&folio->lru, &folio_list); /* * We need to restore the 'stolen' pages to totalram_pages @@ -3292,6 +3349,8 @@ static void __init gather_bootmem_prealloc(void) adjust_managed_page_count(page, pages_per_huge_page(h)); cond_resched(); } + + prep_and_add_allocated_folios(h, &folio_list); } static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) @@ -3325,9 +3384,22 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) h->max_huge_pages_node[nid] = i; } +/* + * NOTE: this routine is called in different contexts for gigantic and + * non-gigantic pages. + * - For gigantic pages, this is called early in the boot process and + * pages are allocated from memblock allocated or something similar. + * Gigantic pages are actually added to pools later with the routine + * gather_bootmem_prealloc. + * - For non-gigantic pages, this is called later in the boot process after + * all of mm is up and functional. Pages are allocated from buddy and + * then added to hugetlb pools. + */ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) { unsigned long i; + struct folio *folio; + LIST_HEAD(folio_list); nodemask_t *node_alloc_noretry; bool node_specific_alloc = false; @@ -3369,14 +3441,25 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) for (i = 0; i < h->max_huge_pages; ++i) { if (hstate_is_gigantic(h)) { + /* + * gigantic pages not added to list as they are not + * added to pools now. + */ if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE)) break; - } else if (!alloc_pool_huge_page(h, - &node_states[N_MEMORY], - node_alloc_noretry)) - break; + } else { + folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY], + node_alloc_noretry); + if (!folio) + break; + list_add(&folio->lru, &folio_list); + } cond_resched(); } + + /* list will be empty if hstate_is_gigantic */ + prep_and_add_allocated_folios(h, &folio_list); + if (i < h->max_huge_pages) { char buf[32]; @@ -3510,7 +3593,9 @@ static int adjust_pool_surplus(struct hstate *h, nodemask_t *nodes_allowed, static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, nodemask_t *nodes_allowed) { - unsigned long min_count, ret; + unsigned long min_count; + unsigned long allocated; + struct folio *folio; LIST_HEAD(page_list); NODEMASK_ALLOC(nodemask_t, node_alloc_noretry, GFP_KERNEL); @@ -3587,7 +3672,8 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, break; } - while (count > persistent_huge_pages(h)) { + allocated = 0; + while (count > (persistent_huge_pages(h) + allocated)) { /* * If this allocation races such that we no longer need the * page, free_huge_folio will handle it by freeing the page @@ -3598,15 +3684,32 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, /* yield cpu to avoid soft lockup */ cond_resched(); - ret = alloc_pool_huge_page(h, nodes_allowed, + folio = alloc_pool_huge_folio(h, nodes_allowed, node_alloc_noretry); - spin_lock_irq(&hugetlb_lock); - if (!ret) + if (!folio) { + prep_and_add_allocated_folios(h, &page_list); + spin_lock_irq(&hugetlb_lock); goto out; + } + + list_add(&folio->lru, &page_list); + allocated++; /* Bail for signals. Probably ctrl-c from user */ - if (signal_pending(current)) + if (signal_pending(current)) { + prep_and_add_allocated_folios(h, &page_list); + spin_lock_irq(&hugetlb_lock); goto out; + } + + spin_lock_irq(&hugetlb_lock); + } + + /* Add allocated pages to the pool */ + if (!list_empty(&page_list)) { + spin_unlock_irq(&hugetlb_lock); + prep_and_add_allocated_folios(h, &page_list); + spin_lock_irq(&hugetlb_lock); } /* @@ -3632,8 +3735,6 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, * Collect pages to be removed on list without dropping lock */ while (min_count < persistent_huge_pages(h)) { - struct folio *folio; - folio = remove_pool_hugetlb_folio(h, nodes_allowed, 0); if (!folio) break; From patchwork Mon Sep 25 23:48:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13398577 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF351E81806 for ; Mon, 25 Sep 2023 23:49:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 779D38D0052; Mon, 25 Sep 2023 19:49:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7014D8D0007; Mon, 25 Sep 2023 19:49:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4408D8D0052; Mon, 25 Sep 2023 19:49:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2F58C8D0007 for ; Mon, 25 Sep 2023 19:49:27 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id E6C4CB346E for ; Mon, 25 Sep 2023 23:49:26 +0000 (UTC) X-FDA: 81276763932.06.732B04C Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf02.hostedemail.com (Postfix) with ESMTP id AA17080009 for ; Mon, 25 Sep 2023 23:49:23 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=eLykC5xN; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=oiAP4i7X; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:dns request to arcselector9901._domainkey.microsoft.com failed: no records with this name}"); dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf02.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1695685763; a=rsa-sha256; cv=fail; b=EfbKhp0Fz/KyogP3qsO4/c8kTl+xH2IpakCFrVL71F7pS2Cra9eWsVj0NoM/hNYWI2lE3d /fJsMtrXc+vv4DS16AFqF8ExUheMbjkBTUAxZGrzWFlWfvHwfDAXtdJCzTGWPXQOvcKBXe 4SqH3/bAma5CpB+RH9m4e91Owd6+TF0= ARC-Authentication-Results: i=2; imf02.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=eLykC5xN; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=oiAP4i7X; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:dns request to arcselector9901._domainkey.microsoft.com failed: no records with this name}"); dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf02.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695685763; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PjGQYnP6GfgFpQZQacG3GP1Mf1Hkr1K9uVvhSO0ZtHk=; b=H76GB6I/hDemArXBO/qEGc/D8yoeNK2I1x907sDUOFpxH1SIs82/x0dQQ2QVEFWnZHh+t5 tQtHNWEItp9qgFyWtOmIxX/YnaV+rNuGdXfz0oIR6vn0Tg3eJ8bpe+Y9E9/p8Ph+g2NKp3 8floZ6GZlfHiZnKuyJiYVTFXFrTV/wA= Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38PNDnst014513; Mon, 25 Sep 2023 23:49:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=PjGQYnP6GfgFpQZQacG3GP1Mf1Hkr1K9uVvhSO0ZtHk=; b=eLykC5xN+L2TYUpniRJIwaJ81qxoDFNvY87MutcHndngFV0mCGVkDJ3zkOH0mant/bPp 0JlHdWsQjLu0nA9B+HRkuJCYd57TWFOZNB/LKudmk4QvYxUMMOBHls6nJXxvlOhAqcVF mDrD0FSjP4QVzLRzaCvgcHL+YigHnhPUiAd2Kw5TexLFwsZM68tI7abBYhGFy1M6LwfW jtwzwpbjcsMes1IpAiTmQDQDAEDVxApjEWU4GjxBMjefi9fXag3lA63glqljcSRavORF yKd0EaKSeuPdEtUAL8h5PZKP6dJicGdTRFUSKlNXjxn9NhqUZDmXkX4V1lTEBzVJi/vj lA== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t9r2dd7db-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Sep 2023 23:49:03 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38PLcn6d030869; Mon, 25 Sep 2023 23:48:54 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2173.outbound.protection.outlook.com [104.47.59.173]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3t9pfbawdu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Sep 2023 23:48:54 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=T0PeOsgyncNO+pHrsSfdVOZN4VWLoKWXgDbtsg9auvmDke5Fket9mY0cpivZ6e7wpsvLSVFvhUB1mytsW2aOQmjXV3hZIlnooxTHLpnhWTR8gyhdshKxtcYGG98EXYzMm95kQkIqovGo8XFQDqccjstX24YdI/cwo1tgGUbpB9l/dxObUWFGWm/I8f1DmVuJzl3hAw5fyHjTzLYp4PvzW9kKwl+meg0/bg8n6c3n+741SfUUf0Df+FTWkK//NM2S5xS5qeR1YDbHTOO8m7qKBy3+l4MVRwj/tgEOuq+HJ9yQsUDhh8jyAkEcMP8AgCZM14XHZOIr8bh6dLAZBd1OXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PjGQYnP6GfgFpQZQacG3GP1Mf1Hkr1K9uVvhSO0ZtHk=; b=AbW2iZDJsJl18/0kiKxTdnQ56Ylr5ihb/ZpPGhMdwqbbZPMdpJT/rioqStfbLuge+2yUdQsdlhc/hQ0B+AljACuYHqoFgNHx4qKPaWj8xT0VfwntauMcOsXt2vuAL7qpQMOggnFyX6uZazdZid6e58cbZV5QUuyQe95wkPOSs6yAj6p2Weph//eYlgTrAkCK2sQkLwWoTR1Du2jIEzKGfvo1OoYV4/WP5Nv478tvSDkDUJX58S3Vf2tCovZR1rmafOcTi2xVt/RRBdacd1ppeG24afHjJA5hVq+/dHLLGKpGIxXn/BSC24Tn8A7vSqe812i4MQ4ErkbFrbtC73uWNw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PjGQYnP6GfgFpQZQacG3GP1Mf1Hkr1K9uVvhSO0ZtHk=; b=oiAP4i7XR6wQTP3cx8UMwJz0WpTBzGcD/frSXf5JdY643PQJ6P/5LLxpGfIviWT927izrz+SgdNl7poU9aKSq4tV9AGIiyIex8xMUq35wSTlI0NZueViFCBk/y5EjoZDoomW7fTMYr6TUv3eluMDgirOgnzI75a3Deh9+G3J6F8= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by CY8PR10MB6730.namprd10.prod.outlook.com (2603:10b6:930:95::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6813.20; Mon, 25 Sep 2023 23:48:52 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6813.027; Mon, 25 Sep 2023 23:48:52 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v6 3/8] hugetlb: perform vmemmap optimization on a list of pages Date: Mon, 25 Sep 2023 16:48:31 -0700 Message-ID: <20230925234837.86786-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230925234837.86786-1-mike.kravetz@oracle.com> References: <20230925234837.86786-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0280.namprd04.prod.outlook.com (2603:10b6:303:89::15) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|CY8PR10MB6730:EE_ X-MS-Office365-Filtering-Correlation-Id: 820338f7-0ccf-4750-a958-08dbbe21f90a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: AMXCN6mr/c8Y1FTiFdSlN+m6rz0cPRvtj7fKh8zlRCNEkg4y4UMmEtcYp0DYUsuX4yYzzk8KcJZLRj49FciNeUT7D0CPkUdz+/PxxFUxwNemh4rSNlgG1k6UzOFMvr0Q2zjoJ3qCarFDa5dgC9KUT/pPmGFTqELP+wAXuZlBYKmpUGv32oABMgo4hg3J3yJyF37ILIE0cY3SJxuvwkS2VCn35cjBlEdVfUZoGtDdyIrwQM99x2l+kX2SXAQR+1wIglHVdMthO/PF9CgZ0XL7f/SPYQsQ/nR5kmlplgCQPp9v8QWjqnBBBRIkvF95O2vqNshyDy9AfOh4ASPY9bnhqZFt9EvVutYfRytPg62mNIx43XeUeVrDz27F4HqywrX4BvnvGeGwqhizrUES5f4M+JCHIEtR70+TYXL2+2oqah8iGbRerw2+YYhm11YTpjHsspO1i6mizG6yaqMiSzaIH4J950py+YYPo8kaL7Jnt5Z99n5IZkZqffe5gK/pM8KFG3yIVPWdF+bCSs43qnvhvAszUhKnZVv/EOq2eEfbD65EVW4+sEKgfwbjqM5OPALR X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(396003)(136003)(366004)(376002)(346002)(230922051799003)(186009)(1800799009)(451199024)(478600001)(6506007)(6666004)(6486002)(1076003)(2906002)(2616005)(5660300002)(6512007)(7416002)(54906003)(316002)(107886003)(66946007)(8936002)(66476007)(8676002)(4326008)(41300700001)(44832011)(66556008)(26005)(38100700002)(36756003)(86362001)(83380400001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 8cvOHMY6p2QaAhtJFzLpilSHWhzD/iuGIN4tfbtRNg7SXgIq1KOeQEymNsY9c786HOm+MxhJKlU4JFj3xaAT2JDebHFbel4lsNo9XEEIUQZHTLBMMV9P6LSjPphXHZKzDltjbkMg5vT8LY1MItMiPiCRiLcEPHC8k1bAMXo+GTCeTEUweaX2Uiih1VjgrrTqyq7x87hdzDTsDPElSWrifO1TZGOUakuanAZoLlOAuw/9R+HF+Ayu7mXgb0qUt7IpFjlZgladHH4/xE5IXlEW+HBl5Au3VG1N9XKlhkuNhr3L/xlyEaMHiVJJ6XGuLyoxuSFYJ+62ViYGRH/6yabZfMjdgEh2yjmKt6d5KReghdOyrI0RGzLlPiEdwKFy2LXcp4oOEIqKf0Bo3VPpTKSTHu7Z8KZO7pO1yYOy5yvxK9QXYMgucC5L5utn+fg1KjmnyMjl46SEPOcFhcMNLw40ByqVRMR/g+TqJZxmWIm2Jv77lnIV39fdLvOxkPOw+X2lkOnSCiidHhR/a1889yMv6AKpJHNHQ/uR/6Xuakvt3qQ9fGq76lLexXoydZvdAuJDv2bPGvquBqfAMVQzUYULTmy22wUHiO0RGcgJFAFWIsmryOboPoVP7ZGHaCvFzq8EkrZrtk1gELcBedG7jDadW0BvbbHcBoqVtg23AxVnW0DWVSVWee6h6F2HMFcMgppcgkuTyjEKe8LpV2hXZBG88S+wJ/t2clLWFyy5v6GVa1EY7oiC06F0f/rSL7qon0NSurSTk+s9CCGJ+TDmz5g2Tm+Lpn2LS2rl+vh1BJq213Pdr7vB4SQybGSUBBWuWZgb0Dt1NchYWEgXvXfp2qMEkXEpZlkx1yRjycPd2zTHA40yjoMklSktNk9sIA+BaINopRXGKnHA6kujwx61/v/0ihHazMaXBNkjmUIuX+C0aY9S1qv3dr5R0xXQxX/riv07jYc+84FhY2SZ1wDaQTWNR0OAHbB9bphHCuk8LepeH89coj9MqtF8l10eIBPSQitC4AY1vl1mXTsTvMvlpSFCGUuum2NHJW8koi3acbryyIc+NU6ESqrJshBTU9sSyCjm2wrPa2xxq9Bc3Fv63f/Xld7pwVnc902R/i0XKNzigjIETMPpdauGmhZHGogObsR1IszNHxpDkweo/+Bq7i2CxC0/HNjMnIDZ28o884jm9ImqA81lzDMPLpLDIm36HfZwvCt7PFr8z85LjQZnnZg2J8yXet0xC9OJ4VJTthUODyEqEOuzQUdqNrYlROo/YV0yOTkabbHORt13T9jJhcKrfsnKVkIlzETW4gCz49AvDguXjaz50wyh1gtOe8J88Dxwx1Fx2IDbhKl6GHZM5fLLptY86OzI9XtuewQqy0qYjnFxgKqXP51jLYvvQw81yJe+wTXaAdLeyBFH+lpH/Mb8J4k1KrOZ2rpnGXpsq1xpAtNGJ3p44LfCNA5YmoHfL2bqTko0LhCL/8vH/5beVpV8JTW3l1j2/S5zF8VgvZgVs0xA4j5iMeJ+s3FBexqRbnVeA+2k1m+9jTVX8UhdrWA1pw1rq8s7qdtTlaXmDC9CrWnkU9vsMnO7AejCGcwua3YVH+29gJb2LKFzbGTez/sPsA== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 8SsQhm1nv49V9sQnPlN+LREHFFU8o8z/o0C8AWco7nGO5Ndqu3LHVyR+gaRxomOCH38aqbqhxCiqPaRqGR2t7MbGmvKE+anhUUUcZgyF8N8/MHUtnMOoFORpXkLWx/iZwwB9dZkkk9/xJ7Nj0vY/vWw23secpAScER42/b3Metu5B5TfAlK/tdBSXF6EBuhgAZGCYWDyYRNcjHvkfx+EBV394+ku2gnbKAe9uk9Vt2s1rQ4w5LTDfUansYL7SpaKybxFQHxW74vpzdSPHN7tyxuZ9cedMu/LvvwrC82LdeOplo9LLfaRFUkLVtXoNVlMRSD0Em326BQee8jqctydGSkWqBqbJHfLkssUyQMnGyR3QqhDqoDgQMoycdqnm393V1VFglDyoKZn93AZZtuE51mrSHXiOjfw4anl2vKbxS5sp5RHea3Dp76OawUmE1CuBDdP4jIFnvWA5Rd2syz0HP9i65+iKhUHaB00wLHaFBwpV9FTMH6x7bTYWhD/s027io/qvBsNarjbTPOJUefrnv7z7Rzz61TmSjffwIa/4tpbNefyWfYFCKauLcgWT+uUGV5PTsZ8XwrkAr5edMkiFkODaPo9PQOw+sbdO6d+ICndzxyMocuizkBOQdXcWEhoTrmLVsDtyOIQMbRgY0/NfBb1V5kqK68N2DitnUlVUutqBbFq7L9L9TRg0Rtn5Z4ZzQW3vvtds/Dy6dFYTMrZ4yxATMAcbafDYXLq+QIuErtAzocMQfGBbUGe7wOtaMD7BHmKUZYhc6QYg4/6xpYZWvq/WZStZ1R+u/OCSCGvuuos6wQbbi+G0FVii0ZhdanaSdALCxNHFMgdXE/YTtCDR3/I1SE2DBjOFFiQEKm1u+VMpuP9Hyj0+NoC0UdTQJOv1rkvywBQYi9Et7P3VIPVMj8EvQJYV+mY5oBKU6C/Sgcs2iOrF5MRq+6nPcm8rUsOyViGFEuz1NrmeUq88zKONbikFNp4w/1gZEOAWZlUqT62O4PuSSZkQZHoeCvc6yNViR3agnRLsY6JZZ6eewL6XYwIM4EEsFe6y0y3cmPKUAHcFa3/OJ/Bug0SdVQChPAXYXHmX2H8lYTfmLnjhv4N/3Zi43CkBkEeMYoPqlXyVn2I+BOxYNmwFU8ch/T9UJZX X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 820338f7-0ccf-4750-a958-08dbbe21f90a X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Sep 2023 23:48:52.5806 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: KIMIYvSVa4LeS0S9p062SgMARxb9CmTLz4qHvMABUFpMtpmTfQOWZp0mJK6FHawykIwI+y289oAIRnwb6F84MQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR10MB6730 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-09-25_18,2023-09-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 spamscore=0 mlxscore=0 malwarescore=0 suspectscore=0 bulkscore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000 definitions=main-2309250185 X-Proofpoint-GUID: lhidaK-TB0svHxNh-hlo_QnCpCSAIlyN X-Proofpoint-ORIG-GUID: lhidaK-TB0svHxNh-hlo_QnCpCSAIlyN X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: AA17080009 X-Stat-Signature: rz9jjpia3mik9ekgbshcsgskxk6555eu X-HE-Tag: 1695685763-927738 X-HE-Meta: U2FsdGVkX1/AYweIkUuavKjEQb0fk1DtPNTsh6sMAmJozPj1MdNpP6UGRAwQ2/73qXCsDqp0ej5uSA/R4l3s5IqLHuQ0OWuPjFrRO4DZdUeEpXg0K0DPQOHBE58AmlGin0M/jKhqK/7u9eclH9c2XR4lySSVQfRjW+4hzxM2kLr6e7y6AH+1n3IgpnUvIqtpMWV7FEGKtlo4N4N97x5hzaCmz3LX7c6X/EXH5jS6Csn6yLNOJRtne5fAO+zn9Wsz9NtOcIXnVLCCXVmF9A0t//Nr1cCtJIV6ARGlLwK2ugX/5SGeNXIB0iCLH+FjUyEReIm4MPbpM7G4EYiKmWLZd7T63rOL8+RWr8Rw+tHtOK4wFYMC3oaWgZCMYyxOdtfsEQ8gCWaWXFHkx02o2OSSyZWIxSC6UtJ5/WxKAvXws/A92F3GsKKc5r816CFdjoIOB6hQTAOQziyXo6dqJS/R3OP9j+bD0YpsDE9VTxfpwQDnO9n9jJ8+0EWxCHj6izg6FrLlbC7Lpkjb0qHENrc3LLgMIX2Prm61bguqwjdDH6NttfINHS5lkkUTQsL1SBgivlOWCTvvVI0TSREtxhWGtNHTiGtiia69MASD/FygLM1VGWClb9ezWSXBgT6nddAneo+8RSQj2K8zVeuciiDtaqjIeWItxgMPBpt1ZKl6qNARu4Jg2DO4SZX33wyLNbkSSmmm24d4eyMmMsOa43Xtuex31i3ZJnJwZlyD8T5+tXU9Kctb1StXCrpqx6usIZ/Pvl1VeCIjumonLWpMBLpy9ERHUT1YFkMzid4KqeD1RhQPePk4bSNE51ohFiN+npKaW4W9AJrILncOJC0BN0rVtwArqwcx4KJM+uUqTY5POuhtGYBVZKk5OT2JuVFGq48b21VlCxMjU0VPoPav/Ank+KiNAGC35UMoFHp58d9pDPY2YAV0gUiNhHy3z54N+3xm7U78uYCSno6yDbcbt20 rJSpqISu ATKLg30FZw4gWGI+ksFBvvSF1xuqRmBcVYYJdt0gbjQ+ZAGw49PJ6UymHHAicY7aaf6InhtLrlKtHrSD1XYZK8Gq8J4A1NCq9D2kOVoG4odyRrqRCqXclC+SzQXI0id1Aki0B/49kPWVkQ0GNtAAnkV0We0Bk+hwsN9ck58ik2GW+YLHa5MT7ChQwL+kZW6600z9CXJ04H2etQ0/4XQ80oH1moxs1Z6kcTf5wvq51OU8cot5Dk2RWdZR4XMzuvIPjWGTn6NbttPBmzdi4Y4sjEEogWW8lmEmzm52HmvMf+7qKYJS1QJnxF15ZTYjAiNqS6wdVU14lgwoQNd2vWTYR7KIGggjoMZg9pGGGSdLILbijmH26o6d7ffSOJw0l1lo7vBwMMOXFUMM/9TFp+hWBT3s0z/z+uIsOl1nxLkSbkU++sHCxKlZu4onFJEg3kN7Wn2X2vSLbWImTBOOfz6TdWpsytmaRCfvtlok89lZhXOshTiHvqtstAsWan2RhpkKYPqt6JpfcKqxXM9Z4ZVbynVG9I9/T9Br2HfBkhU696duZQyVDLEVLLilEMX67Kw5TMJbs5kYfCxnwTRUwZcaNWPa1JEV1ZXGjXqS4f+Mh9eV5WK5OP7FxFCDNDw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When adding hugetlb pages to the pool, we first create a list of the allocated pages before adding to the pool. Pass this list of pages to a new routine hugetlb_vmemmap_optimize_folios() for vmemmap optimization. Due to significant differences in vmemmmap initialization for bootmem allocated hugetlb pages, a new routine prep_and_add_bootmem_folios is created. We also modify the routine vmemmap_should_optimize() to check for pages that are already optimized. There are code paths that might request vmemmap optimization twice and we want to make sure this is not attempted. Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song --- mm/hugetlb.c | 42 ++++++++++++++++++++++++++++++++++-------- mm/hugetlb_vmemmap.c | 11 +++++++++++ mm/hugetlb_vmemmap.h | 5 +++++ 3 files changed, 50 insertions(+), 8 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 64f50f3844fc..da0ebd370b5f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2269,6 +2269,9 @@ static void prep_and_add_allocated_folios(struct hstate *h, { struct folio *folio, *tmp_f; + /* Send list for bulk vmemmap optimization processing */ + hugetlb_vmemmap_optimize_folios(h, folio_list); + /* Add all new pool pages to free lists in one lock cycle */ spin_lock_irq(&hugetlb_lock); list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { @@ -3305,6 +3308,34 @@ static void __init hugetlb_folio_init_vmemmap(struct folio *folio, prep_compound_head((struct page *)folio, huge_page_order(h)); } +static void __init prep_and_add_bootmem_folios(struct hstate *h, + struct list_head *folio_list) +{ + struct folio *folio, *tmp_f; + + /* Send list for bulk vmemmap optimization processing */ + hugetlb_vmemmap_optimize_folios(h, folio_list); + + /* Add all new pool pages to free lists in one lock cycle */ + spin_lock_irq(&hugetlb_lock); + list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { + if (!folio_test_hugetlb_vmemmap_optimized(folio)) { + /* + * If HVO fails, initialize all tail struct pages + * We do not worry about potential long lock hold + * time as this is early in boot and there should + * be no contention. + */ + hugetlb_folio_init_tail_vmemmap(folio, + HUGETLB_VMEMMAP_RESERVE_PAGES, + pages_per_huge_page(h)); + } + __prep_account_new_huge_page(h, folio_nid(folio)); + enqueue_hugetlb_folio(h, folio); + } + spin_unlock_irq(&hugetlb_lock); +} + /* * Put bootmem huge pages into the standard lists after mem_map is up. * Note: This only applies to gigantic (order > MAX_ORDER) pages. @@ -3325,7 +3356,7 @@ static void __init gather_bootmem_prealloc(void) * in this list. If so, process each size separately. */ if (h != prev_h && prev_h != NULL) - prep_and_add_allocated_folios(prev_h, &folio_list); + prep_and_add_bootmem_folios(prev_h, &folio_list); prev_h = h; VM_BUG_ON(!hstate_is_gigantic(h)); @@ -3333,12 +3364,7 @@ static void __init gather_bootmem_prealloc(void) hugetlb_folio_init_vmemmap(folio, h, HUGETLB_VMEMMAP_RESERVE_PAGES); - __prep_new_hugetlb_folio(h, folio); - /* If HVO fails, initialize all tail struct pages */ - if (!HPageVmemmapOptimized(&folio->page)) - hugetlb_folio_init_tail_vmemmap(folio, - HUGETLB_VMEMMAP_RESERVE_PAGES, - pages_per_huge_page(h)); + init_new_hugetlb_folio(h, folio); list_add(&folio->lru, &folio_list); /* @@ -3350,7 +3376,7 @@ static void __init gather_bootmem_prealloc(void) cond_resched(); } - prep_and_add_allocated_folios(h, &folio_list); + prep_and_add_bootmem_folios(h, &folio_list); } static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 76682d1d79a7..4558b814ffab 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -483,6 +483,9 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) { + if (HPageVmemmapOptimized((struct page *)head)) + return false; + if (!READ_ONCE(vmemmap_optimize_enabled)) return false; @@ -572,6 +575,14 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) SetHPageVmemmapOptimized(head); } +void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) +{ + struct folio *folio; + + list_for_each_entry(folio, folio_list, lru) + hugetlb_vmemmap_optimize(h, &folio->page); +} + static struct ctl_table hugetlb_vmemmap_sysctls[] = { { .procname = "hugetlb_optimize_vmemmap", diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 4573899855d7..c512e388dbb4 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -20,6 +20,7 @@ #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); +void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list); static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) { @@ -48,6 +49,10 @@ static inline void hugetlb_vmemmap_optimize(const struct hstate *h, struct page { } +static inline void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) +{ +} + static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate *h) { return 0; From patchwork Mon Sep 25 23:48:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13398576 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37052CE79A7 for ; Mon, 25 Sep 2023 23:49:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B6E498D0051; Mon, 25 Sep 2023 19:49:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B1D198D0007; Mon, 25 Sep 2023 19:49:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9485F8D0051; Mon, 25 Sep 2023 19:49:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 7D7F58D0007 for ; Mon, 25 Sep 2023 19:49:26 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 5301FB33E5 for ; Mon, 25 Sep 2023 23:49:26 +0000 (UTC) X-FDA: 81276763932.12.025F0E1 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf07.hostedemail.com (Postfix) with ESMTP id 1388B40028 for ; Mon, 25 Sep 2023 23:49:22 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=MH3VXNXi; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=e0AH5xuQ; dmarc=pass (policy=none) header.from=oracle.com; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:dns request to arcselector9901._domainkey.microsoft.com failed: no records with this name}"); spf=pass (imf07.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695685763; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TX7Kgu5vLRMl23g6p2+L9qirIhzBjTxtaL2mM0XH1ng=; b=HgS09SpPMuVMsn1HDleZpdRVQBuNLV3fE1cp7yygO8TclsbHGgxYWPKN3Ukod8QQY7brFl sSSpi528R9WrcV0opc2WZVut1HQFe8ymyJLEBZfspN7BDzjdNv05vOTj00uv4QN6mxPamm 7N2nZTDvqKAMOy8e+WaBvxGlGqi2Gkk= ARC-Authentication-Results: i=2; imf07.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=MH3VXNXi; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=e0AH5xuQ; dmarc=pass (policy=none) header.from=oracle.com; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:dns request to arcselector9901._domainkey.microsoft.com failed: no records with this name}"); spf=pass (imf07.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1695685763; a=rsa-sha256; cv=fail; b=eDBKDJkjQLvL+AMKFxnJ4A2X40A1aGTSBki7QQTlmLoF1DGr6w2I6avI0sJTqG2RfXhPjs ffvqePLFHCNZxba+ppOGi6iY4xWfsTgzqbRjVhWosT1HgfSowGV65UbCoo/DpJ7x+DIE7H npEC8j4RAdCmPDFDhEXpUgyGotzOB2I= Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38PNDqPG030138; Mon, 25 Sep 2023 23:49:00 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=TX7Kgu5vLRMl23g6p2+L9qirIhzBjTxtaL2mM0XH1ng=; b=MH3VXNXiykAcZ82q7iGD9SxPU3MKP/6Xu4v4uJuVhqjVWXu6IW3TzAaI8SWEGwfTGzgD yybf/3re2sXZcyLkQthnLIX2Y92/HCTvIC0gOlbh38ufd7BWcC11Ig7ADEK1ZUErt5iV 6oUbXKUvGUpGnVjgGs8wcDKA4eWXGUJW0Es7B9qmkT59WxvbEZwUpS0jqiHqs9VwSnIr o0T/aej2ovz1HIYs3b/0Q0VABGF5aRkirtJT2xJoojezqeuhLZNJ0Z7Sp+k81q30zDhY QOMlLDI+r/yQDQ2reYzWZGoZF3SyOWZBD61+wBI+iFigqr2f0BHirBmjLJaFw2mPAlTM Hw== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t9pt3n62x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Sep 2023 23:48:59 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38PN62tx030634; Mon, 25 Sep 2023 23:48:58 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2103.outbound.protection.outlook.com [104.47.55.103]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3t9pf53054-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Sep 2023 23:48:58 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BRk0MDvnYqEiNAdP95TfQdz2eLol+felrzcqd6ZnhTz8LdtyxSNRC4nwAQE1h4AU//r5kPuJYoP9bjOyZCH4I59Fg2aSo7PHPJ1jLodZlU5CQtNSXTSG9pvHcHutDaOxl55ucboD393i6Zn40jLNFtd0BbVl+XQewdJclW66vFn+nm4La+Zr8PCbE9yDoUnejWUA3f5TpMRSlfzarKFup2nZlWK2yNTor6cR1qbFtxgTwqS1FJr2My4bStxbbuExtuGkx370m5NC6cGgmWOkKQ0+If4ptRZQ6mfI9BwDVvpgfWi8KdNxoo/sN6hV7pz4GuC0RvLbRB3J9l9CW0qtSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TX7Kgu5vLRMl23g6p2+L9qirIhzBjTxtaL2mM0XH1ng=; b=lWLM03FjNcRhF1R+QY54PQwpxFqKgI8ol+krJ2bUwVNqo5iDlMi5wnP5Sj8opvjV0Y8jhQyAN7+WsoB0nXSDTjZ7BvvS/K20gXI4Iob4mtOBMgbRVpJunpTIdve3hWP6RStezfALM/DN6ytLPENKXy7m3o3mIHd1CKAFAhv1ft2nEKJ8ujAIvJwuDFS8EmEw4YY4B21gr2zExmoBYpYVJHMd6sUsLAOsl3i6O4hJMjAqT+N4KSwzI8+MWB2Loc3cEoj7cJlNjGCwFjJtzmhhFfdm023AOJXGXmICfCiKOm8C47eNzfK/DMlHI8tMtXzwMKkymjM+wjTY/8phL7HaDQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TX7Kgu5vLRMl23g6p2+L9qirIhzBjTxtaL2mM0XH1ng=; b=e0AH5xuQkacN7aed44ooaajpz2KCqCbDwI/rq3XJhe2A4Ppw249bCxr6w4gcmbJ43PH/HTUel3ziFLJI73P7LZK3p/Dyh+W3hoLzM+sVjMrjcnHYlZVbN1uYwZqMZTBmEUhbKuWulmGxa4/+MNO1qzZ/531j4opRYV4oSO/oQF4= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by CY8PR10MB6730.namprd10.prod.outlook.com (2603:10b6:930:95::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6813.20; Mon, 25 Sep 2023 23:48:56 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6813.027; Mon, 25 Sep 2023 23:48:56 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v6 4/8] hugetlb: perform vmemmap restoration on a list of pages Date: Mon, 25 Sep 2023 16:48:32 -0700 Message-ID: <20230925234837.86786-5-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230925234837.86786-1-mike.kravetz@oracle.com> References: <20230925234837.86786-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR02CA0027.namprd02.prod.outlook.com (2603:10b6:303:16d::32) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|CY8PR10MB6730:EE_ X-MS-Office365-Filtering-Correlation-Id: 2b17b64f-d85b-46b4-391a-08dbbe21fb7b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: mTi6UPTHZcB/Qqyrfhsd2bb0l0444Ge5YRAG34R8TlmnYaIks05WcGhP9d0aQBO7V5XIvjoyW7DaY86yc325hW0bLOuRDC9nvZh33tlMSEX0gN1CfvC7UpH4k1UwQMODXdd/pvHp9H4w0XGQD/HK29spJLFt976y64koMXM5ERsaB3xH96o8597fJhBifo0HDSGHX4tm29QeGrtCPJwelYdfEyQjjhs3hEBdqZCV1gL0pflBRHLJDcQ897XL/X/pWcRY9Gb0y4tiX9SgtsM+IWEDBZUVBn5RbuOhxB61NCc5LLD7UHqrmGlDpsMjC2Ei9bmzbJo4DdAeWYeqILyyqtlJyA48NfGKEQ/vRDziwawbT73NdrON0QlD9Cv9xSuFgTDRrtn2GqRJq5LMA4Zci/Qx6UBsFIK10FpE+XVNDQNlxgaXExyK+PAZERhYG5PjkGU7+sdfzm003GYo67RtCX2SwYrUvj3MLsuc/yk4s/6UsLjjoKy2DBk1TWpM9NaPHpXgYZ8wmuvdGJdnw9/Tc2iPFZqlOauqQB0opoKX4yvzf03LShLbWZXTSIB+XlHW X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(396003)(136003)(366004)(376002)(346002)(230922051799003)(186009)(1800799009)(451199024)(478600001)(6506007)(6666004)(6486002)(1076003)(2906002)(2616005)(5660300002)(6512007)(7416002)(54906003)(316002)(107886003)(66946007)(8936002)(66476007)(8676002)(4326008)(41300700001)(44832011)(66556008)(26005)(38100700002)(36756003)(86362001)(83380400001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: bVwfuVNE5uYL7exCj70AaN3/FjPtKy19bYgWL/xs+1cWmds1lYUncG52dsUu3oDWenGW8vOWTSgML9vM0WO8QNLEhEQHrBETggfbzZBEnUa3SyryHUUw7xNDsTmDubz9VWG8YcEl2kHkyKYuQpjSxu7e94iT6/bXY2jYEhqP2iInh+YhNRbOX/LmnsdJpoYiGFzRvIU6huVq/CC43gozmprf+qas92Rh7eW1SZagK/4cU2gvQj5//UCPv4t1b9huLlg+ujWelcvjXeK40uhNPe6jtQbOTy7nYixsZwIjmkQM5jzQoo7SKjiIR3/HmPZIHq5m2WBMUROLgX1Ac/nIkB6Q+u2BH6IGJDH1n/XtcYS4yZoQmsmZ/6BQfOA5OTvm64z4+vT4int0bnISqXnYd6DzUUSG4ST1EmS4gGlsGkXa2l9fiBwkl47jgqI6Vea0yqr1g4Vvs3tnhpyv7mfCcMWZXFUvLVbI58lue48mvCNWw3Si1xZfyM2oVZPmPDxJ4+BTe56nOQzTtxZDcN+0AWp4aCe9r3M5FyGZUApCfXGBqBAHlrtxsDQpjYGer/CkFdmp6fdJN6QzZQnmrC8JPwqZyPrmMFT6NnxuI+dqny/lOHh7Xg5yUk7mfaSX32TM5fwSqmS1uAHfMZDAlkdM+EtrpDB30BNyMoe9kcekHfka8Y/3qynf+Aq82vPKQzSXgqvjearj3TV4E2JmVu61FnWKurGMd1D4dcXgBxiduco/xIDqywFcPrmtVtHURDiItkpozLDwHrqHcqYNBhFOe8d0bTEXPh8XXcyHbwGv11KGDXXpbviCYGMwW3IhhY2nOpcf78a2WSXo6WwOgW6aHNqtaUr2xYdYYkIZGFQNLTW0CRXMuT0Pl4JJPpU4gfI6oRiG9jLDs78ES9e+h+8adKa0rfasy1kvXTFeaYB0a0W1f3aG9qdlQ+0NwwO5Ica5oh0TBW2S9dfRjoAHpnBiMR3+NKjU6FT/8raVxvl4am96Wgt0RnxKJOuJaMOQfnFeTO+x2vrXXsvgIEPr5dvN0pAqjXOGHLNujxqPvutlJLUOFvXdbxJCByV5hw8hmmIt8EGoLQp2AmD949UAWnbg099uC1eWKe3nTJT6fom9/Vbn+ZLnmxgkWxor+5MOQ6CnOc3qT19RibWBNedAr3DRL7IsM4x96EL/Jmw97nMTg3wsBK7Xl5vTVEC9fdpoGQrm4pji/q+P/YNrbsKAIM4L2gmx8axWnIK1Kopzg+4zRv1HklSxXeFWweGh93vZGi4tOA6uNhmpjM6XCQ+uTuBNh4ODoRbnngCIcxHU1LjRkrHPtX/clXYCcG0Age/W005o5BnRb+PXpdd0XdGPWQQacInB63lYt3MUtDiA2Fl/6Uk9Go4tilBxuZ8yAv52jklAyy5IWqiAdfyrpngGRJKj7xXZlNmZoE8lpE0wArUWm+ifRMPK3UcnjearqCXjjz8/shnGE/3H9bUG0g7fPBg/pTxNZ5zyPhq7HZ+mSekZq1iJBhrutWDRN9V9jSej9mydntRU5q3ARZ1+weCBS3KkKX3aVBCkUio4Mwa1xsm2gj9mlMrAt17nVVranHxw9xailYo8CxeKKFIiyvqB6zeuIg== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: oyHYJHu6VgRIUw8a4vI6GUb8Gm5SBz6PuFMFTOs9P400o6zVRKcvTq+l74MWhCXlWl4zWZGG79VZij2D+TJ9Nait/eenrVcpOcnhz9q9FEwgvqJ49XycMY1V/H2IKPvMv/AwZFobP2B9IxQAoB+R5HjE4APIQok5S8042g9zeMiyZSqH+VNMdAUbM1EsPD9eFzJPLh79yoG1QCRja10HEA9ziN8SXawBuh/T9LKYC0rZqcxx6WdS0LcYL0XJC0z4XYTjHqUpkaJX+YW6z7wejk0Krk6jM+E/ZOjmfzi01fFVrlSyYFdyyJ2ZyiXSkagCBTS4m6AaY6OYcaULrVNeSU4DEnOED1FcGluHzudjHi8HIdgawV9o7nAs14nv/zlkijvUWF7C3P0MS/pPXKecnmSX+6qW7SlQuUAN/ko7TegSOCuIqnGh3usUylKz9TFFgTixBrR/PWihdp9ILOKoM4eA+Xmx424717S0Eou5ZV4CTeCmTKi11dg8PZ8GF9gYincyE8Cgd0gkxO9JeMUx9Pwt6MabjpqivcfybjPb6TaxfVY96l5K5ssHxBAhYTOIF2qHbYOd4fwvobQt8zcj65oCMsRBMIe7VfrjKpG6Ffkyu7HlyIBF/VBgXYwiuydfQdbn6fmWkU78nlfyqR94uHD4l2tVPu6FA2M+D6BQfGY42MX3fANCxOslIBoTvkcH24mNHDJZ8vOpq2aGXZuQydXMlg3GzMLz4184u89wZF/ygKuIEYbhpvXGgTJbqIbcHoiO6a+5PlzCPn2auHCWmtR8WoDPyL2i01sMUCkoRMTY+S1b1nDLr2GYelnq4KyoOOIZpxT5VoAtMSsmArpZx+EleNMwfvvAURhQPFYp7OSoQTH2gHif2ccPnc8hBm2Cb5I3Ob3mzGzKTJpvw0de9RPUDlVTotQ3JiApmbzimENC5Im85zkyMberGz5mmGTf3GeMUifahu/AGHFdq3eStDEBh4Z0j1wClvWL2/xy52a2Bh6HD9kxdvymySjet1CHTm6Qk/oc+eOdDsmmm4MqZSZSkgSU5dJVTi2tLx5UIIASWWXKq+xW8JOA0VYXegGfyNTj8UANHjc7gBf0mNFVJdqskOOcZt9WCXPsh8XyOYPPYVSGqp9aSlZXOQsdCLQZ X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2b17b64f-d85b-46b4-391a-08dbbe21fb7b X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Sep 2023 23:48:56.6971 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: O/Cl9FVQx8UCEzO3PFPsXUczOnN6V6S/qHZIMCpjxZesnDJXqs1Q4g3iJR/O6cv6M4reNesbpTe6OQS28jIJ4A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR10MB6730 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-09-25_18,2023-09-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0 phishscore=0 mlxscore=0 malwarescore=0 mlxlogscore=999 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000 definitions=main-2309250185 X-Proofpoint-ORIG-GUID: MM-Hq3KOHZXnrrsAtRxcEwdahMlX7fe2 X-Proofpoint-GUID: MM-Hq3KOHZXnrrsAtRxcEwdahMlX7fe2 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 1388B40028 X-Stat-Signature: amoqfouiy8iua4xigskbukswn7ic71bn X-HE-Tag: 1695685762-765689 X-HE-Meta: U2FsdGVkX18LHHU1z9J7LesO9QICQf10DcNu9+Jlp6coApFuk7WbOL2MzMtFPMRPNMJKKSqS4TCGqekzr/qh/odzKjlyMCGkAVrxUttJp2el6fLPkjUE/Y75qTEyPJ7e99AC9rVul3nycldeew/saBTvyeGW+BbVfQZShXoHwUcSW0wOauVean1kbesDpZzL4A1Ov4Mdz4vrkliwKdQQeT3s7dRT5F3LXoA2kbb/61vZNzRhYYyb8lyTVYYXXbS/5KDLjfL8aldyrPlSzEIKA4R7QTIV0LWgTUiaV9lyWH0Kgs49xii6yssKbTWUL7DVrpJhuq7al0w6xdjWF4bdQpF7DUb5k6WUoMuxmEJu0zZPmS2dmL2Qi48Np44SfZ9WDYI7ImlyHC3bNciHzvwurqwOy5AYDvkzuwKHoziOf7wRUfZyNB1Tco6Be01xM8u6utjnv6nQLyfAnAtJP2WU4toJopte0/roL/L0S22bAfOOyZYkxV0MMOpDlxYN8tWejWs1zgDaEdnbv7cL4Z18+ECvRhF/IATXe3Hm3lL+SizPVsX+FFsGbwhG8ALHrIGeCBcNbn7nJWPmxNA+NY9i257IfArPyHOioEF5Gcax60dZ8k2Jwn9jrbk8bzSzL3BNIslyW43RSi/ZJPupytFX0sF9dcMZwwnUd8tzEDFMYetnHRbd2gAEoib0eMbyRKOycxRpfUFwMtbQvaFPL+A0rSGQICvtO5GD/f2ZDZIUUrWDeGT0U5cS8kXYp0a9B4spE9402yB/8rmTZPwbxbAVdHSDEhsaZ3Bj3GUZCp5Kyz/w2sRQ0DKXpUGxfwp4DUhHhv9zcahIkNQOyDYoMcWoKBZ8bdxqKeV6q/plpGh9GMi/Wd+9bTshDE8hl0LDLfIelHbRlJ3OkJyIxZnHFEt7GKsd1DjnuGdaKFFAlgXy8eWPHVrQkiGkl4IuEIDAr5KrJ8dOxyVnCT2HfOnT8Qd k2TWxe2Y WBqRuz1Iume0mfQpN0eVfHf5ONe0dAZhB707gQkthuYawLK3B2Ht4z0cp4nyhbmhW7ldyt0pYxU0ANhZsgFa402YDe7iU9VRxua71CKFTVYVv9KbiEepLlo0qaF5nSZUE6jEZLrvAgSL1n2ye+/2Snd7/sUS3/xZiX7I54yLlTKbz4lSdOrIUYsPDVBWL0TaJIh/cVn1v0Ml5uo4hFLbMVYwJc9d5WzLSae658iAQ/o9GffOMIwUwaWk5AIvXLgoE09KR7+QT12EDVlkf890BmAI5ELKv8N+KRCd8H6OHzQrWUnerXkWPT1emQwxP+gGn0lbvUbgCnByWSDekqyXjU1fBw0D3g8FZ6YKGkcaHG8JNiCs+SR6Q2X7FWJhzhMhySx2IbKWkmFK2a4o7UtAUHi5MzAfFhzD2hwps9oz3YErC25rPS7guTH7dupozV4aEmiD2d3/b5hCTfxseZuBJaqFqs6SduPF+mVs4/nS/IPV84Jxz1ObQZFNfCCacoVzZaT1QhcCyywmlcB1LBWOuf/gqFvp3Y9miWI4NtZqJM4nF9mGoPNziaXxNKQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The routine update_and_free_pages_bulk already performs vmemmap restoration on the list of hugetlb pages in a separate step. In preparation for more functionality to be added in this step, create a new routine hugetlb_vmemmap_restore_folios() that will restore vmemmap for a list of folios. This new routine must provide sufficient feedback about errors and actual restoration performed so that update_and_free_pages_bulk can perform optimally. Special care must be taken when encountering an error from hugetlb_vmemmap_restore_folios. We want to continue making as much forward progress as possible. A new routine bulk_vmemmap_restore_error handles this specific situation. Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song --- mm/hugetlb.c | 99 +++++++++++++++++++++++++++++++------------- mm/hugetlb_vmemmap.c | 38 +++++++++++++++++ mm/hugetlb_vmemmap.h | 10 +++++ 3 files changed, 119 insertions(+), 28 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index da0ebd370b5f..c484bb74201a 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1834,50 +1834,93 @@ static void update_and_free_hugetlb_folio(struct hstate *h, struct folio *folio, schedule_work(&free_hpage_work); } -static void update_and_free_pages_bulk(struct hstate *h, struct list_head *list) +static void bulk_vmemmap_restore_error(struct hstate *h, + struct list_head *folio_list, + struct list_head *non_hvo_folios) { struct folio *folio, *t_folio; - bool clear_dtor = false; - /* - * First allocate required vmemmmap (if necessary) for all folios on - * list. If vmemmap can not be allocated, we can not free folio to - * lower level allocator, so add back as hugetlb surplus page. - * add_hugetlb_folio() removes the page from THIS list. - * Use clear_dtor to note if vmemmap was successfully allocated for - * ANY page on the list. - */ - list_for_each_entry_safe(folio, t_folio, list, lru) { - if (folio_test_hugetlb_vmemmap_optimized(folio)) { + if (!list_empty(non_hvo_folios)) { + /* + * Free any restored hugetlb pages so that restore of the + * entire list can be retried. + * The idea is that in the common case of ENOMEM errors freeing + * hugetlb pages with vmemmap we will free up memory so that we + * can allocate vmemmap for more hugetlb pages. + */ + list_for_each_entry_safe(folio, t_folio, non_hvo_folios, lru) { + list_del(&folio->lru); + spin_lock_irq(&hugetlb_lock); + __clear_hugetlb_destructor(h, folio); + spin_unlock_irq(&hugetlb_lock); + update_and_free_hugetlb_folio(h, folio, false); + cond_resched(); + } + } else { + /* + * In the case where there are no folios which can be + * immediately freed, we loop through the list trying to restore + * vmemmap individually in the hope that someone elsewhere may + * have done something to cause success (such as freeing some + * memory). If unable to restore a hugetlb page, the hugetlb + * page is made a surplus page and removed from the list. + * If are able to restore vmemmap and free one hugetlb page, we + * quit processing the list to retry the bulk operation. + */ + list_for_each_entry_safe(folio, t_folio, folio_list, lru) if (hugetlb_vmemmap_restore(h, &folio->page)) { + list_del(&folio->lru); spin_lock_irq(&hugetlb_lock); add_hugetlb_folio(h, folio, true); spin_unlock_irq(&hugetlb_lock); - } else - clear_dtor = true; - } + } else { + list_del(&folio->lru); + spin_lock_irq(&hugetlb_lock); + __clear_hugetlb_destructor(h, folio); + spin_unlock_irq(&hugetlb_lock); + update_and_free_hugetlb_folio(h, folio, false); + cond_resched(); + break; + } } +} + +static void update_and_free_pages_bulk(struct hstate *h, + struct list_head *folio_list) +{ + long ret; + struct folio *folio, *t_folio; + LIST_HEAD(non_hvo_folios); /* - * If vmemmmap allocation was performed on any folio above, take lock - * to clear destructor of all folios on list. This avoids the need to - * lock/unlock for each individual folio. - * The assumption is vmemmap allocation was performed on all or none - * of the folios on the list. This is true expect in VERY rare cases. + * First allocate required vmemmmap (if necessary) for all folios. + * Carefully handle errors and free up any available hugetlb pages + * in an effort to make forward progress. */ - if (clear_dtor) { +retry: + ret = hugetlb_vmemmap_restore_folios(h, folio_list, &non_hvo_folios); + if (ret < 0) { + bulk_vmemmap_restore_error(h, folio_list, &non_hvo_folios); + goto retry; + } + + /* + * At this point, list should be empty, ret should be >= 0 and there + * should only be pages on the non_hvo_folios list. + * Do note that the non_hvo_folios list could be empty. + * Without HVO enabled, ret will be 0 and there is no need to call + * __clear_hugetlb_destructor as this was done previously. + */ + VM_WARN_ON(!list_empty(folio_list)); + VM_WARN_ON(ret < 0); + if (!list_empty(&non_hvo_folios) && ret) { spin_lock_irq(&hugetlb_lock); - list_for_each_entry(folio, list, lru) + list_for_each_entry(folio, &non_hvo_folios, lru) __clear_hugetlb_destructor(h, folio); spin_unlock_irq(&hugetlb_lock); } - /* - * Free folios back to low level allocators. vmemmap and destructors - * were taken care of above, so update_and_free_hugetlb_folio will - * not need to take hugetlb lock. - */ - list_for_each_entry_safe(folio, t_folio, list, lru) { + list_for_each_entry_safe(folio, t_folio, &non_hvo_folios, lru) { update_and_free_hugetlb_folio(h, folio, false); cond_resched(); } diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 4558b814ffab..77f44b81ff01 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -480,6 +480,44 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) return ret; } +/** + * hugetlb_vmemmap_restore_folios - restore vmemmap for every folio on the list. + * @h: hstate. + * @folio_list: list of folios. + * @non_hvo_folios: Output list of folios for which vmemmap exists. + * + * Return: number of folios for which vmemmap was restored, or an error code + * if an error was encountered restoring vmemmap for a folio. + * Folios that have vmemmap are moved to the non_hvo_folios + * list. Processing of entries stops when the first error is + * encountered. The folio that experienced the error and all + * non-processed folios will remain on folio_list. + */ +long hugetlb_vmemmap_restore_folios(const struct hstate *h, + struct list_head *folio_list, + struct list_head *non_hvo_folios) +{ + struct folio *folio, *t_folio; + long restored = 0; + long ret = 0; + + list_for_each_entry_safe(folio, t_folio, folio_list, lru) { + if (folio_test_hugetlb_vmemmap_optimized(folio)) { + ret = hugetlb_vmemmap_restore(h, &folio->page); + if (ret) + break; + restored++; + } + + /* Add non-optimized folios to output list */ + list_move(&folio->lru, non_hvo_folios); + } + + if (!ret) + ret = restored; + return ret; +} + /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) { diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index c512e388dbb4..0b7710f90e38 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -19,6 +19,9 @@ #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); +long hugetlb_vmemmap_restore_folios(const struct hstate *h, + struct list_head *folio_list, + struct list_head *non_hvo_folios); void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list); @@ -45,6 +48,13 @@ static inline int hugetlb_vmemmap_restore(const struct hstate *h, struct page *h return 0; } +static long hugetlb_vmemmap_restore_folios(const struct hstate *h, + struct list_head *folio_list, + struct list_head *non_hvo_folios) +{ + return 0; +} + static inline void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) { } From patchwork Mon Sep 25 23:48:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13398578 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA335E6FE31 for ; Mon, 25 Sep 2023 23:49:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2E1338D0053; Mon, 25 Sep 2023 19:49:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 294578D0007; Mon, 25 Sep 2023 19:49:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C0FB8D0053; Mon, 25 Sep 2023 19:49:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E9B5F8D0007 for ; Mon, 25 Sep 2023 19:49:35 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B7D551207F6 for ; Mon, 25 Sep 2023 23:49:35 +0000 (UTC) X-FDA: 81276764310.26.B05826F Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf14.hostedemail.com (Postfix) with ESMTP id 513CF10000C for ; Mon, 25 Sep 2023 23:49:32 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=L8jVeAs9; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="JdShc/2Z"; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:dns request to arcselector9901._domainkey.microsoft.com failed: no records with this name}"); dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf14.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1695685772; a=rsa-sha256; cv=fail; b=um5DXK/KToWIJZSF20PDRGA6xFrAx2fUqZtLGD4/L+sVdr+rurTSqlaEiGJ7wpNa0+5UcB 42n5BaOyEcD+QjiKwD2OG+1yWAeQq/hyqIIPoCUix23Ek8jqqICkvlCs3LDCimcP+9rWSg JPIDmx+BfJYW8OIIVBBsrUOJPKnAh7I= ARC-Authentication-Results: i=2; imf14.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=L8jVeAs9; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="JdShc/2Z"; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:dns request to arcselector9901._domainkey.microsoft.com failed: no records with this name}"); dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf14.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695685772; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=o8COXuu6/TtJ5IDPv6UZqVnZDWI+bSmwncS3etnX9L8=; b=gFqGJgRweYXw6Tfp6RjceQJEsfojIj2GHtZCeEWGBCEWdqcb51bkeQRHKg6PdbOUKM1bQY ddsGNtarhDkFEqbBYPUDHU7dtH7h0QNDEYC8f0bxa50gACwPEfgnK1pbZ/rln+IPL7oMZH nmfcU6VE6vHuZe2fQVLhl18QC/hAOZs= Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38PNDuU1027594; Mon, 25 Sep 2023 23:49:04 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=o8COXuu6/TtJ5IDPv6UZqVnZDWI+bSmwncS3etnX9L8=; b=L8jVeAs9Oqs/V8o1E7WXzffvQBJTn+5wKbXP85K/fAZF7b9QnUhzNiMyT8M9DGE5VkhT a+0anpZuXhcsSDqZI087kOi2AfqOVQ5oiyrse5KtYU7vFumQegqfRLPVI1i0WEzSrzMU pJSPseTDI0bGEqRym2TLMj3560Sri2KtFDsWQH7V/PYLlhc2JhQ1Pa5gIZY7oJJYXWj9 qhw7eGg3rkP2ErdO4uO3LnVX+nVqjKdoTb2f+O3qBIj6BoTL97upex+Y1SNhGH4HgKmB KTCbz+k3WuY5zmz4+gaiDkFuMFGRBKgC4CvC4nnlmaS/S+cz+uWdBmcI5hGugujYihrs kQ== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t9pm2595y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Sep 2023 23:49:03 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38PMYNr0030858; Mon, 25 Sep 2023 23:49:02 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2173.outbound.protection.outlook.com [104.47.59.173]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3t9pfbawgu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Sep 2023 23:49:02 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=foopS/ycNYn07ONnmdabdzGYWCIJR6nph6XkkL46bg6hMZRMWVAjaAjzCBnD16t5PNHGKcPNwwQ/Lphtwy/cKoN66FrkE2dSUnKU11Y7unB2wnlqtmvR5tkbQojslFBueVkTc/ZTm9NwzjpuJj+vOn1ZtcjnxSiskcuDC00AS+nY7dcXVj13OY6WkucY+E+9076xrVBABVi0GZDPLax+BewZC5Y6xLxcK+gqd+9yG5rk12oZR9S0ZRjP5TSmeK87Wtth+nQE4FuXWYftlt3BRjv0KUYRXOcg9QtZ2B0BtMMQE1JxKWXOBSGIQJDTDShfPAjcxWiC+etSPlHnjPb/Vg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=o8COXuu6/TtJ5IDPv6UZqVnZDWI+bSmwncS3etnX9L8=; b=cxF5KEIesA2Hj4JYdOFvKhSKwrott4yp6HwcyazTtAQig/hnaq5n7GZjBEg8VFRP0nUwYSuPZ4Gd5ged/9Oi4/OQFDf0cgXZOOqspt2cxNTcruC8dQn3cKs+WD3htWO+Egy9auFimeg8EBjCfZqVvH2W5QBdLv6YH2QPri2wJIrHVHyaqv4u9i0ipiER5t5oYWH2ponw4kQzqYzfnA10ykoMsff/BQXlq5sGNyFoG63ScuxxSY4doSJwxzVxkaH9n/b02kYZYbXH4Kn6hVPICSQtYxPValBKtivqR2pvsJKS5vZ0g3FeeQKgsInSVXvRa/IcEgWY7R1zwbwXtTFgTA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=o8COXuu6/TtJ5IDPv6UZqVnZDWI+bSmwncS3etnX9L8=; b=JdShc/2Zc2AaCspt0hYRkVZXR6gEe81fUZlw/pe10RG4NZXvQ8G51SLazw+3mVQYJKzoq2BYQQ4OSfYVnuSMseQUogdKcDXOGWvfNEEwyzrK9mBCH5WdiXzfv8ALUcgfJJ36hlna/LW5x0VnFEweZDGjGJfnc2it9X+I692gj4Y= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by CY8PR10MB6730.namprd10.prod.outlook.com (2603:10b6:930:95::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6813.20; Mon, 25 Sep 2023 23:49:00 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6813.027; Mon, 25 Sep 2023 23:49:00 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v6 5/8] hugetlb: batch freeing of vmemmap pages Date: Mon, 25 Sep 2023 16:48:33 -0700 Message-ID: <20230925234837.86786-6-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230925234837.86786-1-mike.kravetz@oracle.com> References: <20230925234837.86786-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR02CA0005.namprd02.prod.outlook.com (2603:10b6:303:16d::14) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|CY8PR10MB6730:EE_ X-MS-Office365-Filtering-Correlation-Id: 79ed3d25-09af-483d-c813-08dbbe21fd8e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5ncVl4pVfYVMeOPvF/g59DG37d2xEv9xvvaDXroJKeItKpqTDcbZVs2Zn31bB/NriRmsB45Qbdzt+Y8Bdgu4yfLh/xYh2+R7/APDuz9EOYManxED+WbZ0nepZpr0g7FHmEX0HwFscwg07brRR8CLRVRZtXxkudnBcxPNxIEtW/fP23fD2KIVqTmHp1yxsPkYXJWU4E25WpqA5BO0I4mRMxZNjyJexzjs+rA8IyKq/fF4nq7sWsr4fY67d9DWTI5mLsQ/GOBICwiCWS2jRNfDayCvLO8bVFaibMigmUy+QysSVwO1TZ4STp6C0aBguZ6uAFL4OEStCW+f/yOeqxRURPYw1WVs1G2N1o+KwT3gDOQPMq+zDevuisyA2bjyhn5ES4X8AxmLhnt+Sn42BYD12RfY7wh66fAyxk0kqCOpuu4Y0SmIcP405q4oKgQtZSayUG9i89MCjj9bzV/SbYR/pEmf9l7y9iYiornPKOnT2Ak1DwTv6wnR+2Sh1OoYANG7+RO8ZjQxrvHu9mwXAg676l9SmIG2XiXbhvHfAB/ue6bhV0KAOAEuBGljVfu6zH6d X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(396003)(136003)(366004)(376002)(346002)(230922051799003)(186009)(1800799009)(451199024)(478600001)(6506007)(6666004)(6486002)(1076003)(2906002)(2616005)(5660300002)(6512007)(7416002)(54906003)(316002)(107886003)(66946007)(8936002)(66476007)(8676002)(4326008)(41300700001)(44832011)(66556008)(26005)(38100700002)(36756003)(86362001)(83380400001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: VPXIP6mmrrURJYyacryDztT7xHB5654Jhp8+5J7xSVW/AhNjXclAGxIcl6tyWvjcik+2Jfz+jSQt7lELbLH9Cly1Kggt5H2QNcjA3e3qzuTNpnq1zfWP3m1BFlaYiZHqtSLFlu4n0vE6BuHcdFCaH5NRoUol7KqR1gTD/+u65b7sW1FD6DBzJ2mPoe7emEB2J/wzeTd2eyQmJh4tE/jQafyC3DIb9hlIXdgesg9JgnlnDFVLsOVZqsuLf6alRDnuuq+HCN0Rd5WRUcalryRYnH0yUmavdH/vs+c0Z1F7D6S+M03p42clA8xYH64Ll8LeHKZoI2sjTJno3FN+GHjIC+5ioTcaREE/W2pYRBHyA8mNAhNnGSBAmatSMumsLbr0yKKIOSqI3n9Hr0e+QH0DFv5DnoyMq8MHGNQyJmv0+vzka2X5uqUtBbLr6Dh2DfiHZQKceOeFiD1q7P/ovZbYuN2+IXV1PF564DPxFInyIqCnOYxNgGFtBNz8qUOBrhbqz6ck2Ss/d8FVhrfZ7QftkbtrwezhKgadIsPTQ7fWatwtWooXfHKQxPSmt9XBin6DD3DJENEJZUtQDX1F29w4sHPmXKfpuUXdcsy1hyYlgvLeBk6lVN08imQX32g5xu1m2mJx5vJCaDOj7HE4/SvNGWs3lvUhjxbPiBNXDrYoZ1v2IGVDudXsK/yhxi6FYpm/47PehAQM5tiTeUqppD2Hkew7UQcRQHpaXUMJaQu4KEICeBj6zeNdiGNUujFyURUE8vSJ9wdiQUDF/XolUHGSvvoeFUye1EZ8jEnxkw40G0qu7hdmD5EGX7Ebs6mzRQjJyegvQGJWeWVmxUXvzohq9DPf4XhBKq/58NNLFrBhPRmHEvqTL+zVr/Crsrtyf+aWNrfQj55+J0AoKvMKMWzSW9oUK+qjow1zWBVp7flxkhCokGBHKI1M4B1+rnj+SowO7t2bX3sVl29sreilsOaBS4r/mxOncbbwcGLfucJ9zSjyBhGlwfPw4MEdv4U6g4f5T0jvV9R0HsmM06UZhtMtvI7NVV80W4ULoJPcUWQA7mg7OkYFavRjQ1mnwSrqwiSk5mGBr4NqkYJgFDFZpgK489cAkOon9yY4AHg4WvMV5R4yPN+DiAQCS/I7JSy7iMYo08UFlwFdQIAL8EkX68K3ttGMJ0CTjY9ZKYPD7MNQhkrxEdlJNrAgMPEOpWnNfPc39R1UcO1zRTDkB+mc3UjJpkBbrgTnvhKQeiRcPfu3Zy/7osdSm+b0F/u5rW4Vnx6TUUypXJCakMwewOmaQMmjPHnBUH5mdY+kSfeb2806tWu0iFie+zzt1JbzmSIfV8/xt4VtY56F3zJLM0DfVnE6/0qw+pzVE6yHdOhdBm4bDP18cesrmqtGJ3tWWOmWCLMIBeq0boFdsnvYUHuQ4k1djoiukaG9nK/3gikxzuvs3iSUa6srYwNnL+TLZd72KT3UliKwzRMxepvKlpJ50YnmAqo3ilySt8gEG3wMsDGEof2dDGmB3DH7ZGagQcqNA02KAHuoGHhacPOnuxZK1glAphSlx0uL167uTYT0Jqz+d5b23UiyUOtQ4scW6efEcvUh0JbnucAiL+ZMWF0fOEq4Ig== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: ye1JjnqvYGCdEMQQsW9CJ9Jzjcequq+DDBz4Q8bgOwyfIo4QL2T1WVrf+aRUBjtMHyA6yRnJCf/Jly0Bp7ZOIH/tSiqE4H/fKcpHPxRmdovXYHTxL5PTjSRcG7XntE6ZGQoLq0kDhQ5lqJ24Z1lj42GAhAfAy9H+kk+M96z+VCpDbUCbhPkWkOuwBeRWB8E72mdyHCKbWzAriXo3nGRiYhakiSHXFNcuAXlo3M5349oM1NQO7f3iSZLhq/ZWyIFxoHYVTYYgxbAUbIEw7655j+dLs/0rQVjmXK4MnxAMBVCYxo7WMc54DnUuWlxhYd+uAvZ0KL06kEhsnXqET1bBIyMP83yIwMbECTIq2OhGK0u4WTNzbaBytbbyxpaEfTqRPTIk2vc1RElXQFeBo3OjhuP+uDgzPB0km3Jk1e1A1R9xZCbXKhUdiF7Xittcs9vY0wDaTmc/z0PnHMokTqDc9Nq7q8VoeAGfI0xOC35bYUzX8SJzbFYVnnB8S3IhLXoYZoL8wyJ/p6Qli9srkDPuns74U+n9tWM/aUAd+0AbhKxyHCou+42LWwV3Ps/4/DoNCQMN4VRdEww2mBlqZdcHSuAxEn5xlnxeruVfvpLLl0AuTNZhiA04/JOjYjFPjw9r0ArxRx3IF7/660rW6nlBDsQGySkJKAFtva0yV9tDBbnhcvSxrsdhim9junaBaFOC1QbfjIjn+aQBEuQFBInA4yccBKo8viWkm+4D7hPSRYITi/V5enCFd+Haa077IZfCfiSwz1YI2XYPuG5/HSc/r2FPkCMzN/re3+4It1as2zOhJkPfUt90Qv4lDJIqH7XHKFdBJDPZ/29jio8sa9tD3XzAeA14hmBQUmVdjtkVfY2OWoU37SDvMM42n3O8YMlcP7JM3OeVOFDZVI/43xY63osG4KvdUzWBLzFqz4hAbFjwVK0bZWP6X/FR3GPJZRGOfLoUkk4jWxkSHeIEZP7PwhESkZtbF41O6c9Y/17xd/E5N928Ksc4e0XrBHDQ7Qiy+ThCACX9hvH/0Wmky2z4uO+btJDmnKXl4TEDWNI6nf1pyoN/OLWvWtGamXltXOii38b3CRoM+onzM10dFmEOl71uygr1omnxVEa12MTMsrTxj0Xyk9l3lax1mdX7A36+ X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 79ed3d25-09af-483d-c813-08dbbe21fd8e X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Sep 2023 23:49:00.1709 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: fb9ItMCTFJQt8MGzJ/Ppi6Z2vELRWhBZV7gnp1om+K0BtQ4t9FJuATUhjf31bBVKb8ZWCziKCBAJoZ/Dro+oZQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR10MB6730 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-09-25_18,2023-09-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 spamscore=0 mlxscore=0 malwarescore=0 suspectscore=0 bulkscore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000 definitions=main-2309250185 X-Proofpoint-ORIG-GUID: q671Qzs-QNK0dNZL3hVuOoxUS69WXdWJ X-Proofpoint-GUID: q671Qzs-QNK0dNZL3hVuOoxUS69WXdWJ X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 513CF10000C X-Stat-Signature: ie41z3p9ryw9seq51reiahscuxa9pje4 X-HE-Tag: 1695685772-23334 X-HE-Meta: U2FsdGVkX19xnEdUnCpMmR5JCRCOI5OWL+4TfN2B6NubMNQPHADJytMRzl2/nIniXkiF4M5j4fscY8CeieOVweRAcqB9d70xmihnhYzR5rQJ73uzWd/ThSpMQUZzihgPJEC5yw+Cdy74VFEn6/5uf4stJZ67nlAMdc6ndSFzaCz5MdEzAjqds5+BbnmbA10aWFK2ON5a61fisI/Ki9E/dxVsJjLHFRMIUQXgc99ZvBYCcTGAKwE2yBL+lU9xHhvFHsx74FfBGboQDI8vvYOdsz2dQ8tIBlap3MndYhv07WI1wLk76bhPKeKgj5sQxWT/0m42QFzkoDfMfzykkU+epZjqyEIHrAM5CGS/fWdSIxV5kfqgS0/wl2yz999kMl8Wn636AxQZwIU/fZZbctTCrdqWooMKhggOkWeo6yVmtLBFYOXZ3fjI8mau7NUHR8LhDP4naSzKiafTl6VK40HMQ3r+kE0G4QJes3oXAJjEoGpEg1FKxn38n2b1uMG/bUnb09Qjn6+dCh2LYO5wmdyjvJBgSCKoueEbtc9DsyCaPh1LpsquGF0r7x+DlFmVjII747K6VDMaWxeCV8A4F3AxPA+FBqRyJyTQXo2KBa0Ms+g0LhvD0sDh1X4SZLjJFOTEksVM2KjtvOlE7TpVv6caJSAH/svfmYrAfARJAr8R4MqRsJis8rbEwyRTGczD/9hT+TrFnfiI6z0XsTt8sUHPBWk33w0AcaooOv0kGPOFo3ZRy7/ZupPi+uVoBiBGk2vc6Ml13WhSs278CBgROey8HfitUow+KBWq5xvg+Vvnk4WTRVCHWvQ+S9hFT5vuyMe0hdr0/fB90tjvNUX6TknI7z/pEDRSp2Ngdmdfe8q7ptHIyXdKW7A1vMJCGdyuhXBLyj1ebWZQGEUXgSAcXjSSSrIyQ+LmnWTGGMOigN42m6R48q+zHuLczYtjUoOLjYPPzJ5vkX2JZyWPZNCTTPf eVVM6llT Wqcd89XYbvfkYt8jm/dyuQ/R4Yq3vKNBzE9os1dpN9gG4OAWOvi7HkyjO7T/sjcgI5OY2auA8iPHfniHrZDBDqrrWlszNrr5Z1r1MgsHFKjKiHpunnMuZj6hKqA2ON4xze2EJFWPG5oSM0mXixUS1gGKXimRW4zGnEVX7ZnTptHw1L1m3TyMzFmfmWYCx2Txh69/FO6EKiH/jz0p8nosLMGYd5QE7J5cty+uBL+SgrznRnY7QpOCxkDrlcSo8GRvi7U6XfVwjpMyrhV/STGOOHHI1zzYvgWl+zoKwFuQLe/xZ02BSbxI++z2BZY/frX8ub3XZBDQ5CoyTVrua+T8z593AjOUhVdVGtN4s4dugawVD+2uOL4ddUVUBq7Qqo0ehDB7o7+kwu81VuCAXx8X+NWx2Yz32qfu/QrSXeWsevPFCowJD8sBYREO7Pzjbg0UCxP7TQU946F27Zr0otEKj798iS7Z+gTJLBy2EJEyjedh9IZIxqX8QGkckxfK1nD4ArTFudIcgdeyk0kqiooME6yY2xA3RkWHja9eZMVgUZhOkBn28d8+wjgWZ3Rvb60PvYhL1Ix7Claiv74pjZ1oDTdr4Hz1hv8V6lL/xhcUUHN+6+HVYNBdWQtQ70g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that batching of hugetlb vmemmap optimization processing is possible, batch the freeing of vmemmap pages. When freeing vmemmap pages for a hugetlb page, we add them to a list that is freed after the entire batch has been processed. This enhances the ability to return contiguous ranges of memory to the low level allocators. Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song --- mm/hugetlb_vmemmap.c | 82 ++++++++++++++++++++++++++++++-------------- 1 file changed, 56 insertions(+), 26 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 77f44b81ff01..4ac521e596db 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -251,7 +251,7 @@ static void vmemmap_remap_pte(pte_t *pte, unsigned long addr, } entry = mk_pte(walk->reuse_page, pgprot); - list_add_tail(&page->lru, walk->vmemmap_pages); + list_add(&page->lru, walk->vmemmap_pages); set_pte_at(&init_mm, addr, pte, entry); } @@ -306,18 +306,20 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, * @end: end address of the vmemmap virtual address range that we want to * remap. * @reuse: reuse address. + * @vmemmap_pages: list to deposit vmemmap pages to be freed. It is callers + * responsibility to free pages. * * Return: %0 on success, negative error code otherwise. */ static int vmemmap_remap_free(unsigned long start, unsigned long end, - unsigned long reuse) + unsigned long reuse, + struct list_head *vmemmap_pages) { int ret; - LIST_HEAD(vmemmap_pages); struct vmemmap_remap_walk walk = { .remap_pte = vmemmap_remap_pte, .reuse_addr = reuse, - .vmemmap_pages = &vmemmap_pages, + .vmemmap_pages = vmemmap_pages, }; int nid = page_to_nid((struct page *)reuse); gfp_t gfp_mask = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN; @@ -334,7 +336,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, if (walk.reuse_page) { copy_page(page_to_virt(walk.reuse_page), (void *)walk.reuse_addr); - list_add(&walk.reuse_page->lru, &vmemmap_pages); + list_add(&walk.reuse_page->lru, vmemmap_pages); } /* @@ -365,15 +367,13 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, walk = (struct vmemmap_remap_walk) { .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, - .vmemmap_pages = &vmemmap_pages, + .vmemmap_pages = vmemmap_pages, }; vmemmap_remap_range(reuse, end, &walk); } mmap_read_unlock(&init_mm); - free_vmemmap_page_list(&vmemmap_pages); - return ret; } @@ -389,7 +389,7 @@ static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, page = alloc_pages_node(nid, gfp_mask, 0); if (!page) goto out; - list_add_tail(&page->lru, list); + list_add(&page->lru, list); } return 0; @@ -577,24 +577,17 @@ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *h return true; } -/** - * hugetlb_vmemmap_optimize - optimize @head page's vmemmap pages. - * @h: struct hstate. - * @head: the head page whose vmemmap pages will be optimized. - * - * This function only tries to optimize @head's vmemmap pages and does not - * guarantee that the optimization will succeed after it returns. The caller - * can use HPageVmemmapOptimized(@head) to detect if @head's vmemmap pages - * have been optimized. - */ -void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) +static int __hugetlb_vmemmap_optimize(const struct hstate *h, + struct page *head, + struct list_head *vmemmap_pages) { + int ret = 0; unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; unsigned long vmemmap_reuse; VM_WARN_ON_ONCE(!PageHuge(head)); if (!vmemmap_should_optimize(h, head)) - return; + return ret; static_branch_inc(&hugetlb_optimize_vmemmap_key); @@ -604,21 +597,58 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) /* * Remap the vmemmap virtual address range [@vmemmap_start, @vmemmap_end) - * to the page which @vmemmap_reuse is mapped to, then free the pages - * which the range [@vmemmap_start, @vmemmap_end] is mapped to. + * to the page which @vmemmap_reuse is mapped to. Add pages previously + * mapping the range to vmemmap_pages list so that they can be freed by + * the caller. */ - if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse)) + ret = vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, vmemmap_pages); + if (ret) static_branch_dec(&hugetlb_optimize_vmemmap_key); else SetHPageVmemmapOptimized(head); + + return ret; +} + +/** + * hugetlb_vmemmap_optimize - optimize @head page's vmemmap pages. + * @h: struct hstate. + * @head: the head page whose vmemmap pages will be optimized. + * + * This function only tries to optimize @head's vmemmap pages and does not + * guarantee that the optimization will succeed after it returns. The caller + * can use HPageVmemmapOptimized(@head) to detect if @head's vmemmap pages + * have been optimized. + */ +void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) +{ + LIST_HEAD(vmemmap_pages); + + __hugetlb_vmemmap_optimize(h, head, &vmemmap_pages); + free_vmemmap_page_list(&vmemmap_pages); } void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) { struct folio *folio; + LIST_HEAD(vmemmap_pages); + + list_for_each_entry(folio, folio_list, lru) { + int ret = __hugetlb_vmemmap_optimize(h, &folio->page, + &vmemmap_pages); + + /* + * Pages to be freed may have been accumulated. If we + * encounter an ENOMEM, free what we have and try again. + */ + if (ret == -ENOMEM && !list_empty(&vmemmap_pages)) { + free_vmemmap_page_list(&vmemmap_pages); + INIT_LIST_HEAD(&vmemmap_pages); + __hugetlb_vmemmap_optimize(h, &folio->page, &vmemmap_pages); + } + } - list_for_each_entry(folio, folio_list, lru) - hugetlb_vmemmap_optimize(h, &folio->page); + free_vmemmap_page_list(&vmemmap_pages); } static struct ctl_table hugetlb_vmemmap_sysctls[] = { From patchwork Mon Sep 25 23:48:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13398579 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE0E4E7D0C5 for ; Mon, 25 Sep 2023 23:49:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 504678D0054; Mon, 25 Sep 2023 19:49:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4B3D68D0007; Mon, 25 Sep 2023 19:49:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2934D8D0054; Mon, 25 Sep 2023 19:49:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 12F588D0007 for ; Mon, 25 Sep 2023 19:49:40 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E2F1D1A06DE for ; Mon, 25 Sep 2023 23:49:39 +0000 (UTC) X-FDA: 81276764478.27.D5357EE Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf26.hostedemail.com (Postfix) with ESMTP id 7E00F14000A for ; Mon, 25 Sep 2023 23:49:36 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=RIqHiBBm; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=oPKR+KNW; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:dns request to arcselector9901._domainkey.microsoft.com failed: no records with this name}"); dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf26.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1695685776; a=rsa-sha256; cv=fail; b=fZbPbnVzwYE3gFHky1We5+mI4Z6xoznkkZ7vqzq+cQhM+s5/Yze9fjTyzUPdM+LSmjIZbV 9CcPSNGhoizDObU+qnhG3GG9PLyTGjIvQGjKev1uIFspI0sxjf+n1kHHFEb94qYTkUTAXp 6QTOa9F9CXXYrm7hS0AEY75kjvgJUmw= ARC-Authentication-Results: i=2; imf26.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=RIqHiBBm; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=oPKR+KNW; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:dns request to arcselector9901._domainkey.microsoft.com failed: no records with this name}"); dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf26.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695685776; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3vbONjsZvravfMZFWf/jFmc0JqQCVqxQwTo9QP2uYlc=; b=TwcnLQRIcOg3LmJ5cJJ7loPLawvKI8+8fayOFiNCoxp/bMZFS1BCWt2k0NCPriOoIe3FJf Jivet+Z2c+UYD0XxBeW7+3P9Sbl8g1tWrm0QXjHk2l7OVG9W2pxeSbtWdGzuPKaIwBcwC3 RZETPvz2nNjFlp/rb4iLLkHdmI6PV3k= Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38PNEP7D018843; Mon, 25 Sep 2023 23:49:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=3vbONjsZvravfMZFWf/jFmc0JqQCVqxQwTo9QP2uYlc=; b=RIqHiBBmHQMTyXOEEMefemMO4NV/X7W6OM6BEkhKDf9Ds57/+Y8b4yupOuGm3ttV0VjC wASMlBwocRlRTM6NhUbySNJhjsfoPy9/K7koDMeP2ZyrXcAxjxMhF/CuZV00NCtUEvuw dhTfH0X3lT/XtnILlWiepVL2rbzj4oKwuQd6qI6F0AAUhjNWHTwQXY/7V2lH2yIIgWEl PM9LMOtWg7OuVWmPi1TkEfLDdLePy5FjeVsREPDRCKm4k1d8uMQqAXyZIBwqfhJ8bhBv EK/mKIgVr7i3M0zLp0Op+Q7J3+Hhp78SflQTiio8dzLmtJc+v4QLFkFLibbGZIDmB1Wv AQ== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t9qwbd5k1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Sep 2023 23:49:07 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38PN1bGg039385; Mon, 25 Sep 2023 23:49:06 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2107.outbound.protection.outlook.com [104.47.55.107]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3t9pfbbdek-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Sep 2023 23:49:06 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=A3wMOkjIVefDxSv6LDDWpO0NHJWZSMN6Yb78cNQTrl/0N7WUL/3ci5FQS5oZRbusw7ptbg1BPnfUIEiykYZepWYtBWb+gFjFOIzzp/oIuN5Mv+otcc2n70U2tQ/5Sq5M53gQOhb2UgwA5fp8zMWBCUTC/4KSKA0fTL4QkC4pqFS3Jmro2QhHEigmtI6vWLyIp1Iq9hZ4jzdTOCGJ2LMcsN0ayXnUQVmQ7PNG853fnnY1QtEK2NP0qBFBwsBtZnlGt3K3DVF6Y+Q9o352bjH93GAKZs8gmQgbUjtUArzyOpW/p3epvIuQHb18FYqict7kS/GlXkIUyMqCjGsQTpNYHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3vbONjsZvravfMZFWf/jFmc0JqQCVqxQwTo9QP2uYlc=; b=AJzcPcbkfigl6A/i/2Q8DhmCpexVmeOeUuUF836tvycpFLS2/aAUfjnrh3LhMHLBgBMnPRbYzxtnDczEUkhwe4xD+fWykq2FvaHVrqUUMau34o1rrptLYE4EEkj+0Dun+fpVwzfaDy8Oq07sVgG/TlvVVJc2ntYAO14MSUX7ZyLqWaH1UWOryjNUoIPycmGpXMMBAPoh0tmFnygvF5ZeXiQXOObfvioHBvAb73szYgW8kYkVx6Ri0Uh5EmCtGSVfQyTubYcPC0J2j3mOXEIkXqGVmjRLf9q1XiU+PiMPokmhiOjvrHKUW7P1HHTcWenZaAxmdxOCpl0P2ROOtX2saw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3vbONjsZvravfMZFWf/jFmc0JqQCVqxQwTo9QP2uYlc=; b=oPKR+KNWg+7ly19zJ3ZceAniBf6uouNXV0iPI3EG8B7KhAL0DLTuRDYcBOT3kttmiUU4xwcZo43E/PVSF94kjpC/FgXLbrZjENIQDiQplQqMvauY7RASu2+WpTW43V/bey9fujbw9ETJky5cLGWa6jb1u/nAA/BUuwKLA7wmQXw= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by CY8PR10MB6730.namprd10.prod.outlook.com (2603:10b6:930:95::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6813.20; Mon, 25 Sep 2023 23:49:03 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6813.027; Mon, 25 Sep 2023 23:49:03 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v6 6/8] hugetlb: batch PMD split for bulk vmemmap dedup Date: Mon, 25 Sep 2023 16:48:34 -0700 Message-ID: <20230925234837.86786-7-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230925234837.86786-1-mike.kravetz@oracle.com> References: <20230925234837.86786-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0125.namprd03.prod.outlook.com (2603:10b6:303:8c::10) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|CY8PR10MB6730:EE_ X-MS-Office365-Filtering-Correlation-Id: a2751397-666d-4a53-5270-08dbbe21ff32 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: LArM927dLM33lUpMgw9ceM9vDGwF5A+LYEe9pKEE8ZtUG6l8mh1K5bb9alSFqhPfYZBO4FKHgl8Q1pqs/GZbj4hpjvQUpADz8xmr6MGJD+Kuu7nogf5Q03SUOjFltBxmGet40Ss3Tfr771NSqYLdMM9EbW3OnXCOjrgEOXfaZ/SbJRjDUcWCGaCo9753x0Xmlmfba6ogPqSrBYginR6NRdQmtMQNpsOLbvpBIlnXoOsUZhqFfvaiwjmzhMCx3FNjn6sV9KTO16QxJKOdelX7qFfgvdec7Gg46rJ/bNyHu+c4Z0o1TUmJLWGRi8M/DgfFjstZYsdxyHxPx+hV+k7pHI2jhjwkxizXFnAmOc99hHs5SY6SoCufMKf7eKWH8cannmboyvF3K9iI2CiGoNRpRkSDO4eQZc0GDjWdU5cO9Ynv6Ocy0RrNeG0oTkEc/NT1sUIgoIuMequZsamaUGRifKoTwaAJ3J8vJhjPFlNTF82eMCbI4MlCoewXd91nSCy2JdGzgTIMIK1TwyhnlVab8mt15GfSY5TmKVSehbZNidVEN9dGaCAdAgbnU+I3Dz1H X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(396003)(136003)(366004)(376002)(346002)(230922051799003)(186009)(1800799009)(451199024)(478600001)(6506007)(6666004)(6486002)(1076003)(2906002)(2616005)(5660300002)(6512007)(7416002)(54906003)(316002)(107886003)(66946007)(8936002)(66476007)(8676002)(4326008)(41300700001)(44832011)(66556008)(26005)(38100700002)(36756003)(86362001)(83380400001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: jjYlHVmXbFK9XFU6jg7b0rDR4Kxzxm+sgeSwTVf+UeWzviNrlAhAutxA0mA+iULi151QeovUJQ+LOP5UG4kXUnVMn7GEo0GQ7sGK0WpjvV6NZoPniLqqdEEHlMcDZwTwtrkgVQsfrH0YKOBbnmF+RX5s0CCkjpj/8wrHac5h4rEATV+Q/hiQBuy8DfuQiP5Oc+1RQlX1TufKooZYQ051f4WX2flqU9qJ7q6yOcvAiSkmjQZpYWOvxfCrrnizL5pgAXEjnzlRthwmtyH+jV56MrABm1S6in/bTv+icwEooTxZ7Xj09V8v9eeqL+RkGLCDbtze7ljV//BOCAUTc1/RZaoceqsbpzG9uPmJG2QLFOD6vI7RZIajQW4OTQ+KZEtxSGiZ/e+JS7kahXGDpmwnjOkXhXVaY4cxpkjMtPV3KK28iQvHXF/hoj90pFDzRBfMHJ6HPemdR7JpaNaAb6OZCfgbnjLn6IjMaiiDa+/ZBySuEJETyrALpF93Nj+s+O+ntLhH/AVzK0wozdxG+SodlRIruoul5DZMOS/tWnBhKru9fjO+H0n7C0dRtYrU4RWKXgnpbJ8XXpvW58vQGpT73qEkiPlyv+HwInsKtZEKcbiatNjGOxl4OZf7BA+9IuzGe2tJuOzu8z+qQZ+PLM5lG/ovL2beC4cm5rU6LIv9lpUYpucNlYAoZZMCgw8edP/+krK/O1evEDNvpP+yNAZDI/fnF9t19g2WgEkr6D4ae2HHAMUBBukqKjuXH1W++mIJXhU653zSjlD5uaYR1PHf6Scmjg2q72pVIiJYff5+Zj7K6Ajks72EXoHyVPBe0HCewvrr9lQpQ51L61e1d7YXfUasN1XmzCe1QPpduRL9UvYnZQXXLclpueNqGn8nKO6WhKiE2BN+UDMaCGj+Zgbran7exfDjhgtxi2unh8mpcwddkPHe8bpY5UTThn+ozZzokAILbf83pmHb4nBjrE+A1TOQgWGjUt3aBVH2D0Sg7txJyP060hfbEAvhkyZbvd4B+9z5gg/4qT0+DWXEkbF5S3OpRwKblt1sMnyfrQqdYDUPopeswO+/ues4hTfrp0p98P6Ztv8R6RcpFx4r8ASxkMGI4RQ9CxjGxW3/J6w6ui6+HufT3W+7PFFYmB34DPRL44bAbFbo8zwhTsENWWnJv9dTmOrYlqQuHAD1rEXS4iqEFlgPawNmjGRDyqJVprKfkjVvcxSDpnhmmAdtlFjX9AFN5e0FhU+NU2oNI2C9y4y43EerlvxIzLPzgYtKaYHIzhWGP2vr5geVmqN7/JQieZgeaPTDmg73TUZRxhL6TqTLVCAJayEJtzZ8tMAieymsG4qFuPECN7miGg57HTdnhwCZqnqmpoHRl9xXDuXcXgsccG1imXJ8b3vtRvUCiA3d+VMhYFAOIGBOrL87XAYNlV3hI3iGSylyLBe24MRT78f/WYcb1/Jq0GV6nIn//6/AIy547vuwsiYDHi988XV+16/PdkM0gQ2xIT+Z8MN4INTfKbTOXiwZv5uQhkccSsVYtGZS1cPQN4ErJJcOpzEAD+z6BEZwdQFCcCscJFVeqhwgDqbCbf8TROcxfxv92w6f8uphBdUbyIaLm3Y/+M8f7A== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: FvaRKkpSAsiVFUJIZuumIOLhxNft/fsR5Q2ihL1/hzM8VFbMbXEzbUfmlUOYRsY+VXWep9I3dlCyRteL0h+ZSdDIbJM/oDhnlk3UhU8e1U1dJ8XV2EIc4IOy0R0kwNFhquzTkhObjOEufK7WhHNTemv4OgX9YZ1VDVPAWmNVy7FhQqDmFQ8MyCIjKKSpiH/LQkt+B4EDl/crla1uAZTuTiyyNvLjPoA2iAwcZ43+Ubm9cP5g84td5tcpog46ekzRJ4McXM5rkRGJgguaC5Ys2Ku08OzlpRW2RkuyPxuhqvrlEiaese47hM3Es9/HfbKc7EJ/dfK69uHkuuuBgSJx9bn3MYRSkjtVmKxCcNySOtBc4rH0frw/X5zXc4frrVCRgwTREAhnuszvhPADruxlAQRIXuSrV3CVZNBh6ypOqT+YqB4sqjpvP1rcpivwnxnuGRLPjtvwS9gh82RxwhxjBEBHEzb0L9OP2mEfbe7BqcLQIxMJAeT3lDb6XR5UmXOfYk9hnguJkGd9MwhwmaZAvyCPBCscomtU3i1kNHx5y5wIKIax/WviNNwgREdx8aZLW/ihKpclUtiGmWC5TY73/fNuwBbvHcAhTsfdstM/G0Qboz7Fo0Dt+b8TbbeSVo4NP7kq066o5TyFs58pIVzkvhAZENA5Tb+QsYtHRgqVUW8z48DqAfGCL0qU79uM9BaYMUh47yhcj0Mlv9RthanHtkLun57AjLwpjkCEuXi051DsRRTsMBfNju610C/8kzLJiQvA5e9sQpx+2Gz879AtTouaz05viywQe3YAy/oIJA5o2uzMn9RqmyMcW3jf2eRuXiUt5kEXSvaqO79AS4EHKSFKd9VvUdbLl4hR7sK4vAU7N+U69lfEq7jim2LwUhyA9coMDdBzX4KceZnIZaFpVG26GSKcuFlk1nr0Anado7InFQMmrbFW44bxPHxhVcqxdpAGOxQJ3DU/Sis/QFpE9C26vPqIeAPohZpO0IeAPDDNohTvf+WY4J6lvlanYu7ccDxbU/PwqvoiGG1ccgxpyPdTTT55iJYdnneNCZ9SoqV5PXJqoLdVgiA+IzbOo151K85M4Pay5BXbSSIW9xrQKpgF76iGwbBg4SDjwRFT1HDcrcHIraEEuKIjBfod6l4Y X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: a2751397-666d-4a53-5270-08dbbe21ff32 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Sep 2023 23:49:03.0562 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: u4eGjld0LxFsIGAYClQpmaOJZZC8fvwoILCsx4UKbAQggmJHiK7rcdol5JvFvzt493qbrp+5MnEIhevgI3pANQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR10MB6730 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-09-25_18,2023-09-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0 suspectscore=0 adultscore=0 phishscore=0 mlxscore=0 spamscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000 definitions=main-2309250185 X-Proofpoint-GUID: hvMi1pojaw9HmLZcxevzTBP5YCmST8na X-Proofpoint-ORIG-GUID: hvMi1pojaw9HmLZcxevzTBP5YCmST8na X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 7E00F14000A X-Stat-Signature: xdd9gbohkamdnt9cbsrcmzo9jaa9rbze X-HE-Tag: 1695685776-47438 X-HE-Meta: U2FsdGVkX1/c+shxlJ/vyinGqld4kgPASX3Fxtgi+BLPqYdT1La5lTFq0ibHHGm9t+l+IENCNDtToBUGBZR9+JRnxR5awwXVfHsnzts+RFHuNSjBObXDbAZvHCqNMcefP7QVPrpaOaifkNzHBzZMGwV4pWD8jJP/f3OYvXXaNJvpayFduqHjhHEKQmBrxj0lBhMPwzo2Jp+fD7HlVyYpdThIi10RXzO+u6d+kVc6Do9n/SEhHExIAxSuX0ywi/7ZKlqG8PhnNFz2t9htL2lPHBqcIbhiHRMale7pcad7wJFS0RTLGex30qsWjWKIuZVv3THpzperHO3QlpxvED1TuwUddDvcfvaOxha9jKBq3G2ZTV45ROlgppyYeuBmIrVsYY6jfkDTmbbqmGuhCKryqW2VNUPJMSWnrVmGrfRLzEwc3Jw6ufLCwJ/GVpr21t/QMseF4x1QH6opPaRab/2jDmBu3x0nDBGIqsFdDkNWsIJISuEkV2pnxLvvQCg5vFw9wobkWqWZQ2Py3g63byHzYA5897E54MBuxrVjcnAdSDGBx9kPMMcr/BfPKgfZ96K8KywULq7Yit2xMY7FCNa2ZcPgxBfM7gnHnfGddfd1p/27sJt89+sP8dUxOW8x+5WAOpD3TsFKHhyH/GJnF+U4lNne7xYX7y9l+Kobga5zWgjDKdk1+QE7938gpn+WVE1ufYHnWVhojOjQpaM4Bvwga6/TI6dMWONAWOjVF+YzHtTFlAY/0Fgnzj2I28hbAzZXkQ6G06OjaGRuh36HKglQMuJakXUjtJIx9BkRlkXIMFjwOZ/bsCnwjfy6xjSTwV+kjoIYhfFkv+t0tQLZRxX7VCQ/AXy0qOswuhOu6Da0u/JBRfwAE1qcmODTrhN0dugiGlCMCYqF5i6fwuU5T4FfCnc9KKNEFYm2sFX4mzteCePpr3cJ3s3/PBjmiJc+zIM8AFH2T0F918ioE+nYQiY CTLp0nvb KIndMzSsng/IlZcrazerIXKYnI4TyXVhXNeW5ZVfqdBvSDwUr4sJe4y4KlZVtjHoiLxVORD+lhVwWoPmsMcHMdjhM4pvEVtU9Kv5bRdJPTIFmYkBXsPzLwtm+CLEnuiSXsJ866kdhxR314j7pwSGrE2tk+O2FxAAY0jmMrAzealcCkwXyxEYNWgQu6zGc+uyruMXxCe+7wmCChllyBaaO6OrlwlZELj0aBmSnSrSNAY7atDls5CUBNO5WkrxL8N+gV2AmRg+JPP5XAb88Vlxmi6/BebDqdzdQb1lGUHKtHR4PiFO7yFfcGDk8VNvI6t4nhVhq170jTH9H4/95cnezCLQvP8LlOkzVAmN9cfIyEMR5ROJ5Ajl7wusK4NLs3s7rV4nQ6cMmVGedgbpFTGKXsmBW4lt7aaCDwSIgjgknJxvVdJ1KwGvtrRwEklkptP4gibufKOhuNjhqxm0Y6mUIZ5T8ehQcYnF9Fh3q4Y5s0KywHUIiCDrP6Rs4Evq5Do9nPk3vacNDTWNRrMWwjGZr8O/2eCpgragKSYEqUFSdSc3Cmea7fIzZOLR4jpZhUBr6xcRoswcZlTDXNDWZU+edQ3kpYyqGWJdkSzlwbDVtHKXHxfsD1N0A5HVcpA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Joao Martins In an effort to minimize amount of TLB flushes, batch all PMD splits belonging to a range of pages in order to perform only 1 (global) TLB flush. Add a flags field to the walker and pass whether it's a bulk allocation or just a single page to decide to remap. First value (VMEMMAP_SPLIT_NO_TLB_FLUSH) designates the request to not do the TLB flush when we split the PMD. Rebased and updated by Mike Kravetz Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song --- mm/hugetlb_vmemmap.c | 92 ++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 88 insertions(+), 4 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 4ac521e596db..10739e4285d5 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -27,6 +27,8 @@ * @reuse_addr: the virtual address of the @reuse_page page. * @vmemmap_pages: the list head of the vmemmap pages that can be freed * or is mapped from. + * @flags: used to modify behavior in vmemmap page table walking + * operations. */ struct vmemmap_remap_walk { void (*remap_pte)(pte_t *pte, unsigned long addr, @@ -35,9 +37,13 @@ struct vmemmap_remap_walk { struct page *reuse_page; unsigned long reuse_addr; struct list_head *vmemmap_pages; + +/* Skip the TLB flush when we split the PMD */ +#define VMEMMAP_SPLIT_NO_TLB_FLUSH BIT(0) + unsigned long flags; }; -static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start) +static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, bool flush) { pmd_t __pmd; int i; @@ -80,7 +86,8 @@ static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start) /* Make pte visible before pmd. See comment in pmd_install(). */ smp_wmb(); pmd_populate_kernel(&init_mm, pmd, pgtable); - flush_tlb_kernel_range(start, start + PMD_SIZE); + if (flush) + flush_tlb_kernel_range(start, start + PMD_SIZE); } else { pte_free_kernel(&init_mm, pgtable); } @@ -127,11 +134,20 @@ static int vmemmap_pmd_range(pud_t *pud, unsigned long addr, do { int ret; - ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK); + ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK, + !(walk->flags & VMEMMAP_SPLIT_NO_TLB_FLUSH)); if (ret) return ret; next = pmd_addr_end(addr, end); + + /* + * We are only splitting, not remapping the hugetlb vmemmap + * pages. + */ + if (!walk->remap_pte) + continue; + vmemmap_pte_range(pmd, addr, next, walk); } while (pmd++, addr = next, addr != end); @@ -198,7 +214,8 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, return ret; } while (pgd++, addr = next, addr != end); - flush_tlb_kernel_range(start, end); + if (walk->remap_pte) + flush_tlb_kernel_range(start, end); return 0; } @@ -297,6 +314,36 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); } +/** + * vmemmap_remap_split - split the vmemmap virtual address range [@start, @end) + * backing PMDs of the directmap into PTEs + * @start: start address of the vmemmap virtual address range that we want + * to remap. + * @end: end address of the vmemmap virtual address range that we want to + * remap. + * @reuse: reuse address. + * + * Return: %0 on success, negative error code otherwise. + */ +static int vmemmap_remap_split(unsigned long start, unsigned long end, + unsigned long reuse) +{ + int ret; + struct vmemmap_remap_walk walk = { + .remap_pte = NULL, + .flags = VMEMMAP_SPLIT_NO_TLB_FLUSH, + }; + + /* See the comment in the vmemmap_remap_free(). */ + BUG_ON(start - reuse != PAGE_SIZE); + + mmap_read_lock(&init_mm); + ret = vmemmap_remap_range(reuse, end, &walk); + mmap_read_unlock(&init_mm); + + return ret; +} + /** * vmemmap_remap_free - remap the vmemmap virtual address range [@start, @end) * to the page which @reuse is mapped to, then free vmemmap @@ -320,6 +367,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .remap_pte = vmemmap_remap_pte, .reuse_addr = reuse, .vmemmap_pages = vmemmap_pages, + .flags = 0, }; int nid = page_to_nid((struct page *)reuse); gfp_t gfp_mask = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN; @@ -368,6 +416,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, .vmemmap_pages = vmemmap_pages, + .flags = 0, }; vmemmap_remap_range(reuse, end, &walk); @@ -419,6 +468,7 @@ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, + .flags = 0, }; /* See the comment in the vmemmap_remap_free(). */ @@ -628,11 +678,45 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) free_vmemmap_page_list(&vmemmap_pages); } +static int hugetlb_vmemmap_split(const struct hstate *h, struct page *head) +{ + unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; + unsigned long vmemmap_reuse; + + if (!vmemmap_should_optimize(h, head)) + return 0; + + vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); + vmemmap_reuse = vmemmap_start; + vmemmap_start += HUGETLB_VMEMMAP_RESERVE_SIZE; + + /* + * Split PMDs on the vmemmap virtual address range [@vmemmap_start, + * @vmemmap_end] + */ + return vmemmap_remap_split(vmemmap_start, vmemmap_end, vmemmap_reuse); +} + void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) { struct folio *folio; LIST_HEAD(vmemmap_pages); + list_for_each_entry(folio, folio_list, lru) { + int ret = hugetlb_vmemmap_split(h, &folio->page); + + /* + * Spliting the PMD requires allocating a page, thus lets fail + * early once we encounter the first OOM. No point in retrying + * as it can be dynamically done on remap with the memory + * we get back from the vmemmap deduplication. + */ + if (ret == -ENOMEM) + break; + } + + flush_tlb_all(); + list_for_each_entry(folio, folio_list, lru) { int ret = __hugetlb_vmemmap_optimize(h, &folio->page, &vmemmap_pages); From patchwork Mon Sep 25 23:48:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13398580 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35152E6FE31 for ; Mon, 25 Sep 2023 23:49:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1F76F8D0055; Mon, 25 Sep 2023 19:49:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1AC008D0007; Mon, 25 Sep 2023 19:49:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E29F68D0055; Mon, 25 Sep 2023 19:49:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id CC4608D0007 for ; Mon, 25 Sep 2023 19:49:40 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A52C3C0482 for ; Mon, 25 Sep 2023 23:49:40 +0000 (UTC) X-FDA: 81276764520.23.0FB8AD2 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf13.hostedemail.com (Postfix) with ESMTP id 479992002D for ; Mon, 25 Sep 2023 23:49:37 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=MpgtuMtb; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=vq2OWFc3; spf=pass (imf13.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:dns request to arcselector9901._domainkey.microsoft.com failed: no records with this name}") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695685777; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Tsc70UZvKWYWI+spZQiVZ3u2z8ZAo9dlRTusEOzzs6g=; b=Vof2rNLACT95dVb60jzFWY3MtzqwQxjUTkP9K3cve/RLFcdzqDODFAbSapWzqtMKkTOY7U kISyi9sf9at/cDbBJMn5Cn9ihe68jTUdAO3uV6VPagaEZTikQq0tDOHTDf8Pr6GM2JpISw lyW0kPi5e1EOByVo4g5PpR/U50E5rZo= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1695685777; a=rsa-sha256; cv=fail; b=d8xuXdzaDAl2/aUG5oh6VKFQy5V3FprsqHyc42Irz/uvjoBJzENBj2Jc13cznvUrt/lUNw jpX/FAn5GH4ZzGH/006oOxFgwzFiFMN+rbuS8awosbIIXsWN3RaTUzzWzbJ5CCFiLxapKi HNtc/BXH7pcdXtVJBj1my3Vu23MhITk= ARC-Authentication-Results: i=2; imf13.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=MpgtuMtb; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=vq2OWFc3; spf=pass (imf13.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:dns request to arcselector9901._domainkey.microsoft.com failed: no records with this name}") Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38PNEg29019013; Mon, 25 Sep 2023 23:49:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=Tsc70UZvKWYWI+spZQiVZ3u2z8ZAo9dlRTusEOzzs6g=; b=MpgtuMtbvdYGqxGxUX2CjdkVcRTIe4ieYi04A7RSr9QgiYTuFtkHlUbbhWqLToWrwUwi /2HsHfRulyWFo6RQvCZbPMW2n0P2nIX5Wwq3tq7dk7OZZUksYp5UMOFL5CvqBRYGOzD/ zuVBDu/f2u4x/qVSmyM4/42cYTVr3JVqIHQxr3c1jfm7NEKpg6XApSYoBLM/qBj4ynnB wyCpuGtGKCZn4Pl8E43GjVxPvVdSSCmxV71vW2Xsk0o6g1/CIMBYpHelbV9w4mDPWIGu NhobsVNeflvUvY65+RxaS7YJqmsVt8XENtbZFwj6pJ9cOmJeycxb/eH2LJpD54yYDXIp fw== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t9qwbd5k3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Sep 2023 23:49:08 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38PN1bGh039385; Mon, 25 Sep 2023 23:49:06 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2107.outbound.protection.outlook.com [104.47.55.107]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3t9pfbbdek-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Sep 2023 23:49:06 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nNw3cbgS97vxwMTJ4Elcc5I1umgu7o4yyeWMYMGaPTvzgYQg75KyKW3/Un7OcTO8lZeqGJZ9OMJBSg/inVTJtE7Km4DZa2nwdPkBpMZrhzO419BinbxXIxoyQx5DrnSb092K9FLzC1ToSresoOsSwr24FgHXA19OltAkcofXtKnBkO+9VVO8UJi87jOk4ouSaYdGGs347bKGcIo8Rpot4NFp9R+54NeKwj/jALJ/VNW5jVpFNiGZ4Nm1pjQcYUvsYrLaTQtTI2rtWzpDGEnoMsWsePr0l0OajsRChekwhw/4MUDjDXouErfvi53ohPOaUG5Mk8XOBCFz1qo/Py8P2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Tsc70UZvKWYWI+spZQiVZ3u2z8ZAo9dlRTusEOzzs6g=; b=DDTcYOFGdDmXuISdRxwpL2EqmEPAumtg2tAdpqLAsfOyxNCorxuGGc2hha+epINxCSVk3fdAJIbwd/BLJ7eZR7KQuRxJBrP6fz7HFoiV0G+R0zuoWBOKJ52pATmD3q+ahrusyrown5TLyMjTtbRzqJxOE3UP/pzEepYXhEIUhvtP6T0n0Nmx5lrapP66zoY1/7q+q4nFc0W+tyJTCGYox0B8U/Uz1m2loVaspMUYCqqpqBy6EJ0VwcVlAA/7gox6cTWFkh8vXuMlsd9ODarcCLlpiG7qTIf0wQewIBSlERJeZwN/eextga0kqom9tTJSsoksOBmXRcTjLV67xb8uqA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Tsc70UZvKWYWI+spZQiVZ3u2z8ZAo9dlRTusEOzzs6g=; b=vq2OWFc3ldzeuUXy+riRBy6sOvJBKNFbUUXEkvOzVuyq04wh1lSj2MpifDw8PwwO33CkwCYUeX3oyrFxszvPoJk2t322EcBmj25rbRk8pP7FJ8grcVoi8AnDSjIOWgJ+3bZWNnVu3Vzjs0J3N0WWTHpK7de42bEuQP09Bm8nltQ= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by CY8PR10MB6730.namprd10.prod.outlook.com (2603:10b6:930:95::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6813.20; Mon, 25 Sep 2023 23:49:05 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6813.027; Mon, 25 Sep 2023 23:49:05 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v6 7/8] hugetlb: batch TLB flushes when freeing vmemmap Date: Mon, 25 Sep 2023 16:48:35 -0700 Message-ID: <20230925234837.86786-8-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230925234837.86786-1-mike.kravetz@oracle.com> References: <20230925234837.86786-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW2PR2101CA0011.namprd21.prod.outlook.com (2603:10b6:302:1::24) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|CY8PR10MB6730:EE_ X-MS-Office365-Filtering-Correlation-Id: 265e3905-51ce-4a70-e27f-08dbbe220097 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ACq9scLaREZJQQzSJQNGHaQPeWdQ36JtRxI8Ve03Eusp6nPNJ7Jkk5pb9pQuI8vHCG6E0KEpXQpNUaNPh9xpGK9eekoYWOy7PfD76Ybvm6xulpWAv4l02PiBm/Yd9wkn002wIs8eSw6Xv4FUAB27Z+UGlvZXaqXy0dUsGMv/jQTcpjIl3jDEsKn5oh7VnEKFjjgYrSOqwHKKETmrztQojJ2Uzvlu+48F0Stuf5SJqgdQDmy62oLKNDH2iUE80p8BSxEvGd92bLuJSenmnWmC+do0paPib2cb8cJUQ8BuI01ws6tvb/ovkPkxm10aaGpVA1RPpb8vWG/vRSn3hJi7Nvj6TWGorflnesbmO1XUPnE6IWcRuKJdD2gTXnvrjPN+VbFMq42K9ULM/JsUUKwBn5aqqOEFjyEskPWdec/Wz19u7nBdxqTTInQx1osjbBdvZ3lJQGBivYQ5u2CKi5NXb+D5BbHIuM+1xQ8sWCjMDvFCk9kHi0Pr85RiQhQJ5TtLX3kFHadk/69cD2n7gSyDVNrzBqvLQK4WTHz46SRyGdCNyiu0WOvuXK+JWPgi7mb2 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(396003)(136003)(366004)(376002)(346002)(230922051799003)(186009)(1800799009)(451199024)(478600001)(6506007)(6666004)(6486002)(1076003)(2906002)(2616005)(5660300002)(6512007)(7416002)(54906003)(316002)(107886003)(66946007)(8936002)(66476007)(8676002)(4326008)(41300700001)(44832011)(66556008)(26005)(38100700002)(36756003)(86362001)(83380400001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ZKmLS7zuH8cbCuNleigQuQlw09Q4jKQ47CtlhiQnk//KneBSR/mhXQ1nTWGME1FUyoYjTMWMXU4eBbCjUpAlu8mwkhlKgfjll8e2sreG3cKfEXu/HlY3Wed9D8fbIw+pnnGq/5lP9td0OnQSPC7BDiNbdJJIC0YVUvnJJZVMqEhJ7fyztyykZegvKIdZg7IWzUdcVyW6CzsuKyNXPH6YQ+CS+4B9t49fn8Ez2mVZcAiEAyuc6SHa/ckxrjNg6rHP++Jm6Q91NsJQRRX6OC16xwRoaJPMYdrwz8YYctdVPBFRvLcOILIwaqkn0TFEmzScEfZsynZd/1DB8hxf+MSo3Fysk+arJNoXR4DUA+GQSASTIiKpyIea99/vcnCVoGa00k12hMc1jrY4c/rcNUnSnE3gqInYaKA3AnG/gPUQMHDBz05rlqy6/qMqRL9qOGt9lnhqVn9L8qMEDZrcH1Q5S+p8wsXi4YRiL5Rn43N8Z6NYrs9jhtYdWdAVBtE3ME1Izz1OgVlFfJQo0MY39ni7JqCoD28SzQyDkTEc6/++0YnIpiZ7aYSor/Y97rQlxV4ZXY56aCZF9QZoc5mP+qErrCkVYTQ27UmGY2xf45nk4wsN9F18UNBXpa2jHqt5knNKscB+GGQW5v0c1sDsljrJtnW+sZt1QCAwE5bBlEIrC6Cpch6aiBm4XBbsSrkY5v/yGpOWozlMoNzIHvbfggciHJ/8i7CQWsUKh7/RvdJqyXkihOxDbNprvBPCdzDQ/4qm8eUXO5YXFE1J5XeleA1bwWgfVh6JtmBWH6dLfIo5/Zin5MOhlRWjbjDEIraX77u70cUsC+P5T5u3t7KiViM+lP5xms/mmgSD1oqRCBHYkKhVMmxq+Wj2TvvgkP/nXGmBf2uwvinoIHWD46px7LRqT49IUg8MVu/9gK0qrEJrwHCqjib3Azz4UEYhwGIWcM9OE9Jj1Xqq9aiChNQFA4HmTg44f9UIMJJ0VvlsHyYeDv61wW4Fo8DUOi6AqNlN4D/ERp5Qy5eLt0Ie1nh8MOxufz9A9fAoD1SBjqDWvIHYn3nQevteyiwUuoPg2z2bz0lQWSrx+k1lkBZ1EoWjm+0qyZ3iYBvxR8ltvn65vXPRdrmfqu2zjfFU5zSglP+3KHxZg+APgfFMIgtPtnQhQw23SRjQ2RtvazmpcLocJLsI41GjeffShxxoOXS+CQS4SutSzfmHE2ih3m6cZ+2fSeJt3oP9E/q9m+xx9ZpoV3Nbm+AEaImbEQ90Mdux0m+8Mx3Zuq+G4PDER1T9trjUACtc8I9glAzHvV4Xk8x5GRHIUOmZ5WbrEtiw3OZdIFLQcvJrjTLzqlMVMbggyHrufkVuA+0ObAmG+Dl3aTNAR6kp+zjknbwEfwrrAy2tKG0RXZgBFojmllIk3sbwHy4GLlnN+mBr88vmN9nUFkYKepohXakxoywK+DkEe3+sQiKWUAXwbBoee61gbQsItYkx208Y6W1mFsoU+1Nga7SRhc/H6Mgz/l10MpNP0IA/BHuKZG3JEHKfo5tSDxeDKh7FkYkweakZ87YbJEWtRr6sGVoGorCUTX9DwD14vVvWKZGOoxLGBRjvWUDQyyH4JjQrhHHOFg== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 5AJ+356RewC1DOh9Rb6JHMSLwIXKPQaYmO18STxSPSKRukbY/c6xFUJepHdpRSDlTIzIgbKNnCeJuik0pXVcoqUwPFAlzuSsd98agD0J33EPzAlWGfDztHIG2bdKr/afi62wsQPSyWSwT0D8FZSTCaQaAjyTgdkcH0bFpVlEV37S+AEIwHNNk64aXrU2SMPq9VKVi/M68Nq2I7WJUL6727sP5uD2CHy3StoHXWjwPgJAjNpMow29kcfJxNVSVUfYGTHTnGlm67h4tYAaOpVoJ37cKrxdHhuS//HBSC+aSvbWmMVKzTx8yox9tmXhVXcic11xCDsxyPkAd3UTY5qJv5mTSaFnX4SxvE4Xu8ZvZmyD/LKHAIj8xZsSSz5Lb1i5tukLuCPCbNy4AuZnb1L1AdWDa0gJOA0nM3wNYWhDRdRkXQJW5ssWi2Gv23fRu5BVy6ry0f8VV4QV6AyVmy3q6LEo1m6KOviFcns/aGAJGtfw1h7bPBo37zLdosdhckG2X4/hnwuS4K+4U+bLfqUqRC3lHHK2SPY88POUHqS7wNn3//DECKuMGfQ5AP0lYGaHXyG1M6rPRBCIzwiDqWXM3XLZ6fQhi33znarT78vFALomrO4Qw+XnMSqwAmR9DAf1GS+UL3phIzZfwM22ZD+eORd6AiW9PCW8Px1dBQ9hS3TYdEQMMDoF5+RrB1UfTy/fqQv4qshZ8vXoi884yZdGjoXeqVC/UPCBhVJaRvmpfmoDINVYscV6R9hhvqAQ4x/bsAFPaHCh7UuUurmVUTOKup/jMVcDFc2o8H4D0dZ3QZ8ulL1YMVQmFoI/Df1bZSLXaWR1VxUasShScaJI5dkEGWBiW3wi/+FVL6qneTr5ownKK3aCMzujFuQSpGXyC3bZtnHKgYsJ69godUm+TjfA5XxLCQEi6YTKJIWWo8r30fsw4Z9xhk8It66TGqTyIvGUpt7RhccYMBZgK4urySK2QyNnp5DXIBJSe5l+EWGFElOKD9Qi2l3P7G/3gPUe2zZmmuAEOWyEmMClYCls+gqN23uuh9TjcDeY2DwGsxKRSunL17m+XkqjpGsGb66b/aJYrCJnl+elzE8q3xFo1Pfl8T3b6Ujj/IPaSgaOCUkgRzAKfHN/ELJtbdHB35hByD/b X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 265e3905-51ce-4a70-e27f-08dbbe220097 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Sep 2023 23:49:05.2523 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 0CT6Nb+AcxyncyzFEcEwVvLBWZ9CI/yf1iyEP3Oqkaf2w7zj1UEEhqGHUqw4wvI5B+my7nEwPEsAC2ZlCtOtiw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR10MB6730 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-09-25_18,2023-09-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0 suspectscore=0 adultscore=0 phishscore=0 mlxscore=0 spamscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000 definitions=main-2309250185 X-Proofpoint-GUID: Ue732tyiVC_0vyydzZLVPZCNLDDqpiBx X-Proofpoint-ORIG-GUID: Ue732tyiVC_0vyydzZLVPZCNLDDqpiBx X-Rspamd-Queue-Id: 479992002D X-Rspam-User: X-Stat-Signature: gh5hcqx6w4g8estbzq9h67n4gon6k9gy X-Rspamd-Server: rspam03 X-HE-Tag: 1695685777-302045 X-HE-Meta: U2FsdGVkX18SddfZSl/f+rDQ9JBZHP3c8o6XS4kevHXGhue6dGNqzaxpmdWxomrtE2EPZTMNsScVQwuJ3TXHwQQ3Vkv/W+/nFAmkqZB0zCW0JKyS0wW8g/fOVxrKpI34q/94UnF7Zu6nz8u4YeMRajVsaFegY7f6tEZEtOpJ87Iz1C2BOvRIUIol++uMoLWqqBZPjYJlN4GlmfIVF7mAEQwM394I0XFqcRAvx32FCxpBdsEV8HpOuBQ2gZkp9jt6zwWW0wr9Ef7q0+hrdZPAYoeHrP2v8MfUtpmlkoIExVXAILc6wkNAGBI48X/jdPwKLyuHQ0YFuhCf3jBFKYh0GSGUkT6JS7SEVheju50p6CMA5G4T20mrQ7QuiESE2hiLRF+0cIwhss98v3BFXrAxPYaDxE8/bHTc4AW+WVw5c3ot17ieUzmZVTJ37T4Tu32gVAD0tbWjHyl+KYJjfHqdw95dzwSf5345lFjFjHzEqA80ZW75mt7zL0c5aJE0Q+NphCowGaGL89qyweIeEDtchqvV5OatpVpoOIaraiHrypfkEAhjDu5xH9VCrIIgEUhgOIb6793ms3HgrDziKR4SXojGA+hnuJbdUlm6rPnJcdWBU1lhlyLwWqpL0U+ATt1ywfH817D1CyL2+G8OHeQOdgKGB+TsKRiNR0X9lfjdmzEGsUwERiKl78NgXB7n0dw0UkEmXpagThSRvYYeHej9baiVq7+jrFhdkSDQ7Y6baoGBn+7eKPPykNDA42e/bFgcaE/sPSipUdmD4Iy1zmKctY8PNCyIShT/1vS0BKY7VWNr3a4HJZYTwfHvG5kJwrawsPDBagsp3heSAEenQK/BQ8LZuERtbOd2+FW0CeyHI8zbj+sy04kZ6ACMwglCaRMrjbyCEzL7JC7pjwSIzGbJgxHemavejjOa9P3+e0Kg4UQrw1V1KmlryNfCu6J7+s4Uekvr3TNBRWm6+fhq4jI 6zvlfgkJ GdI6t6zBCkx6Ps6EXWBwIE1E6jkkn0DS4zMEr6+BbJA6389qvIXyVv9Kki4O6au25TnKgTV0WWplT2ADKrDvtAXjd8hyaxmPbvvr0X26um+tiKHs6YpKLDIR0kbtSast9pAupGomcfXzLE22zldAcF59PVKwjiKhCL+TJyz1VYPA25E/0gw5enS0DdKavxMvDzfb/QnPynRxAff6eIUDFGPAaqYTUwWUlGq+6lg+s77tDloXCSpca12ft+pAMhs3Pag0IoL/J0O1MUVP3nvimKDSntkBOwCiccRSzUHw9/OV25Yf87OL9Jod8q8ElZnRQ39dPA8alI0jAUCXAzYL3ZKMdFzIh+K81srtiqcgcXi/mK15DfwvxIa0tNPqbqRKVi04ankpZKjjrsuq8JBlOFsxKLAFxJFIJS6quzEu269BC1ABqhqelSI9zjCgWXh1GCdFl3S2uqhQTIGzZALsWVV1nW3Lih1V1RgBAqj1rcUo7bYP/eQsU9zCwteMWqMD4xoHJP3vQOSLMWSiOKM36LSiG+31IXeS2ZIwwZtHxp3qkFwhTZfKmWTJAo0h+7ENh81DFuFVpJuxJPLFUQPyTYgFIIMpUsRdGjpcXpttRrc2oteIjy4wPfzhguA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Joao Martins Now that a list of pages is deduplicated at once, the TLB flush can be batched for all vmemmap pages that got remapped. Expand the flags field value to pass whether to skip the TLB flush on remap of the PTE. The TLB flush is global as we don't have guarantees from caller that the set of folios is contiguous, or to add complexity in composing a list of kVAs to flush. Modified by Mike Kravetz to perform TLB flush on single folio if an error is encountered. Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song --- mm/hugetlb_vmemmap.c | 49 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 38 insertions(+), 11 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 10739e4285d5..9df350372046 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -40,6 +40,8 @@ struct vmemmap_remap_walk { /* Skip the TLB flush when we split the PMD */ #define VMEMMAP_SPLIT_NO_TLB_FLUSH BIT(0) +/* Skip the TLB flush when we remap the PTE */ +#define VMEMMAP_REMAP_NO_TLB_FLUSH BIT(1) unsigned long flags; }; @@ -214,7 +216,7 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, return ret; } while (pgd++, addr = next, addr != end); - if (walk->remap_pte) + if (walk->remap_pte && !(walk->flags & VMEMMAP_REMAP_NO_TLB_FLUSH)) flush_tlb_kernel_range(start, end); return 0; @@ -355,19 +357,21 @@ static int vmemmap_remap_split(unsigned long start, unsigned long end, * @reuse: reuse address. * @vmemmap_pages: list to deposit vmemmap pages to be freed. It is callers * responsibility to free pages. + * @flags: modifications to vmemmap_remap_walk flags * * Return: %0 on success, negative error code otherwise. */ static int vmemmap_remap_free(unsigned long start, unsigned long end, unsigned long reuse, - struct list_head *vmemmap_pages) + struct list_head *vmemmap_pages, + unsigned long flags) { int ret; struct vmemmap_remap_walk walk = { .remap_pte = vmemmap_remap_pte, .reuse_addr = reuse, .vmemmap_pages = vmemmap_pages, - .flags = 0, + .flags = flags, }; int nid = page_to_nid((struct page *)reuse); gfp_t gfp_mask = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN; @@ -629,7 +633,8 @@ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *h static int __hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head, - struct list_head *vmemmap_pages) + struct list_head *vmemmap_pages, + unsigned long flags) { int ret = 0; unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; @@ -640,6 +645,18 @@ static int __hugetlb_vmemmap_optimize(const struct hstate *h, return ret; static_branch_inc(&hugetlb_optimize_vmemmap_key); + /* + * Very Subtle + * If VMEMMAP_REMAP_NO_TLB_FLUSH is set, TLB flushing is not performed + * immediately after remapping. As a result, subsequent accesses + * and modifications to struct pages associated with the hugetlb + * page could be to the OLD struct pages. Set the vmemmap optimized + * flag here so that it is copied to the new head page. This keeps + * the old and new struct pages in sync. + * If there is an error during optimization, we will immediately FLUSH + * the TLB and clear the flag below. + */ + SetHPageVmemmapOptimized(head); vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); vmemmap_reuse = vmemmap_start; @@ -651,11 +668,12 @@ static int __hugetlb_vmemmap_optimize(const struct hstate *h, * mapping the range to vmemmap_pages list so that they can be freed by * the caller. */ - ret = vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, vmemmap_pages); - if (ret) + ret = vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, + vmemmap_pages, flags); + if (ret) { static_branch_dec(&hugetlb_optimize_vmemmap_key); - else - SetHPageVmemmapOptimized(head); + ClearHPageVmemmapOptimized(head); + } return ret; } @@ -674,7 +692,7 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) { LIST_HEAD(vmemmap_pages); - __hugetlb_vmemmap_optimize(h, head, &vmemmap_pages); + __hugetlb_vmemmap_optimize(h, head, &vmemmap_pages, 0); free_vmemmap_page_list(&vmemmap_pages); } @@ -719,19 +737,28 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l list_for_each_entry(folio, folio_list, lru) { int ret = __hugetlb_vmemmap_optimize(h, &folio->page, - &vmemmap_pages); + &vmemmap_pages, + VMEMMAP_REMAP_NO_TLB_FLUSH); /* * Pages to be freed may have been accumulated. If we * encounter an ENOMEM, free what we have and try again. + * This can occur in the case that both spliting fails + * halfway and head page allocation also failed. In this + * case __hugetlb_vmemmap_optimize() would free memory + * allowing more vmemmap remaps to occur. */ if (ret == -ENOMEM && !list_empty(&vmemmap_pages)) { + flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); INIT_LIST_HEAD(&vmemmap_pages); - __hugetlb_vmemmap_optimize(h, &folio->page, &vmemmap_pages); + __hugetlb_vmemmap_optimize(h, &folio->page, + &vmemmap_pages, + VMEMMAP_REMAP_NO_TLB_FLUSH); } } + flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); } From patchwork Mon Sep 25 23:48:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13398581 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDA45E7D0C5 for ; Mon, 25 Sep 2023 23:49:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 490ED8D0056; Mon, 25 Sep 2023 19:49:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4182B8D0007; Mon, 25 Sep 2023 19:49:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1CED28D0056; Mon, 25 Sep 2023 19:49:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 061688D0007 for ; Mon, 25 Sep 2023 19:49:45 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id DC64DA023C for ; Mon, 25 Sep 2023 23:49:44 +0000 (UTC) X-FDA: 81276764688.30.D8FA277 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf28.hostedemail.com (Postfix) with ESMTP id 79C14C0013 for ; Mon, 25 Sep 2023 23:49:41 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=tTkyREEu; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=xHO55F1Y; dmarc=pass (policy=none) header.from=oracle.com; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:dns request to arcselector9901._domainkey.microsoft.com failed: no records with this name}"); spf=pass (imf28.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695685781; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wop91Vn2thHKTczlqkauN3ZgszKMNCLF+DVzOTx6zso=; b=4khrUDE/YkMaw4AN1Y1G43DHWIDYXT854cL1fQ8En7KYO4VM6r8NdLRdDkk0BPHuWa5Dtp xdMjc4WMA6jqdsRYx+mRzyYk68o9dCEDIvyON6tDcIpmOCYCTlJNwvw35Cdf2lChuF5aDK nIGD8ZCzOa/b5cLUG5mmVsMEITAWkCk= ARC-Authentication-Results: i=2; imf28.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=tTkyREEu; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=xHO55F1Y; dmarc=pass (policy=none) header.from=oracle.com; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:dns request to arcselector9901._domainkey.microsoft.com failed: no records with this name}"); spf=pass (imf28.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1695685781; a=rsa-sha256; cv=fail; b=5MtshlsvfXXyhu3VRX6XlnuEK0Q6Dno/4ItY/D0Ac9OKzPaGmL8HBefwPheYtrrlpFeEG8 3MTRmxhgYEjkwO4I0m2YB5C6nnfjojxGR95jNTL8IqMI4pK8Oef/XZBwpuaxN1b/X0UajU gvpq1/BR/Wvc/L47po8PGe0urpwBR+I= Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38PNECG6011832; Mon, 25 Sep 2023 23:49:12 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=wop91Vn2thHKTczlqkauN3ZgszKMNCLF+DVzOTx6zso=; b=tTkyREEus1vZ0vrDzN8bPKtzQKfpAiGC1GOGPaIkTQapP39ASs7lLyOyDeufLNRhS0Bq mLyLV30tsnRIKhGGCcZ9F9+OnCGxyIjWYIhvoj7jeR7bQ2G/21PoLxrzx3PxR7dkt68u zJAkC9rPtaWVEqSAaaQ5DTvxU0CVJmK5RdfNL9Qr2eS57H0CqH6NDqqcpCJu/cpYcAp+ HmwocPBprZp4f1QkA9sEaUOaeWJi+MogV6ekyWxuy1flr6PuTmPdpU2d9HWiSpiIp3uS 0jtEngGWVq5iwYwk5sm6OEVdA7zRy0RSTHMOPNe0qOOI9ay1Do6UMjQpPqonX0L/f2W2 DQ== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t9rjud6cv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Sep 2023 23:49:11 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38PMKL8A034986; Mon, 25 Sep 2023 23:49:11 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2109.outbound.protection.outlook.com [104.47.55.109]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3t9pf5ghug-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Sep 2023 23:49:10 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NZwOQSCPCLhjP2+ylha47wYrCIv9Z3199JZ0XiV9pWPR+/tYvLBBNsp+8GRXLg1ZgVLiMCExsHoAqv78pAOMu5SqNYO6zl+DywoTncYllbdyKsf7iMyDY8X/BllCK4PojzqnNmvpJJ3O13/oTvqSfhOk3GPUYeVXhp0/J4ANfljq/shKLPhG+dCkscyaXY2Hj0ZOTwvT1QJtqwEWvLqIK4VquQpRwGFb+aM8QdbTj4/QlSJmZ5VYewJ6ntraA+0V+GashGWk9N0d/cHKH8x7KPxx6f9hA2h3A+XmVIEFA+dXgwE6Hw4lXQaYeVWGuRzNq15zNZaMBdo+OOT65B6F1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wop91Vn2thHKTczlqkauN3ZgszKMNCLF+DVzOTx6zso=; b=Ni5vbv/L8c7keanSmSajLhytGwjKnN07Ff40hUksPz/wRiI+RkTahQVOzot+aPDW/SF5f00H9BvFrT8D/JunecGfKmdBi+9t9x1MM1Gg7cnpLJsuhE3mtC4ba7JrTzFHtxdmkszbOSD0s75zG+69B+18LVB7LQty/Wup0KtR3+lQMLk0yslAFGCAdrnYI8msQglcYDo6uRcRVW48SPURYCPNakONNZ5xNRr0Lbk4nZcrHJnP8aPayRtzK/xlBJ1ucAOOHRETfNgstjY8HRMBMtEDOYbYyg2Eu5rANdzS/gnDT9KfyTTVCbPFu4/Dij63sy0qmb16XuJqBg7yEdJ4Zg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wop91Vn2thHKTczlqkauN3ZgszKMNCLF+DVzOTx6zso=; b=xHO55F1Y7vpmQliU/v5oVMTf8eqhsgbGIc6t4ZsffPIPDe9aR9elk9G/RuncmknNnmYNzp7vjORz4DkavSaBXFvR9spKy0VDqINCiL0u7NOIcNNyJN9B84slBy+tWdyNr6WZlp5XG/s+tApQlGXHYGYvY0Jw0hFzuRdVMhl5zdk= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by CY8PR10MB6730.namprd10.prod.outlook.com (2603:10b6:930:95::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6813.20; Mon, 25 Sep 2023 23:49:08 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6813.027; Mon, 25 Sep 2023 23:49:08 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v6 8/8] hugetlb: batch TLB flushes when restoring vmemmap Date: Mon, 25 Sep 2023 16:48:36 -0700 Message-ID: <20230925234837.86786-9-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230925234837.86786-1-mike.kravetz@oracle.com> References: <20230925234837.86786-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4P223CA0017.NAMP223.PROD.OUTLOOK.COM (2603:10b6:303:80::22) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|CY8PR10MB6730:EE_ X-MS-Office365-Filtering-Correlation-Id: 33069808-d3ed-45da-3aa2-08dbbe2202b0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: UzC9s3jxM8bMObKUzSWGxl8NtcI/OTOVbEuxSL5Svf86thYIXYr2PAKjD5Msdj50wCCFI37gwqJpCPfmJOI5aYKBqjlC01sULB+ty6VGrkLgGalomTuKYgubXtSKIvEzuP3mhh2flklVdlPUU4fThjI61FKhDSlkgnWWSnL8nQ6hKMxf42VD7GBKAqa5KdSej2IkQddEoiP1wingXCgMEwjnc2rxCY5DNbLiiolti7L0DzXZ3MFQgIr+1hz0GsXf8kBF3WIrN5vSFfs8w88HiNsvmaAZHvAa8uwWVD3AWteAfeom12DhnMwG07DoyyXch0kjDewxqzH8CAkr0fh/fH03RoS1Krx+TESER11rttTrh1Sq78FIzsC4xvZhEqre3XjD2R+0d5KVxro3hSE+RW0Ogj0zkDc8Ne7t2rKBGWxKWMuI7+FRmcCQ+spRRnInr8H1typsVhRwbY+10OZSUds0Z5CaP3yPZlqxVEzu3pWdcn7ztfoLVP9Jn15oF9z8lCxdu6u92yZhaCvpZSneuflWttURHfkivolMItf8F66X5RKz7aiXrXga8SnoqGgf X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(396003)(136003)(366004)(376002)(346002)(230922051799003)(186009)(1800799009)(451199024)(478600001)(6506007)(6666004)(6486002)(1076003)(2906002)(2616005)(5660300002)(6512007)(7416002)(54906003)(316002)(107886003)(66946007)(8936002)(66476007)(8676002)(4326008)(41300700001)(44832011)(66556008)(26005)(38100700002)(36756003)(86362001)(83380400001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: KRkVZXgshpr0ZwNWrmijfqP32KUN83tktdmkDcQkR33ChILvoMzafRauz7t4S9g58+TK0xU4ygKKYloBhh6yv5K/tbjs/v8F7xXOP+46LENYi81JF3mx3b2knGApkvQpOT6I/gdC9N1PVQY0xRPihh7a9RH+z6hstJIUPLQNdG3e0xADcxYgX9MuNp164tKS15Zusu9j3Nks5B/0OOGt/yJn0p4mHMBhhNtSdhETFLJFIPW9ghxeqaK0cGAzJu91P+41BoRygzvCWQ7ZA+4J9xchemn49B2U48LuhgeJhLUIoCaMI6yCDGkmY3k9o/nnqcqrDMiXPVFtbAPKp+Bc94Z/7YYDigQSqfawtbL8TFJhYjDtjmfxB7U+Wakh2YYzELIilBBEjkyy34gXsIZRgSuxkj1K28zHBH/XbDMXAbjsn0ZqC5v6DFhvYvvI8Q4catZDLayW6uzKO3XijQGQmkZZG4rumaSzgcraKKujLiC/fN8QnarZKL5TP/pSnbhlmxLuJlxyyVPQNPKqU/m7buViyt2WOWK+HD1k5848bjBQFvl4OF3BVmFCMmU9HZvWL+m+Rn2fves/KyMuBrtmClsYnBDMuonetf8ZlplkxmiBEehOnSVvUD7inrp/B5rkVqNKosL2IHXZuK57W8eGaLww3bKmjAZGfnfw+IeMjZLHU1WIgfSoV99Egfev/akJ0kvOJJ3bAu7criGWu3nZFt8zl7aAsFwnF5R8XcsxPo21vcm1HjPQ8K+u0S6j4dD+0p8TbxeNNt9cxp+3leIXnf6qseL9ccQc/bu+lJgRk7Ulrf8M+IqEnATLK0p4bBVcPUBhi3uNPrn3PqX1PLc8aBDExi+ddmrhSG+ENt1qoYI3tZ+YfYkthgZ22X8VDQAJDC+0lnIXrrNS8rR8nw/taPhZWpckJxerwglwyZUvFfmtML5ko06LhHdQQlbiUG53DztZYo3WyRKlhsYwWyeHffbZ4fY/qmar8LVefl5um8MFVE8tMW2DVKSGRNxREkwILTsdWjKHb2/QaFPUtLC/bX7aOzcoFFzsgH3bN3BvctiuRCV9ODWJJ+iXJuYBmtY/tVSXoCJVsQat726COoKS0J3fXy4neZczeHLZJS2SlWOIeRE6cx0XZDs4EGaYzeCFnQy6X10x1vltZZ8HV4vNwqm4YkG+hOQeLlbgMd9R4xCZSNsbb5ORh5q58uQ0U4Y86OmQsssNqH7cuRYMbJONEkMdUzZcjJGKjr8hlAuTCIbIj0+16l8Vksx7utfRMrP/a7XNAs3c4KqeLufMdM8fKQNhINMbylh+11BtA1K2WsCZdb593NcAW/ehT0eVEPGpHZj0I82cYmhewJZ02fsUaj94eSBchi4rId/WCo2l4g/5UXD5gmU3vOLxCCa7OIVClIuv6kYha5m6d1qcFhNs26WaicaFwTALWt03mfpaTrtCHIBZPtjPEmMT02kHd7s4tGkZcTh6BRU7/CP5u7kwnIvoUktTTX6T5glQ88+TGzmjBuH/68fxGM5qXIxMppefPhexvzoEqm96/4K2eSaMJzSP8OvVhTDBzfTjZ4BJdEt9fTE23hCxOKsSKfnW5OEaizuD0U5QlU0ODewnb28b7w== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: qzbJk0zHL3K4reegkmZaWVWTpGKn5JHySJvVXe787q+2sB/zMvXJUWshizXz68fvlgUdmaobfWWwcRF6d8A/c1WrB0bC/HpTwc9edUgUmNAFHLD/MUS9+jCjaZxHy8GRMOkS4jaMnoyAWNdoGr+FjcJh4ol0a5YTrKoNW67DRuOH+PmWJAqr9e1nQzgA/ABw4RwByt8XPNEOWaHynF91zuNgFCG8qeADUiIMEKfKSYley5Vc6Let3m/kTQyTdQXS97kzLMd49uzfVErerW5V0GYfniPIXpFFqC6Y1j8ky6SGoRqWiWF58u7u3gkAgIvPu4935wJXu1Jm0cGwAVQCblaYypXKvOZ+a/fjwOGpaQ6H89A30F2ejKOmcdEx5g9LoPl/JsEA10G4jxtc7orI7fZOYl9dsfa0ct7IcaXIwdOkPlEanAWd7CReBhcyZqeEFqD286R5kfaQnW1JTFgl5rDiLjehlqtfDsQzqjbo1Hs5IkoMDAsP6xkJfI3CspfrasS4pz4nmVpuw4jQ2qjso0jaly2V0D3NLysuPf7GybrcV4XAIIMqnOvdzJfo/OZUavLBpdeXA4aqFak7PS6FIp0QhBWLHNckTIz89IQ217zgX1c/7FhLH7AYBCr2DazeITtGrO0eC85D+UW4kJF70PtA1iRKdcgi+AloECWfAVsLb6fDRpbIUKgKNPJa/MF+mUq1AUZouqwkH30rBUPPx92cpFOmEwDza7JQgmRIN1LgF+1cLcUJUQMk/QxkCuBU1yCRSDtfFinJqm4VAMcMiMOCvvWuQ4H9xtnQtBLkSkO8haIfUkhzY6iJ2i+I1SZQ/DptKCbqB6H4LSBJKl9RyRLrUgkU5lmhJFqprEF1+c2LxO5/ajFL0AfH7md8CwF3aoVOcfB6tVAsmD8md0s2RSS6HkbPGrgOnO88CA+Dg1SKLFcNJSg7Hm2Vu7rYAyBzVoT/lvCL0B0eBkcNsV4AlXdTr5duJiy1B3MP0xG8gbUAGPY/PfIhBUlczlMCst7JQDmAWxPfbvv9yektpKt1rSv31NckRzJJEzwnSqCxzj/KzFuifD/+BfIJ5GMwlOfqIYgGaAIQ3fv+fsuip/zonu9ZJegi2FHiix2mic7mKZeLMzfuc8IaQ4EIB8i3ZsZg X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 33069808-d3ed-45da-3aa2-08dbbe2202b0 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Sep 2023 23:49:08.7588 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: uqkSeEHVZC/+w1DIELPieGGrlz5rzL5ah83wPksISv1qVOkMvCB8QXxEhDiKBQOv1Bl8HE/71RddTeCOr1o+MA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR10MB6730 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-09-25_18,2023-09-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 bulkscore=0 mlxscore=0 mlxlogscore=999 suspectscore=0 phishscore=0 adultscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000 definitions=main-2309250185 X-Proofpoint-ORIG-GUID: 6RnVGTnEGP7DRF1qI71bwVf5vsu-1iSt X-Proofpoint-GUID: 6RnVGTnEGP7DRF1qI71bwVf5vsu-1iSt X-Rspam-User: X-Stat-Signature: 7pt8q77irotxhkact7y8yg84ibrfbw3a X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 79C14C0013 X-HE-Tag: 1695685781-757692 X-HE-Meta: U2FsdGVkX19iyhSLEZRWkBzcJ7jG3pbnB/RTxuSHqVP38KZGUnHJMMVdAYmfgUxNjEGwCrVlEX0v5j6h5ZXzYvEUQkgq9g775xwVvJ55cowErn2HvyEUFlr768BYCt0VUhkUNE2KmIueyF9JghdEN9XFnpn8vjMjsWiQP9XrBDT6YRBs6IoGjKylyUxGqe0Z6uMUWJG9DODurcmuFds85sk0aCcsXuHjJ/36pFcaT6HAdwgjC0PmiC65jeiRFKF1/RZ5RvMS+fF75Cv3ZnxanPpDlY1V2cxUgzbmz1HkseVwnLJMrWAKDiLLz6H87WE9cwHWJfOnBI2zKv1R8G99ru8xVFOfq6Q2n3E4gJzvD4zocoOTVJRQvadk2EaV4HS/Jfco6E6xreFITYoHpc+/T3Lv+iaJvOWE8RG+pt6UqUxBwbeZuDfWYVDOvZwFAmQhnJciXbJNcq1XeS7hO4D9eJRtR977bYqD3j50rMj2McWSw53Hkiob/EOIcF29Ab4/fvlVzYPEMyHmIsK9OOkHfWEa5vi5AveD/wqjskGQJMpuTgG80RvPQOXzGezfNJ6iLA/0ZKZfNaxuJBwqJuXxoVqgbwswbNg+1yoaYmUkX4l6K4k8aEntjmKbpZLNzMzeGS6H9W5BwSP4KCw1uYBK8YsWrfZguffDreVbMq5EkFJVAdeaeQ8eKBRo5aXWkWkn1KbdZNgTW6ltdIII93e62in3qZtpdl08cLXDwKvsK9aiKm26POr9BkZ/U2jzfi8wW6/9ZQ1qC9U1M67SnLrHtWVKBF7r65lRuSX5W9jHbTmOXdL6ejnn89segRyCEQFk0vO4WPTj0uoxpjju9J98Y0AhKWbXJK2gm7rLpMAtk4JUax02NRAsSwBTgWqMMbUOwnEkDawVGNC+sK6GnZ6GE+iwuP5mcxVub0ttmPo9Z+ddAx2Q7QxrhOT/y/sDxmf+QI40HFeKIYWbA1DlgV0 1q6uc4H2 NIN8krGElBpVUYV0mKEQ1Vp/XFK9qGtlmblezKfnxtxYhaV7EwIcjD0dwX73x/4jykyTizE9/vGEajQrAIm537xhLwSFA9YEIPfe8Fr/9SzmKdscc9mTJPduQvBbz1TXEpcPTBKKmUfvJmYpekTjbbmqKxfob6PiG1mF4Im/Z9JSZMLg9pXnj//h7tLdjDwbnn8Vvsw6MaMo1YYWZDOKCL2wM9wbvyjiZL0ENAhPeFZ/+OYDewqPmzZ2qO0WLvxi3MdQz+ogsBBfWdqBApEp0t1WhCTq8MYGEKYGjk9gRvEVO9sYwmHm7h3sWxFMh2P1Q/TkhiQV+QdW6puzhQ1yVZy6uqGgJaz76CKdDHv1UF1QuP6mQ4BcMXt/ojtkMor9HD+9GzA2zfZHpO8sRsD91OiZ01DFUgCGMlMycpLRq8ngNgwtz8zu8g2KE9xrVwod6xzqBAmguYZSb7OY/CTYe2tidB5BsTkyxNe6vksc2Jk/Ia8CFkODvkGwyp0ldE+k+r+5dpLATJkWhjoTqMxoFYkXjrrsDlG9vvUalOZuNbZ3mzvYrp1/I8ulreA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Update the internal hugetlb restore vmemmap code path such that TLB flushing can be batched. Use the existing mechanism of passing the VMEMMAP_REMAP_NO_TLB_FLUSH flag to indicate flushing should not be performed for individual pages. The routine hugetlb_vmemmap_restore_folios is the only user of this new mechanism, and it will perform a global flush after all vmemmap is restored. Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song --- mm/hugetlb_vmemmap.c | 39 ++++++++++++++++++++++++--------------- 1 file changed, 24 insertions(+), 15 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 9df350372046..d2999c303031 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -461,18 +461,19 @@ static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, * @end: end address of the vmemmap virtual address range that we want to * remap. * @reuse: reuse address. + * @flags: modifications to vmemmap_remap_walk flags * * Return: %0 on success, negative error code otherwise. */ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, - unsigned long reuse) + unsigned long reuse, unsigned long flags) { LIST_HEAD(vmemmap_pages); struct vmemmap_remap_walk walk = { .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, - .flags = 0, + .flags = flags, }; /* See the comment in the vmemmap_remap_free(). */ @@ -494,17 +495,7 @@ EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0); -/** - * hugetlb_vmemmap_restore - restore previously optimized (by - * hugetlb_vmemmap_optimize()) vmemmap pages which - * will be reallocated and remapped. - * @h: struct hstate. - * @head: the head page whose vmemmap pages will be restored. - * - * Return: %0 if @head's vmemmap pages have been reallocated and remapped, - * negative error code otherwise. - */ -int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) +static int __hugetlb_vmemmap_restore(const struct hstate *h, struct page *head, unsigned long flags) { int ret; unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; @@ -525,7 +516,7 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) * When a HugeTLB page is freed to the buddy allocator, previously * discarded vmemmap pages must be allocated and remapping. */ - ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse); + ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse, flags); if (!ret) { ClearHPageVmemmapOptimized(head); static_branch_dec(&hugetlb_optimize_vmemmap_key); @@ -534,6 +525,21 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) return ret; } +/** + * hugetlb_vmemmap_restore - restore previously optimized (by + * hugetlb_vmemmap_optimize()) vmemmap pages which + * will be reallocated and remapped. + * @h: struct hstate. + * @head: the head page whose vmemmap pages will be restored. + * + * Return: %0 if @head's vmemmap pages have been reallocated and remapped, + * negative error code otherwise. + */ +int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) +{ + return __hugetlb_vmemmap_restore(h, head, 0); +} + /** * hugetlb_vmemmap_restore_folios - restore vmemmap for every folio on the list. * @h: hstate. @@ -557,7 +563,8 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h, list_for_each_entry_safe(folio, t_folio, folio_list, lru) { if (folio_test_hugetlb_vmemmap_optimized(folio)) { - ret = hugetlb_vmemmap_restore(h, &folio->page); + ret = __hugetlb_vmemmap_restore(h, &folio->page, + VMEMMAP_REMAP_NO_TLB_FLUSH); if (ret) break; restored++; @@ -567,6 +574,8 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h, list_move(&folio->lru, non_hvo_folios); } + if (restored) + flush_tlb_all(); if (!ret) ret = restored; return ret;