From patchwork Tue Sep 5 21:44:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13375123 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E25E1CA1019 for ; Tue, 5 Sep 2023 21:52:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 64E2190000A; Tue, 5 Sep 2023 17:52:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5FEE68E0014; Tue, 5 Sep 2023 17:52:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 451F890000A; Tue, 5 Sep 2023 17:52:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 33DFB8E0014 for ; Tue, 5 Sep 2023 17:52:42 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0177F80B9B for ; Tue, 5 Sep 2023 21:52:41 +0000 (UTC) X-FDA: 81203893764.30.E47FD56 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf05.hostedemail.com (Postfix) with ESMTP id B0D8C100017 for ; Tue, 5 Sep 2023 21:52:38 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=kmyImPsw; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=Eb+qcFrM; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf05.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693950758; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FIBN6YhPsvUwRzPH0L9SXh5tL8FY8pkar6teks6b5sk=; b=0E/cYcq4E0KbVMWXZWPtFyWPmoswhF0XbNrvX2eMyoSvKr/agh+ph1x+7EyD2CqC2RdzMD dApOy/nhlAg3Nmn3fVrECPHYFqVIZ6HqDQwmUCM9SREVNYB4VVwMqh36ZnhORldepNRZe7 B3ZuuU1hFd15cTHNRvGaIHTw4Hc+VxQ= ARC-Authentication-Results: i=2; imf05.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=kmyImPsw; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=Eb+qcFrM; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf05.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1693950758; a=rsa-sha256; cv=pass; b=FEyJN9GmszdK2nzQSXD6X7HpIyX6w4Wk5bFwCh+7A5GHL/Qyx8QVVBcmzkulnbcRKsICsJ qDZW5IVdMAD7+3cKhjmRQh/+a+lxf0fBhMUnVGSpSf3n4Ao5ZOvwmyxUiOr6bAH65ePxp0 zd/nOmjN/nGKu5B687R2GstVzihCCNE= Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 385L9wGu006968; Tue, 5 Sep 2023 21:52:11 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=FIBN6YhPsvUwRzPH0L9SXh5tL8FY8pkar6teks6b5sk=; b=kmyImPswIIXavrg9A7ckMlZ4unYskU8NbtNdPSV4R2GQMy/6v4VZLtboVx8yseJCuWsz lihOxGy609ciu4KfFEVEFNniIyf31L0N6VwJVCxcjtpyBe5PGRbtXmv84AaXgbdW3WSz S9tRk2E1NNGNjUqGP/0HFJbRndrfX8yI0HrrH3pdOtfjIeIZNJ8TD75qu9g9o8BKU9Ho AT8YuZMnjKar8wBEHCqlGQ6NcbRr51Vo7pqbUEY0V2ugK4VSkEWCiX+Fv+78jhq/d8ef zqc5DvyrTj1cvTsZ9CR7V/qZj7DbgLn3ORwCcjr6NwNSIddVoY6I7o+rYkKWfYb6IIY6 fQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sxc0g83vr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:52:11 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 385KA9Hf037099; Tue, 5 Sep 2023 21:44:22 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2042.outbound.protection.outlook.com [104.47.66.42]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3suugbkmr4-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:44:22 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ST5G7kN1J1M43yFRzSZhnxF0odlMqK5i+ADW9AfnLUkLW/bK1vrkjpzyfV+FcFOBMhsimRABsJZuZsyKduLhOIJWWTOl+39d+BZFQ808XP5vmo06Jrc/By6/8swWx8ribFk9BsF11ITdn0Pcqu4YWY4hIJA5WDqKW+M/SAwYwsLcfCKEwyLDrLIRr5v6j7t7HtqIPLZPD2vgX+C6FYgNrXB5vb8ubPdChKHKqsWCWUarAfIqqdRuVx3wKppWUvAw3OwP5KJbRmEptYmGYh/kRR0+OamsK4mcRTkFrDtDc22xFxum90UiMKQC9clLgySDFPyIO8kObPEYtlJXJVM38w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=FIBN6YhPsvUwRzPH0L9SXh5tL8FY8pkar6teks6b5sk=; b=bPghD+J2LtFKubKXF94BYeMy5Lwlqzo55ifZ+lhFiE2L9iwGE7B2SHJD6x21//CWdq1KRPa4mrwn+Fiib0S4vaaN55ftz2mwSYUhP2QYkyojPdtorq3No8hbq1n/o5gTlFg234ybdBk3UN0CA/Q4gTcFAchOJtPOV/i/4/ouS4uipSkcDZqr9VfU2h2etHlGPRSRp/4nHeYLjuVfFMsNpxj31VtpjqkZbBHRAgrEVdHsBfn92pLfm5ZgO28Bk8/c1Ku9pvjHVG8//8oCLfpUO5QLuNvg5zYDlNQMZXoibQ3DqSgzteeCthD47hcLBwGHZNiqjvB+IZKtZo5QujupQw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FIBN6YhPsvUwRzPH0L9SXh5tL8FY8pkar6teks6b5sk=; b=Eb+qcFrMUrFUtj5iJJLd9/CVb4+cmYL1izF4IDuB/JEn2LpQJjCIhpfdFokBnCTXQ8FNmdSkzfQv5De39wbT4wQSqxxW1BDTl3GYqjL1Dqum8TiKXFtUasX0i+7fp2YNGCxnqT1u3SanqioaiWYz9i5El8ScYEBptn4KqPZug1I= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB6395.namprd10.prod.outlook.com (2603:10b6:303:1ea::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.33; Tue, 5 Sep 2023 21:44:20 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715%3]) with mapi id 15.20.6745.030; Tue, 5 Sep 2023 21:44:20 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v2 01/11] hugetlb: set hugetlb page flag before optimizing vmemmap Date: Tue, 5 Sep 2023 14:44:00 -0700 Message-ID: <20230905214412.89152-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230905214412.89152-1-mike.kravetz@oracle.com> References: <20230905214412.89152-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW3PR05CA0024.namprd05.prod.outlook.com (2603:10b6:303:2b::29) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB6395:EE_ X-MS-Office365-Filtering-Correlation-Id: b2e4eb51-c3b0-4302-8f57-08dbae594302 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: aQqduTujRkMeFk15PIjXX7+QdwifXw8/Ue/iYvtas74zlZVmxkZjkiMa57oYQa7eOfvjgs6U9aWweUwqwZ+nADvHDa5iDHRXHnPRa1JKutLqB4CbVnTbfKVEOOsvE0bvVC534+5Lp83zvAzGl06pMJuNPk5xtocPVebGokzsznEWAYJR3yap8Unn+nyCGtJV03cGy9lVS59rABYe4jvrpe7EHcnlGd+RdnZ6dwQeF943ljubDVGPwQVm8nWQZC9EQboEIj5aSlCqkE/kleuVYme3rrUZc9lCskSPqTbEJ+1NFoHG58mzzh2KxSlULSfj7hOUl1szdE1nBF1HSO4P3Y+a7bNNcUsGbqPmKn3A480lP5cAa/85SdnsW2QHOacvifzuhhO5lUWVwDXK0kw/zDhYRwGWO2pAC95xg+CPbGBe8neUDwdrGBDiR3AOY0qH5iIEKaP53cE4k2tJ9j9YJHwSvECajLz2dNaLZHiv9I0VyvNTbGewoEByZDABayXwHpaghZbebwRho6r9PTWbg8wrEsWUjuRGrm87UcwJC7XgngWAEXumhgwWvNEAw69N X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(376002)(366004)(39860400002)(396003)(136003)(186009)(1800799009)(451199024)(6666004)(6486002)(6512007)(6506007)(478600001)(38100700002)(2906002)(7416002)(107886003)(1076003)(2616005)(26005)(83380400001)(86362001)(36756003)(66946007)(5660300002)(54906003)(66476007)(66556008)(316002)(44832011)(8936002)(8676002)(4326008)(41300700001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 7X7LLAtLrXRTHmKLbKC7/EsYnUtPO01VfWjWFFjD0r63JETud0YLz6B08ZRjDTc9dy7v4fgDXNjYEYvxjVHOLuy3TsXZ09AR+318PDlqzJKvkBte5xMXVKo+huawRFUn//VZKz14SlQnkECUavgrxNeoKN1wG1SXGysLBo174j5TzGijDk0KEsq444xbui/0wPW2qNpKzSKvMKHF+SGeqSeKFpaNC0wdS7A82a2PPTDO4rdsw8qB1zGNAXwN5TmxwOWUJ8a02VADxtBAkt7tFo1R1g9LUR/pI6TdD8+7eP/Hc+pT2cGOkKWWn0LrpObGdMOEjoKZAWRP/qdscGsp1RX2pJ+f6EB2i1EUIIMegzhclERsAOUtIJAuVA/lARDQ8UIKJaVKUO7Tp9037iW6NJgm+9YBhegiJXrcEAN1Rln4YJLNCqpLkMNo+cxwQNU8Hi5nQfv605nBV36mHb3w0pMXeDMyE7dQyYIj8tWG9UBA9U2LVJBA9JMexhP43ybBOwKlkgnjA+xwz3Vnv9xDB3Qe1xAitvGBPDgwCtRdvhMzrxtN+W+Nkmsr8kbVed5pc/tjTOMOKJ8q2VuSki5VtxstPyGJO0d+oziCcklXe1RIDKqr8Mp78hQiKm0p7eCMyznjeAS5+fjemTwDH9JW4VeXF9eeKWT/AEQLvILhjqzhywmuqsYVGPlcwCicOhgyVOcKztz2KPOJSpJhMI0sSFw8dcOR+R5FywAKQ67O8IcgtQriHTVle8ijlciR5wxeZ3+NDmURnToDJfy/aK/xx6q9SGtyPlrgkyOM5vn+o5RY0FqcbWRdc8Nt8YklT5HraOnHN63SbEVygvE4j0rXVMs1KLgs/H+n+R9ps4NeBcfT+gtqp2Wn/KBTzlMDtrVN3e4uwje/TsIxG3NUDti3BFw/+2Mx4uByWOtftFnOpfmZ/NrxjK7gACt6THmEtsNNP23sbRohQLSTfqCChH7aW1G/Ubi8IulP5/es5RkamuDNWz1UdOjcbnt9ay7x+xCbfE+l8odfoUWRfhLXLdlRhiaD58qt+4PBGRm8V5+ggQRrNQIrr7OVQRjw5SMLwFpS5YZLCN7gqpCbpmQgwU7zOXj/t1VKnQJ6kJX2nkvuVQhuemJsxxIeiqLeWN9NgpfaY7pF7O20Cvufb/UTl0mGZbiWBRT0vTfA1Sb1CsaRLi2EOSjpi98vBJbBZKzPL4c96Fudv65jHO8jpfgd0U963ZbY6wgERty5klODBH5blGVcaJCtLbSQznqmc9jfOXvmDOKMO8wEOEi2C6oh9vKoW8NerNIj9v2M385RXPKvUHTguajJAz8ebcz5crVYyp2noHl/81jPvp/3iOiHJRcA1Q4z/SiYmPsjqKy/z/X16zeJqKZ4Lvfj2HzYUxCTZN+E37wLR0CJFu3ldipXJXZ9l8ID2/c8o+awYWbX9xobAC5eeWPJJiT6Fm9JFTwVPDjbTwNTZXJVOfaXTNfkHdIJJbKrnktRcwmcEPKImnDU0g5d4sWci8v+X+6r4Y/p2IrG97GzLjnOa5M+xq6FiKRN+qNL+ntAej4/HTrgu2aUh4GEi5QWcI5qSzvluAH0jaYYALe9uFHjalrM8xRrKwczmQ== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: Jq0d5iaXsCM1VUTzfpx4qKr0pbEFY2IGb8ndvC1T1vdddOJGdGBDmAI3fv01zWg8MMZCJBlLWcX6v+AFWhId3AGfm5j/X9Q7Ro8yg7c8lKt1iu75D2Ve9PXYHyxJtdwIJ5npaRELFNLpw80L7JB9iq8+gXA/eo++71oU/s01scKyIroy50JxEDgby+hiIN8mcrakyK+JLmznzb1nglkhYall4o1Nerr/oaRV5hE9Te55mRbwQKY9EX6Uj+XM577iS9uQj73D9t1Rfrki5tJk135F16iTrMRWSvaDlh+ix6rHRjzkKdXfiZ2tTzRwnWO6ED+fFcaWuna3eb62dFPFtlQaoibUzFgzZ7Z2ea4v6RyD9XwPmgX5rzY06UFr/GwA6Bw6N1lNZojBMMOSsvbIfn61f/RWQZEcexY15gbZc1qH4uEFeRVUj9zpDpcAn44oFve+/YYb6cUnrtW/+1IyiRs9RwPgBcL/aePq19/UN1q0IemoiCNaZRuROr6PR8TWUEhWkfvn4Pc1FtbTtxzVsM+HEJGhC8OWYqg+xHLZm8VKRkC5ILF9MMahW2r5cXlTYEmc+3ZgHzIwzJ7rjF6/5GtKk+wY4dDjdI97IASp/Af8Hl6FlEoS6zvFCBJQ/JLGn1oNX8fdzKRvQwi2wIQ2T5G12+IETjR0/R9DddDztiiw1IywW9/0ks9Ux+hcPIo3sVo3OU3Ltyj/IywPJai8PddyBg+WqR0yYOFv2qJjNeCijunpi/ccOaA/0YCFMDL7XNpuYNlJsc0Y22DOKE2LnNxuVziGG4pg4f1ah8c1gEHZOcBzPiBLFw8xWOFIEsg6p7NZk+ARZt01r5B/sveGjhk04sTwg2pdNOMofLxPqIcdQop1hpcVfxJ0Mwc7ESWmXhVP9D6II5GRjXNEz39+tIWJfhPw0WDocYQ6X757ck0cozn+SEYqKNEYa9geWhSQR9w9X7OZpPJXSfr84J/HKbGjauG2o5sulTd3KFlkHMEqzDoA6ZolGuutipOwbNJUd1I1X8e4A0jGiXd9W11HRDq6Yhtli7qI0sbLiXstKOgaZV6Bf6bWX4oNjJKlMVrglvvXOEbS9SfJw2PYDB1pidKd0KtgSOxhO8EPWfM8bMyj520aZwudGUtZb22zqZpw X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: b2e4eb51-c3b0-4302-8f57-08dbae594302 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Sep 2023 21:44:20.3786 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: hxjtT/BJ+eb27xxv5YEprnQzSWgCQO1mNymvpXcUawwyDhSqpsshRNnQkmlOM/WhAaB/nqERWcAZJqcAGRAmYQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB6395 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-05_13,2023-09-05_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 adultscore=0 mlxlogscore=999 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309050189 X-Proofpoint-GUID: jMekqK1luzh1A4vhH0HMzfgFUmQxBFYn X-Proofpoint-ORIG-GUID: jMekqK1luzh1A4vhH0HMzfgFUmQxBFYn X-Rspam-User: X-Stat-Signature: nmojki7x5bdddhdo31kfd9h46qeoibdd X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: B0D8C100017 X-HE-Tag: 1693950758-43528 X-HE-Meta: U2FsdGVkX1/udEa5q14KTb3Snmqa5mSDI+NHouJ8XRGduQfoFs3/zacTZnKlfV0LZI8kJQ9LFGkeKXFndL/DWPWHV4MrjtIs/lQs5u7S61l2sZzsfGGyqBCjy9nO2ee2wIK3pYQEsLNkEk0pm6CI01K3eVXXpQ7dmusFcgOOWhZDADrtAVXIW2QukP3GmWIEoQgKYP34b+yGeoklbLV4jkwQYhAciyT7hliePUzyGWhGZ9Zo9zmhwrh2c+IoqwL8YH+3aBdCCAhLw24q3jTsyHB16M6iwHs3DqpDcz/7JmhewccaXjKYTNjMopcM/bn0YnG5gsu2tPQDslkTfub6eh7ee+q+bM0YXciMDkBLpvC3vlta1M6tXrv//Ysm414WI1pPVhAYiU2JOD+BPk9YTCnqVkrbupy/3vPO8nS2sCcp3N2OnfJvJejEiVoH4tGWW8Cce9Qw3i6Z9srWambGte2kZE53uuuLGANcSvlOyFTpna/vbbu5M+p00mjbya8CuhTxmM505lmlNdj52TPvpTH7WwsA7o4AxhFhze4rr3yVOo+dkCHDpzHIf1+naLwvSamb03bFplAtt22C5Iau98jPb4D0lIfCo4brmOrCeaixXXxYkLEAktsD6+dvHTPAlDmYG/yIQuVegj0fdoWqm9XSCPGyx6/1MFeWGyl+zkLVNRBPyrVADkflUTLsKj0qMQKVDvw7kOx1wLuDDtfAhgTk3zjSvpIRpOacH+eD+SUyTg33Ql7qqKZsDuMie3Y/BydKeulzbBvyMMSjZop4DBUYwy47xxtpxoTbjHxkL+Mp+b+VZ9H7yhHrl2q3t+VqGsu7Jd2spfZ3BfIRbInt6FEd9MxyZyDzx0MyuEWLW8Ur3zNsB3GVAy8uixWdby4QZWoK92d2AG0+4knHOPzAhGjwa6MVExo/v2D3vNbwN7M3eBD8SJsjPYy+EV9xqjcWj7KkF6R3R8N0gqzXiU3 +LEm/pbs YqiFblRPKXI6UjSw9QY+juOFRokd+GUXZI648CJ47TDzdSmuOxnWrqjN/NLWBHlZ5H4HRFyT1zH1+95U+EhmN9NzpoLteF5UAASIgM5u/p+6tmoMbFTRQm4ZkqJ7gs2B9MFJT0SSyQL2biQJKd9+xKuoWup3L1mLcMrn8kHNjSgXPA19ShIpoYpgVoaqtiyLRahu+xUEqroF5CjwCDkqn5g361QKP3MGdkF/h6LoFIbvU58Y1F9BR9g31pARpB0kK/bkafXbxBuR2N7dC3WFoBUA1R0oMAmW3SdhpSioZWh7H1p+iKL2Wzz567Lvs67g/Y2B4fqgHVU2FyjQjr/S0GWV2rpODxJ+iB7Fhua1kfLW1RDbx51Fc4ziKqNAiDg6HWWFXSeeFmL8jFjIFeJvz5JouX7fl2qfoYZBkKWxTSYrxjQCY2+/MH5rmP8ZULWSx63CfstJKmnGOT08pbVlj32l1qZmdyczY8Vw0mSE3Zx6jWq7A/9bKCp+NrWodXZAVlt3X X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, vmemmap optimization of hugetlb pages is performed before the hugetlb flag (previously hugetlb destructor) is set identifying it as a hugetlb folio. This means there is a window of time where an ordinary folio does not have all associated vmemmap present. The core mm only expects vmemmap to be potentially optimized for hugetlb and device dax. This can cause problems in code such as memory error handling that may want to write to tail struct pages. There is only one call to perform hugetlb vmemmap optimization today. To fix this issue, simply set the hugetlb flag before that call. There was a similar issue in the free hugetlb path that was previously addressed. The two routines that optimize or restore hugetlb vmemmap should only be passed hugetlb folios/pages. To catch any callers not following this rule, add VM_WARN_ON calls to the routines. In the hugetlb free code paths, some calls could be made to restore vmemmap after clearing the hugetlb flag. This was 'safe' as in these cases vmemmap was already present and the call was a NOOP. However, for consistency these calls where eliminated so that we can add the VM_WARN_ON checks. Fixes: f41f2ed43ca5 ("mm: hugetlb: free the vmemmap pages associated with each HugeTLB page") Signed-off-by: Mike Kravetz Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 31 ++++++++++++++++++++++--------- mm/hugetlb_vmemmap.c | 3 +++ 2 files changed, 25 insertions(+), 9 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index ba6d39b71cb1..c32ca241df4b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1720,7 +1720,12 @@ static void __update_and_free_hugetlb_folio(struct hstate *h, if (folio_test_hugetlb_raw_hwp_unreliable(folio)) return; - if (hugetlb_vmemmap_restore(h, &folio->page)) { + /* + * If folio is not vmemmap optimized (!clear_dtor), then the folio + * is no longer identified as a hugetlb page. hugetlb_vmemmap_restore + * can only be passed hugetlb pages and will BUG otherwise. + */ + if (clear_dtor && hugetlb_vmemmap_restore(h, &folio->page)) { spin_lock_irq(&hugetlb_lock); /* * If we cannot allocate vmemmap pages, just refuse to free the @@ -1930,9 +1935,9 @@ static void __prep_account_new_huge_page(struct hstate *h, int nid) static void __prep_new_hugetlb_folio(struct hstate *h, struct folio *folio) { + folio_set_hugetlb(folio); hugetlb_vmemmap_optimize(h, &folio->page); INIT_LIST_HEAD(&folio->lru); - folio_set_hugetlb(folio); hugetlb_set_folio_subpool(folio, NULL); set_hugetlb_cgroup(folio, NULL); set_hugetlb_cgroup_rsvd(folio, NULL); @@ -3580,13 +3585,21 @@ static int demote_free_hugetlb_folio(struct hstate *h, struct folio *folio) remove_hugetlb_folio_for_demote(h, folio, false); spin_unlock_irq(&hugetlb_lock); - rc = hugetlb_vmemmap_restore(h, &folio->page); - if (rc) { - /* Allocation of vmemmmap failed, we can not demote folio */ - spin_lock_irq(&hugetlb_lock); - folio_ref_unfreeze(folio, 1); - add_hugetlb_folio(h, folio, false); - return rc; + /* + * If vmemmap already existed for folio, the remove routine above would + * have cleared the hugetlb folio flag. Hence the folio is technically + * no longer a hugetlb folio. hugetlb_vmemmap_restore can only be + * passed hugetlb folios and will BUG otherwise. + */ + if (folio_test_hugetlb(folio)) { + rc = hugetlb_vmemmap_restore(h, &folio->page); + if (rc) { + /* Allocation of vmemmmap failed, we can not demote folio */ + spin_lock_irq(&hugetlb_lock); + folio_ref_unfreeze(folio, 1); + add_hugetlb_folio(h, folio, false); + return rc; + } } /* diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 4b9734777f69..aeb7dd889eee 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include "hugetlb_vmemmap.h" @@ -456,6 +457,7 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; unsigned long vmemmap_reuse; + VM_WARN_ON_ONCE(!PageHuge(head)); if (!HPageVmemmapOptimized(head)) return 0; @@ -550,6 +552,7 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; unsigned long vmemmap_reuse; + VM_WARN_ON_ONCE(!PageHuge(head)); if (!vmemmap_should_optimize(h, head)) return; From patchwork Tue Sep 5 21:44:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13375110 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A348CA1016 for ; Tue, 5 Sep 2023 21:45:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BC699900002; Tue, 5 Sep 2023 17:45:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B766E8E000B; Tue, 5 Sep 2023 17:45:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C8C4900002; Tue, 5 Sep 2023 17:45:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8DD998E000B for ; Tue, 5 Sep 2023 17:45:16 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5FDC780B6C for ; Tue, 5 Sep 2023 21:45:16 +0000 (UTC) X-FDA: 81203875032.22.E58DEE7 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf25.hostedemail.com (Postfix) with ESMTP id 06F3BA0028 for ; Tue, 5 Sep 2023 21:45:12 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=RkolEh49; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=VVBk3cnE; spf=pass (imf25.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1693950313; a=rsa-sha256; cv=pass; b=uLN4EzUaRC5jNEvXMZ6uC66geeDQvosmLIaGDvsLNf0Hw8ABWviB+wbU70/dTSigld2jrZ iadGSo/qKRYy9+lr920iblqP/KzIro8tzmK89hFl+VBvtUTOrXFJJHafgtN2XRHBomV11Y fKBkN6dv7Pau5hrqKKhq8pAthlTnpo8= ARC-Authentication-Results: i=2; imf25.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=RkolEh49; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=VVBk3cnE; spf=pass (imf25.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693950313; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yVlaMs/R0uxiUc/o8fO35CIz9nL9E3sdhKpFc0sSIK8=; b=7DrgLlRG1985k97PK33WZm9eUTYQrzUd6SozcGEHKLsC0CN3WE3pkH/exBSa+zSkOkJhuX KxWwljJPwhKaj4VFD6gKYrLwOsu3sF/+hcEmjClhZbmbn0CNHJ4URVVXdKZVGLAS1ZNsJW pvH67MS6yQ7Z5N2h0H/4adI/mZO1vkI= Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 385Le63d021124; Tue, 5 Sep 2023 21:44:37 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=yVlaMs/R0uxiUc/o8fO35CIz9nL9E3sdhKpFc0sSIK8=; b=RkolEh496gTipOdun4pnsqw1P65/IK5KOUq6kH8XpQXuTE3OzDrZ1BGRDK8dz3t6wpj9 I2nZGSwoEJhz/iMjOQixzJ/VEpCpGfGSVGebpYAYiQcBfEknRgx45gZfhzvO8tQ15hQv vOY55eUr9sTzukPIh8HGCE4j6d4sfHU+LjR48IDrJ7bqKStI5aVyps1LfYNzHKappmqP Ar6PBR+A79is5PQyRdLXmDCKp2D9CH4WwjrmHGXz5BmoNo/+1P+clX4FKEVgpHxJm66E j5xH2N2EQQSbB4auCZ59wzOSGtww/NV4bcrxWMJBuMD+nUrbLW8Xnnn1GpHiaZ/cOIQA XQ== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sxcem8073-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:44:36 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 385KQKu5029067; Tue, 5 Sep 2023 21:44:27 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2042.outbound.protection.outlook.com [104.47.66.42]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3suug5b7hf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:44:27 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=k9qzwMH43TCzdpxVvNzxpM5oTBSW6nuncQW9iQ2XzpaNQpp9DjoxQb9X99D6j/YejpD2AQPZiA6YHQ7l8Y/IWXrAKm5xL650XUbAimZ8GY0+ACHq5RSWtJH9MocQs1WOLxtYZrXHRlnPmTzH7MAO9XBFGh8nZ/dJFHBiF/oKnZV/Tv1D9DvkKqKM86Dtd8htN3jf18lZDwvla4iGPoFDnlXmOu1ukIrr/JI803r/Ys2S9BmA4LF5b+5M8c1ejupfull7zZkZl91QPk7hv1kUEZNn0DOcjXQP5BSEhBYqBiuvL5e5rn/Pc+JgVVeEPJ/HxVQkWwRCxs1xkBLElMYObw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=yVlaMs/R0uxiUc/o8fO35CIz9nL9E3sdhKpFc0sSIK8=; b=JlXIccaC/nI6POnztrhsR/36R3Q268ENb7rTQp+OcUoo88EX2esyWLBYJV252lQYXfoNZxbdyoEEDx+BZ0EsH8n0NML1b6w4EmJvhU5mDn4x45mJw6Dgy1sT4rno6ofcUjAk+AX4ygC4HfUEXE7kxU868GjcQVwFJuBrgHCiUNIk+U18I81ea7wZFqQ8TY+ZIOpz06SD34OHsAiBMhI2sA+RE+x5bq50BRIyG+iB1eOO/2ZilhOteY3kZtSQtGsLt9kN4xHB6I1bt1cYKr9dCtamAEzc1QEjFuwbysA39gr3DtWufpi9pL9AsO7sKUlkeLeq1rwHsmrHQpDEdi+WyQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yVlaMs/R0uxiUc/o8fO35CIz9nL9E3sdhKpFc0sSIK8=; b=VVBk3cnEF5aT3xlkJ/Dsy3SjMQhxjp9QCvgf5hXoQDllCCcAwvIxqXlHTwTdHw+ja4vYy5NOdKnZust3hP/i5XeXb+HBi79pyUMT6GFGv1wGraXXXJiAIX3XMie3CVPgPgLdTjX09yO7iC88G7jw7GEU3rWA44Yn+5/l2wHiOdk= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB6395.namprd10.prod.outlook.com (2603:10b6:303:1ea::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.33; Tue, 5 Sep 2023 21:44:25 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715%3]) with mapi id 15.20.6745.030; Tue, 5 Sep 2023 21:44:25 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz , Sidhartha Kumar Subject: [PATCH v2 02/11] hugetlb: Use a folio in free_hpage_workfn() Date: Tue, 5 Sep 2023 14:44:01 -0700 Message-ID: <20230905214412.89152-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230905214412.89152-1-mike.kravetz@oracle.com> References: <20230905214412.89152-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4P222CA0025.NAMP222.PROD.OUTLOOK.COM (2603:10b6:303:114::30) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB6395:EE_ X-MS-Office365-Filtering-Correlation-Id: cfe4361c-fa70-4091-9ad9-08dbae5945d9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: AB2w0estCYVBlRwZOtCaqFE/wR8a9kqMwre5CEmjluwBvXK+TbqrWRumdO8AQEMma8Bvp1o85BIcA2RRpUZ+nQDY5ab/xFU/6jeiTHkBKengO8dyT3REOwLbBiN9AaubgBKUfDcPfL7o7ZDY1/L3PwPimtupJLA+s5iXtaitFmuhl7zqkZaDuJ1UttjPiyo8PBHIE2fSGriGZDlR/D8VihVuW1wRSt+Arud8Q9Ihl6cPmv5uoPNAsLXYJSuBm9ndx7/i48aT7TAytFKQPK4V0Bwd/Onth2uPzCqW0mgSZyTjhyIV8QHIuBZx/Ey+hRf23C5KfARAInBIp9xOLsnWtfLSCRthwELUaYzhXB5HGk2pwjE4kIMmJUd+NfiATUk1va6PQIH9p3yoWtxa3BtfvlNNvDPRXGnkvF6DRI/EmZzV9vSBKcLxhOqTq2mz+bKRuONfURu3yAETrVxR68KVsPWA1brm8zsXRN6uRw5mmlXWp5SJURBOrUzRrc0+05ps1FEGo9UF9kDFdHBs62Sn1x8pRs/93QUfXIPvYDjTADF/VSxSf0iYKediX5CBaa6p X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(376002)(366004)(39860400002)(396003)(136003)(186009)(1800799009)(451199024)(6666004)(6486002)(6512007)(6506007)(478600001)(38100700002)(2906002)(7416002)(107886003)(1076003)(2616005)(26005)(83380400001)(86362001)(36756003)(66946007)(5660300002)(54906003)(66476007)(66556008)(316002)(44832011)(8936002)(8676002)(4326008)(41300700001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: w7fTVelco3+AE82ISS7lyIm55XYbUTUKbsGLt7dWl8d1u8txtJ/fI0p2+2dHAjfVAQY7V1wjrUGhsoa1aVWYbPZ6bTYGxoTIQvFszRViG2QIExxmP/mNjq49DmP/hpdECZf5ARA+87TOPCPCwTA4g7FPZjFiPfQFucWp+IHwmfuvjk588ncEZJ0ft35w/OhPA/OqbylaUZxnC4AOzx0OajjUVkdoUus2FwtcxSCXzXttOzWT2A2610+oGyEtPk7WXJIROHbiEC8gIrHEfK6eX42M6Ml1AFCoCh81odxVbtnb4dRHbquwvW5B5gfq0Ba/Bb8vpRaa9TSSLj4kXS2l97PUQtwlQmuhvUOmaz817WS6nV2aSq0zqxXdCeL6uZCUGH/lp6mOMHD0L26MrN2O7VB5ydG3SypLxUKfFwPhar3b5KhFyqlBGWbmOtE636+JoS/wj0YIUPqR1eK8BAjvSf8Qdcvo9T8j5GYLcI/v/5qfnGk5P+b3otV1ZW/5MMtGatBtUIaWzmQEBJxCWLkDNBGmPPn4R8wrxViA8ZB5dmL8j7vGsvh67ON2QnZ9o3Gm+Rzlc2LghCFdIiaXdnzc36a5LTyLdPglwW8rgr8/8o/ROl+Wv45X9tfiZjGlFr5MiK1uF6B17LRvCEnKqgasXC1ygxcN6kuia+8XPmfFmZbsMLSPxJXFn3FkB0AXQv3QAPGsB0MiUxtdrJ8cySOPN5kUnwQOAAFW9uRwM6U7apGyZc4sih8wXWLtEPt7qY3P7LN4NXJuRzkC08BrTKGl4SSV4KpitUuCBZULozaJvgNeZ/9ZC9x9gSd9X/YMO47NVom2wj1HxEU3C7euO0xq2PaUvQaSmwGIneGsBLy3r954glmYjr/Sh9AUoZB9f7Muc8yI+rQOxMw8Rw/+cte0QdqT7FJByuBLghU4K3lBIszUnMQ16X6YgVyiYU0pr7BEQqW613c++uSwJZBg2Ik4NT+2eU7ArYsefDBprkEY9AEHQvIJeCnkYTHe5Fn6VAsLC3yQKqZRW5yMmaA/Kl6nnl/74SVDFOFKDuOaEmyfwEPcMDTvf5GGRXx/ldIaHs96KW+RCeFrCNz+cXq4pK61rVcs1ZQEl7Tgf5TgzdMH/8J9X/aVyltUNHiW/TjlzvxJQN+VC0+txwuqRRO+g5vVe5zlBHhAPYeAi2+Uq/c7cfIQyM5k7B+7+09OfyXyrDajsH0jXc0RThYsRi7oVXs70E36iAH2bni1ufXSiwA2HC6Migcq5etIoRI+AEjD4hOqT+5lV1WSwDHsuAdKj/rL1wDhPCsItV4YIfCS1ClehsJs6Rl5CHo61epsa/DqgypTo0aAFKR8AJaMm5xU4rfMrmhByPE2DqKoh6HNxiVALI1DZm5ElsA1URYSFEf6jawm6U5of8BnWrIUswS8MzH1W9+AvBweWvchQNEDuwfOVeokCwq6gYlc1elP7Hg1Yy2s7/7c0Ag9PB9TrLOHnQQaypAbBEm4Fwdh34FWNmSx/AhQ9i3+lLSXZPkDNnBL1UIAo+ANCOXmwKvXLnRW+vNXGxosnAibVPcd6IIERgtAlthViTSVNr27EzZrEHoqEa2MpwTwxscVkqpHtYGp7I72BA== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 2ChgqEdGdpo2p6cFzd6zENYWseDfHHmb5NRzNLOAqQeLCIW+ERjMg6EI9nUqwybw6oEQHMrbeLLZj2n14G8Zjof7YEbdILvQEhUHILa+lvLxD7y/C8fj7oHGH1uCELjvTzpzBOuOHLuHoQNRsX4ZY7f7UAOYtn56VpTFOCbQFXLpZvXEZ6JOTmgQ/THjNZUc7Eh6acyohp1QiokBuCi3lTmxl2N6aOCKwm5bj5z2WQUx3+YbxpXcDyuV8uY0OYCqg5Rlc+5a70ZVS4i5em3G7iXigAK+BiXmsZAaesL1qP215Re11TC71ezZSDy2mwn/JMk9hpmOTuq7RoWE4fY5aaK1H4EsEeTMv+pqB7GYm5pUS1FynsMUDusUgdXn5p25zaWjN1T1XDL9k/KJg6HSorsTdiOBQkbT2/9qTX7EXqv3HDA7vO2hQ9RF/YZrzK83MHa7BhaxDF5AilABOwXSQdAM6Zh/gT1JC1wu2R9IAz+liRNWzyhcacut1agqpjZATMYKFEFa0BDnH46J8SS/n51wTNSK572o2+dhZXIpvONYtEIpJWaBc4t3n5UP80FIkvVd9oxhE+568WU3JyYHwUu0s8MsvkP9bMSXxk9ZAH83T+Eo+Y6kKKd0gEU41Artavu2ckronVLd6Ltnfs2FpRbaNlMuOIQMrzktP9zLiunye4UJIJ5pxrlYAUiO48ojaJbk2kjKXH8FNvX6XJOCH+mzVLKsha4m9X3giSid3OQ2qExaM80Xoql6wfZbCGGjQzExYz34rAm996qNY3sLmoTPmciZS3tH/aFde8a8mVDcMlvpp73pQqa1ka7TLr/M7B6oj/qvqepoa1ymjIF/xAuxV1ug1iUU+yn4n5nmkt1zOApdhn/EDN1Hg7CH5BqDVud46uxwqSbPAGFxEqTTtptlqB9/SYlOhJUQUm7WbCIWd+yH1Da1fcGGsDK+rpI6jWkLnzb8CSY4DmzQnwvWo6v0/azltAKeKt3Bl3GLiggx5583vZelh5i/r2PGtaEdRlVgg0NOQzdjM7JKmuSGgKqvlrCOQVmV7wtSsuDr+oVsG6aobtcIzKQvxjqUXDGTzHNxhf5wMg2yUKZ9Vfcmr3vO/JT4ytQ6TokV+AErGUZeVzg+jLoMgNlZ5X1iboVe X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: cfe4361c-fa70-4091-9ad9-08dbae5945d9 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Sep 2023 21:44:25.1895 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: K0RwKQA+y28CsnC4Y1ZFPxONYZcwP4nuebOgWhxQWbsSXJJzFEjUXVrLBkpmehp4hZ1mHl1iclPszYQhbPYZ7g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB6395 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-05_13,2023-09-05_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 suspectscore=0 bulkscore=0 mlxlogscore=999 malwarescore=0 phishscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309050189 X-Proofpoint-GUID: Qe1AYxgpq0jfqvl4WZgFn_p4iQrR9aZl X-Proofpoint-ORIG-GUID: Qe1AYxgpq0jfqvl4WZgFn_p4iQrR9aZl X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 06F3BA0028 X-Stat-Signature: fpgx5sutxzus6oztyz9utwq4yy6766rg X-Rspam-User: X-HE-Tag: 1693950312-13507 X-HE-Meta: U2FsdGVkX1+YxfVc7XBrHYCi0xzFh3LKeQC+aQNPzUGyUrov4xy6VP9SbMjY5kxt6JYPdzXlkwowt/rX7Udm1XInFvlLR+P1EeeMuAoXloc9Qgr0tFqJ541TFSdPJ9zphf7J97/m4G2akCUWdgC8DcVSD+4LepL/Fetw3I4AB7iy3rC8Sq4fqY4DKenXalbcB6yVotHnDzXh5aiKmqtpg5Fjm2+RO+YRLH2m38rMMyYUgB6rqEmtPJ3RxOLbuWy9IhoNSiV8ExND0SP539jq3Cw44QZ1MEFO7V2TnvqykAb6GqKmZN6xnaKZZ2Q+zin/zO/3yR5XMtOQ+hy5VslEFCS/a3fKI6W8PnIFufhVDnWtKqbol9wxD4aiil7TsK++QgFhwYO/wYJH6b4wwa/h33S93sjTeoMx88q4+MeMAPLHZ3ywYV4SA9/edO7wfmzIC6FRIxlhmfV83/8JNEGKXbrUHXoxfjyDp3w+fTclgIwc7GYm6W5iXfieU3yg6oIM7DNWMGs9QLPlrSY2EbkH9nJh9iUCmo8P9mFh6OkbvUwnLd+v6UdxzyGK0d+2uS94h7zTzdCBTkUqh2e3xQBdvMN/DHjfSjW9ns4c35C2lK40MA/o+uitgS0Mv3p+NV/slJOU/abp7lRbABTweHYCtbL4RJsmQLUgdt0tGlqfA76+f0QfXM5Y0FIejea+0XnsAnLWe4cs1PPRnwJGCNcEc/NXDJjssmYXfYPAaVrnCletWjHLaHVXDGjFut+dFcAEJUE0S4VypUo4Ql7UFtlBiyFAkquyhh3Mn1nQGFjETgLenbKAP+s3r458Ib6ZVU9JVRbWBi2A5nZjhEs1PhoOMFCEN405ihiUMp2oOOL6xbLn4SR39uVCEykXIqFx++RXKiNn5r2i/4tUigneysL9bu2AC01U0BVr0v7s1hSWB6rLKo4E0+Y5jXdnnRa/GjgMo1UaW55PtDmZo974xuY 1YbGhC9h Sw8edosn0lnUL5kfvduOADpcp2QkaajPHtbcrQ98I1c+4RWNWSGEYoSfFVBQJ0nEWewBH1kjXE5P32NN3EwgBYoFpLxj+PY82eoAVnYFs+0QB+6Y0RNIsWxTO1njQTzroHMi566/r7lvg+jrz3Z/R1PfdIEo6F3uq/36PETUC7PeOc7JTauAHsF8KAdIi06EALx8Tb+a4NuCOQZh+gNA9AFtQSbj55EpOD2pxnrsodyDkc2c/lgUWQKVaM0MHNOGm3m5zYPmZpvp+pSmTWk7DliRss8ptPac4oEmkA5zt/ZrOucA6aowd8mwul3X6WnWqoFsfaQ1K/WwJq0KfizhvClGbnoMf4YFTnuq5cB/cbkOFGc427Lf5IF7Mr8aC2UhU7ON2Fwwx/bcUvIPGv2kYTxN9QvkoGnXnrj18b0whPwlf9UGcgBQQ/Y9iSnpiE6Z9HgM/0RlIF7dizq9ULl0OaDncRJEe2CuN1m6SpiUA9FYTlRMEFJ01ed3P97pkv4cY6llNOqp3+hX4UGXpfPilOzok94MS3obv+ipCfkyb4bfJxYGokPmmqIvjxd+KXJyNHv5XgY/NaCut74WrB0yEji1fnMSblSfGVIXl4b+d6wI7F7M9vr12nYCb/BGZj0yPpNBYCFCZXCzyOkOz04j7BkApwfOAR9y4Ke3wGVVRz/obMRg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" update_and_free_hugetlb_folio puts the memory on hpage_freelist as a folio so we can take it off the list as a folio. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Mike Kravetz Reviewed-by: Muchun Song Cc: Sidhartha Kumar --- mm/hugetlb.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c32ca241df4b..a27fcff3350f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1787,22 +1787,22 @@ static void free_hpage_workfn(struct work_struct *work) node = llist_del_all(&hpage_freelist); while (node) { - struct page *page; + struct folio *folio; struct hstate *h; - page = container_of((struct address_space **)node, - struct page, mapping); + folio = container_of((struct address_space **)node, + struct folio, mapping); node = node->next; - page->mapping = NULL; + folio->mapping = NULL; /* * The VM_BUG_ON_FOLIO(!folio_test_hugetlb(folio), folio) in * folio_hstate() is going to trigger because a previous call to * remove_hugetlb_folio() will clear the hugetlb bit, so do * not use folio_hstate() directly. */ - h = size_to_hstate(page_size(page)); + h = size_to_hstate(folio_size(folio)); - __update_and_free_hugetlb_folio(h, page_folio(page)); + __update_and_free_hugetlb_folio(h, folio); cond_resched(); } From patchwork Tue Sep 5 21:44:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13375124 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF166CA1019 for ; Tue, 5 Sep 2023 21:52:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 738D090000B; Tue, 5 Sep 2023 17:52:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6E8508E0014; Tue, 5 Sep 2023 17:52:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5157490000B; Tue, 5 Sep 2023 17:52:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 3ECE08E0014 for ; Tue, 5 Sep 2023 17:52:47 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 147B3B3E58 for ; Tue, 5 Sep 2023 21:52:47 +0000 (UTC) X-FDA: 81203893974.09.E236052 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf26.hostedemail.com (Postfix) with ESMTP id 97B3A140021 for ; Tue, 5 Sep 2023 21:52:43 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=G2ASeubl; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=PURaWeCo; dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf26.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693950763; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=S0ppY54OPBFldkJJgVsH20gQesPTIK8uqEJvN7NBREw=; b=RZkgDAXmYfdpo7PBv9EPXwdby/Wcxf+HIKG9HOM0DeTIEcaw0hL90a+zzyJAn+CdzF0crL Cp48xuYvrJml58kAMOARZD9Q203HQFYXtCODbW41VCrG0xhgiRXvRyNRoOwaiOO72MTpQ4 Cp+oA4oIzJ9q2rqBe05W7G7/QunbrbM= ARC-Authentication-Results: i=2; imf26.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=G2ASeubl; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=PURaWeCo; dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf26.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1693950763; a=rsa-sha256; cv=pass; b=D5dqKYPyT8iHlv1sbpFZQsC8pof2/EAnlV2KF8IZQXPQFmZkjzrNC5XTFHeGEjuw7/z4Jz viq5yF/WN1O7UFlpinX8JQMIcjka7rROqF1wM88iJBi9TqQoq6PDVISQWz5POJDAIHxA1b m3f6B9PEEubSHwCWMFANrQWC5H4G02U= Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 385LB2Wf029608; Tue, 5 Sep 2023 21:52:16 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=S0ppY54OPBFldkJJgVsH20gQesPTIK8uqEJvN7NBREw=; b=G2ASeublQqKek6yxL+x9Cw+MclL9XHbORzM6PZJCrA4FQgQ3zFZ6cDa6Qb93Bst1ma2p Wd5H5CPrr5miEdR5SQ5wx8DNrO7Xe6vwcFeCegN79z7ztZtSRhDcbBf0dX566izHfxJz v0lFPfCml3nKo0KMCFRHvR9eFR2fMkka7pAvg67aveyygJOXRUkWYt1OzUAUW0iIw9H9 wVs8wAWCIlsi2ClUmKHpjo3kLpcnb0ZCulp2a55Z/kRgh+f31iU5l+7/t1hwHsAm5jz5 6M0PyZWzdnf4xDcDdRndWObfcM+5Fk/wdnmHNSlzIzN3y800+nL5MRlQrYQyRtMqAD5a WQ== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sxc0sr27r-12 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:52:15 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 385KKIJr007701; Tue, 5 Sep 2023 21:44:36 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2047.outbound.protection.outlook.com [104.47.66.47]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3suug5c6ur-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:44:36 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=V18DeOYJgXMNXYtek4h9RElCvDMa5B4rU7tjRxWTHLyxs54d6B7ihJE/DkuK/6AqmtE74BNJRHxCU8vd7hsgeSb1hGdlMF6a5q1RGxj7KGu9P58koRXC+KAsuaPfG1B8vX/wivygaibW0by98JwrqoSW6TdMjV2cZ6iO4o5MNGVg87ZYqnB9n0yQWtX3BhRkMi32x5SRMcARfCcXuj8VzKxLrBzpsaTkOpvUHmMcVVm9HofrFaGHKQLTsy+Vtkyw5gvA7SuOZq+CeY0JJ78QJV+K8VK6F+lOioupb8DwbqzRCQRpGkGIXMepYxvM1QpTtoRz9oGdMPaElMK8B9eaEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=S0ppY54OPBFldkJJgVsH20gQesPTIK8uqEJvN7NBREw=; b=ZXfbhtJ7K0fgEO/+JyIyxeVDIDGGO0RWbufd27h9IcrwoNRkrSKeCJkFdBNuw/79e74nlkMOLxf9zmYHep0nxZ0rx3j618AyO0kDxtr65XChue1qG308WhMT4JoNC8v41BB/B7BIn+ZhUKVbbqDpUNN0R1ZX/l0bakkP+HKMRKTpV4rLDLG9m6KAKjhYBNM7P2QQWP09yBwnJdU4DLJ7LWIVG5181rneKCT8HtUi+Ln5McOVOk4XALMWxePhctrr4NMkwwPJbEa53OAjop1D9Um3cV51OgtsdhAmQo7UHoaMaaGBEJKgyrp9tsAYFPmYhNiF2ajLro5YTP6FIlWZBg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=S0ppY54OPBFldkJJgVsH20gQesPTIK8uqEJvN7NBREw=; b=PURaWeCoNdzd/56iUbqbAbbmqMpxlEWgs29PSbjTD62PvYT458atVUCePuQK5jtI/jFSX2cjHMXEzG24TmKl7JMk0vpYw9FgRZLhG3MBqNl2Ch8b8ogd2aO8tEu3MfjeAwj6AHNh/qSo0WcdffYKPmWVA6V+bBcYJZb4tENdWq8= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB6395.namprd10.prod.outlook.com (2603:10b6:303:1ea::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.33; Tue, 5 Sep 2023 21:44:29 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715%3]) with mapi id 15.20.6745.030; Tue, 5 Sep 2023 21:44:28 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz , Sidhartha Kumar Subject: [PATCH v2 03/11] hugetlb: Remove a few calls to page_folio() Date: Tue, 5 Sep 2023 14:44:02 -0700 Message-ID: <20230905214412.89152-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230905214412.89152-1-mike.kravetz@oracle.com> References: <20230905214412.89152-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0153.namprd03.prod.outlook.com (2603:10b6:303:8d::8) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB6395:EE_ X-MS-Office365-Filtering-Correlation-Id: 8f5c52d0-8587-4bab-b4bc-08dbae594815 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: kF4FzkRUx2gxS6JiHld2jV1kYzT4NIh3PXUrcDlPev6YmeY/CHZXErZSVsZ/PasI8a9IPxz4VVG4BrxnzSBBrqeTMWqfgR2gA1Y95oyT16Xq2lEquhOpVAV70wwwo+FUwVkjzAGym8w9hHjbbWIlL7AdK8nyMndWaGawB8rJDoOKWr0upqUOWsc35oR9LO0Cvoh+PfyxZCc3KvZVL2qN9bPrgJN2J7tmGWXltD1uDc43rGE2IQqHp7zd5FKUsYAHRZ0idxNsXyoXpAIYY5B5DBms8Oa8Rc+xfEBAePK0An6l1BcY0E911eKp7qCQuGGolGPIIswIuGwsSGjhV10pbQtMFifBk9UshtjnYpPTDhHXY5YI1bqTsA8F/gIAl8vtYJXieWC3MwMtWOrXgNUJkJGZW/m9ePhcoMZBvoAAyrpNkwC+IJVikaZ0DMW23fAZOE4SZznPcpzHL6hfVKkHRs1QzD6iWCHbBXhVmGqcKaLTNbHIxI/T24HCe/e7QC47heFZfybGSxIP7OyrjNuGajAY671ZZmVHnbPpEukovL1PUsoPIX0hcWR4pGSWpNDq X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(376002)(366004)(39860400002)(396003)(136003)(186009)(1800799009)(451199024)(6666004)(6486002)(6512007)(6506007)(478600001)(38100700002)(2906002)(7416002)(107886003)(1076003)(2616005)(26005)(83380400001)(86362001)(36756003)(66946007)(5660300002)(54906003)(66476007)(66556008)(316002)(44832011)(8936002)(8676002)(4326008)(41300700001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: JZjZe4NPDHQB12n1mBZBytIOo5QArd0sniMMIjCHh5JF+krhKRHSMzxInQIhzD1nyDvgbo5M/yexL8lElndGws2fhdUyx3CM75OKjka7z8sv3fOgZPkeJwqrXLt8lWLsTfZDuqPJvisQPbkd/pAF/xIg69915zXCX/KwOS0+QaHCQgW9D6D9iXQ5NLugWil3a43jrCYzBbxzRrQnCzN9aLsPXhEsrYqEE1XMs6PROFr0Myhd7ZzruS58MxjcApwPDLAwixq0/W7ds1pzEr078TihhE8u/1O4aLrhKhnM4eINEcEpnGxioh5SR1qIMppfKAFrG9HbSLR22pvIGKozw4qH7nPFy07DOFU33QPzz2vq1vJqaUkYELQqIiNRbwiPfTqDaDXONS2tbftRDeyWgvnrI/QALKzhHRAoQJM30HeJ4D9DfXGmQWB4vt9SVp9jRNm/VUbutwKfF5hhqJfnbi+NaqcOcqjYaiCpXDohkoT/i5gkEzIx+4lM1W8wija7OPLCQSfdLPYbqpFmlfZRGSzTGjScJjMq8vJ8r3zRj4DrMZ86FDuWlTPkUDkem+5E12WEOdGGhR7MqAVunVpgBxw4jAlMO4a95o5yLAtFqxUyDl5GC8vg0R1g0XkvqtyTkkOKRG017eEQlfMyUoNWLu/1bAyAvstR/EGwH2huVF6YQrZA/f+L7U2j3c+RBa2j/i9Vr6dRa+Rs/rL1DogSaMmuC81OrTnl3e0S+gw+FvltJXdCUenvONvf14HUIKFagrxdnivfR5Vy1v9imDgx5p4SBc3R10c/sbdVyqMpHWJvuHfkHKCSJ4UUZyHnbgf5CEsEXLuxKM+yAPaNDRnA9Ng9cY+2NXeqguAUbbBHK7S/tV1qrs6BZK2wt2TxLLMSYhq87vJJr2PacnncjLHOf5Ao2VDyoR4cNUAL64Kz9efPSGd2oRB1z7tjOF3DdgKO8L29hY3jttd/K8yUXThklvoJc7gbb09TmFZH3/IwKBwvzksJWteEYNsO9Dsw8xokiguZBQ+yhqFvg9uRlnqbR8RIzxPTtcnWNMa8/qGHiAha2MXBEVO3nuqyHA2lzZhPf4IjCrXOR90L79JQVEEQCwPOOBQv1sMjqomq+HogudzLlwGPtc8ynV/KisF7+8Vy3uK3kfqgTNBPJrSemXP0xRGVV/L4gNZh7elzG4vNJxqdwqde2/qZi5KJRgN6cyUEUa7YpVD+eAHvjVaiUWaHSRpTLJV9RfJLcse9PouWm0ZhY8FmDMpG8uSZInsp1GErX5Z2EKBzvnRMDMZhlnHndwmgs0stLu2/ODLKTOXP2y7SD0fEjAybDjlszfI5dKVv/oaw33Jn65La/txbQXRO/K4ZAZt8Yjr4BEOUfwN7T5bVDGgf8wjlrr10WIVRg+CEKkkJS11oX5SL0kNhkvPHGu3EklyBxzzHwK4KulD53ksvrscO56oTAL9trzOmmSVXCThcWzX/bSW+Wfa2ofncSv9XztUzAPEHbMRHrsvrpwcPyoqosOVWoS9svVoFcOXVUHaFORbd98faG6YrF0R/AlxjG5v6ykmqEpiPf3KCr4T3+j5fPCuTOJoWXrfLj2NbhLxABXBIFD2tDBxRKkMiyQ== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: Smk82Nyr/uL2CkzUoqSOXXpY9gfPg5ZgqL7rHSzJJtt0q2wTHK7FWkZa/I/Eek+yNjwr2z5T6omSot61VGHpaPwxD62pcd16lIu8lViPk6IBGmX9uHtbpkwcmAraz1caf4buXeEh7BK9wynDtSlavmYS8+0oCOFot0TpWo6ekME4r5ptMJUFhm+dOj6ukK7HWb+WLLFzj38n1+wo71cma8Z0SZTbm+VKSftxwalPTY8qPafUYWEabZ5GWqvPGsqMeJQHJhb9C5m9Xz9Ts7/VOXKK4k/EkaS/0xBAnd0rx/Iw8NnojVlI9WU7p6hctddLzE2Oa++2leC49rd0FUMVLixTWfUuorxoGHPBAbGeHE3TjS+72rf0zQ6K+Sa1HSzC6/jmdjsExVzx46S8BXcpY3Gk54zVqDGHRBijLqYj1mkeM4RABOK2+4aufshS9+A8mWlL/fUkRStbNcVFuDra9qVb6d1SlymWv2z3Il75FQyjhm0Y2lkELQ5aK4zKGmTnvwEIWtuYsiwAXQoG5qbWgKQXCYbZ58jC4Smti9AwaBO27Dr5BjYxzy1PNmH0JObNrEl0Juik6Ooojeug88TuVxxuxU629Q6vCodAddlKZoiAJVuXxVUwVJ5reNcEoJ8/vMjFni08G7VS9LigRhgUCpa6sF2fa8RrQ0bci3JjS5wfK/BJ8hP9E+oQ/LCKGIsC7Pz3O4o8274B5L5dT6hd9KLXbg5wqmZJU5QAOZDXDZUu8VvVKSypsOgzVyo4iKgf75lvzJgPQ+JV0EizpGH2hOnd/lFEFrieB92H4Zo6BRba6B/M86FiP3TplaZbitr2EQOqRcl8tUC9kV7tsy0Lc6/wCaA5X4JPXnU6oTQv1vibxiu6nbV1/11aR6YJy6Vhll74k6e1Vo4lpHux5jILz99KPdanh7Xmz3LiQRPQpb1wJLHd862xXZUouoetcqpXxqFr9Bv4R9tixfV5QK8chPXpmXZk85a7MDQ0cgLhk5B6lZ9hzEVvsKvdqPop5+javPDEmzmVhcZznTTUPh/xoUkH0NhUvHcwWVH/PKioiT4x3TCfApU81I5Z6b4haBLseyRxddXv2sBC8fLE4W+3eOwkmd7nuwEExhGYm2bi0QyWdOEue6XhBV3ErMLEeEny X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8f5c52d0-8587-4bab-b4bc-08dbae594815 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Sep 2023 21:44:28.8978 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: euQP1sJItdXaHzpnawJilBFf/F/ihWgIX+LKSA2PxTbtqChC0CVTdOyPZvmlb7/MREywPAK4VBPyIeqx4lOCZw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB6395 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-05_13,2023-09-05_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0 phishscore=0 bulkscore=0 mlxlogscore=999 malwarescore=0 mlxscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309050189 X-Proofpoint-ORIG-GUID: bajyeeLebRXH2TAma_jZXhTUxp2Nm5OZ X-Proofpoint-GUID: bajyeeLebRXH2TAma_jZXhTUxp2Nm5OZ X-Rspamd-Queue-Id: 97B3A140021 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: ainjkin96u6g4hsaq3hr94p7ch1djcmx X-HE-Tag: 1693950763-466014 X-HE-Meta: U2FsdGVkX19UiV/E7H3myl14KsXEuufl909I+I11/pdgPQQmASOhuGe8nUk/bocn8qNvHXhnbzCKZ+EohBGKa8qKIeke303mN9ZcIYQThnNbuaQiYxD9rJ7MFm0/RdVAzV1R1yzhHtCkR/9tlM1eBNFpC6WIdBb4gDXhcp+onxSHOUx7X04CImaP05txLnz0ZSNArMMJtGcXER+vRkpx9/W2+Qd6IZjz/P7C48v1n/df3BB/dveDp3op11OfZvTrBpuz+vQy57REYakWAfS3nV/5nqBUqlhRdO1ilhHzMV5JCl8T/6q85aVBxYS6+foYNQE9FNr4JgJ0X3XFYzs7tZIYnxtXoltAY6WmS7wlhJ4Gw0eEZAKU56lM8XwwgYwNE2rfmtNJp1FtB7g3eUgU3nfe4ibK9rQiEJsW/tRKZDWNElyqpNN4qDRDkBuU/rYL4dY6XPXWc7C3kpJ9twDeEKadGVl0TRhxa5ce5nFJ4zwX0GfZU2LXUPUg6ndcmRogGGUFnyxKElkN+b8QWHcXeqVDmrDXewngRKBcFHxSSHk31sQzxHYwIntOxQKiYC7+0xM23PY5v5zbsBa6p1TtRpC1zUWk3YEhGpSd3jkyTDMN65aL4eNLcD7vJBKh7KlsbPEzGba2ZiX9OeEnt5cQyWDe8w94eO5V4QN3zbAmZtv6kl+LyNfhCRkc3MZuQSmfHfp9h77N7ExLMuTWahGVHEwqW9EwQqTsQDXlsDbd5tbtRgychnKkYxBZp+/8geeOcmNukncFG6gYbRJ9aNgOjKxNHNQpwN52cJcWZczMfdgjJ10C3QabYTk1QffcBzviU6SXnHH6OWX14j2RPtkNxzLqABXUobwaBI0VYXUIypWQszFAzVn4ZwYPOojSHTtasgRoUi4zNSE9bVEFBMvrQNfUhY5+Z+HPIJrGVXtknjeORDxaqLYGduPgvCUwndo245QarQxXIH/BTusVlmw dueLAAkY Jl/QvZL/YQwihJggFmk5XDisnQ15AGlXligRnr7imFcpUNj5DvVkf0olxgzGHc77Dlinwx3d2nsuOjz6tBspll/s3CyK4qH7Vj4aYl9RENjFZWhrH2UQgLcyhQbIuP+50nc6i9eQoZsEssJgdPg6b3Kp5OfcXs8NJJFmKKgDc28s7UbnNXD2ndTrSJWFshATw2M/Tk/RrIxO9PC4y0/kJoTt7b727z17iOhSUgYc82UBMf3atYN39pA/jhuFKnP4E8YdmDOYHA2fLK4WwXcut23WYmyAc3dhhNmUJHkIRv7git+uvU5g4g1lF58zUL90WPbVEBSoQSKYtMMjUmcGubY3H1Iz2G4ODvg1sFUtk6CFqU4NSjN1rAhEjit0EaYvPaWD/utF7osWKNOhpajMnf+GkhRC58qMHAsOufsh6cGLWHJ+j+cndQhXHw80mW4j0XS0HSQxnd4z0xykn9IEWn6NAx/AJ+/nBm48QbspXw2mK3RoCydfIursruYc+cXQuSHt0+WAbnOyIsH8Z1yVaaLWgVCkJcrU9RA82zE2bsnUJ09Nt/jIqAgmdS2ZlEGxwujlOD52hjCS+EAkiAPrO4L53Zw/Ab58H6OetIBAr2GAkDIr68QegEM40rt1AH+EXKhn+pA0Udqf+OMx/mptVfwbmY3N7X0YIsVKsje7HpFe7048= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Anything found on a linked list threaded through ->lru is guaranteed to be a folio as the compound_head found in a tail page overlaps the ->lru member of struct page. So we can pull folios directly off these lists no matter whether pages or folios were added to the list. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Mike Kravetz Reviewed-by: Muchun Song Cc: Sidhartha Kumar --- mm/hugetlb.c | 26 +++++++++++--------------- 1 file changed, 11 insertions(+), 15 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a27fcff3350f..f768fe9aebad 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1836,11 +1836,9 @@ static void update_and_free_hugetlb_folio(struct hstate *h, struct folio *folio, static void update_and_free_pages_bulk(struct hstate *h, struct list_head *list) { - struct page *page, *t_page; - struct folio *folio; + struct folio *folio, *t_folio; - list_for_each_entry_safe(page, t_page, list, lru) { - folio = page_folio(page); + list_for_each_entry_safe(folio, t_folio, list, lru) { update_and_free_hugetlb_folio(h, folio, false); cond_resched(); } @@ -2229,8 +2227,7 @@ static struct page *remove_pool_huge_page(struct hstate *h, bool acct_surplus) { int nr_nodes, node; - struct page *page = NULL; - struct folio *folio; + struct folio *folio = NULL; lockdep_assert_held(&hugetlb_lock); for_each_node_mask_to_free(h, nr_nodes, node, nodes_allowed) { @@ -2240,15 +2237,14 @@ static struct page *remove_pool_huge_page(struct hstate *h, */ if ((!acct_surplus || h->surplus_huge_pages_node[node]) && !list_empty(&h->hugepage_freelists[node])) { - page = list_entry(h->hugepage_freelists[node].next, - struct page, lru); - folio = page_folio(page); + folio = list_entry(h->hugepage_freelists[node].next, + struct folio, lru); remove_hugetlb_folio(h, folio, acct_surplus); break; } } - return page; + return &folio->page; } /* @@ -3364,15 +3360,15 @@ static void try_to_free_low(struct hstate *h, unsigned long count, * Collect pages to be freed on a list, and free after dropping lock */ for_each_node_mask(i, *nodes_allowed) { - struct page *page, *next; + struct folio *folio, *next; struct list_head *freel = &h->hugepage_freelists[i]; - list_for_each_entry_safe(page, next, freel, lru) { + list_for_each_entry_safe(folio, next, freel, lru) { if (count >= h->nr_huge_pages) goto out; - if (PageHighMem(page)) + if (folio_test_highmem(folio)) continue; - remove_hugetlb_folio(h, page_folio(page), false); - list_add(&page->lru, &page_list); + remove_hugetlb_folio(h, folio, false); + list_add(&folio->lru, &page_list); } } From patchwork Tue Sep 5 21:44:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13375111 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F359CA1019 for ; Tue, 5 Sep 2023 21:45:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 675198E000B; Tue, 5 Sep 2023 17:45:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6212D900003; Tue, 5 Sep 2023 17:45:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4AC6E8E000B; Tue, 5 Sep 2023 17:45:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3A1E18E000B for ; Tue, 5 Sep 2023 17:45:17 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 19BF240A91 for ; Tue, 5 Sep 2023 21:45:17 +0000 (UTC) X-FDA: 81203875074.14.1638C33 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf09.hostedemail.com (Postfix) with ESMTP id CB14614001A for ; Tue, 5 Sep 2023 21:45:13 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=cFY7l7fE; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=m3Ln3ldS; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf09.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693950313; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gKX0pbp9UVdp+b8TX61xER/LCQJjhtFXCYpv7vNcYao=; b=hRteSwHIsRKkOb5VlKRICGmNKKL1KvlvEl++Nfh8IUxQO2Ki7ZT3CJxe6qJVAXsi0PlxFk rM6oTcw3XobFgxsWAJno1Pe3jXLSvK7c4hlye8tgr2TGwvZv/O4hvBBoT1EECTZqh/YiGN cFjd3x8FRvAk8bLf42ZoPuM/NZazw1A= ARC-Authentication-Results: i=2; imf09.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=cFY7l7fE; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=m3Ln3ldS; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf09.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1693950313; a=rsa-sha256; cv=pass; b=SQfmCYwsPyLSw4TfjAa6ThtBXaJjnwYWCVwUZKSLILnrKI2ZzTs8JfHYoIZFdMj4MPvo2b X98GHbhb6FwFxjmeTMXYVPeOdYvh4j0D4d41mbSB2Qx9A0SZV1viL/x+LvSP9Cg/RMIjhD fRITB4hXKJn79XbYjtTbXCu4kVN6j+w= Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 385L9vmM006948; Tue, 5 Sep 2023 21:44:38 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=gKX0pbp9UVdp+b8TX61xER/LCQJjhtFXCYpv7vNcYao=; b=cFY7l7fEsDT25/hf2RlJEedMQxryvtG7mlWKb8vbShKzQsmnbseCfaBnBWMjpX91u5Md o64fH7jEceifndY4e7eBgIx60kYWUnfRdfXbcp0vB5Az88KHnvSzCttGZfauvmMUc/WA hkO1D5x6l9mrQyVIcCFm5st1pvOjnV6ihn5EUxG6e40l+GnfJocCF8PTpVelV5i3RzG/ alAqjRFpRt1aYEIaMLmK7mqtf356D4/cCEvi5UhnfJIyafLj/MgCpQCnybkMKBOCfk5O ZWSzp2sZ9VRE/D/kMlx+WKlJ69C0531xhOm/XLzV6meaE4gtM/4MsMH9aMbi6DRmVHPs ag== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sxc0g82sh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:44:37 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 385KKIJt007701; Tue, 5 Sep 2023 21:44:37 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2047.outbound.protection.outlook.com [104.47.66.47]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3suug5c6ur-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:44:37 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PyzPzQwFKvwwOxOAtchRNKUxLwg0Io/aMFvGl1dX9iMo+MzbXv6OYPTVBaD9vy2QrtzhcYWQvuUHfN8o+2PvTQJqykRJw292RllUSuC/LeVObQClH/BNHK63IYrQcLPnpnWIp0IVDTFy35wwO3sTx0jCc0aRMAo9lVw8499501seKx3yZiI7YyatUtHbUhsGIw8jhJ/vDYkvg4P7FNo+wSdry4C6zKeUBU/tzMWnCw/UMITto+iqhMz2unxfaxotYS9+ivw3aqSU+fIMGeXzRimFb9fbFZxDn5+hGfZx6aBbxNRWq5rh57410Zp+ZR0XkQLDgxbAFhf/eYJ+nc1NPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=gKX0pbp9UVdp+b8TX61xER/LCQJjhtFXCYpv7vNcYao=; b=EJmpU9s5jTuMxXgY4W3i1eU5WIEM3ZsHC+1wOapAT9DDgWkqa31JZ3f3oNrroydNRjYEbCVcs5Ajlzg6fNvw2d6mTUz7Od8SkrnPmLxTj1CU7oDf0rkhzS01nUWDJV3Y21YyCSPUXzV77g3iXyPGcG7JO9rUfCDRvgeCVw9p7zdPj0HkzCy5RKLWP3MacZxYZkWj9mrzOQUq19caMGGu7ubb8qUhclvt1Ozqn4XeZwOZXG1pB6qCHAFP10QSPNTD9oILF7GJq990o2TLj0Ii9OzdBqRrVFKtAJbQELKVcNDdXbmkudcdirgQ/Cb5UAWJ1wAqKHCaL8/E1jP/M4kQiw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gKX0pbp9UVdp+b8TX61xER/LCQJjhtFXCYpv7vNcYao=; b=m3Ln3ldSaIIQxgpFE2A9XY2slT/9fTVm/clQjYFGvEgodQQNim6CBHgISD6XFZ4halNzJld1D0bKFSe76ei9VZG0FXuryQtzY9/IQRnGcYFUjKRG/Ls97pWTFpW95fKuMNfpJQ6ZKK/QMyfEmJIDee2+GZptUEn2ASUVaReg1bk= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB6395.namprd10.prod.outlook.com (2603:10b6:303:1ea::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.33; Tue, 5 Sep 2023 21:44:31 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715%3]) with mapi id 15.20.6745.030; Tue, 5 Sep 2023 21:44:31 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz , Sidhartha Kumar Subject: [PATCH v2 04/11] hugetlb: Convert remove_pool_huge_page() to remove_pool_hugetlb_folio() Date: Tue, 5 Sep 2023 14:44:03 -0700 Message-ID: <20230905214412.89152-5-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230905214412.89152-1-mike.kravetz@oracle.com> References: <20230905214412.89152-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0072.namprd03.prod.outlook.com (2603:10b6:303:b6::17) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB6395:EE_ X-MS-Office365-Filtering-Correlation-Id: 40b6f6a6-ccde-4187-f198-08dbae5949aa X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: okeHPF5f0Q4R3EcjnnfN1lSFKANkgzNLYzotHFl2Y52kW/c7kaSPNHiWLw8sUSRzMMzxxPyy4n220mZbocMXf6Z9syWg0pMEBV/f1QOkzrVPvXYRgvFmzxKq4gueB0o4NUicC8xJSVHbf19JDey5HoK6i8q6Lue0rd6vnPrLSvkasnjzgrX4cvOdUjI2fcrVzfIlweTRU2b8gymY5YoZIMOm3QyzevVniqBmV+OWAqvSV0F6HJuEiZuL31x9lZaZK0uIZAld0riO/mFpJHVOc814I0RxxGzS1LvKzsJGFCSvEfrfQGmvbwHRPpBgsP2HJZbiE7mUSnUaCV8bFqnUAVh91HDzhk7XLlcxS252btNd/VMc2jC3hydaNXZmJBxDHaYsbt4hi8ZzeuznB0bJa0V1610nY86Ujvv96lFUTWwJf4T6twFk+PqJjP/665RL0aP67Gd56N6rZ9TbBdYDnDQG9tQHkfmIH2Y5NIwg/j5fxBd/aGHHmBtsILDpSEyoBw4UgGNPuiBSHnVr32is9H4FxEMyowF//9HgXJlG2DHJ6W6/oKcT57u0Ly+N2M0RlCpzgwglH6tiNiNUYz+amw6K28arJKaqZtQNFgz4NXQ= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(376002)(366004)(39860400002)(396003)(136003)(186009)(1800799009)(451199024)(6666004)(6486002)(6512007)(6506007)(478600001)(38100700002)(2906002)(7416002)(107886003)(1076003)(2616005)(26005)(83380400001)(86362001)(36756003)(66946007)(5660300002)(54906003)(66476007)(66556008)(316002)(44832011)(8936002)(8676002)(4326008)(41300700001)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: pNEeRyMeKB9fEWkGuac5EgImumrkkjzDmfOufTNDESAvWGDV/U4Av46kedm8B3b3P3kVIhMjwj4s+gskHoomvC4hBtVm4Ka7TxT/AZbBduOqCgGwf6cS282fLc+EJ8PVl1gA7XykFKCD1nTjsgDSf+dl4sowJXbOwE60L5oZt8WPAloGfVWK6hCVkpJFHzM9ovqyel9KSUiyeyBY6CDRwfcKThSMWwM7L+bbnJ3ZTW1eP7gwOee5nnW3x2BEC/GYrah75kS9ZscNb9KgcDLq9j5wDFWFWAPgWy6JeYVfzEaabH1zx9sP+1WMHRLQv0ySVmwc4wc/3wDtLXSuQq/PAkvabGjmUDBGaEYMMh6U6bNZT6+VRZKkvMypRh83K/bZEKcBCSWhs5JdcjqVQlU6zsGMr6VOPo/sKjPukIRYJqqkaU6+hnkMkkR7bbSsOe1ahAoNnDyTNrEliwtkVn8/HC7aLMyH+4RqyQxNsIh6UkESMCwq4u8qaKGiQdYh/B4SDUCDOfx/eNLXL4UC31a8MTesM8+eFbHzXKNuUY5wCFgqLBtcK0BXFtMBBPX0Wa+nXSylbSvrnM7gajG5uNA3tuMX/OVAtH0ERLecYjLLW0g+2lWSwNDgj3tqm2ydVAA2hnP/QIUcaIhSA5CGOnyt9n2vYnHXBf8jazP/+1RRmlHGWYTP5k4AfDkJAXJFz7uQXdUG7v6ohReZhK68u5gofjq65WPSHEfZK4r5HN4RTUteFoTpwNmAmSVFEYl/iZ7MTfvUeG9hrmkg94JMwkMf+D919MiWxsHZuNzPL6KPgexMw9p8A7O2xVpaVanH+Dvs6+FvEZ9HHC37Q6sZNaMAkESdA4YvCLgrm/3C7Mzo1xaLY3IwYMkYEcugEWMHUK3QA420vbXuEPIVveCov4B35VIBMEh0/0M+2qvQtbAL34mqve8f54SiN4AJxUfMg5dLHgdBpbVqh68dFzfv0SQfTL2xUKRQIqzd/P2odOt47hrVmZ1u3aCPki7ONdmmFZurvs/qaG4mrolddFAj5pn+7hdLUqXOi1TW8oZoHBH+YS09jszpBcZHuo+vDoOD7N7Rti9s6RVvKibtnsYiOhoGgpioodvFFkrcqqyTIDepdcsmjCAfjDmt1zJDL/YxQ+HukmbQbuVAJCOWmvUJtIaewFtInl6enewIXFynTQUUXJ20wMTtYPCmY0B0CNcNDPmQXbEokLcFydv6NlmixE3J4ZoUUc716jb1eeWZCewXydUjne8eWjk79ng7ONbqqqEJKcGkntyNXfXBGxYeQQ91O19Dcvx9bK8RgDt2GZFFH/X/MN3KumXoPTbaRfUHlyxug45Cl3+UmTlTvUVfpBYNkBlNd+sxjQ9EKc0WXVZ461knFxCajMioppqBQTSSzGSiQAfpifTYJzU1/No/vYF2KyOof5xjJQmRFtUd6Hjc4o/lcyrUZp6l3VehMVeZVQ7BvFdzJ4ynKeJrBg+936ffZED836NbcaIw/qYl3SzYXKAgA5Zp5z6raPd1jAoyNPrwQytsWO/yTt4jvSHUZEDRqdm8P/rKG3cTzznMYaQ43pmUjnA3pVRMs7mfQVT9m8w80isFMtJh3/1WovxJP3U8HA== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: c+VYZ97XMCwwXHn4uhMiXfi5W0JrM7sKJ+cMJwoJRSq1u9V34AkHNbUSSoLsgHGcQffPzKb/Ts7am5/HTRacZc6+K4gd4Ys6WptcUpKKwrpWC9urb+WolxkNt0pG/UpiKOxLwos/inCCYu7x1WRD68g8RiYoJ1Nn2idVm/aApCrVrc9tjq26w24ibJDU1K5QyRhNhlqtX1mQ+SmdWxDqaP0HM+UR6SfzL2jiTfnlnSqxlx5/WJ2+/NPX97unwD9DDlDteHN7/59u8aogEL2a78gnBmOXLWHl4O6X0Q1zZHkGrU/QELeHM3sHIq5DvBs5LhBaoYa034E2Hig7fSqSUo7lqgIDjxNu5d1ahrwnoHrXeMWhVDd10wF2dUxjbSznbCBFU676FCAqo8bcloCw9UCpmjsTF5XozTAfMeNH5L2NtRdzcMMTiXRWA7T3Ny1QPfj3YizjZh4UgUMpq9ReNaN8KLywK7Av2imj3C3/i7XeFSQib8blTd5yPXZVWWPcMNRsozYtIDMF6WkGeoNG2N1PsrBERL/PrOlnfCbW8ikiglkpHf+P1ZyT0F+M5xCl6/KLkmidHgndizTyzamoLzOFqGnzcMRBQVQJKQJ88G6cfa2zmutrKIBNY6zJraak5WrTe1D/2xW7r2N/35SFPS1a0TnVHg6o2MI5ZE+luItwNwg1Zk0Pu9fBb0Iso8P5GBZerinfrBdju+RodPKWnX8ahwfS9SR+anRmXk1JUJaqbMiIjK8iPAQ36+SJQB4S3qILnQiH5qwQft2Sa4Ybr/FNMtq/1K0HApaNhL5l5EaguBvJgyAARAAnXNEdS/lDinwFtUUI625IM5SZObUNp2lSKva8kw5r+fXbXSOpIZiiPQepyebVBYpSrNPIUY8tqDVSnZe+s3kJgLFBEPdkPEJvuQ7CMJyFZtZMT2MQxdh8vlREg45v5yRMWzZBhemqeeN7ptQAv8H2fwmqNyTDzIoVHp0NlLBOxpHr/52sDmg4c91UxJfS3WzKIkx0L+1FsNBgfml9fUIpCHYD0zMSm23WnYOJi/1Cs1XmcK/7Q9GU1JlOT3ywHZwGpPsyRzGV1DuOG3etz49MjquNAao7CARjJHu0cS7Op/w5A6HXVIM4+RyzhdqUIF0zghJQQ9vo X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 40b6f6a6-ccde-4187-f198-08dbae5949aa X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Sep 2023 21:44:31.5686 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 9949QE835byPnJxK1yG4R/5u3UYPgLAau/A8/bQv3jBcc1ssEav1GLe+4c+jCwv/SYMGJ0/q7E4jEd8Ag6rmig== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB6395 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-05_13,2023-09-05_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0 phishscore=0 bulkscore=0 mlxlogscore=999 malwarescore=0 mlxscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309050189 X-Proofpoint-GUID: jtRikbc30i9TJT3PyEOSCf6ns33xuH-0 X-Proofpoint-ORIG-GUID: jtRikbc30i9TJT3PyEOSCf6ns33xuH-0 X-Rspamd-Queue-Id: CB14614001A X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: djdde3shbirab4ga5mdzwyj3567ra7uu X-HE-Tag: 1693950313-430417 X-HE-Meta: U2FsdGVkX1+Rr6ZsgAiHgevjVbsNcdAvLuwwrsV65bGOata0CGqT6oMxNJMtk23xA2JHr6w2WiXLdwLTGB2mCRhOTFpo6PU1ZSSkx6npIDZawNMUL2ON1tsDqO1t+VNX1OIPiOiEli4SPjBdktfnUoJ95mZl39jFioMlmQQUcfLNkehNH1LJxVcX9CMjpIAgsAW2E/OOlmY4rLKbNEFlnxrQhBTolvhCrDa6zypXOs1VrvBI+aCxiXPArKBlx3Bc94Q021za41zoOXXr1xJEDkSvQWd8hHVZG2WWs5vGmuO2lNKatHBRyMbyft7Hvw+hmC73UH+Tmz0XLpfGM+VME3d4Rhw2zkWhVPGapvsdQa7YvE7wP3LFvg/UB9PrWfWXo6eROAxLvbz1JrYMMyOVkZUT6egm7PktU1LbicUlsAUg2qH0XK0pkyXyj4DCNlwreU36TnVRIaihwFMwPZILhhzXu8UpSistYt/d7tKHWP31+cRH1GWBJ6B+jJRXWAnt7LNHC4P8ZGN9ad5Q6TP3R9Gv8WzyL0xWdSJGPkWCkkrSChl4q5cY6hmG2na7+RhG/Rk00jYlVWHL9dVC+LWsmkx4DMRWmUM0ihg693hfXLix2TgslEEk+jdH3SgGER+JU9PCVXeAy8xkJNSsF+RRqWKxaW5WLnnTFes9M3gEXTa3jDeMhwAAbn4+O1LgydzFCRajOshc9nlkZdTmi7sS/lN/3eksklh+PADbzr12aLeHNCh6VYsji7XUNpf4lKSO+cIbPbnRHYOzXmB6snvQt3fIvtzTxU4NY4yGV+cjj5r5S32GhXRKcyK094eMCVCAjZRUk2kEQ+qq9FMTrecnHcwRcplGNFaWV2cJ44t7AtktGNajcEvncseznJje9zhjGW1UW9fIYnDz3ydoEkvk9QuAlsd1nWhZ/FEy5jOZJei4dg0zcixpt3JRNIW7LcknjQHHEmHoBN77cFNpm0u LQ1bxQ6m VZHzK6XssL6OrmeIqJN66CdafxW2EF1bXbhqgrJ1p9Kn8IR2OgeYUlGWLS9oBf9OOQEH65PjEiw1eLOUGA/uBLyISyo34dNlCYLuSnqjJonCuFssa1LbqrdlcYBiTnA7GIBNHY62PhuHBouEIJ6BrAuqHHqrJymJx8OxP0VJ0ufX157j/wbydZJiJJHxPh5qmyoJx04wSlmnpEt+w1oXrZC11JRVrWRxNxEb8iw9z5RqW2+u5dvn+EwugBh6Ic+tKBh7DXwS6XgAf1DF+8TJnxEc0C19h8eaY1rYfGZJJshkUrym1DXK1SDdxTYUkZDk1uU6OMqpEGQ7hIuqzrafTjUBXl2BItfX7FnkmOTOEP1DBi6bQh84xqeU1V6h5AXwh2qR3SV/5I5TNLbPqOq3TKxfR83V7kkiYezu/MR6nZIjlelsqpJFHe93pa57+u73R+/4Fve4Yrr2pf7VDAOiAgQZkG+i4dXvCNbWm/anlezPOYPRGfvayZ/uCSWs3BpiMUbz7aYjYcc7xqeo7s7Fuf2jh5NP7fAyoCc2ycm3SQWxJSopRmCjOHXh7LAQ5ZN0r16RwLU2yd/ZtVVcwu5AeLBO+4AWsivjaSAELI3k4wp4LXd0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Convert the callers to expect a folio and remove the unnecesary conversion back to a struct page. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Mike Kravetz Cc: Sidhartha Kumar --- mm/hugetlb.c | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index f768fe9aebad..278c8ae6a36c 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1446,7 +1446,7 @@ static int hstate_next_node_to_alloc(struct hstate *h, } /* - * helper for remove_pool_huge_page() - return the previously saved + * helper for remove_pool_hugetlb_folio() - return the previously saved * node ["this node"] from which to free a huge page. Advance the * next node id whether or not we find a free huge page to free so * that the next attempt to free addresses the next node. @@ -2222,9 +2222,8 @@ static int alloc_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, * an additional call to free the page to low level allocators. * Called with hugetlb_lock locked. */ -static struct page *remove_pool_huge_page(struct hstate *h, - nodemask_t *nodes_allowed, - bool acct_surplus) +static struct folio *remove_pool_hugetlb_folio(struct hstate *h, + nodemask_t *nodes_allowed, bool acct_surplus) { int nr_nodes, node; struct folio *folio = NULL; @@ -2244,7 +2243,7 @@ static struct page *remove_pool_huge_page(struct hstate *h, } } - return &folio->page; + return folio; } /* @@ -2598,7 +2597,6 @@ static void return_unused_surplus_pages(struct hstate *h, unsigned long unused_resv_pages) { unsigned long nr_pages; - struct page *page; LIST_HEAD(page_list); lockdep_assert_held(&hugetlb_lock); @@ -2619,15 +2617,17 @@ static void return_unused_surplus_pages(struct hstate *h, * evenly across all nodes with memory. Iterate across these nodes * until we can no longer free unreserved surplus pages. This occurs * when the nodes with surplus pages have no free pages. - * remove_pool_huge_page() will balance the freed pages across the + * remove_pool_hugetlb_folio() will balance the freed pages across the * on-line nodes with memory and will handle the hstate accounting. */ while (nr_pages--) { - page = remove_pool_huge_page(h, &node_states[N_MEMORY], 1); - if (!page) + struct folio *folio; + + folio = remove_pool_hugetlb_folio(h, &node_states[N_MEMORY], 1); + if (!folio) goto out; - list_add(&page->lru, &page_list); + list_add(&folio->lru, &page_list); } out: @@ -3422,7 +3422,6 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, nodemask_t *nodes_allowed) { unsigned long min_count, ret; - struct page *page; LIST_HEAD(page_list); NODEMASK_ALLOC(nodemask_t, node_alloc_noretry, GFP_KERNEL); @@ -3542,11 +3541,13 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, * Collect pages to be removed on list without dropping lock */ while (min_count < persistent_huge_pages(h)) { - page = remove_pool_huge_page(h, nodes_allowed, 0); - if (!page) + struct folio *folio; + + folio = remove_pool_hugetlb_folio(h, nodes_allowed, 0); + if (!folio) break; - list_add(&page->lru, &page_list); + list_add(&folio->lru, &page_list); } /* free the pages after dropping lock */ spin_unlock_irq(&hugetlb_lock); From patchwork Tue Sep 5 21:44:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13375114 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15229CA1016 for ; Tue, 5 Sep 2023 21:45:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 37BF88E0012; Tue, 5 Sep 2023 17:45:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 251A6900005; Tue, 5 Sep 2023 17:45:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E35CE900003; Tue, 5 Sep 2023 17:45:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id CAD558E0012 for ; Tue, 5 Sep 2023 17:45:17 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A71741A0AA0 for ; Tue, 5 Sep 2023 21:45:17 +0000 (UTC) X-FDA: 81203875074.10.F7FEE8D Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf10.hostedemail.com (Postfix) with ESMTP id 42523C000E for ; Tue, 5 Sep 2023 21:45:14 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=Gg4cWPhS; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=0L6cHcMw; spf=pass (imf10.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693950314; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kRYBSglqq8cl8D8SZbgGK19NxyywhgoGbNK/s4vGIc8=; b=8lzFdlTO5Pdw2NrreRIXm86trZDhn+oIMwXt/czwrWMeAwoa9//5JOZCi/b9Cm+FlmJc0T pn5OD9NKHh1qc9nsDTNlxw9mDK3lyUeR6exYK8XfXTRa8scYgiV6SaYznvgw0fOJ6NgrnV wBbScbCXyyD92Q6Pm4dNiM0nNPlvIZA= ARC-Authentication-Results: i=2; imf10.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=Gg4cWPhS; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=0L6cHcMw; spf=pass (imf10.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1693950314; a=rsa-sha256; cv=pass; b=Pow/77RbftII+mAWB9DYY7w6ZQdr8KLFrZ1Ai+socz2cXtkOLTQE5tgMOwPKULYubWtrdn hiU7x/8qkO/OVMQqSSJrRRPB1WlAJ3lcZiw4KPhfdZFOqMKX2p3L0ZFzJ4za3OLVx8/vzu a4TVZJBWxiEdIrGFjImiNxZPynJPQwY= Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 385LAurb028694; Tue, 5 Sep 2023 21:44:39 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=kRYBSglqq8cl8D8SZbgGK19NxyywhgoGbNK/s4vGIc8=; b=Gg4cWPhSrIeGPRDeV9iDl7kBDNRQJRBd66QnhsCOyjS7vwlCupwf5iBV6yTE1SE8yA29 wGKPnc6Mv8uIIHOXdt6yXaLT3dJUbhmG5+zqnt7+BAST+mQ8LRVbR+kacT0xhQR/iz8e nOUVCs6H7+sbEMJkaSCetEVosjeFO5A/javR5Eh2iYstKCy9jCtHFqRsCuro2Hm+p5Vq CqeJVFCczPWM0afNfDuHm0HtwPVU/AbKu0HjxLY3FvFqEg1eSt/PGs3AX7WwTq6P8f1c S3nrSCenBe+JD3x8Ox9D7HR1QeHtaxqDOS9Gr8V3ftbjr2CCpi+ai6qjnVarRGCh4FHu 0w== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sxc0sr1qf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:44:39 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 385KKIJv007701; Tue, 5 Sep 2023 21:44:37 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2047.outbound.protection.outlook.com [104.47.66.47]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3suug5c6ur-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:44:37 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fEYqLj0R/Dwd5VldT8JlMCTV+F79ttlqUUzSPO0J4Bl3h/g6XFhqz2M/XnEPGHrINhUpH3bI0trKUcyhsMLsdNCrrg72g8u9bkiMMoaqD76u94Hgc5jCRs3iHPgF3+yMDHWRFucpeGlKcFVEbDkxAI9udW0vhwq8LY1wnvJIMmDojKWbFttcNiwu8dA0lJUBNhc+q4Qw0ocuVXu5qnUFQKU2nmsob3wqsJBm+UmajeF2qx1Xk/dGYWArOfsNjQgJswxXF4Lo90BDchOS//MVm6iOcC57CDeWPv+9iVx+42ZZV6hbEM2w3pW3idSuKEc2fuYSJMQe6cRhZIxp7R+t8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=kRYBSglqq8cl8D8SZbgGK19NxyywhgoGbNK/s4vGIc8=; b=Hz+xpn56Mmomzb4CVQR4Fy4QFensbDIGZLGD4JtdmB2IP+4m2QBOvtwnoPomgrSL/CttDE9f6i+NhDb0pziDLBnq8FcXPaTHJFKwN4t2p5nWwS+TAnBNiJetL+PcM7mvj9ymFbFIMPXQZ2qodqFSFqUtFNVcXSmnQEeiBS1SAEyZmYds6hrPO6xedqEhZPCp9mzeqoQ70L5lUxbckD0scXu44aXfr3i0Q585krGjlbYP86RxCdk7XaULxLhCn5A/R2A18onVBkYfiWFIPxI8wi1tA+jy2fynvwTfTCxgz5+l6/IB1y849mL3F6wkT2Ko48rRCm8QsheHGpq3uAJJkQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kRYBSglqq8cl8D8SZbgGK19NxyywhgoGbNK/s4vGIc8=; b=0L6cHcMwXpOCeRdQBFBW176i4fcyYU9cAcYhcWHISfchX5vOhiVimDycImJ8VYcjL6RXs48WpRWoI8qm88RoL1uWxEzZSWa6v1WHjSTCLXXY9opLb/fivF8fWAUh1Zf7O73+oJBLaq/TjA5SgZd62ufVspMUDx2TR/9gXnl4Anw= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB6395.namprd10.prod.outlook.com (2603:10b6:303:1ea::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.33; Tue, 5 Sep 2023 21:44:35 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715%3]) with mapi id 15.20.6745.030; Tue, 5 Sep 2023 21:44:34 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v2 05/11] hugetlb: restructure pool allocations Date: Tue, 5 Sep 2023 14:44:04 -0700 Message-ID: <20230905214412.89152-6-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230905214412.89152-1-mike.kravetz@oracle.com> References: <20230905214412.89152-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0230.namprd03.prod.outlook.com (2603:10b6:303:b9::25) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB6395:EE_ X-MS-Office365-Filtering-Correlation-Id: 70df22fe-df4f-4df5-4cc6-08dbae594ba4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: T3QwQ13jOEcQg71N6od3CcjLB7pg1n9FicAXY+8jiMaqkqva2oFL30OlU2mT08vbeay9vD2V0asedKFukILpheamoQgLV3ehkNlbnfSmsQiJ1qmglgIMoZE/ul2yy7Q1Sg3INPq5xjCgnztuj8P+k9T4T0qMvwlYC8miNkW/8IWeW0FDe72Dpn5+sniNAuTxyDAwHJDX+ktUvksvjRUTc0qAhm24+JrMc5X0z4i/YVFtMqT6BvOIEAY3yq1xwwJNYWlKaxTOKTDm8KbLiy6DJkogDbBwQvmDwRKQcMNfL5Zl5zafMikU5ykr7y1SXF68a+QSf44MDpVhtxIJTBi1FMQ6Ge6MEte4EhM8kZvptz2XMqNh4UdRHEaqSgdU3QCRaVdJgMeWmHBgqN3oxW8Yb14vKMj5lKNRIMitYzHMtWLS15onJghMjG7EuvVSpN0d9vqI6nBAohyGku9bNgFERLgNFJ8XFYwxIA0I3t8iTAfGULOGPa7PACU4bxAxT1WjzZWxWQGP6htWSANNQ6mJrfz6+5T1esXXlX6UKXgsWClptmx1wpKjKdK6MN2f0mNn X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(376002)(366004)(39860400002)(396003)(136003)(186009)(1800799009)(451199024)(6666004)(6486002)(6512007)(6506007)(478600001)(38100700002)(2906002)(7416002)(107886003)(1076003)(2616005)(26005)(83380400001)(86362001)(30864003)(36756003)(66946007)(5660300002)(54906003)(66476007)(66556008)(316002)(44832011)(8936002)(8676002)(4326008)(41300700001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: QNJdy3t0Qk60UekGctIlx5RhmKcGrMx9+1OVKPvRkJ3nfgAz43GrVvI7ECswpezqP/4ABUDy2x75ug1mIS02taA7Cj2rdb1o+HMABImc3Cr3ukSmK/yi8i+nVJGvxQ3zHJORhGqCQg0JV66nkZxkgeRy0smW9KCXVprU/urnnOa/rxXZ1KOKWGtOGfMHggA1S2loxL2CQ6ekq4B6z7jscnKy3UqZp/ejTkF8keWPQYozIPrdC8WgzjM98JC+tEzAsOk9sIiBY2FARcvKnV12G4jvyzgykRxA3OFotwkHjw+D9xXojQ62021W0Du7aGasa3aNUOu3slJBFSCNjMRXvlreW7Gkw4oxG6JPMwrHQ57QEISk8/7+yqb3KbWV4PMlJ+x0cHl18KPkpV1gCXAmGgIpqOoca90rsA9uNtyKu9v50UpkedOjSY+56O1zJPiMLwST+CCK/igO3eaHrNs0G5U6T7TFTtBkXwQuUAsroVaBF25XITgLzp1QlSipMBqzUEXir1RYYvSSHMk3NkaP4X0ikRQS4yHMnekm/vlN8ajs+PFBn1mPWubcz9wj+GTzebxAGRdy9pHT3vJOt/wcTClE8Hfx0L3z/7PiZBNDD5Nskwh1NTrQkfgAtXKO0apfNDUWY/TUwrMdUsCPOvxKFRY1j3aeR5OrPt4Yztt9y2OUvMhMbOS2kzEL5+wNwMFziz1nxQjoV/YTy9NFOSZuWiJiYn5DFT9uMKgGmEtsr/1i5Mu/trSpwRbLbuznso4ZSZer9q3ipaYWgBxUKReMyJJmkDh/zJVX40A1Ce41K5VRZ71YLW3BtxYF96G2BJvIqWu+u0oZZcaSTo7r8a3Whb3V8NwGu+OK4QW0koiwi+P2KgMOW+7d937QEMMU5LqcUEDzemdUul3kDYblzWY7hqKNJAdwgBtNq3DV5Ev4qlglJG2KvYs1FvM63/3zPJZwbCTXlWkR0HTCl0x0cIVzGFFf2bkUqE3uMaub/cH1D7i0bEZZGuvCemeAufYRUnfnoaCvky+N+1Uu3O9e4WLPFEQ71QxjK5M2NX5PGXmk3ZPOPZVWFlqSGSIvPTyP8I4A8E+OIAIStjHdbyR9ex5tArWkbby4dwloD/hg+zPtycZ5E6ktBKuVsu3wkeGa/Spkm0i7+5q7ru0OUWjcJRO34OID6YlZkNZZ4xLukoPOpGvnV5lOW0R6A+463ptzIKsqoBFToXlf+148VrN6bo7vLg9onlWgC/iN4WTRyY+hZAh3KeJc/dolXdM5WolDglD+frE+zou4G9bQ+1kwzR82grLCDjUe1LjK/gsYWMKDi1V8oP664ciaNR4IHQEdB7jgDV5Pm2rgVUULIgp67O2zBIObaSD+iHhInS4fP0EgwoOPtgWsDTG9EfavVdOMyvLQtp4Iqoiy6WmuXJV2ueq6qMMLOptSe0skcAhDlb3RdkSwkDOVBEn6xqmfQlKzif9yhDn1cr2dY1QbtChRl1kOggvTiSl2A8u7EZMYFCajk3sRf1Ugd5oHDczcxsvdvRaCD+wJTND0aHpMtNhrTojK8YASRMzNtWNXvGZ7lUbqnChyzVs9NjqEpmNxqn+PKaxe4e+Va+yblU+tgN9vqA9byw== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: W1kNVxB1Ge6CIE4JstOLMA9tDF87Urokeu0q6MuZ10aSYbA+E/V4idmbLQPM4gmjDxRxephB36oNqAkLue6XG792I9rjC4TdCpzXWlCY1Opdhlsrg5y+RQPUxPvPyoyun8gOpkXcFCNRdJU1n9d91cgxJWdaTrccZO3jiFKpZTeJzoGkghvxqb1v0BilcEBpIigcqFYoTInEcfW5kpMZZKn8T5SCmZyzxajlxgYKgmxb+/dXL3USOz7DCSvv371Di5e3TJHPWHnmOdojxyoflMB0E2rguhQdcH2iBieNej5pNKXGVIPPTQKjqxmBKH8bGTGevnzNGzfduweJhhRlb9F4V2u4fwcsV+F+ET04TqAYH4wtAJPApZfgpXXsQ6ffPXyFKiSXPOvbu9nUM/RjYc/+xrQr352/YG6jrdnfaN9MJxPkowVCzmU7NumJIUVlNKs0v7NHfzKWMwiVnCgAhQN1kBQ5pkDe95Ph+ynH9BncyfZbkgFsBSyFbqwj1Gr5F57UJKy2xsmaOBzxE+N35YgGqAoBj9xljXXlGq9kY0koA5GjuYCoBJCs80uBNeUlhSyB48A0GIDwOaDA+MsSOXtleIkxuzqydMh8uoc9vaPZgui8RMWbYVqNaV3UurwE+XUdcTJfzrJzj1W+eNOz5Qdac0R1d4Wk2q9tFmEFrZTDUZquEray4qh4aVhrfzDmvCtwqLVDRfbcY9f5Bpf2giyY9Dm1kIRRivq97n+uOwyZVL+r3SrKpeKb8zQIRkVwmLkKcprE1uqY0QDxFyEVf5b9mv+XmD4awBBYVHrAt9uBO4YLpGNZxMkjgz89jhQj1ZVTMP6uqKeKORVTxHn9Mm8NKjtEWLwrNMvpmH4dmUosGMoNYjVWcarNZ8leKHsE512Y3W2vxsgaBiem6KIzQz51rjsLgmLu0vrezrJManXu8crDCGy6YsEY8eM0jJ9ZU9GWd68K86WiVtnNqm0MXqnSboZ5IeG3Qb28rcWLtYPu9xjksCxICy4Sol8kTQGc5MxTxvwUQ8YGxBuXsCMNSwcR1lRlGZ8CGiuYHgMxyyYXba9sP+oN0fKEJ6u4lJq+E/McrczLJqoyP3z3IOE6AevPdmgcTs4DQmYMFp55ZjSUjHhiTQhRKKaXE+3CmP/B X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 70df22fe-df4f-4df5-4cc6-08dbae594ba4 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Sep 2023 21:44:34.8997 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 3XN6uRjvbLQdu0MpwdzbkEv+mTvGqH9hcBvRqHq2nHUbbOhf8/fbiMpBx92+oVlr2NzBfrLqTn0hxQyNZgYcWg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB6395 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-05_13,2023-09-05_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0 phishscore=0 bulkscore=0 mlxlogscore=999 malwarescore=0 mlxscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309050189 X-Proofpoint-ORIG-GUID: UmqdcnAC9xJdyR6zNSnR2WrcSTmwkqL8 X-Proofpoint-GUID: UmqdcnAC9xJdyR6zNSnR2WrcSTmwkqL8 X-Rspamd-Queue-Id: 42523C000E X-Rspam-User: X-Stat-Signature: 9u5sfjcqi1nexr4fu6igruhg9z7k7ry3 X-Rspamd-Server: rspam01 X-HE-Tag: 1693950314-292304 X-HE-Meta: U2FsdGVkX1+YNLCHS7e2LUAq/0z+X/T1P2w/sXVZl0bAZ35O5CEG8sN8eGa5bXenLI45j2/z5pSV+0CQo7gEoGZ1Eyvl1Hl18hvyfslVw4UAg8ontvPgAP9k7Bcd8k+yxXvIslW2vVoQFlAOv/Xb61DZAhjmRyUvMVaG9dD9ln3PaN9z/wg6Kwur/WgkAE+wQ+LCuKsuVoKn8Jgc4TJu4xDR3HagrJf5aMBLuljBRxLNV2oY8WphKVsjdAL1nKfCUNooAYV++DVXmq0trH/PsWcOCA3iPJyqyeNjBQJfW7d7RZdSYAZC3dEDccmGRlaWV5f3N317kDs0GjMbfIOluBQJRElWO6wuCE7fX3eLCc5LvDUAc09mEI9hYDMNQ+kFoIlHFsEAHnCcMGJy8H0eE789yeuyaeUOGv9FpAP3F0Dvy50D5XQejg+G6X4LgLrwJBd5l7k6dH73IUebusIbTgxQ10RrtSMY9ISf6mIwcAHHcN+voI4z7tpOZ9yV20iua9pYpXOZhlOtPS6lST4WsxND2MfxXJDT1BAsMNZ+m9Q7JuElmx9MIyvDprwCWqn8fpum1IqOKlMbD9cba28dwyVdpX9yrsF20xEttonufDlggL6jdtTr9qpAlou+6klici6+mx81L8AHR+Hp0vGEdfB1uFqRLq+3+7teakagWwlhFXbskWYWdEICYnZOi0IyDS0Mrvis/6G6/2kSRe49kNZGXI0A4UxzIRjq7KqlvIt9kKjL6mm5FSmnX0RnXvdfp5RUaPsHRPrTciQLnuZjtzRF+BZoTUvifOTeuPEDb3JPjLMv4HAH4bykkqBNcYVFZoXVroWYkVH5WMye2AfupiQ248sJMDvCWCiySYVUYkbyqNbOCoO+uwgVLZHUBOVzkuZOvR4NVtEloFaUPgSRhhLaWBV/uGqRMkYtPlsCIiJmvQA/Y7OkCWNJskmO2zp8c5CTE2qtIGzqE55HlOJ Ijz8wzLQ waV6/CcistqVjZEDjT+7B9NGTIRjHv/hRV132B504ZgB3+l5EvtjTmXpoVHlO5f5sdNojzHU8D0Ln089SwofVNYhp3Ai25RtCS/gZDREkIev0usIsaMRkK4vNjCdUyAfHNKUBcAKflPBrXktH/5v2YkiBEMGd4/Aa1RcJSHhSsNdo+Z2zluXji70MF1iPtGqlwzE1BWHVbfuCeAzU7NGexLy31gWNsvu/XNTG1Z1we9zCOQiNbmKGwenX9deTdpcyGcF97tfJzYm9pjoyaJ1SKZzSysgnRN5ncrR9iSSEfIwe92TIqAgWSNLzVPKfohNX87IRq+cGzZ28TG+Z1W/0inyNeyGukNungFk+dVUgAIAIFXJ++kwJUWU0zYbRDTdWpMBuI8ZG+HjcE+PvARzDTsraGyXyFJPO6ZiYGLkRRdfPIw3GkFmnCn9f/9SUKFTyiHNnEOerngiSqY4x8DoEFKI7N6qXiipl08AnGRop0Ek5gpyF+G0cFxrw39pmH6VxPQtM X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allocation of a hugetlb page for the hugetlb pool is done by the routine alloc_pool_huge_page. This routine will allocate contiguous pages from a low level allocator, prep the pages for usage as a hugetlb page and then add the resulting hugetlb page to the pool. In the 'prep' stage, optional vmemmap optimization is done. For performance reasons we want to perform vmemmap optimization on multiple hugetlb pages at once. To do this, restructure the hugetlb pool allocation code such that vmemmap optimization can be isolated and later batched. The code to allocate hugetlb pages from bootmem was also modified to allow batching. No functional changes, only code restructure. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 183 ++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 144 insertions(+), 39 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 278c8ae6a36c..d1950b726346 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1931,16 +1931,21 @@ static void __prep_account_new_huge_page(struct hstate *h, int nid) h->nr_huge_pages_node[nid]++; } -static void __prep_new_hugetlb_folio(struct hstate *h, struct folio *folio) +static void init_new_hugetlb_folio(struct hstate *h, struct folio *folio) { folio_set_hugetlb(folio); - hugetlb_vmemmap_optimize(h, &folio->page); INIT_LIST_HEAD(&folio->lru); hugetlb_set_folio_subpool(folio, NULL); set_hugetlb_cgroup(folio, NULL); set_hugetlb_cgroup_rsvd(folio, NULL); } +static void __prep_new_hugetlb_folio(struct hstate *h, struct folio *folio) +{ + init_new_hugetlb_folio(h, folio); + hugetlb_vmemmap_optimize(h, &folio->page); +} + static void prep_new_hugetlb_folio(struct hstate *h, struct folio *folio, int nid) { __prep_new_hugetlb_folio(h, folio); @@ -2151,16 +2156,9 @@ static struct folio *alloc_buddy_hugetlb_folio(struct hstate *h, return page_folio(page); } -/* - * Common helper to allocate a fresh hugetlb page. All specific allocators - * should use this function to get new hugetlb pages - * - * Note that returned page is 'frozen': ref count of head page and all tail - * pages is zero. - */ -static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, - gfp_t gfp_mask, int nid, nodemask_t *nmask, - nodemask_t *node_alloc_noretry) +static struct folio *__alloc_fresh_hugetlb_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nmask, + nodemask_t *node_alloc_noretry) { struct folio *folio; bool retry = false; @@ -2173,6 +2171,7 @@ static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, nid, nmask, node_alloc_noretry); if (!folio) return NULL; + if (hstate_is_gigantic(h)) { if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) { /* @@ -2187,32 +2186,84 @@ static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, return NULL; } } - prep_new_hugetlb_folio(h, folio, folio_nid(folio)); return folio; } +static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nmask, + nodemask_t *node_alloc_noretry) +{ + struct folio *folio; + + folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, + node_alloc_noretry); + if (folio) + init_new_hugetlb_folio(h, folio); + return folio; +} + /* - * Allocates a fresh page to the hugetlb allocator pool in the node interleaved - * manner. + * Common helper to allocate a fresh hugetlb page. All specific allocators + * should use this function to get new hugetlb pages + * + * Note that returned page is 'frozen': ref count of head page and all tail + * pages is zero. */ -static int alloc_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, - nodemask_t *node_alloc_noretry) +static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nmask, + nodemask_t *node_alloc_noretry) { struct folio *folio; - int nr_nodes, node; + + folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, + node_alloc_noretry); + if (!folio) + return NULL; + + prep_new_hugetlb_folio(h, folio, folio_nid(folio)); + return folio; +} + +static void prep_and_add_allocated_folios(struct hstate *h, + struct list_head *folio_list) +{ + struct folio *folio, *tmp_f; + + /* + * Add all new pool pages to free lists in one lock cycle + */ + spin_lock_irq(&hugetlb_lock); + list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { + __prep_account_new_huge_page(h, folio_nid(folio)); + enqueue_hugetlb_folio(h, folio); + } + spin_unlock_irq(&hugetlb_lock); + + INIT_LIST_HEAD(folio_list); +} + +/* + * Allocates a fresh hugetlb page in a node interleaved manner. The page + * will later be added to the appropriate hugetlb pool. + */ +static struct folio *alloc_pool_huge_folio(struct hstate *h, + nodemask_t *nodes_allowed, + nodemask_t *node_alloc_noretry) +{ gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE; + int nr_nodes, node; for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) { - folio = alloc_fresh_hugetlb_folio(h, gfp_mask, node, + struct folio *folio; + + folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, node, nodes_allowed, node_alloc_noretry); - if (folio) { - free_huge_folio(folio); /* free it into the hugepage allocator */ - return 1; - } + if (folio) + return folio; } - return 0; + return NULL; } /* @@ -3178,19 +3229,29 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) */ static void __init gather_bootmem_prealloc(void) { + LIST_HEAD(folio_list); struct huge_bootmem_page *m; + struct hstate *h, *prev_h = NULL; list_for_each_entry(m, &huge_boot_pages, list) { struct page *page = virt_to_page(m); struct folio *folio = page_folio(page); - struct hstate *h = m->hstate; + + h = m->hstate; + /* + * It is possible to gave multiple huge page sizes (hstates) + * in this list. If so, process each size separately. + */ + if (h != prev_h && prev_h != NULL) + prep_and_add_allocated_folios(prev_h, &folio_list); + prev_h = h; VM_BUG_ON(!hstate_is_gigantic(h)); WARN_ON(folio_ref_count(folio) != 1); if (prep_compound_gigantic_folio(folio, huge_page_order(h))) { WARN_ON(folio_test_reserved(folio)); - prep_new_hugetlb_folio(h, folio, folio_nid(folio)); - free_huge_folio(folio); /* add to the hugepage allocator */ + init_new_hugetlb_folio(h, folio); + list_add(&folio->lru, &folio_list); } else { /* VERY unlikely inflated ref count on a tail page */ free_gigantic_folio(folio, huge_page_order(h)); @@ -3204,6 +3265,8 @@ static void __init gather_bootmem_prealloc(void) adjust_managed_page_count(page, pages_per_huge_page(h)); cond_resched(); } + + prep_and_add_allocated_folios(h, &folio_list); } static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) { @@ -3236,9 +3299,22 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) h->max_huge_pages_node[nid] = i; } +/* + * NOTE: this routine is called in different contexts for gigantic and + * non-gigantic pages. + * - For gigantic pages, this is called early in the boot process and + * pages are allocated from memblock allocated or something similar. + * Gigantic pages are actually added to pools later with the routine + * gather_bootmem_prealloc. + * - For non-gigantic pages, this is called later in the boot process after + * all of mm is up and functional. Pages are allocated from buddy and + * then added to hugetlb pools. + */ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) { unsigned long i; + struct folio *folio; + LIST_HEAD(folio_list); nodemask_t *node_alloc_noretry; bool node_specific_alloc = false; @@ -3280,14 +3356,25 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) for (i = 0; i < h->max_huge_pages; ++i) { if (hstate_is_gigantic(h)) { + /* + * gigantic pages not added to list as they are not + * added to pools now. + */ if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE)) break; - } else if (!alloc_pool_huge_page(h, - &node_states[N_MEMORY], - node_alloc_noretry)) - break; + } else { + folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY], + node_alloc_noretry); + if (!folio) + break; + list_add(&folio->lru, &folio_list); + } cond_resched(); } + + /* list will be empty if hstate_is_gigantic */ + prep_and_add_allocated_folios(h, &folio_list); + if (i < h->max_huge_pages) { char buf[32]; @@ -3421,7 +3508,9 @@ static int adjust_pool_surplus(struct hstate *h, nodemask_t *nodes_allowed, static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, nodemask_t *nodes_allowed) { - unsigned long min_count, ret; + unsigned long min_count; + unsigned long allocated; + struct folio *folio; LIST_HEAD(page_list); NODEMASK_ALLOC(nodemask_t, node_alloc_noretry, GFP_KERNEL); @@ -3496,7 +3585,8 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, break; } - while (count > persistent_huge_pages(h)) { + allocated = 0; + while (count > (persistent_huge_pages(h) + allocated)) { /* * If this allocation races such that we no longer need the * page, free_huge_folio will handle it by freeing the page @@ -3507,15 +3597,32 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, /* yield cpu to avoid soft lockup */ cond_resched(); - ret = alloc_pool_huge_page(h, nodes_allowed, + folio = alloc_pool_huge_folio(h, nodes_allowed, node_alloc_noretry); - spin_lock_irq(&hugetlb_lock); - if (!ret) + if (!folio) { + prep_and_add_allocated_folios(h, &page_list); + spin_lock_irq(&hugetlb_lock); goto out; + } + + list_add(&folio->lru, &page_list); + allocated++; /* Bail for signals. Probably ctrl-c from user */ - if (signal_pending(current)) + if (signal_pending(current)) { + prep_and_add_allocated_folios(h, &page_list); + spin_lock_irq(&hugetlb_lock); goto out; + } + + spin_lock_irq(&hugetlb_lock); + } + + /* Add allocated pages to the pool */ + if (!list_empty(&page_list)) { + spin_unlock_irq(&hugetlb_lock); + prep_and_add_allocated_folios(h, &page_list); + spin_lock_irq(&hugetlb_lock); } /* @@ -3541,8 +3648,6 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, * Collect pages to be removed on list without dropping lock */ while (min_count < persistent_huge_pages(h)) { - struct folio *folio; - folio = remove_pool_hugetlb_folio(h, nodes_allowed, 0); if (!folio) break; From patchwork Tue Sep 5 21:44:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13375112 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 017D5CA1016 for ; Tue, 5 Sep 2023 21:45:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9F3888E0013; Tue, 5 Sep 2023 17:45:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 97E6D8E0012; Tue, 5 Sep 2023 17:45:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E3D88E0013; Tue, 5 Sep 2023 17:45:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 471828E0012 for ; Tue, 5 Sep 2023 17:45:17 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 14ADE40A8B for ; Tue, 5 Sep 2023 21:45:17 +0000 (UTC) X-FDA: 81203875074.13.E07FE79 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf30.hostedemail.com (Postfix) with ESMTP id 883C48001B for ; Tue, 5 Sep 2023 21:45:13 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b="PHeGAo/B"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=xSWN3PSz; spf=pass (imf30.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693950313; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/QCcumSTTIVe6iwff77MVSYTnNhMPc0aiSwQbQ0i2e0=; b=BvNJTm1S0zKOb/EX417A9XQ+hyyVKrB7mqEhTqekVft9XZCrbGpTr5NKRo7VDAX40p7QNl aztn+9Xekts9XPRIVvidRPH7+uL9Wd20uAyLJZ7ylEg1Or2YVtXHaPg6AS7KY8lj+bC3es p+xisgQuhueuKNwyKxAXuuvNnly55mE= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1693950313; a=rsa-sha256; cv=pass; b=AYtLjmP1pS9L+08QHpEzJek7lqGIZhcC2Bb/uyRcPhQNuQvZ7qrKtTsqnjBHEYjhCg9p/a AtpYq9rWBiCUKK4BNCI8yPOWJyziqnXY7tmF60s4pUwICLnsX21mhmlNwkDV7YR/xSiMfq x1WhJVKUFzVlMPBAB3RJLnyEFEnZ8AU= ARC-Authentication-Results: i=2; imf30.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b="PHeGAo/B"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=xSWN3PSz; spf=pass (imf30.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 385Le1Vl020747; Tue, 5 Sep 2023 21:44:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=/QCcumSTTIVe6iwff77MVSYTnNhMPc0aiSwQbQ0i2e0=; b=PHeGAo/BUfvzfdEC6WXZvOlGru5zcQoHcdFHigcYEZB7w7l0AUlfmcQIHFR/Cp0X/88U m1OdES3Cu7yfFEBsEljAFH5pdwV2252vKErLBtHm1cpacAyDYByHybZbacLDM/meY3DS xGOkjOd45qjJdmmh8QlwPTSSFLH2DjPr054Q1VW92QEDF9Iyj/mgDt3P8CPgk3ueXHhf Lzq0v+Th6gbxMMIR07O3H0uvovB1jzJ4bJXXo7AsmwoPPaJGDQqtEG7Cl3f0VmYB5pPY GHhUezNgW+txVWGQp4f6LOcbwwrzpLgqouRPRWN8sgXpUiB8XQTMe7fNEQy1KLdk2vWC qA== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sxcem807c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:44:41 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 385KMTvF006626; Tue, 5 Sep 2023 21:44:40 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2045.outbound.protection.outlook.com [104.47.66.45]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3suug5u9a3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:44:40 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Teku9pawNoHPPbNZLS5wfxOuakd7GWHeoeq+lzed7emcdTY+Kz/dx8rzPzt1xvPbser/+gbsXUeDC4ifigO8kaZI/7gciqbJ2B7UJk4+pe1VqsD2GVimq5dTKBlpqlhJ3j9utYg94RGebW7LXi20Wmi0SM1FGUhCxeCAwdH5YQxC2y6ub1IE32CGzzY0Eq6hqUlvF3nM6UJ4Bhhmc3xo7JPfIcyeFwLLcjCGgpaNMMjl7CyLUhNlvnGYAiXWGf94vu4ivD13Gy5cX+Ddnmh2ZzQD/kQ/PmbSy/F0LaF2Wtv0abCdSTjx+Q1x6dWSpc7aYuG50JrgMRBiZkUvm1UgfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/QCcumSTTIVe6iwff77MVSYTnNhMPc0aiSwQbQ0i2e0=; b=EZrd7y7SSDSH/ktjCq2+RGAJKjhjdhFz5+sJWxp3e/phkr/x/jxLTHX1tFhOnFebJOZ/Du1ASPG3YF+ktMeEfoN6WqF7eJmDDjw0lEXNKj7NhmQUvZkn8vr9LNRI3IUT9esqkQs/kY/Bkypkz7BTVprcP7jvO1qLKpLhDf+Yc/2jPVzbMPrT6LJbhnXavG846vd4BmTU2RFGo2ZZ1AreAGC2lyEbhrI6OcSRwKRUF+Fam41LJ/j9kuzWsYFDoiKsIbK/o3bhIKfaXMSk3Rxo5KHxOXGRccvE0krzNw8FFlqwWD/GHF3PCVhSRiqwFAwWiVxKSI8xHdt9PdxCAPZ3/w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/QCcumSTTIVe6iwff77MVSYTnNhMPc0aiSwQbQ0i2e0=; b=xSWN3PSzeoWC9YBxMV+QQnYTzjPUQK2MV3KltIc4dvm1gxxB7ycApG1ZlalBKIeZ2+kQb1hD0FmvJAKAlTJMulVuq21KbS3KNr30Sa2b0LIsoOZgNCJJqu4BzpEjkKu718fyxW+4/aBKoNaUer0v1Sdpb9zMURFV6Ow++0z7qyc= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB6395.namprd10.prod.outlook.com (2603:10b6:303:1ea::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.33; Tue, 5 Sep 2023 21:44:38 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715%3]) with mapi id 15.20.6745.030; Tue, 5 Sep 2023 21:44:38 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v2 06/11] hugetlb: perform vmemmap optimization on a list of pages Date: Tue, 5 Sep 2023 14:44:05 -0700 Message-ID: <20230905214412.89152-7-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230905214412.89152-1-mike.kravetz@oracle.com> References: <20230905214412.89152-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MWH0EPF00056D0E.namprd21.prod.outlook.com (2603:10b6:30f:fff2:0:1:0:12) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB6395:EE_ X-MS-Office365-Filtering-Correlation-Id: 8aab66e9-b30e-4c54-95eb-08dbae594d8a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: F/JrOFE92/y6R3qVjFKG7eoTkPAcJQ4yeGvZTHKHOqPqlFtoGGlPv00X5cUgiVvtR7hp71njZxFwRXIaxTOrle0XBvaNgElSuJsbX7ZP4/w40IHU1/I40I4zJOpI7h0a+PbfbCTvGsvM6wpo9kM6juVpB4HMpBzj0j/RXbQ/ORoVYMcYrwhHhwvvSatt/Nd+miiJTxX0w52ZF/KX34bhQb6s9VZESJ9mMsjlnzsiH49fX9uwNAv1QDpZDLaIKi2LW9f/n2E0IKXu6hJI6JJXuMN/Fg/sio6HdunyxRHy7h2bPa2yixxUJkHZoliNM33hucadFdKUkKO+Fxp3NslNaegD5jJu+1k1LcHhaXcNPSn8FBpRgOiqgoBn7XlpZcsjx5ewHk1KlNOzL+Y2A010y/ANpzQh2lY/JVtyjsk89FYs/z0YQiI6Xs7VHWUAcBdb4AaDeWwvEi/R31hQlzpSpsvJN49TMhBzqDO72YxNu8iOr8zrxz9JuZ4r3wUzOYZnxrWn0DQjsJYuXAQwUwc7tPhVots/6RPGuu7FCNwda0OYSLB2jfssqGT55WzIe8Ee X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(376002)(366004)(39860400002)(396003)(136003)(186009)(1800799009)(451199024)(6666004)(6486002)(6512007)(6506007)(478600001)(38100700002)(2906002)(7416002)(107886003)(1076003)(2616005)(26005)(83380400001)(86362001)(36756003)(66946007)(5660300002)(54906003)(66476007)(66556008)(316002)(44832011)(8936002)(8676002)(4326008)(41300700001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 8eoITRbrIOfpfXi1aWgOcy6CDdQXndHuGnijwrXeFUougMfDG0ju11SNSTu6I19W4EToCO1n0TNoZq0xBENuNFPKYh2qMEoynrDP33y746uvmENTDct3BfmDTEWA0cGJEULo+PdtKYGBrnVfMc0RcIA21YjE2GR/gF1A/1QmdcU5cz+EpUuHjUybE60FWkQiv+DOFuXJRfoHKF1ILDThNuSgaj4Iu0vG0NL1OrCf7+daGGafcGc5/O8Oe/hQa4vaf1WJC26rLbkQdoT4TYl7Sf/lG+sjAueV/Gq+cU850crxLiwRHrNiR5LdU5ohXECXHaosONMpxKc4gg5gLA5d/RWYGIbyyNzAj4dnh0xVFl5Twh7qLcQIsCaty/wnyZZlV+8bLyoRPgO6H/Ov7KXztbQZKyCpXi6OUOS0okLn/HHNwFvvx7RDyI0dU21WFHon+n4mO2mFc4/vSVaupXqT8XZHbd+YK5O4UGAsgGM7J84IFzN+fDlktWYaoqU6Zq/stJTkZJlEi1P00+wdoANEBhbCHpH838/QuQUEdn5egWqpkkOzB/QPGS8VlRplePndtHFjoTJtD+k4Vz71Hpp3F/VnvHZiiyD32x0AqoBPO322qBIxSMu+p5U6XR1vdZX3EWPXF8nczk78fqHl8l+KqconP/98ViKvuj5dKI9hJwA5ut7DbJ+119x+7qHH1ed2P9BPTHb68ByCNkOcXkzR3/JNvkHjZ8JqcFbfhsvDczgT8sKffEejl/rh80s9f7XwhDs/r8uLv0DqACdrhKbFB9U1uJzJarjxGoqoKiQ5Ed9m0p6WWfHji5yBW3IPRifKgyHPr2iuHAKpmguYCikLkqfx/c7VCfdyKyT37T/1BrJJBxWM577KPdfWhp6Fx4IWZNasvzT1ob1QG14gAwc54zy9SYcW4V1Hd8VVWS4cHf5Ql8nsm/DOI0Yz8Gq6Yf+XDQBB4QYplDCdLFEIEcQ+wQNMwUzh/LpScmnaLH/a8xMojCC2B7qc1iEh2kYdAsEdbps3xthVMyXMYfJAWAKjY2ZZiB3mb+mGEFsziUCxvM9LzIKGsKyPBYa4l9OFqgC80ab90f7HfyYyevv8B27Yx58bkrHbJIcqfoAMI58AEtBTXOsggDO9ZS75qdNS0Z7s9OXt0yTu2IZqtDJNNUqcabW2R07CP5cb1AlHWZeE/3NVvxPXoekdR+mzWt5sFcikJ+S+hyb2jTUPyJKzzcQMyBK2TBOs6PIFfuxp3+rDArdl6bHHWxHz0MdeBp8yznLiu86wEpZk0HKCYnTboDtsBTVDFKbfxh+GezMR2/zCA2fBTemE56+5MlEjOUAa6m4vld9nUN1YFAaVf1CDF0jUWPNOFZFxn78UFUcG8VN5mSMrmJjFO0Ti1zlXOPiK3Ip9/WuRqHlOuRTtGZ/d319AEAhtoivyWUbt4tOrYpPgL4tYcEUxV/q6a17c30MMsJah+24TdPkQRW/rT1cTNz5cQ9uw60wGPT/HRIAiJEGEZbVR6M4sYp0nW8dsDw4CPZ4ZcgxyqmXnh0mj5ztrHkdpVOadrPUi3o8LT1g2qcyKI84SQuu9rrBMvJNFNAVZRsVOk0iZ1htKTVhlIUyO5/HgDQ== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 3InvulxbifxhVpx65aDDyQGX6vxAI2x5oGsl7eqIOXjkDboRP19YoIWYly0/g1HHDOFtFbYblRG1yZDChm01Cjr5ThuKTbG1vLjKWXggIQPKbkKtDuxZ7Wcmja08obUr+Suo/yCCa7OlMnzXV6gNleWQ564E4wMSRz8I9C8XqjjaXoRASpHj/uqb1+8WIw+CWlX+qZspr2S//fZhlDNN9TGBcq9N7mwaiqv4RL9p/DedBwRQXp7MrbM0TT32k3iUHCkYKUANfX0ROnhb2jkQ3psgQOvbI5+/HWcBzi5bERh/f4mYSOAzk8tKm9drcBTxwt95fGf2KIXExLNBA6iNpYDhBS2HDJXZFT+yKaTpuR/xGWGj2A9jC0uUJGbEdrj3c2k4BicYaGSCWN9YKEDfZGW1HcZ1FGCXbRXr6b45q/FNQKgiPJOOsQnAg2ssO0+gEF3GJeJEhf3/veTfq9VfbDjcPtWbeiF6mIZolm1uNv7pmuz4PJ7S/Vjf4yvWxb1FwthLYJooBOg2lq5jJ1kkeGQUUX9OR38iMnot8YpuDARw9atg9HF6d6aJ9g38Pz55m/cEvncqpYvmmfmKqCbFaR0DXD2p/fddn28jb/lf1nshgaOTQGTvzv7UYqM4W8zURRNYAWORYT017+/IVqRscGlMLBamfLseNhi2VbFUmS4qtW1ap5sUtc+ZBESqK6cAN789H2cFYUKDRCGmGHh8ocRkDNwLnITE4IiguPnPsPwOcbO8hTMhEG0+eeJdSjXOX9305NwB6Pb+AleFNpw74AdxMgiVEb6CJ7kDow0Nf7hVrkncMydgg4q89tOn1fmS7Yt0W/ROYH0VhaYriROZbOPN0gCc+mi684vAgocvsATa0aZQecRfq2UpE4jXUXWUcegiZfQwwzHJNGtl9dwyOlLVQrvWcdkhxEmjY7zItCuWJVNpifJKDLc5UJyx9cKg1p1C6cV9XexuTn23fAX5Ieesjh4XGgHlSSTmGy2u8kDP9fFeHFL9MqryCT5DGx2sdTpswvIEUp//ZLtpdnB85JsKfekvJWBRGwWlXaZl3ZgCEVCX7Lj3T78KZhNVkznT4/YEa7cNsMsLnr3SYmalApzpggUz3TppHCoUkIlMfxIN+IJUYVV5W379kAsVnwWw X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8aab66e9-b30e-4c54-95eb-08dbae594d8a X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Sep 2023 21:44:38.0931 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XQwZrMWiLBro5bYwl98TwlpYCr9xyUhHc7Dunot+jAHBpOzQuZGtxQiQwPa8VfjRSr9PBsM6DGsDGKhsD0q8yg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB6395 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-05_13,2023-09-05_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 phishscore=0 bulkscore=0 adultscore=0 suspectscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309050189 X-Proofpoint-GUID: RJDh7pq58084jrsK0vboPqbAWoc1unDB X-Proofpoint-ORIG-GUID: RJDh7pq58084jrsK0vboPqbAWoc1unDB X-Rspamd-Queue-Id: 883C48001B X-Rspam-User: X-Stat-Signature: z6epynup4ma7x5s78ry56z49h4dzd5bw X-Rspamd-Server: rspam03 X-HE-Tag: 1693950313-398943 X-HE-Meta: U2FsdGVkX18DMKueLbkSVFqct2H+A+O1djRFHKfm+iGt/FovnkDeVFR6ZWtjDjio6vXa0+GDk2j6egNJmLH/HcJ8ZYk01XvDUI4IK9Gr/McCJWWhDOuB5pCSthBLmThJm1KRM5EbzaJo3E0umo3a8konp/TRJN4bUvfiWZIbx7x03hMzz/tOTU3QOhrkEPDtbt86NbTfE5k/R3e4HOtORPWmu6HK4GpQ/+iLmNWcn7OqFAw82wLMWDm/aFVFLwDYM8H9AnwXQsmmCRHOzEtK9+RG9+KNuzZ0MqN/mg//CprC5yK9ejZ+GxUwpl91RCiLm7Ezwg7vcHd0jz1xRBtD8Ups1Vj7iu1wl55tTtzwAAse7qtor0SPstFwAsQkwPlk/tts1yfaoZQiHUhtm2SmqdDICqd3KUXv7IPglbJ86yLN/V8oMWrWmFHo6ubvba/2phrU1/UeSS02cATfV/SddAX2Spxs4N+agjkyseid81xBeOmek1Duzjy6TCOK4KGNvyEcRAT8lJQRhuuwVFj9qdI2ZB2A85RTwJM+Ao0z7bT3zFPwF5B6bAnoGRG2puCQYKQlRrI2Lq3hfAK4JKXnyICEgGfsyyKX51C0P2obgA6E9e/OOrtGLEwR3obodo4u/iEPS1z6A6VfGBB26QyUIynGeHN420SeCQLfA18Qz4eVM9BiFkdqFtzCy6JeHVScbR36zVUNpuBLwv4Dx3NJIP2/cd8ivQ/299wBXd69Os5kVUaAMJYzwec6y1I0QNGmqNLc8AnoIumzy7u/tNgdWkL+A9OyjP3QdwBDvLXuhSVEVpf5SbtpIajxV0rKk+jNdTIcmTDEgjJ0JdbGpxvIIJEYVqkWdf4+GtN9r/mmncJMtZbEaPwWK1yF4kA0fwA9RN2ZNM0loO9tSYFJXLOsccMuNh0WhXlBlAt4myU5FRHMmjep4MkdhwXOt9jauP3R6AdtUmbs4j7ppJXPfOg pckzBtID +7fybWyxjnPiK0zOaNz7LdEodig5mQnGQsoebpYRFdUw9ngg5VwotDosGpUzveIbaDRUthPPy+lFysi+7hkknwT5sjYb6trPSWmqzXf6caQ2xlmRLFAFNPbxvsXXNU8hVXXjgW5vjmIO4QFXAwUxgyevkF2YyEBJbzxGUWWIgstP+ELGkUI11T09o7NTm3A8eJZvXqXrTmJlZP9fRBF3cP5MICo4L/ZWEb8A3bUtmibPi06jflOSQCZGkJHWgDq+G2I4AKpQd8dqyN7WjlupbCA55NgJFNlGOoGCZRUdQwei09EHvP9hx3o77XboO9Z67qj85UuQyTNquxRFAOfV1IVOKgmE6R/2Jd8J3Wb+6+jtBkY1ePfk0qpLkEY7XMhxmZmIQSgz/mmrJiFQwyx1Rt3II8nb1f7GQUM19U1YABgHDVuQKU8iVsgzm0ekMG2+DEioZr4tGpKTozuhl/L9egHF/cqm210SZ8qwA1jNOANNywSoWr3E3CG8m7+Rs4kS2O2pd X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When adding hugetlb pages to the pool, we first create a list of the allocated pages before adding to the pool. Pass this list of pages to a new routine hugetlb_vmemmap_optimize_folios() for vmemmap optimization. We also modify the routine vmemmap_should_optimize() to check for pages that are already optimized. There are code paths that might request vmemmap optimization twice and we want to make sure this is not attempted. Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song --- mm/hugetlb.c | 5 +++++ mm/hugetlb_vmemmap.c | 11 +++++++++++ mm/hugetlb_vmemmap.h | 5 +++++ 3 files changed, 21 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index d1950b726346..554be94b07bd 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2230,6 +2230,11 @@ static void prep_and_add_allocated_folios(struct hstate *h, { struct folio *folio, *tmp_f; + /* + * Send list for bulk vmemmap optimization processing + */ + hugetlb_vmemmap_optimize_folios(h, folio_list); + /* * Add all new pool pages to free lists in one lock cycle */ diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index aeb7dd889eee..ac5577d372fe 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -484,6 +484,9 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) { + if (HPageVmemmapOptimized((struct page *)head)) + return false; + if (!READ_ONCE(vmemmap_optimize_enabled)) return false; @@ -573,6 +576,14 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) SetHPageVmemmapOptimized(head); } +void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) +{ + struct folio *folio; + + list_for_each_entry(folio, folio_list, lru) + hugetlb_vmemmap_optimize(h, &folio->page); +} + static struct ctl_table hugetlb_vmemmap_sysctls[] = { { .procname = "hugetlb_optimize_vmemmap", diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 25bd0e002431..036494e040ca 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -13,6 +13,7 @@ #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); +void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list); /* * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See @@ -47,6 +48,10 @@ static inline void hugetlb_vmemmap_optimize(const struct hstate *h, struct page { } +static inline void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) +{ +} + static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate *h) { return 0; From patchwork Tue Sep 5 21:44:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13375122 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2219CA1019 for ; Tue, 5 Sep 2023 21:52:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0CFA3900009; Tue, 5 Sep 2023 17:52:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 07FEC8E0014; Tue, 5 Sep 2023 17:52:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DED70900009; Tue, 5 Sep 2023 17:52:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id CC3718E0014 for ; Tue, 5 Sep 2023 17:52:38 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 9045D120B12 for ; Tue, 5 Sep 2023 21:52:38 +0000 (UTC) X-FDA: 81203893596.28.ED52F58 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf12.hostedemail.com (Postfix) with ESMTP id 895EA40006 for ; Tue, 5 Sep 2023 21:52:34 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=JUkq20M9; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=xaRoojgL; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf12.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693950754; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=e7JmAFpN0aQBgHF7jZBKMPSyY+ViT4J/qMJefAZX8As=; b=A0QNKFk0qDKU+TICDMfhJAmqzvgusFPrMR8taDf2yUv2dTo4UENNW+MynewEIB1M/PSWJB IUBxNugVRWnKl4v8uP5HmdUtwEOrX1cqBvrxJWc+OtVyXjfNXg3Fzpj++9RmjiQcnfqAb5 /I7DoDrRwcwOQRFoBsr3MVRVGpvXfS4= ARC-Authentication-Results: i=2; imf12.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=JUkq20M9; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=xaRoojgL; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf12.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1693950754; a=rsa-sha256; cv=pass; b=3IxX/Yk3aDl5kdfjmhMHa5YN07pU0jvOEXiMBSEYX6cItRPhe0PgYfmjYMM0GQWFOYxPHt t+uCuEK9qVoQpYnk5LjvDwLXYSpHjKqpxR6R1xowyUkUmJNvnVSkwl9VMjuEKiiFYwSSca MFMMgaPLu9Y22HL8UNF3DUXd+B0mGO8= Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 385Le64a021124; Tue, 5 Sep 2023 21:52:04 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=e7JmAFpN0aQBgHF7jZBKMPSyY+ViT4J/qMJefAZX8As=; b=JUkq20M9Z/PG85y5aDNZGaWkSiQvvjvyKilvjul+qewCY5KBOWJAliAtFpcTjyIQRRfG XGGu7KFj0Tiu3eNp1oI3JYoGxTFLSwdZ9VirdYFi/sEjeeW8ksLQzlgxLqg/SrICeXcK X/AcOi5bsPPyuNoGkP6C0Cea6f3pwIvQouUGb1uqr5XqcwiFVSl6oGctZpnVE3zOmTIV 5DWqrfIT4HfYb9T0HsXb5NuOO5i8iVrY2/KqYUMSET9SdPiY+ZEjS1rtmkpve3gcGfI3 R2T/wepGB/bHYBn80QMoL9h0s3k0EWhEppGbCkidMnojbtZmcrWSdAzpltlyBc0YORK7 6g== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sxcem80mg-8 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:52:03 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 385KHWWC006563; Tue, 5 Sep 2023 21:44:44 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2043.outbound.protection.outlook.com [104.47.66.43]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3suug5u9b4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:44:43 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gur55vx/uX1sroM709/Xjjk8AafjVoWaCGEiGducSlpPXxcoDyyjn7j387DFqXfb/9ZoKt1TQJj8AyYHyJ1Xqhx9s0C2kDj65v1y1KI4kivov88Gc9rxunfUKRt6oTuhDrWw3HkLZw2dErjKjQYQgr6fvf4AR+mqLB1UPlknxuA6IAzyVZqTc6puXo+9q/80mfQGjaKtKQjNVG9EgnpiGpbBGLKlFIJtLpVsFnZjD/SmIQylcFleDMe3oNkJgwHcNAZLIg5X3s03JSW3XbtkN2winIaHuoDan3442IMsQiVUPTnfJZ0L7u7Mngi1jciCg/xZBRhGX9N5A/wKaxevqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=e7JmAFpN0aQBgHF7jZBKMPSyY+ViT4J/qMJefAZX8As=; b=YuoithHQh9L2A4+kkWyX70ou/NyflIK8tuh0+4FOkgAO61ojATXcp9FnbZxLGLANWirx+XIDL3PQAxyvnIL2t+TAZ7mjV5pRYHw4Y7oXLePNaO6VnHVHhXsIgTdjIphtEpHAPE7fOcLBxxRpu0lmp6Am2bh0xiOaw7Vm6BueFo6WTYR2eMn869KtHYRIWp2pFbRh3YEmwLr8RrmeL8n252wuX+JJEjrqe/tVyjy1qUzPOw/BNeggpw4rmIG5WSeFldCuB6DbUBqC35R7kjHe/h+7anTmi4zVM64vNX0e3gZV4q4CQ6FYVrHD/WrFt03pb6JwFm9vTOZIP0j22qKmBA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=e7JmAFpN0aQBgHF7jZBKMPSyY+ViT4J/qMJefAZX8As=; b=xaRoojgLm49ElUTnzOHjmNO+tdawBGRiNwBjmH7hqRn2Bdrib4yIsW548oKkoDmh59J1cd5sodCpM8eKDqFIfiokQkbiE7sBEhNrPZcsnZBeKclEwUFT64vuRgioHSCeCnY8MmIeYh8OrNg2BL2G8KqIZ/Sm2RGtmqUAYtW0StM= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB6395.namprd10.prod.outlook.com (2603:10b6:303:1ea::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.33; Tue, 5 Sep 2023 21:44:41 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715%3]) with mapi id 15.20.6745.030; Tue, 5 Sep 2023 21:44:41 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v2 07/11] hugetlb: perform vmemmap restoration on a list of pages Date: Tue, 5 Sep 2023 14:44:06 -0700 Message-ID: <20230905214412.89152-8-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230905214412.89152-1-mike.kravetz@oracle.com> References: <20230905214412.89152-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0148.namprd03.prod.outlook.com (2603:10b6:303:8c::33) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB6395:EE_ X-MS-Office365-Filtering-Correlation-Id: 7cf5c1d3-30a9-4357-6258-08dbae594fb8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: g9KgQ8y7zZjaHL5vIn/DfUJrU2V80EFDrVY91ttiGaQY2UMsjUPztg2uSHfzr3tf9YVEjFis2afwvNJJGkhspIM/ssiWrfcZt8vuYK1U4z8Gs8Jg8XD9TlDvxnDwPQqDsrtfRxVk5y+UPJivEp5qV/oVT2xBmkSUxHGxB7MMzg3HqZYzq6F5AxH0GxUtyf/SnKp9o4cGQo/BD2cZ+OtVlQiRTt9nTzrODfn6DxZtNbFJcKlJbg4ZvaEVEh/IkcsEcIf+vXT85x/Xk5XxEFQSOonlrtSwnNCWiuS8IES48OaXYWbBnUNPofCpGDIDwjpNUsHyJP+tVoPcR2Sy+ks8G1SELEsvEsPvCBbKpXHEGdVyf7/viwgYShUOK8PwlWuhEXJcMh4dhfGoAIhoA6ZWDzURlPqEN1zqN8oLBEmqASluCpI/dxQcUiMtAO0YVmWvipum0yi9VdXd1UvR3nVGdXJgEwsHSGxyHxoRmi0FTWKtGxSZe5gkFxNAZQLniZG0sVYGIyudVsRtAYMczyHU8Vl9P4gtPoIThBKsUNjbWvyTfSv5QXBHTill82TXxxPD X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(376002)(366004)(39860400002)(396003)(136003)(186009)(1800799009)(451199024)(6666004)(6486002)(6512007)(6506007)(478600001)(38100700002)(2906002)(7416002)(107886003)(1076003)(2616005)(26005)(83380400001)(86362001)(36756003)(66946007)(5660300002)(54906003)(66476007)(66556008)(316002)(44832011)(8936002)(8676002)(4326008)(41300700001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Py39vyL4NOo4znmyxkrIZmc25O5FXwqheGlQb+iBn3eHQl7fzeSjAUvcvDvy/Sm6DaCa/lOo0Gfi4QkRg8s1aw6dDxJ0CxGuy+B1tKmMlEGjrbDihrHM5EL90k74GKMHMGpS3T7s/NMPtRG7mD2K63lu5lrcu6NLrvICiQzvFHrZv5Y3oo1plH9mQ/hju1nHJ31zLZaKlruwvrLyIkOSIHpNN0WMtaxlaIYoaOputnz6otETuN5m2En9qY4PVuoLpBHaYomcAxH9C6PpRq+5gMYdxLlMr1tLC2zA7cIFdVKRtq7uylomT+qd6oTJtsMr671N81lyoKWwfj0Vz8AGpUJAfOhF9/UIywihiA67+ne6dEdoetImJmJC2YPPEy2dcgeXy2bflaoKa2nbaJQpB+vW4qG8nhGmZtX7yXDRmyoTYNA36wTJOX5VKHJzB0po6aqdjMnTbE4VDEaEXrzBXJ6W570Mx7RpVX/eZ9mnOlcKOrclhYaRzv8eXhWBZLkhYaTxyEiMtsQRKMj/BRlEdPLpnR78y95aeqEkDtlX42+jvx+Pfs3isYMLe56pTGeUhWks8YSJ0i1wz8rlAquxQ4yY4McmLrRqCBh33PJ4zxZ1oghn32sjwtrQtRoO7trYTYLb+JQ9mRUFXsJk/FEPqdncp8nAHo/EFs+kOjlDYadbsVUw4/cp6RIS0/MdRye8ABcLq4hVxbEC625yNois+N6rW3FgwT5yHQW10I+FwSPRUiBShPfq9h1wcv9gvW2ZC+2Xv1ixMG/k5lYUeiH5z35kUnWLLcY8Whqd/bNFO8uENWW7gM4e1MFJNrPtk/q4mD3wcKJYNa3goEMfk3pF409af6cn+iUa2Cz+YO/gLu7YRXsy4p78H7KHlGWKFCvwf8A61f1X66iSayLGplLZ0g6ejVVHl1WeysZ/+AfmyASz+njLZiKisR8M/F4vqWwTT3FQ6MPXWZOJAJPqAUWjeWayLquy/D49XaprC5YEp+j4zF3KqNEW77GOhat6JLcKZLuiTK/hDzdynV+lJ7yFyDXezsINDfLsMBQZvVnyY1lgVJLTnua8OBlqbeNHnOVQZh/d15iLq3SFEn/QPO0CLwfvp9hZ5kEKSUVXbh5caIFino7CAqa3WZQKHSOSGHILZn2cHUdbQY8CXcyo5f49m14TgLa/AhZNc1icpwutQZdr/kdsbFL4dfkMpPQE2LnmIl/IJWO3dgzUXwg0xw0bnNjR6Y4qgjxCWdrp8xKE6srcHjxfNgF3qiaKPhbFfsoeqhISRRM8OOGpZCJs04jlq/+CisgAsx1vKeP+p+dZ6to2yyfmctrgxuSASfVRyIz1UyuqoEKX1lZeC60kGgc41Y9cdDQ+1U98ky8u20Rk+A8eLSB53d3eqNjp50FdE6kB62nNGrlawBTbvxGCu7xAnXM5OYcSFbKhkLS4EnfFhwdMHEi5yETT7232FW/GkeoVwQi4GKdqiFK1i/lP96vhwBT90OLZRlU2ZMRirIONvGxkiD8VkdhWhQumH3M3uVJc6jgduxAFVZdOMWQPloY2HGGP62n+iydzugjYnvk+ECV+Gfo/ApSqkiY0PzG7UER7mlx1IiXtO7Lk3DjA9MCaOg== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 9jb6KEefcsU69GM5vdb26/qZwbV4lcp/URyWMRKO5Q1th+xXuSma/Kc2gGU2a+l7CByfF7XMBPVd2QL2pybHPrjuHcYkMvoq4vhrWQNEvQDitspLXtc2W3UuyL/GDehApbk7udecBWOFWPAYHIItJwNZLkmvp43GXwPPMPzQPUsqT4SuwpuwmRHggAx8mV5A1KTX81h89JkcaBPcOGGyar+YnF1lwpJWewejNZn64E1Ccuz95LL4lKyd/vxOvctBJHYiQrASTj0fKI7X6/kfAc4OgxZk9TQMDDng6hzdjiwXeWBkB26sBVnKZYlfYFgsDN3cN9BFqp0l7ByA81L4XfqgxQgfZ4NB5Rkc13ukuKmh5fLqWpKxTtSeNdO/4VatXoeYxzM/cebeMBDiSlLHli0snB3udM1ZDjqtnOOkbANeRLQHFRPQK74cNbrhAN3gogiIFN/RMrCMuqq8o6WdAYGvXNqkL2S8UKRxj2vxEU6a2SQrVKq78Y6+2j1T+QfnptoiSnXYtBl/ZaWkYVk+5O2yVS1YcC+ccxNsOmW1cWjgXvlpE6sOziRUjN6LnFBA7FWS0xxkBs9UC3+NvhJWu01mXz3IHwe/3bkxITM2cO7AKvz7FB5bE2yKCnBuSq9E6jsL5ZSN+mQPsX034NGtlFxOj+SfNr9NmaYHCSMDSToI06SXGHQ0C8+2nn3kQzE6IV37z0vRqYbUoYNXJDwSQAfklNTZBAu4FgQgMFHBdLWn+q5n2R01xd6NV8V1r5oWD1B+GKNUVzHLYizxuroEXdw+N4Qz8zeK2y7f9cwPk0PbZT74zggttA04MFR7aPlF8f+d75z3F8lSeBNWUwTXXqSTMWHxsXn3xt94/iwr9V3VCt7AJZfHdA5oEvLKCAIkndKr0t1582fjWmIqFaMwLKmIoDhqUFeSWZUzkStmz6yYJLbk8SV8kFp/8XJ/fE3+BjYgaEGB0Ew6lV8fDP9ezRAnHKm5vvP2Q43fk6A3HKZrtFaNDw046L5ZIWtsKHGgi71qSAVyX05l5jn4A1oowPS7K0TNeRXFMSrJoT8c8IbzDq/Kf/+HUcWwd4xTz2+KL88BsEwhfiKlg/L/ypes8xIrO6kHTLHCfR+1TqQnN/u7POAxNj6nBDdxb1e67ZL2 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7cf5c1d3-30a9-4357-6258-08dbae594fb8 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Sep 2023 21:44:41.7277 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 2Evl8rM0NiprtqoxV4BYPYMXVuXRgcOvDLr/lBwGX+CMtMONLGIEZHc1ihZ+yPxArLCpHZCIlXeTsJje2bERHg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB6395 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-05_13,2023-09-05_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 phishscore=0 bulkscore=0 adultscore=0 suspectscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309050189 X-Proofpoint-GUID: 4O-z_C99ESvs9HZN75xT3N3kckjoy8SA X-Proofpoint-ORIG-GUID: 4O-z_C99ESvs9HZN75xT3N3kckjoy8SA X-Rspam-User: X-Stat-Signature: f7mbs5ktprddzzqga3foak3zczdih89h X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 895EA40006 X-HE-Tag: 1693950754-968204 X-HE-Meta: U2FsdGVkX18cQ3NIGLNnhQaCBEl1u7v2KO8ybBRGs4OfVzlLkYpYlZsLopwiRaAHhp+MaPwrheehVl1mXGdfY4pLwYaacLZSBXkf5gRohiBhgC5QXh6ysU0hvlyY9oedH8gKXFNkbj/1Rq/BnAdjKldDIXHhNyZE9h+2t3p1sgJ+CUjqBXUsD/phrRdXqQsE6c6pdKJ03uOEtf7cFK10T0IgdDyHxY6RqZMkCSLvpkZ4PsHxseIpOOFoCxOPC7jNPDBi1y5ADZLFrbGBA4lqnwb2MCWZCDQxt6TeEw8OfNa974SZxpOIqVIuSfDj6t/hF1twp4icWYtQ3lU2AR3aB3NBVjOYtkhl+dvTL6WJGYXs5ZJ+dr4UH2gJjEg3CBhleTW5tmIPmkl+oR1wJp30nB/ye7sZ5mqlYWCRBgKgs0I9RAyrcxO6hi6cSDy5C9W2smcImUFAx6Ma4tdOl4+PDy/lyIiQsey/22N9e+zZJudazaRkJwMrXIAjWy05fnyHYwQw88nsqUV1LEIG9gQn80y+TRbNnfhGoFsV0ejyuuqcwk/SQMdCJ+wFW//ADqbCV7SWxME6Bz9w77xynWKdKDe2yjQxB3XBjSaqw0d2QT9jYhhlhdk9VKYvh1AYH9KInuSBNImMGP+HugBKGd/hdZ/n1yB1jUCP2Ha8+FBAnNClWVGSAAMdbJE6tllZIbgvopKOqqzbMx8lAQ37os1O54allvQwH4XTxE0xEMyMA/ye1smQh84jP9Xmg+OMsBdru7rGl84OskWHcvbpC+RsJrr3Pk8U8SImkYyW6+A5dls+EiPnPC8y51JuxOc5ax6VtmYZH4QVcrksMB0WBi0OCke+icXo89hGRbI0xXNmROd/UgdlWya0f7+VZtwvnE76oM3r6rs0FRZ9b6J9h4YN/TMTJ48xEj9SaWz4qFGJhMqO31kfhyK1SDi87sP1txOCGiCLMalk7VaDxiQvdER 2v1NABtd 62urPfWoyR4iDC0lo6KOyO/dVaN/a7SfHGWZYNF8pagz4HrhwI9HWLzH6ld4G3e1U4FDMwg62S9dgSP9h1Hv0HE7P1i9kXjXu6IkYCtD6qe5EYkAkJLhv+4EDRewXP8MeP+UaJ6PkqTky/RegGqda+j7l8kpt1hALb0tj6nsNmdmIS+9gHAvwYxi0JFPrIQzFt//ew1pH02H42BzG1Wjm/FD4rydctDgsGgQOKgjgcg5Kx4wrSJkkbgNWZdVTuaPSbPS3ozFeJGM+0emKn3N1QXyxjw1oHfbhkpRsm0STRq0+LyKzZY7GALFwJFn639+r5C5D6nV3HlPwLkESfqC0YLn8hU1iDoLv2cYPkzdK4AmJvNFWzQlOcYPy3KRkdzrdKY4Br8dKeLjHFlV6pYAS3uiOTUJbnvEYAUyMmVdsM3e7Gqph9QJ/oKRZskClS+EO+5v+TBbjyS7/mhgqRdgYaqgnRko8CnH5qu+A X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When removing hugetlb pages from the pool, we first create a list of removed pages and then free those pages back to low level allocators. Part of the 'freeing process' is to restore vmemmap for all base pages if necessary. Pass this list of pages to a new routine hugetlb_vmemmap_restore_folios() so that vmemmap restoration can be performed in bulk. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 3 +++ mm/hugetlb_vmemmap.c | 13 +++++++++++++ mm/hugetlb_vmemmap.h | 5 +++++ 3 files changed, 21 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 554be94b07bd..dd2dbc256172 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1838,6 +1838,9 @@ static void update_and_free_pages_bulk(struct hstate *h, struct list_head *list) { struct folio *folio, *t_folio; + /* First restore vmemmap for all pages on list. */ + hugetlb_vmemmap_restore_folios(h, list); + list_for_each_entry_safe(folio, t_folio, list, lru) { update_and_free_hugetlb_folio(h, folio, false); cond_resched(); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index ac5577d372fe..79de984919ef 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -481,6 +481,19 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) return ret; } +/* + * This function will attempt to resore vmemmap for a list of folios. There + * is no guarantee that restoration will be successful for all or any folios. + * This is used in bulk operations, and no feedback is given to the caller. + */ +void hugetlb_vmemmap_restore_folios(const struct hstate *h, struct list_head *folio_list) +{ + struct folio *folio; + + list_for_each_entry(folio, folio_list, lru) + (void)hugetlb_vmemmap_restore(h, &folio->page); +} + /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) { diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 036494e040ca..b4ee945dc1d4 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -12,6 +12,7 @@ #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); +void hugetlb_vmemmap_restore_folios(const struct hstate *h, struct list_head *folio_list); void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list); @@ -44,6 +45,10 @@ static inline int hugetlb_vmemmap_restore(const struct hstate *h, struct page *h return 0; } +static inline void hugetlb_vmemmap_restore_folios(const struct hstate *h, struct list_head *folio_list) +{ +} + static inline void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) { } From patchwork Tue Sep 5 21:44:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13375116 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B4A2CA1019 for ; Tue, 5 Sep 2023 21:45:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B614900005; Tue, 5 Sep 2023 17:45:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 167798E0014; Tue, 5 Sep 2023 17:45:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EAD77900005; Tue, 5 Sep 2023 17:45:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D60438E0014 for ; Tue, 5 Sep 2023 17:45:20 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9DF84140AEB for ; Tue, 5 Sep 2023 21:45:20 +0000 (UTC) X-FDA: 81203875200.04.BA2FE2A Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf06.hostedemail.com (Postfix) with ESMTP id 0EBF818002B for ; Tue, 5 Sep 2023 21:45:16 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b="JD/CLYXM"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=JBpISu3+; spf=pass (imf06.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693950317; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9OHDnOM7r2zXSFQchXv90fCms9LW5vnW7SnIblTgvg4=; b=mW1kRUKRT8bSeQAkPGzT+983J5Wxs8nK6KYQQWd4PS+i/zo96utl4Z1WJqCqeGUNGytF/c YRwwJmF04AX1MDjemFUSWrCfhkEyLRuwGXqq0RLDWF7QwRwZXffM4rOOt64ZJSlPONJrNC CyqX9XRdFmgCHYaXzxFWwD0bkvb689A= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1693950317; a=rsa-sha256; cv=pass; b=wxnNmZj0NUylmjvTe3fP80oH/NUaizFlQa9BLNhKVruOeWkM29rHHiq7Z6DENC4HSPggyZ CPTesibT29S3eguGJXX18zKbLFs3EZH9Ehm5ppm5KfAdK9K7fpJAuY24HVk4HtFHyq2Zj+ AE55FmBo3D2OpSOGzIk5L4LBwraWcXI= ARC-Authentication-Results: i=2; imf06.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b="JD/CLYXM"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=JBpISu3+; spf=pass (imf06.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 385KnUr0009073; Tue, 5 Sep 2023 21:44:47 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=9OHDnOM7r2zXSFQchXv90fCms9LW5vnW7SnIblTgvg4=; b=JD/CLYXMNcfjyU9zGUIRWuw289c8sZEP4xb83rcUg5XllDy+SUMxKNF1p2TxgmQeLNqM NePN9cjIlB1sRAs9nbAQv4W/pF4Byv3C3AzQritmPr7hU4jxaC+O5IIpfbBzOAR6MG66 VkBoV+Ma0b9TpNRuqUtETAtGynn/p9RhM/yTdH9/LD575QEe7Vh6Ftjk2PBHXmx/nPtd THvBcTqIB+yASpzauDM1gC24eoSz42rCTZeUljPIe2tgOpwhW94QU45GZ4vNtniPfp1o 7uzzJai9Ommf+sbx5DSlSkH5/XoTiCZz33lPgSDo6gWjYOm3URJcN/aV12tTet+0P9mq /w== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sxbq383ry-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:44:47 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 385Ju1RQ028130; Tue, 5 Sep 2023 21:44:46 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2047.outbound.protection.outlook.com [104.47.66.47]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3suug5bd40-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:44:46 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=C66doB5EX305ipq1I904tXWTrj6YEOf4hwvjfKfuYSeOAeheYrMLO2hCDzcxIkZPOED3kELtPnQcf0RY37q/herkQD+tuPRmhJ28bTFpcv1NejdIuJ8dISjAHtUIybjuxpzTQAYFDGgy7S26my+L4mK1hZznYYZUBC6qaQvduLq7U3Ez+c7Tmwww+5jKWX572ayPsc/KdEwv7nNeYeGWLZJoVluitxno3Nc1CYVDDsgxFvK9WqyCq9DJwPFsQ9LlLYBW9wX4ZhZmTLwfFlhgx0Wm8x0pjOKRcybg4JHkk5y/637N0G4/mW7x16kimCTrlQ9hVjFzREoHUw3TXCiWug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9OHDnOM7r2zXSFQchXv90fCms9LW5vnW7SnIblTgvg4=; b=kGLRpt0LjeTOsXwe+KtnQZKt8W9SKK+cvJdZ4XeriQpn0KoaIqyx3Gts9jPEEU5enbMvVUzcT70Iznq0lZ5Wq2eLWDWcRRUoG+q8k+nj91h7Htv5EXS8KpfBWCOWvpyspZFRf8Y4SUEORwI6na7YfgVZhT69rJDaRpb5eEPy6HYvqWMmAVlJo6lm7N28XBBQQVWhRLWajuPaLQpthRFrT5cVLkTB8bE0IfnWEvCMGH9qJqbeqHmfHkvhW9Wk4Ql/cXie/84Z7opWhW9nbqGwc/I8N6corneEHHDgfZGsEOTMxJEIM6kK6Wb0tZMWXyEoHpmTnQicA7sit9dw8/9vAw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9OHDnOM7r2zXSFQchXv90fCms9LW5vnW7SnIblTgvg4=; b=JBpISu3+fXqYxr2r7LawlV4hI25J0MMlxp5xXY0onI9+UJKlenpilEwAZQjWythjLjQpLW6p7bUJE2OJV8q89UouXTRR2RUOifSLr+69+y6mBln7G0XlydLJCJopLpf1wkLNu28/3TkBEOJs6zkxuiOTn0vG4TvpNUgaLrPg/BI= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB6395.namprd10.prod.outlook.com (2603:10b6:303:1ea::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.33; Tue, 5 Sep 2023 21:44:44 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715%3]) with mapi id 15.20.6745.030; Tue, 5 Sep 2023 21:44:44 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v2 08/11] hugetlb: batch freeing of vmemmap pages Date: Tue, 5 Sep 2023 14:44:07 -0700 Message-ID: <20230905214412.89152-9-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230905214412.89152-1-mike.kravetz@oracle.com> References: <20230905214412.89152-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MWH0EPF00056D13.namprd21.prod.outlook.com (2603:10b6:30f:fff2:0:1:0:16) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB6395:EE_ X-MS-Office365-Filtering-Correlation-Id: b300aa6c-3d1a-428f-ae49-08dbae59517f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: acLoCcx0XUDElInN06zaVDaXD9S3Gpl7HeOkCghmz/7QN1mH3QbohRCpwBzgYVe8sBOwP/4VPaDGAi9hV7aCguZoL47YjPNe0JJYznU3saHCEIAYaq4BWzcKfok/7f7wMPfmOhJL/0rxbRSYTxSJzNX9l1IN0Npdg75xPhCxv8fIByRttx5AGi5M8qhgm1ALvL979vvKla5JtAyb2dN0zDPT0ivpdyZfBWLRabkSGf9AJZFJFV3y7R66uezyDqycXubvG95/9MUBzU5gNnqRslitBQjS9TJMyKnENoK2jZs0GV5Ae8GVj9JmlxP8YPm+JyC8+PvOtQj9bwncrEkkNMUK1D8VSikEMrv1sKMGgLjkbFy7RJ8XSva/Jmnjg1BWSF43LdRnIQL/BwOD7xznn85svLCgaCu7A+UhqtzlcFwN8Sh9QkdO1GrXgWtE1keCUCOOPcMGlY7edJkZV70aKNPeIlVyhfWB35WeGtdFhAlC1wbE5wxpFGaji08zFusObeiKnf/HaXQJyG8EydVc/sWwkUJGbnzWoJd8HPry8jJw49+l8r156WR/g8zkvoze X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(376002)(366004)(39860400002)(396003)(136003)(186009)(1800799009)(451199024)(6666004)(6486002)(6512007)(6506007)(478600001)(38100700002)(2906002)(7416002)(107886003)(1076003)(2616005)(26005)(83380400001)(86362001)(36756003)(66946007)(5660300002)(54906003)(66476007)(66556008)(316002)(44832011)(8936002)(8676002)(4326008)(41300700001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 4cejnJjIGvgFWisxePGHPVDX8+52ZQosziP9oIkt9QGXTp3HlM5h0vahPlhhMo0m5tDoGpywIjerOVUkEIu14kPQHUkWGyAHtzFuNEtXDaUJ1wy/3t3+criRB/w/mNsS6XzOOssHv0AbaiRBhI8olCYFsLXigv18RDKiYOUsYzor88gHjt3d7zNGw5cJDbeblg9ux3VQwLfeJvJnLH/v0M2vozpDC/JIyVZyIOugl+o6aIEtZAg6weNCzZFtBdTB1285jXTd7hiPKK14qSdY/SI23AO3n/gB8OMJuTL7VX5X5h4AA9gx4hlXPoN0Jv7l4wJX3fSR4fjk0+X1PyklMSyTMOegc7T1FljHeHrQS7/A0vIJ5BYbU3YZpdQNAerXDPBvca3uDbMuZPJ8VQL5UsrlO85xdPAOROJa4SnyXID9TrkoPqu7GLVu0lB8GcaRDjRz4tXJzyKENwd3UFTJFT1yykIbN3X11lcn4Hqk0b/oO99KIEIPEjH0mK5jetd9rzSqxHyJPpsxugstTzSQigUBOix1zA8+hwuNE1JaKSRKwVQ+OcMe1bXAl6T4JMAqmgg7xCmSYqVnEixlePdSbO5bvf/FNlKvl1O4fq/OW7d0GyJZ03fr7C6pLQNaX/huOQTg/miVig081X2S/gN1xb7isdWcLac5hpuAeceqMSMU1C2epNGD3nVhYbpiVgh4m2hCG7ti4I9heJzU/T66umCaS0mqTC+AXuKwGDehg3gmez/qTYjqC3gVX/ix160royhtddyvd8NDiru9NGQeIC+zwLu/HZ0lnq1QqMGIN3ezs0HeGts5Z9X1RrACHZg7fWCAqMj0/AOGKBgbG/donybShya4Aw2tTmQUvgS0TP+ZQep/Uh3Ksl6YtCxxooFPypEacoeO0MrYancyivGQjTPs8OxU0/0Yr2d5RvvQEbrr4rzqpkQqqUTTH09m1iNMzZOdShV5c52g5H9KpU5oiHQXnsiwyQKyGKHzI2oHNJJA3diwPv/MMz/svLh2UrYb4h/pc+SuB9vGH6i2cn/U+hCiA9pFeC50r0hpRix+x6Y3teZQG+IMZRkR1G5ddAPRIwW8ISYuZ3EDMpn2D+7+5Z1Y7Vl5s6WMoxBUiTyc5W0W8z8O7CKxsBeysRHDLO9jmj4Ge6B8kUA4Dl4aGDAwwU2UNkJRkgX6/35gV21UCmYoL9B1o1BNjlvQJXz8+hdjTNHWQgcZ/60wnofVdEq6BmSRrS45iZaVe77iwsa0OhcuNMs6TBrGqOsXmFj/Tb/OOxsYayGjNNBoI/tVPaXwSJ4/Ju3i06VFb1VHMpllR7FFr2WIgRCAsMlwQFNC94NCwT2PBmUkYTh0sHD9EHzf4tP1+e9KUFN5Hn53z37927wmnn3Lrwe4BNSqUvrYkkwzkq923WuYIaWKKbAXmoSIoPCcVh9lbiaxjJ9mEviCzMZAcY8/4OjkSu9HcWl/Hurf7/PFF8KvDGQNTQGJbkvzVuTYNhdr986BrOwLsqa/+CvEYd38OXvH31rQrdHpE43r9J38hz7k8hMtUkyNKUsRFRt1INbH+LJfJAd9I8o+ICeCLbuYLWUQwyIncoWyU9eGIk6uizE44j7uV0nEeRHBGQ== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: lMnKV1pzyydX1lFrd0p4wMPQAjPQ4YhsTDN7eBSTLcTAsA+B1RfZCEbHQ7bR538oWP5Pbo2/8FfipTpXAmC7oS56C7+4Dp2tV91y652CP4SgGb9j4yB7xC/AlLpaY7SnPGBSR0LK1MhoviZU18eMFXl5R+t5NE7oYzTpnMerHFt4iXdUH0q4EsJRiu3pAZstKqStWRXfPjCOhdCv3JsrQD9CYSgAr5qzhIlntqWh8BCvvRESigdjOmzckCjkMP5li7e8L4ztMjePxvDWpP/7NGRhKuEyYiZOtMANkQomRiOMDqZMX9pwugFWHBVlUv/QRnyTHxV2+eKpcEjrCrzk4GC1ZSgtbSsWdR10oclx9NyfgAw4ZIFBnyUiZ4LZP2xUmZ7u4otXPmbofPxD7lvcdeRyMBMiePMDZgr4yX1cJZHLXrCqyhtwY0wELJoF4iv2rYrdidkY3Jx42QM3hSthOKEO4WihkzHvD7aAFz9r0F4BZtJARQt1Wd4oGDg39K3bvmqq3toqIpV61WF+KkLLsLWg1wnug2l//Q0t4BfPdpJyQN0vzbHECEbnPHA4dTYZiETOcF9mr1S3pnstGz2EICD/k/6I21WPH3KaQdIlUMLNx0PqnWNPTIAaKSxwnttmLl/i/WVSo7f1WkcFLaY45hAI5ReEsoBWFonw+K91+zsJ9h4hYqxsMJoKUCAhamIKpOOkdSlsm+D7hXR9xTQ6BL5BpywZ7Zv1s5Gipo+rqp1ljqo79DXk0EyAVVi22OjKFWwddxUUBw8rlJ04waO+yqaIGVEJigamgs4PofvrRiniX8YMMYMnWLtcR+ZuBkWFdHvkXKdMRBQTJUYmz5O5fAiHGFSw8iEjh2b+m1vT6Tyzg6/WBX/KM1b/bNDgHFbgPUjb2X3euaxJ6rGi1mNqEwrn8iJmSVFnyPYPWJxRowndZPO5br3d8lYxeZTJYu9tL92gRjg89feLqllaE4xjSDPE+lLr4qZQtDQ9upmMXM1n7Jxqvs6QX353r969B+xp4eGTFXCUKf84Z5hv0up48BJZImbTV7K9w+KIyu6jwiCMm6wN3X3J2k65JKySmyt2Oo9dMsxppEcdU/fG25G8g3dlgn48Di7NBRgVUo2ZJFMs59zBpcAcfkZstVY9W28Y X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: b300aa6c-3d1a-428f-ae49-08dbae59517f X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Sep 2023 21:44:44.7016 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: zf8C2iTmbE9lhFCG7MR15a0nW2EdzW8LSBKT5pFYJAuRgvOLWBLUHuxnhb+Rz3ICtDqMkuKqnvsZypln2vsv0w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB6395 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-05_13,2023-09-05_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxlogscore=999 adultscore=0 phishscore=0 bulkscore=0 mlxscore=0 suspectscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309050189 X-Proofpoint-GUID: VNfSrgJj8je6MQ2GCXhsRo5SHFZk5JTG X-Proofpoint-ORIG-GUID: VNfSrgJj8je6MQ2GCXhsRo5SHFZk5JTG X-Rspamd-Queue-Id: 0EBF818002B X-Rspam-User: X-Stat-Signature: m79d8kmph6n5cifmfcutcfz3x7qrxbo1 X-Rspamd-Server: rspam03 X-HE-Tag: 1693950316-563409 X-HE-Meta: U2FsdGVkX1/Txfxf9r1fTFPR1H/Hr5iBp0qwMwxxXYR8HQXCDAKkjtjwSTPlvJe/xQ/NapLAeg3dRqjCyhD4NuVei5TPWn2ulbliGYwWlz9p8uZ8staq58dx4QbDGrZ8J9yqXbAHtiRYeN1Hp7iWt1Rw41k7RLIgX+1nyx/quxkNdc7EqoC4anUHhiur7AmvauAY5K8skscsZUdvnz365lG7QY3Z7GrG14rr/RXPQ72OGAWgBC4w2ntOnmyMjrn5DQcoszqsh2Gf1m0c7yNNOwNqC9kU0vHZ8PyX3460qe/mLigpRa1Z06AZ0YEYqe54qbdhm9LgmA50HB8Xzns9vJ6GI+nULXarrXkSoIrrmV9AciFoOFwy39e+R1LHzYLBPhoWkyUoZrJUDoGsqv/kb8R4hhNL2PGD4Bu/bb5aQbNkG2d+rKlOBKXXB4q+bt1ZOPdEfwn+MQ9w+9ZTmumR0e1U9H1PKOb6afKNYuPRL1LU2OSYCFzkLdnVIGq3LmC0wWCRtAYE3sdHvDu/C8HDASONWWRlSjDulRFgam73Pr0EOpI3jV1r36ZnYcr5fnB5sFBA5QOiCzHCa/paVbpmZI5jdfaAZpCTeKfrfLBJv1LrxC3wIhMu8P3ffCiaPqp9kblDfNHJ6GRmBeSrHeK/H7MGa3h3tZ9sEPpo5qrXyBWiCh8Vhurd1cF7oUxbjxGlhLwSeTc9Qu5hrjRiObnT5OJCcAqvRIIZM2Ht/6sJjyAyRx/02RW4+cQvUArsq3/i2dVVYnFUKOTa2VMxyjUFHnxHyiR0LhF9C5U8tcjsHWfh1QUKu6pluHjFov2GBzCgHMJypjL0DBj4aaPpOqgKr86KaHlRebE5BEBYotNxpeXbUpd6p3gj0blA8isUbbc5tVwmIHqq0ANBbt/d/IvV4o7rFEvPgbZLHZ63f2QcFyOlJADXmZc3OM40EiOD+ZTxZJFPAhgTr4STb8QI3/5 uzVrW4L0 SiIsE3i9RBYZ8dzZfOuUV7dZV6X9phLRjGu8JevVjvPZGvSGWc0hB7gVSu4Ud31CQJkRr4q/Y+QUIim3q5ce8Qu27pOCIQCYFb0YRoYbUy+X6aASAzWC2izaAk3mA+Np6i4MdtgA9lCISzQYwQBk98jZHhDP4w73O0tm/Xi8rv74XKB+H9WNZR+oxYROFUFct1M6q4t+sdcCNYdR5PK5ueWover0LOplfwLZfa13ittxabK3lZyMOc87ejZZulyQRfqHcIiUsXdz5m43+YEK8g/sfGyVn18rkEpkEjOx0MLnzJ0S+1c8sb8OPRxtzHsxPDoL/xXUruHi4c238SnKRvl77VLp8FbZW54avzPFGCJAxnhdnjFQERGjPWC7y0QcKfex8xn2SEjQvTfdWQno0lauawYHwZJ0LOo9Ga3Kbz7/33+XAVfheht690J1lQMunNbkZ4wYogw9O8BfrMp9amT0EEC+iMtWbJF7GvH23gEgh8lwIazx632rX1OVtvMrG6Xdt X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that batching of hugetlb vmemmap optimization processing is possible, batch the freeing of vmemmap pages. When freeing vmemmap pages for a hugetlb page, we add them to a list that is freed after the entire batch has been processed. This enhances the ability to return contiguous ranges of memory to the low level allocators. Signed-off-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 60 ++++++++++++++++++++++++++++---------------- 1 file changed, 38 insertions(+), 22 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 79de984919ef..a715712df831 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -306,18 +306,21 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, * @end: end address of the vmemmap virtual address range that we want to * remap. * @reuse: reuse address. + * @vmemmap_pages: list to deposit vmemmap pages to be freed. It is callers + * responsibility to free pages. * * Return: %0 on success, negative error code otherwise. */ static int vmemmap_remap_free(unsigned long start, unsigned long end, - unsigned long reuse) + unsigned long reuse, + struct list_head *vmemmap_pages) { int ret; - LIST_HEAD(vmemmap_pages); + LIST_HEAD(freed_pages); struct vmemmap_remap_walk walk = { .remap_pte = vmemmap_remap_pte, .reuse_addr = reuse, - .vmemmap_pages = &vmemmap_pages, + .vmemmap_pages = &freed_pages, }; int nid = page_to_nid((struct page *)start); gfp_t gfp_mask = GFP_KERNEL | __GFP_THISNODE | __GFP_NORETRY | @@ -335,7 +338,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, if (walk.reuse_page) { copy_page(page_to_virt(walk.reuse_page), (void *)walk.reuse_addr); - list_add(&walk.reuse_page->lru, &vmemmap_pages); + list_add(&walk.reuse_page->lru, &freed_pages); } /* @@ -366,15 +369,14 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, walk = (struct vmemmap_remap_walk) { .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, - .vmemmap_pages = &vmemmap_pages, + .vmemmap_pages = &freed_pages, }; vmemmap_remap_range(reuse, end, &walk); } mmap_read_unlock(&init_mm); - free_vmemmap_page_list(&vmemmap_pages); - + list_splice(&freed_pages, vmemmap_pages); return ret; } @@ -553,17 +555,9 @@ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *h return true; } -/** - * hugetlb_vmemmap_optimize - optimize @head page's vmemmap pages. - * @h: struct hstate. - * @head: the head page whose vmemmap pages will be optimized. - * - * This function only tries to optimize @head's vmemmap pages and does not - * guarantee that the optimization will succeed after it returns. The caller - * can use HPageVmemmapOptimized(@head) to detect if @head's vmemmap pages - * have been optimized. - */ -void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) +static void __hugetlb_vmemmap_optimize(const struct hstate *h, + struct page *head, + struct list_head *vmemmap_pages) { unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; unsigned long vmemmap_reuse; @@ -580,21 +574,43 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) /* * Remap the vmemmap virtual address range [@vmemmap_start, @vmemmap_end) - * to the page which @vmemmap_reuse is mapped to, then free the pages - * which the range [@vmemmap_start, @vmemmap_end] is mapped to. + * to the page which @vmemmap_reuse is mapped to. Add pages previously + * mapping the range to vmemmap_pages list so that they can be freed by + * the caller. */ - if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse)) + if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, vmemmap_pages)) static_branch_dec(&hugetlb_optimize_vmemmap_key); else SetHPageVmemmapOptimized(head); } +/** + * hugetlb_vmemmap_optimize - optimize @head page's vmemmap pages. + * @h: struct hstate. + * @head: the head page whose vmemmap pages will be optimized. + * + * This function only tries to optimize @head's vmemmap pages and does not + * guarantee that the optimization will succeed after it returns. The caller + * can use HPageVmemmapOptimized(@head) to detect if @head's vmemmap pages + * have been optimized. + */ +void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) +{ + LIST_HEAD(vmemmap_pages); + + __hugetlb_vmemmap_optimize(h, head, &vmemmap_pages); + free_vmemmap_page_list(&vmemmap_pages); +} + void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) { struct folio *folio; + LIST_HEAD(vmemmap_pages); list_for_each_entry(folio, folio_list, lru) - hugetlb_vmemmap_optimize(h, &folio->page); + __hugetlb_vmemmap_optimize(h, &folio->page, &vmemmap_pages); + + free_vmemmap_page_list(&vmemmap_pages); } static struct ctl_table hugetlb_vmemmap_sysctls[] = { From patchwork Tue Sep 5 21:44:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13375118 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 486F3CA1016 for ; Tue, 5 Sep 2023 21:45:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 51238900007; Tue, 5 Sep 2023 17:45:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 49C838E0014; Tue, 5 Sep 2023 17:45:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 29E2C900007; Tue, 5 Sep 2023 17:45:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 142448E0014 for ; Tue, 5 Sep 2023 17:45:22 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E01B2120B0F for ; Tue, 5 Sep 2023 21:45:21 +0000 (UTC) X-FDA: 81203875242.21.0F02A98 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf24.hostedemail.com (Postfix) with ESMTP id 6C54B180030 for ; Tue, 5 Sep 2023 21:45:18 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=gy7xCZ4k; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=PKAiOh2k; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf24.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693950318; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FjiEFcLPHO4DD4C9WEFm/uQerVdf8Ih+bp5vWwT92M0=; b=aoAelaub8rgureJaB0YwPaI9ho2GW2kfBQ5fs6OmMWQgNIKPvDARjQInTcOT+qnzOT4hxE FW2432Uc8P7yyrK6j776eq+xK3cUcILw5RpFFU2EEaz0QH49CfG7k4eYhcoauJ6caFApZA ScY199hpEiYVtXAve5Q4wSsSKHIPSg4= ARC-Authentication-Results: i=2; imf24.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=gy7xCZ4k; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=PKAiOh2k; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf24.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1693950318; a=rsa-sha256; cv=pass; b=wVFpsH+IZIx0fJx3LAehTCcomWsgllb/vqDm25Dp/Q9K+zNz7nJHTyr+0duqPoxmjL/O+E i5XmkNc/728f7dJfJ7FE03NZJcEl9qxxb/6N/QwBJgy7WzvOt9JObTm4fW2TNBcPxEa927 fJ5GHBLu11Ok/tvPmsD1MJyrWcDVH3Y= Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 385Le0AP020248; Tue, 5 Sep 2023 21:44:50 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=FjiEFcLPHO4DD4C9WEFm/uQerVdf8Ih+bp5vWwT92M0=; b=gy7xCZ4kgj5a3fRwXzHd7coO34aKD5aRzVed6A8PeERPUcObk3LFMmWfy9otvsS2Z7NL XFdL6m5OAKrmgikxSL5sWlUqzC9kQoDu+qmqmlJhHKC83T8q2stmkIaqvgqQEVTadsmj V6N+9kNIvQA/EZjwCngMaff3UGovlqtYmxw7vnN/cNMoe3/ZwWmEBqi6f7MOGtvNRlN0 aANPDEUxzeuVfyXDZ87B8HpqmvbSws1W7UYlw0mCT8UvGl9GG2Xr9ab9hJU+wGdf6xJf Vcf+ClAdZSSQvH3PFF0nnk1admUCuXvSLehHBt6JstJ2VHIc5My4dfvFdjBkkno+B1Xy zA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sxcem807s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:44:50 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 385KEUfs028145; Tue, 5 Sep 2023 21:44:49 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2044.outbound.protection.outlook.com [104.47.66.44]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3suug5bd4s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:44:49 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=V3ybtoeyRcmB85WHcCOzInlgfEBbTDvLNYSfwzg7tLn76dUlbVlUszKGvLJq4l9l7wYM7e+9f8fS9vo+Cl9ti9Fvvw5zRR4HJ+XXSbhVLEfbHMBJ5FeR81mJgHCf7wTpvIC8hK1sOfj7awcB4dXeO3w49fATEXmIQ546+fe+/140Rg5zYaT6Id34dME4lsCTwqwEFb8wpp6xmwIsVWFbo2ESZUzOFEks/kE5P8h9DEslzZhhBCcx4L9ocS/MMlhNR3HPR24eGVBYVNr/rgga3WU19PD4pa5nOY/RmU8X+a4Bw7Kz3bkz4TvpiG7mHob5rjCTExhbNUtgpadsMWbk8g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=FjiEFcLPHO4DD4C9WEFm/uQerVdf8Ih+bp5vWwT92M0=; b=dW3mGvdc1V1fCprAdYIUcA7WTegsbFyuwSza5aUm2VJRDZy5/n4S0oIOlZoE/x1puQ5/W01FkbXlROZvCxS7B1r0ZresDSbIqadlXHxVn3QaKXsZMqh7gRYEXLtZsq//vjCO3Rj4V7bH1f9PLjMHv9p0+q7Kqbl2AuAX68XVOOgQQw2nDiYbSGp/p6OJktLpey7pG2Hu7q2ejYvovx13NvL6CcUovSrXhirufSEgnncAmkxY1h1LkDn+S3umpBiDRkvQCqY53g+HhqgVstdG8kavCNf+wscYfT4X6exyyZttpt+kDs/EzI4X5W88CRL4cCx3ajJn/2/5E5Lf7EON3Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FjiEFcLPHO4DD4C9WEFm/uQerVdf8Ih+bp5vWwT92M0=; b=PKAiOh2kJNenqYgOoAr3lOxVGocim5d1u7zZ7V3z+BkmbTucSgAqBh959wNw0cLaL998PLNfFco3McFBHvoXXDjGlHXIFe7CFEsf47m/WqOe+JbJgozxw0gkqZpR5Aiy/k1DpOVZxpeDHcy1XpLnndzUivaRABrOBqhoz+LZxb8= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB6395.namprd10.prod.outlook.com (2603:10b6:303:1ea::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.33; Tue, 5 Sep 2023 21:44:47 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715%3]) with mapi id 15.20.6745.030; Tue, 5 Sep 2023 21:44:47 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v2 09/11] hugetlb: batch PMD split for bulk vmemmap dedup Date: Tue, 5 Sep 2023 14:44:08 -0700 Message-ID: <20230905214412.89152-10-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230905214412.89152-1-mike.kravetz@oracle.com> References: <20230905214412.89152-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0080.namprd03.prod.outlook.com (2603:10b6:303:b6::25) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB6395:EE_ X-MS-Office365-Filtering-Correlation-Id: 06b18e07-c825-4acd-4d6c-08dbae595317 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: JMc8wGd9JLazJvIhvcsxaXnk+Wpho5bVm4aWBZOdOarbY2ip+DOj4GHs2rDKBl5rVGWQAjO+EjznfjlOo0IX0v00EVzST/lDZg/8xLqFrb+hpAlFZ6s+XfJ1po5xOiNgpNUaRV7bwg7lXL9fjmzR4QF4jtCvZ8QNOoVZasiKPHkTOs5e9Vb++k9wghW9JJh8Kqq6zBSAIxF3UBztwy3C2Ppyp8IVO2l18XyQrOnFk1n2nmBwv8bLg+T27avlPOhC9sqYFxXIoEO1NTHki/6RJkxFdTQCEFZmJeUi8BMvdK7PFoYy1D9iUinZShIZWvHCIQ0cd51y9JWiN+Hc2GoXKFpKzebtQ2DR5T/u3elQg/Wm9kYbGBN6v+1yOdirNAjIUU2rBgCbb2RDu+NP+/DvNEbJ97FqqO6eFbTvkrCTL7WA7eh1NRZQE92Y66+v+R4/U1orySejubpudC3ieMy01HLXH5ynNmDexiRYE2JrFsVVkFL/lyXIT8CCGveC/rBNLRyFkzriVjWZumLEQCwcpXJxI8xKSliLnFuwmLOmbi1UIcga57K01/abPu3kPikk X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(376002)(366004)(39860400002)(396003)(136003)(186009)(1800799009)(451199024)(6666004)(6486002)(6512007)(6506007)(478600001)(38100700002)(2906002)(7416002)(107886003)(1076003)(2616005)(26005)(83380400001)(86362001)(36756003)(66946007)(5660300002)(54906003)(66476007)(66556008)(316002)(44832011)(8936002)(8676002)(4326008)(41300700001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: MKsyrStyfowNZO6nsFCqQKY5aMpsjBAUYEleu8eZd+tq8cHlNfnq/2Lu6xDbk3N+iRvxQ6wD8WPSVx36KRF9a/6mhbBKMGYpkV6R8D1FY5mEF3pk/hdbDLpUPn1DlfXTiqqbnugmOiVKTyNeJ8dEIYq/LZvh9IL84/KMsLYAeSGfF8Mn399OeqAsBU2cdeIgHPczkuZ3DU9gIW9MHkMY/c+XTBKzhAs4tK3oFmG5/kPwjh2BxlM8bjEOUz8RktISKkK2JqfDhNUQeUQo6pgzJMPnuMFZoG4dslFgCSiBpNHRpc/kU6gehy/mx++SJc/R+2UI7LzIjdiZNLGu+NuO/QaACMUQjmkx2K8ZgsuaB9u4KNb3yOI5wm/ArxB9Xg+1TeA2YkJbzKKMzkq7tSpm/pRriJEYJ4F2VgFORl6kUZ9WfKDohKecZg5xJWyCDkelBXKMCaiOEWJDHpRlezGG0umiZtHO9whdb/rQXy3HYAosru7BBDggQAWNmKseYIQE6qhXrkqU704u0yCaNCTgjRziLgUJMq+DGjrskNE8TgUq09qVnkunX8WnNH0V036lJtd5XeqWzuACAMNaUK7ESo4xjKDynbA/yt5Do5mftdFdqwO7+rQVVf/iFSajdDNIYPTEV6cvVlPEYuqeC05RUAHFiZ2jzXmQu1Wk7YYMT2aT0kv3UbaDnIrI3Brtn0pGfe5/iwzua8xLlgYQe4r01P/fZbpX3FC7NzWOaHxmpbY0rW2/WkNGWsJm7wjEtjtQPc8J+aJU+FWcqGN1cBm4jB2fDNutuF6IaG+1+OcQcYJSbrfMHQU1xR8F7AOVh3BuIoMdmQvb8e/XXsKpFh7grhUspIFs6moGaKJ+z6Rs+VSDzDN4iMA4dQCqv6K5rV+VUsi9nbMvSl4LC2X3/smJqsgGpD+VEck7X7VeuohEb7f73UAyBRrnpnNsf8/sJrqpGt/70hSCVr+KkHtEY1V+qB+LkS8vGPiCCYiR7W2mFq7cfByz7xrN/EhcfXMrBYtAVcd/XtRT+UiIwoM0HgOzxVBj05rGS9QI3C7ApdP/MZvxWT3XQLZ8rJxbS0ISS9PDle6bFl+IETSgTVGdKW8SnATsU3KT/EPV1oJQtgswKU/ZbHvDxncAgvhwPnHqWO0iIuK8cQXlpVKex9OZ9OZyvgLQpTL9ipbCiqGwEUer/+8s4xX88iUjDL+x585m+0TPURcDEoVRny1zGHj3pCTdLr2Kbj7WRv0ylMBiYvP3RskYjB1vcsWvhKds9sgEfdVJIsdt+B4zZWikg7cqwDkMjZtYEf3Rd9dlZcoX1R0vZ0G9xIqCKzzxqzCyU5QQQZ7SNTFiT2VRMV3Apvr3/aNd0uotHgHczvOcEp+Nd2oxf425VMwo8wfrQ/hClxpMPME7Wm3gcUauGm1f/qAr3GTEkHVLFpLi0NrHUX32i0VHOVBZ0zRLwk0L+p6nvWfEQtZKz0y+yRVg09bgX3+EfMZY0De6cMFsUcW3dHdZ5jMcH8PawOEpoWhrclgjVbRV2ZObSn1qsY6CXUbzX9JNY+um6ss2sVUx70EPfgiv2pYp5enPYqbJWxkuZw/kHRA4xDPg7gN37EAPrS4n8rqX6C8Ntw== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: CQ4OEmir/yiudi9dtt9Fyl+ifmT0u0O9QGzilCxE9GzNz912+1aj3UEoOeVvvcvB4pj5R90+EeGq/xeBLiCQFnmJX/1ou9a/31PDWbEfUzSlrwq7X1xwb1yns6u4yHxHcekPnwdibEK6G9+Eda56oZcSsPHGEAdjVFUdsbmQfgGOAnhgGXyfWNiBWNWY+0vn1PU+IpHdzo8KCFG4zuwwIjyWfddFtYZpQBZrizXqWxNd4HFpLB7QllnzYb/YJ8Q2CQY+rArY5Gut9OCMyl4Eg1GRCS1yjQS7FNq6rw+91J4A+SYOg8U6UBvXx0vavJW78WM4aMKWnz9/wk9oWRcCItvyh4zXXYcNCCLdn7omVitAHCc6VPkvz5mMaabJDS2py1iOETCb76hPqY9b7Bsxvw2kjdT+WNeX8pAc6IgKOq8mKoYMTZhO2/xBjMbh2wEsd+ynjqWFOULZ/azOM1wdpGzft4jr6ypO49DN+f+7QemVWd0pD8zUVPacJGlbnQMQW+Refsm13E1Ko+4IFLjvlBOE3ddSCnz/qDbqwlZLX0Ch6ldPiyjbjJs1NrfutQ+oRQI2iTteG1V8SR3D2qqUxZGzx9o7xbEkM35moakoAEsaZxeqlQoS8+CZBf23uqRHDxk1dZ31K9lWzoVhPnZajbLnmyrxYvM7d8SvPnLKoUgkkp21N0ZJo8xR+2/uo1I7OcAjpimlvq0kzs3WNMtfP5bc41fERNF9h3XpO7fcpTPHU4KQpEOtc6GPSuSpM2RLJVsG5XvBnNHLjZgTve2x+lduN/6uB0LyQVjEtSwGxM2x5oZEGnOdL8CIb9/lhbf+tsZmtAtZmP01ggFfUYSzSSA3LgJjzGno+GTmnpK4wc2x7lo95fSzgr6tC0JS3lAJ0h7I6P7Lwk0wuOzD8P9EyNWgYCvL8NRlnoRvWtHwrUXAQ0/ggCv7t9c7jfwe7giB+nwnrr5LQJcPgEpoiIn5vX1kjqhmsm5IptveSqMMhc3L7/CoauGWQPGNAqnmiZZFOHquvYquLx5lkAOtVopIGslAai82Y5MIIsMZpLxyy+UbsnkPug/ChHnhltdjyOZ8kANcdicLttQVnG7xgy26sQwiCorg4wREpYhMz3N2slUoWub27eFxZfXGU6TtWt0V X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 06b18e07-c825-4acd-4d6c-08dbae595317 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Sep 2023 21:44:47.4098 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: z23esX+Hqqxc95HZxZIW2fTuO0GbwYuGyzcj/rad4jDOidbER4yz2GSvFsxfcE8jC9RKcOnIKdVvmJDXJ07nbA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB6395 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-05_13,2023-09-05_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxlogscore=999 adultscore=0 phishscore=0 bulkscore=0 mlxscore=0 suspectscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309050189 X-Proofpoint-GUID: 2ebrnMTaCSdFEW2h5N-BVX-FdItjkdss X-Proofpoint-ORIG-GUID: 2ebrnMTaCSdFEW2h5N-BVX-FdItjkdss X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 6C54B180030 X-Stat-Signature: mcdqpte1tpdpwjyopfqh9qpng87mq6sf X-Rspam-User: X-HE-Tag: 1693950318-320291 X-HE-Meta: U2FsdGVkX1+irlP8mfsDHkMx7vgwHyCh/q9sWsjCZDaM7fEx5sZO1xDaYaNvrTHasm/nLjxCElJ/1V4NbLSgzzeZkFNgvu5YQlFvsIl16r++fonOnRP5s6HI0rH3eyJSRfuCPsH+ZOOwH8nPYE8oQ+3GbdAg9AYvGumSzitF81sWKsr9g8/jcfZ3Vlpjrmky3yh2dNYW0dq6NlTIXCLmJS3kDdBfM46D9QwalAyqlYSGgWmKEaBkUr5WH2jz0xxrmqmwxE26Fee8/adq24SBk4RBB2KKLFzLBe1zd+hnatVdI/nISBIrymv89iovqZIhcaCm71pdrGKkWbrx4DfsR5ETltmn5xHT4kMiBlfzEB3w0R5sFJ4S8o/md2cPeBqnxMkxpjH2hzLGOQcUX3f1jrXMpE53IqwSntE2glNXB2jctVH5T98CQzmkOFTPCJXV2/6JufCet6ph6SKjg1Q32Ew1TdwL0ml9isLX/3MHiWDEAIWIYIhy7Jf7nvAUjmocmpugHUYWw5XxlOQdTQpSbk9o5HSqjYSNCM4QCmYdDRJKGTMSMJ4dcso4cwQbP6H9m36U85op1FBX6oHrcQqeEbXM0Tzx/dhm6TKdC31ykAGnYsjUaMqTopk/by5Wb0cNpFnrcV6dsIr40jDonnNNIN95RUEJJ6IMqOzMFFjmC2C4SOgk4ZcKIJA05SsJv0qOo55+iK0GfFx7dEA+LbyrCtoR4Rzmj8ASh52+33JZTV+CTo7mGCfDWkVOJ+Fyt+ogyUablmYESTPuikpGxPRKXzJaO18dooDe3JXQ5/JydGQWIkkS7/beC4Nrpy+R4Jq95HKOMS260f83locFD46eQ+SMy43fcjLcvSGOUT7vl2IF5EDjBwYX4T3caaPiEw7xF2odzB/CAv/Z3QBGUr1r7f1sF0z+J2yxPzpSdpEK3uIxpa1SXbVy7YUz8egMD5IxkZY7z/BuycXdJblbmCM uYAqYq84 C+JAMwV8DK8pgSNXLRLgO2/Y6zLCYNrPOvOjqRnljfwLm/BOLGQTxUfBuan2TbmJuldpWsjB21fdH1tAwIBFUH1Vi8ekn5SrzIgbe87fzWDU+9EQDyKVMZ8Q7pSOROGbvX/XH4Kii7bJT+SoNLshmT5D0vRNnVghMcAE7YMQ5v105l54qRSKRvoj27JpnkyYQ2WRSK3vXU6soknzKuIINSqCYPNXIilNqVwTUy4LULkXFYyLI6DRsweX3cJc1xvOSHb7FN2QPS1KpRfI7yIheXqfsIE3RPNtX/PZjzwO93Hz3/5+4B33TOoOU0Jq5ePZkTCHKV05aCMVNOOepLVXUtD0azN260hK0d3LsVZFcfHwiWKVa0kxVN6rM/+6QtUlohGbGidYsLpj9a0UpkYyTBXrRHZGuyomtohVNdh6npgdnnOZSyvKkCDGyKKKTFRFZ2dPKmHrrGe/JDkGvz+AI89EElFQn684kReRQ9OtNlCt2FoNv/Nh9ECy/8XGNeHWM6Zmh X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Joao Martins In an effort to minimize amount of TLB flushes, batch all PMD splits belonging to a range of pages in order to perform only 1 (global) TLB flush. Rebased and updated by Mike Kravetz Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 72 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 68 insertions(+), 4 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index a715712df831..d956551699bc 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -37,7 +37,7 @@ struct vmemmap_remap_walk { struct list_head *vmemmap_pages; }; -static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start) +static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, bool flush) { pmd_t __pmd; int i; @@ -80,7 +80,8 @@ static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start) /* Make pte visible before pmd. See comment in pmd_install(). */ smp_wmb(); pmd_populate_kernel(&init_mm, pmd, pgtable); - flush_tlb_kernel_range(start, start + PMD_SIZE); + if (flush) + flush_tlb_kernel_range(start, start + PMD_SIZE); } else { pte_free_kernel(&init_mm, pgtable); } @@ -127,11 +128,20 @@ static int vmemmap_pmd_range(pud_t *pud, unsigned long addr, do { int ret; - ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK); + ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK, + walk->remap_pte != NULL); if (ret) return ret; next = pmd_addr_end(addr, end); + + /* + * We are only splitting, not remapping the hugetlb vmemmap + * pages. + */ + if (!walk->remap_pte) + continue; + vmemmap_pte_range(pmd, addr, next, walk); } while (pmd++, addr = next, addr != end); @@ -198,7 +208,8 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, return ret; } while (pgd++, addr = next, addr != end); - flush_tlb_kernel_range(start, end); + if (walk->remap_pte) + flush_tlb_kernel_range(start, end); return 0; } @@ -297,6 +308,35 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); } +/** + * vmemmap_remap_split - split the vmemmap virtual address range [@start, @end) + * backing PMDs of the directmap into PTEs + * @start: start address of the vmemmap virtual address range that we want + * to remap. + * @end: end address of the vmemmap virtual address range that we want to + * remap. + * @reuse: reuse address. + * + * Return: %0 on success, negative error code otherwise. + */ +static int vmemmap_remap_split(unsigned long start, unsigned long end, + unsigned long reuse) +{ + int ret; + struct vmemmap_remap_walk walk = { + .remap_pte = NULL, + }; + + /* See the comment in the vmemmap_remap_free(). */ + BUG_ON(start - reuse != PAGE_SIZE); + + mmap_read_lock(&init_mm); + ret = vmemmap_remap_range(reuse, end, &walk); + mmap_read_unlock(&init_mm); + + return ret; +} + /** * vmemmap_remap_free - remap the vmemmap virtual address range [@start, @end) * to the page which @reuse is mapped to, then free vmemmap @@ -602,11 +642,35 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) free_vmemmap_page_list(&vmemmap_pages); } +static void hugetlb_vmemmap_split(const struct hstate *h, struct page *head) +{ + unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; + unsigned long vmemmap_reuse; + + if (!vmemmap_should_optimize(h, head)) + return; + + vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); + vmemmap_reuse = vmemmap_start; + vmemmap_start += HUGETLB_VMEMMAP_RESERVE_SIZE; + + /* + * Split PMDs on the vmemmap virtual address range [@vmemmap_start, + * @vmemmap_end] + */ + vmemmap_remap_split(vmemmap_start, vmemmap_end, vmemmap_reuse); +} + void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) { struct folio *folio; LIST_HEAD(vmemmap_pages); + list_for_each_entry(folio, folio_list, lru) + hugetlb_vmemmap_split(h, &folio->page); + + flush_tlb_all(); + list_for_each_entry(folio, folio_list, lru) __hugetlb_vmemmap_optimize(h, &folio->page, &vmemmap_pages); From patchwork Tue Sep 5 21:44:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13375115 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49CC0CA101A for ; Tue, 5 Sep 2023 21:45:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5AE95900003; Tue, 5 Sep 2023 17:45:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 55D238E0014; Tue, 5 Sep 2023 17:45:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3399F900003; Tue, 5 Sep 2023 17:45:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 1F33B8E0014 for ; Tue, 5 Sep 2023 17:45:20 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E6B24120B07 for ; Tue, 5 Sep 2023 21:45:19 +0000 (UTC) X-FDA: 81203875158.11.78206E8 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf14.hostedemail.com (Postfix) with ESMTP id 8BA4B100039 for ; Tue, 5 Sep 2023 21:45:16 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=33M2LwQi; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=BmmpGOyP; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf14.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1693950316; a=rsa-sha256; cv=pass; b=a0uF8Ws8X71xpZSScGSaBj8m7tGSIzgFR3dvdTTS6qkD+xW2xzHtiIGYj/7b1Z0s/44Egy mFEgLa8Qx+eTw6DHfjFT1TVABbwzQxvXXFVmgSMrw2coAXWIkkHlNas//Ad5/2Awp5FFSq lNSJT1cIANP9DWdQCPEnFpbdotOu8gs= ARC-Authentication-Results: i=2; imf14.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=33M2LwQi; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=BmmpGOyP; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf14.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693950316; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cvreQOpNKMda5aobqHfJE2Kae8a6KoZTDy8YOltS1Rc=; b=obYXwe5dJieM/6Ygn4V06pDZ8i2qS1J0r8Y7oU0dBi0/HzXoKn8PmDgjmTpAzjXuXiuVEp HIF270NVjnuqyyELS/pmCLCIvdlM5qvPFozD+XBhao+Q7/TDZ1GwWjJ8YKCvfNUsU70Mki e/+O9DBSIxa6favXPdWhsgWdT5qAWYo= Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 385LA6La007133; Tue, 5 Sep 2023 21:44:53 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=cvreQOpNKMda5aobqHfJE2Kae8a6KoZTDy8YOltS1Rc=; b=33M2LwQiSgPzY/OhY1W1/+DD3wdv90l5Y8GBBAThfqBHJw48hFBa1suTLdOkWqDCL3lg w57vMxCpIXwJsqnMWeif0nillZssHRZLjC2NvR0fvNOpUlfVHzemY2BBJm5DrqmsyeEK BCTfLuCGGEBUJki8iEe05s3Nfp3TIU5q4R46+/33cnefoOSoMHWYoXnLWquW2E4eoKsU 0HOW/+UbgiyW5UJEux2pbg0ydmAWJgPYTOs/mn+NjjLkHEDUDE35fndAqa10shcx/7eC mh5mMi1SWv86t/fgacLDFa2qexpJzjmY6epF2HQpma3KPCqAPn/K5qC4glYCG8omU7C/ 6Q== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sxc0g82te-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:44:53 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 385LV2ns007675; Tue, 5 Sep 2023 21:44:52 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2045.outbound.protection.outlook.com [104.47.66.45]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3suug5c767-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:44:52 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=T10+dpJzm4hSKFDdQd7sFBQyXNVxU1jdjAQeV6FNQguLrUN099nJpG7F7R/kJCh0RMY7frFmY2l9RLCOscLdC8CsfAg1gqg/4UkzaPwPlrBZrMvXfjjEEnLd2cnvinsm3VSgXxLscHblWBP2FKiJ1CgaxW+a05nKhgvf+aGf3vbU59p0/QwVIo9aLocByAJ7oN9ySimOIce6F2aZGa8GXObO0eNmNwrPfUpNMwwr53VGUvcefNfPwLq2KgRfEDEx/XY4SScjP4yP1dyYlAnvEDqQjedj8oRr2BB4doNFxxnrFPpA/Pct4NThl1RRv9/kerOF0xuLjU94lEEpWcm6Qg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=cvreQOpNKMda5aobqHfJE2Kae8a6KoZTDy8YOltS1Rc=; b=LcEsXxtxCKwRyBETrF18+nuRs6kwe8ymm+ALVGtBthELyzuIYp2RwzOHO4ipW4/qA1h92JFDhjxQbO1w3fgJu87C8v0ir0OfYQ5Q7O7xD8TrYH5LTyOeN0TfJzuwsuBb0exyWVThQsrJmNZKybATbtDWdBDeNyNZF/pr8jmrIzjGdnB4gijPZdftBKnKU21gIRuIUbEI9W93K5en1qpNGaTcBPQsysoGjcapiO+VFQFNjYJbLtr5jOSBFQRfzblpxn3VScP0gMOyHz5czXV3qoSb/KIPIhYEPLN4pbNuCSuhaoPSVkqyukhYXupcpAF/Uy46yiAQXDE/rvhDG1oR2w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cvreQOpNKMda5aobqHfJE2Kae8a6KoZTDy8YOltS1Rc=; b=BmmpGOyP3ylzvtcOSRl9m/CIP5wvzErBGaIPslFu9xW2DvUWt4ZFQy9WeHTXCDi5bR8gwj/0B4gP/ibhQVA+PMJEwh/zPoh42XxD0P6Zr7nO/iU+xi0x4mHatwGGfytRim/xhGnpfz6AKp202QuhrQq7MTreS/RYs+hFPjZw7YU= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB6395.namprd10.prod.outlook.com (2603:10b6:303:1ea::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.33; Tue, 5 Sep 2023 21:44:50 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715%3]) with mapi id 15.20.6745.030; Tue, 5 Sep 2023 21:44:50 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v2 10/11] hugetlb: batch TLB flushes when freeing vmemmap Date: Tue, 5 Sep 2023 14:44:09 -0700 Message-ID: <20230905214412.89152-11-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230905214412.89152-1-mike.kravetz@oracle.com> References: <20230905214412.89152-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MWH0EPF00056D11.namprd21.prod.outlook.com (2603:10b6:30f:fff2:0:1:0:14) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB6395:EE_ X-MS-Office365-Filtering-Correlation-Id: 607d7e75-f540-4546-281f-08dbae5954aa X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 0YWm8+ZXa8Kk1htabM77hg8DYwpGFijTVPG9lsE3yYkY9jZ2NuiGJ36+oJgP47GTsFz0cvZ4L84LNhAtqhfy/Av2ypAVytAEc+Rcs6UMAyVwAyT5tKU2LMB6XviFtt5PURIPqqtP32oaLYdja7Rfl5Nv8QeNWz/Ll6vKHuD+EMxgggrwPUdh8gomP8oQVE7z8EKCB681CkJaxdEiw3Dt6umlI1L5H/TdTFNyf6dBdg5Ab0ykUJ8JB8mSCLZ8+zrS7J+e45KZbEZ5t3V9WRFRIWndaln/o8TPN6OWwuNvLgCldhsmygArROqyPK7E7eDP8yWehwxuxuIfI1PtXrM7+EMhs9GV4BVC153CYu/nwgRStMj3WqSc+UnfxtLphnBfQmk6A1ETW2BCy0Sd7uAGl8h3FBsh9QiOeJmGUAev7aJH6+gmzLbmJJkTw0tHYGQXt31a4PxuUf4I62E7MAuQe8Daj4/ywn3WGABy7FwQ9IglaVaOhP3/v3dPtUbyPVx/PlhNar0KIN8lznY15JE9Q21QlZbMCbGSxf4t1MSs8TQUiMzTtFrJ6IJN2UPinPjk X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(376002)(366004)(39860400002)(396003)(136003)(186009)(1800799009)(451199024)(6666004)(6486002)(6512007)(6506007)(478600001)(38100700002)(2906002)(7416002)(107886003)(1076003)(2616005)(26005)(83380400001)(86362001)(36756003)(66946007)(5660300002)(54906003)(66476007)(66556008)(316002)(44832011)(8936002)(8676002)(4326008)(41300700001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Jr6uKhdzVizJYNGnLPhXEjPDtG19OD4UaAK25NtqAkT07O+FCRV/bcgBZ51t6WmWVQigDMMY1V0GHlQUKrd+nzPX+iUh8cNb7jOghdiTuS3oVLfVEYSUoz7f2KFiZp//PwGhka2wnKEcKylvQXMxXfSSBkzJgv80WpyEh6dWYdu2PqvlbEZyCU0r4+ZJ/xtoOevMXdHh20rptEYqnItcamk0tEczQAgMOYkEiJ2P24nPBPpC+PtOmd/n5uPiw/r/rh9XpS+gP9m4fDn2zOprDUdAvBNR8x+kSD+9cJ08Y7x/xR7AcsRTO9TaAnRCI8LG57z8PohosX9jvmy3Yr4Ge+5h56L/UivudWgB6CDe647gAUbv8J80QCu7IVWH3dISZeLCNtDmzrOdp6PsNVY57gvaTegl6djHSDFb/obtooOczfvDwJTbeFvt1qLOO5KjQiqlkSYGDKkrRsAkRiAvpym4uLdkQCmrPcUF9vcJJE420EjDgopflkDvWRabYJdjnSZRQwfW3QA7trVFRwYa59YrCQQ+nNEG50wLOtMT6GoltLGGYT3mrWtSGqU7+QEe71fpeziTvZMXLm+YlhucPL9Uyv8KE9dzjyyvJbNyIiAux7q2u05XqzPbGymEzOosVSQvVIT0ozp7yNdLVGprCDwHyx74+Najb1rOemZI6n/TJfueScx+nYtUAkm1A0f1MHkmM6sKbYcHTsetZHsn9/xT25USMd5qEpNBVu8PdI2c0RJEiXlc/qC7ITlRPQzt/T9mXwjM4BLquhbrYXOTJee4MA0U79vRVseNSY7nLRajnFyk3cDQ3P7PzUwL5vLofL7s6LDJ/YZwYE+KmgM0TyWBuqTlwfBufiPoeZRNE4hSCeLCGvbixbtqm3Apl/oXG5tdlsnDBzxQF+dS/u+Tji8yg+uDDi+ZDvZuPJYrRmJ5D667FRH9Eyrdb0Ga5JpQG4M2iJVApu13reRJCSUFIDxuJLqME0dW8v5ob+UTBqXZaET9h5CsS+JSWl8i9UH2FTH25SD8/nuQUssfji8g+KMg03SAFB8DCcxjSfA4CekPz/DcTIwNs9Zz7Vj/DyVBBlv+VZlS6tSmmgk79Nx8n0xGh1IirXgV8gdWolOz/AyhM4oi1khROs8NZzFq9fQQ02AzpN3+avgwEFUtQW6V4PQErKpR366w5Ni3DsomdzMFcSur5DDP4GMcJxm3L8AJyjuomnBTN3BPa/+osfxrsLUiI93LUFeNCeCNXXwds6xjFbsO5Q9XsEcE3XJi3ziUYvwk3avPvuGgkjKFnTAszCANwnyMQG2mOkBnzTf7SdJ526o6XHYgFhHufrIPT55zjsBQ7nFdddkrZc/+iDu5bzVQC5yTCSv4cBUv189sGTBJGEU55X9CeSPjBFxe1kmHqWTX/QWhgroqEsTqMNQmXguKQ8GevIfPYRZSW5oqMQSWHNCnwDZFmsrWXcG5vv6FmTzlg8O2Vg9YYpbWl2WIfkjKJ+snf/hPLt7YcVzg9w7rj9JwcqmfSi3Qyvl8J85ZmLWxDZFprXosojarQ9/TmpUA7DHUVXTn+nzL7dRnVBTjIuDWmu2gd8g04jo7vKk25z6RzvWzSHraLEJWno+l9g== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: QKnaUpw+70WBUYaSZe/HDJaIaTOSySI3UeXkn+reQInoVSGdhuvGqYZAqvZ5Af0aMaxknV2pxHwgqrkureYhPihVFELSOuZw9l74RXIsgXZU8QJydkyc1H38kct/NGOc8A3u1E5Sx8zplIj2VOI/t+cm20eIfrlTdKpk8Smxp65uwGfQUsRHX0vwaAUOEy81x4BD2GtH8l1Vjebqcmh5O1mzydnO5yJYLvpfGMqox1k43cKt/IlmEDu8Z9tICrzrgCDjuSPitox+EjTT9X/GjljgWYbfR5+xM985VlFnbtZixwhRgCrrF/9LKiwRA7L4C+7obAeiSDMe/WAPucdRUtIE17E61KTZf+V7qWvuaSHutNbChIumwwrP+WSsfE3oBcIV2Q/xVY8duGS2OcrkVEGWdgUncdTGhqPYu0juvGYkRh2aaxNf79MuMiC36cqUhPpK+RsotXWTfRgBf5LqOUj7DHnpqWsY9ogOHmE2v1TC59WacSogAEtHUl5l0eJpmJlbHkLsf21cv9Y4CC2442eQna2xw2t107m1bTH2VXjP2k+K2a6MafDW+BguXzNOLEkT9g/NchBekhp94oZfBmVlx0SgRbKuUrXgzrSYncRip1FEJzBKUBr9lR7FujjlEAx0sAgjWFsEUFK6hwMSPZK1USDLk9Z0XM+nc6AmFj14vAfaPsJ9JOKks260bhimPs+kRDQ8hY7MydiRSwJQtAXV8LbfqX0NkKj+nXZQ5Ai9pr9FreoA3cjZ4ZezWQE5PopI2ykwOhh199KxLcxsp2yXFNrTDFXd5ZGoJuoNhFDsSlmBMmrAf0cazBvGgNB7g+6PfQTR22xSNtipbVHU2BLWp0xkMfHZaE/A5YnaQo5FJ/NQhRLg9sugdoUIS8neEnLmTj8mBd9EMBTn78FL9f0aPPLymFey5C2wjNnYNxm9v3v2OoanbNrLq9zHMXS/nvOnTXg5/DWxlg9PmOe3AdxbsS/liylySemyU/1EtpNLjT74a7J7MD3g89QP3nLwO8S4sV8i+AXaOKGtqGA1nMboDD4cOCRQasnC5f46iJJkKWWh4YYzj03RFw/d8dE/v1WLS33/3OpeOrCneJY2o5aN1kbLZvNcS90HET9KzcM03nbsf9JClSODTo5VndyO X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 607d7e75-f540-4546-281f-08dbae5954aa X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Sep 2023 21:44:50.0122 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 4AmqggF1xTn7jLUJYAXudQFm9N4dS3HlSSKAfx4UQ5An1ges9Cp2e34prW3OUBRDbHZYj6U7qmJqykjJarQnEg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB6395 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-05_13,2023-09-05_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0 phishscore=0 bulkscore=0 mlxlogscore=999 malwarescore=0 mlxscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309050189 X-Proofpoint-GUID: di88WQWgAm9obJiwKwKkiIXORxKdnhwS X-Proofpoint-ORIG-GUID: di88WQWgAm9obJiwKwKkiIXORxKdnhwS X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 8BA4B100039 X-Stat-Signature: j3gdkgpj5stiq9t7drg31i5mx4feb6mx X-HE-Tag: 1693950316-308699 X-HE-Meta: U2FsdGVkX1/VdJ3YxDjjPXzxAhds8vqqrvcbCYj0ZU1zOt/k5nF362GsmIfN1pieQieFUf7OzOmB79I6zffNwewuulHWpCTeGKPCe/f8wFeytxq9JPf0otNwZBx++VMGH3YxGZtddwS1ZA9YQfX0w0dHpQwKAVjs8Cacm3vM7vUbFiZSVFxTn3+svGTLXMIg0t4HuPFkZ49O8IcPMNEgFAnHyj8uDEDtWY+NrbAVZfdhJqYFzyf1G/PaSiCRNdpbeXb00CwB6zI8lD/EcPdBv4IYZPtoPH3DHjF0cTS9XH/UQjbO+4SbkMdq6OkbWS09uoC4/GqmDyZgJLcg2L3ccmNZ1vQVav9NJ+Z19fBIYJlIUD3MdjN+Xuqu7cRVB4BRmS6xIN8ZN3RMd+Ia9+OFJ5BLYrS4chGg2FSd8qlGpxO4l9O2ooDbRf0rWQAXf/aB8nD23yMzk5eqMPS8PcVoXqJyXzQl5bykSBspYYyKPsaXx/XuTV7H9MGwgWmCyZIlhd5nZG4VNTN6wCcxqNHd1MgpfMkic3bPvw58FuXl95YmZ4Km3P0I/mjhFFeyp1Qb6sfC1xjndEvlklaqeVrvGJVZH4Bzi03mMJtRN+rICStXucpqOo2kldXRp4+4PSvCMjoXmjxSPRfGFOeB8rpXnVsJaSVAZzXs/A3W1+zT0l+8OHWF3lKgCoF1ngjAN8DjTIM6UTjDe4cg9mpUi9dWUxy7rIllx3RtoRrT9lid72jvu3WO00jRIfJsgkEUnLY4KWfQiV+UCfLn0RGb3gxRGsVy/vHYhdUr0kzPtfT6atYW7j8dsdW0HFT+vOg8vSMRrYkJgmu8GBMYjzKLwRoD5+wgZgoc9ayt24bsZPVvBnuWCMZA1Y0bMT1zkw1WdZKedyOMZOYo2LyfMC7T9Abx2Cm6hbMojaQnkmswWBiu6qYvz7zYN4NA/l2KewWZ02x8S/5tHWm7n3YXWpp9F/N KFyXNw6K rpniFhHEvDw8Fz9XiUVOhiR9gsvyMkYT4nF8HyG00edUp4g7EBUyggm3pdhKC7RueGFK6tzaPElgbWvLAKQ72kpPwIJ+L7eDYCxCxeB0CKg1eFM9Y7pTRf38mBpx5PL08W2MHA1OovqJLMAZpSbtK8Q5s4NW9O6uSN84Rsek7pXNortX/c76tELqazSObhCY73EYyOOYkFuOAtI537Sl1h4qGgFF4QKL99DX2nsf8ItTS8GSX5886FZg9f9kjAq5syP/LTYx57cBZxLPyUs0qbwdgy653kzS7PB7gM7wzJuhe3UgzJNXJ55mhvR/JI/ybBxPJlia77o5s00qfkohRT2wgBU3ck+PsQ160al6M6gWXbszDHtPkSXfGofDLWRH+4JmS9VPGCvTgqc1RcTouBgV8QYuNfqHWjAXBC6Vt0jIesxIcnAbiMwq1Yvve2fFPfV/5xDuQgmrDLv5ey+B82q/UcURGbJWRKRA9wB9Fd2u6kp/F1vw1FdvxMMjqgoCYU2zu X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Joao Martins Now that a list of pages is deduplicated at once, the TLB flush can be batched for all vmemmap pages that got remapped. Add a flags field and pass whether it's a bulk allocation or just a single page to decide to remap. The TLB flush is global as we don't have guarantees from caller that the set of folios is contiguous, or to add complexity in composing a list of kVAs to flush. Modified by Mike Kravetz to perform TLB flush on single folio if an error is encountered. Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 38 ++++++++++++++++++++++++++++++-------- 1 file changed, 30 insertions(+), 8 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index d956551699bc..8c85e2c38538 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -27,6 +27,7 @@ * @reuse_addr: the virtual address of the @reuse_page page. * @vmemmap_pages: the list head of the vmemmap pages that can be freed * or is mapped from. + * @flags: used to modify behavior in bulk operations */ struct vmemmap_remap_walk { void (*remap_pte)(pte_t *pte, unsigned long addr, @@ -35,6 +36,8 @@ struct vmemmap_remap_walk { struct page *reuse_page; unsigned long reuse_addr; struct list_head *vmemmap_pages; +#define VMEMMAP_NO_TLB_FLUSH BIT(0) + unsigned long flags; }; static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, bool flush) @@ -208,7 +211,7 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, return ret; } while (pgd++, addr = next, addr != end); - if (walk->remap_pte) + if (walk->remap_pte && !(walk->flags & VMEMMAP_NO_TLB_FLUSH)) flush_tlb_kernel_range(start, end); return 0; @@ -348,12 +351,14 @@ static int vmemmap_remap_split(unsigned long start, unsigned long end, * @reuse: reuse address. * @vmemmap_pages: list to deposit vmemmap pages to be freed. It is callers * responsibility to free pages. + * @flags: modifications to vmemmap_remap_walk flags * * Return: %0 on success, negative error code otherwise. */ static int vmemmap_remap_free(unsigned long start, unsigned long end, unsigned long reuse, - struct list_head *vmemmap_pages) + struct list_head *vmemmap_pages, + unsigned long flags) { int ret; LIST_HEAD(freed_pages); @@ -361,6 +366,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .remap_pte = vmemmap_remap_pte, .reuse_addr = reuse, .vmemmap_pages = &freed_pages, + .flags = flags, }; int nid = page_to_nid((struct page *)start); gfp_t gfp_mask = GFP_KERNEL | __GFP_THISNODE | __GFP_NORETRY | @@ -410,6 +416,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, .vmemmap_pages = &freed_pages, + .flags = 0, }; vmemmap_remap_range(reuse, end, &walk); @@ -597,7 +604,8 @@ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *h static void __hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head, - struct list_head *vmemmap_pages) + struct list_head *vmemmap_pages, + unsigned long flags) { unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; unsigned long vmemmap_reuse; @@ -607,6 +615,18 @@ static void __hugetlb_vmemmap_optimize(const struct hstate *h, return; static_branch_inc(&hugetlb_optimize_vmemmap_key); + /* + * Very Subtle + * If VMEMMAP_NO_TLB_FLUSH is set, TLB flushing is not performed + * immediately after remapping. As a result, subsequent accesses + * and modifications to struct pages associated with the hugetlb + * page could bet to the OLD struct pages. Set the vmemmap optimized + * flag here so that it is copied to the new head page. This keeps + * the old and new struct pages in sync. + * If there is an error during optimization, we will immediately FLUSH + * the TLB and clear the flag below. + */ + SetHPageVmemmapOptimized(head); vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); vmemmap_reuse = vmemmap_start; @@ -618,10 +638,10 @@ static void __hugetlb_vmemmap_optimize(const struct hstate *h, * mapping the range to vmemmap_pages list so that they can be freed by * the caller. */ - if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, vmemmap_pages)) + if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, vmemmap_pages, flags)) { static_branch_dec(&hugetlb_optimize_vmemmap_key); - else - SetHPageVmemmapOptimized(head); + ClearHPageVmemmapOptimized(head); + } } /** @@ -638,7 +658,7 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) { LIST_HEAD(vmemmap_pages); - __hugetlb_vmemmap_optimize(h, head, &vmemmap_pages); + __hugetlb_vmemmap_optimize(h, head, &vmemmap_pages, 0UL); free_vmemmap_page_list(&vmemmap_pages); } @@ -672,7 +692,9 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l flush_tlb_all(); list_for_each_entry(folio, folio_list, lru) - __hugetlb_vmemmap_optimize(h, &folio->page, &vmemmap_pages); + __hugetlb_vmemmap_optimize(h, &folio->page, &vmemmap_pages, VMEMMAP_NO_TLB_FLUSH); + + flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); } From patchwork Tue Sep 5 21:44:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13375117 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7231CA101B for ; Tue, 5 Sep 2023 21:45:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C66E6900006; Tue, 5 Sep 2023 17:45:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BEBFB8E0014; Tue, 5 Sep 2023 17:45:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A1918900006; Tue, 5 Sep 2023 17:45:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 889E58E0014 for ; Tue, 5 Sep 2023 17:45:21 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 5E6F1A0AFA for ; Tue, 5 Sep 2023 21:45:21 +0000 (UTC) X-FDA: 81203875242.01.6DD8962 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf29.hostedemail.com (Postfix) with ESMTP id 0D6A9120013 for ; Tue, 5 Sep 2023 21:45:17 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=3gZYo+M3; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=n1swxgOF; spf=pass (imf29.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693950318; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=65VawqEObe3ud0azxInS6sVwyPtLcLotMrZynBuz84M=; b=D0JD1meiviJXgmT81YnxiytCnEergw14KTWjEbh9RDr1p7ZzfThYpisc7IhD0aVIM2r/Po rUiujgzMM+szPtasaA1KP3G21cO0WkOGe3DLmiiMjqugF9OYzZUjsFABQ7tnu9NEerd39k OrDMlIlQPzTDaeRYfasKwJTyFNGdkwM= ARC-Authentication-Results: i=2; imf29.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=3gZYo+M3; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=n1swxgOF; spf=pass (imf29.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1693950318; a=rsa-sha256; cv=pass; b=bcY0Ri/GG6Btj4jE9n6filLYepPDA4DNWLo4HyXDXgVe02wppPE1HiYyjzNW/4o0vzgFN0 h7+LCkwLAxDlNW3emfmaLsqwu8DwkHNCgsuAlCVMvWJ3bA2IQ34Im5hwc1XEC2zQxcH9BH z319HayestRsV714CEtkleOv7pfOujs= Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 385LA20k007098; Tue, 5 Sep 2023 21:44:55 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=65VawqEObe3ud0azxInS6sVwyPtLcLotMrZynBuz84M=; b=3gZYo+M3S2ZHKHQyEN/9V5zt2jLzLA3lahK2soGCZalQybTA/7oTf5hYQr0S4065LWzS gvRAzlCnjMgk9ybO41yjd96naP8gol30RfhaXOiJpn68BaRSe+60xqbbYUHamcJ7nWmP cVaUt9AFQIwzlQg9Lc+hou1194XzW02dLg6WeISU0XJBbHkquPnLVxhcTycIPsuaSpQt 2Fdzl/oNAOsFKNlvGQFdP1wuTqSVpRGpDRZvj+DI2HhZAgBJpWYXlm7DJGVzD1W+n7yI lkQTf1edvi6QfxBysPMfL8qFEb3i9tito10KsAjjE58BPdmOBoMmmCDwDcuVkIDM45BB 0g== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sxc0g82tp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:44:55 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 385K1v9i037100; Tue, 5 Sep 2023 21:44:54 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2107.outbound.protection.outlook.com [104.47.58.107]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3suugbkn9a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Sep 2023 21:44:54 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jmXFOctDdZPfqXEnx1mEXqCLAFRitN9/fg/jKFIdV8736dtaQfjQAKFmQ3dQdoDLDmCpvIHeo8HZbuR7zfTSCghy38fB1PCBTZbQG+8zsJT5PnuXAWKOCGhVlTq0EqQpY9dsMEMg7LjIGhE1+Mb2qC0JX3Yk3hNcMCsrlM9LXzeSMTaL5VypTf91hKh/f++sS3RiQT9R1zoJIGfQ8RlD8M1QSbnrO5u6Ktcwq/OiP/UOa6iGM/QZ5nU2IqNE3LrJV3xtuWbHoBHuPUqjboa1ofAaeQgWIdearLl6bWPvbWbPDc5RjjhJX0yXA90SlYqb5DH88wPw3u3K4XVrzi5n6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=65VawqEObe3ud0azxInS6sVwyPtLcLotMrZynBuz84M=; b=MzbrvRypVf84n+Tt+LGfwsdOJgnfY4Rc3FG1cfDileCDQ3wlMomszwhEwLALfL8Tshk825jlWAGrowOG0ikmiWqj/UEZ4+75g3lHPohb635qsgogofP0+n1QCE2fFkIZ1ZDGbyNTpLeVv3JVsj/R4ByPxDgnthmE8pxGf6cUfpgy/cXsfWl7+CSQq8EVq3D0HjiQfGxE0bEE6Q1vMaHIlDY/kiE6nprxIPv7gSno7W4hFOzLgO9WhsZsnDvYpysT3E5jmTeLcE6en8khwoLZzPx501Vfx1eXNnZCsMwfEvaQRDAkrvndRpzGPy23fh8pF5Wgng2seVqgqcsesJM6xQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=65VawqEObe3ud0azxInS6sVwyPtLcLotMrZynBuz84M=; b=n1swxgOFK9wtd0EXaGeqztOX7a+e2pocAzuq9tuyOjWiHmTyY+FtpoGdtu3RQRgSQl7YrECBUssDJk6mit1f9qfBxeAc6YbbUTLLD8NSJnErqtSvsYNX7rS8YCQApFoVAWakngHr1VluAilBuP0nA+4GD3Sx6Ak4tjcYyDPRbig= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB6395.namprd10.prod.outlook.com (2603:10b6:303:1ea::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.33; Tue, 5 Sep 2023 21:44:52 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::58d8:cf97:ae4b:1715%3]) with mapi id 15.20.6745.030; Tue, 5 Sep 2023 21:44:52 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v2 11/11] hugetlb: batch TLB flushes when restoring vmemmap Date: Tue, 5 Sep 2023 14:44:10 -0700 Message-ID: <20230905214412.89152-12-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230905214412.89152-1-mike.kravetz@oracle.com> References: <20230905214412.89152-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0302.namprd04.prod.outlook.com (2603:10b6:303:82::7) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB6395:EE_ X-MS-Office365-Filtering-Correlation-Id: 31f78125-a4c6-4a12-02d4-08dbae59563d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: s6mssAv+T6t7N3e57Tpv4QZJYrbeCZc192hl1jP+foO2hGveDwn3cSJ54ut879HspZio3rFMyMivve+24UAsH8cHQ2OtKuSCnO9YoUzF0rv/5hNOgpvjs6hwNYxvwv+VCsdopw0OpDu788XRieFv/C0+qTLyPj1JrJ+ZyHoK+NMgOildMbS/oR7Wwf8GfzIiYA9fcjtlgINadS1W20rEmcB1LrN7sJcH6S9BvPEbvhpPmGZUDoYo8OzG+zzOjqCDPKkLrwwCxfJaZxmo6PduI5kaHSYtvu6zF0srY8JZ388r7dAUfezzpF5iiAuu6bk4tlL9NzXIJ34W/J8H8tyeUJGIBiIfxhCUoOJxB8DmYJAKHBH8Ipba8Z/4vNLOxztAh5yZ8AXLcSfton84JThVXG2y/8yCab4D3BQKezn7ovCiBxCoarC3Q5VCjEqekI82SgHToy3lq9P7SspHgdY6ZInb4aa4vx5cvDYdSH7EmKY4PCXwbLEwkM1fo3yd+4lg3v5Z5YO7LkGcprHeKW0vJaC+e1UwB6LuzK3q2CF5jXHEWGKXDvrifJ55cdNUVmJD X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(376002)(366004)(39860400002)(396003)(136003)(186009)(1800799009)(451199024)(6666004)(6486002)(6512007)(6506007)(478600001)(38100700002)(2906002)(7416002)(107886003)(1076003)(2616005)(26005)(83380400001)(86362001)(36756003)(66946007)(5660300002)(54906003)(66476007)(66556008)(316002)(44832011)(8936002)(8676002)(4326008)(41300700001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: W4pYRwXTm7p0/vd9hnQ0pBL9sQLW3CQoBGvzZbRpT/CdAtUMu1UQNiIm0ROFcqSEtRK0FgX5oFgnl+yuPZ+3wz6inJ5VPPD0eW7Zw8Y5n/iGltPTLIhqsQzfejiVxAIt4+w+hAT6oGWlYvB60fM+InpkzH/qP2FEb67Nk9TBT021prvrFYnK9ggwWXHpldu/OBbjlf9KEQ74ptuWg6fTYD5hFlsVg+gJjDcRs/uULI81H8qataB8twmDubVf9miJ0ZYfU6HLKx39m+CRYX/7dft347yELgnxkQ2WgK/+e+9T7DfK8GXsKFS/YXAVfTox6c5M2BDvuEmdnDPH/VdYUb+I/cjYmbxjsuAvhgLvPDMyftDaroD+8LhBHFCXnksqVmduqGqnGQBvNkeBwIqYk90T+b5xzWSMSyx1ECKZ6L9EqRWdQqPwpt+O5XD+Nz7hirvLECZ2TD2JJlQFXO8/gxniJv+VPrwxKZO+ucax3b50QPfpD3499/2pO0YOp77enzITiRW0fFjIyy5rl48HcRtS78Zx4Dj1rGLY2kIIRHvZCT4TqVmCN1hNalDrwc/5kxY2iL6kXLBmT4J9FXQF1E6TTtZvkRrUqCpZS62hmg3ZXh+Sc6Dswi9+HA81+Ueh1vheXRMONJWUV6y3M9fklXD/4crGGB7LRsmTaCjkCGxOtoxG0kl048nPFAfilFn2knPCyD7dGOiKbyH1W5eyotO2dNmt8iy5+zENN1MGGX1VX3Ji6wQQePNMUoV307Tg/xDh7MxDLNk0LWRN583q31xEXHoUUqrx1Sa5Uhn0rsUY4qhfw4FKeMF2WdECeJfcO91LJV633x/TyzE2cQffQgrn4o6WxCMxcMdeWDdScOJZEYypjOgRJkdL3KLZm1r9P/coDfYimv6K5ctjl4lunQjV2jqT1oqDdrT5un9Qz5fXPWFp4Nr7qlT8dEeKXpyyn4GQZogdQnVSa5UtrW3BUPY4DuBp0KnmdcMohf4WZQNkvtuV1WHy09eePSSFSofWiDnW6aXWpi6WXoxNfSVzc6gf/G1Aycd0XqC//wa+q//5LHfkYMV8dxPisPOFWqK47h7SBtXfpH89P77h9R61zEcfFeMADIWyn+Ehx/k3XycQad/gCrU4y5meeFW3JJsoXnTOWnNVWj45ZZz1teykRJ+i5wNOV77iwBFZdLv9pOyTIsNlJg70zVX/U7Fe7OvK+66STDt6d6izG0fsBj7ZcmM9t135Xo2+Gk9z8eYNLSOhjghSehgWkwbnQLcKg5nS+Q1m+Sx0ExLXH96zw/6WA2dmBp6BXkxM4tZT075OyrK3QZyCOgLRxROucleEF2Ix1xU+6JC3wkfPL/7475L46KVTMJvL9hvG1k6RI9l2i21K1+B//XASJs3xNEg1F0vvuHMPnjzkYE4aWUgSWnauSqn2NN1fmVW3GtfR5slpJrqjOeI7k7ibF7OdwTqKtzzk73wKe9slYO2YiF3B2H+hsrANDLUyyKkbDkb1iG4NxEMpRViJJ3NBXRqvH8UN6V4FFAFBM04+FhGBFjAi7eZASwAEEpoZTf0v0UjRE/6D9g2dOZIphDmYl2YAuwQ0ZeuxyzcNZKgsiSBuCO4RakoDnA== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: D3qZyfPXygtt7xGMjxVu3WLXm8JD4jhBT9xbDsKO82u2pexUpliIcB8aBfmwZVRpPz9DKb6OLTiApq5GZ/WIRIpiLV97SyVdolBENTzLgDJ5O7Wmc8/sDl2wUQFB3m5QQcegrKSnh98DmgVPIu+w4GQkVXGfKvqjGuVKDBiMVU8UkRAx7DYsK+dHr8D376TD63ZcipxTJ1dVPwbAcig6GkhNyac+txk06eR8q0hJjHmL4+Nq9ihKtHfnaIEPDHm/bgMWEooBLNawmAA/fAcdmflCGFifN0/+Q9JFeOKBr7tS1XnWVNIyg8rIoZ62vz2gEC/fq4rmcz/Nd5w9CVoOWDyGjC9I4Ng39gF1dVKqhyXZBmj1cZRrqLlpVnYgDdhtPkePSPzYKyF4zpyiL7SF9VMZaWJssUioMNIQ4X2bwNtua0CbyrXUJ9kmdjJsQMPV3fEOi5AIUcfWpfw74rXNafBWrQBY5slKppwjQRePRqpz95gP05rPzDRq9QQHpgyVgBtJKHzQUNfjhhcYJtgPyNGsVp+vYeTrM3ScTvWPUDbvbMxu2v/02Yn/YTycBIPSN5cOHTItB+O9fHQsB4LD8zwe/342BwIYT4MHIH28VvbBPPnGsQ1DO5q/fr+gOVnS+XBfRw2fGTW8Z0naH+EYI09nrCnngAaQalLtDTNcGMp6/SgN92vd8xCkIZhBxhrgau5/9TnIkBHJMNz/Tn+c5oltRo8PicEbR0NoSsqrze6BuHUvz26dQRlu0tPITMSeEd+x0lUZsgRsM/H7HATkEUJq1Exb4SLiVrSDyCGtdVTO2nvU12FIzJheXGt8GwPddSgcDoM9FNKP3wZKizSmmJ7m2Fpp1twoooXigK7y2U27ud3v55YW5RPM8etsFeIIeXtnilS6lhWM3l00iuppbGmGuHwRVVaG2DIjP5fFTVmuXNKEbXjI0/LvIoBaxnrMYnkNvNk9dHcKOklmaomQzkIVO71z/DLhG3i9I8rxIDzD8eJCc8v1T5gCn4+sdIuCEflV+2guY2hbX/ivY3OhdcMYuq8VOTk94brE/ratof/XQoGo+IcPD7Q77OCB4QKRhCCWuXEyPY2w5uBPmrcR8ClBJ40SxX4MAJ+ogIs8q6VSj4DnhBBJhCfRj4ZAl9dS X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 31f78125-a4c6-4a12-02d4-08dbae59563d X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Sep 2023 21:44:52.6340 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: IcGPDIz++ivtslhqJHdGTUImehjDF6EjjsJwMyB6eG01+BvPqvdsDufovCYAYI8UPNBSB/64XvVo0l/QvCUtpA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB6395 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-05_13,2023-09-05_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 adultscore=0 mlxlogscore=999 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309050189 X-Proofpoint-GUID: RFDLOWTO6skv1qnA9hIr00iahrAWKeCQ X-Proofpoint-ORIG-GUID: RFDLOWTO6skv1qnA9hIr00iahrAWKeCQ X-Rspamd-Queue-Id: 0D6A9120013 X-Rspam-User: X-Stat-Signature: yxcyxagkep4cpankmpkhecerdtnpo4ju X-Rspamd-Server: rspam01 X-HE-Tag: 1693950317-394891 X-HE-Meta: U2FsdGVkX1+R2s5fbCTziZYifcFikjpToK+GSrtKUgXLUbc5iszsChtyWy8tnK0pAYRdojjvkrIoXA9fK7iVuFa7dK4VSmh3vIbZ+6143QTvlv/JE6k51sa34qb/tW0Cb7fCVjJgqB+wP5c5pjcVBiLTb4kxSpXWkL+Ghj9xfhH0lzoaWf8hdH3t0X2qTQidbx71R0HnWJKIx5yGKD386z9AEKI9WUpaOgIBCybACC8rTOvjNmPneGrtISCgF9U0cHSFpJrG2R4N3/Ihwbe8JQJyTcZyiTHaYU8XePmXRln/qgm7zUhWseBGnJgv8CTVT5WmrCQ51fjFufLY4YWo63xJ8ZKHg0NEDXp6JOSK3wRoKrPvNbpyxPjy0cEwMsmdQ3lKjbv4XSudmnUlROOQXi4GndMohQJ+hrcXC0TfCq2nhPNV6ZxsKzngGoiLa9r7vPF6bNGkU6WG2S/mpFcIcQYZPV8h6jR+cstfenoVDB2ypXZzRQrqZyfS7jW+4x05lzhp55rhKiuCtTemVTcK6yVh9ZZqFf7Fp8TMfj3Ln41KHl/4jp1adhqdwG8JuDXB1/Oav0Wo5qBfmDNDHGtIVbzrop/pQe1mGqDJMI5msD5tF/+6bvC4sv3AXN8JjGRvDqru011uJBUjye9lNpuL9iNTQcWiduzSc7ODZJyarBBc4lDQzWObQ0nhpZ23po1bDislq6J3wnBnjob+fQGQbF6ZCzKOZLguOp6dB6rR45ObwFhAZjaJnTIzAWgKxPqyf4l2IvT56JZSTKvNqJ1A3Z02T5PJ+rWowzpRu/+dUqD+AfJpXgqyqSg5xFIiEB5LBeWuADJjOaTSo9XqC0Rmsw5JRUAA0u2vwzxZG+4ku8maiQqIFnV/JpEEq6Li9xVpX3/Dk/EX4iHgPYeRC183P8iPx/ELnNxg0ap7kxDRPs/PhTZNtUDTI3Nx/x02VX78nyCNCWQl48pN1KnJXg1 i8MMjihO i/6UufRXQQMhBr5NZqtAsq1GKM4x2J/wm7Q9CT+dS8fZorbxfqxef0fr5P/IlD3T/AjhJdvn5nCWLHret3YXP/+AP3uXWMiweDwB05zTSsib0wSF2YPi1MQ/3LPRhQDmcZ+TQl+2RJeudmd++r77nzSgxUOezJJBFa2AgeuCn+BEyz9+ZX+OzShuV24fAObRBJm0PPRxmLmBs+dlC0OPO8GR41uxDBpe5avHML/JsNk6cF+KI64OHDdG52mDLhoLXNPbtrCMSs9MzHQ55gwq/Yx55dkMqbwFeFaVQ9+njw2CfWSW90bDHE5SgXcLhEbD/AAM4etAg4vJXW5bZ0tynOVjp5iXjRItb77Q2ygBS3zQkNaHkz+XqRXLEzECcHDbHJ1Y588jnMMlpDYGPV9YjlVPzrYY6kXxvMVppdhMD5iwdSJtrVpiOS8w/KSVqtrMRfGBhnzyg5UgOX+ah92IEn+pz9g5LpEbGyXDUnH2h8DVxiTjBzEummqFCA1InZitxmD8B X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Update the hugetlb_vmemmap_restore path to take a 'batch' parameter that indicates restoration is happening on a batch of pages. When set, use the existing mechanism (VMEMMAP_NO_TLB_FLUSH) to delay TLB flushing. The routine hugetlb_vmemmap_restore_folios is the only user of this new batch parameter and it will perform a global flush after all vmemmap is restored. Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 37 +++++++++++++++++++++++-------------- 1 file changed, 23 insertions(+), 14 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 8c85e2c38538..11fda9d061eb 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -458,17 +458,19 @@ static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, * @end: end address of the vmemmap virtual address range that we want to * remap. * @reuse: reuse address. + * @flags: modify behavior for bulk operations * * Return: %0 on success, negative error code otherwise. */ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, - unsigned long reuse) + unsigned long reuse, unsigned long flags) { LIST_HEAD(vmemmap_pages); struct vmemmap_remap_walk walk = { .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, + .flags = flags, }; /* See the comment in the vmemmap_remap_free(). */ @@ -490,17 +492,7 @@ EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0); -/** - * hugetlb_vmemmap_restore - restore previously optimized (by - * hugetlb_vmemmap_optimize()) vmemmap pages which - * will be reallocated and remapped. - * @h: struct hstate. - * @head: the head page whose vmemmap pages will be restored. - * - * Return: %0 if @head's vmemmap pages have been reallocated and remapped, - * negative error code otherwise. - */ -int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) +static int __hugetlb_vmemmap_restore(const struct hstate *h, struct page *head, unsigned long flags) { int ret; unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; @@ -521,7 +513,7 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) * When a HugeTLB page is freed to the buddy allocator, previously * discarded vmemmap pages must be allocated and remapping. */ - ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse); + ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse, flags); if (!ret) { ClearHPageVmemmapOptimized(head); static_branch_dec(&hugetlb_optimize_vmemmap_key); @@ -530,6 +522,21 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) return ret; } +/** + * hugetlb_vmemmap_restore - restore previously optimized (by + * hugetlb_vmemmap_optimize()) vmemmap pages which + * will be reallocated and remapped. + * @h: struct hstate. + * @head: the head page whose vmemmap pages will be restored. + * + * Return: %0 if @head's vmemmap pages have been reallocated and remapped, + * negative error code otherwise. + */ +int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) +{ + return __hugetlb_vmemmap_restore(h, head, 0UL); +} + /* * This function will attempt to resore vmemmap for a list of folios. There * is no guarantee that restoration will be successful for all or any folios. @@ -540,7 +547,9 @@ void hugetlb_vmemmap_restore_folios(const struct hstate *h, struct list_head *fo struct folio *folio; list_for_each_entry(folio, folio_list, lru) - (void)hugetlb_vmemmap_restore(h, &folio->page); + (void)__hugetlb_vmemmap_restore(h, &folio->page, VMEMMAP_NO_TLB_FLUSH); + + flush_tlb_all(); } /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */