From patchwork Fri Sep 15 22:15:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13387782 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F40BCD37AE for ; Fri, 15 Sep 2023 22:16:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7FAF96B0379; Fri, 15 Sep 2023 18:16:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7AAC36B037B; Fri, 15 Sep 2023 18:16:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5FBD86B0384; Fri, 15 Sep 2023 18:16:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 509AB6B0379 for ; Fri, 15 Sep 2023 18:16:34 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 212851405C5 for ; Fri, 15 Sep 2023 22:16:34 +0000 (UTC) X-FDA: 81240241908.26.C8FC3C1 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf02.hostedemail.com (Postfix) with ESMTP id 9A4248001C for ; Fri, 15 Sep 2023 22:16:30 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=bKdst1ZO; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=JQ9xccCD; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf02.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694816190; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HDEIvt03Ie6K7+bZ4EHI6Dnv/+9YtZuhxJ2Q9zE/vFs=; b=ZXAwgFtXTct6vUhQWSwy3G3mKIEI2v9rl1w0QDGXR4/+Vgo6U3SQy5LUSBpU54enDLkUAp wRkOIMZgxUt+I2PdqajEydYgSF24fIyNBfE8sp5HJQ58QO4d12gz2ub3en22AX5gN/7aBj 1dz4E55TaJJlR/bNNhjcsUcBv/u1zU4= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1694816190; a=rsa-sha256; cv=pass; b=ja+8e/Dwlc1OPRWhq/ihYre8B6NLZuH2xAMQksZ8D8SpwUaOO+Wr7INxOnpBwNq49KkRvR B0Ud11KK7ZjeXnJAf0S5RiF/EStRFP+hBM1EB6yI8y8ium9YMzjhp0fOX8uPBNt4IYbeQb dJFN0WZA4QtNSODrGK6kO3Vr2RyrsUE= ARC-Authentication-Results: i=2; imf02.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=bKdst1ZO; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=JQ9xccCD; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf02.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38FLuFa1023039; Fri, 15 Sep 2023 22:15:59 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=HDEIvt03Ie6K7+bZ4EHI6Dnv/+9YtZuhxJ2Q9zE/vFs=; b=bKdst1ZOMgWBe5TzpcmNeLwSbJgSVWFJbtMFcc0Q18tl4mVixxtUmvxO1yLjV+bamHqM n4Ah0xDFdG3WSWTpKdEEs+e0z7aJmVv1lcYUkX8qfWChfTV5HB1VplumjMx56TWx4O7F 6RpfyM1umSM1Wjx8aGtxzDQKYupF6LbIUxb6veajr85+iaAaRtbjh1uWWUOaZFHF0OCG vPE8RMRGB2gYUqSYK3PMVPvxsLxgzSk6nQwit4fsQvuGdnyCw2uBBLw108qN2cawntWr f2YpvcPgt8j4vA7029Q+nKA3DC/mPOTM7rGgA1L2raVn+oZ83V8mLIESfVy3rmdOgmAy DA== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t2y7n0k48-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:15:59 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38FLQxuf036173; Fri, 15 Sep 2023 22:15:57 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2170.outbound.protection.outlook.com [104.47.59.170]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3t0f5gx8s5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:15:57 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cb0xQeuTdDBk9XJ2YSkBw2RS5aUKOdDU6X9Ej7egnaNo8lr/1upsTh+MFAkRCuRz8e4bbKdyuQhJCJ5iY05Rt3lL64BU0npxPk4zFEB4Ts8F0V6aJWeHGMV75ARtVo+Fq0UO3KOsMkblRVpQx5z/GhYlLlth6tbF+SFUgAw2wIZkHkVtIKmZ61ROSjuFwwEjJ26kKPDEePHxms0T207v88OdR7QU3GT8HtgBkv3Dzudyh/p26+oqxtAkePscPCCeT4akRQPdA+1KIF7L28+kJu5EFz/9oWuDBPN5MHq1jpNUiNaVQoAj7HCz/pVVd8GB1BD5x95wp0337C8c2pQBbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=HDEIvt03Ie6K7+bZ4EHI6Dnv/+9YtZuhxJ2Q9zE/vFs=; b=Wg7KXq0a0p4k4PJXrR88sHDOjyscVqKfzSugUNYHYKbMaFzCi4SyKsCxPKLtXMRuEymtIhbN2WvBvjPUUnALfxiWSPh3osJJd/ReBIe8h57cmpS8kgBuR96kcQoHHxRdOY5ZZMoMTKm3fXnBjLPzUs7bljXf06zQB46W7vbJWrJH58RDGt7YHunwtcDiAwm5mGPyvx90Ways6fKqwnQkgoE1XLuGvCJPNPaY7zoLE2Qp7kF4FhlmAnxVKBlH9d/qvRa8X9wqDHX/u5VPvJgrfLXB2tErFNTqiemmecT7hDJs4VN1xTgvL6hYUvaV/8KIa2bBbVVfteaVFDkn1/uEAQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HDEIvt03Ie6K7+bZ4EHI6Dnv/+9YtZuhxJ2Q9zE/vFs=; b=JQ9xccCDZInfaQWwK98vt52yFPXcqvFdlaTqSz1JvDykCvasJYH4/scpe9izWjrDSzlquKrnnGKHoXbohXnyfIYmvKltTrxtXT2jXLgGYC/qzsprqXw6/nY7glm7B7138yKDBliH6qSz+FhpEDhWXb4HHggFiWaANoaU39P+jng= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB5861.namprd10.prod.outlook.com (2603:10b6:303:180::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.34; Fri, 15 Sep 2023 22:15:55 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.021; Fri, 15 Sep 2023 22:15:55 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Johannes Weiner , Mike Kravetz Subject: [PATCH v3 01/12] mm: page_alloc: remove pcppage migratetype caching fix Date: Fri, 15 Sep 2023 15:15:34 -0700 Message-ID: <20230915221548.552084-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230915221548.552084-1-mike.kravetz@oracle.com> References: <20230915221548.552084-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0344.namprd03.prod.outlook.com (2603:10b6:303:dc::19) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB5861:EE_ X-MS-Office365-Filtering-Correlation-Id: b043899b-d920-4566-d9a0-08dbb63954c2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ctxyRL+zTQRoubd+1QYu+DyYlW/CkQQU7g3iPS3IDGul9cdBjC8ZfhQhFlmdIbIDHxk8P3LjuptiNOQAymt5NnkXwYNQ9NCfGasOEm7xfrR5hgh2jBZ56vbigUmuVkyZzJWyrBUXXpVAEn5e7jSHRT7IgxTbEFeUNtlAnPPw/1hVZ3lk+uHGoSdhUrLSOuwNezvHZi2x4ipcncIHRdYZtq85tuV9SnnzyiujcBDYvhCYaisCbdgVJpa+mLkDgMneJ/IiewfTghuDGlhYjl1gobDDD8IS7ibw/EUYaOjDRbcuShWRx3wqEDiDrk0BYMUZSuhz5iq8n9v5xEuzXVXXKKVphQ2TEvr0x2YV9xiMZSsKihDVXhMGv0Ob8ckaiYMNUvK+/LGGAJgfN/iRqPRcTYbxzKOMbBGisJkzpJURA0Si2f4812RKHFGRc7K5u3Od4c8X7u4YL48YP3WOy4zzhEAF9T96M93ftGxyFLjzP0TaJL8hLB8QWSL1SSNLOKaRmbpCplYN2oL2OQWp7CyJEXBxCgH6VfzuPUJnsl//hZSqApqFKGyBicQc9Lt83Wtv X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(346002)(39860400002)(366004)(376002)(136003)(451199024)(186009)(1800799009)(44832011)(86362001)(36756003)(83380400001)(2906002)(38100700002)(478600001)(1076003)(2616005)(26005)(107886003)(4326008)(6512007)(41300700001)(5660300002)(6506007)(6486002)(8676002)(6666004)(66556008)(316002)(66946007)(8936002)(7416002)(66476007)(54906003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: efnAExhU5yLvkr7Toxfeg/Jez9lXg3a7N9jorR9Tro5NgHLdIMAm1Ygh5u7XianZOr+TxSKx+i6pVaNsqRYsJgHxJi5zp7wiuEQONLPTnbnbPdniYSza3bzIBSTjPBZtJIfK14AdF959hIz2SI8XCL1SDCmKd0cTbSBNyhQoRHkRX4LMF+D5yt3GU21+mas8BtYaVhFvUZlZSM05YqQ9SWS0xE/SQEbo4JMvsVb7tLz6w711hDn74hpxMI+QaUanGwWm58Hm0ckv/7MMXO5nTKFIR8W1PFXuQpOXDIzJldDT/lKXxb67BhEv60baIf8iiQSyfTrdAVpu/bROxV4/1/GavYuG6MT94bSPwYbjzoVsWh+iUOEIIfOQWvGoMrD3WQd0a5GqxnBur52GZZrlNnr82Kr7Epg6f2lIMbbvVySXBFktRFtJiBf42HvHFm0IcD0aiXqH9z9k8R0rLz/vnvFnxUvU3pNCPCgH61EGIjpHYbk12EDKGVgVFIC/soMaHbm+w5hqGf60lHUlUtZkZ0QDQV2hIj8kKUCWdVMrDwVqXMpQVlWr1FYBxoDnEhFN1C9WQgXZk7aLzX/6qVs1U8BrXt87tp+tO7m6OuwmD5HK9mWiIsjXLCIjshH5OY/grhqMomwKmxqSLnNPGPBWyPyMeEsH4LnKsEClKWsf0yEirPLP8gMsSMVrbS794eJ1+vo8/pEd4lXf2ogA0QV2Qm1aQppefyLGlw+dk6wvNhFVR5cBulRjybMvnvk5Fl59I9WekJjJqUqiqhkTswXATu7Bop9LX1offVm5lJFQkt0gLdnRK2Lb2d0ilbQFL9c/H87rRYvH563qM5o3DEAGubC5K6pdwGXmuvayPLG4/2cPsjcd0FTp8qo+dNEXpjnk+b58UtvWJrSduNdb9Phn5rqNzPtEVgqrVJnhT/5vvN5ybmAhKKhXM8MpPDVWVC4UhUwF1mZxmZghQO8pHYV0qrqAm3zXdTHUQBaSpGwZxKr0+FOdSim6YbJ3npUgVFgCXgHNbNq7zNKM6fupBCtYpbzp88pYnmGkpUrrAhnwexZC/mR1vB8ENaGrv6SyDg9KBOP09xjyX4OutE/kYL7gAmz+DVDuuTF/agNnf5L4vTrbLLe4o2teQIugzzyIFJYWdW/YxOTXF3VOvNIasUQiGdBaLcBWq7GAWTMAlt0tSzOclSbSw/ErjkslYLqOo6j5Im5InZ2dSJxs5vjjq82SbWPzF7RjuLCpc8sgzQm+c9xCUTATwDwCirQT1IX0l/0+XLiCrv8xcus032krk21gAWJ9TPZXpPOJMbOUu5afBrLYrq9Dpv8ZaXs3vQ8cSPUnz/fJmL8aZTgy4ZR71iNYK1IWqcFHoOwNZHlIGpTvT2+5Wy78L7lRk8Dh09uA8ehh3zo0PV6Hy4qlJQ2kU+vSzL1XbDMsZhTQu/vlZ301VMAaaeuHCbmv5TM9Olu1+MprsfbwrU+zoSheFzsKaBQAYLC79HrjVhxTXulVoOQP4EKNfwBfMvxLJF69ysxbDatDp6ck/7w0TlUCIGwQxzPFOc2WLmfeJgTi0JgdR0yQSvDrkpMNw33C3MeQkENyusbGwIxOuDThttrfOeFQWGqTlQ== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: tuxkzUNt7i2jfVKEd3L+lScLztyIvhGcnbdS4loeZ4KDSjsjfN9F01iDrjVpvBNyUskCtnEmnHt7toNna4f6yOzTDvQlEh4YS4tt9f6NUMH1hzS1XI1NmpIJXlPJv1LEhK7VubQMLVqXgZXYTFbXxu+kBZbZSA0bpiwgcj/3+wEXB0xOzIW+rJkGXaIifHiK+8p/gnjEv/ixorKIRW3tiOo8xrjILS8C5Ig5tTq9fOZT6jAkO52cO/XS+l5kqDMBFlMjchZcb40bWypiW4ycDxnmyBWh9e70e7Axgd12L5rV9aIKMOPejiTvajXzlc1iX2Yb4yowEA2XSv3xtbx4QOOYoVCbqau31gvdOxSMLy7x/isXPsWOqgCN7xjhDAwl+R3TxJdPWj9xM7bfVGRsRSNn3x8f4CzLU63eMcGTzhCE8kUedwhoe0KZUX7tWggg8ISzFQ2BXSxGpZDSwb92HevQ8h7zbXfe0nuswjxtpST6naYQwXVhNOLs+jlrAqpPJndE5hnv211WoSx1a6CPRKj9XZMwITrLWLU+rWTetJnIOWbC0bMscEABpTAPXlGkDBD0dBoeYbQ+kq5vCTz2J0qtSTzYwPoBMb8O4AJ2+BzI3/8sPHtFyrM/G3IkmQi1Lx4eOAsOeuAXjbK0NQTOwM7SQPnaq8RUmrsRXblW9n1FntK55VTYkrI3MeG8p223RvLDOTZmimjM6dZq9afnP1N4I16kJMM99u+9QuBs14SEM2Sow+cOe8ls2kWNj53IlIsj9I9jzU6XvRleE2lBzUwbW/qYbMpZHn10KWAvDTbT2fGE7QoZARWkvJohoY+Tmwr2Vhm6T855rCd0P98+6BgNbGteM0aTi5mTwqMSQuZssvlv5h0m+oIYA6oVrCAvpXSdLhFLPwmOXZBYL65lhFdyp2rvbGdDmvwDBybHr5/0MF1/otKhkpxBBojuooyow5n8jiGKzF2g0kEmCuHQxaBqnX+Q9QqLUrRJo5tTEgKoBdqZMQfi8e08pgeYuVKsZ0i9sW4MK7NbwMeXL7HchwFliPz0GjPOBd0pEFahQGuEc0yLAKAetMzik+zJGlM8WbCn+S902T0xdwfM4NyX+bzCz08Tfy4vubVZ8rw2RpdXKebw9u3VFt8b/BpzQAQVvMfuMaDN+YnfMsQq7+i5JFMpmOUPDh2CCL74BbSnjFY= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: b043899b-d920-4566-d9a0-08dbb63954c2 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2023 22:15:55.6285 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: U1CSj4+bjK6mmX3L7dVLmWPKCF5algYrOO7qWZbzRVTZMTuxotYwCdHLE86vYBedcE4ulT0WsxbI1DzARs+vzA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB5861 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-15_19,2023-09-15_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 bulkscore=0 adultscore=0 phishscore=0 spamscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309150200 X-Proofpoint-ORIG-GUID: GLjFc5f5vEtIIjGI424KWEp8jFgJuIk2 X-Proofpoint-GUID: GLjFc5f5vEtIIjGI424KWEp8jFgJuIk2 X-Stat-Signature: btttpx4cqot9ezskgie3ydht86k66yxt X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 9A4248001C X-Rspam-User: X-HE-Tag: 1694816190-815395 X-HE-Meta: U2FsdGVkX19yqIWovFEPomx/bFLf7rWWtU1GkD7Sf7t5OfyTJsU/u+IHlOn5L8XxyN2kWztpitj2xy4gYJZf83+3H2Py27vABsvf7dT+iZbU22WiXlgFvx2RgmRx74oCFqId8wx+xB9WLsEgLyCS36oJ1rX00qTV18Hl48qwSTVaFPQrRWRTQ46TU5cwTIS6f0Xmk721EVai1yJngCVaTE8M25C4VN3ozdSx1A1fos1CTzLrNbOvNimqKQyVTDeAbzbNcUC+IfRGdOV5FtQQPDhpPzuFlZrweD9pxigZFEOa4tkluYo/K48Kpnej8gdNKV/F1PX86fagBgTc6d47/swucOz8GyS7Tpf0jl5oP/h1qC2CSrltHTmiEr99RdFxK/J2r46HdSFd5ye2SrXMxnXd9+YtEfWmocNHuVLHGjiiBh2J0HCFmnyrO3Oq7zkFP1B9dv9zEzwBOVRd/13JuxaUmqQBS5AiU/5vCuOxxrs14SbIpHIOvJl5g86dydsd5lfWfVZxf0WWL5y+9emLu1dFibiLFD+sEtFF0Y9Pk6wJ6LpRrHnxlUfzzqMAwMZZ5fWH61/DVCx6sihL5hWDGIKOxK+OXirCNsVV7qgqFqRekXlYSijCjYdvCwvaAlXCRsRDuzekTN1AoqiW7Fz+8lS2Ld4tm0nyN7KFabf2nTcjMMP2aPqXZuY+xelJSNR6P3Az5nFqkGdPhJPUqC0ktE3ZmE2OQYxmQBiNcIt4dBCZquObXHNAwXbkyB2HaCxHzJBmxceZub4BwknW0WNIh4r0ZbrgpUCEdW8F0nZfJ57ZU8VS2Gfu9ebGEXRRLhDeog+eW8WSsbaQ2C7D/QKqHyqnFFR8zkIZwezwQPulMFkEHKpdF2wyH1d5DGVTJEZIpVq88kjjsi6KEX+cM8ARX+n2Oze9DOINQYFBtWczTW1UcG36lK4Vp8pX5Ubvtr/nfG/EQygIrzbS9qApYBX 8hJyRuEP 4v1d3FtS8n4skUJWqawiy5x7mLSciyKN2dKQ9Fszt3vukUzU0gTnuOtmnfylPy681Y4KylMndmo9sQHanumF10CmIEXt6m73OrEWPEgtSW8ki1VkLC1rZP7IcbU9fvzccdeKywOebdWasV6raybvEnAa0OPM5ACzBnFfanZf/TsCzbBR0xqXu+D4+IaPpkFQrybsUdB8zqZk+bPlOggTSJs95/wAc7ErhwrLEBV5e0EvxG2/ZFJ74wxnE8I2cf3480seOynFt8Cfu1x9iXz+rJqrkEJRG/sQjY5s8qw4Y4HrteUeUMTObGfSj0+w0m67SWySPmyMOkQKtveI6mZc0XLl1LB2Wse6RX/BoL6RBZFG+iH/s2+AuvHZ3JSX+CnuLFKEynOXBiTvXtAupGi4KSiS7HRiFFipXWMXEz1/xR0oizsoXwbGCzSxRkIEpeTY4teY3GTEaUJT1uuy3u2TnVJqAv1NYRlwim1j54SSTpEoe1+tiYC494EveHTq2cngEq80MNDddxunOCE8X3Psx3jIezwp8i5h2pxPeExbqJpXWMkF8azkeAdCAzQbTiAjBxZFR1ZMN3auKwJve1Rp8LHhp+e+bMXx0YD2bqtPOQ9oQ1lA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Johannes Weiner Mike reports the following crash in -next: [ 28.643019] page:ffffea0004fb4280 refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x13ed0a [ 28.645455] flags: 0x200000000000000(node=0|zone=2) [ 28.646835] page_type: 0xffffffff() [ 28.647886] raw: 0200000000000000 dead000000000100 dead000000000122 0000000000000000 [ 28.651170] raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000 [ 28.653124] page dumped because: VM_BUG_ON_PAGE(is_migrate_isolate(mt)) [ 28.654769] ------------[ cut here ]------------ [ 28.655972] kernel BUG at mm/page_alloc.c:1231! This VM_BUG_ON() used to check that the cached pcppage_migratetype set by free_unref_page() wasn't MIGRATE_ISOLATE. When I removed the caching, I erroneously changed the assert to check that no isolated pages are on the pcplist. This is quite different, because pages can be isolated *after* they had been put on the freelist already (which is handled just fine). IOW, this was purely a sanity check on the migratetype caching. With that gone, the check should have been removed as well. Do that now. Reported-by: Mike Kravetz Signed-off-by: Johannes Weiner --- mm/page_alloc.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 123494dbd731..1400e674ab86 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1227,9 +1227,6 @@ static void free_pcppages_bulk(struct zone *zone, int count, count -= nr_pages; pcp->count -= nr_pages; - /* MIGRATE_ISOLATE page should not go to pcplists */ - VM_BUG_ON_PAGE(is_migrate_isolate(mt), page); - __free_one_page(page, pfn, zone, order, mt, FPI_NONE); trace_mm_page_pcpu_drain(page, order, mt); } while (count > 0 && !list_empty(list)); From patchwork Fri Sep 15 22:15:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13387783 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD035CD37B0 for ; Fri, 15 Sep 2023 22:16:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9A7DE6B0387; Fri, 15 Sep 2023 18:16:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 957796B0388; Fri, 15 Sep 2023 18:16:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E81F6B0389; Fri, 15 Sep 2023 18:16:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 58F046B0387 for ; Fri, 15 Sep 2023 18:16:38 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 001B1407C8 for ; Fri, 15 Sep 2023 22:16:37 +0000 (UTC) X-FDA: 81240242034.06.C91F871 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf19.hostedemail.com (Postfix) with ESMTP id 9FA3C1A0019 for ; Fri, 15 Sep 2023 22:16:33 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=FWQpehyU; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=NGVJn2ID; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf19.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694816193; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Z0pbkesaKqvYJeLsJht1bKFE7pKSDUHfig44BRUqGi8=; b=Bb9Zq4DUAfJMOo/VHTq7K1h0x/IISYq+2+fHG2/H3UvjuwPnv6/vJmTFS3btotKa6KhhAV CcdQ8/q3reggLJJPWWZANCIRvNI4c+X/gRsHhIvUzpS8eTBIUmffY+is3/syBHmIqID+cI ytZ9slRjW063xq5Iz+9/uLCXK2kJCS0= ARC-Authentication-Results: i=2; imf19.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=FWQpehyU; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=NGVJn2ID; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf19.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1694816193; a=rsa-sha256; cv=pass; b=q07tfU06DzvPCMxzDLPryadYpr7SjX2DptZwKIYITknkUZIxXu3Rh1FHU9GihJjZlY6K+d HgTKE+W5STCpDcAx9NmJbZjIV8g61Au4+N/YNoTFw/EKIKU9h5RBWaryonFa+MhoSBHXQn T+2hWVDbjVk3GDkA3W/lMyLD6akitZI= Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38FLwluU027810; Fri, 15 Sep 2023 22:16:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=Z0pbkesaKqvYJeLsJht1bKFE7pKSDUHfig44BRUqGi8=; b=FWQpehyUwthwsc5kxVZDdu8WUc702caUPjl2rJHESCbDlSWfXujXqEgTm9S6E9kGalqv bBM3qUvyJU01jb2O8UzL/eISJqOtG582f1aismjC+RdusIFpgkdjC4S8BWO5eqy47X3L 4eirCjE1Iqo/Do1pX6mADoEi1TvQGNGp/czK5sQLakPjwvwuTb/EwlFOeIi4V5axN6FS asNqNOwxt5LwBj4FKw+1S2Oi1J6XUVpBqMrar6NyoWLGc0A9vN+rFnWbViSZGCh9sU6t O5tv1C5Iwcd6lYiEWDmkoECR37JZPPwFA3zXnPb9AJ/xRyhz5Ld4CBXz5GepwY6BgaTV Hg== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t2y7kgsrk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:02 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38FL2HxB003116; Fri, 15 Sep 2023 22:16:01 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2176.outbound.protection.outlook.com [104.47.59.176]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3t0f5anfkj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:00 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LYrwTpIytPZK7iR9/l42nR0ceMsb5fiAJ8rjwvCEdWYP/dj0RgZnbNs0FS5HOKXzhY4cTZyo+KrFFjfQh8rNx+BAraO03zfx4u+ld/o9zN8wJmL7aYerB7UqUzmrdGFsmb0LuVffV471Io7OVQziPxFS4sZWWJy7pJeUTmcVFsTJQ12LTEQu462tvwss5UJOAWgzT7f/wFPoQie6y6gcFK0yjlZdUBzfRuJHeKfZQ4iowGNuNBAtAkDifK4dmmJHplLM0Al/A//qtCvfLvfeBlAFNxfn0/5A9bMV5Tvlmf19k+tBi9yLxEkGGYVEbIMJBoJvHrRftCOKO96V4FjSsw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Z0pbkesaKqvYJeLsJht1bKFE7pKSDUHfig44BRUqGi8=; b=ofsCfJGDoXPt3etQFhlenECJjwrwkWC5gwVvDZ7vOQsqFG1ihmGVPTQQwOV1zsrQ5O+cpcd5Qoek/jnBse1es6/wpRI1f7Shi/O3ukgcCeI8aPx78XKDiO5BmTJlNbXOv3/9SdhGJWh3iVQUledgW3akslWPUD8caJ/sqpsIrcloJXXkZ6gdeZBHkqQeFRhIEt8QXSI+euta/yD6NOXFL2loAScTm9foY+NkU+UCTtn/KeK5X3vUNa//6ZjASAChMeMPIgub5CTDQ+FIzjsHPZDpYMC4rXA/RnwS1kIn/FsHFJsQxEPQfB8QISHBYXwPdbXAgonbDYp8WcESzbRdTw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Z0pbkesaKqvYJeLsJht1bKFE7pKSDUHfig44BRUqGi8=; b=NGVJn2IDT4zc1hPa98+kVgBndMIbmhGPDxL0SsekkGC/4ITE9Fkm4ymnyx63cbmHRAJYih7HpcEscjKabr3z8hzsKr8CNPV6xn7zsTYJGNp1b3xnL8eBnlgznNcDsZWH3BSkld7gDBRTuMiBpYlJ9KaoPCf+pXbT2HSjPH5000c= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB5861.namprd10.prod.outlook.com (2603:10b6:303:180::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.34; Fri, 15 Sep 2023 22:15:58 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.021; Fri, 15 Sep 2023 22:15:58 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz , Sidhartha Kumar Subject: [PATCH v3 02/12] hugetlb: Use a folio in free_hpage_workfn() Date: Fri, 15 Sep 2023 15:15:35 -0700 Message-ID: <20230915221548.552084-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230915221548.552084-1-mike.kravetz@oracle.com> References: <20230915221548.552084-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW3PR06CA0014.namprd06.prod.outlook.com (2603:10b6:303:2a::19) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB5861:EE_ X-MS-Office365-Filtering-Correlation-Id: ae097464-1a2d-4a22-db14-08dbb639565c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: OTSdzMljbPRDqINwlsRZmDdjF2FeTHyVCcokk23ZKa6vvZJj052eY+eQfeYYFqjp3OHhB73tVjjcik2cE5lo2fRWIh5V9MftidxX+8/KCp4kZ2qY0GmhiBKj6rpGNgOM4Cx4M8vJFVphIY4sDyZ0vCfnKDMLfmq/IUzLua1SbBTLMatDzzPPfzfIU2TWa9Aew1ihZv9ioJLPMf/dyGhy9vUnXelvJVNakvGJAM091QpEj46V9KQFPsEMi48UiHqHngX3cZWh5simqcp2XlkVpgUBGfHwBf2ks286X4rfrRjtEKqZ/VqEG9ekzX5wm1/oHvc99fM9zCZNlaQenpPMsmYGn3Hiplcp4O7FRbfRcuyjQ/mAqm5od60DG/nPbKrYRO5P8WeQd/RjEyGCQpLTZU+OMyVU2a4rapjWydyvXp/MmlFc/IbdsqT3BkRZ9bt1A5XUCYh/QdCV1lNWKZkV/HzzzBpUePCyihX8RguR5m9VRuHhEbLXu0M55YYzuE2VuPCamjo5+W7INIL4ndaVg1CBqHlLW4A/C03MqNDh/Lj5fd/4g/veeScM/BUYqlLS X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(346002)(39860400002)(366004)(376002)(136003)(451199024)(186009)(1800799009)(44832011)(86362001)(36756003)(83380400001)(2906002)(38100700002)(478600001)(1076003)(2616005)(26005)(107886003)(4326008)(6512007)(41300700001)(5660300002)(6506007)(6486002)(8676002)(6666004)(66556008)(316002)(66946007)(8936002)(7416002)(66476007)(54906003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: PNphD1lWJG9deYnTq4QDT27cUjkEgBRolEgjy8OYmNH/+Xch6i7oROZwJDs2n9Txggqu8UuCw0BQ+pBqditJIXPkcZao8nf30Get9iOmit2MSp2bAx5BAg9YpI8jY5Yw9nLxQBJYxCC+7UaTwgEjbHC0vEBbQxhzHMByXCtT/ewUsKr1dTPK8J3jeKkis8b0NxT3FXFqV3qkXlLyirTRzG2u7GEsH/35C8K4xXnOdi71sak1erN+QeT2h1czeh2wI4Vb16Rx2Grfl76Cqz7GWrxLJUjbECVX1PrtD6ObDrQEdn7VbL0xT3xzV8CuHfSs6OiSsTPhCIELA+8TeJzpJcizFmXi0zr4gQn9HaoyZwbaF+l29WwJ9BWZ3Z+CtcGbqB9gDgVSFUilPe8LGmIJX7wMdS0w8K2+KsccZ5X/sAQ15yvBAL5h3lhvIFsf8OB6u6G7ZED4wpz0xmQhX3Oc6iKJUI3jP1ULA2vODkPRBY9hymmehoNwYKowtJk+VFtlNH0r2T6fyGBdpn9Laft9DYKnJnC06pg7G2Hg8g7/DZMMnhxBB3EdrsHqq4d0M8hkntkKgJuQxC/2wcMw0T6tJdlCzubXvNkWM1OAlDbQ7Ceqo1u2+bYddMXW9KBcuOiC2M6na8UiQMkbz4QeOhmnuVKo4JNltFxqr+nDv7uuQnGo8puqyL65iCRYTZJoGuxvMafNT4equCXFP1PX9rqSBJf/Col9k0K+QB0mtWdKJF2V9jn1m3Efe7HxbR1Gi5EphLacvITln6BtiEVjvjRvEH8lGcfTAxaAlZIK0xQLqTi4njlik+iY8l8T3rKTNoysgCl9Y2Xy2zlPE4KdPZULIQZPA5fzUUtR1mEp443HCuRbCdvkk3y7ti97TsT1+3mVplLBHSr7rfGuzvMhYSDWfIYTLt+Az0q2OfcLjuhFwn9Adk6+vE6xzLKgjOjf8oI7TAkHTdu+q1NBkDzfzbTlu8Ku3eGd5dkxVpLCtmnkM1m5dRUkWVljAmGiBUo1D332lQc3b+etAgFulVID1uY8PmqtpgY+cMkwcmhFPEumLWOJBpYIofOOWWEXYJKsqcbAZQ/bsLLt+W+apab3IS6P7CMzdAOaIx8nxgYofhQLsHDOjuVGCROFcD2UCxZd9iIKtgzf6jHkZar42IM4J/SSHQkA2vL8cmhg7TtRFfl4/dqYoN8jozmM0LC1LPWF6wf/asRXST9Zl6puiSHz3PaPpL0eVZBQ1shFOxrIBWzMx8w2MWJTC985hHoxlODOTKE5wAgX5BMO7ky6duR5/RTzutFdhafSmxXQuRcOC0izF7xgYu3Ee97EYby5lTkbM0qBER8rfuah7BOXLrNURo9mhc5nwCcA2Ujy+px3MrWOmMNtXpF7ybtfp6NEegQM9G1RzL0y2MxvB/nPYki+Zl41pGG22rL5skKSI1GxPIqpuEGgcexVvHuqLrO+MrtRKd014N23MPZLeVAkcVdhAK+4rEuTs5L8vSu1R/dOvMLt+Nms1dOfrmeOtGFYuOR9Yfpd5lWw8lUROazz8ULTxFLaB6/gb8FTkZJTECIkOwnH0fg/Qu1atNuIxK2xJNOxBlfn0X88m+4FhEJZiLMCYSYlRA== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: z/IVJvVLVFN17xcfbIAzkB1617YCkhPMMRu93FJDaz4Sbdr9JRdrt0a+/pvTKGtnqw5kvQIocBH/qxFN58SJgTa2Ot3C6ry8x/uaFCkuTZ/P1ZMAD7NxRAmsxoB+3uY2KzuUFa7sFoYr04rMmGC1HhFnf0i7NhjjbTJBejo/4I73IvTkXMe8WZz+UMgE+JjyRSgLty1uxBQ0/wmz95+NFBP9JTyxWEBgGTkxhgdqFTfq1gqNyHSUvsSnslUI3qBxkA/LTZ7mXqpHMHBeHDL1tm34HCX5Lr7wgifH2GX83r5UvizqV2cu8Tmohrs3TA3WuP5jJyqL/pzZzsayyTpYKd+kP0Ygl3d0/kwjjGHvHR4MEpUGYjpy93RjrSF8WbHM7/JNjeZuO9+YDBJfULWXz50mWv++ee+XCZ7cCEoGoNvLHNDOm63niF845QZ/rL2q1Da2qV/zxYvnGZR9UCA75kdBJjL8VvAmFnY/FklUI6jctzJNu2nbQEkPyAxpQmDfHMDDlpEH2lE8GRj2p+KAWYB+6zQVRJn5Qu06t0VOUEhD2ZdzXDzGa+t+RiJvcDhc+G6D6Cgli3CiJi7GEUxhcjLVbFL9CmjPivYWfQm9W7IsSppKxlyuNszmx3xppnIOSJfOTfIqa/la7yzg3ADzfsVaVyAdxP4i8H1Exl2KgAyB/0XgXM4M2UvJq9SZXrDsjaJ3T3L8SusfuT0atC+wk5j47xuoXRGJYgHZa3mGF3nosEvYLmdPoJWpWPoOECcZi4lucFKWkDeoPGHWRla/xObVkAKxFGeQRIZCKqRRhhqhky/edwLfk4BlYL/cNsuwlh3pZbmt0EV8yX/tWUdrMi5VP5aJuyzUQSXtBDlSenD337MS1uoJMSJ+hVItqs1xyaiOcDfnIXI01ZmeU8dzn+N3fMJIKvGLL1MRsQlftRuTdwmYufyX+013c/jE4ggqeCqJf4TqAM2TOTOZ3k9nq+KnEc3xqTYbtNR9E8wipXsOAqNgKTtx37hYWM/RhIrfnCwrMgTCN9ELs4SOHlyy5L3UYjsIQ9F0vY0+UOK/+qmCUnGXeyj9l6TLsY6NCNMnYE9I/L/sovm6ceK5uYiEyHOvkdJ8rKuqmP9GuW8nNNHUVKy2wUgZohYzMokCbEJA X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: ae097464-1a2d-4a22-db14-08dbb639565c X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2023 22:15:58.2964 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: PISC3D/BETXJx215CPZKZ6fipM5XzhFx27p3AOkyw5pNJjcVbv/IqAjOBy2TIGq0trOxcjELuR/6PkrDjZ/4cA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB5861 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-15_19,2023-09-15_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 suspectscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309150200 X-Proofpoint-GUID: hzSeL9k9eZgHxR8RKFNmecbR62ozPBZz X-Proofpoint-ORIG-GUID: hzSeL9k9eZgHxR8RKFNmecbR62ozPBZz X-Rspamd-Queue-Id: 9FA3C1A0019 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 1uuxpq3pixoqx35bdywcy348zkrxj5x4 X-HE-Tag: 1694816193-303566 X-HE-Meta: U2FsdGVkX1+Gz5KtY+OHVEvbdcMgikKqZyKljuI4Je5IiXL+OF4+pDObb/qpgZnlQw5gq/JObliiEAgk3ldKKiitCpB6BRrCkwQG4xY5+VbQNjNypQm68WrcatdmNmJtu+0YySsllQBY03gY1Z7Xn0Cf21sDZuiJ7Jm0PFgnOefy1UnsmuFsGLnRHYzCeTiCdBWs7k99m3nnMe3azWTt3d4vqY7KmRWmUGwCtMCLxYe8ggKw9/Zl48rI7xcxbDTkDNoy5+8oWAErfVn8aNev6wsQpLvxbofcDppZTU+QaED4eX46ZwDlirZD2UxP0TbWFxjKPLqeXcQGbld1rzmHEBrSad7EkgWcf8Jun8Pit+Jyyd8Ccn6lOLhM/cLuDKRWWL/dgjeeERNO6dcq4MoFxuSMkao95ACBhWQ7nlm6kJnULfk+JJoTU4xUs0Qt+HXgEOFEfi4Uan49bSosxO6oQUJ58F+8NDbVnJ8BGcP4MJEFviuhLnsnqPxPsiyOlM7OshgT6l0Tf8UhVC9D2iXUitPkqQ/qqlfAbze6INJXaZcesHgjI+puhZL+DNzobVoex0P1ZSI1dZvTvF9boGYG41nGf9GjthV2SeNxgtZnFx0NiUfRazPZ0URWv8QdRPtCarmIFO01nEmkkxFiRwxxqt+jc4JzPZygTEvPZuFsazBaTm1Hf3VjMf80fZ2X9LaB8qmwsRYokiY9+ZOAePN2d+F0/1Kk99r/AUbkD3W376qr+VS8rY16Dbi8Jt6DaDc2kKxOAHW8ygJtKoU8sVW4IBekY1OdQjYXVpz62Mesi5dRpn29lBvcn/dHH+xYInlTfo4Ji08VS2vH0OVto9x5DlJiWYiR+JNTQGQdbxMZ9CizUJ047Q1A293jQsY+hXadHnylUx9Lh1FOxKfa25mDKv9HPsVC4Fz6/cPkUqEO3bzHvmr/U6tEp3YeZuOMyIdwAWz92bvRh3Fl1vzIHYv N9YF8IUX ud8Dv5Lk3hi/e9EtkBqS3FwTAV6fpqwVRa5Sym4sC0Q3Vt5fyYdeC+cLi/Km2/xnyY96NcYJagYk+cHFw2LfjLR1gnwK+E5+3jxO5ZgYZYgBuQIImMJ7JJ4iHvN4KZA5leb4XMhoUfgS2pyw2Z0d13szBKS1riV3xVEPvzfFrZ8/Uma7kVp0J4w5p7T9SN94iehxCfkkxWjAki7oUouj7Y4gjQZNdaAMX5uywGtdnA1OtE3JVlcrZHGRbigaOII05w+l3+h1EF2s889dRZmjHvXGNsLR3OIvqrejAbdPcaQpL/V1xWRnp9xpqm3pnU8K1QtqRyxpar9+HXQ8lkxx7Hrf9RCms/dfQYVg49C7i1/8TIPH2r943k6FD6YbhRXpUJIc0ddTdrDapAIMwMsGH2RoppT8RTrRy9kQ3AhIN1q9N/KeLZqFY8p1KVodODY+9NOq3l3TYd/zqiBbS1F2xDuJIgkOyWu8ftt898mI9Xzls8+6OzbYK+dQuocWX0t+Ys8E6MpYZHdxX/Ow/iHItc/IMmf2oJXR0CVBcBpehjIR6s9m7JRPlrR2Ulla3eNrFl6GxKOktrUwPERem1nNET0FfXOSsyeARQxSb6IK2gnhHc0cIVQFgV7JuiMcF1/U0ta9EwbJRvWR+FYRWZf2/1kkoALRrUAbqnBED+5oJOJ7aI8Q= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" update_and_free_hugetlb_folio puts the memory on hpage_freelist as a folio so we can take it off the list as a folio. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Mike Kravetz Reviewed-by: Muchun Song Cc: Sidhartha Kumar --- mm/hugetlb.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index af74e83d92aa..6c6f19cc6046 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1780,22 +1780,22 @@ static void free_hpage_workfn(struct work_struct *work) node = llist_del_all(&hpage_freelist); while (node) { - struct page *page; + struct folio *folio; struct hstate *h; - page = container_of((struct address_space **)node, - struct page, mapping); + folio = container_of((struct address_space **)node, + struct folio, mapping); node = node->next; - page->mapping = NULL; + folio->mapping = NULL; /* * The VM_BUG_ON_FOLIO(!folio_test_hugetlb(folio), folio) in * folio_hstate() is going to trigger because a previous call to * remove_hugetlb_folio() will clear the hugetlb bit, so do * not use folio_hstate() directly. */ - h = size_to_hstate(page_size(page)); + h = size_to_hstate(folio_size(folio)); - __update_and_free_hugetlb_folio(h, page_folio(page)); + __update_and_free_hugetlb_folio(h, folio); cond_resched(); } From patchwork Fri Sep 15 22:15:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13387784 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB48FCD37AA for ; Fri, 15 Sep 2023 22:16:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C67B6B038C; Fri, 15 Sep 2023 18:16:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2CC206B038B; Fri, 15 Sep 2023 18:16:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 008806B038A; Fri, 15 Sep 2023 18:16:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E113B6B0388 for ; Fri, 15 Sep 2023 18:16:38 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id C26B5B438A for ; Fri, 15 Sep 2023 22:16:38 +0000 (UTC) X-FDA: 81240242076.27.F38EFFE Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf19.hostedemail.com (Postfix) with ESMTP id 674191A0025 for ; Fri, 15 Sep 2023 22:16:35 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=Bm5lLyHg; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=jaBnIbJ6; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf19.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694816195; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=70YZXjYHeXJTUfHpeVFoC+vRXVEfJ62tlb/36hn+mlw=; b=o/9GMVS3/ZcEjJSmR/7NCrCQgyXXVG9/YU4ukKYlm3qw45KJXzLgNyvkXDif+sehWG0ZxX SN0glUDVnVjkaV1TmSSXm7u4hULulJpj3nuPzBmgZqeHlMu0ePuP7uN2oqwJnReYQG7vHr SsyR59/S5SpaMbVkKRLwTU9nzJe21gw= ARC-Authentication-Results: i=2; imf19.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=Bm5lLyHg; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=jaBnIbJ6; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf19.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1694816195; a=rsa-sha256; cv=pass; b=3nBEBD4HrTW/Ex9GLx9p47Grt0+ZXTi+KSqzbwT+HF+zbM1K4xDyOfFs6zOSET6SgYChLf MwExBOLyIR5dQw0+Grbqx7f3NzSCkekr1k1qYASaEVTgEBckLmVCqOAcRRBfLILZUaFyXY PBQOctuPjyAvppIVni1uw9HeXLn2htc= Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38FLvDqx025304; Fri, 15 Sep 2023 22:16:05 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=70YZXjYHeXJTUfHpeVFoC+vRXVEfJ62tlb/36hn+mlw=; b=Bm5lLyHgAv4VQclF5sJAonlsRLrM3Aice3CgG6rFyCMkn/Xznq12ABUKJ/txkdui6K1K F4kA6s/1SQFemNwSClF2OWj+Q2HqhMO6saQFfI52zopj3hlycM5MmsV60HK8nrmBEI47 4EfTQaVDGm3CVK9ze6PYdr4N8igMZMz2wRWjMVB8qoIQZuXIx0JrP3U2r27EKvpYttsV 2WS1pXUPFFFGaRX/azkCr8NhkN6h+BgUBvPup6eI0jD89/oygJrR13R2+jsEAheDtKrx 6zEh7SxMNk/WUSwA3oTXBAYJuGsv22xudon332fZPnEoVjF/RQMNAPxtcJAhx8atvA3U HA== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t2y7n0k4h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:04 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38FKi6Hw036207; Fri, 15 Sep 2023 22:16:03 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2106.outbound.protection.outlook.com [104.47.58.106]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3t0f5gx8ua-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:03 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jfTHon2UVRZvWWcHCeaslJknwCl5M/qsXAa3ExeRBoN8aEj/eypccX0wLN8HUlXllsE8uD7rAwW3fwPLqVOz0e5RjTzXVAnkJOLgqCgS5pT35hv+X3xiWkSiDo+kjRPoaJpiv2LxM/u2q/D2C7A55x7/ZHhsEbSgnUo8SlhEdq5kdMjlO8SrUhuoRkdfLmNYdzHNbBX++kUDNU1P//sDDPTSIriSRLR6cTB2ffXCu1lVMqS6otX6x6uM97z+IZdQgNLb6U+Q4SgPVsKSBSZaC9SbtP4YZ3NVO0sswTkFcZpLxwYtTQ3f+ZgHHOlfdjiGG+h5qb+EvgqQ9Hh1tOS9yg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=70YZXjYHeXJTUfHpeVFoC+vRXVEfJ62tlb/36hn+mlw=; b=P6Sl/7Vem0LcwoZbCRcmM4sIIQbJWL7cTtVki0rMLrC05+MnyfhWVrdLr0Nm1r7kvE4sdapX9T/5XlzCfF/bP+gNV1u+iCQHl07VAoDe80U8mD0Luq97UbGGsrzB+wL/sPTJFfi7UD3J6A0KnpdteQtHM58enP3hK8giaR3bEIENbw3Uic1dTWqXIBHJ1a2Py9seu7vwSBvbJAk6SA6n0fdRkLz9Obnfn6tYAkZB7iLX6pElP92nfQlUQSupfSwKCFmrPCTOEZw2Lt0KAdmj/9X6wd+NM6MFjClrk0636tRNZ64ZROZV5RYRC1j2z0kEZM43AKSdcSBmeOqC+dFxwA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=70YZXjYHeXJTUfHpeVFoC+vRXVEfJ62tlb/36hn+mlw=; b=jaBnIbJ6Jd63l+zHm4GiCaEvx6B/h/2SKlJycITDOLTxo71AeNKE5n815C/8Q4sjAMZxWRyG5vzgNYe6S9A/bgBGk9NPbZn83murb0irLDN7TKygEVfJhXf56rSuWR4UIKWF4EQmqLhTHe8VoKXYkk0oJ+d7+qh37Oy5XkSwAM0= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB5861.namprd10.prod.outlook.com (2603:10b6:303:180::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.34; Fri, 15 Sep 2023 22:16:01 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.021; Fri, 15 Sep 2023 22:16:01 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz , Sidhartha Kumar Subject: [PATCH v3 03/12] hugetlb: Remove a few calls to page_folio() Date: Fri, 15 Sep 2023 15:15:36 -0700 Message-ID: <20230915221548.552084-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230915221548.552084-1-mike.kravetz@oracle.com> References: <20230915221548.552084-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4P223CA0023.NAMP223.PROD.OUTLOOK.COM (2603:10b6:303:80::28) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB5861:EE_ X-MS-Office365-Filtering-Correlation-Id: 402c1fc4-1750-4fd1-656d-08dbb6395800 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: CFHK0shqqS64ZR+cLpIhqLN86Xb8fokj5PQzfkKqVNkQpU8pEcRwU8AQjMPzHPwZU+oHALO7QdVZY03zoESkqa6IFfbRWjccqXOtlLxTXjwhzEBXKElJU2Q1sMOvlwcz3XqhBRQPr5myGyEm9G0pmqvR+y+d5kNUGnyo/stEp3CgZlq9q/K08IiWTiEDhfnWTXuOHc2LUp+2UEeGUYBpLnhhnGkUzRtoCPqzcBddwaXzzjwv3eQiVOFZueTc7IgXKLtJAbuu/9OFW/OMQ3bSQ32WQBKkAKOx1+EAPfEmP4SByeVEoCX0iLeTNRz5DAiPOVCHtPM+iT7Zgv6fRYSh73dIXpV3TMzomEERSYni1vLj07dtAPjPkfS+tcdBjdqoxcAPEWJKQrQIPcRNL2f7tprjMq8MPBlseUXMyRUrnWFTv4m44xz1JT2KXHRqbe6dLNM8Qjs1auupZOe3BeUNTzdR1rHbAhBAW1nhPst1CteVpxBXxxVMymSArOv1LJaoYBjMxz2bPaDCiI0cMoZkyO1nf3qFz1DLuIUE2gfs4KAJEs1kax4r6iFN2bkC4qL7 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(346002)(39860400002)(366004)(376002)(136003)(451199024)(186009)(1800799009)(44832011)(86362001)(36756003)(83380400001)(2906002)(38100700002)(478600001)(1076003)(2616005)(26005)(107886003)(4326008)(6512007)(41300700001)(5660300002)(6506007)(6486002)(8676002)(6666004)(66556008)(316002)(66946007)(8936002)(7416002)(66476007)(54906003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: pSOH73TS1b+Y8Qz5azwPTzkii3qTYt7411BCCpC6yOaAVIZQX7EPOmLcVjVKZi8eCy52y9e3YZv70uBR5qJK3qE7Ocx2XDDAYG7P5DR9Si10i2cveuRj6wc5d95L2tYXNF4gtvsNYyoYjcROe6gSyEgp37/scmxWr6Ha9ChLbM2z7Ho4sRWlFfHt+4hw6t3zoMn0pcydS+47MtxGZ78WygpEvxYb13DNUCCX33nZhgGiq0zBW3GXElKuU3Hlb8Ofvon7egvgJ+N9+2fdLySdVd41W8u5EzMBiOL7RgjgtOGABy5EEpLIsTh7IHw953wXhxAVoxUTyf1c47S0gFsOB0o7ZCbw6aIsFhrpYcVU11OMLRtuny8HnknWwKBWmDf22tx9jHgtrzzUXekhU4T7KSZvjtCtyUF3iu823SH4lW1e3h8SAShhJjk2yYuCBohrsjiDqL2sMN+XKR5wQbwbjYYCS7wNYQ4NA9X7ZIWm9wFzgSwztubevq0HiYbzzFu6b2VQyU6gN4WtOEMlm7Y0lh3xUtF6sNvWIQEIHdg1fZl7J/a9EZ1uy9QZMCDp3D7+wECpy1GBNVwarWqW0fguQ/OVaDpb60XDI8PMxoRiknKmuwxBg6qJj61WrSAkZ+c/U3xGGmiQ95nZUsH0Rs47TjwLUXaP8W+Rnqli6ZhJSXBDTDJcEHQQD6p2eidYRnQnePcqBmYW5N6Sm98sb1+bQ6wqhGgvkZwKWaSy4mqYyxLANZMyZgQWhkGDk1KJrNZza5+/ZKlBxgVlkj8RU5lVnJng3qyt33yetfbSdT2nqW9oiI0/kXWnHzaL5VkebGK5mKYKBD1Wh2Jh+MKdCqgW/BbH7L/jjJ82BVAQNESRpilkJ9I25BSZDOFwIhF+W9BF1R94MOAaZMAUPfq9XAPYhqMsJVKC3aHDiUfg12BOTxSIiS3MMYu1yD4JchhaMGBM4JBbpaIyUKBBSmpO4BcSKrZhiaPccbm/s2quQEYeiA8qQZ+H1eI9kN/q5IX0aHcRbWa1xsUlnVfLNBZqMSPieUOYKotxCvVdkVq/AXZ4IJotbdFmob8MwOznVoT36663FDXHWTUWBYsi1U5medjdF9THIen8YBmVEvhTfhV7JbbMfviaXaAmn+aJ0SOM99hxwLBw2M53z4B+l2duXjMsddYgFZFVMo89aiPK5zACYBVyGQSAoinfht/NoGFHAjkHG4OkPn/2m5W7duglqpLjudga/if0sBmc+F0dueIecF10ACk8mpjY8wS2YBdebB3eqmJNwutcooUnFfmpcJF5zUbwxuGNcKe6TAnHpvXfQGj8u5TZ979c+WlkUfKXrBusjudw4EROmQ9BdhplhVTdsSNvhQ+qIRg8H0HWN6fCi4DY/X2513AMBAiZx+K7+someizVGaIS4Lq0d2mVYduZ3WJzU9Gvbl6mx2n8WhtVAox+wKmqqOhDzTwWWtsqjJDLaAp8zsXLoR6V/EeZZ02Qp3xEOCbXM2cGh5LZUB3FYcs//MAsiOyFISSUG7i/m3mcCPOS5huO4kkwmYCoW7ZRnsEdl6QSiDP7I7vWImDeM5TCLE6q0vdCziF1zMkfvpbTPP5YFR9+tSApK4GGZ4K2Hg== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: lc+2ho3THmk+O1thGUJGWmCnx3oGw84m+qaag5iqkrOpgZV50fRv+csGxKXkkO6CiOiy9NsPwgJRXGl82sGW4YwI9SK14DZFQq96WrS8RL48l0miNZK9fjBFz70dxI11AULF3ATHGqQLTmvA7Ashk0Hu5Y+FwADxPawrQYtvdLaqoowxHst+6OFk9knQnNal+t7gWPC8LvIydlOKnsLPvIp/odMssspxhA8Wc06Q0ogjjya3CFGQhaLFx31BjTUKFgKo7Zd7wWZ70MrSFKizHjvF2bHb+NfWBMSVgMBrOiy8RZFouWhveoCbHYidUpfunPnJmhnZQHA1I7I3e/ggPutm8u9B1nXS0bnPfM+TRYvCATn0PmRiqooWbLLUe+Lpd14CG5R28R5TmMA24KrV5c6+ZXJHym4wG+vRpnUkQhCqxeRJQ2EcZDAW/6IXEn5lS7slgQkJaPM+bcsN+QXLUNYsvNHHKFLxXpdN/BCvTKN7FDXpBBwyot58gTnVvV0t2xeiTONKOejT+/nCVeTO6mOrm/ml6G2vBk7JHMaJ0MAxK7VB+6IDDNQiDrsFE98zdFzbHd5lKnsUIw4lHYC1hsTMmswbntVNj8ayg1N7IBjrUWozQgLBe1RWOTrjIFcZ5J5k6vt0R/u6t0um0CJq4lCTXtFOuAwElf5h8NHQhZ8lSJLwNsNsNWATi7ErnfGsiJpbocqdzrFlyzNcokdglFcYeDwLQHB0jIxY9OXK9V36Z7v3mIj878olBZwcycsxRGP5IDp9BKjkbEIJV9I2zbsqbrsXDpb4fwV8VYqUIHQFDMa3frOrWFOv2jVAU0KT5un+pz3v9vy/8aKjr7c6M+4ueYVbRhEw82fsSGyFutJS8Y4ePZG21JWO/9V7/ee13T4Gup+ydNEblcomaeTNt1XGkhEgp+kn++XRhgzLrCJm3194LtumBXKjXvgWAO0tSzEEBULKlwekAEE1CE3uwtMjjBW+tgQCFv/5mEQzq/0w9jAtUnEYNuONQcDJjwA4VeW3YxPt6oA8jdFRKO35Dmm/QOHqRYY7jckXexaP7f9de2C6B8QyWzvi3tPhn9hNHwmAf6yTFG8TzBdVZHf5kiz8i8M/IL3NEuzbFzxaLVdtqtIcIkA0Lr36XtS4A3TP X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 402c1fc4-1750-4fd1-656d-08dbb6395800 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2023 22:16:01.2295 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: b9MrJyBuaJckDndEyeAZscUjW9FgOxgfQMU5Y+7+t+OuPIDyEgkT3AO0iM6eHPVDuJM5MU3JliUmR21IVOeSjA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB5861 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-15_19,2023-09-15_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 bulkscore=0 adultscore=0 phishscore=0 spamscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309150200 X-Proofpoint-ORIG-GUID: 3cqM8Cb3M-SDhvPsn_zfF0D1U2ZPdhY_ X-Proofpoint-GUID: 3cqM8Cb3M-SDhvPsn_zfF0D1U2ZPdhY_ X-Rspam-User: X-Stat-Signature: tmcp9zk6iju6spnji3397y6suqmuqowa X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 674191A0025 X-HE-Tag: 1694816195-880852 X-HE-Meta: U2FsdGVkX18cq2pPt8xtj42sczpfMqGoFxF50+h9mUavF/b4VpBmz/1UwOniPyaCUV7R0eNEllNRYBIwwhn280bgNPXe/FD71ZfSZOlpqfuxgqYbNQmSvR+ObitzPBwaIFGcCzA0oZ6fFgLwsdJORl4OjNV5uO0qI1CCOgbqNdEHVoOgfhJZUvSIINf/Jg0Hp6LBE0z56HAA3O0MrgNPcRDQCYk3vQd+DICUz0xrpDAxsxioDzi2R9z/V8wGrxU43E/qYiEiHDzBWT/uB7QRGCD06zcNhbSvv9mTOhbQvzKVukb6DePoJcFxLxuXZOrYklNSkOXE3BRdiF/KD4HyUDrz+AnW+rvbySGiaIKN6Hnz0KVArgx5s+9RaxnE31d5HkxKLD5U6dZYEFZF/hiKZE56FFHzRCvEP0E+yfKRY93GNsUUrxxaFrbya8gOdYmqP3n2aGQf1O4lokaNB8gB0aCIMNHF2lOaQVxb67GiowK6+w9BQ1L4A9eEVBomJorp7KSf3IsgymDY8kYChwPGc7UrfWFak2/WjoJMbLTKbtLACmkiFw37GViW0kEelj6WwyaxI+4wjWsn+M/7BD67Db4HrRVhwvWeSaa7XgyNi7h4kwa4Xjj3/QQ1U3xJU34nSzpgTZFSBjFUIpOGe9CbWbHc6bSE7k6DirMCbjYRdLFC5tRqt0FIbxWlE2X1e2Qho3Bi/mQd65gmF3JujiEzHBCjaIhVrDOA0KOi4f0L0I+5vuWe3cS2mIL4BFols4m1AuXKwSL/mRdg8JpGEF6MUsS3jxqjF3voDw86XimMF/gFtXteSbq8d2SFs+FxmypOdjQemiCDEENq2YhD2X+B6+Zrslnc8GM2HRWvSSTH3I2rhQZ6Y0ohPOyYYvuVUhpUD+42WPNcJ9UE7D9pPaleKlF8gnJa/lni1GwtWU6iN/S8+eITZWe5P5qmcSi/yxCOx38dYVmGTZt+1l3fRJb 8tONiD3P KebFbpLWKXYCRmCC6698pEpEmg5eAXVG4+D7GExPoYYN041eOrABXBUzWFv3Ab76zxQVP1iGAvW5WF26CWxldHX2oQosEP5cMQsmHRVpR+YGJM/ZXjkSnlk68gmnbUaxSlvLrY05gxU2kLBQjtK+zXzZOfcYsXD8o2noEFHM6vuh8K322dmb4dBYIwbaCw0M6Hls+MVAyPNNIibtAw/2Me0qf3/1QfVfoRC1mAU3dtA3vjCi712XSzh25fXGNYqfxePkgsXDvt8ckYldnUhLb/UQbAPVMgKKArqfLPe63tWhW06sYKzfdL05qi/pqXMjGevPK5GjWwU1K1BB86nGCOyZqp8ld3HRXmIqtxtTcc3CKZBUaF8f8g89qdIuJrIa3aiS1F8cJKpCDi5oCmGwhzh3kD8aV2cqPWJNTlOw2W+E5wjLQsE8l6nu6oMAoOucJ4/sfTqAUaY1V6XN1l4vLmr2PfZCo5Zj7/udtPYI738G5fpjedLexgxeU0VrJ80Ndal6cpanD+DxQU/ZQXYFnDOMCy8ny+HTUmxn3EQP229vI2lfUJSH1Yii8HO9UjE+XF1P5AqQqOMz6bXxNG0nITUILLgnWIywC2Kb9bZIKEmCwQBGXO/KujrrI89/NlxkXf+U01M717+wiF7GOOQ3Ls29V5rJAmwgmCPcAURfGmA5+0kw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Anything found on a linked list threaded through ->lru is guaranteed to be a folio as the compound_head found in a tail page overlaps the ->lru member of struct page. So we can pull folios directly off these lists no matter whether pages or folios were added to the list. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Mike Kravetz Reviewed-by: Muchun Song Cc: Sidhartha Kumar --- mm/hugetlb.c | 26 +++++++++++--------------- 1 file changed, 11 insertions(+), 15 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 6c6f19cc6046..7bbdc71fb34d 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1829,11 +1829,9 @@ static void update_and_free_hugetlb_folio(struct hstate *h, struct folio *folio, static void update_and_free_pages_bulk(struct hstate *h, struct list_head *list) { - struct page *page, *t_page; - struct folio *folio; + struct folio *folio, *t_folio; - list_for_each_entry_safe(page, t_page, list, lru) { - folio = page_folio(page); + list_for_each_entry_safe(folio, t_folio, list, lru) { update_and_free_hugetlb_folio(h, folio, false); cond_resched(); } @@ -2208,8 +2206,7 @@ static struct page *remove_pool_huge_page(struct hstate *h, bool acct_surplus) { int nr_nodes, node; - struct page *page = NULL; - struct folio *folio; + struct folio *folio = NULL; lockdep_assert_held(&hugetlb_lock); for_each_node_mask_to_free(h, nr_nodes, node, nodes_allowed) { @@ -2219,15 +2216,14 @@ static struct page *remove_pool_huge_page(struct hstate *h, */ if ((!acct_surplus || h->surplus_huge_pages_node[node]) && !list_empty(&h->hugepage_freelists[node])) { - page = list_entry(h->hugepage_freelists[node].next, - struct page, lru); - folio = page_folio(page); + folio = list_entry(h->hugepage_freelists[node].next, + struct folio, lru); remove_hugetlb_folio(h, folio, acct_surplus); break; } } - return page; + return &folio->page; } /* @@ -3343,15 +3339,15 @@ static void try_to_free_low(struct hstate *h, unsigned long count, * Collect pages to be freed on a list, and free after dropping lock */ for_each_node_mask(i, *nodes_allowed) { - struct page *page, *next; + struct folio *folio, *next; struct list_head *freel = &h->hugepage_freelists[i]; - list_for_each_entry_safe(page, next, freel, lru) { + list_for_each_entry_safe(folio, next, freel, lru) { if (count >= h->nr_huge_pages) goto out; - if (PageHighMem(page)) + if (folio_test_highmem(folio)) continue; - remove_hugetlb_folio(h, page_folio(page), false); - list_add(&page->lru, &page_list); + remove_hugetlb_folio(h, folio, false); + list_add(&folio->lru, &page_list); } } From patchwork Fri Sep 15 22:15:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13387781 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF248CD37AF for ; Fri, 15 Sep 2023 22:16:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7858C6B0384; Fri, 15 Sep 2023 18:16:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7353C6B0387; Fri, 15 Sep 2023 18:16:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 53B656B0388; Fri, 15 Sep 2023 18:16:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 432756B0384 for ; Fri, 15 Sep 2023 18:16:37 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 1D7A0160801 for ; Fri, 15 Sep 2023 22:16:37 +0000 (UTC) X-FDA: 81240242034.17.FEC4A68 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf16.hostedemail.com (Postfix) with ESMTP id A138318001B for ; Fri, 15 Sep 2023 22:16:32 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=ly8gISQI; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=vMj+Us7h; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf16.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694816192; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=E2k0SQT9AAGQbQtl+bCcx+uvnWZgNmigsOuLewx2GeI=; b=MV8hUOUtQ283j094UuK0JtcG8onstk2nFKdhjJlsyJb8loAVNeN2RlnuwcGRJE0uTC9seb pGHN2LeTFjwFmCF93D1RvIVNtmPK47Xi1HK7+Kg1jIkdJsEi+y0tRJ0aBDDfT9V8HvpZXF b/AwqoBcS46ZKc5tlV9dETlvK+/HyaE= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1694816192; a=rsa-sha256; cv=pass; b=NFtNm1woAX6sKqFW2OmQRE08uDj2AlPKoynRNVkVRFrVZWk7UXqXLoVjsPCT/JxzM8e0Xe BOtuC/KQ6Gn2o7pLVmZD66r3NhxA7/8nGrrnKqs/tk5RbasmS/I1XUbYxmD4JOQl5PGwPf sH7adjCIFGt3M7iebFs9fTq2PDHjru4= ARC-Authentication-Results: i=2; imf16.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=ly8gISQI; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=vMj+Us7h; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf16.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38FLuFXh023028; Fri, 15 Sep 2023 22:16:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=E2k0SQT9AAGQbQtl+bCcx+uvnWZgNmigsOuLewx2GeI=; b=ly8gISQIduGECkkwZ06y9RXpS2fA+IIhrOqw4lUgKl+J++zsyGNPSHylboQo6MVd7jfS JxX4kLgg0UJ4m2Dz1kUgfVDAkJQejPJnmLa83bT4nU3V9xTWBCRRDyrAj14Q3FIKUeIA bLvaIrF3VzG4VyN+BgfZYUU+Be7rDtbBGkfgku+pJvdLGDhg6pGCwOJvk2Dk3vlls3nI tk0SBlDY5lcV9G/1fxPHbDImYIkIaY1vTS3lv3DyLUbu3k6enERNNjdLRlPmN2p0xTPS W6ng9Ku7Y5rh2YheOkpveg+C69lSj9Doaw1ZBpK5WxLogTNAXcn4N7Vnl9i9tv3rpoPX sA== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t2y7n0k4k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:07 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38FKouGE036194; Fri, 15 Sep 2023 22:16:06 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2176.outbound.protection.outlook.com [104.47.59.176]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3t0f5gx8vp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:06 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YSQXZmZt423On9BPQEGFLrZRo6NNTHrdrZFiuBsvLJfzsHYAwY7kWLwqvJFCamIE1I89/jg/t+7IWMrbUhAM3PCCFp6GuSziSlKRUwa7zATka+HdhicXSlZBbSirghcr1IqtubqOE50F6XmXku6PeHr/sxlV88s/XyJ4mnsABydgbEFOAZ2Mq5e/BBqi3TeMGr4cCFKLYv3S7hvVIpmU9GtA9redlW8xsqANcmtF5oYG9upin3rnKqs/WF9S2jz6XH1tAUuqGVxtoH51LZHeGDnewQc1C8g9rgVrxJfMK0oGqu5lxwr76Pn2HgH3/K+dESzHRnaKRIZGoiud7jpNvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=E2k0SQT9AAGQbQtl+bCcx+uvnWZgNmigsOuLewx2GeI=; b=OCTxwuXW9SwQlCmVR5PlqvQMOgT5YqkfOq60akXMKYAfD0qRu3FStw75grmYl4GUqqrcbbkJyZOPi9a0rW2mx8isCgtghrrQnVIi+QV9Fv5/kiX3OerSv1qv48N7d5OFCBf7kGA4eEsnTMNmfuIXUFQfaZB7ucm6ddZ6AszsJwfACXIoAuEbeA5k+DQSvxzgwruqJ2pTbJql4gpQVHsOjSS9dzUx2AZ40pg6aJr4J2AaR6J05OcL99HCVMJx9FtQ1IgFw9GDwtoh3KCQcrQVkQ189kmdd/IhP2q7FdBG2M3H4gJEVPsjg2wc/uaBqLtOY6sLBP6Qbr/0HYw6/ci1UQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=E2k0SQT9AAGQbQtl+bCcx+uvnWZgNmigsOuLewx2GeI=; b=vMj+Us7hWWunwFmjXN1MK9fNTcUM8YBilbhPWn/SpbU9Zd1fCIdr9h0LfyrpfAxFPNH3Ymf38L2ADjeNSapKldE3tMco7seaLF5Y+WZWjUe9qPGWZFCwbII0kQPfAilgmugQ9knfy7zTT53ykJNbzbpTbduORuyKEw5YqPlWKSI= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB5861.namprd10.prod.outlook.com (2603:10b6:303:180::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.34; Fri, 15 Sep 2023 22:16:04 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.021; Fri, 15 Sep 2023 22:16:04 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz , Sidhartha Kumar Subject: [PATCH v3 04/12] hugetlb: Convert remove_pool_huge_page() to remove_pool_hugetlb_folio() Date: Fri, 15 Sep 2023 15:15:37 -0700 Message-ID: <20230915221548.552084-5-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230915221548.552084-1-mike.kravetz@oracle.com> References: <20230915221548.552084-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0063.namprd03.prod.outlook.com (2603:10b6:303:b6::8) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB5861:EE_ X-MS-Office365-Filtering-Correlation-Id: f08e84df-54bd-480b-259c-08dbb63959dd X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: W2eFw8VLFi0cIAzJau3IJqeB9mLZzIgxYgmle+RLCtK+a1Op/bCUSayXcuheR3BJjnRTE4iHYaIIS/oSkvhZnmaYlHxNXa3rchzNvhx1rwsS+g3ls7OUy9wKJp4JGHXPuquCPS9tC30nKWYETgfDFqSA+6iIKweNj4ATHew+3cd4BDyDbpZo6BE1WE9ccIHQdxVNdZhSx8GoSIxcQSVvSFGUM3b4kvYFg7gs9VpUkaBY7dbPC5BnaviE9Sw3eR3JP3/8WbBSPHcvTEyu7xHypy3T67QUW1dn/L8Pu0O4TqX+5SrqpHirTJW4zRZuWjOuAPlPCuhVcNVAEnsWx9KnoT3PkTx52TTvTJzp78tf57fqwe5Y1M5jPpBZOWbSLMezqNXI5GScimpLGLr/dlAdB+P63x5mgl3APE8R4/+iibcEKxybF5Q9XHzy98EkqrUaMdkGBZ3lxA7Tl3mIR6pT5RCG9ghmOtpRS36Ku14YEbAhmsABdSI2HpL7GbzDDc65Fhdfg/GnSUC4gg+CqImaW90Wfz+zTe0e7h4nyhjTjZXNyEW6SCwbVElG0pYh2VVTZwwMp5ncoecCFFgGAZuvmv5cHhm/IcTIirVV8U2qfl0= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(346002)(39860400002)(366004)(376002)(136003)(451199024)(186009)(1800799009)(44832011)(86362001)(36756003)(83380400001)(2906002)(38100700002)(478600001)(1076003)(2616005)(26005)(107886003)(4326008)(6512007)(41300700001)(5660300002)(6506007)(6486002)(8676002)(6666004)(66556008)(316002)(66946007)(8936002)(7416002)(66476007)(54906003)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: gRRLcbcqyPn5YYlMAlDzeplPVRPUeP0zX4vvVRqZpdzcpm+fBE84wXbv32PWevFQ8rtBWtlJZgXhKAmpIQtk5VeXnBl0ttOmjN5OQAZGfiBgomw6c4bOQGFT8lzvYwC8EK+re5TVoQp9m15GRz4j/m2Wuf0/j+WXJEMtfHe/QxXv/u/OJTWXfEEZCIef/VlioFAIKW5FLX3gTPrlY7+K0Anwew45NnSeyIlt8Kkm4bKVuBOvtCu6vw3BLUqDHy5aZhZbScpFCNjn9ARuFjB+ZLY133kluOkN8xgWwscUuOKMGcgr2K17J5t9+Z5pfsQifXuWWC1LwHene69ZAUkP3q7tOUg/auKnV+57QnvbNpLmk76E1KCgD3ZwSKGODm9/qytx5RsAZqaDbp2XRJ+NBGJnsfHnH+Sdgt5jLc4ahtRp+efN8c5+Y8xtTZ5xFQVD0g8eSQvE8FKpZZXfNWfCP5IUcage83mFy678/7piyXCKWIEZ5Ha+z7PPFORsNcx4Zo1QXebfXnFW5YX6XD+wAOl91FDR0WC2JtVgUTtYt3OjDf2mtNqzwCe4mwXjWb9QnuvUXfckRuPZHFpvNiTMIZGgczVPZjfvWvXs0ssvOnBRM4mFlzMxkaiRPmIV+ouv99tTT78BI/9EnvaZq385fSUF/Cd8i7v1Gr2kPqhNKpHF2p0tK1pJn6z1riJlWFOz1Op6CEfMhmPYMkuXfwfl4+vtE4LlKUryMguKecdyjN5SCaNXbuwegw2kI+oWz7wYV0u8duH5amA4c73VW8fhLzcoYwLRkeC5zR3QqBE87iCxsWs3ZzoHO/+/2i2JjDsIIyESYNj0p4rGxQgHgwQenJtzaQ0abYGHqOcI21iLjIkYCD5DBwC2vjgPbvWVchdVejPXxN6L2Zwxzeqe8hvKu908HB4V4LgN4IgUQfS3+TSNZD0IUjQdkxI5dOFVonXCCDWIjxTDcZ3TlfvLnwGveQhBjT67g0XkKO5ASwL30kzBfz9nQKQiU0Iw21Uv0Xo0Q5RIxb49K4zCm81OLtLsuZ1EIty/o+9RtKUzNZasYI2nr6GKZU0YYqu1KElKEEecUe9i15kT4kSAd5u4HlyvqIsVtbo/2/ySS03P9Qz1BxqUWS02xreV/5Ep2jDAYaE2DvIZ/SepfZU4Vjr/M8M6xlZTxHT/EQkxZEk0INeHpFNx32YVuSjT9jlBDvlB7h2Ja0qr17k9mb+PjRfBPaCBezEUi8r5tIAkj2EoKSl7vxeN6Qe5f2aKCnjQed/IOmxlk5Q35Z/EnnKdOu+ZOrfLDRvoMEImFLDLjtU+XtUm8cJFO/5J2bWk8V4LFI58d/Y2da24A4C0ZQltFwwNBtF6Z35j663MYlBzpgQ1uxL3KB2hZ785+IWPh0jJtQCGBuuyFMNhfSrPyShvSUoOZTvJ8QbJwk2lNLNTzFaR9C9bcU+Nx2+nMCeJEQ7xtWTqlcWYDocPOfViEMu3G97HAf3v3KTGY55A6uEK1ocrjsmzDb5lcBEFJDNg5RFe6YQn75DkLPUbr26H6M/yp272ZDDQ+nwFXAD2WfrcsIiT6QUAuDBVmwYL6R9v4xhwhEyCy1ff26JgMdIf2a2cfqFZkc2O/w== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: Kxru561f015MUwRRcV5V/q6KhauhseexIv22o2YqKOSyYyBrFPw1eVbzt4ZkV/+qoxiwQeBj/CFa2Ul9ybW2PCJp1hIh34VRvUzKOkgFA2pBQANMbqnIthAWg9YYFYKyecH1BDId88B4tJ8NdyiIvXogmKKk5+UZkigCvRRwhZneSXZeUPR3Mwq9/gAYnkrgCSlscJ/mfjmnyjyTcGGocAM70hhOpiGBcglo+cKusLUKASgzIeb/DbKTwW1xhvfAt2zgzyJaoiCAFApp7StmdDNL/NREZzYVR2J0NJDZ72X68o6R2eTLSRZJYVDN8FmuS3uyG6BBguO554DvqfPgGJ4dCthQXepvkDEZJVxMmzGuUfd9YgI/z0dJ1HQ+Xj8EAbxIrOx5vrZW8t9uG7RHLlMTt9u1Iwfa0YKhILtwFA8QhmILa/2MPZate2fhMHvFiboMlJI6gK5pLS+aoFD512kIjfNLPM34GpkpeTPxejs4CnGAentrmCXk9wEcXrQWzyl1sQ3uvU/rXeoI+QpPS9b/HbJM1FlA3DFGLENxganzdlEJdEe8t+bO3utVXw5Ax1UO1kLug7kcFQomqCnaEeH32/YVMI5BhY/iF/zDbVDjpf2HFpRN/8Xb3AJ76BA817GPwjnOXorKJ8r2fASnTXVNG8gf5X+d4f8l1S+QZnEQ4bHnNG2aMw/0p0EQgC8sdNz9eZbIYXI8vdNomZNoZtpT+WRWHnPgjlPLBMbsKZIcaaKQ44LzAKooC45bpPENx2Tdh+hLTZ1jhfRyq4SGC39tA3HWpeEpqny2y4mnvIbLS3KF/GTOJx16Cvj+mDVEvHjwOAK2LNv5hlCAzo7xTAeEustfn4jIMoncZdK2gv+K6HBqbg8fQzEnhY1Qvw/cbx2/G0T6LE7FzF5z2OogsbzC223xO6LXRSUJXKX0My0RT30uoXB0cYImw1rw4tfl4PKt4nxkm2HTq/3D1KcYWpHm9RE3gPiA+ge7Ace7zdY8AZXzKPuNZpmwdcyeputyVoAaqgmOsJ17BJcGlN2sJRS5t8DJ/Enln1QbC9f3OjQcd3KPEGrdvUUMKyWhNviimhafha4lTFIbvbH7nmxHB3EzN5+kCfBBTzHhf5XNboq3UEzJAxLYBbbv8mpNNQgg X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: f08e84df-54bd-480b-259c-08dbb63959dd X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2023 22:16:04.1538 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: g5wgSyNR6G09XhUHTGGdZ4lUBaziCOffsQYidofdwKijflM0C1DaxD0RpQMvLxTsgv3kNQMrps4R6wql6zIawg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB5861 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-15_19,2023-09-15_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 bulkscore=0 adultscore=0 phishscore=0 spamscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309150200 X-Proofpoint-ORIG-GUID: PipOTme2fxSCDHfAyj52zXp3pQW0KwHa X-Proofpoint-GUID: PipOTme2fxSCDHfAyj52zXp3pQW0KwHa X-Stat-Signature: 7cdnjghhhwrywj3e9co96m59cyhhyo4i X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: A138318001B X-Rspam-User: X-HE-Tag: 1694816192-641168 X-HE-Meta: U2FsdGVkX1+U2+rHCoEbAlIWXyjX9OJT/GmbnBE208A0AXce/20onrF9InkCd/OqdqyZSzxOpQcO9s9w21ZgkL483OjVF15O2vgMdHz49yjjxQhvgo10q3dTCW5pvrjRphK8sa7nSKFzXG9oduOqO7xMOH9R/ioBR+l+y483g6jpRdr3qFMWor6iLywbyr5owIlnI2VsjR7aoa8Ps6BfpgWGLMrbRc19WscSmvoqN27+YhOXBjdPCZb8eNGEllRWJPI6aD4QcY3oHw9sDzpbPF8xlvcS5ugFR+9vnbyXE6fcrv5tJdlnbhmPlGLpKFZY8sO19I+iDrqkLBih3aYrW5pON5Qq3Ah9tjSsxsowGwS2dCAOeV/TWM3OFc2hiHEwqlIo8J0uioeraM319lWdDlTMtqEnpAjk4wem7JVkEQC20D1eW2f4+3Ir8CQXNwIYSmsgz97L1MMR9GI4HjKBaYyuqmZ6IQ11TdqBJjeGvRSY+3cJli9oTI8ox6rEYJmLdwD8XvD8eumXdd4nvkQcZI7M4FJF+PGKGpE2cm2CXiRH9Vuaj4gP7AWyyKTjIKiafDLOg69yMiksdTVqF0q/gkX11bK8K755PLHVeUeEJ8S+FvFspl5bULYKI9yb7pEzl+FEiF2nXc2XHH8ZqAXqQ8+ECtYBKuFg/agETrVUNeeTB5bs5mVRkHhfNlhu/g+7zfByvmfzEck1LeT/wPDDkVahhxuY2xp9DCv4bqiNGKn5d9jg3v7vyM8s/jI/ig6g5QXEKch3SN87NAWkzVHLc8kieniqo4ZKKP1I98bvLj/aH8tmuogZ5KXBpxvSKM8F0K6pm2tuajDpq3mlwH3peT2jgVNgHYzJeCVWEDSWtb8d4VS/pyWNXLbxc5Dl3kX2fhZMBHaLIsT9IJf5JEZpU77sDIAbWJ04Nx608LFkU16Od7Vz0Nsvv/AYk/h7M+4bJgLFlKyNBhxzccf05dm TTkNrVyl 8ojg51TyMV6uxTPlRkrFh97uDiSZncIiR+uxwZ/sCZ6xyHdD8pHFSLrPDfBA/H2lrP/qXmG/vNcWiN4MLt7TtqDQMXAiaWcEQtHkET0hDqS4UdDyEbv/lDiBtobYWYdRwdYmSP3kn0xvoQg6IIWcHpWgBzGYKvUwm1o59J+XuoVDvcqwJQvFZ9v28eO6AbobTPJrejq+fy2jBdYKrvf+7KjIkQtd0zZau8V2zWzfE34v+HiaGBkuwEv9d4tf7UimyjV4HaRaBUNBbNEx0TTx4QVVHw1uzC2fkTU4uyc5OX3a9vGUzlY2f/ThvBH/Njs7G9Q7IkUTWdmvxKSVbEXrXkf+BHqxO3DQ+bNWf6iujlDj9KJVw56DvX+fVWnNII0PmuhDDAUNMIl2j+Z3sqHfTdo85ZoZpW844dZnPc6gRqiFAt7E9HuEjJAvJ2ncxslgP+20NmO/aSynNe3BPHYZq3M6t1NHS39oPJ+qoKtvuiAbueNyMvxEB60LkP6dtFjqVyu0RofD/n2wOJhxzhOHuUfkdvCpt3Gzxi66YzM1mpGD3f0DaN9WJ1Ubj8TovIcBGoGsswghMh8f+mJSXMR7DH3TB+5YF5vUMMV4yOeRJDj3wCU4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Convert the callers to expect a folio and remove the unnecesary conversion back to a struct page. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Mike Kravetz Cc: Sidhartha Kumar --- mm/hugetlb.c | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 7bbdc71fb34d..744e214c7d9b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1439,7 +1439,7 @@ static int hstate_next_node_to_alloc(struct hstate *h, } /* - * helper for remove_pool_huge_page() - return the previously saved + * helper for remove_pool_hugetlb_folio() - return the previously saved * node ["this node"] from which to free a huge page. Advance the * next node id whether or not we find a free huge page to free so * that the next attempt to free addresses the next node. @@ -2201,9 +2201,8 @@ static int alloc_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, * an additional call to free the page to low level allocators. * Called with hugetlb_lock locked. */ -static struct page *remove_pool_huge_page(struct hstate *h, - nodemask_t *nodes_allowed, - bool acct_surplus) +static struct folio *remove_pool_hugetlb_folio(struct hstate *h, + nodemask_t *nodes_allowed, bool acct_surplus) { int nr_nodes, node; struct folio *folio = NULL; @@ -2223,7 +2222,7 @@ static struct page *remove_pool_huge_page(struct hstate *h, } } - return &folio->page; + return folio; } /* @@ -2577,7 +2576,6 @@ static void return_unused_surplus_pages(struct hstate *h, unsigned long unused_resv_pages) { unsigned long nr_pages; - struct page *page; LIST_HEAD(page_list); lockdep_assert_held(&hugetlb_lock); @@ -2598,15 +2596,17 @@ static void return_unused_surplus_pages(struct hstate *h, * evenly across all nodes with memory. Iterate across these nodes * until we can no longer free unreserved surplus pages. This occurs * when the nodes with surplus pages have no free pages. - * remove_pool_huge_page() will balance the freed pages across the + * remove_pool_hugetlb_folio() will balance the freed pages across the * on-line nodes with memory and will handle the hstate accounting. */ while (nr_pages--) { - page = remove_pool_huge_page(h, &node_states[N_MEMORY], 1); - if (!page) + struct folio *folio; + + folio = remove_pool_hugetlb_folio(h, &node_states[N_MEMORY], 1); + if (!folio) goto out; - list_add(&page->lru, &page_list); + list_add(&folio->lru, &page_list); } out: @@ -3401,7 +3401,6 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, nodemask_t *nodes_allowed) { unsigned long min_count, ret; - struct page *page; LIST_HEAD(page_list); NODEMASK_ALLOC(nodemask_t, node_alloc_noretry, GFP_KERNEL); @@ -3523,11 +3522,13 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, * Collect pages to be removed on list without dropping lock */ while (min_count < persistent_huge_pages(h)) { - page = remove_pool_huge_page(h, nodes_allowed, 0); - if (!page) + struct folio *folio; + + folio = remove_pool_hugetlb_folio(h, nodes_allowed, 0); + if (!folio) break; - list_add(&page->lru, &page_list); + list_add(&folio->lru, &page_list); } /* free the pages after dropping lock */ spin_unlock_irq(&hugetlb_lock); From patchwork Fri Sep 15 22:15:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13387785 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9E9DCD37AE for ; Fri, 15 Sep 2023 22:16:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 872FE6B0389; Fri, 15 Sep 2023 18:16:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 821C56B038B; Fri, 15 Sep 2023 18:16:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B5DF6B0389; Fri, 15 Sep 2023 18:16:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 2110D6B0388 for ; Fri, 15 Sep 2023 18:16:39 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E9220120507 for ; Fri, 15 Sep 2023 22:16:38 +0000 (UTC) X-FDA: 81240242076.30.ADDB7A5 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf20.hostedemail.com (Postfix) with ESMTP id 84AFB1C000F for ; Fri, 15 Sep 2023 22:16:35 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=HulXkWT9; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=rNVHRQQU; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf20.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1694816195; a=rsa-sha256; cv=pass; b=Ws4w30WQ2w+XtOr7PnEoTCy84XLTC176bYpmPnh5PM9nxdm5Bd6EENrAKBplNGdwNdhpm2 hHVTz83U013y1LExLUJ3Fr6sq44UNxFHZhimwWSSE84BK3OZK1/WmtwtjJUzwNsntYfGAf 6EXGYFwJ/oCs34dyk5lm1p9FMbif9nE= ARC-Authentication-Results: i=2; imf20.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=HulXkWT9; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=rNVHRQQU; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf20.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694816195; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6WdNctGaCYcEcx93nm9VBTMI1xvm9ngqGmtgPK5aZYY=; b=HZiZ0e0uWNqxcVA78hQbwSpx2jIQ+fbEZljgrjwlMK6awfjxm1mLvuRfYG8qvrb3XntuAl cA+M3g5hmA1IxB7x8NdxZNACK8J0nBlupIuJ1+DB2i1p+G0QJP9N0rvMy2UILXgjRvM5YT GFwG1KTgyr/SjQuNrmauDKpPEBXXmxM= Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38FLvpmK026895; Fri, 15 Sep 2023 22:16:10 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=6WdNctGaCYcEcx93nm9VBTMI1xvm9ngqGmtgPK5aZYY=; b=HulXkWT9rIvq0D1cDorZ+ZLmoJKWpu4KcsM9uUdCW1zM4glBwbz8ba2m7p+t6ZuaM9Bs kqmLB9CZlJdpHhwz9/laduLnsCwIgcH85Hhs8bwEhBg1DHU/7tAwph8NR/A6fIuAWNyj BOCtK8X2OPQnzpJXao8yztWnrFkj9bnjfKKO76hBMSBhvYX5EypASihzgG56omdOCA0j saSlftHgRkCtTkgCQNyP2fKQSttK6MvR9AfFeSDvjkbYwsyDdwpkZyVyR3kwt6xtl2E5 2NKVC1Mpx5BLuAxFw7DoUCTuBmkwsOYE7Vwvc0FRyvgmiU3eXa1ZbQW38iI28HaJhk5s OA== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t2y7krpn0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:09 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38FLA011007422; Fri, 15 Sep 2023 22:16:09 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2172.outbound.protection.outlook.com [104.47.59.172]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3t0f5bd5qd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:09 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ioIjP8nPJeBxzzcWWW2bNVEnZRR+7b8f7zETXAVStai6grXiZfC6Sh0NsZVZb74G1TuKdnp9rBguWgRNDoaPWiVtwS/e/XHFnkJWZvs6GznrdjryKbSLgGpfsqMPe/KnoXcKINY2AG57rNkEDvoNdBDBWEK1NBzRSlTRle/J62a3rxUC2o/+F8WgfiLFERLd6UYg5r+scfJHGMGyAJiO5L++IGoNavQ7ttxJz1KhUN0XF49FG/HpkhBfc0prkratcosD6o9PmURxXChbWRwqa8gp4cmi5/d0JRl2SJDie/7uT+NsmxgpmWIT/EToxGDAXOpKrNeKc6Z1ivbZhcXNhw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6WdNctGaCYcEcx93nm9VBTMI1xvm9ngqGmtgPK5aZYY=; b=gap/b/CzlIIofeFu3HLTsLqldyimCvEhDOeM3bB/HR+1F3YqUHTwcdzCz5SpN13jX23MynlPN5/F/1HnfQzLSCbd17w0bK1P6dNU4dAycLilE8LmQcOpOjFnbr/d9ci43ZrVHSSev8jZiPGOgOSeac7l496OATmFYyBw0/9RzoA7FweJhlBbQIEcKB7iCOoBZtme4R6xy7y714Hs8maJQ0YeWcRNJxQSxPjsCqiULt6rsEXZezrjnm+2Xuv0CWr/R2uBfsvIEbNeA04YPIKyMqMtjzTolo40YSsUKVeDyoZTWWCRnthiNJXGa5L/i5QGQOXs5wEiC+OQKmtrsoRGWQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6WdNctGaCYcEcx93nm9VBTMI1xvm9ngqGmtgPK5aZYY=; b=rNVHRQQU6hYSsJhrnIZuCQGhyEnrs2yVHlHmjtL4finQHzMJvC5N6EhbzRRve4aOExQLAobjEkQG9uT75Nwghj/Lvgb0c3rYERl+31+DVombNv5RnYuj8X6Zt/3PXmGGsHjT8n+xamp37kqJFUKxd8HYIAf3TJWqBHj2XcHS+Hw= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB5861.namprd10.prod.outlook.com (2603:10b6:303:180::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.34; Fri, 15 Sep 2023 22:16:06 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.021; Fri, 15 Sep 2023 22:16:06 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz , James Houghton Subject: [PATCH v3 05/12] hugetlb: optimize update_and_free_pages_bulk to avoid lock cycles Date: Fri, 15 Sep 2023 15:15:38 -0700 Message-ID: <20230915221548.552084-6-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230915221548.552084-1-mike.kravetz@oracle.com> References: <20230915221548.552084-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0378.namprd04.prod.outlook.com (2603:10b6:303:81::23) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB5861:EE_ X-MS-Office365-Filtering-Correlation-Id: db473531-2370-4800-dc52-08dbb6395b49 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: OU4JdLrwrCxYsXATQaERkEcoBI3vVVSOleH/pOwMDTqLvRWJE/N/AyrHQ6JXbsSzKfJoKNDvj4qB3+B9rEYgPljnfsMLXbCzMgg0QGIFstWMWkgeI28LI/KONwIy/JVwRYTZwuGX3sSoWETn9faJ2YeQiOdT9fUj7DHE83647c7COAb35ZQ986YbUSurk06calwCpO3fdRSN4IqrjEIuPE6oqaRCVW+FHhO+8P80ek6rZNMEjD8XOBCGNayLeygy/Yhv7nrAmLyaL/UE+wQuX6FntNFQm2eciBG60KIp/gsQPIISCSIlji+c/NWJQ1IJrs2iqnndJvYHnwQVOrjuyEpZa05zaaaMVoJ3ZXd0tWqiqdZor1E8H0JfzBpUofeNgqDz/QacYTibrl3njcEP/4kxlMp73Fe1MzOOlCU7YoxENatHBaKUa8oyhm4ZloSbbg1ZAUugyiOV9NPGjPMx7ICFu2b5yaF39XAWOYVhzVLBYdJJAyBB6KIPsfTFh9uG3Kb1Zjbic1VFmVncSLymhjtOp3pQaQWEvLrhSsDZzLDe9WUofCWbd6wtn8Lq76Va X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(346002)(39860400002)(366004)(376002)(136003)(451199024)(186009)(1800799009)(44832011)(86362001)(36756003)(83380400001)(2906002)(38100700002)(478600001)(1076003)(2616005)(26005)(15650500001)(4326008)(6512007)(41300700001)(5660300002)(6506007)(6486002)(8676002)(6666004)(66556008)(316002)(66946007)(8936002)(7416002)(66476007)(54906003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 86O3UTS3vzhe9ZIdLpgLJBakHwtBmXMF8J6sTRAhhs6mEJAhXJW6PT+5cMcBSogdqpNGTHsYCK9Ij5BaUvIdT2EZ1J9m8o4O9FVLD5yByEjcEnA5LmNld1naaOL/Ppk0dby6Ogyxe5i9oKVrnsD+ZE5cqBgexazflWtDpSJO/voXw3CRGgbW2KKIsGd0aIu3/edCD8wNly7fC+KlSlkC6xhJA3eL6B774iehJjc9szbJniqTEmZmwl49ycKmc1i0tODfvt1HXMo2hQfukPXn+Yjlf1BLLi7R+YMtXs3Bzd2QDBjeGjh6Nq/oPsiklUMlwBx6cDL/OKFtlO4ygSuR0t30z7YP/1GmDzWR9HLE68WBdGsHi83Ogh8p2gvnnTb57IvuQygLrh9NQFlfrpPE5Z5+rvw0WWF8P4PkilAhP5uwbZ5zjzmgj19SOJHmo60eQV83/iu9X2Tp6CGMgMOnoSBMMZOx1YeFvGGvIztPArvdL6P+VB8pzqTmB5TGl6WiW9lwRGeD4O5pCEa+8SHCe9FRvB7MLjpXxvETi1F+AsyldB10cc3O5GoEQ+0x78WuiDUOfdb2ELrpeoAtunH+9k4iIX1xz3TzYEmUD5tf8WN78vNe56UDpzqqgQVQuK3qPnMOicfRRU5T9GD1CJlLyV3csqLqTBrKKZLlVO2LBvEwdQewmImfWcRHfhtcAfitQJUPh1JD/MxR3/wEF4oD/oyvCSojqpdIOaT/cXbDnw04ZTR4YG5WisfQMz3H4lLZbw3rxw2gfjz5lqeZrOj1BcabzieXcp8Hfq6lEeh99DB3JeUhOBdw86duhOggA5c2h5kMLz8F3Bwm/XtZFLM6SI0HKYY/pAPvxrfdyZXUDYq/AZHighqqv+TmFL7E84hA+gkYwBvlHhv//xazQBPsJrI/vc1k1eAwVwSisnAefK01tM82zSq0FCpV/8aYTxusC2yb7SDUIw/jRfZnt4HAeENVK4xJlXLMDLbF4+StRaHbOnUCXsFx0jZHjjk4IOiRODWtv9+BImQ8sPYVrZ8CxQmirsGmQUojhRf2xAPJphyxjjpdBDk6kCG7qKFlOcaJj5yEQeKV5eILFC6zm8NHbXGD9FSGM7/ohVBQVZA+SCR4p3ILE0AufazWkm27RFW03gJF3ckSJIrJl70jNRaM1sumJa6iI6GYhuTUPIKOg1SG7UJNR77qWs7Fvckid/kih5Btezh3ymdMUD0dFAAdWAI7Mdcblwit7N7Xgi2HQojxdtuzeg7+IKCzwSIkNuMnUpwwtvwI/frKH1/mRlj1bBd092CZ9UVcROB+wfYU8ectrDDl86+u1RSv4gG3AkUPm0qQ3eoCKwg5SptjrdzZIZ463hSKLJ1KN7dIHrCAkq7xeaK7C9KSV2UZu6I3SNBaYSnJgt5BBkoSr/X6BYpPtDmD2MSC17VzZKD8FWaZo9MeL40N5G80esT4t0sNknM+8RAU8+JmP0lXXA9gbyHXYeOIB0RF4F+WA9dQ0nKf0HYWK3C6HrrNu9pPzm+pUHsxxh+gYKvlpZdVYLdmLEBU1VfwKqcrg8NbG2Tbze3l3RnCGJAiXZTKjcnIH9rRCny0fByycg7MvSy5OFQI0o5pUg== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: PthCyh8nsiDOzGkG7as6+tShOlpIHgB0q31ApQZ2Xaq8B+e0GA8gxqx8fNpEr1YdMTJzMfT5YF04uZ1vWwM0Wz2b/LMDwrZIadX6K+PPQJZfMPzHEK0cRdr1XqfiCjCttFcsfrQLMCrTe2C6NvbV+O5RwY/tL+A21k5oIfVa57xNjGnv+3dwNQK36+n9A/wU6qny9PyZ4vb+UUz0+fO1ElT8XlgLoJeLjndCDQd8/yJTd8ZOh9G1jOymuTwiFd1ALtMJfGSxgJVZ7Aa5d5lbM/gtOxxs5HeybQn4iPsHzMRw6B9U39tIqRIWQdj26tR3TRuZdMGNGF0IiBmptROHre3uw84e7oVk+oOTbbM1t6p6/lCRLgzD38m+WK/GmhJSlKmCADz/gl+yZvQidRu4j63XqBFjRODh04W6mR9bljctBVOc1yOmDGsg2XinXMkvWup65/pwvnaDYEWbWqeJeXbpzvEit/TbtldAc4QcjC1QFz4qPhfIVwBx3MmYfKVC0bl0stq7Bp2WyU01Pl8Zlki7As4sqqAMd/kZ3VtAYJdR+HVUj8+uoGBBKzOQoOD1WE8HVLitd7gbCJ/AZSm8qzLjqyctO8vGDUA6p/QFjuB5X5en7gMsSG5hpoLFXAgcDsBDaItqXoATjqK6QeEyIS/qQuYB786rjXysnGy6lGTnCwCmQNBGPSm7y3Qy+5MiBug8xpS2Tj+jDtD9DyQiWyGagxlO19y9sMgeUZQ0gkrF98fv2yUWBvHP4P+/GtUJ9XVG8aY0HvCygtNW9Ji7H11chQbtW6R/pJiDtJGN5liQJmz9mhB1BRk7bTSNGbwxGCkSBCSZm5/2kimJxKKQfGNAk7tRIhOYha5T1MMLLqbbWe7YV/hliYmpwWeNhyr1EhVV73g3wtu1p9Jzb2OCyXe/MvYwoiw7Bb21U7xu7KMNoUSv/lNcJ/eVMa4tBBhkCojA/X40kwqO/2m4IBgk/3ELq4624UCuWdtFolWLLuanHaogiX6jrhIPLHLK3UpnDnInYEPjN3uR9BgeIsGveuLi6Qs/ZndegSdpfSIuKFpaougpdAQTRmeij6h9GDH2Tcj63ohOTGc466/VghNbjWXQ1ieJRPPGaKSTjLvpQTbq7AHAOksM5afh76Eikp2U X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: db473531-2370-4800-dc52-08dbb6395b49 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2023 22:16:06.5426 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: lw9WLHjOAZFxUyrcepf6pskcjt5yktHzIfo0N2vVf0qcodeMX6TN+zKllZMGIDNUrW8/uxLCvVPeD2UbGsCwfA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB5861 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-15_19,2023-09-15_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 mlxscore=0 adultscore=0 phishscore=0 spamscore=0 suspectscore=0 malwarescore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309150200 X-Proofpoint-GUID: tszrv8NPMR4A70xj1-3LxstI4rhn6i8U X-Proofpoint-ORIG-GUID: tszrv8NPMR4A70xj1-3LxstI4rhn6i8U X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 84AFB1C000F X-Stat-Signature: spi6wig5w3umddeemsg9odj7r9wy6shj X-HE-Tag: 1694816195-190435 X-HE-Meta: U2FsdGVkX1/8o9kpSdWbtNVGuRudtVyPtrZSmPPvPV6ehp5Ha+8XkAvvJXxhwSsmDhdriPRjE0wIzkBnEHlQWzZliqRnj4USPR4cMF7FKvmLra8xwIWXNkgLJjDh3nlWKxngHQpTNQikaxMwwo80mh9gb4OmUqFqqaprRdXCUwCvx+lBGu/U079vUgLLS8n8NP3yDg5Q201Vf3rM5ICDCALlN+IsknicmX1O3e2FZyLbgNN18Hjk4KeG3IHQCTDWBjMRjFVnoIxoUrPgUmcxrFBMbnGPeju1di9eZo/wPBTSgOjUsM7V4lLhSnVSVkvU7rnHNIl7d5NvN/Id9PK0PlpwzQF1jwYlAX4ZW6BD2YI/ruyJn7t4J8uBMX15K6GiwcrfH1aQRbO60/JH3Bber6j8mbTQHy7TgzjFCZL45+Z1ZYqytJjUzH2UrvsHgS2I3i8YQKKx+cXpJNhiFJWbWEhnq1w7rUyaZRNDI2UMcKXuMwh+MvJ+u3lYz1wl82gg77cdyhMH0Vq2e7jJCfdGfjnqAHqHYNvn4hOaKwwBkJ8JE+sUyX6bt1laFVZHCkaDwOk6JJJQIwKv8R4JoGXWWab8CT2ERFCz9EVwF9RrUzETpUf4o6TBQ0aEJO1OWoEwRWkHM5HqegfjeAQnKWdbxJwU2BqvIBnJMmFxDSreKa0ME9RyJWeU12To+iBOcwCmwk0KscP2r0twmvLMw7717dVeok3BpM3dX7ApBWzaEv7rjNiJet6mMUCul6oh3gWZp3BRrDMQUMYx0tMv2EMBW4c4qs1QFEVyVIxhNKCYmnFFMN6/Yp3BVNtl9cZI/1TKMA3BB0/6eX9xfJHy2i/70VbJiF3xNHH763WFSXsSQ+mb1PAj8II41FaF9Z1VzFw/mAQp9ykQ3+grnvKhVarat59L7J8dfaqlT1rva0apJBGm8wOV6ZGUoyc1lMUvLZYBz38EM/M71BPcDJgNSeJ /c40x4k9 gK992hqfT/p1Ytle/ni5L6xYcLQkBZa9uzaJZZnPsZf1fhVRXl+Ss3x++hA9UEE9DH1unjm66IOAgpaOIPBczuebq8XIUoUk8siJX/cgNh6XUVJSP0BVJYqu6qADgjVcLIo1nQAJUijjxSq+Tsnd6sSDzHY0hAkTVBQortaxkslDnQukkka/v5b2UJzobfvdM5zO5ujgevlby7rpZ3n1nVkqbuRwX4M5cLV5AtqJ2mEZwiPIxjWFVnD5lChovabP/6jlCAPZe0XrW029AOkzkMl1tk/WzPZ7uz8DWumITVXooay7wcjmtNliXima0wJukl/XNZkrDwyy46JhpSMCfpqu8tjfL/+tbrbux/FIzrc2xcXso9TCyMc0kuOyDe3VyLQ2MswbEY4ThwB5SgDBbfZcE8dvZTCVGbsOiXSgBYMijfzELybFyvXNq4jUCxeh22Jaanm7NE1z6d+GnFdQokmn5zDC+oR2Fifb7gyfyTyk66BFMrCLCLvVKz3vV5TpFXX+gh2y1RyeVIpF/THJhN5pzuqHFwKfbrpL+VO4MUEnBZVk1EXyo45LCy6SPY3JUB7pjO4++nf843xfO59ZAJxnUnHfHsD/bvqHjE3s9R2FAb+AySJyIRzQCietThh3GEUvP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: update_and_free_pages_bulk is designed to free a list of hugetlb pages back to their associated lower level allocators. This may require allocating vmemmmap pages associated with each hugetlb page. The hugetlb page destructor must be changed before pages are freed to lower level allocators. However, the destructor must be changed under the hugetlb lock. This means there is potentially one lock cycle per page. Minimize the number of lock cycles in update_and_free_pages_bulk by: 1) allocating necessary vmemmap for all hugetlb pages on the list 2) take hugetlb lock and clear destructor for all pages on the list 3) free all pages on list back to low level allocators Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song Acked-by: James Houghton --- mm/hugetlb.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 744e214c7d9b..52f695222450 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1830,7 +1830,46 @@ static void update_and_free_hugetlb_folio(struct hstate *h, struct folio *folio, static void update_and_free_pages_bulk(struct hstate *h, struct list_head *list) { struct folio *folio, *t_folio; + bool clear_dtor = false; + /* + * First allocate required vmemmmap (if necessary) for all folios on + * list. If vmemmap can not be allocated, we can not free folio to + * lower level allocator, so add back as hugetlb surplus page. + * add_hugetlb_folio() removes the page from THIS list. + * Use clear_dtor to note if vmemmap was successfully allocated for + * ANY page on the list. + */ + list_for_each_entry_safe(folio, t_folio, list, lru) { + if (folio_test_hugetlb_vmemmap_optimized(folio)) { + if (hugetlb_vmemmap_restore(h, &folio->page)) { + spin_lock_irq(&hugetlb_lock); + add_hugetlb_folio(h, folio, true); + spin_unlock_irq(&hugetlb_lock); + } else + clear_dtor = true; + } + } + + /* + * If vmemmmap allocation was performed on any folio above, take lock + * to clear destructor of all folios on list. This avoids the need to + * lock/unlock for each individual folio. + * The assumption is vmemmap allocation was performed on all or none + * of the folios on the list. This is true expect in VERY rare cases. + */ + if (clear_dtor) { + spin_lock_irq(&hugetlb_lock); + list_for_each_entry(folio, list, lru) + __clear_hugetlb_destructor(h, folio); + spin_unlock_irq(&hugetlb_lock); + } + + /* + * Free folios back to low level allocators. vmemmap and destructors + * were taken care of above, so update_and_free_hugetlb_folio will + * not need to take hugetlb lock. + */ list_for_each_entry_safe(folio, t_folio, list, lru) { update_and_free_hugetlb_folio(h, folio, false); cond_resched(); From patchwork Fri Sep 15 22:15:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13387786 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51C5BCD37AA for ; Fri, 15 Sep 2023 22:16:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B7DC26B038A; Fri, 15 Sep 2023 18:16:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B2FAE6B038B; Fri, 15 Sep 2023 18:16:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 932546B038D; Fri, 15 Sep 2023 18:16:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7A6306B038A for ; Fri, 15 Sep 2023 18:16:40 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 5413F1A1036 for ; Fri, 15 Sep 2023 22:16:40 +0000 (UTC) X-FDA: 81240242160.11.98E48AE Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf22.hostedemail.com (Postfix) with ESMTP id 09241C001D for ; Fri, 15 Sep 2023 22:16:36 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=iVRntUqY; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=TFnCOJn8; spf=pass (imf22.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694816197; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9cpMarytvHGbkfELIePaEAGmrxRtGSUtw1A3TpXlgaQ=; b=4aa5q6RHffwx85Ktp5G4BAAXwdvXyrYCXhTCx63Txgrr5wlpradJVTEBxo+s7dRZ9EV1IY 1JHvHBc+TECxDrkdvIXSLcOhtYAmIFpvIwd1kQlBwsfO9e11Fd7LpTTK+WijD2/ZJLiIv8 MIVKZVYNyJ2D1y8gUDpNy9Q2BgKERAI= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1694816197; a=rsa-sha256; cv=pass; b=1em2nVLXlK8vGau35+lycoO4Bon+UYD0mDJ0FG+MMJxG9d4iOLPe0cbpuyXoetaoa0GvL0 1Hr+RuWK+qFSaU79H9QIuU8n9DLF8zDRFNH0qAh3FCxUp/uWr0McXD5dlAo58Gc4RC5kHm IqqIahxIz6rpFHzQXvTTLsHBNvcacSM= ARC-Authentication-Results: i=2; imf22.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=iVRntUqY; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=TFnCOJn8; spf=pass (imf22.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38FLxFCn017035; Fri, 15 Sep 2023 22:16:14 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=9cpMarytvHGbkfELIePaEAGmrxRtGSUtw1A3TpXlgaQ=; b=iVRntUqY6vn7EmhH2/Q/boKaJ9cdb8/w/wo+mOmRW5xQP01rlNJ0wf9ewWX1iAZPta5O WXdqIWQEa+ULO38FgHkRnWEzobuzgOx+gaJ4xmY7fWGO11bGLHED1sxNVDB9uKPzKA9m 4eXPw+KTHwfzafhR2tPXR3lJMNQCg7j2Ht7DkYa2Z4Yn/M4Vg1t26MSPE6VczCI7Cwrr l7QkdjO+0qB5V2zOZK5TwU5WeGVZw3SkuEgj8vzloxXrqfi48jf8+2+QK7n3RrUknU8W 8yfjmGaHJlWGel++/suVBfPXM2GKDWh/AJXwkXOzzW9TST0q3WuVedjD4I3SL9eaQNP5 /Q== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t2y7rgxc0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:14 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38FKi6I0036207; Fri, 15 Sep 2023 22:16:11 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2175.outbound.protection.outlook.com [104.47.59.175]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3t0f5gx8xv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:11 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=a8tCsbvPRqYBfKLpmWCMacXHYMLI//z3mDDJLhZV2tIKcTBFUqRsbn1cHXRD4O26xYcPItoLstTRq2SAWX1hsZbfWvkQ7df/OCEhdOQ06dEQ0bSPbv/2F0HhQKv65OvaRmP14TfK60vWiv7GGKvSKjI1bBdj8Z+oUoArAK7MiwxVUfTmDKwfplQ1RD31ueIGodcNp7j5NYjlT3ZDvk1iZMRT8CfhRGNKhBss3gW+WGmtAyX5Wn7BnJ/qCzZE82uNfpkOWr3jI3uCRcrCGeHG3i12ubqh7gWv7fiTThpkD2EbdBUU/uaCnLS1a/McISTMwOjejQNH6bEWzkFJzcqwgA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9cpMarytvHGbkfELIePaEAGmrxRtGSUtw1A3TpXlgaQ=; b=G0QEd9asIC7QXiXKFo1RccqCom6u/3DnIGr7q75thMNkEOvM2BUbEGgJR3oTlNkEImriZtPjBU7nNUYfCpbx3bhdCAgDd81h0Zr/YilcJJgfe1SAFgLRVRtBqvdqH8iAjFrczoRPibVjJegyk7lPbAOGC4nisiS9yb6NLRlmC2lg9E9bTKqEeiJwBIyl2XmHeOFhL5q73Af/IeVFGLWUO/WpN70rGnUV3yB8GlaKg1WCg9g+RRa209hXsOYYAsexVq6Xh/u2gpDLoS7iUdf4MnJefIZjgR59AMoq0iwiN/0RVSbRN+cRpPzI34jQpWUz/VbqgeT5SAqE48yIr2B22g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9cpMarytvHGbkfELIePaEAGmrxRtGSUtw1A3TpXlgaQ=; b=TFnCOJn8GSr7hAFaxj6dUMHKB8ArmeceJGlcAMsYnNIPRwaADDqmYNrZl7v2fkh/HjIaVQfxBxOh+AnbAMK4zyfdFI/P6mjsDmeBUcJ6JrqglD12ZNXkhK1RoAxwehyj+fX/J8mUenE2D4CT9MY1dcHMJWshsKlrwssWqW8x0WA= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB5861.namprd10.prod.outlook.com (2603:10b6:303:180::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.34; Fri, 15 Sep 2023 22:16:09 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.021; Fri, 15 Sep 2023 22:16:09 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v3 06/12] hugetlb: restructure pool allocations Date: Fri, 15 Sep 2023 15:15:39 -0700 Message-ID: <20230915221548.552084-7-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230915221548.552084-1-mike.kravetz@oracle.com> References: <20230915221548.552084-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4P220CA0004.NAMP220.PROD.OUTLOOK.COM (2603:10b6:303:115::9) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB5861:EE_ X-MS-Office365-Filtering-Correlation-Id: 0b1828a1-c844-493a-42e4-08dbb6395cc7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: BZB+1sTU4I/jA9cNCjUw9gEp40h21/CesPm0+nnYUFylFqL2M1nu1gVU+HOrce5gF5GInqF41nelUEfRaFtIsTbB5hQt+6LFMVmdKEnExc1OB4IRCUvQRrR9Ay17n374nO6Xtvr6OsiknpUnVLDFkU3t8+XaVimJkZYVgO4Pi7GjP+xrM3mewjrfGzJ+mobzjdwEgRmNJ8Q2OAebtr5DSE72P7Zs7oBy7O4RgH2+IPfqSfPt5wP7jeVI098inFXuAs085QyXm4cqOGR5q8u/vP7975AunvYaQRXnDhuXcpVqWye+ifz3o0cbpIBq5M25jVIjFjMG7dhsSnRTZ+atPfluWfSCFigl2Qf9pikRImrsO0eEBTG9+XyB6Ngcf1Vgm9mxWPqVWQ4QLA5NAUsGhc7CVqEOC9CV4/G5UJnwLk2goCHiurJ9NymjUyOqZ7Nyr9Uthow4A/Cs0MeRjtIlyJsMAYj6LIJ2kCPGjqQuysuheWKKEpFL4rkUmXlaZevkA+7ZIQPxWWW0sL1A2WPbjv/IpivQe407IlkEyfJKR/aESyfsEI/WgBAr66Bf+nU5 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(346002)(39860400002)(366004)(376002)(136003)(451199024)(186009)(1800799009)(44832011)(86362001)(36756003)(83380400001)(2906002)(38100700002)(478600001)(1076003)(2616005)(26005)(107886003)(30864003)(4326008)(6512007)(41300700001)(5660300002)(6506007)(6486002)(8676002)(6666004)(66556008)(316002)(66946007)(8936002)(7416002)(66476007)(54906003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 8uy2Scp7BgU4odLXIztEx2zkfjkGtR5nWdir9fyKA+hE9Sg747RwNHadab6QfKMyJvW60KeVZPXeUPV5XDyotc8228iOwZqPrGLZ+8vH7zg+JBQ0CgeIziD8rOLGLOnBIQOLqjdkiTTzO+c6jGaHJQDwhhtdbVG2oAOWqFO+45ljRIJrqQev0tuFawn6R/ONPwX6JPWUCJQM2vKoSBipkQj4Px5prjDkeNFSqg0t7G8eN49QcH/foY4/Dd2uJrQVHX1mIx+4vB9YN+9NSaEz+n0zhIWUG2L+sDkx/MstsU2dqOKrx698vR9+Hj6QaVsQmuL6lFWlDRLFT6lFi7GGVbp9E9cKM4V57tWdbiX/zwZDarlsKffDEEGUuTPd9CAXJzH8u+blZ3PRi/H4BT3yP8DfXJuiENQbIrw7vLnunO8nbceJXOtdosWRaz7ZtRVORtdHpH4u0+6+POsZrhNZot71dSksUST5NbF4uhvTIkFG4K4qZMwvU5LgJizdoVGRwVc3BqTZ0vikE0BfFzB6Dq6vEVNTfxV8p+3vGlR9KXoiwYVJItdVAV/ojdUjzv3vIyh6C8sfGpJprYZyqJOOme1Ro2Oa7smKhRadbs0fKGURWFvl2+sYBO2yBKpWMAvT9TPR5WixtAgw1WvLaOa4os97PBIszHAqzmCm+zJsyIBjEGoPyZ4qk5nHc/LRpl9YMab3zqiIVu3CbqXD2L2f+t5hOzPUb1IV+zCyYG0K8tpTlUnkeGFyVewAQo6R0qSFliGpKzmOrqtq/6E7I0AjHZSOdNC7K33iYOF7RKuB5UbSKF6sXwGX0g7XYmJSB9o8E12JzmRkPJ39DH9aP1JrmJ9Fw8GP+gXuj39bgc1pCEq5wOuiQAG1GEeBY+eqc0wIwHMGMUm1eLzf6Q6TlChaOoWpBQx9nwvyVXvO1RdOb/1G3mIb1qE77Bykz1MGBveo9xGilhRyGDf644EBSGF4t34oIrrcbhmhFaFEODHpUO0BC6l0I5pmP2TMGbP9zhZH+QOi1sico6AMP9n+574P2z0MeIdnZ63BL2wZbdwnclUNT8+v/Uto/ubkF7KkdftEti//EA4zmsPTYCYE6cwJ7TAAZL+l21SPwM1rxNloEv4UK38N1g0+ckuohKHKdDtl/JnDAj6WcvaGZSQb8oHC5E1/M3iMYJsA9/xXbhmQk8nXsvXQHBEF2ui7NT+VUVCT95yJ90/fHyNdVMGiSyRtalMXYEERtIU9Pxt6gcdZaKJMwq6Eff5YmvEdbFF9+Quc8OsRdoKs8GXRO2l4sm0+3O7/1YibzetGgIOJD88l5yiYMuj4yREf12dF5BSfB7uRljIXt0+TdaWyCqj/BsWGqKgchEkUIW/fvOwb6bVd0Bev9ESkVx7T5k5hQMaypfPnme7gcGmo4BJY9oqOZC/5PkPC3+7S9sTkc5GjgvQfrfv4GvR2kY4BpusvRE/UFHA5Imx8ueinn7Xk/tXhXe6i3SBQghLjM2OoEs3S9f6poddos0zV+6SWTbORxpRpLd5OSCCgWK++VL6gIlVfvQgxlhZNDcd7IDWFSM9G4Dn+MHZB/rb0JWTusq3blMLq1KmgIRIWcxnVBxiTYLhpAvwx5g== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: TZhEnqBU/5EZ3tCZoY7qRsDQWqcruOuwCRWBPt+A6a1qog4hSaRqfLFS+HyOpgY2dv4HujnjZoPJ4AKqjd/vIZvtOW9JWDVlN3RYBIm/9iVHcUZoNs/HjomoefKajqrX1AdSUUsNmWlD/dr76JGfKmhxdjJMLmi8ab8X2q2L59ek7gmchvC+Sq2ZnVPRk+fVZoDr2OG1yYAhiUX5pq5PMKI1jutXTzN5K9LK4ZQnm/SMFm6EkwD2NIfI7NZUdN7cW3rM8L+ZJnKF4Dh3Qb5jlm29M3phK5may8sPTIGYTqoxwZwwaoZ1kzghBVAT/WAOLFooje3DoGamyb7lhrU84BKeftIwd/Icjzm6YBL4USBvBgQWgxONJYvLKJR0RPXT9miwXBbdwLWh0fn5uaqel30HIkRMNrm2hzLeRfd2KZszuKHULwvPne8B77JfD50bxCzS6mDesoN3dZz05SgiyC6tAZtM9ta7ohRB71LS0LpqLvWYGmgzwXoKG5tAcjtTmZ1FYZuk6sVWlDPzrNSBXrvxte6nMsEIY/6epBMcW6vZxly7SjPQffbsYgDB0Zb7BWknSLGbVBz9pxTdqUs4RIunuIp6UJlLePoohBDIvBBcBV2ee2lJbdrEcw2XVowz7rXALCW2wB6rdSfWeXo4ISbqENetTxhrM01J0QmF94EAFwXAFglSqo0f8to1G39XiLJcFCR98Qk1AtkkDZ648bgImzHGlPfk/XvbnQLW93txCPFRu9/rm+tfwAIVxB9QEgJ6uDM5kCjQksLxhAY39UnwhiDhMLW+5ReN/HqkaFg6R+1b+brZ2Gdc8RhRh94lQ+azr8YfoVEPmQnwH2SJMVZk8jMnwlh/QcMAeCqBGtZlZ1UU0EBMWeSRkY+qkmu5uIPz3IRYOXOE23hD49rfHgT8dJFeZCx9cBmGyIUXlxDc6qvZFA0rbyhOpmheSQlDMJRW/cxUtjFmMq9ZTiWZfuWzV+TefrFdpLIt/s4vJC3QTNr7DJLR8pCzRTH1iTlA9xetJ8VgoZ4JhNSwb0JHrXrF7kiTQ52BWNzeJvfgtS89syc/DP8lgZGVIt4LagNaHkdfCsA+kZpKhyyGfkXgjHHLgy/NKSXZHh0FhsYBYM/ynLxtm6XLiEJd6JiTNHZj X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0b1828a1-c844-493a-42e4-08dbb6395cc7 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2023 22:16:09.0587 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: bLXmHMzMIsm/cGhc/2RSF28XS7jxTINZIOfskSv1ooUoIG8rWpJ1+eq5y+AqbvyD7/KpVRx5lJxzonLhpUkiDA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB5861 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-15_19,2023-09-15_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 bulkscore=0 adultscore=0 phishscore=0 spamscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309150200 X-Proofpoint-ORIG-GUID: CPlRNB8FOKYYRd0A00-ads4OsbUDU1DJ X-Proofpoint-GUID: CPlRNB8FOKYYRd0A00-ads4OsbUDU1DJ X-Rspamd-Queue-Id: 09241C001D X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: ami169zabfqr8rkyth7toz4d6nyobxm8 X-HE-Tag: 1694816196-747661 X-HE-Meta: U2FsdGVkX1+lCAIi/f2hMo56PjiI7q5SPcURofjIP413/ix9IXQbYLFKE1XBjdsnvJmJiBqNkJ2fSdbav/jBt4D72hYbtyNjCVE80yEJMNmygkKnGUVu6DbKhs3WFI1Kioc6tcmrxVNq718DgEVbgIjDqdef3y073pGde9kAMbBPiiVsIMkVY7cA6r6/mdT2imGNGgykjoa4anWO2f9LCfwXKDQ83uUOOjauFYSgEwOf0xOkoQbWVYRjTISbeN0/uOC4Wok1WiVYkJA0T1rwCuU1voKfkQ5vYI/lRIlzwepuBpZPBR/Ab9C14JjHGBzc0Yx9Ul+ib6TRG/qBncEbaaWDvsPIkf2HO8rYXttQvhp0ERBBpxi+CojXZiz1Zf6dvE0ov1v/xvd5tvDqe6toQA+FNNyZGlN9qkjrLHrqL7KL61kWnidutVJ8vPQLwZDE8XDjaCIv1MBzKYaItQ+OBy+9FxxFNPf7uftU788e0b/+6x4jMrcsNQJG1JFlDND9oWW5igAeXvDaFe/2veB/yJVEeKR5W9vE53AJx8YW4ydXlC1gaNo5VGHWv6X/eTlgM1L7VkuTnMTuF4f3/UI3RuPiQJQfu7XpnFA0FeIO1bapqsCPiaFTF4bpAt/vFvNOHVZHxpAsxMzM+EaZCDBJVCIpYTVnMyNPdt7NyzKgs7Lb3uLYg4s5+c+yg2PoRI6WnsKFYtvDaxxD+TfJbCzbyXXxj+OTgWSBpPhGd5rLYKpyLNFq+kHjOJKOV7HIeNpmFlSuStsB3YouBK6NzCFHJfwtZXTb4FZZ0ay8GXfUAQgt+GJ94gal/jlvpe69N+nioxWiX0mY0tN4u9pmzyQOd8JAB3iPymVCou7KMIeBezI220T6wYdQ9Vb2kZ+0PJ3ivNFkwRaDvXiufupcEcUJlFt8dtgTkpYK0pVK7VzL+BX0ZAymU7AyITP+BpZyEnBpxUhdarAmCFpjYAVts1g gDJ4DA27 qGGTyc07MX2F9IW8+q5GNHO4VwzDly7ztYWvMpf4le7ddRlu+4WI52/EOljsBR0ax4q1Hz4fv2s2qsN0SniEHd1wAnYi1D58RRnrLyXXqeeAFeQgTWJMsS4vKH8Qdla8Y19PW9Koc6AzTB8MBOaxgycmNNT+fhPcb9jbiRP6gSVfx2F+jxoLB4HL92dBhrcbBob5u9Mb5+H8Zk8t877/wNkDEsEY4VbIU64C3icPDHPiGfF99GTGZj8rJ9/Hr4J01iyd6XGJTohVa9+oz6QQQAIVp/XBqwiv8gqprYSkq6v8xClQmE6u6pWYy0h7KpmQrBtN6ICW8JcUHfrXegNoBjwGdFnQePSgK55l7hG0rws8+5ZsR+hbJ16iBvkaLjpG9h3B0mh5bada5fekdSBqH3nhdjGipbpx3NqtUAZyhzM9Lld6koSHkJ+HL45iCWNAyr8JriV703b5QruixgN9QxaqY+CAPSy4Wm9pA X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allocation of a hugetlb page for the hugetlb pool is done by the routine alloc_pool_huge_page. This routine will allocate contiguous pages from a low level allocator, prep the pages for usage as a hugetlb page and then add the resulting hugetlb page to the pool. In the 'prep' stage, optional vmemmap optimization is done. For performance reasons we want to perform vmemmap optimization on multiple hugetlb pages at once. To do this, restructure the hugetlb pool allocation code such that vmemmap optimization can be isolated and later batched. The code to allocate hugetlb pages from bootmem was also modified to allow batching. No functional changes, only code restructure. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 183 ++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 144 insertions(+), 39 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 52f695222450..77313c9e0fa8 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1963,16 +1963,21 @@ static void __prep_account_new_huge_page(struct hstate *h, int nid) h->nr_huge_pages_node[nid]++; } -static void __prep_new_hugetlb_folio(struct hstate *h, struct folio *folio) +static void init_new_hugetlb_folio(struct hstate *h, struct folio *folio) { folio_set_hugetlb(folio); - hugetlb_vmemmap_optimize(h, &folio->page); INIT_LIST_HEAD(&folio->lru); hugetlb_set_folio_subpool(folio, NULL); set_hugetlb_cgroup(folio, NULL); set_hugetlb_cgroup_rsvd(folio, NULL); } +static void __prep_new_hugetlb_folio(struct hstate *h, struct folio *folio) +{ + init_new_hugetlb_folio(h, folio); + hugetlb_vmemmap_optimize(h, &folio->page); +} + static void prep_new_hugetlb_folio(struct hstate *h, struct folio *folio, int nid) { __prep_new_hugetlb_folio(h, folio); @@ -2169,16 +2174,9 @@ static struct folio *alloc_buddy_hugetlb_folio(struct hstate *h, return page_folio(page); } -/* - * Common helper to allocate a fresh hugetlb page. All specific allocators - * should use this function to get new hugetlb pages - * - * Note that returned page is 'frozen': ref count of head page and all tail - * pages is zero. - */ -static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, - gfp_t gfp_mask, int nid, nodemask_t *nmask, - nodemask_t *node_alloc_noretry) +static struct folio *__alloc_fresh_hugetlb_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nmask, + nodemask_t *node_alloc_noretry) { struct folio *folio; bool retry = false; @@ -2191,6 +2189,7 @@ static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, nid, nmask, node_alloc_noretry); if (!folio) return NULL; + if (hstate_is_gigantic(h)) { if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) { /* @@ -2205,32 +2204,84 @@ static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, return NULL; } } - prep_new_hugetlb_folio(h, folio, folio_nid(folio)); return folio; } +static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nmask, + nodemask_t *node_alloc_noretry) +{ + struct folio *folio; + + folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, + node_alloc_noretry); + if (folio) + init_new_hugetlb_folio(h, folio); + return folio; +} + /* - * Allocates a fresh page to the hugetlb allocator pool in the node interleaved - * manner. + * Common helper to allocate a fresh hugetlb page. All specific allocators + * should use this function to get new hugetlb pages + * + * Note that returned page is 'frozen': ref count of head page and all tail + * pages is zero. */ -static int alloc_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, - nodemask_t *node_alloc_noretry) +static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nmask, + nodemask_t *node_alloc_noretry) { struct folio *folio; - int nr_nodes, node; + + folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, + node_alloc_noretry); + if (!folio) + return NULL; + + prep_new_hugetlb_folio(h, folio, folio_nid(folio)); + return folio; +} + +static void prep_and_add_allocated_folios(struct hstate *h, + struct list_head *folio_list) +{ + struct folio *folio, *tmp_f; + + /* + * Add all new pool pages to free lists in one lock cycle + */ + spin_lock_irq(&hugetlb_lock); + list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { + __prep_account_new_huge_page(h, folio_nid(folio)); + enqueue_hugetlb_folio(h, folio); + } + spin_unlock_irq(&hugetlb_lock); + + INIT_LIST_HEAD(folio_list); +} + +/* + * Allocates a fresh hugetlb page in a node interleaved manner. The page + * will later be added to the appropriate hugetlb pool. + */ +static struct folio *alloc_pool_huge_folio(struct hstate *h, + nodemask_t *nodes_allowed, + nodemask_t *node_alloc_noretry) +{ gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE; + int nr_nodes, node; for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) { - folio = alloc_fresh_hugetlb_folio(h, gfp_mask, node, + struct folio *folio; + + folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, node, nodes_allowed, node_alloc_noretry); - if (folio) { - free_huge_folio(folio); /* free it into the hugepage allocator */ - return 1; - } + if (folio) + return folio; } - return 0; + return NULL; } /* @@ -3196,19 +3247,29 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) */ static void __init gather_bootmem_prealloc(void) { + LIST_HEAD(folio_list); struct huge_bootmem_page *m; + struct hstate *h, *prev_h = NULL; list_for_each_entry(m, &huge_boot_pages, list) { struct page *page = virt_to_page(m); struct folio *folio = page_folio(page); - struct hstate *h = m->hstate; + + h = m->hstate; + /* + * It is possible to gave multiple huge page sizes (hstates) + * in this list. If so, process each size separately. + */ + if (h != prev_h && prev_h != NULL) + prep_and_add_allocated_folios(prev_h, &folio_list); + prev_h = h; VM_BUG_ON(!hstate_is_gigantic(h)); WARN_ON(folio_ref_count(folio) != 1); if (prep_compound_gigantic_folio(folio, huge_page_order(h))) { WARN_ON(folio_test_reserved(folio)); - prep_new_hugetlb_folio(h, folio, folio_nid(folio)); - free_huge_folio(folio); /* add to the hugepage allocator */ + init_new_hugetlb_folio(h, folio); + list_add(&folio->lru, &folio_list); } else { /* VERY unlikely inflated ref count on a tail page */ free_gigantic_folio(folio, huge_page_order(h)); @@ -3222,6 +3283,8 @@ static void __init gather_bootmem_prealloc(void) adjust_managed_page_count(page, pages_per_huge_page(h)); cond_resched(); } + + prep_and_add_allocated_folios(h, &folio_list); } static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) { @@ -3254,9 +3317,22 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) h->max_huge_pages_node[nid] = i; } +/* + * NOTE: this routine is called in different contexts for gigantic and + * non-gigantic pages. + * - For gigantic pages, this is called early in the boot process and + * pages are allocated from memblock allocated or something similar. + * Gigantic pages are actually added to pools later with the routine + * gather_bootmem_prealloc. + * - For non-gigantic pages, this is called later in the boot process after + * all of mm is up and functional. Pages are allocated from buddy and + * then added to hugetlb pools. + */ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) { unsigned long i; + struct folio *folio; + LIST_HEAD(folio_list); nodemask_t *node_alloc_noretry; bool node_specific_alloc = false; @@ -3298,14 +3374,25 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) for (i = 0; i < h->max_huge_pages; ++i) { if (hstate_is_gigantic(h)) { + /* + * gigantic pages not added to list as they are not + * added to pools now. + */ if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE)) break; - } else if (!alloc_pool_huge_page(h, - &node_states[N_MEMORY], - node_alloc_noretry)) - break; + } else { + folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY], + node_alloc_noretry); + if (!folio) + break; + list_add(&folio->lru, &folio_list); + } cond_resched(); } + + /* list will be empty if hstate_is_gigantic */ + prep_and_add_allocated_folios(h, &folio_list); + if (i < h->max_huge_pages) { char buf[32]; @@ -3439,7 +3526,9 @@ static int adjust_pool_surplus(struct hstate *h, nodemask_t *nodes_allowed, static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, nodemask_t *nodes_allowed) { - unsigned long min_count, ret; + unsigned long min_count; + unsigned long allocated; + struct folio *folio; LIST_HEAD(page_list); NODEMASK_ALLOC(nodemask_t, node_alloc_noretry, GFP_KERNEL); @@ -3516,7 +3605,8 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, break; } - while (count > persistent_huge_pages(h)) { + allocated = 0; + while (count > (persistent_huge_pages(h) + allocated)) { /* * If this allocation races such that we no longer need the * page, free_huge_folio will handle it by freeing the page @@ -3527,15 +3617,32 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, /* yield cpu to avoid soft lockup */ cond_resched(); - ret = alloc_pool_huge_page(h, nodes_allowed, + folio = alloc_pool_huge_folio(h, nodes_allowed, node_alloc_noretry); - spin_lock_irq(&hugetlb_lock); - if (!ret) + if (!folio) { + prep_and_add_allocated_folios(h, &page_list); + spin_lock_irq(&hugetlb_lock); goto out; + } + + list_add(&folio->lru, &page_list); + allocated++; /* Bail for signals. Probably ctrl-c from user */ - if (signal_pending(current)) + if (signal_pending(current)) { + prep_and_add_allocated_folios(h, &page_list); + spin_lock_irq(&hugetlb_lock); goto out; + } + + spin_lock_irq(&hugetlb_lock); + } + + /* Add allocated pages to the pool */ + if (!list_empty(&page_list)) { + spin_unlock_irq(&hugetlb_lock); + prep_and_add_allocated_folios(h, &page_list); + spin_lock_irq(&hugetlb_lock); } /* @@ -3561,8 +3668,6 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, * Collect pages to be removed on list without dropping lock */ while (min_count < persistent_huge_pages(h)) { - struct folio *folio; - folio = remove_pool_hugetlb_folio(h, nodes_allowed, 0); if (!folio) break; From patchwork Fri Sep 15 22:15:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13387787 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F388FCD37AE for ; Fri, 15 Sep 2023 22:16:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DEF466B038D; Fri, 15 Sep 2023 18:16:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DA5566B038E; Fri, 15 Sep 2023 18:16:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA7CB6B038F; Fri, 15 Sep 2023 18:16:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 9E68D6B038D for ; Fri, 15 Sep 2023 18:16:41 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 69036C083B for ; Fri, 15 Sep 2023 22:16:41 +0000 (UTC) X-FDA: 81240242202.24.8FD70AB Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf30.hostedemail.com (Postfix) with ESMTP id 2597A80024 for ; Fri, 15 Sep 2023 22:16:38 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=R239j1iH; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=gGDRmG+y; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf30.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694816198; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PvDEfXofWnbA/mu8Nfn0j1Qjeexj3L00jJmMYHtlJvg=; b=4MXfvjlV4J2gz2oK6aiHce7obRnN8WxdoLWRhUj2tTJ8qkUzuMRmrdyhDF62YOz0XCF9JV dr6C5rQNJZyjhNQG4z4tSumbd9pGu8LLJ1H77OOAyPjKO/MuNF8yG8n9Rru9SvQH+TfiFV mhvnMiZQsxgpZDeVOpjeQCCwyPjqFLE= ARC-Authentication-Results: i=2; imf30.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=R239j1iH; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=gGDRmG+y; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf30.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1694816198; a=rsa-sha256; cv=pass; b=qVE1XpuCeiZcKI6IMidkaQlvxP3+Xc3F9/xAmjE0fbAfvzmS7H1E/gWJ/Kg1eNRR9648xu y4uK4kbtKrw9nr0xuWRyH6tGkwZF6jDEzSsAOo4GAhdi9QbmU0IBj/zdhh9P/5j86qXqfB 7LQDATudo8p3JtQSI200X7eo2Pzs1Ms= Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38FLwmhf016392; Fri, 15 Sep 2023 22:16:15 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=PvDEfXofWnbA/mu8Nfn0j1Qjeexj3L00jJmMYHtlJvg=; b=R239j1iHA+WD06NMRyUJZpZ1I0Gy876CQxjJ1Bv4lzQI+mOezvDOPudzXsSv3qO6DYAN 9IEFnwSXCRgXX7Z8Azd1/VR/OcOQLBHe8NsDYfSetbll36+8vPUIbehgjRgEW3XHH6eW 1J9uJA9LIiITQGhIwx+M4/AgtT5vtbTFqceZF2tKE9emFHIDUmnkeJDgOCr0uSK2vm76 fYqF6kKkJfYrM6DqdK7DI36Xgh/G2x8n0DQphIFcNpoLtwhTVd4IZQyB45jDIBC1fygT //shbY1eCWqeHvyQv6619OYAMjJa1EP4QPpiVjPv8mYtvj2YI9MOJV0wb/ktBgAzJf3A XQ== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t2y7rgxc3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:15 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38FLb0xG008807; Fri, 15 Sep 2023 22:16:14 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2177.outbound.protection.outlook.com [104.47.59.177]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3t0f5awd0r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:14 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IQMlrGTMv3BBC4FCDlXSdY1CQnpYwu5Wadozso6PKkDemZpZFNO/uh6YwdS2LiZ/0A/v4YPhNcRjmc4DCTHbCqmfFTM2uws8fCn41tDl4syq9tpsJcYrKK/Sl9H5ZGl+zpffth31oORbX793WLFDIzyGr30nQq4ZO2BjFeDrSS55L3EJBWftUoGy8OhqOKD8HqrlaHqKr/y9rAX9FjlCNf+WIHTrHbITqeT0myGZQ4A0qxzXC0l9tSijWQBAUwKGC8rJkz9nWoyP/QBmisZvmxsjfWrh6b4LxFE8x1rrq2x2ZDO3fOzGtkIr0uSKlq97tW0VpbFsyUO36FGSPz1bXw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PvDEfXofWnbA/mu8Nfn0j1Qjeexj3L00jJmMYHtlJvg=; b=MM26kZ2a5m9l6Fo6xlt90FU5G815ioyqBQYJvg8FXgWKC4OEuwe1803INqUWzv9f87qciMeOhFoUun7R1/aqsjLZwWg2C1+vDz8HktjhEU7c1adqG6qXxHXMcWPRz/47BG06G0R2EISjnaA47Qx/0nGOhAqQXjFMTshLjs//K8QOmzEx/NcvMsK2I1gRkXsBnerlJblE69k2jbSTkCpFQ1tD8I1uM5oAT5mUaF5Ydsr31JHB8pGlebWjIPDZbgMmMfQbhiEp2zI7WnYNzEydyRv3hPzg7FnO22hGTdE3UhpXRdkg8/rLUHiO5DRe/GOweWMLXo1k+e0QvaCaDGHQ6g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PvDEfXofWnbA/mu8Nfn0j1Qjeexj3L00jJmMYHtlJvg=; b=gGDRmG+yrWn7fm3Bvtlc4CiezTdxDjOVZZxxSPcdWW42c0omncNGkIu7v6ajDz1JRaA/LgTtbX4tbdgz8oiUwK0Q5J7koVC3GXW7nrloDvl5+uVbWt1qYl79DdTremUtEScqf/HdlOes9iL7J3ugyabVKEcyIgTc03Ef2wALS34= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB5861.namprd10.prod.outlook.com (2603:10b6:303:180::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.34; Fri, 15 Sep 2023 22:16:12 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.021; Fri, 15 Sep 2023 22:16:12 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v3 07/12] hugetlb: perform vmemmap optimization on a list of pages Date: Fri, 15 Sep 2023 15:15:40 -0700 Message-ID: <20230915221548.552084-8-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230915221548.552084-1-mike.kravetz@oracle.com> References: <20230915221548.552084-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4P220CA0009.NAMP220.PROD.OUTLOOK.COM (2603:10b6:303:115::14) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB5861:EE_ X-MS-Office365-Filtering-Correlation-Id: 25efbe30-169c-4b56-1ab4-08dbb6395e79 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: dYssCvivKBqfP6m8oKvo1tPAGYquavbkRDYt0iPkxbQ3OQT1gMd8HHjKzKTbRo5avo2pyK6NMGdpvOPq5HLn6bCHGD10TuYBou5prb3Clk5ISzId+UR06ZwkoAfTRlPjiOmbzi4qn2u46DOvgiXRT6q5sfplMyt4QRcZo0PH6CH+whvdpvKPFevs0BwQA9OEExi0zJnTThhpI2JEzLMYmdEftxEcp9VslZXWqrQqiromDkBXGhBQsh4c5wqh9O0h8FcjnwgrMgkFkvo44TtSWECzMeZTVw3iSz43krUzEfZpGTUXpr5xYx9mynkk6JMERACAURHcQcD5gF+VF+WcrhEZ4+T7eYIr/obf2k4ZeSqUN7DWuQ1PtQ4AuTC708l5zCRvTrRALjE4oaHpUa8wggWKC5YeNqJHEJpFbWeR32S5230kwjVcGW6fyx/zIyNwpeVczyZgbW2CLDGtpv+eGi7fkl3xyBrwUORFZS7fsCJvNswtaxfJGm1LZScnkZZuFkUHc/GKUB4OLqQMwJqy+hwXhNRR9IdykzW/BtnyaLvCjXya/tsHruoaMoVqRVsN X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(346002)(39860400002)(366004)(376002)(136003)(451199024)(186009)(1800799009)(44832011)(86362001)(36756003)(83380400001)(2906002)(38100700002)(478600001)(1076003)(2616005)(26005)(107886003)(4326008)(6512007)(41300700001)(5660300002)(6506007)(6486002)(8676002)(6666004)(66556008)(316002)(66946007)(8936002)(7416002)(66476007)(54906003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: TwEqCjTIvRg+vOBcMjMtCfh3HLe3aomuU2yTAhJXnrLWWbXAeJmOzf+il7Pgb+cUM4uLkNXRc4/mVrRaDvm+MAOD32YARDCjA9aXX6G8Tyl+30FOe3yTmRyM2t3mmDbtJbLWsh2sWnsgrToyIHflaeHVg8+jJsenyCf1wxXTSRbJtfpd5nGgKtIVNFW5l4BF/Q6rYkNQ7qdKNCyx2b3IX0fqSv8taaIjVRjbxCsfRyk67PJXlwK1NHcgUH9UW/CtfVhzqacsCBPuKSYHdIRD2XC01UlDN02HnZpEtJBNpExTrqJQyKt7fTQgeVcUA6qIoW2q84+8ww8rk0WbE/Mn8LKubuozzXenzTV9+0JWwz6PUjnlGC2k1agygF5vKinBbhWLk6wms6kKNwPf7UzBrqtmS3Ljn0G9FC1q5Wf2pNTmqnHNIFygXXmhF+iR1yzYENc2bJ87vzC3YSGb9xCdT16W7xJ3UKbH2dJDLaB7iI22q41DUqMHshjj7xVlUIIeXNv5PJDSh7mvujjBhe0K6kSfeJVXCWlhCKu03YZz2SZrbGVELqyX5p1S37Dtn8Ld1eNR5u6LBVSxQ92wcyCNm8onB3LYnMGFUNOjQzawdpI25Gr4R/jU5wbQ71I7kUPK8S7HbwGnmKWuUPbfWTqsrSCiUiffxYS0RKiaiIGRTq3v1xHglVCwmrPAFTt6g6TrlBbPnHit6++cU3SveZORfMoozHzRy8o0OWKTwlMOxTzMhuEzYNjEfKImtlj4UTH2Q6O60dluJpoVpYdvFlGKHmdYbRm5mMy+vFpdLn7gXHu+afWgStiElUDEGNo4fq58BEsy8QkMlDCUa2IBhcGaBTclV+3eUWv5LY5TyWZtb9Ju7pld2eWokcWmONViJfSI7hsno+9AW5DtNfSKdP993eNDoGuURdAuKhVr2sDotkIN6fSMMOrS+8SnQoA/gQnQorYd0qpB/bgIrAskaVcsr4QHaB3qF5X0KV7Y7BFhhphjg47/UmfCqu0FX+v3WEbb78zJX89kPNNosnqiH00C3/i+k2vvG6+MPgxwZ9bWd2xmFxavjAsU01j1MjWLWlTRQLLk3cYH/Q9ITevLsZ0O4OT1utgug2bvDZW9JwAmVbEwqRCqWaoMVmVCb5fpvN7Tc2T00pLfqdB8OQiIEmIAzyzLcecnLBGStLDXG820Cif1XeMB6A9oP+rIzTV9UuZtzDgng2X+13+tu7xSH8e2jHj3aHkspYQfh5PGL3cR12eaYS4cAWZh4XSaKJAo3E4vdrKjA1zTjUDM5mCGiFlerSVpbp4EccbqsG/RpZ3Ox6cYAQocZVCvphgpT+1/qaAyRTm26md9BjpBmCjFlVu4Yi8NG14khMPl9yVugsP5pZjlgBedUc+HTzjzO9h3S+ObPLlZbtKOgIbw/rsMFyeVoO2XS/GtLOtmiliwYkDbQNkkXqStee7Ir2NpzWZMLiBf3/UYpEqfGAxfoYX9n+GhR827ToVWC4jWkrlJQPM8n/SdCnycI/lSnnx2k1c1Qzq2DOsSrSYNellcPXpJjONlvUvknUr8ELam+bAUCyYB+UMfg/2dyF8m4pEl/U5baryGBjxQE89gO2ZN/WPemlf9tQ== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 1CN/tGvphJwXn3dpFfbR1jgD5CF0IOYxPvcvtYSJWfWs+94pqiu+a5g4xaQycSd5JkauATGKaZJJN/sH2mkzKapE+sOWjglGIaH7KJr1pPomRQ02H3h6poe4gPeZYTiHDKFvBRu9BsDlc+TzaQSDp1evb9CFx30xCRJPhqKezGBcJnRg2n4YrvUNuYuuMCqzJkOS9TcNqN3v8IrBahpE/mdZtESqL3xb77zxqn50PFBY5dNq61yht4IXwnXfJfuM5sswA+RPU7TZmbUvU36n46HkfA+8kwqTudQSYh/ZrkCZkN1iDYXcmaxLqN71x3Q3By9InviTS94E5jxxMVKteu5nN63xAORwPeSgkd7syKHg7N5Lut3UVCYUIM1miFs1PMfkjyHJybn9D/7REJyvCBJcW8jy3vuIvGYnaYm9AkSf0Dmtn3KsiTekayLxvvHV9/MGpXrnnHhttFVwpDcgSnghfNWrDOU3/0NbBWM3z7ofn0hEUEu/LZ+kTfqwWQSoueCT/4apyLZzvhD6pFiISJEm+LSsbdt2Fco0TmhajrwcW1hDRQPx+z4hV35p1wR9rEljRie/iz54YzZJQg2omSwASCUilcwFnctSaT+go88ABjxiZsdWnG7TEj078TI1gnJ9MG/LDKkC4dmPy/TMgUfLq4ddg/1zff8QdFiqj/b4NHHDbuvQU38yX66MEjl/RvjoHSow4iMcpa5Mh5s+E9XI7by41EBcN63f0grsqN1BJO6P1cj+vAkciKqlu1KV5PT1ty0JOc4ANKLL7weFFGQQLursCzQNvte/DZmZThN5EzobBY39HYvDO23EpTcTmUQYr0ewIsdRkBs1VcXs3tGgkUxE3RO4VCTfpe1hmgQFRdZoLh4ij8TJ2VjR6XmsFb1yjWFqM8FhKGQkzq9plQIy7DTSnHRiKqoeqFgeQACc1d6Fh3fmbTTD5l6tAx8btEjNKC4SoZMdJK2udb8lEOkoriYlj2dIjn9uIxG2xBRdE9mprM9NZ9rsTgYVW3A1S86WaTaEl5DgQYGJDVTgsJaJJLNbhEc4r4bW4/HJLVcrC9APofWYgyX0fm1aF5Zb2//viap82b6yDlthKrFjVTCqf0OI6urN6NW/fEPOebXotuIGgZ/ZYQZLNbepcL4a X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 25efbe30-169c-4b56-1ab4-08dbb6395e79 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2023 22:16:11.9181 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: H2SGDHWQr9kMEx/0piHnE+V065w8n32w9Sp487NfEmUqkgRuCUu0BdYODL1/7W5fBjoM1ydpoNL1um7vUBHYZw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB5861 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-15_19,2023-09-15_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309150200 X-Proofpoint-ORIG-GUID: 7eoyWQtg0vZ_C6KjTxBNa9nbQIokJEDs X-Proofpoint-GUID: 7eoyWQtg0vZ_C6KjTxBNa9nbQIokJEDs X-Rspam-User: X-Stat-Signature: 3zbqjo93kifniaxno9qgphh3emqpotob X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 2597A80024 X-HE-Tag: 1694816198-84867 X-HE-Meta: U2FsdGVkX1/0uaZ76yDM4ITeMjGfh7PG4TCVAlEYlCMpP9cCUtpXFU71b8DUCW3eZrhjFuXOHbI/+iRi9Mps9qNo6XR6Fd08w6fAGrdJttQjHoZRGJiTG3Fo1/Y4sC+G/JrVpt01RCiku5QWK1hmbHPrVcC4FldlTFc8j+szO/QPgAJzbx5f6gvJ1yuihZKv4HhntmO7tPpq00jElg6sg1umTkQB6eDFOmD3X0YsNYwkcnOTwQAgInmJhwRdJoOEdPxExLWcIgOYgeh52BYW6PASu/6ebb4hUiquxeJB3noCnPlVmaT3LjDFiU1/CzqXTdEQbWwrEK1p4KhpeVMAqRQ7LVCsg/jXJUw84ZvP40CnUZ8VBu6alcf6u9c2RPZwrk8v03KARsCF/IbhBa2/NS8X4p6dj04TW5sdtw1ak1/9eIcPxFQf24K+ix+QdZ/SX/GcTmnl90OxcCoOvDYu3Y7EKLNR/C6X0/4Yx6FBfB5xLJHgJoQFYo/1pdWJTdN4jJHCMv/u3BFLr0Et/8T+QtfjGfdJqf1pZaE52JnhL4PPhl5oYmjrNV091fj2hnc3NqeLYRwhs7ehGhtiNeeYaZ11wiWnrRZ1ERWMEIn0fOBP9pXQzeBSTfqdAe0AAaVdCf51vH04BYWonIoaGS1UDgJ9Vpj1Hp36ePrL7LP21nqP9Mpcsi20YCbsqPhexq9Yro/riKc9rqRFjP9/q9Lu5FXbREvC3YFri7Feh4euPYyj42bxpnrTJBhkkb7vu3wHmVoX/zyiPvmFK6bT1JuKZUb8XFsD32VoE4cInEOF9WPhwtYktSh8FMdV2oe1SsgLgMGCBefBhckDGeb0+SoNYHf49uwyrAs4wqzIAASxOUfP4GyRc4owa6ieOqvUXVryPxAk0Im9uOJAS5Zuw5Q7+CJ8ZhI/T48Urg/OXhz5rNRZJzwe+CERNws/Zdlt/QfyIyX2n+qkcHsmJoOBUJK LtZ7FNuz MiKr3IkYdgJ46XmgRqrzsqZ+kk86dbBW1lW2dZuVOL9V8YECPsFBCbM4leZIHAp18CTsnaFPBZJvwPyg8TN3ta0qkXVTVahKYIqWRNf/ni3ccg8oTlWoPCHx3O4wTRr+nfV1y19HxS2gsWu4KN6Qf27EFO5yznxMRqHpwAUFp17AVyP2MvAe8dVQISBkmnFvUpJmRxhdJKyatxF8fHQYIES7s/fNp1/lWYOCLn6W79Hgp1pssz2FAUEkwsB/D0j/dR0mz4Dho48JgxBKJiTKEOEjfboJJdjch7d89Wh1Ivx1yrsyr/nOJTSvln/sAsguO7ZRCDiQG1WcQk3tE8HEeMZtAMIDYf20ipbWgQNLmCp+waJR/0xMDxhG6zoSxfXXeY3YMw2mo/RZ5OIntS6bu61UEnWc0fdq4J891SYRo9+dkw44Q+gPpwD6tEcVIfPQKB6+eVrtiTrQNU5j2HQg127Yym8EhKDZr89vH85WhKQS+qGuEVC1Nh1UkJ6tZCWKqC03F/LanVn78+6ZNGkGKqvrDWYTzkwlkUqmCu4LD3Nkjxk9O49RJ1t++Isy4PD7/OKQd X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When adding hugetlb pages to the pool, we first create a list of the allocated pages before adding to the pool. Pass this list of pages to a new routine hugetlb_vmemmap_optimize_folios() for vmemmap optimization. We also modify the routine vmemmap_should_optimize() to check for pages that are already optimized. There are code paths that might request vmemmap optimization twice and we want to make sure this is not attempted. Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song --- mm/hugetlb.c | 5 +++++ mm/hugetlb_vmemmap.c | 11 +++++++++++ mm/hugetlb_vmemmap.h | 5 +++++ 3 files changed, 21 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 77313c9e0fa8..214603898ad0 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2248,6 +2248,11 @@ static void prep_and_add_allocated_folios(struct hstate *h, { struct folio *folio, *tmp_f; + /* + * Send list for bulk vmemmap optimization processing + */ + hugetlb_vmemmap_optimize_folios(h, folio_list); + /* * Add all new pool pages to free lists in one lock cycle */ diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 0bde38626d25..c17784f36dc3 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -483,6 +483,9 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) { + if (HPageVmemmapOptimized((struct page *)head)) + return false; + if (!READ_ONCE(vmemmap_optimize_enabled)) return false; @@ -572,6 +575,14 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) SetHPageVmemmapOptimized(head); } +void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) +{ + struct folio *folio; + + list_for_each_entry(folio, folio_list, lru) + hugetlb_vmemmap_optimize(h, &folio->page); +} + static struct ctl_table hugetlb_vmemmap_sysctls[] = { { .procname = "hugetlb_optimize_vmemmap", diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 25bd0e002431..036494e040ca 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -13,6 +13,7 @@ #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); +void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list); /* * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See @@ -47,6 +48,10 @@ static inline void hugetlb_vmemmap_optimize(const struct hstate *h, struct page { } +static inline void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) +{ +} + static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate *h) { return 0; From patchwork Fri Sep 15 22:15:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13387788 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EDA9CD37AD for ; Fri, 15 Sep 2023 22:16:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D565D6B0393; Fri, 15 Sep 2023 18:16:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C8D176B0395; Fri, 15 Sep 2023 18:16:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A949D6B0396; Fri, 15 Sep 2023 18:16:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 82A716B0393 for ; Fri, 15 Sep 2023 18:16:49 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5860240E0F for ; Fri, 15 Sep 2023 22:16:49 +0000 (UTC) X-FDA: 81240242538.22.86E2390 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf22.hostedemail.com (Postfix) with ESMTP id D0D83C0027 for ; Fri, 15 Sep 2023 22:16:45 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=mKt1cFDX; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=k6YUlS3x; spf=pass (imf22.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694816206; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oDZlE4S/P8gMy5+qqWQXiX898yEOVoj4504q4XjHrvY=; b=NFmymu5Idgx4pbxvsQn4cLKtrYXyYYhL26QQuYPbGOdgXHpol08r9s8NwBtqqNPF3gBwAY P6gujU5Yw272l2FdywEef4jEqI24XNW8BdshqOSNAl+VEdaNlvnkvlKsHfz1E6KximVXZX GFXgJqeg7LjG4/vjyPGd/eQIdPrYCxE= ARC-Authentication-Results: i=2; imf22.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=mKt1cFDX; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=k6YUlS3x; spf=pass (imf22.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1694816206; a=rsa-sha256; cv=pass; b=5YVpwcbJK6ln6+ocdHSQTl3mfseGp9PplZQSyRr4QIBZSU2NZGYUdsiustfwg4XOYLti5g fezaeXfp9sdIwm8FUBlcXEb/S4tuBrFj9jrgf2CneyHbXcqTB5RtDH1NQzB0jKzKpxNVNZ TrZmak/hsm9cjGZoOKgqdX7wccBgsmo= Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38FLu271027077; Fri, 15 Sep 2023 22:16:18 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=oDZlE4S/P8gMy5+qqWQXiX898yEOVoj4504q4XjHrvY=; b=mKt1cFDXMM3bX/MOtmFd2+pfZcx8bJ6GVvIcv/H5BRhqDahR1YIACPUmdz9ayx20xvZS nBdlAZMZ1lxao2kbLNskeO2CXq9OPyBwNrFo5dXNyP6TTZ/uZHLnnhhH/9C4PO7XbMDa Y05xW/b2XWU0PvmpSSm/0GhVsH1KSWwkMrJlr//QNxrcXxcIE1aJ2Z61gEU2sLwQhVIG HLhVhtf7DpKZ6xO4Q7u0yKGa9TeCWMlhLKN2Q16o7Pf5YIK5gSgHDBnKH0K+y+y2NO9N x4hvIAOoOQ+9uWbMQY1h82Se1Xcy6b0l3whbgVrqj7NB45fufbGGH3adTVi9b1beou4A 4A== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t2y7prp00-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:18 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38FJkubT036245; Fri, 15 Sep 2023 22:16:17 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2170.outbound.protection.outlook.com [104.47.59.170]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3t0f5gx906-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:16 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kzAGC7o1JrV0jbbWIkfX5afd5Rq+5FWocmprUQ+1XPkQS0TcKRoyIIVq+eNZOQfkLcf/tB+uoEnzZyc6+qOsWTxpjaZHXQRTsDAI9aJ4Ywzgyt/TF8wdK6uQd2qCeEd25eHACuM30T/SqFGY8zkmn8YWn90dOn3TBQ6kM3mxv5vGgLLOlVuTCF8y+xzCknw1K2XbQ6NRQpw+3TQ4A0A8YOXOYgQ9xMuBtjR8x0aDIgRgEbnCXXHduKiToqGu4O9D+Cw40wad3frGZNKizlBIHQpM36Os3ufWeJIbrH9lGvQ1QBMWVBvhCeKWGXj5OWeFtbsv0c/LPCT2+YBrKggvkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=oDZlE4S/P8gMy5+qqWQXiX898yEOVoj4504q4XjHrvY=; b=M04lL7M70r6xL1rQDM0gLzso/xnE5wlT+LHRk69y3fCDUWLR4YGYHN0Y15t3Wyma3N7W8AVUjEYPGfgWkso5v9LWKDX6jvQ64QnMXiTi7GcacOg5CU9nDUR/xbP+jzPmSYlYJhi/lES3c9MxBkM1baDlsOkH6vKVjn1+DQO0HaRtilxnGQlxZfwUr4CEtqyssrETtys9fIxa2RzQwAWMjmk4WLk6ZmYxqjJrUvSfeG/bMPb0w1oJR8hAO204tsmQWYmHr7CqV/ePRCpeFDvXeS7rBq3LsPCLNpMjRjVuPQ2NMKuSGUI/8azlqWMytzl4uimZYCPvuXHpub8p/AZ8Sg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=oDZlE4S/P8gMy5+qqWQXiX898yEOVoj4504q4XjHrvY=; b=k6YUlS3xhC5+3QWiCGxy+0ERPfQpLzvmKmMoBbHD4dzYkdwf+Dh5UOUMKcAPKuHYLaKYJfnB3vy8K8zMm1wweShowIx+OVE9ThPOyUOlotRlju94WzKjITF4Tu3KyP9o1YOU5ef+OVQ9Va6GfYGrel8h8Okz6Wn7WgGgEZwyE4U= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB5861.namprd10.prod.outlook.com (2603:10b6:303:180::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.34; Fri, 15 Sep 2023 22:16:14 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.021; Fri, 15 Sep 2023 22:16:14 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v3 08/12] hugetlb: perform vmemmap restoration on a list of pages Date: Fri, 15 Sep 2023 15:15:41 -0700 Message-ID: <20230915221548.552084-9-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230915221548.552084-1-mike.kravetz@oracle.com> References: <20230915221548.552084-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0022.namprd03.prod.outlook.com (2603:10b6:303:8f::27) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB5861:EE_ X-MS-Office365-Filtering-Correlation-Id: 2e2649fa-a489-4861-b9b6-08dbb6396010 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Q4ZwTQVhXh9lCE5/f+eot+ZjOENl061bUSIa+DFnKR7MbtGK+Pz7dAQPw7sTS9rXN+BlgWYUpBjjRxi8TYa1Zn6iOCtgYOHIo7oYbTtfDKarUmX1c3aJ7EDI/UbtLejoCSAqixxF4hDtw/l/W8JCg16OntcZIW9fWz9EKR1duPJQmNlqEg8GM2bTObbsRdyRzkle2Q/PxJF4I6LMJCSB3oyiuo7eq2FjI3iIfrLRU/osxFohSsXWbLBcMomwEXeR+9BX8YUUmXA3C/o0auHgxqORgPwWMr2Qf+EbHjAwzQek4bmxkytElLN5wzh73fE9Qv8g+BrtOT3iBxdC6AV7fRtGdoiVs628ozoSCZgY6C3c0IolFW1FE6klOYMTjlN1OtgkpFX7e28tARlyq4/GWOSVwDhFw4z2pNV8kBZAMTclJw9B6hK6Hf4G6+C8bGL3h3wRfrj5DaqXIL8KmJz8mtMHeNKbX+vIhCZFmwdnw9ORhaIebbISnHJ2aNa5xcumOKW12TVoLUyMLn51ZRO7c1s0/LdQzMt3LKkNTKyf1t+nTnRPjyddOeRZoKMQcRkL X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(346002)(39860400002)(366004)(376002)(136003)(451199024)(186009)(1800799009)(44832011)(86362001)(36756003)(83380400001)(2906002)(38100700002)(478600001)(1076003)(2616005)(26005)(107886003)(4326008)(6512007)(41300700001)(5660300002)(6506007)(6486002)(8676002)(6666004)(66556008)(316002)(66946007)(8936002)(7416002)(66476007)(54906003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: dIzXfKaNi7kSWYx/yPd1aQd9JWjL5MdHxSmORWf+T2qwzSB9finOeKdQC81aDT/lA6SeSSRUyDj+5uRZY8KMQmY983PJwF9mm0DfooGG3zDKCyZkJUF4rW9mPIXecqZnHOa9HyR0xooeqXWKFRxWAamduJqK9B6yh1caclqYkUb8wzVHOE2+CMoYeol1BuSQOY7XXVs3jEquVAEXKA/92jYj5tZXwzAMhym7sTJMqaw8SWECZe8bjqLpYur2LxmqAQk+wKSxcBLtdTDeX2/CZuVrKYPD2yKTUSLtraZ9RnmtInQC8lzN1nySTSLXgLIof0qr2zoKw2wkiGuLen9Kyrcm17/CkoE5EYW7uYIjGQITqaE7cVt6XbSRs1CZ3+OUU9UxvdmDjIqtmaqCtpZtMIgtyV7TK/ssMrPrDUNyoJhvNjmBHK9+zZbH2fXQqJp1oDkKyTds7wC41BjKrhffbJSfJYezetZaNx9UgHYSsd0KuxMC5ouGsLcIUjvhv4iB4Guj2QzkvFqeVtG4GkmhsfSEJ8Ik0M2mE1KK6xnIOnSUGD0vw6DqPy7K4k3f70rphhSM2tyjSgVPRj5RuVR8Z1Oh5opFMATIvrwYEZP6W0A7ARxjvqPVWoBBG0gPh5I4ntTtms4ib+42S5Bv+C4s4cL4GIKwdbOl301Le6GMQlL7YdazViI1FbN0WOQQJK8N+iPt8nheLQCl+ob1Y06+Fs5NmQIKBCX1fa+3AmYQFECufP+0x7knDke5BmCPV/62qCPMx0jBHFykUMWSaCP91Pbeu84qdN5wJuuwHayndIonT73e5u3wsZTf3+WdgDmvq0M/+JS36Bv6FeiC3Ot2V3K9fSYB10hvCOfSrFrFOc3trRpqdyGiuIfAsAJ8biKGvHQMXkL8MlBuxlGgw3oQcoHSCWsh9HSaHFeIvYiU+d+MbINj9ml8WT0g/GQtTb11vKfXunfww/A9yFiFxItfXrIwEeH3O2mTXdiYCbVfaj2GweQjIPN1exlz/+rLB2Y6L6wN5QTBYnWKbnaBJLh3DceibWWEqdSm1lsbhjbeJ838hvkh5a4GZp4T7h0iYE6TkTCfPPPi/1xDIF7qJ1rslk+GMRwqrqIM/XpX2/uaJAk9zk5ifwtRnLklfvg4jBJyAJNXsbq1uHbD1EahPkW/MpZzrVpyVfUoTM5+cn/9DBW1tQZVvzayjVKtHmgyVyrhJbKhRUEh8CGrONspZmOccZHbTCo9U1hQHwaVF9IRcrYPynHoG6C+87L48Q+QEjQG3/L8hG3UbDtwpW/98NMmUvosVaE/StXQkOKspw3EWMKIDR5ElfPFvD7P3anWQtqB6aVSpfWqsEQedEy81y8Y4j8At2SOTE4py3gEFQGpQYPrqX3Wwj6OJNL1UsS5tmgxLUx7ZGxWzN3PeN0XDbUUttbmKqfTJGQ0B9KuCGqtGT5AdbI6blpTFmQvuKjpTIGwZ4Kkhqp3QjtRVPtMTewhrNcFihhsKSQ/fiQrB9PCMxwnk8J8TnfHu3c/RRoC+22eGIyfyhlW9kuXgEisHgzjWW4cesWbvCRmzpCUWmOtVOkKVtfCkrvmQsbbJ9FEYfeZplfBEWgTAX7A+7o5Cvh5Og== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: oZIHqSeKh9vKbPteDEfHfjF+U7AWyoF/CYzgfEXXtkbewjbL7IeCMH1WbVqyebFiNMGxy+aCM/UCQnzicO+dA9LJx2JEN1daqMeFdqSi12mXz4NwDInuQfKBUAyXU/Hp1u++CZDaAQoQIYT7bdtysdUB5/T6ul/B8FFdijZwaOR+FsGYO+IoOIBHP6FjWLkVeknzAjIJP0TUY/3Abaiqf78/gGOPKxn9Wg7rQ8SnUXZOPrYMT+RBc0zA3lzdlBECVCsup0pW/6FEnzB7UK6FVXYU4h+HX3rkZLNWgzZSj/eTKutGLz5v3CIReADQHALh4Rc07XefBCWBhlJ/g8yCUVsf/FG2MiZfy5yzaiNYJ5FL6XVbRCNovgwKDIfd6ymevIc8cZa4FPgg6CfojAXUNNyDw2apQum0SfmSZACpNwg2zti9h7Kb4GnxcuxhVRO5F40roucKHelwGdHiENxdCGanwsIs2X1+YWbSx81GccPe5aUVDCReTmxxcUJLr+J/QhM3x2G3cx19L0uku0HZ1UdtJsxzefhE37jUR2Kw06GQHeY79domJgGYj0/7Q5JqYAFqB1sVCa171kwSxcvjoxBkHyKHBfLlnzsoHWv7r9zbYFfL9lwJQcnUGXu234blJhav/klyCU6TgE2YEAYkj3iFHdjeN4GR646HlumAr0xxWSqGS9VulJl3a0p0myaNxcY4RIkTvSmWc8GJn2GLYH6dFzOBC8nHjd2NWos6YTJ9o7/p7QUyrBGii+7KoapPv9UhGQ02WMacefE6fX6s9ieTRJjoC6iOegQDoxK/VYtlxI0W0OHKqYzlHF6mDd7K5va3GlJdQIYyRnXwPC65DrQkFUyiQ7EoH9bEA5prZgcDVkl6v2OHRKF83K64/CQARp5HRGXGuXlgc81RcVe/D4pa86Zkwa7CV1NCoZVO1KxZZkPWJ7Ox1s3a2mBqn5DUaHggHcric3i6K0ABLK+kjBh/bG1vI011j2x/qtRVHguSCN4nzeD62Mb5kn7F85gBjmRSSeiERONh6KPEsypP06dh5XuQvMpkscQz1wH0kLtQK1XToj+DvRZZc217LswQR9d8/od9rITAq3lOwqD83jQ0o0/LVjAa4XbDlyLSHXULTT9wwHNVrITOo3nRiHPL X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2e2649fa-a489-4861-b9b6-08dbb6396010 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2023 22:16:14.6150 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cVflspdx53e/Yve1leD8OpzAhG3ScWSztSflE04Mzf0pEBtZcvKXWsANcPj86611TOJ6druBWWGmlApZfylf6A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB5861 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-15_19,2023-09-15_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 bulkscore=0 adultscore=0 phishscore=0 spamscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309150200 X-Proofpoint-GUID: HX_rZgjwfQoMcw1znlN5y-AMmK6nOOF- X-Proofpoint-ORIG-GUID: HX_rZgjwfQoMcw1znlN5y-AMmK6nOOF- X-Rspamd-Queue-Id: D0D83C0027 X-Rspam-User: X-Stat-Signature: 8jatjprhxa7iijrk53mmt3zncxzcwnfw X-Rspamd-Server: rspam01 X-HE-Tag: 1694816205-523249 X-HE-Meta: U2FsdGVkX18qRVRycjI9WuGztgwiO+TmNsTbJDrvkIutgA0ReQr2EI2NBE8F9whSHG3b2h/Hr7Szx+wWaQhfTARe8m/4VFjDrG6wrRzppBA6nGFDj1H3sM+8lAP0ZTdiFQnkivNukWEucYjL/hgJjZ5k8FnDRkkWaGZJF31kyRxC9zYkqqsmlDXzy4pznD9RpUWBN/FSYaw9Le27kvbbVwVjqJhaVjYy3udsK05NSEYyqZPfSwjAYnEY2Fk0iz5WWzs1kdwyEtbBvvEWkJJ5pttQCLex8xzq/w69ySCsomNC84tZ4YtIL+meanuq1DRllc8JNv5DRmaJoOjSFOhpfT87pn+4YDF5R70nKbiwzYvppSrg48ohoNSUOvbZwBtesV+OsPcDqXXViA7CCA8hEvwiMC8muTeVoVtOI+YvVZjq0eoOicMCapZ1uoUvPhk4mLknWPaL8OkKy/YKaTe4cNFQL85KcAQoAxYLk7RJfF/5UomtQkfxG0jO8ozy3azCVFfWt8EGoHJFY30N+3LIHhasLcQD6vYsxplID3xMKiyE7cGa8ZGjUORMoHaLGPR1qowowVmuXC7OAlPyfbWJi7EklmrbsyYY3IKDuagnJr1sYCv1WnSj7nbrMpy+jIEYPszWSHN1NMz4qZftP8i+U9wzN8ET4PbodlgiJy6MUKMVjVH1EVNJf9wgUfoZ+EQfYlagMWx1ntfWa3ZAAq++WeAALcujL/F8sS4RVycL/ie8mBmTq8fbtqyXF6k/RjNjW9k8ZFgO1GLy8MBEq/BYc6RXf4AWEAOpXSB2czbYjqeMfKv2RGaFUP1kLm72YKfyQakhms6uKEdB2YRDTIKHdHKIttOdeo1wC2g5mkpvg57hFL6sZTFTEfvZIh6BIEE1nNl6IpBRpM/o1CJkLOhC6zMSEVcDYlnTPrZzPk4mY+Q25J8+b7kNdRcGlKLoWxOeAKg1RFbGy8N0zl0Z2eS r/jA5fx0 GLhhFOeSL1qsx+/6ewg4YiYctHum/+fEwMmaKqbUJgs6AtHa9JhzDEvKrJCySuBBDRTCqsajYUwXKpvixsK5K+YUUozYgeA0znWvbh752GMTikbItdTmSsseUkl/g9DzBNLwCZFIVkYuS8B1tC7ny/jvuNpvOPvYaYhVzOjkZdZp8fla8GGNhz1C3oS84nB+WHdiR8dFOawJdTfIlef2o5gZWBT75eaRVBCese+0XHIiMK5kg4un5Ft7zHd2MGiwB+emmGjhRTnty01TMT0xjoNxMVHb9dB1ubBekP7id4e01pEUQNjQKmfGRR5lw3xlk/e+laKd9s0i9ZRRROAp1biDLlvcHLjSUlNlB1RmuIBv1l7zB+T7n0Wp1ObcDjvCnilYxwkq3PgS8pLpfwYWn0xxf3Tckje0UYAnHo+pDTcWG+5Z3dwWLI4eEMyFeXVPbSsnacYsDihkFicGTQvtMU5MtjmOgsbBkyP/i8cgVlz/EzHg8LPp6MDjq3/rNhQCcfIZm X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The routine update_and_free_pages_bulk already performs vmemmap restoration on the list of hugetlb pages in a separate step. In preparation for more functionality to be added in this step, create a new routine hugetlb_vmemmap_restore_folios() that will restore vmemmap for a list of folios. This new routine must provide sufficient feedback about errors and actual restoration performed so that update_and_free_pages_bulk can perform optimally. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 36 ++++++++++++++++++------------------ mm/hugetlb_vmemmap.c | 37 +++++++++++++++++++++++++++++++++++++ mm/hugetlb_vmemmap.h | 11 +++++++++++ 3 files changed, 66 insertions(+), 18 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 214603898ad0..ccfd0c71f0e7 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1829,36 +1829,36 @@ static void update_and_free_hugetlb_folio(struct hstate *h, struct folio *folio, static void update_and_free_pages_bulk(struct hstate *h, struct list_head *list) { + int ret; + unsigned long restored; struct folio *folio, *t_folio; - bool clear_dtor = false; /* - * First allocate required vmemmmap (if necessary) for all folios on - * list. If vmemmap can not be allocated, we can not free folio to - * lower level allocator, so add back as hugetlb surplus page. - * add_hugetlb_folio() removes the page from THIS list. - * Use clear_dtor to note if vmemmap was successfully allocated for - * ANY page on the list. + * First allocate required vmemmmap (if necessary) for all folios. */ - list_for_each_entry_safe(folio, t_folio, list, lru) { - if (folio_test_hugetlb_vmemmap_optimized(folio)) { - if (hugetlb_vmemmap_restore(h, &folio->page)) { - spin_lock_irq(&hugetlb_lock); + ret = hugetlb_vmemmap_restore_folios(h, list, &restored); + + /* + * If there was an error restoring vmemmap for ANY folios on the list, + * add them back as surplus hugetlb pages. add_hugetlb_folio() removes + * the folio from THIS list. + */ + if (ret < 0) { + spin_lock_irq(&hugetlb_lock); + list_for_each_entry_safe(folio, t_folio, list, lru) + if (folio_test_hugetlb_vmemmap_optimized(folio)) add_hugetlb_folio(h, folio, true); - spin_unlock_irq(&hugetlb_lock); - } else - clear_dtor = true; - } + spin_unlock_irq(&hugetlb_lock); } /* - * If vmemmmap allocation was performed on any folio above, take lock - * to clear destructor of all folios on list. This avoids the need to + * If vmemmmap allocation was performed on ANY folio , take lock to + * clear destructor of all folios on list. This avoids the need to * lock/unlock for each individual folio. * The assumption is vmemmap allocation was performed on all or none * of the folios on the list. This is true expect in VERY rare cases. */ - if (clear_dtor) { + if (restored) { spin_lock_irq(&hugetlb_lock); list_for_each_entry(folio, list, lru) __clear_hugetlb_destructor(h, folio); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index c17784f36dc3..0eeb503d8a4c 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -480,6 +480,43 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) return ret; } +/** + * hugetlb_vmemmap_restore_folios - restore vmemmap for every folio on the list. + * @h: struct hstate. + * @folio_list: list of folios. + * @restored: Set to number of folios for which vmemmap was restored + * successfully if caller passes a non-NULL pointer. + * + * Return: %0 if vmemmap exists for all folios on the list. If an error is + * encountered restoring vmemmap for ANY folio, an error code + * will be returned to the caller. It is then the responsibility + * of the caller to check the hugetlb vmemmap optimized flag of + * each folio to determine if vmemmap was actually restored. + */ +int hugetlb_vmemmap_restore_folios(const struct hstate *h, + struct list_head *folio_list, + unsigned long *restored) +{ + unsigned long num_restored; + struct folio *folio; + int ret = 0, t_ret; + + num_restored = 0; + list_for_each_entry(folio, folio_list, lru) { + if (folio_test_hugetlb_vmemmap_optimized(folio)) { + t_ret = hugetlb_vmemmap_restore(h, &folio->page); + if (t_ret) + ret = t_ret; + else + num_restored++; + } + } + + if (*restored) + *restored = num_restored; + return ret; +} + /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) { diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 036494e040ca..c8c9125225de 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -12,6 +12,8 @@ #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); +int hugetlb_vmemmap_restore_folios(const struct hstate *h, + struct list_head *folio_list, unsigned long *restored); void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list); @@ -44,6 +46,15 @@ static inline int hugetlb_vmemmap_restore(const struct hstate *h, struct page *h return 0; } +static inline int hugetlb_vmemmap_restore_folios(const struct hstate *h, + struct list_head *folio_list, + unsigned long *restored) +{ + if (restored) + *restored = 0; + return 0; +} + static inline void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) { } From patchwork Fri Sep 15 22:15:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13387790 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6E96CD37AE for ; Fri, 15 Sep 2023 22:16:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 00EE46B0398; Fri, 15 Sep 2023 18:16:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EB0A16B039B; Fri, 15 Sep 2023 18:16:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C40886B039D; Fri, 15 Sep 2023 18:16:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id AE2E66B0398 for ; Fri, 15 Sep 2023 18:16:53 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 806EDA07F5 for ; Fri, 15 Sep 2023 22:16:53 +0000 (UTC) X-FDA: 81240242706.07.895202E Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf09.hostedemail.com (Postfix) with ESMTP id 34CEA140022 for ; Fri, 15 Sep 2023 22:16:49 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=yKe4AgVA; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=nzqn3FtN; spf=pass (imf09.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694816210; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qMEVjuvfkFjYvAgifMfI1ykIObOzu1AqZltnchTcTjQ=; b=ibnG1qNhi6wQjTkxysOJVBwU9UH16t6pHwyiE6bwv1JA4/kMCF8tJrK5bZCihs7xthzdDx +ZcPcHOlVjDojrtqqR3yVRIrj5cWuDNVkjrGhgxx+ZKvl/ekriaGW1GV8fVF0BIDeLMml+ OiDIM5Z098Sf6fjzRuLO7+t02PqFZmw= ARC-Authentication-Results: i=2; imf09.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=yKe4AgVA; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=nzqn3FtN; spf=pass (imf09.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1694816210; a=rsa-sha256; cv=pass; b=cVOtlTjBhDDggELhBarG7U9V+9yJ1eksC+L7cEUbRBi0eaTDT5rIi7uzMr4PrfInM20X6V gv3NeMcM8SFwP/sSNWGGDETlKoCkR+ojh6eVbxXAJ+W/VNft+7NNzZ6Rm5qGQ1OXZ4YaB6 hotRFfSSx+/HTNdiuqoEswdZ0ZqjApA= Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38FLx9pG007419; Fri, 15 Sep 2023 22:16:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=qMEVjuvfkFjYvAgifMfI1ykIObOzu1AqZltnchTcTjQ=; b=yKe4AgVAeecCRxLkm8a0yu3K56ssTMsX/b1DldJY/PwJhQATjWxCktUBGqDy7YzjJljg A1nHI1mFvxrWHAeMn4tEaA4rQ323e7Pwi9K8rXgCdq1IkbqHnc3x/T/LO/ZcnbXdaSUF 1JGRa+9C5cpeHRIcifcVAjaT7wcrj8mdngjo7CX2nSnJiJ58H9VEoqdh76l5jFJza130 mPdHdUBFSje8TnFaijpWEjE4A4z3/ghspwQq+hiEN61ph+6bUzJtKcQh9L81KmpHrDUf /EvEgdJgweBsY7l7aClpjGS8GXgZVzGZ+j8s4+9mlRfNsxnD5Lcu7ThTXTWSRJen4U8m /w== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t2y7hgsw5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:23 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38FLT2W5030436; Fri, 15 Sep 2023 22:16:22 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2174.outbound.protection.outlook.com [104.47.59.174]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3t0wkky7bg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:22 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XIjsB/ARC1KioLR0Lnr9fyYS8Ue+SXOCc6LxSeEqHviOsKSN4w8b0s2To9KHTxBzG8idflUTCVsZO1c5zCAU8QhH8LJ/gmOsvZh58Osj3zbct9aHi1bygXRX8I13AKtwv1DvCnJy1eLm5tuxgt2j0HU9uP2lgmDL2i96ALfoJtSK7FdSHQCKXVmqzpz0oZCo8ccyQNQNQfte2uEQHs/RDOG/McKD2aiJpqPy3iRhYeiOdwM350UEAuI5ZJqWquyXjgadPVQkVUlJ8xIHmp7X0Jj199N7JzmtnxauZox7G3UwuyluTuVvg+eYhnDokHG+bMRmzQ4DbmHNiE2gct+/kA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qMEVjuvfkFjYvAgifMfI1ykIObOzu1AqZltnchTcTjQ=; b=kup2QUgVPEHlI45bkXqLlHZbB3Df+Kullt66f8pDFCHKUqu2A2HemaKA5a1hnhCLMI2VkNiySo/BFSvnSLHFzSes1+yZnlQA50tH3D3UbnKO1vwvNn58BvRo+ZBK5ntdWiltj2CXAWgWUQniQFmIaWvBCp6nuvZ+N8KSjRjW/k9p7q2iTSgRTnTAW4q9CbLFFgKsBhy/YDKed7uA0E54Q7BMqcj8I6YvBzIfQsdGT6MH71rcrTOuo50qRS++FkkMfzijl1VSR2q3c9du4L/XD3MX0BFDcFl1WexYAuPpizTrnBQaezJ2miZiQKChpUN8DbQQ/VMeN/Xqfc6gkOUPSw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qMEVjuvfkFjYvAgifMfI1ykIObOzu1AqZltnchTcTjQ=; b=nzqn3FtNt88Av3X4XlmOd5b5YEuPibh/F1MOkad1ze9uMsEx6V+wlkk9VK1ZWI0ZuC1mfEG3r+2Brr7R8wZfMQcnP+7oGyCDqXgHBbNfvgs0XBrCQm8+fKhC27Sj0EHAftQYgK88dCE3eofN3eRtap/dKivQ1/i+sR5XMjpE0bE= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB5861.namprd10.prod.outlook.com (2603:10b6:303:180::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.34; Fri, 15 Sep 2023 22:16:17 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.021; Fri, 15 Sep 2023 22:16:17 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v3 09/12] hugetlb: batch freeing of vmemmap pages Date: Fri, 15 Sep 2023 15:15:42 -0700 Message-ID: <20230915221548.552084-10-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230915221548.552084-1-mike.kravetz@oracle.com> References: <20230915221548.552084-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0063.namprd03.prod.outlook.com (2603:10b6:303:b6::8) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB5861:EE_ X-MS-Office365-Filtering-Correlation-Id: 1e2c3b76-2789-44b8-2034-08dbb639618e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: oSU7QTzK/v4SvSuOoilAkoV80j8viuRa1HfM1/stxBO8fGuCmRW0IYxHuYf7bIhhit9/KgTQsrHu2tb3RJ0z9HI3UpiUsbwWCtY3TXMrH+G7gmqV6QGLIISFhIDHDf/luEXtHsGuDebhmtbtoEFq8u5a1ljUMxVCHUD3OYqRtYAXTJGKiRvW3d+vMIIbCrWxENQJvs0XftujCWAYCdb4d1C27vURfOGt00+hlsSRCtHpma6OpxnaGFaHV30sNWtvbEjRDFT4h7jNr1zXTESZxQSjwxhGA5z6lVKuG1YJUV5Ln3C8dXZUaf7tLu+Ms0uJeVI66WHwSzkRJt5Fq/V+Z1wQJLx62hiM+oCiG25HzcsNNpbcpipTIEBbhH8FJ2HHRnCRM6ouhJ19SucbJ3OsodXV9OamexJKGdzdRJXfasusScg/V734XPQlcg4r5MiE0jL7Y0ywhqg4WvJWe1ETTZ8ZD5e3WV9w6CosI9bWmtP1l9QrCfMs5dyPRAxznmexh+vUQU5LKDJBR35d1vs5hwcXKQqdhDXj/YwzrM2lrnyU4mxXcjBKjYG+pwvXs3g6 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(346002)(39860400002)(366004)(376002)(136003)(451199024)(186009)(1800799009)(44832011)(86362001)(36756003)(83380400001)(2906002)(38100700002)(478600001)(1076003)(2616005)(26005)(107886003)(4326008)(6512007)(41300700001)(5660300002)(6506007)(6486002)(8676002)(6666004)(66556008)(316002)(66946007)(8936002)(7416002)(66476007)(54906003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: iFrVz4s6v4uIN5jTg6lJouiLySvuAWKJ+Rlt7377C+D469XIA0hFR4LJ6Gg9WLtnEnXU854ZFbQKWNOwLcaLCmPXr2zi0I5PeMU2NqXt+S9BY8DSOuWGv1weNFU/fH61ERr4glCFPZBvJfwO6XbBt+TQc6Wj35ARC0atva2A1PU9D8v/tTDm5/QJGGP0EZVM+odr4NG6kiSIk/nlQe5zVGCV0HHcA/PaEzWg+txLznct2iET7l3fS3xKC2Wxe2Pdba/Nsmov4slkUXs3b3BSqMJ2lnZ7+c2YW+8etaB9dJGcV8PnbOA/QZFR3ZaOjdf0l1c0ODjaz1H6+51OBMQfQeMLvIBA3gPbY2hGUhiVUfFraMe4hKbqduttUBnA0UnNK6Bv1GSVRZcRb0j2AQcDEie792kuacY4E7Tk2B/MghaN7PX4q/3WgR8LXKZUU4nQkzkpzBdgZnMfNIN/yopIVSWBltSNVyEj2PaJZIQHC52lko4eUsFQs0UUicvhSfr0229zzXtgOzrINptg11iPVB1Y5Rps1NBchThJomiDK5Oqk7q3D2qlwbTITBLP/8cN8YtAGngx+DgvRsXeOeq5s8wfuIr3Kuo1Dt+5Wc2uR7+dqYqEttXfFi+9KgKnkgm4/vrSQ+HnFUY6EALIb4arMU8Bzd8qy9sb/cGEB23IQzo70aoQUmf5N/+5UaAGbDGh5APSZsPEM543+WU3l2njXYYFoG5IuypY/w7wzqTszJJVTVLp0J5ahkSvcplx+yzCjqi+6af5i5NYVqXHVGaHqa00W1cSEa62pw6oEZMpCInV9itNxfxjAjoJCrbp+BLD56g/THqCFBA8UTbuu0C18yDFEpu4Z6JD/T8Myet6Fl33x0nTsHMWlGdZX/ABvwG6spnXWc1ZPnWvxrEVyaRv5UfDTKhxcfY1Y8M4AB1TwgZrEpbgohBffTgALh119UV44cm36f23GyzX1qYzOoWaS48i4r3MpdGQtJdTk5XZpKt+LbUuEsmTMRZvzJvEa0Ch/2QaJFe4V3r6K69zH2SYQxl1rxiG4fiiZsZqFax9xhpJRLX2AdmWgz4lhwI2kPVSo+R9WrOfU96cPnPSf3b63uZ/1+S74Jcl893ll1YCEZAZDvXy04V0yLayZZyUAsZFVo/SpgVonkT2CCbOokgDWlnLvuplr5lKWRD2xL61DEjmXtlSD9OTKmiVm+HJ10j5+cstbiJF1zjHV33NdYPxI/i9e3/YDwMHTDofff+0FqsIfctfc5pN384gsEXkEAk3vZ7xDyvfSc9Np49hIBlk5eM0ldNx15caoGdkCFQH0OFVXW3a1dgbpIsUCnFKRHAh2Izqococ4j3miZO6gPJT3Moo91IXutx5npDypDjm7OthcLVoASIMIZRvj2IRn2CarVMJCn7aoZriH0czPrJkjRWNiq/B2iykApxZpKG8yR/i5XYsCe+zbft/wukekWXWFmsOfhJMSubu8bjGL16N6PziPOPoUNSP2i/kDIfZzgFCIbpYlQezQbHPhYAeS3X7dRRLh5aMPZaRwQ3m0K4tbgHW7ryzLbo7LQxkOW6ZfSFbrt2W2+gLTUhdpQwiuN9LTwI4gNZi+ZT0Qfikb8xPNA== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: ps0r0JVQXB4tlo465QFf+bb24uD88888b/BhSFqwyj/RPBKEiF3KM5F6yKAtvZdNPPEPmR82LCVPsFyFwW1ZLBW/JXPOpw4Oi/yjUcHWaPdDf2ZyEfhlUaVK2ZUycvdijipt8TBqyuTVWC3iFXhUFgO/bja9xnvX5bXBZVr6NoLmeUD/yZXjxvBT1GNNIiWkIxE6YCgZjhi2yehnM8FdelsCOWvw0OWwEoOsOixx+Bk7fDou+xW+rhdPVrtwQ2bTLl6ZsSZzJB70IfPfJmf0m1TnGN8TRUcA8oU+hBw/52ow1TA5M22iomzJJKi2CORtEj2/6iWUBCokcnDzL1c+k9l7SbjtheARVTmrXU7CRVQFYmo0mMLSkSb5lldTsSXNdK+da6hMylDjVCoekIkB8p/TB7k7GkjO62IiGicVyz2EtG0nY1CjkhFTV1nr9esTVGl5qbpN5PEHxAanx7ON2NtjCMJ9yjWNBjR0uhP/PZiD2BMbqVgncjCMJgHpE8ZMn38lJTGVsEpqmrOXFwCfv5fOgrQJrdvgKIj8Sy1SXzRyTQk4Tf5U7LdNWH3i8w13jU5UgnvNJAQ1GuWeLYt5lphzYIJCIK4IsBXzrQ6wKXEc7oAeeoB686aMWliXjx7RmjnCinIgpfpB8BdFsUFk02rp3TYQ2sfc3CmQsjF9xaoflaxoUwOlatgrP+IvumVYsh38qAV1L+GDNsyFFpz/XLnJR/zb9Hlw9Dnu/De5j+57YxGXoBy+peCH5KhgR/2Kw2OtmxPGSsuhWpZl3gdvK+OLl6tGxnGJQYCUJ1Ht/E2Q8Q/tDKvOvBtwZzgvv/GMY4lMK2ASrLZwPfuCJnLSNYT8n/ipEUTuWwHZbdESL7uhbExEgLI3jTSuIZPP5NdJECH5ZQWSVjK0zxzd72Ls/GYzlJ2rfpry9x7qMjhGSJM9AkSpKzEZGHa7Iq4ViWxUsnq3m3ul+AdZu47rUQapOOLoe8S3gijXo82+G2gyjNeDIM/kbFUmKIz+IlEFZehUgVnfvwt5v7s5Mf+8H82r2gyCYU+OYwZ2xnXe+CFdvSKOxnEwF+d2CwsySd4VE88nk6TAB/+dGhUrQLEl2qRstzZKuwfSSfAtUUP3NtEjkKgfx6HeGFEPQM2JZE5fM4pn X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1e2c3b76-2789-44b8-2034-08dbb639618e X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2023 22:16:17.0488 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: f+qqFJ6mvOfgJ/Vf6y2CkI3FG7KFafnmZNedrdQMI/XYzIq9JeJzWIWp0vi1ocVzWfSpvwj7YKfqz3DAgkksOw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB5861 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-15_19,2023-09-15_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 adultscore=0 phishscore=0 spamscore=0 mlxlogscore=999 bulkscore=0 suspectscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309150200 X-Proofpoint-ORIG-GUID: GBV07K0Cu-JuI4Xj5UkNQK0it1kmM2rJ X-Proofpoint-GUID: GBV07K0Cu-JuI4Xj5UkNQK0it1kmM2rJ X-Rspamd-Queue-Id: 34CEA140022 X-Rspam-User: X-Stat-Signature: yz7eh1cz3udakt5f4hk8cw67w83t5mht X-Rspamd-Server: rspam01 X-HE-Tag: 1694816209-244608 X-HE-Meta: U2FsdGVkX19snEn+qGlm/f2d34C/q++G4yTaV5ecvfcy1AWKwP+v24TSZT065+NtZySM6sIEfpwwjILEj2tTarhY6Az3iz9iOZikGAMKWjhKfTJl8Umflcj4qpFaiMJ3kCt5upcXfrWqIZK5Ug4k3n0oyUQ6g7uqFs672IIH4tbdWOI/kx7SMn+BIq16owcpHwejCmDVSpCOqx4K1ENEiUGHpsLN4VCxYC93QMsjYcDuskaVUsOOvWdkib9yTo2lp7hsGRfXlzHzCb4xPS5vthqXG6FHs0uL0abE6iV2ERZKR2ICsqr775Q/r4LCEwFXZK6YQxEUQe21dupw9nU5IjpjivgZ1wbBXE4livPXs3dMagVWLxR5R5xVt3ryfLJBVEqQkF1VNVer5f51vpXaFAiVLBWR0vWDGzTDqBfomjwo18gxzbidbp5y2xjjFIFaic/t6zrVIJK9aKcHmStBHPtjlBSyA+OhAvzGQVVNlyMuejDF563kr/Wo76hrJmBfgNvW2pRuIYtfmgKSLtwoZPHGC/OHp1dJN4EozeMxHq2HIA90ncb2r1LS5UdNyXXumHcYk9kpNlaq2Y2dUIG8X+nrJzGa6tF3tKk8o6liDWS0VMC1QWEbrAF5x8A3ammCFr3NjTFjf+7S+SvBPiGpUJ8P7G3mK0dMiV0+eKviPZseJufU3F80k3vpJyX7DqtiJ2BSx6tNZM51T2aabzMMrK2zKc9rAsfhGkBENrmNY7M9dl6WT09G+cSK2bD5WOrErLYAlnbO2ZAszDqQ6+X8kNIq0N+Fmnq0pMHr2EjoeM3taysHtI3ici2l0Idxqztqn99n8YDUsCElrGUbhiyEc/Y5fAmvtlsEUE7JOLwowi+9V1FZ6Zx3qYx2ZulPlgZba/2pnbf6b3dvwMZSEcuI6pa4RqWZtu3WIzxANkuRVRvnE5fWlukpMhkDT2a3U1dbc0VqgjvebkpHMwIuxy1 AptlA5Ap n+ygeuEVL3yV74IQVjQIq72hWfu8jjXzFQWrmhVnQOek+UhkDiupAy7zjSQMAUbKW5BC6+MHKathPWcfzcqwRkPbv+6xBm/Dvkq2hPmDl5Ic/FqY4LzulTsqqxxH2CUPMoHobuUScNhSGZCH7ipYuvHy8q2ewQICvje8SEBwu3yqY6RoVyukyeg55wWwPYMkk+pr4fDESGUl30jC5vb0HjV7bNdpz01pyt/QT4fFl+M0AF2vxEAAaHcs0293roNAkuGlsYDU5REqXsYPYTs7hFZKSwaWPdHx1VaetQy7goCv8nQGt+9vJmGcT8NG9OCjit4HejoAfwTZuEhCAEN2hik79us89jv2eJTt76dEDDSiWfA9lZieAj4tAosvHwgrzsQns7FFDuqTolVo0R0LpIFEPc+hwcS9+hwT8eEABop3PTcSw8on4zVuQhqnJQurTMSniDKtRvqDYDG4r5RQD1q3TJLXVceTIVqyo5R91HUKl6cx7VXgh2rNzuX7XmsYYEHJ4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that batching of hugetlb vmemmap optimization processing is possible, batch the freeing of vmemmap pages. When freeing vmemmap pages for a hugetlb page, we add them to a list that is freed after the entire batch has been processed. This enhances the ability to return contiguous ranges of memory to the low level allocators. Signed-off-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 85 ++++++++++++++++++++++++++++++-------------- 1 file changed, 59 insertions(+), 26 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 0eeb503d8a4c..8f8a559ff6ac 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -222,6 +222,9 @@ static void free_vmemmap_page_list(struct list_head *list) { struct page *page, *next; + if (list_empty(list)) + return; + list_for_each_entry_safe(page, next, list, lru) free_vmemmap_page(page); } @@ -251,7 +254,7 @@ static void vmemmap_remap_pte(pte_t *pte, unsigned long addr, } entry = mk_pte(walk->reuse_page, pgprot); - list_add_tail(&page->lru, walk->vmemmap_pages); + list_add(&page->lru, walk->vmemmap_pages); set_pte_at(&init_mm, addr, pte, entry); } @@ -306,18 +309,20 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, * @end: end address of the vmemmap virtual address range that we want to * remap. * @reuse: reuse address. + * @vmemmap_pages: list to deposit vmemmap pages to be freed. It is callers + * responsibility to free pages. * * Return: %0 on success, negative error code otherwise. */ static int vmemmap_remap_free(unsigned long start, unsigned long end, - unsigned long reuse) + unsigned long reuse, + struct list_head *vmemmap_pages) { int ret; - LIST_HEAD(vmemmap_pages); struct vmemmap_remap_walk walk = { .remap_pte = vmemmap_remap_pte, .reuse_addr = reuse, - .vmemmap_pages = &vmemmap_pages, + .vmemmap_pages = vmemmap_pages, }; int nid = page_to_nid((struct page *)start); gfp_t gfp_mask = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN; @@ -334,7 +339,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, if (walk.reuse_page) { copy_page(page_to_virt(walk.reuse_page), (void *)walk.reuse_addr); - list_add(&walk.reuse_page->lru, &vmemmap_pages); + list_add(&walk.reuse_page->lru, vmemmap_pages); } /* @@ -365,15 +370,13 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, walk = (struct vmemmap_remap_walk) { .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, - .vmemmap_pages = &vmemmap_pages, + .vmemmap_pages = vmemmap_pages, }; vmemmap_remap_range(reuse, end, &walk); } mmap_read_unlock(&init_mm); - free_vmemmap_page_list(&vmemmap_pages); - return ret; } @@ -389,7 +392,7 @@ static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, page = alloc_pages_node(nid, gfp_mask, 0); if (!page) goto out; - list_add_tail(&page->lru, list); + list_add(&page->lru, list); } return 0; @@ -576,24 +579,17 @@ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *h return true; } -/** - * hugetlb_vmemmap_optimize - optimize @head page's vmemmap pages. - * @h: struct hstate. - * @head: the head page whose vmemmap pages will be optimized. - * - * This function only tries to optimize @head's vmemmap pages and does not - * guarantee that the optimization will succeed after it returns. The caller - * can use HPageVmemmapOptimized(@head) to detect if @head's vmemmap pages - * have been optimized. - */ -void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) +static int __hugetlb_vmemmap_optimize(const struct hstate *h, + struct page *head, + struct list_head *vmemmap_pages) { + int ret = 0; unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; unsigned long vmemmap_reuse; VM_WARN_ON_ONCE(!PageHuge(head)); if (!vmemmap_should_optimize(h, head)) - return; + return ret; static_branch_inc(&hugetlb_optimize_vmemmap_key); @@ -603,21 +599,58 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) /* * Remap the vmemmap virtual address range [@vmemmap_start, @vmemmap_end) - * to the page which @vmemmap_reuse is mapped to, then free the pages - * which the range [@vmemmap_start, @vmemmap_end] is mapped to. + * to the page which @vmemmap_reuse is mapped to. Add pages previously + * mapping the range to vmemmap_pages list so that they can be freed by + * the caller. */ - if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse)) + ret = vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, vmemmap_pages); + if (ret) static_branch_dec(&hugetlb_optimize_vmemmap_key); else SetHPageVmemmapOptimized(head); + + return ret; +} + +/** + * hugetlb_vmemmap_optimize - optimize @head page's vmemmap pages. + * @h: struct hstate. + * @head: the head page whose vmemmap pages will be optimized. + * + * This function only tries to optimize @head's vmemmap pages and does not + * guarantee that the optimization will succeed after it returns. The caller + * can use HPageVmemmapOptimized(@head) to detect if @head's vmemmap pages + * have been optimized. + */ +void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) +{ + LIST_HEAD(vmemmap_pages); + + __hugetlb_vmemmap_optimize(h, head, &vmemmap_pages); + free_vmemmap_page_list(&vmemmap_pages); } void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) { struct folio *folio; + LIST_HEAD(vmemmap_pages); - list_for_each_entry(folio, folio_list, lru) - hugetlb_vmemmap_optimize(h, &folio->page); + list_for_each_entry(folio, folio_list, lru) { + int ret = __hugetlb_vmemmap_optimize(h, &folio->page, + &vmemmap_pages); + + /* + * Pages may have been accumulated, thus free what we have + * and try again. + */ + if (ret == -ENOMEM) { + free_vmemmap_page_list(&vmemmap_pages); + INIT_LIST_HEAD(&vmemmap_pages); + __hugetlb_vmemmap_optimize(h, &folio->page, &vmemmap_pages); + } + } + + free_vmemmap_page_list(&vmemmap_pages); } static struct ctl_table hugetlb_vmemmap_sysctls[] = { From patchwork Fri Sep 15 22:15:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13387789 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17D9BCD37AA for ; Fri, 15 Sep 2023 22:16:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A21086B0396; Fri, 15 Sep 2023 18:16:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9A75B6B0397; Fri, 15 Sep 2023 18:16:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7852B6B0398; Fri, 15 Sep 2023 18:16:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 638146B0396 for ; Fri, 15 Sep 2023 18:16:52 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3AF1A41054 for ; Fri, 15 Sep 2023 22:16:52 +0000 (UTC) X-FDA: 81240242664.17.E6DA715 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf06.hostedemail.com (Postfix) with ESMTP id 3A0C518002D for ; Fri, 15 Sep 2023 22:16:48 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=A7BjxVX8; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=hekVPJiq; spf=pass (imf06.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694816208; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=i7fwZVFpbWQwFa0ZuX0bGyigYcZrLBRp0FddpocwtE0=; b=GGx33+VXrX99vSSjZWBoQUbOVg9MEdHHqTxulGidARSEDZrOeZ17kN4PsZ3fcfjtJoTOR8 eTe1z4A7GE/hviSSn1EmxsOH4d6DrmdwinXu7ZdUpQr+WOebiHRMMl9FuKitaWvVITYbSY IUU1nYBOYsCXH5Cq8wdr6mY8OuB6gUE= ARC-Authentication-Results: i=2; imf06.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=A7BjxVX8; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=hekVPJiq; spf=pass (imf06.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1694816208; a=rsa-sha256; cv=pass; b=JTB68yiZ7+qA37mRj484mSo0yC9O+zpa004vDEiJvCnfVWSdVFXQgXmMjUNULeMVWIQfn1 Xb8gWSZjUsLGVf9gsR4n0Lsv74JvfNcbvvFoKkNbkTPGyw3wvuOYFjSMILbyhyHy9qNcCK rO/IGX1rjlx0Avb9uenbAqegWxz7Nq8= Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38FLxYgm007900; Fri, 15 Sep 2023 22:16:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=i7fwZVFpbWQwFa0ZuX0bGyigYcZrLBRp0FddpocwtE0=; b=A7BjxVX84mtGamQPYAdWr/fSLgiivgO9lgcMX3tj7vM+XWzaMwSackgQMX5h8NgHAipk Wc6Mnq4kGFIIXetCANWRqJ40qW6/h0WpvQmqKdPfudls+71INbjmGYLQD/RdPyQ9qtVG 6dxxledYOa9WnD02QPhn5D7s49PkNqoCQvaQtIYJurKrFlNdnRrZuE7jNTns/2+Q24sl uq6u5xwgrqvlfxy1uHFDdrGRrlh+eI4I5EllboN+E830wqSQ2smCipGT1l11x3jBvqMC HyB/aVvDBEdyexP5xcoSsi9X5XF4OBmWZ4LGmuoJv2jofcx871pfIZEplKmTNNSjHau0 NA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t2y7hgsw7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:23 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38FLT2W6030436; Fri, 15 Sep 2023 22:16:23 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2174.outbound.protection.outlook.com [104.47.59.174]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3t0wkky7bg-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:23 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DoAJ7T9CcPyCNGrhpHLb5UQgyUonfGeo9Sj1CDdsQxK7iDzywDFjknRvHrkTg+UGAQQIdQ985wryxriioTOU89F3tIueiWcxtbfTyTa4fSW6xkvC2cnU7yzrNzs9V0QfxIC4kfS9w142qerkHKNGJw3rDy/cYh2db3byYStmxIUYt1xq/zAAUx9pTbn719+mJi+YMPMusZhJB/fsjgQHHVAE7TLoHmf/aZT0+PvzXp3cfsjshQt9hFRuT16VzvqbLS6gM1a+kc4CEvmF1cH+tQgw5OyF0BJkCp2nlXXU82kgvJTocZ2a/huhFrY05KVXWho6667O3xDETTww0uM9aw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=i7fwZVFpbWQwFa0ZuX0bGyigYcZrLBRp0FddpocwtE0=; b=BnzzgRRJMfe2HYtAphmEhWidmSGhwDSKYDxVE2Jb5C4wMRDZ1I9AFStG+Qwyu+aCf8mJxxVxwFU0Y7ZT8f4M0/SGC30PU7Rgd2q+11xRrVUe9DqkAY2Oz8o4gz7N9MOlquSkPjQJOKdccLdJbclR5cc8gSvCIa05dPDiGNV9VtmXBwQBWfg3jUc+Q0lE1uP6C5LWRS9MwQyDLu7gHtNJjRPL2l+P6rNtf9iplncfxCZ4rfpct1nkW9LJ85EZiTrv8Haca7GVNUMjdJwQm8RvNGOmjN57g2DRn3O1noCNSX33p3A6ahurZoduHmUJATCWHNtkPMTK6e77OEh1qPbsZg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=i7fwZVFpbWQwFa0ZuX0bGyigYcZrLBRp0FddpocwtE0=; b=hekVPJiq3PNVpaYka1ea+OkS/95a+PRwnDEj11aAVso5UyVV4mHvWRyl4IEDxyavSEIZAKaiSHax32kcDfYyxFC76BR5gu+1MrkQ14K1NwjP+20m1JU5FyqJyaIqUXC33C9owIONZlO0TFAmsoH+NOHVa2byqmw2Pgcve3cPFzg= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB5861.namprd10.prod.outlook.com (2603:10b6:303:180::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.34; Fri, 15 Sep 2023 22:16:20 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.021; Fri, 15 Sep 2023 22:16:20 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v3 10/12] hugetlb: batch PMD split for bulk vmemmap dedup Date: Fri, 15 Sep 2023 15:15:43 -0700 Message-ID: <20230915221548.552084-11-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230915221548.552084-1-mike.kravetz@oracle.com> References: <20230915221548.552084-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0015.namprd03.prod.outlook.com (2603:10b6:303:8f::20) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB5861:EE_ X-MS-Office365-Filtering-Correlation-Id: 83779850-a1f4-4c4c-577a-08dbb6396361 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4FYJdwDnKtduyUk9VDhYRSzya2gG/SN7La5w3Nwmq+l0ITUDSkKsrFRBUYcTM9TWk5hi0LBnx1CKzEwS95fpumZvJZzIp84YhP1lgJSsh+TZmVYpMFOkgwEEaJr7txEtA9PPbtgT66EiEMfLI/jvM/Sq3641aWakGYfetkA2yseR+1aQK7rHUXgGIBdBwLUOY4iXDal/d653okPLyR8crhcEGI8aZYKIxPFiOBNHfYuk72jpzvnzH8c8avd9rku7hvW0ntUm1uuzRlURnxyGOFFk0okPOKt9h1B2Gon8+jcig6+EjWhPzYqaZU/bVulyyjArtmQaJIHT1xp2m/SCSWQqS9zkcVnjPFJq4iazqROe5fFsfGNATGtNrEMPNV/HqPAzYQZdcbZV2UmG74Y9K2tt+yMMAxJOmeZU7glBUMChviRLG1QJLBiVtPLthtg9ipHY1SRv23VwafUEZ0TYK6B7uYTaYZ9IpoGVhPMc2aXs0bZpV+DPq/8OJysC4FyLlTj2MPrav02aFVx1yeuevm7tWyptJ5A4vd8L/GOXaWq8iq44UXGMGp2jl300NlLp X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(346002)(39860400002)(366004)(376002)(136003)(451199024)(186009)(1800799009)(44832011)(86362001)(36756003)(83380400001)(2906002)(38100700002)(478600001)(1076003)(2616005)(26005)(107886003)(4326008)(6512007)(41300700001)(5660300002)(6506007)(6486002)(8676002)(6666004)(66556008)(316002)(66946007)(8936002)(7416002)(66476007)(54906003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: //J4X2tl3RBtZ4X5hGZ56j9TFYAm2xMwRlkCea5yrQwPP4mHsRr8whc+OjhifNeEHrD5DBc5Xm0nVctYyP4ynCIOxzUMkcKb4J1Dcb3VwCFXG25/lP/tP2VF0ckSMsnjjB/m1aB078l8vBP9270RnU0tALo/EV8ygV4mrjfLEPVe+68cz68lnOStd0CF17kLOcHReTjGPm2G+mnjZexNRr9QnYvw6Jg4tzVuE5VK7rxQf5TTOLPMazCcIDTbuc5SnSsqpqZINPCAsEVPV6+GIP4yqPYXs+R64JvQREA3+89iHbtCyzf5XnBbqd9f4faQ6ckTayesqaqOkUCAVXClbAkAqQgQpNQBIGkCbp+AkIS5lj+u7vpuj0d+uVEV1Sn4s5pt1gu/gw/7JE2U+/d1gTuy1A7zKOI2bpS77daq0tDcOJTfzm/jyyjAbdoCRSltkKEvvwrJjtSYustJUnp3rB2y6NDl143wdiWhtIRijrDaQdL5KwNC6nAH1SS7tpeu0tqtY9iX2uDoz6UP9nhwl+NuFNoeXj+gCqDUyE431CvssRHc93Lpwu8H1iRtT1Ss1PAgJO+Q2eEqp0bX4gSOQbWlKrsb2IGSXSZLAGdoApJw7jBSZuzVZ/ZBhZWggcgxuRGRNJUrQbt6r/lxnQIQM21lCUOYgt2XBSbnVy4jKJK5J5cDsZCDEOrDTXjBhvAMaEs8Je9ZjVz9UXFwtv2yWTNZWJlCEru3OK6XEAYCJZoP2UBRZlFQiYnu0A6q453ix+/2lYgUTtSXLWi1AmlHN0LumUnSzxV8cIvcrCmScZ9lKAgnJMisgGex39IubGW5no+uIsDljhDMleieqQxHGHq/2SbD1/eDxYgxqcs+btT8LnXv1ZpHcV3p+Jy/mPT1HJpnS5fJ1r6jDhxp59yvXNc6Z/RIs2ui+dRjYo/GX5BBJXfG1rr+zoWiO8rYcjHNO6rYE+DjXNIYqwl2cpHPWcLjSmONzgTTRW/1Z53f8U6ZvFPvAu5QIxIktRraz9m3ZL6fRooTiatO7/33CeZGJApIyoPJBgrlZzqJIIMftyCkO4bNrCw1QNp8X+jfhFQOvTZOe+fhh7EDjhTjS6amjQ3728WYzdiGETgeZjrPt+jGf6L1PsSDhhXPSr0ZukWa+UbBxWe2i+L+nsZgx7MKAL2/ufBl4M5cXh1BAUx2eIwfKrAlSUMIyj06asenX89jYGBnZBX4onPLfWEFqRup61hHamSV+FlYrXAIZ5ioAuhGALZDgr7jrFDTvP1is//fLvHHIs5acDVWHsf5GmsFkPE/hfkEHrt7Iwzb8/EvRKBC077Gi4/6IQVqMRCsrWOsvDDTjg6KeV4kBBgKHHmHSK8Yv+xbH/GDb6UBEj8JFZ4YYdTWWLBB/3jGJbDgSV5TeJHzMKVOe0u5Sa+3UcKNrUAzVyfNedtGQINOh/A6QxUX9RiUI2NL1j4Q/wL84ndC2utqNpCkbzc0R9IEsWA72DlJG2npXU08u9zikN8omKOsJNvIvi/+zNGNweR3k1hrYSsyupQ17n8T4mkVxQdefdlTRVMAnSRWctfuDJ4NT7R4UMGjZcwPMWHhMAt2fJJ/TZ+ah1HLlL6ZJhMgXHI8iw== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: pjTQymCEGuf2VPI3AuQgZtjpfeOhmdbpvTlGPQq/2a1aU0m5FwH/L0/RuPBZxQBIchVrDxNF9iX1Y39GFgQw7JJ947opYwVYfutyx2nRtUrFUAFmQlLYi/nTR7rym3Z3H1J7xVWXqqPIU6eepZ0Am/e4n/X+F8cNQtOfGSBMYYSYuVwXtPytj5thx1mxkie8xxRDYU2SbNkT1KBrA8opWbqxHxtaXll74xUuC7L9y7iaBslqlWONI/vRAwfC8mTJIsZ7GBaoBL7QQlbVGxjJFrVk2XvaKz1XMw/hwxAX1j+kKiW+cilhIIy5bMCauShWcZ75CBqu+wCII0w6PYAbH5az+368Y46ovQwLbRalaQAUOFtJTFnKvnsUaxaJyhNNGS+tENrv8gecktT5JAKN4tJ8GETzK60hIhETTgNyGzlO7gzlvJAPXCJK1chIieYtJCGmOIbMw6xUjsAq/9r8m/PE0Cb+5IplhvDV6ui5hRJHrISWl9VwoaeIYyhgOyS9c/cPWXEJncaNlJwIq3URhgbvq1vtr4wph8NlBh+CVRd/ujVm7x4uZax3frxE16tN6pIZ9cUpswHNrb/6hla1TqCYiLxjTHqdIyXsokXrx7SljwMcL8uh+eo092hPFruVieqt4V45aSzIxVtJBJJPy/poUa4dy6WSgGfbGy5rRhmP1E+zvUZp3soeKQFCmy236CA2WWFHq+dE8SQzLfxux69NdJaQ8w3ScZ/lpceaTPCUjgpL5H1IZPIp4YkT+Ora6Se7e2gilsJT4McFjItTqHNJrBLqVXGf5OdkUUNWmRXDSw1NiK1D6RZHieSFUm6jiWLzppjgxV4XZm51pF4OhIDWjaNTXBqIpq7kuJrJlv24QiT8GAXQfat7SeEKcnDwMjBxbXfs8Lv3zBxusJTwMcLgMlE7T636obMiGx/v9mN/PV3Rl9UeJ8jnj7oKNi1EV/tFtsz4DtMrvzmtPoXXeMEKp7LqS2XjsRWuSTPYfhInWlSerH5M+SEIEGnBgQzAbXS259U5afF2pdzV3GhG95mfHL16ZZQDyagiKwdSkhdTjK/qELN59lSBPA5O3VqhaeNmoqtYd39sjngjZW8hDjm7Z9OBD3u3eHiu7TpWf8IB5CIyY1EXtQgsidKFUC0U X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 83779850-a1f4-4c4c-577a-08dbb6396361 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2023 22:16:20.1263 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: qEQa972aHiADMUJ74b3mxrCowS/6u2UmJPGdchF7RHLsEpjugpMMJ/mitw6Izzl7CjqhHFmpOxc4QLcswcYXXg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB5861 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-15_19,2023-09-15_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 adultscore=0 phishscore=0 spamscore=0 mlxlogscore=999 bulkscore=0 suspectscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309150200 X-Proofpoint-ORIG-GUID: KuDrkyMSSY7FmhcBC3uRDuUOa8b8d87G X-Proofpoint-GUID: KuDrkyMSSY7FmhcBC3uRDuUOa8b8d87G X-Rspamd-Queue-Id: 3A0C518002D X-Rspam-User: X-Stat-Signature: 9a1awhgsebukcqzq3zktcxp9di96i9d6 X-Rspamd-Server: rspam01 X-HE-Tag: 1694816208-72936 X-HE-Meta: U2FsdGVkX193xvsDZTe9ROKeKuTY6Gtxh886XcLbvd5NEz0tSLUHpRWgpz5+Dlqii1FquhPtoJyJ0kSfsACxlxgdq+Qi+z6E72NyYfGrkenDCscE03mn9dsKicD6//xOCvA17igzkYAmppmxzkbv/p9LR/k9a9JuYFBPzYErCs9ILR4fYRE3ASUr68uUH0NUN0ntbd6WM0L9JMZsEwGctZlhIMuYE1MQrFddM1eVF0cEjQkGlmyF5I9f7BkdcNW4v47QFhHW71dlZAsJZcNiX91AUKj6B8IvsZ62WBEg5ncLuCbDRBCh+JkIr0B7PZRGH1GfbOgXzkMDhSIHG1fRFmS8YL+Q5z1LCbDSkWVUwWjhOu6jad9bkqrmbtOiXLPV8qtVc4HtbX/LJol/cIIA9nS2czHr7DVYdPZfrGHis8/gIx+R0Z7zU684qqUqMVNaXZEr82dsieGhwlw20nb2OATtWdRPvKXhV+pTgUu/e6kTvv1j5DljsRj9+YInRvW5JxZM0gSI7HRtfOewRXBcd09RFeVq9CEV/hGtlmTU+VlAAzrhULC/kI69a9T7cvT0LYPUKSw/6XNrx3OBc9ZOdiJhiwe2ciESRpS28dNIxQz4UiBXWwLN7BIur7HYeHVJZUO0b8O9Qy3w6EJ2CYCC+/vMy7G8LT+zJ7HfMmxvF+8IaBqv4dR4tYdnOnjY26dp4Q4RS5N4ZHBds4KmIHecv0jjStgXTK0mtCIyPAGcFvM8XIMcXWsUFjrAw0OSsMtJYGpylIvb6rejD6SzOssIACQKhodWtydAazrS8d5bXxYCccxRIMIlX9e7Y66d+fQywKJyNjp7vzb24bWhoeInO3grMPpH3bxNCV9XSIw71IaFm3d9L76CEOaNjFrMKCDLC1zCLQUl0I4ykTzsmO+GH7IcqyWSK0yqDgTKuZaajAf2fwBjP9kQhcH+N1+cnTFltV1PoO1EmLCnh7tgJ9D bsrasKIy arftV7EGcoQqYvkIArXs7z1MaO4y++Nml1miRl9od2Qxjr5SCqqIVqGXOPYFtCbhVdAh5hiptiZXVYv9AS8jL3H/7e4C6HFSLAnoGOt+ceRVg7bcEI1rta1swqPxGHhZ4Jzrk5m9BlAC0ewQWB6VCZLCKySbgzd/oZqZoHoqU2mXQj9hRFdlQsQ+hYExw6NsEvh0Fi1bC4HttB6AoWVtH9jCrzsJKEtU3yTcrPgpQI5huobxqvX6Hn9elbtSKzW/EmK22xqfT2J7qaU6EScAU1ar7RINfmem0MKiOR0pTcZD7E0M73owab5H7g1ABDAiG100xK9sIDLLsnrsxMVBtR8VnS0NQK0IKobEKdm2regmqwwEyWrmM5wKvVG13R6U6oBPEBVFZ2kmbGdWbrPxRUH1fq5S9hmqrWVuAPb2nwARoCCdLy52WfozyN5sT2OKiIpARj5w53KGib4IB0HYbqxwUk7QyKFdKps/CZA2XHmBeQPxdcj5lbzGtEzXNIoXYIpqk X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Joao Martins In an effort to minimize amount of TLB flushes, batch all PMD splits belonging to a range of pages in order to perform only 1 (global) TLB flush. Add a flags field to the walker and pass whether it's a bulk allocation or just a single page to decide to remap. First value (VMEMMAP_SPLIT_NO_TLB_FLUSH) designates the request to not do the TLB flush when we split the PMD. Rebased and updated by Mike Kravetz Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 79 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 75 insertions(+), 4 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 8f8a559ff6ac..c952e95a829c 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -27,6 +27,7 @@ * @reuse_addr: the virtual address of the @reuse_page page. * @vmemmap_pages: the list head of the vmemmap pages that can be freed * or is mapped from. + * @flags: used to modify behavior in bulk operations */ struct vmemmap_remap_walk { void (*remap_pte)(pte_t *pte, unsigned long addr, @@ -35,9 +36,11 @@ struct vmemmap_remap_walk { struct page *reuse_page; unsigned long reuse_addr; struct list_head *vmemmap_pages; +#define VMEMMAP_SPLIT_NO_TLB_FLUSH BIT(0) + unsigned long flags; }; -static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start) +static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, bool flush) { pmd_t __pmd; int i; @@ -80,7 +83,8 @@ static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start) /* Make pte visible before pmd. See comment in pmd_install(). */ smp_wmb(); pmd_populate_kernel(&init_mm, pmd, pgtable); - flush_tlb_kernel_range(start, start + PMD_SIZE); + if (flush) + flush_tlb_kernel_range(start, start + PMD_SIZE); } else { pte_free_kernel(&init_mm, pgtable); } @@ -127,11 +131,20 @@ static int vmemmap_pmd_range(pud_t *pud, unsigned long addr, do { int ret; - ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK); + ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK, + walk->flags & VMEMMAP_SPLIT_NO_TLB_FLUSH); if (ret) return ret; next = pmd_addr_end(addr, end); + + /* + * We are only splitting, not remapping the hugetlb vmemmap + * pages. + */ + if (!walk->remap_pte) + continue; + vmemmap_pte_range(pmd, addr, next, walk); } while (pmd++, addr = next, addr != end); @@ -198,7 +211,8 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, return ret; } while (pgd++, addr = next, addr != end); - flush_tlb_kernel_range(start, end); + if (walk->remap_pte) + flush_tlb_kernel_range(start, end); return 0; } @@ -300,6 +314,36 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); } +/** + * vmemmap_remap_split - split the vmemmap virtual address range [@start, @end) + * backing PMDs of the directmap into PTEs + * @start: start address of the vmemmap virtual address range that we want + * to remap. + * @end: end address of the vmemmap virtual address range that we want to + * remap. + * @reuse: reuse address. + * + * Return: %0 on success, negative error code otherwise. + */ +static int vmemmap_remap_split(unsigned long start, unsigned long end, + unsigned long reuse) +{ + int ret; + struct vmemmap_remap_walk walk = { + .remap_pte = NULL, + .flags = VMEMMAP_SPLIT_NO_TLB_FLUSH, + }; + + /* See the comment in the vmemmap_remap_free(). */ + BUG_ON(start - reuse != PAGE_SIZE); + + mmap_read_lock(&init_mm); + ret = vmemmap_remap_range(reuse, end, &walk); + mmap_read_unlock(&init_mm); + + return ret; +} + /** * vmemmap_remap_free - remap the vmemmap virtual address range [@start, @end) * to the page which @reuse is mapped to, then free vmemmap @@ -323,6 +367,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .remap_pte = vmemmap_remap_pte, .reuse_addr = reuse, .vmemmap_pages = vmemmap_pages, + .flags = 0, }; int nid = page_to_nid((struct page *)start); gfp_t gfp_mask = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN; @@ -371,6 +416,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, .vmemmap_pages = vmemmap_pages, + .flags = 0, }; vmemmap_remap_range(reuse, end, &walk); @@ -422,6 +468,7 @@ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, + .flags = 0, }; /* See the comment in the vmemmap_remap_free(). */ @@ -630,11 +677,35 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) free_vmemmap_page_list(&vmemmap_pages); } +static void hugetlb_vmemmap_split(const struct hstate *h, struct page *head) +{ + unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; + unsigned long vmemmap_reuse; + + if (!vmemmap_should_optimize(h, head)) + return; + + vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); + vmemmap_reuse = vmemmap_start; + vmemmap_start += HUGETLB_VMEMMAP_RESERVE_SIZE; + + /* + * Split PMDs on the vmemmap virtual address range [@vmemmap_start, + * @vmemmap_end] + */ + vmemmap_remap_split(vmemmap_start, vmemmap_end, vmemmap_reuse); +} + void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) { struct folio *folio; LIST_HEAD(vmemmap_pages); + list_for_each_entry(folio, folio_list, lru) + hugetlb_vmemmap_split(h, &folio->page); + + flush_tlb_all(); + list_for_each_entry(folio, folio_list, lru) { int ret = __hugetlb_vmemmap_optimize(h, &folio->page, &vmemmap_pages); From patchwork Fri Sep 15 22:15:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13387792 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3596CD37AA for ; Fri, 15 Sep 2023 22:17:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6010A6B03A3; Fri, 15 Sep 2023 18:17:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5892D6B03A5; Fri, 15 Sep 2023 18:17:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3406B6B03A8; Fri, 15 Sep 2023 18:17:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 1F2DA6B03A3 for ; Fri, 15 Sep 2023 18:17:01 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id EB73E80F7B for ; Fri, 15 Sep 2023 22:16:56 +0000 (UTC) X-FDA: 81240242832.26.AD75008 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf13.hostedemail.com (Postfix) with ESMTP id 6D3212002D for ; Fri, 15 Sep 2023 22:16:53 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=DgP6mbDb; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=MqTiVVZB; spf=pass (imf13.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694816213; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KVXGH08x3ax1vR6HN82mHK+6ICeAb1chYX6HzievKKg=; b=PstA6T2ylUkjsy0Ojpkkdm63M+ZvkrUjxHPN7sFSdROQmo/3AraIpXY1gI7QUigMdMecrG K/pySvAiyAdfJLkXqH872F0gxFxEKnFMXim7bXzbO7A8LWsT5Pp49fMmIsirnhKmYXTSOU 5s2zA5gsRUxnBMNyJjfiGazevbCf1os= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1694816213; a=rsa-sha256; cv=pass; b=hfoXyFwUmwf/1rdiFLR7J/31mj3miUtIK51MgCHhn1JB+H2YgNABpI4KBLnfoUaILNEwAv bWIFL1hCiiQlaW6M31yn5s1zqtummNMcwFgARjYesGtROlt4iHPtyKKrWi1yr6rRIWIQGF l71MIehwndxlABtCMoQW7t8+A2CFsnc= ARC-Authentication-Results: i=2; imf13.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=DgP6mbDb; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=MqTiVVZB; spf=pass (imf13.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38FLvnCf031156; Fri, 15 Sep 2023 22:16:26 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=KVXGH08x3ax1vR6HN82mHK+6ICeAb1chYX6HzievKKg=; b=DgP6mbDbVJa1qzsYcAqVKrO8ymVx9c5epq8lydAI12ygLnk+ZZEOxhUbHq/aZULbSVfd 5A4UxSJlDIRU5RDi0dFKcY7Ws2MCcPgc1x5Rz54yWI6pR3pA71J29bJQmNA9Rvm6rqXx NkfWBZhUIUawGOUHsuRNm8prXxOmQG3gu3iM2y/8tZBujoo7iMD/BarBpfvKlIgWI6uS bdlSYyYzPdLHyPqzvsL+JYgIYB+XddLmG0cEfiepXyiZ0ASifKPzW/y8zO9ZWkR9eWro x064rmECKZsg5sddoet7qQ8mePWDtkT5gymYH3cbG94Dmri8X6SYmO2YeVAAIFRRjv6m Sg== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t2y9m0nyn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:26 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38FJwJrf030387; Fri, 15 Sep 2023 22:16:24 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2172.outbound.protection.outlook.com [104.47.59.172]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3t0wkky7cr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:24 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=f+MyylBfOtq9yEKf2s2SZWYmUjUWX0MCEHNdJrfOAE/MTADxtUMKZx/JuQOLrf96/TEzPDPYEGcGEDutlW66W0v2+FY9jmkwEo2AKTx59cWOkM4eyq0DhrUIWazD2X7w6jkGnSltUANC5RGABTKSnS0D/3pU9nMcHsQipwFwyEfdP5iIK/Rd8+5Yv9F8/5Um1yazw+RDmRSkZBLqae4UyOTfCxlvkkuXWB2PaqsVDhTzYdjTJYMdZICm5AWUnTCDxMOfNICZQnng0V+2peE/kgtKnBByI0DDQ0snl+2vxDS+aao3mTPiOGdf7r4qxLKxpUzTuWlzla/PRIambwIhgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KVXGH08x3ax1vR6HN82mHK+6ICeAb1chYX6HzievKKg=; b=JKzcIjqHr6zSYb6Dm8DHQB3esaFbZ7i7ZKmRDcson6ZwjFv+hx6tFaUm+XusSW3tcpE3T1l0ILsXQQ2lyLWlELrziNIkFnM+jPoKukc1IOdUfBpAKD/JiHtS/rDj6CurlLXGNbiM8dxFsU2b7mgIzBHk4YL2rAFqC2yPi8opNZOKMfAj9UNWVru2DGlDiXvi8IQe7WGjAu1TGnM415Gjih+t4jJZU4n85owOR6+3j33Vlu/Ehvb37i7W6UHNWuJ26bcKN0HoALSbO2lU64kW3hSWzoOTNBFjQZKxKYfrxEVCAosclM59BPVlV9ibp4TYzvcwikmwX5oNA7YFDDY2TA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KVXGH08x3ax1vR6HN82mHK+6ICeAb1chYX6HzievKKg=; b=MqTiVVZB8yshw1PoVV3hAUuyFr+6Vtux5EvhsSykzw2jSVHHcowqD0pki823Zoun2RlqrWtumKWR5GxN/G5I9NEAdcKY1ZZ8j87BneUOF0cyTI3dPqnYTBqZLhMP9lkdrhSMQCMQuEB7Q2p9AXf1vRMpAzfSPIB5aqjK5ziZJAs= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB5861.namprd10.prod.outlook.com (2603:10b6:303:180::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.34; Fri, 15 Sep 2023 22:16:22 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.021; Fri, 15 Sep 2023 22:16:22 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v3 11/12] hugetlb: batch TLB flushes when freeing vmemmap Date: Fri, 15 Sep 2023 15:15:44 -0700 Message-ID: <20230915221548.552084-12-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230915221548.552084-1-mike.kravetz@oracle.com> References: <20230915221548.552084-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0039.namprd03.prod.outlook.com (2603:10b6:303:8e::14) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB5861:EE_ X-MS-Office365-Filtering-Correlation-Id: a9445a06-5465-44db-5ba4-08dbb63964dc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: rLGGHI3ilkJTaiF9zRX0mO2qMjMl/hGWkFrS9U8cfLFTJVeSRc74rMmKoru6Rbv6YpqLCn1NEPRRmtJSqUwZdbj0P0l0v6auqYQFXMNwVrDYhSjluQxfZW+/w9+cWp18WBRXAALEuXB5LeHtv34+j8rwWShxmFw22v2oLMptxWGD29eD/utPmLFfQN1iMwLkvR1TAdVFdMlyBhHWPNWpc4b5qqVaseSfxRcTSTl0RPdXYpx3mnEQN9Wtp+k7sLFrMiGagYzGecgwqr/dGD9JqDB/4AOEOrgq2bfK+tXvX9k6OC1LtZ6b6z+7IcRCSICTN7XJ1Cl4cu3G5Oa7BBmMTJBFCEacaX+BPk7SuNmbvuxlLtxQRqaDuVhTDOwTMxVN4pU0haCfVHJz1LM9jOx/NAqVYPkTBCCKcIKayJPM6WnPwFRC8PUPI2dpfgdwocFHxs7x9WDjaQgrpMljJ7EzsZTIg+C+auqBJk9ZglGS8oRQ80N6T8BLasr321erXaVAvbOOwswGwCeVCyj+VEEAeCPdq/BQ3PbLsRU91TK7pU8gTesO3w7tpnakL/+UL5Vj X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(346002)(39860400002)(366004)(376002)(136003)(451199024)(186009)(1800799009)(44832011)(86362001)(36756003)(83380400001)(2906002)(38100700002)(478600001)(1076003)(2616005)(26005)(107886003)(4326008)(6512007)(41300700001)(5660300002)(6506007)(6486002)(8676002)(6666004)(66556008)(316002)(66946007)(8936002)(7416002)(66476007)(54906003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: qpVFlkuNaQld2kWan3MVFklbwP3urrvuCAYJ1C91vbnhKATaWH06c52k/2UvX+8Yx/QXXNQduLKELLYR9HQx1ongxa1POqUsm3Vn0BKJDN61akb90p2qhxs2fU5HMkz86O27z1mh8hfdTNGmijUbK/EUNRrCu9tCEHiZ5L+kRlJ0lTKZpc7aBmOuXEadYkenKFgnaGz9GUZ61C+B09DGgkRlt8oy3v3fpEYGSqaXHrxhboEHrcfAeOWsn6rrfLtY45PqtRm9lgsco1lR0u94bu7qXxBgjIbiN2zpnSZFKoQisyRootaHYYA7V/sE14bMQPyvZo3uavEZySV5c34y5BpHvUmldwfAoNErQTotcanZKl0k0ESaBtZrlllW0d5nhVIiq4z2Dl/UP7hU7mWV6KdhsoBMKF4ftw1+8bGCYjNP+6zg60cUiQY1Wuu2ECrEo/Uix+uPQtdcsXxfJIlg0RrbjNs3AH0uUE2sPPVllOnQ4XNgpQeavtPn8s0kthUbABlHMK3Taxe7hv3THVGPEpquQiva3e1962sWgF4U4szhOG/hNWhm2qX1ihihqdQ5Vb+9EcFrZKOWQG2tS9INezG9Q/rkJJGOL9h7SMpOTAsTqk5WseLzVhhQUb7fS9dqPJp5sLSTBXqR22YB1rb3thm9d3VBaZomkRHh0JfK9LoOOOrctmnEgIZBPRks3UCN1Df1K74fNutHECC5tWUE29NMFdgksQCJnEqWbt2eGYszu/LMY0bzd4/iwHUEiR4O+Yl5MkYxM0ZOFYNo2MnWrCMRB0lVZWWwDapURhfhVR1QqcWENRusPy4fGNaz+0Fs9BCKYcQ8fg8SSo3BSlcNBmVFLtHYvoki68njo2HfoJ8ZeJHbXwB6sfGVgZyx3jxmOPsKQvbHxwuztw0QXlNpfr8iDNL2Un3PZWXj5Y2himZ1iAzZ0R7VpMlFflx2KmBRSAFn7QUJsdbX1izT0YbdcktdznHuEwU70QQPcWjKjsHtseBXil4eP/zkYpD5xujj6zOVrgCpWwuNSJAlL7Yjv9NzM5rLX2yvQ1eHdv0UlITNs562yxTRyMcQ8udfrE0unsmOi9NBdzh4gDCoBBgF3j8dXQSwLuBkchlAtKE/BNXlWx96Jq9CzyjWXcoqJ2UwiULYfRJvy4d3on3uau0AoxCssjfzOlyIMQWSpoQao7VGyrY0vzbuqLLy2RdCN8qYfSzzsT8yWSEe50qYL/F77oCVZsQ9EXlmv/OPll+08YNrxFlJacg1PralCmxXZeuUdZWDjdOyRoagWksZ3B8+QeED/BnWeSgCGopyvXmyOG+2gZgy0lhLsfZRVeFUrWGMUlQ/nIv+ccpPu887TI5SeGE5d27GyIqUWibT9fIBQLq42zCeXIKBKjhBbAcZ/TIxo1b5FWmKR3EaHGZo8VGWeZiFPXtMomvQzNglbDV8KTr7ZFOFt8ks+4A7K7TLuzWf+8PELSZFUBZUP3Dgmrb3/Zl4tGiLM/Pxi7of+gI5C273YfV+KZJQee42QDz2BsPwa2mzWc4TFBInzIjjIM8y+x0oU9S4dwfAM9rKzxOqc16TX1kcj8PAzNutYmfK5CMnBc/m4RzcH4cv8U7EEQyn9A== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: LLt4B3ge3EZq1PzKHTdip/EFr35PGYcJsT/27O3qoAIHHf87vrqj+aw5KB9F59iR94Kmp5hrqrexFgY9JK7Ouo93k8ErBe0UgTbB9tBRjQ9cI7cxj3gunzW7Gb2KMR4fYce1QTTVnQYLfQ4/E9yww1Z/gN3zW2bwLcyVWU3qBU7hZ4Lh+eE9ix0Bs7OEh1Y+uIlFHt4yZNMpP9tiIYU4WiO/pgTPWXei4XXTXVz3lTYj3rt7s+hmzaN9Z6zCI0B5Ve3BWigvHv0OO2G33K4OGX9g3I5RdMPqK/7JrF3Xt0wdBmyTvdacJknf7T/tC1E3x27DQYbkuHlbE460wQu3sNwMzhxAY8CtAiOGPsr+kfoaqoLFZ45FBOBwByhTfpWstPsLhACbTbdWKucXv9KH9iJ0P/R8XmssA3QyRQYlzq2v9TfMWOR1ok5b5lnJ3zBU9XE3ANl4yVtgLR0n85lHt/avQPpO7RcipjfiOL6H7JLfrULu/vPrtYkJe4+0QIgpGqHb6QGVMDgxeLqyCRvzS7JNuNoM5DJCeKhU+5lTbH53LCC9OxQh8QpZhb2HxVxF0IsQeEo3CpdMoI4kHA9GOsr7xql/9EIJjgfYuzOkC184nCj3ezgLTiakTapNzCOj5GGOk5+dn51HVw3Qqc35h+aWC0SuL1U6uam/tRl6hdHtAY6s4okovydryNZiyvZsYpp0ktxOxtWlmUqFcCrsLsFVpAFHg8D7zaeT3lLlHNGh5FQs4+Kupiuv5M8z+qobAExvcK+PS/qouZEUfEBrr8OUb2LnGEPJ8Etr67chOT4cPU5nb+KJuXxqwlI+4usd5j2k+VOz+K+ruLj9JVO2GB7OqN5v9lysHpQewOzjo/6TMU46SAIkZWyBwwQzb0NIQwdna66IrGoUkLhPP7WVuy2zQlCOuoyNlrg2WjF2P0seG8pPdf9C2Ch9Uxp4onBq6kn167GuDDQMYkxKljiMYh+tLbz6ONzgMcpmCaMeT/A5fqKygQm9NXJFD34wsdejk5WiM1cugPQ3VNAgr3mSrSWDK5PTzuYo7pHQ6In2o0y+twXGV/oCYLpRT/a1/sdGCgnX+gAdnXBLxOLvRkiCoLqJeAcxWTI8N5ZAXxiqx+bWCaLS/SSvNxc4b0lqKndb X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: a9445a06-5465-44db-5ba4-08dbb63964dc X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2023 22:16:22.5921 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: g+VB81NrBPXra16DfyJoa6+XvNIx1KquXGYW3eCn2NTqitWtYVA9TrzA+0oAXp63JqHQyXDqN5xsih+Ws85ACQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB5861 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-15_19,2023-09-15_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 adultscore=0 phishscore=0 spamscore=0 mlxlogscore=999 bulkscore=0 suspectscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309150200 X-Proofpoint-GUID: bJMmjlEics-sIz0b4OFHXYuD1U3EifRW X-Proofpoint-ORIG-GUID: bJMmjlEics-sIz0b4OFHXYuD1U3EifRW X-Rspamd-Queue-Id: 6D3212002D X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: u4kg1k79e7comm1tj9sgsbksxckytuwi X-HE-Tag: 1694816213-604130 X-HE-Meta: U2FsdGVkX1+j5jKLqbrJWW+hjXPtXFP/4bmUcRVNzZ8gHHkmnuJB7LYriKlUPBumgF6o1PWnU1tNfNb6b3qgK4zxJvhOL2UlieMCvTsNn2LBfIWFIVpKar8AP5DEG/m2kxW7DznWqfOBgFZ3jzzLTqbEKKEmy43GgHdIMUuMwtOGSAXow6EZhK7BVtISuEml322brUbW5b/HtSg3OSGqCLKWGjroSIzNVRDjB9FCqPYrwresoN3dzffUp1Zi6zGLJ5jciNEeEo8cmKpSxgB27sg+1waYKdajlZGU0eQ7xs1WulkO5C07z8bHGeccJ7gDUJj57/zBEt7RFETHKbdKOAJyFOOHL55Kg7pUal9giskv8LUR8KVZTQHAbo0f1W/ZYccYDRD9HvhGhJSCyWvrgSHI0PLvwujW1ZubJtg/EFHXVTmFljaTT7qlka5kDhuI3juPSsh4d253vfM3CnCpKjGAr7HVRiKzZ6LDI0NwK4U+ReMRQr4dWCx/tqFqV/pjkAMyxJnHXGcGiG/4wPkvnClF6rIgUTQ0o7VpPEHQhCjKoT3T3GCHHKQasAvx+jEkFKzw2RRkw/zu7gdbmIEP3fzWiDSh1hn9uiszK2CZq3cQgAa1P+OVABjdlUGY/rM31q2iWnnqp/M4gU40k9I2ZJh8aQ3Lmo3ps8rD35P1WcXv4TP/V/5WinO6i/wWP7Cu0rdKaG5wYU/yZYC8OfMuTeCN6Cwh6XN8WJLRWAYio5qWvQ9VRWEJxfYIe88HDy2bVw7Pk+dhl7VG09tZMUUEsvaMfQLwyEKd2glxqiu4IYU6si/u7UDL46SDjKOvy29pxFVrb2t/YokhPXEsDci+QO+gRv96DfRoUbC9oujBczpFGmNaCoNXXnE/ctw+KTYCK930VFT0ajTO3K1lUPyhSiNJsBmqtd/5rwh/4WBnuBn/ElLF/1kSUpmuevrAU8EnvmuigEpmHi2vTlpi+1B lZ4n2L0I cMOcKrDrX19d4j27qqQ2jAUoZHEO/gC5SrH8CcPGpSlfUjU/e5KC/2kpSqj2k4W6A8e4UccS8DAIoThql5ILlCfacRZ6cDN59P5mS8WJSeRq8hlg+B4rhEBN7N2sUGZLAxtyvw6eQA6LJGOzTTUVPqAUGEZW0Dygb0a4/0ztCAHyx59KLpcV2B7PN2k/gO9CfYWomTS33oqhhM7TIu7+3a8c8TIIl1zWHxqRc7pGjLSnI8snMwyn7uOqIl4pzU4sxn7Eau1UYeL5Ckvah4p6Q/KQE76GrjAr9HW6insrqY9OrzvxLPAyLAh+GK2UYzcb+PqwwSnJbtvg2jNbdVvxjGvrtSDnHBqtPgbilofgoS/xbREk8Vf6EnfNMPF1kGJF07FzDR/pG1BOLg63KzpRiSJv1837RRtUF3A20xEQNbNafRbegeTjd3JHfk0cYMslCOglfRbdkFlIsn7g4xM8UgonFFn7XIXiWc56H8dwcrEveU0soDTvAHIT5yhSXTJnxpQKA X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Joao Martins Now that a list of pages is deduplicated at once, the TLB flush can be batched for all vmemmap pages that got remapped. Expand the flags field value to pass whether to skip the TLB flush on remap of the PTE. The TLB flush is global as we don't have guarantees from caller that the set of folios is contiguous, or to add complexity in composing a list of kVAs to flush. Modified by Mike Kravetz to perform TLB flush on single folio if an error is encountered. Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 44 +++++++++++++++++++++++++++++++++----------- 1 file changed, 33 insertions(+), 11 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index c952e95a829c..921f2fa7cf1b 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -37,6 +37,7 @@ struct vmemmap_remap_walk { unsigned long reuse_addr; struct list_head *vmemmap_pages; #define VMEMMAP_SPLIT_NO_TLB_FLUSH BIT(0) +#define VMEMMAP_REMAP_NO_TLB_FLUSH BIT(1) unsigned long flags; }; @@ -211,7 +212,7 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, return ret; } while (pgd++, addr = next, addr != end); - if (walk->remap_pte) + if (walk->remap_pte && !(walk->flags & VMEMMAP_REMAP_NO_TLB_FLUSH)) flush_tlb_kernel_range(start, end); return 0; @@ -355,19 +356,21 @@ static int vmemmap_remap_split(unsigned long start, unsigned long end, * @reuse: reuse address. * @vmemmap_pages: list to deposit vmemmap pages to be freed. It is callers * responsibility to free pages. + * @flags: modifications to vmemmap_remap_walk flags * * Return: %0 on success, negative error code otherwise. */ static int vmemmap_remap_free(unsigned long start, unsigned long end, unsigned long reuse, - struct list_head *vmemmap_pages) + struct list_head *vmemmap_pages, + unsigned long flags) { int ret; struct vmemmap_remap_walk walk = { .remap_pte = vmemmap_remap_pte, .reuse_addr = reuse, .vmemmap_pages = vmemmap_pages, - .flags = 0, + .flags = flags, }; int nid = page_to_nid((struct page *)start); gfp_t gfp_mask = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN; @@ -628,7 +631,8 @@ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *h static int __hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head, - struct list_head *vmemmap_pages) + struct list_head *vmemmap_pages, + unsigned long flags) { int ret = 0; unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; @@ -639,6 +643,18 @@ static int __hugetlb_vmemmap_optimize(const struct hstate *h, return ret; static_branch_inc(&hugetlb_optimize_vmemmap_key); + /* + * Very Subtle + * If VMEMMAP_REMAP_NO_TLB_FLUSH is set, TLB flushing is not performed + * immediately after remapping. As a result, subsequent accesses + * and modifications to struct pages associated with the hugetlb + * page could be to the OLD struct pages. Set the vmemmap optimized + * flag here so that it is copied to the new head page. This keeps + * the old and new struct pages in sync. + * If there is an error during optimization, we will immediately FLUSH + * the TLB and clear the flag below. + */ + SetHPageVmemmapOptimized(head); vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); vmemmap_reuse = vmemmap_start; @@ -650,11 +666,12 @@ static int __hugetlb_vmemmap_optimize(const struct hstate *h, * mapping the range to vmemmap_pages list so that they can be freed by * the caller. */ - ret = vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, vmemmap_pages); - if (ret) + ret = vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, + vmemmap_pages, flags); + if (ret) { static_branch_dec(&hugetlb_optimize_vmemmap_key); - else - SetHPageVmemmapOptimized(head); + ClearHPageVmemmapOptimized(head); + } return ret; } @@ -673,7 +690,7 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) { LIST_HEAD(vmemmap_pages); - __hugetlb_vmemmap_optimize(h, head, &vmemmap_pages); + __hugetlb_vmemmap_optimize(h, head, &vmemmap_pages, 0); free_vmemmap_page_list(&vmemmap_pages); } @@ -708,19 +725,24 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l list_for_each_entry(folio, folio_list, lru) { int ret = __hugetlb_vmemmap_optimize(h, &folio->page, - &vmemmap_pages); + &vmemmap_pages, + VMEMMAP_REMAP_NO_TLB_FLUSH); /* * Pages may have been accumulated, thus free what we have * and try again. */ if (ret == -ENOMEM) { + flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); INIT_LIST_HEAD(&vmemmap_pages); - __hugetlb_vmemmap_optimize(h, &folio->page, &vmemmap_pages); + __hugetlb_vmemmap_optimize(h, &folio->page, + &vmemmap_pages, + VMEMMAP_REMAP_NO_TLB_FLUSH); } } + flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); } From patchwork Fri Sep 15 22:15:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13387791 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48E85CD37AF for ; Fri, 15 Sep 2023 22:16:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B33026B039B; Fri, 15 Sep 2023 18:16:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AE2696B039D; Fri, 15 Sep 2023 18:16:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 871C86B03A0; Fri, 15 Sep 2023 18:16:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 72D3C6B039B for ; Fri, 15 Sep 2023 18:16:55 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 49D54160FF6 for ; Fri, 15 Sep 2023 22:16:55 +0000 (UTC) X-FDA: 81240242790.16.FAC6932 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf07.hostedemail.com (Postfix) with ESMTP id D186F40027 for ; Fri, 15 Sep 2023 22:16:51 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=cVlxG1zi; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=Rvq6XiU6; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf07.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694816212; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fR0foe/sV5Sumv5EvmNzyoFAm34AhjLE3kM7pxcILow=; b=rL2cHpTXt2p5vXr7ekgfm4vBBM5hMol6XQ0flhQG/6csge9lft8ytVcgPinZixah7CLQtX aH4muFiRA6R6OQLV12XBjWpSbUow6w3gU7X1TANAfzO6OdRux4sxBQ5qN7kILWC7ksMMJm YrneyXXVZY1GW2HgDFuJ88wJgfiYcEM= ARC-Authentication-Results: i=2; imf07.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=cVlxG1zi; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=Rvq6XiU6; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf07.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1694816212; a=rsa-sha256; cv=pass; b=j2EEKbaUEYvt801iF+dk70Jrl05sYdWfBB66fTvPTU6X04I6mC8YMEVoubN+rE6nx8PTr1 sXMlTKHk+v/bzJ9pVgqzmnJQIR1e9VCtPj9jZfa7/fpH8hZACvaTzmiOyY3Uet33r+Pke/ JYXiAUENpcyUj0dO+a+5oT0fsCUcBYA= Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38FLviaD025851; Fri, 15 Sep 2023 22:16:28 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=fR0foe/sV5Sumv5EvmNzyoFAm34AhjLE3kM7pxcILow=; b=cVlxG1ziPRjon1lSiAP7Gx8+qeBaUba/zMsYSP7cnMvh3ZghrI0ThFgEDtn3LNoODhh1 3oBxSCiDq71i2Ufua6jP/c1IjrJWXgxFWLAV2LTXMPyIxXcdEvQCjtakluxEGugEmU32 rwQq7+Xpt951BEgucBtIeBL8UnPXGA2MUJdXN2JZSgjvAq3p8RiyqNXnHHADjSb/u0Xw +hzVLERmorw36n2OJfVhHUU0doEBBc6RawYrfU0TKfKwQixlGEegvx5XmTmVkBBJZyt9 PeWdySeJVEtEC3Wf4X2rVdS1pP60GtXjOchu7IMgzSbOpatEdRApvZ2mIz27FUg/CV1v jQ== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t2y7n0k4y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:27 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38FLb0xI008807; Fri, 15 Sep 2023 22:16:27 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2174.outbound.protection.outlook.com [104.47.59.174]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3t0f5awd55-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:27 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nMWIzJuAObHBG52dU6yhF0tD5WRm4WNISyH04Tt116RG0MWW52Futa/X2OSAJpjejjH23xZSManhbgFlm32yac/Mk+/wDu4rQlcbRJCwoVBmxbtDZUOO/aM91J0YvzcG8jm8ncIQPYoKbW35rixP6kvxxt3LCM5Fwq1TWC+PrWEDoZ61XVN5r+UzPE88OlJY2DMnAtg96cw19hyqZL4pEOsy1ZX/IUzs7OH4ME7j2NW9q2LPWT+Pk0NDDF7erT/VshsTbZAY+H/qggO2ntDrw7Ad6e3phwiWwOXsldiabCj+6wyWP5MXBfSM4bsNYVlWERPT2jZ9xvvowWdRRnBwaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fR0foe/sV5Sumv5EvmNzyoFAm34AhjLE3kM7pxcILow=; b=npA5NsGIMmG1a+7X0arzQ70apfAQCOlVoYEh8KsI/B1/bdVY/2PvDT2Qf2KN4a12aw0XBZhvvgSdazS8I9IRJC7DvOS9zh2i4Q8/o0j0MP+Q3OgavWiXDBs247PrmcrA7FVWNE0RAhLXwcu+3/dxNUov5O8c3k/AYlGnoNEtu/r89dTRMM9aQhBIIYUx75FsZG8k0jQflJdpEuJaWlx3RGqQ2jovT8BhZTYNSVVTGgxVyLcQi4rBokHSLF0+losnr8fuIydvXgh1Dnbtkn0M8GuJlL1PLicVMOU3KdDXfHoNmQ/WKZRicAXrxZ+ExBE2ofz0E9EyoMZXizASW/muTg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fR0foe/sV5Sumv5EvmNzyoFAm34AhjLE3kM7pxcILow=; b=Rvq6XiU6wjNWckoP9ZvRctqY9JXUAJ5ksgqfl3X/H0fQDGzGvQ5qtaZSx+vcTrBUTvaZVVk5l9G4oLaAP2XBUsb1enpp7oEXZhiBwbEjxptljtpVJiLw09MQdLsEX1BQjSpyW4SZAH6iEC1HGpXo5zaBSgy0EkzFtUXpPyfgH+U= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB5861.namprd10.prod.outlook.com (2603:10b6:303:180::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.34; Fri, 15 Sep 2023 22:16:25 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.021; Fri, 15 Sep 2023 22:16:24 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v3 12/12] hugetlb: batch TLB flushes when restoring vmemmap Date: Fri, 15 Sep 2023 15:15:45 -0700 Message-ID: <20230915221548.552084-13-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230915221548.552084-1-mike.kravetz@oracle.com> References: <20230915221548.552084-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0201.namprd04.prod.outlook.com (2603:10b6:303:86::26) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB5861:EE_ X-MS-Office365-Filtering-Correlation-Id: e8ca82fd-ba7c-4bd1-62c8-08dbb6396638 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: FnYUU+jr7yYOLNKTYoVsW5nqTom7s3SrhSiJoExhqGgoC/rdPlqItVgBfBmM1U4gogovk7gQLLVVNHoy9OKdqH9RYtXWaJ9ncNVoc9EOyM6FpL+S1DYm9tDyAojHbi4TEbreEia7BDJd35fxKYiUDqt0HvsM7aVNAJPbCTiBHy0WgQo5i3DHt/5q7dwcnIqmiNP1EhS7Jilf5X7LbKE02/E4/EIcXwqVIQprvmI1tX96BXcx0mwWINxZS58inx5jjv/kNGCSB8G5RnTmdCs7BjZR06sU2OADkz0gMD/1nXnyaBkx+EC/XXII18K5FRI152fWm8rw1vidLgFPPiiiiFRCLTCHhIR0YaUmXHoFC6mMgCaDDLYhdJRoGQLku4vpfQ6bd24ey+6LIj4gkHyz8DYc4usiglDCgLppGLqgoTMox4sW1Tycl08X+/8uXSGbY7ZCRlfvyy+Z7vt3DF3Y5/LHhP2WWWBXQr+jHA2cx4kXNoOaad0rcY8uX/FXoBqTNN5BIFF3bzV6SGlUoBijC8FI93otxx/3WlvXAeYZqJr+ULe+EoNJYW7apT/rzxfn X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(346002)(39860400002)(366004)(376002)(136003)(451199024)(186009)(1800799009)(44832011)(86362001)(36756003)(83380400001)(2906002)(38100700002)(478600001)(1076003)(2616005)(26005)(107886003)(4326008)(6512007)(41300700001)(5660300002)(6506007)(6486002)(8676002)(6666004)(66556008)(316002)(66946007)(8936002)(7416002)(66476007)(54906003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 0uJwDlDZxbAHEdtFmkAyMsyQdOMk8QWEWHvgpWCHt0RBK59yIJW+FPIOrCOTGfbQMkmlAz6G8zX7thrqx4d9LrbqLZpYHaClHeGRmfkMR8HwkOkXkODrr+vBizD6H3Eew+4gf7AxoHo4gjkPD4Fi8wmUgTV0tUx/fQQOZmtT4RKZhlPkC4OZBV4zVekkFVvbRX+fi7T7MFf56lAfksRMmW1az5yPctW6PoTvu7pcr/e396IRbIsRn/mQRkSo/GOjuJ34VtqYhez3icf2PAnW/5IYWT8BN/ABNnqAahC+oNtSTauQmHSFAAC8ofImVhageKRVBwOGYaXFvAp+mczVA9My24lkxBuIWjOLKf85yLfqpaXMdpBnAoBFYwYq2QaIV173QqUDkJBS/xX48pWkDzw3277DkVj2pEbhhLUSfkfTYxk3M501iBLB0LZ431XsBbh3yYCFlsR3kNbrL7qQobii0WAHyDZBC1W6AnIzo4DAV3ypTd/w7IuqHKT7RXbR+xPmYSEAlzloSnIl20AdF4l+C2Y3qcJonaylm6HVZxhb+yi051pbvDYpmasLzil3GGXIQIUkmA73RkXvxjjgLgU3ZbyGqyyQDkDEs6lLKmlvmST+YSKZO4U3tjhP5pUeBMMGLZBpY9epVm8RdYFKEDPAfgQAJotd78CDVEwq9iEI7v48Tousi1jRgt4YdOS9I9XLv/TKbS7PhBl03hNv3dEFECtfeJanueRlilZzek7/HyHhqUStX28SSOYseeS9tKjQAhi36acSogCstzNezsWO4xFLHfUQj040Et0nZE3jYT0k8YX5TjrRmovitp1cfe8dj7zHVlKgECEQlUX2GDJK+IA6a4OmxqRzxB7PLkz3DV0zYyVKZVhkhv3/nJ/heEEfRXlhHTdMZPgXKHSwEhemKRsncnn7DaHFWnnGRl6xKAazHhffuQAG3mV95qOrytJEKY7cjCoqPc2cWShKHr05rbUQv7KAFXYYiBlGbVGFgsyqLwU8vE3wFGWY74Q5ubwenz4NZpLhCUL2FL+MLrK7XuYbPBiVnrnGiTqIJGCL4Sf7pTEkJeP864wkjW1WrBB+7ewbaufVJ9Kj0r9KP14yGnRLwrgrpuZ0pJlA4FJ1dMXvBkqvFDx6I3epWAer3ltYL6RxHfQtE3WA6vsR714y0McvLWd7g8OR7lZLZEheEG/ytOBpOSKo7oFfydyk+YXK8vVyYTP1Ki/43O8+wIo618pjjxtE+bGtShtDMUDqjPviGt3vhWZTsBUGNlTTanGfnrJX4JsyJWUKU+p6xtUVOQz8GvaPA8A2zOR1dMGlOLZ+U2uTlkqIGUKxDCYLf9nGMHn2X3uHoYn0TluVCGGEJvkN0dDuYR76xP/sbSp8acSwmwmL0bpGTmblk/wB+RrV2IGLuvf1SfVbPRX7TGuBmGjxyTnWWSc2Ff9FaleZwCyCZm7L2WJVb7subBsIkzMPZWnsYbJJKVWwUIrt3mHJ0xRph3vJO8h4/l/nusGXzmh1QG6D8LPJnbeaG+5JRZxMGJDGN2pHVOURkYS/H99/HGEiW6wL7OiD+qUiDeMYcWnntwo4JJ4ghA6cuVZ2kSiUQ51/VcSlEbMHeK8DtA== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: noXzunz+VW1vfjvQamsOS6IEmED47Z7TSxbIp4gbo+QT6Nj7XcImYythUBKPj2t8KtZkjzvXhR5uq8tNWENn/VjkWPWhdLHSVcK8EQLj65Hun3q+ZOBkrb9pLFqzgw8SRZnei5ufFaZF9iPUjm7b3/+M+3qeFMiTejFIwxKQEDLLhWAUS+mwnjiIUx9HALwF/occtqh09A2b1SnWebAGNOfZ97F/oKNLBGAnTsPq3VstTEgRvR6OKC+owg7/yE1sWz19jQLguGFC9VbaRVnGQRW5zFVqD7/aWiw1Olr/fXKSLCm2tXhQE0vORYJ5RQilp4e8UqOdg6jR4/UEeFVmxeYusy22hP1lVl+Z8uAPSnqSC4+jHGZU2nkg8SnsRU93XsYFMwDO47TYyINskhHV5dx76YegUAaVzANp3RGMEIZkbZZKe3kArv8fQJOiVjKeLvhC+Prq9bAQafdvFjsgJVgrSl6DNsBfVRGZhNOTkSByCZzTQ0lV0f9k4bMboYwdunE4lBuNEfqeNVV4BA/aQWJcDyuqDJ08taSsC6ZChcTPASOa+/5uTsmy64ijCoovAmX4EYBgrEsKRbYu4fc+vmKlW61Xhotzv2owus0HqVUghUp5e+8lco8U+F/sA1LbvFTszLwkz2oD4fHBXS8eEGLN/gWafYDqQ0r1L/L4qYiJSD1FAlvHAwmosxNjI5n/E5Qf9cF/aYXPRYUB50GWLmbD+aKWF3DT/qds3DpPCIeL7NCQAyIumq6is0xss7p4LEPxFoq+awhXU4AXv3gpiE0/AWRSudkmCZInjKgMaccm6FW0tbiGmh24gbWsHzH9yIry3e7kjI97KpY1XhXein6Nu2cFC/YzTJTrGhwRYZpCni4cka1Wwti/jvnpUwTlM0p2scVkxY8oHo2AW6HCaksrbIj/mSz2ow5pXT63bmCMPf3mHXvO+dkcbt93pIq4kFHyDOVaBNK+b99iD9YKjJPmCsQp7MDuvmnq+yf7w909yhX9MUtna4aq/fuucHudebMM11HzSvJ2GNj+RqCxWd9lGbGE2Sdp0EChsJWKWtmP4IF3/G1AfL2pujxfSzFQZBRuTVqqYlu/7XkeC3Dk8YRrHIuM7myMATN3I1wr2T1RiR2ohsykU36cKEUc/Eh8 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: e8ca82fd-ba7c-4bd1-62c8-08dbb6396638 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2023 22:16:24.8774 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 4xh6rb46FVTquit+qOmfQh4Nx7cmfoIC8fMJuy28A8LajsLFEnXguRwk4dLstxJ1KqY1mgDNZl2HCGSCWSB5Mg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB5861 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-15_19,2023-09-15_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309150200 X-Proofpoint-ORIG-GUID: _7KwDLtVsLlD14n7b_LTNmnQAai-P1V_ X-Proofpoint-GUID: _7KwDLtVsLlD14n7b_LTNmnQAai-P1V_ X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: D186F40027 X-Stat-Signature: jd84oob648cjqqgfidwfdwbrs641nt1r X-Rspam-User: X-HE-Tag: 1694816211-168655 X-HE-Meta: U2FsdGVkX1+RWWO8C23Z8YR395jH21C6ABXpHhzFUZPrhrRVTos+8x7n5qUsiQLC5M0S740OtJcjg4fkaXKIlAcKFvVoly62g1BIibw0iAap6M6FzHWMfgeCWLDCfAKqvLc9NnZA2yGKfN7N6b/J7hoDRURRmJ8lss8uvljuDngpe80BIymfcbyZCkRubX+shMz1WsZC4ChH3/lfkZoHHZvFpwEZfJtwyqjxvqe/da3EiILlw5sxWuP9aQAowrjaugwFCuFlTWLhHcNGDIaNjC2kS0SuW/8yLxlncWOaUaJsTAveJ6PKaHZFdYSZ5EHcsutp/OBfPhp6uOc2jg+BoQ/a86mtW73m1gqY1hk7rY8k0Qc3IYLQ8IrQOuqlCLc6YjAYdSX5BHAF0P+ZDJ5QcYkhURw0YCO1G97zDM2j4AjXGOhHGHO+gvgbCuLMjPFXt+KscHPzk3Uhq78iEzYUDPklIQS5khW+FEPnltLFt1LirXYrX4muQujGjHcy8a2QYD9IRjse+9/LMh91PDVAy3O+Yw/mmUBBiPluS68CegP0O7JZqeZrdIycRJ5RXWekrglsnOPme/DRy+UVjf3MOGQgfQfS4N46m2LuiHSfoTtkh42OEp8AaFQIIZfMYfJgPLHqipxIraB3/N5QeeSO2ZOxTM4NpdAIWqgJ6pdbZLmmx5GxwjPk1zrorOY8jJ1RXZwac5QwQAY1JLREuEkpQ4RNJJIRy/Vc+DNi0LrJR9rIxCbS6x8oHLURm1nckRokmjkI2z6Z26zI+9Chi9ASJcwZtLlRyC6fsCRuINdWV4t5X2blsSGpjEdQXCeiZgA0RvDM8w83IPq1S7ikXqFHdSE4Lejou4s7m5d/aNFl7AtoSE0ZTy+t+QIK1KDeURty/2RHAtN0iFZc2MWmdsNqhbNA6CTRZMXb6WASiu7lG1px/Hy7zT7EmzEBICHg3WbgPhjb4bU0NzKDvxJTKlM 21V4wvoQ dDBwOAlbc8a/00doNAPmQkZrjWWqJzqMpTKfHxzCjzoVKAXSeiRIJ03Eq94MKk4GrZ/pPsBQGL5Q3Gwqe8y3dMPtsrSC5pJxGoFvt/7ujPzKyEOFv/493gwoD6UdWRqhE5R9G5coAQmYDrqVvu0kprszyJIug6Na1NQwhbL5GS+0a/HYx/1jpbI8fq9DbX6I6i8CBTPrJpfjxYFMcmL0zmpmUJy0Ow5c1YyrFUDKWw2xBYlufynxBB6cWCEb98s2sj2+8FQ/8nhAFMVRRICLzpB0pS9G77kUq4Hgxp82sgJ5iH9f5xfDG2qebyuEljRTO/IPymzP3+WjiKqHBYWzuuqrBiN4lXetMTU7k7Ncb9b77RVwAIFKe5Pa2gHDW9Jh7SfP/yZr3GDnBjMm+siaGoQE8z5l9kKIMA8fyYJfxaLnMh0EpARL9c88bnYeVFAxLli613fHGlcRjpWWLv7oIHc796a3XptntQ9gc4mb8K6vY2KJq7KQBPKrRry6nwYCvufqk X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Update the internal hugetlb restore vmemmap code path such that TLB flushing can be batched. Use the existing mechanism of passing the VMEMMAP_REMAP_NO_TLB_FLUSH flag to indicate flushing should not be performed for individual pages. The routine hugetlb_vmemmap_restore_folios is the only user of this new mechanism, and it will perform a global flush after all vmemmap is restored. Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 39 ++++++++++++++++++++++++--------------- 1 file changed, 24 insertions(+), 15 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 921f2fa7cf1b..0e9074a09afd 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -460,18 +460,19 @@ static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, * @end: end address of the vmemmap virtual address range that we want to * remap. * @reuse: reuse address. + * @flags: modify behavior for bulk operations * * Return: %0 on success, negative error code otherwise. */ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, - unsigned long reuse) + unsigned long reuse, unsigned long flags) { LIST_HEAD(vmemmap_pages); struct vmemmap_remap_walk walk = { .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, - .flags = 0, + .flags = flags, }; /* See the comment in the vmemmap_remap_free(). */ @@ -493,17 +494,7 @@ EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0); -/** - * hugetlb_vmemmap_restore - restore previously optimized (by - * hugetlb_vmemmap_optimize()) vmemmap pages which - * will be reallocated and remapped. - * @h: struct hstate. - * @head: the head page whose vmemmap pages will be restored. - * - * Return: %0 if @head's vmemmap pages have been reallocated and remapped, - * negative error code otherwise. - */ -int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) +static int __hugetlb_vmemmap_restore(const struct hstate *h, struct page *head, unsigned long flags) { int ret; unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; @@ -524,7 +515,7 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) * When a HugeTLB page is freed to the buddy allocator, previously * discarded vmemmap pages must be allocated and remapping. */ - ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse); + ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse, flags); if (!ret) { ClearHPageVmemmapOptimized(head); static_branch_dec(&hugetlb_optimize_vmemmap_key); @@ -533,6 +524,21 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) return ret; } +/** + * hugetlb_vmemmap_restore - restore previously optimized (by + * hugetlb_vmemmap_optimize()) vmemmap pages which + * will be reallocated and remapped. + * @h: struct hstate. + * @head: the head page whose vmemmap pages will be restored. + * + * Return: %0 if @head's vmemmap pages have been reallocated and remapped, + * negative error code otherwise. + */ +int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) +{ + return __hugetlb_vmemmap_restore(h, head, 0); +} + /** * hugetlb_vmemmap_restore_folios - restore vmemmap for every folio on the list. * @h: struct hstate. @@ -557,7 +563,8 @@ int hugetlb_vmemmap_restore_folios(const struct hstate *h, num_restored = 0; list_for_each_entry(folio, folio_list, lru) { if (folio_test_hugetlb_vmemmap_optimized(folio)) { - t_ret = hugetlb_vmemmap_restore(h, &folio->page); + t_ret = __hugetlb_vmemmap_restore(h, &folio->page, + VMEMMAP_REMAP_NO_TLB_FLUSH); if (t_ret) ret = t_ret; else @@ -565,6 +572,8 @@ int hugetlb_vmemmap_restore_folios(const struct hstate *h, } } + flush_tlb_all(); + if (*restored) *restored = num_restored; return ret;