From patchwork Mon Sep 25 23:48:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13398579 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE0E4E7D0C5 for ; Mon, 25 Sep 2023 23:49:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 504678D0054; Mon, 25 Sep 2023 19:49:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4B3D68D0007; Mon, 25 Sep 2023 19:49:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2934D8D0054; Mon, 25 Sep 2023 19:49:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 12F588D0007 for ; Mon, 25 Sep 2023 19:49:40 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E2F1D1A06DE for ; Mon, 25 Sep 2023 23:49:39 +0000 (UTC) X-FDA: 81276764478.27.D5357EE Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf26.hostedemail.com (Postfix) with ESMTP id 7E00F14000A for ; Mon, 25 Sep 2023 23:49:36 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=RIqHiBBm; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=oPKR+KNW; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:dns request to arcselector9901._domainkey.microsoft.com failed: no records with this name}"); dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf26.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1695685776; a=rsa-sha256; cv=fail; b=fZbPbnVzwYE3gFHky1We5+mI4Z6xoznkkZ7vqzq+cQhM+s5/Yze9fjTyzUPdM+LSmjIZbV 9CcPSNGhoizDObU+qnhG3GG9PLyTGjIvQGjKev1uIFspI0sxjf+n1kHHFEb94qYTkUTAXp 6QTOa9F9CXXYrm7hS0AEY75kjvgJUmw= ARC-Authentication-Results: i=2; imf26.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=RIqHiBBm; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=oPKR+KNW; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:dns request to arcselector9901._domainkey.microsoft.com failed: no records with this name}"); dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf26.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695685776; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3vbONjsZvravfMZFWf/jFmc0JqQCVqxQwTo9QP2uYlc=; b=TwcnLQRIcOg3LmJ5cJJ7loPLawvKI8+8fayOFiNCoxp/bMZFS1BCWt2k0NCPriOoIe3FJf Jivet+Z2c+UYD0XxBeW7+3P9Sbl8g1tWrm0QXjHk2l7OVG9W2pxeSbtWdGzuPKaIwBcwC3 RZETPvz2nNjFlp/rb4iLLkHdmI6PV3k= Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38PNEP7D018843; Mon, 25 Sep 2023 23:49:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=3vbONjsZvravfMZFWf/jFmc0JqQCVqxQwTo9QP2uYlc=; b=RIqHiBBmHQMTyXOEEMefemMO4NV/X7W6OM6BEkhKDf9Ds57/+Y8b4yupOuGm3ttV0VjC wASMlBwocRlRTM6NhUbySNJhjsfoPy9/K7koDMeP2ZyrXcAxjxMhF/CuZV00NCtUEvuw dhTfH0X3lT/XtnILlWiepVL2rbzj4oKwuQd6qI6F0AAUhjNWHTwQXY/7V2lH2yIIgWEl PM9LMOtWg7OuVWmPi1TkEfLDdLePy5FjeVsREPDRCKm4k1d8uMQqAXyZIBwqfhJ8bhBv EK/mKIgVr7i3M0zLp0Op+Q7J3+Hhp78SflQTiio8dzLmtJc+v4QLFkFLibbGZIDmB1Wv AQ== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t9qwbd5k1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Sep 2023 23:49:07 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38PN1bGg039385; Mon, 25 Sep 2023 23:49:06 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2107.outbound.protection.outlook.com [104.47.55.107]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3t9pfbbdek-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Sep 2023 23:49:06 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=A3wMOkjIVefDxSv6LDDWpO0NHJWZSMN6Yb78cNQTrl/0N7WUL/3ci5FQS5oZRbusw7ptbg1BPnfUIEiykYZepWYtBWb+gFjFOIzzp/oIuN5Mv+otcc2n70U2tQ/5Sq5M53gQOhb2UgwA5fp8zMWBCUTC/4KSKA0fTL4QkC4pqFS3Jmro2QhHEigmtI6vWLyIp1Iq9hZ4jzdTOCGJ2LMcsN0ayXnUQVmQ7PNG853fnnY1QtEK2NP0qBFBwsBtZnlGt3K3DVF6Y+Q9o352bjH93GAKZs8gmQgbUjtUArzyOpW/p3epvIuQHb18FYqict7kS/GlXkIUyMqCjGsQTpNYHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3vbONjsZvravfMZFWf/jFmc0JqQCVqxQwTo9QP2uYlc=; b=AJzcPcbkfigl6A/i/2Q8DhmCpexVmeOeUuUF836tvycpFLS2/aAUfjnrh3LhMHLBgBMnPRbYzxtnDczEUkhwe4xD+fWykq2FvaHVrqUUMau34o1rrptLYE4EEkj+0Dun+fpVwzfaDy8Oq07sVgG/TlvVVJc2ntYAO14MSUX7ZyLqWaH1UWOryjNUoIPycmGpXMMBAPoh0tmFnygvF5ZeXiQXOObfvioHBvAb73szYgW8kYkVx6Ri0Uh5EmCtGSVfQyTubYcPC0J2j3mOXEIkXqGVmjRLf9q1XiU+PiMPokmhiOjvrHKUW7P1HHTcWenZaAxmdxOCpl0P2ROOtX2saw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3vbONjsZvravfMZFWf/jFmc0JqQCVqxQwTo9QP2uYlc=; b=oPKR+KNWg+7ly19zJ3ZceAniBf6uouNXV0iPI3EG8B7KhAL0DLTuRDYcBOT3kttmiUU4xwcZo43E/PVSF94kjpC/FgXLbrZjENIQDiQplQqMvauY7RASu2+WpTW43V/bey9fujbw9ETJky5cLGWa6jb1u/nAA/BUuwKLA7wmQXw= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by CY8PR10MB6730.namprd10.prod.outlook.com (2603:10b6:930:95::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6813.20; Mon, 25 Sep 2023 23:49:03 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6813.027; Mon, 25 Sep 2023 23:49:03 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v6 6/8] hugetlb: batch PMD split for bulk vmemmap dedup Date: Mon, 25 Sep 2023 16:48:34 -0700 Message-ID: <20230925234837.86786-7-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230925234837.86786-1-mike.kravetz@oracle.com> References: <20230925234837.86786-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0125.namprd03.prod.outlook.com (2603:10b6:303:8c::10) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|CY8PR10MB6730:EE_ X-MS-Office365-Filtering-Correlation-Id: a2751397-666d-4a53-5270-08dbbe21ff32 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: LArM927dLM33lUpMgw9ceM9vDGwF5A+LYEe9pKEE8ZtUG6l8mh1K5bb9alSFqhPfYZBO4FKHgl8Q1pqs/GZbj4hpjvQUpADz8xmr6MGJD+Kuu7nogf5Q03SUOjFltBxmGet40Ss3Tfr771NSqYLdMM9EbW3OnXCOjrgEOXfaZ/SbJRjDUcWCGaCo9753x0Xmlmfba6ogPqSrBYginR6NRdQmtMQNpsOLbvpBIlnXoOsUZhqFfvaiwjmzhMCx3FNjn6sV9KTO16QxJKOdelX7qFfgvdec7Gg46rJ/bNyHu+c4Z0o1TUmJLWGRi8M/DgfFjstZYsdxyHxPx+hV+k7pHI2jhjwkxizXFnAmOc99hHs5SY6SoCufMKf7eKWH8cannmboyvF3K9iI2CiGoNRpRkSDO4eQZc0GDjWdU5cO9Ynv6Ocy0RrNeG0oTkEc/NT1sUIgoIuMequZsamaUGRifKoTwaAJ3J8vJhjPFlNTF82eMCbI4MlCoewXd91nSCy2JdGzgTIMIK1TwyhnlVab8mt15GfSY5TmKVSehbZNidVEN9dGaCAdAgbnU+I3Dz1H X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(396003)(136003)(366004)(376002)(346002)(230922051799003)(186009)(1800799009)(451199024)(478600001)(6506007)(6666004)(6486002)(1076003)(2906002)(2616005)(5660300002)(6512007)(7416002)(54906003)(316002)(107886003)(66946007)(8936002)(66476007)(8676002)(4326008)(41300700001)(44832011)(66556008)(26005)(38100700002)(36756003)(86362001)(83380400001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: jjYlHVmXbFK9XFU6jg7b0rDR4Kxzxm+sgeSwTVf+UeWzviNrlAhAutxA0mA+iULi151QeovUJQ+LOP5UG4kXUnVMn7GEo0GQ7sGK0WpjvV6NZoPniLqqdEEHlMcDZwTwtrkgVQsfrH0YKOBbnmF+RX5s0CCkjpj/8wrHac5h4rEATV+Q/hiQBuy8DfuQiP5Oc+1RQlX1TufKooZYQ051f4WX2flqU9qJ7q6yOcvAiSkmjQZpYWOvxfCrrnizL5pgAXEjnzlRthwmtyH+jV56MrABm1S6in/bTv+icwEooTxZ7Xj09V8v9eeqL+RkGLCDbtze7ljV//BOCAUTc1/RZaoceqsbpzG9uPmJG2QLFOD6vI7RZIajQW4OTQ+KZEtxSGiZ/e+JS7kahXGDpmwnjOkXhXVaY4cxpkjMtPV3KK28iQvHXF/hoj90pFDzRBfMHJ6HPemdR7JpaNaAb6OZCfgbnjLn6IjMaiiDa+/ZBySuEJETyrALpF93Nj+s+O+ntLhH/AVzK0wozdxG+SodlRIruoul5DZMOS/tWnBhKru9fjO+H0n7C0dRtYrU4RWKXgnpbJ8XXpvW58vQGpT73qEkiPlyv+HwInsKtZEKcbiatNjGOxl4OZf7BA+9IuzGe2tJuOzu8z+qQZ+PLM5lG/ovL2beC4cm5rU6LIv9lpUYpucNlYAoZZMCgw8edP/+krK/O1evEDNvpP+yNAZDI/fnF9t19g2WgEkr6D4ae2HHAMUBBukqKjuXH1W++mIJXhU653zSjlD5uaYR1PHf6Scmjg2q72pVIiJYff5+Zj7K6Ajks72EXoHyVPBe0HCewvrr9lQpQ51L61e1d7YXfUasN1XmzCe1QPpduRL9UvYnZQXXLclpueNqGn8nKO6WhKiE2BN+UDMaCGj+Zgbran7exfDjhgtxi2unh8mpcwddkPHe8bpY5UTThn+ozZzokAILbf83pmHb4nBjrE+A1TOQgWGjUt3aBVH2D0Sg7txJyP060hfbEAvhkyZbvd4B+9z5gg/4qT0+DWXEkbF5S3OpRwKblt1sMnyfrQqdYDUPopeswO+/ues4hTfrp0p98P6Ztv8R6RcpFx4r8ASxkMGI4RQ9CxjGxW3/J6w6ui6+HufT3W+7PFFYmB34DPRL44bAbFbo8zwhTsENWWnJv9dTmOrYlqQuHAD1rEXS4iqEFlgPawNmjGRDyqJVprKfkjVvcxSDpnhmmAdtlFjX9AFN5e0FhU+NU2oNI2C9y4y43EerlvxIzLPzgYtKaYHIzhWGP2vr5geVmqN7/JQieZgeaPTDmg73TUZRxhL6TqTLVCAJayEJtzZ8tMAieymsG4qFuPECN7miGg57HTdnhwCZqnqmpoHRl9xXDuXcXgsccG1imXJ8b3vtRvUCiA3d+VMhYFAOIGBOrL87XAYNlV3hI3iGSylyLBe24MRT78f/WYcb1/Jq0GV6nIn//6/AIy547vuwsiYDHi988XV+16/PdkM0gQ2xIT+Z8MN4INTfKbTOXiwZv5uQhkccSsVYtGZS1cPQN4ErJJcOpzEAD+z6BEZwdQFCcCscJFVeqhwgDqbCbf8TROcxfxv92w6f8uphBdUbyIaLm3Y/+M8f7A== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: FvaRKkpSAsiVFUJIZuumIOLhxNft/fsR5Q2ihL1/hzM8VFbMbXEzbUfmlUOYRsY+VXWep9I3dlCyRteL0h+ZSdDIbJM/oDhnlk3UhU8e1U1dJ8XV2EIc4IOy0R0kwNFhquzTkhObjOEufK7WhHNTemv4OgX9YZ1VDVPAWmNVy7FhQqDmFQ8MyCIjKKSpiH/LQkt+B4EDl/crla1uAZTuTiyyNvLjPoA2iAwcZ43+Ubm9cP5g84td5tcpog46ekzRJ4McXM5rkRGJgguaC5Ys2Ku08OzlpRW2RkuyPxuhqvrlEiaese47hM3Es9/HfbKc7EJ/dfK69uHkuuuBgSJx9bn3MYRSkjtVmKxCcNySOtBc4rH0frw/X5zXc4frrVCRgwTREAhnuszvhPADruxlAQRIXuSrV3CVZNBh6ypOqT+YqB4sqjpvP1rcpivwnxnuGRLPjtvwS9gh82RxwhxjBEBHEzb0L9OP2mEfbe7BqcLQIxMJAeT3lDb6XR5UmXOfYk9hnguJkGd9MwhwmaZAvyCPBCscomtU3i1kNHx5y5wIKIax/WviNNwgREdx8aZLW/ihKpclUtiGmWC5TY73/fNuwBbvHcAhTsfdstM/G0Qboz7Fo0Dt+b8TbbeSVo4NP7kq066o5TyFs58pIVzkvhAZENA5Tb+QsYtHRgqVUW8z48DqAfGCL0qU79uM9BaYMUh47yhcj0Mlv9RthanHtkLun57AjLwpjkCEuXi051DsRRTsMBfNju610C/8kzLJiQvA5e9sQpx+2Gz879AtTouaz05viywQe3YAy/oIJA5o2uzMn9RqmyMcW3jf2eRuXiUt5kEXSvaqO79AS4EHKSFKd9VvUdbLl4hR7sK4vAU7N+U69lfEq7jim2LwUhyA9coMDdBzX4KceZnIZaFpVG26GSKcuFlk1nr0Anado7InFQMmrbFW44bxPHxhVcqxdpAGOxQJ3DU/Sis/QFpE9C26vPqIeAPohZpO0IeAPDDNohTvf+WY4J6lvlanYu7ccDxbU/PwqvoiGG1ccgxpyPdTTT55iJYdnneNCZ9SoqV5PXJqoLdVgiA+IzbOo151K85M4Pay5BXbSSIW9xrQKpgF76iGwbBg4SDjwRFT1HDcrcHIraEEuKIjBfod6l4Y X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: a2751397-666d-4a53-5270-08dbbe21ff32 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Sep 2023 23:49:03.0562 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: u4eGjld0LxFsIGAYClQpmaOJZZC8fvwoILCsx4UKbAQggmJHiK7rcdol5JvFvzt493qbrp+5MnEIhevgI3pANQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR10MB6730 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-09-25_18,2023-09-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0 suspectscore=0 adultscore=0 phishscore=0 mlxscore=0 spamscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000 definitions=main-2309250185 X-Proofpoint-GUID: hvMi1pojaw9HmLZcxevzTBP5YCmST8na X-Proofpoint-ORIG-GUID: hvMi1pojaw9HmLZcxevzTBP5YCmST8na X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 7E00F14000A X-Stat-Signature: xdd9gbohkamdnt9cbsrcmzo9jaa9rbze X-HE-Tag: 1695685776-47438 X-HE-Meta: U2FsdGVkX1/c+shxlJ/vyinGqld4kgPASX3Fxtgi+BLPqYdT1La5lTFq0ibHHGm9t+l+IENCNDtToBUGBZR9+JRnxR5awwXVfHsnzts+RFHuNSjBObXDbAZvHCqNMcefP7QVPrpaOaifkNzHBzZMGwV4pWD8jJP/f3OYvXXaNJvpayFduqHjhHEKQmBrxj0lBhMPwzo2Jp+fD7HlVyYpdThIi10RXzO+u6d+kVc6Do9n/SEhHExIAxSuX0ywi/7ZKlqG8PhnNFz2t9htL2lPHBqcIbhiHRMale7pcad7wJFS0RTLGex30qsWjWKIuZVv3THpzperHO3QlpxvED1TuwUddDvcfvaOxha9jKBq3G2ZTV45ROlgppyYeuBmIrVsYY6jfkDTmbbqmGuhCKryqW2VNUPJMSWnrVmGrfRLzEwc3Jw6ufLCwJ/GVpr21t/QMseF4x1QH6opPaRab/2jDmBu3x0nDBGIqsFdDkNWsIJISuEkV2pnxLvvQCg5vFw9wobkWqWZQ2Py3g63byHzYA5897E54MBuxrVjcnAdSDGBx9kPMMcr/BfPKgfZ96K8KywULq7Yit2xMY7FCNa2ZcPgxBfM7gnHnfGddfd1p/27sJt89+sP8dUxOW8x+5WAOpD3TsFKHhyH/GJnF+U4lNne7xYX7y9l+Kobga5zWgjDKdk1+QE7938gpn+WVE1ufYHnWVhojOjQpaM4Bvwga6/TI6dMWONAWOjVF+YzHtTFlAY/0Fgnzj2I28hbAzZXkQ6G06OjaGRuh36HKglQMuJakXUjtJIx9BkRlkXIMFjwOZ/bsCnwjfy6xjSTwV+kjoIYhfFkv+t0tQLZRxX7VCQ/AXy0qOswuhOu6Da0u/JBRfwAE1qcmODTrhN0dugiGlCMCYqF5i6fwuU5T4FfCnc9KKNEFYm2sFX4mzteCePpr3cJ3s3/PBjmiJc+zIM8AFH2T0F918ioE+nYQiY CTLp0nvb KIndMzSsng/IlZcrazerIXKYnI4TyXVhXNeW5ZVfqdBvSDwUr4sJe4y4KlZVtjHoiLxVORD+lhVwWoPmsMcHMdjhM4pvEVtU9Kv5bRdJPTIFmYkBXsPzLwtm+CLEnuiSXsJ866kdhxR314j7pwSGrE2tk+O2FxAAY0jmMrAzealcCkwXyxEYNWgQu6zGc+uyruMXxCe+7wmCChllyBaaO6OrlwlZELj0aBmSnSrSNAY7atDls5CUBNO5WkrxL8N+gV2AmRg+JPP5XAb88Vlxmi6/BebDqdzdQb1lGUHKtHR4PiFO7yFfcGDk8VNvI6t4nhVhq170jTH9H4/95cnezCLQvP8LlOkzVAmN9cfIyEMR5ROJ5Ajl7wusK4NLs3s7rV4nQ6cMmVGedgbpFTGKXsmBW4lt7aaCDwSIgjgknJxvVdJ1KwGvtrRwEklkptP4gibufKOhuNjhqxm0Y6mUIZ5T8ehQcYnF9Fh3q4Y5s0KywHUIiCDrP6Rs4Evq5Do9nPk3vacNDTWNRrMWwjGZr8O/2eCpgragKSYEqUFSdSc3Cmea7fIzZOLR4jpZhUBr6xcRoswcZlTDXNDWZU+edQ3kpYyqGWJdkSzlwbDVtHKXHxfsD1N0A5HVcpA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Joao Martins In an effort to minimize amount of TLB flushes, batch all PMD splits belonging to a range of pages in order to perform only 1 (global) TLB flush. Add a flags field to the walker and pass whether it's a bulk allocation or just a single page to decide to remap. First value (VMEMMAP_SPLIT_NO_TLB_FLUSH) designates the request to not do the TLB flush when we split the PMD. Rebased and updated by Mike Kravetz Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song --- mm/hugetlb_vmemmap.c | 92 ++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 88 insertions(+), 4 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 4ac521e596db..10739e4285d5 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -27,6 +27,8 @@ * @reuse_addr: the virtual address of the @reuse_page page. * @vmemmap_pages: the list head of the vmemmap pages that can be freed * or is mapped from. + * @flags: used to modify behavior in vmemmap page table walking + * operations. */ struct vmemmap_remap_walk { void (*remap_pte)(pte_t *pte, unsigned long addr, @@ -35,9 +37,13 @@ struct vmemmap_remap_walk { struct page *reuse_page; unsigned long reuse_addr; struct list_head *vmemmap_pages; + +/* Skip the TLB flush when we split the PMD */ +#define VMEMMAP_SPLIT_NO_TLB_FLUSH BIT(0) + unsigned long flags; }; -static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start) +static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, bool flush) { pmd_t __pmd; int i; @@ -80,7 +86,8 @@ static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start) /* Make pte visible before pmd. See comment in pmd_install(). */ smp_wmb(); pmd_populate_kernel(&init_mm, pmd, pgtable); - flush_tlb_kernel_range(start, start + PMD_SIZE); + if (flush) + flush_tlb_kernel_range(start, start + PMD_SIZE); } else { pte_free_kernel(&init_mm, pgtable); } @@ -127,11 +134,20 @@ static int vmemmap_pmd_range(pud_t *pud, unsigned long addr, do { int ret; - ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK); + ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK, + !(walk->flags & VMEMMAP_SPLIT_NO_TLB_FLUSH)); if (ret) return ret; next = pmd_addr_end(addr, end); + + /* + * We are only splitting, not remapping the hugetlb vmemmap + * pages. + */ + if (!walk->remap_pte) + continue; + vmemmap_pte_range(pmd, addr, next, walk); } while (pmd++, addr = next, addr != end); @@ -198,7 +214,8 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, return ret; } while (pgd++, addr = next, addr != end); - flush_tlb_kernel_range(start, end); + if (walk->remap_pte) + flush_tlb_kernel_range(start, end); return 0; } @@ -297,6 +314,36 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); } +/** + * vmemmap_remap_split - split the vmemmap virtual address range [@start, @end) + * backing PMDs of the directmap into PTEs + * @start: start address of the vmemmap virtual address range that we want + * to remap. + * @end: end address of the vmemmap virtual address range that we want to + * remap. + * @reuse: reuse address. + * + * Return: %0 on success, negative error code otherwise. + */ +static int vmemmap_remap_split(unsigned long start, unsigned long end, + unsigned long reuse) +{ + int ret; + struct vmemmap_remap_walk walk = { + .remap_pte = NULL, + .flags = VMEMMAP_SPLIT_NO_TLB_FLUSH, + }; + + /* See the comment in the vmemmap_remap_free(). */ + BUG_ON(start - reuse != PAGE_SIZE); + + mmap_read_lock(&init_mm); + ret = vmemmap_remap_range(reuse, end, &walk); + mmap_read_unlock(&init_mm); + + return ret; +} + /** * vmemmap_remap_free - remap the vmemmap virtual address range [@start, @end) * to the page which @reuse is mapped to, then free vmemmap @@ -320,6 +367,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .remap_pte = vmemmap_remap_pte, .reuse_addr = reuse, .vmemmap_pages = vmemmap_pages, + .flags = 0, }; int nid = page_to_nid((struct page *)reuse); gfp_t gfp_mask = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN; @@ -368,6 +416,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, .vmemmap_pages = vmemmap_pages, + .flags = 0, }; vmemmap_remap_range(reuse, end, &walk); @@ -419,6 +468,7 @@ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, + .flags = 0, }; /* See the comment in the vmemmap_remap_free(). */ @@ -628,11 +678,45 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) free_vmemmap_page_list(&vmemmap_pages); } +static int hugetlb_vmemmap_split(const struct hstate *h, struct page *head) +{ + unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; + unsigned long vmemmap_reuse; + + if (!vmemmap_should_optimize(h, head)) + return 0; + + vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); + vmemmap_reuse = vmemmap_start; + vmemmap_start += HUGETLB_VMEMMAP_RESERVE_SIZE; + + /* + * Split PMDs on the vmemmap virtual address range [@vmemmap_start, + * @vmemmap_end] + */ + return vmemmap_remap_split(vmemmap_start, vmemmap_end, vmemmap_reuse); +} + void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) { struct folio *folio; LIST_HEAD(vmemmap_pages); + list_for_each_entry(folio, folio_list, lru) { + int ret = hugetlb_vmemmap_split(h, &folio->page); + + /* + * Spliting the PMD requires allocating a page, thus lets fail + * early once we encounter the first OOM. No point in retrying + * as it can be dynamically done on remap with the memory + * we get back from the vmemmap deduplication. + */ + if (ret == -ENOMEM) + break; + } + + flush_tlb_all(); + list_for_each_entry(folio, folio_list, lru) { int ret = __hugetlb_vmemmap_optimize(h, &folio->page, &vmemmap_pages);