From patchwork Fri Sep 15 22:15:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13387789 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17D9BCD37AA for ; Fri, 15 Sep 2023 22:16:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A21086B0396; Fri, 15 Sep 2023 18:16:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9A75B6B0397; Fri, 15 Sep 2023 18:16:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7852B6B0398; Fri, 15 Sep 2023 18:16:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 638146B0396 for ; Fri, 15 Sep 2023 18:16:52 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3AF1A41054 for ; Fri, 15 Sep 2023 22:16:52 +0000 (UTC) X-FDA: 81240242664.17.E6DA715 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf06.hostedemail.com (Postfix) with ESMTP id 3A0C518002D for ; Fri, 15 Sep 2023 22:16:48 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=A7BjxVX8; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=hekVPJiq; spf=pass (imf06.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694816208; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=i7fwZVFpbWQwFa0ZuX0bGyigYcZrLBRp0FddpocwtE0=; b=GGx33+VXrX99vSSjZWBoQUbOVg9MEdHHqTxulGidARSEDZrOeZ17kN4PsZ3fcfjtJoTOR8 eTe1z4A7GE/hviSSn1EmxsOH4d6DrmdwinXu7ZdUpQr+WOebiHRMMl9FuKitaWvVITYbSY IUU1nYBOYsCXH5Cq8wdr6mY8OuB6gUE= ARC-Authentication-Results: i=2; imf06.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=A7BjxVX8; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=hekVPJiq; spf=pass (imf06.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1694816208; a=rsa-sha256; cv=pass; b=JTB68yiZ7+qA37mRj484mSo0yC9O+zpa004vDEiJvCnfVWSdVFXQgXmMjUNULeMVWIQfn1 Xb8gWSZjUsLGVf9gsR4n0Lsv74JvfNcbvvFoKkNbkTPGyw3wvuOYFjSMILbyhyHy9qNcCK rO/IGX1rjlx0Avb9uenbAqegWxz7Nq8= Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38FLxYgm007900; Fri, 15 Sep 2023 22:16:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=i7fwZVFpbWQwFa0ZuX0bGyigYcZrLBRp0FddpocwtE0=; b=A7BjxVX84mtGamQPYAdWr/fSLgiivgO9lgcMX3tj7vM+XWzaMwSackgQMX5h8NgHAipk Wc6Mnq4kGFIIXetCANWRqJ40qW6/h0WpvQmqKdPfudls+71INbjmGYLQD/RdPyQ9qtVG 6dxxledYOa9WnD02QPhn5D7s49PkNqoCQvaQtIYJurKrFlNdnRrZuE7jNTns/2+Q24sl uq6u5xwgrqvlfxy1uHFDdrGRrlh+eI4I5EllboN+E830wqSQ2smCipGT1l11x3jBvqMC HyB/aVvDBEdyexP5xcoSsi9X5XF4OBmWZ4LGmuoJv2jofcx871pfIZEplKmTNNSjHau0 NA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t2y7hgsw7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:23 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38FLT2W6030436; Fri, 15 Sep 2023 22:16:23 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2174.outbound.protection.outlook.com [104.47.59.174]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3t0wkky7bg-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Sep 2023 22:16:23 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DoAJ7T9CcPyCNGrhpHLb5UQgyUonfGeo9Sj1CDdsQxK7iDzywDFjknRvHrkTg+UGAQQIdQ985wryxriioTOU89F3tIueiWcxtbfTyTa4fSW6xkvC2cnU7yzrNzs9V0QfxIC4kfS9w142qerkHKNGJw3rDy/cYh2db3byYStmxIUYt1xq/zAAUx9pTbn719+mJi+YMPMusZhJB/fsjgQHHVAE7TLoHmf/aZT0+PvzXp3cfsjshQt9hFRuT16VzvqbLS6gM1a+kc4CEvmF1cH+tQgw5OyF0BJkCp2nlXXU82kgvJTocZ2a/huhFrY05KVXWho6667O3xDETTww0uM9aw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=i7fwZVFpbWQwFa0ZuX0bGyigYcZrLBRp0FddpocwtE0=; b=BnzzgRRJMfe2HYtAphmEhWidmSGhwDSKYDxVE2Jb5C4wMRDZ1I9AFStG+Qwyu+aCf8mJxxVxwFU0Y7ZT8f4M0/SGC30PU7Rgd2q+11xRrVUe9DqkAY2Oz8o4gz7N9MOlquSkPjQJOKdccLdJbclR5cc8gSvCIa05dPDiGNV9VtmXBwQBWfg3jUc+Q0lE1uP6C5LWRS9MwQyDLu7gHtNJjRPL2l+P6rNtf9iplncfxCZ4rfpct1nkW9LJ85EZiTrv8Haca7GVNUMjdJwQm8RvNGOmjN57g2DRn3O1noCNSX33p3A6ahurZoduHmUJATCWHNtkPMTK6e77OEh1qPbsZg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=i7fwZVFpbWQwFa0ZuX0bGyigYcZrLBRp0FddpocwtE0=; b=hekVPJiq3PNVpaYka1ea+OkS/95a+PRwnDEj11aAVso5UyVV4mHvWRyl4IEDxyavSEIZAKaiSHax32kcDfYyxFC76BR5gu+1MrkQ14K1NwjP+20m1JU5FyqJyaIqUXC33C9owIONZlO0TFAmsoH+NOHVa2byqmw2Pgcve3cPFzg= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MW4PR10MB5861.namprd10.prod.outlook.com (2603:10b6:303:180::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.34; Fri, 15 Sep 2023 22:16:20 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.021; Fri, 15 Sep 2023 22:16:20 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v3 10/12] hugetlb: batch PMD split for bulk vmemmap dedup Date: Fri, 15 Sep 2023 15:15:43 -0700 Message-ID: <20230915221548.552084-11-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230915221548.552084-1-mike.kravetz@oracle.com> References: <20230915221548.552084-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0015.namprd03.prod.outlook.com (2603:10b6:303:8f::20) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|MW4PR10MB5861:EE_ X-MS-Office365-Filtering-Correlation-Id: 83779850-a1f4-4c4c-577a-08dbb6396361 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4FYJdwDnKtduyUk9VDhYRSzya2gG/SN7La5w3Nwmq+l0ITUDSkKsrFRBUYcTM9TWk5hi0LBnx1CKzEwS95fpumZvJZzIp84YhP1lgJSsh+TZmVYpMFOkgwEEaJr7txEtA9PPbtgT66EiEMfLI/jvM/Sq3641aWakGYfetkA2yseR+1aQK7rHUXgGIBdBwLUOY4iXDal/d653okPLyR8crhcEGI8aZYKIxPFiOBNHfYuk72jpzvnzH8c8avd9rku7hvW0ntUm1uuzRlURnxyGOFFk0okPOKt9h1B2Gon8+jcig6+EjWhPzYqaZU/bVulyyjArtmQaJIHT1xp2m/SCSWQqS9zkcVnjPFJq4iazqROe5fFsfGNATGtNrEMPNV/HqPAzYQZdcbZV2UmG74Y9K2tt+yMMAxJOmeZU7glBUMChviRLG1QJLBiVtPLthtg9ipHY1SRv23VwafUEZ0TYK6B7uYTaYZ9IpoGVhPMc2aXs0bZpV+DPq/8OJysC4FyLlTj2MPrav02aFVx1yeuevm7tWyptJ5A4vd8L/GOXaWq8iq44UXGMGp2jl300NlLp X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(346002)(39860400002)(366004)(376002)(136003)(451199024)(186009)(1800799009)(44832011)(86362001)(36756003)(83380400001)(2906002)(38100700002)(478600001)(1076003)(2616005)(26005)(107886003)(4326008)(6512007)(41300700001)(5660300002)(6506007)(6486002)(8676002)(6666004)(66556008)(316002)(66946007)(8936002)(7416002)(66476007)(54906003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: //J4X2tl3RBtZ4X5hGZ56j9TFYAm2xMwRlkCea5yrQwPP4mHsRr8whc+OjhifNeEHrD5DBc5Xm0nVctYyP4ynCIOxzUMkcKb4J1Dcb3VwCFXG25/lP/tP2VF0ckSMsnjjB/m1aB078l8vBP9270RnU0tALo/EV8ygV4mrjfLEPVe+68cz68lnOStd0CF17kLOcHReTjGPm2G+mnjZexNRr9QnYvw6Jg4tzVuE5VK7rxQf5TTOLPMazCcIDTbuc5SnSsqpqZINPCAsEVPV6+GIP4yqPYXs+R64JvQREA3+89iHbtCyzf5XnBbqd9f4faQ6ckTayesqaqOkUCAVXClbAkAqQgQpNQBIGkCbp+AkIS5lj+u7vpuj0d+uVEV1Sn4s5pt1gu/gw/7JE2U+/d1gTuy1A7zKOI2bpS77daq0tDcOJTfzm/jyyjAbdoCRSltkKEvvwrJjtSYustJUnp3rB2y6NDl143wdiWhtIRijrDaQdL5KwNC6nAH1SS7tpeu0tqtY9iX2uDoz6UP9nhwl+NuFNoeXj+gCqDUyE431CvssRHc93Lpwu8H1iRtT1Ss1PAgJO+Q2eEqp0bX4gSOQbWlKrsb2IGSXSZLAGdoApJw7jBSZuzVZ/ZBhZWggcgxuRGRNJUrQbt6r/lxnQIQM21lCUOYgt2XBSbnVy4jKJK5J5cDsZCDEOrDTXjBhvAMaEs8Je9ZjVz9UXFwtv2yWTNZWJlCEru3OK6XEAYCJZoP2UBRZlFQiYnu0A6q453ix+/2lYgUTtSXLWi1AmlHN0LumUnSzxV8cIvcrCmScZ9lKAgnJMisgGex39IubGW5no+uIsDljhDMleieqQxHGHq/2SbD1/eDxYgxqcs+btT8LnXv1ZpHcV3p+Jy/mPT1HJpnS5fJ1r6jDhxp59yvXNc6Z/RIs2ui+dRjYo/GX5BBJXfG1rr+zoWiO8rYcjHNO6rYE+DjXNIYqwl2cpHPWcLjSmONzgTTRW/1Z53f8U6ZvFPvAu5QIxIktRraz9m3ZL6fRooTiatO7/33CeZGJApIyoPJBgrlZzqJIIMftyCkO4bNrCw1QNp8X+jfhFQOvTZOe+fhh7EDjhTjS6amjQ3728WYzdiGETgeZjrPt+jGf6L1PsSDhhXPSr0ZukWa+UbBxWe2i+L+nsZgx7MKAL2/ufBl4M5cXh1BAUx2eIwfKrAlSUMIyj06asenX89jYGBnZBX4onPLfWEFqRup61hHamSV+FlYrXAIZ5ioAuhGALZDgr7jrFDTvP1is//fLvHHIs5acDVWHsf5GmsFkPE/hfkEHrt7Iwzb8/EvRKBC077Gi4/6IQVqMRCsrWOsvDDTjg6KeV4kBBgKHHmHSK8Yv+xbH/GDb6UBEj8JFZ4YYdTWWLBB/3jGJbDgSV5TeJHzMKVOe0u5Sa+3UcKNrUAzVyfNedtGQINOh/A6QxUX9RiUI2NL1j4Q/wL84ndC2utqNpCkbzc0R9IEsWA72DlJG2npXU08u9zikN8omKOsJNvIvi/+zNGNweR3k1hrYSsyupQ17n8T4mkVxQdefdlTRVMAnSRWctfuDJ4NT7R4UMGjZcwPMWHhMAt2fJJ/TZ+ah1HLlL6ZJhMgXHI8iw== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: pjTQymCEGuf2VPI3AuQgZtjpfeOhmdbpvTlGPQq/2a1aU0m5FwH/L0/RuPBZxQBIchVrDxNF9iX1Y39GFgQw7JJ947opYwVYfutyx2nRtUrFUAFmQlLYi/nTR7rym3Z3H1J7xVWXqqPIU6eepZ0Am/e4n/X+F8cNQtOfGSBMYYSYuVwXtPytj5thx1mxkie8xxRDYU2SbNkT1KBrA8opWbqxHxtaXll74xUuC7L9y7iaBslqlWONI/vRAwfC8mTJIsZ7GBaoBL7QQlbVGxjJFrVk2XvaKz1XMw/hwxAX1j+kKiW+cilhIIy5bMCauShWcZ75CBqu+wCII0w6PYAbH5az+368Y46ovQwLbRalaQAUOFtJTFnKvnsUaxaJyhNNGS+tENrv8gecktT5JAKN4tJ8GETzK60hIhETTgNyGzlO7gzlvJAPXCJK1chIieYtJCGmOIbMw6xUjsAq/9r8m/PE0Cb+5IplhvDV6ui5hRJHrISWl9VwoaeIYyhgOyS9c/cPWXEJncaNlJwIq3URhgbvq1vtr4wph8NlBh+CVRd/ujVm7x4uZax3frxE16tN6pIZ9cUpswHNrb/6hla1TqCYiLxjTHqdIyXsokXrx7SljwMcL8uh+eo092hPFruVieqt4V45aSzIxVtJBJJPy/poUa4dy6WSgGfbGy5rRhmP1E+zvUZp3soeKQFCmy236CA2WWFHq+dE8SQzLfxux69NdJaQ8w3ScZ/lpceaTPCUjgpL5H1IZPIp4YkT+Ora6Se7e2gilsJT4McFjItTqHNJrBLqVXGf5OdkUUNWmRXDSw1NiK1D6RZHieSFUm6jiWLzppjgxV4XZm51pF4OhIDWjaNTXBqIpq7kuJrJlv24QiT8GAXQfat7SeEKcnDwMjBxbXfs8Lv3zBxusJTwMcLgMlE7T636obMiGx/v9mN/PV3Rl9UeJ8jnj7oKNi1EV/tFtsz4DtMrvzmtPoXXeMEKp7LqS2XjsRWuSTPYfhInWlSerH5M+SEIEGnBgQzAbXS259U5afF2pdzV3GhG95mfHL16ZZQDyagiKwdSkhdTjK/qELN59lSBPA5O3VqhaeNmoqtYd39sjngjZW8hDjm7Z9OBD3u3eHiu7TpWf8IB5CIyY1EXtQgsidKFUC0U X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 83779850-a1f4-4c4c-577a-08dbb6396361 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2023 22:16:20.1263 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: qEQa972aHiADMUJ74b3mxrCowS/6u2UmJPGdchF7RHLsEpjugpMMJ/mitw6Izzl7CjqhHFmpOxc4QLcswcYXXg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB5861 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-15_19,2023-09-15_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 adultscore=0 phishscore=0 spamscore=0 mlxlogscore=999 bulkscore=0 suspectscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309150200 X-Proofpoint-ORIG-GUID: KuDrkyMSSY7FmhcBC3uRDuUOa8b8d87G X-Proofpoint-GUID: KuDrkyMSSY7FmhcBC3uRDuUOa8b8d87G X-Rspamd-Queue-Id: 3A0C518002D X-Rspam-User: X-Stat-Signature: 9a1awhgsebukcqzq3zktcxp9di96i9d6 X-Rspamd-Server: rspam01 X-HE-Tag: 1694816208-72936 X-HE-Meta: U2FsdGVkX193xvsDZTe9ROKeKuTY6Gtxh886XcLbvd5NEz0tSLUHpRWgpz5+Dlqii1FquhPtoJyJ0kSfsACxlxgdq+Qi+z6E72NyYfGrkenDCscE03mn9dsKicD6//xOCvA17igzkYAmppmxzkbv/p9LR/k9a9JuYFBPzYErCs9ILR4fYRE3ASUr68uUH0NUN0ntbd6WM0L9JMZsEwGctZlhIMuYE1MQrFddM1eVF0cEjQkGlmyF5I9f7BkdcNW4v47QFhHW71dlZAsJZcNiX91AUKj6B8IvsZ62WBEg5ncLuCbDRBCh+JkIr0B7PZRGH1GfbOgXzkMDhSIHG1fRFmS8YL+Q5z1LCbDSkWVUwWjhOu6jad9bkqrmbtOiXLPV8qtVc4HtbX/LJol/cIIA9nS2czHr7DVYdPZfrGHis8/gIx+R0Z7zU684qqUqMVNaXZEr82dsieGhwlw20nb2OATtWdRPvKXhV+pTgUu/e6kTvv1j5DljsRj9+YInRvW5JxZM0gSI7HRtfOewRXBcd09RFeVq9CEV/hGtlmTU+VlAAzrhULC/kI69a9T7cvT0LYPUKSw/6XNrx3OBc9ZOdiJhiwe2ciESRpS28dNIxQz4UiBXWwLN7BIur7HYeHVJZUO0b8O9Qy3w6EJ2CYCC+/vMy7G8LT+zJ7HfMmxvF+8IaBqv4dR4tYdnOnjY26dp4Q4RS5N4ZHBds4KmIHecv0jjStgXTK0mtCIyPAGcFvM8XIMcXWsUFjrAw0OSsMtJYGpylIvb6rejD6SzOssIACQKhodWtydAazrS8d5bXxYCccxRIMIlX9e7Y66d+fQywKJyNjp7vzb24bWhoeInO3grMPpH3bxNCV9XSIw71IaFm3d9L76CEOaNjFrMKCDLC1zCLQUl0I4ykTzsmO+GH7IcqyWSK0yqDgTKuZaajAf2fwBjP9kQhcH+N1+cnTFltV1PoO1EmLCnh7tgJ9D bsrasKIy arftV7EGcoQqYvkIArXs7z1MaO4y++Nml1miRl9od2Qxjr5SCqqIVqGXOPYFtCbhVdAh5hiptiZXVYv9AS8jL3H/7e4C6HFSLAnoGOt+ceRVg7bcEI1rta1swqPxGHhZ4Jzrk5m9BlAC0ewQWB6VCZLCKySbgzd/oZqZoHoqU2mXQj9hRFdlQsQ+hYExw6NsEvh0Fi1bC4HttB6AoWVtH9jCrzsJKEtU3yTcrPgpQI5huobxqvX6Hn9elbtSKzW/EmK22xqfT2J7qaU6EScAU1ar7RINfmem0MKiOR0pTcZD7E0M73owab5H7g1ABDAiG100xK9sIDLLsnrsxMVBtR8VnS0NQK0IKobEKdm2regmqwwEyWrmM5wKvVG13R6U6oBPEBVFZ2kmbGdWbrPxRUH1fq5S9hmqrWVuAPb2nwARoCCdLy52WfozyN5sT2OKiIpARj5w53KGib4IB0HYbqxwUk7QyKFdKps/CZA2XHmBeQPxdcj5lbzGtEzXNIoXYIpqk X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Joao Martins In an effort to minimize amount of TLB flushes, batch all PMD splits belonging to a range of pages in order to perform only 1 (global) TLB flush. Add a flags field to the walker and pass whether it's a bulk allocation or just a single page to decide to remap. First value (VMEMMAP_SPLIT_NO_TLB_FLUSH) designates the request to not do the TLB flush when we split the PMD. Rebased and updated by Mike Kravetz Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 79 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 75 insertions(+), 4 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 8f8a559ff6ac..c952e95a829c 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -27,6 +27,7 @@ * @reuse_addr: the virtual address of the @reuse_page page. * @vmemmap_pages: the list head of the vmemmap pages that can be freed * or is mapped from. + * @flags: used to modify behavior in bulk operations */ struct vmemmap_remap_walk { void (*remap_pte)(pte_t *pte, unsigned long addr, @@ -35,9 +36,11 @@ struct vmemmap_remap_walk { struct page *reuse_page; unsigned long reuse_addr; struct list_head *vmemmap_pages; +#define VMEMMAP_SPLIT_NO_TLB_FLUSH BIT(0) + unsigned long flags; }; -static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start) +static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, bool flush) { pmd_t __pmd; int i; @@ -80,7 +83,8 @@ static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start) /* Make pte visible before pmd. See comment in pmd_install(). */ smp_wmb(); pmd_populate_kernel(&init_mm, pmd, pgtable); - flush_tlb_kernel_range(start, start + PMD_SIZE); + if (flush) + flush_tlb_kernel_range(start, start + PMD_SIZE); } else { pte_free_kernel(&init_mm, pgtable); } @@ -127,11 +131,20 @@ static int vmemmap_pmd_range(pud_t *pud, unsigned long addr, do { int ret; - ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK); + ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK, + walk->flags & VMEMMAP_SPLIT_NO_TLB_FLUSH); if (ret) return ret; next = pmd_addr_end(addr, end); + + /* + * We are only splitting, not remapping the hugetlb vmemmap + * pages. + */ + if (!walk->remap_pte) + continue; + vmemmap_pte_range(pmd, addr, next, walk); } while (pmd++, addr = next, addr != end); @@ -198,7 +211,8 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, return ret; } while (pgd++, addr = next, addr != end); - flush_tlb_kernel_range(start, end); + if (walk->remap_pte) + flush_tlb_kernel_range(start, end); return 0; } @@ -300,6 +314,36 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); } +/** + * vmemmap_remap_split - split the vmemmap virtual address range [@start, @end) + * backing PMDs of the directmap into PTEs + * @start: start address of the vmemmap virtual address range that we want + * to remap. + * @end: end address of the vmemmap virtual address range that we want to + * remap. + * @reuse: reuse address. + * + * Return: %0 on success, negative error code otherwise. + */ +static int vmemmap_remap_split(unsigned long start, unsigned long end, + unsigned long reuse) +{ + int ret; + struct vmemmap_remap_walk walk = { + .remap_pte = NULL, + .flags = VMEMMAP_SPLIT_NO_TLB_FLUSH, + }; + + /* See the comment in the vmemmap_remap_free(). */ + BUG_ON(start - reuse != PAGE_SIZE); + + mmap_read_lock(&init_mm); + ret = vmemmap_remap_range(reuse, end, &walk); + mmap_read_unlock(&init_mm); + + return ret; +} + /** * vmemmap_remap_free - remap the vmemmap virtual address range [@start, @end) * to the page which @reuse is mapped to, then free vmemmap @@ -323,6 +367,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .remap_pte = vmemmap_remap_pte, .reuse_addr = reuse, .vmemmap_pages = vmemmap_pages, + .flags = 0, }; int nid = page_to_nid((struct page *)start); gfp_t gfp_mask = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN; @@ -371,6 +416,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, .vmemmap_pages = vmemmap_pages, + .flags = 0, }; vmemmap_remap_range(reuse, end, &walk); @@ -422,6 +468,7 @@ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, + .flags = 0, }; /* See the comment in the vmemmap_remap_free(). */ @@ -630,11 +677,35 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) free_vmemmap_page_list(&vmemmap_pages); } +static void hugetlb_vmemmap_split(const struct hstate *h, struct page *head) +{ + unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; + unsigned long vmemmap_reuse; + + if (!vmemmap_should_optimize(h, head)) + return; + + vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); + vmemmap_reuse = vmemmap_start; + vmemmap_start += HUGETLB_VMEMMAP_RESERVE_SIZE; + + /* + * Split PMDs on the vmemmap virtual address range [@vmemmap_start, + * @vmemmap_end] + */ + vmemmap_remap_split(vmemmap_start, vmemmap_end, vmemmap_reuse); +} + void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) { struct folio *folio; LIST_HEAD(vmemmap_pages); + list_for_each_entry(folio, folio_list, lru) + hugetlb_vmemmap_split(h, &folio->page); + + flush_tlb_all(); + list_for_each_entry(folio, folio_list, lru) { int ret = __hugetlb_vmemmap_optimize(h, &folio->page, &vmemmap_pages);