From patchwork Fri May 27 22:58:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12863872 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2ADE4C433EF for ; Fri, 27 May 2022 22:59:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0963D8D0007; Fri, 27 May 2022 18:59:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 01EC58D0002; Fri, 27 May 2022 18:59:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB0B58D0007; Fri, 27 May 2022 18:59:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id CA5F18D0002 for ; Fri, 27 May 2022 18:59:13 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A431020D1D for ; Fri, 27 May 2022 22:59:13 +0000 (UTC) X-FDA: 79513040586.11.035183F Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf03.hostedemail.com (Postfix) with ESMTP id 9768720028 for ; Fri, 27 May 2022 22:58:59 +0000 (UTC) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 24RKldMU006274; Fri, 27 May 2022 22:59:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=3zEOXDOnNVDX1cmb24BXpclDgrucJjzYv60skjVuz1w=; b=WMD0C1dbAFIJPBuBh/Sdm+Qi14EjYr4+N7c7cwgYnCtFZahFTipWPMlU88DhKPsoL0cZ KnSZSF6YZ24IFNCrx/ADeLr8SQmNlRe5ypSfFmc1AM8oft9GXl4MYRGadJGtl3/ZuciU hP4GIaLftcld66wKzd+H+vVZV57I6pvubE3k7QG8g6b9HCiS0gveyQw+VFbJSSthpSCS yFSUiI+xaw/Tjx4jVAgvViSdJ0/soghAQSLTMkcEj9yquYZ6npDyMoy4OYn8ejbd+A0t 3nUdtGBqslzzNwhn1np42Q8T2I89ou9qm8+alDqbli0X3/OcUa44ngktk+S0IIXi1Evb Hg== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3g93tc8cex-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 May 2022 22:59:06 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 24RMoHiH009606; Fri, 27 May 2022 22:59:05 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2168.outbound.protection.outlook.com [104.47.58.168]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3g93wys19p-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 May 2022 22:59:05 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=drJBCzgf4OQAbPL2G5T8lzOWjquKOHWhtg6Rn0/jpMHFTE9DfOYl3hHUtKM0bX9kARIydyZ1kjrYMcL0WiV913ImkiNnCzbMPcF1yWQrShQZ+YLaq6h53VIeUlqIeD320S7vWtA0zVofOsfEBWcj5ZbAk8ws89aW5MQb5H5PzCJBNaS4V05oUi5X1U1229KdV+jFtxIZy4Yp/NO5lIy2edvvId3TMCfrc9QNIoQrC1TgAJAiGv7mXTAYD5vWHA/yCr2KWkSc1xxyl8HhPD+KoFM92Yq0nhr4wIE5VYGOBibJznvVDvPpabzaJ5noR5+GPyL6I7n5J9M981BXGFOcVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3zEOXDOnNVDX1cmb24BXpclDgrucJjzYv60skjVuz1w=; b=IwQ13nSzn9r40iYiyDLQku/McObryMt38vYOtAqz5k8gDnArESAivoPB98UEMTZgsBwdcLA611ILrtt0UGRu6XKhzTvC21e0kVlfbBfJj638GWCH5ShyQJSqhuAN5wk3GWNv68P9/zym7BJX/GWDdhQ9UjxAGiVdPdNsuW4iy9nesCutJBwbI7WQlp5ucf4ME3XpGhZ3hFWUlOYxe4yt1jT/jWvzvlZ/ecVboAEZ+V4b1I+ryBuYgbqVPNYzKEWZIL3w/5igV+8ht64FaRg51uy6vR3pz1/AkDsKTsdrPxfn8XgbY/z+FMzTKIREFTnuGBzWL6AVatcIH7OMHYHhtA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3zEOXDOnNVDX1cmb24BXpclDgrucJjzYv60skjVuz1w=; b=xl0MoRvStaJtP3btr386iQCnY3nY8K+FQbx23BGO+fYKza9nE0ZWDCMnJ5DTcTXbwXpzHpQodCZ+TetaY+wUdrsD/RLVg+PeKpgyosZHKj0rRqNGXOWQkSbQFkYBwX/K8ig1dSWdn0PJZamB2XKEsn3AA9Tf1yv8b/sMx+36xJc= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by DS7PR10MB4831.namprd10.prod.outlook.com (2603:10b6:5:3ab::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13; Fri, 27 May 2022 22:59:03 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::2125:9bb7:bfeb:81f9]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::2125:9bb7:bfeb:81f9%9]) with mapi id 15.20.5293.016; Fri, 27 May 2022 22:59:03 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Muchun Song , Michal Hocko , Peter Xu , Naoya Horiguchi , James Houghton , Mina Almasry , "Aneesh Kumar K . V" , Anshuman Khandual , Paul Walmsley , Christian Borntraeger , Andrew Morton , Mike Kravetz Subject: [RFC PATCH 1/3] hugetlb: skip to end of PT page mapping when pte not present Date: Fri, 27 May 2022 15:58:47 -0700 Message-Id: <20220527225849.284839-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20220527225849.284839-1-mike.kravetz@oracle.com> References: <20220527225849.284839-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MWHPR17CA0091.namprd17.prod.outlook.com (2603:10b6:300:c2::29) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c51018e1-f5bf-4799-7b85-08da40347e65 X-MS-TrafficTypeDiagnostic: DS7PR10MB4831:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: DNBpEErFZEsfyJQKOQDlQFsgtURvw4ysEM9U/zLpmmEg63lwD6hXu1Z7xvaJdVWk5v5+AQTEDR2UpIyGCN4BNz9jZmpmAq5uXN4U4ey1yNAM4XJx6LclGqSF5pmQzRLGqCSax8ztI0cKUfTWnAtXr2FezE6hvlPTudb0EVJT7soAMJoQTLHihOV+hxtxLov+EbGIztHpooHlijtuAuc7bixjiZJpB7phOfqHqjFmhX/QglKmmEwG8AjgzHSwF7Jqq0TIngmkjvkhz8hh1vTWiVKt5yUleDz2BwBZWU4VrJ+xnB7sg9y2aoZ59XVv93gWGGrFgNBPCojUK/pryjebKrXg+rdQ5e73Jw7S2S18yIVxBoQfljjKLqrMC+v3rPN/2A2SpgajscRNTdbGAXukC4rxUUzHuZEo4+RKFUrbqwsomlW8TDkEyuTQLP5B+iss+dXqPNIHmmAU2zEsh6RP6ldk7ytiQn55hZDQt5PnkdN90vrB6WeejiIM7HzVf3S5qvAqtkhgp6e0JYk5g9KKFc8vC/+ibwFCTgB+Wl974/MT09eP/0kiqMEA97qqERVCh47BWfzW9BddGlYxMSoE75wfx5HBaUwIpk1iWYmxAiek3hTh/6oegX4y1M32iEuZbj2/gwesW7C2dHLU9ZfOPRlXIeEDvIFKBZFEn0BBEvJqhV58DGoM4xJr/sgOBedFN5fwz5XmjhzOmYpK+ksocw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(26005)(6506007)(6512007)(6666004)(5660300002)(2906002)(52116002)(508600001)(44832011)(8936002)(316002)(86362001)(66556008)(66476007)(66946007)(4326008)(8676002)(6486002)(54906003)(7416002)(36756003)(83380400001)(38100700002)(38350700002)(2616005)(1076003)(107886003)(186003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 2lDWOWtqURVwYl4lbdgQr8szMg1ITLFvfAQ2psuUack8Z85Smj3B+Kz4Wu6GTdsr3gujDhyg+IFzCO3qaZ8gDU5qGhblekF/DDClP3+xduK9LCdJodKsdo6YERwzAWFiwiX/K4kkxVFl4XsK/BId31IMYNM65yH7tweniVM7+N/f67LQ3u7yQkZmijOArvH1mrm34XCOPlhH1v55hu4hnBYEH+fWyGXrV7tqKhrzbGFnRXamhws2gVpqBrdkUk7OfZ30pAREhhAQHLEwRlklByTJ6T20wBcoYMQTjbk7bHoVSKjRf0cV0j4VQHDcIXdIFz2+gMM1cQRW6nJMwU2vcPlVDVwLbClIxlpveb7Mh7ngPTG4qWNTEitHGvA/m4NOXZrAIGXksCkPllj0Cdn2OZv0oA41r2iRuDVwwoNP17FiI1D/pSYvYy8aw/1H7IV6UESXqyWhZwSHXfPA6bc5jEGQmdSfJ6J4vwJHw8qJGpHtdEfYMXc3V2FtEkut76emWIdyEjfdn+PaBxfQV19YkL0w5wG6Kc184oJV5opV4gmvAJzDKViSycZxSPuPgTfIRwepG+ZZktz+9YAvXWuC61LIW0hjhQ62eXjmMGhV+bRIvCUeeg7nYOwRSAQTkz8thRiI1Vysb249eGA/OPvBe+thCdGA97flMWWXmy3MGEsBiljbvkQHF0vugf5Srtczvnxxpvl7D+jZLMl82AigNRXOm88/pvluLV1Tu307OagO91brO56lEWzLjcyjKTfAAKtmCO3sU7g3vQ7bWTh0KXxLxFQEeYzLfT5ZNQS77PekH354PqvxCqBiHF8AAIrZYuADr/MJM2wEPMo8wdg9SKdk0/45pcN4iATk3gdfn2LHqJOaWp4EiQEb73XLHHf7VoYgoEM0sV631E/h0qYMdfuIwRwPHBytO16EbzcnJZalGorIYiFXBOiLqyjMknfoJUzEnRg6D7r0x4IwwnaJ+zCR12gFKq7MpKXtLq+LfU6rnCkm7CkGChQgBqJYPgIEWz/TlxMv0ZWp8jO98NMWZkGGaDxPGEpvySisJ8V4YidlpRvmG9rWfo2aKFGLat3GEK+NzFnTYLEH0e+ozb3uwwf6kSHxjf97mx6sdEOkgxrQbYTYp6k+Ph1CVnROzcyx/Kuh30nDVtyvd7ZPKpgVmoTvoO1ESwnTFbO1lcpl6p/ODRwO3ZfI6b3Uc/9vZwDeqQ6HgTu7jbfmZNvLSBLPADIAbNStccv5Mz9A1/o7Oaw3MzQUG+HnWE5Fu3oAaDpHRRrLHbigrZ8voSDZIwNobufYJ/24he4B8LTk797vlrBZk4spPOOB9mPgS2hZskCBnHGMdtrLIpxplye+qGKqfbX5Zv05O/Q4POVUsaX83zeDVNDLeRC4wheEKFgqm211VYx7WX5LXQuqIzSRNzySvwcwWIFrQyube0nOoGbAFiCg/tc6OwwvKmOkkFGwuz4iNrQzVy1qXOoQEyR585/UeIMnBfqlE61CAxnud6zZtbjq062nTn8XgAN7DT2gvQpvtw/GvHCFYzxDarcqcLDxpptIJ1k/n1bDYj7Sfui+bYQgJCItrj3g0+x/IqDVxDu9Sxx6V34v/zh+uTtT078Bi6RyYx824nv7c7VV8icfVR4y0+YnytKT4PeccaOUUYLutVaCGaRhAkRfvJ563iDHmuclXor5AVgRgzw71XwFZhcXlG4gAjKOAj4rKCu+CC8osPh1LIS0yb4aZ/7iyZacICXi3Dn8DruMJ9tPaxfPnmU= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: c51018e1-f5bf-4799-7b85-08da40347e65 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 May 2022 22:59:03.0652 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: kpzUJwew93OyEjoQzxSxSeIwZ2I5bxBuvlgJNeWiJM13UGaY5nlNWRCFfwZUM2Jf4MrGD20s/SkXpxtZZWo6kg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB4831 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.874 definitions=2022-05-27_07:2022-05-27,2022-05-27 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 mlxscore=0 phishscore=0 spamscore=0 mlxlogscore=999 bulkscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2205270111 X-Proofpoint-ORIG-GUID: gYnZrLLda-e_fdbSDCoMVcYP8red8kdU X-Proofpoint-GUID: gYnZrLLda-e_fdbSDCoMVcYP8red8kdU Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=WMD0C1db; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=xl0MoRvS; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf03.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=mike.kravetz@oracle.com X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 9768720028 X-Stat-Signature: gbwuzkmmisk9p471kq6or4xcycr75wx5 X-HE-Tag: 1653692339-643839 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: HugeTLB address ranges are linearly scanned during fork, unmap and remap operations. If a non-present entry is encountered, the code currently continues to the next huge page aligned address. However, a non-present entry implies that the page table page for that entry is not present. Therefore, the linear scan can skip to the end of range mapped by the page table page. This can speed operations on large sparsely populated hugetlb mappings. Create a new routine hugetlb_mask_last_hp() that will return an address mask. When the mask is ORed with an address, the result will be the address of the last huge page mapped by the associated page table page. Use this mask to update addresses in routines which linearly scan hugetlb address ranges when a non-present pte is encountered. hugetlb_mask_last_hp is related to the implementation of huge_pte_offset as hugetlb_mask_last_hp is called when huge_pte_offset returns NULL. This patch only provides a complete hugetlb_mask_last_hp implementation when CONFIG_ARCH_WANT_GENERAL_HUGETLB is defined. Architectures which provide their own versions of huge_pte_offset can also provide their own version of hugetlb_mask_last_hp. Signed-off-by: Mike Kravetz Tested-by: Baolin Wang Reviewed-by: Baolin Wang --- include/linux/hugetlb.h | 1 + mm/hugetlb.c | 58 +++++++++++++++++++++++++++++++++++++---- 2 files changed, 54 insertions(+), 5 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index e4cff27d1198..25078a0ea1d8 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -194,6 +194,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, unsigned long sz); pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr, unsigned long sz); +unsigned long hugetlb_mask_last_hp(struct hstate *h); int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long *addr, pte_t *ptep); void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 7c468ac1d069..a2db878b2255 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4731,6 +4731,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, unsigned long npages = pages_per_huge_page(h); struct address_space *mapping = src_vma->vm_file->f_mapping; struct mmu_notifier_range range; + unsigned long last_addr_mask; int ret = 0; if (cow) { @@ -4750,11 +4751,14 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, i_mmap_lock_read(mapping); } + last_addr_mask = hugetlb_mask_last_hp(h); for (addr = src_vma->vm_start; addr < src_vma->vm_end; addr += sz) { spinlock_t *src_ptl, *dst_ptl; src_pte = huge_pte_offset(src, addr, sz); - if (!src_pte) + if (!src_pte) { + addr |= last_addr_mask; continue; + } dst_pte = huge_pte_alloc(dst, dst_vma, addr, sz); if (!dst_pte) { ret = -ENOMEM; @@ -4771,8 +4775,10 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, * after taking the lock below. */ dst_entry = huge_ptep_get(dst_pte); - if ((dst_pte == src_pte) || !huge_pte_none(dst_entry)) + if ((dst_pte == src_pte) || !huge_pte_none(dst_entry)) { + addr |= last_addr_mask; continue; + } dst_ptl = huge_pte_lock(h, dst, dst_pte); src_ptl = huge_pte_lockptr(h, src, src_pte); @@ -4933,6 +4939,7 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, unsigned long sz = huge_page_size(h); struct mm_struct *mm = vma->vm_mm; unsigned long old_end = old_addr + len; + unsigned long last_addr_mask; unsigned long old_addr_copy; pte_t *src_pte, *dst_pte; struct mmu_notifier_range range; @@ -4948,12 +4955,16 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, flush_cache_range(vma, range.start, range.end); mmu_notifier_invalidate_range_start(&range); + last_addr_mask = hugetlb_mask_last_hp(h); /* Prevent race with file truncation */ i_mmap_lock_write(mapping); for (; old_addr < old_end; old_addr += sz, new_addr += sz) { src_pte = huge_pte_offset(mm, old_addr, sz); - if (!src_pte) + if (!src_pte) { + old_addr |= last_addr_mask; + new_addr |= last_addr_mask; continue; + } if (huge_pte_none(huge_ptep_get(src_pte))) continue; @@ -4998,6 +5009,7 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct struct hstate *h = hstate_vma(vma); unsigned long sz = huge_page_size(h); struct mmu_notifier_range range; + unsigned long last_addr_mask; bool force_flush = false; WARN_ON(!is_vm_hugetlb_page(vma)); @@ -5018,11 +5030,14 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct end); adjust_range_if_pmd_sharing_possible(vma, &range.start, &range.end); mmu_notifier_invalidate_range_start(&range); + last_addr_mask = hugetlb_mask_last_hp(h); address = start; for (; address < end; address += sz) { ptep = huge_pte_offset(mm, address, sz); - if (!ptep) + if (!ptep) { + address |= last_addr_mask; continue; + } ptl = huge_pte_lock(h, mm, ptep); if (huge_pmd_unshare(mm, vma, &address, ptep)) { @@ -6285,6 +6300,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long pages = 0, psize = huge_page_size(h); bool shared_pmd = false; struct mmu_notifier_range range; + unsigned long last_addr_mask; bool uffd_wp = cp_flags & MM_CP_UFFD_WP; bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE; @@ -6301,12 +6317,15 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, flush_cache_range(vma, range.start, range.end); mmu_notifier_invalidate_range_start(&range); + last_addr_mask = hugetlb_mask_last_hp(h); i_mmap_lock_write(vma->vm_file->f_mapping); for (; address < end; address += psize) { spinlock_t *ptl; ptep = huge_pte_offset(mm, address, psize); - if (!ptep) + if (!ptep) { + address |= last_addr_mask; continue; + } ptl = huge_pte_lock(h, mm, ptep); if (huge_pmd_unshare(mm, vma, &address, ptep)) { /* @@ -6857,6 +6876,35 @@ pte_t *huge_pte_offset(struct mm_struct *mm, return (pte_t *)pmd; } +/* + * Return a mask that can be used to update an address to the last huge + * page in a page table page mapping size. Used to skip non-present + * page table entries when linearly scanning address ranges. Architectures + * with unique huge page to page table relationships can define their own + * version of this routine. + */ +unsigned long hugetlb_mask_last_hp(struct hstate *h) +{ + unsigned long hp_size = huge_page_size(h); + + if (hp_size == P4D_SIZE) + return PGDIR_SIZE - P4D_SIZE; + else if (hp_size == PUD_SIZE) + return P4D_SIZE - PUD_SIZE; + else if (hp_size == PMD_SIZE) + return PUD_SIZE - PMD_SIZE; + + return ~(0); +} + +#else + +/* See description above. Architectures can provide their own version. */ +__weak unsigned long hugetlb_mask_last_hp(struct hstate *h) +{ + return ~(0); +} + #endif /* CONFIG_ARCH_WANT_GENERAL_HUGETLB */ /* From patchwork Fri May 27 22:58:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12863874 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 822ADC433EF for ; Fri, 27 May 2022 22:59:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9A32F8D0009; Fri, 27 May 2022 18:59:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 94FD48D0002; Fri, 27 May 2022 18:59:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7A0C88D0009; Fri, 27 May 2022 18:59:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5A7918D0002 for ; Fri, 27 May 2022 18:59:17 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 28DB435E63 for ; Fri, 27 May 2022 22:59:17 +0000 (UTC) X-FDA: 79513040754.01.FF6D468 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf06.hostedemail.com (Postfix) with ESMTP id D68E8180042 for ; Fri, 27 May 2022 22:59:12 +0000 (UTC) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 24RKmdHX024170; Fri, 27 May 2022 22:59:09 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=Kg2wNYIJEkQndu7x1BiocycJELO1I9KZm0qOzqkH0XU=; b=YGXP9G+iS5HfdIaszJDKWKA5MVJuhBji8zD9uzUxOoKG/4YbODGebL6kJWP8u6K7Ae1k l5gzIv5LQl4nf+RHGOy+xzSJLbBpKAXPdH/3hEvQIn+UisNTLAjXxhGac+vxvA9+gQ5K +pG/jRXRs7+F9SdTaqyGKmYhVkmaDDq1dx9ZnabRrvgIJ/NOWfLBSaQzUOmEKXR4Tyy+ mK/PWHuvvNcpmP1dr/9MtPgL8PYE8scHFv1IzytOY1/HtTpsJSWFedlCtFbuRyPRBBEG 16chjfNbl/i13Oga5BJJtLYllileP1uACE+Ie34x2kGV3ahEuEN9fraZHoRIkdAS7sml TA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3g93tb0cbp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 May 2022 22:59:08 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 24RMpXnH031770; Fri, 27 May 2022 22:59:07 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2171.outbound.protection.outlook.com [104.47.56.171]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3g93wsts0f-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 May 2022 22:59:07 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cirKwvHnACku+qjS3vCc0KFIxKlFTENSse2El9tDJlOiV5ZuxWemxFppl6bvybaJ6nGmintLgOklFFWA5I+oVm98BgzS1pJaWzTrFWPhpdO2dhFUBp1eGKOm8AKUsMRSPNrot9asjM/0FdZWF0LSze1r0NR8iPDrD1vMFLs+GPzXlb22LpFSa2ztBTOBMugNqLE+VD+/fc3Bys4UuqzmT0dt7O5JiWHSE7bXxmak58wCtD0LD4bpig7YS1MTzIu79cpKnPy8YBBUTeMA8ctzEfqBcwWH0LcqHrsEHV6/v5rExVHix+xA9gZnceMgRdA+oP4Fgf/Dla3Dd7RnFG6WRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Kg2wNYIJEkQndu7x1BiocycJELO1I9KZm0qOzqkH0XU=; b=bCDm6TJzGNHhL+eWqyvgznbE4Q8Kth9LsBdj6TyMJ0O1VB60dyL82psdMGYAtSzdESv4RnIIfki7m0htcUYsP1nbrFkO5OmTGU699TWe9gkhvP7LFkOGNPYJXNpKX/Szh+WrkZRLQW1fI6rmAaC6RLiNwJ4N5WnPw7j9ikTpDUvG2FdAol4On640WPtU1td1ofoLI8YKuthJ9BGWZ30Nc9CgpG8MYmd0pusyTgLzWHFU9ZvBcMql0xWevFMbYICOszIuqPGlMR2fzjmZYLOCjCAShxX8mFJO+AZNWPANaxhm2O7xQA4Ye/K50Ose6TeBJBvuXeh7jnasCK7jEQ8bwQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Kg2wNYIJEkQndu7x1BiocycJELO1I9KZm0qOzqkH0XU=; b=alfQuvlss2Tj6pUSqac5vXY6fL9B6RvBzIGH+wZ4yIJY8mdeJ9hkaoGPIaSUAJc4LBAV6a5F8225ltOB53gfshsT4s+kcVY+d+GZyXU4x5NJBD36c53CeyF1hJXQ6+CQG7958Yq63ga7W7BlGMiQh3NWTDLmydl/7GkHmEXX5d4= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by CH0PR10MB5068.namprd10.prod.outlook.com (2603:10b6:610:c7::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13; Fri, 27 May 2022 22:59:04 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::2125:9bb7:bfeb:81f9]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::2125:9bb7:bfeb:81f9%9]) with mapi id 15.20.5293.016; Fri, 27 May 2022 22:59:04 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Muchun Song , Michal Hocko , Peter Xu , Naoya Horiguchi , James Houghton , Mina Almasry , "Aneesh Kumar K . V" , Anshuman Khandual , Paul Walmsley , Christian Borntraeger , Andrew Morton , Mike Kravetz Subject: [RFC PATCH 2/3] hugetlb: do not update address in huge_pmd_unshare Date: Fri, 27 May 2022 15:58:48 -0700 Message-Id: <20220527225849.284839-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20220527225849.284839-1-mike.kravetz@oracle.com> References: <20220527225849.284839-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MWHPR17CA0091.namprd17.prod.outlook.com (2603:10b6:300:c2::29) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 673bd517-9cf6-44da-b946-08da40347f64 X-MS-TrafficTypeDiagnostic: CH0PR10MB5068:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 2NxYY/Rpe6iyp3IIpw8ixSAdkqos5Ftptyc4uo6WLSfpGW2xix6rz+0m8cgsivx/t3PALnhEYIDccShrDrEYt0hDlCk1WS0xcbYqt4t+9BAqAeqc4I5cLQ9TYzblOZmctAx/VpXbaNy6soJW0/WlCTw7QmdlMgU9zQRQPwMCz9+0jTnAgaD3zDe4WH70PcuJ3Yj/aMMjYCg6zxKWtFrhedNhAOHDLDiXO2aEGSBML7rQNUaxc4kjJp4NyXe5Abp4R5KW2alOsr4wCG+P6KXR6oWn46ZpTKqjNsyJHTFEUQd6fPh/efGlxT1oqdwm5fEUnMPnkzN7e1Y3xsUprEsWLAd4Wy3kJpjmrD+d9ZQ8X8ABRAcJaFvA259NTLhJP2u/JUgfMyS5tvgRwwrQ8io4KNFNJIYkvjNFsnAQdNUkCv9FOK3mqyKyleTdrZ1FKSmrVlURcQwyuDyNKOdrOxb7USHp0tnf9K1W6cNLz0fU79jnt16P/Ow8ckMrCuVu5RR5WfUX7pivhkRTEOWq2Oehp70POMIhBBsSTMw5IJahE7pNlXXyEXc9Hgomzn6UqoHNM2mAAXa6fh3jtWg1oXfC85BK6kwOjAIUEVdlCNY/4e2BLSArkkxm+6Sg6j+AIuZxsi/h6GZjZNaWZIehzQJ0sdUNl8kaeZg0kF6hP6CjzS2tAm90vFeGeLmYJvcHS+koBsVP4wJG8r0m7j6MUWvJb5NTUQDFaj6DrI5FhYzrUoQ= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(6486002)(8676002)(36756003)(107886003)(54906003)(86362001)(4326008)(316002)(66946007)(66556008)(7416002)(52116002)(8936002)(508600001)(44832011)(66476007)(26005)(15650500001)(6506007)(2906002)(83380400001)(6666004)(186003)(2616005)(38350700002)(6512007)(38100700002)(1076003)(5660300002)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: vk1bUjEFIz+LdaSqiBOJajAqpl9sb+zg2w0mjtEwmKbVlZ9eI+2vVW0jQXN50+r2ephpO3AMAFhFl5m39XVjtnK20eoI0bU+JQw96AKrFrrZnAJp7YInuxqOEQZeo8cUIpoFs8Z5YZVBepjHpYqosnsYG7UyutoxJUkJBj/Iwmf2gsLnNF8yRYIr9XY0AuHLSjOwqIJG/aNj0EEYwcEChwH3JNTtjeFldBXa18ogq9v3tMMI3Fl9R7vITSXNU6ixDr3eeh36a5c5mzEnqOtNIbEJBsoqdZj/6uE56dRkKXU0N9H/iUw+KbjJ1cOCLxkOpYJPnRLe054oVN3OvF6qG6xXKerFWQxOAh+lk/aeXumowqhyNqidQxO69J8vI/XROhcp1UsaGlH3bKNulAwImKSfBcwk0MfeySShN+yEHm+0x4FOhmdir7b6FqEBG3Gn6SV/OV48z6+Y8KDqUkbu0xu3OlQWM3s9klviMoS6nQGjXrO/gcR9KEHnBIK+ChyZ/URDh1Oa8hmCEylAw9nnBctuUVWc8QoQc1hIpuqF8fE+PKQ+HF84pd1byylzky1kjynBZsaEtqazl+6WufY6p3RUfbCewjzm6TLssgI9Mx4/F1gtVPSKQhK/rhpWExV/xeR3XdWW/Javp6nJ3j7sYJaSdBRzx/SqlpbMFCHMet3BuLVthmzN9twfWgC3Icq6BAvcqXPp2vReUchbJX6YPuBKMHOQgUsuE8Up4JIXhoXuzTcVmtxwGj7l/mhkjZCT0rVjMID5chjE9bt1J+WWqVO2J08SzarpXJGs/Ff2tblTEfVyS6d7GwGkr5H7/icSKQrOdbcwlfrOt+V6AcSUaVcmMscZ2IsDgXCdL6kolxXoSyKDyqKuLMzMNL46PMKFjGYwEr/C/+jMCbUHmc5wn+pjhQKLhTU1ZrYV9dQp+dnpzHak5Io0pqxBdx5T1nrZFa1J3kVqpbpvIIpm7oIcvS42lu2uOaWIacQmf76xx5Ox/jqhZgd1JxoOXU4OR9+2PJbqPeYzB7Li7DCWl/u1VnlkiHVQl37CuLsAr4wWDHU3c4BT34QX9k/WT0PbcUgf234FZ2rEQRyLafEPOIhCfPQ8IymYpOeVNgEjn8wzW66OPlZjFb4Ey1+5kEX/MsP+NkALANpr56YzE3C89d2gLpcTYWVUDLJQ6O0Nk/4PGfrvGFFbe4U1RDARoPd1juqjP0J+kBPmm1SMIfSRoMNRvLOqeqmGo+NxSyxKE4pN349Wsm47276n0Qq4eNPeE/oZNfrF6iHAlvOpQaoe/3TsCIFvv6TZISLudTDFP02y13OeW51CH7ctu+Ab376BFEd4ee3nKJ7W+cxUMYTBZV7PNnMdSwvTMVefkDXkp70t48rZbCBiUC+69HparD/nGtvf7o/Mtga24dDjHwrxVS9ltqkHOYcROQM+BThYCRPc1Q+m7TMzBNYMawmWbZNKhpU+w/u7vlum4WSO/sVsL2INCei2pB5uFQ9FOfqPo9SFKTNT693UUahEGt0JbUIC4uqPC2rKSrX8E6J51HAJN/ICsh3jAEKp3zGGXc81IoX9CIki1NGu4QNv4QLsagA8vyljpfN60M6jd8UqCwhh0IuryEFG8Jla66gClvdH7NULlFSKjan47qM9dDhtxFLrGWX+3yljgVihdg9KpzTadixvamRr6JxbpwEbuBuCC7O7+Vi96xgWdwOmDkxqsCKU95T5FHTbLjlD3nVSmKNVGmnVMOyLAziZ/eSZ+QeY+vDYn+U= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 673bd517-9cf6-44da-b946-08da40347f64 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 May 2022 22:59:04.8474 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: bgB9tGkDMie7KxZrrHHo84Gg7pwqqqAITR8rg1tLN3uv0r+o7sWYHHOy4xo5JU2+3TcVvk7uzvh+hLQaaVPwYA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR10MB5068 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.874 definitions=2022-05-27_07:2022-05-27,2022-05-27 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0 malwarescore=0 phishscore=0 mlxlogscore=999 adultscore=0 mlxscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2205270111 X-Proofpoint-GUID: ak9l4zlU8484jy7Y2cVramGWqttby6aF X-Proofpoint-ORIG-GUID: ak9l4zlU8484jy7Y2cVramGWqttby6aF Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=YGXP9G+i; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=alfQuvls; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf06.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=mike.kravetz@oracle.com X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: D68E8180042 X-Stat-Signature: 7br7m33t9oy8ztor8zoadh3psiywge1u X-HE-Tag: 1653692352-253104 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As an optimization for loops sequentially processing hugetlb address ranges, huge_pmd_unshare would update a passed address if it unshared a pmd. Updating a loop control variable outside the loop like this is generally a bad idea. These loops are now using hugetlb_mask_last_hp to optimize scanning when non-present ptes are discovered. The same can be done when huge_pmd_unshare returns 1 indicating a pmd was unshared. Remove address update from huge_pmd_unshare. Change the passed argument type and update all callers. In loops sequentially processing addresses use hugetlb_mask_last_hp to update address if pmd is unshared. Signed-off-by: Mike Kravetz Reviewed-by: Baolin Wang Acked-by: Muchun Song --- include/linux/hugetlb.h | 4 ++-- mm/hugetlb.c | 46 ++++++++++++++++++----------------------- mm/rmap.c | 4 ++-- 3 files changed, 24 insertions(+), 30 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 25078a0ea1d8..307c8f6e6752 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -196,7 +196,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr, unsigned long sz); unsigned long hugetlb_mask_last_hp(struct hstate *h); int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long *addr, pte_t *ptep); + unsigned long addr, pte_t *ptep); void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma, unsigned long *start, unsigned long *end); struct page *follow_huge_addr(struct mm_struct *mm, unsigned long address, @@ -243,7 +243,7 @@ static inline struct address_space *hugetlb_page_mapping_lock_write( static inline int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long *addr, pte_t *ptep) + unsigned long addr, pte_t *ptep) { return 0; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a2db878b2255..c7d3fbf3ec05 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4940,7 +4940,6 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, struct mm_struct *mm = vma->vm_mm; unsigned long old_end = old_addr + len; unsigned long last_addr_mask; - unsigned long old_addr_copy; pte_t *src_pte, *dst_pte; struct mmu_notifier_range range; bool shared_pmd = false; @@ -4968,14 +4967,10 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, if (huge_pte_none(huge_ptep_get(src_pte))) continue; - /* old_addr arg to huge_pmd_unshare() is a pointer and so the - * arg may be modified. Pass a copy instead to preserve the - * value in old_addr. - */ - old_addr_copy = old_addr; - - if (huge_pmd_unshare(mm, vma, &old_addr_copy, src_pte)) { + if (huge_pmd_unshare(mm, vma, old_addr, src_pte)) { shared_pmd = true; + old_addr |= last_addr_mask; + new_addr |= last_addr_mask; continue; } @@ -5040,10 +5035,11 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct } ptl = huge_pte_lock(h, mm, ptep); - if (huge_pmd_unshare(mm, vma, &address, ptep)) { + if (huge_pmd_unshare(mm, vma, address, ptep)) { spin_unlock(ptl); tlb_flush_pmd_range(tlb, address & PUD_MASK, PUD_SIZE); force_flush = true; + address |= last_addr_mask; continue; } @@ -6327,7 +6323,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, continue; } ptl = huge_pte_lock(h, mm, ptep); - if (huge_pmd_unshare(mm, vma, &address, ptep)) { + if (huge_pmd_unshare(mm, vma, address, ptep)) { /* * When uffd-wp is enabled on the vma, unshare * shouldn't happen at all. Warn about it if it @@ -6337,6 +6333,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, pages++; spin_unlock(ptl); shared_pmd = true; + address |= last_addr_mask; continue; } pte = huge_ptep_get(ptep); @@ -6760,11 +6757,11 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, * 0 the underlying pte page is not shared, or it is the last user */ int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long *addr, pte_t *ptep) + unsigned long addr, pte_t *ptep) { - pgd_t *pgd = pgd_offset(mm, *addr); - p4d_t *p4d = p4d_offset(pgd, *addr); - pud_t *pud = pud_offset(p4d, *addr); + pgd_t *pgd = pgd_offset(mm, addr); + p4d_t *p4d = p4d_offset(pgd, addr); + pud_t *pud = pud_offset(p4d, addr); i_mmap_assert_write_locked(vma->vm_file->f_mapping); BUG_ON(page_count(virt_to_page(ptep)) == 0); @@ -6774,14 +6771,6 @@ int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma, pud_clear(pud); put_page(virt_to_page(ptep)); mm_dec_nr_pmds(mm); - /* - * This update of passed address optimizes loops sequentially - * processing addresses in increments of huge page size (PMD_SIZE - * in this case). By clearing the pud, a PUD_SIZE area is unmapped. - * Update address to the 'last page' in the cleared area so that - * calling loop can move to first page past this area. - */ - *addr |= PUD_SIZE - PMD_SIZE; return 1; } @@ -6793,7 +6782,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, } int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long *addr, pte_t *ptep) + unsigned long addr, pte_t *ptep) { return 0; } @@ -6902,6 +6891,13 @@ unsigned long hugetlb_mask_last_hp(struct hstate *h) /* See description above. Architectures can provide their own version. */ __weak unsigned long hugetlb_mask_last_hp(struct hstate *h) { + unsigned long hp_size = huge_page_size(h); + +#ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE + if (hp_size == PMD_SIZE) /* required for pmd sharing */ + return PUD_SIZE - PMD_SIZE; +#endif + return ~(0); } @@ -7128,14 +7124,12 @@ void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) mmu_notifier_invalidate_range_start(&range); i_mmap_lock_write(vma->vm_file->f_mapping); for (address = start; address < end; address += PUD_SIZE) { - unsigned long tmp = address; - ptep = huge_pte_offset(mm, address, sz); if (!ptep) continue; ptl = huge_pte_lock(h, mm, ptep); /* We don't want 'address' to be changed */ - huge_pmd_unshare(mm, vma, &tmp, ptep); + huge_pmd_unshare(mm, vma, address, ptep); spin_unlock(ptl); } flush_hugetlb_tlb_range(vma, start, end); diff --git a/mm/rmap.c b/mm/rmap.c index 5bcb334cd6f2..45b04e2e83ab 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1559,7 +1559,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, */ VM_BUG_ON(!(flags & TTU_RMAP_LOCKED)); - if (huge_pmd_unshare(mm, vma, &address, pvmw.pte)) { + if (huge_pmd_unshare(mm, vma, address, pvmw.pte)) { flush_tlb_range(vma, range.start, range.end); mmu_notifier_invalidate_range(mm, range.start, range.end); @@ -1923,7 +1923,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, */ VM_BUG_ON(!(flags & TTU_RMAP_LOCKED)); - if (huge_pmd_unshare(mm, vma, &address, pvmw.pte)) { + if (huge_pmd_unshare(mm, vma, address, pvmw.pte)) { flush_tlb_range(vma, range.start, range.end); mmu_notifier_invalidate_range(mm, range.start, range.end); From patchwork Fri May 27 22:58:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12863873 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C6E9C433F5 for ; Fri, 27 May 2022 22:59:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B73158D0008; Fri, 27 May 2022 18:59:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A5B9D8D0002; Fri, 27 May 2022 18:59:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 80D8C8D0008; Fri, 27 May 2022 18:59:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 712ED8D0002 for ; Fri, 27 May 2022 18:59:15 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 4306521734 for ; Fri, 27 May 2022 22:59:15 +0000 (UTC) X-FDA: 79513040670.05.9DB7AFD Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf31.hostedemail.com (Postfix) with ESMTP id 8983620049 for ; Fri, 27 May 2022 22:58:37 +0000 (UTC) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 24RKYxNt001972; Fri, 27 May 2022 22:59:10 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=WklxNc8i9CPt8PvCnvXar2TGFKBokVagIFoHNTNhGKo=; b=vWfGeNexUz/QaM91XNIvfPAXg+NfTqmd+5BlCpKINKH30S89xXAfOmgXrEfoL0NQQLtA 3FYII0jqGETdlNCoIKvt87Wz9dm4WWI+tlTM0ch2BKQAe6TwtpNisJ5KOBAHBZzBiQbM 7JAS0N49NDsoQC1+Xm0RGjaizEuWdeKCs5Mjp51EyC8hlfDngMN+SurVX+kGk+TcsO88 nRojxZaK13BFmycAdVYzFYLitsO7fobnzFawg5bLJh4FPc3z5+IJ0ZJ8wHUTTp0a+25a DcBJQa041kGD9wHnwUQMtibO2NaXjTXO5Qacj5vYO2AyYRvxQkX7HVPIcWlWC0h5aJmQ Bg== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3g93ta0ane-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 May 2022 22:59:10 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 24RMoF9m005421; Fri, 27 May 2022 22:59:09 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2171.outbound.protection.outlook.com [104.47.59.171]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3g93wxq4m8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 May 2022 22:59:08 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZgxzuAPXsLaz+swesbL4pGqZDfivS8TCOh/LZBGePAHqXZvTBbqv+9jw8NFmvsZK5ojvQDo/6CoUTIV9hpmtjn9ncX6dMpiUSAe/lVHqpw6r4LNBE8nV0jNsZz4UrTgijPA8Hm5QTtYH8zgpFnjyGQraQ3tFnF2dqgYFG6LUhDGrcB1Apcv2yciwNtr3yBzgXqS5I0RCQKpDdDKDuah0xFAxo/IMSIX5r1TZx/u7Qx2B6gGwQT4W7RDmsE1KsW6sjWGmsiMUUqTIWaPUPMb8Zwh3lNkY2Bf5S2rr3nNZBIcuzk0TIfFTtklnfK172+cW3AuqHNLOICFQUVx5IBe8Gg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WklxNc8i9CPt8PvCnvXar2TGFKBokVagIFoHNTNhGKo=; b=AnAkDNz/vZsFiGprJuQdYtuaNijkaMQEymHzzVG5V6zHSyLV7aYiYobZ5FVN+sUWDvu07YYRvGVidZSwkduWAhM5VgHksBSJ9a9YjTxbJdWlGiChM+jSn51VJvjJSWbLExIzFZ/UnTHY/f2rHCVrFtU4xi1HA+10ulOtuu5mRx9VNw8BJSV0HOj1cXvpjKVDaTvcWovY6DiiEM4tCDYfMK+opheuXH0Qgm424fpvVze2fgezecYWBh3DRjhY5jMGrqXqKsI7cgM0PVCHd9M0tuVQo/Ajgh7SHY3fLLt4FMOaMV0hrr9BsaJM1A/j9IUqdAl4rPBLDaKYkibX19H5Cg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WklxNc8i9CPt8PvCnvXar2TGFKBokVagIFoHNTNhGKo=; b=M7yb6RZC0JrFz7YRGsX4HN85sAma4q91tEtlvwGlHWhsDFjmPuqp7SNQVhzk32IOtSjNAVws3cFAyyV+xdEawP7vis/e/vhcV+mAYFDdpFG1O3Hbx5NgrXprqHEfmv0JYhtCJHGrXSUfM7IwKd5WlVN9jKD+Mx28dO66l3U0W4s= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MWHPR10MB1472.namprd10.prod.outlook.com (2603:10b6:300:22::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13; Fri, 27 May 2022 22:59:06 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::2125:9bb7:bfeb:81f9]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::2125:9bb7:bfeb:81f9%9]) with mapi id 15.20.5293.016; Fri, 27 May 2022 22:59:06 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Muchun Song , Michal Hocko , Peter Xu , Naoya Horiguchi , James Houghton , Mina Almasry , "Aneesh Kumar K . V" , Anshuman Khandual , Paul Walmsley , Christian Borntraeger , Andrew Morton , Mike Kravetz Subject: [RFC PATCH 3/3] hugetlb: Lazy page table copies in fork() Date: Fri, 27 May 2022 15:58:49 -0700 Message-Id: <20220527225849.284839-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20220527225849.284839-1-mike.kravetz@oracle.com> References: <20220527225849.284839-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MWHPR17CA0091.namprd17.prod.outlook.com (2603:10b6:300:c2::29) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 38b1ad82-696d-4096-e333-08da40348050 X-MS-TrafficTypeDiagnostic: MWHPR10MB1472:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ZJSVGhB49zG4p2niBOSFXZEvmFyDpAhVIvrgPt0iNS798Qg27wYo1eDbyukAWEPbU0J3l95Naw/kRBT5rwWhTF5VwIsuvbvyNMAA6QwQt3bP41I2uHCkSqy0hMPCIgQkCZK2GtCQ1SgQzISJ1iEohK2tfT/5CCKKaEYHIDaQvmM3yUOSrprfMwLxMqp4yiP0Tm12JOuRzgJ9aszow16w6vASD17yhOehcPWN0Tla6G0THmNKhntTm31kEjGJxvNVwD1Ec/vaV/H71ild90JtxDAelrOjeXpKX/2dsbS+8lPrkkVa9ESBdj90SidF0lJG+ITlvCiGBlsxyoZZfts1sAKJ9v/4189C8rsv5c4p7Fn01aLve2QstvXx7lg2KuedGsFRBZwNWdvT5tLMzU8tO+Y92FuJ9rtsMlNaJ+LI8+Kcxh8jlBA/5wpUJmEBkqvN4PEq9B5OWjImVP5RlmQ+RRVe6MWuZ8MiHLofyQ6a+KPK/b663gterapU1yZyF8aN2o6vMEfICHKYCy3Grd0ga1DahS/JhxBAKTRZ+ByLFTwOnyic5NtjOWravF3APrmEqq6di5A0JPCLSqDgo/0y+/3n8stYfrGHrbjc9XNuOjrc+lBn2lIVHVtyYxWMPcS7BZcK4857MTe0QqbfQ1H6J+YW6XK9ceky0ZN17fWi6AXUARq6ppGKyg1sQfWGZhHutYYtdKjPRvIzJ0ogDUFTkw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(2906002)(508600001)(6486002)(38100700002)(38350700002)(66476007)(8676002)(4326008)(86362001)(66946007)(66556008)(6666004)(8936002)(2616005)(54906003)(44832011)(186003)(6506007)(5660300002)(316002)(107886003)(1076003)(83380400001)(26005)(36756003)(52116002)(7416002)(6512007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: pWoDRWnGQaHX7j/LNAxxTQVP/tgUkoSMlVwVsGtZvk2Uc+4V/kwERBfuwvyBeLB9WAV9SbtZcC/WhW0X+2TDJWfE8Md0H35t+RV7ThbbJLTQ946HVb2u9QhVeuLnG46HcCb/o9WGsXRGV7xcfv+R82IyxUZz9T4polLWVEhJVKGyrSY9DSV0BYltASTcAvKtZNRPXlGxJzZHFmkkoclyz0NPMUMd1kjZ65nqujTuE8XLKi6776ptDRWLikW0srHyk2KXUxWaBWt45L8r9Uq1YlK2kh5p4DwTVB2UhOjwpypD7siFSywGcWP6GBLFNlqjB05FWbfHcTSb68Tv//hts3RBPK6oPh8zwXqS2F9FSPnaUuaycHM/ldcasCblfCVnZe9beKu6sfV+D4J9KRAYPoQwudOVqiv2DpEQ4wZqqE7dDFYipxVHqR0Unf/VASoCMFbOef6Uk5HpGR9CcAJ2oitJ8tsYoo7gr2SIPzyLdd+tD8e00qKg8YDduSXhsMVg5TiOp9nQpeUC3Vc3+FtCBM8J4NZX7iVt56uQEeuUz4G02e1y1GWPB2/sVZZ0gqp2zUy73Kk9vim9zrnsaA4JHSANCnLC2TxgT8ED7ahFooGo5yE78s83yKTeZsjTGs9xKT7JE9Dnf5SHZCddRgYCzqQMyA76CAyngaAFweZvqTFsaLZII5GApU2RwePRAx0lNm6H3h6ThRyA1XBMX3j7lxRzg2XJ6jDMPQUigL1njjW4e+hzsPA8NfVmLJR+HHu2qtt7xze5WfNgw3pCQ/gtSmHmbv85EltNDZhnFK1m8byXZu+RhBLpOyRmjee3Ht1sK/IUzu7QKhZNpjv4Uku08L0/V9BZmWl4TqkKevjy1ELGq2dyq+P6Rze2We5eqzRDjQccE1tJEBJJIabmTZd8b0GZw5fphgA9ISCDYWHpq9B0WGCyqwJjwmTx2WgiLKtOlFQIDa58dPZBFRsTL083E9VY8ID5rIDI7EnBYYxUsGUMCz75XTb/K+q2hUQmc1/Kdp2pa5FreYvNy0sP37Ovp2l9Wsx7YBkXPmgktsvzZkZkCGJMOpY/xko1EqXbqdjfNpZ8kJnzyhtHykP8sdYWRasXN4Vl0+AsvHOG4cRHG9fYDI17Pxa8c/PeRSzRT1Xk4D2TIGWGc1NYsfsDtAG4Z5An2xD7T07f3Odyh43DXKyBcF4X9sn/guwMCKqH7z9J/edD19X7pDXrzbjvU5WKD7sev+7RYhY2fdskXTOQyXbAMnuB0GFmygWymxoFEvVHqm14NtD8b57rLA8lgyXEMCbBoNn1lYNpLm9r85zD8BUo5TVt41xBJljwPcn3lAZd5kocWrZK7byi/isqojgo0NmwX6ehdSAKYTrWPjaMe7+7mRB8b5W34ZHsNHL4RofiBZWnefc4pYAwEaqEdFBGCi89M0Pr9vGxGxgh7kqDcg20du3kWljwBB09Au9N4Yz/G4s27sPGGoSr8FevHxmQbMbpllJf5wb8+0+KqU5aMlzP5JAykeVab7l5REP1Xr6FkRdMY2f7co6qJGlbb3yWWjc/aasbNiZ7zFEjdQMDQMzLOk9mpIID2fWuMPDqh1iS9Xv6n/PORfJ4CAHl6oEarsNn8a8r0cVtDIRCYY7acA966iYPAq+7wx9xt+Jg1J5dy1lPfHXCDat4Jk1688SGrBuhbDOcG9BFQD3p53da55QlQcDjq/+cOcfDXUWTd60dgbEV1suQyFeGaRJd1SA32nS2tbtOQqronDJfbL1hAZc= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 38b1ad82-696d-4096-e333-08da40348050 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 May 2022 22:59:06.3482 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Q3QPVXQo9Srv3oFBg3RAMwoIYp7oPwTS5rOcqmG95OrbI7J93O2NcfQMEBAA+p/1OxbEwpJdon9/WZogvLcnXQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR10MB1472 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.874 definitions=2022-05-27_07:2022-05-27,2022-05-27 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=984 malwarescore=0 adultscore=0 bulkscore=0 suspectscore=0 phishscore=0 mlxscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2205270111 X-Proofpoint-ORIG-GUID: TFrEPee4ZyygY5vXV5rQUurJbLpg7WSk X-Proofpoint-GUID: TFrEPee4ZyygY5vXV5rQUurJbLpg7WSk X-Rspamd-Queue-Id: 8983620049 X-Rspam-User: Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=vWfGeNex; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=M7yb6RZC; spf=none (imf31.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Stat-Signature: ck8fuzdm9jzrmdqn4m6uyd98b3785o7o X-Rspamd-Server: rspam05 X-HE-Tag: 1653692317-222258 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Lazy page table copying at fork time was introduced with commit d992895ba2b2 ("Lazy page table copies in fork()"). At the time, hugetlb was very new and did not support page faulting. As a result, it was excluded. When full page fault support was added for hugetlb, the exclusion was not removed. Simply remove the check that prevents lazy copying of hugetlb page tables at fork. Of course, like other mappings this only applies to shared mappings. Lazy page table copying at fork will be less advantageous for hugetlb mappings because: - There are fewer page table entries with hugetlb - hugetlb pmds can be shared instead of copied In any case, completely eliminating the copy at fork time shold speed things up. Signed-off-by: Mike Kravetz Acked-by: David Hildenbrand Acked-by: Muchun Song --- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 21dadf03f089..f390cc11886d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1245,7 +1245,7 @@ vma_needs_copy(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) if (userfaultfd_wp(dst_vma)) return true; - if (src_vma->vm_flags & (VM_HUGETLB | VM_PFNMAP | VM_MIXEDMAP)) + if (src_vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP)) return true; if (src_vma->anon_vma)