From patchwork Tue Jun 21 23:56:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12889955 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C001AC43334 for ; Tue, 21 Jun 2022 23:58:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:Cc :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=5wHkY+EJIuSL21G/rCcqR29fl3GhmpTKrvGzhT70bj8=; b=jjZSSFn4vBjYSS Fz74IbJOTOI1FKUnk9AWqekJ+JIpf4TI6f7wK6ClooB3aZkxYSitidTn09PRcy2XCWusxbixQaL3X Z1f9Zy1gqQvjnmMDTXgmA/U9/ZNr7sDfvFLIe60UG3YWoHGz2d1lNyVPB9TkkDnnnIDzWqJgZ5Elz Hmq1aF51JR1X25NoCoMa0T7syK/7mRMY8Jcg4iuJMGtzpNvdKMVLOvWl16YAc7/fD38F4eCKd7WNl KSmTFu4kwUmE6BPIMNuyh/Qs+dEA9EswGCUrP5HaeCEfiiyoUJN6YDHI/paCcH/ynSNrf4ncivig1 buHWry5pJ51SDVlwPidw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o3njs-007iYX-Ol; Tue, 21 Jun 2022 23:56:57 +0000 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o3njp-007iWw-Qq for linux-arm-kernel@lists.infradead.org; Tue, 21 Jun 2022 23:56:55 +0000 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25LIvLGb007514; Tue, 21 Jun 2022 23:56:32 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=JCAUTkU7DIVAPFGJ68CqLPcMG2CgrmMo+eXKPgxjQds=; b=OGgFKvfqewyRdxGbMvTFmN7R46E9M2zor+dSO06ICHUdVrQY8TdERVBHci3Zo/7MrZO2 ZBli2H1+Er9jyhExT8EJWofxMz1vU74uAmZhmrgJ1B/oMWIxkyPIDCZ+ITX3O4ZqP43u qJ92gy8zMw00Ej4CTEMP5S06xCZmOSfvAi3w0zgvU0/jAtruAMrDcwYoRnME5ehLEaI0 AE7WN+ADDbGkoJxC4M+JgSNOrJB6qgP8737ARCNuiVkEqAF3ThNWltdnVjR115NA8HcB iVKZ51bfGKfhb83JSLTdzlGZBMRuMPAucAMn1yYt+rrgdOMGsY3DanuHsEh6CgUQs0M9 rw== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3gs5g1y162-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 21 Jun 2022 23:56:32 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 25LNoVC4013270; Tue, 21 Jun 2022 23:56:31 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2171.outbound.protection.outlook.com [104.47.59.171]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3gth8wwugy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 21 Jun 2022 23:56:31 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FQWUjIb25+67jIRslMcUSLGrSr3IuOeenxWmhYrP+9wuEUEYyZEKrbaD+5ZWhwfGdI76PhaPRboYJzAD20nOqHLFPgypWNIBHyT4ho2WEXaXpbnGVhhq2VF4KPMyjbt5tKXoI8Lj8QQXSORe7ykjQZJGxM1sohMOwRqIGNJqpQlwzpdCGr7zc/J0BNhSlr2jeRbm7ql/uBQ1PTeFzdLuVSoj0F+SMhxtivyuCYeUQ2l+BuN2ZvTuzp1MNw/JqniE2SeQcPJpqBEn9V1IQjA/VJg07mLOO6VZ2yvvcvB5XZcDCseAkhhCJhBQewOJ0zPVYhGM6sC9dXnaavZmf1oAhQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JCAUTkU7DIVAPFGJ68CqLPcMG2CgrmMo+eXKPgxjQds=; b=iIuP3tV/cGyDKvumoM0lWPkb4X6SCrmUUi6F7DsjEztxh7q89PbN5k8sJoUoCKcIjGUjN5hx22q1D3E9vkJjyLPYC15U7/+S9AezplTs0pSUDYqtkQ5OFXanJHLnxw/a0WgQL748Gij7/e6HixQ/BRZGROulozHDOS4Xw+sE2PnmTicHU1ohMo8lt344LzbOXtb7LDbH9Xy/KBk0338B7IM/wkJSi+tRBudDuvG90RQ1UwltNfwvSLvCH+t0Nr3a5aA0Wn0adUhy857A+9P2w0qevltt0Ai4WuDifSTfiIVa4Ib1lka9dWAJpmG5jm9kIjjbyNTm25wONxqDFFFQFQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JCAUTkU7DIVAPFGJ68CqLPcMG2CgrmMo+eXKPgxjQds=; b=XgtIMwFyBpHQZTTw3xvmpwUBATumvgDBpkA8nTYk6uLYBbJoHjuEAmlvn62JUiTWyqkJwF8HbLsuzU9kTsQBBumiRtVpQSH9ikLy2grnlpPuhUyKM6WcyG9H5p4ZzoGPqY9bqQa9Gm8As/r98db8nKd008ePu0Vo8Huxops2S0w= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by MWHPR10MB1501.namprd10.prod.outlook.com (2603:10b6:300:24::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5353.15; Tue, 21 Jun 2022 23:56:29 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::2125:9bb7:bfeb:81f9]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::2125:9bb7:bfeb:81f9%8]) with mapi id 15.20.5353.022; Tue, 21 Jun 2022 23:56:28 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org Cc: Muchun Song , Baolin Wang , Michal Hocko , Peter Xu , Naoya Horiguchi , James Houghton , Mina Almasry , "Aneesh Kumar K . V" , Anshuman Khandual , Paul Walmsley , Christian Borntraeger , catalin.marinas@arm.com, will@kernel.org, Rolf Eike Beer , Andrew Morton , Mike Kravetz Subject: [PATCH v2 0/4] hugetlb: speed up linear address scanning Date: Tue, 21 Jun 2022 16:56:16 -0700 Message-Id: <20220621235620.291305-1-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.35.3 X-ClientProxiedBy: MW4P221CA0009.NAMP221.PROD.OUTLOOK.COM (2603:10b6:303:8b::14) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9785a83f-1ace-4793-e9ff-08da53e1a894 X-MS-TrafficTypeDiagnostic: MWHPR10MB1501:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: RiU1UfSSQ04hHu/uKT7++qlyZrE/95sX1vsSC8unUDyhTylc832sHKTaXw2n6QTsD9SwY/61UBOwD7YTu8s2JWOxzc/DxbWg9irzIZEghMHRyYjtozD+q76lniohVTLrD55It84AimUhOPvJSgxGmmpaS65VoguFQe4bjOO7eZCjkdCYfPbImaXeY40K+qMdh+j6BsPmQVBnEAxzZXO64JHvHk6h+5DXZF4f29IUZLxjV5vj3cEPymVTCJEJFcc2fxUsOmMxjSpYEsLGG3XbRd1tq3Dj4WD+XTHNaJDmu6zIClS0PCT/gJfa+aMe7ftpjq3IQOi86DgPwi6eCvqlNP9DSkosK932GRdFBmS31x5EMOgBAcFHDA9wHkPYqlzk45uOBfb3aaIBJITCD872Gxz7sQHk7I8Kb3XClqiN/32g5epLldhFh5wnLEBP5Qp/LeDFnUrZ5tw5U8N1Dq2c8lCC0Yo11Te3fsJzpQ1sHpWFN6JqZGtA5cngnW2yi9RMSIlf+WurDl3nR2CtnMoqNGx4ZbP980ys1EFmV5xb/UgTwoV9xgnTNpgfqhhB1rtqZEuP50IgGOWRfVfvoNXxPD0BLHmqbxYwSrqJRCqwuUVXgMCb3a1pdnm3h3gvRHT6zXwktgLx0uTbSNSRgzNVPAtRP8UYCCgmtUhBXF2rIsYKVemfVoybbWVy9ZgD7AcU X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(376002)(346002)(366004)(136003)(39860400002)(396003)(7416002)(38100700002)(478600001)(316002)(6666004)(186003)(5660300002)(8936002)(2616005)(1076003)(107886003)(83380400001)(41300700001)(2906002)(86362001)(8676002)(66556008)(26005)(6512007)(66946007)(36756003)(66476007)(44832011)(6486002)(4326008)(54906003)(921005)(6506007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: GFwHZ4yRaYMcynGzv/uO+EosR+2JdmrkzPR392ovpEFAJnhMWhDneaRaQphzHqs99Opf8R1j+GjOHBncr1ub18/BpnqCj0e8zUKqBHxZsxzlrkotTkZmzkC9JX/buitJ4QguopUf3S1NcJ4NYRAU5Buj4UOllt5y+uKTTXou7ZAThI5++n0biNwYXNaYFYVZydblveWiGzcv+MVQeSLCva3UJiDaIvCbiJJp7IQA34RLSDGpSNadhn86xTleQZdlwLaPPmNbtCNK7rBA0vQr/O2FzD7ouyhaQ21Au9zXFma4d3BX2xHaOd97UOsXOpuugNS/XJkabehwC9aSqjj8Sr4exty8fqs7mz8WzFkVW+/dxrUSdqcEhKZX1/WwdB75bsEEQ5aFH9r3vpsUiF62e56fM6G88vG7fuXCmt6yAd3m4c9hkWabme/6odLRMCK0RCYo05J+spN7hDlU1I4VK7Per9lkdutS+0s0sR7qSpWoKyi0khXXHbEWFSb08ABovSzbtXkWkTRaH9ZsVdZDInaxUFM7iid/JAx849Gg4WGRfJlNFGbZAfesjM2mobl+NEWZdfbWb988TX2dSJV46/y6vU2mMbcyiMLJvbU7Ox9ZpV/D4LmKL7/undllsm3UjnhN5QN0sA68PLFJs0zJYtZ+zz+ZOOEMb2Sp8yVTTLbJEwrMbwna79M65oLd+nG2ka1XrW8IOdAWuuGlZ0py9z6G3cte14414UJPxEWyRGZEyYGue7mlA1nJ2bNVuyKEp3jMyBjAZicG8sax2WdGT61r7ICKEj3NdhebvVHI4NgMbVJShK3/gwujVNlnAGuWFdvyip4sv7RbPps+qvsWP/qp7GLI9ZcuzyZBBsZrLLRHy10c2UOxk9xX5AEyniqjCrzZfADEwEDtMT5Yt4XNKrpD80osjzx7Bzp55YoKNNB6NG6p1FkPdtPn9WS/RZMRutBL5Eo83YzWfmivQb4Xe6xMh1XXzgJWnUWEblhP6blc1Jt3uQb+ixr8YzcbwLUPYzHnASkdU84E4XB5neTuunZxmiQjsje27RfwhS5rF0d5efyAUlkrlAiX3h3whKF6GQTFLAxC8aTSmpG1t6kBy29vtOpq2hGy/YyIgybJcGly1LDmnOyqi3Ckpd2TYT8cSjQRT/baS2po95+LQc+Vtq/qqXiZBlDTQUmG6BtLGuLQpwwBJP1UgxW3Q10tHvYF63WAt5hGKl69EqGPrh7oq4aRZwT/JzO0DzO8nIXa4ciCKuDgrmvfWQ8a6X/Z7rugbTjzap3p1YWaLS9KEeoa21tXTf2mCIgfw8buDi2e28kmSJDmCyth6Ex7TTedTyHZnzqbhzniJ0KcBBpDNEL2k8JqDtr7zsd3GW9G/nK4erSk/Tpshb0kf6TbysQJvGL8+owj+hG+ZATlRS+dUe195QbXSnYO86KegluAKCBUPvJ8AFI65RfciKulbyFVKFtyxxhRPrRjg0bKC7z/5zkybkTfy9b2TBmvYcgUUEF9vErzu3VwYWdA+VBLhoOQkprqPayP9gWyFKP0E126VBOsE8veNQWJmbGorZuwVKlBnzaltqkPhoT29MpQFD2uDZ+A94/ZHfOLsOZrcv/iZwkrySHV5Ae6QvYeFoJ2g3y+OveoHTldLvZKxfCoNh+UYRdlwXEmREhEPaOrHHX6IyJGRKY0Nm9AmGY4HIZdP6T4RlHtwKCtvEQLIktkMa2sEFpzSoiMI7PbQNIJoqV+f4HUFxZd92mNQQ7imc+nKY2uiUc= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9785a83f-1ace-4793-e9ff-08da53e1a894 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jun 2022 23:56:28.8987 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ETCSvQhVGKOQ7c3CAVVHTS4+Lb7nhqeO0sp1OPUZ8Msu2FDM2fVqlq+IO44j1+W1OxlHcyXQN/MQdYX3PAtOsw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR10MB1501 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.883 definitions=2022-06-21_10:2022-06-21,2022-06-21 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 malwarescore=0 mlxscore=0 phishscore=0 adultscore=0 suspectscore=0 mlxlogscore=738 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206210105 X-Proofpoint-GUID: HbXXjC84H3vS9iRHVh-73hQJ5rzE4_R7 X-Proofpoint-ORIG-GUID: HbXXjC84H3vS9iRHVh-73hQJ5rzE4_R7 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220621_165654_014981_FE4BFF88 X-CRM114-Status: GOOD ( 18.56 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org At unmap, fork and remap time hugetlb address ranges are linearly scanned. We can optimize these scans if the ranges are sparsely populated. Also, enable page table "Lazy copy" for hugetlb at fork. NOTE: Architectures not defining CONFIG_ARCH_WANT_GENERAL_HUGETLB need to add an arch specific version hugetlb_mask_last_page() to take advantage of sparse address scanning improvements. Baolin Wang added the routine for arm64. Other architectures which could be optimized are: ia64, mips, parisc, powerpc, s390, sh and sparc. v1->v2 Change hugetlb_mask_last_page default code to 0 instead of ~0. Peter Fix build issues on i386, including going back to if-else-if instead of switch in hugetlb_mask_last_page. kernel test robot Update commit message. Rolf Eike Beer Changes were relatively minor, so I left the Reviewed-by and ACKed-by tags. Baolin Wang (1): arm64/hugetlb: Implement arm64 specific hugetlb_mask_last_page Mike Kravetz (3): hugetlb: skip to end of PT page mapping when pte not present hugetlb: do not update address in huge_pmd_unshare hugetlb: Lazy page table copies in fork() arch/arm64/mm/hugetlbpage.c | 20 +++++++ include/linux/hugetlb.h | 5 +- mm/hugetlb.c | 102 +++++++++++++++++++++++++----------- mm/memory.c | 2 +- mm/rmap.c | 4 +- 5 files changed, 96 insertions(+), 37 deletions(-)