From patchwork Fri May 31 16:32:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Liam R. Howlett" X-Patchwork-Id: 13681890 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 147A816936F; Fri, 31 May 2024 16:33:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.165.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717173207; cv=fail; b=QRzFissMO6ymrTqLqKxJJAXVX8n7c5i7a4SQYtE+uWgXQVHHrGgANj5JxvaQYVA9tlqLfojQLyW0VY7Nh+Ag151MSHp0wi9X2Px1gav0ZQXSWJa9ac8X3LhD2z+G0vGH6YGBjQyRE9EWj/0qYAByevMwa+3rQJnI4TV4b7v6CMI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717173207; c=relaxed/simple; bh=pC5ikyU6Lrj8OCtU9/kcZgyIbFUWIlhCLB7xDJOTJDg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=Bgeb2e0Z0HKeFFNgx8dgCzRR+PJqnkZ11ErfO1LfDIC9+X6l7quNrsLP/SxOfdU01cpD0q/P0p6Q9Ch/oklVD05khr+HNkkz4Furq/MVPVkxE46U6ASyht5V97ihQWVFM489Wv5E6gqelSnE+3NV66k2BcWUrwEnHFxTIFELL1U= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; arc=fail smtp.client-ip=205.220.165.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 44V9VWXO030176; Fri, 31 May 2024 16:33:13 GMT DKIM-Signature: =?utf-8?q?v=3D1=3B_a=3Drsa-sha256=3B_c=3Drelaxed/relaxed=3B_?= =?utf-8?q?d=3Doracle=2Ecom=3B_h=3Dcc=3Acontent-transfer-encoding=3Acontent-?= =?utf-8?q?type=3Adate=3Afrom=3Ain-reply-to=3Amessage-id=3Amime-version=3Are?= =?utf-8?q?ferences=3Asubject=3Ato=3B_s=3Dcorp-2023-11-20=3B_bh=3DcaSvmY9bE4?= =?utf-8?q?1fDaqsXdYRpUUnMr9miPHUopMtUxgxinc=3D=3B_b=3DFnM3ReBkjxbV4ag6Y0Us3?= =?utf-8?q?qM+Vk84i4wDB6FawhfdCk8CD5WgXOCsJRIYDjE92FxhHnDu_DB4kNnL79LOdd9EHT?= =?utf-8?q?n6CVp+LLQtHlCc9fmi4IqLwgpk5VLR9EzgczJ9iKYIQjme0JX6L_DdCyDRBLwu24D?= =?utf-8?q?I1+8ObiIq1bTrTkpixq9AJNCAvNoc1VMuLykjPXbUxENHuwstdw5Adi_1Mik9VaRa?= =?utf-8?q?+kJAP2cbWoDx0ke1zmBEi6zXUTgPptc5ZmF5//i0TUgmzKlAUeIU/tIcHRW_YXQAO?= =?utf-8?q?hc47FtwcyxcW7QWHK0OF5NPWKITnHxw6Kqk3/ODBQGKWPlaiI8stA3bbBa1Lx0Y_3?= =?utf-8?q?A=3D=3D_?= Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3yb8hgbk84-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 31 May 2024 16:33:13 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 44VFaLYI006199; Fri, 31 May 2024 16:33:12 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2168.outbound.protection.outlook.com [104.47.56.168]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3yd7c8kmkf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 31 May 2024 16:33:12 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XOUTz3IHJm2XCe9qsN3X3eAQEIJejHak0yuyacq56VX6IIWWb/bcAWgw968WAKoDU/ntKS228hbgHZQbJ18/IbdhJi1BArQ8cE6NOBZJVuoAaHfGboFBmwI6DL2TpDzj1UwVgByW7B3JpefD+REDMmofSFzRxj6BZtDLApNdAyk9w94q0I2fgwAxVmQzprEaQQ2Y3lJXsaYk0lGDd/dhuCODIbxXycBU64La/UbpoK/t7rgfQEYg4Yo3YPW/+hQlDsKPA6p4RQt1pS+z7L3RPVBZrGP7ed2tmdXeHdMe5TKyBXMVApZczR0QEBylUGb3ik9zsxpTxZ8mKsvWS4ea1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=caSvmY9bE41fDaqsXdYRpUUnMr9miPHUopMtUxgxinc=; b=IrLDRxXfj2RI1pgRK8RscYBcv47sbCIBWurPz4fWqTO6EEZkSlFfx6kPUZM0iqWIGBQ0TsqEkO9Uf9SQdfzYne2kgZr2cPpY1alwcHPut8csTTB/g2P5nd2BHLt87qIhQDM/FmOyZZEpTVq63zkWBFDCBMs+HqA4WNpQ2c3OhDaaQ6qtEtqEXvcERckFerrz+7POzsC7DC/YcEenWSQ5hoqrJIoyIgwTDysbqhsnM4Q6Xt8hIcckcf5u8Db2HVDYAkBOt9AzQF1jQFNUoTd+RxL/r5dbKFeEuRGXCsQFzYScQWLGT3VCuTEtwXYthS7m8uRButS6LDYwELgj/Fse8g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=caSvmY9bE41fDaqsXdYRpUUnMr9miPHUopMtUxgxinc=; b=HCpTmrnvDTM7rwYibiNdDIHlngkq8QdDzje4dAtHo8y5tgF5ESrF/78fD4ykYxnoqBF/nudrgYrl41RItO9YHPLqFbdn99eX2ipbAZCRDXk7CLDebPBUQKmJ7Gb1GohUSfIuwhDZi4Z7sjTzjNmyVO0xizRPxSxiEsEtW3kEonQ= Received: from DS0PR10MB7933.namprd10.prod.outlook.com (2603:10b6:8:1b8::15) by SA1PR10MB6318.namprd10.prod.outlook.com (2603:10b6:806:251::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7633.24; Fri, 31 May 2024 16:32:44 +0000 Received: from DS0PR10MB7933.namprd10.prod.outlook.com ([fe80::2561:85b0:ae8f:9490]) by DS0PR10MB7933.namprd10.prod.outlook.com ([fe80::2561:85b0:ae8f:9490%7]) with mapi id 15.20.7633.021; Fri, 31 May 2024 16:32:44 +0000 From: "Liam R. Howlett" To: Suren Baghdasaryan , Andrii Nakryiko Cc: Vlastimil Babka , sidhartha.kumar@oracle.com, Matthew Wilcox , Lorenzo Stoakes , "Liam R . Howlett" , linux-fsdevel@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 1/5] mm/mmap: Correctly position vma_iterator in __split_vma() Date: Fri, 31 May 2024 12:32:13 -0400 Message-ID: <20240531163217.1584450-2-Liam.Howlett@oracle.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240531163217.1584450-1-Liam.Howlett@oracle.com> References: <20240531163217.1584450-1-Liam.Howlett@oracle.com> X-ClientProxiedBy: YT2PR01CA0026.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:38::31) To DS0PR10MB7933.namprd10.prod.outlook.com (2603:10b6:8:1b8::15) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR10MB7933:EE_|SA1PR10MB6318:EE_ X-MS-Office365-Filtering-Correlation-Id: ba740f81-6630-4eeb-4798-08dc818f4c6a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|1800799015|376005|366007; X-Microsoft-Antispam-Message-Info: 38YLa/oX6v78eoCtQzq5wdXN5Qcc1gu4b3gTg0MdDlJcW/LeKJYjWzF7mZod1S7c0GjMtUW6BzpjQEWslxAqYhxdKlIVFlm2N1wH0bbdlTolqPdCrpFJMv04Q9IhNQZSVoOQcCgyOsGCWm0SoFJI6A7gv48mi56bVaKqEZ/qg5J/wnUvzKB5ZwzeO+v+xBgcs7x2axN4Ofx8Cj9AvlwpilWe0fdXNLtcN5DdQX456EjAn5r+yrmC/6W6gKjbNtUwMN5RYnj+gI/PV/jjulQMcNRjXzHeRonS8kYS+a/zOFD+yGgqAqVrMR8AwJC9zwpMhozvmewdhM0x6YZx26trBmwoAAyalHXjCAQmYpLuzyTGt4V3HOztLH/CMGXTscx2OZR6H686I69fhNxTl8meHalowk5YoMrLMNNmXvnYG/6P+i2g9ZKBsa9SKuP6DlxDO3hoz46FtNQrgSJ3VGROOfmMopsVPqLS9RdisnFeAbpJpa7/kS2pld1Sk5CRFXOyH3Furjx2VKBr9d5bGKNy1E0tMnM+wHCwenp6NczXxMJZy/qhQVYpStvw1Vas8pLdNItTPhiij+iJq7hpXCQhlyPvmBIlaGPjyWNRXc+TTg7RBBAN2PCqqUuYB0PD8ic2KQ+C977RoXAA4zJPi08Q9m+Hi/jG0kL/maO0qb7hxlavromjwSb/2iyPjeWtI2maEakaxAO/uqDJCSsa43Dd0BpZAxNSzocRu7zc0RZgD+UG/No00RQCpF+NTwC6c2Al0YjQUB9niYdQ4IykMQiqwLr5pAICRjPQVgqlBCsZAKoOEKmttsDDnCWgiTrt/11h+ZY7Cz4jBGtaXq4D99crL8lJWFHteO+1+0tNdXdkw6tUzJT4dXmHqKsgmAbcHGy0Nvsf/yYoJUHoClEt5UpCQHiJTEfxrWizSH4bWGPSVVYo7jdhExRiu6QfbeQZuKpFzgRYyk/5MmebHCS6h7DZAhtQrVnVB7wlWq/7sGIqU+TVL4zZbJHycFzMGVzRQi66HAXqZXvezddYTk/6CpyDZPiDGobd6mvTfjabnZW8scAA94k/g/jK+6ZRKjRmdFFZI6/LGqINsmspm9FLFg5vu/MH7KgQ69mOdNFI4odjqlYi6txl+0kpJADGpnNbK8wNxGC+Suc7uZznbMW2F7wGmEfDLx+uPBmQPI8+wsQIpaiQuJRhjut/IGVm+K7DB3CLNmIZDXCdmJS0m8RjKUKiS+/JmQa3jZB01UeebqKttOmtyxyjLa1cl8lJew5uKgnlUPf+jWkzc1zXKT558f8FHw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS0PR10MB7933.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(376005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 1sWvOmpEhG8Fglmgl3EUdjXw0khZczBYPWllIF2etVc8WKVAxxw1/OYnghIs2Ou6U0m+B2zTG3ZIEvq0GTt7aiKlWX9UED1i5DeB4ksvU8bDnS96N7ZEBdtzhs83ONA7/m09Vb6cZTtELkcGbqUQ/VulgywdWokaawY7pZnJap7gEum/fu5W5iZb44c+7Psw1SFYt8amaTH6GNDKRQa6keTY6183gyPuPOC4dMFRnVm2+Lt/Y6Y4vtUqlyY9M3FoY/wlxSHoukFsKdM9ibTCyj8/u4cP+YcSSWVuZ8+SDPr5kbznM8xJSYKnmgEsmORmPGFqxCEMkgkEI2EGvePlmEeMq/ym3Pt/DrzKYV2SGk2RDlUBciL+zHm1Fu/osoqxmvMxF40BnVZbTDuofVT5UsoV23Z3ccmvJC5/22o+RD23EBTkZbjwuWKAu5AaHQ4Y6gZ9EzCScHYndsEYkjsFv3k2ubZXFCEGrpMxb7Gwq0RYWPNmC/h4nN7/sUwN12rHlcFTjO96lCfV2uvioaH3b+x+WHXpvACfkWPCmXAQHnO+jiFc0UgoEYQL+uwLTJwQVH9yUoZn+O6ilD2+fdI6I9duJtvUSpKIbiBCO4KANVLIVAAYOHr35vTDZjVf+uiSpJsxZ3ywtprki7NbHHje6MkMuta0PPGyKb1vgUGMnvtknV/H/Le+sv78kmzod0sIZm1aa11xEwsN9JKAIrkm0xnUUevuh2Zzs0iMpOcYeXAqIuIHCwDTd5pOoQup01fObjMRrglRrnKUs0BC7q2JjXqxrfDTLy/eODHj7UGduZdY7tkFA2dvzbkXU8EeZ9l3TSmJLCx4uzb0SQdnPpRuDkBCvB8JkylhwU0eQRjb/AToC+spXIdgVeKGogLVEz4AUklihVLZbJpPsnUibjcl8ZbsQCncECc5IcAz4HTl/E5bij0NPZkV/Y5dId3F8lS5QO1nPGqRBJgX54MH1fvZizuFMzrWI1Fu7nYZ2NnaA9LlKQx0qBo74z45UGMeVPS3zF8YyMzyXk4qlvg8IS07RHddm1wBB2OlvLjSHTD0NL44fCtTc3BnTpeD9KCcdnwC0LsKt4o4QFtm2l5IgU4biOBFHCGeodNglcjrt/PzhAsZ7pGeUvpHriEN+kGdhXaG4UWNaMVgeRO12fQYdpMCN4G5rWEUdA+NYaU4WLrVfMu5SK0FCugAukkUn+WV9D/Z1pMyhJGbO7TeE3uimaGpR5Ox4DSqxz+VvLkysL+7gRJa+P3tQTrubKuQ38+w8s0QMp0Q0XRzs8dNSR4UuTVZHpnyTaLyo1svVm24jr4Qbsh+YeZX0aBYy6GU923A+Rj60QLU0E8NaIuFW7S2LfWHnVYX/w6KD3ejdkfkCjDStrECf9n0Kpkt4KxP+fmYda9XpSOAX+/dZP3jOzF5dJIDgPqse6YEXokdQ+YuV4QuoqLs8r3qOzH6CG3EKuYhG06GXM5dH0HU1VPEhUhNPbLwBUuQfPlS/sXx3JwzNMlbvPnvzuXrI+Y3ASNAD1r0UT+f06EMMgsFcj2GVnIR2CAv8QRx9ax4ivutix6fZ/PfKMB+pDHgfdzKVWwlDXoA+MUX4TApIgAlDj5iNx/jDrzTFA== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: n4jyp9I15MuhdfTt0Cb0vlrP9ksgH5D7JXXncFANoYckv0CZBbftLGwSQhSVcw+lSasijbI38ymYpDtk50aOZHwexTnRUtZi5K20mU0/3G4vUWQfUX67CVkCsIF3cvIznr3/I4P4RjuVoktfiInZLdRxxa1gcBHBFKkguGRNkJWVX+amslTzyiQmH5sxhCGmCF5tc2+pvb3S6TmGYCYSBnOS5X3AtQU2uGFPaS0Ygl/Vwm8zFTUNGsnUY1aBDt5r/XpN3nkwMX/h5wC7tfeT3Hv/Y1SQBsmRGNMvWETG/exrK1ozRKFftXOdmokZ7I7xH/cVH6Gh1cF6Z6YVQ9FXLnwsPDVSKoJRSvGDpxf+mdEGZGLpJMWmDtn3/81TZ84bBBLFGVKY4DV8oj4QCMrB3l/doA0LX1Qx0cIbQMACdDFCBc4iPvOoQJvlYuIFR1eHYoLSBgj/h7IxITN0XkO7tVvudZa+MA9/CMr+EIV98vcNBHeCLqZFNQz96BOHziulEeIpRQGwLpZV2ojzpsp2ZfyTn4UL3BFaVbdehFaCfZMnNbOUxCiYImnda4+OjRy9oYGkWc1UWDiYV+7VTWesAxZJZhBqTKNEIMd2ECvvcXA= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: ba740f81-6630-4eeb-4798-08dc818f4c6a X-MS-Exchange-CrossTenant-AuthSource: DS0PR10MB7933.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 May 2024 16:32:44.2820 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: yH7ZoTYauYkZAPJ34yalWauRh3MyPbJcSUp+Z9Sr/WyL64b8xN9rfc7XslZlBNRNeUAMkZl+PEuQuLMN9Geq7g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB6318 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.12.28.16 definitions=2024-05-31_12,2024-05-30_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 bulkscore=0 suspectscore=0 mlxscore=0 adultscore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2405010000 definitions=main-2405310125 X-Proofpoint-GUID: 4vE1ATAQnR7pmSNfj2aNkeEbCblKTWyh X-Proofpoint-ORIG-GUID: 4vE1ATAQnR7pmSNfj2aNkeEbCblKTWyh X-Patchwork-State: RFC The vma iterator may be left pointing to the newly created vma. This happens when inserting the new vma at the end of the old vma (!new_below). The incorrect position in the vma iterator is not exposed currently since the vma iterator is repositioned in the munmap path and is not reused in any of the other paths. This has limited impact in the current code, but is required for future changes. Fixes: b2b3b886738f ("mm: don't use __vma_adjust() in __split_vma()") Signed-off-by: Liam R. Howlett Reviewed-by: Suren Baghdasaryan Reviewed-by: Lorenzo Stoakes --- mm/mmap.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 83b4682ec85c..31d464e6a656 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2442,6 +2442,9 @@ static int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, /* Success. */ if (new_below) vma_next(vmi); + else + vma_prev(vmi); + return 0; out_free_mpol: From patchwork Fri May 31 16:32:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Liam R. Howlett" X-Patchwork-Id: 13681894 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 27920161B4D; Fri, 31 May 2024 16:33:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.177.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717173216; cv=fail; b=NIX2Fkid39F71dVyNc1K9FL0O0EgJ0T7WcrRS08IGwjTAwJ155D1q92wBzsFy9csHOPS5drTZdNj9lNqWC2juMS8zd8+q+kZgbWwLYXB47gqV/Pm1n+na62uyeivhbmzp1u2divLQa92PGbRo4yvxMy5YTYhlNY7lM7DCeNeRng= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717173216; c=relaxed/simple; bh=rm6/bFIL6p1MHxYBTQ/ROrm717XpKX2PmgkMitR2IYU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=XqTZPkngwEeqMRSdm6CS/kXPj0tcHIweD9b1XbkluMnvqTpYpamlnBH0Z6ErfwXtop1R8OLfxOIUyr61e/xdGfG59nK+a0htuhgjwBIGBlP/BgzvyFIkM9RTtYnxZVCi0dE1oizV67QRBzijS0g4WNIK95YVvMHBMTabHOzU0Hk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; arc=fail smtp.client-ip=205.220.177.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 44V9T2EM019194; Fri, 31 May 2024 16:33:19 GMT DKIM-Signature: =?utf-8?q?v=3D1=3B_a=3Drsa-sha256=3B_c=3Drelaxed/relaxed=3B_?= =?utf-8?q?d=3Doracle=2Ecom=3B_h=3Dcc=3Acontent-transfer-encoding=3Acontent-?= =?utf-8?q?type=3Adate=3Afrom=3Ain-reply-to=3Amessage-id=3Amime-version=3Are?= =?utf-8?q?ferences=3Asubject=3Ato=3B_s=3Dcorp-2023-11-20=3B_bh=3D4FXhuH3znV?= =?utf-8?q?7KCQLXqKilTUAjWsm/phpAGBkoe5rd4bo=3D=3B_b=3DbmhSaa1b7KHlFQcbcbpCD?= =?utf-8?q?/0eUX9stddVrHHC+F4G1HxkBDurMCAD8JroyHts1irSEId5_7OZwhVB/Mlwx8KaaM?= =?utf-8?q?qE7Qg4WpVQQkGT6SBixcyr5AFRv76hbQk8JmsJ1zFKK4Tz1wNkd_ajpfJJqlQsb8r?= =?utf-8?q?P7VTjXJ5dRGeaUw5fPfWTfnv1Ak2PhWy8NlbmRHtXHfU48338Yo/3Nz_/sZcZv2nR?= =?utf-8?q?Q4VTH89NlyMVcZHOaOAYU/qQg51IX3SwDJu98Ha5KY2cxsyBIRwd9sCljOE_Msdi5?= =?utf-8?q?q7WA7XlYSKJSxNW4fxTwleHr22TAbguFKE9093ffFCg6NAj30Z/GIQ2hoDraOdd_s?= =?utf-8?q?g=3D=3D_?= Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3yb8fckk7k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 31 May 2024 16:33:19 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 44VFaLYY006199; Fri, 31 May 2024 16:33:18 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2168.outbound.protection.outlook.com [104.47.56.168]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3yd7c8kmkf-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 31 May 2024 16:33:17 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dUdkZP/Of3YJrHRueIbiKTU0A4K4MFBb30WH3yzwb5o3nQdo4a3xVfkvvrgb2l3DOtrxtq9cqZEtBh6Pou/f93wvRIW0k+Tmw6DQc96NlJc6dsLrxjDRLor6g/lnFR2IwQz3wIOvuoS+jD9jta/es32Dog6Bh3xLCjZn9gTi/HcxlyiCpo/veSAc8ZPqVli6xzYoupmibW6QPJmVmTBvA/lW0foqLBBgqy5eDR8JHomolu2R4sGCqb5ixp6wP0dbi1lwvI5j6P43Bcv+FXxw5HKCzCuqGj9WzxSjD1PbuhU+bfM9d4vQ1exREKYDF/9z1Qu6ULgPJCb9D0XLnfRDDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4FXhuH3znV7KCQLXqKilTUAjWsm/phpAGBkoe5rd4bo=; b=UtpEqIYwmgvuH7BsfAw4rCbsMkOImVJBhe+2S8gQ+P3nH19xReIYGpVepgYkQwoWyj8gbBamWoeCEfWH7FGa7CKLZhh+gT6WApqA+sBITATXdqqtXML6fTwvd0YCUQsQY9WineP69lwhjYTSpXDW2cf03Pqk7o+2PJ9Ma+l2Qgit1jIpnwSTl4S5Q+LC7qEeK/lSw4/BWOaIX1tmv6Fk43S62b2Uc3i691cQ1Lr6j0O5cz7YJ6KEsmlwuwa/wXuh2xa+mMQB6hSuwUkSdtBrxk0uXHg+1jEkgNLFgWXv4A8NUtsDfEHXWOqEKAY0GjgtQQqfImBJ8SVkzMlFo9SvvA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4FXhuH3znV7KCQLXqKilTUAjWsm/phpAGBkoe5rd4bo=; b=yQemrvhqkCvhAxLoGKnlN+xsOyXHnxJ8mE58v/Pwy/QbUD13PGEMPSBWC/h4qSQPuR2PrqAqfEpbFY7JZi8q0U3vLhhAbHSVNoPGETOxt2OotI4isPfWt/iIxy/rnKcYdVya/De1B6j1RQevdXMD5n9C0eKMs//Qb4hYOWaLOT0= Received: from DS0PR10MB7933.namprd10.prod.outlook.com (2603:10b6:8:1b8::15) by SA1PR10MB6318.namprd10.prod.outlook.com (2603:10b6:806:251::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7633.24; Fri, 31 May 2024 16:32:46 +0000 Received: from DS0PR10MB7933.namprd10.prod.outlook.com ([fe80::2561:85b0:ae8f:9490]) by DS0PR10MB7933.namprd10.prod.outlook.com ([fe80::2561:85b0:ae8f:9490%7]) with mapi id 15.20.7633.021; Fri, 31 May 2024 16:32:46 +0000 From: "Liam R. Howlett" To: Suren Baghdasaryan , Andrii Nakryiko Cc: Vlastimil Babka , sidhartha.kumar@oracle.com, Matthew Wilcox , Lorenzo Stoakes , "Liam R . Howlett" , linux-fsdevel@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 2/5] mm/mmap: Split do_vmi_align_munmap() into a gather and complete operation Date: Fri, 31 May 2024 12:32:14 -0400 Message-ID: <20240531163217.1584450-3-Liam.Howlett@oracle.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240531163217.1584450-1-Liam.Howlett@oracle.com> References: <20240531163217.1584450-1-Liam.Howlett@oracle.com> X-ClientProxiedBy: YT1PR01CA0082.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:2d::21) To DS0PR10MB7933.namprd10.prod.outlook.com (2603:10b6:8:1b8::15) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR10MB7933:EE_|SA1PR10MB6318:EE_ X-MS-Office365-Filtering-Correlation-Id: efe14402-7f27-471b-ed3b-08dc818f4dee X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|1800799015|376005|366007; X-Microsoft-Antispam-Message-Info: XMuHApLxl3QAI/5/yy5cUGp18ToelNpi9Zm/Fuj+U1EmnzFZjiZpOQ7AbfiShEEZG4UabSwvdyzCp0KyYVOnV5GBGusZJ+IKdiBiq9iYu+EInZoXhM26uWD/mApuMgpO3h6yig3FvnpB2WUEULM++ian6BmkQ4DnLGaTAEjRQXP84GTHQqtcAEcZEBzSU+J3YiBymnl6CkrT63Zc3ciQKKUhO39DWc6lDBk4TKfECr6H8om+qfLmpYQqceeC+Pm5BKqTWK4tFdEdGdH7PLp1jqbbuVNUCdku+HXpWdfGYKLA5W8caV2ZMiNFx+vH9wajITF1UAH+FUaxlG1RAwuaEJXDeyilz8M0pGqxOfGR0MBUllcnon/oybUJYTfC7JFGAP32niCGjHsK9kDN2t0fTj1KnR00s8e9EssV+hgPEgIoyzjHXYhI67ufbwyIQsNIpBWNo428EjV30Cx5TGoTsscVwBtqFk+xjPZo8OYEscKhR85LdNVrC0Z/eXFq2tb2K2mjEDDo86PLZBSJPIwn9juFzG5Te9Mwqsnmx6GHqIh+X0OC3wA0nue0FreEO9jA55G+Wgb5/T2kcQct07xN7ynKZWdrbHyJxo73QEjmtX8zjeyN3Ng4kkiX2CnoSOffSi4OV10QUlEbSSVU0TDjosDDKuTP6y9fwewEnVah1pmlLbnS6kowqWKU0nzK4BbCeWLpSn0tbIiBRihaHR1R/YSwGFuqrw4jYci+4zZ/qCA5ypI72yNrh7BxNG6ewvOzioPac/CRRIfUWVgyh9YD0R7hE5mkcMC8lz4zjAricOQzep2FT9h5gMcgvO8quIoXIBU6YoVubw3YPREkh2A9hV/V4Nug5GZquH1NzhqoLX/xREESYvdbDZlyf8FCS6McqioM8romyl57ZDcEzDJ9Peooljz/uP7rfncGYLoRxTSaYHbcYakRPZ7KHn87sRBc3f42xOe+sHbjy2VUYDMeoMfnCrVMWNDPdiIUIPCIjYdvy3bWc7bEFxWWJyBO8dFwBevvMOEvJFCCSq/FtY/GFSdcUphm4TVnVO07uUVI5MKnQ78gbvFSxKIPI9cUSYcJl3fR0AmRK1dXCKU00YxTSaUiK+RH9NiSwHOH1knkaLvXoWSaT6u/+F6ZIYvVRwwACsXzneNzLUPUxiiB/wbD/tJz/lXgbF39YGvOF0MsQqXR1ljb8B94ovqLnLUHa/641mdMyxtxdPzTnp+b/62bBxrAe65lKhEgYdB5ZErp+DsHQ20JbRdag+EioOqho4VWUNaHY5MfgcVYK1dKGVHF/g== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS0PR10MB7933.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(376005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: R1Lhe4WQRxri8XnNvL0aUgRoywxjSMc5smfh80jbeOvVl/iVJgUjTSLAeqTVf8VwredilUTfbONHlF1i7Q/LRC2Mrpvl4TLfuKQNK+knB7ESk0COpBeFxkW7Me1imFcCCQAbyfa8w/QrNG33mKGIeuvzQe4cKuLM0TWWOR+H6+6OdezbJs2EJWze8+stfdgWocu9Q66M2COiWhqzlRqUiQgf3MOG1lC3u02w7M6xK4zn1aawg6iaXZBpsNX7vJjXOahXnPZpwxTse9roNpLXvk4StFbXnSC1/wPBAjja1eCJ3aIWd6R6BhXtwphQRs9wAHA5s3gLFq9kz1Z7sSaV22qfr/60bfD8talGdwQu8PMBKxu76ox21OvxqzHVsurUap+6fLM7eVapMSnbATPNpt83JfzNn5ZbqeJo3I2kpBR03k13ytlsklRatk/Y6i5CXLuuCBAdAeWXhUiUcKLDdzbr4l+Pnnq8j+LGe1XXtBqCu2Db1xnsW2Z5SoUTHlzQZqWk95Hre6j31+mA6TkPuLn5WhD/9rwACwXkMoiXrXXhbpHKKGssSTUBmG8XmBVwntTV1HPHDu4VGVLLmENqQyIdIYKEJp/m/N9HpVXiyx9W95hvGf80+K3II5pWXEQxctlBbJS7M+FAhE7k4Toyse8ANGkCFAiaygHkHtrDbHtmyp9xOa2F5gRA69ruX6Bj1Edhmx/EsuH92kl83OPUFSdCly31G878OL9Hmkzv9m9RWyAJa+reLCuLHHGQz2rFQONu8RjuXh0326NCFksLmYlwtVZb4HKiS+/gJwdVuojv5yjaq4mgadg5jZVe3cHIfmRgGMPKIcaE1n6wd5AstFazviH8XNJPX7WAmQOoRo14bGruyOnk8wakB0Lefe+3hNz8HK+hvrMQE8Z/aw/ecmX1bR79bXmjlTCuv1KYZaV8+ttp8KGQlNABXnY0yJuIloaCmwvCZ8Rf5dgc08nK8dhQh2zNvOAyrVEYd4BFApp/mAqfcgwPW59adxbO5XkFn58F5VrSYfS5BC5QDwOR7mFJwsOG8x/SVEUCvVp6JZBatQSiKIb8gke7U+3ZMEZQ9jE9xvyeTJA6GYuQZB6lnMj90jqDU7hnNLhu+L/HvZXeqKR0IDrFfcv9ScaqhI7clNnwAiXSJM2PooGekoqqgZQcGiM+NF9r6aT0zHBrwrBTqbhnN/Rd2WrgHNLhslGtY8phUV0paSEur7zM+glatmLzcllYBc3KoIIrMwn51Ty2bzv1DsqKrdt5jpUfzzG/wW2mo2/csT3+B1sWTKMP1Mo4AtGXzJ5sXATfAezBGdewNzmput1uKHMBRwgwQ4Rw9GogY2S4FnFz/Ad0SReR4UeFl7EmryZoTjEVtyKb1ppp0bh61wLO1G7/QA/Uo0z3y1ACdOWXgyap772/PP3dFj5Dgdrjdk8i8YeZcgwHcpxU4UrC0cD+Fn3s3NqrXw4TbA4bUnP854zCgQkj4kjYM/yn8rFHCCCj56lB9rmvH8VO/2Yv3czCIKkHqkHnJI9FFnoyxGmld3bZHHCAKsJME9YB14XmkCHrVEiqFsrBkFaOjZb4q44sf4319UVfEn7A X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: t4BWhhr6ZsDxooWz6kcB6GvzGQOy17EkG1Vp33TKmUqcqBE0HsNMq9U+fGJgDlDIA2Kt6oMk+8+ih1seZB+ThJdT+D+NMM27CyNUaAaTcOsp2xlxW7kU2BfhrlZGMI7tVp2N/C/dwGk2nbAYVv8iMuMLp4sQmLH0riKfXKKpLShlHs+Ss7EmDHJ0csOf0C+FyVQtkargZSzNtHEZNgtKKK8N2C6R7tPygoAPLpXSV+lEkr+lgSFMx4/XfisgO3fEs7ccIbGRPEQyQz6kHB0gTaCHRRthgLkYFnI76Xas+zY9QsFrKJqO7lqR8T/vZNiTe9K5Jge0I5SmDnJPySOXZLKjFFKeL2FW+/LfLuRQhSJyvZb9tLHhw1hUt53MOMoBDiXTMohPw9+/Smyftga6HS4wABOTg3hNzAESv1JnM1I+sqhA3ESjmTre+EqggqKIns73v6Sd3jCPbfOF23JSNSyV2QRuVj8EZrMD3x+8dwMsF8udwRr9dxaIlQV8glHFhf4FtzOmW8yqouSax+SYdGfPYVzebmIBzGJ9uFANF6uykc/2E0a3R4bN88Cbud2gBcUcPvh5hTXkbf8mmSQH7jChfch4mqPNChBHEBIyRTE= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: efe14402-7f27-471b-ed3b-08dc818f4dee X-MS-Exchange-CrossTenant-AuthSource: DS0PR10MB7933.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 May 2024 16:32:46.8486 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 62O5WHhUf8tKBX6aHjI7xvi7rCdIzdX37yVgRwJiLORMsefJiV+h6zxz9JqOatvf4rPDO8LmCcX5hu9zzAZrgw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB6318 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.12.28.16 definitions=2024-05-31_12,2024-05-30_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 bulkscore=0 suspectscore=0 mlxscore=0 adultscore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2405010000 definitions=main-2405310125 X-Proofpoint-GUID: jtQTfN2JonQERTfMAMf_FtcFY6I-HA3J X-Proofpoint-ORIG-GUID: jtQTfN2JonQERTfMAMf_FtcFY6I-HA3J X-Patchwork-State: RFC Split the munmap function into a gathering of vmas and a cleanup of the gathered vmas. This is necessary for the later patches in the series. Signed-off-by: Liam R. Howlett Reviewed-by: Suren Baghdasaryan --- mm/mmap.c | 143 ++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 101 insertions(+), 42 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 31d464e6a656..fad40d604c64 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2340,6 +2340,7 @@ static inline void remove_mt(struct mm_struct *mm, struct ma_state *mas) if (vma->vm_flags & VM_ACCOUNT) nr_accounted += nrpages; + vm_stat_account(mm, vma->vm_flags, -nrpages); remove_vma(vma, false); } @@ -2545,33 +2546,45 @@ struct vm_area_struct *vma_merge_extend(struct vma_iterator *vmi, vma->vm_userfaultfd_ctx, anon_vma_name(vma)); } + +static inline void abort_munmap_vmas(struct ma_state *mas_detach) +{ + struct vm_area_struct *vma; + int limit; + + limit = mas_detach->index; + mas_set(mas_detach, 0); + /* Re-attach any detached VMAs */ + mas_for_each(mas_detach, vma, limit) + vma_mark_detached(vma, false); + + __mt_destroy(mas_detach->tree); +} + /* - * do_vmi_align_munmap() - munmap the aligned region from @start to @end. + * vmi_gather_munmap_vmas() - Put all VMAs within a range into a maple tree + * for removal at a later date. Handles splitting first and last if necessary + * and marking the vmas as isolated. + * * @vmi: The vma iterator * @vma: The starting vm_area_struct * @mm: The mm_struct * @start: The aligned start address to munmap. * @end: The aligned end address to munmap. * @uf: The userfaultfd list_head - * @unlock: Set to true to drop the mmap_lock. unlocking only happens on - * success. + * @mas_detach: The maple state tracking the detached tree * - * Return: 0 on success and drops the lock if so directed, error and leaves the - * lock held otherwise. + * Return: 0 on success */ static int -do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, +vmi_gather_munmap_vmas(struct vma_iterator *vmi, struct vm_area_struct *vma, struct mm_struct *mm, unsigned long start, - unsigned long end, struct list_head *uf, bool unlock) + unsigned long end, struct list_head *uf, + struct ma_state *mas_detach, unsigned long *locked_vm) { - struct vm_area_struct *prev, *next = NULL; - struct maple_tree mt_detach; - int count = 0; + struct vm_area_struct *next = NULL; int error = -ENOMEM; - unsigned long locked_vm = 0; - MA_STATE(mas_detach, &mt_detach, 0, 0); - mt_init_flags(&mt_detach, vmi->mas.tree->ma_flags & MT_FLAGS_LOCK_MASK); - mt_on_stack(mt_detach); + int count = 0; /* * If we need to split any vma, do it now to save pain later. @@ -2610,15 +2623,14 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, goto end_split_failed; } vma_start_write(next); - mas_set(&mas_detach, count); - error = mas_store_gfp(&mas_detach, next, GFP_KERNEL); + mas_set(mas_detach, count++); + if (next->vm_flags & VM_LOCKED) + *locked_vm += vma_pages(next); + + error = mas_store_gfp(mas_detach, next, GFP_KERNEL); if (error) goto munmap_gather_failed; vma_mark_detached(next, true); - if (next->vm_flags & VM_LOCKED) - locked_vm += vma_pages(next); - - count++; if (unlikely(uf)) { /* * If userfaultfd_unmap_prep returns an error the vmas @@ -2643,7 +2655,7 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, #if defined(CONFIG_DEBUG_VM_MAPLE_TREE) /* Make sure no VMAs are about to be lost. */ { - MA_STATE(test, &mt_detach, 0, 0); + MA_STATE(test, mas_detach->tree, 0, 0); struct vm_area_struct *vma_mas, *vma_test; int test_count = 0; @@ -2663,13 +2675,29 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, while (vma_iter_addr(vmi) > start) vma_iter_prev_range(vmi); - error = vma_iter_clear_gfp(vmi, start, end, GFP_KERNEL); - if (error) - goto clear_tree_failed; + return 0; - /* Point of no return */ - mm->locked_vm -= locked_vm; +userfaultfd_error: +munmap_gather_failed: +end_split_failed: + abort_munmap_vmas(mas_detach); +start_split_failed: +map_count_exceeded: + return error; +} + +static void +vmi_complete_munmap_vmas(struct vma_iterator *vmi, struct vm_area_struct *vma, + struct mm_struct *mm, unsigned long start, + unsigned long end, bool unlock, struct ma_state *mas_detach, + unsigned long locked_vm) +{ + struct vm_area_struct *prev, *next; + int count; + + count = mas_detach->index + 1; mm->map_count -= count; + mm->locked_vm -= locked_vm; if (unlock) mmap_write_downgrade(mm); @@ -2682,30 +2710,61 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, * We can free page tables without write-locking mmap_lock because VMAs * were isolated before we downgraded mmap_lock. */ - mas_set(&mas_detach, 1); - unmap_region(mm, &mas_detach, vma, prev, next, start, end, count, + mas_set(mas_detach, 1); + unmap_region(mm, mas_detach, vma, prev, next, start, end, count, !unlock); /* Statistics and freeing VMAs */ - mas_set(&mas_detach, 0); - remove_mt(mm, &mas_detach); + mas_set(mas_detach, 0); + remove_mt(mm, mas_detach); validate_mm(mm); if (unlock) mmap_read_unlock(mm); - __mt_destroy(&mt_detach); - return 0; + __mt_destroy(mas_detach->tree); +} -clear_tree_failed: -userfaultfd_error: -munmap_gather_failed: -end_split_failed: - mas_set(&mas_detach, 0); - mas_for_each(&mas_detach, next, end) - vma_mark_detached(next, false); +/* + * do_vmi_align_munmap() - munmap the aligned region from @start to @end. + * @vmi: The vma iterator + * @vma: The starting vm_area_struct + * @mm: The mm_struct + * @start: The aligned start address to munmap. + * @end: The aligned end address to munmap. + * @uf: The userfaultfd list_head + * @unlock: Set to true to drop the mmap_lock. unlocking only happens on + * success. + * + * Return: 0 on success and drops the lock if so directed, error and leaves the + * lock held otherwise. + */ +static int +do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, + struct mm_struct *mm, unsigned long start, + unsigned long end, struct list_head *uf, bool unlock) +{ + struct maple_tree mt_detach; + MA_STATE(mas_detach, &mt_detach, 0, 0); + mt_init_flags(&mt_detach, vmi->mas.tree->ma_flags & MT_FLAGS_LOCK_MASK); + mt_on_stack(mt_detach); + int error; + unsigned long locked_vm = 0; - __mt_destroy(&mt_detach); -start_split_failed: -map_count_exceeded: + error = vmi_gather_munmap_vmas(vmi, vma, mm, start, end, uf, + &mas_detach, &locked_vm); + if (error) + goto gather_failed; + + error = vma_iter_clear_gfp(vmi, start, end, GFP_KERNEL); + if (error) + goto clear_area_failed; + + vmi_complete_munmap_vmas(vmi, vma, mm, start, end, unlock, &mas_detach, + locked_vm); + return 0; + +clear_area_failed: + abort_munmap_vmas(&mas_detach); +gather_failed: validate_mm(mm); return error; } From patchwork Fri May 31 16:32:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Liam R. Howlett" X-Patchwork-Id: 13681892 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3FD0C1649D1; Fri, 31 May 2024 16:33:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.177.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717173215; cv=fail; b=OiKWo2XZ3W7BwofuBP+hpACn5KMfZQk4RH0IeTFrgwolYun4cxtAJmPdVbLk+0k/Hz/aFKuC+7TIrjduqrZeZrpS7Pxw5UilS6R4i3ao/BgHCweW24sxB+mZWTxPHPbauPdm5WTlaoKOwLUcGLpG8X/Btpu0WB2mbsKyGKwrPk8= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717173215; c=relaxed/simple; bh=Co/pl70fYMu2sj7o1fDOxN5OoHkk0lAPZRamHbaBOZI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=oX9kiOaUq5Rfu/7GZGRj70XNGJ4Dc16IKC+xk2HeEcT66Etdh2mG464mAWB5CemjiW750NA1QlMue1RbgKYyWvhLmcyjN88jSWcYbKm5E0/bRAJNJ1gBEHcYrLALkc40gw4+gFha7mc5D3HqMcy+DrqwIxAsySaqxyqncZ96mcM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; arc=fail smtp.client-ip=205.220.177.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 44V9T95H006927; Fri, 31 May 2024 16:33:20 GMT DKIM-Signature: =?utf-8?q?v=3D1=3B_a=3Drsa-sha256=3B_c=3Drelaxed/relaxed=3B_?= =?utf-8?q?d=3Doracle=2Ecom=3B_h=3Dcc=3Acontent-transfer-encoding=3Acontent-?= =?utf-8?q?type=3Adate=3Afrom=3Ain-reply-to=3Amessage-id=3Amime-version=3Are?= =?utf-8?q?ferences=3Asubject=3Ato=3B_s=3Dcorp-2023-11-20=3B_bh=3D377W4Om/B0?= =?utf-8?q?uMZ6r68PRB5C4y7gbR0BZHCpVT9AmPfiA=3D=3B_b=3DNc5BWX3ZqaFj9IcP94zLj?= =?utf-8?q?vBXkGxszs40HZHHCcHP8+Axn1d1y+XJ5o9hugPvH0rOy8kt_JoTTBab46XbuN80mw?= =?utf-8?q?8cC/UvI7RE5z7Ayo7RvRZhYjZkBmQCJG22+DVDYySHASrmYfw5j_G7bPitQLU3B/K?= =?utf-8?q?wJDyGvYm+CeihS0AmAuCQpI1XKWbBLknRf+bGg3MhVx6wq1q6WeMmN2_XCyxg3pG4?= =?utf-8?q?+mqhv6bMNXB+G4DPJEj2RtpzFKH+bW9gHID4CoON9Ey5HOzd3L0hU/Qj6kc_aIKF0?= =?utf-8?q?0mIobiFjg9SWCuDacQM+2+tnJagmPVDogbYamE4qx/yAumTqgWt7vU3SvNivEXL_H?= =?utf-8?q?Q=3D=3D_?= Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3yb8g9urhk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 31 May 2024 16:33:20 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 44VFaLYb006199; Fri, 31 May 2024 16:33:19 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2168.outbound.protection.outlook.com [104.47.56.168]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3yd7c8kmkf-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 31 May 2024 16:33:19 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=D2VW0eyokTJJ13ulzssSpElTKSnd/MDL86qKxoE0peNuSsSwWKM9Z2FX+s2ZxnNG/1HZHc4tk4TpBdzJSbUxTxJfFXAXlIansBbao7uPwymsuSCPMds0Kk/r6SIZnQNY3ZKu6nQphjzOyaveYP8vi3wITqSOSLlWHHHUGvMFUlYEDArMqFly6ksOU/5VTpYn7N9RNyzT60D3BWqg8W9dHAvnDXrDscQwznFbT4i3o2N2G+ML/odKRk/tCf14g+J1TZ7tBfrB+2HCDbA83RNYHvXpH8+kSwDaag65BCP/n/r3EBf1L27o3MrAiRZW2vCumgbMDmbyxhwB37xTYQqH5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=377W4Om/B0uMZ6r68PRB5C4y7gbR0BZHCpVT9AmPfiA=; b=ZRrMZ0JynuzPfaa2UWEibn6s3n0Bd9S1kHHPeBVgyBNXm71pwEah46NvE/39gKXSDWLqOH1aU8fDJBcngjigYsKseRamDv8CFDooJyhtnssb15YjphjIEQhSCrEsFfH2/xmqEPQkm3L6X1lpkayI8gsr3nuy6bJQyikYBPnaoIfQJsZvdpBEoHo3gGJuGhW33nL4Dkyb8eMAv41X19toPrEnMY5uSRE0COZtsC3Ub0G3VWfP4D/BIspLcmn7zh2by4+HuV+VJbGMwkV2QPO1/VDo6tpgnq1J37oAAj0uXapwSZWr0/tLlt7AYBT6APJuSH6WaF55GNaTx9BS3CA4ow== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=377W4Om/B0uMZ6r68PRB5C4y7gbR0BZHCpVT9AmPfiA=; b=WPLVunZgPnwGGg/LC9X1PtOyaZBUpSHn6Y+yD66UfdTrPtZxEwh9u3Vp9C/A80wo+6WViFxMVYkbyi2RYfmgHageV58UkBytrLmOej01cb90xzjDFjU08Tw0yiGpe6e4Vka6ddzmMrVHKp2VvpWAYG7/yBd25eVPUKKo2NbrQ88= Received: from DS0PR10MB7933.namprd10.prod.outlook.com (2603:10b6:8:1b8::15) by SA1PR10MB6318.namprd10.prod.outlook.com (2603:10b6:806:251::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7633.24; Fri, 31 May 2024 16:32:48 +0000 Received: from DS0PR10MB7933.namprd10.prod.outlook.com ([fe80::2561:85b0:ae8f:9490]) by DS0PR10MB7933.namprd10.prod.outlook.com ([fe80::2561:85b0:ae8f:9490%7]) with mapi id 15.20.7633.021; Fri, 31 May 2024 16:32:48 +0000 From: "Liam R. Howlett" To: Suren Baghdasaryan , Andrii Nakryiko Cc: Vlastimil Babka , sidhartha.kumar@oracle.com, Matthew Wilcox , Lorenzo Stoakes , "Liam R . Howlett" , linux-fsdevel@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 3/5] mm/mmap: Introduce vma_munmap_struct for use in munmap operations Date: Fri, 31 May 2024 12:32:15 -0400 Message-ID: <20240531163217.1584450-4-Liam.Howlett@oracle.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240531163217.1584450-1-Liam.Howlett@oracle.com> References: <20240531163217.1584450-1-Liam.Howlett@oracle.com> X-ClientProxiedBy: YT4PR01CA0436.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:10d::8) To DS0PR10MB7933.namprd10.prod.outlook.com (2603:10b6:8:1b8::15) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR10MB7933:EE_|SA1PR10MB6318:EE_ X-MS-Office365-Filtering-Correlation-Id: a2ae8b52-7eeb-478e-c1a3-08dc818f4efd X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|1800799015|376005|366007; X-Microsoft-Antispam-Message-Info: +JN2XF8F2r3p9g/1Xx/JsLr+MMuRyXHU5gZUF2soCPhpaz+Y1DgZ6Z22ApKBOz7pKssgUO6dPqRwKv/hPAdYWDU5LbO7DUmTKfAoPPuOfKLG662foxVQ6BqTTtQ5Si2n1EnWSS1j5Bmgt6fC5kT9Zads53JUDbO/qRJgu9W9WFmGsGt99H54iwgqu9R7NANCuQOgCmb0QrkoJZNUv1NN12erH4jzEbSJGdNj/ozZC8SOaxtn8Jyyn32ge9IpFreBOzeJZkmARR1U61W+lh+RrWgYPdYAZYVgEq7VJs61ePIf6liYj+DeYM8YhKmXdDVuVqi1BcmILSMf9taLL0E6KKUalgwVRIZjFokOorQclZMmvMIaFj/N3jM7outXUVKZ3cv2z3GuRLKUrYHficsx5fAzgHlFjHYsQt1aciuMpOasxmqiPUpKG2KmVEVwhJRz+BIA+IHlP4zLRWcLF80lrKiV9fUmqXaSMWsdS+J8Afpd6LaSn8AUcrpPFfBG2T4o3hXCZ6vnVLlNiVqgD50jdt1gTu5uSmPBe8ANBlon4vfJ/IaGNiFms4iZYORd47eBXl4+szfeWLaCGNKPL1qqnOUJSglhd1ye9fC+a+n9zI0yrO2c7vM+dNexgDkgR/gMWS7TVnQ8gnI/k00SN6T3vyXzpsKYGWJyo+JKsYe/+SHDIiJ8hmBViAwBuDxQyErL+nIgKaBZXmpscIFWMKUB/j2zPlHIuSCANvd4KPFeRZSwkOusN+TkgGJ4Ye5NwcmA6f8618b2OQ2FZHF5YuOAD5QrZa2PrdM/qxP9EZFLcP5yVMa9WfZtGT1c5jvKxyX+Lm5csq6PMEASFz/dXzbsi5jLGkg7lienTpvp2JjtqP3CB5YVdUwKPPgzixhyNGw37APIgRYH6J7AfptLuqLuzjg56Bd09odO28WVkFHbdLO1LvxtjoO6lta1x8Vp7x9i3FqB1frY7uKpIJeFamtnD8n3tX7z1MZjo+M9yJLYPns09ZYR3C3pyNodTbEaIkszPtgSM/LnuFJ6BXSpniFl0TYlB14B2EfrOnvHlhZEd+169hx87tl+BzwrvjX7DLsBYpxawr8U9NV4NbtyCg9Zpde9sPMsnmHV+l8+DHhqTnOUngyYT+rSw4IuXJvaN65r7k2rIOkYRtwTLAgYctsb0ymdXYeBDQlPvL22PH7PZtFWLVBa2Qq44wVfHL77MuVSg4S1Fh/SBuUzeC1ghAJ7sX6MXlkIknCp7F9qMt3qBVyW/luAQ1f8Z6H+V8Gh0EEQh4HDX1LfGxK05mZVpWHObw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS0PR10MB7933.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(376005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: n27W3g7Eiil5kPYrS2vQaMj4H48E+87bHEpAZ5Kawd0u3AahCvkrbhc6/sKkOOoF9I03XbjM18q1u4fLfhSgH4IwDPzOmB0p9SC8huTERKWknfb8ukJE7BQAEPXamidxB54gl+Ex1D28iq0FhOMUIyRm9OJ3KrM9239khe3qCpJ8C7t961KHC+bJPBB9g7aMCMPfn5OBk3p10ttAXWXDerPP8MXtPwHQ6lVe+6dCUkfT45tfhdjcBMuXcYGxFC/4hR4xsWPEv4avbqGMZ4aIo/SWJkdpgr1GeO97+KpkFWUGjozC3eoQvzgq/dTSM5mt2zg1qW9Y/+6yJa5Zg57w/Ahwz/IO89LvK8SDPaYKbJ7l+fIlh1XyUixUucu6DS97QJEx1AlagVQ+8MEvEO6qoMdUjzwynJCN77YghOngUxP7UZ34Ju9bb2JYaBLrNRkWVvCltdTVLDnRxVnzNxA168XVN+CHh/TdXseQvbF56q6dy7MssirYLy5pQvaCxfGihH1gkItUyeMfs9GX2KKUabRL7e4QyWAN0/XPINQBnYAqK08MrUjj+lK2ItxDDFCO7zpcUlrb41JPgXWRowonj/hDnwvjTPpFgseqDb/mPbc515cqt3xN/UYetzF5S1Rh10gXGCzsuz9F4s7GXeIaXiF5bac2uxWEjlqIvoJeB+wHcFAl4qpS6fGSNudaISNk6XMVC2i58QMAw2sMsKNFehoczsBO1XPxBreOHi+2AGYK3K3hLBBhz1jKA/bKF0KwuACdS3mH8GPewNqLYUC4cyoh2WA35+4etfRtS7Zue60/B2vYPl4MW+z8YznFCVhsN+OewqpV0eErVV81Tbfrj6Nro+t6Ed1tyfAkxowxbCNMli25LKjP/veeCc+LTvXROrge+0MZoBOY9IfIOQzk9uMsGNRuna1IVW2NKuObX8i3eL5EuQhyiC+KuQ37D/4ObvDIWs9qCrwdTaKDNxaCtVlNrBKi4jRNLmb7BHnfDISbGfDzCTknmQnzwFiUtjlwFS+yVze6rp0vRCmrgNgAYRxag72iZVx3Ng/gjKDHedjYmMbkuI1YiQc5mlOpdc8qeI6YDPyGuAyy81fFvDi/cDKBNIK239DCp2u5VDoaFW1gXF2epNsAdRpje4GnfERpf8QpVKEQWZlHYQxpBqOySuFWVwuOz1WxAE4W06OH0+qnF/VohydoFQUWOvuxYvpHGrO3CRDjoGMjLSw7wsFDvVv7pNQZgR700tUvOXjI9kRdrizb5OuIyAaOiVZrAPrLJ6pLct37mnsi1JlUwjoM7iLHj/rPHM9YI8p6CfwMxxWEkLCqi6Dak612Z425oJMiXCsH/+nyKd2Q1MOlmqL9R7CPjWQM7AgFqAoXzBjAth9i0dqr1yxUOwMwIzgAoYd3nB8ThmCGjlx1+FhhVR0Mpv6/nsstOxU1eMsu9JVjCbg2SfYQo4WGYryp6SWGBjk3KobsqT1PQmq3LX34UqYOTd6rOwuhO35duFiiAFxq7iPqNykv65MuBDOhTdGzl+HutSaSU5WnVg4xB+Ml9BuJKXYnPfXMIoxrWNV4i8clC7c4tescVqPhH489olUmQYoA X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: OLS7ICUOL3wQeweKR8PBFPeYcMwNP+gZuUkPPT6ZTudBKpDwvEjF8JSwpwLqlfe3hiP4YdXu59ICsYT9InimNTgu9g8Fdt5rB8kTa+f0djF5Nr7EuQEEOnVMwTSZf026+eAPeTVWRZ/c4eNMw5Sf15ZWwqryzqfyByOLb9MXOphGzoDFo9T5JmCmTalQ/wMjQokkXsqIWqcBQ3vMj3ZTDEfMuaPtZrbVO27hTTE0THmkUtj9MWFtUeGMsVII8eU4qq1ehhZ+oCk77uiLtVWfmGj/lQ6XRHHsccgjlqlTHK5p03FH7wg9k5YCSdNqKsn2va3TyvA3qHZXf7ykQ2Y1k1+xDfdzjjdApvtnguWcRcF+XBKNJAy95L9S4+nhKAhThQDXkHitaeqn1u1Xn4lnhFFTnZEDPUJBkJGcr0zeRv/k+8xnCZVRyEdFP8/+dKNfuy9lAd4HMsczVm/L/7v3LWaZBUHk8kr34OnCpc2TYJuRDrU+/lwzlgC3hmIyq5vwgGh9gpSqvmsIldY9tNVcUOkdOS+xyT5Ft1LWKA5EmqiAcZuwewnpa+Bm+yibkHW5q8wcXQubQn4uMLWK68LfitNXMHaf2PqNQ81jGnyj52Y= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: a2ae8b52-7eeb-478e-c1a3-08dc818f4efd X-MS-Exchange-CrossTenant-AuthSource: DS0PR10MB7933.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 May 2024 16:32:48.6502 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: fR0D2Oeu1t71C3gBCiKE0QnXptzfLzc6zVYLK0KBWId7pQ30ZdULmcgRIMObgNqqj5frYzBhZ8NYEszM72rZ4w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB6318 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.12.28.16 definitions=2024-05-31_12,2024-05-30_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 bulkscore=0 suspectscore=0 mlxscore=0 adultscore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2405010000 definitions=main-2405310125 X-Proofpoint-GUID: lkIdZBPl9AUwRW4GLUKwWtsXNxBc3z4T X-Proofpoint-ORIG-GUID: lkIdZBPl9AUwRW4GLUKwWtsXNxBc3z4T X-Patchwork-State: RFC Use a structure to pass along all the necessary information and counters involved in removing vmas from the mm_struct. Signed-off-by: Liam R. Howlett Reviewed-by: Suren Baghdasaryan --- mm/internal.h | 16 ++++++ mm/mmap.c | 133 +++++++++++++++++++++++++++++--------------------- 2 files changed, 94 insertions(+), 55 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index b2c75b12014e..6ebf77853d68 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1428,6 +1428,22 @@ struct vma_prepare { struct vm_area_struct *remove2; }; +/* + * vma munmap operation + */ +struct vma_munmap_struct { + struct vma_iterator *vmi; + struct mm_struct *mm; + struct vm_area_struct *vma; /* The first vma to munmap */ + struct list_head *uf; /* Userfaultfd list_head */ + unsigned long start; /* Aligned start addr */ + unsigned long end; /* Aligned end addr */ + int vma_count; /* Number of vmas that will be removed */ + unsigned long nr_pages; /* Number of pages being removed */ + unsigned long locked_vm; /* Number of locked pages */ + bool unlock; /* Unlock after the munmap */ +}; + void __meminit __init_single_page(struct page *page, unsigned long pfn, unsigned long zone, int nid); diff --git a/mm/mmap.c b/mm/mmap.c index fad40d604c64..57f2383245ea 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -459,6 +459,31 @@ static inline void init_vma_prep(struct vma_prepare *vp, init_multi_vma_prep(vp, vma, NULL, NULL, NULL); } +/* + * init_vma_munmap() - Initializer wrapper for vma_munmap_struct + * @vms: The vma munmap struct + * @vmi: The vma iterator + * @vma: The first vm_area_struct to munmap + * @start: The aligned start address to munmap + * @end: The aligned end address to munmap + * @uf: The userfaultfd list_head + * @unlock: Unlock after the operation. Only unlocked on success + */ +static inline void init_vma_munmap(struct vma_munmap_struct *vms, + struct vma_iterator *vmi, struct vm_area_struct *vma, + unsigned long start, unsigned long end, struct list_head *uf, + bool unlock) +{ + vms->vmi = vmi; + vms->vma = vma; + vms->mm = vma->vm_mm; + vms->start = start; + vms->end = end; + vms->unlock = unlock; + vms->uf = uf; + vms->vma_count = 0; + vms->nr_pages = vms->locked_vm = 0; +} /* * vma_prepare() - Helper function for handling locking VMAs prior to altering @@ -2340,7 +2365,6 @@ static inline void remove_mt(struct mm_struct *mm, struct ma_state *mas) if (vma->vm_flags & VM_ACCOUNT) nr_accounted += nrpages; - vm_stat_account(mm, vma->vm_flags, -nrpages); remove_vma(vma, false); } @@ -2562,29 +2586,20 @@ static inline void abort_munmap_vmas(struct ma_state *mas_detach) } /* - * vmi_gather_munmap_vmas() - Put all VMAs within a range into a maple tree + * vms_gather_munmap_vmas() - Put all VMAs within a range into a maple tree * for removal at a later date. Handles splitting first and last if necessary * and marking the vmas as isolated. * - * @vmi: The vma iterator - * @vma: The starting vm_area_struct - * @mm: The mm_struct - * @start: The aligned start address to munmap. - * @end: The aligned end address to munmap. - * @uf: The userfaultfd list_head + * @vms: The vma munmap struct * @mas_detach: The maple state tracking the detached tree * * Return: 0 on success */ -static int -vmi_gather_munmap_vmas(struct vma_iterator *vmi, struct vm_area_struct *vma, - struct mm_struct *mm, unsigned long start, - unsigned long end, struct list_head *uf, - struct ma_state *mas_detach, unsigned long *locked_vm) +static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms, + struct ma_state *mas_detach) { struct vm_area_struct *next = NULL; int error = -ENOMEM; - int count = 0; /* * If we need to split any vma, do it now to save pain later. @@ -2595,17 +2610,18 @@ vmi_gather_munmap_vmas(struct vma_iterator *vmi, struct vm_area_struct *vma, */ /* Does it split the first one? */ - if (start > vma->vm_start) { + if (vms->start > vms->vma->vm_start) { /* * Make sure that map_count on return from munmap() will * not exceed its limit; but let map_count go just above * its limit temporarily, to help free resources as expected. */ - if (end < vma->vm_end && mm->map_count >= sysctl_max_map_count) + if (vms->end < vms->vma->vm_end && + vms->mm->map_count >= sysctl_max_map_count) goto map_count_exceeded; - error = __split_vma(vmi, vma, start, 1); + error = __split_vma(vms->vmi, vms->vma, vms->start, 1); if (error) goto start_split_failed; } @@ -2614,24 +2630,24 @@ vmi_gather_munmap_vmas(struct vma_iterator *vmi, struct vm_area_struct *vma, * Detach a range of VMAs from the mm. Using next as a temp variable as * it is always overwritten. */ - next = vma; + next = vms->vma; do { /* Does it split the end? */ - if (next->vm_end > end) { - error = __split_vma(vmi, next, end, 0); + if (next->vm_end > vms->end) { + error = __split_vma(vms->vmi, next, vms->end, 0); if (error) goto end_split_failed; } vma_start_write(next); - mas_set(mas_detach, count++); + mas_set(mas_detach, vms->vma_count++); if (next->vm_flags & VM_LOCKED) - *locked_vm += vma_pages(next); + vms->locked_vm += vma_pages(next); error = mas_store_gfp(mas_detach, next, GFP_KERNEL); if (error) goto munmap_gather_failed; vma_mark_detached(next, true); - if (unlikely(uf)) { + if (unlikely(vms->uf)) { /* * If userfaultfd_unmap_prep returns an error the vmas * will remain split, but userland will get a @@ -2641,16 +2657,17 @@ vmi_gather_munmap_vmas(struct vma_iterator *vmi, struct vm_area_struct *vma, * split, despite we could. This is unlikely enough * failure that it's not worth optimizing it for. */ - error = userfaultfd_unmap_prep(next, start, end, uf); + error = userfaultfd_unmap_prep(next, vms->start, + vms->end, vms->uf); if (error) goto userfaultfd_error; } #ifdef CONFIG_DEBUG_VM_MAPLE_TREE - BUG_ON(next->vm_start < start); - BUG_ON(next->vm_start > end); + BUG_ON(next->vm_start < vms->start); + BUG_ON(next->vm_start > vms->end); #endif - } for_each_vma_range(*vmi, next, end); + } for_each_vma_range(*(vms->vmi), next, vms->end); #if defined(CONFIG_DEBUG_VM_MAPLE_TREE) /* Make sure no VMAs are about to be lost. */ @@ -2659,21 +2676,21 @@ vmi_gather_munmap_vmas(struct vma_iterator *vmi, struct vm_area_struct *vma, struct vm_area_struct *vma_mas, *vma_test; int test_count = 0; - vma_iter_set(vmi, start); + vma_iter_set(vms->vmi, vms->start); rcu_read_lock(); - vma_test = mas_find(&test, count - 1); - for_each_vma_range(*vmi, vma_mas, end) { + vma_test = mas_find(&test, vms->vma_count - 1); + for_each_vma_range(*(vms->vmi), vma_mas, vms->end) { BUG_ON(vma_mas != vma_test); test_count++; - vma_test = mas_next(&test, count - 1); + vma_test = mas_next(&test, vms->vma_count - 1); } rcu_read_unlock(); - BUG_ON(count != test_count); + BUG_ON(vms->vma_count != test_count); } #endif - while (vma_iter_addr(vmi) > start) - vma_iter_prev_range(vmi); + while (vma_iter_addr(vms->vmi) > vms->start) + vma_iter_prev_range(vms->vmi); return 0; @@ -2686,38 +2703,44 @@ vmi_gather_munmap_vmas(struct vma_iterator *vmi, struct vm_area_struct *vma, return error; } -static void -vmi_complete_munmap_vmas(struct vma_iterator *vmi, struct vm_area_struct *vma, - struct mm_struct *mm, unsigned long start, - unsigned long end, bool unlock, struct ma_state *mas_detach, - unsigned long locked_vm) +/* + * vmi_complete_munmap_vmas() - Update mm counters, unlock if directed, and free + * all VMA resources. + * + * do_vmi_align_munmap() - munmap the aligned region from @start to @end. + * @vms: The vma munmap struct + * @mas_detach: The maple state of the detached vmas + * + */ +static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms, + struct ma_state *mas_detach) { struct vm_area_struct *prev, *next; - int count; + struct mm_struct *mm; - count = mas_detach->index + 1; - mm->map_count -= count; - mm->locked_vm -= locked_vm; - if (unlock) + mm = vms->mm; + mm->map_count -= vms->vma_count; + mm->locked_vm -= vms->locked_vm; + if (vms->unlock) mmap_write_downgrade(mm); - prev = vma_iter_prev_range(vmi); - next = vma_next(vmi); + prev = vma_iter_prev_range(vms->vmi); + next = vma_next(vms->vmi); if (next) - vma_iter_prev_range(vmi); + vma_iter_prev_range(vms->vmi); /* * We can free page tables without write-locking mmap_lock because VMAs * were isolated before we downgraded mmap_lock. */ mas_set(mas_detach, 1); - unmap_region(mm, mas_detach, vma, prev, next, start, end, count, - !unlock); + unmap_region(mm, mas_detach, vms->vma, prev, next, vms->start, vms->end, + vms->vma_count, !vms->unlock); /* Statistics and freeing VMAs */ mas_set(mas_detach, 0); remove_mt(mm, mas_detach); validate_mm(mm); - if (unlock) + if (vms->unlock) mmap_read_unlock(mm); __mt_destroy(mas_detach->tree); @@ -2746,11 +2769,12 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, MA_STATE(mas_detach, &mt_detach, 0, 0); mt_init_flags(&mt_detach, vmi->mas.tree->ma_flags & MT_FLAGS_LOCK_MASK); mt_on_stack(mt_detach); + struct vma_munmap_struct vms; int error; - unsigned long locked_vm = 0; - error = vmi_gather_munmap_vmas(vmi, vma, mm, start, end, uf, - &mas_detach, &locked_vm); + init_vma_munmap(&vms, vmi, vma, start, end, uf, unlock); + + error = vms_gather_munmap_vmas(&vms, &mas_detach); if (error) goto gather_failed; @@ -2758,8 +2782,7 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, if (error) goto clear_area_failed; - vmi_complete_munmap_vmas(vmi, vma, mm, start, end, unlock, &mas_detach, - locked_vm); + vms_complete_munmap_vmas(&vms, &mas_detach); return 0; clear_area_failed: From patchwork Fri May 31 16:32:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Liam R. Howlett" X-Patchwork-Id: 13681891 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 278C915D5B6; Fri, 31 May 2024 16:33:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.177.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717173214; cv=fail; b=NCfQF47I3x2bi6a+AvKhHg0KTutn3XBEvFg8Zr+BBC3gzIrX3QMsABHWDN9yBHENXtNYlCS0VXBcDHkFXF3hrVGZf/ZtYhV/OWun8I8dCkQhHPeHb+Ew4+pWORR+SpkYDtCtNWlH88l2c4GQ5UdIrKfHFMBobct4x3PMKURIlVs= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717173214; c=relaxed/simple; bh=VuueP+ZtLgsfFDTdX/zzHpbHSPSgkI1wr/i+2oHi6CE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=Zu7Uf62dj4MdqClF0GQT1Y5R3jmI0uP5TLhiIHH0ZRntmakd9q3wy4Sw8TyJbuMOP9nk1aYoVMGyqXJGAZqM/ECRfGPkiz0xOI/XhERmWIImjYtpd45bGF6KDgM8eiUL7EKSMAIXdqJ2ohdGEOOWfPetcANbosvZArY2XQSioeQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; arc=fail smtp.client-ip=205.220.177.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 44V9TGCi025400; Fri, 31 May 2024 16:33:21 GMT DKIM-Signature: =?utf-8?q?v=3D1=3B_a=3Drsa-sha256=3B_c=3Drelaxed/relaxed=3B_?= =?utf-8?q?d=3Doracle=2Ecom=3B_h=3Dcc=3Acontent-transfer-encoding=3Acontent-?= =?utf-8?q?type=3Adate=3Afrom=3Ain-reply-to=3Amessage-id=3Amime-version=3Are?= =?utf-8?q?ferences=3Asubject=3Ato=3B_s=3Dcorp-2023-11-20=3B_bh=3DkcuZ6HWEjt?= =?utf-8?q?3pdm7wXmr0/LqSv+K42t9RFLa8SzV94aQ=3D=3B_b=3DbmpNQHRenD+MHzdp2CFYX?= =?utf-8?q?f4gEIlvhPPfjcgoz+OOwrT5nCohZ34uIXvsg/uMKK5jdKyp_c41Xa1LCCHefGC2nf?= =?utf-8?q?4xWsgaAKRWzxPxkfSJCRsRDnnCJJ2v5tVXAV2R5ZFrXNtsHy7GG_Opb7CCF/TUrEW?= =?utf-8?q?5AFYAzsVXsfTXQNGPeZQDqDIAnm855lH8eMZMtONYxPLvb4nYbGZJee_V7mD+OTe1?= =?utf-8?q?x0ns2mANOeikf1iUOuR+sR2wTc/epduPMzDeEbktH4f/dPChbCImQ7N1qq2_Ldf18?= =?utf-8?q?APCWEhcrukbnl7AtznPiLIPv6RUMyHQVDh+5ytxFKnJ98AO2MOAkYUAL+mmjyTr_Q?= =?utf-8?q?g=3D=3D_?= Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3yb8j8besx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 31 May 2024 16:33:20 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 44VFaLYd006199; Fri, 31 May 2024 16:33:19 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2168.outbound.protection.outlook.com [104.47.56.168]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3yd7c8kmkf-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 31 May 2024 16:33:19 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=O6s7FcKOaBR0Ibh5zs+yA4omCokneZHhZfUR18HMkr5Ax/ZgPTbYDZF2rG50AtOd1dwo//INjCgOgrKkuBW2mFrPBr1H6qbfBJnOGISaG2DS8/NuFMSA0DMQnto6b9QW0wHJN9jJkKdyuhfFOVJ8fHMawOlhh3tWj3p1pqaqgezXXE1AzTaU6zHZhntXjOhApRhAHB6SCieYiQbbRmMUih69nvUGMd52LS48fBEUiP3eOCfMohDEb0qdx4+GySPYPZ00AdDj1e8ZrLxCSwOOcMXnpxseYDtCnESbd2vPLOGDVJ8MK7GZpi/mZ8A4ngu653iKQndTAu/dUNvNMlac5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=kcuZ6HWEjt3pdm7wXmr0/LqSv+K42t9RFLa8SzV94aQ=; b=T8muiu4oDyR4k3SXEjgB1WsiY6x1XV21dxIpUrbJuH29CIz3G631nPLjdZC4wLEvcC6aIPtwG+Cvc/dvJwk6wpOTp6iS8Ea4dfLC3GggBX3GR6C4MmWDnUYsmyRcUr/eC33gwCy3kPNIIkKYG6gRiC7HphpoEYpispANn3sO/yXU9zM5TRZuFj+Bzd2/bugEM0LPQaIa4mrZgRDka2n8h58xDJ4//HS0MARke8uXrHNp1tg0FHkl3iw0fLlKCKhV0TAoamLPO806voi7YkFazSJEYZA79/EhqmLSejKrGZX0FmEUs2rrq4qU8BVHNURBBrDe7ACMSmBSe+qsTnOjVA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kcuZ6HWEjt3pdm7wXmr0/LqSv+K42t9RFLa8SzV94aQ=; b=WxvShYD2ZxQk0Zo9QZPFZM6V+7vN4BdZU/n7JYRCTZNS14cgOBuSDEiwIjBEbgGGzE4qSumqbXtw/TiLnk0MyygJ3xVYp3OTMpca4C6Hae8q/ErvI7bbxJ0UmmZFuyE/b1J4f4TxQvMnuQBx2mtUdit/cXt+ZW7/220fPAtCrx8= Received: from DS0PR10MB7933.namprd10.prod.outlook.com (2603:10b6:8:1b8::15) by SA1PR10MB6318.namprd10.prod.outlook.com (2603:10b6:806:251::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7633.24; Fri, 31 May 2024 16:32:51 +0000 Received: from DS0PR10MB7933.namprd10.prod.outlook.com ([fe80::2561:85b0:ae8f:9490]) by DS0PR10MB7933.namprd10.prod.outlook.com ([fe80::2561:85b0:ae8f:9490%7]) with mapi id 15.20.7633.021; Fri, 31 May 2024 16:32:50 +0000 From: "Liam R. Howlett" To: Suren Baghdasaryan , Andrii Nakryiko Cc: Vlastimil Babka , sidhartha.kumar@oracle.com, Matthew Wilcox , Lorenzo Stoakes , "Liam R . Howlett" , linux-fsdevel@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 4/5] mm/mmap: Change munmap to use vma_munmap_struct() for accounting and surrounding vmas Date: Fri, 31 May 2024 12:32:16 -0400 Message-ID: <20240531163217.1584450-5-Liam.Howlett@oracle.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240531163217.1584450-1-Liam.Howlett@oracle.com> References: <20240531163217.1584450-1-Liam.Howlett@oracle.com> X-ClientProxiedBy: YT4PR01CA0483.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:10c::22) To DS0PR10MB7933.namprd10.prod.outlook.com (2603:10b6:8:1b8::15) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR10MB7933:EE_|SA1PR10MB6318:EE_ X-MS-Office365-Filtering-Correlation-Id: 117e2752-cbc9-43cc-cdce-08dc818f5057 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|1800799015|376005|366007; X-Microsoft-Antispam-Message-Info: qKk9okYMg3mNjACHrmH1egStdP6cgEdcvw4LoKf2aK3T7k2RXM0RMXIlsFY5JchfXa1v6TyjWz+7VGHqrHm97PQOPwHsTr7JOjcz20SIwBDi/pg+Owr5+7YkBluYQr+rGNSr/bM9m/ZrvLtLhSP6gTtufRHYYNM1mRZcisfpzL0VECfZ8zX5LkOZmfvY5qgzRSIC/Cxp818XbCWi/iCRzuhKIiQsusyY9AoBtBD3ihK5Tv//MyVM82k/xavnDBfNX4IxaGoXP44dhbYTc+vUJygI7WXJsEGNdYCXfMzOKFB/lyGRT4mv1lS1MD5eBUal8wWhrFMNrg5axrjAOGGSaWaiF2QXtmmjVM3d+1gdxzYK95BRbXkt3D1eepBuygZuXsBoQhjTEwhNWEZ+ir81eYpIgt2yNw2WfDJKRdZkoO6f4ASg7OR0pXVhvjr+J0Ih2is9vcfZ61FbFtOokqdPFUQpAk4c+WvUcm+VMWLuSi77fpImyYeT+CWgQ2Zyto9BycUwK5B7KkgqX0IHDj/QJFaUjlmKVqdv6o2yI28wdF8NBwWyyQFkWylq8lJ/omR2vCFapWqlyR/M+hGPk+IH5WgB4fR64m/a85xzbjF77UM0Jxz+41Hm8Hg+GTp687wn7GZDrsj03fZyOrDQpiXgVW4f3kaxLmH8pIaANxz3ieVfNzcbjcnhzmFPW0HF7+ZhBVIwyLgHs9DN3vt53geNxbl9Mad7RBKRm/9lc5ZjRQZzfnD+pb+vlfrt/SD2yiTLkw5uUELeWZIfA4GJrQ/IjMtjvqXIWjMFOUzIcRTU8IY6i1JyUcd8Cz0gN4CR5FUAXqQvNyZ5qN2DdTpa//2/+w0+pYGkzCHL7Rd9mSNg8ZuhqVm0ZKoJWCXTI4SqmF2Aqt0EjcDAXCbuj3QzxuX+gS01HYmX8JP5BuE30Wu8y1dfkGgLwe0hOkMPxqK7Lp4MQK9WRg+Td+BnR7gRp9Glof7ERBy2u1pUD/EADapBuBwO3U6PDU63LZVYO4PKbRDjemEMymOJa+gkrJ9hxhfpXXyX8TJZOeB3HAaDLja6eoTY6+RCX6FyjUTmMkLlSmHMrR7ExbFWuQRjGTnZbvc5cWyEBJIdWwn5/19w2IxkItCwamtaAA3r758JhkC33H6Mz3Pl1injkw2v+7VgdqHBmwyWU0I0jxYcWmAeHfDzfJg7WVMo/Xent8CZ8LeELddDyr2Y7a8m81f48YWZCUZoRFqLLIiXqPIXij1yZ/ov4qSIIhx09SOya5oh5sdHQT2LTpbJYdQR42HbGPAejxbnkQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS0PR10MB7933.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(376005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: C8OlPvGcUbbWVO180IsXqj2EiG6RDO41Kn6QsdUYFqUQyUCGYiJWCf+iVQmy3GZEiI9VPsBk+8vEOxIR0i0gZyodpY0rHhUFlArPcRoQ3pvS0JvPyVhdCnZEVIt2p4mKf6hZHYCMw/+tQZucjUZaIybnhPWB91oCog7WekU67qajJJ9znHeI5HxiDVvc3XEAltMW7sWIv3zaEVINJgtVjJDI+LPmHdOxdoi6m1OIRPPuSnhLBY5huTEC9nQyTVC14ERGtozaKdmZ9dgdh37wdyCOm1pNGCNhhhhhm0it5IBj1W+IrqOPmukAUuGVGvBRs+zCcFdqH8iuLVPnT+IXXhKKS+4Rlk9g3CH92tKyjtOEtZSpxHAOrqKMXeijkUYKmiBuVyFkTEua0ZyJkJM3z/km6AExGkYbemTf0TCPksuLSa0oHml03/EphreclvoNRulZAMh4e23zXs1DUPzRQ/uqKgf99e1SGKZTp9/nYEx4+yv8Vyq4bfYTz2dET6TAJ7pbOH7P3CCWlXfp/ZDtwFnySzal0ZnDTpEttN4TkqGwVybmDX/1HVkbC9CYglJYdXHdqZdfM177nyonaZ3kVS8P6xVGBBQuIz1aH4i/2ww/MjDGfvhzc4FsSHMoiCiLU6jybOjjxCDTIgUacECXyC41PpF2Qm/WmZAA4p/rKlCYOySnYhwnk4TCwKDeVLGFVdaFT4+Afds1c16z4lQXJ6ifqB6R2axkEIGsqvEdGfCVxMmXG6YYJ32COdjSFzrY6cyIkCKHUtHXQS5vpobSYDC/5b77cadGBI4wNvA6MwpJa2oP64kpG2Lg+zdD63WhR+q6GAIQ1lrQdC+OaHrOUU5QbgLJ+7wEO2xq7/w976ev9Uh+y9uLyfVPZDXOeefi1e2r9PiiDwYTzXiSoAs8P5ApAdoUhVdDlmdEEQBSAiAYLtX72UBFRWTtLbp22hjH/AMzwIs8k5WR3pPN85ItIjsxHpHIOwjIImJsOrBmBQopK4Y9DSIA6/8Vp8L+xjrqQ5tsChal3bsN1ZUBHj0KSnrIyfwB+QcuDNVVvTXg2upjl0UOPHMB9L287VIi6ZOVDePE8IrsvAwUvBAwhdG0DpXVxUjNK/LdD4k51sdLKQTEpRV9WuK79XW+K0vWUht/jrB1qq+in7djI5I2V5yZpGtvCiISYVz/UyFHxP7wjsVnV79EZjIj4wDH7rx2YIKeJd9ElnBf0N5QLknJ6HVB7nrvcdxoXbV/c0DcfOLkX1vgGpIvI/m1nf4QPdPsDwPhFhK84lCMvV96ZUOg+lPLfQxMO8pRWCC0LPwdT2rJ+WCzntLG80yGHsx8iihUHf6JH6NNBroInEIc4DSJLwV85FmayZDgcqWBPbH6eOAD6ZdYCLf5tK4kcDzfTQX/RYpvWhj/D1A6dPukDvySGE1d3ssAOTdXJn1hScdMZxmS3idsVP45DoverqVzGBHhfujFKrIlwrrmrb/osQZSPwL0c7oI8cA3sFkQevqwC2lrOf58rzFqzoVf/vzxg22JAaE+B4UMDVYhStJE1q+oMuXsoN5/otGs0G+tUnOSmpIDoKbbPbElf3Mcd2ucCLwbpaow X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: zMh/hA9oBDZbHPgA+gmVhNQOODBHOp/NYu0QGxUPWJ36mtZweuVZm4NB0xjTLeeVm1DmB2ihNIwjvAhTMobZHsgRJlEnKqe9KLHCYYehUz7WiWvw6MQ+VNyldosmVi/50pykgGeab6ZA56bby0qgAIYy3rnMnkSBDNPLUdiMwNQNEx/mxYqi5N6CMheDa8/lTOOySstOs2azil0E/kyXPYyhWzjy59BEyi5RcysXZU/oVorwZjZWw1EF6Irl631b7jokSiPpaaQ9SyyUtVPM87KKX3wiCMVvIypqTtyMjBn3QeEs9AYHwgCQlok70VAfWzdovxoP1UcCjLo5fiMcSczAWaWarqkC09iAu/9qIjPb2PJo23k4JmJayavDw8pOV8TfKzoj11ys15325GqoFKnhE+Fg5cInfc0Pcq814tz3rAGpLrcXCSJ0Ijdm4NhAUd2iZooGvUKCm51PK7qNwFAogZLuYKf+GH4xo+uOv93d107aIvScRMYn8gNFZUbc2NxqVTD/PtAfxHrUwlyfn/g//hJDAQCRbp1vEQB2rp79NW7ZwZLtyoGheFbOo8e0rSSf+TSzRf6PI1j3XvCXoZM+hRzKbr3Mf8OOfYhFQ18= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 117e2752-cbc9-43cc-cdce-08dc818f5057 X-MS-Exchange-CrossTenant-AuthSource: DS0PR10MB7933.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 May 2024 16:32:50.9091 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Vj49urUZk1GQ+Uvz8ZMe6fPKd1ABwlmUFjhw1Wl94xi8Y9kn4f5baepgZEkOMp9HDJ5DXjfNkAsObh58E4V6ow== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB6318 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.12.28.16 definitions=2024-05-31_12,2024-05-30_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 bulkscore=0 suspectscore=0 mlxscore=0 adultscore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2405010000 definitions=main-2405310125 X-Proofpoint-ORIG-GUID: qBm1ZB_6H5b_O9ARpghh6vTZ0NV7a-Tz X-Proofpoint-GUID: qBm1ZB_6H5b_O9ARpghh6vTZ0NV7a-Tz X-Patchwork-State: RFC Clean up the code by changing the munmap operation to use a structure for the accounting and munmap variables. Since remove_mt() is only called in one location and the contents will be reduce to almost nothing. The remains of the function can be added to vms_complete_munmap_vmas(). Signed-off-by: Liam R. Howlett --- mm/internal.h | 6 ++++ mm/mmap.c | 85 +++++++++++++++++++++++++++------------------------ 2 files changed, 51 insertions(+), 40 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 6ebf77853d68..8c02ebf5736c 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1435,12 +1435,18 @@ struct vma_munmap_struct { struct vma_iterator *vmi; struct mm_struct *mm; struct vm_area_struct *vma; /* The first vma to munmap */ + struct vm_area_struct *next; /* vma after the munmap area */ + struct vm_area_struct *prev; /* vma before the munmap area */ struct list_head *uf; /* Userfaultfd list_head */ unsigned long start; /* Aligned start addr */ unsigned long end; /* Aligned end addr */ int vma_count; /* Number of vmas that will be removed */ unsigned long nr_pages; /* Number of pages being removed */ unsigned long locked_vm; /* Number of locked pages */ + unsigned long nr_accounted; /* Number of VM_ACCOUNT pages */ + unsigned long exec_vm; + unsigned long stack_vm; + unsigned long data_vm; bool unlock; /* Unlock after the munmap */ }; diff --git a/mm/mmap.c b/mm/mmap.c index 57f2383245ea..3e0930c09213 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -482,7 +482,8 @@ static inline void init_vma_munmap(struct vma_munmap_struct *vms, vms->unlock = unlock; vms->uf = uf; vms->vma_count = 0; - vms->nr_pages = vms->locked_vm = 0; + vms->nr_pages = vms->locked_vm = vms->nr_accounted = 0; + vms->exec_vm = vms->stack_vm = vms->data_vm = 0; } /* @@ -604,7 +605,6 @@ static inline void vma_complete(struct vma_prepare *vp, } if (vp->insert && vp->file) uprobe_mmap(vp->insert); - validate_mm(mm); } /* @@ -733,6 +733,7 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma, vma_iter_clear(vmi); vma_set_range(vma, start, end, pgoff); vma_complete(&vp, vmi, vma->vm_mm); + validate_mm(vma->vm_mm); return 0; } @@ -2347,30 +2348,6 @@ struct vm_area_struct *expand_stack(struct mm_struct *mm, unsigned long addr) return vma; } -/* - * Ok - we have the memory areas we should free on a maple tree so release them, - * and do the vma updates. - * - * Called with the mm semaphore held. - */ -static inline void remove_mt(struct mm_struct *mm, struct ma_state *mas) -{ - unsigned long nr_accounted = 0; - struct vm_area_struct *vma; - - /* Update high watermark before we lower total_vm */ - update_hiwater_vm(mm); - mas_for_each(mas, vma, ULONG_MAX) { - long nrpages = vma_pages(vma); - - if (vma->vm_flags & VM_ACCOUNT) - nr_accounted += nrpages; - vm_stat_account(mm, vma->vm_flags, -nrpages); - remove_vma(vma, false); - } - vm_unacct_memory(nr_accounted); -} - /* * Get rid of page table information in the indicated region. * @@ -2625,13 +2602,14 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms, if (error) goto start_split_failed; } + vms->prev = vma_prev(vms->vmi); /* * Detach a range of VMAs from the mm. Using next as a temp variable as * it is always overwritten. */ - next = vms->vma; - do { + for_each_vma_range(*(vms->vmi), next, vms->end) { + long nrpages; /* Does it split the end? */ if (next->vm_end > vms->end) { error = __split_vma(vms->vmi, next, vms->end, 0); @@ -2640,8 +2618,21 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms, } vma_start_write(next); mas_set(mas_detach, vms->vma_count++); + nrpages = vma_pages(next); + + vms->nr_pages += nrpages; if (next->vm_flags & VM_LOCKED) - vms->locked_vm += vma_pages(next); + vms->locked_vm += nrpages; + + if (next->vm_flags & VM_ACCOUNT) + vms->nr_accounted += nrpages; + + if (is_exec_mapping(next->vm_flags)) + vms->exec_vm += nrpages; + else if (is_stack_mapping(next->vm_flags)) + vms->stack_vm += nrpages; + else if (is_data_mapping(next->vm_flags)) + vms->data_vm += nrpages; error = mas_store_gfp(mas_detach, next, GFP_KERNEL); if (error) @@ -2667,7 +2658,9 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms, BUG_ON(next->vm_start < vms->start); BUG_ON(next->vm_start > vms->end); #endif - } for_each_vma_range(*(vms->vmi), next, vms->end); + } + + vms->next = vma_next(vms->vmi); #if defined(CONFIG_DEBUG_VM_MAPLE_TREE) /* Make sure no VMAs are about to be lost. */ @@ -2712,10 +2705,11 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms, * @mas_detach: The maple state of the detached vmas * */ +static inline void vms_vm_stat_account(struct vma_munmap_struct *vms); static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms, struct ma_state *mas_detach) { - struct vm_area_struct *prev, *next; + struct vm_area_struct *vma; struct mm_struct *mm; mm = vms->mm; @@ -2724,21 +2718,21 @@ static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms, if (vms->unlock) mmap_write_downgrade(mm); - prev = vma_iter_prev_range(vms->vmi); - next = vma_next(vms->vmi); - if (next) - vma_iter_prev_range(vms->vmi); - /* * We can free page tables without write-locking mmap_lock because VMAs * were isolated before we downgraded mmap_lock. */ mas_set(mas_detach, 1); - unmap_region(mm, mas_detach, vms->vma, prev, next, vms->start, vms->end, - vms->vma_count, !vms->unlock); - /* Statistics and freeing VMAs */ + unmap_region(mm, mas_detach, vms->vma, vms->prev, vms->next, + vms->start, vms->end, vms->vma_count, !vms->unlock); + /* Update high watermark before we lower total_vm */ + update_hiwater_vm(mm); + vms_vm_stat_account(vms); mas_set(mas_detach, 0); - remove_mt(mm, mas_detach); + mas_for_each(mas_detach, vma, ULONG_MAX) + remove_vma(vma, false); + + vm_unacct_memory(vms->nr_accounted); validate_mm(mm); if (vms->unlock) mmap_read_unlock(mm); @@ -3631,6 +3625,17 @@ void vm_stat_account(struct mm_struct *mm, vm_flags_t flags, long npages) mm->data_vm += npages; } +/* Accounting for munmap */ +static inline void vms_vm_stat_account(struct vma_munmap_struct *vms) +{ + struct mm_struct *mm = vms->mm; + + WRITE_ONCE(mm->total_vm, READ_ONCE(mm->total_vm) - vms->nr_pages); + mm->exec_vm -= vms->exec_vm; + mm->stack_vm -= vms->stack_vm; + mm->data_vm -= vms->data_vm; +} + static vm_fault_t special_mapping_fault(struct vm_fault *vmf); /* From patchwork Fri May 31 16:32:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Liam R. Howlett" X-Patchwork-Id: 13681893 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2796416936F; Fri, 31 May 2024 16:33:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.177.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717173215; cv=fail; b=g2gOwaVyX4ybpKrgsL2pbtfxbT8s9Eynv/4DWQXpntLsi99FfSJ+1/8AuoEzcglDZThUypiRfkKG1l96fmO8ntXBuPY/gbVkmZp29q4Da3NWV+YLnTvHPL4PfnPiYEAQox8rhrGRchZd06HFQN8V5pj2hjInWpepHgTlBCISMEo= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717173215; c=relaxed/simple; bh=SRwFQKtjCmJ83mjUv4rYOpA0MKijt/SfMF8g9KcNM38=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=f4zI+ypbZ5SCbArKqtwjTVB0WopbeGlDoE9YUiLXkgfMkb8071W2n2PoXfauFJN+dvgnjrzp8poLekuaaLTTdpkIfoaf18f/OYG5odo8rHP0F9zpDmhNuyud4fgQmd+v/MMRFMaxoY5O3SaW+ZUpU4ApNdg0PKXDkBGOLFGdc3M= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; arc=fail smtp.client-ip=205.220.177.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 44V9TCfx020158; Fri, 31 May 2024 16:33:22 GMT DKIM-Signature: =?utf-8?q?v=3D1=3B_a=3Drsa-sha256=3B_c=3Drelaxed/relaxed=3B_?= =?utf-8?q?d=3Doracle=2Ecom=3B_h=3Dcc=3Acontent-transfer-encoding=3Acontent-?= =?utf-8?q?type=3Adate=3Afrom=3Ain-reply-to=3Amessage-id=3Amime-version=3Are?= =?utf-8?q?ferences=3Asubject=3Ato=3B_s=3Dcorp-2023-11-20=3B_bh=3D92o/3ASbgy?= =?utf-8?q?xOKlqshYmXS4iV2H2wPgMUuURig1jWyTw=3D=3B_b=3DZqwxUPGoqG7sfApe+fWU2?= =?utf-8?q?hbG8IMNIn0j9602LxIiEJKCZH18YXlJ2M3wPjtfsElvY0Df_rxex/iGUfF5j/rE01?= =?utf-8?q?kJf3h0mCcZ9BdCCHgPTF+D5zqxq2lEBnNR7Jf5WEmwz+ZjYiNJo_WsPhUtww0+Aln?= =?utf-8?q?60hr/xaPstLTWMAMQOi/6mirCKaKjD9/hfTMTdPsMmVppDrXcv8OhMZ_7DRMtxZ9o?= =?utf-8?q?rYdIcFJgZ8hkDxFWAxKDcMQELiLaYJzS9g3wPSN46v6BCxJlRKpPLlwYCi5_+P+nn?= =?utf-8?q?hKWWZPGO84PfErwUINewOhK90eD9OzqVOlaRhZ41gzubevQwOa0u7/YZPXcUlIW_I?= =?utf-8?q?Q=3D=3D_?= Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3yb8kbbm4c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 31 May 2024 16:33:21 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 44VG0HPV010628; Fri, 31 May 2024 16:33:21 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2100.outbound.protection.outlook.com [104.47.58.100]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3yc5125p0g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 31 May 2024 16:33:20 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=m5RvLZR2wjPmN3nud7FSQp0uMim/Ndo61TTTW6a4vCBcLHNEao+oeGRngmzeUP9NBiDCwD7+Og2YlvmuZpef0ri5M1/kgObwqZZzyeAGtxPSiAb8MYqpY/ZEVitpdNdp/q7Lz2neQiX+etfkn2LIlEtx6pOLcKbYgPRb2kkA/w6722mk9bw0RbkbiCPgHl3JbTr4c75uyxtjzqjSaqJb8XH8+O5j8TUz6cFUihXyPRSC7BU9hsPNalgBty9d/or1xkhT3w9tDGAd7++qodwc8EI65eAbw5PsLR9RxP1RzA1k/Bz/l1vKCaE+Wbk4QPgzFUSStOjfrEb77JJJZqUE4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=92o/3ASbgyxOKlqshYmXS4iV2H2wPgMUuURig1jWyTw=; b=lthRaFiC52HIal7T7OkVbioTXqEUwV1coXE3vDThkperYr7fiyqvEBjkT0J2RJD2xQONHSo3KHL6qgkjvrRxHVP2JhhMZe0Ywo6Kwo7zqDu5PMgslRph/EVMXj6ZaHEJnvfJck1B3fCSCFKuhd5cLZ+tjGP8BW1o4D7EMSk7ESLCRyti/brwzG6DSSox+y2zjt0RN9GyaMthtroclzIlamYuocwlHNylRZ8ec1YNvN1Uj4lw832rJIhEWJELmupt7u+djZU9Qhv383E8YRIhG3iMN7sJKsz/2rVxW7KU5EFdqYb5hLEW5LH+WsA9mGD8O37ghty8MlMJgYm2MINT9Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=92o/3ASbgyxOKlqshYmXS4iV2H2wPgMUuURig1jWyTw=; b=q+G+Ihr85RiNd9VQvZC7JDZrLLq0s6YzT6n6yPJJW9wLCs+IPsCt2aLWyoouCwuP2hEaS3+ZXUuFII5of6xryIql6ffV2wtTAsUO3DXud161nIijG2TkofXI/GrY5K88AUCzr1brNX3Tw/RHzgmBFQhCg1GjdX+xOmqzEOwcK1Q= Received: from DS0PR10MB7933.namprd10.prod.outlook.com (2603:10b6:8:1b8::15) by MN2PR10MB4238.namprd10.prod.outlook.com (2603:10b6:208:1d3::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7633.22; Fri, 31 May 2024 16:32:53 +0000 Received: from DS0PR10MB7933.namprd10.prod.outlook.com ([fe80::2561:85b0:ae8f:9490]) by DS0PR10MB7933.namprd10.prod.outlook.com ([fe80::2561:85b0:ae8f:9490%7]) with mapi id 15.20.7633.021; Fri, 31 May 2024 16:32:52 +0000 From: "Liam R. Howlett" To: Suren Baghdasaryan , Andrii Nakryiko Cc: Vlastimil Babka , sidhartha.kumar@oracle.com, Matthew Wilcox , Lorenzo Stoakes , "Liam R . Howlett" , linux-fsdevel@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 5/5] mm/mmap: Use split munmap calls for MAP_FIXED Date: Fri, 31 May 2024 12:32:17 -0400 Message-ID: <20240531163217.1584450-6-Liam.Howlett@oracle.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240531163217.1584450-1-Liam.Howlett@oracle.com> References: <20240531163217.1584450-1-Liam.Howlett@oracle.com> X-ClientProxiedBy: YT4PR01CA0252.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:10f::24) To DS0PR10MB7933.namprd10.prod.outlook.com (2603:10b6:8:1b8::15) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR10MB7933:EE_|MN2PR10MB4238:EE_ X-MS-Office365-Filtering-Correlation-Id: 040325da-7007-4d81-d549-08dc818f515f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|1800799015|376005|366007; X-Microsoft-Antispam-Message-Info: Cn9B/vQ82Y6/07PeXU8BqZaU20HZVjWl9NM5ufNha+G5BmjhrjFJ6kBUpp5blJqGsCkakONIhmLUDCJIEfOc7FmL1ymZkKgW9MHPEFnfbGDUKp2Umvw/4//QBtMYgs4o4kVYIBu/XGBTmG4bsK5OWKy+NNu/Uq3KbIZMQyjFoXAKg1wT70wpiMIawyKPmp5I8208x8Eabu9ZHR5W/SYTubYGqMOpnhMgGeBKi03CvYKS1GcRwHFauTxCcU+tFiIcDp4jdvnWF4eKXqiF5E8I4QThJAApC2+Tj8ckGM3OCRsvtxTsaoo48+XOPM9XLhbCwjeIjUJ3+WvvcrJ35le8Y/npBxujNI5F9Z1Ub1QmTFJ1TldYPiiTAwTeOViYUDEBZpAnxltJvjURFpY+pvvfjvlT9tTbkYFALes7dZr8YaUIiSOamEWgVixwl74Cm4b1jMk0YkIFwZt25+H0fP0iInrRCKDf7nYWndC/GykZZnV3iePpml/hE2+3BSvPSwsRY2XeGqs/BM6MqtCtRYW4xeiV6MzRUf+0RuhhA7gzOZXvXQVVfEgPdRNo7fERHATI1t+EWf/fKQ4GTsh0vAUS7kvyzZy2JMfYVjbLWAlU7oVB5foqZ1JteLW9XDiLXe94XCpwmZECH8b9lQxOWB0ZRDkwgxgc5WOPifV6wfM1WgUjjrw/03Dm1IcQD5wIX+6TCPpQr8FNDLA3dFMPeCk6XaGrlt63QSChkLaUEbf/VSOs5RnOO1Zn4llFCngiGLdYruf17xfQ1wK5Y+JxWMuWH0wCxU90iVUTGNfKgDOfChrcFYuAUyn8JBE0wCqZkat3+1dE9yqI+jvPMb8WDDoSAiE0rA+jyU3Z8YqosXPHy24Dk9dJdgpmFaR7sUg3lcdUsD89i72KWTWpGJe2es0FLc0jc9SQO/zi7L6C7pk2eQ1pC9W4pdjlVfxKHFjjT7AHw0ONq51PyQu0Sahr+NpyAq+9xQVsGPbj5Zgcb4KbtLhDJOS/wDI4AWSXQ17YgcbcOW4bVSXRmp4OseUO81hHC3sLXnDs7KZkoGqvgPG81KccExhMlw+A1UxjzffYTpDl1Ln3Ld1YJH/+dqeUr/vijJZ+fw+8BJ1GjUczB6VUf2Ngbwk80h5dZPynYWN9dgdW6Hnw9oFwapzqREamD4n+uv0TnOpe3V0ybqgraVhsrtTWBjzMvFWGMPSb2ZWJKUpc50VFpCO7JvD0YDtTAjvknJdkSB5Cx2+SvkdK/vLaruhaKMZY8EwZ43KhEwCk+6H9lqtCwALpuu4CqUunoLa/Zw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS0PR10MB7933.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(376005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: UFLXOusTuO8x3pejriTHl3NYb2YA11J5aFQOdRNL7kNdN0bsgC183F34+PygkJBTnPReD2ZbVf7RDinYu5oExN0MR8bGu2FLTgdw7yT9TxpllirJTnlvUhBoE8ZPTwoIs+ZUBI+OWBcKPnU/x2prKoUN6yoV00SC96w3tDreq/KHeWntfxPm4RuJkuxvIjGsm2zgBP3+ZGQdxv8ZGmZEZ+GEuGxmivB4GNp7BOZ/UjNFJTkeKF5gunLZESZguOr+5+yy01dBYCNZAe1ugE6o52jcZNJDPeP5V4+aDjzikj2U2iyn42fiVBB+aZGNHEysrl+82SFc1lmk0xwWpsJ45cL6W5fzZ7ijBvAP9SZmt4IfCLs9NANUFep0uiN/1/BYTAoODHpBSHAzRT2O1ityyBG0X0YbUlmza4BNnOuX/epQMUmQzIjHRpFEZqqQisAyFrB+dTK6Ad/+TgMOWoQCOyBNtKeT8F94zp7W94mczlGCWXv6Ux92lELuEyfhiIHPDaCWqIzmi4HbmFFvdA3plzX1TKQiV431O37shjB4rTBbnR7becCUy9MI8IY41BT9c0LCm2EC/zI8hedoZg2Th5vKPeIj24StDzI541YZ4anfSVNHkCbJOCaGsVKS0VTuCWdLFHOQjisRI/ryLNxdtD8bV+jyD8yFhKWk7TcoFEoUhujnirx5GJXm7UUktMuAsZMQP7zIhVsQxv07Rw2pB/Yw/aJsKmvajmm8sQlYYIj/DZMRt/F8o+L8mL4kbCxni7KgPlpT1ed+Yt1+XipXZvC6Eiraw7Wj5Xx5TAFYfFNM2LR3WiYFvmmzRInajX+AkLOVazriLw5babaO1WV/F8YnHdT/VRV/QDzhBnSAH4Niz5TmOg1TDEmH7lmBU2Ilc3CSUCAOY/sGyb+HNNXjKj8ks9LTVJcfSRvNAqGaJuB4QdOcWEOVHf6mqzalW1AYRUuyJ9whv9HJEUHb1K7PwnSWJXjbloNH8cOsKequSzuaE/Jv7ftfDV4KRC4qYrcHaB8sQxY5znamG5ApWHn7qpy+ewZCKYBGKTGVq8sbqD1lGj7HyWuLNKNR+5Id+odYkqDldjr6ZKnZ9l6DW+spRxbConHBSby9jzEdssP07s+YL9yGF7P6swo9LhnH/+awIn/KFhvDnz9K9444kFV4EuHfFuPD33rQJMyPu0K3B6t0F3b1BVQ3OBeSLNvpFqPiAkln+yuQs47OPZxpar2DVfjSwsgVEzfib7RRPSrf5pRDB18aJE2UYyC7UuLVX1MvNlxk8ZYQeJUxwOsNc3nWt/VxIAMGGPrZ7wzASfNlbWTfU4xUht8Iq+tjH2thT78lb1+4QfMI1AKDKtSfiDMY09N5xzUoULXhDCa5zhts7B6rEWt0Th2vLfLWpqAKvMep3kEFGXQQbyOMw36d6VwZhk+0t9bPA9NhjgBmqAljWRmSqitCY5HEUx5mjW5bZVfLOuMoJZ6EtvGni3svTuii7kZ2mHt+Krjssqsyhf3D3CtWTIA6xd6wkEwquMLX0oEUJOVO5EKrft1KVieA3CEzQOngAUbi2TZCqt74an2WKFO8v2Ny4Vc46eU06OgUFwOzNmhMkUwlj84TmZ9gx4JqVw== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: XbcGSN66aHjmxdZ2GVLkOT5CvWwzNgbE0fwH/OGGMJzFKyevZkb/gapik9VMEHlJZyneqZFvpWKN/tCHeNVbafrcmjblAs7InscO8VLmn4TUDkQRtIL+dZYDFXzRa7p6V8Zw/jOSJUR7xWnMdvQoMM85oleEmwIvBVPC2RhCiTOGW6Kw3p07KtcsYUzMo64bHc9x3oXY5Vq1+FoxWXIu52U/A50TSGdUolbHtAmGPlcvHUs7TCbVJ0YeqD2+zNmqVwyN2LtZFxrP9PQ/f/5T03M2sbnApLE4yugi97rpuCGKTLW5qOSudeuDdZ8xxyBxosanKaNh8dPZ18bHjp34VO4NtQnkSCPqd6tVJyPPPSV4l+iL0vfENjaSf8O2eAb3ILakOaG4/xVr7WWntDhF1tZMxcZ7DzqAsoYMkn8vTJD2Sbd0Tw5cRB0GAawW/Cugoa8+PzV1XZej/vc9Ij9MZMdNidqkLn1/aMw+mT6mM/FZR4ylQeOyd94iKHpOxcMjDOTMyCDIsHf1PsXPG3NoLNL+USLNEpzdhIaZ0rTs0Rh9Dm7vaLE2Jt3ZwBlAsQSkk+pQbQ2vKQUibBpK3mob/IkLghfXmedhnejtspt8JMk= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 040325da-7007-4d81-d549-08dc818f515f X-MS-Exchange-CrossTenant-AuthSource: DS0PR10MB7933.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 May 2024 16:32:52.6149 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6Hq6ORONMimXAGWbvrENRZSTSw085k87WF+2ENSKnRqVu9NIGDTGTswKgRRzwnbU4+E4Jtme82aGAjS3dyR4Eg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR10MB4238 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.12.28.16 definitions=2024-05-31_12,2024-05-30_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 malwarescore=0 spamscore=0 adultscore=0 mlxscore=0 phishscore=0 suspectscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2405010000 definitions=main-2405310125 X-Proofpoint-GUID: i1yIdGP9yCBuwoSiJ295m-z6MP0-lG38 X-Proofpoint-ORIG-GUID: i1yIdGP9yCBuwoSiJ295m-z6MP0-lG38 X-Patchwork-State: RFC Use vms_gather_munmap_vmas() and vms_complete_munmap_vmas() to prepare and execute the unmapping after the new area is written to the maple tree. Delaying the unmapping avoids RCU readers seeing a gap in the vmas, which isn't supposed to exist logically. Gathering the vmas that will be unmapped allows for the accounting work to be calculated prior to checking if there is enough memory. Using the number calculated during vms_gather_munmap_vmas() allows code to be reduced in mmap_region(). This removes the only caller to count_vma_pages_range(), so the function has been dropped. This does have the side effect of allowing vmas to be split for unmap and fail may_expand_vm(), but the number of pages covered will not change. Note that do_vmi_munmap() was previously used to munmap, which checked alignment and overflow. These checks were unnecessary as do_mmap() already checks these cases, and arch/mips/kernel/vdso.c arch_setup_additional_pages() uses predefined values that must already pass these checks. Signed-off-by: Liam R. Howlett --- mm/mmap.c | 84 +++++++++++++++++++++++++++---------------------------- 1 file changed, 42 insertions(+), 42 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 3e0930c09213..f968181fafd5 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -364,23 +364,6 @@ anon_vma_interval_tree_post_update_vma(struct vm_area_struct *vma) anon_vma_interval_tree_insert(avc, &avc->anon_vma->rb_root); } -static unsigned long count_vma_pages_range(struct mm_struct *mm, - unsigned long addr, unsigned long end) -{ - VMA_ITERATOR(vmi, mm, addr); - struct vm_area_struct *vma; - unsigned long nr_pages = 0; - - for_each_vma_range(vmi, vma, end) { - unsigned long vm_start = max(addr, vma->vm_start); - unsigned long vm_end = min(end, vma->vm_end); - - nr_pages += PHYS_PFN(vm_end - vm_start); - } - - return nr_pages; -} - static void __vma_link_file(struct vm_area_struct *vma, struct address_space *mapping) { @@ -2863,47 +2846,61 @@ unsigned long mmap_region(struct file *file, unsigned long addr, struct vm_area_struct *next, *prev, *merge; pgoff_t pglen = len >> PAGE_SHIFT; unsigned long charged = 0; + struct vma_munmap_struct vms; + struct ma_state mas_detach; unsigned long end = addr + len; unsigned long merge_start = addr, merge_end = end; bool writable_file_mapping = false; pgoff_t vm_pgoff; - int error; + int error = -ENOMEM; VMA_ITERATOR(vmi, mm, addr); - /* Check against address space limit. */ - if (!may_expand_vm(mm, vm_flags, len >> PAGE_SHIFT)) { - unsigned long nr_pages; + vma = vma_find(&vmi, end); + if (vma) { + struct maple_tree mt_detach; - /* - * MAP_FIXED may remove pages of mappings that intersects with - * requested mapping. Account for the pages it would unmap. - */ - nr_pages = count_vma_pages_range(mm, addr, end); + /* Prevent unmapping a sealed VMA. */ + if (unlikely(!can_modify_mm(mm, addr, end))) + return -EPERM; - if (!may_expand_vm(mm, vm_flags, - (len >> PAGE_SHIFT) - nr_pages)) + mt_init_flags(&mt_detach, vmi.mas.tree->ma_flags & MT_FLAGS_LOCK_MASK); + mt_on_stack(mt_detach); + mas_init(&mas_detach, &mt_detach, 0); + /* arch_unmap() might do unmaps itself. */ + arch_unmap(mm, addr, end); + init_vma_munmap(&vms, &vmi, vma, addr, end, uf, + /* unlock = */ false); + /* Prepare to unmap any existing mapping in the area */ + if (vms_gather_munmap_vmas(&vms, &mas_detach)) return -ENOMEM; + next = vms.next; + prev = vms.prev; + vma = NULL; + vma_iter_prev_range(&vmi); + } else { + vms.end = 0; /* vms.end == 0 indicates there is no MAP_FIXED */ + vms.nr_pages = 0; + next = vma_next(&vmi); + prev = vma_prev(&vmi); } - /* Unmap any existing mapping in the area */ - error = do_vmi_munmap(&vmi, mm, addr, len, uf, false); - if (error == -EPERM) - return error; - else if (error) - return -ENOMEM; - /* - * Private writable mapping: check memory availability + * Check against address space limit. + * MAP_FIXED may remove pages of mappings that intersects with + * requested mapping. Account for the pages it would unmap. */ + if (!may_expand_vm(mm, vm_flags, (len >> PAGE_SHIFT) - vms.nr_pages)) + goto no_mem; + + /* Private writable mapping: check memory availability */ if (accountable_mapping(file, vm_flags)) { charged = len >> PAGE_SHIFT; + charged -= vms.nr_pages; /* MAP_FIXED removed memory */ if (security_vm_enough_memory_mm(mm, charged)) - return -ENOMEM; + goto no_mem; vm_flags |= VM_ACCOUNT; } - next = vma_next(&vmi); - prev = vma_prev(&vmi); if (vm_flags & VM_SPECIAL) { if (prev) vma_iter_next_range(&vmi); @@ -2950,10 +2947,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, * not unmapped, but the maps are removed from the list. */ vma = vm_area_alloc(mm); - if (!vma) { - error = -ENOMEM; + if (!vma) goto unacct_error; - } vma_iter_config(&vmi, addr, end); vma_set_range(vma, addr, end, pgoff); @@ -3075,6 +3070,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, vm_flags_set(vma, VM_SOFTDIRTY); vma_set_page_prot(vma); + if (vms.end) + vms_complete_munmap_vmas(&vms, &mas_detach); validate_mm(mm); return addr; @@ -3100,6 +3097,9 @@ unsigned long mmap_region(struct file *file, unsigned long addr, unacct_error: if (charged) vm_unacct_memory(charged); +no_mem: + if (vms.end) + abort_munmap_vmas(&mas_detach); validate_mm(mm); return error; }