From patchwork Mon Jun 6 20:20:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870834 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A983C433EF for ; Mon, 6 Jun 2022 20:22:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DE70E6B0078; Mon, 6 Jun 2022 16:22:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D96506B007B; Mon, 6 Jun 2022 16:22:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B74D56B007D; Mon, 6 Jun 2022 16:22:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A1AC26B0078 for ; Mon, 6 Jun 2022 16:22:24 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 8436560A4F for ; Mon, 6 Jun 2022 20:22:24 +0000 (UTC) X-FDA: 79548933408.08.D00577E Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf06.hostedemail.com (Postfix) with ESMTP id 92559180013 for ; Mon, 6 Jun 2022 20:22:19 +0000 (UTC) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256KJjoC030086; Mon, 6 Jun 2022 20:22:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=6BJ963cSHLwoWXx+Sn1JXEa7kfrO5qnc4QbhLDXzJfM=; b=XSThaI5efd5j+aWpu4xuZcVqnFgJClYUuiZswgGeBPUfdrF/YofPWTV4RNIiJSSy7cSG zpK7LZCFmPjxl+CpnmH0sBqCSZGEzCR9/KUsEDv7BN09bOW/DnKxFuf7CAJ3h3rkgKn2 /+f8aDK8vBnrxywjjo3XyZGLxmZrvliOBsun3xid3nAwh9zaAeSxYOirOS1fOJKQ6Udz UeX0VEdyk1TPm8WR8zSQCbFW38zQrsBJQqBQMUocgAp0QSO5mBOpPz7eZsrhcx7lnXle 4JlxJ86umZkiuBJwfz2W4sWnGgmVH3AvlwKOrh7fhcYMqP1WJAbq6mV3wav7x/jc8q2z WA== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ghqad84qy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:22:08 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KGEXr003280; Mon, 6 Jun 2022 20:22:08 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2175.outbound.protection.outlook.com [104.47.58.175]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3gfwu2bt3r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:22:07 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=md3YSVD2WL410xId5KsVGrrgBh97VuNC4dmOa2zf9YTTxREVkajpdWa5ZHze3j87NVldkEbxTYzVRkvgYcW4KpLvZua2weH5E5h+BsjsSgg8uZeaxmZagdN62FEygA46+7K4yYnsjX9xVmAvHINThEI6oyCFkO/MYWZt08PtS989fGNX0WoQIaNPGdcefVMnyskVVt34YgyR/vIxVJd8rfvdYQ+OnH4paxkZgoemW3kai5XjIaN8HQnC3tDQYHq87fW/q5LsFccLt9F4qOaX6qXREfHVKqEbLbPHc7GNzTBX/RRiuwjk66TgU14uOXX7DKXqeEYMBudaeyuzr6eR2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6BJ963cSHLwoWXx+Sn1JXEa7kfrO5qnc4QbhLDXzJfM=; b=Pw5xFKv53wTk0iYxs1/f0XOgTWpgQzIN+MAGLPQl2TM2DfdDL+GC1qeatZaRO/JQtme1X+XAEFSLYCNU/iVV+wgR2HUx2EMZ1EtekYk0rAK0H68xfTmN3OpCJ3poF61dzLKVH6qvLGi2MUtAqDBu2mb93O2tMul6X01GEV8ho0V6yIHCa1VUs/6opGiOYPzhhYFmvazCXAw/7dPclcnTwgJZOA5JvEOx9N0dMRyOiRjHK4wkMyufWKX4dpFZy5aFtOcciDOcmrUU43vMLw6bxwVMYgeKA1f3ibJ8/8RIg3d4OUCgWF40N8+Vdb0ktGV4MQCW1Vbt1TPWJeWQuA9IWg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6BJ963cSHLwoWXx+Sn1JXEa7kfrO5qnc4QbhLDXzJfM=; b=NWdT/+Y2K4pGMYi7xbHKmQSeTL8tkQm2A6pnou0u4mKZ/XUzr0egvjkNt/oACen0CpUF0vZZ4js+1d5YmW+hqbcyKKyxSYUO2kIeTYtn85v8pICBC+3TOORE1Z2fz1Z6ijq8UJNiB3m2hT9+kRPZebINKfKIK1fIaku+g94xysg= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by DM5PR10MB1772.namprd10.prod.outlook.com (2603:10b6:4:11::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.13; Mon, 6 Jun 2022 20:22:05 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:22:05 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 04/21] mm, clear_huge_page: support clear_user_pages() Date: Mon, 6 Jun 2022 20:20:52 +0000 Message-Id: <20220606202109.1306034-5-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: BYAPR01CA0034.prod.exchangelabs.com (2603:10b6:a02:80::47) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e76cfaf1-db89-4c79-6177-08da47fa3948 X-MS-TrafficTypeDiagnostic: DM5PR10MB1772:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: YK6dU+WVvxawlCO6baPU3XgPpQMrOkN04BqU6vXolYAQoDtuGld7NzufJDLr5yhvjszGJzxhWeeWPFbb/yUl4OKB9VY9hX1NVcGfm+/R/3okMufB9H/jV03dStNZ8QIDmnXLyjctOKIzRwc/EbT4Y8TToBdYDAZuf7T4F5v/e2A7EgQ36UbKct0QPgHmV9+ikmRO31qMIwD7DlhYKOlJI5fF1iGM1WilcKPO8kSp6JWJ4B53oHEEFOWWUcS3QFz+//SUUD3s48d7EZnocmyXjE8iSJZKpiHO7r7/Zfg8zflIKMblPms8YrE/SABwiXz51HlBNdfnfvsxsxwof7E3NK27VDsv/4vgGGgcgSQuHZj1tbDqvCAgB0jL1K3LBlreVPRsIpn1glAyLNOaoU4F0rKayT9K2R6mZnBND6fKqBfNBcKT9pS9RuBUs8Pp4k+XSHUEuRQioGyDXlA+Y5VHkUsK4VGro8o/L2ZPBessZv1bxePs75ZwXfD3rr/wHa07lF2fsBRSBg7ce/2unWtYu2V84bDppElzPvQ1hDmZFf6bKWapEiHV3nYt2N8D84VfwqD4AEASHufUPYO32MhRDp8ZeXPxCpEpoq/xuseKhOzKoLa3yTZ4+ks4RjDZfEK361dtbcrP75GKWCFKXFvW8kiS1A1qcnEynyVzNiYkvPXHXN1bXR6bLeUY2XKSVBX/8Nx9/aptu9904jGluI73zSVJWJSp7snF1g1Z00NRcRw= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(8936002)(7416002)(6506007)(83380400001)(107886003)(66556008)(36756003)(1076003)(38100700002)(186003)(6666004)(66946007)(508600001)(6512007)(26005)(2616005)(5660300002)(38350700002)(4326008)(8676002)(316002)(103116003)(2906002)(66476007)(86362001)(52116002)(6486002)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: eIGiphxJVBJHg/dOZ2mWIy7aJflRUe16uqgUYp2oRx1+OguaDfVQb5GpbijpSyOn9K8kE32kRVPdYm6P0MQUUYIbPDi+ptKgi2EKjzd2rcsxvRYMwCnBNslvrf0CrxTW1gRat3cLw3jJxpLt+JhfIZEsCeciLK4zN4uE77KEQxI1KUy8bESLGxSWbuaFW1ynsZIrOtGysd6Egd0svED8CjrliSXq9ibLv44n3g43jxclm3LkR6BeVer3XM5olK0AEBZBOEo/mvx1BBIq5zBZgxU6tOmGxMQtMuYeCrMlaf+KfYe1Xc5fDgyWwaQC6otl30HTUPTtCdOHJyGmpMNhMW25Ol2s/dPD1mnPEzwVT3HIs/ARaIB849XjMH779UiLjPy6T1QD/9bY/bRtUtAC8UzxjXIp8LTY3+aliAb5rGqpv9ydkLOvwdKbeDZK0pPW67nAlaYwh/9KLhx9umSPO76KG8Po9NTPSLGr6dRiqR6xQr7qUa9ZBwUuK+JSGOWeY+vyOcZDpApbuN9DmQo+SO6rgRZ+bm5ez4mplu41VTJHYmcUss0CnZA5EeBf6dMtNf8knP/KFCh/AAcZfNuQeZahgXE3tsQTqvyf0J+FUMk7X9Tvjbp+RePisQGTj/f/C6NwCkqoJU/AYB1nWeadJogEDmAJ/aPjRFifkWfabdu7O9JZ8+K3Ur+oBr0pW6UUHx+30rdG+3KMTXXS237WXk6KZHmGt0bsc8YV4jRB+IMeJCBIcioFlYVxQcChLiDophEu62tFNhW5J8xo6j/wbfa9GqPB6v3dxWd2TDKimKvwL7us/0L2DsXlD33T2DX/r+E2Vie7r+Eg1sW7JV3c6JipahUEDkJ9vXxS3G4X3bvFCQkN4EYFO6MFBNb/ckgLRu6QgLnDfzZZrqQIn4bO4mPR4IRAEAGIf/MaKeEbyjavilcXtKvqXIyKRud1C+lRgte3FmHHswbXr+mUTWWqloCFtQ3tQH8zIA87W8b6QlKAGzyWPIBOTb9tkFKUihng1hnWBNqCx+UqHzznEqIGCDVL8LrKSgz+AZZ2oxXpx9ilx5LBJr5wPbIsCJMmq1oWqeyOXVnHPr7SFpZ1PKmdeSDRjybuKdtFXbUS+ai417OUofCU65dRELG19Vgxx56cLu0HmTmauB8B7kKpo6s7QyW7eXBdsJjZ+VjJIA6KnofR+lBBiP2MX4OV9gbVzGMKmf0SdQPowjn5ae8lziBZblPAMlDFUJtk/Vjy9TXUoTh89U06VxYopXPR4r88OcFWSv3aDFYt1tTZtwYGJLHbHw4NArB63AKQ3IE454jD91QKmXdqDvw8AZNfdolXMLX02TiiKrOAOfcdkb5aMArRzMmdmXOcfdNWGSj6AKs8hFASQrlKlNijYDHI3cebl81/scN59fba1/p4lUDtHC7B/CiqVd4mT8Nx0RwSa5w/KLAgA8+E1J8lwkmWcMDLMJ+pTaSnZWqiNowQGinjiSBK5+eiLGmcBZj3E2WCawc/dHg+mUJKWXW8EJcuc6ci2C+jI5c7QBMEtEuswfS4CtVceBEAsRse0oIwolO28OMg9/MMj1MY9VVsI/HP0Lf3J5g5MH7xXD+w3OCadQD17uJ3RMpJ9bKqyHtIvesUnwhQh3+Hy+5gnndSN/3SDPuQtzQMx9a9FLn2tavfHIPluvVPiGpuRaW54yiGqyih6oV2CL8ja4yHDJbGPQrh3ITIZZr/Xhg+J348H4WJAQqmCV/xe7DoGZLM+Y5HEN/IGx5vq8U= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: e76cfaf1-db89-4c79-6177-08da47fa3948 X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:22:05.6818 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: t8vkoxPS9odXsOp5gm7LcQuVAP+g6fvtqh/MEbErxdXyO88H9Ob/o7DtsghT3horY+0fKfZNgQE6GQIc9QcHQ16PsGb6gaWZaT9tnSiJMPI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR10MB1772 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 bulkscore=0 malwarescore=0 phishscore=0 adultscore=0 suspectscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-ORIG-GUID: qqGtsA2CUbLUkubRmSxzRBXAEcra_0nE X-Proofpoint-GUID: qqGtsA2CUbLUkubRmSxzRBXAEcra_0nE X-Stat-Signature: o47rkabksfarccikgb438u59k7u1bwzn X-Rspam-User: Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=XSThaI5e; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="NWdT/+Y2"; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf06.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=ankur.a.arora@oracle.com X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 92559180013 X-HE-Tag: 1654546939-446670 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: process_huge_page() now handles page extents with process_subpages() handling the individual page level operation. process_subpages() workers, clear_subpages() and copy_subpages() chunk the clearing in units of clear_page_unit, or continue to copy using a single page operation. Relatedly, define clear_user_extent() which uses clear_user_highpages() to funnel through to clear_user_pages() or falls back to page-at-a-time clearing via clear_user_highpage(). clear_page_unit, the clearing unit size, is defined to be: 1 << min(MAX_ORDER - 1, ARCH_MAX_CLEAR_PAGES_ORDER). Signed-off-by: Ankur Arora --- mm/memory.c | 95 ++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 69 insertions(+), 26 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 2c86d79c9d98..fbc7bc70dc3d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5563,6 +5563,31 @@ EXPORT_SYMBOL(__might_fault); #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) +static unsigned int __ro_after_init clear_page_unit = 1; +static int __init setup_clear_page_params(void) +{ + clear_page_unit = 1 << min(MAX_ORDER - 1, ARCH_MAX_CLEAR_PAGES_ORDER); + return 0; +} + +/* + * cacheinfo is setup via device_initcall and we want to get set after + * that. Use the default value until then. + */ +late_initcall(setup_clear_page_params); + +/* + * Clear a page extent. + * + * With ARCH_MAX_CLEAR_PAGES == 1, clear_user_highpages() drops down + * to page-at-a-time mode. Or, funnels through to clear_user_pages(). + */ +static void clear_user_extent(struct page *start_page, unsigned long vaddr, + unsigned int npages) +{ + clear_user_highpages(start_page, vaddr, npages); +} + struct subpage_arg { struct page *dst; struct page *src; @@ -5576,34 +5601,29 @@ struct subpage_arg { */ static inline void process_huge_page(struct subpage_arg *sa, unsigned long addr_hint, unsigned int pages_per_huge_page, - void (*process_subpage)(struct subpage_arg *sa, - unsigned long base_addr, int idx)) + void (*process_subpages)(struct subpage_arg *sa, + unsigned long base_addr, int lidx, int ridx)) { int i, n, base, l; unsigned long addr = addr_hint & ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); /* Process target subpage last to keep its cache lines hot */ - might_sleep(); n = (addr_hint - addr) / PAGE_SIZE; + if (2 * n <= pages_per_huge_page) { /* If target subpage in first half of huge page */ base = 0; l = n; /* Process subpages at the end of huge page */ - for (i = pages_per_huge_page - 1; i >= 2 * n; i--) { - cond_resched(); - process_subpage(sa, addr, i); - } + process_subpages(sa, addr, 2*n, pages_per_huge_page-1); } else { /* If target subpage in second half of huge page */ base = pages_per_huge_page - 2 * (pages_per_huge_page - n); l = pages_per_huge_page - n; + /* Process subpages at the begin of huge page */ - for (i = 0; i < base; i++) { - cond_resched(); - process_subpage(sa, addr, i); - } + process_subpages(sa, addr, 0, base); } /* * Process remaining subpages in left-right-left-right pattern @@ -5613,15 +5633,13 @@ static inline void process_huge_page(struct subpage_arg *sa, int left_idx = base + i; int right_idx = base + 2 * l - 1 - i; - cond_resched(); - process_subpage(sa, addr, left_idx); - cond_resched(); - process_subpage(sa, addr, right_idx); + process_subpages(sa, addr, left_idx, left_idx); + process_subpages(sa, addr, right_idx, right_idx); } } static void clear_gigantic_page(struct page *page, - unsigned long addr, + unsigned long base_addr, unsigned int pages_per_huge_page) { int i; @@ -5629,18 +5647,35 @@ static void clear_gigantic_page(struct page *page, might_sleep(); for (i = 0; i < pages_per_huge_page; - i++, p = mem_map_next(p, page, i)) { + i += clear_page_unit, p = mem_map_offset(page, i)) { + /* + * clear_page_unit is a factor of 1<dst; + int i, n; - clear_user_highpage(page + idx, base_addr + idx * PAGE_SIZE); + might_sleep(); + + for (i = lidx; i <= ridx; ) { + unsigned int remaining = (unsigned int) ridx - i + 1; + + n = min(clear_page_unit, remaining); + + clear_user_extent(page + i, base_addr + i * PAGE_SIZE, n); + i += n; + + cond_resched(); + } } void clear_huge_page(struct page *page, @@ -5659,7 +5694,7 @@ void clear_huge_page(struct page *page, return; } - process_huge_page(&sa, addr_hint, pages_per_huge_page, clear_subpage); + process_huge_page(&sa, addr_hint, pages_per_huge_page, clear_subpages); } static void copy_user_gigantic_page(struct page *dst, struct page *src, @@ -5681,11 +5716,19 @@ static void copy_user_gigantic_page(struct page *dst, struct page *src, } } -static void copy_subpage(struct subpage_arg *copy_arg, - unsigned long base_addr, int idx) +static void copy_subpages(struct subpage_arg *copy_arg, + unsigned long base_addr, int lidx, int ridx) { - copy_user_highpage(copy_arg->dst + idx, copy_arg->src + idx, + int idx; + + might_sleep(); + + for (idx = lidx; idx <= ridx; idx++) { + copy_user_highpage(copy_arg->dst + idx, copy_arg->src + idx, base_addr + idx * PAGE_SIZE, copy_arg->vma); + + cond_resched(); + } } void copy_user_huge_page(struct page *dst, struct page *src, @@ -5706,7 +5749,7 @@ void copy_user_huge_page(struct page *dst, struct page *src, return; } - process_huge_page(&sa, addr_hint, pages_per_huge_page, copy_subpage); + process_huge_page(&sa, addr_hint, pages_per_huge_page, copy_subpages); } long copy_huge_page_from_user(struct page *dst_page,