From patchwork Tue Jan 24 20:34:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13114784 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 411A7C61D97 for ; Tue, 24 Jan 2023 20:34:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3059D6B0082; Tue, 24 Jan 2023 15:34:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 290E16B0080; Tue, 24 Jan 2023 15:34:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0B8E06B0083; Tue, 24 Jan 2023 15:34:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E0DC26B0080 for ; Tue, 24 Jan 2023 15:34:51 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id AF248AB1BB for ; Tue, 24 Jan 2023 20:34:51 +0000 (UTC) X-FDA: 80390846382.07.5ED24C2 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2041.outbound.protection.outlook.com [40.107.236.41]) by imf12.hostedemail.com (Postfix) with ESMTP id 169E04000B for ; Tue, 24 Jan 2023 20:34:48 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=pxEDXGZ2; spf=pass (imf12.hostedemail.com: domain of jgg@nvidia.com designates 40.107.236.41 as permitted sender) smtp.mailfrom=jgg@nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674592489; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eF0zU2ZoBV9oclS+2djXfPffPq07D2lWplMza8/TPgs=; b=PpK3zsNcAAUP7SBYDmKfTWeJzZyKz+TNnK32MHONp05+tH+PltG4vtRt96zwIn9XEZEJxq M0IKoeUDoWvpsSc++sbuuf4GffPTY6vpVqok9GKNJWBG+AxvuMIARISLLBoLcvsqn4lrMP ZgDgFCU+fVWPSszBk7PvMimpqQPujwU= ARC-Authentication-Results: i=2; imf12.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=pxEDXGZ2; spf=pass (imf12.hostedemail.com: domain of jgg@nvidia.com designates 40.107.236.41 as permitted sender) smtp.mailfrom=jgg@nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1674592489; a=rsa-sha256; cv=pass; b=8nypJiRwsAc3rTdrQGTrozuWiiamanpeHWiWG9dCuOD28nwZZGTNRIKRBXxv/RQ/UfEx0y +UfqlXCKMlKA2I1rS+r6Ha09MBSy1Hm0LgMHKr7vOW0IjlnpjRUv/TpuYft+R+iR/zaCLQ cTr+efrgEbRS8sS3FSxkTwCdt0Oa+qc= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PuS0mzJfwNDoY7wCYTXsHv+/JBRLhlDiAlPul+X5bshJN9fbPrPXU+BwOS4LYcf5dp+xnF+/MYMezq+yksOSqzNZt/qDhcj8uBP1/V7t9A64sr2QStl+d8bgAjBjkLvPkG0i09bsVNOHTpymUmvmMOB+z/dLr5Qgxkg/BCewK9EhdrNla9GnOlLRMYxoaIs8cwE5vjUI4ZRi8JuhUlntl7h0BXPXCYXKnDutSGoU//OFXqI7jidBHT7mLXKRVgH2TtgHfLzIDiouLr1km/J57zoURhUlAEAVaIyz0w/SCO/TZ8s9+1qBwvtnzF5IRjdZXYqeGrh3Qrs7VySv9HGpwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eF0zU2ZoBV9oclS+2djXfPffPq07D2lWplMza8/TPgs=; b=WiJoeTBAUnyGicbtKF/dJPOCz6eR+QBFijD+/dsVhloenAfW8lM7DkgqDeE6bTclRqQCztr+zdyrA2UVj+QiHNPLyiRZmrtgTr7jJK8jroc3oskQjK3yeTF7BQrY6ltcLlVUunYRpbrAoN/WCykeucPC/ZC6Wb8L919446ZA4AsO47Fv4boU0cOv/s79GM07iMfpiBGLCG/eUXBlIxL7JloU8fvEcL6L4fnSF73PktnBFe77kfqIxcUX74qwS/YUNvJPudTvj2o3c3L5y3xJWHkV8RrK2rDQ2aYCTbQlXAU5xrpOnCp0tX4HdPr8BVeJpIHN1TyBueahbvhWuRTADw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eF0zU2ZoBV9oclS+2djXfPffPq07D2lWplMza8/TPgs=; b=pxEDXGZ2NBEDtwLYRDKN1K5yI8Bp+W9uQnTKR0N3OGThtgFoLTuJ4/ugX5xdQfukyDpiMuf+FuNhzJu7+hnNDCJHzjG4wdwQ1nhTvyxj8JMAZLX7D8abKjcOtCBjnZ2eC1fINNSmp28+kPR1H5Px9zkCljb9QCP98trSXRWW5w6Jqx6VB3exlfUVlkLGjlAGWjjG0e6rFuXb8nIc7CiPPTRZwFwHuxHeaY92U4gjoDc+D2Ur7kWL+kw2ff5XtfHztvLMBfyV7ddw2CKJH6IINCfXPZsKyb7U1tD1BxEfwxJznElCZyQ8yucgZerLVYqqBsq/wk4pi62v9CKueLrTVg== Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by MN2PR12MB4095.namprd12.prod.outlook.com (2603:10b6:208:1d1::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.33; Tue, 24 Jan 2023 20:34:39 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::3cb3:2fce:5c8f:82ee]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::3cb3:2fce:5c8f:82ee%4]) with mapi id 15.20.6002.033; Tue, 24 Jan 2023 20:34:39 +0000 From: Jason Gunthorpe To: Cc: Alistair Popple , David Hildenbrand , David Howells , Christoph Hellwig , John Hubbard , linux-mm@kvack.org, "Mike Rapoport (IBM)" Subject: [PATCH v2 08/13] mm/gup: add FOLL_UNLOCKABLE Date: Tue, 24 Jan 2023 16:34:29 -0400 Message-Id: <8-v2-987e91b59705+36b-gup_tidy_jgg@nvidia.com> In-Reply-To: <0-v2-987e91b59705+36b-gup_tidy_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR20CA0041.namprd20.prod.outlook.com (2603:10b6:208:235::10) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|MN2PR12MB4095:EE_ X-MS-Office365-Filtering-Correlation-Id: b3c3d6fa-0e19-436a-c8dd-08dafe4a68b9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 6nPxDeZMWsl1ZwjH4t2PZgEphtRd3jRqGFr7jsuJ9sGnBAS4o2fpF0exD5VXJhOINZAoTeS0I6+wUUKabFxARulHvfIAk0068J9MNrHDlvicoLx3nrEalG24vtTiGzh+hlKwsRYx7xJZMhdM6ZPcRPst6YVPJvb+QzLXsMYMrO7i8m57Kj4rGr1AF9fboEo8fXbidMzz4SfPvLh8+ZZ03i7/8EwIrjt2KpIwfK+n5BzgQD2B1IVpLsQBs35e3r1Igek9C9v8NzfJH2RgyH1yzRNlxHykBDAB8dRIJpQ//lVj+0XfrNhnQqjcJbc0xK6lGgscE1XiOxr7OjmtBbbiOWlQFINbUxPfcStkh2hznoHLkMJK9NLQM9UPaNvH/qxVb9jUr05hspJbCwer+hQyDzey4GQIHxL/O6vBbs084wqTPO1DAScJKAFm+qqs9qPV2s5D1eYpRRRSgtFHok5bhMD1nUjRIjHcjtdz0pAqx/LRhxOYukUIBmp9z0MdoJYv6Urc/d2NuJtHMe1MQDkHKjvuFwOarrvrtmNGUMIoqjfSb1PGweUL21/Wz/Op+XeBHtDQBA7xbkIHN/wdtHBc8x7pQqI3PMOXg/+20IOiemiE8hUUS1L5NynOzkrcO4hO1SJPcX0Y5JVKrOpElI6kqXdPMoiu1jYKxK/p0oxFm7FezVwGA4m7ip2J1bkBvEhjx50rPz/p4exUVrfZ3yYMUg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(366004)(396003)(39860400002)(376002)(346002)(136003)(451199018)(109986016)(38100700002)(83380400001)(5660300002)(41300700001)(86362001)(2906002)(26005)(4326008)(8936002)(316002)(6666004)(6512007)(6506007)(8676002)(186003)(66476007)(66556008)(54906003)(2616005)(478600001)(6486002)(36756003)(66946007)(4216001)(266003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: HQCQJG9ektNR2vikEK2fS1AjFGBBcOrhAn8ryn5nBjwEuu3vhC/SqCiHtTMaIjvoTPwPY9G1RqUTe+83R3VRUL09tHeXt55keCHUAm0d+odaUL40REzPcKNBa0uYdOx8XLuPRNIQWYgfQf1ZrKlXgdvJnq1hzvk9W7bRyNO4BOkXeDcAVWnArFyA/FyK68rrQPC0XX69bCFn4GkHy60+aJXEGUlJxRvV39rfVn0lxmOSdlEUx5y2XKW71nz6vXmJ3nAWpwirRR/pIvZ6QzUFdk8SXDinM80dOIZYB9Rlkpw0JXF3JNt7ImzsgahgaoAf5jMnJ4mwaM5cfqKgcRrHJZd4Hbmrhn+yCW9VALrIExQFDiuFW/Pys5vnsYcV0b32OKC1sZlqZl7Ub1Byl7qspqjx9BJtXIkazdazNmUW59uLZSpZPtF6VSjX76o/aORtg2aCWHFRjIGdtAAq2K/puKpCOVxJP0HAz/e4y7duQlz63U4SgrkdKNv/mwDlltTlZn1E6fSBs6De4f91SC4ar76FC5dT2om5RQTOe5uuBI9sqhuq4IU+dz49lgb1HCBQcg1/dfZy5rDmYTEKo4lk81/zeULdawF08aVOvXsNCDdWRZs7SQcmLMViQbWK3vbbmqjkKFumW3yaphsaGEfsplS/Bft4rr37rYVaMYQji20JWsr+eLQ7bgpC1iokqXMkkinw3tpO4cflKvIXgLO8W4dSy6M26K/JJJVQaA2d+P2AytkOK59nRddnduO3B+lxhOGAc4j4+dIt6dZpK4fDUgRamCY48TPHEg6wNnPXuR2PMH2TsmnNYm4bqoCA3srgREFG3jVEcjddigWth2dHOBH1p0bgcdIhbknVnHk7vC40erLsKLuA2rOTc7sDmQvkPKCnYPq5I+JUqgCueMAEyk3dQmcuRqmHkawRfABENJE9jKtVB3XHdHU2qT/6ukjqn9xqNbNi9mJGyRmL/t8oexpTF5vyDcbjskxfe8rdo25/93tIvkt5fXH3GJUXSYsY+U9vw3sndlrXq0pog9s2aottJRBZoNF+7mQroK0jM7T6kXNrEw515PqzHpmMI2cwdPq+Q81BMYFl7B0cQiJnCAR2G1b83nBNYwsXE8yUfGTsBe4HgXOt+K7pMhr2vX4FHrMMUVKo8JHAtYKU2PXL1jQnnEIQkNw/ivkhS0MMUw9m1ssoYeR+4luJt1pEI9wldsrgDedbyvWNLnjoj1GUP1rd4HQCV/SnnSaOBYEY7+nkv5lVX+WL4utnuvqrNar+M4+euVbBWgbMugnlzfLgFgxj+pup8qUWHTfCnEC3d+tZYcLKzrzoLV0JLyPgKobSNQp5sW77OQXDDTac1xwbZjRnl5CBg6Zpk7v+cEoaSIFnQPS4K8SWfn7p8rt6V4aTVHRLkoW1LPDVoOxMA/984x2TJIk96esfypratm5/u0G7D4NzW5TbUKvaFQ3VqATsuAWHl4WS2hD+6C1NIglP370ztf7mI0QsnrAW4QHHXJTo2HVL7PyQ3XhIglq+WJcxe1rxf1VyTRe73+cYBeFDwgjN+7EBFB5ykArnqN/ubC89w8O0hY1+bDPOWkwvXO1K X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: b3c3d6fa-0e19-436a-c8dd-08dafe4a68b9 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jan 2023 20:34:36.5481 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: lRbUBT/tVOfIZUM07saLpPHZNQ09YWjfpj2DIHHa4noLcZ5SJOzv9zx/h8EaFBkC X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4095 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 169E04000B X-Stat-Signature: 35qrtkzy1oyarq99k9baqwpxf6xxengf X-HE-Tag: 1674592488-368798 X-HE-Meta: U2FsdGVkX1/paPs+ORjR9tyG1Q2C0J8lDxjW6NR+zZzwwxs0qFCmzELF3QK//W4dSWRJ/dbaDQUINvg9MPy1nkqcLOxUQUhy/U5cDIaJsg+Y9hK4LEml2k4vm2rQd8gOPl8kd8O1hu+UeGjohhz1tlNnzc71Lj1cGTC8Ztm/Hm6ePdMmJvKfY0BLtvY5d+Fq9vslg8/PU8tl4mus/iUDd7V490+L1JS7v4cJhBwKmkGzCE00D7UenWhxDo9eUYQxuMpYn6/onMvEoGK1aUOQSlYER6yETkmsPzFiniRqE/4l9iRFfDRacMFphCXiau3yrBdh6nnPE3FKQt5166Ab5jVT306KYSfsLKnV75EqiOGm+N6ECyt0L1A0aqXoqUupiKUuT6zehSwjfW9tqRQoyC8Re0iyky5UupenaiAA0RjGPKNqhLU7XmqUGmXdhqMKdRuYE9sMZaj+xXHkiHBk1k+n/Jh+FlFKt9glmQMmPMOjmONbFmbtJOQHjEyRjKexRNFQTR98ZkmjvVOTcydLnm/IezFkdochU76p+enEUcTgHt4pJOw9Kz7nkrKom3BDMxFTRjd0jpvAgWP86Htep4UVYmVf6DZpOr7EAv9Flue824aKFO3OamDrhZRgJTCMxHRmaJfjzUt0KXqrBRMg9StvHxVyp2ROtLbDRFIetZp5xK8wMPW4lMIpUbj2xYhYMEuoGHRK4oRIQiidglQE0Frb5FT/La8iyBk2g1tT1OGz0StR0UiiGY2tSzYLkRSqECrOft2IZVQnfqawJWVhbzLRfq+jmZ8VbUX4CNRuthMdqYyQSwEG0rQfufWCP1/HgsC00hUTTzyTGIFZHr802nTl3Fxs6FYvSxbvflUSHzetLKV9Hx4bGie8eeTRBu+O4sDRO7EFR5/8jf6utdVFCEZFIwAU8UR7hYPxMsvPZQvl0QTgQH+qQ4RY1v6heWeRf8qfRbW48nX95H5FFQ3 uJ/hyC6i 4ErkTmVE5QOkVdvJaquQFE4Ys/8HeQhDAKzrQYpB1UmwoqxM71kWrA7j1VAyWZz/jocXNadEXSgHfLk6TzYSI1M7oc2c2+AfnGXVZdt+WTSnpPzwluSOIX5h2+8xlQgUF/KdjlFJCyftcoLJeIj5+7QqXot86TW4PWu26tne/6uZJz3ZdflG4du9ZhVwGNvEQRcG+mP7Sbo6QmxHJGsLN1Uy2JMfpA1bl3l0an2FM8vCPG8l4Us1+i7At28Lf55uze5s4MgK4yNAwbFccdaJzKrKYp7wVvr1BG7zWo0jCX3851/iRfR17zkIHBadB+6cxayuGoj9k3Q1REx0ELvGJEfI5Ag== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Setting FOLL_UNLOCKABLE allows GUP to lock/unlock the mmap lock on its own. It is a more explicit replacement for locked != NULL. This clears the way for passing in locked = 1, without intending that the lock can be unlocked. Set the flag in all cases where it is used, eg locked is present in the external interface or locked is used internally with locked = 0. Acked-by: Mike Rapoport (IBM) Reviewed-by: John Hubbard Signed-off-by: Jason Gunthorpe --- include/linux/mm_types.h | 1 + mm/gup.c | 36 +++++++++++++++++++++++------------- 2 files changed, 24 insertions(+), 13 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 8971a40c120e38..518617431f8085 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1103,5 +1103,6 @@ typedef unsigned int __bitwise zap_flags_t; #define FOLL_FAST_ONLY 0x80000 /* gup_fast: prevent fall-back to slow gup */ #define FOLL_PCI_P2PDMA 0x100000 /* allow returning PCI P2PDMA pages */ #define FOLL_INTERRUPTIBLE 0x200000 /* allow interrupts from generic signals */ +#define FOLL_UNLOCKABLE 0x400000 /* allow unlocking the mmap lock (internal only) */ #endif /* _LINUX_MM_TYPES_H */ diff --git a/mm/gup.c b/mm/gup.c index dfb315b3f2950d..37849fcd962e9a 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -896,7 +896,7 @@ static int faultin_page(struct vm_area_struct *vma, fault_flags |= FAULT_FLAG_WRITE; if (*flags & FOLL_REMOTE) fault_flags |= FAULT_FLAG_REMOTE; - if (locked) { + if (*flags & FOLL_UNLOCKABLE) { fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; /* * FAULT_FLAG_INTERRUPTIBLE is opt-in. GUP callers must set @@ -1382,9 +1382,11 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm, for (;;) { ret = __get_user_pages(mm, start, nr_pages, flags, pages, vmas, locked); - if (!locked) + if (!(flags & FOLL_UNLOCKABLE)) { /* VM_FAULT_RETRY couldn't trigger, bypass */ - return ret; + pages_done = ret; + break; + } /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */ if (!*locked) { @@ -1532,6 +1534,9 @@ long populate_vma_page_range(struct vm_area_struct *vma, if (vma_is_accessible(vma)) gup_flags |= FOLL_FORCE; + if (locked) + gup_flags |= FOLL_UNLOCKABLE; + /* * We made sure addr is within a VMA, so the following will * not result in a stack expansion that recurses back here. @@ -1583,7 +1588,7 @@ long faultin_vma_page_range(struct vm_area_struct *vma, unsigned long start, * a poisoned page. * !FOLL_FORCE: Require proper access permissions. */ - gup_flags = FOLL_TOUCH | FOLL_HWPOISON; + gup_flags = FOLL_TOUCH | FOLL_HWPOISON | FOLL_UNLOCKABLE; if (write) gup_flags |= FOLL_WRITE; @@ -2107,12 +2112,20 @@ static bool is_valid_gup_args(struct page **pages, struct vm_area_struct **vmas, * interfaces: * - FOLL_PIN/FOLL_TRIED/FOLL_FAST_ONLY are internal only * - FOLL_REMOTE is internal only and used on follow_page() + * - FOLL_UNLOCKABLE is internal only and used if locked is !NULL */ - if (WARN_ON_ONCE(gup_flags & (FOLL_PIN | FOLL_TRIED | + if (WARN_ON_ONCE(gup_flags & (FOLL_PIN | FOLL_TRIED | FOLL_UNLOCKABLE | FOLL_REMOTE | FOLL_FAST_ONLY))) return false; gup_flags |= to_set; + if (locked) { + /* At the external interface locked must be set */ + if (WARN_ON_ONCE(*locked != 1)) + return false; + + gup_flags |= FOLL_UNLOCKABLE; + } /* FOLL_GET and FOLL_PIN are mutually exclusive. */ if (WARN_ON_ONCE((gup_flags & (FOLL_PIN | FOLL_GET)) == @@ -2127,10 +2140,6 @@ static bool is_valid_gup_args(struct page **pages, struct vm_area_struct **vmas, if (WARN_ON_ONCE((gup_flags & (FOLL_GET | FOLL_PIN)) && !pages)) return false; - /* At the external interface locked must be set */ - if (WARN_ON_ONCE(locked && *locked != 1)) - return false; - /* We want to allow the pgmap to be hot-unplugged at all times */ if (WARN_ON_ONCE((gup_flags & FOLL_LONGTERM) && (gup_flags & FOLL_PCI_P2PDMA))) @@ -2140,7 +2149,7 @@ static bool is_valid_gup_args(struct page **pages, struct vm_area_struct **vmas, * Can't use VMAs with locked, as locked allows GUP to unlock * which invalidates the vmas array */ - if (WARN_ON_ONCE(vmas && locked)) + if (WARN_ON_ONCE(vmas && (gup_flags & FOLL_UNLOCKABLE))) return false; *gup_flags_p = gup_flags; @@ -2280,7 +2289,8 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages, { int locked = 0; - if (!is_valid_gup_args(pages, NULL, NULL, &gup_flags, FOLL_TOUCH)) + if (!is_valid_gup_args(pages, NULL, NULL, &gup_flags, + FOLL_TOUCH | FOLL_UNLOCKABLE)) return -EINVAL; return __get_user_pages_locked(current->mm, start, nr_pages, pages, @@ -2968,7 +2978,7 @@ static int internal_get_user_pages_fast(unsigned long start, pages += nr_pinned; ret = __gup_longterm_locked(current->mm, start, nr_pages - nr_pinned, pages, NULL, &locked, - gup_flags | FOLL_TOUCH); + gup_flags | FOLL_TOUCH | FOLL_UNLOCKABLE); if (ret < 0) { /* * The caller has to unpin the pages we already pinned so @@ -3195,7 +3205,7 @@ long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages, int locked = 0; if (!is_valid_gup_args(pages, NULL, NULL, &gup_flags, - FOLL_PIN | FOLL_TOUCH)) + FOLL_PIN | FOLL_TOUCH | FOLL_UNLOCKABLE)) return 0; return __gup_longterm_locked(current->mm, start, nr_pages, pages, NULL,