From patchwork Tue Jan 24 20:34:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13114788 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9306C54EED for ; Tue, 24 Jan 2023 20:35:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A593C6B0085; Tue, 24 Jan 2023 15:34:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 942BE6B0087; Tue, 24 Jan 2023 15:34:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F9EF6B0089; Tue, 24 Jan 2023 15:34:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 5E7F36B0087 for ; Tue, 24 Jan 2023 15:34:57 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 39463AB1D5 for ; Tue, 24 Jan 2023 20:34:57 +0000 (UTC) X-FDA: 80390846634.23.A131987 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2041.outbound.protection.outlook.com [40.107.236.41]) by imf12.hostedemail.com (Postfix) with ESMTP id 82B4A40018 for ; Tue, 24 Jan 2023 20:34:54 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=DYKFYF0t; spf=pass (imf12.hostedemail.com: domain of jgg@nvidia.com designates 40.107.236.41 as permitted sender) smtp.mailfrom=jgg@nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674592494; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=S5ttauOJcEpPx3GZLCR6G/PephTpgeGFxt4/6gTcmLc=; b=313Ey4Mj3CvF3R2oTzNcsowQgFso5xgp+hnbJjG5/I1oY7OnawXqqVkXQ7Wd8EoefuH4jf h5pz/qHkklbXHZx79TDO2Yuo57f+IjeTzgeDsaqRxlR46UPxPemgRXMmnsRzyg1DNg1Alq XyH3ML8LLmhV2Mb8F6pKRP4awq/6mxQ= ARC-Authentication-Results: i=2; imf12.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=DYKFYF0t; spf=pass (imf12.hostedemail.com: domain of jgg@nvidia.com designates 40.107.236.41 as permitted sender) smtp.mailfrom=jgg@nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1674592494; a=rsa-sha256; cv=pass; b=xdXPjBGyrKsoIu26v30giTbvPYeoAhHX2hSq2CO/xL5L0IsBWzmjlNtC4gn0tZ2aboRaS3 EzAqL5ulD60ScxILJ1YefV0ahLepmF0htHYVaHNRiEN+hNJ3Nkn44lioxZvNOnpEn2/Ykh Kqk6UgHWVSzPwoVW8AIQq80OuTtbRF8= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ff3/cED01mXGv4Siw6k/9gbsGpiH5jAN6BXDZoZPJ0mrMFRyGkZEIyFedtr/sE5U7Rq5P9RSPUE+4emMgCGSnL6QylF5Q7BT6zECzRYIPZBSyDoNaQgNwAeAl39eOtKm4vOWmDvYjz28WWbx+Q4qIbGPPgZ1i74TJw2L0SFPwdzAeyR7+n3l7ZySLLcXqIRrUGkE3Jei5Q8cM2thkvj0zDArj6QV/ob1q1MNcgumrVFqyCqG+HWlPkg4PhAP+6HzlNDeztwicXCnnzusXDBzmZBGlDpIuXmUaNOhvkZaeUwV1MtdQv16q+nxY7BBgMLLLASAfXoNnC4z2oJgSb6roA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=S5ttauOJcEpPx3GZLCR6G/PephTpgeGFxt4/6gTcmLc=; b=bNg7jfk087d+rkjJAQh4svEsxz8ZpLDEQb68VJAy/BoSBdbz5m3tilggjFUB3D/h7HcQwVRMcADtmRKasbtomOSrM01uT81KdmPolqoISepu5K5+xZuczrC1uondKEtn59xaYp9Bl3mlX1B3SjobRgWBH/o9P/whhs8lTTmkTwjLBDSlAVeDAxXaH0aCmB/Qyxx1W8gaaDwYlzJmwswaR4uo6/r+6Q9vthYs4EvFkRWtd6vt648yFAZcoz9Lo7KEFhxWNPgPaE5qVcA0rFanvEYwk/2F4ByUtl270hVj5zsJHzfHlSPIFZWZFP4Oz+lBRU4YI1YkK9SzzrCVkISokQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=S5ttauOJcEpPx3GZLCR6G/PephTpgeGFxt4/6gTcmLc=; b=DYKFYF0tEQg5HJiFEJQqNoDCUXwZ1H5SfrkMv3N50pLbMbErbLk/eMZbBFyjmsacJzd1egS52WWeofuEOFMnJ1OX9tyTGbmARKcTJ19boIfbcnEGqEIhC07WSk4GT8GJItTtLbQrScK9EQAwKeWO9DCruVxTFUA0Sz4UerCkjbIM7y/Du7RuE32RI+vi7wRUcg4cIU9r6WI6rA9o4jWJwEowUSF7boc/gMyT8vXJtHBWvDNa51TOaiS09zDCTGasMNOoy9p0SBiYDiWIw2qjr3JrPsi2wIsXLksMpE2WmJxKWSj6yux4s5n75jyCDKRD9U/b5LkCJT+IvuWSq5XvZg== Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by MN2PR12MB4095.namprd12.prod.outlook.com (2603:10b6:208:1d1::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.33; Tue, 24 Jan 2023 20:34:40 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::3cb3:2fce:5c8f:82ee]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::3cb3:2fce:5c8f:82ee%4]) with mapi id 15.20.6002.033; Tue, 24 Jan 2023 20:34:39 +0000 From: Jason Gunthorpe To: Cc: Alistair Popple , David Hildenbrand , David Howells , Christoph Hellwig , John Hubbard , linux-mm@kvack.org, "Mike Rapoport (IBM)" Subject: [PATCH v2 09/13] mm/gup: make locked never NULL in the internal GUP functions Date: Tue, 24 Jan 2023 16:34:30 -0400 Message-Id: <9-v2-987e91b59705+36b-gup_tidy_jgg@nvidia.com> In-Reply-To: <0-v2-987e91b59705+36b-gup_tidy_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR02CA0093.namprd02.prod.outlook.com (2603:10b6:208:51::34) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|MN2PR12MB4095:EE_ X-MS-Office365-Filtering-Correlation-Id: 975c8d53-c552-44d7-7a63-08dafe4a6920 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qdBAHHxyOurPxow9DXjB32Im4IbemwDoaMKmLVZhvSvXU5nLtIBm8gWpgd/GWcuyLH9sbBGsLw1sl1hhaXQAnfyeLlF0Bb8WWZ3eSgqu6lbkudpLXBk3y5N/RxA/2joZd1qc+oxIyzZKKKqMcJ19Ts+Pw77oH2+ZQt/Xhd2GOK7UsC3FjkZKHVNpXx7H9GBsDghJbQSprk4pnKQURNDTzcxq4GX+CBIRGhqfOa5mU83S662afy8jIthx3a7vomA8qxzl5Vptosi1vW1IyjJyVJnl41zzV4i5K0Luz1ySibsEJBZzT+yCnLIOjERN0RwSwALhN2hWkWSFXjOxskYI3r5C2TaN0g0EyCZWsFAq8ZIY0uU7Lee8aU5Md3bJVJd+co5zgbFflh2ymplTtvJgO+Q3exO3lt48j4TJWBVqmhNgKCx350aonCPAn63CcWWMd4322vrYLu1BdbhSAUvrvSVpFugslqbAVpm4pndOngOnBNnyzjwSN3Iag5XywTgS47672BGXWDWmTIyiHZUtJRMb7ut124lUtzsNnUqMHleCGBYV/0rkip9HtzWuqYM2WVJ7eTJKV2QvulcymPlGE0dBZlPji+3GRywBPeOqv3VSRSrTn9fDQ9qKu8+312/T/7k38EwBYb0xyk8g18/Y/VLvEb5+jARPv/GNmezsQUFvC6ZjHYW+eJLITptNgGIfuVF0CPOzuyUMAdDkcr5GEg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(366004)(396003)(39860400002)(376002)(346002)(136003)(451199018)(109986016)(38100700002)(83380400001)(5660300002)(41300700001)(86362001)(2906002)(26005)(4326008)(8936002)(316002)(6666004)(6512007)(6506007)(8676002)(186003)(66476007)(66556008)(54906003)(2616005)(478600001)(6486002)(36756003)(66946007)(4216001)(266003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: MGHIy5CxrS/hK7gIHUQPqvy+DrZ/rqm2JBwrI3zIm0HtOnXP+z3xfMoT9hsj/VCI/bDCsojpnXtBhBdO9QZxmlbNfVhBKXSjBuEeIocY74Dq2GCjtcdiOZuf3BYWXQxxPhXLvJAm62jzAlSJ0DC1St2jaDdEaKjZtWQKpp/7IoT/8bkjLIud7gVgY/HgIqQekC4tXaHNEwbsQgWZSh6Dl5cpgR2X5HQS5f2CSdjnwb3NeLb8qEX39FTIBLoGF/B7Rs1KCifu1MHJY67OLHNWtfNAU0puHkzJ6HEfhZUt72/1qrPC8UxZNR1l0DuQJ6EP8DfouCEcrjtYCFvyRES4oMVmSoNf/TozpCfhhCTD/roHkQj0NYiUVl/Nif3F0E6mZ+mUWwYNeAraU/mg5KOxITllHROV3cluEPKty1JOMO3WayaQR2JUA47IaYgHZgOKufHkUpnBWdLWP5PwZ5nzyvPI2KQFfmRxOT864dLG9Df5MS4ytDTmKHcfSVsHmhP7IZgeRUt5AL+2Y3m3svpnTnQwX5gfwAbfYU8bSjoDTXPpEH8fu5PBRB+oA+ac9KGbGuZGFoWBDmQPmgBX+9PUzYKHBIH+9swrMobrdPP8t/wsiGgyrZE2jw7Oesrgsa3A1esCw0xt5SUYUR2abh5IESiBOUtAdvuNb7BQKLLck+JcIPE5d+CETwKi4IEM6HtboEEZUvWm7gJZb8uDmG6xjhR8p5jlCWQLIvsItna3kKKO/SFwGJtr1TWVIp15RaWN1fxw2v04NkdtDCWDUp9dAbVFjw+B31xZHCZJ1N8vXg71VYMTBDX4h7nlM7PMPxUrvwBhWumY92/wACfPxqPuv0GF0sC37noc+r1/YnfIEqWRA8QaCdIa5nubwUa7TRmIkHVW006eeSxTRp5TOYqXLrJbe0e8aYObWJp9Qx7B7kGY+HqHo7NbxqGqkrQ1MicszBtQL6gBS/NgN9zsTkwcHCOeGDbOIryywaekaWFQujUKoQvpuSUToqwKGV+gYYksIQbH0Ito/tjNoSg8V1wi29rfDQlkAOvPgjxPxLdKDclPfcHft+H8q4WOx/uo4KbByg+x38EMdrUMFuQN07ZZpbiAcHlkSGcaymnTQq8QcrzOHUOw0ZORNX3ANSfBMimBpWN2jv+rrR1g+9DJLjMHI91FWDcd2KdaC/Ja/bTZvfptg//HhQ0aEk6QekMQ1s3ppZa5oKiIZlN8T83GtJveJ1t9/+qYVGMim0yWszt6882E5jqO4nKg/NRxwSQPOCCJnAZSsNjJ3fA0qBgijYGGJrSMDo4QS/R55a/Kf9ch6cMaq8o+//0Q1T2//fBO/O2rfdMfqrNeEvVOzWgNeM4xK9z5UB809n8Of3CNLXwne0F6PfMSSkuG+vVyQq9IbbSFzt5NCtdTExeQpPGmLEsCnTRnzsz0qaxgY66QxhMi7XDpI1OEXgEnSoyUE7SUhXr0Y5NnyGEG//qX9kKSzpyQFORRHdscotsRC4WdlSvRI9mq5iOWWfJvso4whe2HZZ47HFuakrFw+RiXCd0thY+DAqAk1imiugCuVNxzr3kt49IYATlI3LWBS9LZWLhlCiyY X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 975c8d53-c552-44d7-7a63-08dafe4a6920 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jan 2023 20:34:37.2523 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: RKP4fhw+X12kgL3J1dlXUAlzqFvw4vVWvMKvA/YF4AlmSXE5c1b3Wg6D1R/1ygFw X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4095 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 82B4A40018 X-Stat-Signature: toyphu8zrsuetqwr5hgcjcnh59im1z3b X-HE-Tag: 1674592494-129174 X-HE-Meta: U2FsdGVkX1/kVsV/REqcfcBm1eEvVxG3qRV0WrJY5PAEYFFoIZEuWdF28Q+/MJTNoDo0DPnC75Ue8yBOuQ68f3STGoTgmzqx5MOTYtndd8EXKf+YIISzbNBhp6fg6t+XXgzQW1casrhDKO2m8vYbSlGcT6hVzWiuec9OULTjNXfD6Nyq6fzeyQVH57KH4d/VX1SKLWHm61aI337tXUiU8V8pE7fpsYWy3EX2B2vTa0Er6GK9Ep/gYNZhtrtdc5BYZJW/Syd4W8UCJwb5tUu8BGPGQlzB3yQ/Tl1VVbiKphzb5VAEksiFFEO3HQrACxssZ+JpiTEd3Cd7tOZVaDzXt9+LpzAxzm6C0kKiKs9DiMOEhDqpEwrlcT+N3JtZO09XNcRVRGNU18PbaKpparvU6WDFTVXQk4vpo5kI39NgGTxqf6HSZSK2wX4XZ+HNQluak5ZylsBYXKYzaLkM5ziIvuqqCqPkwa2Pmbbi36pELUrgq7ro20lDJTB2zOPs+62w8PefYSRpbuumv5KUN1Gf/oHlrgHk3g0JCNnvp+U1vNDAqRPF/B+LMXkN59Eu14Cnu023u0H7okie2pvaN0Sc2mN+gdpiwvAev87wp2K9CihDd9odYk367CwRB8AJoKaKhA/OHlq5AuZmJw6mDG/cKCHObEqCcA09KL06hEfozLueXVijLe6OQDJ3Cu1vy55R1fDeEO8DzeQDLQzOsADw+LYWj2Sm3qFvYgO3+5j0GGYRD6DM1UwQJZ8Y2vHT7WNGfrAiIArkxFQN6+7P8kefTqSHLYdW/+oZPUtMaqPvUJTMjDTk8e0HeeGubR2uhciRSeBrYINPZZJUBfKJdQ6rU0eHVcwHqWknMShicJpPF+XRjyBqvL2jLy+CIZ3W9b7QyUwLjR/X2fz2g2PpcznkTkRxRfp8/3OqjHHfzMUTIgKjyENmPws3XWZ2NSGTy6tKgK7Vlzs7f10lbo62wzI sTvWZaSN nokMbpuC4VRg/F1gjPDZcyAKw7/l5Tjzt6uuHsfR0aa5wGOk2bQfzxqr4EXHkcJ6N9X5CIwS9ivTu8bsq3W01BFYBgwYLvZ6AESQRAEl07HAAR3yaYSsm86hZOBKA91BbRoxulVEXf5dsFjizk2lO/wPE+wx9fZyjFZQaspgFbRZldmh46VLjP1Oqpq/AHBKBHaAqwB2EdWhWn8NoXsjrk4DIotiy5lKHg/HqXy7F5IYm6+TyMsxXwHjm5IeSxobRk8BgVRW9YhfjnpMvlNC4Mu6RBpMQNAdZdqDs12uTRo6YbSc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that NULL locked doesn't have a special meaning we can just make it non-NULL in all cases and remove the special tests. get_user_pages() and pin_user_pages() can safely pass in a locked = 1 get_user_pages_remote) and pin_user_pages_remote() can swap in a local variable for locked if NULL is passed. Remove all the NULL checks. Acked-by: Mike Rapoport (IBM) Signed-off-by: Jason Gunthorpe Reviewed-by: John Hubbard --- mm/gup.c | 51 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 30 insertions(+), 21 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 37849fcd962e9a..932a2339613c2f 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -879,9 +879,9 @@ static int get_gate_page(struct mm_struct *mm, unsigned long address, } /* - * mmap_lock must be held on entry. If @locked != NULL and *@flags - * does not include FOLL_NOWAIT, the mmap_lock may be released. If it - * is, *@locked will be set to 0 and -EBUSY returned. + * mmap_lock must be held on entry. If @flags has FOLL_UNLOCKABLE but not + * FOLL_NOWAIT, the mmap_lock may be released. If it is, *@locked will be set + * to 0 and -EBUSY returned. */ static int faultin_page(struct vm_area_struct *vma, unsigned long address, unsigned int *flags, bool unshare, @@ -930,8 +930,8 @@ static int faultin_page(struct vm_area_struct *vma, * mmap lock in the page fault handler. Sanity check this. */ WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT); - if (locked) - *locked = 0; + *locked = 0; + /* * We should do the same as VM_FAULT_RETRY, but let's not * return -EBUSY since that's not reflecting the reality of @@ -951,7 +951,7 @@ static int faultin_page(struct vm_area_struct *vma, } if (ret & VM_FAULT_RETRY) { - if (locked && !(fault_flags & FAULT_FLAG_RETRY_NOWAIT)) + if (!(fault_flags & FAULT_FLAG_RETRY_NOWAIT)) *locked = 0; return -EBUSY; } @@ -1062,14 +1062,12 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags) * appropriate) must be called after the page is finished with, and * before put_page is called. * - * If @locked != NULL, *@locked will be set to 0 when mmap_lock is - * released by an up_read(). That can happen if @gup_flags does not - * have FOLL_NOWAIT. + * If FOLL_UNLOCKABLE is set without FOLL_NOWAIT then the mmap_lock may + * be released. If this happens *@locked will be set to 0 on return. * - * A caller using such a combination of @locked and @gup_flags - * must therefore hold the mmap_lock for reading only, and recognize - * when it's been released. Otherwise, it must be held for either - * reading or writing and will not be released. + * A caller using such a combination of @gup_flags must therefore hold the + * mmap_lock for reading only, and recognize when it's been released. Otherwise, + * it must be held for either reading or writing and will not be released. * * In most cases, get_user_pages or get_user_pages_fast should be used * instead of __get_user_pages. __get_user_pages should be used only if @@ -1121,7 +1119,7 @@ static long __get_user_pages(struct mm_struct *mm, i = follow_hugetlb_page(mm, vma, pages, vmas, &start, &nr_pages, i, gup_flags, locked); - if (locked && *locked == 0) { + if (!*locked) { /* * We've got a VM_FAULT_RETRY * and we've lost mmap_lock. @@ -1354,7 +1352,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm, * The internal caller expects GUP to manage the lock internally and the * lock must be released when this returns. */ - if (locked && !*locked) { + if (!*locked) { if (mmap_read_lock_killable(mm)) return -EAGAIN; must_unlock = true; @@ -1502,6 +1500,7 @@ long populate_vma_page_range(struct vm_area_struct *vma, { struct mm_struct *mm = vma->vm_mm; unsigned long nr_pages = (end - start) / PAGE_SIZE; + int local_locked = 1; int gup_flags; long ret; @@ -1542,7 +1541,7 @@ long populate_vma_page_range(struct vm_area_struct *vma, * not result in a stack expansion that recurses back here. */ ret = __get_user_pages(mm, start, nr_pages, gup_flags, - NULL, NULL, locked); + NULL, NULL, locked ? locked : &local_locked); lru_add_drain(); return ret; } @@ -1683,7 +1682,7 @@ static long __get_user_pages_locked(struct mm_struct *mm, unsigned long start, * The internal caller expects GUP to manage the lock internally and the * lock must be released when this returns. */ - if (locked && !*locked) { + if (!*locked) { if (mmap_read_lock_killable(mm)) return -EAGAIN; must_unlock = true; @@ -2222,11 +2221,14 @@ long get_user_pages_remote(struct mm_struct *mm, unsigned int gup_flags, struct page **pages, struct vm_area_struct **vmas, int *locked) { + int local_locked = 1; + if (!is_valid_gup_args(pages, vmas, locked, &gup_flags, FOLL_TOUCH | FOLL_REMOTE)) return -EINVAL; - return __get_user_pages_locked(mm, start, nr_pages, pages, vmas, locked, + return __get_user_pages_locked(mm, start, nr_pages, pages, vmas, + locked ? locked : &local_locked, gup_flags); } EXPORT_SYMBOL(get_user_pages_remote); @@ -2261,11 +2263,13 @@ long get_user_pages(unsigned long start, unsigned long nr_pages, unsigned int gup_flags, struct page **pages, struct vm_area_struct **vmas) { + int locked = 1; + if (!is_valid_gup_args(pages, vmas, NULL, &gup_flags, FOLL_TOUCH)) return -EINVAL; return __get_user_pages_locked(current->mm, start, nr_pages, pages, - vmas, NULL, gup_flags); + vmas, &locked, gup_flags); } EXPORT_SYMBOL(get_user_pages); @@ -3158,10 +3162,13 @@ long pin_user_pages_remote(struct mm_struct *mm, unsigned int gup_flags, struct page **pages, struct vm_area_struct **vmas, int *locked) { + int local_locked = 1; + if (!is_valid_gup_args(pages, vmas, locked, &gup_flags, FOLL_PIN | FOLL_TOUCH | FOLL_REMOTE)) return 0; - return __gup_longterm_locked(mm, start, nr_pages, pages, vmas, locked, + return __gup_longterm_locked(mm, start, nr_pages, pages, vmas, + locked ? locked : &local_locked, gup_flags); } EXPORT_SYMBOL(pin_user_pages_remote); @@ -3187,10 +3194,12 @@ long pin_user_pages(unsigned long start, unsigned long nr_pages, unsigned int gup_flags, struct page **pages, struct vm_area_struct **vmas) { + int locked = 1; + if (!is_valid_gup_args(pages, vmas, NULL, &gup_flags, FOLL_PIN)) return 0; return __gup_longterm_locked(current->mm, start, nr_pages, - pages, vmas, NULL, gup_flags); + pages, vmas, &locked, gup_flags); } EXPORT_SYMBOL(pin_user_pages);