From patchwork Wed Jun 5 09:10:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 10976497 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C731A6C5 for ; Wed, 5 Jun 2019 09:10:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B8DCB28684 for ; Wed, 5 Jun 2019 09:10:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AB1E6288D8; Wed, 5 Jun 2019 09:10:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2CB4D28684 for ; Wed, 5 Jun 2019 09:10:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 32B626B0008; Wed, 5 Jun 2019 05:10:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2B3866B000A; Wed, 5 Jun 2019 05:10:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 12DBD6B000C; Wed, 5 Jun 2019 05:10:45 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id C90896B0008 for ; Wed, 5 Jun 2019 05:10:44 -0400 (EDT) Received: by mail-pl1-f197.google.com with SMTP id g11so15711753plt.23 for ; Wed, 05 Jun 2019 02:10:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id; bh=A0F5TwKY/VCSLcp+XKNpiRnciFopNfIc4l3QtF9bGnE=; b=rOrbHj5qh+5WjaQzJ9ccATU00EAR62BVoQLd0G+AxiaWzqSl1KYckw5IlBE47nacLZ IFHN4lUalf1SZ5CK8BCSj4x4d6f9/E7wi2iYXBpIj1AMeyPJm70GNK4hQ0Tr9JbBRzNs nQon3cehkKmqgYtNL8HAUWVG35iKSA0fmKfl4Kviox2+eD/eENw1iBzVaC0iB/+LdOD2 3uhByLBpUDj1NBmKLtbf1yAr4NqFS9N237bAXUWZ06DEz5NPMOrN0C+d4A27ISeV4p4/ j57zzltbBK7X5oQNzsXBlHU2szoGV/o0plydV+yy8ZGnY0PNX5O5wrBBhbAKz7JPCZ/V 7YjQ== X-Gm-Message-State: APjAAAWQgtDqfk4bQ9m4Kwu6tVw/fClQFSvaHXKqvqzrbQg5T8aMAYUa fJ0AgZyyFE4O4rSX0MjJ7nBvernJHMdGc4NPc65kZGlcfDbII237qOVAGOjCsmE58v7mpUvdQO9 jj/DxqdupDxWgwK7DJN3eyr1423ngX3v0DVuuqA6U140VjLRPYEIMfDAKM72qVc7Esw== X-Received: by 2002:a17:90a:bf02:: with SMTP id c2mr15758723pjs.73.1559725844285; Wed, 05 Jun 2019 02:10:44 -0700 (PDT) X-Received: by 2002:a17:90a:bf02:: with SMTP id c2mr15758625pjs.73.1559725843221; Wed, 05 Jun 2019 02:10:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559725843; cv=none; d=google.com; s=arc-20160816; b=HsxivcSQFrwPc0pdfo72pDex3DHLJa/GoNDvPHbjba4VZOdjtIQASEu1uBEZJaPoQn RWu0Ki3LyIRlwB2SwVMtLJu8I8v/HzGN74LK2js6M48xLBAd3hfvZkDk+PmJtVOYOLA7 pqS5+aYMLo1U61mexcBsDMjtQ8oOHv4dX+ghwXgFxZJd9+NCgaHleM2WQ9hQehxh9+m3 neb0wem3IBYxcDQqwrgbkjFyR0XtoD2PUXOatPLNbcpdaxaWOpNTHO4jUh+zZfKALnxb 4g8r/SFaJkccLsg7TFHRs0vsOL/SFYI5PM1S0x7pDEAecWHgf7PBaMuNKtn4yOLGGnJ9 yF+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:date:subject:cc:to:from:dkim-signature; bh=A0F5TwKY/VCSLcp+XKNpiRnciFopNfIc4l3QtF9bGnE=; b=UUIJORyAeziuQCjLbpW95g+TbhgKqBXueyABs/a/3/XtqatTMHEGhrB9OULsqqjZXF zKu72B8KqlBm15YDse9J9aB2lZpvP+QKpy9rI+qti7MYy9vsi8q21GTza7ypI6bdIU+n BNRxfN+WTeXqZwCvXslpE+Mper06JZWETV6ziOy21es+3/E8cUXfIwektzTw5xgu6mMw 07+WO+NJ6tEKkeUqSaWz5VtFYyb6r/+blpWPvdOrziBnzlTD4OSoLODxDXszWpStHeV9 EslFLhxhjz5ojBc4VlVq8fwgd5Ys2ZabwUV+vaZXHr8GVDZt0KxBNvx9bTW2HLuvsmVX Iulw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=qsKJodQe; spf=pass (google.com: domain of kernelfans@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=kernelfans@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id m32sor10391289pld.47.2019.06.05.02.10.43 for (Google Transport Security); Wed, 05 Jun 2019 02:10:43 -0700 (PDT) Received-SPF: pass (google.com: domain of kernelfans@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=qsKJodQe; spf=pass (google.com: domain of kernelfans@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=kernelfans@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=A0F5TwKY/VCSLcp+XKNpiRnciFopNfIc4l3QtF9bGnE=; b=qsKJodQeWHQrpBSUsCV/81Z7vFl9IiPrLe7Wx+TTMYwOb8vRhY7qMbyumwt8N6nzUp huDpz/WgJhGZQJNcubX5sDMOT5m/6LbguxcPIte89UJaj/B9lKADRPVtqIepVB+oX5sd hnXgkLK8L2MAW5lPvWDC99pJ4HSaC8QE332V5c6l5n9kmFfM5U7iJgD+3HoP8nJY6/oe T2TGY4+lUrdaa/iyrZ9+h1U0EN1aUJNorBJTwurwt7/FrvQoDKhFU75jWXstBB47BFp4 vZG8Ouj9I7fWnXUOm8nkebeHXwU6K1NcK2iJ+B0SMP3aK6IXE+yb2bmKcQto4teTeueg RHKw== X-Google-Smtp-Source: APXvYqwPrprfYIFcfp2U4RhBi6aA36+dRA/hHPLVJndUrShnel3Up//HynanoblmvqR0aoxy92SoVg== X-Received: by 2002:a17:902:624:: with SMTP id 33mr42733906plg.325.1559725842640; Wed, 05 Jun 2019 02:10:42 -0700 (PDT) Received: from mylaptop.nay.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id w36sm11844525pgl.62.2019.06.05.02.10.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Jun 2019 02:10:39 -0700 (PDT) From: Pingfan Liu To: linux-mm@kvack.org Cc: Pingfan Liu , Ira Weiny , Andrew Morton , Mike Rapoport , Dan Williams , Matthew Wilcox , John Hubbard , "Aneesh Kumar K.V" , Keith Busch , Christoph Hellwig , linux-kernel@vger.kernel.org Subject: [PATCHv3 1/2] mm/gup: fix omission of check on FOLL_LONGTERM in get_user_pages_fast() Date: Wed, 5 Jun 2019 17:10:19 +0800 Message-Id: <1559725820-26138-1-git-send-email-kernelfans@gmail.com> X-Mailer: git-send-email 2.7.5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP As for FOLL_LONGTERM, it is checked in the slow path __gup_longterm_unlocked(). But it is not checked in the fast path, which means a possible leak of CMA page to longterm pinned requirement through this crack. Place a check in the fast path. Signed-off-by: Pingfan Liu Cc: Ira Weiny Cc: Andrew Morton Cc: Mike Rapoport Cc: Dan Williams Cc: Matthew Wilcox Cc: John Hubbard Cc: "Aneesh Kumar K.V" Cc: Keith Busch Cc: Christoph Hellwig Cc: linux-kernel@vger.kernel.org --- mm/gup.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index f173fcb..0e59af9 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2196,6 +2196,26 @@ static int __gup_longterm_unlocked(unsigned long start, int nr_pages, return ret; } +#ifdef CONFIG_CMA +static inline int reject_cma_pages(int nr_pinned, struct page **pages) +{ + int i; + + for (i = 0; i < nr_pinned; i++) + if (is_migrate_cma_page(pages[i])) { + put_user_pages(pages + i, nr_pinned - i); + return i; + } + + return nr_pinned; +} +#else +static inline int reject_cma_pages(int nr_pinned, struct page **pages) +{ + return nr_pinned; +} +#endif + /** * get_user_pages_fast() - pin user pages in memory * @start: starting user address @@ -2236,6 +2256,9 @@ int get_user_pages_fast(unsigned long start, int nr_pages, ret = nr; } + if (unlikely(gup_flags & FOLL_LONGTERM) && nr) + nr = reject_cma_pages(nr, pages); + if (nr < nr_pages) { /* Try to get the remaining pages with get_user_pages */ start += nr << PAGE_SHIFT; From patchwork Wed Jun 5 09:10:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 10976499 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 546D76C5 for ; Wed, 5 Jun 2019 09:10:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 467C3288D2 for ; Wed, 5 Jun 2019 09:10:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3A706288DA; Wed, 5 Jun 2019 09:10:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AF076288D2 for ; Wed, 5 Jun 2019 09:10:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A2F4C6B000C; Wed, 5 Jun 2019 05:10:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9B7C26B000D; Wed, 5 Jun 2019 05:10:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 880AE6B000E; Wed, 5 Jun 2019 05:10:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 4A68D6B000C for ; Wed, 5 Jun 2019 05:10:49 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id d7so14390888pgc.8 for ; Wed, 05 Jun 2019 02:10:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=qe2qEqCwB2kLhiPVN6wzjgcS8llGOmV8LcU4p/YtuR0=; b=ukgaMaQJK2K8qz7wXX71YKJof/r0esjhj/rbNRz2ReRaEhIr+fp7D2DIhGq6Yr2Sju yidaBQLpZ7eOn7+9qya5FfYTRYoY/YIu6qdQf1CovkPjNOk/0pCNqiQpZVtaVfvXcKKT h9PKi92YD8u8RC3GDThrlguFZkq/ZqJ37P8CSD3WQVCsCEdFxeci/F4wxQnblpILcdm6 l4i2rf8kFnyxq3qVAMpEcrM1UM2HeeL/ntaR42EmT6EZCApjBFOmA0j3uiX1kWrFB9+4 2AjsBPqVkyEbuHsuVDvtXTmA7I/+zyXyCL82t12uoK1ie01zyECp+LQaTbfmDyoZgXAF xTfQ== X-Gm-Message-State: APjAAAXdDr7CVg6rLJ09hbUspFU+60InqzSXAPgsCt7mrHJX2lGvuSjh adjxXa6yhCeuNFCzgvdrupEUEMTDwPhPMKjRDw5NuzmfCRyIavO+gA8jw9DX8vxumrdCaw5qAum OyKyBFLRo7cN5mTfUHaK5YQPLVF/Oiq+yWVPNgrg8wYY/bL5tihpUuhJQGc1ErXZCrw== X-Received: by 2002:a63:d008:: with SMTP id z8mr3018036pgf.335.1559725848733; Wed, 05 Jun 2019 02:10:48 -0700 (PDT) X-Received: by 2002:a63:d008:: with SMTP id z8mr3017925pgf.335.1559725847458; Wed, 05 Jun 2019 02:10:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559725847; cv=none; d=google.com; s=arc-20160816; b=dxSU9jcq8uxgvNFC+KxRul4s6nBI89KAgi1UuIlymuo4xlAHsZrjuWB3n2PV+kpPV+ rglEO1ED+3qsWh5KPqeWPfV4Ia38EXwKr3i4hgltDMhDj9Ls/7g+qlxsYe5Vp3aDf2E1 uuJCgk+69kLlOAojwkaTRH6R+HszbY4ZLl4Gg74TIEAwsipEk1kkV3y/QRV3DNrH6A68 ZelEgPfNd/180mKeb4cH8opx8cE1z8tqYD+DmZApZ62tk7pFidI4E9++GVnCHcnzuxCB +a5UhRsjxvX871FB/hsCvbRwQ/VomJmhcLvAmT0o4q1WLJa1qWHITQOE1zDDZH54m1OG kq5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=qe2qEqCwB2kLhiPVN6wzjgcS8llGOmV8LcU4p/YtuR0=; b=sOsGDsCuz7qMSdPBFlwHEGep3DBW+VJRddXRJuBCCOfa+8I8dK9tmQynn4xfLA4UXf 0ROqNVCyJrTc6waOB7H6zxIy29rI70FaFt+Y7SM4vpISbfiexnMfiKi3OjM7XNqJPixL fld4sB1XTw3IB3Fb42Otf4E8h69vTxEuYgrVSbtkaA0XVMSZTxR36StYkTIU4BuzySOK 8MP5jhAPPt17qeGF6pMonBBdTsNQTPaZP8nxscbqAx3LvZ8uZq7maTUG++jWMGaZYmhB 323vm/N+2eK2yA459FPtQMplRLLIe+nCPVP+oG1Ip9olZXrQPz+Fn1Swq9ONhZg5tBFm OpRw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=k8PDFRD6; spf=pass (google.com: domain of kernelfans@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=kernelfans@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id j35sor20860994pgm.60.2019.06.05.02.10.47 for (Google Transport Security); Wed, 05 Jun 2019 02:10:47 -0700 (PDT) Received-SPF: pass (google.com: domain of kernelfans@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=k8PDFRD6; spf=pass (google.com: domain of kernelfans@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=kernelfans@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=qe2qEqCwB2kLhiPVN6wzjgcS8llGOmV8LcU4p/YtuR0=; b=k8PDFRD65GfqIRKyHH1gKGhC28AUBf1rIqpRtBVKZkFOyfSofJ+xIM+bLtiUdsa1ig BfBeMXE4TTb+OvlBSRfaFAWMxhPvW7zD7u0gshBiyiG+f/AP+hFcw5/3p//x5e6yNF8i iPCPlE6TLiaKPZK4piGiEE9Pkzh5T8527HyNmPxS/jzZbdznDq+UHq0PUsYFS68PCjVH c6haPJ8Zsy54mqCByUbYktyztQAztfiraWDZZawfYhmUvogLTVoFiPHNA4cKvrB/L4eM ZBGGJxngz4OmNGVs5i1rAmncPcv9RqFAq3rg7aA50c5lr/VrCFjmg28FjvRrq5P13Hx3 Xgog== X-Google-Smtp-Source: APXvYqxhrX6wAssVXs2ZclXfwzF30GmChMOrWvQir8bDvycAg5gzUgp1/7H0hRmAZ6tIjdILLAT+Sg== X-Received: by 2002:a63:4c54:: with SMTP id m20mr2997813pgl.316.1559725847045; Wed, 05 Jun 2019 02:10:47 -0700 (PDT) Received: from mylaptop.nay.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id w36sm11844525pgl.62.2019.06.05.02.10.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Jun 2019 02:10:46 -0700 (PDT) From: Pingfan Liu To: linux-mm@kvack.org Cc: Pingfan Liu , Ira Weiny , Andrew Morton , Mike Rapoport , Dan Williams , Matthew Wilcox , John Hubbard , "Aneesh Kumar K.V" , Keith Busch , Christoph Hellwig , linux-kernel@vger.kernel.org Subject: [PATCHv3 2/2] mm/gup: rename nr as nr_pinned in get_user_pages_fast() Date: Wed, 5 Jun 2019 17:10:20 +0800 Message-Id: <1559725820-26138-2-git-send-email-kernelfans@gmail.com> X-Mailer: git-send-email 2.7.5 In-Reply-To: <1559725820-26138-1-git-send-email-kernelfans@gmail.com> References: <1559725820-26138-1-git-send-email-kernelfans@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP To better reflect the held state of pages and make code self-explaining, rename nr as nr_pinned. Signed-off-by: Pingfan Liu Cc: Ira Weiny Cc: Andrew Morton Cc: Mike Rapoport Cc: Dan Williams Cc: Matthew Wilcox Cc: John Hubbard Cc: "Aneesh Kumar K.V" Cc: Keith Busch Cc: Christoph Hellwig Cc: linux-kernel@vger.kernel.org --- mm/gup.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 0e59af9..9b3c8a6 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2236,7 +2236,7 @@ int get_user_pages_fast(unsigned long start, int nr_pages, unsigned int gup_flags, struct page **pages) { unsigned long addr, len, end; - int nr = 0, ret = 0; + int nr_pinned = 0, ret = 0; start &= PAGE_MASK; addr = start; @@ -2251,28 +2251,28 @@ int get_user_pages_fast(unsigned long start, int nr_pages, if (gup_fast_permitted(start, nr_pages)) { local_irq_disable(); - gup_pgd_range(addr, end, gup_flags, pages, &nr); + gup_pgd_range(addr, end, gup_flags, pages, &nr_pinned); local_irq_enable(); - ret = nr; + ret = nr_pinned; } - if (unlikely(gup_flags & FOLL_LONGTERM) && nr) - nr = reject_cma_pages(nr, pages); + if (unlikely(gup_flags & FOLL_LONGTERM) && nr_pinned) + nr_pinned = reject_cma_pages(nr_pinned, pages); - if (nr < nr_pages) { + if (nr_pinned < nr_pages) { /* Try to get the remaining pages with get_user_pages */ - start += nr << PAGE_SHIFT; - pages += nr; + start += nr_pinned << PAGE_SHIFT; + pages += nr_pinned; - ret = __gup_longterm_unlocked(start, nr_pages - nr, + ret = __gup_longterm_unlocked(start, nr_pages - nr_pinned, gup_flags, pages); /* Have to be a bit careful with return values */ - if (nr > 0) { + if (nr_pinned > 0) { if (ret < 0) - ret = nr; + ret = nr_pinned; else - ret += nr; + ret += nr_pinned; } }