From patchwork Thu Aug 4 03:22:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 12936077 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E5CDC19F29 for ; Thu, 4 Aug 2022 03:23:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD1D78E0002; Wed, 3 Aug 2022 23:23:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A80D28E0001; Wed, 3 Aug 2022 23:23:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8FB978E0002; Wed, 3 Aug 2022 23:23:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7D06C8E0001 for ; Wed, 3 Aug 2022 23:23:00 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4C27E80663 for ; Thu, 4 Aug 2022 03:23:00 +0000 (UTC) X-FDA: 79760463720.20.589083D Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2069.outbound.protection.outlook.com [40.107.94.69]) by imf26.hostedemail.com (Postfix) with ESMTP id 1CB74140039 for ; Thu, 4 Aug 2022 03:22:57 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CPvonWM4e/uXm51BsF5yBiI+Rafyos1JJjFi+0ilKST/NHtvOOPYcZ+rGzOV2iOuEu7keSgGXpLheTenWI+YqBAdVMNMX0ZeIEz8rMlXWa7HC5ptksQwRXzTG2A1XmHqRPxQ17tSRhaF7P7bCaj/8pkTQPNRCbcSHU2ylNZRphFOLrA2NF3dfJ4UihCR6FzekBO0iJRHpIXkv3TfQzbnskM7AMeAIUKYV6SSslr4UJRzYctNybOxTxLo02zATKWp9JrxpLDYPkofk7A7XDAwxToGudxKZPBVCI3onUSv1+1m721PwKAgSJWHAs+nRUitu50ONMp8BWY6XYCwma+ABg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ie3E69NNFimjNpTukhM9g1+eptQKwtizqSs9Kze3k04=; b=TNgDjro+n+1JqGgiUlgB9M+xQuJk1XimoUWNrz/Vbjfxb9464x1wbzp4GnuS7EE0Qo/W9O1ivQRvMqPR77Jv75g1OQqPsr7HpVIbDSUtGP0nfBuq3XbwGMuuCZ/wNwk4yTB59dQcwAwE1/g73vO3T0jLLeZ/LFVK/1a3f+54Eb/5W/7gfnh6g8EHhvVTDtYJ5/VEPablAM/QHm3ZXpHA79L3o8s7UA23JDcUVPXGpYWkenwIel45/xbe21fqVnEl/voy1yS3XAcRDqS1Gll9wEcb0iwXuH22XTSPiHcJOk5a1BAnqDQp+Dn+RWz12JmZ2Wl5mgyjORxu9l1Y35ftqw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ie3E69NNFimjNpTukhM9g1+eptQKwtizqSs9Kze3k04=; b=P3bNY3lRK3GR+VhpLbx7O9mkMO9Ju9xeA9Tct+5X0QjJdSt17WcTimg3SkdEZODxMMJiyyd4Gpr/H+WRcBSDDwm0Xiyw1spkHcE/GVfFRr2HlnPy8ZLeSuDFhjZ+T/fggZwElRB8z3G4PjN2RaHS+gfXzCHyoNVL38tccBDqmSLD5kVN1iCWOPE8w7fSMD2owb0twAg/2y+LsKg9fE3wtxGBrAjr/FinOPwjmvdMdLIE3T4d3Qq2HboC/ZaigV4AVEY7oUJZ2270a6YjocdnL6pUb/KpxIGcLwK1FAthM7krrUqPizcr445JcVRz8KDh/O02upSeoXUhauPedzyjwA== Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by DM6PR12MB4988.namprd12.prod.outlook.com (2603:10b6:5:16a::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5482.12; Thu, 4 Aug 2022 03:22:55 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::eca6:a4a7:e2b2:27e7]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::eca6:a4a7:e2b2:27e7%5]) with mapi id 15.20.5482.016; Thu, 4 Aug 2022 03:22:55 +0000 From: Alistair Popple To: linux-mm@kvack.org Cc: akpm@linux-foundation.org, linux-kernel@vger.kernel.org, Alistair Popple , "Sierra Guiza, Alejandro (Alex)" , Chaitanya Kulkarni , Dan Williams , Felix Kuehling , Jason Gunthorpe , John Hubbard , Logan Gunthorpe , Miaohe Lin , Muchun Song , Pasha Tatashin , Ralph Campbell Subject: [PATCH] mm/gup.c: Refactor check_and_migrate_movable_pages() Date: Thu, 4 Aug 2022 13:22:41 +1000 Message-Id: <20220804032241.859891-1-apopple@nvidia.com> X-Mailer: git-send-email 2.35.1 X-ClientProxiedBy: SY6PR01CA0126.ausprd01.prod.outlook.com (2603:10c6:10:1b8::19) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3a7411ce-1d32-4a16-6fb1-08da75c89f04 X-MS-TrafficTypeDiagnostic: DM6PR12MB4988:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: xYknlPcWdhciHx3gJPtcI3ze32LDUflEnYNEwtoxAntuIzfGCR+8ubwIQ2hFe3g3eqaKeJJyLktA4s3ye/XxzFaw73Yp8P5ne75sqfGDvWsHlbRfekVClufmIWKPe5cKzBRSodTOiM2wmZwrL4x7lBym/3I9EaHeV398a9h0xKcspl25QvHALllC52/3LXMhUSmlafJUaXhXE+O1CtbzGn/dV2FRc+3EB4KktOGbhzk2bg5bG3bXbezqzwEHPOFgs57HqSsSt0a3oa+XiHWcoBs3qXeF/Y3vplHVTBwX92PCDScEIVpwIgyoBwzK/H2UnwvSC1fVhH2DDgAnScWsZW47KPTklIdsMFTqoW9ea4O5BLQt9r4Mlatc1HNOCsssrt9PNg9O0rBO2/SiIlRzZLM7fdQjsilWm5NG1z54fHbzVj1vQIq2MirJdzkzFbwwZcD50Pn4ss482xjl0lgcowBzzHCpZjH57afhgEjTtDq5vGYH/SNrp5HDLgAQMUrloyDLgG1xsD3B0LtvldQehlYnc7azRwnqn7MHnZpG+QVzGqj9bVS8+rc2h35pfCi9Ab/fxE8sHyFGEwxzzoeVZHcEhm+k8KUzZs0qtZryN4PV6UETuLeOt1WYcK2W/dhRmnc3mPWYomq/lFzTrwIieW6a3fxrldIIrwOL+lsrjTlXvfAJO5mrqfPeV5JvKdZHYAQ4SBDUaBFjMvw3nc/Nmf+sViVTBI8kq4++K0XLif7QwSI2+edXJtlqM5WpC6gHt61V2w27W3rYhxkioWM2dlEz+OxSiqq0ru+FBylhPzMAYSta97s3q3b7GTySbJj2wpIGsmX2Ul2A/r1mu4LP1A== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(376002)(346002)(396003)(136003)(366004)(39860400002)(86362001)(83380400001)(26005)(6666004)(6506007)(186003)(2906002)(41300700001)(38100700002)(6512007)(6916009)(316002)(5660300002)(1076003)(6486002)(966005)(54906003)(4326008)(2616005)(8676002)(107886003)(66476007)(8936002)(66556008)(66946007)(478600001)(36756003)(7416002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: UpWjfcVvshv/oLREmResnbZQt/lvDswYKI6X24t3fA3HI1eJeIoOGf1d22a1OI4iXXeao1l0qIlcPYTLX2BPwydjfpacacCUXUMoAdnzvmtCvFbKg623Pfrqij5yzi1NOLSEfTIK/M/pH+KlonlnKwAslJTtE/Q3EWJQmoSZUIXp+Mg/uuPgQ9tRFVhdp0JB/QUkBGFehVAI8H7cEpk8F76v9Aj6cXIUDQQAkGOzGbphYdxg3l5ik9L4IH3OWvI3ylrOEFax1uHsDeor6OIFxbG+b/IfJCOZXlCpoNZjOo0nryGT8zwaLOOGPVNnrwSkovKEnbSszHSMT3e5s+IepLh/GQLZuqMUHjrH8QzeTfs5OKrUXB2lZE9yCDqlvmAK0A2J0dKaI/kwETZJuNuwGltZ51PhVr2Ts7KaUef8mG6pPzRoT6zyr4xiyGUoctbUfSXvZu93JXejp0BeqWx3cpdavaqNuvNcQ8dIom3xUqzuojrj8ku0GscvnKKe85LsWSaKRv9JE0X4dOL6pLKr4Yw7guv3xb21WglwVw5Bx0D6johhHhLWM0AtuZk27jnjaLqFXtQg39s+U/gEN1CbVxX/mQ9o+QKNPS8vr3muSxaWAYSewgA0pYhNblInCSmJBhrx54/1dQvy3MeDK15GbKbwR9J8B+QTFO6/rmpUAGOT3jdFWVDxPXFdEUBFo8riSZFTysg8ZIp6I3xA4HWnsgI7to3FYZQ1SRnLuhZVGUuiG5ooi4iaNB4QDA+9+ppdOXgUpR8UaOy70+Z8+eFMa/y1gk2yBC1974Um/cIuB5pDiHdWgpI/qQVDZViKMd1Jw8N1HIipORTAp0OGi1IQh+1IbIKAWFM6e/IEdzKZrXk/Y4YDnDNJoS2McTd1pwyaLS+CiG3Ajl/TeuOXASE0nloKW2IVwuxejZp0hR/9zIrW6A5V52yMSB5vDDiTRcqA4BgQmqgV7E5tOUZglvljXqKInNv8SSxtY8Et5F5AdvsS7qJpZBtHBEy8/RKwMce0Pr2MAzgIiHml3OXSZ8twSEbyqIpR71msuiVcYhTKrRwXfj/RE0Y0weZCaEZIF7tkglfHsujzOe0LzMtiIH7GDLb83FlkywGThMR7wKZbWbOS52jF/xxAoFpQGCJCmNqJY5mJh26g1kz0ILpuanjVxBUwYh75bd4ab3C5uRZadRLQbdm1RLXDkpExjj7LuX4wgCaEtEmQwe3FEh0Uc8Bt6zFJNe/6ARYcxmu+ucPvIYA0Q4bpVqAqkKpVuKKhLrNQyYm4T+9XeyjcgeKpF9yVd895FKcC42r+BYFC562G+GlohrxPJHj7m3u9+jfTjM7lZZOnN8WBYIzFTYMbdnMiawZTm2DJCgHHtFFDRe71Zv2ZxWXAmnoL7FLDdZEMCH67spr6XD6ijm9Iu0PQtPkAXMzpm/mKLd73/s361TXWWXJfHE/IP2I1RLzWJLO2ZxyZiD1+aso7YwrI/uj3vQX03ZkZGgm0nVI8CX6A2HPk6Uznu69H4NAWXse7FuFYjhGxjfxki0lZg4mfcQaCRUkmxq4TF/2GxbQkgQ6qjp9kZbj4nvD0phi/zGCNhlfcRT2T X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3a7411ce-1d32-4a16-6fb1-08da75c89f04 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Aug 2022 03:22:54.9998 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: WikvfDg7JXiKWJHwr+Hv2eYJaSYQgWlOV93gqvpjfKLfRu9tapOzgmVumLCnCMAXsi1MigpnRhv5Vb1XhQF8FQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4988 ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659583380; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=Ie3E69NNFimjNpTukhM9g1+eptQKwtizqSs9Kze3k04=; b=fMArho9/7TWaPUIXNN0zFzpIP9YFMx0DzihGOxvl6d71XHpZs2LrYAkxA7wRD4ni+btCBv 7pOFJ5X73P67HPeGEgNUgoHRfsoziFhnVmjxoZbz3ReOPyPHi+CkPXpcFDhpyqY+IX3XBA WJhgDYbin7XMj7LCxy2iizsxIL3MzpQ= ARC-Authentication-Results: i=2; imf26.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=P3bNY3lR; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf26.hostedemail.com: domain of apopple@nvidia.com designates 40.107.94.69 as permitted sender) smtp.mailfrom=apopple@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1659583380; a=rsa-sha256; cv=pass; b=nkZyrguZ2ljfoAUSRSLBqD3mLztpwK1k9XS9C9fG5rLehiM+GROBDmHb8uTYMk+uKjMDvO sXn0mfSnct3U65fRWSeFbIO+CxCzSpyqzgIYDay9naHJ/DDfSsU3B9wN4a/6Ey5hBQUdVa iqI025eyM+ZKdv/qKRphtNc9Kn+ffTw= Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=P3bNY3lR; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf26.hostedemail.com: domain of apopple@nvidia.com designates 40.107.94.69 as permitted sender) smtp.mailfrom=apopple@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 1CB74140039 X-Stat-Signature: owwdwyfaa6q8jknp9kjf546pk8hs34q9 X-Rspam-User: X-HE-Tag: 1659583377-658924 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When pinning pages with FOLL_LONGTERM check_and_migrate_movable_pages() is called to migrate pages out of zones which should not contain any longterm pinned pages. When migration succeeds all pages will have been unpinned so pinning needs to be retried. Migration can also fail, in which case the pages will also have been unpinned but the operation should not be retried. If all pages are in the correct zone nothing will be unpinned and no retry is required. The logic in check_and_migrate_movable_pages() tracks unnecessary state and the return codes for each case are difficult to follow. Refactor the code to clean this up. No behaviour change is intended. Signed-off-by: Alistair Popple Cc: "Sierra Guiza, Alejandro (Alex)" Cc: Chaitanya Kulkarni Cc: Dan Williams Cc: Felix Kuehling Cc: Jason Gunthorpe Cc: John Hubbard Cc: Logan Gunthorpe Cc: Miaohe Lin Cc: Muchun Song Cc: Pasha Tatashin Cc: Ralph Campbell --- Originally posted as "mm/gup.c: Simplify and fix check_and_migrate_movable_pages() return codes"[1]. Changes from that version: - Restore the original isolation failure behaviour and don't fail the pup. Instead retry indefinitely. - Unpin all pages on retry or failure rather than just failure. Jason - I dropped your Reviewed-by. I had to remove the changes to make error handling follow convention as we need to always unpin the pages. We also need the list_empty() checks because we may or may not have pages in the list if we found coherent pages. So there isn't much I could see to simplify, but let me know if you spot some. [1] https://lore.kernel.org/linux-mm/814dee5d3aadd38c3370eaaf438ba7eee9bf9d2b.1659399696.git-series.apopple@nvidia.com/ --- mm/gup.c | 61 +++++++++++++++++++++++++------------------------------- 1 file changed, 27 insertions(+), 34 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index c6d060dee9e0..5ce0106a92f8 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1901,20 +1901,20 @@ struct page *get_dump_page(unsigned long addr) #ifdef CONFIG_MIGRATION /* - * Check whether all pages are pinnable, if so return number of pages. If some - * pages are not pinnable, migrate them, and unpin all pages. Return zero if - * pages were migrated, or if some pages were not successfully isolated. - * Return negative error if migration fails. + * Check whether all pages are pinnable. If some pages are not pinnable migrate + * them and unpin all the pages. Returns -EAGAIN if pages were unpinned or zero + * if all pages are pinnable and in the right zone. Other errors indicate + * migration failure. */ static long check_and_migrate_movable_pages(unsigned long nr_pages, struct page **pages, unsigned int gup_flags) { - unsigned long isolation_error_count = 0, i; + unsigned long i; struct folio *prev_folio = NULL; LIST_HEAD(movable_page_list); bool drain_allow = true, coherent_pages = false; - int ret = 0; + int ret = -EAGAIN; for (i = 0; i < nr_pages; i++) { struct folio *folio = page_folio(pages[i]); @@ -1946,10 +1946,10 @@ static long check_and_migrate_movable_pages(unsigned long nr_pages, unpin_user_page(&folio->page); } - ret = migrate_device_coherent_page(&folio->page); - if (ret) - goto unpin_pages; - + if (migrate_device_coherent_page(&folio->page)) { + ret = -EBUSY; + break; + } continue; } @@ -1959,9 +1959,7 @@ static long check_and_migrate_movable_pages(unsigned long nr_pages, * Try to move out any movable page before pinning the range. */ if (folio_test_hugetlb(folio)) { - if (isolate_hugetlb(&folio->page, - &movable_page_list)) - isolation_error_count++; + isolate_hugetlb(&folio->page, &movable_page_list); continue; } @@ -1970,29 +1968,25 @@ static long check_and_migrate_movable_pages(unsigned long nr_pages, drain_allow = false; } - if (folio_isolate_lru(folio)) { - isolation_error_count++; + if (folio_isolate_lru(folio)) continue; - } + list_add_tail(&folio->lru, &movable_page_list); node_stat_mod_folio(folio, NR_ISOLATED_ANON + folio_is_file_lru(folio), folio_nr_pages(folio)); } - if (!list_empty(&movable_page_list) || isolation_error_count || - coherent_pages) - goto unpin_pages; - /* - * If list is empty, and no isolation errors, means that all pages are - * in the correct zone. + * All pages are still pinned and in the correct zone. */ - return nr_pages; + if (list_empty(&movable_page_list) && !coherent_pages) + return 0; -unpin_pages: /* - * pages[i] might be NULL if any device coherent pages were found. + * Unpin all pages. If device coherent pages were found + * migrate_deivce_coherent_page() will have already dropped the pin and + * set pages[i] == NULL. */ for (i = 0; i < nr_pages; i++) { if (!pages[i]) @@ -2010,15 +2004,14 @@ static long check_and_migrate_movable_pages(unsigned long nr_pages, .gfp_mask = GFP_USER | __GFP_NOWARN, }; - ret = migrate_pages(&movable_page_list, alloc_migration_target, - NULL, (unsigned long)&mtc, MIGRATE_SYNC, - MR_LONGTERM_PIN, NULL); - if (ret > 0) /* number of pages not migrated */ + if (migrate_pages(&movable_page_list, alloc_migration_target, + NULL, (unsigned long)&mtc, MIGRATE_SYNC, + MR_LONGTERM_PIN, NULL)) ret = -ENOMEM; - } - if (ret && !list_empty(&movable_page_list)) putback_movable_pages(&movable_page_list); + } + return ret; } #else @@ -2026,7 +2019,7 @@ static long check_and_migrate_movable_pages(unsigned long nr_pages, struct page **pages, unsigned int gup_flags) { - return nr_pages; + return 0; } #endif /* CONFIG_MIGRATION */ @@ -2054,10 +2047,10 @@ static long __gup_longterm_locked(struct mm_struct *mm, if (rc <= 0) break; rc = check_and_migrate_movable_pages(rc, pages, gup_flags); - } while (!rc); + } while (rc == -EAGAIN); memalloc_pin_restore(flags); - return rc; + return rc ? rc : nr_pages; } static bool is_valid_gup_flags(unsigned int gup_flags)