From patchwork Mon May 22 05:00:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86717C77B73 for ; Mon, 22 May 2023 05:00:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 283FD90000A; Mon, 22 May 2023 01:00:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2345A900002; Mon, 22 May 2023 01:00:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D57E90000A; Mon, 22 May 2023 01:00:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id F13EA900002 for ; Mon, 22 May 2023 01:00:22 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id BBEAEC045B for ; Mon, 22 May 2023 05:00:22 +0000 (UTC) X-FDA: 80816689884.08.117DEC1 Received: from mail-yw1-f180.google.com (mail-yw1-f180.google.com [209.85.128.180]) by imf20.hostedemail.com (Postfix) with ESMTP id 6F6151C0021 for ; Mon, 22 May 2023 05:00:20 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=gG5CM9oz; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of hughd@google.com designates 209.85.128.180 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684731620; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UuVwG0QveaZLADVtUaHSgqKzTTEqSjST+aR027Vb5lo=; b=nFuug4z8RtzYiwfH+Fw3I5A2S/Y2J/puSTkzahZIG7nY1cYsIgljDWLb6mRqKE0vQy7KPg k0mQX1mGlmKgN93KpMuMVTJbEMLK2rR07lHwgkz8zQrL165VBT1p3bELnk7nyzp89r1pMo wglxqvtI5Ra01Du/HeLIyOzcxhO0S/o= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=gG5CM9oz; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of hughd@google.com designates 209.85.128.180 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684731620; a=rsa-sha256; cv=none; b=3cRJB8QSQ2gG9Qbyecy/M5Lfob2OjiT+CCOCzRHuNVpr0cR2n747z6a/xOICndIoSygFN9 +LaZSWQzk7V4HamTpGoo2L2wJb3sN5uUj2/s53uOrva7kk4KhDPLQaAj2APlt25hZs6cxQ JwJbPkRBCvoSzFXtM3qp8qBfs/gZOEc= Received: by mail-yw1-f180.google.com with SMTP id 00721157ae682-561c5b5e534so74214067b3.2 for ; Sun, 21 May 2023 22:00:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684731619; x=1687323619; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=UuVwG0QveaZLADVtUaHSgqKzTTEqSjST+aR027Vb5lo=; b=gG5CM9oziEo3d2xuTQZP2LZGLbyhx6n0Gd07VVIcPAplxjY92fK4ZBhk/PKkZAtjDN 5xbQCSYDeQd+vsPz3UctEBNNQiAW1PG9SnA0aX9yoaPh6erzg7iRa3/rTzxeaqzGLp7M lSM/wuLv0LneLKuxocS16HoYAdzNdVswju/csw1C62EZM/zSg7SkkxlvZsPb/qR4YDiI JxhcXWwQQZon4YC+DZWlfAheKEVu+DJEgm81Ja53+XH3TYiVcdE+XisUHVOIOH3c+bhB 84wXaTSFtqGDpcIfa2zJuBidBN4AfMzBHzdLZY7lpccTQVYK/Jpluc7HpY0MDlN7jzqs jXCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684731619; x=1687323619; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UuVwG0QveaZLADVtUaHSgqKzTTEqSjST+aR027Vb5lo=; b=gaZ5GswZW38atTz6uHarHkuwJZH0cPmySeYk3RcSiCsW4ckJMjTBNHPsFoIPbgPV38 Wi8I/S6Bry3tgnQdYrUxxPzgTkxEnGo3xz+sBsWd17iOm+pfAcG2B5FXU4ItmDtl6oMX bxMFbBpk54CGQmtjHTGvVOK6R/77YAeRKm7KxU38bMQ3C2E+XEONYnnlzkBgt+W9OtOV Oww3Z2gQ3mi4TttoO6AAIsit9zEcHCtpqqwS2m2RYiKF1dhMdbV4QDCUlyzJmb1WH8Ju XAenoYhqgHLAxU+Y9WLH0DkN5nym5yZnUXVDgDWNICYkv7xsdPbPSUPTlTJS8A/zi8ZK ws5g== X-Gm-Message-State: AC+VfDzEWKB/g9PTD1AXyiDuTUouBOoWzaqEqH5T78uvEtSd1ggXe/3V A6ERdLJphEwZCyG6AE6Beh8Spg== X-Google-Smtp-Source: ACHHUZ7OcXDmRntzMQt9n+hv5KPWL1NcuSYc2shkgSn7zkVgG1IKDeMbUgq8Rk9jU0J7GbmzBFEDoA== X-Received: by 2002:a81:6d09:0:b0:561:902e:dc0a with SMTP id i9-20020a816d09000000b00561902edc0amr9837501ywc.32.1684731619134; Sun, 21 May 2023 22:00:19 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id r63-20020a815d42000000b00555df877a4csm1794565ywb.102.2023.05.21.22.00.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:00:18 -0700 (PDT) Date: Sun, 21 May 2023 22:00:15 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 09/31] mm/pagewalkers: ACTION_AGAIN if pte_offset_map_lock() fails In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: <6265ac58-6018-a8c6-cf38-69cba698471@google.com> References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 6F6151C0021 X-Stat-Signature: tcg5gnptxnqtja9icmrf569r9troji8g X-Rspam-User: X-HE-Tag: 1684731620-213481 X-HE-Meta: U2FsdGVkX1++eZlvW2jAl/4dyQu0okHUrBM/PxMS8W5W0yqiYO8hW/i+7mB7JfL9zyGe+sxlRD6cJJFvUSqNFiZlxqurnJ4616RPmWzLGrFfDgZznTPWiSrvX3vnx3fY70PeK4azPVNGLwpY90A3SO/aTwiYCLwbG/5FabBzJWM8mzKYgWozVyA9WBC4f2uh5lBlzxdPUr+0L0A/eiSm+FqddyvRjBTsgo2wkRLtbIdHwgP/ZfhAuPPKxodu/jUYg8ruwFK6tkdT0W68+G8yLs/k6GpoFG1v1JZ1qe8A8ybIHoxRKxLQxQHibnRQVUGTN9bwvGC6jOdhKY0bgxxIz5GItcbdqoid80WZ82DD2ZqgOUw31zB3jT6LvWfRYIj0/W3KrKKQjS1oHaD77iS+G472JHW8+tcQeoVLQ1h3hx1Z+XEiEvyiN4zNk7dtWwOirFUQQSvffnS10EtEWPUGyoPoIZyj5pbscN7SLgZYQ4kvDtO+YMxzzRWchDmIvmjCovYxDfrnzU1p95TXOnEy6rDO8W66gUpKo/t3Cs38q0iFTEbCseimg0ZJ9GOgBbm67+sS2C8gnSBXxOX+mb1qcmVUaQkBCHqgJZhDKQCoA/1qcHEnoDV2xKy0bbxN9vOfoa0IL0eS+SvpOIYemkPoSjHlTZ9qtK2zZYt+wHPYiQtcwHElbBB/LgUsW3Dt4y0UpnY65PVDrAUGd3t3zGWV5aRByYAQuRPwTIQiWMxE9x/OKEahBbPxFO+lLNWskyxNM0kjCqugwXF+TqJ1l9VnqMSkjpHbZKNgqOgpUKvH6AJ9W1M0ocOVdxTXWR4bmyY4blF/VXg2l7PNntdVczAUSHylNL3uhZGaP/werI1JEm8tIDe78Gxlu/aqkf0Lo7o1Hm3V0pH0Ge9f1vzJNvv3kdVP1LDsaZcyNaKi37DUjrdxvLXIFJeHu/urffmTDxzQA6Vha68FaCWRvrnA1GB dqUhFNVd 6vEU/RhOy3yiW6VBaql9xuPM1MKdW6L0cMe6Q15iZvXILIv3VcHo2Axz7fvStwmOPV/zptyEcIkgKC+Go+++OSnX7NEhG8xmx8Jl5ysS/5Y6Z3E7/suHbd5FynJZTVvQtEVfFsUR756TqbaVeHqHazK1JaqQQmCCZa8G29S9BPI+phE9/E7/9xOufguyyRmrTDKBjOQA1OsGaVhi97zD0TjGp3cWjoOdeJcOtLGdgVGlVsafsTm6M1mmnXJoAcZm1vH0t1AY6kaGDqnV42jSTLhG8uAuUFMWporh22Y3sAkBfQM2uxm2dcjFQCu9iXXK7iDpDg0wP3Z5boMDvhKxitr+acBVUlLFn8M3chxT/qs+ZhtNzMPpyc+35BDoo8CwXeFcA6RUvGM1UBTMHnIYySstRZEg70tGwQ+B98cPOzZZer5BCSHCGCRF/fB7csYjUBEFNyXY8dtYEfMZjRssp11Zkqw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Simple walk_page_range() users should set ACTION_AGAIN to retry when pte_offset_map_lock() fails. No need to check pmd_trans_unstable(): that was precisely to avoid the possiblity of calling pte_offset_map() on a racily removed or inserted THP entry, but such cases are now safely handled inside it. Likewise there is no need to check pmd_none() or pmd_bad() before calling it. Signed-off-by: Hugh Dickins Reviewed-by: SeongJae Park --- fs/proc/task_mmu.c | 32 ++++++++++++++++---------------- mm/damon/vaddr.c | 12 ++++++++---- mm/mempolicy.c | 7 ++++--- mm/mincore.c | 9 ++++----- mm/mlock.c | 4 ++++ 5 files changed, 36 insertions(+), 28 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 420510f6a545..dba5052ce09b 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -631,14 +631,11 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, goto out; } - if (pmd_trans_unstable(pmd)) - goto out; - /* - * The mmap_lock held all the way back in m_start() is what - * keeps khugepaged out of here and from collapsing things - * in here. - */ pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } for (; addr != end; pte++, addr += PAGE_SIZE) smaps_pte_entry(pte, addr, walk); pte_unmap_unlock(pte - 1, ptl); @@ -1191,10 +1188,11 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr, return 0; } - if (pmd_trans_unstable(pmd)) - return 0; - pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } for (; addr != end; pte++, addr += PAGE_SIZE) { ptent = *pte; @@ -1538,9 +1536,6 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, spin_unlock(ptl); return err; } - - if (pmd_trans_unstable(pmdp)) - return 0; #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ /* @@ -1548,6 +1543,10 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, * goes beyond vma->vm_end. */ orig_pte = pte = pte_offset_map_lock(walk->mm, pmdp, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return err; + } for (; addr < end; pte++, addr += PAGE_SIZE) { pagemap_entry_t pme; @@ -1887,11 +1886,12 @@ static int gather_pte_stats(pmd_t *pmd, unsigned long addr, spin_unlock(ptl); return 0; } - - if (pmd_trans_unstable(pmd)) - return 0; #endif orig_pte = pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } do { struct page *page = can_gather_numa_stats(*pte, vma, addr); if (!page) diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index 1fec16d7263e..b8762ff15c3c 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -318,9 +318,11 @@ static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned long addr, spin_unlock(ptl); } - if (pmd_none(*pmd) || unlikely(pmd_bad(*pmd))) - return 0; pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } if (!pte_present(*pte)) goto out; damon_ptep_mkold(pte, walk->mm, addr); @@ -464,9 +466,11 @@ static int damon_young_pmd_entry(pmd_t *pmd, unsigned long addr, regular_page: #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ - if (pmd_none(*pmd) || unlikely(pmd_bad(*pmd))) - return -EINVAL; pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } if (!pte_present(*pte)) goto out; folio = damon_get_folio(pte_pfn(*pte)); diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 1756389a0609..4d0bcf6f0d52 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -514,10 +514,11 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, if (ptl) return queue_folios_pmd(pmd, ptl, addr, end, walk); - if (pmd_trans_unstable(pmd)) - return 0; - mapped_pte = pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } for (; addr != end; pte++, addr += PAGE_SIZE) { if (!pte_present(*pte)) continue; diff --git a/mm/mincore.c b/mm/mincore.c index 2d5be013a25a..f33f6a0b1ded 100644 --- a/mm/mincore.c +++ b/mm/mincore.c @@ -113,12 +113,11 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, goto out; } - if (pmd_trans_unstable(pmd)) { - __mincore_unmapped_range(addr, end, vma, vec); - goto out; - } - ptep = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!ptep) { + walk->action = ACTION_AGAIN; + return 0; + } for (; addr != end; ptep++, addr += PAGE_SIZE) { pte_t pte = *ptep; diff --git a/mm/mlock.c b/mm/mlock.c index 40b43f8740df..9f2b1173b1b1 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -329,6 +329,10 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, } start_pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!start_pte) { + walk->action = ACTION_AGAIN; + return 0; + } for (pte = start_pte; addr != end; pte++, addr += PAGE_SIZE) { if (!pte_present(*pte)) continue;