From patchwork Fri Jun 9 01:17:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273072 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C917C7EE29 for ; Fri, 9 Jun 2023 01:17:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E77A58E0002; Thu, 8 Jun 2023 21:17:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E27A18E0001; Thu, 8 Jun 2023 21:17:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC8048E0002; Thu, 8 Jun 2023 21:17:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BC8268E0001 for ; Thu, 8 Jun 2023 21:17:33 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 945591202B7 for ; Fri, 9 Jun 2023 01:17:33 +0000 (UTC) X-FDA: 80881446786.12.3599FDA Received: from mail-yw1-f170.google.com (mail-yw1-f170.google.com [209.85.128.170]) by imf05.hostedemail.com (Postfix) with ESMTP id C8C11100005 for ; Fri, 9 Jun 2023 01:17:31 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="hfFZS/To"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of hughd@google.com designates 209.85.128.170 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686273451; a=rsa-sha256; cv=none; b=XWOmGOh2bp1XzYCtB7Ah1hx6iiZpD28lalwbQnhWRdX5pFNF9y5P44ZAyQ57zD0X0s9Fl7 FTOD3sRfqZDHtHiMxtxrUjfbLfbF6XUv8s5L5WqOME3pwoDZbpAY1gKKY78sdsNfOlkTU1 2Y3mZLkB7BqcRAFZbt0LAti9FcVZ4Lw= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="hfFZS/To"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of hughd@google.com designates 209.85.128.170 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686273451; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YGxXR2sylNgXOWjswYon8LmdFOYpbwThTVM1QZPxixo=; b=IxgM6bSP3cZQSF3DryuNK9BFuiv3/zs6Chrp3+ZD0IgyjlaeV4jsLnCt7ZMzJBUEzBjy1V 4SNgirXvuBIzXG0ewVWqXC91hHVfGoYB91J0isEkuAkujkvL2nKdEpvdAuGLYjRA78jMCH ygbN6DI10H6agw3IXOgV0LiGEEC8Ht8= Received: by mail-yw1-f170.google.com with SMTP id 00721157ae682-569fc874498so12103617b3.1 for ; Thu, 08 Jun 2023 18:17:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686273451; x=1688865451; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=YGxXR2sylNgXOWjswYon8LmdFOYpbwThTVM1QZPxixo=; b=hfFZS/To38y/kHK62NcQe3NeUtPQadIbTbMGv9CwCwnaOm3+ubft1ovMBDYwgIcJHN maH31bmCKIXTcoH2mw+7AlwbT//z013fnJricqAgo1Y8wR2cLWz5jidfA/WWWdNn++DD eExfA1ytyJkqY/La00m6AnHJyexbZrQSpOFK5OQ6AFOk6cyYk4BIHtO7IYbubUELYE5/ t2x0w817NAACm7JQwyi+uQozGB37VzsQ0dIb5qTadGN0KMsSpWs2GUyHK1Uhm4UV4EVa KWOR3EiG1vfrWrdRdz5JYnJcT4eEEx83Gbf/GYzP39cr1/xlfqJ/0ZZkApmwpKDFSRuJ KEIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686273451; x=1688865451; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YGxXR2sylNgXOWjswYon8LmdFOYpbwThTVM1QZPxixo=; b=JNmJ1X1Do0dRHkA4wR1KB2HoFUlmqsnkpu4iQDLIrl0uTsx/ZW/vtwR2PJuXvV92MC ChGHf1RmFlnxv4CIO4JqW+jai5+yutE5j2w2Dhi9rjDQr6rXvo4NdKnaBrfvhbwZ0n1J QHuXlUZFQ/7kg33YlxML5QBtJLQGTKx4kDwIyop58Om+MU8uJxqg096lgOkvZho2ELmO yuXjdjwb3MGoXpuuNZBUrVs6oFTgNTnniJDVvqhjRD+josWb8uZrQ2QtZiI7ODRTGrfz zpGxd9gPVgpULGothVvZVafSaavIFRIzH1UPsLo8aZyZJFQgUmYFUS5F9nK1Hs01EueJ P8PQ== X-Gm-Message-State: AC+VfDyNaf0kxdvlSpfk/zM5WY2FNSgjfVtOSFGNrh/2V0i9s2u5ET8q zfQGYHXD1m8sdsheTe89i/gpxQ== X-Google-Smtp-Source: ACHHUZ684nq9AHXRq1UVfeo4n1qoOEQQWARDc3yoeKJKitFHVG0vbG1oJERSIO9SpyMcUhVMBrYK3g== X-Received: by 2002:a81:6cd5:0:b0:564:c4db:631e with SMTP id h204-20020a816cd5000000b00564c4db631emr1257858ywc.5.1686273450756; Thu, 08 Jun 2023 18:17:30 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id p131-20020a817489000000b00560beb1c97bsm287394ywc.97.2023.06.08.18.17.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:17:30 -0700 (PDT) Date: Thu, 8 Jun 2023 18:17:26 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 09/32] mm/pagewalkers: ACTION_AGAIN if pte_offset_map_lock() fails In-Reply-To: Message-ID: References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: C8C11100005 X-Stat-Signature: wiz7ipamtzjqjda73gjszgdhck8c8t3y X-HE-Tag: 1686273451-999198 X-HE-Meta: U2FsdGVkX19TLZKvp340vPcYg0NdUjofadvbGjmiGsEN7F7xMfSYo3gxk2FlOalYJUsJAJgFnHGuoyTX0+KH5iBZntu7CPhTw+luFfi0BCrCZqDHZKe5nmmiT0JKgIKkPkk0XJV6Neg9v33DRrJti62jmrzhDBDoD7eIxXOGU5SqIYPES29Da8rUgFSJcaD8XKROk8HjEvpAq7iLY3q94LjEMyCubj+MVYUWEy05JA7SKKPjwfDAutlCdXA+eTY6aChQa1gwZRK5BeHVmIzLuc9c/470jXGj0Fvm4X6+4ZQEDeojvxU4yVqpjQcNVUA/Ws3acZTKDLGALv4TD0G2HrqKCP0YFfO7hhaSKLeGzCMiW4EfUf5aX93HpgLM6oPpp8Vwm+4zg2i7Otlup32fYNHm+dZfAAAS1GCEWDt48LgW0MAH2/fELweYTpYio+4TenzqRmjfuKHQnxqVwxDKMWik0ivCtVk/z+8cZrSzmtnxW9+cwIutA96ENPJwPDyRyRbaRCATTs/hHpVRve6ikHblO8oVJQxUqA8RnCEuL3Bvc6UrxhcXUK9OZSOAhPUJoVEuWmQzUierTnH84NuOAQx3T0tq0RNKMwzjwP1/lfXCkru6PpfOK0UREZQTgrpaeiU6qNAIVpv/lwFwkXONJ8Y1isxxDQw21cm9/1RYZhuEirb4sVa3N7lm4bRq0mxU9tJ5qx6e8A9ww6WV1BJkQX44HqCjDXHUbkCQV/aCqGHFyIWd8rdIZWhr0FKNnBqMMAFQBXQ3WODRJ7KJz2ii26wKKc3S8Ex24NRM65cYAboOga3uFjdkpvWtsvzXMSUlxD9IVH83yggqlf3YncUxkfFkJQqVWcqiS/TYkIQj6U3ZbDOcd5weBPWhA//clutRVpWHjVPH8a1pH8BWnUJuSCa3aylqVoIdeQqWb4s8w3a6g+H2OroE4HRSABrYrZ8SNqsDPW2Kcz2d/Vlgvja 71xtePYr r96j48gFm8IhyLaQldt2Y+5ZO0hqbzBGpLzUQqxU3/KnbDHbOM6U/rdp4Jus7Sbp1GQZk3+o3EJZGbaW41ZHfjGIria/WnlCRW/suXQZlnvkGhELepRIbu0Tt91L0k/szMTnpWWIPbxjG5e1xs9bNLgVwTAytZm5D4c72/TU//brYfirS0HmGESeca9VBN64OTSgwlD9fRJzrRcGh7JF5MYm1Uh8T4KZcoxZ9ztjsmIlaiz0Pt/92Q6XSQnK1kREgaDfFy4pDpHUFR4TNhLDoLtTpj1lHSQqZD7VxxJzI7mtTdrV8rhfzLMsv5C9SXIOcKO/KbZPaTOcy4j697bQjOjCDWZDCeSbXUO2AcgW8wUtEXq69141Tq87mFZahNtC0SsmhvtLvc9kn1A96BQ2OH05jBBUX0joDLmM2kwapjm6t/b18whyt0vnPM+Ux/t26tna7CU4iZiZy8T2SLWQhINpM5xTjDOHmPi24fl/s15cYH7E= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Simple walk_page_range() users should set ACTION_AGAIN to retry when pte_offset_map_lock() fails. No need to check pmd_trans_unstable(): that was precisely to avoid the possiblity of calling pte_offset_map() on a racily removed or inserted THP entry, but such cases are now safely handled inside it. Likewise there is no need to check pmd_none() or pmd_bad() before calling it. Signed-off-by: Hugh Dickins Reviewed-by: SeongJae Park for mm/damon part --- fs/proc/task_mmu.c | 32 ++++++++++++++++---------------- mm/damon/vaddr.c | 12 ++++++++---- mm/mempolicy.c | 7 ++++--- mm/mincore.c | 9 ++++----- mm/mlock.c | 4 ++++ 5 files changed, 36 insertions(+), 28 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 420510f6a545..dba5052ce09b 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -631,14 +631,11 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, goto out; } - if (pmd_trans_unstable(pmd)) - goto out; - /* - * The mmap_lock held all the way back in m_start() is what - * keeps khugepaged out of here and from collapsing things - * in here. - */ pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } for (; addr != end; pte++, addr += PAGE_SIZE) smaps_pte_entry(pte, addr, walk); pte_unmap_unlock(pte - 1, ptl); @@ -1191,10 +1188,11 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr, return 0; } - if (pmd_trans_unstable(pmd)) - return 0; - pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } for (; addr != end; pte++, addr += PAGE_SIZE) { ptent = *pte; @@ -1538,9 +1536,6 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, spin_unlock(ptl); return err; } - - if (pmd_trans_unstable(pmdp)) - return 0; #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ /* @@ -1548,6 +1543,10 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, * goes beyond vma->vm_end. */ orig_pte = pte = pte_offset_map_lock(walk->mm, pmdp, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return err; + } for (; addr < end; pte++, addr += PAGE_SIZE) { pagemap_entry_t pme; @@ -1887,11 +1886,12 @@ static int gather_pte_stats(pmd_t *pmd, unsigned long addr, spin_unlock(ptl); return 0; } - - if (pmd_trans_unstable(pmd)) - return 0; #endif orig_pte = pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } do { struct page *page = can_gather_numa_stats(*pte, vma, addr); if (!page) diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index 1fec16d7263e..b8762ff15c3c 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -318,9 +318,11 @@ static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned long addr, spin_unlock(ptl); } - if (pmd_none(*pmd) || unlikely(pmd_bad(*pmd))) - return 0; pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } if (!pte_present(*pte)) goto out; damon_ptep_mkold(pte, walk->mm, addr); @@ -464,9 +466,11 @@ static int damon_young_pmd_entry(pmd_t *pmd, unsigned long addr, regular_page: #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ - if (pmd_none(*pmd) || unlikely(pmd_bad(*pmd))) - return -EINVAL; pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } if (!pte_present(*pte)) goto out; folio = damon_get_folio(pte_pfn(*pte)); diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 1756389a0609..4d0bcf6f0d52 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -514,10 +514,11 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, if (ptl) return queue_folios_pmd(pmd, ptl, addr, end, walk); - if (pmd_trans_unstable(pmd)) - return 0; - mapped_pte = pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } for (; addr != end; pte++, addr += PAGE_SIZE) { if (!pte_present(*pte)) continue; diff --git a/mm/mincore.c b/mm/mincore.c index 2d5be013a25a..f33f6a0b1ded 100644 --- a/mm/mincore.c +++ b/mm/mincore.c @@ -113,12 +113,11 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, goto out; } - if (pmd_trans_unstable(pmd)) { - __mincore_unmapped_range(addr, end, vma, vec); - goto out; - } - ptep = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!ptep) { + walk->action = ACTION_AGAIN; + return 0; + } for (; addr != end; ptep++, addr += PAGE_SIZE) { pte_t pte = *ptep; diff --git a/mm/mlock.c b/mm/mlock.c index 40b43f8740df..9f2b1173b1b1 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -329,6 +329,10 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, } start_pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!start_pte) { + walk->action = ACTION_AGAIN; + return 0; + } for (pte = start_pte; addr != end; pte++, addr += PAGE_SIZE) { if (!pte_present(*pte)) continue;