From patchwork Fri Jun 9 01:15:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273071 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6688AC7EE25 for ; Fri, 9 Jun 2023 01:15:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F3B2C8E0002; Thu, 8 Jun 2023 21:15:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F12248E0001; Thu, 8 Jun 2023 21:15:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DDB5B8E0002; Thu, 8 Jun 2023 21:15:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id CEC988E0001 for ; Thu, 8 Jun 2023 21:15:51 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id CBF13C031B for ; Fri, 9 Jun 2023 01:15:50 +0000 (UTC) X-FDA: 80881442460.10.C471257 Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) by imf23.hostedemail.com (Postfix) with ESMTP id 037E0140015 for ; Fri, 9 Jun 2023 01:15:48 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=qGqUUKrU; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of hughd@google.com designates 209.85.160.179 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686273349; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9VYOFAJpcVHpYrrS1TQZcIVdYEqSjI0GrwCgA5USRTU=; b=CrQtUuwJaQedCm9e1Rw4oLBWnzISNEOSTAK82qtFTXUSfUTc+I/MDh06JUvlm+DzLWeHqf IWzUTocw9fRmLwT8eCXTEkvWVSC5oksfW2td2kuKVFw6jRRPW4FQxCSeOh/Ox+R2CAHRyx SGG8JYLtHP8dhq3KAq9SiEBZcIjAAuM= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=qGqUUKrU; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of hughd@google.com designates 209.85.160.179 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686273349; a=rsa-sha256; cv=none; b=FDjUN+bai+sHNNunc1tV+ml0IxpLqzoo8a+dXtKHrczrTPjqYmUg9OKvTF+I4Bt4CclqcY O1PY020qHGlsmuXst9F3PqC87aI5cmG09bCoYphAsCcHSYcNfbjE90yJZ4fmDYLgve7ZsI Fenjm6gdiw//0oo4sv9k1Cvb6MDs6gc= Received: by mail-qt1-f179.google.com with SMTP id d75a77b69052e-3f8008cb772so10534141cf.3 for ; Thu, 08 Jun 2023 18:15:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686273348; x=1688865348; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=9VYOFAJpcVHpYrrS1TQZcIVdYEqSjI0GrwCgA5USRTU=; b=qGqUUKrUhXZqdfIpdRpru1enURAlAM37w3JnFCBdqcN5NOVKva/5ZB+Wx1AncWuaoB 4NOGT/sluCqCWe9k46AbMdrfO/y7S/uoPCSIv3YcbyPtfcHQLmMhf87/VSB5iKcTyY2z DvH2xbXgY53bvpfsWKoRD42G3UAVdO/UJ+NkrWhrUDvDf4DfgkGc7vx+86uzLhaB2Vdl OBAdZ3Qvm5QESmYlbn3dnRPJ/xpFtDzek+36j2PNxi4y2bmdWvVJycdWxpou775822Zd vv/wKkp98uy9cbO4mRL/ywca8CgLJugp1BRNkerDGGNXhj+j6x2eNtUbygfDvk9GNFDY 1+3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686273348; x=1688865348; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9VYOFAJpcVHpYrrS1TQZcIVdYEqSjI0GrwCgA5USRTU=; b=UIbx9YWmuHg4tRsZ2CwYCdeRKYhgTs/9NsvcJ7yBwqSQxv8w7+L1QEjOqe4MBOSN24 EAFaSLVIekglHrCqQ4pXWzO/yx5MU8HbJ5eugG+VcJG+sY2+t/fhdngIvmv9j0TKqHSm sKJFCwhEDc1Dhp7FJvO3aNz1HeargOLvTkI0po4c8GvtclCIehAvsJEcex8Iw+ciJl3X ca2dPvOVvczoo7Lsgg7ppKzOEl3mangz6MpL55pJYmk+B3/maywqaWRNzmn2RKBbaYqp gKm3MRfUUo0lXbOC3VmGnXXsIMt1AA6fQPulYArzhKs2slrJHYkeVtnUml8MWR9xk/Sm tJNw== X-Gm-Message-State: AC+VfDwnX77syhohI3nqFyvya5nMXzx7+S71KK3lpI452Yne70m2uDEW VhNIDf/KLRWSW9fDioap6itQfQ== X-Google-Smtp-Source: ACHHUZ5RPLaTpoCJzmmgtuS8+EduaIDi339U1d6YVPQERnp7XZzVYgUh9cqEFrnN8L2JonZ6XbEg1g== X-Received: by 2002:ac8:57d2:0:b0:3f6:9a18:e67c with SMTP id w18-20020ac857d2000000b003f69a18e67cmr42699qta.66.1686273348055; Thu, 08 Jun 2023 18:15:48 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id u15-20020a25840f000000b00b9e2ef25f1asm583095ybk.44.2023.06.08.18.15.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:15:47 -0700 (PDT) Date: Thu, 8 Jun 2023 18:15:43 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 08/32] mm/page_vma_mapped: pte_offset_map_nolock() not pte_lockptr() In-Reply-To: Message-ID: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 037E0140015 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: osfwnknrqjzc11y6n8g7ui7mrmufbpkp X-HE-Tag: 1686273348-207053 X-HE-Meta: U2FsdGVkX1/wEfW3DZdcjFK5HwtB+pAO964SCfjfMZFZH3vRRaPDNU8orwQEYRAfHUEAhIIVcU4FhI5fuwKVuXAjmT1sAe6b+yms8iO9aosHM+Y5lIIQqxal0rk8hi2kvLKKUiGCm2rRWAQdsOlhDSk1BI2IfDTkUMwmES5PugJIQrLewZQiAh/uU3Ezhxj2Rqe1DfTXBADlAuIJfkp+BfzQVB+vjI/IY3cHCkcxNaksjrIPkf2zUQg15tA/SdnTBw8PbYdMAAG2AFHoTAZvABTSMxdHhI0HuKki7jybT4qOq3DmUMXP4nn8T1+A0eR7bDi82Q54lkzUTxdBcKvQksgCb3Ne5EtYOq50fr9BiY2PA5i6Jt42pcGv/lL+zyE7t/vDWo9LRTuDj6JqH8fmkic/mixmvWoXrIpY/txo2REkoTpHNiBLSiIg9rU/xQIa139jjWLApFs73daH5Q6/O2yZYx5AKHIBLeNGuo6/YLbV8AqypD+0Z90oGUXIECtCflLLlCsgRLpitOuivkIncShiFqGytmMPzGWa4K1mAoweWTS7Z3MwFh6NqhKzAcTI69bOUpE8Cin5zzAfbi0MFyX6rcJuVrF02ItMNURvS2giQo7pqEOQPh4BcvN7I0W921oK0ukujJuh5A4zxmLdfnNGC6CfQ6fzY0VjiRLes/HrP1XdYKy/F2U3/DjP6ExF5QKu+qjmzsE6SkEw2WZf3qmJLPnELUgXPGXlrqKjKQueTv/rsfggSg6oFUCxSX4CbVFfXUeY5q4ad6hHtUwobWcyWGeGlPDyVopaB/4Ivc5yuDhUBgtJ6i1ogpLwUBxU/wS1ctTI4UAY8rHskgeW8EE6n0u8XOa06Mpyze/vYCLofi9hIP7/WeYqRA+i0UCIe2QmltsKv48Gi/ie+YBXDbL202jsL7RQYZEry7io4ni3BiBpagfPV7xe3AQkPoabvJXnwlnFxQnMlzmbjeE rDe78KAJ yhODVIBgXjndrQgGu/swy5MAS9Xl9narHtG2Uv0zYGwDWms9TPqhW2ZjY9sVUjsmqecUXdt2iMZb9rfEvWrAWnqRedx2hMeiXBFFL2+Eo5KTlNZlWWN7HGXH9UYge7VKCWHLR80WcUuoQpGkOwR0SJAaAJDH/x72sLSWC85douPKnEwJ3JnUwJKX1kZl2fhLm59ISGUaoDYfV+Jb+PffhSPVbFEOrVXtX/PJP02tDDry5TDf09f397IJG/AOORW9hfvxhIcN70wn8HrOqEIzQfJevP6r1ZrYMn9bj6TjsWM1tRsXZR/wL7iUvJQpAyq4WTZpXce+nUX4sKyr+lgDIMfn/TQPYlRxlvTyEMP02MaJF9fmg14tmXPMB0WwFZL3aIUIjP/0y9rKw+ekZ9Xbmsoay/b2Lpm3T9TSZZyP/3oHtbNWBNUjFQjTNpw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: map_pte() use pte_offset_map_nolock(), to make sure of the ptl belonging to pte, even if pmd entry is then changed racily: page_vma_mapped_walk() use that instead of getting pte_lockptr() later, or restart if map_pte() found no page table. Signed-off-by: Hugh Dickins --- mm/page_vma_mapped.c | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 947dc7491815..2af734274073 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -13,16 +13,28 @@ static inline bool not_found(struct page_vma_mapped_walk *pvmw) return false; } -static bool map_pte(struct page_vma_mapped_walk *pvmw) +static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) { if (pvmw->flags & PVMW_SYNC) { /* Use the stricter lookup */ pvmw->pte = pte_offset_map_lock(pvmw->vma->vm_mm, pvmw->pmd, pvmw->address, &pvmw->ptl); - return true; + *ptlp = pvmw->ptl; + return !!pvmw->pte; } - pvmw->pte = pte_offset_map(pvmw->pmd, pvmw->address); + /* + * It is important to return the ptl corresponding to pte, + * in case *pvmw->pmd changes underneath us; so we need to + * return it even when choosing not to lock, in case caller + * proceeds to loop over next ptes, and finds a match later. + * Though, in most cases, page lock already protects this. + */ + pvmw->pte = pte_offset_map_nolock(pvmw->vma->vm_mm, pvmw->pmd, + pvmw->address, ptlp); + if (!pvmw->pte) + return false; + if (pvmw->flags & PVMW_MIGRATION) { if (!is_swap_pte(*pvmw->pte)) return false; @@ -51,7 +63,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw) } else if (!pte_present(*pvmw->pte)) { return false; } - pvmw->ptl = pte_lockptr(pvmw->vma->vm_mm, pvmw->pmd); + pvmw->ptl = *ptlp; spin_lock(pvmw->ptl); return true; } @@ -156,6 +168,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) struct vm_area_struct *vma = pvmw->vma; struct mm_struct *mm = vma->vm_mm; unsigned long end; + spinlock_t *ptl; pgd_t *pgd; p4d_t *p4d; pud_t *pud; @@ -257,8 +270,11 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) step_forward(pvmw, PMD_SIZE); continue; } - if (!map_pte(pvmw)) + if (!map_pte(pvmw, &ptl)) { + if (!pvmw->pte) + goto restart; goto next_pte; + } this_pte: if (check_pte(pvmw)) return true; @@ -281,7 +297,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) } while (pte_none(*pvmw->pte)); if (!pvmw->ptl) { - pvmw->ptl = pte_lockptr(mm, pvmw->pmd); + pvmw->ptl = ptl; spin_lock(pvmw->ptl); } goto this_pte;