From patchwork Fri Jun 9 01:06:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273061 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A47D9C7EE23 for ; Fri, 9 Jun 2023 01:07:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0BF9D8E000F; Thu, 8 Jun 2023 21:07:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0705A8E0001; Thu, 8 Jun 2023 21:07:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E7A518E000F; Thu, 8 Jun 2023 21:07:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DBB298E0001 for ; Thu, 8 Jun 2023 21:07:06 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A4FE41A0390 for ; Fri, 9 Jun 2023 01:07:06 +0000 (UTC) X-FDA: 80881420452.21.FF943C0 Received: from mail-yw1-f180.google.com (mail-yw1-f180.google.com [209.85.128.180]) by imf02.hostedemail.com (Postfix) with ESMTP id BA80980010 for ; Fri, 9 Jun 2023 01:07:04 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=F4iPSaqG; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of hughd@google.com designates 209.85.128.180 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686272824; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FYWG+Dmea1iVO9u1anz9ea+vH7b9XYdNpJnPYqrcQ8g=; b=hT4ORVWF2FKjypi4w2K+bfmiSwYNiLFjMOVRe0v9lqLGvGuRJKHt7ATXMCARvu8I44mGub UKRSMwoHCiec71Ig3bIRNMrJkG4BJF4M1gaG47Q5DBYf0Uj30J6ElQNB3EESbt0JFZtzOJ 69vmRMRamxEe8NRsV5xls5OGxrMDvQU= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=F4iPSaqG; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of hughd@google.com designates 209.85.128.180 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686272824; a=rsa-sha256; cv=none; b=t2mBGTptZdlkXcYWg068YHPuj+wzAnjVEnbd9O7nXGLEPyS6xHvYu4StIkWowMl+cZsiVi GMlk/Sp/LAQq6qlCuIUTA+fs5MP5+X0yWiX4X1zavMKCpr9liLAkB0X7dDj05V4vCgs7YM UU1KlmORQ7srRdHy9Qo7MwS7MHs2TD8= Received: by mail-yw1-f180.google.com with SMTP id 00721157ae682-568ba7abc11so11400847b3.3 for ; Thu, 08 Jun 2023 18:07:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686272824; x=1688864824; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=FYWG+Dmea1iVO9u1anz9ea+vH7b9XYdNpJnPYqrcQ8g=; b=F4iPSaqGVauSHCN3PPx1HPCWJrbnjh20XWi+31Rxj6kybI3XEkrDo9g8TlN/DUHnID cA24qUKu4G6Vj7NJbRdZASctuqFdhdFDB+bQFh7yqdfrUkggC5zThxCrXeltsHoErZDt xA3Doa45fnOu6mBH+w4Zu0i9kibQ9g3niiOe6jmkqXtnnKBsq4Rlqd1i923q7ctcGoK9 NpGckUH03ViLvNgz5x9NiKAf8htS6SXGATYEGV3wZ9h6z7DQULhRQ42RGKscWpa7yH/X UrzgAwS5xf827WDV7tLvd4mt4ZGJ4FS3qwkbBLtNfkKh0VkQABmVM7ywrrTk/wDqi4Mi rRxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686272824; x=1688864824; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FYWG+Dmea1iVO9u1anz9ea+vH7b9XYdNpJnPYqrcQ8g=; b=JN9HWqwu+Pd4VNkI2S6nZWgOayb6VQlJnGg98BAuC1R9TaPOzRG8FkjMeUzugeDhEi /VBcjw86k+aL62F/mMhOBixt+qDAUbH5omhcEqO6627ZldjRN+9wYROVLL29VUN4WLt8 kAwsV4wsRFsEqjHR7kzB3sb1hjawQmJ/2iWKs7pnXsPQXz74p/XgGglAOJA8Yt+ixV+E bFySLwDs48QVCzySzVyYYtoWGMK+f8TBmRQtxvD0Y0j+8Lp9btYuhc1QiksDkCwmHJd7 OGhBYRhYimksKVk/1nljauH6bso1peaMzfYjeqE7ze6T4wUxHPJDmFnRv+fHp38bEnJ0 iNFg== X-Gm-Message-State: AC+VfDxXLMwzaqZX+EuA6+iPzwqmFLLN+Y7Tn+AYCTT+cqqrhfUezKmr hmqq1ppGeRpKhMZ03KhBmY/lxA== X-Google-Smtp-Source: ACHHUZ4e9gtItKHo/uKH6fckN0QfmcYW2cVTmMNzekmPXGojmR2zjeAFUImej/93RTPFo775/xnN5A== X-Received: by 2002:a81:7b0b:0:b0:561:afca:5b4d with SMTP id w11-20020a817b0b000000b00561afca5b4dmr1387844ywc.3.1686272823636; Thu, 08 Jun 2023 18:07:03 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id y4-20020a81a104000000b00566e949fb9esm282673ywg.82.2023.06.08.18.07.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:07:02 -0700 (PDT) Date: Thu, 8 Jun 2023 18:06:53 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 01/32] mm: use pmdp_get_lockless() without surplus barrier() In-Reply-To: Message-ID: References: MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: BA80980010 X-Stat-Signature: k3o8of3ajqefh6qzyjp5i5tcmbxcfaea X-Rspam-User: X-HE-Tag: 1686272824-821239 X-HE-Meta: U2FsdGVkX1+H4k5+xSZdSEypJ8tTCJmwG7+lq+XinTEAFlSIPsUnIqbZi+GNd5MzfjtaaTdC2o7HlZ+A5luwjIncA5BKMXypi/xvyFwo6FnLLwsZcCvBh26pDKgza6mJeYQvJMCslMwgBG8o+3dgfw3YiDcfVjvmUU0XuWmFpiYKnTSntqWAN464MOu2zW2lJHnn/qxTnYEKaWISKeMiuz/36fk5DQy1h0Cr/Xx37+pBI24cL5jGcsBZy9F9Ph+TKUWjgDbmzg0obfmxSWMR+8i1c5ocY7eiC5aqUMIHXA23uGTMILKVDB2fuEq9DUx8vagI26hmeTdqPnX82Gzz7U2LrK8ZBbX2EvsSpuGt616QIlwEhBgRBDz6Mjf/YBUTA6KrqMPo7gHQx7sTOkoTiLxHvnXblUFYadPwTIBU+yHuiYGRJeQGooYE+FX0ZnG82eDBEDTX1nnPkc0fD/4yvHsx2LaGDDn9wLvdUNRY2kuIRXla2+6zTsXIWqt8zTuYc+OSDN2SuyLbU+16DywWOZUOlcoINQmtG+i4EX0VcGT2CqU+ixOc6qyyjvxFg7vy1xE35ckT0Y75JFxqYwY8zRb/67ZnOFfZyJ1zu8/+6PkzzE57Ya1oi7yHo6uZ1gslQJexTJ9B68spOVFm4/31jzMMcpVZol/7kp4kJC+xyvSqE+MzQD5vTMcvuLzjwKVy2Ti0/H179efAOzXn0BiWXg3CJEuubSaquX2t810tDypUYbf4IUG1QXnT3gKdqJa2SeNYbsBcPp4KivPI9ObNoDfr7Is3x9pV3tgY28qSROI4/apn+GDkNNIjsRflioDtoz/Oz3UonYlnQe05iqfYJj71X3MlWxUonNKzF3resUiEoMlxMaQeDksgQoNEdvCMnd8bccZkUmu2eQs00V7Ux2e8XpDlIaMSWVl85GESF4R7FjhKCqHHoiv8Ed4yfkmkZgi1/bynD0YYDuoEWJX 8EJwJ9ze mKXghBa9xHHYdqFW3IWddfpNgNDjX29NyUD0gRegUDzac0WcdDTxB6Ceh7Q1azv+VhVV7djQmse/7JAVn9mBCN6mz4f9kCM3ND6ui8if0ob13UeaEj/d2PwJz82q/HGSccGIQLivqQ1LZzBkf/zZHMLNEXY3WwzuYNycjX0gWc7UUMTpCHc+N9ykiOxWJNtZ+SNdJOaLOpn76re0kuTX1bxlsz4eQls42kHSYJDs3BsvgYrYS0x0Gj319pwy/PeEVIyARkZ5Rk3ojyt3qsmv8tWloFVwWiwRx61ofhkplTmYQQbHvSSzPvxs3XvYx9EXwclG5qgtHKdEN4pzHMUe2NwZM8ePmAcofhKd3jye2ohedr+iKcfeNmVvb3K/Zvyck9L76GZqfm3EZnVTiHv6bNvlMDRVhsk7GUh/+j6AOk8e3NdhoxHjxRoYU67/X6hffDRhmuHi0hB1rWGc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use pmdp_get_lockless() in preference to READ_ONCE(*pmdp), to get a more reliable result with PAE (or READ_ONCE as before without PAE); and remove the unnecessary extra barrier()s which got left behind in its callers. HOWEVER: Note the small print in linux/pgtable.h, where it was designed specifically for fast GUP, and depends on interrupts being disabled for its full guarantee: most callers which have been added (here and before) do NOT have interrupts disabled, so there is still some need for caution. Signed-off-by: Hugh Dickins Acked-by: Yu Zhao Acked-by: Peter Xu --- fs/userfaultfd.c | 10 +--------- include/linux/pgtable.h | 17 ----------------- mm/gup.c | 6 +----- mm/hmm.c | 2 +- mm/khugepaged.c | 5 ----- mm/ksm.c | 3 +-- mm/memory.c | 14 ++------------ mm/mprotect.c | 5 ----- mm/page_vma_mapped.c | 2 +- 9 files changed, 7 insertions(+), 57 deletions(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 0fd96d6e39ce..f7a0817b1ec0 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -349,15 +349,7 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx, if (!pud_present(*pud)) goto out; pmd = pmd_offset(pud, address); - /* - * READ_ONCE must function as a barrier with narrower scope - * and it must be equivalent to: - * _pmd = *pmd; barrier(); - * - * This is to deal with the instability (as in - * pmd_trans_unstable) of the pmd. - */ - _pmd = READ_ONCE(*pmd); + _pmd = pmdp_get_lockless(pmd); if (pmd_none(_pmd)) goto out; diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index c5a51481bbb9..8ec27fe69dc8 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1344,23 +1344,6 @@ static inline int pud_trans_unstable(pud_t *pud) static inline int pmd_none_or_trans_huge_or_clear_bad(pmd_t *pmd) { pmd_t pmdval = pmdp_get_lockless(pmd); - /* - * The barrier will stabilize the pmdval in a register or on - * the stack so that it will stop changing under the code. - * - * When CONFIG_TRANSPARENT_HUGEPAGE=y on x86 32bit PAE, - * pmdp_get_lockless is allowed to return a not atomic pmdval - * (for example pointing to an hugepage that has never been - * mapped in the pmd). The below checks will only care about - * the low part of the pmd with 32bit PAE x86 anyway, with the - * exception of pmd_none(). So the important thing is that if - * the low part of the pmd is found null, the high part will - * be also null or the pmd_none() check below would be - * confused. - */ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - barrier(); -#endif /* * !pmd_present() checks for pmd migration entries * diff --git a/mm/gup.c b/mm/gup.c index bbe416236593..3bd5d3854c51 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -653,11 +653,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, struct mm_struct *mm = vma->vm_mm; pmd = pmd_offset(pudp, address); - /* - * The READ_ONCE() will stabilize the pmdval in a register or - * on the stack so that it will stop changing under the code. - */ - pmdval = READ_ONCE(*pmd); + pmdval = pmdp_get_lockless(pmd); if (pmd_none(pmdval)) return no_page_table(vma, flags); if (!pmd_present(pmdval)) diff --git a/mm/hmm.c b/mm/hmm.c index 6a151c09de5e..e23043345615 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -332,7 +332,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, pmd_t pmd; again: - pmd = READ_ONCE(*pmdp); + pmd = pmdp_get_lockless(pmdp); if (pmd_none(pmd)) return hmm_vma_walk_hole(start, end, -1, walk); diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 6b9d39d65b73..732f9ac393fc 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -961,11 +961,6 @@ static int find_pmd_or_thp_or_none(struct mm_struct *mm, return SCAN_PMD_NULL; pmde = pmdp_get_lockless(*pmd); - -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - /* See comments in pmd_none_or_trans_huge_or_clear_bad() */ - barrier(); -#endif if (pmd_none(pmde)) return SCAN_PMD_NONE; if (!pmd_present(pmde)) diff --git a/mm/ksm.c b/mm/ksm.c index 0156bded3a66..df2aa281d49d 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1194,8 +1194,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, * without holding anon_vma lock for write. So when looking for a * genuine pmde (in which to find pte), test present and !THP together. */ - pmde = *pmd; - barrier(); + pmde = pmdp_get_lockless(pmd); if (!pmd_present(pmde) || pmd_trans_huge(pmde)) goto out; diff --git a/mm/memory.c b/mm/memory.c index f69fbc251198..2eb54c0d5d3c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4925,18 +4925,9 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) * So now it's safe to run pte_offset_map(). */ vmf->pte = pte_offset_map(vmf->pmd, vmf->address); - vmf->orig_pte = *vmf->pte; + vmf->orig_pte = ptep_get_lockless(vmf->pte); vmf->flags |= FAULT_FLAG_ORIG_PTE_VALID; - /* - * some architectures can have larger ptes than wordsize, - * e.g.ppc44x-defconfig has CONFIG_PTE_64BIT=y and - * CONFIG_32BIT=y, so READ_ONCE cannot guarantee atomic - * accesses. The code below just needs a consistent view - * for the ifs and we later double check anyway with the - * ptl lock held. So here a barrier will do. - */ - barrier(); if (pte_none(vmf->orig_pte)) { pte_unmap(vmf->pte); vmf->pte = NULL; @@ -5060,9 +5051,8 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, if (!(ret & VM_FAULT_FALLBACK)) return ret; } else { - vmf.orig_pmd = *vmf.pmd; + vmf.orig_pmd = pmdp_get_lockless(vmf.pmd); - barrier(); if (unlikely(is_swap_pmd(vmf.orig_pmd))) { VM_BUG_ON(thp_migration_supported() && !is_pmd_migration_entry(vmf.orig_pmd)); diff --git a/mm/mprotect.c b/mm/mprotect.c index 92d3d3ca390a..c5a13c0f1017 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -309,11 +309,6 @@ static inline int pmd_none_or_clear_bad_unless_trans_huge(pmd_t *pmd) { pmd_t pmdval = pmdp_get_lockless(pmd); - /* See pmd_none_or_trans_huge_or_clear_bad for info on barrier */ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - barrier(); -#endif - if (pmd_none(pmdval)) return 1; if (pmd_trans_huge(pmdval)) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 4e448cfbc6ef..64aff6718bdb 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -210,7 +210,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) * compiler and used as a stale value after we've observed a * subsequent update. */ - pmde = READ_ONCE(*pvmw->pmd); + pmde = pmdp_get_lockless(pvmw->pmd); if (pmd_trans_huge(pmde) || is_pmd_migration_entry(pmde) || (pmd_present(pmde) && pmd_devmap(pmde))) {