From patchwork Mon Oct 3 19:34:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12997757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1FE9C433F5 for ; Mon, 3 Oct 2022 19:35:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 040E36B0071; Mon, 3 Oct 2022 15:35:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F33106B0073; Mon, 3 Oct 2022 15:34:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DADC98E0001; Mon, 3 Oct 2022 15:34:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id BBED76B0071 for ; Mon, 3 Oct 2022 15:34:59 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 8A3B48056C for ; Mon, 3 Oct 2022 19:34:59 +0000 (UTC) X-FDA: 79980641118.22.278DF24 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by imf02.hostedemail.com (Postfix) with ESMTP id 3A03780019 for ; Mon, 3 Oct 2022 19:34:58 +0000 (UTC) Received: by mail-pj1-f44.google.com with SMTP id x1-20020a17090ab00100b001fda21bbc90so16203749pjq.3 for ; Mon, 03 Oct 2022 12:34:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date; bh=XpdXNPBzi3+2LRRbY9xCMr4whphrF0pEZu5KjI+tgrU=; b=TmK2I6xcRG3UlmVmDyIhbhHW5FH3mqGaG579Akpwk7gXeKp6ffrjM2msXzZUJ9YmU/ DBGTI56Th8R38uOlONLCpNnU/2Et/mjsLCCVu+rxG42NU0Ea9zWCyBwbzZ4VD3qcPugE 5hlnGv4P+uWGQM4LzIgL3f67Dkk0Hg0n2h607MngD1GCzss6sV4L/GM5VPdYQRV1/2Uc Ei5nZfRr1ZfYBRz4kS35OHNMchhrDZk5LW+No/7z5LPGYn8nPL8DgMJMkTEhs8nWdx66 QZjw3Fe3VFsI3piKHl/zkzifauxtTFOzCAo9nko4EtHBW6rKSuKyoH0K7acC7gnd+GuF KZXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date; bh=XpdXNPBzi3+2LRRbY9xCMr4whphrF0pEZu5KjI+tgrU=; b=dvOFeN7yB1kFc2mDh0on3WxQaMnyIakQ42NRU9U2E+Lgruw3oVHyoQFEZZd8hrQ0eQ K0hZbIBL9izuvQekk0SU34ORUTqlWN4Z4FN9wXsQkSwqgfs10anU2vPNnUE7uG6cVZbb LHppowaCfhLhNU2XqJSFILDQUsxvUPAdDzoj3tsmdoieFb50sf6FHJDuSAIt82PubZrR y7QEjHk5oFx6uR78Jm9AolKI/uvvjNa3NZcMIB8MHqKKVpnRRK7IaY40S2LjzUpT8mFX WDbgbc0OMIzSKOOHQj4pGtzlV0/jdGTiox4oL3NsKhX3RdJZYYHgLSzOTgM1YNoYsGmT 7SKg== X-Gm-Message-State: ACrzQf1PI4sLCtyaq6kRQ3WktlSA0US60L6tt0fyEriyM27HzSOGjsud 1nbhvHXQwz0HPBQNh5k/C1M= X-Google-Smtp-Source: AMsMyM7GeLOdSHsxpl+w7D/FApIYullKrm3K37WExlSK/Ijcp7VXbtiNFVxsMaJ+nvkh1IUBukgJPg== X-Received: by 2002:a17:902:c410:b0:17d:4d86:4d91 with SMTP id k16-20020a170902c41000b0017d4d864d91mr14512697plk.41.1664825697045; Mon, 03 Oct 2022 12:34:57 -0700 (PDT) Received: from localhost.localdomain (c-67-174-241-145.hsd1.ca.comcast.net. [67.174.241.145]) by smtp.gmail.com with ESMTPSA id x26-20020a63171a000000b004308422060csm6809601pgl.69.2022.10.03.12.34.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Oct 2022 12:34:56 -0700 (PDT) From: Yang Shi To: gregkh@linuxfoundation.org, akpm@linux-foundation.org, aneesh.kumar@linux.ibm.com, christophe.leroy@csgroup.eu, david@redhat.com, hughd@google.com, jgg@nvidia.com, jhubbard@nvidia.com, kirill.shutemov@linux.intel.com, mpe@ellerman.id.au, npiggin@gmail.com, peterx@redhat.com, stable@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [5.10-stable PATCH] mm: gup: fix the fast GUP race against THP collapse Date: Mon, 3 Oct 2022 12:34:53 -0700 Message-Id: <20221003193453.1625076-1-shy828301@gmail.com> X-Mailer: git-send-email 2.26.3 MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1664825698; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=XpdXNPBzi3+2LRRbY9xCMr4whphrF0pEZu5KjI+tgrU=; b=fhruS6zpsugGC8/Hs0lpY38KqJ5JiKe+bOXR5SIDhAEYXEvLf4bj7XHgdApLY2KregvZAY vK8Llbw/KDQ5yIEoIJ01pN22a464QzG3jBC6qbF+MXvhgT/27OD8WSirJqr1wahBfYmmUB i7XncUvF3n2K1B6+SqBx049zHEbShkQ= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=TmK2I6xc; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of shy828301@gmail.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=shy828301@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1664825698; a=rsa-sha256; cv=none; b=vcuKGVkqIuXbfY5wZvt/M66+HEJskwA0vmxe8R1Um2iZzGkzR6q2C62zXJasQjaHSJvPsg n6FyijEXPE0n8TG0l8llXNDOYSizgJ4Hyd7v6Xs49Ms/FWEgT7HRTpyDQoAb5jG7BnS9Vk a1TrJ7sE6sUAmShiH3QiOXsl6HhtMNQ= X-Rspamd-Queue-Id: 3A03780019 Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=TmK2I6xc; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of shy828301@gmail.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=shy828301@gmail.com X-Rspam-User: X-Rspamd-Server: rspam10 X-Stat-Signature: cmpkt8dsdf1bnd69qus9agmczief3idi X-HE-Tag: 1664825698-787934 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: commit 70cbc3cc78a997d8247b50389d37c4e1736019da upstream Since general RCU GUP fast was introduced in commit 2667f50e8b81 ("mm: introduce a general RCU get_user_pages_fast()"), a TLB flush is no longer sufficient to handle concurrent GUP-fast in all cases, it only handles traditional IPI-based GUP-fast correctly. On architectures that send an IPI broadcast on TLB flush, it works as expected. But on the architectures that do not use IPI to broadcast TLB flush, it may have the below race: CPU A CPU B THP collapse fast GUP gup_pmd_range() <-- see valid pmd gup_pte_range() <-- work on pte pmdp_collapse_flush() <-- clear pmd and flush __collapse_huge_page_isolate() check page pinned <-- before GUP bump refcount pin the page check PTE <-- no change __collapse_huge_page_copy() copy data to huge page ptep_clear() install huge pmd for the huge page return the stale page discard the stale page The race can be fixed by checking whether PMD is changed or not after taking the page pin in fast GUP, just like what it does for PTE. If the PMD is changed it means there may be parallel THP collapse, so GUP should back off. Also update the stale comment about serializing against fast GUP in khugepaged. Link: https://lkml.kernel.org/r/20220907180144.555485-1-shy828301@gmail.com Fixes: 2667f50e8b81 ("mm: introduce a general RCU get_user_pages_fast()") Acked-by: David Hildenbrand Acked-by: Peter Xu Signed-off-by: Yang Shi Reviewed-by: John Hubbard Cc: "Aneesh Kumar K.V" Cc: Hugh Dickins Cc: Jason Gunthorpe Cc: "Kirill A. Shutemov" Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Christophe Leroy Cc: Signed-off-by: Andrew Morton --- mm/gup.c | 34 ++++++++++++++++++++++++++++------ mm/khugepaged.c | 10 ++++++---- 2 files changed, 34 insertions(+), 10 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 6cb7d8ae56f6..b47c751df069 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2128,8 +2128,28 @@ static void __maybe_unused undo_dev_pagemap(int *nr, int nr_start, } #ifdef CONFIG_ARCH_HAS_PTE_SPECIAL -static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, - unsigned int flags, struct page **pages, int *nr) +/* + * Fast-gup relies on pte change detection to avoid concurrent pgtable + * operations. + * + * To pin the page, fast-gup needs to do below in order: + * (1) pin the page (by prefetching pte), then (2) check pte not changed. + * + * For the rest of pgtable operations where pgtable updates can be racy + * with fast-gup, we need to do (1) clear pte, then (2) check whether page + * is pinned. + * + * Above will work for all pte-level operations, including THP split. + * + * For THP collapse, it's a bit more complicated because fast-gup may be + * walking a pgtable page that is being freed (pte is still valid but pmd + * can be cleared already). To avoid race in such condition, we need to + * also check pmd here to make sure pmd doesn't change (corresponds to + * pmdp_collapse_flush() in the THP collapse code path). + */ +static int gup_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, + unsigned long end, unsigned int flags, + struct page **pages, int *nr) { struct dev_pagemap *pgmap = NULL; int nr_start = *nr, ret = 0; @@ -2169,7 +2189,8 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, if (!head) goto pte_unmap; - if (unlikely(pte_val(pte) != pte_val(*ptep))) { + if (unlikely(pmd_val(pmd) != pmd_val(*pmdp)) || + unlikely(pte_val(pte) != pte_val(*ptep))) { put_compound_head(head, 1, flags); goto pte_unmap; } @@ -2214,8 +2235,9 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, * get_user_pages_fast_only implementation that can pin pages. Thus it's still * useful to have gup_huge_pmd even if we can't operate on ptes. */ -static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, - unsigned int flags, struct page **pages, int *nr) +static int gup_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, + unsigned long end, unsigned int flags, + struct page **pages, int *nr) { return 0; } @@ -2522,7 +2544,7 @@ static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned lo if (!gup_huge_pd(__hugepd(pmd_val(pmd)), addr, PMD_SHIFT, next, flags, pages, nr)) return 0; - } else if (!gup_pte_range(pmd, addr, next, flags, pages, nr)) + } else if (!gup_pte_range(pmd, pmdp, addr, next, flags, pages, nr)) return 0; } while (pmdp++, addr = next, addr != end); diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 969e57dde65f..cf4dceb9682b 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1144,10 +1144,12 @@ static void collapse_huge_page(struct mm_struct *mm, pmd_ptl = pmd_lock(mm, pmd); /* probably unnecessary */ /* - * After this gup_fast can't run anymore. This also removes - * any huge TLB entry from the CPU so we won't allow - * huge and small TLB entries for the same virtual address - * to avoid the risk of CPU bugs in that area. + * This removes any huge TLB entry from the CPU so we won't allow + * huge and small TLB entries for the same virtual address to + * avoid the risk of CPU bugs in that area. + * + * Parallel fast GUP is fine since fast GUP will back off when + * it detects PMD is changed. */ _pmd = pmdp_collapse_flush(vma, address, pmd); spin_unlock(pmd_ptl);