From patchwork Fri Nov 25 21:37:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jann Horn X-Patchwork-Id: 13056268 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CEE9DC4332F for ; Fri, 25 Nov 2022 21:37:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 38E066B0081; Fri, 25 Nov 2022 16:37:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 33DCB6B0082; Fri, 25 Nov 2022 16:37:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1DEAD6B0083; Fri, 25 Nov 2022 16:37:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 114C56B0081 for ; Fri, 25 Nov 2022 16:37:47 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id DBC93C0EA8 for ; Fri, 25 Nov 2022 21:37:46 +0000 (UTC) X-FDA: 80173276932.13.9EAB9BF Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) by imf15.hostedemail.com (Postfix) with ESMTP id 773C5A000D for ; Fri, 25 Nov 2022 21:37:46 +0000 (UTC) Received: by mail-wr1-f48.google.com with SMTP id n3so8521929wrp.5 for ; Fri, 25 Nov 2022 13:37:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SXD6TekHCwWyaJjEnEUCIQwhKm8R4syqj2EShSG9CG4=; b=LbycPTZagQ51nPLmloFB/7NtnvTiSeUQAuZZ2EkPPlQgtWmKZxxpz1La+0Sg5Wa4Rr QPK0+z28OGsP2RhtKkhY6uwU9FRTAt/NqXjZ7MjCe3YETc37AnNoh2YcVnwLSIA/A2in ozUtyauDDVkAhz9m29tRAE0JI3b3myHPK1RVcbBAl0asvNRR5Vy+CDZXCw/8tuXxqhjS yrv6noxZgbshgZx84/f8bQc2LkkMA/Z9vXXk6Bo0tqg2j/ykUC5HaVHtCvv4sb+G8fPj 1eJ657mhGUY43wuEPXURLQhISq427MEO8FteYgZbLF00C0vgmD5VPh021QAYYnfkDmNV 3Caw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SXD6TekHCwWyaJjEnEUCIQwhKm8R4syqj2EShSG9CG4=; b=h5/3IOFe9QCXVQTBTesfkQ4AsqAhPw7yycA+nrdKd4vRWRwMCEoZFQwz+ZMqrASfk/ a0rtpVhFt1RJg/d88Hg3JhqeuKYdr6ocZbX9mGzPZwp69HtLafm9WvAZXLNeLDgdZtq2 /hDjLMFgKKeoC5c093M65jPV1eDX7p02Dm5HHCFD9dsk//QfiS13at2wnyWXVN1utYD6 Ay2wVIkKNOuyEU3voe2/p6RVHM/lRtENE0EFyPi+liHnwXo8YwJYFa/99hTOt2RwEuag PwVzK6S7NenOZ4P9mO0lt1qKlvNcyip5aL8G2MzNa4dRoiHIr5E4hrB1kif3e1rp7U0e u48w== X-Gm-Message-State: ANoB5pmaYLcyytnVOPq2KQO0nURUJS0mfSLLoHzT7k1+mOkjZ+zlkuGl OOuEVSp2NX3UBzVWLJD2d1JjJg== X-Google-Smtp-Source: AA0mqf5MLHrAyzs+7KAfomgqrFtT5xPA83/WPLEYaIucLGWlV7Rf/Y2a73erzIdA6X9yklrsVtx2Nw== X-Received: by 2002:a05:6000:235:b0:241:bd2b:f718 with SMTP id l21-20020a056000023500b00241bd2bf718mr23371777wrz.89.1669412265015; Fri, 25 Nov 2022 13:37:45 -0800 (PST) Received: from localhost ([2a00:79e0:9d:4:9104:e316:5846:858e]) by smtp.gmail.com with ESMTPSA id o5-20020a05600c510500b003b4ff30e566sm16931581wms.3.2022.11.25.13.37.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Nov 2022 13:37:44 -0800 (PST) From: Jann Horn To: security@kernel.org, Andrew Morton Cc: Yang Shi , David Hildenbrand , Peter Xu , John Hubbard , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 2/3] mm/khugepaged: Fix GUP-fast interaction by sending IPI Date: Fri, 25 Nov 2022 22:37:13 +0100 Message-Id: <20221125213714.4115729-2-jannh@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog In-Reply-To: <20221125213714.4115729-1-jannh@google.com> References: <20221125213714.4115729-1-jannh@google.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1669412266; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SXD6TekHCwWyaJjEnEUCIQwhKm8R4syqj2EShSG9CG4=; b=p61AVqnmi4UbpmXnm25A9h+mfa3YwUfGeWqQoQkqdy6+COy2gOFvSzXChXxn5FkUrNr+RA +ftD1HeSwoKcNrD+dGLOkRPW6aXPM9CKSgKSdgU4yXQMI8ki16RKWTejb4n7QIyLVu/saY BguVthkx37z4gQObqKuz1V8L8OW79oI= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=LbycPTZa; spf=pass (imf15.hostedemail.com: domain of jannh@google.com designates 209.85.221.48 as permitted sender) smtp.mailfrom=jannh@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1669412266; a=rsa-sha256; cv=none; b=0sIaVAJdCbQ4JmVs45UDknay4DFrRwWiBxZmU/cnOILKJZ7dzQXGrFTdOz2kyX0zZwyvyU IFOlhDlxW4VSgJgQ3uZGMzA9XSDNfLUb8+q5UIpaxxTAYwC9/EkBoN9MdXap76zz+SzOaz yAU8swDXLTpkOtNxIih7ob4cB80eTNE= X-Rspamd-Queue-Id: 773C5A000D Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=LbycPTZa; spf=pass (imf15.hostedemail.com: domain of jannh@google.com designates 209.85.221.48 as permitted sender) smtp.mailfrom=jannh@google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 6nfzho7hah313kif6dothnubpcwjnrg9 X-HE-Tag: 1669412266-410197 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since commit 70cbc3cc78a99 ("mm: gup: fix the fast GUP race against THP collapse"), the lockless_pages_from_mm() fastpath rechecks the pmd_t to ensure that the page table was not removed by khugepaged in between. However, lockless_pages_from_mm() still requires that the page table is not concurrently freed. Fix it by sending IPIs (if the architecture uses semi-RCU-style page table freeing) before freeing/reusing page tables. Cc: stable@kernel.org Fixes: ba76149f47d8 ("thp: khugepaged") Signed-off-by: Jann Horn Acked-by: David Hildenbrand --- replaced the mmu_gather-based scheme with an RCU call as suggested by Peter Xu include/asm-generic/tlb.h | 4 ++++ mm/khugepaged.c | 2 ++ mm/mmu_gather.c | 4 +--- 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 492dce43236ea..cab7cfebf40bd 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -222,12 +222,16 @@ extern void tlb_remove_table(struct mmu_gather *tlb, void *table); #define tlb_needs_table_invalidate() (true) #endif +void tlb_remove_table_sync_one(void); + #else #ifdef tlb_needs_table_invalidate #error tlb_needs_table_invalidate() requires MMU_GATHER_RCU_TABLE_FREE #endif +static inline void tlb_remove_table_sync_one(void) { } + #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 674b111a24fa7..c3d3ce596bff7 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1057,6 +1057,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, _pmd = pmdp_collapse_flush(vma, address, pmd); spin_unlock(pmd_ptl); mmu_notifier_invalidate_range_end(&range); + tlb_remove_table_sync_one(); spin_lock(pte_ptl); result = __collapse_huge_page_isolate(vma, address, pte, cc, @@ -1415,6 +1416,7 @@ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *v lockdep_assert_held_write(&vma->anon_vma->root->rwsem); pmd = pmdp_collapse_flush(vma, addr, pmdp); + tlb_remove_table_sync_one(); mm_dec_nr_ptes(mm); page_table_check_pte_clear_range(mm, addr, pmd); pte_free(mm, pmd_pgtable(pmd)); diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index add4244e5790d..3a2c3f8cad2fe 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -153,7 +153,7 @@ static void tlb_remove_table_smp_sync(void *arg) /* Simply deliver the interrupt */ } -static void tlb_remove_table_sync_one(void) +void tlb_remove_table_sync_one(void) { /* * This isn't an RCU grace period and hence the page-tables cannot be @@ -177,8 +177,6 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch) #else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */ -static void tlb_remove_table_sync_one(void) { } - static void tlb_remove_table_free(struct mmu_table_batch *batch) { __tlb_remove_table_free(batch);