From patchwork Mon Nov 28 18:02:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jann Horn X-Patchwork-Id: 13057845 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B44C1C4332F for ; Mon, 28 Nov 2022 18:03:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F07056B0073; Mon, 28 Nov 2022 13:03:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EB6D06B0074; Mon, 28 Nov 2022 13:03:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D7E596B0075; Mon, 28 Nov 2022 13:03:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C7FA76B0073 for ; Mon, 28 Nov 2022 13:03:10 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 82A81160E4F for ; Mon, 28 Nov 2022 18:03:10 +0000 (UTC) X-FDA: 80183622540.11.50CA28D Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) by imf25.hostedemail.com (Postfix) with ESMTP id 8F2FDA003C for ; Mon, 28 Nov 2022 18:03:08 +0000 (UTC) Received: by mail-wm1-f48.google.com with SMTP id o30so9117189wms.2 for ; Mon, 28 Nov 2022 10:03:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RUow5Vogs/7PnC9nrxFlLM/L16QCcApD5uJXFy4USuY=; b=rU/5WuLN/iM6g1+U2lBomEDv/g6EreorKfHwKtqRkF8SblU0VsIoZV8lMj7la44xOo RFhODdlKSvqR653yK+HRv3a22IBJ7tLBK5Z6ja7TLaM28AE7gzd8hZGHoFRg1XxIlqjJ WzZLuZdJo6sJsyS+ZZSUvjrHnUPfa6Ac26czHLanj7NP3nju9ONX3HUkVJHKHv9yZ1BF xmWKi8CRGLpW4bKOJghlq05QZvrVOUN3iDX+uFzjp2XCnvE1dkTCbY2bZ7loNCgKJ9US 4XoznfiWY3eu0D8pWYcvLjab/VADOYaVOhIRlWOyHDbaAxmQm1iUD8calDDGZ6mT7Dcv kYVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RUow5Vogs/7PnC9nrxFlLM/L16QCcApD5uJXFy4USuY=; b=sqQaRmQkDpJgN1ywZGRefVJsmhXMKxAwFhBDDSiDb1mDIG7vhRt2XGIgwzG8eGQM3i mX2x3962L11EpsLsEN1Ardhd89YEnXF8FunDcUPpCCJO6nslGKzTkjdDBLWjF1rATwHE rcxpuhYLVzHVjI4aRWzc2F7WLmgPDyzcxYdFf5KJwIkycqxa1CydGWuaIugVW4CcpDuz 6iaBimyk+e1XXeLZB/zYxpA4whX8vXLOaQeuigOxBEU0tyF/kLNLwHBrbgwvivgKOd+k xNbaS1VEB3Oik1FCT5YDdFupnCVcsvY1PzmExMVEYbu9iBRfUxhrTRxLD4mesjqg+kHG wE3g== X-Gm-Message-State: ANoB5pmfPqSm5CbbgasE0vWQYT/rNh0jsENC84jyLMiORh2N3L4RKaWM ctwYt5x2C29QZXSYbN66b79z3w== X-Google-Smtp-Source: AA0mqf55TgU3FF0+HFze1Tgck+4ISKVwlxDQBSJiTP5rct0txTHiKl+xLdG68s5H54ewewN4ONkTvA== X-Received: by 2002:a7b:cd18:0:b0:3cf:a359:de90 with SMTP id f24-20020a7bcd18000000b003cfa359de90mr28732913wmj.122.1669658587158; Mon, 28 Nov 2022 10:03:07 -0800 (PST) Received: from localhost ([2a00:79e0:9d:4:f4d1:b340:8675:e840]) by smtp.gmail.com with ESMTPSA id l11-20020adff48b000000b002420dba6447sm6628727wro.59.2022.11.28.10.03.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Nov 2022 10:03:05 -0800 (PST) From: Jann Horn To: security@kernel.org, Andrew Morton Cc: Yang Shi , David Hildenbrand , Peter Xu , John Hubbard , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v4 2/3] mm/khugepaged: Fix GUP-fast interaction by sending IPI Date: Mon, 28 Nov 2022 19:02:51 +0100 Message-Id: <20221128180252.1684965-2-jannh@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog In-Reply-To: <20221128180252.1684965-1-jannh@google.com> References: <20221128180252.1684965-1-jannh@google.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1669658588; a=rsa-sha256; cv=none; b=5fEdFdoDaM6NPjRbmGcl8gRuAonrmu0cIgcvrpX9xyXTJF/KKZwY3uz6QCpXQW9uC3V3yW 6iTVRKeu0LwXz6OrAj78B5RcWvvl7mbzejGNltDegA86fHCkSlcTnyS5RoKggtqDVAJrIh EtfZ6lExG9dA1B/6jKNc1vuo5UTAYjI= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="rU/5WuLN"; spf=pass (imf25.hostedemail.com: domain of jannh@google.com designates 209.85.128.48 as permitted sender) smtp.mailfrom=jannh@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1669658588; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RUow5Vogs/7PnC9nrxFlLM/L16QCcApD5uJXFy4USuY=; b=jtaHauZw1TupnJc0CHET8JdZYwkm4tZPdlruhZruqUT+UorJS6b5o4Eg8IHZi/A6nW0Fa8 Jmd2dwHG/pkLZ654VlgVuGzZ1rv69/JFUKMz6l6xF8xnzd5r7kyfM9nhOsJnBE3AxGaMe1 bf9XFgB0ooRBe/kYaIvAXYi4wT5YsTI= X-Stat-Signature: fmce8mhkkafobbpwj43ps13qp3rxsc88 X-Rspamd-Queue-Id: 8F2FDA003C X-Rspam-User: Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="rU/5WuLN"; spf=pass (imf25.hostedemail.com: domain of jannh@google.com designates 209.85.128.48 as permitted sender) smtp.mailfrom=jannh@google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam05 X-HE-Tag: 1669658588-820203 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since commit 70cbc3cc78a99 ("mm: gup: fix the fast GUP race against THP collapse"), the lockless_pages_from_mm() fastpath rechecks the pmd_t to ensure that the page table was not removed by khugepaged in between. However, lockless_pages_from_mm() still requires that the page table is not concurrently freed or reused to store non-PTE data. Otherwise, problems can occur because: - deposited page tables can be freed when a THP page somewhere in the mm is removed - some architectures store non-PTE information inside deposited page tables (see radix__pgtable_trans_huge_deposit()) Additionally, lockless_pages_from_mm() is also somewhat brittle with regards to page tables being repeatedly moved back and forth, but that shouldn't be an issue in practice. Fix it by sending IPIs (if the architecture uses semi-RCU-style page table freeing) before freeing/reusing page tables. As noted in mm/gup.c, on configs that define CONFIG_HAVE_FAST_GUP, there are two possible cases: 1. CONFIG_MMU_GATHER_RCU_TABLE_FREE is set, causing tlb_remove_table_sync_one() to send an IPI to synchronize with lockless_pages_from_mm(). 2. CONFIG_MMU_GATHER_RCU_TABLE_FREE is unset, indicating that all TLB flushes are already guaranteed to send IPIs. tlb_remove_table_sync_one() will do nothing, but we've already run pmdp_collapse_flush(), which did a TLB flush, which must have involved IPIs. Cc: stable@kernel.org Fixes: ba76149f47d8 ("thp: khugepaged") Acked-by: David Hildenbrand Signed-off-by: Jann Horn Reviewed-by: Yang Shi --- v4: - added ack from David Hildenbrand - made commit message more verbose include/asm-generic/tlb.h | 4 ++++ mm/khugepaged.c | 2 ++ mm/mmu_gather.c | 4 +--- 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 492dce43236ea..cab7cfebf40bd 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -222,12 +222,16 @@ extern void tlb_remove_table(struct mmu_gather *tlb, void *table); #define tlb_needs_table_invalidate() (true) #endif +void tlb_remove_table_sync_one(void); + #else #ifdef tlb_needs_table_invalidate #error tlb_needs_table_invalidate() requires MMU_GATHER_RCU_TABLE_FREE #endif +static inline void tlb_remove_table_sync_one(void) { } + #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 674b111a24fa7..c3d3ce596bff7 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1057,6 +1057,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, _pmd = pmdp_collapse_flush(vma, address, pmd); spin_unlock(pmd_ptl); mmu_notifier_invalidate_range_end(&range); + tlb_remove_table_sync_one(); spin_lock(pte_ptl); result = __collapse_huge_page_isolate(vma, address, pte, cc, @@ -1415,6 +1416,7 @@ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *v lockdep_assert_held_write(&vma->anon_vma->root->rwsem); pmd = pmdp_collapse_flush(vma, addr, pmdp); + tlb_remove_table_sync_one(); mm_dec_nr_ptes(mm); page_table_check_pte_clear_range(mm, addr, pmd); pte_free(mm, pmd_pgtable(pmd)); diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index add4244e5790d..3a2c3f8cad2fe 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -153,7 +153,7 @@ static void tlb_remove_table_smp_sync(void *arg) /* Simply deliver the interrupt */ } -static void tlb_remove_table_sync_one(void) +void tlb_remove_table_sync_one(void) { /* * This isn't an RCU grace period and hence the page-tables cannot be @@ -177,8 +177,6 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch) #else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */ -static void tlb_remove_table_sync_one(void) { } - static void tlb_remove_table_free(struct mmu_table_batch *batch) { __tlb_remove_table_free(batch);