From patchwork Wed Aug 21 08:18:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13770990 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DAA6CC52D7C for ; Wed, 21 Aug 2024 08:20:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6948C6B00CB; Wed, 21 Aug 2024 04:20:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 642E26B00CC; Wed, 21 Aug 2024 04:20:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4BDAC6B00CD; Wed, 21 Aug 2024 04:20:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2A3C46B00CB for ; Wed, 21 Aug 2024 04:20:18 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 97F0D1409A4 for ; Wed, 21 Aug 2024 08:20:17 +0000 (UTC) X-FDA: 82475555274.30.8200679 Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) by imf13.hostedemail.com (Postfix) with ESMTP id C222E2000D for ; Wed, 21 Aug 2024 08:20:15 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ARp+xsnr; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.46 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724228335; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ATsR6rCpxZFOFORcauHss5xESVmegH/7Abibhpgyw90=; b=kirXcut88OjEz2zGW9+xg1M9W8ljFA6lY9bF9wrEATc55uwJOgcpozqtSZkdW3jqkf9K3r K2hWW6+buWU5h2BReOKF6boeuwlo3/s2VcFg9LBoeEwOD85m88jAfK4FE9eZNDl8o0NqAH dsJa5oGOq/uLt0gPwuK8bozg36B2In8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724228335; a=rsa-sha256; cv=none; b=qFauzgrJxTGSCDuUo9En+iPk5g+gKMgsVPt8lkjvWxZzGp/hihT20ikmfzpUl4+7wMWqww hU726gs5+G+pZGOlV034oWOjCCkNph/672GCp63g2X0t/ZL0nFgV16Sk/g4Z2d0p3MYgS5 opzXtO5K1+6b1G840j5YKJyBYsz2fsE= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ARp+xsnr; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.46 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pj1-f46.google.com with SMTP id 98e67ed59e1d1-2d3da054f7cso3951823a91.1 for ; Wed, 21 Aug 2024 01:20:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228414; x=1724833214; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ATsR6rCpxZFOFORcauHss5xESVmegH/7Abibhpgyw90=; b=ARp+xsnrbLXGzgahhdCk1M27IbMsZm4i3Z03IW2gIvucn9ryCvOASP9xHL9AiRMfU3 mvNp2oObAtFvLC3qBVwMrpVTvczybZeGsTeeEB4tVGD0RXvhWZ4hBGcD+78ez82ZYT63 ZNeNTHuCWwkEa1U+N/gRA/cBPvlw7uvCn6+t5Oodt97eu16suqCXzP5HUtiGBXimtOKJ 4Ha2lNwkEenRhPFT+WTd2HHYyN1VzjzgPg7RfO7xuVAk6FzyybdDWHvGKrav9OrkDtXM uJt9BPQJfoFLTzhC5+SNsOg9WSEAEazQVh8anUrnB7RGA/Y+IcOz9u90tZTV3MSJoLgy KCLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228414; x=1724833214; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ATsR6rCpxZFOFORcauHss5xESVmegH/7Abibhpgyw90=; b=u0fVJZUfxqP/Qbt9K/MFKhUz3sLhCS9sMQ2d3Ys6Rb2JGZXycmy90GBq16Y6/NGAes diPwIdGXBDXlzNbItaadPYos8Rc54QUTYUuHJn++MOxM9rUEjxkQWsFkWYRRVMH7mDRN zjtM6gywMi/yvHsGeD9zU6cqkYnYEKOtXkSV6jSQ8v/BsIlXP7/d0lNMcJUCHjw3WFZs UfQecwYRaPaDxbouIAN7gBkrUhD3inonC74VfHsTTqITYd3M8meCVvbhOFo/tBxO+kY5 /kUVxw84yDtLpWLuahl0niYtKCYaCJ+fru99oQVkEdQIKR6GLcPleJW309HwJuMWahvQ qoDg== X-Forwarded-Encrypted: i=1; AJvYcCV1Jyp9SUvWbj/iDj3Gi4hk8AHHR6J+ZZEK6r4DX1mPH2gftPJCGPP8YZgMGCAzphAyqnQV5RB8+Q==@kvack.org X-Gm-Message-State: AOJu0YybOGtryS5wxFPePiRg9a+ZqjGpSO6+bMEuxpt5ShDoalUOd8C8 mYMR5m1SJR2i2EoQf2UqpsaPLJ6Er+OQCfb+hOm2CFDXu7a5eSpXM3x8F/8pGLM= X-Google-Smtp-Source: AGHT+IEASO/tYfbmKEqODBO5ptoLkIYqajvyb9bjTzXqrTBGFe58odiOJRUvoUKiSb3oI+1Cc8Ov6A== X-Received: by 2002:a17:90b:1c01:b0:2c9:9658:d704 with SMTP id 98e67ed59e1d1-2d5ea4c9ab9mr1366119a91.40.1724228414402; Wed, 21 Aug 2024 01:20:14 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:14 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 07/14] mm: khugepaged: collapse_pte_mapped_thp() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:50 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: 7q6dabai3aubed4rzs5ckpfy7174xumx X-Rspamd-Queue-Id: C222E2000D X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1724228415-14292 X-HE-Meta: U2FsdGVkX19ObH5P47tsQEDjJ7IpPMuVN1kU6YGz4NsQBlUPGt1LtysywXnDepOkEr58ifjuo0AQljgQYdZ6hfXbj4vuseEjuN97kYDReX9QwLpjmW2GnFY7Lzx3M3rdm818NkIvgUK8TfxOq8A0B4ctZaNMz0vh4DcnBu9Jqyt/q0eYNI5eVR3yLar6MwiUdu9rS3YGkT9oxoIT31kT+Ys8Y/SbdGU3yz+ptUnBoP4x70JeSUL4CVx72hfzUYi2xPAfxqKxQWtLfJH0TfzCM8c1eK7CK1pzYEoC4gt5MlIca2RCW1S+mE6C2sIgqYqhVQh3CGlByLiubJxFZlW4i94eKsogSeU+5RYgcRFwabhC/sMrm4mNsG072XR3WhtxsZ4LKbdb30MPCFOXE+glS34F1cVqYRDlOipv5vcPVrysfD5NyzVYmRht/SgdcPcxu3jAVkqN8P1NAJNpILn/uRTVY+pAGbhrdxVac8SkyPHm8Ephm2XO5e5q1fFKfzVOCjTudCW8u4DAc01VpsapQTrC59w7fpz8YY+9gpAY7GhfPfdkQ1/UwfUvYJC76wMpVMG594XIraLIjXSiwjxJWV1P8fnbCetYEQnTSh6cUgI4LdC6Mox7Do/18rHw/76WxlK2wFQFYQrdYJvmkpSG0xUfZcDyEyH1oh3CIrZ5Y/hk5k/N5EAceIU149hj+yjUswbfylAM1nF2H7Q1n5BTN1p7USGQ6LQBH84qa79anE5DA+7WIvOnd+iPPafQfAkNBo3MxAEF+xRwkYuxVS1hsc4NXjiQT946h5wDb6v031e8wFYzwaf9YVT5gnyKhKx2tUD0GQK790UloLRF6c/qsyz0yNnPTrWwOId+D7TRUPsP5tCU+G0EyaNdckjKMTqO3YzRm1NXk4o0yRK4KvMVvZ5dzQgE1gYICcx6kP+2Ab5daalZXI3N7d8khq6LOqGwcK0CHYFzjZjNUDmJsut ZnjLPRrk FEWyw1tPQqwxqcA4lZZndtCG9lYsy+CJQ5FyZJoZPqjGnRT5eFvC93BSfJ1jyLf0Avxf7j5lk6m4dcyqhrjXBzOW1Xy9gp/6Zx801nMj4u+pzoL4qjjuZqn7L2BowZcs3ToZpNAs6e+u2dqcxevDreNDCuh4Xwsuhfmg17cdGW30FiVE3zzfubYSDGw73KpFWJWmLx29dUpM4wXXgfSgAO72Sld0KrMCVSR+pKfH11LkWg8FZnc+gUkBvtwZBysEd6iHbMTiXroBep+/yFzHJcMDAYNeBcY7LRlTuLQ7Db4arjc0+fgKR2ltQc3/WqE5fUQL8Bf4WuOl2u5IpKSQ+9wQyPftXFjrTos/9QB5kwGWyrAkL95psBGO589JG4R0NoSiqrRXMIHHnMUZ64jZ+xa96tAeXEwRwGThQKN5+/XnM5c20ush6D3w+lg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In collapse_pte_mapped_thp(), we may modify the pte and pmd entry after acquring the ptl, so convert it to using pte_offset_map_maywrite_nolock(). At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the PTL held. So we should get pgt_pmd and do pmd_same() check after the ptl held. For the case where the ptl is released first and then the pml is acquired, the PTE page may have been freed, so we must do pmd_same() check before reacquiring the ptl. Signed-off-by: Qi Zheng --- mm/khugepaged.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 26c083c59f03f..8fcad0b368a08 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1602,7 +1602,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) pml = pmd_lock(mm, pmd); - start_pte = pte_offset_map_nolock(mm, pmd, haddr, &ptl); + start_pte = pte_offset_map_maywrite_nolock(mm, pmd, haddr, &pgt_pmd, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ goto abort; if (!pml) @@ -1610,6 +1610,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, else if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) + goto abort; + /* step 2: clear page table and adjust rmap */ for (i = 0, addr = haddr, pte = start_pte; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE, pte++) { @@ -1655,6 +1658,16 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, /* step 4: remove empty page table */ if (!pml) { pml = pmd_lock(mm, pmd); + /* + * We called pte_unmap() and release the ptl before acquiring + * the pml, which means we left the RCU critical section, so the + * PTE page may have been freed, so we must do pmd_same() check + * before reacquiring the ptl. + */ + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + spin_unlock(pml); + goto pmd_change; + } if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); } @@ -1686,6 +1699,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, pte_unmap_unlock(start_pte, ptl); if (pml && pml != ptl) spin_unlock(pml); +pmd_change: if (notified) mmu_notifier_invalidate_range_end(&range); drop_folio: