From patchwork Tue May 21 04:02:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Yang X-Patchwork-Id: 13668892 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB849C25B74 for ; Tue, 21 May 2024 04:03:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1CC596B0085; Tue, 21 May 2024 00:03:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 17CD26B0088; Tue, 21 May 2024 00:03:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 044746B0089; Tue, 21 May 2024 00:03:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id DC4826B0085 for ; Tue, 21 May 2024 00:03:19 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 44F3F4091A for ; Tue, 21 May 2024 04:03:19 +0000 (UTC) X-FDA: 82141058118.18.D44FB6F Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf03.hostedemail.com (Postfix) with ESMTP id 86ABC20010 for ; Tue, 21 May 2024 04:03:17 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=EBVVEQiS; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf03.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1716264197; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=spLqBETeC+qWfSJg+4pVx1XrstM8E2ZSksh7ykN8FD8=; b=b/sBi5cZ5Jl0M3gqYrM/m+lVyFf74DoV6HBVbJpZM96/rV3oKuo+mbn+vMzBvvdiVmKRvb N9Rstl1thEzxxGRCwc0A0+pfR8OcoymfXrYhnCP29mIJN9Yvx0t5lFHTcFVCJJkwToTn4z ougGZ1kK57Ec9cHv9WyA3eC7BWyaF5w= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=EBVVEQiS; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf03.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1716264197; a=rsa-sha256; cv=none; b=ieNQlPeRSKNDl+3mncyhkeBwj2/rvlJXYjeZWYZsEXjkORa+XdhrALaoqWu4FHLiIgI+l5 gD6Z5z2Pif03xl0gZ+6sRaafT1wz7hGiX9Z06bDIW0TNNv4zuTdxVme119kclUn3iJAIa9 LSHtnKTTk9OMVgWexC5U7dMFPzMDIv4= Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-1ed41eb3382so95624495ad.0 for ; Mon, 20 May 2024 21:03:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716264196; x=1716868996; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=spLqBETeC+qWfSJg+4pVx1XrstM8E2ZSksh7ykN8FD8=; b=EBVVEQiSI10hCMJS6nses+cAiQ7dXw+KmyrlL7l4+/kMwCxTLZd4mYVENL6VvIyFSy KipbU/X7LfC94nQIM4DrWQ7oqFii8L+VoLAjFyKdHgxYGMpSCybwd729tKQ51ReWxRqc +SzV5nmbNRQcHpw85TxVw2kw5aStmTSARfiKMS2p9FMfqSwQCq7oTHe7c7dXd424ihv0 5Yq8fusZaYVxerqyjhkRRKdqtIjqmqNEh2FtVbXUlWoF8vrN+Zu/NewPfXBqX40B+gD8 NgFQ2rxzjL20lpuTqPNRQ427CRFuauHtUGzav0AA4OLwq6Dp3zAc8flnCW3NOFr1ENoT pDtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716264196; x=1716868996; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=spLqBETeC+qWfSJg+4pVx1XrstM8E2ZSksh7ykN8FD8=; b=QGDvWq2mFKYxaB2zIS2dQFyPlwp9vamzCHXRYff0mvc23Hp4+U822DobR71+cWsuVp HXIlUEj055Y1tbnWctR0GB0KvvKOzOPeOiFCONf6w5yiSVFCf37Q2PP4kJvoI8Yk8K8T pm0ihRjCvyL/MPStneVpRA74xrAZ4OPA7c+hK05TmCiBA2N3XH0bWFtxD+5dAhLvc6cr 7aCmU2FNmQSFtqES/m32h7UQuiGZ02+ApBYcGJDyKiJ7Mpr+ncWKXBwen0OFrKF2vwfN Zm4gOhwdU2hJ0dqxNdLBpMBDFfL7n+fKs7IqtzbrAYeoJ/rRm5gPcXvJgIHM4W/RvaM/ 0ZFw== X-Forwarded-Encrypted: i=1; AJvYcCU6TIIuikyNmhBAX2EL2aXd8nt7b/e5fY92xZS138/iOhuAZTu0+iSjbtCyo9RT60lWLvSIFs1S81xeQThbo2Hd5zo= X-Gm-Message-State: AOJu0YxDpgIgSJ0APC7VtOCvUKxAMH12f79Gf3EheI5HGv8UinJDTdOk cKNnnOX5YE38xsXSQDS09tzxEPJcQ2raJZ1qiQR+yjuSr3EiQ3Ad X-Google-Smtp-Source: AGHT+IGliB0CTAHaxPyiVxnLz68yFNjKn75clHMn+AefFU3zI1HVriRKt3gDwUekPuWH6LbqIsZZug== X-Received: by 2002:a17:902:d4c6:b0:1f2:f043:774e with SMTP id d9443c01a7336-1f2f0437a36mr127361215ad.25.1716264196190; Mon, 20 May 2024 21:03:16 -0700 (PDT) Received: from LancedeMBP.lan.lan ([2403:2c80:6::302d]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1ef0bad62bfsm211683945ad.74.2024.05.20.21.03.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 May 2024 21:03:15 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: willy@infradead.org, sj@kernel.org, baolin.wang@linux.alibaba.com, maskray@google.com, ziy@nvidia.com, ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, libang.li@antgroup.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH v6 0/3] Reclaim lazyfree THP without splitting Date: Tue, 21 May 2024 12:02:41 +0800 Message-Id: <20240521040244.48760-1-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 MIME-Version: 1.0 X-Rspamd-Queue-Id: 86ABC20010 X-Stat-Signature: czphxo3ow48ygtcxx93xhcabhoz17s5b X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1716264197-728660 X-HE-Meta: U2FsdGVkX1/6G0DD6WXDSlguk4bW5xrgR0VwEhdNjVI0EpmwSATui22Fr/8DKzY+HCj417W2awjaOv54oiGAGEvguReHi2ZWnpihT4+ZTy/+U4IjNcmRFzi1iN5BxKJKcPdOERSLtmnZvxBnR63Bork0UGt5Qxvzc1cnzmPFINJXBAzWvGzDmudkPA2PHt029obGhSqqmlpY7mGKxQ+8es6YaVehKCaQiqEYyYG47l1de6di2S2hwxqVYITSkodZagx5MskeNQUzzKKbe9uCZrDDBDc1a5866U+EhB7DqOhC6MLvCwurEry/cx/W9e0wpkjOBtvBPRhYR8Bnt5jS+Pg1o25J/Lndj2lhfSneb7qyB4qIb+CYaWLJPeUVZ9jJHUoQ6LF9pFLADr4mOc/ZHijmKWtS7Q4zFYn9rxyceiHayJEfychkiIYkByxYtO3pATGLZY91i2WM/2/tI5CmD7WvT7tpPMV0xqJ+t6/s2K/RgKxGBFoTssyFB0yxVWEt2g/K9EW+F7+v2NrWCi/9dKdxdvr0k4dAIlizB15TucAmh6eGJn18jZEco8C5Pt8xQenkdOA5oJNrCe/DOv9upPmkFp9tx5gPM6uyQSJ4zQ5OoYscBENa5Kox7aFXQUtADyzpOL+kShCuKHd1YFlofB/JZLOU+W92a1HzPS4laRXwKtNY3HPrdSE+t9HI0k1cDxdeVYR1pTClwERnvSX9oW6KbK4BOi74hX6ML5fn181okPU+wssIVYr5/1H0mVYrS7fViCgPVbpr0kyKVMCkG4KPlpdLZK/Da0i5TpzViAVuGwz9IClQ5jz6EsmtohUDLIDCu2he7UgyTbyDAXkqIa1J+2weR6az39dSKj3RNQinJ9FjIHW/jYqYH/ldZB7mi0lfTbqbkhWHSQL8wGrAAwOjLIne8RWK0r7Y85hxhKzHXVqTjeErmLhpSeJegouekNBecyzKx3JDFrL7mP2 ghKXdoVP diQOFMd4roexK2JehoLvypK+MZs6UdRnOsdXhIzO0m9/tuLl0csiUIBk/VfO3UNwnpXkh+Gyj7UQ9NUBhRr489qs6bUBV/TksBqAGGAjAWDn7awFwlgxVPojLl4Uxw4r2TJz/hZqUxDclUWfG+zDLLYxunKrW1PcxRzcqg4ondQdEdp+Kj+tVi0HrqZXaPojb6ZQ6zbyoA27YtCgXIU/bl2yK0+ANqXOOkFTN1gq/Xo7iXb2+DE/no3j+UndQZjyWl8sLE5SQ+ZUhJc8qRTogDkr0tCUPbjkipTWJpBBfjuNoMb4zlWXg4iUuqmFHycRrTOYIiiOa6MYvYULw95YO3MYbM2Ctz0TKDXLSIbN5AbV3zNOSGzx3staS/imh9IrNTULOz2Pt7N02o7ZgjOJI38QwKkbUS7siqkrbgC+MZCSHZ2beVQ3fqPmaWoPlDVb8zCbkO6sIn8yc7OfpYuKRJhKzbqxAPqUR+kIEQczD2Z+wtgaVJPyd5d50VxC4/sb0+dy6AVW7UTI0KMw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi all, This series adds support for reclaiming PMD-mapped THP marked as lazyfree without needing to first split the large folio via split_huge_pmd_address(). When the user no longer requires the pages, they would use madvise(MADV_FREE) to mark the pages as lazy free. Subsequently, they typically would not re-write to that memory again. During memory reclaim, if we detect that the large folio and its PMD are both still marked as clean and there are no unexpected references(such as GUP), so we can just discard the memory lazily, improving the efficiency of memory reclamation in this case. Performance Testing =================== On an Intel i5 CPU, reclaiming 1GiB of lazyfree THPs using mem_cgroup_force_empty() results in the following runtimes in seconds (shorter is better): -------------------------------------------- | Old | New | Change | -------------------------------------------- | 0.683426 | 0.049197 | -92.80% | -------------------------------------------- --- Changes since v5 [5] ==================== - mm/rmap: remove duplicated exit code in pagewalk loop - Pick RB from Baolin Wang - thanks! - mm/mlock: check for THP missing the mlock in try_to_unmap_one() - Merge this patch into patch 2 (per Baolin Wang) - mm/vmscan: avoid split lazyfree THP during shrink_folio_list() - Mark a folio as being backed by swap space if the folio or its PMD was redirtied (per Baolin Wang) - Use pmdp_huge_clear_flush() to get and flush a PMD entry (per Baolin Wang) Changes since v4 [4] ==================== - mm/rmap: remove duplicated exit code in pagewalk loop - Pick RB from Zi Yan - thanks! - mm/rmap: integrate PMD-mapped folio splitting into pagewalk loop - Remove the redundant alignment (per Baolin Wang) - Set pvmw.ptl to NULL after unlocking the PTL (per Baolin Wang) - mm/mlock: check for THP missing the mlock in try_to_unmap_one() - Check whether the mlock of PMD-mapped THP was missed (suggested by Baolin Wang) - mm/vmscan: avoid split lazyfree THP during shrink_folio_list() - No need to check the TTU_SPLIT_HUGE_PMD flag for unmap_huge_pmd_locked() (per Zi Yan) - Drain the local mlock batch after folio_remove_rmap_pmd() (per Baolin Wang) Changes since v3 [3] ==================== - mm/rmap: integrate PMD-mapped folio splitting into pagewalk loop - Resolve compilation errors by handling the case where CONFIG_PGTABLE_HAS_HUGE_LEAVES is undefined (thanks to SeongJae Park) - mm/vmscan: avoid split lazyfree THP during shrink_folio_list() - Remove the unnecessary conditional compilation directives (thanks to Barry Song) - Resolve compilation errors due to undefined references to unmap_huge_pmd_locked and split_huge_pmd_locked (thanks to Barry) Changes since v2 [2] ==================== - Update the changelog (thanks to David Hildenbrand) - Support try_to_unmap_one() to unmap PMD-mapped folios (thanks a lot to David Hildenbrand and Zi Yan) Changes since v1 [1] ==================== - Update the changelog - Follow the exact same logic as in try_to_unmap_one() (per David Hildenbrand) - Remove the extra code from rmap.c (per Matthew Wilcox) [1] https://lore.kernel.org/linux-mm/20240417141111.77855-1-ioworker0@gmail.com [2] https://lore.kernel.org/linux-mm/20240422055213.60231-1-ioworker0@gmail.com [3] https://lore.kernel.org/linux-mm/20240429132308.38794-1-ioworker0@gmail.com [4] https://lore.kernel.org/linux-mm/20240501042700.83974-1-ioworker0@gmail.com [5] https://lore.kernel.org/linux-mm/20240513074712.7608-1-ioworker0@gmail.com Lance Yang (3): mm/rmap: remove duplicated exit code in pagewalk loop mm/rmap: integrate PMD-mapped folio splitting into pagewalk loop mm/vmscan: avoid split lazyfree THP during shrink_folio_list() include/linux/huge_mm.h | 15 +++++ mm/huge_memory.c | 122 +++++++++++++++++++++++++++++++++------- mm/rmap.c | 83 ++++++++++++++++----------- 3 files changed, 167 insertions(+), 53 deletions(-)