From patchwork Mon Apr 29 13:23:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Yang X-Patchwork-Id: 13646952 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BFB7C4345F for ; Mon, 29 Apr 2024 13:23:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 645AC6B0089; Mon, 29 Apr 2024 09:23:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5F5356B0092; Mon, 29 Apr 2024 09:23:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4BC7C6B0096; Mon, 29 Apr 2024 09:23:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2E5E46B0089 for ; Mon, 29 Apr 2024 09:23:25 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 947861A19F6 for ; Mon, 29 Apr 2024 13:23:24 +0000 (UTC) X-FDA: 82062635928.26.96E3FF6 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by imf11.hostedemail.com (Postfix) with ESMTP id AEC9B4000F for ; Mon, 29 Apr 2024 13:23:22 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=GQBDRk4d; spf=pass (imf11.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714397002; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=AIUyZIi7TzY1sCJM9AY9M3QfemgaOle6/RPmVuxxVxA=; b=EZBGxB2YDrDI9wuCzaAZRcovl1lGg0IAQ86uPH8rxJEwIGfjYVLwrEvVOYeDry5X6xAlv3 sIStgzCfVDlOQfsndFBdio5wvlp6v1xqHXgMZd7Eh1CtnKAJ/2Zb1FHI066MGVYIdtd6kH Ug6XG12oYAZIrJMlw9s5gk/uGoAHrIs= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=GQBDRk4d; spf=pass (imf11.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714397002; a=rsa-sha256; cv=none; b=gK+rtZ1JA2yqL0cZGTA/9DRTp8upBeKkPpOSviMt+cMePiYNmvPfJTJbYfU3keAuhMRlNA ij1Qi744HIsFrQotb779YQSdLttRE5L4DcfsmNeLmbPZY3rkLYlr5jAlV4nWSOm1efVEWd B1erlUwGX+Yha4KNLaMmjua+BHj+SBg= Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-2ad8fb779d2so3681669a91.0 for ; Mon, 29 Apr 2024 06:23:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1714397001; x=1715001801; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=AIUyZIi7TzY1sCJM9AY9M3QfemgaOle6/RPmVuxxVxA=; b=GQBDRk4dM5zKpAX7scChSkDCt08KVpHyn+3o9WLNiByC7+51iYgCwktTvJUxWT4OGX 2jxPTLc3ytlAnb0qSUcxoFNBfxNCMqhTA9ymENs5ZBrx54f/jwchVuOugLPWnMCn/6yQ dilJKgBVZIAy2R1QaH21jkAr85M5Wjev0MPR2hHDJbtTRQU5qPz/aJmfyYv1b98u8rD7 af0f8vsfc0HkZIxXpKfn3y+VT7nNqd1OKzu9lacNl2o4lkp+pXETu7nLFgg8Lb+7qrMT czs/dhIlJY9q6F0xuPJ8YzBcn1oB/CIxW7Eptu7q6BFy3Zl+zPy83MEykQGzfDhayoHQ bQ7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714397001; x=1715001801; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=AIUyZIi7TzY1sCJM9AY9M3QfemgaOle6/RPmVuxxVxA=; b=l7CQfRU/w74wlNceMEWklRJf5VOe30lSkq26lxUjAGV9D5iBZkxB2MeeO7If+0kmTI ABLnV/P7S+K9ozdNa2Zery//4kK9MRWr/yzwqSpcQQQvHhPaP51SDwABuFHLGhmkyQ+9 KcSa4P04CUp5j2/yuy20l1w29uLVYKW3lXqRj6ab73jOC/7YeocpeT1CHIsV3y4bwgHe 5WQri8Fu3b1pYxiPWVSb+VwNKBV/jI80a+ZYMId2F8I+Pd1xLlN9VgjzvAdK18kcGt2j Z7bISieqH+n2WvvsDwuSSxlCfkXOi31B6513HpDRFvsYO3dleQwGxoRMIsxOZo4Oiyw7 UwAw== X-Forwarded-Encrypted: i=1; AJvYcCUyW/UdShYH5eJncrYJt0xXAuetI5ruE92WtSbViSRHHK720tW6jkWrERw5WVnsw9lwkTKacoeQ0Ulc6UdYrB14Eo8= X-Gm-Message-State: AOJu0YydehcDwi/Q53SqNMhW1nAm0X0z3n3MHjwR4afcVV7ktbHTE/6n uKgw/PIp/ZK3N6HQssoVtemvr5tY1RzYfn9TyC8CA8qHaOY8nzFl X-Google-Smtp-Source: AGHT+IGCo9M+a2pMrkha4e/8Rr8qhMrUJ+s7wzyuSxRDLnHms8qMV3zHtJCRRj8F9yPHPGVyB/vniQ== X-Received: by 2002:a17:90a:6c88:b0:2b1:bb1:7061 with SMTP id y8-20020a17090a6c8800b002b10bb17061mr4615812pjj.44.1714397001267; Mon, 29 Apr 2024 06:23:21 -0700 (PDT) Received: from LancedeMBP.lan ([112.10.225.242]) by smtp.gmail.com with ESMTPSA id pa5-20020a17090b264500b002b113ad5f10sm3562203pjb.12.2024.04.29.06.23.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Apr 2024 06:23:20 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: willy@infradead.org, maskray@google.com, ziy@nvidia.com, ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, libang.li@antgroup.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH v3 0/3] Reclaim lazyfree THP without splitting Date: Mon, 29 Apr 2024 21:23:05 +0800 Message-Id: <20240429132308.38794-1-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 MIME-Version: 1.0 X-Stat-Signature: udj1gcjzbbit416ojhspyx9wk37mjefs X-Rspamd-Queue-Id: AEC9B4000F X-Rspamd-Server: rspam10 X-Rspam-User: X-HE-Tag: 1714397002-198592 X-HE-Meta: U2FsdGVkX191mdKRsYndzyUMoWh7c+/QFKyKGXQRD5cxJfbt5feVcnZCYTx05unxQRtpQ7dbFK1OfYsQsa+tGeACIzVPT/GuiPMchHvgb7Iu/0RE17MrMiEgdqUyV9UVvi5TWxnNrYiKN0MKadsNav+KZOBCq7RIyZJNB3Z01qHAl/ZDUaXAeqdzjJw5xIkE2bWKpEUsD72NhLqTq2Tfz6z1T0UKg09oIaliHUPVRrw64JWcZe9FLLXxRdXRXOJrxvXWG7aUu4okFl+IwQOv137/WSCjFu4mzREHYPDfq5efo3Wu/3T7umMWrY9pzV8AmfJNW2hxEisnr0964GoBfPGjzb2SqBRnUGMAF5dGcjEN55b3h0gmv10ETI/ZhC3neq4z6tXdB7nOjQs6Q0ypIqo20NpDoazyKF1ITNeCjHWvT7eXBUPTAHoOPqq9u4zsdhJnfc9VPD3J+HAHk6+u44vPXcM2jsxkFXUMBDPL65xQXrHfapiUKexYXXjZkmrfg9afyC4SyJBw3wdNnIqelDvAuC+G0hd20SenllpPqY3Zj8BwnXvtN4AElwUPdH5TrPqeIbjfdvAX46HO5q1cORPF32o47uY5aQ27J0Zwk+yAYjO0325BSAY/hjr1WLWAx9lS9iSA/YacXIH8JUVnXFRvMfJjO9d5Hq+fEEtd/RauaiBErQujAUPBVJKM0/aW/RtkDM1DOS6S5hzLfwLvco5NSZmGBUKNXy1AcLQTnyz1uf1UBVnoWG4zTMce5ebiYPRwMTawcoxKx5pj3gRMb0Ohg/fwtp9YPHNiE088FUl40cu5XtPAPRuLDaXnVJ2jn60wO5tF+FhYgfDpUUKWHUa4w3dQ3Fb3LCJiAZus8fN0iIXwpfp2RPGbPuFQ0wDsfVhWejSxDbuLButiBg+IGp/7M1eYHauE5RCENEsCattkQlbw/yzVDKuofEYdy+wHdrcG87NkeRFy8UfUHSH 4M6nwxRv Va+tzcpmDyKbscYX9NyFGTU4dPSKag6Xj5t9CIaLyOZGSxFIGgAeT+rDk/qLRxWWs/Hyhp59msNvieCRc+EgxcQSl8WjCdO6wNrx8Kafc6PjqeNYw/4y8r8giXg2Kmc0qMKwjrjcjmQLV/Erq6xHLKsMDC+o/uw6Cd3F57QJUWFFrCY/0GfDrsFGbtHGkwJvl0XMO3IpSmNwS/5WWWCGcqsw61NF7LILMuT9Sqhk1tAyM0Kso3yIb9dsEjNwxCHkMSjWVnd+10BzSMX9wTIXYEw4pvnXtko1swE1yvpWIELMTUm6dMFuhrL8jFmE+vmfVVN8wMOp82ZnWXkcLACnJ1H7wAL18W6kdxfODUFYIxsVpN5tVS6yqlNdoSpX4NLlSYdHfCA/JOdwQdlBhj6NvBttBRSertV+VvF/BdMRpwTI6XSEV9OPQllaQ6t0nuJi1ghhoslGtfeF1LsUSI9nEmMdfElu/Do4dte587lmW03v9BbSK2sfahFQEuWuIpK9Xpeci X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi all, This series adds support for reclaiming PMD-mapped THP marked as lazyfree without needing to first split the large folio via split_huge_pmd_address(). When the user no longer requires the pages, they would use madvise(MADV_FREE) to mark the pages as lazy free. Subsequently, they typically would not re-write to that memory again. During memory reclaim, if we detect that the large folio and its PMD are both still marked as clean and there are no unexpected references(such as GUP), so we can just discard the memory lazily, improving the efficiency of memory reclamation in this case. Performance Testing =================== On an Intel i5 CPU, reclaiming 1GiB of lazyfree THPs using mem_cgroup_force_empty() results in the following runtimes in seconds (shorter is better): -------------------------------------------- | Old | New | Change | -------------------------------------------- | 0.683426 | 0.049197 | -92.80% | -------------------------------------------- --- Changes since v2 [2] ==================== - Update the changelog (thanks to David Hildenbrand) - Support try_to_unmap_one() to unmap PMD-mapped folios (thanks a lot to David Hildenbrand and Zi Yan) Changes since v1 [1] ==================== - Update the changelog - Follow the exact same logic as in try_to_unmap_one() (per David Hildenbrand) - Remove the extra code from rmap.c (per Matthew Wilcox) [1] https://lore.kernel.org/linux-mm/20240417141111.77855-1-ioworker0@gmail.com [2] https://lore.kernel.org/linux-mm/20240422055213.60231-1-ioworker0@gmail.com Lance Yang (3): mm/rmap: remove duplicated exit code in pagewalk loop mm/rmap: integrate PMD-mapped folio splitting into pagewalk loop mm/vmscan: avoid split lazyfree THP during shrink_folio_list() include/linux/huge_mm.h | 4 ++ mm/huge_memory.c | 117 +++++++++++++++++++++++++++++++++------- mm/rmap.c | 69 +++++++++++++----------- 3 files changed, 139 insertions(+), 51 deletions(-)