From patchwork Sat Apr 13 00:22:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Yang X-Patchwork-Id: 13628385 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0208EC4345F for ; Sat, 13 Apr 2024 00:22:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 158FF6B007B; Fri, 12 Apr 2024 20:22:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 10A046B0082; Fri, 12 Apr 2024 20:22:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F11FC6B0083; Fri, 12 Apr 2024 20:22:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D2BD16B007B for ; Fri, 12 Apr 2024 20:22:45 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4B3EF810B8 for ; Sat, 13 Apr 2024 00:22:45 +0000 (UTC) X-FDA: 82002607890.26.D4B9BB9 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf14.hostedemail.com (Postfix) with ESMTP id 7F17E10000B for ; Sat, 13 Apr 2024 00:22:43 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ZiJ1Yhpr; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712967763; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=H56jlVSkImjG5Con1Y1IwJCEpB0vQXFHrk5E2u8OB/Q=; b=NjlrPwLTps7m9/Rvmkewr9TPBb65SNkrJwWCP5aBxkP94Md9bNrNwRGVxt4d9aUm8sqXhj IdKsFKXKtcLSXzMDU6QlGlmwhGY8fTxfOz9VYIzJZ+BoSSJE1h/BJyHvC6A6BWs5VMURcV 3agROSe6edWBnFx7ufsLHgM9xY2R3bw= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ZiJ1Yhpr; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712967763; a=rsa-sha256; cv=none; b=IRKpkkkVLIp2gBQd3qH2bSdG6gBxufPvFpngz+xIX9WJ82ZdgRW1jFR4HiIB96WY2whb5w 5vyK87fHtgKwX+UF7CyYqjpxpkKaI4PMSA0Mt65ninJSk48CG8630Yuw7SEmEaE0kbA/lk Y/hTR1+xO9LYfh1sLuiFG6nmDDGvTQQ= Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-1e4c4fb6af3so8213955ad.0 for ; Fri, 12 Apr 2024 17:22:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712967762; x=1713572562; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=H56jlVSkImjG5Con1Y1IwJCEpB0vQXFHrk5E2u8OB/Q=; b=ZiJ1YhprgeK8uAPh+Ip7PMXxMCmEg41dmk1aWjhz+57V7dp2bJXNqAdU7PLP1Dk7n9 /4ScZNxI/BFa5D6dVCU+Aro333dBImfInHbIfOe6KgrI1qtGog4qL3UCOUQQ1ipyFig/ RF5sxFgE5g0bY+LZxBUUzkr34uKusA2oWwy6PwFWBPvlYw6/6bjZD7beyqw63b+e5WKE hWee0CAiRxjSWQk0k4J5F4KkwtOUsvww3NwG/8Qknuzzz5G1fRc9gEuL9u8AQ6Z90ewe 8c/mSMtEIasW02cETKFySScl6obZROSPGHfrvKaL84xUSBzB8n4HZUEefjolZCLXNM9p zUDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712967762; x=1713572562; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=H56jlVSkImjG5Con1Y1IwJCEpB0vQXFHrk5E2u8OB/Q=; b=Eb9MDxrkA8RLAFgF818sLdBy8AthgFDLXtbf4HscL5m/RKzM5mu0RlhiVpFsEGg8jK CwKk1bUu1B7Td7PijxJ/BKHGxnXIe37csM7w75099MMv3hyTQtJGxtVET1l/rd08LNbI ZOf60FH9HkLlcJ/ugMABjEQQcDjvY9Ki1igY2K/jVrgVoXrvfX8L/NC5o2oWIu0k4FMH Zj2obYq9SVm1clfN3bKSn495w9yRbIO+dadCCXmmw68iDBmzB0xQnglB+zF5aUo6BJ0X eXubXIRmAo7xIBS5qeAa/G/1K4aMeexMtkPfj2JljbeHBpEvBZ2LEb74okNLi2ES86JJ BM6A== X-Forwarded-Encrypted: i=1; AJvYcCWFA+JZFrS8WkG2F2cMDUQO7NTdQ+ocSdwuyrE7lltLwO3Xnh7+SCjOIarpjrJ7YQqgvHnadrc5GIimms0CvLg6kvg= X-Gm-Message-State: AOJu0Yy7pLV2oJzyhzDR+3ESIdhRTBvgKTUl1yEfEWb7ZsbkF9oReqJr flbpVyd2FZtK3o843GtRnS4b8RCqHMisZ9PSVjCtAofmbg/ThgT6 X-Google-Smtp-Source: AGHT+IHDPb7KYH6+g22qQLutSNBT3fGe7ZKFU6yWCJMO/AHYf1F2UvmYB+FQBtiQwybK2ATAoW47bA== X-Received: by 2002:a17:902:cec4:b0:1e4:55d8:e15f with SMTP id d4-20020a170902cec400b001e455d8e15fmr9761137plg.32.1712967762227; Fri, 12 Apr 2024 17:22:42 -0700 (PDT) Received: from LancedeMBP.lan ([112.10.225.217]) by smtp.gmail.com with ESMTPSA id a17-20020a170902ee9100b001e2a4ac7bf9sm3569618pld.111.2024.04.12.17.22.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 12 Apr 2024 17:22:41 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: zokeefe@google.com, ryan.roberts@arm.com, 21cnbao@gmail.com, shy828301@gmail.com, david@redhat.com, mhocko@suse.com, fengwei.yin@intel.com, xiehuan09@gmail.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH v6 0/2] mm/madvise: enhance lazyfreeing with mTHP in madvise_free Date: Sat, 13 Apr 2024 08:22:17 +0800 Message-Id: <20240413002219.71246-1-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 MIME-Version: 1.0 X-Rspamd-Queue-Id: 7F17E10000B X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: gi96ptmy4i97hprj49dnb5xjyui4m7yn X-HE-Tag: 1712967763-118373 X-HE-Meta: U2FsdGVkX19Gf91+9Jc5f5HsEBu1MNCPwjwH2vJNaDeRZ78zlM5vkXjSEXaZePiTOs+oAjLzm9uamm1HsoMi/bX8IeR0m3Fbnok9KWHhiJZ1qY1AT9ENd0L1HnwpCaTpg6KWr4LJ1Rl1YKp+cMbWExGnVLftRlbQ3CrI0OomnB0+eHBrwDRAPSV3wcgdSRFSkMa+pYs0cfan7+5k6Y+j3xXvtLf59HAUOwO4DECgV/V6m3P+E1z7729Y8sRwdncdBjjxE9DFTOoB5nHc2Dq3vT8pZROaEW+DO70ygoz6gL+/5cbfP9xNWBWizDXqbUqw9GmV926GGISQCp415ApGn6mtZehu9KqK2Z6XRQbQRphtaQt7/UMNn+rshYEWXSXkqukh3i+qGHc3FjfaliSQJZUArn4y0vjwoNuw32EbWTb/sTFhqi2qh6uj4R4W0e2bmuM8lkaPYvgpih/Avtdqolu6jN6FaBOHXjvhNfYeSTvejHUh5arGse5rs778F2ZDASlDA3eMvcbpnADLgbRqiF4nWW1+Uf6O9Jvn/pYYHPUEAR4eNooJ9F9xe3S2eXdrPUDJsKq2rkKRvIHPhCLXYTz9WVZ2rnkDIy+Aqm6NG5YPxUWDDalzF3DFlwW8hB/bLjYOX7xkgjBs2YAFBZQ7oFygBGLWsHQmGLI+narB4ye9mlAmzHXCGWI8HAlhJZuadlieOpEpLWlG6xYDI0kNgDiXUYW0QimGiBWekbGsqpZMdQcmCMrr4bxeaY0yiXDWqNby4PgT/xUPjdyQyu1qnKmbNH5w0zppXpEjan6wuWXvEmqGZxl31aFeCEx3oeAcJOEYXA5mtAYwVQZClcrqHwQI2GQtYAtc34O2hm4VFrqLq2z+/EJHm2GFySGiEHI22AFdgXYYprKdv4xIaPoBz+XLtR9jPZTYW/FruuY8IofTKPEn7aQIf4uek3L2DlrtlDK+ghtqTb2gXeC5Ng8 oIWbewiK Ygr5cxn/MRvz3ta/S1bBvFgIcCILzpzImNwp3UwwvJQUX6ADymmg9DGTkleqrGcOiedqqSYhsh4OE2yv6a9oePPC4BYG644Si7osdvTaluYKz+5v5qlzfOEsrRqQYNGBT0E7H86oWaHFLFgea/tLEyXjR2+oqHUIW13Jx2Agi7XYtE3eDbxe9y+gdNUlpIsYUUU2X7Pz7RtyjBLRxrhHmnwOlAL1uswGxupgKQC+y9XTUiil3AmAbAZ4JzjFZEPJ/8h8IuKqtjk1irW+vKIAePesE8FBapF3d89pwBDViJghstK9el1bJgwe6YdkUWA2M8NAacVKQlSBYy46yIFv4WmeL0VYhSAxD50/3joDqfLn8LUg9SqAj9MT/556DCS7FIbIlp3Q8I9oM0oEGbE1MeEt3Bfkul/is1dfM45HY33ubiHvXrDm1AxofYW/x6/zB8HN0VrJDNk2Ks6SVICU/AF6RLypetwvXdHf1fn0B7mllLAGZ+hursUJ+X8S5KPl23uf0Z57/WcNfFyGG3FDUCfdTF6XeO4dsNIeS6EVAI4+Yt4Y= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi All, This patchset adds support for lazyfreeing multi-size THP (mTHP) without needing to first split the large folio via split_folio(). However, we still need to split a large folio that is not fully mapped within the target range. If a large folio is locked or shared, or if we fail to split it, we just leave it in place and advance to the next PTE in the range. But note that the behavior is changed; previously, any failure of this sort would cause the entire operation to give up. As large folios become more common, sticking to the old way could result in wasted opportunities. Performance Testing =================== On an Intel I5 CPU, lazyfreeing a 1GiB VMA backed by PTE-mapped folios of the same size results in the following runtimes for madvise(MADV_FREE) in seconds (shorter is better): Folio Size | Old | New | Change ------------------------------------------ 4KiB | 0.590251 | 0.590259 | 0% 16KiB | 2.990447 | 0.185655 | -94% 32KiB | 2.547831 | 0.104870 | -95% 64KiB | 2.457796 | 0.052812 | -97% 128KiB | 2.281034 | 0.032777 | -99% 256KiB | 2.230387 | 0.017496 | -99% 512KiB | 2.189106 | 0.010781 | -99% 1024KiB | 2.183949 | 0.007753 | -99% 2048KiB | 0.002799 | 0.002804 | 0% --- This patchset applies against mm-unstable (37a4ecbf36cb). The performance numbers are from v2. I did a quick benchmark run of v6 and nothing significantly changed. Changes since v5 [5] ==================== - Convert mkold_ptes() to clear_young_dirty_ptes() (per Ryan Roberts) - Use the __bitwise flags as the input for clear_young_dirty_ptes() (per David Hildenbrand) - Follow the pattern already established by the original code (per Ryan Roberts) Changes since v4 [4] ==================== - The first patch implements the MADV_FREE change and introduces mkold_clean_ptes() with a generic implementation. The second patch specializes mkold_clean_ptes() for arm64, providing a performance boost specific to arm64 (per Ryan Roberts) - Drop the full parameter and call ptep_get_and_clear() in mkold_clean_ptes() (per Ryan Roberts) - Keep the previous behavior that avoids locking the folio if it wasn't in the swapcache or if it wasn't dirty (per Ryan Roberts) Changes since v3 [3] ==================== - Rename refresh_full_ptes -> mkold_clean_ptes (per Ryan Roberts) - Override mkold_clean_ptes() for arm64 to make it faster (per Ryan Roberts) - Update the changelog Changes since v2 [2] ==================== - Only skip all the PTEs for nr_pages when the number of batched PTEs matches nr_pages (per Barry Song) - Change folio_pte_batch() to consume an optional *any_dirty and *any_young function (per David Hildenbrand) - Move the ptep_get_and_clear_full() loop into refresh_full_ptes() (per David Hildenbrand) - Follow a similar pattern for madvise_free_pte_range() (per Ryan Roberts) Changes since v1 [1] ==================== - Update the performance numbers - Update the changelog (per Ryan Roberts) - Check the COW folio (per Yin Fengwei) - Check if we are mapping all subpages (per Barry Song, David Hildenbrand, Ryan Roberts) [1] https://lore.kernel.org/linux-mm/20240225123215.86503-1-ioworker0@gmail.com [2] https://lore.kernel.org/linux-mm/20240307061425.21013-1-ioworker0@gmail.com [3] https://lore.kernel.org/linux-mm/20240316102952.39233-1-ioworker0@gmail.com [4] https://lore.kernel.org/linux-mm/20240402124029.47846-1-ioworker0@gmail.com [5] https://lore.kernel.org/linux-mm/20240408042437.10951-1-ioworker0@gmail.com Thanks, Lance Lance Yang (2): mm/arm64: override clear_young_dirty_ptes() batch helper mm/madvise: optimize lazyfreeing with mTHP in madvise_free arch/arm64/include/asm/pgtable.h | 37 ++++++++++++++++++++++ arch/arm64/mm/contpte.c | 28 +++++++++++++++++ include/linux/mm_types.h | 9 ++++++ include/linux/pgtable.h | 42 +++++++++++++++++++++++++ mm/internal.h | 12 +++++-- mm/madvise.c | 147 +++++++++++++++++++++++++++++---------- mm/memory.c | 4 +-- 7 files changed, 212 insertions(+), 67 deletions(-)