From patchwork Tue Apr 2 12:40:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Yang X-Patchwork-Id: 13613886 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 203CFCD128D for ; Tue, 2 Apr 2024 12:40:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 85A546B0085; Tue, 2 Apr 2024 08:40:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 80B8B6B0088; Tue, 2 Apr 2024 08:40:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D2876B0089; Tue, 2 Apr 2024 08:40:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4EEBE6B0085 for ; Tue, 2 Apr 2024 08:40:42 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D5522408DA for ; Tue, 2 Apr 2024 12:40:41 +0000 (UTC) X-FDA: 81964550682.05.607E66C Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by imf04.hostedemail.com (Postfix) with ESMTP id 09A8E4001A for ; Tue, 2 Apr 2024 12:40:39 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="WU5m/qXZ"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf04.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.210.173 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712061640; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=z5G6GpIHkbgDMvuomt94iyreeKkPmpGXaASUVbfOhgI=; b=tsuqYqcjVmSul0T/sL2s1opJ2ml24HBeDB64GqOGAYIqEh+xEg0wv48i548veGijr7rdlf OsHK3dFEThPBqhgfSmi90TPNNTjoXHN30f28//lPhcqIMi2A6hJHjtrnw33iVWL4fzA570 sVDPAiHSLwQoTU2jBk/RLXMydzPHF+w= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="WU5m/qXZ"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf04.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.210.173 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712061640; a=rsa-sha256; cv=none; b=GO591L0Xk9yZ1sIG0BTFIoLX8DaCaR+00jQkCwHXLuKd3kk3vbkLaHIxT0Qt8GR24bPAm1 h4Hpl2Kj+/jYlOIwHN+NkpatY6BTzmIogLDt9XAMu3nVI6Fu6dtj+S3qElime3PI+kkW0n 6tRyBH//vA5OmRDUWb/x1Ou1rNStqpk= Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-6e6ca2ac094so4669227b3a.0 for ; Tue, 02 Apr 2024 05:40:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712061638; x=1712666438; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=z5G6GpIHkbgDMvuomt94iyreeKkPmpGXaASUVbfOhgI=; b=WU5m/qXZ7fOnE1VIBgdaxJGkDvkLhprI/6rl3myidBxJTRL/hVPLwLr+nOt99XFPG5 J1JZ75ARZ587UmSdgBKYzoe83PvfAeuIVD/DVH1uYfClF3w7saT934Fi7UyTIoSwhVm7 hoeaK5ChVYhgqrFa73L4vFTMnKIySGbgv6GNLkMxuerxhVPBhoZxWj1mhZZQNOdhVjGW jMNMg0n+l2lohHCPvtHG4Gk9V3ukg6YHQYyDL6jGN/1KR8pn+3siLKtKsCwypfXoJfXt wrrJKpuXVVxjiy9XsABKnJ5e91IyhtoOHhZmzrQ6A1yvojj7DxqC2AG3g9fmcWRlcpeE QN3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712061638; x=1712666438; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=z5G6GpIHkbgDMvuomt94iyreeKkPmpGXaASUVbfOhgI=; b=CBXdNpd61YnGYyjWGyPN6rx1CN+sbkkcavVh3d4XxN32ZL4XhQb9DK8bN5lJODlKaL OXLIPTbTBcpc8C/sUVNCwE1whEaSVUY+x/VjmMhabbrglqr4ny6U3gnW+YlWhwAdNqCE y8AqERSpSLfB/Zy1fJBus57knFlfPV8/jMim4hQDJFM+xmwPKpCZPwZGUKiKsFRPUoJc 6kczQ2K7JfhRV1p5p+IgCKIt19JG6aTnewg8BI98E3e1KE3anHvrtn9+3Y5+2Ifu2Hy8 Kya2I/38qD7nJEv56YLHSlilqyqtf6NMxEa3MYEI0mfbDOCZlCgejDUdRmawqbWUb3cC bLxw== X-Forwarded-Encrypted: i=1; AJvYcCWPZ1bKCbxIwN3XQt5PN9InYt4q7EpCyooPq1CoxGfhNREaoIbYec7PmpSPxl4QrTED0h8u3FuNgfBPFyi0VC/MBRc= X-Gm-Message-State: AOJu0YwJPY0QoDglDKA2liwpJtlNNwt42bfRNbpYsZEiWG0qGnZTMS/h c/w6q0RAvjp2Ko/EDTA7kGlnRBqSXifqzf1+kyw5CKo7EiDJCc8Z X-Google-Smtp-Source: AGHT+IFU6JVlTIGq308kE/gMP4m+lWmDGW6KTg9wTf197WtftPYl4LndEzB7/UfG2/2McnwmBsYTvQ== X-Received: by 2002:a05:6a20:3947:b0:1a5:698e:505c with SMTP id r7-20020a056a20394700b001a5698e505cmr11809339pzg.10.1712061638605; Tue, 02 Apr 2024 05:40:38 -0700 (PDT) Received: from LancedeMBP.lan ([112.10.225.9]) by smtp.gmail.com with ESMTPSA id h5-20020a62b405000000b006e664031f10sm9708232pfn.51.2024.04.02.05.40.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Apr 2024 05:40:38 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: zokeefe@google.com, ryan.roberts@arm.com, 21cnbao@gmail.com, shy828301@gmail.com, david@redhat.com, mhocko@suse.com, fengwei.yin@intel.com, xiehuan09@gmail.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH v4 0/2] mm/madvise: enhance lazyfreeing with mTHP in madvise_free Date: Tue, 2 Apr 2024 20:40:27 +0800 Message-Id: <20240402124029.47846-1-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 MIME-Version: 1.0 X-Rspamd-Queue-Id: 09A8E4001A X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: uktj99wnfrdyft4ja54icn5738gz9bx6 X-HE-Tag: 1712061639-23028 X-HE-Meta: U2FsdGVkX1/sz/Lgd9pwONLo/mTe3OiGtC836PMGLYCYSL6VakbEIgJ/RQo3IFZrsP33WByUknEZ5hyFBDpxtYtMjFogvqXZp1dwdVw4wuHemukhZWpRjoLBlmQTRi3QKFG9XITbfdngpi44uBUv/Z1hCQU9kqgYASmtg9OP9n5Jwhrw3RR7g5Zuubd+qDF7rRSyZSXL1sTtDDLkuGFkqB10CSHAqXt6c+LdHNjRdKbI0lpTknmXS5e0Oslx40nL46dqsc5wxC/0yCvEIWa4WBSce9iI2XzI9vLlrcGdq2T93P1I667mdRwTZ9OVwCP7+jYOUhTkysxhT/D2PSWRZjBsI4gILeGVcX0ptV+yGbyd+oytgoNkXqtapHY0tIMsjNeJykX6jJDupHt7ghwEHDjsk/0ye4TTeHSErUXCqVwM2C08MPINPXc3zGApcrkleMKK8AX9TFwcbIKth6vAp8nx4Hg6xj8q2D2fuVOC58aD0i1jyQDTEdxL0kYwZkEGZKWPIfgaf38x5omS7yqwkG/rM060oUFBKDmdjN9ORY6EtxkA1QVQ0fIl5hvcBO0dtLIAZj+iGU8IDksAbU1ZUl6lfHxI5iOVGwVirEBdKVfHTa0ooo835xfSLSmQnl9SYwLPC+XDI01gZUOv2BBUMpTBfsc+hbcHjQGT9bT2hXWecG2xWEvv2iuUQ6675H2EhRRmFXZoUHzZkvSuVGLbAWm5Tz1aga86zrG57zZvRzo8UQU+eBC25FsQzBcvgiZfS0ItgZ5SqVv1+MskbR797EvltNeLgnDYWSRKLB4oa202w8Nry5z1jhYmq0MI4lfcrC1pYDdJKdtSdS3vrCi8TIA/8hZP0xeMDvx333LMky12QFrkLrEP9ZDUGGcvFwaw3bGsnDEb0Z/8ZJXuDrzsj04qaPFOlRKigW0LT684nCYB0mOfqxwzPYPXZKAqXmT+j9wBAnb1vNqSIYrd+Be QYAFhoHU rG9U2JTmT5kuj37CCHd/RDjqSlCrideVPcEqFC0tRLi+LzdSyIpKUpmDEPRip2aEBIkymn95+LR/Zw8bi8vciDPjauYU7ZeDpD0LRA7z+k28euMd5Bju+w7Lu6Pw/LbjnnX7pWmDrlHnBU2Ns20NjdP+C4l0kSZa5NMCeW9hfz9KOKUdadhqSaNOhvh4JapRxccTwZrTiqBnQUuBxHI3Hc3mqol4dlG70W+STU1onE/XnQFlor3VBME+s2nFvRsOX6Uuj76O40kTOI7xS7RFxyVRwc5nwR0MZGs0uhlBzE+IOf5VNmXzqE55Lsx6XMSg4lwfanpFF/Yd4yQFoyHuI3fnga3Aka9wdxdBeSxUqqm4ac2WK6SdfUzAhOF589DNZ7Hul1ymoY7GupFFZCIzqvHa11SRdHFTa6bBCxvLxcrwcpp8uWTakbHvF48x8rVQog8HIbunq2IjibmtorkNHTqS8sEStzw+RfQD+Hp0p3zaSfWEcl6ewnP1JHLVBDKqdo45Ce+2MqC5wjrJHAEp1zBy7kZ6g/LoRgqNOwJDK1pZk4Lg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi All, This patchset adds support for lazyfreeing multi-size THP (mTHP) without needing to first split the large folio via split_folio(). However, we still need to split a large folio that is not fully mapped within the target range. If a large folio is locked or shared, or if we fail to split it, we just leave it in place and advance to the next PTE in the range. But note that the behavior is changed; previously, any failure of this sort would cause the entire operation to give up. As large folios become more common, sticking to the old way could result in wasted opportunities. Performance Testing =================== On an Intel I5 CPU, lazyfreeing a 1GiB VMA backed by PTE-mapped folios of the same size results in the following runtimes for madvise(MADV_FREE) in seconds (shorter is better): Folio Size | Old | New | Change ------------------------------------------ 4KiB | 0.590251 | 0.590259 | 0% 16KiB | 2.990447 | 0.185655 | -94% 32KiB | 2.547831 | 0.104870 | -95% 64KiB | 2.457796 | 0.052812 | -97% 128KiB | 2.281034 | 0.032777 | -99% 256KiB | 2.230387 | 0.017496 | -99% 512KiB | 2.189106 | 0.010781 | -99% 1024KiB | 2.183949 | 0.007753 | -99% 2048KiB | 0.002799 | 0.002804 | 0% --- This patchset applies against mm-unstable (d4cd6840d1dc). The performance numbers are from v2. I did a quick benchmark run of v4 and nothing significantly changed. Changes since v3 [3] ==================== - Rename refresh_full_ptes -> mkold_clean_ptes (per Ryan Roberts) - Override mkold_clean_ptes() for arm64 to make it faster (per Ryan Roberts) - Update the changelog Changes since v2 [2] ==================== - Only skip all the PTEs for nr_pages when the number of batched PTEs matches nr_pages (per Barry Song) - Change folio_pte_batch() to consume an optional *any_dirty and *any_young function (per David Hildenbrand) - Move the ptep_get_and_clear_full() loop into refresh_full_ptes() (per David Hildenbrand) - Follow a similar pattern for madvise_free_pte_range() (per Ryan Roberts) Changes since v1 [1] ==================== - Update the performance numbers - Update the changelog (per Ryan Roberts) - Check the COW folio (per Yin Fengwei) - Check if we are mapping all subpages (per Barry Song, David Hildenbrand, Ryan Roberts) [1] https://lore.kernel.org/linux-mm/20240225123215.86503-1-ioworker0@gmail.com [2] https://lore.kernel.org/linux-mm/20240307061425.21013-1-ioworker0@gmail.com [3] https://lore.kernel.org/linux-mm/20240316102952.39233-1-ioworker0@gmail.com Thanks, Lance Lance Yang (2): mm/madvise: introduce mkold_clean_ptes() batch helper mm/madvise: optimize lazyfreeing with mTHP in madvise_free arch/arm64/include/asm/pgtable.h | 36 +++++++++++++++++ arch/arm64/mm/contpte.c | 10 +++++ include/linux/pgtable.h | 30 ++++++++++++++ mm/internal.h | 12 +++++- mm/madvise.c | 147 ++++++++++++++++++++++++++++++++++++-- mm/memory.c | 4 +- 6 files changed, 164 insertions(+), 75 deletions(-)