From patchwork Mon Apr 8 04:24:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Yang X-Patchwork-Id: 13620557 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 579D5C67861 for ; Mon, 8 Apr 2024 04:24:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 621066B0082; Mon, 8 Apr 2024 00:24:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5D1676B0083; Mon, 8 Apr 2024 00:24:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4980E6B0085; Mon, 8 Apr 2024 00:24:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 2B3926B0082 for ; Mon, 8 Apr 2024 00:24:52 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E2925120231 for ; Mon, 8 Apr 2024 04:24:51 +0000 (UTC) X-FDA: 81985073982.05.0E803BB Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by imf18.hostedemail.com (Postfix) with ESMTP id 1FAF11C000D for ; Mon, 8 Apr 2024 04:24:49 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="VizM/Ejh"; spf=pass (imf18.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712550290; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=lvZDaxqmUkHGSXUI9YaQuvWtKQrnJIvEACQeGbEAIqM=; b=M3nynnMIo+aH3csKQ+eHt5WkH0EV11SOl38ub7pK/uej93xw2nfXkjjX2sBT7cSHvfewjV 1kP7NVgoGcmWn/qI/OciTbTSEbZhARUZce5zHNGmc7aXNoJxcCH1j6EUK9xQALdHv81sot HDBjohoPtZ48PQrlK5t66g6tNHl0xBo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712550290; a=rsa-sha256; cv=none; b=4PcrTGvAO+aZdyh/ITjEMTkW1IfDKOJr/0D66HLO8boyRonKrYaOzvdClh38f0/AEjrtlD yG6/oSgukKEgIvkiuKN1KcPDuZKQEKonS3QcPdexsmr9Y1kEIZKj4+LR/YZDGagOpFlJhO YUZJrQbCaQANeOb+oAdZLN2/wRmVKTE= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="VizM/Ejh"; spf=pass (imf18.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-6ecff9df447so2778176b3a.1 for ; Sun, 07 Apr 2024 21:24:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712550289; x=1713155089; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=lvZDaxqmUkHGSXUI9YaQuvWtKQrnJIvEACQeGbEAIqM=; b=VizM/EjhOYu9pUYpRyC2RaPDt2w0HEe853iPahY+ZnipzD8C0ojS/ZZrDo8TEEUMHG qsXDq0n3nZ5b9x3SjpQP66mdjb2FjEdspO3O43dlmnTOg8A2+tA3CP7vrGkVotTUMRJN nkrzS1kXKmF/eEIFyGYTcvUFc+7ytde1wh/VQPIPoVyGcubBnDFXhod8kZMvL1lVRuLz 9EgT0UKszPmoeWcTh4rs9jblj+E8rdYS9u4C0ms8N8MwaGa6mg+TPhpcahMQL9txaNi4 bdLMCkURFTe8dHOR+lvoZioXTQQ3Z1RXD7zZVVjvaoHcIoe2oUKt4TM5busKpQZPfOE6 enYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712550289; x=1713155089; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=lvZDaxqmUkHGSXUI9YaQuvWtKQrnJIvEACQeGbEAIqM=; b=FKff5YDFiQ2xLidhV/Q1KhD20Gz2834+hExbRWTroY5gkcycJ4a0yKwUoPF9v1c2yG ttgXxHs4/2i3VtoJnE+F4BSa+8KQ4/2tbMjwYVOn3lOKIpH45rx4vbafbT+/KzYoYYAj 6x2M3n1zkV/saMmbCsSDoDAkUVPKQ6eDfFiTTHRYX5610qney41pxPQqpWJlTjeemwEy H6yR6cVRwChH8XzMYcLUTn/S2M1DC40BIFXp2MQMD/qdC/Ntk0CUVbyQctnqhvzg70JE vM0Iuar7vKNSACeJ5UDrAgTkHpEBUQbH3u3tN374kGnIiixIJEkuQUg3/Vri/LGwJ5UP N4Dg== X-Forwarded-Encrypted: i=1; AJvYcCVwx1cEHbUoJkcc37I2mCvC9SWSerIUR1KetafhTq/1JNeoNdtdIilQKI7t9Qc54BYA/oLsYJ88uvLfHsJJL49nF4c= X-Gm-Message-State: AOJu0YxPeaxliONtK74TGLd1svpVTHjgy69NSA90JoxaHS6OZ/TElqAG X7D65H1qHp/4qDz9jLF/GPXK5se9gegSbH/q/F/G8Qm+MgDSzLQl X-Google-Smtp-Source: AGHT+IGbZOrCLrZJaHQy67i/FShVBjxhbnrmp61AK8agpPOVcn4KW5h0zlVJ9D+vneksS+P65au0Og== X-Received: by 2002:a05:6a00:810:b0:6ed:5655:a094 with SMTP id m16-20020a056a00081000b006ed5655a094mr37371pfk.32.1712550288579; Sun, 07 Apr 2024 21:24:48 -0700 (PDT) Received: from LancedeMBP.lan ([112.10.225.217]) by smtp.gmail.com with ESMTPSA id p20-20020a056a000b5400b006eab6ac1f83sm5465628pfo.0.2024.04.07.21.24.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 07 Apr 2024 21:24:48 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH v5 0/2] mm/madvise: enhance lazyfreeing with mTHP in madvise_free Date: Mon, 8 Apr 2024 12:24:35 +0800 Message-Id: <20240408042437.10951-1-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 MIME-Version: 1.0 X-Rspamd-Queue-Id: 1FAF11C000D X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 497t8cwmmmpcq9dwygahkq7968dh67wa X-HE-Tag: 1712550289-290266 X-HE-Meta: U2FsdGVkX19l2d9iTDgYWinLf5k+PCPh4wZmLSUg7bzGKw+eYgoeutiIqXHp/ipL3OFzo8sDl7avkPu1wctDFBapTut0CC/zXjfN8hAHP7CoHaisP1VQ14JeXUMCK6jiVANzvM+xxnheaPjpWH3RC4DtLIYKSrOcmiX31nhtoo7UhMYDlr45DjqiAHVzCKpPx5VUar7e0tHrlMQ/nbfhd8Z2kDLP0Rf5Brs4qPdRRpv4taoR+q1+oy/vFTKlQeSfapRFahtk7XyldZBD4VXfJiIxED/ChDSJrrGPXjkc31dIX67v8cD5UH9PP33Wn8JsF9WmYRzUOs1VA6ebA1fz7BD7m7EQDnZLxnQwvRZKFrV+KHJf3fGdu81OQRFy1My+mLNxW7WAcDdn78X1V0t3GCTXkJfODzGTEy+9Hc/ttEIUsoMNoodXXPTRHeuhOCMeYfUGSU98upd0TJoFQIWcvwM57WbzVqY/9HFr3/zZwzbyxZsBER+CWW080ftecyJ86NpeDER3J7XI5jZqwhA+NAlfdpLaWjn9XD2+rQrqr96KSo4uO2Ty8vx2CKTL7cu3xmz0a9Q5vhet0ChmVbEW76MvZfrjAbVYXk8Obro6qNPISAk/RlsOJcA7GC2BUhV1e+tEic7kh6zuYGIPZHdqKq5VMyEmP61FWUKx1gygucKmtJ17PmfuEneS3f64PHyH9FsbSd6BCMvHQw623jdX6PcszvwULtw5IZU8uuGB2ro2aYuWyfDYnUEp7jXLWhFVUoozpYfJ02PLi7vQnhyavWCNuOIuiOu6fa2g9pm/tZYecpJquIf4bB11lQ8ARRyaw1GGmGnNgoePJiUAU24dCG0kHWJKdD1eADWhiYSWeIM0xaj5tDn7kH7ZbSCgfNN1e9BuHfBZ8kwxFdCJUx+SEutF0YBilsfgbASSOluGDuPonbCNAcRq9iLvQTQ+SfTVOLIz1otAk/olfRBaDKm UB+itqpk wTcbSZlxHqkR1U7fwddslLlbH35qGYXvJHCdIddqjWBF3ccQJGphSgauTWYmOOET7yCZP7rfBTQk02kyATHh95a/64/iIApPCGswwYnqPHGdnQarxBcdzcKtfy7OWIACSXPTSzS89Yjn7iuOaCJnnj0bWkjqgV0R9nS00CjLcIQCeq8rKOiwo6ql2Lf2MKlGNmayhRlvh5AoqKVN5znVtQmuTMWrC/x3Ffe4eeMuL80bS3pySI56yHKO5r25pdVhfHscB9FsmRbQ1WZX8CYBXot0KXCYkOGbcu8U8YbpXQcqA7L7oYkDG01cnoRCCA+Hs9peFp12no3Jlem93xBTcTPvvNDDlryG4f+NtaYJoyydzRaMuHLC7TpJiUOn6Ih2HT4bEA2/hbZTKrKEjFMkazKB66wQV3NJNH66Uwt4YkR1PjNPXV2oDRPiRT4Re8/lbRe/rtIU3Aurn+DAeST3JRBcy5P+OyBVUbvKQrGVVjMbmvSY7kqRvgSZ3GWjRjkUS12pJ71hQNMKLcqJNw5Y5YpbhGsuMHeBvQ0OG4UdTe/BGx3c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi All, This patchset adds support for lazyfreeing multi-size THP (mTHP) without needing to first split the large folio via split_folio(). However, we still need to split a large folio that is not fully mapped within the target range. If a large folio is locked or shared, or if we fail to split it, we just leave it in place and advance to the next PTE in the range. But note that the behavior is changed; previously, any failure of this sort would cause the entire operation to give up. As large folios become more common, sticking to the old way could result in wasted opportunities. Performance Testing =================== On an Intel I5 CPU, lazyfreeing a 1GiB VMA backed by PTE-mapped folios of the same size results in the following runtimes for madvise(MADV_FREE) in seconds (shorter is better): Folio Size | Old | New | Change ------------------------------------------ 4KiB | 0.590251 | 0.590259 | 0% 16KiB | 2.990447 | 0.185655 | -94% 32KiB | 2.547831 | 0.104870 | -95% 64KiB | 2.457796 | 0.052812 | -97% 128KiB | 2.281034 | 0.032777 | -99% 256KiB | 2.230387 | 0.017496 | -99% 512KiB | 2.189106 | 0.010781 | -99% 1024KiB | 2.183949 | 0.007753 | -99% 2048KiB | 0.002799 | 0.002804 | 0% --- This patchset applies against mm-unstable (f43b3aae9451). The performance numbers are from v2. I did a quick benchmark run of v5 and nothing significantly changed. Changes since v4 [4] ==================== - The first patch implements the MADV_FREE change and introduces mkold_clean_ptes() with a generic implementation. The second patch specializes mkold_clean_ptes() for arm64, providing a performance boost specific to arm64 (per Ryan Roberts) - Drop the full parameter and call ptep_get_and_clear() in mkold_clean_ptes() (per Ryan Roberts) - Keep the previous behavior that avoids locking the folio if it wasn't in the swapcache or if it wasn't dirty (per Ryan Roberts) Changes since v3 [3] ==================== - Rename refresh_full_ptes -> mkold_clean_ptes (per Ryan Roberts) - Override mkold_clean_ptes() for arm64 to make it faster (per Ryan Roberts) - Update the changelog Changes since v2 [2] ==================== - Only skip all the PTEs for nr_pages when the number of batched PTEs matches nr_pages (per Barry Song) - Change folio_pte_batch() to consume an optional *any_dirty and *any_young function (per David Hildenbrand) - Move the ptep_get_and_clear_full() loop into refresh_full_ptes() (per David Hildenbrand) - Follow a similar pattern for madvise_free_pte_range() (per Ryan Roberts) Changes since v1 [1] ==================== - Update the performance numbers - Update the changelog (per Ryan Roberts) - Check the COW folio (per Yin Fengwei) - Check if we are mapping all subpages (per Barry Song, David Hildenbrand, Ryan Roberts) [1] https://lore.kernel.org/linux-mm/20240225123215.86503-1-ioworker0@gmail.com [2] https://lore.kernel.org/linux-mm/20240307061425.21013-1-ioworker0@gmail.com [3] https://lore.kernel.org/linux-mm/20240316102952.39233-1-ioworker0@gmail.com [4] https://lore.kernel.org/linux-mm/20240402124029.47846-1-ioworker0@gmail.com Thanks, Lance Lance Yang (2): mm/madvise: optimize lazyfreeing with mTHP in madvise_free mm/arm64: override mkold_clean_ptes() batch helper arch/arm64/include/asm/pgtable.h | 57 +++++++++++++++++++++++++++++++++ arch/arm64/mm/contpte.c | 15 +++++++++ include/linux/pgtable.h | 35 ++++++++++++++++++++ mm/internal.h | 12 +++++-- mm/madvise.c | 149 +++++++++++++++++++++++++++++++++++---- mm/memory.c | 4 +-- 6 files changed, 202 insertions(+), 70 deletions(-)