From patchwork Wed Apr 17 14:14:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Yang X-Patchwork-Id: 13633426 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE334C4345F for ; Wed, 17 Apr 2024 14:14:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3B8F66B0082; Wed, 17 Apr 2024 10:14:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 368316B0087; Wed, 17 Apr 2024 10:14:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1E2426B0088; Wed, 17 Apr 2024 10:14:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id F1ADC6B0082 for ; Wed, 17 Apr 2024 10:14:53 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 91CE31C0C15 for ; Wed, 17 Apr 2024 14:14:53 +0000 (UTC) X-FDA: 82019220066.05.D45DA75 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf11.hostedemail.com (Postfix) with ESMTP id C77D84001E for ; Wed, 17 Apr 2024 14:14:51 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=UhsM6qIX; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf11.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713363291; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=SOdqNlpobw5BkPj0KVJli6oAlknzNEgphraB7EENvoU=; b=d5lXPJaE8KOKMrE9bar5N5cE4+Cal/Bamj3yEELyJMXEdU4SQnk+IbqTdlOpwvMf4SAda5 ErGJlBnroX/d7tWZ2pm6kDSn96Lttg4XQZFOFAFU45/EXXZQD1j0DZXAstxrCUf0mAaYFc nNEPcx2ZcMefeHWXiqBdoVKO4FygN7o= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=UhsM6qIX; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf11.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713363291; a=rsa-sha256; cv=none; b=S7i8HqVn0Ml75W0djsXbREx6OXo4rR05QRwNF0uaLjPzVTBYU+Mfa+v06CRZYaKFzZjySQ tQqXjHlHr38rc8u7VTX3++cgu+zUXqeJs/cHxks8uDM8PH0SIZxVMJB6fkG+8bXuCS90Yr SYWjXFEC1jV2Gptt9NuQcgVyY/hQ638= Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-1e3f17c6491so48448545ad.2 for ; Wed, 17 Apr 2024 07:14:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713363291; x=1713968091; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=SOdqNlpobw5BkPj0KVJli6oAlknzNEgphraB7EENvoU=; b=UhsM6qIX8Uf8qFs+B3nxpYSimwVZ310Z6pUEgNfD0RC1mWJSJ+2ia8wnR0zl1DhD3C 3QoOQuz9JlJG67W+ZAI9StqbD7Kiq985fW2KCvbY2030o0uuL5bVVrPJ5Bw1LMA7FkCS NwWz/yH7OoNOpyNzoGJySldt6kJ1KkPdwX2O/4P6pXfrFzCtj95syvBHOb4dqXPL2lCx T9NRcSqEs5/AV/7G8s/ADSfG09jD8nKwRBb5eQgy+4iw1qKkOlXGOUxRKtwDGUFFZnwn GIj95x7mMOt7S20HfP/iVcvoVErA/jWDZ9UcuqzmVya+WKu2OvN8BpBXeXiLTw+ncM+F x1CQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713363291; x=1713968091; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=SOdqNlpobw5BkPj0KVJli6oAlknzNEgphraB7EENvoU=; b=IZb2TIjrCl9bUO/ealmwFCWVlBBw1KxD54iA3YLW00NDxhy15EAMkvCKCC9qlOG4tz NVzF+eEawm4XxMew6QZw28t6niB9oQzPchBleMTr3zGm6DkND2m3Ji1BSeCoNrRB3r1h GfA914OqFb3Mo7GygHfVi0hlyO06Ckp8tOrFRB6CTohW2Z4Gscyt54l7mR+0TC9Z40rf mv1XXQnTOS72Q3ugJCu0qu50eIJIGmgC9u1QxmLmuPvW9KPwz2M8TN1ULc6yRdCjREgF jwX8rJgJ3OYuoTYVu26289HmYPBCXy+tqgGhNeHkO2B7vBi5dUX+Z6BNvsuY2UnVqznJ +P7g== X-Forwarded-Encrypted: i=1; AJvYcCWkAE6ozult+CrfKrpKnnyEQbuDTAkyFUjI7XILd8cUXW/U8UQxOz0QnRUzdF5BQnLJrbDYKEmCoB2kvb/VusclY8c= X-Gm-Message-State: AOJu0YytYNnHsgMwhXflwfDqP/6m8ql83W7EXn/ZTfxbf7hxd1mkOn3F plO9JufJcNLaJD3CiToblaqgVvFcBLKxYm43IctWZc/4jMQ6kUat X-Google-Smtp-Source: AGHT+IF65xXZWE/D8lMwGBdhlH4F1sGFx4vMdAiKWMYZXKnzO5Ri5TkXgMwrOdlF4Zlr/S9Ifg+5FQ== X-Received: by 2002:a17:903:2452:b0:1e8:26e4:d089 with SMTP id l18-20020a170903245200b001e826e4d089mr2296668pls.54.1713363290409; Wed, 17 Apr 2024 07:14:50 -0700 (PDT) Received: from LancedeMBP.lan ([112.10.225.217]) by smtp.gmail.com with ESMTPSA id s14-20020a170902a50e00b001ddc83fda95sm11613562plq.186.2024.04.17.07.14.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Apr 2024 07:14:50 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH v8 0/3] mm/madvise: enhance lazyfreeing with mTHP in madvise_free Date: Wed, 17 Apr 2024 22:14:33 +0800 Message-Id: <20240417141436.77963-1-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: C77D84001E X-Stat-Signature: f5ai1ma7du8fhy7cjbkg3udx51mow3ps X-HE-Tag: 1713363291-789514 X-HE-Meta: U2FsdGVkX1/BqczS4E6k0sLTJdRsJp71WO3pzJITV2c65tZNmelSTcCIv+0xY3bOnuzHoyEJGaSIinZoCyYG0vvZDPrFvFl9oQHX+PEKKffzvqdQYNI8VLhx23ZJFvpVKakzLuX6L9F/u6w3rDBtN8xGr8CnTrPciI6gtiRK8BUUAxRktP+gDMbAAyFFyE3uLVS+bk3ObO0rowxv+LHzr5elo4kCjR0DGtYgRgEveL6Gu1EjBe358zKizEVfZoapKEy6Ztww3GcmQeiu80OpzAvkk3SqHEb5Dtobn9X0Q/bUk6/UoQrLjmgCSFwy8PTfMfhng4diAM1eeRg/aEOBM8iTYf+ENcptQFKxZO4DUhDacvz82sONoO2CjPzKxSwx3XClHKaXodEZ4n/OvTAPmRMH+RP3Huoz7mNaHQNMV4XI3uPgSikl/Aze/nF+TzfFwt0AWKOuy7VvmyGvjdXdYlCuXumo4KPGbB4A1UduN/WOhCZP6YI/U+kyEGAcHyWP7wlHe0s+2Hhj8Kgm6+q6cuV4bWKqWmCRZrRTPMUJ3Wi2yZwwB6+NQPSLWG4J7AgsnpJQWfVXfiUW+ZHH3ov5l4wra1PEXOisDTcwY/rfQaXb7sPDnNK8zZcylb6rstIzNU4oL9hO6XxSdnoKGJUFX8GKFjT5Pvhwyvst4/cvu0K4TjJKMRkfrfTeRFJYRIiTJvXLC5IrvT/Fpz+vRuS1E0wNexikRZY0FPPJz47uUpZPVBHXf44g7/nhVAqPT1yg2k6r7Zb9bjaMV86/RKyP7SaNOhExnn9orK3AUuBDB5HGqJLn/BO19u/ybVNqlkL6YR/SGDe71weFXhqoWnAd+/sCUZ/nxPDBxesFpKfl6iOYlU8jwByn74j2eeH6CEeAN0XOCXtGkeauN3F05wk6wXDNrXp4b+uC6R/XlCE00w8GzIG5cBPEJC3s+ui1epChSGekvcmNK3W63ESucdK 9cEsJ25J ZWAE9/y6tLnVbW096dg0+hQKQLxHJOuQGgAkKwkMVgz+Mca1iCSuOMvGmt6ckSJi0ogEpH+olojGN8LwkXEfIBxoW0g2C9eqo5XGIUay22WesXLE5ZEMRfC2qYXK43klBJXm3z8RiUCTh/N2b8FSaYTzrYP+mPdUQRWK0pQCaM5GOdQ9Oo6xkBgyCtccVR7Z8Ah6NECtMlIHj4nuwUi6zjC36SurKzQsOiv77Wq+QYwNhHuT7TgECnr2kHXct3qOKaeqxu8hHDSIe8i/037MRcRPg4Tcypsk6cz658kXC9pX6np0/bk1n9Wum4XecGvevw+jysThsIZyg75QcJKmIFGh5cSF3PTw8Omd5smk5yjjmHM7F+rJ9RFPN7IUwGAlqf2iXCXRGRqvNbr0oI0k8eVmODji/kR+l3zbra4oWl+sLHEoTW4sC9H1+nvBi84zNptcSm4qIHIkzHdLaqRMmt/FPpKxOJw4DAYbN7Q8ZaCnfB6R6D7NeafU6SVNu06W01YXQGGa+DDvzsPdaJKqOBKouX9dtRONrw2UrKq3/ZLdC63c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi All, This patchset adds support for lazyfreeing multi-size THP (mTHP) without needing to first split the large folio via split_folio(). However, we still need to split a large folio that is not fully mapped within the target range. If a large folio is locked or shared, or if we fail to split it, we just leave it in place and advance to the next PTE in the range. But note that the behavior is changed; previously, any failure of this sort would cause the entire operation to give up. As large folios become more common, sticking to the old way could result in wasted opportunities. Performance Testing =================== On an Intel I5 CPU, lazyfreeing a 1GiB VMA backed by PTE-mapped folios of the same size results in the following runtimes for madvise(MADV_FREE) in seconds (shorter is better): Folio Size | Old | New | Change ------------------------------------------ 4KiB | 0.590251 | 0.590259 | 0% 16KiB | 2.990447 | 0.185655 | -94% 32KiB | 2.547831 | 0.104870 | -95% 64KiB | 2.457796 | 0.052812 | -97% 128KiB | 2.281034 | 0.032777 | -99% 256KiB | 2.230387 | 0.017496 | -99% 512KiB | 2.189106 | 0.010781 | -99% 1024KiB | 2.183949 | 0.007753 | -99% 2048KiB | 0.002799 | 0.002804 | 0% --- This patchset applies against mm-unstable (96abec3c15c2). The performance numbers are from v2. I did a quick benchmark run of v8 and nothing significantly changed. Changes since v7 [7] ==================== - mm/madvise: optimize lazyfreeing with mTHP in madvise_free - Remove the duplicated check for the mapcount (per Ryan Roberts, David Hildenbrand) - Pick up AB's - Pick up RB's Changes since v6 [6] ==================== - Fix a bug with incorrect bitwise operations (Thanks to Ryan Roberts) - Use a cmpxchg loop to only clear one of the flags to prevent race with the HW (per Ryan Roberts) Changes since v5 [5] ==================== - Convert mkold_ptes() to clear_young_dirty_ptes() (per Ryan Roberts) - Use the __bitwise flags as the input for clear_young_dirty_ptes() (per David Hildenbrand) - Follow the pattern already established by the original code (per Ryan Roberts) Changes since v4 [4] ==================== - The first patch implements the MADV_FREE change and introduces mkold_clean_ptes() with a generic implementation. The second patch specializes mkold_clean_ptes() for arm64, providing a performance boost specific to arm64 (per Ryan Roberts) - Drop the full parameter and call ptep_get_and_clear() in mkold_clean_ptes() (per Ryan Roberts) - Keep the previous behavior that avoids locking the folio if it wasn't in the swapcache or if it wasn't dirty (per Ryan Roberts) Changes since v3 [3] ==================== - Rename refresh_full_ptes -> mkold_clean_ptes (per Ryan Roberts) - Override mkold_clean_ptes() for arm64 to make it faster (per Ryan Roberts) - Update the changelog Changes since v2 [2] ==================== - Only skip all the PTEs for nr_pages when the number of batched PTEs matches nr_pages (per Barry Song) - Change folio_pte_batch() to consume an optional *any_dirty and *any_young function (per David Hildenbrand) - Move the ptep_get_and_clear_full() loop into refresh_full_ptes() (per David Hildenbrand) - Follow a similar pattern for madvise_free_pte_range() (per Ryan Roberts) Changes since v1 [1] ==================== - Update the performance numbers - Update the changelog (per Ryan Roberts) - Check the COW folio (per Yin Fengwei) - Check if we are mapping all subpages (per Barry Song, David Hildenbrand, Ryan Roberts) [1] https://lore.kernel.org/linux-mm/20240225123215.86503-1-ioworker0@gmail.com [2] https://lore.kernel.org/linux-mm/20240307061425.21013-1-ioworker0@gmail.com [3] https://lore.kernel.org/linux-mm/20240316102952.39233-1-ioworker0@gmail.com [4] https://lore.kernel.org/linux-mm/20240402124029.47846-1-ioworker0@gmail.com [5] https://lore.kernel.org/linux-mm/20240408042437.10951-1-ioworker0@gmail.com [6] https://lore.kernel.org/linux-mm/20240413002219.71246-1-ioworker0@gmail.com [7] https://lore.kernel.org/linux-mm/20240416033457.32154-1-ioworker0@gmail.com Thanks, Lance Lance Yang (3): mm/madvise: introduce clear_young_dirty_ptes() batch helper mm/arm64: override clear_young_dirty_ptes() batch helper mm/madvise: optimize lazyfreeing with mTHP in madvise_free arch/arm64/include/asm/pgtable.h | 55 +++++++++++++++++++++++++++++++++ arch/arm64/mm/contpte.c | 29 +++++++++++++++++ include/linux/mm_types.h | 9 ++++++ include/linux/pgtable.h | 74 ++++++++++++++++++++++++-------------- mm/internal.h | 12 ++++++-- mm/madvise.c | 144 ++++++++++++++++++++++++++++++++------ mm/memory.c | 4 +-- 7 files changed, 230 insertions(+), 97 deletions(-)