From patchwork Tue Apr 16 03:34:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Yang X-Patchwork-Id: 13631162 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9955C4345F for ; Tue, 16 Apr 2024 03:35:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5E39F6B0083; Mon, 15 Apr 2024 23:35:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 593706B0085; Mon, 15 Apr 2024 23:35:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 433EA6B0087; Mon, 15 Apr 2024 23:35:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 259706B0083 for ; Mon, 15 Apr 2024 23:35:13 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 17B22120846 for ; Tue, 16 Apr 2024 03:35:12 +0000 (UTC) X-FDA: 82013979264.07.A3C7D45 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) by imf21.hostedemail.com (Postfix) with ESMTP id 5BAA41C0002 for ; Tue, 16 Apr 2024 03:35:10 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="DFgsb/G/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf21.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.210.170 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713238510; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=GjN0nEuCNdhZqfr9jbPn+Tu2kvqYDx3m99uXEZwJL3o=; b=iM1BGw8xTH6HqHaNNwi2OGXapJPRgZIMrqwpW2396gfZqMIY7NWIY7n1xvt45x+SxZNHXL 7uEvV9LTdo+aIaFn+6JR2LjktQ/ZjuZw3FvdgaEML9DDZRl885/8aRssr6gEizX/RZPRJU uPbvJtCvn+M8P/ddV4BdLL3MxxLkGz4= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="DFgsb/G/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf21.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.210.170 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713238510; a=rsa-sha256; cv=none; b=uYFQppT/rsjs8VFf68zq6vjl+xK5CJbzT8PwgTgGEylhFhaIn2V0mI2nW2lFtbQhpv/NIX z8ACl4Cd2vlthu4VYL8y/ER0Y56CEdz7NJMbUscrOtHlq5Q3nHT4/4VmSJxc74F61Qz3xp oQcB5RMlaLyStmsdUSqNICX14GJ4QMc= Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-6e782e955adso3924445b3a.3 for ; Mon, 15 Apr 2024 20:35:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713238509; x=1713843309; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=GjN0nEuCNdhZqfr9jbPn+Tu2kvqYDx3m99uXEZwJL3o=; b=DFgsb/G/hyayDglQ7CgIPxDMg2HIuQ5w0VPvzsVb0MKUWbXDbJeDsuzIxS5c77rGzM ya1BrPAWu0nWPZ/Ffa9zH8xsJBKhTxEkAmXnYVbASJ9omdEdgrs11iXyRz9d/KVyw7wF xIM/oIciMKkzppNQAUruAR5QPShWmCUFQZd7BK/aG6KElgSUeTYlI7F8e/V8Ne+sO8EV 4g8tUNuUB0e7N5GXFPYn0jYHar1Vymwun5CzUBQE4EV8XztSjPXuYvkSdJ9I/buKtRfL CG9veOsuOoemqT40wVsep5aBwVcUjcYen2GVSZWDa/JcCJZMLjrwuW7+kC8hXdageHRI zFLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713238509; x=1713843309; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=GjN0nEuCNdhZqfr9jbPn+Tu2kvqYDx3m99uXEZwJL3o=; b=U2UvLh4hvGms4hnBqbOR5GD92Aeq6OjYHLbBQ33Ns86RnnAsmrp4AQQ/Nmc6J9X98M 4Jlvn94LR15LyzlHV4/ch4yMcs4SKssLxvkmvw0bSXFiznGIBEG+/aQoSJdCWrWfGGha uc47Ton3zTF+q6BuiMLI/hIWWJy3qj1Knh7xlcTrCL6RV6BbjysVUGlfrp9ZSFvpzPoY KWTUZ3x1QqACouIniUPL9u8PtPcgMF9YewlkqCuDbBKp3Ogth7JZ8aVeqE+CWsOtKMfA mRz6QcawXv2nSa4LnsZD3r/D60FncoZKJz9ouhXaNDr9162PMKPkTmUmflqbQDDPTow3 mc+A== X-Forwarded-Encrypted: i=1; AJvYcCX9GcG9SAQNSq6ZM9nxq/k1kZWF+p9dga9XxOkx28sDgDkSj4Wj6MQnB5zIL9m9lnfw1pJtpoLXkjRllEfpCbY0+VM= X-Gm-Message-State: AOJu0Ywr8ZRIyubP01DTT1OAqEA8ARBWCtc6B68tZxZIzZ9WRqHsV39s h3ThOAgsQNXN30KWcfiVuE1BOsq2mA5ja8ypnJN4jzUzKJER7u40 X-Google-Smtp-Source: AGHT+IHurkClch6FM5hxvUFrUajtAvIvdqPYxjE9XSaOZa03kXjqn5uOlh4k6emQasjAJPbR+XdJzQ== X-Received: by 2002:a05:6a00:928e:b0:6ec:e785:98b9 with SMTP id jw14-20020a056a00928e00b006ece78598b9mr15255924pfb.27.1713238509085; Mon, 15 Apr 2024 20:35:09 -0700 (PDT) Received: from LancedeMBP.lan ([112.10.225.217]) by smtp.gmail.com with ESMTPSA id ka13-20020a056a00938d00b006e57247f4e5sm7906737pfb.8.2024.04.15.20.35.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Apr 2024 20:35:08 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH v7 0/3] mm/madvise: enhance lazyfreeing with mTHP in madvise_free Date: Tue, 16 Apr 2024 11:34:54 +0800 Message-Id: <20240416033457.32154-1-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 MIME-Version: 1.0 X-Rspamd-Queue-Id: 5BAA41C0002 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: wck4baef4ig4eqyfitjt6h6zwtx9eiea X-HE-Tag: 1713238510-81140 X-HE-Meta: U2FsdGVkX1+iXuNDnPhYkO0SpSjT5Fb1JP/VzTc6kyCLXYnkT3/WeTBS7/kL+KxiHZ8vsKfTB4YEMmGQ4URbPRTuGsYaHmniAhedVRQ8n3EwWBurfnltc0MSMObatQD0+UUbGZOlF5zo4gadqhz5PZbD1EHSOUNsIMIG+Ci/uX+MqfHyVv0qrYt2Kiqb9gj46XtEaLRuywADZbXMBydaYTqvpuvXWoDVefa7hRl08JNMHamBVIeGa2fZtYfTH2X8THS3st7H1sKiHTzyIT+rNtj+umH0Blm8dh4gxm2br7H9i7yTW8+/9dtWBldQ3G8KusCWmENgiMhNS8tb+Mophw1ygc52apHAQ57qB0wQH80v3WwetPZjWgFt83L9k9Xwz4y+RDLyhsC5TtfaEiKTE/G5QsF/ext0Ppv2yBfr/4CgUA1pD69MdYMGEnDBdRz4sdKeubtXXwCB2Mu1GL/2f06YprLOE2jQnCBaPuSkwohYiC/F883BeNzOGjSv3oFuYriwOlSC9t2y9y9KGrXOLcbO8W/h2uzRD0+arrZfQfGaPPrGs54Xzj/vvSPtBZo5+4iVWnwvZ792Z92vNazb/U/iWcBJG1mJNPiFNom8xb+vq7rKTf7aF+JGZp8dsudb5DC8IAyeqD/9WwKhREmzti5Q19HaBHchh0yRNjF10XHuZMihCurNRUuxu2GQNcBDhAal9CjakgXuR/bjCtnerSG9ocYUcQOIfLv9rN7zbOAdDj8aME98MMv8YGBIJJv5BVdjJcRvwx3X/71JphJtsjyrXd6EBDdZOAOz0u1I//V2flNOKz3sa65pyKIfwARWdhqiZ06K4gFC2sDHUQyc2rA8/HoAHHzJreB8yTfiCDZCmLohL+Zv3Vk1TBKJWHErDnaBXh1xYR7Y/lqmwZGrkWXcX3z7HyXhEdLG7GspxqoJ5FLzxltGk1XCC8QGfwWUEiJ/JouWGjCdOszD1q3 zUh18tCH cb5o2+eiwid9Rj2qzXJZoEPTU0syzIA3WGiIX1ostVQU+h8yo103wovxShVzxF2+agJ1T50M0shmUi+VOFdyKoxB6yVEvtqvDX1ZQzLurVZNjwj7WRdw9/ImCA4MgGdWwvYZTWkKGgci6OtlQNLUUVc/c/JqHXQNrQ5qtelgBIChAJ58lRG6zIBpGcoQPm/kqCQxRDsfCqCSOabzkv3vq+icmMlgC/ZmTm6BIDgswlgDfrYW4/ShFN/B67kCSZaW5PH9+EQWBPCvXV3cbXt1Owx+CWFU9XtLhV1XtPhJsLWPYwjASpVkQNcPP/Wbp8wZ24LwoUrOPPgClfR0k96JceZbSrU06+SIVr2ngDEYMcq48JDmmoq7hgcquYeputp01GxcyJagtb35+cjz/KIpZVmWmJ3TF+bx3xfuOUzWk+cD441v5UxF4DpprlgOkP0XvCI6EMSjlXUw8pQgi9IgCtt1VVVJa1caFpsVhyNlU+wVdaE2rfJmSw224ALTF4UsU3BKeRUVdRr2wkZmm00aNUX5G/I4x/NAJZ9hYnC3oRde6UPg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi All, This patchset adds support for lazyfreeing multi-size THP (mTHP) without needing to first split the large folio via split_folio(). However, we still need to split a large folio that is not fully mapped within the target range. If a large folio is locked or shared, or if we fail to split it, we just leave it in place and advance to the next PTE in the range. But note that the behavior is changed; previously, any failure of this sort would cause the entire operation to give up. As large folios become more common, sticking to the old way could result in wasted opportunities. Performance Testing =================== On an Intel I5 CPU, lazyfreeing a 1GiB VMA backed by PTE-mapped folios of the same size results in the following runtimes for madvise(MADV_FREE) in seconds (shorter is better): Folio Size | Old | New | Change ------------------------------------------ 4KiB | 0.590251 | 0.590259 | 0% 16KiB | 2.990447 | 0.185655 | -94% 32KiB | 2.547831 | 0.104870 | -95% 64KiB | 2.457796 | 0.052812 | -97% 128KiB | 2.281034 | 0.032777 | -99% 256KiB | 2.230387 | 0.017496 | -99% 512KiB | 2.189106 | 0.010781 | -99% 1024KiB | 2.183949 | 0.007753 | -99% 2048KiB | 0.002799 | 0.002804 | 0% --- This patchset applies against mm-unstable (3aec6b2b34e2). The performance numbers are from v2. I did a quick benchmark run of v7 and nothing significantly changed. Changes since v6 [6] ==================== - Fix a bug with incorrect bitwise operations (Thanks to Ryan Roberts) - Use a cmpxchg loop to only clear one of the flags to prevent race with the HW (per Ryan Roberts) Changes since v5 [5] ==================== - Convert mkold_ptes() to clear_young_dirty_ptes() (per Ryan Roberts) - Use the __bitwise flags as the input for clear_young_dirty_ptes() (per David Hildenbrand) - Follow the pattern already established by the original code (per Ryan Roberts) Changes since v4 [4] ==================== - The first patch implements the MADV_FREE change and introduces mkold_clean_ptes() with a generic implementation. The second patch specializes mkold_clean_ptes() for arm64, providing a performance boost specific to arm64 (per Ryan Roberts) - Drop the full parameter and call ptep_get_and_clear() in mkold_clean_ptes() (per Ryan Roberts) - Keep the previous behavior that avoids locking the folio if it wasn't in the swapcache or if it wasn't dirty (per Ryan Roberts) Changes since v3 [3] ==================== - Rename refresh_full_ptes -> mkold_clean_ptes (per Ryan Roberts) - Override mkold_clean_ptes() for arm64 to make it faster (per Ryan Roberts) - Update the changelog Changes since v2 [2] ==================== - Only skip all the PTEs for nr_pages when the number of batched PTEs matches nr_pages (per Barry Song) - Change folio_pte_batch() to consume an optional *any_dirty and *any_young function (per David Hildenbrand) - Move the ptep_get_and_clear_full() loop into refresh_full_ptes() (per David Hildenbrand) - Follow a similar pattern for madvise_free_pte_range() (per Ryan Roberts) Changes since v1 [1] ==================== - Update the performance numbers - Update the changelog (per Ryan Roberts) - Check the COW folio (per Yin Fengwei) - Check if we are mapping all subpages (per Barry Song, David Hildenbrand, Ryan Roberts) [1] https://lore.kernel.org/linux-mm/20240225123215.86503-1-ioworker0@gmail.com [2] https://lore.kernel.org/linux-mm/20240307061425.21013-1-ioworker0@gmail.com [3] https://lore.kernel.org/linux-mm/20240316102952.39233-1-ioworker0@gmail.com [4] https://lore.kernel.org/linux-mm/20240402124029.47846-1-ioworker0@gmail.com [5] https://lore.kernel.org/linux-mm/20240408042437.10951-1-ioworker0@gmail.com [6] https://lore.kernel.org/linux-mm/20240413002219.71246-1-ioworker0@gmail.com Thanks, Lance Lance Yang (3): mm/madvise: introduce clear_young_dirty_ptes() batch helper mm/arm64: override clear_young_dirty_ptes() batch helper mm/madvise: optimize lazyfreeing with mTHP in madvise_free arch/arm64/include/asm/pgtable.h | 55 ++++++++++++++++++++++++++++++++ arch/arm64/mm/contpte.c | 29 +++++++++++++++++ include/linux/mm_types.h | 9 ++++++ include/linux/pgtable.h | 74 +++++++++++++++++++++++++-------------- mm/internal.h | 12 +++++-- mm/madvise.c | 147 ++++++++++++++++++++++++++++++--------- mm/memory.c | 4 +-- 7 files changed, 233 insertions(+), 97 deletions(-)