From patchwork Thu Oct 17 09:47:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13839727 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 271BAD2126E for ; Thu, 17 Oct 2024 09:47:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD5886B008A; Thu, 17 Oct 2024 05:47:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A85B46B008C; Thu, 17 Oct 2024 05:47:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8B1AF6B0092; Thu, 17 Oct 2024 05:47:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 6CCEF6B008A for ; Thu, 17 Oct 2024 05:47:53 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A84C6411F3 for ; Thu, 17 Oct 2024 09:47:46 +0000 (UTC) X-FDA: 82682617122.09.71A2AE6 Received: from mail-pg1-f174.google.com (mail-pg1-f174.google.com [209.85.215.174]) by imf15.hostedemail.com (Postfix) with ESMTP id ECCC3A0009 for ; Thu, 17 Oct 2024 09:47:40 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=LbG3tvNc; spf=pass (imf15.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729158327; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=Her0gSR4BBHnMz6qkjoFdQSLfH6EB46Dm9TwWv8aciM=; b=e3WDWzxi421+y9AjYrGbWZSEtQAV6bXDas+K0FeJtU1SvvJxb7l2UqEnVeyMQs+M7klDVg sv/EyphcxWYNnKJoHykEE9TuVVEq5SGR7dulOwCljlz5E/SmOde9A4a/y1mb8b9HgLgFJV syGWbO9/xT04dj2izv5iqCFO77G6FUk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729158327; a=rsa-sha256; cv=none; b=P/R240ebNywpYsPI+x/IO41xw959fYELS/m+WFX/20NmdMQGWhu1XMkUDwo4WiID8I7BHN XX1TfpZLx+Og7kA9LOXHDDyoQujQd1QUGI/RxyiMWxmxbxrD+/1tDs/v5YoBGHVDuGdqxe 4bpKouKrKbi6zuW8zlnSALZn2o2qFvg= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=LbG3tvNc; spf=pass (imf15.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pg1-f174.google.com with SMTP id 41be03b00d2f7-7ea7d509e61so371913a12.1 for ; Thu, 17 Oct 2024 02:47:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1729158469; x=1729763269; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=Her0gSR4BBHnMz6qkjoFdQSLfH6EB46Dm9TwWv8aciM=; b=LbG3tvNcuiRRkl8YwGoOVDSACep5R6gjJE0sZDAxOg2TAD97W0xy8toT2J37L174HS 7m/l5PJwqzW7igul5UvliuTyzzlfzu4ZYglupRRGa+mKdJDfpu2PGzUZN7tAH8/OGg+8 5YT9isg4GAOnPTInPlN13MS73V0iH5Q5dnspc8UAJqFQgzxpZW0gKdVrVZtQ68J6RUH1 KJ5BQi5ckRd2OYQBsP5pDIcOkSooTf0tp6sniJWlT47CeMkHUKueqSIeWteWYlFqA+Yw +aeLcdfMVF547d2nbv+a6p323yxx3wrGwHZtXYaWahYSENP1GJXUtcFwK/2te+XX/Vpv saKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729158469; x=1729763269; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Her0gSR4BBHnMz6qkjoFdQSLfH6EB46Dm9TwWv8aciM=; b=FHQE0+sFk4Hp1CToow8Go8347gtdL4Wl4mjyh4ub/90RncBKrLHtR6cZb6lErHRJmM 5357/ShRnc+P+qSYrD9MHvXE/8/uKbfY978PsLpAe8pZL/E8yuP36SOrUmZaQrWwf+QH SQVcFFH7KXlq/CKvihvTPc/BJxmPtTXWZbb91yTKtUHyTmMy59sWtzLUP9hRg/M66uqU LjFYqcdZbTbq6sZovCuPftDaTOuUcSbMyMEvLcMQqYrV+rRrA5nzw5CS5YoQIf5ZIXz9 ld+TEkk1h2qzOINIYGknTPAYKtP/VnexFL8b7ChyG6PqTg1xO4oAT9wEF8dsh3PTX3Qv 49lw== X-Gm-Message-State: AOJu0YyliVRFjJ3pPuFgc5LLyTJYo4cYP/SOGpsHyWnFRLqc11gjf2az GrAivjtxqhTfR6oDS8VgJQGnD6UC8V3vBvCwPK5xI/2HNe1IOS9bSnIIWLnQELk= X-Google-Smtp-Source: AGHT+IE1v6XXIN4uy6wpGXlNlRaRH/EFn1RgvKGG6B8slpDy+FU4v/9ncWciKSoJRaJla+Op+ATmuw== X-Received: by 2002:a05:6a20:cfa5:b0:1d2:f09e:c15b with SMTP id adf61e73a8af0-1d8bcf42381mr31104053637.25.1729158469098; Thu, 17 Oct 2024 02:47:49 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71e774a4218sm4385365b3a.120.2024.10.17.02.47.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Oct 2024 02:47:48 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, mgorman@suse.de, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, zokeefe@google.com, rientjes@google.com, jannh@google.com, peterx@redhat.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, x86@kernel.org, Qi Zheng Subject: [PATCH v1 0/7] synchronously scan and reclaim empty user PTE pages Date: Thu, 17 Oct 2024 17:47:19 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) MIME-Version: 1.0 X-Rspamd-Queue-Id: ECCC3A0009 X-Stat-Signature: p4ino3bw4gf6q7bcm5u5p89gjcxd3tnj X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1729158460-366922 X-HE-Meta: U2FsdGVkX1++eKaKdb6BWlkXP3tu8vVCZZHtkde6FQ5Vd6aA3gpkcokWE1h6X9NDQzXhWdWLrGaAMh1K64hej7tpiEt41euYJGq4VmJxOd671p+IS3hUoIsO1R/fYA+aL98jTUX1phcF3K0UwtpyAPbq4IM0qWP6w28sahFmLMX195WvXT/6y1K/orQ8NLDQKvsVVJzfUU9bQN2x9qNPYsDmIO72aZ+nJ7awgG0PI2MQdpJwpcwLOlw9zd7mAPJEZXy6cSbn0QuKJxmXkr/J07Fc9GQtyHwCj2nv6rHMqZ2QAbf4dbs41SmLoS2/jFEO+UfCYRlbNCOfS5sUfjJOxI0hYPmsvgxhBaS8XCqFejvTuP6ukc4x0wkRiSrOrL1k4L2NWx74bP/GvUGYrmjB+krT0KqtkhdiTqzLKfmD2rlJFioi1rvdtRH1o4ZrBkcB93qymRY77KsP6O+C9QMtqOaK1Jr4SgGjFyEJmCbf8ROafm3k3qQQKYtqZlwl8NqhFRd3nuP2Vmtvoyhl4qf3oqab9jc8JCJmTO161av7ml77Ir3WTe9uQQoDHM15fjFD29BJlAhXYxt6ctSvDpiCpdyoABgqeW0YDKAP0rvDkDphBZtyX3UPJeuS/HfBaEkh6+6jNw9CvgQs/709shZH33cqOSTU/bXF+iWeCT5gX0k88oBPTcXfDErKLwg1YF0b+JJhfCrBNge8mKqqQg9BalWBzCwEaAULWbdP8r2ldqmPEZEfTUAU7M4SPv7M9wfEjwity7Fqgc4PN5SEDXndg0Fr80DZT6d0hvkCnoBU4v78hDXotpc1BF5yLAkeAniDEwcXjFnSQqRw5lnjbPYO7rkhI3HXUiFXBaDsels6sGlXzw7mwAa7rmoiyEs4Yx0sm/KgBEYxcn1RKnNF9Dx/njFjMXKR19l3luSXU9d2MPZKZ8GX9BHGsYRmLzKRaii67K4srW6BwJvCYorq6rI gmokNLGO luiiKTQpbdYpew/wAJp0+Y2MBTs+AaBkQDj0Yb9ybEZbPuwqVXCRC1hRC+SiVESlSlL7Qcrvip+fK09YvlaJ/ppGJoQzwrxOew1VGhfzl6v033dpLj7J1M58hUJ4JPkQbgwr+pbWVjFqMuTL8z4Eknn6r4ATeO9VkSlLfv2cBxZFvmF06BF7lRUQaygYiTKW7Tihc7+h+ueaFfMp72HPNTwlr4tVmAi5XvV0/mJ0fe3c1R1s7RyglFo+Y2rcphAZXC/gd8t7hqHb6HvrcD2OFg1JcCPWyJfpgWd+H1RJNonLC3LdIHvTAFtwaEvb6rt/yMU/OZLQQcubeJqUsnz4xHGP1Mg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Changes in v1: - replace [RFC PATCH 1/7] with a separate serise (already merge into mm-unstable): https://lore.kernel.org/lkml/cover.1727332572.git.zhengqi.arch@bytedance.com/ (suggested by David Hildenbrand) - squash [RFC PATCH 2/7] into [RFC PATCH 4/7] (suggested by David Hildenbrand) - change to scan and reclaim empty user PTE pages in zap_pte_range() (suggested by David Hildenbrand) - sent a separate RFC patch to track the tlb flushing issue, and remove that part form this series ([RFC PATCH 3/7] and [RFC PATCH 6/7]). link: https://lore.kernel.org/lkml/20240815120715.14516-1-zhengqi.arch@bytedance.com/ - add [PATCH v1 1/7] into this series - drop RFC tag - rebase onto the next-20241011 Changes in RFC v2: - fix compilation errors in [RFC PATCH 5/7] and [RFC PATCH 7/7] reproted by kernel test robot - use pte_offset_map_nolock() + pmd_same() instead of check_pmd_still_valid() in retract_page_tables() (in [RFC PATCH 4/7]) - rebase onto the next-20240805 Hi all, Previously, we tried to use a completely asynchronous method to reclaim empty user PTE pages [1]. After discussing with David Hildenbrand, we decided to implement synchronous reclaimation in the case of madvise(MADV_DONTNEED) as the first step. So this series aims to synchronously free the empty PTE pages in madvise(MADV_DONTNEED) case. We will detect and free empty PTE pages in zap_pte_range(), and will add zap_details.reclaim_pt to exclude cases other than madvise(MADV_DONTNEED). In zap_pte_range(), mmu_gather is used to perform batch tlb flushing and page freeing operations. Therefore, if we want to free the empty PTE page in this path, the most natural way is to add it to mmu_gather as well. Now, if CONFIG_MMU_GATHER_RCU_TABLE_FREE is selected, mmu_gather will free page table pages by semi RCU: - batch table freeing: asynchronous free by RCU - single table freeing: IPI + synchronous free But this is not enough to free the empty PTE page table pages in paths other that munmap and exit_mmap path, because IPI cannot be synchronized with rcu_read_lock() in pte_offset_map{_lock}(). So we should let single table also be freed by RCU like batch table freeing. As a first step, we supported this feature on x86_64 and selectd the newly introduced CONFIG_ARCH_SUPPORTS_PT_RECLAIM. For other cases such as madvise(MADV_FREE), consider scanning and freeing empty PTE pages asynchronously in the future. This series is based on next-20241011 (which contains the series [2]). Comments and suggestions are welcome! Thanks, Qi [1]. https://lore.kernel.org/lkml/cover.1718267194.git.zhengqi.arch@bytedance.com/ [2]. https://lore.kernel.org/lkml/cover.1727332572.git.zhengqi.arch@bytedance.com/ Qi Zheng (7): mm: khugepaged: retract_page_tables() use pte_offset_map_lock() mm: make zap_pte_range() handle full within-PMD range mm: zap_install_uffd_wp_if_needed: return whether uffd-wp pte has been re-installed mm: zap_present_ptes: return whether the PTE page is unreclaimable mm: pgtable: try to reclaim empty PTE page in madvise(MADV_DONTNEED) x86: mm: free page table pages by RCU instead of semi RCU x86: select ARCH_SUPPORTS_PT_RECLAIM if X86_64 arch/x86/Kconfig | 1 + arch/x86/include/asm/tlb.h | 19 ++++++++ arch/x86/kernel/paravirt.c | 7 +++ arch/x86/mm/pgtable.c | 10 +++- include/linux/mm.h | 1 + include/linux/mm_inline.h | 11 +++-- mm/Kconfig | 14 ++++++ mm/Makefile | 1 + mm/internal.h | 29 ++++++++++++ mm/khugepaged.c | 9 +++- mm/madvise.c | 4 +- mm/memory.c | 95 +++++++++++++++++++++++++++++--------- mm/mmu_gather.c | 9 +++- mm/pt_reclaim.c | 68 +++++++++++++++++++++++++++ 14 files changed, 248 insertions(+), 30 deletions(-) create mode 100644 mm/pt_reclaim.c