From patchwork Tue Apr 26 14:44:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Zach O'Keefe X-Patchwork-Id: 12827320 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EC8FC433F5 for ; Tue, 26 Apr 2022 14:44:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5733F6B0073; Tue, 26 Apr 2022 10:44:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 522686B0074; Tue, 26 Apr 2022 10:44:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C22B6B0075; Tue, 26 Apr 2022 10:44:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.27]) by kanga.kvack.org (Postfix) with ESMTP id 2C69E6B0073 for ; Tue, 26 Apr 2022 10:44:20 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id EA48C1207C9 for ; Tue, 26 Apr 2022 14:44:19 +0000 (UTC) X-FDA: 79399300638.15.683D36A Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf14.hostedemail.com (Postfix) with ESMTP id 82F36100054 for ; Tue, 26 Apr 2022 14:44:18 +0000 (UTC) Received: by mail-pl1-f202.google.com with SMTP id u8-20020a170903124800b0015195a5826cso11379570plh.4 for ; Tue, 26 Apr 2022 07:44:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:message-id:mime-version:subject:from:to:cc :content-transfer-encoding; bh=5JWaX/EucWreXtsR4xW4ViBbb5gC7VglyXpVgnEikDQ=; b=PVhUeaNbknFPXx3irDljB4p+zq7J/iH1c9g3toxJKyJN5m1nkmYImIcQ+ID/hXLYTq SKs2lgTu9tyHb9R+evXrEURGIy5Dw2Zi3A0OEhlS1jTUDr5HKRHuHWlWLU74yCp4DcZH wuj4kmt0Wwz9AhFPqhg8sPTHqjfsfdwcScWRCE+DvwPHrJRvsCDuTFx8fn2NEKtXMCeT H9+WmXihA/XelZKu/DMxC80gzZN6ZpYblFzL5v2G7yHG4JuDRr18/Qv8sw13/xrVii85 i+rJ0s0ztuN4U/iPCl11rENsoXDJvZjd80Kl2ynfl5/Wz9fru9r2zYR7+WopUBw9U+8Z 3uew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc :content-transfer-encoding; bh=5JWaX/EucWreXtsR4xW4ViBbb5gC7VglyXpVgnEikDQ=; b=HBxO/+T4QTJY5XRW9LQyHGTTypiHGtxA83UTQbhUUvrJrYfbXi6syYMGsIYhkBfClC 2h+nGWWILGPGIybBxb5qUcvOtLxP3ztPtP3YVas5Zb8LhAxeDhkuXlmVh52/yDyOOV14 87oHQJIHFej0hfJyKi8Sk2ckAm1qqzJxZwQW9GMnyatnfA0JvsTrqfB+bwLh0cLi0Bgp 6mQt1giFLb45MGA2bpYDevQUv66y/NQxu9m7bNVhamVuYdsuL0HoA9sgmJELxa0dOA3J SiCuPMIJEGKDv+r3B7JpodUw8ktjxzLiqM2c/8h/EyWqAwGnvFt4Ubo8A8+YAwQU0C7r YyNA== X-Gm-Message-State: AOAM5306pcgiwoN5mbJsu9VB2S78erHvmiBIu4BUZv8yl/1WsgYbrSgG Mpzce3cu4tnjDA/o5icl6Q43el+cH0lC X-Google-Smtp-Source: ABdhPJyBGzP/pgZRGO1EpUPMpDGtGSVXbjMv47OmnxzpIPZUYZDJWgdGmZMsp7UGnIk5Jtn44t9nZnkklHag X-Received: from zokeefe3.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:1b6]) (user=zokeefe job=sendgmr) by 2002:a05:6a00:1152:b0:4be:ab79:fcfa with SMTP id b18-20020a056a00115200b004beab79fcfamr24689255pfm.3.1650984258125; Tue, 26 Apr 2022 07:44:18 -0700 (PDT) Date: Tue, 26 Apr 2022 07:44:00 -0700 Message-Id: <20220426144412.742113-1-zokeefe@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 00/12] mm: userspace hugepage collapse From: "Zach O'Keefe" To: Alex Shi , David Hildenbrand , David Rientjes , Matthew Wilcox , Michal Hocko , Pasha Tatashin , Peter Xu , SeongJae Park , Song Liu , Vlastimil Babka , Yang Shi , Zi Yan , linux-mm@kvack.org Cc: Andrea Arcangeli , Andrew Morton , Arnd Bergmann , Axel Rasmussen , Chris Kennelly , Chris Zankel , Helge Deller , Hugh Dickins , Ivan Kokshaysky , "James E.J. Bottomley" , Jens Axboe , "Kirill A. Shutemov" , Matt Turner , Max Filippov , Miaohe Lin , Minchan Kim , Patrick Xia , Pavel Begunkov , Thomas Bogendoerfer , "Zach O'Keefe" X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 82F36100054 X-Stat-Signature: d4tjtwudfa6ypt4osq84w5miq3tqmp1p X-Rspam-User: Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=PVhUeaNb; spf=pass (imf14.hostedemail.com: domain of 3QgVoYgcKCAc6vrllmlnvvnsl.jvtspu14-ttr2hjr.vyn@flex--zokeefe.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3QgVoYgcKCAc6vrllmlnvvnsl.jvtspu14-ttr2hjr.vyn@flex--zokeefe.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-HE-Tag: 1650984258-395720 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduction -------------------------------- This series provides a mechanism for userspace to induce a collapse of eligible ranges of memory into transparent hugepages in process context, thus permitting users to more tightly control their own hugepage utilization policy at their own expense. This idea was introduced by David Rientjes[1], and the semantics and implementation were introduced and discussed in a previous PATCH RFC[2]. Interface -------------------------------- The proposed interface adds a new madvise(2) mode, MADV_COLLAPSE, and leverages the new process_madvise(2) call. process_madvise(2) Performs a synchronous collapse of the native pages mapped by the list of iovecs into transparent hugepages. The system-wide THP sysfs settings as well as the VMA flags of the memory range being collapsed are both respected for the purposes of determining THP eligibility. THP allocation may enter direct reclaim and/or compaction. When a range spans multiple VMAs, the semantics of the collapse over of each VMA is independent from the others. Caller must have CAP_SYS_ADMIN if not acting on self. Return value follows existing process_madvise(2) conventions. A “success” indicates that all hugepage-sized/aligned regions covered by the provided range were either successfully collapsed, or were already pmd-mapped THPs. madvise(2) Equivalent to process_madvise(2) on self, with 0 returned on “success”. Current Use-Cases -------------------------------- (1) Immediately back executable text by THPs. Current support provided by CONFIG_READ_ONLY_THP_FOR_FS may take a long time on a large system which might impair services from serving at their full rated load after (re)starting. Tricks like mremap(2)'ing text onto anonymous memory to immediately realize iTLB performance prevents page sharing and demand paging, both of which increase steady state memory footprint. With MADV_COLLAPSE, we get the best of both worlds: Peak upfront performance and lower RAM footprints. Note that subsequent support for file-backed memory is required here. (2) malloc() implementations that manage memory in hugepage-sized chunks, but sometimes subrelease memory back to the system in native-sized chunks via MADV_DONTNEED; zapping the pmd. Later, when the memory is hot, the implementation could madvise(MADV_COLLAPSE) to re-back the memory by THPs to regain hugepage coverage and dTLB performance. TCMalloc is such an implementation that could benefit from this[3]. A prior study of Google internal workloads during evaluation of Temeraire, a hugepage-aware enhancement to TCMalloc, showed that nearly 20% of all cpu cycles were spent in dTLB stalls, and that increasing hugepage coverage by even small amount can help with that[4]. Future work -------------------------------- Only private anonymous memory is supported by this series. File and shmem memory support will be added later. One possible user of this functionality is a userspace agent that attempts to optimize THP utilization system-wide by allocating THPs based on, for example, task priority, task performance requirements, or heatmaps. For the latter, one idea that has already surfaced is using DAMON to identify hot regions, and driving THP collapse through a new DAMOS_COLLAPSE scheme[5]. Sequence of Patches -------------------------------- Patches 1-4 perform refactoring of collapse logic within khugepaged.c and introduce the notion of a collapse context. Patches 5-9 introduces MADV_COLLAPSE, does some renaming, adds support so that MADV_COLLAPSE context has the eligibility and allocation semantics referenced above, and adds process_madivse(2) support. Patches 10-12 add selftests to test the new functionality. Applies against next-20220426. Changelog -------------------------------- v2 -> v3: * Collapse semantics have changed: the gfp flags used for hugepage allocation now are independent of khugepaged. * Cover-letter: add primary use-cases and update description of collapse semantics. * 'mm/khugepaged: make hugepage allocation context-specific' -> Added .gfp operation to struct collapse_control * 'mm/madvise: introduce MADV_COLLAPSE sync hugepage collapse' -> Added madvise context .gfp implementation. -> Set scan_result appropriately on early exit due to mm exit or vma vma revalidation. -> Reword patch description * Rebased onto next-20220426 v1 -> v2: * Cover-letter clarification and added RFC -> v1 notes * Fixes issues reported by kernel test robot * 'mm/khugepaged: record SCAN_PMD_MAPPED when scan_pmd() finds THP' -> Fixed mixed code/declarations * 'mm/khugepaged: make hugepage allocation context-specific' -> Fixed bad function signature in !NUMA && TRANSPARENT_HUGEPAGE configs -> Added doc comment to retract_page_tables() for "cc" * 'mm/khugepaged: add struct collapse_result' -> Added doc comment to retract_page_tables() for "cr" * 'mm/madvise: introduce MADV_COLLAPSE sync hugepage collapse' -> Added MADV_COLLAPSE definitions for alpha, mips, parisc, xtensa -> Moved an "#ifdef NUMA" so that khugepaged_find_target_node() is defined in !NUMA && TRANSPARENT_HUGEPAGE configs. * 'mm/khugepaged: remove khugepaged prefix from shared collapse' functions -> Removed khugepaged prefix from khugepaged_find_target_node on L914 * Rebased onto next-20220414 RFC -> v1: * The series was significantly reworked from RFC and most patches are entirely new or reworked. * Collapse eligibility criteria has changed: MADV_COLLAPSE now respects VM_NOHUGEPAGE. * Collapse semantics have changed: the gfp flags used for hugepage allocation now match that of khugepaged for the same VMA, instead of the gfp flags used at-fault for calling process for the VMA. * Collapse semantics have changed: The collapse semantics for multiple VMAs spanning a single MADV_COLLAPSE call are now independent, whereas before the idea was to allow direct reclaim/compaction if any spanned VMA permitted so. * The process_madvise(2) flags, MADV_F_COLLAPSE_LIMITS and MADV_F_COLLAPSE_DEFRAG have been removed. * Implementation change: the RFC implemented collapse over a range of hugepages in a batched-fashion with the aim of doing multiple page table updates inside a single mmap_lock write. This has been changed, and the implementation now collapses each hugepage-aligned/sized region iteratively. This was motivated by an experiment which showed that, when multiple threads were concurrently faulting during a MADV_COLLAPSE operation, mean and tail latency to acquire mmap_lock in read for threads in the fault patch was improved by using a batch size of 1 (batch sizes of 1, 8, 16, 32 were tested)[6]. * Added: If a collapse operation fails because a page isn't found on the LRU, do a lru_add_drain_all() and retry. * Added: selftests [1] https://lore.kernel.org/all/C8C89F13-3F04-456B-BA76-DE2C378D30BF@nvidia.com/ [2] https://lore.kernel.org/linux-mm/20220308213417.1407042-1-zokeefe@google.com/ [3] https://github.com/google/tcmalloc/tree/master/tcmalloc [4] https://research.google/pubs/pub50370/ [5] https://lore.kernel.org/lkml/bcc8d9a0-81d-5f34-5e4-fcc28eb7ce@google.com/T/ [6] https://lore.kernel.org/linux-mm/CAAa6QmRc76n-dspGT7UK8DkaqZAOz-CkCsME1V7KGtQ6Yt2FqA@mail.gmail.com/ Zach O'Keefe (13): mm/khugepaged: separate hugepage preallocation and cleanup mm/khugepaged: record SCAN_PMD_MAPPED when scan_pmd() finds THP mm/khugepaged: add struct collapse_control mm/khugepaged: make hugepage allocation context-specific mm/khugepaged: add struct collapse_result mm/madvise: introduce MADV_COLLAPSE sync hugepage collapse mm/khugepaged: remove khugepaged prefix from shared collapse functions mm/khugepaged: add flag to ignore khugepaged_max_ptes_* mm/khugepaged: add flag to ignore page young/referenced requirement mm/madvise: add MADV_COLLAPSE to process_madvise() selftests/vm: modularize collapse selftests selftests/vm: add MADV_COLLAPSE collapse context to selftests selftests/vm: add test to verify recollapse of THPs include/linux/huge_mm.h | 12 + include/trace/events/huge_memory.h | 5 +- include/uapi/asm-generic/mman-common.h | 2 + mm/internal.h | 1 + mm/khugepaged.c | 598 ++++++++++++++++-------- mm/madvise.c | 11 +- mm/rmap.c | 15 +- tools/testing/selftests/vm/khugepaged.c | 417 +++++++++++------ 8 files changed, 702 insertions(+), 359 deletions(-)