From patchwork Thu May 30 02:04:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13679631 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0711C25B75 for ; Thu, 30 May 2024 02:04:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 36C636B0098; Wed, 29 May 2024 22:04:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 31BC16B0099; Wed, 29 May 2024 22:04:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1BDD36B009C; Wed, 29 May 2024 22:04:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id F14116B0099 for ; Wed, 29 May 2024 22:04:41 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 75F13C0A46 for ; Thu, 30 May 2024 02:04:41 +0000 (UTC) X-FDA: 82173418362.15.7518BCB Received: from out30-100.freemail.mail.aliyun.com (out30-100.freemail.mail.aliyun.com [115.124.30.100]) by imf21.hostedemail.com (Postfix) with ESMTP id A712A1C001F for ; Thu, 30 May 2024 02:04:38 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=oEn0SIPO; dmarc=pass (policy=none) header.from=linux.alibaba.com; spf=pass (imf21.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.100 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717034680; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=S1iPg+ik4InPIm0h6COwUgYMHQTxsJZhhLbc4ho8+Oo=; b=paCqdpkCgjqO8pDHTOL3aH9CazaruPeTxI3Yz48RVlpOjoOcZzQ+9ynQzuqTm6wxS03tVs JmUnqc+32/TOldMVetNYPxqkBm4AM8rj0Ptp3aUKMfOd4v7RS9hIBq+Xqz4AC/lZt8JjUc C60AAlRiwGkSg5Q4T6l/7zdfcBbW1lM= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=oEn0SIPO; dmarc=pass (policy=none) header.from=linux.alibaba.com; spf=pass (imf21.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.100 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717034680; a=rsa-sha256; cv=none; b=z90goMBZm/fyHss7z4wLcbHohhKnCiWSVwSucnWQ/22mUQm7MW//JnqiWcQxGutE5BF9Pl N6kPqRic4h+dDVffBVAMZL5h3BsvHs1lOHjrO6ZslUHi7muG8PIU+G4G3DAddyMYIGOq+M VZsVT36LOkAj0yk3e28+vQmF2vSaYnk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1717034675; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=S1iPg+ik4InPIm0h6COwUgYMHQTxsJZhhLbc4ho8+Oo=; b=oEn0SIPOigO8KFcKdLFmtABM06/anD+N9I36/R4kvuK8lSh72553otV5oKNjS9AFz1aNODylInOgCU03AX0ow2nk3umt8//E/2dnRT+kYVZv+YUmMtDiXocWtCJXwAIOOr/pZe/dtneHuTHB9JepE3P6gnszRqln79Dbv2ICTaY= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R111e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033068173054;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=16;SR=0;TI=SMTPD_---0W7V1J4O_1717034673; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0W7V1J4O_1717034673) by smtp.aliyun-inc.com; Thu, 30 May 2024 10:04:34 +0800 From: Baolin Wang To: akpm@linux-foundation.org, hughd@google.com Cc: willy@infradead.org, david@redhat.com, wangkefeng.wang@huawei.com, ying.huang@intel.com, 21cnbao@gmail.com, ryan.roberts@arm.com, shy828301@gmail.com, ziy@nvidia.com, ioworker0@gmail.com, da.gomez@samsung.com, p.raghav@samsung.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 0/6] add mTHP support for anonymous shmem Date: Thu, 30 May 2024 10:04:11 +0800 Message-Id: X-Mailer: git-send-email 2.39.3 MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: A712A1C001F X-Stat-Signature: p66rqcqn8eummq4ud9r5zmgkukjx75ew X-HE-Tag: 1717034678-255716 X-HE-Meta: U2FsdGVkX18aEyuO5G1kCbHvQMQZroMg1qfQzPJ7Aug1hDCNHaiDgzu0ThzJPQGeCUtdWHojzFwweluF5EJJWVyQSK9dLBC3lKXH/Bz8YTqifO0ZTfqi6aSUpo0NT6OPpn/GFeVSfZ4+dqQ7gCZsV5BTs0bocdGS9PsMa2gzG6QBql2caWA2OqTE+rPauitKNVhW76Q1CShRsz4kJFN6oTOAJyyv30oqH4ZEbh/fq1IeAzwRfkIF4LrQwjwTUYaI1SvceRFn/WAa165fMSQVJD4qXmpSD4OQOAGASdIzsX+Xu44inmH2MfCo4/zpXSbffK0Em7G5GNtC+Jk1ntwJ0VZd2DWdVcuUdI499PcnwB10B2YEeD/Y4eKzpZBeqD/VPvqmlVHzeWjpCTbTlPRHgQG8vUep3VFufTqgSnwPekAAtEj+0oSkw7Yzl/HSW489ysYxhMNaPiOy2yn3iHFxZ8hb8UB07fVASlJu8Xby1JqGLboUr0TZMI6romlqut4BWn/xx3kVWK+UPkF5p3BW2vs/sioGd52Yjpl/kr8EHSjpnU7iskpDQalRSWn6dz9YcaG1fEqv4HjLYMYyCRP6SrcdP/M61ktbuVuFmnuIxeUKMLWIj3+betUCZX74tRSOQkMlB3mj7HckAdiFtRI4cdRfItjYxGfoz/bnGrQ602BgldRA5KXTaI62JwCZvcsst+nx97tsvV9eRBx3gvtmwEjENT1FhUTwSxHSjQnXu2gcoXYFCLwfd+NF13DqWVKBZiuMMR4lYynes17dFf1Vb5pFt4MXSnsqe5LgU+Hqo6L4Y/TKrAtQapvq6mnzL2QI6iBLREOPt3z7MfSGZV4IAdZ95JzmZHK8SFAb47Bh6mdbKenPbEKxUnyKl7OxromSKJEPvOx4VWvf4uonInXdlYSfJObndYzBx09sdXu7FdhIcr3neIC+jl0pkQRCbcOtwJ01RGwCBk5EnFjwGoQ lJH0NkqT gvF9b2P2ykSg1fr9lEF6h+fOlFlCJoXD2ld89S7XLu5h8hnbLqVvhK2gAIZhLKvnpll0fbrWMukvbhfOB9E/vWf1CUWMzobIFKxdxnvaGMmngoEcWJe/C4cjGtBFY0jDlMTMM+SbKBdwKmDBYD96KajjLn+3QOPcNiVCqySmWC0fhhu4PmbTBA6GhpJfyfDOefr2xls2KOuVuY9JBVeccDTA4VQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Anonymous pages have already been supported for multi-size (mTHP) allocation through commit 19eaf44954df, that can allow THP to be configured through the sysfs interface located at '/sys/kernel/mm/transparent_hugepage/hugepage-XXkb/enabled'. However, the anonymous shmem will ignore the anonymous mTHP rule configured through the sysfs interface, and can only use the PMD-mapped THP, that is not reasonable. Many implement anonymous page sharing through mmap(MAP_SHARED | MAP_ANONYMOUS), especially in database usage scenarios, therefore, users expect to apply an unified mTHP strategy for anonymous pages, also including the anonymous shared pages, in order to enjoy the benefits of mTHP. For example, lower latency than PMD-mapped THP, smaller memory bloat than PMD-mapped THP, contiguous PTEs on ARM architecture to reduce TLB miss etc. The primary strategy is similar to supporting anonymous mTHP. Introduce a new interface '/mm/transparent_hugepage/hugepage-XXkb/shmem_enabled', which can have all the same values as the top-level '/sys/kernel/mm/transparent_hugepage/shmem_enabled', with adding a new additional "inherit" option. By default all sizes will be set to "never" except PMD size, which is set to "inherit". This ensures backward compatibility with the anonymous shmem enabled of the top level, meanwhile also allows independent control of anonymous shmem enabled for each mTHP. Use the page fault latency tool to measure the performance of 1G anonymous shmem with 32 threads on my machine environment with: ARM64 Architecture, 32 cores, 125G memory: base: mm-unstable user-time sys_time faults_per_sec_per_cpu faults_per_sec 0.04s 3.10s 83516.416 2669684.890 mm-unstable + patchset, anon shmem mTHP disabled user-time sys_time faults_per_sec_per_cpu faults_per_sec 0.02s 3.14s 82936.359 2630746.027 mm-unstable + patchset, anon shmem 64K mTHP enabled user-time sys_time faults_per_sec_per_cpu faults_per_sec 0.08s 0.31s 678630.231 17082522.495 From the data above, it is observed that the patchset has a minimal impact when mTHP is not enabled (some fluctuations observed during testing). When enabling 64K mTHP, there is a significant improvement of the page fault latency. Changes from v2: - Rebased to mm/mm-unstable. - Remove 'huge' parameter for shmem_alloc_and_add_folio(), per Lance. Changes from v1: - Drop the patch that re-arranges the position of highest_order() and next_order(), per Ryan. - Modify the finish_fault() to fix VA alignment issue, per Ryan and David. - Fix some building issues, reported by Lance and kernel test robot. - Update some commit message. Changes from RFC: - Rebase the patch set against the new mm-unstable branch, per Lance. - Add a new patch to export highest_order() and next_order(). - Add a new patch to align mTHP size in shmem_get_unmapped_area(). - Handle the uffd case and the VMA limits case when building mapping for large folio in the finish_fault() function, per Ryan. - Remove unnecessary 'order' variable in patch 3, per Kefeng. - Keep the anon shmem counters' name consistency. - Modify the strategy to support mTHP for anonymous shmem, discussed with Ryan and David. - Add reviewed tag from Barry. - Update the commit message. Baolin Wang (6): mm: memory: extend finish_fault() to support large folio mm: shmem: add THP validation for PMD-mapped THP related statistics mm: shmem: add multi-size THP sysfs interface for anonymous shmem mm: shmem: add mTHP support for anonymous shmem mm: shmem: add mTHP size alignment in shmem_get_unmapped_area mm: shmem: add mTHP counters for anonymous shmem Documentation/admin-guide/mm/transhuge.rst | 29 ++ include/linux/huge_mm.h | 23 ++ mm/huge_memory.c | 17 +- mm/memory.c | 58 +++- mm/shmem.c | 340 ++++++++++++++++++--- 5 files changed, 406 insertions(+), 61 deletions(-)