From patchwork Wed Nov 11 20:40:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 11898509 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C3082921 for ; Wed, 11 Nov 2020 20:40:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 58D1420797 for ; Wed, 11 Nov 2020 20:40:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b="rEuH0PxI"; dkim=temperror (0-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="mmV3IxgO" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 58D1420797 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=sent.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 50F496B0068; Wed, 11 Nov 2020 15:40:45 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4BF9C6B005D; Wed, 11 Nov 2020 15:40:45 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F6F76B005D; Wed, 11 Nov 2020 15:40:45 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0203.hostedemail.com [216.40.44.203]) by kanga.kvack.org (Postfix) with ESMTP id D05F36B005D for ; Wed, 11 Nov 2020 15:40:44 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 7534D8249980 for ; Wed, 11 Nov 2020 20:40:44 +0000 (UTC) X-FDA: 77473306008.01.art65_450a10c27300 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id 426FD100473D9 for ; Wed, 11 Nov 2020 20:40:44 +0000 (UTC) X-Spam-Summary: 1,0,0,cabedb66269b73a4,d41d8cd98f00b204,zi.yan@sent.com,,RULES_HIT:1:2:41:355:379:541:800:960:966:973:988:989:1260:1261:1311:1314:1345:1359:1437:1513:1515:1521:1605:1730:1747:1777:1792:1801:2196:2199:2393:2559:2562:2987:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:4052:4250:4321:4385:4605:5007:6119:6120:6261:6630:6653:6742:7576:7875:7903:8531:8957:10004:11026:11473:11657:11658:11914:12043:12296:12438:12555:12679:12895:12986:13255:21063:21080:21324:21451:21627:21990:30003:30034:30051:30054:30056:30064:30070,0,RBL:64.147.123.26:@sent.com:.lbl8.mailshell.net-62.18.0.100 64.100.201.100;04ygbct63xktz7o39qbd9a8671y5jype5kxtzhur46wn3tzwg7b3whmk51gftxb.bd747udit6bq37fger9hfwxcyyktm1gm5beadkpk4bf5k1mme9x3cp593au9fk5.4-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:111,LUA_SUMMARY:none X-HE-Tag: art65_450a10c27300 X-Filterd-Recvd-Size: 13227 Received: from wnew1-smtp.messagingengine.com (wnew1-smtp.messagingengine.com [64.147.123.26]) by imf50.hostedemail.com (Postfix) with ESMTP for ; Wed, 11 Nov 2020 20:40:43 +0000 (UTC) Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailnew.west.internal (Postfix) with ESMTP id 51FFDB3D; Wed, 11 Nov 2020 15:40:41 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Wed, 11 Nov 2020 15:40:42 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=from :to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; s=fm2; bh=vIYJoxuU5tKu/ hyJrnZuBp+v7rlQ+F8xeJcONsNriU0=; b=rEuH0PxI3Ns3MdKNLXtW5dHUajYUw MkSOswCbZijE/eYkI1/YUCfAUjoGR5Jy7HwMBPWvEI9bURP8DIq/HFpAOVgUY7DT HTerQY/NFDRsCgEGfXEuqMTlKXyOE2cE1j1L4h4chdESunEnV9OUzlx0uo+7W3Ns VRwRS0rxUcr8CV0KcJgU4oJ5RC3BJv3fq1slKy4LndWlOXzsk0RklUmSGD41rnhe dS/9jFHVFl2d6b2LJgVmYg14q16ggYZEP2Kv7TSDCnsNcItU6C5fccZw6ijDUkLN 2NRSjnCwgnayzD0Ze7FccTLbRaR6Ks7tVykyAwqPDsGaSX8y6pC3Xpp7g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:reply-to:subject :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; bh=vIYJoxuU5tKu/hyJrnZuBp+v7rlQ+F8xeJcONsNriU0=; b=mmV3IxgO RXxP1quVBOCmymvDWIZnKCPcfPZTGXL716YDnRuC0+phHyGMdF8lttuvgA3/UIp+ iLG+/drOK5/Ex+iXwQ3YGHGCWCCF/61PGlgfhU9VeKP9GkTnfOWw3RMRA+rRb5Yv ZKpS8kYqKFktj75TWybTPIPS3k2va+MSYVusVpVuJ+/ba1vfRUY5gA4M/yfmojiw csvVTC9Cowem5Gv74ZgqhOlmt441xAfhwkWRW2N5qYZ2rqXarHsT3GpIm8YL9nm3 MfMZPX9TPr53U017iSgXaWWWcrbWFDGVCzfYnUCumX0Hy2scW0WWOOQROcKIgMPs NKdfa9A93Cs0Qw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedruddvtddgjeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfrhgggfestdhqredtredttdenucfhrhhomhepkghiucgj rghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepieejue dvueduuefhgefhheeiuedvtedvuefgieegveetueeiueehtdegudehfeelnecukfhppedu vddrgeeirddutdeirdduieegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpe hmrghilhhfrhhomhepiihirdihrghnsehsvghnthdrtghomh X-ME-Proxy: Received: from nvrsysarch6.NVidia.COM (unknown [12.46.106.164]) by mail.messagingengine.com (Postfix) with ESMTPA id CA8FC328006C; Wed, 11 Nov 2020 15:40:39 -0500 (EST) From: Zi Yan To: linux-mm@kvack.org, Matthew Wilcox Cc: "Kirill A . Shutemov" , Roman Gushchin , Andrew Morton , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Yang Shi , Michal Hocko , John Hubbard , Ralph Campbell , David Nellans , Zi Yan Subject: [RFC PATCH 1/6] mm: huge_memory: add new debugfs interface to trigger split huge page on any page range. Date: Wed, 11 Nov 2020 15:40:03 -0500 Message-Id: <20201111204008.21332-2-zi.yan@sent.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201111204008.21332-1-zi.yan@sent.com> References: <20201111204008.21332-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan Huge pages in the process with the given pid and virtual address range are split. It is used to test split huge page function. In addition, a testing program is added to tools/testing/selftests/vm to utilize the interface by splitting PMD THPs. Signed-off-by: Zi Yan --- mm/huge_memory.c | 98 +++++++++++ mm/internal.h | 1 + mm/migrate.c | 2 +- tools/testing/selftests/vm/Makefile | 1 + .../selftests/vm/split_huge_page_test.c | 161 ++++++++++++++++++ 5 files changed, 262 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/vm/split_huge_page_test.c diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 207ebca8c654..c4fead5ead31 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -7,6 +7,7 @@ #include #include +#include #include #include #include @@ -2935,10 +2936,107 @@ static int split_huge_pages_set(void *data, u64 val) DEFINE_DEBUGFS_ATTRIBUTE(split_huge_pages_fops, NULL, split_huge_pages_set, "%llu\n"); +static ssize_t split_huge_pages_in_range_pid_write(struct file *file, + const char __user *buf, size_t count, loff_t *ppops) +{ + static DEFINE_MUTEX(mutex); + ssize_t ret; + char input_buf[80]; /* hold pid, start_vaddr, end_vaddr */ + int pid; + unsigned long vaddr_start, vaddr_end, addr; + nodemask_t task_nodes; + struct mm_struct *mm; + + ret = mutex_lock_interruptible(&mutex); + if (ret) + return ret; + + ret = -EFAULT; + + memset(input_buf, 0, 80); + if (copy_from_user(input_buf, buf, min_t(size_t, count, 80))) + goto out; + + input_buf[80] = '\0'; + ret = sscanf(input_buf, "%d,%lx,%lx", &pid, &vaddr_start, &vaddr_end); + if (ret != 3) { + ret = -EINVAL; + goto out; + } + vaddr_start &= PAGE_MASK; + vaddr_end &= PAGE_MASK; + + ret = strlen(input_buf); + pr_debug("split huge pages in pid: %d, vaddr: [%lx - %lx]\n", + pid, vaddr_start, vaddr_end); + + mm = find_mm_struct(pid, &task_nodes); + if (IS_ERR(mm)) { + ret = -EINVAL; + goto out; + } + + mmap_read_lock(mm); + for (addr = vaddr_start; addr < vaddr_end;) { + struct vm_area_struct *vma = find_vma(mm, addr); + unsigned int follflags; + struct page *page; + + if (!vma || addr < vma->vm_start || !vma_migratable(vma)) + break; + + /* FOLL_DUMP to ignore special (like zero) pages */ + follflags = FOLL_GET | FOLL_DUMP; + page = follow_page(vma, addr, follflags); + + if (IS_ERR(page)) + break; + if (!page) + break; + + if (!is_transparent_hugepage(page)) + goto next; + + if (!can_split_huge_page(page, NULL)) + goto next; + + if (!trylock_page(page)) + goto next; + + addr += page_size(page) - PAGE_SIZE; + + /* reset addr if split fails */ + if (split_huge_page(page)) + addr -= (page_size(page) - PAGE_SIZE); + + unlock_page(page); +next: + /* next page */ + addr += page_size(page); + put_page(page); + } + mmap_read_unlock(mm); + + + mmput(mm); +out: + mutex_unlock(&mutex); + return ret; + +} + +static const struct file_operations split_huge_pages_in_range_pid_fops = { + .owner = THIS_MODULE, + .write = split_huge_pages_in_range_pid_write, + .llseek = no_llseek, +}; + static int __init split_huge_pages_debugfs(void) { debugfs_create_file("split_huge_pages", 0200, NULL, NULL, &split_huge_pages_fops); + debugfs_create_file("split_huge_pages_in_range_pid", 0200, NULL, NULL, + &split_huge_pages_in_range_pid_fops); return 0; } late_initcall(split_huge_pages_debugfs); diff --git a/mm/internal.h b/mm/internal.h index 3ea43642b99d..fd841a38830f 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -624,4 +624,5 @@ struct migration_target_control { bool truncate_inode_partial_page(struct page *page, loff_t start, loff_t end); void page_cache_free_page(struct address_space *mapping, struct page *page); +struct mm_struct *find_mm_struct(pid_t pid, nodemask_t *mem_nodes); #endif /* __MM_INTERNAL_H */ diff --git a/mm/migrate.c b/mm/migrate.c index a50bbb0e029b..e35654d1087d 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1851,7 +1851,7 @@ static int do_pages_stat(struct mm_struct *mm, unsigned long nr_pages, return nr_pages ? -EFAULT : 0; } -static struct mm_struct *find_mm_struct(pid_t pid, nodemask_t *mem_nodes) +struct mm_struct *find_mm_struct(pid_t pid, nodemask_t *mem_nodes) { struct task_struct *task; struct mm_struct *mm; diff --git a/tools/testing/selftests/vm/Makefile b/tools/testing/selftests/vm/Makefile index 62fb15f286ee..d9ead0cdd3e9 100644 --- a/tools/testing/selftests/vm/Makefile +++ b/tools/testing/selftests/vm/Makefile @@ -42,6 +42,7 @@ TEST_GEN_FILES += on-fault-limit TEST_GEN_FILES += thuge-gen TEST_GEN_FILES += transhuge-stress TEST_GEN_FILES += userfaultfd +TEST_GEN_FILES += split_huge_page_test ifeq ($(ARCH),x86_64) CAN_BUILD_I386 := $(shell ./../x86/check_cc.sh $(CC) ../x86/trivial_32bit_program.c -m32) diff --git a/tools/testing/selftests/vm/split_huge_page_test.c b/tools/testing/selftests/vm/split_huge_page_test.c new file mode 100644 index 000000000000..c8a32ae9e13a --- /dev/null +++ b/tools/testing/selftests/vm/split_huge_page_test.c @@ -0,0 +1,161 @@ +// SPDX-License-Identifier: GPL-2.0 + +#define _GNU_SOURCE +#include +#include +#include "numa.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define PAGE_4KB (4096UL) +#define PAGE_2MB (512UL*PAGE_4KB) +#define PAGE_1GB (512UL*PAGE_2MB) + +#define PRESENT_MASK (1UL<<63) +#define SWAPPED_MASK (1UL<<62) +#define PAGE_TYPE_MASK (1UL<<61) +#define PFN_MASK ((1UL<<55)-1) + +#define KPF_THP (1UL<<22) +#define KPF_PUD_THP (1UL<<27) + +#define SPLIT_DEBUGFS "/sys/kernel/debug/split_huge_pages_in_range_pid" +#define SMAP_PATH "/proc/self/smaps" +#define INPUT_MAX 80 + +static int write_file(const char *path, const char *buf, size_t buflen) +{ + int fd; + ssize_t numwritten; + + fd = open(path, O_WRONLY); + if (fd == -1) + return 0; + + numwritten = write(fd, buf, buflen - 1); + close(fd); + if (numwritten < 1) + return 0; + + return (unsigned int) numwritten; +} + +static void write_debugfs(int pid, uint64_t vaddr_start, uint64_t vaddr_end) +{ + char input[INPUT_MAX]; + int ret; + + ret = snprintf(input, INPUT_MAX, "%d,%lx,%lx", pid, vaddr_start, + vaddr_end); + if (ret >= INPUT_MAX) { + printf("%s: Debugfs input is too long\n", __func__); + exit(EXIT_FAILURE); + } + + if (!write_file(SPLIT_DEBUGFS, input, ret + 1)) { + perror(SPLIT_DEBUGFS); + exit(EXIT_FAILURE); + } +} + +#define MAX_LINE_LENGTH 500 + +static bool check_for_pattern(FILE *fp, char *pattern, char *buf) +{ + while (fgets(buf, MAX_LINE_LENGTH, fp) != NULL) { + if (!strncmp(buf, pattern, strlen(pattern))) + return true; + } + return false; +} + +static uint64_t check_huge(void *addr) +{ + uint64_t thp = 0; + int ret; + FILE *fp; + char buffer[MAX_LINE_LENGTH]; + char addr_pattern[MAX_LINE_LENGTH]; + + ret = snprintf(addr_pattern, MAX_LINE_LENGTH, "%08lx-", + (unsigned long) addr); + if (ret >= MAX_LINE_LENGTH) { + printf("%s: Pattern is too long\n", __func__); + exit(EXIT_FAILURE); + } + + + fp = fopen(SMAP_PATH, "r"); + if (!fp) { + printf("%s: Failed to open file %s\n", __func__, SMAP_PATH); + exit(EXIT_FAILURE); + } + if (!check_for_pattern(fp, addr_pattern, buffer)) + goto err_out; + + /* + * Fetch the AnonHugePages: in the same block and check the number of + * hugepages. + */ + if (!check_for_pattern(fp, "AnonHugePages:", buffer)) + goto err_out; + + if (sscanf(buffer, "AnonHugePages:%10ld kB", &thp) != 1) { + printf("Reading smap error\n"); + exit(EXIT_FAILURE); + } + +err_out: + fclose(fp); + return thp; +} + +void split_pmd_thp(void) +{ + char *one_page; + size_t len = 4 * PAGE_2MB; + uint64_t thp_size; + + one_page = memalign(PAGE_1GB, len); + + madvise(one_page, len, MADV_HUGEPAGE); + + memset(one_page, 1, len); + + thp_size = check_huge(one_page); + if (!thp_size) { + printf("No THP is allocatd"); + exit(EXIT_FAILURE); + } + + /* split all possible huge pages */ + write_debugfs(getpid(), (uint64_t)one_page, (uint64_t)one_page + len); + + *one_page = 0; + + thp_size = check_huge(one_page); + if (thp_size) { + printf("Still %ld kB AnonHugePages not split\n", thp_size); + exit(EXIT_FAILURE); + } + + printf("Split huge pages successful\n"); + free(one_page); +} + +int main(int argc, char **argv) +{ + split_pmd_thp(); + + return 0; +} From patchwork Wed Nov 11 20:40:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 11898511 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 42B7D16C0 for ; Wed, 11 Nov 2020 20:40:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C3C6E208CA for ; Wed, 11 Nov 2020 20:40:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b="Ir3b4tGV"; dkim=temperror (0-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="fwIe3Zwi" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C3C6E208CA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=sent.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7D5B66B005D; Wed, 11 Nov 2020 15:40:45 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5F7EA6B0070; Wed, 11 Nov 2020 15:40:45 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4710C6B0071; Wed, 11 Nov 2020 15:40:45 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0112.hostedemail.com [216.40.44.112]) by kanga.kvack.org (Postfix) with ESMTP id 020446B0068 for ; Wed, 11 Nov 2020 15:40:44 -0500 (EST) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 9D7FE3625 for ; Wed, 11 Nov 2020 20:40:44 +0000 (UTC) X-FDA: 77473306008.10.way94_360784b27300 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin10.hostedemail.com (Postfix) with ESMTP id 7BEE016A0BE for ; Wed, 11 Nov 2020 20:40:44 +0000 (UTC) X-Spam-Summary: 1,0,0,0355557fc5d0bd14,d41d8cd98f00b204,zi.yan@sent.com,,RULES_HIT:41:355:379:541:560:800:960:973:988:989:1260:1261:1311:1314:1345:1359:1437:1513:1515:1521:1535:1542:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:2731:3138:3139:3140:3141:3142:3353:3867:3868:3870:3872:4117:4321:4605:5007:6120:6261:6653:6742:7576:8603:8957:9010:10004:11026:11233:11658:11914:12043:12296:12438:12555:12679:12895:14181:14721:21080:21627:21740:21966:21990:30054:30064,0,RBL:64.147.123.26:@sent.com:.lbl8.mailshell.net-62.18.0.100 64.100.201.100;04yrx778cix4p3sq9ttou8aaq4d3gocwdu7xx1wsfgnfkm9dcdk9d5dakfmgtng.qkuexu37z19nxosb5h3khu5hma3tgxk6hf5dtr8s3e8b4t5bj91i63nbt1wjq7n.1-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:71,LUA_SUMMARY:none X-HE-Tag: way94_360784b27300 X-Filterd-Recvd-Size: 6550 Received: from wnew1-smtp.messagingengine.com (wnew1-smtp.messagingengine.com [64.147.123.26]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Wed, 11 Nov 2020 20:40:43 +0000 (UTC) Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailnew.west.internal (Postfix) with ESMTP id 5A4B4B40; Wed, 11 Nov 2020 15:40:41 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Wed, 11 Nov 2020 15:40:42 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=from :to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; s=fm2; bh=C1UmKosirAS8x RBgjcrsjxki5ADf6IPDpymKmd+0TMI=; b=Ir3b4tGVCsZhn5bMAVwbtio4VFv/D uvFcXmsU+FgU/MWp6ywhrcvjUy810sDC+0EMgLzlWYK9ffWlLljuGMBCCGM44IRk P+TigjnK8WgpvVD19UDiSOYE1ou8Lp7hJCZ/7UeX7qcZDDo+cuxdxSDem2ft7RoG PuwXGGzZLbzdDMCza1lb9Go3E2I0qunqYbv3kMaLMgmiPfE6cXDeKXRMAGB9m4hZ E5xUC1dJavxz9tifS4M+2uzJnCMsSFx2jNAHV545RNGMXNaqSGv6ZLHoA2SG8Po1 gpdNWPzpPpBlb3iIgE6YTfdobD5YwFoo9vA3355/CW7WgGkBAqVJOr1bQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:reply-to:subject :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; bh=C1UmKosirAS8xRBgjcrsjxki5ADf6IPDpymKmd+0TMI=; b=fwIe3Zwi /hTa3Z5fKz8I7PeJm7uSJJICOcBJvxIP9yuRqJoiczmUYpqmsdH7XK5PZ4XGIPmS T9S8fZmz6uH/UGu12BXR3jkogZDl/EBejFY9FIQ+uIEi0Bsvp6CEDEkQT5AwTxr4 h8lgPjYr+qYrVj6oObzC9hwT28PwpPXqrDHnql825AEpBZhiQW+aa3WEvlyMi7mr LhKF9/2LAuugMNROu5EXDqfw2geQ/FkST+Njwl/ZUPtTCUtJB8a+DKB7JBPt9Tx1 G9qxsgleSYagSMpgT9Xo9aMDb8lYlBW55nfl02T2YFkymLp6viVlv9plDhvciRGc dV5YHsCunzI87g== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedruddvtddgjeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfrhgggfestdhqredtredttdenucfhrhhomhepkghiucgj rghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepieejue dvueduuefhgefhheeiuedvtedvuefgieegveetueeiueehtdegudehfeelnecukfhppedu vddrgeeirddutdeirdduieegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpe hmrghilhhfrhhomhepiihirdihrghnsehsvghnthdrtghomh X-ME-Proxy: Received: from nvrsysarch6.NVidia.COM (unknown [12.46.106.164]) by mail.messagingengine.com (Postfix) with ESMTPA id 1C5A3328006D; Wed, 11 Nov 2020 15:40:40 -0500 (EST) From: Zi Yan To: linux-mm@kvack.org, Matthew Wilcox Cc: "Kirill A . Shutemov" , Roman Gushchin , Andrew Morton , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Yang Shi , Michal Hocko , John Hubbard , Ralph Campbell , David Nellans , Zi Yan Subject: [RFC PATCH 2/6] mm: memcg: make memcg huge page split support any order split. Date: Wed, 11 Nov 2020 15:40:04 -0500 Message-Id: <20201111204008.21332-3-zi.yan@sent.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201111204008.21332-1-zi.yan@sent.com> References: <20201111204008.21332-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan It reads thp_nr_pages and splits to provided new_nr. It prepares for upcoming changes to support split huge page to any lower order. Signed-off-by: Zi Yan Reviewed-by: Ralph Campbell Acked-by: Roman Gushchin --- include/linux/memcontrol.h | 5 +++-- mm/huge_memory.c | 2 +- mm/memcontrol.c | 4 ++-- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 0f4dd7829fb2..b3bac79ceed6 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1105,7 +1105,7 @@ static inline void memcg_memory_event_mm(struct mm_struct *mm, } #ifdef CONFIG_TRANSPARENT_HUGEPAGE -void mem_cgroup_split_huge_fixup(struct page *head); +void mem_cgroup_split_huge_fixup(struct page *head, unsigned int new_nr); #endif #else /* CONFIG_MEMCG */ @@ -1451,7 +1451,8 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, return 0; } -static inline void mem_cgroup_split_huge_fixup(struct page *head) +static inline void mem_cgroup_split_huge_fixup(struct page *head, + unsigned int new_nr) { } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c4fead5ead31..f599f5b9bf7f 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2429,7 +2429,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, lruvec = mem_cgroup_page_lruvec(head, pgdat); /* complete memcg works before add pages to LRU */ - mem_cgroup_split_huge_fixup(head); + mem_cgroup_split_huge_fixup(head, 1); if (PageAnon(head) && PageSwapCache(head)) { swp_entry_t entry = { .val = page_private(head) }; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 33f632689cee..e9705ba6bbcc 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3247,7 +3247,7 @@ void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size) * Because tail pages are not marked as "used", set it. We're under * pgdat->lru_lock and migration entries setup in all page mappings. */ -void mem_cgroup_split_huge_fixup(struct page *head) +void mem_cgroup_split_huge_fixup(struct page *head, unsigned int new_nr) { struct mem_cgroup *memcg = page_memcg(head); int i; @@ -3255,7 +3255,7 @@ void mem_cgroup_split_huge_fixup(struct page *head) if (mem_cgroup_disabled()) return; - for (i = 1; i < thp_nr_pages(head); i++) { + for (i = new_nr; i < thp_nr_pages(head); i += new_nr) { css_get(&memcg->css); head[i].memcg_data = (unsigned long)memcg; } From patchwork Wed Nov 11 20:40:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 11898507 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A47C416C0 for ; Wed, 11 Nov 2020 20:40:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 28934208B8 for ; Wed, 11 Nov 2020 20:40:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b="VDFMUHUe"; dkim=temperror (0-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="WO7cVfVL" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 28934208B8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=sent.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0CB0F6B0036; Wed, 11 Nov 2020 15:40:45 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 078B16B006E; Wed, 11 Nov 2020 15:40:44 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E0E1D6B006C; Wed, 11 Nov 2020 15:40:44 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0041.hostedemail.com [216.40.44.41]) by kanga.kvack.org (Postfix) with ESMTP id ABA456B0036 for ; Wed, 11 Nov 2020 15:40:44 -0500 (EST) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 47202181AC9CC for ; Wed, 11 Nov 2020 20:40:44 +0000 (UTC) X-FDA: 77473306008.26.mice84_4908e4e27300 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin26.hostedemail.com (Postfix) with ESMTP id 219411804B65A for ; Wed, 11 Nov 2020 20:40:44 +0000 (UTC) X-Spam-Summary: 1,0,0,8387c614c44d1b2e,d41d8cd98f00b204,zi.yan@sent.com,,RULES_HIT:41:355:379:541:560:800:960:973:988:989:1260:1261:1311:1314:1345:1359:1437:1513:1515:1521:1535:1543:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:2731:3138:3139:3140:3141:3142:3353:3867:3870:3872:4118:4321:4605:5007:6120:6261:6653:6742:7576:8957:9010:10004:11026:11473:11658:11914:12043:12296:12438:12555:12679:12895:12986:13215:13229:13255:14096:14181:14721:21080:21324:21627:21990:30054:30064,0,RBL:64.147.123.26:@sent.com:.lbl8.mailshell.net-62.18.0.100 64.100.201.100;04ygfxjcnpo1moc65z4hoapbx9a1mocbdqycp5qj5icohaz6exf6r9k9d8dbj6c.s97juwewt1hekgrje56c5y57h57p31qaycqrjuy4u9db41e3fwzj4z343mix8w9.h-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:153,LUA_SUMMARY:none X-HE-Tag: mice84_4908e4e27300 X-Filterd-Recvd-Size: 7135 Received: from wnew1-smtp.messagingengine.com (wnew1-smtp.messagingengine.com [64.147.123.26]) by imf47.hostedemail.com (Postfix) with ESMTP for ; Wed, 11 Nov 2020 20:40:43 +0000 (UTC) Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailnew.west.internal (Postfix) with ESMTP id 5B022B43; Wed, 11 Nov 2020 15:40:41 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Wed, 11 Nov 2020 15:40:42 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=from :to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; s=fm2; bh=e2DmXMYHMkbqy Lg+bqDlVe/Knfvv9mMQFeJGw1T5RTk=; b=VDFMUHUegXKz8lI5ggnc/LZRtDOoo 4EcN06pcjMGbGGmm2696Z3q8HyQugMw4H/NEnKhaL4GN67y2E9LTsDm0LEmdFJF1 bHGSDFaUsA6uiEtw/ELZf3dagGtnuYV4+k3uoXLje7aHXJB7uLRQDVoe4xNWT4id NWYv7+Fl7qMpixcMBqSWvSkz1o/8yxD5hOyskNn2+budKYAWQq5Yftmfu/LYSqyK 6SxHNW7CvHIJD+t/679n/pF/zXdDDDwxl0hE7xx80x4G6L2gUOPkmIN338tOiJY7 omH1Ag95o9zssQHhVO6s/0I1+Y+oa1GJCL7AvwcQU1qa8llp5Nbz6u7Xg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:reply-to:subject :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; bh=e2DmXMYHMkbqyLg+bqDlVe/Knfvv9mMQFeJGw1T5RTk=; b=WO7cVfVL Kf6OID8EvuIBRLlxwacD0XiqRBnjxV6zyv4sxMRsLI4AnFVWf/pXSod5W4b0tM46 ZPXCg2m23QPJrGhTmfp55myfP2ldMRBNq8SiAEjCr+OuksZxowHnZ1AqnwePGS47 G4TOJdyo7KTuFuV44vncUpgIy0RtE33UPbuv+fAbaMSAxna49v7WOJqPfGzKUMv0 l2zHUSng02djviihm+p7RPsY9UsssAxq3kRP3dMFC6caLQVwZUTvVUFX5kUBIJAn oe8bMIfD0lsJgf0nV9dDcGrPGEP/ef77fhgMrx3LyBiQw8tZl8/llh+6P/z2tk8U 07xK3sXkd2UzgQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedruddvtddgjeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfrhgggfestdhqredtredttdenucfhrhhomhepkghiucgj rghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepieejue dvueduuefhgefhheeiuedvtedvuefgieegveetueeiueehtdegudehfeelnecukfhppedu vddrgeeirddutdeirdduieegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpe hmrghilhhfrhhomhepiihirdihrghnsehsvghnthdrtghomh X-ME-Proxy: Received: from nvrsysarch6.NVidia.COM (unknown [12.46.106.164]) by mail.messagingengine.com (Postfix) with ESMTPA id 61CD43280067; Wed, 11 Nov 2020 15:40:40 -0500 (EST) From: Zi Yan To: linux-mm@kvack.org, Matthew Wilcox Cc: "Kirill A . Shutemov" , Roman Gushchin , Andrew Morton , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Yang Shi , Michal Hocko , John Hubbard , Ralph Campbell , David Nellans , Zi Yan Subject: [RFC PATCH 3/6] mm: page_owner: add support for splitting to any order in split page_owner. Date: Wed, 11 Nov 2020 15:40:05 -0500 Message-Id: <20201111204008.21332-4-zi.yan@sent.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201111204008.21332-1-zi.yan@sent.com> References: <20201111204008.21332-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan It adds a new_order parameter to set new page order in page owner. It prepares for upcoming changes to support split huge page to any lower order. Signed-off-by: Zi Yan Reviewed-by: Ralph Campbell --- include/linux/page_owner.h | 7 ++++--- mm/huge_memory.c | 2 +- mm/page_alloc.c | 2 +- mm/page_owner.c | 6 +++--- 4 files changed, 9 insertions(+), 8 deletions(-) diff --git a/include/linux/page_owner.h b/include/linux/page_owner.h index 3468794f83d2..215cbb159568 100644 --- a/include/linux/page_owner.h +++ b/include/linux/page_owner.h @@ -31,10 +31,11 @@ static inline void set_page_owner(struct page *page, __set_page_owner(page, order, gfp_mask); } -static inline void split_page_owner(struct page *page, unsigned int nr) +static inline void split_page_owner(struct page *page, unsigned int nr, + unsigned int new_order) { if (static_branch_unlikely(&page_owner_inited)) - __split_page_owner(page, nr); + __split_page_owner(page, nr, new_order); } static inline void copy_page_owner(struct page *oldpage, struct page *newpage) { @@ -60,7 +61,7 @@ static inline void set_page_owner(struct page *page, { } static inline void split_page_owner(struct page *page, - unsigned int order) + unsigned int nr, unsigned int new_order) { } static inline void copy_page_owner(struct page *oldpage, struct page *newpage) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index f599f5b9bf7f..8b7d771ee962 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2459,7 +2459,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, ClearPageCompound(head); - split_page_owner(head, nr); + split_page_owner(head, nr, 1); /* See comment in __split_huge_page_tail() */ if (PageAnon(head)) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d77220615fd5..a9eead0e091a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3284,7 +3284,7 @@ void split_page(struct page *page, unsigned int order) for (i = 1; i < (1 << order); i++) set_page_refcounted(page + i); - split_page_owner(page, 1 << order); + split_page_owner(page, 1 << order, 1); } EXPORT_SYMBOL_GPL(split_page); diff --git a/mm/page_owner.c b/mm/page_owner.c index b735a8eafcdb..2b7f7e9056dc 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -204,7 +204,7 @@ void __set_page_owner_migrate_reason(struct page *page, int reason) page_owner->last_migrate_reason = reason; } -void __split_page_owner(struct page *page, unsigned int nr) +void __split_page_owner(struct page *page, unsigned int nr, unsigned int new_order) { int i; struct page_ext *page_ext = lookup_page_ext(page); @@ -213,9 +213,9 @@ void __split_page_owner(struct page *page, unsigned int nr) if (unlikely(!page_ext)) return; - for (i = 0; i < nr; i++) { + for (i = 0; i < nr; i += (1 << new_order)) { page_owner = get_page_owner(page_ext); - page_owner->order = 0; + page_owner->order = new_order; page_ext = page_ext_next(page_ext); } } From patchwork Wed Nov 11 20:40:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 11898519 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 65A21921 for ; Wed, 11 Nov 2020 20:41:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D2B172100A for ; Wed, 11 Nov 2020 20:40:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b="nNqAHO1c"; dkim=temperror (0-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="mK6KtfCe" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D2B172100A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=sent.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AEBF76B006C; Wed, 11 Nov 2020 15:40:45 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9A45B6B006E; Wed, 11 Nov 2020 15:40:45 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D0736B0071; Wed, 11 Nov 2020 15:40:45 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 43DDD6B006E for ; Wed, 11 Nov 2020 15:40:45 -0500 (EST) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id D3FE6824999B for ; Wed, 11 Nov 2020 20:40:44 +0000 (UTC) X-FDA: 77473306008.15.pie41_0e0c1a427300 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin15.hostedemail.com (Postfix) with ESMTP id AB5031814B0C7 for ; Wed, 11 Nov 2020 20:40:44 +0000 (UTC) X-Spam-Summary: 1,0,0,a7749a7806d872e9,d41d8cd98f00b204,zi.yan@sent.com,,RULES_HIT:1:41:69:355:379:541:560:800:960:966:968:973:988:989:1260:1261:1311:1314:1345:1359:1437:1513:1515:1521:1605:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:2636:2693:2731:2898:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:4250:4321:4385:4605:5007:6117:6119:6120:6261:6653:6742:7576:7903:8603:8957:9010:10004:11026:11232:11473:11658:11914:12043:12291:12296:12438:12555:12679:12683:12895:12986:13161:13172:13215:13229:13255:14093:14096:21080:21627:21740:21990:30034:30054:30064:30070,0,RBL:64.147.123.26:@sent.com:.lbl8.mailshell.net-62.18.0.100 64.100.201.100;04yrcsk5wczmh1ce7qm5moqa3c319ochban7b4qc46pzepq9hoxmopb3scenczr.5pnupbmukhhgtageurhzgpff9khqfdfzn5rq3dmn3kgu9a9itf777q88hc4s6qq.k-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:104,LUA_SUMMARY:none X-HE-Tag: pie41_0e0c1a427300 X-Filterd-Recvd-Size: 14216 Received: from wnew1-smtp.messagingengine.com (wnew1-smtp.messagingengine.com [64.147.123.26]) by imf07.hostedemail.com (Postfix) with ESMTP for ; Wed, 11 Nov 2020 20:40:43 +0000 (UTC) Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailnew.west.internal (Postfix) with ESMTP id AB7BAB44; Wed, 11 Nov 2020 15:40:41 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Wed, 11 Nov 2020 15:40:42 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=from :to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; s=fm2; bh=FFtNPcXr4xGJ9 kwfOrLCBoFxL6hUUpz35sirIAoWiiA=; b=nNqAHO1cgSD8k1/k0UX7u64YpLvHJ RN5VLlFA+6q9RP8y1geK6BZ7qOXGB2Vj7TK6CgJYSoFtjHogYDPLSQo8H3Qpzpmy ptvVSkPedS4IMPYqVpC07nq2rRnpsqqEEbfGfb69TFwdGsSQtaJp7anDMO5hJLLa zUKruMxkg0ydPQfR48JmlUOa86lFAvMK/AOV8nLqBAa+2MKUV027HDR4vId+hT1U XpVlA8ZmpL4PWyQZTQjqiTvSvEystURTvYGiUtruj0b2ShBOfRmQJ68Fy+H8kJzy fGe+ETvhTsbH/+CMNc2ZgR40tXX9fw4L3w1M5//cYoUb2h++qam66zrPQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:reply-to:subject :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; bh=FFtNPcXr4xGJ9kwfOrLCBoFxL6hUUpz35sirIAoWiiA=; b=mK6KtfCe pdcav9+CQDBPGtDBrzCpjPeMOimHbPcD8pH1KlyCwypNLkfsKha04nRt1L04xLKi MWRF5g5gjJX0TpjkmnVs2lItP4RUV/KOSYUNNaSW5QbnTbiHMDrv/mPRQX7ArfNk X0JxmzknIBwDrZPo/mYr4GfmfJGA8o6hgQrPtC3LiHafWaD+S5XGY+rUjedKVEvM KbbvuC/y5FHBLOjzo/cPOojOCdVxj0ASwF2VvA2+oS4PL8czI1MVS9C0qp9NdFfs GbWF0fZhC12vb3mE3Ix0DzzEtVBzyClm6875kgX8FX1Gg6Jhhu45mDQAM0IET38W MIdxx2bYCc1QVQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedruddvtddgjeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfrhgggfestdhqredtredttdenucfhrhhomhepkghiucgj rghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepieejue dvueduuefhgefhheeiuedvtedvuefgieegveetueeiueehtdegudehfeelnecukfhppedu vddrgeeirddutdeirdduieegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpe hmrghilhhfrhhomhepiihirdihrghnsehsvghnthdrtghomh X-ME-Proxy: Received: from nvrsysarch6.NVidia.COM (unknown [12.46.106.164]) by mail.messagingengine.com (Postfix) with ESMTPA id A773B3280066; Wed, 11 Nov 2020 15:40:40 -0500 (EST) From: Zi Yan To: linux-mm@kvack.org, Matthew Wilcox Cc: "Kirill A . Shutemov" , Roman Gushchin , Andrew Morton , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Yang Shi , Michal Hocko , John Hubbard , Ralph Campbell , David Nellans , Zi Yan Subject: [RFC PATCH 4/6] mm: thp: add support for split huge page to any lower order pages. Date: Wed, 11 Nov 2020 15:40:06 -0500 Message-Id: <20201111204008.21332-5-zi.yan@sent.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201111204008.21332-1-zi.yan@sent.com> References: <20201111204008.21332-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan To split a THP to any lower order pages, we need to reform THPs on subpages at given order and add page refcount based on the new page order. Also we need to reinitialize page_deferred_list after removing the page from the split_queue, otherwise a subsequent split will see list corruption when checking the page_deferred_list again. It has many uses, like minimizing the number of pages after truncating a pagecache THP. For anonymous THPs, we can only split them to order-0 like before until we add support for any size anonymous THPs. Signed-off-by: Zi Yan --- include/linux/huge_mm.h | 8 +++++ mm/huge_memory.c | 78 +++++++++++++++++++++++++++++------------ mm/swap.c | 1 - 3 files changed, 63 insertions(+), 24 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 60a907a19f7d..9819cd9b4619 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -189,6 +189,8 @@ bool is_transparent_hugepage(struct page *page); bool can_split_huge_page(struct page *page, int *pextra_pins); int split_huge_page_to_list(struct page *page, struct list_head *list); +int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, + unsigned int new_order); static inline int split_huge_page(struct page *page) { return split_huge_page_to_list(page, NULL); @@ -396,6 +398,12 @@ split_huge_page_to_list(struct page *page, struct list_head *list) { return 0; } +static inline int +split_huge_page_to_order_to_list(struct page *page, struct list_head *list, + unsigned int new_order) +{ + return 0; +} static inline int split_huge_page(struct page *page) { return 0; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 8b7d771ee962..88f50da40c9b 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2327,11 +2327,14 @@ void vma_adjust_trans_huge(struct vm_area_struct *vma, static void unmap_page(struct page *page) { enum ttu_flags ttu_flags = TTU_IGNORE_MLOCK | TTU_IGNORE_ACCESS | - TTU_RMAP_LOCKED | TTU_SPLIT_HUGE_PMD; + TTU_RMAP_LOCKED; bool unmap_success; VM_BUG_ON_PAGE(!PageHead(page), page); + if (thp_order(page) >= HPAGE_PMD_ORDER) + ttu_flags |= TTU_SPLIT_HUGE_PMD; + if (PageAnon(page)) ttu_flags |= TTU_SPLIT_FREEZE; @@ -2339,21 +2342,22 @@ static void unmap_page(struct page *page) VM_BUG_ON_PAGE(!unmap_success, page); } -static void remap_page(struct page *page, unsigned int nr) +static void remap_page(struct page *page, unsigned int nr, unsigned int new_nr) { int i; - if (PageTransHuge(page)) { + if (thp_nr_pages(page) == nr) { remove_migration_ptes(page, page, true); } else { - for (i = 0; i < nr; i++) + for (i = 0; i < nr; i += new_nr) remove_migration_ptes(page + i, page + i, true); } } static void __split_huge_page_tail(struct page *head, int tail, - struct lruvec *lruvec, struct list_head *list) + struct lruvec *lruvec, struct list_head *list, unsigned int new_order) { struct page *page_tail = head + tail; + unsigned long compound_head_flag = new_order ? (1L << PG_head) : 0; VM_BUG_ON_PAGE(atomic_read(&page_tail->_mapcount) != -1, page_tail); @@ -2377,6 +2381,7 @@ static void __split_huge_page_tail(struct page *head, int tail, #ifdef CONFIG_64BIT (1L << PG_arch_2) | #endif + compound_head_flag | (1L << PG_dirty))); /* ->mapping in first tail page is compound_mapcount */ @@ -2395,10 +2400,15 @@ static void __split_huge_page_tail(struct page *head, int tail, * which needs correct compound_head(). */ clear_compound_head(page_tail); + if (new_order) { + prep_compound_page(page_tail, new_order); + thp_prep(page_tail); + } /* Finally unfreeze refcount. Additional reference from page cache. */ - page_ref_unfreeze(page_tail, 1 + (!PageAnon(head) || - PageSwapCache(head))); + page_ref_unfreeze(page_tail, 1 + ((!PageAnon(head) || + PageSwapCache(head)) ? + thp_nr_pages(page_tail) : 0)); if (page_is_young(head)) set_page_young(page_tail); @@ -2416,7 +2426,7 @@ static void __split_huge_page_tail(struct page *head, int tail, } static void __split_huge_page(struct page *page, struct list_head *list, - pgoff_t end, unsigned long flags) + pgoff_t end, unsigned long flags, unsigned int new_order) { struct page *head = compound_head(page); pg_data_t *pgdat = page_pgdat(head); @@ -2424,12 +2434,13 @@ static void __split_huge_page(struct page *page, struct list_head *list, struct address_space *swap_cache = NULL; unsigned long offset = 0; unsigned int nr = thp_nr_pages(head); + unsigned int new_nr = 1 << new_order; int i; lruvec = mem_cgroup_page_lruvec(head, pgdat); /* complete memcg works before add pages to LRU */ - mem_cgroup_split_huge_fixup(head, 1); + mem_cgroup_split_huge_fixup(head, new_nr); if (PageAnon(head) && PageSwapCache(head)) { swp_entry_t entry = { .val = page_private(head) }; @@ -2439,14 +2450,14 @@ static void __split_huge_page(struct page *page, struct list_head *list, xa_lock(&swap_cache->i_pages); } - for (i = nr - 1; i >= 1; i--) { - __split_huge_page_tail(head, i, lruvec, list); + for (i = nr - new_nr; i >= new_nr; i -= new_nr) { + __split_huge_page_tail(head, i, lruvec, list, new_order); /* Some pages can be beyond i_size: drop them from page cache */ if (head[i].index >= end) { ClearPageDirty(head + i); __delete_from_page_cache(head + i, NULL); if (IS_ENABLED(CONFIG_SHMEM) && PageSwapBacked(head)) - shmem_uncharge(head->mapping->host, 1); + shmem_uncharge(head->mapping->host, new_nr); put_page(head + i); } else if (!PageAnon(page)) { __xa_store(&head->mapping->i_pages, head[i].index, @@ -2457,28 +2468,31 @@ static void __split_huge_page(struct page *page, struct list_head *list, } } - ClearPageCompound(head); + if (!new_order) + ClearPageCompound(head); + else + set_compound_order(head, new_order); - split_page_owner(head, nr, 1); + split_page_owner(head, nr, new_nr); /* See comment in __split_huge_page_tail() */ if (PageAnon(head)) { /* Additional pin to swap cache */ if (PageSwapCache(head)) { - page_ref_add(head, 2); + page_ref_add(head, 1 + new_nr); xa_unlock(&swap_cache->i_pages); } else { page_ref_inc(head); } } else { /* Additional pin to page cache */ - page_ref_add(head, 2); + page_ref_add(head, 1 + new_nr); xa_unlock(&head->mapping->i_pages); } spin_unlock_irqrestore(&pgdat->lru_lock, flags); - remap_page(head, nr); + remap_page(head, nr, new_nr); if (PageSwapCache(head)) { swp_entry_t entry = { .val = page_private(head) }; @@ -2486,7 +2500,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, split_swap_cluster(entry); } - for (i = 0; i < nr; i++) { + for (i = 0; i < nr; i += new_nr) { struct page *subpage = head + i; if (subpage == page) continue; @@ -2620,21 +2634,39 @@ bool can_split_huge_page(struct page *page, int *pextra_pins) * us. */ int split_huge_page_to_list(struct page *page, struct list_head *list) +{ + return split_huge_page_to_list_to_order(page, list, 0); +} + +int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, + unsigned int new_order) { struct page *head = compound_head(page); struct pglist_data *pgdata = NODE_DATA(page_to_nid(head)); struct deferred_split *ds_queue = get_deferred_split_queue(head); - XA_STATE(xas, &head->mapping->i_pages, head->index); + /* reset xarray order to new order after split */ + XA_STATE_ORDER(xas, &head->mapping->i_pages, head->index, new_order); struct anon_vma *anon_vma = NULL; struct address_space *mapping = NULL; int count, mapcount, extra_pins, ret; unsigned long flags; pgoff_t end; + VM_BUG_ON(thp_order(head) <= new_order); VM_BUG_ON_PAGE(is_huge_zero_page(head), head); VM_BUG_ON_PAGE(!PageLocked(head), head); VM_BUG_ON_PAGE(!PageCompound(head), head); + if (new_order == 1) { + WARN_ONCE(1, "Cannot split THP to order-1 (no order-1 THPs)"); + return -EINVAL; + } + + if (PageAnon(head) && new_order) { + WARN_ONCE(1, "Split anonymous THP to non-zero order not support"); + return -EINVAL; + } + if (PageWriteback(head)) return -EBUSY; @@ -2720,18 +2752,18 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) if (!mapcount && page_ref_freeze(head, 1 + extra_pins)) { if (!list_empty(page_deferred_list(head))) { ds_queue->split_queue_len--; - list_del(page_deferred_list(head)); + list_del_init(page_deferred_list(head)); } spin_unlock(&ds_queue->split_queue_lock); if (mapping) { if (PageSwapBacked(head)) __dec_lruvec_page_state(head, NR_SHMEM_THPS); - else + else if (!new_order) __mod_lruvec_page_state(head, NR_FILE_THPS, -thp_nr_pages(head)); } - __split_huge_page(page, list, end, flags); + __split_huge_page(page, list, end, flags, new_order); ret = 0; } else { if (IS_ENABLED(CONFIG_DEBUG_VM) && mapcount) { @@ -2746,7 +2778,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) fail: if (mapping) xas_unlock(&xas); spin_unlock_irqrestore(&pgdata->lru_lock, flags); - remap_page(head, thp_nr_pages(head)); + remap_page(head, thp_nr_pages(head), 0); ret = -EBUSY; } diff --git a/mm/swap.c b/mm/swap.c index 14c3bac607a6..6c33e6165597 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -983,7 +983,6 @@ void lru_add_page_tail(struct page *page, struct page *page_tail, struct lruvec *lruvec, struct list_head *list) { VM_BUG_ON_PAGE(!PageHead(page), page); - VM_BUG_ON_PAGE(PageCompound(page_tail), page); VM_BUG_ON_PAGE(PageLRU(page_tail), page); lockdep_assert_held(&lruvec_pgdat(lruvec)->lru_lock); From patchwork Wed Nov 11 20:40:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 11898515 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8C767921 for ; Wed, 11 Nov 2020 20:41:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2F949208FE for ; Wed, 11 Nov 2020 20:41:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b="G28shBuM"; dkim=temperror (0-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="jbfenD4g" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2F949208FE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=sent.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DAA486B0070; Wed, 11 Nov 2020 15:40:46 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D284F6B0072; Wed, 11 Nov 2020 15:40:46 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 86C9F6B0073; Wed, 11 Nov 2020 15:40:46 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0253.hostedemail.com [216.40.44.253]) by kanga.kvack.org (Postfix) with ESMTP id 178A86B0070 for ; Wed, 11 Nov 2020 15:40:46 -0500 (EST) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id A88EF180AD801 for ; Wed, 11 Nov 2020 20:40:45 +0000 (UTC) X-FDA: 77473306050.30.fall63_0711c8e27300 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin30.hostedemail.com (Postfix) with ESMTP id 7D34D180B3C8E for ; Wed, 11 Nov 2020 20:40:45 +0000 (UTC) X-Spam-Summary: 1,0,0,8952cd76c081b386,d41d8cd98f00b204,zi.yan@sent.com,,RULES_HIT:41:355:379:541:560:800:960:973:988:989:1260:1261:1311:1314:1345:1359:1437:1513:1515:1521:1535:1542:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:2731:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3872:4117:4250:5007:6119:6120:6261:6653:6742:7576:7875:7901:8957:10004:11026:11658:11914:12043:12291:12296:12438:12555:12679:12895:13161:13229:14096:14181:14721:21080:21324:21627:21990:30034:30054:30064,0,RBL:64.147.123.26:@sent.com:.lbl8.mailshell.net-62.18.0.100 64.100.201.100;04y88s4h543a6iunqwq8ne8m6i6zeycw64pt6b93rqt7rmui4k3iws743d7uo1h.aiz1rc6seujnmhpyxiy43nepiir5xgw5nykx6wa8jznqrtyzxqg6htnzxh935jp.n-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:70,LUA_SUMMARY:none X-HE-Tag: fall63_0711c8e27300 X-Filterd-Recvd-Size: 6128 Received: from wnew1-smtp.messagingengine.com (wnew1-smtp.messagingengine.com [64.147.123.26]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Wed, 11 Nov 2020 20:40:44 +0000 (UTC) Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailnew.west.internal (Postfix) with ESMTP id E2A62B45; Wed, 11 Nov 2020 15:40:41 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Wed, 11 Nov 2020 15:40:42 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=from :to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; s=fm2; bh=Ao4OQzNEMkLUB rt8fQw2khsG2IVAwlRTCnIBOCh4AzI=; b=G28shBuMvus0pki1eu5Vkq4Grf0lO b34p4MR90hWIeKICBMvCssewrtQ3rLrRBNfzSeNDp8ARtpsflL2oo4bOWjrCn+yc 5h8y8K9DzFYRM7yR7Crb7VnlIDqYG8S4BcAm9KYa5T5WNZk/m9zmepDofqC2iIFw Mx+dQAq4ftmG7+MBAYkKeY+TAbfGGeQ8wdJ5yDSfvxnFsPVvErvKk3rAAFMN5j3I N99KJvcP2cyos451wXIsc0x18J/TJPA9ZWarZ3IuIYg6jA5UYqcXp6+bQeHbKxoY +S5KmSrMhCjlJbjGKkAKAdsfOboH99oQUy6pr5p0hDflQqOZfniVT9zuw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:reply-to:subject :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; bh=Ao4OQzNEMkLUBrt8fQw2khsG2IVAwlRTCnIBOCh4AzI=; b=jbfenD4g zwnLr0RWKlmTz1To0IBFs7Tld1sqLWJrUZUm6le2AJdhi7D86YBjXf3zfOsb0MA0 mtuq6K+t8lFOht4xCpVZD2h4+y8T1EDzKqJpRDiqSMdt3Y3V6VkvsEs/tO6ZgVYq /HuJD2sQm2vAO+PVYD3s825UenK0atu7XqIgJYBhtQBxIL5o8gunwGLwMLMPk4zO VAXc9rpf2P1OA2lKGbKN7ssGONkwVZRHcg0xllPNrR8icvmUZjZAbauP5qHu4nyS 6nOiZXuVxzrbbOEVFoG577eKcQeuSFGD0z3I//Q3dYGJjHMq58N00tkA7VzJGizq OlJ/lIa8FV6dsw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedruddvtddgjeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfrhgggfestdhqredtredttdenucfhrhhomhepkghiucgj rghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepieejue dvueduuefhgefhheeiuedvtedvuefgieegveetueeiueehtdegudehfeelnecukfhppedu vddrgeeirddutdeirdduieegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpe hmrghilhhfrhhomhepiihirdihrghnsehsvghnthdrtghomh X-ME-Proxy: Received: from nvrsysarch6.NVidia.COM (unknown [12.46.106.164]) by mail.messagingengine.com (Postfix) with ESMTPA id ED343328006B; Wed, 11 Nov 2020 15:40:40 -0500 (EST) From: Zi Yan To: linux-mm@kvack.org, Matthew Wilcox Cc: "Kirill A . Shutemov" , Roman Gushchin , Andrew Morton , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Yang Shi , Michal Hocko , John Hubbard , Ralph Campbell , David Nellans , Zi Yan Subject: [RFC PATCH 5/6] mm: truncate: split thp to a non-zero order if possible. Date: Wed, 11 Nov 2020 15:40:07 -0500 Message-Id: <20201111204008.21332-6-zi.yan@sent.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201111204008.21332-1-zi.yan@sent.com> References: <20201111204008.21332-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan To minimize the number of pages after a truncation, when truncating a THP, we do not need to split it all the way down to order-0. The THP has at most three parts, the part before offset, the part to be truncated, the part left at the end. Use the non-zero minimum of them to decide what order we split the THP to. Signed-off-by: Zi Yan --- mm/truncate.c | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/mm/truncate.c b/mm/truncate.c index 20bd17538ec2..6d8e3c6115bc 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -237,7 +237,7 @@ int truncate_inode_page(struct address_space *mapping, struct page *page) bool truncate_inode_partial_page(struct page *page, loff_t start, loff_t end) { loff_t pos = page_offset(page); - unsigned int offset, length; + unsigned int offset, length, left, min_subpage_size = PAGE_SIZE; if (pos < start) offset = start - pos; @@ -248,6 +248,7 @@ bool truncate_inode_partial_page(struct page *page, loff_t start, loff_t end) length = length - offset; else length = end + 1 - pos - offset; + left = thp_size(page) - offset - length; wait_on_page_writeback(page); if (length == thp_size(page)) { @@ -267,7 +268,24 @@ bool truncate_inode_partial_page(struct page *page, loff_t start, loff_t end) do_invalidatepage(page, offset, length); if (!PageTransHuge(page)) return true; - return split_huge_page(page) == 0; + + /* + * find the non-zero minimum of offset, length, and left and use it to + * decide the new order of the page after split + */ + if (offset && left) + min_subpage_size = min_t(unsigned int, + min_t(unsigned int, offset, length), + left); + else if (!offset) + min_subpage_size = min_t(unsigned int, length, left); + else /* !left */ + min_subpage_size = min_t(unsigned int, length, offset); + + min_subpage_size = max_t(unsigned int, PAGE_SIZE, min_subpage_size); + + return split_huge_page_to_list_to_order(page, NULL, + ilog2(min_subpage_size/PAGE_SIZE)) == 0; } /* From patchwork Wed Nov 11 20:40:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 11898517 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3827116C0 for ; Wed, 11 Nov 2020 20:41:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B292420797 for ; Wed, 11 Nov 2020 20:40:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b="tgYh9jFI"; dkim=temperror (0-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="iri+JizH" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B292420797 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=sent.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AB7566B006E; Wed, 11 Nov 2020 15:40:46 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A3EEB6B0071; Wed, 11 Nov 2020 15:40:46 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5FBF56B006E; Wed, 11 Nov 2020 15:40:46 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0053.hostedemail.com [216.40.44.53]) by kanga.kvack.org (Postfix) with ESMTP id 1E0AC6B0071 for ; Wed, 11 Nov 2020 15:40:46 -0500 (EST) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id B57F71EE6 for ; Wed, 11 Nov 2020 20:40:45 +0000 (UTC) X-FDA: 77473306050.02.home39_060e82027300 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id 9755B10097AA1 for ; Wed, 11 Nov 2020 20:40:45 +0000 (UTC) X-Spam-Summary: 1,0,0,be3dd4c93bcef2a5,d41d8cd98f00b204,zi.yan@sent.com,,RULES_HIT:1:2:41:355:379:541:560:800:960:966:973:988:989:1260:1261:1311:1314:1345:1359:1437:1513:1515:1521:1730:1747:1777:1792:1801:2196:2198:2199:2200:2393:2559:2562:2731:3138:3139:3140:3141:3142:3355:3865:3867:3868:3870:3871:3872:4049:4321:4385:4605:5007:6120:6261:6653:6742:7576:7875:8603:9036:10004:11026:11473:11657:11658:11914:12043:12296:12438:12555:12679:12895:13255:21080:21324:21451:21611:21627:21990:30054:30056:30064:30070,0,RBL:64.147.123.26:@sent.com:.lbl8.mailshell.net-64.100.201.100 62.18.0.100;04yriif9wig46mpxr9byhet4nim14yptnp7p1ham1ta11b9r6iruafqkygtuqqc.w5ypiux6q3y6mu6tct41jug3ebtamza4hitioeeabscmx4ih6izrqui9dyxbczw.q-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:68,LUA_SUMMARY:none X-HE-Tag: home39_060e82027300 X-Filterd-Recvd-Size: 10155 Received: from wnew1-smtp.messagingengine.com (wnew1-smtp.messagingengine.com [64.147.123.26]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Wed, 11 Nov 2020 20:40:44 +0000 (UTC) Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailnew.west.internal (Postfix) with ESMTP id 32681A17; Wed, 11 Nov 2020 15:40:42 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Wed, 11 Nov 2020 15:40:43 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=from :to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; s=fm2; bh=3tDLBowVRIwdz IAbCVfBnGR6mqNvKC6lGI+I9eRGP1c=; b=tgYh9jFINu/CLM52sZOzJ3PyVhFcB F+QMBFot3z1lIpdUSlvLbQzmo45w3X3WxHo0YYyFPLnNKR0A9ry0LcwX5q9kmpFr tCtdsmx7trFjT6hNAslVM7bRZREBgI0ZXLLoKlGuNwvWar2DXNARDEUrb6I8C2b+ zCO7P6Syt3VmUa/16Lf2NDPcWD8jqCFXFQJ/93szTG2ceM67gKYLNHNwWoKbvNRg OvdXMzCWj7YAwdv1MB/FELN+149Hj0Dx6TQOhdbRYU83WcUQEHOVJ9ObC8MUu4Jr xhSm7VUm5AuJVIr8VRud1LTP8M4HqkFKSM4I8mHjxc1fRMn4DrpPR8lLA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:reply-to:subject :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; bh=3tDLBowVRIwdzIAbCVfBnGR6mqNvKC6lGI+I9eRGP1c=; b=iri+JizH HvHd8FyUbpHT/ZkIsS+0/aYycXg8HMpN7EM465b6RmbagT7hC+OD2JMv0D9TYuHi 03A8jIvoNH5EYIskaQgE8VQR1r/rGKse25soUj42Bg4gcyrgE0ouHwRcGehZMAKs N6fzdgqbSzxoXhpcaGB8E4Oq4MBvyZ8uI5iu/uQ3mdtXaoZ9x+7FAySUV4a2KqHh lMTz89MH4st/I611ycmkSqyMHem7GMbPiALOIEh6PTuurfZGDO50AtuGRJg6cufi nh8MM2fECYHZq1r2tLG/LJlthaAeF24aAnKS3dSsfroq0t2KBxJ66yK4vWURY+VO rZDxeuT7JiR3RA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedruddvtddgjeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfrhgggfestdhqredtredttdenucfhrhhomhepkghiucgj rghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepieejue dvueduuefhgefhheeiuedvtedvuefgieegveetueeiueehtdegudehfeelnecukfhppedu vddrgeeirddutdeirdduieegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpe hmrghilhhfrhhomhepiihirdihrghnsehsvghnthdrtghomh X-ME-Proxy: Received: from nvrsysarch6.NVidia.COM (unknown [12.46.106.164]) by mail.messagingengine.com (Postfix) with ESMTPA id 3EDFA3280063; Wed, 11 Nov 2020 15:40:41 -0500 (EST) From: Zi Yan To: linux-mm@kvack.org, Matthew Wilcox Cc: "Kirill A . Shutemov" , Roman Gushchin , Andrew Morton , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Yang Shi , Michal Hocko , John Hubbard , Ralph Campbell , David Nellans , Zi Yan Subject: [RFC PATCH 6/6] mm: huge_memory: enable debugfs to split huge pages to any order. Date: Wed, 11 Nov 2020 15:40:08 -0500 Message-Id: <20201111204008.21332-7-zi.yan@sent.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201111204008.21332-1-zi.yan@sent.com> References: <20201111204008.21332-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan It is used to test split_huge_page_to_list_to_order for pagecache THPs. Also add test cases for split_huge_page_to_list_to_order via both debugfs and truncating a file. Signed-off-by: Zi Yan --- mm/huge_memory.c | 13 +-- .../selftests/vm/split_huge_page_test.c | 102 +++++++++++++++++- 2 files changed, 105 insertions(+), 10 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 88f50da40c9b..b7470607a08b 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2974,7 +2974,7 @@ static ssize_t split_huge_pages_in_range_pid_write(struct file *file, static DEFINE_MUTEX(mutex); ssize_t ret; char input_buf[80]; /* hold pid, start_vaddr, end_vaddr */ - int pid; + int pid, to_order = 0; unsigned long vaddr_start, vaddr_end, addr; nodemask_t task_nodes; struct mm_struct *mm; @@ -2990,8 +2990,9 @@ static ssize_t split_huge_pages_in_range_pid_write(struct file *file, goto out; input_buf[80] = '\0'; - ret = sscanf(input_buf, "%d,%lx,%lx", &pid, &vaddr_start, &vaddr_end); - if (ret != 3) { + ret = sscanf(input_buf, "%d,%lx,%lx,%d", &pid, &vaddr_start, &vaddr_end, &to_order); + /* cannot split to order-1 THP, which is not possible */ + if ((ret != 3 && ret != 4) || to_order == 1) { ret = -EINVAL; goto out; } @@ -2999,8 +3000,8 @@ static ssize_t split_huge_pages_in_range_pid_write(struct file *file, vaddr_end &= PAGE_MASK; ret = strlen(input_buf); - pr_debug("split huge pages in pid: %d, vaddr: [%lx - %lx]\n", - pid, vaddr_start, vaddr_end); + pr_debug("split huge pages in pid: %d, vaddr: [%lx - %lx], to order: %d\n", + pid, vaddr_start, vaddr_end, to_order); mm = find_mm_struct(pid, &task_nodes); if (IS_ERR(mm)) { @@ -3038,7 +3039,7 @@ static ssize_t split_huge_pages_in_range_pid_write(struct file *file, addr += page_size(page) - PAGE_SIZE; /* reset addr if split fails */ - if (split_huge_page(page)) + if (split_huge_page_to_list_to_order(page, NULL, to_order)) addr -= (page_size(page) - PAGE_SIZE); unlock_page(page); diff --git a/tools/testing/selftests/vm/split_huge_page_test.c b/tools/testing/selftests/vm/split_huge_page_test.c index c8a32ae9e13a..bcbc5a9d327c 100644 --- a/tools/testing/selftests/vm/split_huge_page_test.c +++ b/tools/testing/selftests/vm/split_huge_page_test.c @@ -16,6 +16,7 @@ #include #include #include +#include #define PAGE_4KB (4096UL) #define PAGE_2MB (512UL*PAGE_4KB) @@ -31,6 +32,7 @@ #define SPLIT_DEBUGFS "/sys/kernel/debug/split_huge_pages_in_range_pid" #define SMAP_PATH "/proc/self/smaps" +#define THP_FS_PATH "/mnt/thp_fs" #define INPUT_MAX 80 static int write_file(const char *path, const char *buf, size_t buflen) @@ -50,13 +52,13 @@ static int write_file(const char *path, const char *buf, size_t buflen) return (unsigned int) numwritten; } -static void write_debugfs(int pid, uint64_t vaddr_start, uint64_t vaddr_end) +static void write_debugfs(int pid, uint64_t vaddr_start, uint64_t vaddr_end, int order) { char input[INPUT_MAX]; int ret; - ret = snprintf(input, INPUT_MAX, "%d,%lx,%lx", pid, vaddr_start, - vaddr_end); + ret = snprintf(input, INPUT_MAX, "%d,%lx,%lx,%d", pid, vaddr_start, + vaddr_end, order); if (ret >= INPUT_MAX) { printf("%s: Debugfs input is too long\n", __func__); exit(EXIT_FAILURE); @@ -139,7 +141,7 @@ void split_pmd_thp(void) } /* split all possible huge pages */ - write_debugfs(getpid(), (uint64_t)one_page, (uint64_t)one_page + len); + write_debugfs(getpid(), (uint64_t)one_page, (uint64_t)one_page + len, 0); *one_page = 0; @@ -153,9 +155,101 @@ void split_pmd_thp(void) free(one_page); } +void create_pagecache_thp_and_fd(size_t fd_size, int *fd, char **addr) +{ + const char testfile[] = THP_FS_PATH "/test"; + size_t i; + int dummy; + + srand(time(NULL)); + + *fd = open(testfile, O_CREAT | O_RDWR, 0664); + + for (i = 0; i < fd_size; i++) { + unsigned char byte = rand(); + + write(*fd, &byte, sizeof(byte)); + } + close(*fd); + sync(); + *fd = open("/proc/sys/vm/drop_caches", O_WRONLY); + if (*fd == -1) { + perror("open drop_caches"); + exit(EXIT_FAILURE); + } + if (write(*fd, "3", 1) != 1) { + perror("write to drop_caches"); + exit(EXIT_FAILURE); + } + close(*fd); + + *fd = open(testfile, O_RDWR); + + *addr = mmap(NULL, fd_size, PROT_READ|PROT_WRITE, MAP_SHARED, *fd, 0); + if (*addr == (char *)-1) { + perror("cannot mmap"); + exit(1); + } + madvise(*addr, fd_size, MADV_HUGEPAGE); + + for (size_t i = 0; i < fd_size; i++) + dummy += *(*addr + i); +} + +void split_thp_in_pagecache_to_order(int order) +{ + int fd; + char *addr; + size_t fd_size = 2 * PAGE_2MB, i; + + create_pagecache_thp_and_fd(fd_size, &fd, &addr); + + printf("split %ld kB pagecache page to order %d ... ", fd_size >> 10, order); + write_debugfs(getpid(), (uint64_t)addr, (uint64_t)addr + fd_size, order); + + for (i = 0; i < fd_size; i++) + *(addr + i) = (char)i; + + close(fd); + printf("done\n"); +} + +void truncate_thp_in_pagecache_to_order(int order) +{ + int fd; + char *addr; + size_t fd_size = 2 * PAGE_2MB, i; + + create_pagecache_thp_and_fd(fd_size, &fd, &addr); + + printf("truncate %ld kB pagecache page to size %lu kB ... ", fd_size >> 10, 4UL << order); + ftruncate(fd, PAGE_4KB << order); + + for (i = 0; i < (PAGE_4KB << order); i++) + *(addr + i) = (char)i; + + close(fd); + printf("done\n"); +} + int main(int argc, char **argv) { + int i; + + if (geteuid() != 0) { + printf("Please run the benchmark as root\n"); + exit(EXIT_FAILURE); + } + split_pmd_thp(); + for (i = 8; i >= 0; i--) + if (i != 1) + split_thp_in_pagecache_to_order(i); + + for (i = 8; i >= 0; i--) + if (i != 1) + truncate_thp_in_pagecache_to_order(i); + return 0; }