From patchwork Mon Sep 28 17:54:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 11804491 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F352A6CA for ; Mon, 28 Sep 2020 17:56:30 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A9CF322204 for ; Mon, 28 Sep 2020 17:56:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=sent.com header.i=@sent.com header.b="XEu86+j+"; dkim=temperror (0-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="lsRyCnmS" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A9CF322204 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=sent.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5CA2290001C; Mon, 28 Sep 2020 13:55:31 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 52C7790001A; Mon, 28 Sep 2020 13:55:31 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3A02890001C; Mon, 28 Sep 2020 13:55:31 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0229.hostedemail.com [216.40.44.229]) by kanga.kvack.org (Postfix) with ESMTP id 0A6DC90001D for ; Mon, 28 Sep 2020 13:55:31 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id C46BE365A for ; Mon, 28 Sep 2020 17:55:30 +0000 (UTC) X-FDA: 77313222420.08.mouth47_1d090d327183 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin08.hostedemail.com (Postfix) with ESMTP id A4EB81819E76F for ; Mon, 28 Sep 2020 17:55:30 +0000 (UTC) X-Spam-Summary: 1,0,0,f3be5bf35097e05e,d41d8cd98f00b204,zi.yan@sent.com,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1261:1311:1314:1345:1359:1437:1515:1535:1544:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:2693:3138:3139:3140:3141:3142:3355:3664:3865:3866:3867:3868:3871:3872:4119:4250:4321:4605:5007:6117:6119:6120:6261:6653:6742:7576:7903:8784:10004:11026:11473:11658:11914:12043:12291:12296:12438:12555:12679:12683:12895:13894:14039:14040:14181:14721:21080:21451:21627:21795:21966:21990:30051:30054:30064,0,RBL:64.147.123.17:@sent.com:.lbl8.mailshell.net-62.18.0.100 64.100.201.100;04y8e8ffdyhm4ib944dk346hm6fhyycmieagne47ihaf7ttn66ddxnnwj6hts6e.tjs1ju9onobt55btg8o114ezmpfwz57pbhnccg7eyog9xsiy7174jnnfkf3irtb.4-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: mouth47_1d090d327183 X-Filterd-Recvd-Size: 8819 Received: from wnew3-smtp.messagingengine.com (wnew3-smtp.messagingengine.com [64.147.123.17]) by imf17.hostedemail.com (Postfix) with ESMTP for ; Mon, 28 Sep 2020 17:55:29 +0000 (UTC) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailnew.west.internal (Postfix) with ESMTP id 1C73EEB0; Mon, 28 Sep 2020 13:55:28 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Mon, 28 Sep 2020 13:55:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=from :to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; s=fm1; bh=fKNGPn4b5sGwL Rh+ItgDgX7lMBGsorph0sms9B238k0=; b=XEu86+j+e7gcMHEv92P5s4xJp20rp zMFVKS65FnuMSKgAA77Gtj5Ut+FUrecE44+tUeEk6LFeW9NtYgTpYMm120lIOWGx HKht3fcFgqTYoq5eaVGgHK5PK8AhelDj/mEBUuq3cgVpjZ/OUSU++DpMZI8K84LG YIjEjqIXnkDi3RUfoY0XSLSIu8G2ETQ2XsEyId1umd5KLN01DJxt3rkbIv7rr1tm +KOna1sKXCp+cZEAZPGdRN+hzk+oS+y6HttZbrD3q2+llWNhzOpswf1GxRE0V73e SjuI2T4VXRU3LOUm/1Wvi0b8EELHFNSaY2bSuHMRsFt7//jDROOkxq96Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:reply-to:subject :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=fKNGPn4b5sGwLRh+ItgDgX7lMBGsorph0sms9B238k0=; b=lsRyCnmS nAcktrPPtkfFAhB7BBGBbHqNo/gPpg3cChaqKww8YC0T6Y+zAygHI2S+g8Z4bVjX QRR7zu1tFpCbO4aHf6IPZAAd5fF2hhY/x+P0wHq3Kh/6KQDOZCpvOGuMFZ98KziJ Ex+CzzF4D+kBV6HuNyknRSav09t+z9bSTYrRziX7HxBtd9kOVQ7dxnLFhVXHSAwl HuN9o7AsduqJLdKQCbPw6wlAdkrTCqJki99q1sUS5+CN8FYdGRBspor/Y9zn7UZc wP7XsClW4DokSWczCjfxpj78fTugYYgGOlTD2TEFkL+BZ427ytqKodqCX7ZSEiE5 +9DfsXEmWWATMQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedrvdeigdeliecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhhrggfgsedtkeertdertddtnecuhfhrohhmpegkihcujggr nhcuoeiiihdrhigrnhesshgvnhhtrdgtohhmqeenucggtffrrghtthgvrhhnpeduhfffve ektdduhfdutdfgtdekkedvhfetuedufedtgffgvdevleehheevjefgtdenucfkphepuddv rdegiedruddtiedrudeigeenucevlhhushhtvghrufhiiigvpeduvdenucfrrghrrghmpe hmrghilhhfrhhomhepiihirdihrghnsehsvghnthdrtghomh X-ME-Proxy: Received: from nvrsysarch6.NVidia.COM (unknown [12.46.106.164]) by mail.messagingengine.com (Postfix) with ESMTPA id 4C6BB3064686; Mon, 28 Sep 2020 13:55:27 -0400 (EDT) From: Zi Yan To: linux-mm@kvack.org Cc: "Kirill A . Shutemov" , Roman Gushchin , Rik van Riel , Matthew Wilcox , Shakeel Butt , Yang Shi , Jason Gunthorpe , Mike Kravetz , Michal Hocko , David Hildenbrand , William Kucharski , Andrea Arcangeli , John Hubbard , David Nellans , linux-kernel@vger.kernel.org, Zi Yan Subject: [RFC PATCH v2 26/30] mm: thp: add a global knob to enable/disable PUD THPs. Date: Mon, 28 Sep 2020 13:54:24 -0400 Message-Id: <20200928175428.4110504-27-zi.yan@sent.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200928175428.4110504-1-zi.yan@sent.com> References: <20200928175428.4110504-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan Like the existing global PMD THP knob, it allows user to enable/disable PUD THPs. PUD THP is disabled by default unless user knows the performance tradeoff of using it, like longer first time page fault due to larger page zeroing and longer page allocation time when memory is fragmented. Experienced user can enable it and take advantage of its benefit of suffering fewer page faults and TLB misses. * always means PUD THPs will be allocated on all VMAs if possible. * madvise means PUD THPs will be allocated if vm_flags has VM_HUGEPAGE_PUD set via madvise syscall using MADV_HUGEPAGE | MADV_HUGEPAGE_PUD. * none means PUD THPs will not be allocated on any VMA. Signed-off-by: Zi Yan --- include/linux/huge_mm.h | 14 ++++++++++++++ mm/huge_memory.c | 38 ++++++++++++++++++++++++++++++++++++++ mm/memory.c | 2 +- 3 files changed, 53 insertions(+), 1 deletion(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index c7bc40c4a5e2..0d0f9cf25aeb 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -119,6 +119,8 @@ enum transparent_hugepage_flag { #ifdef CONFIG_DEBUG_VM TRANSPARENT_HUGEPAGE_DEBUG_COW_FLAG, #endif + TRANSPARENT_PUD_HUGEPAGE_FLAG, + TRANSPARENT_PUD_HUGEPAGE_REQ_MADV_FLAG, }; struct kobject; @@ -184,6 +186,18 @@ static inline bool __transparent_hugepage_enabled(struct vm_area_struct *vma) } bool transparent_hugepage_enabled(struct vm_area_struct *vma); +static inline bool transparent_pud_hugepage_enabled(struct vm_area_struct *vma) +{ + if (transparent_hugepage_enabled(vma)) { + if (transparent_hugepage_flags & (1 << TRANSPARENT_PUD_HUGEPAGE_FLAG)) + return true; + if (transparent_hugepage_flags & + (1 << TRANSPARENT_PUD_HUGEPAGE_REQ_MADV_FLAG)) + return !!(vma->vm_flags & VM_HUGEPAGE_PUD); + } + + return false; +} #define HPAGE_CACHE_INDEX_MASK (HPAGE_PMD_NR - 1) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 61ae7a0ded84..1965753b31a2 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -199,6 +199,43 @@ static ssize_t enabled_store(struct kobject *kobj, static struct kobj_attribute enabled_attr = __ATTR(enabled, 0644, enabled_show, enabled_store); +static ssize_t enabled_pud_thp_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + if (test_bit(TRANSPARENT_PUD_HUGEPAGE_FLAG, &transparent_hugepage_flags)) + return sprintf(buf, "[always] madvise never\n"); + else if (test_bit(TRANSPARENT_PUD_HUGEPAGE_REQ_MADV_FLAG, &transparent_hugepage_flags)) + return sprintf(buf, "always [madvise] never\n"); + else + return sprintf(buf, "always madvise [never]\n"); +} + +static ssize_t enabled_pud_thp_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + ssize_t ret = count; + + if (!memcmp("always", buf, + min(sizeof("always")-1, count))) { + clear_bit(TRANSPARENT_PUD_HUGEPAGE_REQ_MADV_FLAG, &transparent_hugepage_flags); + set_bit(TRANSPARENT_PUD_HUGEPAGE_FLAG, &transparent_hugepage_flags); + } else if (!memcmp("madvise", buf, + min(sizeof("madvise")-1, count))) { + clear_bit(TRANSPARENT_PUD_HUGEPAGE_FLAG, &transparent_hugepage_flags); + set_bit(TRANSPARENT_PUD_HUGEPAGE_REQ_MADV_FLAG, &transparent_hugepage_flags); + } else if (!memcmp("never", buf, + min(sizeof("never")-1, count))) { + clear_bit(TRANSPARENT_PUD_HUGEPAGE_FLAG, &transparent_hugepage_flags); + clear_bit(TRANSPARENT_PUD_HUGEPAGE_REQ_MADV_FLAG, &transparent_hugepage_flags); + } else + ret = -EINVAL; + + return ret; +} +static struct kobj_attribute enabled_pud_thp_attr = + __ATTR(enabled_pud_thp, 0644, enabled_pud_thp_show, enabled_pud_thp_store); + ssize_t single_hugepage_flag_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf, enum transparent_hugepage_flag flag) @@ -305,6 +342,7 @@ static struct kobj_attribute hpage_pmd_size_attr = static struct attribute *hugepage_attr[] = { &enabled_attr.attr, + &enabled_pud_thp_attr.attr, &defrag_attr.attr, &use_zero_page_attr.attr, &hpage_pmd_size_attr.attr, diff --git a/mm/memory.c b/mm/memory.c index ab80d13807aa..9f7b509a3aa7 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4282,7 +4282,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, if (!vmf.pud) return VM_FAULT_OOM; retry_pud: - if (pud_none(*vmf.pud) && __transparent_hugepage_enabled(vma)) { + if (pud_none(*vmf.pud) && transparent_pud_hugepage_enabled(vma)) { ret = create_huge_pud(&vmf); if (!(ret & VM_FAULT_FALLBACK)) return ret;