From patchwork Wed Sep 2 18:06:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 11751483 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 879FF722 for ; Wed, 2 Sep 2020 18:06:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 445E6206EB for ; Wed, 2 Sep 2020 18:06:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=sent.com header.i=@sent.com header.b="JoTG8PpH"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="jFA2qhOu" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 445E6206EB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=sent.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AD0F9900018; Wed, 2 Sep 2020 14:06:34 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7A42C90001A; Wed, 2 Sep 2020 14:06:34 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3D4B9900012; Wed, 2 Sep 2020 14:06:34 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0029.hostedemail.com [216.40.44.29]) by kanga.kvack.org (Postfix) with ESMTP id DC678900018 for ; Wed, 2 Sep 2020 14:06:33 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 9D1FD2485 for ; Wed, 2 Sep 2020 18:06:33 +0000 (UTC) X-FDA: 77218901466.11.work61_0708107270a2 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id 69BF6180F8B80 for ; Wed, 2 Sep 2020 18:06:33 +0000 (UTC) X-Spam-Summary: 1,0,0,fa2fdf57119fcf62,d41d8cd98f00b204,zi.yan@sent.com,,RULES_HIT:2:41:355:379:541:800:960:973:988:989:1260:1261:1311:1314:1345:1359:1437:1515:1535:1605:1606:1730:1747:1777:1792:1801:2198:2199:2393:2559:2562:2693:3138:3139:3140:3141:3142:3867:3871:3872:4119:4250:4321:4605:5007:6117:6119:6120:6261:6630:6653:7576:7901:7903:8660:8957:10004:11026:11473:11657:11658:11914:12043:12291:12296:12438:12555:12679:12683:12895:12986:13148:13230:13894:14096:14110:21080:21451:21627:21939:21990:30003:30054:30064:30070:30079,0,RBL:66.111.4.25:@sent.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100;04yr9gwmqrhdopqyksfp8x4p3cn1wocxhxnnbi1da9gxerd1dnsnghe7ph5kidr.d96yth43wmgg1xikb9bohanrpr8jas6fik6ndezhhge7tcb6cfk8tbs3fbg8hdn.q-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: work61_0708107270a2 X-Filterd-Recvd-Size: 8949 Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Wed, 2 Sep 2020 18:06:32 +0000 (UTC) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 86FAE5C01A9; Wed, 2 Sep 2020 14:06:32 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Wed, 02 Sep 2020 14:06:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=from :to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; s=fm1; bh=clFcHF8RSMWnt gVxpYKMJOSLAdjV1uGHbUSJNEeaPm0=; b=JoTG8PpHCt6+MSuCBazIZdfaAO886 st903/gbM04E2nYNv8727gPF9HMBh/oqKFgx1YmRs84dnUSNs/omDvdmPpX4gwG/ Ew4Uh4RGJ0fETazGEuI+cM5vbwgnGSYkQfET8StsjilICk/J9PlJqpnRBjLtuuwE BCjfUSEVtbYMUTkrSm7bH16yPjuMvxGTwmXYtm5EAtYmepVM+02YKgqljUtZYV/Z yCKbbHDoLDZtWTWFygbC3/hMzadv+COQfC05AqSECv1sTi8svldpXn5SCRxW9m8i CPUxH9XwY78LZ1QuX76yuy7LK31PFU1lsLazJv3gxOMPBvFGmEAJNSahg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:reply-to:subject :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=clFcHF8RSMWntgVxpYKMJOSLAdjV1uGHbUSJNEeaPm0=; b=jFA2qhOu FrMd+jy72pdOO8/f5IEk4cde1d2AYkQ/c0RGD5QBuAQAEfkgFXD9EYoQR5jIDK0l gC9JNi8W/jE3TB1bZrrB8IP2rYmkvfRmzTiOmdNIlOXYWOkWUBqokj1mR/9tmd/e jarC8xIX9RSNkaryI9WsfbMxB0CwmtQ9iI7CN6p1QaSFpyG85Rp3AQp5+cyqOwBL 1fMvaiT+jdoEfsX86CLN7DCiqpE3zVy3tfd2ziuXMHB0TQymw3s3TCd/c2cn77N7 FvXfmOZrGyL3qy+TTU5XfX4OicNdx+GB7kc3CvO+/qFowjEZFJN24LBRfZws0VrX 43aIJ6WAOuqCKA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduiedrudefledguddvudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffufffkofgjfhhrggfgsedtkeertdertddtnecuhfhrohhmpegkihcu jggrnhcuoeiiihdrhigrnhesshgvnhhtrdgtohhmqeenucggtffrrghtthgvrhhnpeduhf ffveektdduhfdutdfgtdekkedvhfetuedufedtgffgvdevleehheevjefgtdenucfkphep uddvrdegiedruddtiedrudeigeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmh epmhgrihhlfhhrohhmpeiiihdrhigrnhesshgvnhhtrdgtohhm X-ME-Proxy: Received: from nvrsysarch6.NVidia.COM (unknown [12.46.106.164]) by mail.messagingengine.com (Postfix) with ESMTPA id A7D5C3060067; Wed, 2 Sep 2020 14:06:31 -0400 (EDT) From: Zi Yan To: linux-mm@kvack.org, Roman Gushchin Cc: Rik van Riel , "Kirill A . Shutemov" , Matthew Wilcox , Shakeel Butt , Yang Shi , David Nellans , linux-kernel@vger.kernel.org, Zi Yan Subject: [RFC PATCH 04/16] mm: thp: 1GB THP copy on write implementation. Date: Wed, 2 Sep 2020 14:06:16 -0400 Message-Id: <20200902180628.4052244-5-zi.yan@sent.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200902180628.4052244-1-zi.yan@sent.com> References: <20200902180628.4052244-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Rspamd-Queue-Id: 69BF6180F8B80 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan COW on 1GB THPs will fall back to 2MB THPs if 1GB THP is not available. Signed-off-by: Zi Yan --- arch/x86/include/asm/pgalloc.h | 9 ++++++ include/linux/huge_mm.h | 5 ++++ mm/huge_memory.c | 54 ++++++++++++++++++++++++++++++++++ mm/memory.c | 2 +- mm/swapfile.c | 4 ++- 5 files changed, 72 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h index fae13467d3e1..31221269c387 100644 --- a/arch/x86/include/asm/pgalloc.h +++ b/arch/x86/include/asm/pgalloc.h @@ -98,6 +98,15 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, #define pmd_pgtable(pmd) pmd_page(pmd) +static inline void pud_populate_with_pgtable(struct mm_struct *mm, pud_t *pud, + struct page *pte) +{ + unsigned long pfn = page_to_pfn(pte); + + paravirt_alloc_pmd(mm, pfn); + set_pud(pud, __pud(((pteval_t)pfn << PAGE_SHIFT) | _PAGE_TABLE)); +} + #if CONFIG_PGTABLE_LEVELS > 2 static inline pmd_t *pmd_alloc_one_page_with_ptes(struct mm_struct *mm, unsigned long addr) { diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 7528652400e4..0c20a8ea6911 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -19,6 +19,7 @@ extern int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD extern void huge_pud_set_accessed(struct vm_fault *vmf, pud_t orig_pud); extern int do_huge_pud_anonymous_page(struct vm_fault *vmf); +extern vm_fault_t do_huge_pud_wp_page(struct vm_fault *vmf, pud_t orig_pud); #else static inline void huge_pud_set_accessed(struct vm_fault *vmf, pud_t orig_pud) { @@ -27,6 +28,10 @@ extern int do_huge_pud_anonymous_page(struct vm_fault *vmf) { return VM_FAULT_FALLBACK; } +extern vm_fault_t do_huge_pud_wp_page(struct vm_fault *vmf, pud_t orig_pud) +{ + return VM_FAULT_FALLBACK; +} #endif extern vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index ec3847392208..6da9b02501b7 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1334,6 +1334,60 @@ void huge_pud_set_accessed(struct vm_fault *vmf, pud_t orig_pud) unlock: spin_unlock(vmf->ptl); } + +vm_fault_t do_huge_pud_wp_page(struct vm_fault *vmf, pud_t orig_pud) +{ + struct vm_area_struct *vma = vmf->vma; + struct page *page = NULL; + unsigned long haddr = vmf->address & HPAGE_PUD_MASK; + + vmf->ptl = pud_lockptr(vma->vm_mm, vmf->pud); + VM_BUG_ON_VMA(!vma->anon_vma, vma); + + if (is_huge_zero_pud(orig_pud)) + goto fallback; + + spin_lock(vmf->ptl); + + if (unlikely(!pud_same(*vmf->pud, orig_pud))) { + spin_unlock(vmf->ptl); + return 0; + } + + page = pud_page(orig_pud); + VM_BUG_ON_PAGE(!PageCompound(page) || !PageHead(page), page); + + /* Lock page for reuse_swap_page() */ + if (!trylock_page(page)) { + get_page(page); + spin_unlock(vmf->ptl); + lock_page(page); + spin_lock(vmf->ptl); + if (unlikely(!pud_same(*vmf->pud, orig_pud))) { + unlock_page(page); + put_page(page); + return 0; + } + put_page(page); + } + if (reuse_swap_page(page, NULL)) { + pud_t entry; + + entry = pud_mkyoung(orig_pud); + entry = maybe_pud_mkwrite(pud_mkdirty(entry), vma); + if (pudp_set_access_flags(vma, haddr, vmf->pud, entry, 1)) + update_mmu_cache_pud(vma, vmf->address, vmf->pud); + unlock_page(page); + spin_unlock(vmf->ptl); + return VM_FAULT_WRITE; + } + unlock_page(page); + spin_unlock(vmf->ptl); +fallback: + __split_huge_pud(vma, vmf->pud, vmf->address); + return VM_FAULT_FALLBACK; +} + #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ void huge_pmd_set_accessed(struct vm_fault *vmf, pmd_t orig_pmd) diff --git a/mm/memory.c b/mm/memory.c index 6f86294438fd..b88587256bc1 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4165,7 +4165,7 @@ static vm_fault_t wp_huge_pud(struct vm_fault *vmf, pud_t orig_pud) #ifdef CONFIG_TRANSPARENT_HUGEPAGE /* No support for anonymous transparent PUD pages yet */ if (vma_is_anonymous(vmf->vma)) - return VM_FAULT_FALLBACK; + return do_huge_pud_wp_page(vmf, orig_pud); if (vmf->vma->vm_ops->huge_fault) return vmf->vma->vm_ops->huge_fault(vmf, PE_SIZE_PUD); #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ diff --git a/mm/swapfile.c b/mm/swapfile.c index 20012c0c0252..e3f771c2ad83 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1635,7 +1635,9 @@ static int page_trans_huge_map_swapcount(struct page *page, int *total_mapcount, /* hugetlbfs shouldn't call it */ VM_BUG_ON_PAGE(PageHuge(page), page); - if (!IS_ENABLED(CONFIG_THP_SWAP) || likely(!PageTransCompound(page))) { + if (!IS_ENABLED(CONFIG_THP_SWAP) || + unlikely(compound_order(compound_head(page)) == HPAGE_PUD_ORDER) || + likely(!PageTransCompound(page))) { mapcount = page_trans_huge_mapcount(page, total_mapcount); if (PageSwapCache(page)) swapcount = page_swapcount(page);