From patchwork Tue May 18 20:07:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12265583 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1DBBC43462 for ; Tue, 18 May 2021 20:08:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8868C611AC for ; Tue, 18 May 2021 20:08:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8868C611AC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2A80E8E0047; Tue, 18 May 2021 16:08:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 27EE38E002F; Tue, 18 May 2021 16:08:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F7E48E0047; Tue, 18 May 2021 16:08:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0094.hostedemail.com [216.40.44.94]) by kanga.kvack.org (Postfix) with ESMTP id CB2F58E002F for ; Tue, 18 May 2021 16:08:20 -0400 (EDT) Received: from smtpin32.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 5851D181AF5F7 for ; Tue, 18 May 2021 20:08:20 +0000 (UTC) X-FDA: 78155438760.32.0671068 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf06.hostedemail.com (Postfix) with ESMTP id 00A59C001C46 for ; Tue, 18 May 2021 20:08:18 +0000 (UTC) Received: by mail-pl1-f171.google.com with SMTP id n8so508169plf.7 for ; Tue, 18 May 2021 13:08:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MNGAefi2XGquv7ADvXSLLg1AB0DvLJWZ+nDSpBzysJE=; b=po/9chrKD5sFYOH0pueuDwSaCPgabrhMyc2CjLFVET34fJ/hAWd+g1uwVdnYlIgkM+ 4Ky3Ze4gxG+3rMHG++vd50nQbD+GcH4CDBrS/fB25iJHtU5C/ltZNj5/KFryTVVVNMzU qG9Dt/5SuplVU08G+DtdnC3924xIFn9edObevi2Z+XpcL4HytFsRdqZRiWhjkYBxKvgf CDqFs4wFYo1uiFVTX0FlMzkBw9zeV4McZ/CGXsnz1M8OgfSuX3Soq8WWn3gY/gwtIA1r t4z6U0AsV5+gSgz7Ku0p4VbMplFDqIX3xDmWGvAOCGVfAMRc5hYokfVpRBnhBGKSWyZv 9xGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MNGAefi2XGquv7ADvXSLLg1AB0DvLJWZ+nDSpBzysJE=; b=J0S9dP0otx9FG5pksZBUUDn1hS/btXWf6TuvdokdgLG363/Sf3GJnFms8beiVSBbVs 97okndRsMxDf0g7HyAHBOxTpSpiABCrYWIvysWAOLS8z1gszdhaUA3Ou1buoTFd4T0Nk lKL2D4NGJkuOcKfJrfaaVhV2PdObBenoAw/+oEJu/rGzuX4xTbxYiZXF19SfYvRgfU8v 0FnklMquV7GPBAZkDhXUMZzI6VJ0slCJbJhtBjWuVvTsZ5JpcJp9GohU5uVKqBnntMzO DScLlo8PSgL/mYIz8GkeFfAEO7wjKXoMa8E0pVPCbcyQVpzIFXsuIJNRIsQMyNYzT8gv TzTw== X-Gm-Message-State: AOAM532L0ClCUsspB6hTWP1B6Y9miw7f+ZsX1DgB141TamDy9TvABHz3 t5xzoooBtYVwoOHfFBZkj5Q= X-Google-Smtp-Source: ABdhPJzGW7Ar92/6nEIX2ysdxrWuQCZB2TGBhuS0zyU1hYinjuGSb+/BC0JRw+C4p2hD4TR/SSpEcw== X-Received: by 2002:a17:90a:e7c6:: with SMTP id kb6mr7137260pjb.81.1621368498964; Tue, 18 May 2021 13:08:18 -0700 (PDT) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id r11sm13456600pgl.34.2021.05.18.13.08.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 May 2021 13:08:18 -0700 (PDT) From: Yang Shi To: mgorman@suse.de, kirill.shutemov@linux.intel.com, ziy@nvidia.com, ying.huang@intel.com, mhocko@suse.com, hughd@google.com, gerald.schaefer@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, borntraeger@de.ibm.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v3 PATCH 1/7] mm: memory: add orig_pmd to struct vm_fault Date: Tue, 18 May 2021 13:07:55 -0700 Message-Id: <20210518200801.7413-2-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210518200801.7413-1-shy828301@gmail.com> References: <20210518200801.7413-1-shy828301@gmail.com> MIME-Version: 1.0 Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b="po/9chrK"; spf=pass (imf06.hostedemail.com: domain of shy828301@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 00A59C001C46 X-Stat-Signature: 5yfhm17rogsefkm3uhs3uwuxok5ode5h X-HE-Tag: 1621368498-24587 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add orig_pmd to struct vm_fault so the "orig_pmd" parameter used by huge page fault could be removed, just like its PTE counterpart does. Signed-off-by: Yang Shi Acked-by: Mel Gorman --- include/linux/huge_mm.h | 9 ++++----- include/linux/mm.h | 7 ++++++- mm/huge_memory.c | 9 ++++++--- mm/memory.c | 26 +++++++++++++------------- 4 files changed, 29 insertions(+), 22 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 60dad7c88d72..77d6d454df90 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -11,7 +11,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf); int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr, struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma); -void huge_pmd_set_accessed(struct vm_fault *vmf, pmd_t orig_pmd); +void huge_pmd_set_accessed(struct vm_fault *vmf); int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, pud_t *dst_pud, pud_t *src_pud, unsigned long addr, struct vm_area_struct *vma); @@ -24,7 +24,7 @@ static inline void huge_pud_set_accessed(struct vm_fault *vmf, pud_t orig_pud) } #endif -vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd); +vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf); struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, unsigned int flags); @@ -283,7 +283,7 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr, pud_t *pud, int flags, struct dev_pagemap **pgmap); -vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t orig_pmd); +vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf); extern struct page *huge_zero_page; @@ -429,8 +429,7 @@ static inline spinlock_t *pud_trans_huge_lock(pud_t *pud, return NULL; } -static inline vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf, - pmd_t orig_pmd) +static inline vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) { return 0; } diff --git a/include/linux/mm.h b/include/linux/mm.h index 6ce7bac74530..b3a4b405364b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -550,7 +550,12 @@ struct vm_fault { pud_t *pud; /* Pointer to pud entry matching * the 'address' */ - pte_t orig_pte; /* Value of PTE at the time of fault */ + union { + pte_t orig_pte; /* Value of PTE at the time of fault */ + pmd_t orig_pmd; /* Value of PMD at the time of fault, + * used by PMD fault only. + */ + }; struct page *cow_page; /* Page handler may use for COW fault */ struct page *page; /* ->fault handlers should return a diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 19195fca1aee..692788988530 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1245,11 +1245,12 @@ void huge_pud_set_accessed(struct vm_fault *vmf, pud_t orig_pud) } #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ -void huge_pmd_set_accessed(struct vm_fault *vmf, pmd_t orig_pmd) +void huge_pmd_set_accessed(struct vm_fault *vmf) { pmd_t entry; unsigned long haddr; bool write = vmf->flags & FAULT_FLAG_WRITE; + pmd_t orig_pmd = vmf->orig_pmd; vmf->ptl = pmd_lock(vmf->vma->vm_mm, vmf->pmd); if (unlikely(!pmd_same(*vmf->pmd, orig_pmd))) @@ -1266,11 +1267,12 @@ void huge_pmd_set_accessed(struct vm_fault *vmf, pmd_t orig_pmd) spin_unlock(vmf->ptl); } -vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) +vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; struct page *page; unsigned long haddr = vmf->address & HPAGE_PMD_MASK; + pmd_t orig_pmd = vmf->orig_pmd; vmf->ptl = pmd_lockptr(vma->vm_mm, vmf->pmd); VM_BUG_ON_VMA(!vma->anon_vma, vma); @@ -1406,9 +1408,10 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, } /* NUMA hinting page fault entry point for trans huge pmds */ -vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t pmd) +vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; + pmd_t pmd = vmf->orig_pmd; struct anon_vma *anon_vma = NULL; struct page *page; unsigned long haddr = vmf->address & HPAGE_PMD_MASK; diff --git a/mm/memory.c b/mm/memory.c index 97ca3e51cf85..f9df3ab12813 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4252,12 +4252,12 @@ static inline vm_fault_t create_huge_pmd(struct vm_fault *vmf) } /* `inline' is required to avoid gcc 4.1.2 build error */ -static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf, pmd_t orig_pmd) +static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) { if (vma_is_anonymous(vmf->vma)) { - if (userfaultfd_huge_pmd_wp(vmf->vma, orig_pmd)) + if (userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) return handle_userfault(vmf, VM_UFFD_WP); - return do_huge_pmd_wp_page(vmf, orig_pmd); + return do_huge_pmd_wp_page(vmf); } if (vmf->vma->vm_ops->huge_fault) { vm_fault_t ret = vmf->vma->vm_ops->huge_fault(vmf, PE_SIZE_PMD); @@ -4484,26 +4484,26 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, if (!(ret & VM_FAULT_FALLBACK)) return ret; } else { - pmd_t orig_pmd = *vmf.pmd; + vmf.orig_pmd = *vmf.pmd; barrier(); - if (unlikely(is_swap_pmd(orig_pmd))) { + if (unlikely(is_swap_pmd(vmf.orig_pmd))) { VM_BUG_ON(thp_migration_supported() && - !is_pmd_migration_entry(orig_pmd)); - if (is_pmd_migration_entry(orig_pmd)) + !is_pmd_migration_entry(vmf.orig_pmd)); + if (is_pmd_migration_entry(vmf.orig_pmd)) pmd_migration_entry_wait(mm, vmf.pmd); return 0; } - if (pmd_trans_huge(orig_pmd) || pmd_devmap(orig_pmd)) { - if (pmd_protnone(orig_pmd) && vma_is_accessible(vma)) - return do_huge_pmd_numa_page(&vmf, orig_pmd); + if (pmd_trans_huge(vmf.orig_pmd) || pmd_devmap(vmf.orig_pmd)) { + if (pmd_protnone(vmf.orig_pmd) && vma_is_accessible(vma)) + return do_huge_pmd_numa_page(&vmf); - if (dirty && !pmd_write(orig_pmd)) { - ret = wp_huge_pmd(&vmf, orig_pmd); + if (dirty && !pmd_write(vmf.orig_pmd)) { + ret = wp_huge_pmd(&vmf); if (!(ret & VM_FAULT_FALLBACK)) return ret; } else { - huge_pmd_set_accessed(&vmf, orig_pmd); + huge_pmd_set_accessed(&vmf); return 0; } } From patchwork Tue May 18 20:07:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12265585 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A156C433ED for ; Tue, 18 May 2021 20:08:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EDCDB61074 for ; Tue, 18 May 2021 20:08:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EDCDB61074 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6C5928E0048; Tue, 18 May 2021 16:08:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 69C6E8E002F; Tue, 18 May 2021 16:08:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 517A48E0048; Tue, 18 May 2021 16:08:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0066.hostedemail.com [216.40.44.66]) by kanga.kvack.org (Postfix) with ESMTP id 171668E002F for ; Tue, 18 May 2021 16:08:23 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id A334E8249980 for ; Tue, 18 May 2021 20:08:22 +0000 (UTC) X-FDA: 78155438844.02.A2511E1 Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by imf29.hostedemail.com (Postfix) with ESMTP id 9985B798 for ; Tue, 18 May 2021 20:08:20 +0000 (UTC) Received: by mail-pf1-f177.google.com with SMTP id d78so7378569pfd.10 for ; Tue, 18 May 2021 13:08:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/SAb4GmBYcE1RUtznSnoMs9bVmPh+Fmyxo0FfvNKkrs=; b=qTXntNfCiZwDGQEMfk2x4vS/wIHq7eq4cub1djq6BDGbwCBDouuYnFDUTkEFw3sFa/ zXWj8Nzwt0d0vegm5TRX283djZ5lKzUQjPLS0U63PhhBUsHvuJ8Z79joQnd2+wXFpAns JmojxPXqAm+C6/6DgPjRJi0A2Sth57/vRHGakaHuw6l2qEBDclnVpk8tQXA3eQeMSxHy Y/tpdSguaArIG6gQtegc9JdHB3DAWLvaY5cKnmMxkJtKy/Fg9Y8Zyj/OjuauGSTkT48M +AaZmb86n3/I2wGP7wDfP7ikDmqCiHxKmGvCRdRNA1fxGVW0rDjnKBgFnJ/95hefATz4 yIEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/SAb4GmBYcE1RUtznSnoMs9bVmPh+Fmyxo0FfvNKkrs=; b=VC5acx72SN4RNgFY82jKuKk6TKcVB5tIliMa2+CXwVbcFFIJ5wai+GRgPvTgGObKnA lz5eC6Y4AxIcK41FB0KXmu5gdLvBHC107++ppu7/a1OxKuT4q3NbsKQK51NA7F+2Gnqa DBJuENxA3salz1PvMECuyJuBgaOKBgZq32w98E+trx3HOHxFVRDAbz+QRvkf+Ma2FaQZ 6AeBxWYrhTiq1624Jn8wq1AkOqC+M+3k26ARFYVh7xJ/iLuV3ZVENVQR4jwpDU5Ocjcf h8BbuF9BRm/Zo9/pDDs/9YSQ/Trr8IxMrV5/Tv6CQSiMWwmw0RKq8XXdbFZO37X4T1W5 9IkA== X-Gm-Message-State: AOAM533l2uY8MZfrw6SgYM0kvRb181D+usEbpRkeUpGkBZu2AUHR5z8c WgozTa514QSce4Jc/hvp4Gw= X-Google-Smtp-Source: ABdhPJzE79+gi8tH257NdQjO8LFM/9zlnlCFxksOY0avdvvvpts3EqA18TbkrxA3iV9ASa/7lH41oA== X-Received: by 2002:a65:50c5:: with SMTP id s5mr6792668pgp.138.1621368501491; Tue, 18 May 2021 13:08:21 -0700 (PDT) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id r11sm13456600pgl.34.2021.05.18.13.08.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 May 2021 13:08:20 -0700 (PDT) From: Yang Shi To: mgorman@suse.de, kirill.shutemov@linux.intel.com, ziy@nvidia.com, ying.huang@intel.com, mhocko@suse.com, hughd@google.com, gerald.schaefer@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, borntraeger@de.ibm.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v3 PATCH 2/7] mm: memory: make numa_migrate_prep() non-static Date: Tue, 18 May 2021 13:07:56 -0700 Message-Id: <20210518200801.7413-3-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210518200801.7413-1-shy828301@gmail.com> References: <20210518200801.7413-1-shy828301@gmail.com> MIME-Version: 1.0 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=qTXntNfC; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of shy828301@gmail.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=shy828301@gmail.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 9985B798 X-Stat-Signature: q9drwjgb5ehxsi14xeqz9ujib9k8bc15 X-HE-Tag: 1621368500-551447 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The numa_migrate_prep() will be used by huge NUMA fault as well in the following patch, make it non-static. Acked-by: Mel Gorman Signed-off-by: Yang Shi --- mm/internal.h | 3 +++ mm/memory.c | 5 ++--- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 8a176d1d0176..28fc471b4669 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -653,4 +653,7 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end, void vunmap_range_noflush(unsigned long start, unsigned long end); +int numa_migrate_prep(struct page *page, struct vm_area_struct *vma, + unsigned long addr, int page_nid, int *flags); + #endif /* __MM_INTERNAL_H */ diff --git a/mm/memory.c b/mm/memory.c index f9df3ab12813..a2f2c2a8d28e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4129,9 +4129,8 @@ static vm_fault_t do_fault(struct vm_fault *vmf) return ret; } -static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma, - unsigned long addr, int page_nid, - int *flags) +int numa_migrate_prep(struct page *page, struct vm_area_struct *vma, + unsigned long addr, int page_nid, int *flags) { get_page(page); From patchwork Tue May 18 20:07:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12265587 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9FC80C433ED for ; Tue, 18 May 2021 20:08:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3706461074 for ; Tue, 18 May 2021 20:08:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3706461074 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CB9038E0049; Tue, 18 May 2021 16:08:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C90818E002F; Tue, 18 May 2021 16:08:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ABC9C8E0049; Tue, 18 May 2021 16:08:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0139.hostedemail.com [216.40.44.139]) by kanga.kvack.org (Postfix) with ESMTP id 67D558E002F for ; Tue, 18 May 2021 16:08:25 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 09E4F181AF5F7 for ; Tue, 18 May 2021 20:08:25 +0000 (UTC) X-FDA: 78155438970.09.15610E9 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by imf17.hostedemail.com (Postfix) with ESMTP id 553854080F44 for ; Tue, 18 May 2021 20:08:23 +0000 (UTC) Received: by mail-pf1-f169.google.com with SMTP id 10so8261033pfl.1 for ; Tue, 18 May 2021 13:08:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=C45PnyOdAKKdP0uikZzFqWjcMiIEbofP6QQMFTRkh6w=; b=CL5nPnvoI5mOaaaHiLyHZEkQf8fXrv1vtkCvFBFuJJWCGuizVhnpjdxPXTgG8jc5aj 7WBfc1oKuABFJRwsoT2TuDRlLiozNYNenpcDGSAc5C/u782mJiRI0LvXiAQ7fA3zlVl8 HqqhnRzd9a0d7rSUmWEw0U8nBQSMDxjxMGcGsgihitU7kbItmgZ6KLEdiSzoLhrG4v/B UV3Q4wY7WOfK4EWNLngwfFClgW3bqz0hf7f4iaWQLb9Xl42tqHJAODRB/kKw9fCvBWig cBVB/CL7bgWKQ0Zlq7R3rxlXCmaet0OPGhyFSw6FGvAhACPA4D0R67PLdngekij7Awmi yF3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=C45PnyOdAKKdP0uikZzFqWjcMiIEbofP6QQMFTRkh6w=; b=nS0WEiYlRsiBKgBgN87Jigwlo8rqUNIVie5vkZXpmK3WLNMvjCJruDMUp5ZnympWP5 6J7kvvtLcHhGQSQwuOu7DVtREvf4NpwK8EmscV8j+2XHimfFNRvjPGEOoTGlfbnk7HxZ FjM3gpPDTc9cZsElZXqwRx7DiuZzIfM4dH2AFypghaOELabvMEFo2MXXSrNrb5/fSGVe fEuI2ypUk0QVWtcOn1JWA2DvyOVZeQGf2FDgCHBEkv39DTzwpPI9oL/3htZz0+0u+XN4 KGaopr6Hi5HCzvEwV1gf8XAOEvWb1AByz1EVwL1+3u5Yk2ORUCoFPXbvROUvrUwzGy6t 3AXw== X-Gm-Message-State: AOAM532fogdCf1y/vsZL38N8LG8fpUf0EgSVEygREqhQr5MbvRQ/ZGRs WUh+ULVCcc8x+sWxO7u34kM= X-Google-Smtp-Source: ABdhPJzXfSSTZQ+PAcYbUE/rxP6sZZ3uYNwuAXI65lO98OQt0sI9gQGMWwN+U6wdxl7ad9jFCUvCPQ== X-Received: by 2002:a63:416:: with SMTP id 22mr6870972pge.363.1621368503764; Tue, 18 May 2021 13:08:23 -0700 (PDT) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id r11sm13456600pgl.34.2021.05.18.13.08.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 May 2021 13:08:22 -0700 (PDT) From: Yang Shi To: mgorman@suse.de, kirill.shutemov@linux.intel.com, ziy@nvidia.com, ying.huang@intel.com, mhocko@suse.com, hughd@google.com, gerald.schaefer@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, borntraeger@de.ibm.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v3 PATCH 3/7] mm: thp: refactor NUMA fault handling Date: Tue, 18 May 2021 13:07:57 -0700 Message-Id: <20210518200801.7413-4-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210518200801.7413-1-shy828301@gmail.com> References: <20210518200801.7413-1-shy828301@gmail.com> MIME-Version: 1.0 Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=CL5nPnvo; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of shy828301@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=shy828301@gmail.com X-Stat-Signature: 14mcwir9gi9wimpm18n93ejdg9zs5kpw X-Rspamd-Queue-Id: 553854080F44 X-Rspamd-Server: rspam02 X-HE-Tag: 1621368503-849576 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When the THP NUMA fault support was added THP migration was not supported yet. So the ad hoc THP migration was implemented in NUMA fault handling. Since v4.14 THP migration has been supported so it doesn't make too much sense to still keep another THP migration implementation rather than using the generic migration code. This patch reworked the NUMA fault handling to use generic migration implementation to migrate misplaced page. There is no functional change. After the refactor the flow of NUMA fault handling looks just like its PTE counterpart: Acquire ptl Prepare for migration (elevate page refcount) Release ptl Isolate page from lru and elevate page refcount Migrate the misplaced THP If migration is failed just restore the old normal PMD. In the old code anon_vma lock was needed to serialize THP migration against THP split, but since then the THP code has been reworked a lot, it seems anon_vma lock is not required anymore to avoid the race. The page refcount elevation when holding ptl should prevent from THP split. Use migrate_misplaced_page() for both base page and THP NUMA hinting fault and remove all the dead and duplicate code. Acked-by: Mel Gorman Signed-off-by: Yang Shi --- include/linux/migrate.h | 23 ------ mm/huge_memory.c | 147 +++++++++++---------------------- mm/internal.h | 18 ---- mm/migrate.c | 177 ++++++++-------------------------------- 4 files changed, 79 insertions(+), 286 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 4bb4e519e3f5..163d6f2b03d1 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -95,14 +95,9 @@ static inline void __ClearPageMovable(struct page *page) #endif #ifdef CONFIG_NUMA_BALANCING -extern bool pmd_trans_migrating(pmd_t pmd); extern int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma, int node); #else -static inline bool pmd_trans_migrating(pmd_t pmd) -{ - return false; -} static inline int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma, int node) { @@ -110,24 +105,6 @@ static inline int migrate_misplaced_page(struct page *page, } #endif /* CONFIG_NUMA_BALANCING */ -#if defined(CONFIG_NUMA_BALANCING) && defined(CONFIG_TRANSPARENT_HUGEPAGE) -extern int migrate_misplaced_transhuge_page(struct mm_struct *mm, - struct vm_area_struct *vma, - pmd_t *pmd, pmd_t entry, - unsigned long address, - struct page *page, int node); -#else -static inline int migrate_misplaced_transhuge_page(struct mm_struct *mm, - struct vm_area_struct *vma, - pmd_t *pmd, pmd_t entry, - unsigned long address, - struct page *page, int node) -{ - return -EAGAIN; -} -#endif /* CONFIG_NUMA_BALANCING && CONFIG_TRANSPARENT_HUGEPAGE*/ - - #ifdef CONFIG_MIGRATION /* diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 692788988530..b8526b9b041a 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1411,94 +1411,22 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; - pmd_t pmd = vmf->orig_pmd; - struct anon_vma *anon_vma = NULL; + pmd_t oldpmd = vmf->orig_pmd; + pmd_t pmd; struct page *page; unsigned long haddr = vmf->address & HPAGE_PMD_MASK; - int page_nid = NUMA_NO_NODE, this_nid = numa_node_id(); + int page_nid = NUMA_NO_NODE; int target_nid, last_cpupid = -1; - bool page_locked; bool migrated = false; - bool was_writable; + bool was_writable = pmd_savedwrite(oldpmd); int flags = 0; vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); - if (unlikely(!pmd_same(pmd, *vmf->pmd))) - goto out_unlock; - - /* - * If there are potential migrations, wait for completion and retry - * without disrupting NUMA hinting information. Do not relock and - * check_same as the page may no longer be mapped. - */ - if (unlikely(pmd_trans_migrating(*vmf->pmd))) { - page = pmd_page(*vmf->pmd); - if (!get_page_unless_zero(page)) - goto out_unlock; + if (unlikely(!pmd_same(oldpmd, *vmf->pmd))) { spin_unlock(vmf->ptl); - put_and_wait_on_page_locked(page, TASK_UNINTERRUPTIBLE); goto out; } - page = pmd_page(pmd); - BUG_ON(is_huge_zero_page(page)); - page_nid = page_to_nid(page); - last_cpupid = page_cpupid_last(page); - count_vm_numa_event(NUMA_HINT_FAULTS); - if (page_nid == this_nid) { - count_vm_numa_event(NUMA_HINT_FAULTS_LOCAL); - flags |= TNF_FAULT_LOCAL; - } - - /* See similar comment in do_numa_page for explanation */ - if (!pmd_savedwrite(pmd)) - flags |= TNF_NO_GROUP; - - /* - * Acquire the page lock to serialise THP migrations but avoid dropping - * page_table_lock if at all possible - */ - page_locked = trylock_page(page); - target_nid = mpol_misplaced(page, vma, haddr); - /* Migration could have started since the pmd_trans_migrating check */ - if (!page_locked) { - page_nid = NUMA_NO_NODE; - if (!get_page_unless_zero(page)) - goto out_unlock; - spin_unlock(vmf->ptl); - put_and_wait_on_page_locked(page, TASK_UNINTERRUPTIBLE); - goto out; - } else if (target_nid == NUMA_NO_NODE) { - /* There are no parallel migrations and page is in the right - * node. Clear the numa hinting info in this pmd. - */ - goto clear_pmdnuma; - } - - /* - * Page is misplaced. Page lock serialises migrations. Acquire anon_vma - * to serialises splits - */ - get_page(page); - spin_unlock(vmf->ptl); - anon_vma = page_lock_anon_vma_read(page); - - /* Confirm the PMD did not change while page_table_lock was released */ - spin_lock(vmf->ptl); - if (unlikely(!pmd_same(pmd, *vmf->pmd))) { - unlock_page(page); - put_page(page); - page_nid = NUMA_NO_NODE; - goto out_unlock; - } - - /* Bail if we fail to protect against THP splits for any reason */ - if (unlikely(!anon_vma)) { - put_page(page); - page_nid = NUMA_NO_NODE; - goto clear_pmdnuma; - } - /* * Since we took the NUMA fault, we must have observed the !accessible * bit. Make sure all other CPUs agree with that, to avoid them @@ -1525,43 +1453,60 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) haddr + HPAGE_PMD_SIZE); } - /* - * Migrate the THP to the requested node, returns with page unlocked - * and access rights restored. - */ + pmd = pmd_modify(oldpmd, vma->vm_page_prot); + page = vm_normal_page_pmd(vma, haddr, pmd); + if (!page) { + spin_unlock(vmf->ptl); + goto out_map; + } + + /* See similar comment in do_numa_page for explanation */ + if (!was_writable) + flags |= TNF_NO_GROUP; + + page_nid = page_to_nid(page); + last_cpupid = page_cpupid_last(page); + target_nid = numa_migrate_prep(page, vma, haddr, page_nid, + &flags); + + if (target_nid == NUMA_NO_NODE) { + put_page(page); + goto out_map; + } + spin_unlock(vmf->ptl); - migrated = migrate_misplaced_transhuge_page(vma->vm_mm, vma, - vmf->pmd, pmd, vmf->address, page, target_nid); + migrated = migrate_misplaced_page(page, vma, target_nid); if (migrated) { flags |= TNF_MIGRATED; page_nid = target_nid; - } else + } else { flags |= TNF_MIGRATE_FAIL; - - goto out; -clear_pmdnuma: - BUG_ON(!PageLocked(page)); - was_writable = pmd_savedwrite(pmd); - pmd = pmd_modify(pmd, vma->vm_page_prot); - pmd = pmd_mkyoung(pmd); - if (was_writable) - pmd = pmd_mkwrite(pmd); - set_pmd_at(vma->vm_mm, haddr, vmf->pmd, pmd); - update_mmu_cache_pmd(vma, vmf->address, vmf->pmd); - unlock_page(page); -out_unlock: - spin_unlock(vmf->ptl); + vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); + if (unlikely(!pmd_same(oldpmd, *vmf->pmd))) { + spin_unlock(vmf->ptl); + goto out; + } + goto out_map; + } out: - if (anon_vma) - page_unlock_anon_vma_read(anon_vma); - if (page_nid != NUMA_NO_NODE) task_numa_fault(last_cpupid, page_nid, HPAGE_PMD_NR, flags); return 0; + +out_map: + /* Restore the PMD */ + pmd = pmd_modify(oldpmd, vma->vm_page_prot); + pmd = pmd_mkyoung(pmd); + if (was_writable) + pmd = pmd_mkwrite(pmd); + set_pmd_at(vma->vm_mm, haddr, vmf->pmd, pmd); + update_mmu_cache_pmd(vma, vmf->address, vmf->pmd); + spin_unlock(vmf->ptl); + goto out; } /* diff --git a/mm/internal.h b/mm/internal.h index 28fc471b4669..d9a5d00bee1a 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -375,23 +375,6 @@ extern int mlock_future_check(struct mm_struct *mm, unsigned long flags, */ extern void clear_page_mlock(struct page *page); -/* - * mlock_migrate_page - called only from migrate_misplaced_transhuge_page() - * (because that does not go through the full procedure of migration ptes): - * to migrate the Mlocked page flag; update statistics. - */ -static inline void mlock_migrate_page(struct page *newpage, struct page *page) -{ - if (TestClearPageMlocked(page)) { - int nr_pages = thp_nr_pages(page); - - /* Holding pmd lock, no change in irq context: __mod is safe */ - __mod_zone_page_state(page_zone(page), NR_MLOCK, -nr_pages); - SetPageMlocked(newpage); - __mod_zone_page_state(page_zone(newpage), NR_MLOCK, nr_pages); - } -} - extern pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma); /* @@ -442,7 +425,6 @@ static inline struct file *maybe_unlock_mmap_for_io(struct vm_fault *vmf, #else /* !CONFIG_MMU */ static inline void clear_page_mlock(struct page *page) { } static inline void mlock_vma_page(struct page *page) { } -static inline void mlock_migrate_page(struct page *new, struct page *old) { } static inline void vunmap_range_noflush(unsigned long start, unsigned long end) { } diff --git a/mm/migrate.c b/mm/migrate.c index e7a173da74ec..4e0c2644256a 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2045,6 +2045,23 @@ static struct page *alloc_misplaced_dst_page(struct page *page, return newpage; } +static struct page *alloc_misplaced_dst_page_thp(struct page *page, + unsigned long data) +{ + int nid = (int) data; + struct page *newpage; + + newpage = alloc_pages_node(nid, (GFP_TRANSHUGE_LIGHT | __GFP_THISNODE), + HPAGE_PMD_ORDER); + if (!newpage) + goto out; + + prep_transhuge_page(newpage); + +out: + return newpage; +} + static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) { int page_lru; @@ -2083,12 +2100,6 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) return 1; } -bool pmd_trans_migrating(pmd_t pmd) -{ - struct page *page = pmd_page(pmd); - return PageLocked(page); -} - /* * Attempt to migrate a misplaced page to the specified destination * node. Caller is expected to have an elevated reference count on @@ -2101,6 +2112,20 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma, int isolated; int nr_remaining; LIST_HEAD(migratepages); + new_page_t *new; + bool compound; + + /* + * PTE mapped THP or HugeTLB page can't reach here so the page could + * be either base page or THP. And it must be head page if it is + * THP. + */ + compound = PageTransHuge(page); + + if (compound) + new = alloc_misplaced_dst_page_thp; + else + new = alloc_misplaced_dst_page; /* * Don't migrate file pages that are mapped in multiple processes @@ -2122,9 +2147,8 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma, goto out; list_add(&page->lru, &migratepages); - nr_remaining = migrate_pages(&migratepages, alloc_misplaced_dst_page, - NULL, node, MIGRATE_ASYNC, - MR_NUMA_MISPLACED); + nr_remaining = migrate_pages(&migratepages, *new, NULL, node, + MIGRATE_ASYNC, MR_NUMA_MISPLACED); if (nr_remaining) { if (!list_empty(&migratepages)) { list_del(&page->lru); @@ -2143,141 +2167,6 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma, return 0; } #endif /* CONFIG_NUMA_BALANCING */ - -#if defined(CONFIG_NUMA_BALANCING) && defined(CONFIG_TRANSPARENT_HUGEPAGE) -/* - * Migrates a THP to a given target node. page must be locked and is unlocked - * before returning. - */ -int migrate_misplaced_transhuge_page(struct mm_struct *mm, - struct vm_area_struct *vma, - pmd_t *pmd, pmd_t entry, - unsigned long address, - struct page *page, int node) -{ - spinlock_t *ptl; - pg_data_t *pgdat = NODE_DATA(node); - int isolated = 0; - struct page *new_page = NULL; - int page_lru = page_is_file_lru(page); - unsigned long start = address & HPAGE_PMD_MASK; - - new_page = alloc_pages_node(node, - (GFP_TRANSHUGE_LIGHT | __GFP_THISNODE), - HPAGE_PMD_ORDER); - if (!new_page) - goto out_fail; - prep_transhuge_page(new_page); - - isolated = numamigrate_isolate_page(pgdat, page); - if (!isolated) { - put_page(new_page); - goto out_fail; - } - - /* Prepare a page as a migration target */ - __SetPageLocked(new_page); - if (PageSwapBacked(page)) - __SetPageSwapBacked(new_page); - - /* anon mapping, we can simply copy page->mapping to the new page: */ - new_page->mapping = page->mapping; - new_page->index = page->index; - /* flush the cache before copying using the kernel virtual address */ - flush_cache_range(vma, start, start + HPAGE_PMD_SIZE); - migrate_page_copy(new_page, page); - WARN_ON(PageLRU(new_page)); - - /* Recheck the target PMD */ - ptl = pmd_lock(mm, pmd); - if (unlikely(!pmd_same(*pmd, entry) || !page_ref_freeze(page, 2))) { - spin_unlock(ptl); - - /* Reverse changes made by migrate_page_copy() */ - if (TestClearPageActive(new_page)) - SetPageActive(page); - if (TestClearPageUnevictable(new_page)) - SetPageUnevictable(page); - - unlock_page(new_page); - put_page(new_page); /* Free it */ - - /* Retake the callers reference and putback on LRU */ - get_page(page); - putback_lru_page(page); - mod_node_page_state(page_pgdat(page), - NR_ISOLATED_ANON + page_lru, -HPAGE_PMD_NR); - - goto out_unlock; - } - - entry = mk_huge_pmd(new_page, vma->vm_page_prot); - entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); - - /* - * Overwrite the old entry under pagetable lock and establish - * the new PTE. Any parallel GUP will either observe the old - * page blocking on the page lock, block on the page table - * lock or observe the new page. The SetPageUptodate on the - * new page and page_add_new_anon_rmap guarantee the copy is - * visible before the pagetable update. - */ - page_add_anon_rmap(new_page, vma, start, true); - /* - * At this point the pmd is numa/protnone (i.e. non present) and the TLB - * has already been flushed globally. So no TLB can be currently - * caching this non present pmd mapping. There's no need to clear the - * pmd before doing set_pmd_at(), nor to flush the TLB after - * set_pmd_at(). Clearing the pmd here would introduce a race - * condition against MADV_DONTNEED, because MADV_DONTNEED only holds the - * mmap_lock for reading. If the pmd is set to NULL at any given time, - * MADV_DONTNEED won't wait on the pmd lock and it'll skip clearing this - * pmd. - */ - set_pmd_at(mm, start, pmd, entry); - update_mmu_cache_pmd(vma, address, &entry); - - page_ref_unfreeze(page, 2); - mlock_migrate_page(new_page, page); - page_remove_rmap(page, true); - set_page_owner_migrate_reason(new_page, MR_NUMA_MISPLACED); - - spin_unlock(ptl); - - /* Take an "isolate" reference and put new page on the LRU. */ - get_page(new_page); - putback_lru_page(new_page); - - unlock_page(new_page); - unlock_page(page); - put_page(page); /* Drop the rmap reference */ - put_page(page); /* Drop the LRU isolation reference */ - - count_vm_events(PGMIGRATE_SUCCESS, HPAGE_PMD_NR); - count_vm_numa_events(NUMA_PAGE_MIGRATE, HPAGE_PMD_NR); - - mod_node_page_state(page_pgdat(page), - NR_ISOLATED_ANON + page_lru, - -HPAGE_PMD_NR); - return isolated; - -out_fail: - count_vm_events(PGMIGRATE_FAIL, HPAGE_PMD_NR); - ptl = pmd_lock(mm, pmd); - if (pmd_same(*pmd, entry)) { - entry = pmd_modify(entry, vma->vm_page_prot); - set_pmd_at(mm, start, pmd, entry); - update_mmu_cache_pmd(vma, address, &entry); - } - spin_unlock(ptl); - -out_unlock: - unlock_page(page); - put_page(page); - return 0; -} -#endif /* CONFIG_NUMA_BALANCING */ - #endif /* CONFIG_NUMA */ #ifdef CONFIG_DEVICE_PRIVATE From patchwork Tue May 18 20:07:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12265589 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6AE5C43462 for ; Tue, 18 May 2021 20:08:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6C13A611B0 for ; Tue, 18 May 2021 20:08:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6C13A611B0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0F9998E004A; Tue, 18 May 2021 16:08:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0D28B8E002F; Tue, 18 May 2021 16:08:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E685F8E004A; Tue, 18 May 2021 16:08:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0041.hostedemail.com [216.40.44.41]) by kanga.kvack.org (Postfix) with ESMTP id B5A2A8E002F for ; Tue, 18 May 2021 16:08:27 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 5663DAF6F for ; Tue, 18 May 2021 20:08:27 +0000 (UTC) X-FDA: 78155439054.13.D23C161 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf08.hostedemail.com (Postfix) with ESMTP id A456F8019393 for ; Tue, 18 May 2021 20:08:23 +0000 (UTC) Received: by mail-pl1-f176.google.com with SMTP id i6so896303plt.4 for ; Tue, 18 May 2021 13:08:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NJy1kyPl+oDN1LphbCW3pYnL0yTMuSAszJt41Bp1UBM=; b=dz0rHWSH3hdqt902RCPgGvJ7ufXgHBvMlvWjyWU9TgOnB6+uISl/zfQAZqijTUm9lj gIzdnBYXvA5OTtf9+vGaIo0jwxQR+cpjiZ8Vd4kLF3Bm8qKL7Ftb1DGPQJu9vcww60/n K6Jp65JDLmLw4WXCJhZ2Ac7BxaN3ybDLaIDvgZV5WD+AHy+sTTeguiQA4C80YOCigIqg NR0wPWRtJ9pmeno0Ng6pa+ewucpiAMwHYVS7J58cd9LaxHDe4c5Uobo9zm460NtPGgPR gGgvKGG5Tp36HLsIbanO9KPOZfJYy3UdU+g2N+rEZnczLphoMPuOFtQYUy2H6+85PvwG sHYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NJy1kyPl+oDN1LphbCW3pYnL0yTMuSAszJt41Bp1UBM=; b=CX61NHsBGF7Tt9NJEr59TG64TFW2DxaiJijXKNE75Vpuj6lJzeVyHzl1QULueKYcUg WvJZ+kQWL93w40XahnHnXDzJJwlTR4wH3japDBBG5aZLPQQsJiDnR9iUrjD/iTb6j50O Q22E4qpjNaZtuOr7Tx+JWeoGypbLSYQ77Bk0KfmB0siSA+CYo7ypQYySQnAa4pjnxrDY /kMFey6VPmvsfDjPHwoRE9KVhO7p9KnW7JpsK3nlXrA1C0iXHSawbvB4k5siQfuTpS1V m+PnsH6rdVvZDkSIQN94vlmaSWGK9krPYQBsf16DxqnICKLXnBZHVvYHe+rAGUGueA0p AK8g== X-Gm-Message-State: AOAM531JAzwRV8nO67CaveYi/7ZE0y+iDpaWrYTocxaRZ1u742S2SNWl bTr3kxl7Hai6fpEjkF7SsEc= X-Google-Smtp-Source: ABdhPJxC5xHJpdW0gj/fkethiDc7bPdRQpp9SNv+bII0C3hZHPfOmRFe/+aSGlwmrPy4u/vr0u2k9w== X-Received: by 2002:a17:90a:ba16:: with SMTP id s22mr7191848pjr.12.1621368506220; Tue, 18 May 2021 13:08:26 -0700 (PDT) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id r11sm13456600pgl.34.2021.05.18.13.08.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 May 2021 13:08:25 -0700 (PDT) From: Yang Shi To: mgorman@suse.de, kirill.shutemov@linux.intel.com, ziy@nvidia.com, ying.huang@intel.com, mhocko@suse.com, hughd@google.com, gerald.schaefer@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, borntraeger@de.ibm.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v3 PATCH 4/7] mm: migrate: account THP NUMA migration counters correctly Date: Tue, 18 May 2021 13:07:58 -0700 Message-Id: <20210518200801.7413-5-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210518200801.7413-1-shy828301@gmail.com> References: <20210518200801.7413-1-shy828301@gmail.com> MIME-Version: 1.0 Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=dz0rHWSH; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf08.hostedemail.com: domain of shy828301@gmail.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=shy828301@gmail.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: A456F8019393 X-Stat-Signature: 3md77h5t4dhuyxm9haqkonjsykreh1rg X-HE-Tag: 1621368503-948442 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now both base page and THP NUMA migration is done via migrate_misplaced_page(), keep the counters correctly for THP. Acked-by: Mel Gorman Signed-off-by: Yang Shi --- mm/migrate.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 4e0c2644256a..26f79f516100 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2114,6 +2114,7 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma, LIST_HEAD(migratepages); new_page_t *new; bool compound; + unsigned int nr_pages = thp_nr_pages(page); /* * PTE mapped THP or HugeTLB page can't reach here so the page could @@ -2152,13 +2153,13 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma, if (nr_remaining) { if (!list_empty(&migratepages)) { list_del(&page->lru); - dec_node_page_state(page, NR_ISOLATED_ANON + - page_is_file_lru(page)); + mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON + + page_is_file_lru(page), -nr_pages); putback_lru_page(page); } isolated = 0; } else - count_vm_numa_event(NUMA_PAGE_MIGRATE); + count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_pages); BUG_ON(!list_empty(&migratepages)); return isolated; From patchwork Tue May 18 20:07:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12265591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6DE10C433B4 for ; Tue, 18 May 2021 20:08:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 166B061074 for ; Tue, 18 May 2021 20:08:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 166B061074 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A709A8E004B; Tue, 18 May 2021 16:08:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A46D38E002F; Tue, 18 May 2021 16:08:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 89C608E004B; Tue, 18 May 2021 16:08:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0118.hostedemail.com [216.40.44.118]) by kanga.kvack.org (Postfix) with ESMTP id 556AB8E002F for ; Tue, 18 May 2021 16:08:30 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id EE01C8249980 for ; Tue, 18 May 2021 20:08:29 +0000 (UTC) X-FDA: 78155439138.11.0FEFBDE Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by imf09.hostedemail.com (Postfix) with ESMTP id 1A94C60006E7 for ; Tue, 18 May 2021 20:08:26 +0000 (UTC) Received: by mail-pj1-f41.google.com with SMTP id t11so6140814pjm.0 for ; Tue, 18 May 2021 13:08:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xsOsjkhPCcPV0zv6oCcbmotnPtBUzCdIBSrvKgiDikA=; b=SwtmZy7QhVhjTdJlzMLxTCWs9t9a8LPJVRDTfGexGYehagSrQXPy+sobHKasW8WhMi vyV+EDOWiHNG0QHiCEaYBWXlmDvgeZnSCrI4kVdoDW1fnMCb5fLVB4oqPzCldtWSd+Aq gnIYMHgbuzjF1EoItH0nV+nzV6vuIjtVYMQ0NhpjLnAt37nvq8soPHEWH/eA6ajm/r13 IB2pUlLLDtEFKgoRUKuJp9XPRlvKj8c8YlJtdggxxjE6Kx+uMm6CBKme163wi42wIKX2 azfWKcO/Ws8VlDvHQOVcmOUNGTp7+WJM1G1SkP4IQhPMyODP/qleLPrnGGRja+3FeoXW Nifw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xsOsjkhPCcPV0zv6oCcbmotnPtBUzCdIBSrvKgiDikA=; b=phltRc9YipV2qJeacRS0v1hVR0h0dtT8Z4cmR3ZaHJXWmXwDoQ8eDcvGv+CU8Vm8w+ Nsf9iNhBfBslF4cMpeUbO3ETtRWLjJtOqXdquYPJ1GMRF1E4s18s97+S9qE+gNeobjLb VDDHCFBwxIXZIWxDMznlqZfjtw/sQJ2zueO+2DK91FROcotWWC+1a/Iz6ARuGxwHOhWf xrLKC+rF5YcRCyYW3wkPKdX4K4hfPePGBJHjuvwYo2ngDub6+eppMMdtrzC+gMyBVyFs vGPckq7lcdd8F5IIqO8H9a0Lafv5Ca2wZzbe9KPYUUvbbXjbcXK5N8BU+0QI7prv1Hdo W7ow== X-Gm-Message-State: AOAM533wnkdK8SvL+6F02kahmufc8PXqZTduojhboBW5EPmeN09fWIxr U+Q/eloNU0AvGFuzodrbSsw= X-Google-Smtp-Source: ABdhPJytDWDaq+aC4veI99lmSFliP67cUhY8lIZXHRGfmTdfTnvkUJxAwyUWmh809yNfMsIcSzth1w== X-Received: by 2002:a17:90a:6402:: with SMTP id g2mr7237187pjj.82.1621368508608; Tue, 18 May 2021 13:08:28 -0700 (PDT) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id r11sm13456600pgl.34.2021.05.18.13.08.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 May 2021 13:08:27 -0700 (PDT) From: Yang Shi To: mgorman@suse.de, kirill.shutemov@linux.intel.com, ziy@nvidia.com, ying.huang@intel.com, mhocko@suse.com, hughd@google.com, gerald.schaefer@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, borntraeger@de.ibm.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v3 PATCH 5/7] mm: migrate: don't split THP for misplaced NUMA page Date: Tue, 18 May 2021 13:07:59 -0700 Message-Id: <20210518200801.7413-6-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210518200801.7413-1-shy828301@gmail.com> References: <20210518200801.7413-1-shy828301@gmail.com> MIME-Version: 1.0 Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=SwtmZy7Q; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf09.hostedemail.com: domain of shy828301@gmail.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=shy828301@gmail.com X-Stat-Signature: 66d8mgtnecioh96ny6asmyuoe1btmhd9 X-Rspamd-Queue-Id: 1A94C60006E7 X-Rspamd-Server: rspam02 X-HE-Tag: 1621368506-579910 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The old behavior didn't split THP if migration is failed due to lack of memory on the target node. But the THP migration does split THP, so keep the old behavior for misplaced NUMA page migration. Acked-by: Mel Gorman Signed-off-by: Yang Shi --- mm/migrate.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/mm/migrate.c b/mm/migrate.c index 26f79f516100..035daaec1ca0 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1420,6 +1420,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, int swapwrite = current->flags & PF_SWAPWRITE; int rc, nr_subpages; LIST_HEAD(ret_pages); + bool nosplit = (reason == MR_NUMA_MISPLACED); trace_mm_migrate_pages_start(mode, reason); @@ -1491,8 +1492,9 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, /* * When memory is low, don't bother to try to migrate * other pages, just exit. + * THP NUMA faulting doesn't split THP to retry. */ - if (is_thp) { + if (is_thp && !nosplit) { if (!try_split_thp(page, &page2, from)) { nr_thp_split++; goto retry; From patchwork Tue May 18 20:08:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12265593 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34E01C43460 for ; Tue, 18 May 2021 20:08:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DFCDB6108D for ; Tue, 18 May 2021 20:08:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DFCDB6108D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7D70A8E004C; Tue, 18 May 2021 16:08:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7AF568E002F; Tue, 18 May 2021 16:08:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6532F8E004C; Tue, 18 May 2021 16:08:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0227.hostedemail.com [216.40.44.227]) by kanga.kvack.org (Postfix) with ESMTP id 244308E002F for ; Tue, 18 May 2021 16:08:32 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id B14308249980 for ; Tue, 18 May 2021 20:08:31 +0000 (UTC) X-FDA: 78155439222.27.04A2873 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf02.hostedemail.com (Postfix) with ESMTP id 1FFA44080F55 for ; Tue, 18 May 2021 20:08:28 +0000 (UTC) Received: by mail-pl1-f179.google.com with SMTP id t4so5735618plc.6 for ; Tue, 18 May 2021 13:08:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Bft5/S3sIep+UIOe0wjEBCs4qeqthplQvAas8/8e+yo=; b=nHLS3OJzElgnE2B/Z8SKCricVXLIx3iJxQjJ0syYMD0st4Cu/HagKgnXREM54zswjw XMNbtKvBMaoANopTaCd8U7u2Hm86+M3RuZtminAUpNjSsKZa0ANHy7i82oYV5IUBo3y5 9as9AvR0a6Tf/pms+EH7p0fkg0BSIgxX0pHVwN8P5Wx2tzfzJWe1qn5USVLiIJCq+FuE BSJupLShAnX1rbhtkb46N4UqYZ8EQt19Oxj+xOEtlDuZCPe1a2dfSKZZsE6riRl+RTUN VZadMbE778WhVEQIFJBh5I6kdzgTgYVgZZeinQpMkA7oXwHQKhc9R+s52vZyIwRN3/vc 4HjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Bft5/S3sIep+UIOe0wjEBCs4qeqthplQvAas8/8e+yo=; b=fLbtqtFlCmDFJ4mwcaUeBeBA7ZCbdfd4ES2I6hkqjBa/Unb9V6EU24xRknyGM1c9Rz CeizyIYHM2UVZIIAwT32w80nS2HEoQLwGVN/LN2codncQSXb7e0jxgxB8yLFdvor1RoX rHL8gqOWruIE7kWeuFD109grbzuZ8pEjfpFcK3HF3Glmim1OmR8nrJ5NcXL745Tjqfw+ 7OC0IFJCHb7yHXniBsaHPmhZKgMDuBaqLi2zobWZMXbDBRI1RFprLLhklUImjHsPGAse n0QKEscWZbjj0pjSUWkaXaOYSi4Hr5q57ZugScr9FCud/r33bXE1ubIOB0mG95K8QvQG j3ww== X-Gm-Message-State: AOAM530wynkR/sQMNMQW4PNr6BfftG+Hv+pc+VQq9vjgNbkbeYTMOcWi UYKF58AcoDPy9+zCbhZo/zw= X-Google-Smtp-Source: ABdhPJycucOy8RWBt3ZSLaK0DULZYOHapGP8SDJZBX29cfFbM7ELURa2SaI9yfmt82bS4uEWVqBwiA== X-Received: by 2002:a17:902:e007:b029:ef:9dd2:be6 with SMTP id o7-20020a170902e007b02900ef9dd20be6mr6495035plo.12.1621368510709; Tue, 18 May 2021 13:08:30 -0700 (PDT) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id r11sm13456600pgl.34.2021.05.18.13.08.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 May 2021 13:08:29 -0700 (PDT) From: Yang Shi To: mgorman@suse.de, kirill.shutemov@linux.intel.com, ziy@nvidia.com, ying.huang@intel.com, mhocko@suse.com, hughd@google.com, gerald.schaefer@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, borntraeger@de.ibm.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v3 PATCH 6/7] mm: migrate: check mapcount for THP instead of refcount Date: Tue, 18 May 2021 13:08:00 -0700 Message-Id: <20210518200801.7413-7-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210518200801.7413-1-shy828301@gmail.com> References: <20210518200801.7413-1-shy828301@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1FFA44080F55 Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=nHLS3OJz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of shy828301@gmail.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=shy828301@gmail.com X-Rspamd-Server: rspam03 X-Stat-Signature: o5mdnzr654pn7cwurb8hcefiswx85p5r X-HE-Tag: 1621368508-542250 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The generic migration path will check refcount, so no need check refcount here. But the old code actually prevents from migrating shared THP (mapped by multiple processes), so bail out early if mapcount is > 1 to keep the behavior. Signed-off-by: Yang Shi --- mm/migrate.c | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 035daaec1ca0..77215f28502b 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2070,6 +2070,10 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) VM_BUG_ON_PAGE(compound_order(page) && !PageTransHuge(page), page); + /* Do not migrate THP mapped by multiple processes */ + if (PageTransHuge(page) && total_mapcount(page) > 1) + return 0; + /* Avoid migrating to a node that is nearly full */ if (!migrate_balanced_pgdat(pgdat, compound_nr(page))) return 0; @@ -2077,18 +2081,6 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) if (isolate_lru_page(page)) return 0; - /* - * migrate_misplaced_transhuge_page() skips page migration's usual - * check on page_count(), so we must do it here, now that the page - * has been isolated: a GUP pin, or any other pin, prevents migration. - * The expected page count is 3: 1 for page's mapcount and 1 for the - * caller's pin and 1 for the reference taken by isolate_lru_page(). - */ - if (PageTransHuge(page) && page_count(page) != 3) { - putback_lru_page(page); - return 0; - } - page_lru = page_is_file_lru(page); mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON + page_lru, thp_nr_pages(page)); From patchwork Tue May 18 20:08:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12265595 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A48A0C433B4 for ; Tue, 18 May 2021 20:08:35 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5224D61074 for ; Tue, 18 May 2021 20:08:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5224D61074 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D84078E004D; Tue, 18 May 2021 16:08:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D07DB8E002F; Tue, 18 May 2021 16:08:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA7CF8E004D; Tue, 18 May 2021 16:08:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0237.hostedemail.com [216.40.44.237]) by kanga.kvack.org (Postfix) with ESMTP id 7F9658E002F for ; Tue, 18 May 2021 16:08:34 -0400 (EDT) Received: from smtpin35.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 201B3ABE8 for ; Tue, 18 May 2021 20:08:34 +0000 (UTC) X-FDA: 78155439348.35.4232264 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf21.hostedemail.com (Postfix) with ESMTP id 5B82AE000111 for ; Tue, 18 May 2021 20:08:33 +0000 (UTC) Received: by mail-pl1-f174.google.com with SMTP id 69so5736792plc.5 for ; Tue, 18 May 2021 13:08:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/vcL4h8Y/iDJ4PQ//oXXgHbE7D0ndwuNE52sYaiQ5uA=; b=Rm/Otlj3gP5XkV3BLLxJkLQZNy2jj12x4U46YT0W9OzEwpis3BcOIrYA7TVCiGcBYd Y/A52de9C1ucmsAiRKng/fYtrQYoCX40xawvO30vOcJUOan7Bv2m4ddeDNmr4hxw0Qq0 6jl+8hWGd+pTu+rgkD94iUuIYKjREOUB2cvP2M5h11Udgoyr7RPzlPbEi5XXf4WweaUf ZpLLVOF0qWA1+HfY6ZP6JGee3F1z5RIbsZofK3zJeI+YbVgxyJb61AlXkNcywq7xNgtx S8bSVLZUqAGKZLV9CHJpPhGdkhrgfZCRAoZ+R2kGYxTIKwp5+kZWKvxXdkwpQeMKM6cu +FHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/vcL4h8Y/iDJ4PQ//oXXgHbE7D0ndwuNE52sYaiQ5uA=; b=Ta3zga8hygkMTg8YBXE6AkEjbYRGBqHGUv3/fSQd/Ghpg+v+3UK204Si1xrXOAEHfF TnTsyHDYpbkn9CNPstdrXeBTIFC3bbadh+VudLuoSteVnFDaz6/4iwSTeKE3zFTbMUgL NO1NWSt5RXrX3crCOROocFNp+bQ8sqPGZvUtuuCbxeUAuFiLIeIZxLLJkhaG47Vlr8Ze AFDXEcPwmK81ZrObVBhYvugm/nM9vUWeNx1+Z/B+s0AiU52NBH9s6KawhUFd53yUtYO/ lH7xSPGnq31fleFplaxJDtjPxS5UySf0NLs+OQ0Dwc5FD+mLJJVwH/hYNhzCEyE5yO8y lZKQ== X-Gm-Message-State: AOAM530NxAGqd6CWmojNnZ3Db/COXrIXzNXBzzpcNPKCokdFyApwM9RR 2l6HrnWO7OfjVDasM0+OgZM= X-Google-Smtp-Source: ABdhPJzHjDupKD9HBvIBLGRi6l5SuWQ81rgs7GzcOZnGf3LhF6AGiUZhf89yxQ3FkYWXt210nsqHGw== X-Received: by 2002:a17:902:8c91:b029:f3:b4da:4600 with SMTP id t17-20020a1709028c91b02900f3b4da4600mr3168651plo.30.1621368512990; Tue, 18 May 2021 13:08:32 -0700 (PDT) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id r11sm13456600pgl.34.2021.05.18.13.08.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 May 2021 13:08:32 -0700 (PDT) From: Yang Shi To: mgorman@suse.de, kirill.shutemov@linux.intel.com, ziy@nvidia.com, ying.huang@intel.com, mhocko@suse.com, hughd@google.com, gerald.schaefer@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, borntraeger@de.ibm.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v3 PATCH 7/7] mm: thp: skip make PMD PROT_NONE if THP migration is not supported Date: Tue, 18 May 2021 13:08:01 -0700 Message-Id: <20210518200801.7413-8-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210518200801.7413-1-shy828301@gmail.com> References: <20210518200801.7413-1-shy828301@gmail.com> MIME-Version: 1.0 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b="Rm/Otlj3"; spf=pass (imf21.hostedemail.com: domain of shy828301@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 5B82AE000111 X-Stat-Signature: pe833skjwhfyfqtmxsn1rahfjon85o48 X-HE-Tag: 1621368513-760958 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A quick grep shows x86_64, PowerPC (book3s), ARM64 and S390 support both NUMA balancing and THP. But S390 doesn't support THP migration so NUMA balancing actually can't migrate any misplaced pages. Skip make PMD PROT_NONE for such case otherwise CPU cycles may be wasted by pointless NUMA hinting faults on S390. Acked-by: Mel Gorman Signed-off-by: Yang Shi --- mm/huge_memory.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index b8526b9b041a..fb984ef5a761 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1735,6 +1735,7 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr, * Returns * - 0 if PMD could not be locked * - 1 if PMD was locked but protections unchanged and TLB flush unnecessary + * or if prot_numa but THP migration is not supported * - HPAGE_PMD_NR if protections changed and TLB flush necessary */ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, @@ -1749,6 +1750,9 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, bool uffd_wp = cp_flags & MM_CP_UFFD_WP; bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE; + if (prot_numa && !thp_migration_supported()) + return 1; + ptl = __pmd_trans_huge_lock(pmd, vma); if (!ptl) return 0;