From patchwork Fri Jun 22 03:51:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 10481167 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 6A44260383 for ; Fri, 22 Jun 2018 03:56:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5D9EF28F91 for ; Fri, 22 Jun 2018 03:56:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 51D8628F98; Fri, 22 Jun 2018 03:56:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CB6EC28F91 for ; Fri, 22 Jun 2018 03:56:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 851BA6B027D; Thu, 21 Jun 2018 23:55:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7B0656B027E; Thu, 21 Jun 2018 23:55:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 64F096B027F; Thu, 21 Jun 2018 23:55:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf0-f198.google.com (mail-pf0-f198.google.com [209.85.192.198]) by kanga.kvack.org (Postfix) with ESMTP id 1A4586B027D for ; Thu, 21 Jun 2018 23:55:51 -0400 (EDT) Received: by mail-pf0-f198.google.com with SMTP id v10-v6so2548128pfm.11 for ; Thu, 21 Jun 2018 20:55:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=yW2i1Ez//chvcvsQ5lTNVepZ3OlvOOaAeTclwLP/FqY=; b=n8pmIBA0w/Hjb1e/ToQyMdQTJlRgMceSLt7qVKgiz8IN1PPwPFJeyJWYh+YP7ctBDc 0BMu3CR0KoYhRisn2DPqVWfMtfWBXWbWs0QXEtxDUtoz7QWDN5/fECSROyV9NTRVWilJ xHcvUKT6oIMLjkac9vlxYXt5wJgp62AqKMJtYxtST5uer64/CGTBydViLZFZLK7Ri8vY EsXxRZE093MlNNjgb891GkyJQdUL74zVeT69v0SfFaqQuOAV5DtKrj3oX3kg1k9cvdEl APiADkHRU6hME4zYXZIlxn7vrsSOunyt5VyMbZSM1CUmQQo5+pMxgS2apniq+R+2JEDk bP7g== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APt69E2HLTOQCrEybEE6Rzexe8/M3zC5cI9YTtrPMPPg3UGTnxUsao7k YUDfDumw4MRT1fXvpzyvOP0v3kGi4P97BLgn2911gmTe2OM3FffGrM/QBVTcAOqtTS/oOMFeDr7 tnguyspSJ0tEYjfS3ANNyQelalFhesxbH5cGyY+ZW2O14AZF14YUr6y5NkVsdt6Y9Nw== X-Received: by 2002:a17:902:b604:: with SMTP id b4-v6mr31208810pls.18.1529639750723; Thu, 21 Jun 2018 20:55:50 -0700 (PDT) X-Google-Smtp-Source: ADUXVKJd9CLrGCEm94hl6/TkZg3aHdSywmNXkJOTc21iAGklG0bCykFsitxVx8S+Wvdq45yHDriK X-Received: by 2002:a17:902:b604:: with SMTP id b4-v6mr31208770pls.18.1529639749963; Thu, 21 Jun 2018 20:55:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529639749; cv=none; d=google.com; s=arc-20160816; b=u5XodAeln2O+ibfIaIfjFZ/Y/lv3YxYSqpiLNPFYp/s9LybMYxjQzF3g+AdX6yb5Aa MDtGnAz/8CPaSfCWxswt3+DmhDoxCx7gtXXYUEqJv7mJLesagDkQg2Ek7QZfuMwrzj6e VaUBKx0udm+FWATUJ2LKB5rzzh+yjlzqXtjA/i0sectwZv5LTNxvTEdR3F9p44hupvvL E4g+0+1AWcdeZyKgT6fCCg2961CdY9R3Jlqo5R5XCDN5HcRYHlqVu+MvNOnfdemTd6s9 T1b97go7jwU+5w0303xB35IF3b70rBcqsYNoiHcCZZOOf+J5L63Ve4l0LFwS8y1mMOlD VS5A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=yW2i1Ez//chvcvsQ5lTNVepZ3OlvOOaAeTclwLP/FqY=; b=XfvRRL5+8bFKz/5Zj4r0IXjqdWlyoarT4U929hBeyrrZoycwUkZ5Yg2t8ArjNgTg5U XLqFuNsKd4+yZeW5iIsERHpj7u4OOaXVxMGBf1VqyLTViSnPZHRkCSROxXF8V+2YwExM S81daLL1ZyF+AFlqfy8JZb9urG6U9Qjey8l9Yy4Nwl/x6mQQ1+G1xIKcU+Mf7dsecH7B D1eEHyQXUI3tG1gH3SNH4f0Z2Rvergie9CeUhf27QPxolUX2ZtXndSBt78v65X0j1cw6 FLeyHa190K0hRFKOh0e6ub4RfFJMhDL3NFD450zZuBfjr7uts9NQiZ+MT/ncb+3ILsgn kkXQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga01.intel.com (mga01.intel.com. [192.55.52.88]) by mx.google.com with ESMTPS id n61-v6si6160066plb.256.2018.06.21.20.55.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Jun 2018 20:55:49 -0700 (PDT) Received-SPF: pass (google.com: domain of ying.huang@intel.com designates 192.55.52.88 as permitted sender) client-ip=192.55.52.88; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Jun 2018 20:55:49 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,255,1526367600"; d="scan'208";a="65335172" Received: from wanpingl-mobl.ccr.corp.intel.com (HELO yhuang6-ux31a.ccr.corp.intel.com) ([10.254.212.200]) by fmsmga004.fm.intel.com with ESMTP; 21 Jun 2018 20:55:47 -0700 From: "Huang, Ying" To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , "Kirill A. Shutemov" , Andrea Arcangeli , Michal Hocko , Johannes Weiner , Shaohua Li , Hugh Dickins , Minchan Kim , Rik van Riel , Dave Hansen , Naoya Horiguchi , Zi Yan , Daniel Jordan Subject: [PATCH -mm -v4 15/21] mm, THP, swap: Support to copy PMD swap mapping when fork() Date: Fri, 22 Jun 2018 11:51:45 +0800 Message-Id: <20180622035151.6676-16-ying.huang@intel.com> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20180622035151.6676-1-ying.huang@intel.com> References: <20180622035151.6676-1-ying.huang@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Huang Ying During fork, the page table need to be copied from parent to child. A PMD swap mapping need to be copied too and the swap reference count need to be increased. When the huge swap cluster has been split already, we need to split the PMD swap mapping and fallback to PTE copying. When swap count continuation failed to allocate a page with GFP_ATOMIC, we need to unlock the spinlock and try again with GFP_KERNEL. Signed-off-by: "Huang, Ying" Cc: "Kirill A. Shutemov" Cc: Andrea Arcangeli Cc: Michal Hocko Cc: Johannes Weiner Cc: Shaohua Li Cc: Hugh Dickins Cc: Minchan Kim Cc: Rik van Riel Cc: Dave Hansen Cc: Naoya Horiguchi Cc: Zi Yan Cc: Daniel Jordan --- mm/huge_memory.c | 72 ++++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 57 insertions(+), 15 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index adad32d54ff2..38c247a38f67 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -941,6 +941,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, if (unlikely(!pgtable)) goto out; +retry: dst_ptl = pmd_lock(dst_mm, dst_pmd); src_ptl = pmd_lockptr(src_mm, src_pmd); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); @@ -948,26 +949,67 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, ret = -EAGAIN; pmd = *src_pmd; -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION if (unlikely(is_swap_pmd(pmd))) { swp_entry_t entry = pmd_to_swp_entry(pmd); - VM_BUG_ON(!is_pmd_migration_entry(pmd)); - if (is_write_migration_entry(entry)) { - make_migration_entry_read(&entry); - pmd = swp_entry_to_pmd(entry); - if (pmd_swp_soft_dirty(*src_pmd)) - pmd = pmd_swp_mksoft_dirty(pmd); - set_pmd_at(src_mm, addr, src_pmd, pmd); +#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION + if (is_migration_entry(entry)) { + if (is_write_migration_entry(entry)) { + make_migration_entry_read(&entry); + pmd = swp_entry_to_pmd(entry); + if (pmd_swp_soft_dirty(*src_pmd)) + pmd = pmd_swp_mksoft_dirty(pmd); + set_pmd_at(src_mm, addr, src_pmd, pmd); + } + add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); + mm_inc_nr_ptes(dst_mm); + pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); + set_pmd_at(dst_mm, addr, dst_pmd, pmd); + ret = 0; + goto out_unlock; } - add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); - mm_inc_nr_ptes(dst_mm); - pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); - set_pmd_at(dst_mm, addr, dst_pmd, pmd); - ret = 0; - goto out_unlock; - } #endif + if (thp_swap_supported() && !non_swap_entry(entry)) { + ret = swap_duplicate(&entry, true); + if (!ret) { + add_mm_counter(dst_mm, MM_SWAPENTS, + HPAGE_PMD_NR); + mm_inc_nr_ptes(dst_mm); + pgtable_trans_huge_deposit(dst_mm, dst_pmd, + pgtable); + set_pmd_at(dst_mm, addr, dst_pmd, pmd); + /* make sure dst_mm is on swapoff's mmlist. */ + if (unlikely(list_empty(&dst_mm->mmlist))) { + spin_lock(&mmlist_lock); + if (list_empty(&dst_mm->mmlist)) + list_add(&dst_mm->mmlist, + &src_mm->mmlist); + spin_unlock(&mmlist_lock); + } + } else if (ret == -ENOTDIR) { + /* + * The swap cluster has been split, split the + * pmd map now + */ + __split_huge_swap_pmd(vma, addr, src_pmd); + pte_free(dst_mm, pgtable); + } else if (ret == -ENOMEM) { + spin_unlock(src_ptl); + spin_unlock(dst_ptl); + ret = add_swap_count_continuation(entry, + GFP_KERNEL); + if (ret < 0) { + ret = -ENOMEM; + pte_free(dst_mm, pgtable); + goto out; + } + goto retry; + } else + VM_BUG_ON(1); + goto out_unlock; + } + VM_BUG_ON(1); + } if (unlikely(!pmd_trans_huge(pmd))) { pte_free(dst_mm, pgtable);