From patchwork Mon Sep 3 07:22:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 10585579 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B7D9814BD for ; Mon, 3 Sep 2018 07:23:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A127C28785 for ; Mon, 3 Sep 2018 07:23:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 95820287C6; Mon, 3 Sep 2018 07:23:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 04F4E28785 for ; Mon, 3 Sep 2018 07:23:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C8136B66BE; Mon, 3 Sep 2018 03:23:00 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 29EF16B66C0; Mon, 3 Sep 2018 03:23:00 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 18BC06B66BF; Mon, 3 Sep 2018 03:23:00 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id C43396B66BD for ; Mon, 3 Sep 2018 03:22:59 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id s22-v6so7731798plq.21 for ; Mon, 03 Sep 2018 00:22:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=pwSeESJGFFTVsNtUrsZfBu4js7ORgUq+NaSC8N5CVac=; b=tDoO2wWgL9GwDBC7C+92+ZJguv3aKJkwzdZcHG7hMtX7h3IgPMYz/K1h2e7VRMqar+ BNovZYkjEewHtMb1EZccYx9VTPeBIVTi1DusbGWpQwB+i8GoFCUX8FnH7UEiUbaxMPWp 00RJADGgq8zzRe10icISyJrCuEStxoc/GHv4+qeX3p+VcDluCf65L9H+4+HchKYXBjdd HTqTlIlUXz4P90e3OifTT4W5SuREPXrqEaTTSUbxQM74Jb6HQwWO307U7oc7RLJHL5ZU fgqcQgUXQK2Pm/yej4JMcgmds0iNNsbX0xkatOXqxM62SiecGxw9rfVZ+3NKnoUoLdZU rnqQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51C5+jUQfeRPWv8s/6egOqDZ4835dWWOKHu7XNIibywgh0R2PKOp eyARB1E7KG49/x7m/upzFGgWij4xaW0+DYjEAZ0GnOe71ZQPz8PQqLh+TGt48VKoE+4PpccnvPt wgkKdxfZUQx+2UvgXqoSSqQ6J6U7OtmlfP0GZ5Xs/hSP398KNI9++RtggDYb+3ktHGQ== X-Received: by 2002:a63:67c3:: with SMTP id b186-v6mr13306575pgc.5.1535959379483; Mon, 03 Sep 2018 00:22:59 -0700 (PDT) X-Google-Smtp-Source: ANB0VdbGg6pBxiYY+fnfiGCF2WIVGk/08vwXZjyfuT2an3+n3Tuc4gXIqEBAA60i7DbQ8casE6rs X-Received: by 2002:a63:67c3:: with SMTP id b186-v6mr13306549pgc.5.1535959378775; Mon, 03 Sep 2018 00:22:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535959378; cv=none; d=google.com; s=arc-20160816; b=BGItmRXKvW/HTBwDBaCGMZ5uUGpMCuYM6raG8W7O+O7KNtGdrujU9n3xzgZ5lyaTNe ahARLBHmHq3fKa2RIrQJ8TDFDfMu7bZd/iFB856ho5Y3HMyfftzGcUdQIy6xfw5crMZz X35MZ2ypp66l7+CBDZIGQ5BK5dbSlu/jemRWh66bvo0YVDD729mwyf4sSczpv/+OM2i5 U5J0BdwFKoPQM07pHBfbXZUfM39PGKZM+cX8e7J6AuOILYYb/0Ry3xJr3ze9Dad8SYH3 qDsb63z7aaqeB7x4qA4KGl4h5TsuNJRbIpt9fOgul+iLtHvBZgy/23M+O8TKjTduA0Th CU1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=pwSeESJGFFTVsNtUrsZfBu4js7ORgUq+NaSC8N5CVac=; b=vLj8x8lz3fuayDybb5h9vRc3qU+uRP6HTU6AAjq/vTutgBQIvJKG1bQBAvr4obK86H I03+q9iJCM2Nxn2Bjzn7u/ZooZB6uxs7kfmJl36QJBIFgHFK/ZC/kNmFkl/poJKYzzi1 VvcbEyQiVrzrw8158Abnt0S4PLV8ULVLULq+ZM6mGBHx0kViaYneOcAX7XkxdIiUTVT6 kg5Vm3t3NLzT3tBTg0z3M4+EvjgBc/1E9Ddfu8/GqJq5kIedLcxsVg0sdGdHSNglzTUC sI9ocn82dqsOhYmkSNQUch8yKHE3UL5kv4UaX+CYbGqOJ08jWhY1M/+qAK4xJVRD50I6 xNCw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga07.intel.com (mga07.intel.com. [134.134.136.100]) by mx.google.com with ESMTPS id az1-v6si9327648plb.513.2018.09.03.00.22.58 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 03 Sep 2018 00:22:58 -0700 (PDT) Received-SPF: pass (google.com: domain of ying.huang@intel.com designates 134.134.136.100 as permitted sender) client-ip=134.134.136.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 03 Sep 2018 00:22:58 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,324,1531810800"; d="scan'208";a="70146947" Received: from yhuang-mobile.sh.intel.com ([10.239.196.86]) by orsmga008.jf.intel.com with ESMTP; 03 Sep 2018 00:22:55 -0700 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , "Kirill A. Shutemov" , Andrea Arcangeli , Michal Hocko , Johannes Weiner , Shaohua Li , Hugh Dickins , Minchan Kim , Rik van Riel , Dave Hansen , Naoya Horiguchi , Zi Yan , Daniel Jordan Subject: [PATCH -V5 15/21] swap: Support to copy PMD swap mapping when fork() Date: Mon, 3 Sep 2018 15:22:08 +0800 Message-Id: <20180903072214.24602-16-ying.huang@intel.com> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20180903072214.24602-1-ying.huang@intel.com> References: <20180903072214.24602-1-ying.huang@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP During fork, the page table need to be copied from parent to child. A PMD swap mapping need to be copied too and the swap reference count need to be increased. When the huge swap cluster has been split already, we need to split the PMD swap mapping and fallback to PTE copying. When swap count continuation failed to allocate a page with GFP_ATOMIC, we need to unlock the spinlock and try again with GFP_KERNEL. Signed-off-by: "Huang, Ying" Cc: "Kirill A. Shutemov" Cc: Andrea Arcangeli Cc: Michal Hocko Cc: Johannes Weiner Cc: Shaohua Li Cc: Hugh Dickins Cc: Minchan Kim Cc: Rik van Riel Cc: Dave Hansen Cc: Naoya Horiguchi Cc: Zi Yan Cc: Daniel Jordan --- mm/huge_memory.c | 72 ++++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 57 insertions(+), 15 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 292c16b21442..56b12f533a64 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -941,6 +941,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, if (unlikely(!pgtable)) goto out; +retry: dst_ptl = pmd_lock(dst_mm, dst_pmd); src_ptl = pmd_lockptr(src_mm, src_pmd); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); @@ -948,26 +949,67 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, ret = -EAGAIN; pmd = *src_pmd; -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION if (unlikely(is_swap_pmd(pmd))) { swp_entry_t entry = pmd_to_swp_entry(pmd); - VM_BUG_ON(!is_pmd_migration_entry(pmd)); - if (is_write_migration_entry(entry)) { - make_migration_entry_read(&entry); - pmd = swp_entry_to_pmd(entry); - if (pmd_swp_soft_dirty(*src_pmd)) - pmd = pmd_swp_mksoft_dirty(pmd); - set_pmd_at(src_mm, addr, src_pmd, pmd); +#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION + if (is_migration_entry(entry)) { + if (is_write_migration_entry(entry)) { + make_migration_entry_read(&entry); + pmd = swp_entry_to_pmd(entry); + if (pmd_swp_soft_dirty(*src_pmd)) + pmd = pmd_swp_mksoft_dirty(pmd); + set_pmd_at(src_mm, addr, src_pmd, pmd); + } + add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); + mm_inc_nr_ptes(dst_mm); + pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); + set_pmd_at(dst_mm, addr, dst_pmd, pmd); + ret = 0; + goto out_unlock; } - add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); - mm_inc_nr_ptes(dst_mm); - pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); - set_pmd_at(dst_mm, addr, dst_pmd, pmd); - ret = 0; - goto out_unlock; - } #endif + if (IS_ENABLED(CONFIG_THP_SWAP) && !non_swap_entry(entry)) { + ret = swap_duplicate(&entry, HPAGE_PMD_NR); + if (!ret) { + add_mm_counter(dst_mm, MM_SWAPENTS, + HPAGE_PMD_NR); + mm_inc_nr_ptes(dst_mm); + pgtable_trans_huge_deposit(dst_mm, dst_pmd, + pgtable); + set_pmd_at(dst_mm, addr, dst_pmd, pmd); + /* make sure dst_mm is on swapoff's mmlist. */ + if (unlikely(list_empty(&dst_mm->mmlist))) { + spin_lock(&mmlist_lock); + if (list_empty(&dst_mm->mmlist)) + list_add(&dst_mm->mmlist, + &src_mm->mmlist); + spin_unlock(&mmlist_lock); + } + } else if (ret == -ENOTDIR) { + /* + * The huge swap cluster has been split, split + * the PMD swap mapping and fallback to PTE + */ + __split_huge_swap_pmd(vma, addr, src_pmd); + pte_free(dst_mm, pgtable); + } else if (ret == -ENOMEM) { + spin_unlock(src_ptl); + spin_unlock(dst_ptl); + ret = add_swap_count_continuation(entry, + GFP_KERNEL); + if (ret < 0) { + ret = -ENOMEM; + pte_free(dst_mm, pgtable); + goto out; + } + goto retry; + } else + VM_BUG_ON(1); + goto out_unlock; + } + VM_BUG_ON(1); + } if (unlikely(!pmd_trans_huge(pmd))) { pte_free(dst_mm, pgtable);