From patchwork Wed May 23 08:26:19 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 10420615 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 7850A60224 for ; Wed, 23 May 2018 08:27:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6B07028E3F for ; Wed, 23 May 2018 08:27:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5F02F28E54; Wed, 23 May 2018 08:27:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B407C28E3F for ; Wed, 23 May 2018 08:27:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CE9D06B0270; Wed, 23 May 2018 04:27:15 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C9A106B0271; Wed, 23 May 2018 04:27:15 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB9C06B0272; Wed, 23 May 2018 04:27:15 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf0-f198.google.com (mail-pf0-f198.google.com [209.85.192.198]) by kanga.kvack.org (Postfix) with ESMTP id 75D516B0270 for ; Wed, 23 May 2018 04:27:15 -0400 (EDT) Received: by mail-pf0-f198.google.com with SMTP id l85-v6so12618021pfb.18 for ; Wed, 23 May 2018 01:27:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=r5fJWceke6VA6XqCta8sbDWJd2+freUpXAQOaYo1tYI=; b=Ky4q1zBj90uXYEof9A+ymwQ9cWMOcbyLo5u96D+7cr6SUeu3z2+rs+EAqWu4zRefDf gzOfJJCsTui7x4J+Lgp/jjB5yOnM3MBp2duvmOex5D+q/frT1/4nygbxCtxPA+euU6ic Ya6HAePmKwz3OfCPdHaFCgq8o7Sm0lcCYQGis5Wp2uoDhRt88J5lwPRFfQ6Vrp+ezckO GZ98Gt+QDygjBOMsXz02YIEPitBAk5Dl+FBVpG331507iBsHkje9suBQHGpC3SN0Zp6d lqbv6BwrS7y9/Gbo70WA6NwT5h84k+HRc0iT8HNp38QEzQfOiUGBUiOekNXc/UCc9WpX qMpw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ALKqPwfTHlOCYSHryG/Q1o9ej16fMixzFZrPrqMv4JGXfZkS18cq4lZP ohuwSMk8psHvapGKGmlUhJOHqYqCxe4SJJjLhUs6O8a2KF9q46pgTxQOMCxdV9CHiT0wqnK/wU6 lQ0DNgES06P2kL/6tZGwDaW6Q5I4GWDKy59bnnLNOzsYcoZAPaNAfS+DW8I0UWUJHrA== X-Received: by 2002:a65:4204:: with SMTP id c4-v6mr1533470pgq.26.1527064035165; Wed, 23 May 2018 01:27:15 -0700 (PDT) X-Google-Smtp-Source: AB8JxZoqy+DzPmIUCjYfYC+DhGmmn2Nhwyg4Bj1ZU2zh7bP9Ngoz0n349qFaSlLqfYbG6b8IHXcx X-Received: by 2002:a65:4204:: with SMTP id c4-v6mr1533436pgq.26.1527064034476; Wed, 23 May 2018 01:27:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1527064034; cv=none; d=google.com; s=arc-20160816; b=SzGXn8Qdm+nD4+idD8+rSXD4KxXWH1MXjyeXriUGs7F/pA7YzC6fgb6SWxRSM4P91S ef35Dcc40VtJWPcpyU7kuFHUgFs1+vfnXUyJACWvF/RxL+eh7ozgnFFAoEBfAUw/cvcr VRbYadJjSral2xvxTXw9+txQsRhBCxgHfcA+EL7La1s9cOBOVBLBVjp2j1m9LzEV8PbG rna8FNdQcISbv1vkgXxvMJlDYkfjs6FQ4Hhy8a3HJ7mYexJ/2sCKhK2TMYV5yaSbvsY7 Zf9W7lFHDuFK65lrtRm0H/Cq3nud6YPAL/rmAAFGE9sy0CH9NvFQrQPOPBXRcwIGrmby 8COw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=r5fJWceke6VA6XqCta8sbDWJd2+freUpXAQOaYo1tYI=; b=l50zukjBv6ukotyntQUGD2ZKZaQwPVrGAcvRVRNTsIYXqCGT63OT1IwpnLnHX+tHSd 30q2L/c+o/cKNpp/J/yGZKQlJWNF6gPs1aysu9bMxLXhX24fUTrZlJtkD3sLLe1pUVKi 5k208Iyh8nvxyc1etyktMtje3F3bC2Qy9aJSHhWI7CkHP1zLySNDC1TEgfwXpi+0Ak9a lxGsbS84QXyITQl1eTpi1sSczdQqhzVv5Au5Jobxnfcm0OpntVTemXUYFwqYW7oqHB88 6Kvez7/1cYcIQodgY8Ojl77qc0xSNW6poNfnFN7ANNcbGJJbNu7S8ql21i/HoKXCLvnK UZMw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga17.intel.com (mga17.intel.com. [192.55.52.151]) by mx.google.com with ESMTPS id d2-v6si18038840plh.387.2018.05.23.01.27.14 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 May 2018 01:27:14 -0700 (PDT) Received-SPF: pass (google.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) client-ip=192.55.52.151; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 23 May 2018 01:27:14 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.49,432,1520924400"; d="scan'208";a="57726216" Received: from yhuang6-ux31a.sh.intel.com ([10.239.197.97]) by fmsmga001.fm.intel.com with ESMTP; 23 May 2018 01:27:09 -0700 From: "Huang, Ying" To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , "Kirill A. Shutemov" , Andrea Arcangeli , Michal Hocko , Johannes Weiner , Shaohua Li , Hugh Dickins , Minchan Kim , Rik van Riel , Dave Hansen , Naoya Horiguchi , Zi Yan Subject: [PATCH -mm -V3 15/21] mm, THP, swap: Support to copy PMD swap mapping when fork() Date: Wed, 23 May 2018 16:26:19 +0800 Message-Id: <20180523082625.6897-16-ying.huang@intel.com> X-Mailer: git-send-email 2.16.1 In-Reply-To: <20180523082625.6897-1-ying.huang@intel.com> References: <20180523082625.6897-1-ying.huang@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Huang Ying During fork, the page table need to be copied from parent to child. A PMD swap mapping need to be copied too and the swap reference count need to be increased. When the huge swap cluster has been split already, we need to split the PMD swap mapping and fallback to PTE copying. When swap count continuation failed to allocate a page with GFP_ATOMIC, we need to unlock the spinlock and try again with GFP_KERNEL. Signed-off-by: "Huang, Ying" Cc: "Kirill A. Shutemov" Cc: Andrea Arcangeli Cc: Michal Hocko Cc: Johannes Weiner Cc: Shaohua Li Cc: Hugh Dickins Cc: Minchan Kim Cc: Rik van Riel Cc: Dave Hansen Cc: Naoya Horiguchi Cc: Zi Yan --- mm/huge_memory.c | 72 ++++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 57 insertions(+), 15 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c4eb7737b313..01fdd59fe6d4 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -941,6 +941,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, if (unlikely(!pgtable)) goto out; +retry: dst_ptl = pmd_lock(dst_mm, dst_pmd); src_ptl = pmd_lockptr(src_mm, src_pmd); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); @@ -948,26 +949,67 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, ret = -EAGAIN; pmd = *src_pmd; -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION if (unlikely(is_swap_pmd(pmd))) { swp_entry_t entry = pmd_to_swp_entry(pmd); - VM_BUG_ON(!is_pmd_migration_entry(pmd)); - if (is_write_migration_entry(entry)) { - make_migration_entry_read(&entry); - pmd = swp_entry_to_pmd(entry); - if (pmd_swp_soft_dirty(*src_pmd)) - pmd = pmd_swp_mksoft_dirty(pmd); - set_pmd_at(src_mm, addr, src_pmd, pmd); +#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION + if (is_migration_entry(entry)) { + if (is_write_migration_entry(entry)) { + make_migration_entry_read(&entry); + pmd = swp_entry_to_pmd(entry); + if (pmd_swp_soft_dirty(*src_pmd)) + pmd = pmd_swp_mksoft_dirty(pmd); + set_pmd_at(src_mm, addr, src_pmd, pmd); + } + add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); + mm_inc_nr_ptes(dst_mm); + pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); + set_pmd_at(dst_mm, addr, dst_pmd, pmd); + ret = 0; + goto out_unlock; } - add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); - mm_inc_nr_ptes(dst_mm); - pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); - set_pmd_at(dst_mm, addr, dst_pmd, pmd); - ret = 0; - goto out_unlock; - } #endif + if (thp_swap_supported() && !non_swap_entry(entry)) { + ret = swap_duplicate(&entry, true); + if (!ret) { + add_mm_counter(dst_mm, MM_SWAPENTS, + HPAGE_PMD_NR); + mm_inc_nr_ptes(dst_mm); + pgtable_trans_huge_deposit(dst_mm, dst_pmd, + pgtable); + set_pmd_at(dst_mm, addr, dst_pmd, pmd); + /* make sure dst_mm is on swapoff's mmlist. */ + if (unlikely(list_empty(&dst_mm->mmlist))) { + spin_lock(&mmlist_lock); + if (list_empty(&dst_mm->mmlist)) + list_add(&dst_mm->mmlist, + &src_mm->mmlist); + spin_unlock(&mmlist_lock); + } + } else if (ret == -ENOTDIR) { + /* + * The swap cluster has been split, split the + * pmd map now + */ + __split_huge_swap_pmd(vma, addr, src_pmd); + pte_free(dst_mm, pgtable); + } else if (ret == -ENOMEM) { + spin_unlock(src_ptl); + spin_unlock(dst_ptl); + ret = add_swap_count_continuation(entry, + GFP_KERNEL); + if (ret < 0) { + ret = -ENOMEM; + pte_free(dst_mm, pgtable); + goto out; + } + goto retry; + } else + VM_BUG_ON(1); + goto out_unlock; + } + VM_BUG_ON(1); + } if (unlikely(!pmd_trans_huge(pmd))) { pte_free(dst_mm, pgtable);