From patchwork Wed Sep 12 00:44:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 10596549 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 04CE76CB for ; Wed, 12 Sep 2018 00:45:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E7BA629AC1 for ; Wed, 12 Sep 2018 00:45:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DC12729AD2; Wed, 12 Sep 2018 00:45:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B49B029AC1 for ; Wed, 12 Sep 2018 00:45:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D1A3F8E0001; Tue, 11 Sep 2018 20:45:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C7A5D8E0012; Tue, 11 Sep 2018 20:45:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B4CD98E0001; Tue, 11 Sep 2018 20:45:14 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id 729E18E0001 for ; Tue, 11 Sep 2018 20:45:14 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id v195-v6so122036pgb.0 for ; Tue, 11 Sep 2018 17:45:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=iqcb6BcSEpiA1xXvEIvVxaqOr9xLhQaEiPiikqIif+w=; b=g/dvaxbVaVuYyUMp8ZAn1bMYTDuFbEDGXEWQ+7b4Uj8mDyhPPEOmOEVC5JRR6Aiw8/ /Ks7R5kEMSc6YUq5aWyZYMnmdiSsOBbDThVog2YaMnOCizYgU4lkRviVOLBGJkOyvuag yq1p4adAkSjPbqctxcgnd4g5zhUl+4JwLuQEzuOftMQ0ZkSNN60/T1trpmk3ly0OV0MK B1f2I+Bqd8xC/+oN8U2PnXrtKdu1c7uFj0mhJ1SteAp1ty7f2igvi/xZ6C2d+HMXZQkF EVEccftnc0amqyraEvCLOXbClmd7Oxa5lTabHt+Y0c0iLtijfOrsiiIbPxiy8ZLeJ8F4 xrdQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51DQZCIT9vhKFzB10NJYxmau7vDPXD8J0Jc73xWsntAGNgLz2hw3 YuJL1Tuup+mQOVVz/oMXhFDpzbknlHGWrghBTtztsF6Ek3tOkxgo44Oeh26C5pSfA9VBvy+RoUW zGMOirU3SsqzNqg5kIRFkuSLiMZazIHe5uIOwG9L8E4dmyl/j11VKej0wCcw91qHDTw== X-Received: by 2002:a63:7d48:: with SMTP id m8-v6mr31544535pgn.0.1536713114139; Tue, 11 Sep 2018 17:45:14 -0700 (PDT) X-Google-Smtp-Source: ANB0Vdakl1dLVh2aKs1NLsZyCfyog89xldDsiO/f77jyiyzw6E53TcIuBtV8Cn9fTh+pt1+1W6pO X-Received: by 2002:a63:7d48:: with SMTP id m8-v6mr31544500pgn.0.1536713113424; Tue, 11 Sep 2018 17:45:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536713113; cv=none; d=google.com; s=arc-20160816; b=BSh/7a/XCLgJHwMrK1pmuKgqFNxC0p8W2dov/VRZ6ZUJva71C2JrVp9EstI4KrEyPf G7LIn+5oebZ8I4oRYetVqplyJ1rjTcfKocHaZk4twYdofuI2+mlt3jH1bTD7N9SjOkDL 7f9BVLpzTz73uKFjKEHBg6fDuedRejckmbSQVaSzjEOMJLjjc5Ytv2Z87S2XlHXUdkkp 0EnsRVw8Cn45Zt5FZqjc7lCgjslO2fDoJWd24McAI0jGDwAU0zs+nxu8qSWVJBRsMQfE IdX3uIXILZCsQtX9H5s6YsE5+4bO7ievduoGG3zvWEt5rktAvvyRpkMIMMzniKAeVYTN yRIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=iqcb6BcSEpiA1xXvEIvVxaqOr9xLhQaEiPiikqIif+w=; b=lGC9vET7Q/6PIMOs+c9EDKpId3XYPf1p6pgOIUQZeamiGB6dm9W2Nr5SHJw5R5Jds7 h8tFJWJndmsz+y0CLkKfw082cr2Z7wmXtWH5ZQr9orFHo1EWz61AajIEBn7fYGo5tZbB aPl4or7B2aeL7iYura8N8+HUh2lHJknyE4l9p1JCjqTVJVsk9BRbiSXVc4EF5eSy9AC3 enGHqwNphdyEhkCcwM35Y0awDWgT0Coze1TyrDNrjeD+h21ntMkRqdgP3kCCXu6xQ1Aw +hHWLZ7c/haBe/1Qxz3r85dqjcwYCxehEXjdEOAFlOb+2vLLdjcvyp2sml9z1QbyyhSb R5kA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id e8-v6si22134418pgl.498.2018.09.11.17.45.13 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Sep 2018 17:45:13 -0700 (PDT) Received-SPF: pass (google.com: domain of ying.huang@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Sep 2018 17:45:13 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,362,1531810800"; d="scan'208";a="69283993" Received: from unknown (HELO yhuang-mobile.sh.intel.com) ([10.239.198.87]) by fmsmga007.fm.intel.com with ESMTP; 11 Sep 2018 17:45:08 -0700 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , "Kirill A. Shutemov" , Andrea Arcangeli , Michal Hocko , Johannes Weiner , Shaohua Li , Hugh Dickins , Minchan Kim , Rik van Riel , Dave Hansen , Naoya Horiguchi , Zi Yan , Daniel Jordan Subject: [PATCH -V5 RESEND 15/21] swap: Support to copy PMD swap mapping when fork() Date: Wed, 12 Sep 2018 08:44:08 +0800 Message-Id: <20180912004414.22583-16-ying.huang@intel.com> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20180912004414.22583-1-ying.huang@intel.com> References: <20180912004414.22583-1-ying.huang@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP During fork, the page table need to be copied from parent to child. A PMD swap mapping need to be copied too and the swap reference count need to be increased. When the huge swap cluster has been split already, we need to split the PMD swap mapping and fallback to PTE copying. When swap count continuation failed to allocate a page with GFP_ATOMIC, we need to unlock the spinlock and try again with GFP_KERNEL. Signed-off-by: "Huang, Ying" Cc: "Kirill A. Shutemov" Cc: Andrea Arcangeli Cc: Michal Hocko Cc: Johannes Weiner Cc: Shaohua Li Cc: Hugh Dickins Cc: Minchan Kim Cc: Rik van Riel Cc: Dave Hansen Cc: Naoya Horiguchi Cc: Zi Yan Cc: Daniel Jordan --- mm/huge_memory.c | 72 ++++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 57 insertions(+), 15 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index f98d8a543d73..4e2230583c53 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -941,6 +941,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, if (unlikely(!pgtable)) goto out; +retry: dst_ptl = pmd_lock(dst_mm, dst_pmd); src_ptl = pmd_lockptr(src_mm, src_pmd); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); @@ -948,26 +949,67 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, ret = -EAGAIN; pmd = *src_pmd; -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION if (unlikely(is_swap_pmd(pmd))) { swp_entry_t entry = pmd_to_swp_entry(pmd); - VM_BUG_ON(!is_pmd_migration_entry(pmd)); - if (is_write_migration_entry(entry)) { - make_migration_entry_read(&entry); - pmd = swp_entry_to_pmd(entry); - if (pmd_swp_soft_dirty(*src_pmd)) - pmd = pmd_swp_mksoft_dirty(pmd); - set_pmd_at(src_mm, addr, src_pmd, pmd); +#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION + if (is_migration_entry(entry)) { + if (is_write_migration_entry(entry)) { + make_migration_entry_read(&entry); + pmd = swp_entry_to_pmd(entry); + if (pmd_swp_soft_dirty(*src_pmd)) + pmd = pmd_swp_mksoft_dirty(pmd); + set_pmd_at(src_mm, addr, src_pmd, pmd); + } + add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); + mm_inc_nr_ptes(dst_mm); + pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); + set_pmd_at(dst_mm, addr, dst_pmd, pmd); + ret = 0; + goto out_unlock; } - add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); - mm_inc_nr_ptes(dst_mm); - pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); - set_pmd_at(dst_mm, addr, dst_pmd, pmd); - ret = 0; - goto out_unlock; - } #endif + if (IS_ENABLED(CONFIG_THP_SWAP) && !non_swap_entry(entry)) { + ret = swap_duplicate(&entry, HPAGE_PMD_NR); + if (!ret) { + add_mm_counter(dst_mm, MM_SWAPENTS, + HPAGE_PMD_NR); + mm_inc_nr_ptes(dst_mm); + pgtable_trans_huge_deposit(dst_mm, dst_pmd, + pgtable); + set_pmd_at(dst_mm, addr, dst_pmd, pmd); + /* make sure dst_mm is on swapoff's mmlist. */ + if (unlikely(list_empty(&dst_mm->mmlist))) { + spin_lock(&mmlist_lock); + if (list_empty(&dst_mm->mmlist)) + list_add(&dst_mm->mmlist, + &src_mm->mmlist); + spin_unlock(&mmlist_lock); + } + } else if (ret == -ENOTDIR) { + /* + * The huge swap cluster has been split, split + * the PMD swap mapping and fallback to PTE + */ + __split_huge_swap_pmd(vma, addr, src_pmd); + pte_free(dst_mm, pgtable); + } else if (ret == -ENOMEM) { + spin_unlock(src_ptl); + spin_unlock(dst_ptl); + ret = add_swap_count_continuation(entry, + GFP_KERNEL); + if (ret < 0) { + ret = -ENOMEM; + pte_free(dst_mm, pgtable); + goto out; + } + goto retry; + } else + VM_BUG_ON(1); + goto out_unlock; + } + VM_BUG_ON(1); + } if (unlikely(!pmd_trans_huge(pmd))) { pte_free(dst_mm, pgtable);