From patchwork Mon Aug 13 03:41:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 10563867 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 87E3114E2 for ; Mon, 13 Aug 2018 03:41:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6380228E5D for ; Mon, 13 Aug 2018 03:41:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5638C290AE; Mon, 13 Aug 2018 03:41:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1621728E5D for ; Mon, 13 Aug 2018 03:41:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AAA436B0005; Sun, 12 Aug 2018 23:41:40 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A5B396B0006; Sun, 12 Aug 2018 23:41:40 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 96F486B0007; Sun, 12 Aug 2018 23:41:40 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk0-f199.google.com (mail-qk0-f199.google.com [209.85.220.199]) by kanga.kvack.org (Postfix) with ESMTP id 6BF256B0005 for ; Sun, 12 Aug 2018 23:41:40 -0400 (EDT) Received: by mail-qk0-f199.google.com with SMTP id x204-v6so15866104qka.6 for ; Sun, 12 Aug 2018 20:41:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id; bh=Pilx790CvHyY7hwm6VSZj46Ygh+pR4v37iu/tSwUVWM=; b=GzAlACGm0sVhxISzeRofgyUAx3OKheTrU7lmMDAjbRSat8dtS2T1pIJAbNmbOrnJzq VVBQ/790Y+/yyl72/38KOlrhN/qRWKRG9mmaZ0+vtWxySM7zQiIU1PgiQEOUekZlWCs6 xe9P87Su4eHRS77fakQ48pQ/GJzcvRYxltnp2Xonkhx/wCpoaOWC7UjcZB9V/O+x2ZiJ xifEHQC+TMnLrrM42o+bEHiIikZcnzxJ6A7SmbWfdIX6ruPY6Oe1lBVkgkHUGczGfis7 wN2ebk38vvriWamt/CqI2mWTqy35iaDw60X8FVRrOE5d4DV4wJdNDbaNRWYAUmN7mL8R ZS8w== X-Gm-Message-State: AOUpUlEcbodJRqEsOZ9VII1ODpK4WR/qzvVM7qyDbiiDCrAW0UMrsLO2 G8Lwv9y2puIBE62fDrdZzYGQXQD7kJQzDRpE/NcqJYqQvAnwRFRZJ83wIpA04UJR2Ypj2FOHyrM 1ecer66vXhzLtyELKt2Zxyf2PRC2ICheRsnwafpK7xgcRXbhtwPixGO2VWCVGt7rfUg== X-Received: by 2002:a37:39c7:: with SMTP id g190-v6mr12628499qka.66.1534131700141; Sun, 12 Aug 2018 20:41:40 -0700 (PDT) X-Google-Smtp-Source: AA+uWPxVUBYfo35H9mZpm5M3Vf+vgSmitSSdXELZ0ERU/DGrbSpD5VAaqva2f4KudJOer0lNxMoO X-Received: by 2002:a37:39c7:: with SMTP id g190-v6mr12628485qka.66.1534131699437; Sun, 12 Aug 2018 20:41:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1534131699; cv=none; d=google.com; s=arc-20160816; b=rA3EBWFFrsppZP7Mj+D8aSTMDnZrJ+D/0sYVtFn4dyw5pmswCHgSNseyh0Ng0md057 u2sRt1PpKOCaXqhlfC73tRCCbmJE8Mgm7aseeMkSAWwHh20c/DFi2JxrfM6mSfEES6YR 0QuO/WQ/9YLzLdsVamkG1pfq3fqfcG0LdEziH01o6U/GnVzc0dLdwEd0Ak7Seb6ihGbW wC1tOMPwPIce4gopKJ+w1DjHnaBlEZSGuNBBqrLDLjM7yt7DQ0PRsarovXS7cqeQAEB8 mj25Sv4gejdL2h/StnaM1nWbjnK7tC+hSLJrIgZTibHmQ8Gr14YYODKJTu50WMB/EBbt rr0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:date:subject:cc:to:from:dkim-signature :arc-authentication-results; bh=Pilx790CvHyY7hwm6VSZj46Ygh+pR4v37iu/tSwUVWM=; b=gUYU+WU/xDjQ54Vhk4I5LymjR+1SSBlgnwyj+nPcjD5LIrwsIiMXoVxb6gDYn8xqtf k4c/Z+R6U0bvda/p24qjd6Al5gxuy6uO9nCAgx3uPPUCp3yfCfpXznjcbcl3/e6GOkaz tDeRbTqaT1oaQZ8YhuxtTlmNqUGcpS6UDfrNsaZU62bqpDX5cnWWE01FhOutgrsrX74M 8eGy2kgCjduqDJuGYydtLG5CddwArNVBHoF2fVcTQC3Ivh1u035WdSxZywXGkszPDEc1 3xzHxzKwWnsddCgN2G81UzAH4XcnD3IyZBfcb94GEfpyhJop0akvpDKoqpq5AkuU5PEQ 87uw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=k0oSYYzC; spf=pass (google.com: domain of mike.kravetz@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id k1-v6si3268942qkc.104.2018.08.12.20.41.38 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 12 Aug 2018 20:41:39 -0700 (PDT) Received-SPF: pass (google.com: domain of mike.kravetz@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=k0oSYYzC; spf=pass (google.com: domain of mike.kravetz@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id w7D3e6b2124913; Mon, 13 Aug 2018 03:41:32 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id; s=corp-2018-07-02; bh=Pilx790CvHyY7hwm6VSZj46Ygh+pR4v37iu/tSwUVWM=; b=k0oSYYzCFLPacFtJXZ9vG+b8eXDFZ3H4VVA6sp1GRenua7t4uM1x2Qv0hopqpiFk9goP tKAUJsCeFGkixExLZs6J6mwFZ5zfp58IeJGtE+tiTheH2UUHymlyiqrRI6wk6I1hz2PF zBa/V6BgJEzzjaGDlDSLRxQL0q61C7bqcrX98cu+RFHJFSgZopu5V6QB9gVYTIJ/NvI9 zc2iNbssY61u1DEywil34z6D7e7zAgdf/GSv6Zx18IzFXPLYePQRZGkM+vlAKG41tbqX Gvn/TtlYf4xW869MozDSy2ii9P85EjHoEOJqRHBJv6tCMLtfKjUL6U6VzAv2eSEUwlvH fw== Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp2130.oracle.com with ESMTP id 2ksq7t2g3n-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 Aug 2018 03:41:31 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id w7D3fUOa024950 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 Aug 2018 03:41:30 GMT Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id w7D3fThs012107; Mon, 13 Aug 2018 03:41:29 GMT Received: from monkey.oracle.com (/50.38.38.67) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 13 Aug 2018 03:41:28 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: "Kirill A . Shutemov" , =?utf-8?b?SsOp?= =?utf-8?b?csO0bWUgR2xpc3Nl?= , Vlastimil Babka , Naoya Horiguchi , Davidlohr Bueso , Michal Hocko , Andrew Morton , Mike Kravetz Subject: [PATCH] mm: migration: fix migration of huge PMD shared pages Date: Sun, 12 Aug 2018 20:41:08 -0700 Message-Id: <20180813034108.27269-1-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.17.1 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=8983 signatures=668707 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=913 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1808130039 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The page migration code employs try_to_unmap() to try and unmap the source page. This is accomplished by using rmap_walk to find all vmas where the page is mapped. This search stops when page mapcount is zero. For shared PMD huge pages, the page map count is always 1 not matter the number of mappings. Shared mappings are tracked via the reference count of the PMD page. Therefore, try_to_unmap stops prematurely and does not completely unmap all mappings of the source page. This problem can result is data corruption as writes to the original source page can happen after contents of the page are copied to the target page. Hence, data is lost. This problem was originally seen as DB corruption of shared global areas after a huge page was soft offlined. DB developers noticed they could reproduce the issue by (hotplug) offlining memory used to back huge pages. A simple testcase can reproduce the problem by creating a shared PMD mapping (note that this must be at least PUD_SIZE in size and PUD_SIZE aligned (1GB on x86)), and using migrate_pages() to migrate process pages between nodes. To fix, have the try_to_unmap_one routine check for huge PMD sharing by calling huge_pmd_unshare for hugetlbfs huge pages. If it is a shared mapping it will be 'unshared' which removes the page table entry and drops reference on PMD page. After this, flush caches and TLB. Signed-off-by: Mike Kravetz --- I am not %100 sure on the required flushing, so suggestions would be appreciated. This also should go to stable. It has been around for a long time so still looking for an appropriate 'fixes:'. mm/rmap.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/mm/rmap.c b/mm/rmap.c index 09a799c9aebd..45583758bf16 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1409,6 +1409,27 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, subpage = page - page_to_pfn(page) + pte_pfn(*pvmw.pte); address = pvmw.address; + /* + * PMDs for hugetlbfs pages could be shared. In this case, + * pages with shared PMDs will have a mapcount of 1 no matter + * how many times it is actually mapped. Map counting for + * PMD sharing is mostly done via the reference count on the + * PMD page itself. If the page we are trying to unmap is a + * hugetlbfs page, attempt to 'unshare' at the PMD level. + * huge_pmd_unshare takes care of clearing the PUD and + * reference counting on the PMD page which effectively unmaps + * the page. Take care of flushing cache and TLB for page in + * this specific mapping here. + */ + if (PageHuge(page) && + huge_pmd_unshare(mm, &address, pvmw.pte)) { + unsigned long end_add = address + vma_mmu_pagesize(vma); + + flush_cache_range(vma, address, end_add); + flush_tlb_range(vma, address, end_add); + mmu_notifier_invalidate_range(mm, address, end_add); + continue; + } if (IS_ENABLED(CONFIG_MIGRATION) && (flags & TTU_MIGRATION) &&