From patchwork Mon May 24 09:01:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 12275721 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 407F8C04FF3 for ; Mon, 24 May 2021 09:02:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DC47F60FDA for ; Mon, 24 May 2021 09:02:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DC47F60FDA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 16699940059; Mon, 24 May 2021 05:02:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 12765940055; Mon, 24 May 2021 05:02:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B00D0940059; Mon, 24 May 2021 05:02:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0063.hostedemail.com [216.40.44.63]) by kanga.kvack.org (Postfix) with ESMTP id 6FD1C940055 for ; Mon, 24 May 2021 05:02:06 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 01FD56C3B for ; Mon, 24 May 2021 09:02:06 +0000 (UTC) X-FDA: 78175532652.25.EA7AA16 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf22.hostedemail.com (Postfix) with ESMTP id 8B71FC0042D9 for ; Mon, 24 May 2021 09:02:00 +0000 (UTC) Received: from pps.filterd (m0187473.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 14O8v61u023666; Mon, 24 May 2021 05:01:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : mime-version; s=pp1; bh=7eX2gy/zuhS8OjiDxr2VcenOd74UiuXCuRJmJCTOZVs=; b=VEgEPFVYUm9YUeHa31Vq6bMIXPd/xHDfCVyw2NAwUGtmng3R21v0p8Uc2ehYdIayIC3o ZUwmoE8PqvLkdjsMDlAz3OvkLQSobj5Wk7TqvxJZ6aNPyR/T0Rdqc+fKqVXlEfTi+tl0 r+9NdkAaGacTnisGTcbL+eA8qgVG4a+iR+TpP+JzNwQiUvaTE3HtrHE8w+Gkua9L1CSm NlTDTykjCOfC5lYnb190mLMPitdPlcbFjOoiwfZCNKwP5RkUQ88OA10fXhlBeZiOtstU GwdTgRaxVASMH/5e+bIqqPWC9Z1UzfL4rBZmH/4ZMEw9bNklSKOiHrClzPUmZRSuRAJy mg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 38r93803yb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 May 2021 05:01:56 -0400 Received: from m0187473.ppops.net (m0187473.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 14O8wi3g027225; Mon, 24 May 2021 05:01:56 -0400 Received: from ppma04dal.us.ibm.com (7a.29.35a9.ip4.static.sl-reverse.com [169.53.41.122]) by mx0a-001b2d01.pphosted.com with ESMTP id 38r93803xm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 May 2021 05:01:56 -0400 Received: from pps.filterd (ppma04dal.us.ibm.com [127.0.0.1]) by ppma04dal.us.ibm.com (8.16.0.43/8.16.0.43) with SMTP id 14O8nPrF028826; Mon, 24 May 2021 09:01:55 GMT Received: from b01cxnp22035.gho.pok.ibm.com (b01cxnp22035.gho.pok.ibm.com [9.57.198.25]) by ppma04dal.us.ibm.com with ESMTP id 38psk8s76a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 May 2021 09:01:55 +0000 Received: from b01ledav005.gho.pok.ibm.com (b01ledav005.gho.pok.ibm.com [9.57.199.110]) by b01cxnp22035.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 14O91sh636045212 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 24 May 2021 09:01:54 GMT Received: from b01ledav005.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A366FAE05C; Mon, 24 May 2021 09:01:54 +0000 (GMT) Received: from b01ledav005.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C52A5AE060; Mon, 24 May 2021 09:01:51 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.102.1.240]) by b01ledav005.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 24 May 2021 09:01:51 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, kaleshsingh@google.com, npiggin@gmail.com, joel@joelfernandes.org, Christophe Leroy , Linus Torvalds , "Aneesh Kumar K.V" Subject: [PATCH v6 08/11] mm/mremap: properly flush the TLB on mremap. Date: Mon, 24 May 2021 14:31:11 +0530 Message-Id: <20210524090114.63446-9-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210524090114.63446-1-aneesh.kumar@linux.ibm.com> References: <20210524090114.63446-1-aneesh.kumar@linux.ibm.com> X-TM-AS-GCONF: 00 X-Proofpoint-GUID: E9Jsi4cLb3qfzd726OVY4c0xb9TlqyVJ X-Proofpoint-ORIG-GUID: 5g1Qlm1no-PwpUDg4QhE5RZcILplYtPo X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.761 definitions=2021-05-24_04:2021-05-20,2021-05-24 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 clxscore=1015 suspectscore=0 adultscore=0 lowpriorityscore=0 bulkscore=0 mlxscore=0 priorityscore=1501 mlxlogscore=999 impostorscore=0 spamscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2105240067 X-Rspamd-Queue-Id: 8B71FC0042D9 Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=VEgEPFVY; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf22.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com X-Rspamd-Server: rspam03 X-Stat-Signature: xmsgkbht8m4hcxuh884oft4qpgxou13t X-HE-Tag: 1621846920-866106 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As explained in commit eb66ae030829 ("mremap: properly flush TLB before releasing the page") mremap is special in that it doesn't take ownership of the page. The optimized version for PUD/PMD aligned mremap also doesn't hold the ptl lock. Hence flush the TLB before we update the new page table location. This ensures the kernel invalidates the older translation cache before it can free the page via the newly inserted translation. Fixes: c49dd3401802 ("mm: speedup mremap on 1GB or larger regions") Fixes: 2c91bd4a4e2e ("mm: speed up mremap by 20x on large regions") Link: https://lore.kernel.org/linux-mm/CAHk-=wjq8thag3uNv-2MMu75OgX5ybMon7gZDUHYwzeTwcZHoA@mail.gmail.com Signed-off-by: Aneesh Kumar K.V --- mm/mremap.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/mm/mremap.c b/mm/mremap.c index 000a71917557..8967a3707332 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -266,11 +266,13 @@ static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, /* Clear the pmd */ pmd = *old_pmd; pmd_clear(old_pmd); - + /* + * flush the TLB before we move the page table entries. + */ + flush_pte_tlb_pwc_range(vma, old_addr, old_addr + PMD_SIZE); VM_BUG_ON(!pmd_none(*new_pmd)); pmd_populate(mm, new_pmd, pmd_pgtable(pmd)); - flush_pte_tlb_pwc_range(vma, old_addr, old_addr + PMD_SIZE); if (new_ptl != old_ptl) spin_unlock(new_ptl); spin_unlock(old_ptl); @@ -313,11 +315,14 @@ static bool move_normal_pud(struct vm_area_struct *vma, unsigned long old_addr, /* Clear the pud */ pud = *old_pud; pud_clear(old_pud); + /* + * flush the TLB before we move the page table entries. + */ + flush_pte_tlb_pwc_range(vma, old_addr, old_addr + PUD_SIZE); VM_BUG_ON(!pud_none(*new_pud)); pud_populate(mm, new_pud, (pmd_t *)pud_page_vaddr(pud)); - flush_pte_tlb_pwc_range(vma, old_addr, old_addr + PUD_SIZE); if (new_ptl != old_ptl) spin_unlock(new_ptl); spin_unlock(old_ptl);