From patchwork Thu Apr 22 05:43:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 12217689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29B33C433ED for ; Thu, 22 Apr 2021 05:44:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BC2616140F for ; Thu, 22 Apr 2021 05:44:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BC2616140F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4F2326B0075; Thu, 22 Apr 2021 01:44:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4A0CB6B0078; Thu, 22 Apr 2021 01:44:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 233876B007B; Thu, 22 Apr 2021 01:44:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0123.hostedemail.com [216.40.44.123]) by kanga.kvack.org (Postfix) with ESMTP id 022386B0075 for ; Thu, 22 Apr 2021 01:44:12 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id A8E1BA2A6 for ; Thu, 22 Apr 2021 05:44:12 +0000 (UTC) X-FDA: 78058912344.27.64A28D6 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf26.hostedemail.com (Postfix) with ESMTP id A2F4740002C3 for ; Thu, 22 Apr 2021 05:44:05 +0000 (UTC) Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 13M5YKSw099319; Thu, 22 Apr 2021 01:44:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=h6cvove2QbbYLRFIT5pj9eXqp3dfloayLyKMJ6xNrKM=; b=d92YZDx50HJMamXl9i4qNKqF2NC3nb8Q4+BQN16qLUclPAzMPDMPV1h2wKo7Ew9egZwD X4GLDD6wl6eCVL3bO9bBlVjQ7cwyq7QMirkKScA9tQikV3fK3JboIZY4aKIMtkkPOp5h 1FAtmOOz2Jf6H4BSnGJ+7pjdw22220ufVmrUHamXCuQyunjJd8DgBFEiFI8DOEEQAtXJ WuxsIRyp/+3hNgn7F4kbnXzEee/Wy2SedUbEMYaEwlkYTqO0wOyLI3goY6rijYNTyoAW SquueDX2T98l+OFestK9oCVIrjPP6R21bfiomyfbGGVC7uqXy0OUrUuLCLVV10AsjwlD Yw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 382xvenknm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Apr 2021 01:44:05 -0400 Received: from m0098396.ppops.net (m0098396.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 13M5YenA100281; Thu, 22 Apr 2021 01:44:04 -0400 Received: from ppma03wdc.us.ibm.com (ba.79.3fa9.ip4.static.sl-reverse.com [169.63.121.186]) by mx0a-001b2d01.pphosted.com with ESMTP id 382xvenkmb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Apr 2021 01:44:04 -0400 Received: from pps.filterd (ppma03wdc.us.ibm.com [127.0.0.1]) by ppma03wdc.us.ibm.com (8.16.0.43/8.16.0.43) with SMTP id 13M5ghfE021411; Thu, 22 Apr 2021 05:44:03 GMT Received: from b03cxnp08028.gho.boulder.ibm.com (b03cxnp08028.gho.boulder.ibm.com [9.17.130.20]) by ppma03wdc.us.ibm.com with ESMTP id 37yqa9fna2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Apr 2021 05:44:03 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp08028.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 13M5i29o21496230 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 22 Apr 2021 05:44:02 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A160378064; Thu, 22 Apr 2021 05:44:02 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 99C5F7805C; Thu, 22 Apr 2021 05:43:59 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.199.37.130]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Thu, 22 Apr 2021 05:43:59 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, kaleshsingh@google.com, npiggin@gmail.com, joel@joelfernandes.org, Christophe Leroy , "Aneesh Kumar K.V" Subject: [PATCH v5 7/9] mm/mremap: Move TLB flush outside page table lock Date: Thu, 22 Apr 2021 11:13:21 +0530 Message-Id: <20210422054323.150993-8-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210422054323.150993-1-aneesh.kumar@linux.ibm.com> References: <20210422054323.150993-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: kxKDNx-VgCd7FdPF0DJZjyumBzgeJqh8 X-Proofpoint-GUID: Ps2uTNLFbI7ur_Co88181xFlqNhxweJQ X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.761 definitions=2021-04-22_01:2021-04-21,2021-04-21 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 priorityscore=1501 spamscore=0 bulkscore=0 lowpriorityscore=0 mlxlogscore=999 impostorscore=0 mlxscore=0 clxscore=1015 adultscore=0 phishscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104060000 definitions=main-2104220047 X-Rspamd-Queue-Id: A2F4740002C3 X-Stat-Signature: ffagozpeebhirn9math5rujud78mw61n X-Rspamd-Server: rspam02 Received-SPF: none (linux.ibm.com>: No applicable sender policy available) receiver=imf26; identity=mailfrom; envelope-from=""; helo=mx0a-001b2d01.pphosted.com; client-ip=148.163.156.1 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619070245-164325 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move TLB flush outside page table lock so that kernel does less with page table lock held. Releasing the ptl with old TLB contents still valid will behave such that such access happened before the level3 or level2 entry update. Signed-off-by: Aneesh Kumar K.V --- mm/mremap.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/mm/mremap.c b/mm/mremap.c index 109560977944..9effca76bf17 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -258,7 +258,7 @@ static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, * We don't have to worry about the ordering of src and dst * ptlocks because exclusive mmap_lock prevents deadlock. */ - old_ptl = pmd_lock(vma->vm_mm, old_pmd); + old_ptl = pmd_lock(mm, old_pmd); new_ptl = pmd_lockptr(mm, new_pmd); if (new_ptl != old_ptl) spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); @@ -270,11 +270,11 @@ static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, VM_BUG_ON(!pmd_none(*new_pmd)); pmd_populate(mm, new_pmd, (pgtable_t)pmd_page_vaddr(pmd)); - flush_pte_tlb_pwc_range(vma, old_addr, old_addr + PMD_SIZE); if (new_ptl != old_ptl) spin_unlock(new_ptl); spin_unlock(old_ptl); + flush_pte_tlb_pwc_range(vma, old_addr, old_addr + PMD_SIZE); return true; } #else @@ -305,7 +305,7 @@ static bool move_normal_pud(struct vm_area_struct *vma, unsigned long old_addr, * We don't have to worry about the ordering of src and dst * ptlocks because exclusive mmap_lock prevents deadlock. */ - old_ptl = pud_lock(vma->vm_mm, old_pud); + old_ptl = pud_lock(mm, old_pud); new_ptl = pud_lockptr(mm, new_pud); if (new_ptl != old_ptl) spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); @@ -317,11 +317,11 @@ static bool move_normal_pud(struct vm_area_struct *vma, unsigned long old_addr, VM_BUG_ON(!pud_none(*new_pud)); pud_populate(mm, new_pud, (pmd_t *)pud_page_vaddr(pud)); - flush_pte_tlb_pwc_range(vma, old_addr, old_addr + PUD_SIZE); if (new_ptl != old_ptl) spin_unlock(new_ptl); spin_unlock(old_ptl); + flush_pte_tlb_pwc_range(vma, old_addr, old_addr + PUD_SIZE); return true; } #else