From patchwork Wed Jan 27 23:53:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 12051279 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28746C433DB for ; Wed, 27 Jan 2021 23:54:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D1F3064DCE for ; Wed, 27 Jan 2021 23:54:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D1F3064DCE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 776F66B0070; Wed, 27 Jan 2021 18:54:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7045B6B0071; Wed, 27 Jan 2021 18:54:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5A0F86B0072; Wed, 27 Jan 2021 18:54:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0062.hostedemail.com [216.40.44.62]) by kanga.kvack.org (Postfix) with ESMTP id 41C8C6B0070 for ; Wed, 27 Jan 2021 18:54:15 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 0D6F2181AEF21 for ; Wed, 27 Jan 2021 23:54:15 +0000 (UTC) X-FDA: 77753211270.22.owner41_140e4a62759b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin22.hostedemail.com (Postfix) with ESMTP id EB8471803E48D for ; Wed, 27 Jan 2021 23:54:14 +0000 (UTC) X-HE-Tag: owner41_140e4a62759b X-Filterd-Recvd-Size: 5852 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Wed, 27 Jan 2021 23:54:14 +0000 (UTC) Received: by mail.kernel.org (Postfix) with ESMTPSA id 50E8F64DD7; Wed, 27 Jan 2021 23:54:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1611791653; bh=qKlq1koZlj44TavSerJXZc2iPqP23OH65idsqi/sK5Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lKljCK8hhOFol+aCFdIASHc6u2ky3vKHJ5g63uIUxW1GEL1DDtig0WgcH7PFXAg4T CQfw98Y+3nc3zRKDQ0zuFCgAxfrM0BSjhD509FrDyM6ee1Hl5oQmJXvNRSxolVq0GF pJipcdH4kIcP4+9kZjc+ugTTl1+035Thn9W5b7zee82eOiqFfrRh0U9UeiPF8n3rK8 QaKmaI+B+HEIdMASO0FHzYfV4Rw+fICnP5Yf1ViTvqFoW5RORrfbOMgaeLYyU/0I1k RdxPaWaHuiX7TUU8aIKoOQD+TigHCA6SUSq4Tb6hSOE7i5bPLZabLF3CBIhAbLCRYL EeB6zLgZgnEcw== From: Will Deacon To: linux-kernel@vger.kernel.org Cc: kernel-team@android.com, linux-mm@kvack.org, Will Deacon , Yu Zhao , Minchan Kim , Peter Zijlstra , Thomas Gleixner , Linus Torvalds , Vlastimil Babka , Mohamed Alzayat , "Aneesh Kumar K.V" , Nadav Amit , Andrea Arcangeli Subject: [PATCH v3 3/6] tlb: mmu_gather: Introduce tlb_gather_mmu_fullmm() Date: Wed, 27 Jan 2021 23:53:44 +0000 Message-Id: <20210127235347.1402-4-will@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210127235347.1402-1-will@kernel.org> References: <20210127235347.1402-1-will@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Passing the range '0, -1' to tlb_gather_mmu() sets the 'fullmm' flag, which indicates that the mm_struct being operated on is going away. In this case, some architectures (such as arm64) can elide TLB invalidation by ensuring that the TLB tag (ASID) associated with this mm is not immediately reclaimed. Although this behaviour is documented in asm-generic/tlb.h, it's subtle and easily missed. Introduce tlb_gather_mmu_fullmm() to make it clearer that this is for the entire mm and WARN() if tlb_gather_mmu() is called with the 'fullmm' address range. Acked-by: Peter Zijlstra (Intel) Reviewed-by: Yu Zhao Signed-off-by: Will Deacon --- include/asm-generic/tlb.h | 6 ++++-- include/linux/mm_types.h | 1 + mm/mmap.c | 2 +- mm/mmu_gather.c | 16 ++++++++++++++-- 4 files changed, 20 insertions(+), 5 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 6661ee1cff47..2c68a545ffa7 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -46,7 +46,9 @@ * * The mmu_gather API consists of: * - * - tlb_gather_mmu() / tlb_finish_mmu(); start and finish a mmu_gather + * - tlb_gather_mmu() / tlb_gather_mmu_fullmm() / tlb_finish_mmu() + * + * start and finish a mmu_gather * * Finish in particular will issue a (final) TLB invalidate and free * all (remaining) queued pages. @@ -91,7 +93,7 @@ * * - mmu_gather::fullmm * - * A flag set by tlb_gather_mmu() to indicate we're going to free + * A flag set by tlb_gather_mmu_fullmm() to indicate we're going to free * the entire mm; this allows a number of optimizations. * * - We can ignore tlb_{start,end}_vma(); because we don't diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 1fe6a51298a6..e49868bc12a7 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -590,6 +590,7 @@ static inline cpumask_t *mm_cpumask(struct mm_struct *mm) struct mmu_gather; extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm, unsigned long start, unsigned long end); +extern void tlb_gather_mmu_fullmm(struct mmu_gather *tlb, struct mm_struct *mm); extern void tlb_finish_mmu(struct mmu_gather *tlb); static inline void init_tlb_flush_pending(struct mm_struct *mm) diff --git a/mm/mmap.c b/mm/mmap.c index 7a9f493a4b83..4eac7c63edbe 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3214,7 +3214,7 @@ void exit_mmap(struct mm_struct *mm) lru_add_drain(); flush_cache_mm(mm); - tlb_gather_mmu(&tlb, mm, 0, -1); + tlb_gather_mmu_fullmm(&tlb, mm); /* update_hiwater_rss(mm) here? but nobody should be looking */ /* Use -1 here to ensure all VMAs in the mm are unmapped */ unmap_vmas(&tlb, vma, 0, -1); diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index b0be5a7aa08f..5f5e45d9eb50 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -261,8 +261,8 @@ void tlb_flush_mmu(struct mmu_gather *tlb) * respectively when @mm is without users and we're going to destroy * the full address space (exit/execve). */ -void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm, - unsigned long start, unsigned long end) +static void __tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm, + unsigned long start, unsigned long end) { tlb->mm = mm; @@ -287,6 +287,18 @@ void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm, inc_tlb_flush_pending(tlb->mm); } +void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm, + unsigned long start, unsigned long end) +{ + WARN_ON(!(start | (end + 1))); /* Use _fullmm() instead */ + __tlb_gather_mmu(tlb, mm, start, end); +} + +void tlb_gather_mmu_fullmm(struct mmu_gather *tlb, struct mm_struct *mm) +{ + __tlb_gather_mmu(tlb, mm, 0, -1); +} + /** * tlb_finish_mmu - finish an mmu_gather structure * @tlb: the mmu_gather structure to finish