From patchwork Sun Jan 31 00:11:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12057455 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B715C433DB for ; Sun, 31 Jan 2021 00:16:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 037D864E19 for ; Sun, 31 Jan 2021 00:16:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 037D864E19 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E8B3E6B0074; Sat, 30 Jan 2021 19:16:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DCC786B0075; Sat, 30 Jan 2021 19:16:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BCCC56B0078; Sat, 30 Jan 2021 19:16:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0177.hostedemail.com [216.40.44.177]) by kanga.kvack.org (Postfix) with ESMTP id 95B416B0074 for ; Sat, 30 Jan 2021 19:16:15 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 5FCE4181AEF23 for ; Sun, 31 Jan 2021 00:16:15 +0000 (UTC) X-FDA: 77764153110.24.scene65_6312f89275b5 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin24.hostedemail.com (Postfix) with ESMTP id 3F95C1A4A5 for ; Sun, 31 Jan 2021 00:16:15 +0000 (UTC) X-HE-Tag: scene65_6312f89275b5 X-Filterd-Recvd-Size: 7717 Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by imf06.hostedemail.com (Postfix) with ESMTP for ; Sun, 31 Jan 2021 00:16:14 +0000 (UTC) Received: by mail-pf1-f173.google.com with SMTP id e19so9062033pfh.6 for ; Sat, 30 Jan 2021 16:16:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=B1n9KuHbkwDteeyX+EGvgbidSs5/e5CI67AfWMUmp8s=; b=SKZEOydA0mMWkDa5c+uNDDq0+qMOl+Zb6dqieV/jCTFS++mV7yMbTyfUAEFSfwFcL2 ya1tZMpINcor6UQYYqa5YSPVwsgJwmtDtXjfoVUxQhTlq3wC7rn7uhgnvpiTAijnzQ5J DOjVqicU91j+tmfyiWaGWEcL9Kj/vkOFJdbPV/ZE8hP51yiruPoTE9yTgfFXptPlh9ok Iahtt+cLVVjWY3zHdcUenrf4+tyHlaNTzuZ36PebMzYas0Xz1Pr4A/ILg/aDGBYou02R vR0q3WWSXwG2iv4f3C53S0tpyjBf3PsMIYTG7KBudtFNHH6ySwoMHXUKRczItmaUieXX IFig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=B1n9KuHbkwDteeyX+EGvgbidSs5/e5CI67AfWMUmp8s=; b=PqkCvdvvlkjkn7VcUK9e9ZgfNHTRY9Wh/PE8q7bNYspV/Rek48+Sj6m0y0wOhlessq 1VPF71ZeWQeCyDY3XISnahc3HAi8VdxRXxZtAa+Wl0MVJ0KskK2C3qiu8gPJsjfwB+PZ AaRlEh+TBsI1DYxB93atzv3papZXazzllqrfN4y4cbCJeHGYYJwLJrHVLorOIaiSWju+ JYoDD5AnzrdyccsIRcHcz6RSRFjymfmfyDMWIU6lfkJqGZj0EmBlsFfOJMGmeWk6e39R SZPVWZXyiqCcO87O4KG/l1rqec2yowoY8EswfDIx6SmaV2fH+9FbCQdR4NafUhxjyYkL 91+g== X-Gm-Message-State: AOAM530SSMZPCCklngB9oQzzGj9DQcTc9tnBu785b3Fd/NT7E+k4T4mV GPm6lRhKKo9mNebVNydb621Zb4L4U5Y= X-Google-Smtp-Source: ABdhPJxF8xo5w6K8Nn+HFQfi/CPEUbKsBx5EUV2igQkFgijrTJKmcZ10by10k68OT6k9qdD59CoIoQ== X-Received: by 2002:a63:1261:: with SMTP id 33mr10809432pgs.213.1612052173517; Sat, 30 Jan 2021 16:16:13 -0800 (PST) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id e12sm13127365pga.13.2021.01.30.16.16.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 30 Jan 2021 16:16:12 -0800 (PST) From: Nadav Amit X-Google-Original-From: Nadav Amit To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Nadav Amit , Andrea Arcangeli , Andrew Morton , Andy Lutomirski , Dave Hansen , Peter Zijlstra , Thomas Gleixner , Will Deacon , Yu Zhao , Nick Piggin , x86@kernel.org Subject: [RFC 08/20] mm: store completed TLB generation Date: Sat, 30 Jan 2021 16:11:20 -0800 Message-Id: <20210131001132.3368247-9-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210131001132.3368247-1-namit@vmware.com> References: <20210131001132.3368247-1-namit@vmware.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit To detect deferred TLB flushes in fine granularity, we need to keep track on the completed TLB flush generation for each mm. Add logic to track for each mm the tlb_gen_completed, which tracks the completed TLB generation. It is the arch responsibility to call mark_mm_tlb_gen_done() whenever a TLB flush is completed. Start the generation numbers from 1 instead of 0. This would allow later to detect whether flushes of a certain generation were completed. Signed-off-by: Nadav Amit Cc: Andrea Arcangeli Cc: Andrew Morton Cc: Andy Lutomirski Cc: Dave Hansen Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Will Deacon Cc: Yu Zhao Cc: Nick Piggin Cc: x86@kernel.org --- arch/x86/mm/tlb.c | 10 ++++++++++ include/asm-generic/tlb.h | 33 +++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 15 ++++++++++++++- 3 files changed, 57 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 7ab21430be41..d17b5575531e 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -14,6 +14,7 @@ #include #include #include +#include #include "mm_internal.h" @@ -915,6 +916,9 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids) flush_tlb_others(mm_cpumask(mm), info); + /* Update the completed generation */ + mark_mm_tlb_gen_done(mm, new_tlb_gen); + put_flush_tlb_info(); put_cpu(); } @@ -1147,6 +1151,12 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) cpumask_clear(&batch->cpumask); + /* + * We cannot call mark_mm_tlb_gen_done() since we do not know which + * mm's should be flushed. This may lead to some unwarranted TLB + * flushes, but not to correction problems. + */ + put_cpu(); } diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 517c89398c83..427bfcc6cdec 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -513,6 +513,39 @@ static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vm } #endif +#ifdef CONFIG_ARCH_HAS_TLB_GENERATIONS + +/* + * Helper function to update a generation to have a new value, as long as new + * value is greater or equal to gen. + */ +static inline void tlb_update_generation(atomic64_t *gen, u64 new_gen) +{ + u64 cur_gen = atomic64_read(gen); + + while (cur_gen < new_gen) { + u64 old_gen = atomic64_cmpxchg(gen, cur_gen, new_gen); + + /* Check if we succeeded in the cmpxchg */ + if (likely(cur_gen == old_gen)) + break; + + cur_gen = old_gen; + }; +} + + +static inline void mark_mm_tlb_gen_done(struct mm_struct *mm, u64 gen) +{ + /* + * Update the completed generation to the new generation if the new + * generation is greater than the previous one. + */ + tlb_update_generation(&mm->tlb_gen_completed, gen); +} + +#endif /* CONFIG_ARCH_HAS_TLB_GENERATIONS */ + /* * tlb_flush_{pte|pmd|pud|p4d}_range() adjust the tlb->start and tlb->end, * and set corresponding cleared_*. diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 2035ac319c2b..8a5eb4bfac59 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -571,6 +571,13 @@ struct mm_struct { * This is not used on Xen PV. */ atomic64_t tlb_gen; + + /* + * TLB generation which is guarnateed to be flushed, including + * all the PTE changes that were performed before tlb_gen was + * incremented. + */ + atomic64_t tlb_gen_completed; #endif } __randomize_layout; @@ -690,7 +697,13 @@ static inline bool mm_tlb_flush_nested(struct mm_struct *mm) #ifdef CONFIG_ARCH_HAS_TLB_GENERATIONS static inline void init_mm_tlb_gen(struct mm_struct *mm) { - atomic64_set(&mm->tlb_gen, 0); + /* + * Start from generation of 1, so default generation 0 will be + * considered as flushed and would not be regarded as an outstanding + * deferred invalidation. + */ + atomic64_set(&mm->tlb_gen, 1); + atomic64_set(&mm->tlb_gen_completed, 1); } static inline u64 inc_mm_tlb_gen(struct mm_struct *mm)