From patchwork Sun Jan 31 00:11:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12057473 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7988C433E6 for ; Sun, 31 Jan 2021 00:16:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8D04964E15 for ; Sun, 31 Jan 2021 00:16:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8D04964E15 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 82B836B0083; Sat, 30 Jan 2021 19:16:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 787CA6B0085; Sat, 30 Jan 2021 19:16:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5DAE86B0087; Sat, 30 Jan 2021 19:16:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0137.hostedemail.com [216.40.44.137]) by kanga.kvack.org (Postfix) with ESMTP id 421616B0083 for ; Sat, 30 Jan 2021 19:16:30 -0500 (EST) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 05CD63633 for ; Sun, 31 Jan 2021 00:16:30 +0000 (UTC) X-FDA: 77764153740.21.chin40_240427e275b5 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin21.hostedemail.com (Postfix) with ESMTP id DCD31180442C4 for ; Sun, 31 Jan 2021 00:16:29 +0000 (UTC) X-HE-Tag: chin40_240427e275b5 X-Filterd-Recvd-Size: 6446 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf27.hostedemail.com (Postfix) with ESMTP for ; Sun, 31 Jan 2021 00:16:29 +0000 (UTC) Received: by mail-pf1-f181.google.com with SMTP id t29so9063712pfg.11 for ; Sat, 30 Jan 2021 16:16:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kt89wMExxB9mDZA9vWtyDT+tDBx3/YIxqezv1vec+68=; b=Z5HJbKiNov3/1MqP6XXO23ukd068I8ix9I99unM52IccOHfT9Hj9BoH9R8X1vm2fgE JDk7hr2hNN3oxblJMNiZ7K2IeCvHh3mrtboU70WDcxA03xXVGv8TUwLYbq5mMCt73Zoq UytzF2crzPOyu4jT7ytbq3xuthDJcdJN8rX5OGFqu7jjNON5BLB/sM+j8twhIoxIcXIZ JHGjQvRX4asWSyEQVvYHDQG7hQpLHmn+TanLbrdt7oQzw5S2SZKtooN6t/r1jQV7bKQ0 OljxSOcmcQy6UXIx0Ig5hMgRaFKhhkVwWb9Me2tokcWFccRzJJDsJjNUnLC1cCclYDAs pEhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kt89wMExxB9mDZA9vWtyDT+tDBx3/YIxqezv1vec+68=; b=bpMvMpX06VVAklWL/H6mnu8RDnkeInyf4pI6/g3gvYizNCxQB351QPelaxKCGBYLiH biRo3b4NnTfvMlxei4/oObuR/J5dTlk2S1qeOi7tJRGMfqyL4m67Jf6UDg2G+8Ax6ocQ n3BlRo+27REJcNth+kzBVmNTvbEYrFmOY6UkUB+iYh757wBKvJED6D7FK1cZ8BsFwEt1 r/NggdOxkasPGpT0P5680Jm1tFZ67bF1LnVvs4OGSuuSFaW/GLv0QCNcMfEVLa2cApZk F3FWZ6rC6/VIN+qm3RoW0tljQt1FBURuAdn53/LkzGcHUqat9bdqmQNGZXNLFtxf/Mkt Ohqg== X-Gm-Message-State: AOAM531ZcUzNH+Zh52Mfrby9Xj8P3gw1SWWujgrqvNjDZGtYZ4DvwDWT Iu4QQgz9t+Qp9KOiJ0q1s5rX0TkBXuw= X-Google-Smtp-Source: ABdhPJz+SyTFVAhW+P9pLT3RRUrcjfybFvt58ygJGhoPJwFKwTcLrwg+PMGlshJUUkIYq9kJ0kl6Sw== X-Received: by 2002:a65:68ce:: with SMTP id k14mr9338963pgt.401.1612052188222; Sat, 30 Jan 2021 16:16:28 -0800 (PST) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id e12sm13127365pga.13.2021.01.30.16.16.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 30 Jan 2021 16:16:27 -0800 (PST) From: Nadav Amit X-Google-Original-From: Nadav Amit To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Nadav Amit , Andrea Arcangeli , Andrew Morton , Andy Lutomirski , Dave Hansen , Peter Zijlstra , Thomas Gleixner , Will Deacon , Yu Zhao , x86@kernel.org Subject: [RFC 17/20] mm/tlb: updated completed deferred TLB flush conditionally Date: Sat, 30 Jan 2021 16:11:29 -0800 Message-Id: <20210131001132.3368247-18-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210131001132.3368247-1-namit@vmware.com> References: <20210131001132.3368247-1-namit@vmware.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit If all the deferred TLB flushes were completed, there is no need to update the completed TLB flush. This update requires an atomic cmpxchg, so we would like to skip it. To do so, save for each mm the last TLB generation in which TLB flushes were deferred. While saving this information requires another atomic cmpxchg, assume that deferred TLB flushes are less frequent than TLB flushes. Signed-off-by: Nadav Amit Cc: Andrea Arcangeli Cc: Andrew Morton Cc: Andy Lutomirski Cc: Dave Hansen Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Will Deacon Cc: Yu Zhao Cc: x86@kernel.org --- include/asm-generic/tlb.h | 23 ++++++++++++++++++----- include/linux/mm_types.h | 5 +++++ 2 files changed, 23 insertions(+), 5 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 74dbb56d816d..a41af03fbede 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -536,6 +536,14 @@ static inline void tlb_update_generation(atomic64_t *gen, u64 new_gen) static inline void mark_mm_tlb_gen_done(struct mm_struct *mm, u64 gen) { + /* + * If we all the deferred TLB generations were completed, we can skip + * the update of tlb_gen_completed and save a few cycles on cmpxchg. + */ + if (atomic64_read(&mm->tlb_gen_deferred) == + atomic64_read(&mm->tlb_gen_completed)) + return; + /* * Update the completed generation to the new generation if the new * generation is greater than the previous one. @@ -546,7 +554,7 @@ static inline void mark_mm_tlb_gen_done(struct mm_struct *mm, u64 gen) static inline void read_defer_tlb_flush_gen(struct mmu_gather *tlb) { struct mm_struct *mm = tlb->mm; - u64 mm_gen; + u64 mm_gen, new_gen; /* * Any change of PTE before calling __track_deferred_tlb_flush() must be @@ -567,11 +575,16 @@ static inline void read_defer_tlb_flush_gen(struct mmu_gather *tlb) * correctness issues, and should not induce overheads, since anyhow in * TLB storms it is better to perform full TLB flush. */ - if (mm_gen != tlb->defer_gen) { - VM_BUG_ON(mm_gen < tlb->defer_gen); + if (mm_gen == tlb->defer_gen) + return; - tlb->defer_gen = inc_mm_tlb_gen(mm); - } + VM_BUG_ON(mm_gen < tlb->defer_gen); + + new_gen = inc_mm_tlb_gen(mm); + tlb->defer_gen = new_gen; + + /* Update mm->tlb_gen_deferred */ + tlb_update_generation(&mm->tlb_gen_deferred, new_gen); } #ifndef CONFIG_PER_TABLE_DEFERRED_FLUSHES diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index cae9e8bbf8e6..4122a9b8b56f 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -578,6 +578,11 @@ struct mm_struct { */ atomic64_t tlb_gen; + /* + * The last TLB generation which was deferred. + */ + atomic64_t tlb_gen_deferred; + /* * TLB generation which is guarnateed to be flushed, including * all the PTE changes that were performed before tlb_gen was