From patchwork Sun Jan 31 00:11:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12057447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D91FAC433E6 for ; Sun, 31 Jan 2021 00:16:11 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 93CF164E17 for ; Sun, 31 Jan 2021 00:16:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 93CF164E17 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0F8C16B0070; Sat, 30 Jan 2021 19:16:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 086936B0071; Sat, 30 Jan 2021 19:16:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E186F6B0072; Sat, 30 Jan 2021 19:16:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0193.hostedemail.com [216.40.44.193]) by kanga.kvack.org (Postfix) with ESMTP id C47B06B0070 for ; Sat, 30 Jan 2021 19:16:08 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 91E0C3632 for ; Sun, 31 Jan 2021 00:16:08 +0000 (UTC) X-FDA: 77764152816.24.bulb53_57154b9275b5 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin24.hostedemail.com (Postfix) with ESMTP id 603151A4A5 for ; Sun, 31 Jan 2021 00:16:08 +0000 (UTC) X-HE-Tag: bulb53_57154b9275b5 X-Filterd-Recvd-Size: 6974 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) by imf11.hostedemail.com (Postfix) with ESMTP for ; Sun, 31 Jan 2021 00:16:07 +0000 (UTC) Received: by mail-pf1-f176.google.com with SMTP id o20so9090929pfu.0 for ; Sat, 30 Jan 2021 16:16:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jBC+01VJDydhQXc0ahXrqYuyDmXo9pxN+3IIrkulC2k=; b=TVaH6VkAFpqdsk3/O5YnMODhdAAk+rzD1GW94VeMtYiLPCpNcRRex1R5curwd/F/u2 JoZO+6hwCClkfatFg7nvWtmamjIgVWUi6E8GJpxIWv2iPzxgDU/qaQNi3EZ9zSaSkdcr JeafsVuqp25qdIhEsQH1BBdCItzsiDYGctTkAvqDLxfpb74yY1KmKU2sjHqS2M0xFeVc YIU60KL8eTty3Ml5hP60IR5UfA1EwnwH4r29RX9A0wlgF123L+y962rKwK3V6mzC1u1r JIrMzJzUFgndPpsk4dnSac3Hja48wtwRVU1AJ865LmviTsl5s5ntTPYH88jXCOxx/4V8 nuXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jBC+01VJDydhQXc0ahXrqYuyDmXo9pxN+3IIrkulC2k=; b=JXRX/QvkVXBU68XtO3rpFFtZPBJ9ADFvDou8wIlU033iw6faT8K4skxzyTty4YQh8/ hA2EPjIbtCTYKlKzOVhL+0iFxGJuECp15c7SGBTl7M4UoF6KeaHNuN4Rqw/zl1dSxkvr dHBms+itAQep8hjUgoSRob3Rd4FLu30vxPAh0ZJdbF3J+yzDkP5WLr6dcJo0GvWFsT/Z rEWStbia4vD5Z+4Q0cBEsL/Av8XmRFWHhKfHhF3FvT3s9skCkoATzPx5e3zt4TBRgV3O xH/B1Rb9ffgYhKaMYJ/1HW1TypQCyWrHBGGExNc8Tr9kv6HsJUPmOmlM/sibtAwN7fCn Pr4g== X-Gm-Message-State: AOAM532H14InH2rYgKUXouyfCvKwE36nCsM/2vQiUSyNCZvlozMqViAE fp40rxoQCmkHMg1rrxQXjR3gcvC8Gwo= X-Google-Smtp-Source: ABdhPJyqoXCkGVfO0VoTpCz9roEewxlxHC7XiUyi9KFDCIAE2zizHYqngYx2P5q4ZIJvw6BChNykYQ== X-Received: by 2002:a63:43c6:: with SMTP id q189mr10561751pga.245.1612052166645; Sat, 30 Jan 2021 16:16:06 -0800 (PST) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id e12sm13127365pga.13.2021.01.30.16.16.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 30 Jan 2021 16:16:06 -0800 (PST) From: Nadav Amit X-Google-Original-From: Nadav Amit To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Nadav Amit , Andrea Arcangeli , Andrew Morton , Andy Lutomirski , Dave Hansen , Peter Zijlstra , Thomas Gleixner , Will Deacon , Yu Zhao , Nick Piggin , x86@kernel.org Subject: [RFC 04/20] mm/mapping_dirty_helpers: use mmu_gather Date: Sat, 30 Jan 2021 16:11:16 -0800 Message-Id: <20210131001132.3368247-5-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210131001132.3368247-1-namit@vmware.com> References: <20210131001132.3368247-1-namit@vmware.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit Avoid open-coding mmu_gather for no reason. There is no apparent reason not to use the existing mmu_gather interfaces. Use the newly introduced pte_may_need_flush() to check whether a flush is needed to avoid unnecassary flushes. Signed-off-by: Nadav Amit Cc: Andrea Arcangeli Cc: Andrew Morton Cc: Andy Lutomirski Cc: Dave Hansen Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Will Deacon Cc: Yu Zhao Cc: Nick Piggin Cc: x86@kernel.org --- mm/mapping_dirty_helpers.c | 37 +++++++++++-------------------------- 1 file changed, 11 insertions(+), 26 deletions(-) diff --git a/mm/mapping_dirty_helpers.c b/mm/mapping_dirty_helpers.c index b59054ef2e10..2ce6cf431026 100644 --- a/mm/mapping_dirty_helpers.c +++ b/mm/mapping_dirty_helpers.c @@ -4,7 +4,7 @@ #include #include #include -#include +#include /** * struct wp_walk - Private struct for pagetable walk callbacks @@ -15,8 +15,7 @@ */ struct wp_walk { struct mmu_notifier_range range; - unsigned long tlbflush_start; - unsigned long tlbflush_end; + struct mmu_gather tlb; unsigned long total; }; @@ -42,9 +41,9 @@ static int wp_pte(pte_t *pte, unsigned long addr, unsigned long end, ptent = pte_wrprotect(old_pte); ptep_modify_prot_commit(walk->vma, addr, pte, old_pte, ptent); wpwalk->total++; - wpwalk->tlbflush_start = min(wpwalk->tlbflush_start, addr); - wpwalk->tlbflush_end = max(wpwalk->tlbflush_end, - addr + PAGE_SIZE); + + if (pte_may_need_flush(old_pte, ptent)) + tlb_flush_pte_range(&wpwalk->tlb, addr, PAGE_SIZE); } return 0; @@ -101,9 +100,7 @@ static int clean_record_pte(pte_t *pte, unsigned long addr, ptep_modify_prot_commit(walk->vma, addr, pte, old_pte, ptent); wpwalk->total++; - wpwalk->tlbflush_start = min(wpwalk->tlbflush_start, addr); - wpwalk->tlbflush_end = max(wpwalk->tlbflush_end, - addr + PAGE_SIZE); + tlb_flush_pte_range(&wpwalk->tlb, addr, PAGE_SIZE); __set_bit(pgoff, cwalk->bitmap); cwalk->start = min(cwalk->start, pgoff); @@ -184,20 +181,13 @@ static int wp_clean_pre_vma(unsigned long start, unsigned long end, { struct wp_walk *wpwalk = walk->private; - wpwalk->tlbflush_start = end; - wpwalk->tlbflush_end = start; - mmu_notifier_range_init(&wpwalk->range, MMU_NOTIFY_PROTECTION_PAGE, 0, walk->vma, walk->mm, start, end); mmu_notifier_invalidate_range_start(&wpwalk->range); flush_cache_range(walk->vma, start, end); - /* - * We're not using tlb_gather_mmu() since typically - * only a small subrange of PTEs are affected, whereas - * tlb_gather_mmu() records the full range. - */ - inc_tlb_flush_pending(walk->mm); + tlb_gather_mmu(&wpwalk->tlb, walk->mm); + tlb_start_vma(&wpwalk->tlb, walk->vma); return 0; } @@ -212,15 +202,10 @@ static void wp_clean_post_vma(struct mm_walk *walk) { struct wp_walk *wpwalk = walk->private; - if (mm_tlb_flush_nested(walk->mm)) - flush_tlb_range(walk->vma, wpwalk->range.start, - wpwalk->range.end); - else if (wpwalk->tlbflush_end > wpwalk->tlbflush_start) - flush_tlb_range(walk->vma, wpwalk->tlbflush_start, - wpwalk->tlbflush_end); - mmu_notifier_invalidate_range_end(&wpwalk->range); - dec_tlb_flush_pending(walk->mm); + + tlb_end_vma(&wpwalk->tlb, walk->vma); + tlb_finish_mmu(&wpwalk->tlb); } /*