From patchwork Wed Jul 25 15:52:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10544249 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2C706139A for ; Wed, 25 Jul 2018 15:53:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1B2B22A62C for ; Wed, 25 Jul 2018 15:53:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0F7482A5D9; Wed, 25 Jul 2018 15:53:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4644C28C1C for ; Wed, 25 Jul 2018 15:53:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7F9446B0295; Wed, 25 Jul 2018 11:53:08 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 784286B0296; Wed, 25 Jul 2018 11:53:08 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B1CC6B029A; Wed, 25 Jul 2018 11:53:08 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 129FE6B0295 for ; Wed, 25 Jul 2018 11:53:08 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id e19-v6so5014599pgv.11 for ; Wed, 25 Jul 2018 08:53:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=bnIhgs4ewuvZ6PC2wzBBOY45XHEVcX8KLV28U6C8Xdw=; b=DzNdytEyVB93tSq7DT5ugIOgtAaDw+fz5HQSQ5GQfit86usmdbTdUrDnomg1NZ0SZJ E7+RL4XkM48uB2hbVx0evuEVwrecvxop6JavB7Nfq3FojFKwCkl2D7lfbKjXRRPn70fR qVn47/NhyY/2lMzhVtAng3n50s4P4Hu1JckdblrHb2iI0UE81YKx2Ozf69eFSzGnSYSr 71Op5UFnRnDxed9oQh7QiR7+xAjE1uvVg6msvOIx6DsvcaRVv9wYsbm2CeabmNb+uKCi kaMCAVM79gv1Li1KW3Cq91jjKnq5UOLaQ33G1z5xtRCffGp037+WiXTKAzzTaJRFZLr1 oTAQ== X-Gm-Message-State: AOUpUlFdnR02G9utYf7UEp+OdtCkAJBj8f+EtJlN8oFl27/bmq596SqK SRBSXSbAQPkXfap0j6GpvFeavYs3UekKZTus2NS9q7tCvDTyw1subR0dU6TtF25AvzMMznnnfpy 0imxyz4wv8kgVRmsV6iQ4hpZgpUxDqKW5PT/o51YFHt63/nqi92HfIRAaXMFDKxnQXst8nI5BIY XI3Cds1mu+OdbXfoTw5sr3oZvE/eQZkzV9TiMasLrqoCNhqSKKzHOWmsL7z7R5fp9Zni6E/nBnl QixZbSrVJfFQLLFMSOI4WZX86VwXdrLa69qsbAeA5EqQWodKx07GcA0y1Y8QUxcu93IPaoEuga1 /QWiWmJ6kgs5uM6NBGU4k9q9b8/iwpV3Kh1HyksKlA65XH/oX462JG1LSlyfvy7AH6XIH53FOXu 0 X-Received: by 2002:a63:8449:: with SMTP id k70-v6mr21232993pgd.309.1532533987744; Wed, 25 Jul 2018 08:53:07 -0700 (PDT) X-Received: by 2002:a63:8449:: with SMTP id k70-v6mr21232955pgd.309.1532533986818; Wed, 25 Jul 2018 08:53:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1532533986; cv=none; d=google.com; s=arc-20160816; b=Y7SQoHluL1Cyc1MdPFLnEWrqsmHvgeZFp7ed3cYYLdOTMVIuEg6Eqx4K6uwPvgtrlC yRIqFSEjbxAFjY+Gc4Ce2pDQDG3E4WocPWQmJyVby/H3tmnXythbsWnIZBDMm57aUl4I tpC8IiRXSiqOlBozPh66OO8EW7xNNu/tMj4PWMV0YozleiuDurX37FJhpsdusB5osden cyv/L0uFeadTiEFazaanie9geae2ZuBhWpA29FM1Oeh6q8MtALYBV6Ajf4H2pn7PzRdS UPhr21hKoPIUshmgy52NstO58QvwJ+H0zEvWGRK4qAGIyfbcy6IF9MeqyM/K7v7bAqv4 Y6gw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=bnIhgs4ewuvZ6PC2wzBBOY45XHEVcX8KLV28U6C8Xdw=; b=SZxvhlZ9/1nU/3zNITHnaoGxC/ZKgJziSftroDhFlLUQ4CjKgTIiOTNRB4Aok1nZZ9 nI6Cd/nNq909xP8c7lVjAEdJXXDiNBGc8Zl5+t+np+BBt65QGL1ocvL1maffwogYC9qz CPue+lDFF6hSSA9FIBccht1Qd4Z1iF5KwRS5wDQZu4WANmi7MD30kB8gonvYR6jM91rt vSIXfCWMoFus9V7zp49VGAKNDG0dlR7nFTygK1emV7VNsciE3DARLr5HD51R9H99WPmU 4pVhKyA1YihCLYb/BCtXzvzVe9wyjXj/KHcB1CyaxYaLqloBnVMzD6g18ENl8mAfdBKW iw6Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=fIbI3ZkQ; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id i13-v6sor4057961pgi.127.2018.07.25.08.53.06 for (Google Transport Security); Wed, 25 Jul 2018 08:53:06 -0700 (PDT) Received-SPF: pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=fIbI3ZkQ; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=bnIhgs4ewuvZ6PC2wzBBOY45XHEVcX8KLV28U6C8Xdw=; b=fIbI3ZkQvC7teSReRGUJngCCNZLNRAJX+WKnES7UFEmGRqWvemxnctWyAvuPAbFdnq yHzup+UDM24mP94mz5Q3oD4vKXCduPJh+kT4chgBDubrSp9JA/uIn44miuGJfgypFlJT kBIHgW7AvVyS1o11IiUEtOLvgao9ZWASOjAcdwT6t3uaExkqEfLMbU4aApP+g00nHZaC fT2AdJBpmHa1MQp6OQJxwNtJU2BJMd/hZrcuDgqqL1/wXgSdQ7uoTJAwwNUb2k5fFBOT JCFrEHyXIdr3tIZUD/OASgG96j1XNhCJHJEODkN4XQNbZRPcuftcL+04VCdflCWddBL2 s72w== X-Google-Smtp-Source: AAOMgpdMcLV/nJw0jrLWzUg9SiqBKpLLKIhWFnhAzvz55fvNAPf+pYIQ6Fdcxm7YtQfQ9SxWN3ml6Q== X-Received: by 2002:a65:40cd:: with SMTP id u13-v6mr21456233pgp.334.1532533986356; Wed, 25 Jul 2018 08:53:06 -0700 (PDT) Received: from roar.au.ibm.com ([61.69.188.107]) by smtp.gmail.com with ESMTPSA id j72-v6sm23438315pge.19.2018.07.25.08.53.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Jul 2018 08:53:05 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-arch@vger.kernel.org Subject: [RFC PATCH 4/4] mm: optimise flushing and pte manipulation for single threaded access Date: Thu, 26 Jul 2018 01:52:46 +1000 Message-Id: <20180725155246.1085-5-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180725155246.1085-1-npiggin@gmail.com> References: <20180725155246.1085-1-npiggin@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP I think many tlb->fullmm users actually want to know if there could be concurrent memory accesses via the mappings being invalidated. If there is no risk of the page being modified after the pte is changed and if the pte dirty/young bits won't be modified by the hardware concurrently, there is no reason to defer page freeing or atomically update ptes. I haven't gone carefully through all archs, maybe there is some exception. But for core mm code I haven't been able to see why the single threaded case should be different than the fullmm case in terms of the pte updates and tlb flushing. --- include/asm-generic/tlb.h | 3 +++ mm/huge_memory.c | 4 ++-- mm/madvise.c | 4 ++-- mm/memory.c | 13 +++++++------ 4 files changed, 14 insertions(+), 10 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index a55ef1425f0d..601c9fefda8e 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -100,6 +100,9 @@ struct mmu_gather { /* we are in the middle of an operation to clear * a full mm and can make some optimizations */ unsigned int fullmm : 1, + /* we have a single thread, current, which is active in the user + * address space */ + singlethread: 1, /* we have performed an operation which * requires a complete flush of the tlb */ need_flush_all : 1; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 1cd7c1a57a14..8e3958a811f0 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1703,7 +1703,7 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, * operations. */ orig_pmd = pmdp_huge_get_and_clear_full(tlb->mm, addr, pmd, - tlb->fullmm); + tlb->singlethread); tlb_remove_pmd_tlb_entry(tlb, pmd, addr); if (vma_is_dax(vma)) { if (arch_needs_pgtable_deposit()) @@ -1971,7 +1971,7 @@ int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, * operations. */ orig_pud = pudp_huge_get_and_clear_full(tlb->mm, addr, pud, - tlb->fullmm); + tlb->singlethread); tlb_remove_pud_tlb_entry(tlb, pud, addr); if (vma_is_dax(vma)) { spin_unlock(ptl); diff --git a/mm/madvise.c b/mm/madvise.c index 4d3c922ea1a1..d9e1f3ac8067 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -350,7 +350,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, continue; nr_swap--; free_swap_and_cache(entry); - pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); + pte_clear_not_present_full(mm, addr, pte, tlb->singlethread); continue; } @@ -417,7 +417,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, * after pte clearing. */ ptent = ptep_get_and_clear_full(mm, addr, pte, - tlb->fullmm); + tlb->singlethread); ptent = pte_mkold(ptent); ptent = pte_mkclean(ptent); diff --git a/mm/memory.c b/mm/memory.c index 490689909186..471ba07bc7e3 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -221,8 +221,9 @@ void arch_tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm, { tlb->mm = mm; - /* Is it from 0 to ~0? */ tlb->fullmm = !(start | (end+1)); + tlb->singlethread = (mm == current->mm) && + (atomic_read(&mm->mm_users) <= 1); tlb->need_flush_all = 0; tlb->local.next = NULL; tlb->local.nr = 0; @@ -300,7 +301,7 @@ bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_ * When this is our mm and there are no other users, there can not be * a concurrent memory access. */ - if (current->mm == tlb->mm && atomic_read(&tlb->mm->mm_users) < 2) { + if (tlb->singlethread) { free_page_and_swap_cache(page); return false; } @@ -1315,7 +1316,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; } ptent = ptep_get_and_clear_full(mm, addr, pte, - tlb->fullmm); + tlb->singlethread); tlb_remove_tlb_entry(tlb, pte, addr); if (unlikely(!page)) continue; @@ -1330,7 +1331,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, * the old TLB after it was marked * clean. */ - if (!tlb->fullmm) { + if (!tlb->singlethread) { force_flush = 1; locked_flush = 1; } @@ -1367,7 +1368,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; } - pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); + pte_clear_not_present_full(mm, addr, pte, tlb->singlethread); rss[mm_counter(page)]--; page_remove_rmap(page, false); put_page(page); @@ -1389,7 +1390,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, } if (unlikely(!free_swap_and_cache(entry))) print_bad_pte(vma, addr, ptent, NULL); - pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); + pte_clear_not_present_full(mm, addr, pte, tlb->singlethread); } while (pte++, addr += PAGE_SIZE, addr != end); add_mm_rss_vec(mm, rss);