From patchwork Sat Dec 12 05:31:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Collingbourne X-Patchwork-Id: 11969985 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-21.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D493C4361B for ; Sat, 12 Dec 2020 05:31:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F214423137 for ; Sat, 12 Dec 2020 05:31:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F214423137 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 334916B0036; Sat, 12 Dec 2020 00:31:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2E5556B005C; Sat, 12 Dec 2020 00:31:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1D4486B005D; Sat, 12 Dec 2020 00:31:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0009.hostedemail.com [216.40.44.9]) by kanga.kvack.org (Postfix) with ESMTP id 07CFB6B0036 for ; Sat, 12 Dec 2020 00:31:57 -0500 (EST) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id B88E5181AEF3F for ; Sat, 12 Dec 2020 05:31:56 +0000 (UTC) X-FDA: 77583508632.23.rod90_17045db27407 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin23.hostedemail.com (Postfix) with ESMTP id 943E537604 for ; Sat, 12 Dec 2020 05:31:56 +0000 (UTC) X-HE-Tag: rod90_17045db27407 X-Filterd-Recvd-Size: 6106 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Sat, 12 Dec 2020 05:31:56 +0000 (UTC) Received: by mail-yb1-f201.google.com with SMTP id m203so13521560ybf.1 for ; Fri, 11 Dec 2020 21:31:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:message-id:mime-version:subject:from:to:cc; bh=mX5T9FW+2l/1R352XqQyejcKzawMBrrLG2SU6FLPDJs=; b=Qb8K6oiWp1Dh99hZP1NFLr0KaQwd22zUWJFepMgXty8BLOboxAKTVoVYKHqAZpo2pJ VZbVBJi61TrFiyuyeK7CUUFeoIgOCwHf+pcwbiq3120r0/hg63L3sgRDyI+D6jykUgRX U9mLSnNgpntgl429WYEcszFtbryZAks1qEJm1CYMdfi5Wlh4oUtxy4GjNwWi7h3/rUaB J8zwlDxC2F3HiHOuXx6esTjlRoQIyf5O0GBi6yNrj+LrunqZsjyV6dgm//mBmpJxCfVb R3j+PkSoL3dBF+0WELWmtW+TzxwS4p2Ky184tZoBTQ7Ja31gVBWOxKnGekneZsZnnMEb CloA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:message-id:mime-version:subject:from :to:cc; bh=mX5T9FW+2l/1R352XqQyejcKzawMBrrLG2SU6FLPDJs=; b=T+quFIgekMVh5eNx3OnieJyozhBOIiaGhiau0+gzMjC28ifY6KebsY552AzuZvwziQ ooJoGuu9HLgwNCKO76bQHDv2nbCcMnolIE0Y28zR0nXl/G4enHSwCf+M/Q/i3q2Hd9xn YDMpiJjOEtBPQOTPRg/PtMERZvURlKox9yXe/Wo6b4d0NjNAFoLiLzbAi3LygYHp8DKi c/xs9ilf99hfJIniIHoJ22DCI32y+9IWr4dYwj7skjWUI4VaGMMTKaSvg49DCVwf+Hsx 1Ur0ID7d6LjBiJlTvdDzDlGBKx7C068TlkTeAC1v1bgrq4TXw8ht3p0NP2Ulo9sVQZtQ Y2XQ== X-Gm-Message-State: AOAM530LRd8cHNs5De/VgPtaTHbUOSpGFLnA4GLQ1UOJ8NdSWvAwJSWX O+FCwTwTk3p17A7Ihjk+Qr/FJGo= X-Google-Smtp-Source: ABdhPJxNBU16avEDmcbz7C7/qakSpo46s+wL47ygPByviscozXmpzAXbqi6u9zZCHTUI7B5xCxwTItw= X-Received: from pcc-desktop.svl.corp.google.com ([2620:15c:2ce:0:7220:84ff:fe09:385a]) (user=pcc job=sendgmr) by 2002:a25:1843:: with SMTP id 64mr22883191yby.80.1607751115348; Fri, 11 Dec 2020 21:31:55 -0800 (PST) Date: Fri, 11 Dec 2020 21:31:52 -0800 Message-Id: <20201212053152.3783250-1-pcc@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.29.2.576.ga3fc446d84-goog Subject: [PATCH] mm: improve mprotect(R|W) efficiency on pages referenced once From: Peter Collingbourne To: Andrew Morton Cc: Peter Collingbourne , Kostya Kortchinsky , linux-mm@kvack.org X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In the Scudo memory allocator [1] we would like to be able to detect use-after-free vulnerabilities involving large allocations by issuing mprotect(PROT_NONE) on the memory region used for the allocation when it is deallocated. Later on, after the memory region has been "quarantined" for a sufficient period of time we would like to be able to use it for another allocation by issuing mprotect(PROT_READ|PROT_WRITE). Before this patch, after removing the write protection, any writes to the memory region would result in page faults and entering the copy-on-write code path, even in the usual case where the pages are only referenced by a single PTE, harming performance unnecessarily. Make it so that any pages in anonymous mappings that are only referenced by a single PTE are immediately made writable during the mprotect so that we can avoid the page faults. This program shows the critical syscall sequence that we intend to use in the allocator: #include #include enum { kSize = 131072 }; int main(int argc, char **argv) { char *addr = (char *)mmap(0, kSize, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); for (int i = 0; i != 100000; ++i) { memset(addr, i, kSize); mprotect((void *)addr, kSize, PROT_NONE); mprotect((void *)addr, kSize, PROT_READ | PROT_WRITE); } } The effect of this patch on the above program was measured on a DragonBoard 845c by taking the median real time execution time of 10 runs. Before: 3.19s After: 0.79s The effect was also measured using one of the microbenchmarks that we normally use to benchmark the allocator [2], after modifying it to make the appropriate mprotect calls [3]. With an allocation size of 131072 bytes to trigger the allocator's "large allocation" code path the per-iteration time was measured as follows: Before: 33364ns After: 6886ns Signed-off-by: Peter Collingbourne Link: https://linux-review.googlesource.com/id/I98d75ef90e20330c578871c87494d64b1df3f1b8 Link: [1] https://source.android.com/devices/tech/debug/scudo Link: [2] https://cs.android.com/android/platform/superproject/+/master:bionic/benchmarks/stdlib_benchmark.cpp;l=53;drc=e8693e78711e8f45ccd2b610e4dbe0b94d551cc9 Link: [3] https://github.com/pcc/llvm-project/commit/scudo-mprotect-secondary --- mm/mprotect.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/mprotect.c b/mm/mprotect.c index 56c02beb6041..6f5313d66d00 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -47,6 +47,8 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, bool prot_numa = cp_flags & MM_CP_PROT_NUMA; bool uffd_wp = cp_flags & MM_CP_UFFD_WP; bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE; + bool anon_writable = + vma_is_anonymous(vma) && (vma->vm_flags & VM_WRITE); /* * Can be called with only the mmap_lock for reading by @@ -136,7 +138,11 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, (pte_soft_dirty(ptent) || !(vma->vm_flags & VM_SOFTDIRTY))) { ptent = pte_mkwrite(ptent); + } else if (anon_writable && + page_mapcount(pte_page(ptent)) == 1) { + ptent = pte_mkwrite(ptent); } + ptep_modify_prot_commit(vma, addr, pte, oldpte, ptent); pages++; } else if (is_swap_pte(oldpte)) {