From patchwork Sun Dec 20 04:55:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11983779 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0C42C4361B for ; Sun, 20 Dec 2020 04:55:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A363923A5E for ; Sun, 20 Dec 2020 04:55:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A363923A5E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3A19A6B005D; Sat, 19 Dec 2020 23:55:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3527C6B0068; Sat, 19 Dec 2020 23:55:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 240566B006C; Sat, 19 Dec 2020 23:55:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0092.hostedemail.com [216.40.44.92]) by kanga.kvack.org (Postfix) with ESMTP id 0D9A16B005D for ; Sat, 19 Dec 2020 23:55:47 -0500 (EST) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id D5E7D3641 for ; Sun, 20 Dec 2020 04:55:46 +0000 (UTC) X-FDA: 77612447892.16.cart37_4a05edb2744c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin16.hostedemail.com (Postfix) with ESMTP id B840C100E690B for ; Sun, 20 Dec 2020 04:55:46 +0000 (UTC) X-HE-Tag: cart37_4a05edb2744c X-Filterd-Recvd-Size: 4764 Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Sun, 20 Dec 2020 04:55:46 +0000 (UTC) Received: by mail-pj1-f48.google.com with SMTP id l23so4510497pjg.1 for ; Sat, 19 Dec 2020 20:55:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xWd672dBo9Oa6ix0+hhyLcTLYZH2Zvk+xoJHykTA92c=; b=DRvKHlCOwLBnLdj3M27nD7OS7WS1CL9dsWFdy1VEQRPr5A9ITSXQlysyPVhfv3grG2 P6jelf3s1kJINJzugRromFaG6a1yXfHKS8f8hS6+VByTMNtdZytrrINXb8MTjN5PKvIR IOsvbEktdy6Ys+SjFPdqOFNx8l24/yG5LHQCoYRZLZy3UCJvUvrMY01QfP1/Kh5Aczbe VFYZKiqKLuAuEbLh97qb/g1OFzzIO9MY5UA1NnSCyBHEtlbvwjIlLSH1CKVym19u5PDw 476YaZV5+AbXpfO4LXareb5t5u0pDmBFUxsRpypVESAxUeSCcyhdbyrvjAjMLvPw0Rzm rJow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xWd672dBo9Oa6ix0+hhyLcTLYZH2Zvk+xoJHykTA92c=; b=bk7KlXloPPWqrKm3df4fYfQHfdjY7Jgn9eUxDEK168C6N7pXo1HK49yHLmidnmumSc lEnXVU1HN+ye6xfXIpI/CUa7z8j4yjCSzJtgHX7eQqYAahYSA1LBM4O0ApJnZI+rldOm eWrWcvIXE+QsWnHVzH5n6ztN/buRx2lTBs2hPypKGAZszHhNxxyMPf2XwzFywN2vrrXi vMrZjpOI6pfOyCA4UzNhdOjQCxYSQHSX5kE8l6/Na6RBZ6GgQ+4FqktWk3yKScKl7jnw 1UprRXr3acbpJDk1+Y1XITMVIlQAuOVuMhT0DrkezJV254wURfIszJcUsPd1AciobqqV hKwA== X-Gm-Message-State: AOAM533/ixiflxH4B51lSAyEl37awvRDsD35PCr3KQyettQLKnk+HExZ Qru/e4qcC95ydsRIqrFppZjg+Q1lk30= X-Google-Smtp-Source: ABdhPJwbY7xRsSUBpwC4t7vnDEpbEz/VAOd+XRsZ5CRDtV8WUNErUyJVBgb65Kvo4OifwU6asHgKGg== X-Received: by 2002:a17:90b:3682:: with SMTP id mj2mr11599960pjb.27.1608440144987; Sat, 19 Dec 2020 20:55:44 -0800 (PST) Received: from bobo.ibm.com (193-116-97-30.tpgi.com.au. [193.116.97.30]) by smtp.gmail.com with ESMTPSA id l13sm13632529pgq.51.2020.12.19.20.55.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Dec 2020 20:55:44 -0800 (PST) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , Linus Torvalds , Bibo Mao Subject: [PATCH v3 1/3] mm/cow: don't bother write protecting already write-protected huge pages Date: Sun, 20 Dec 2020 14:55:33 +1000 Message-Id: <20201220045535.848591-2-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201220045535.848591-1-npiggin@gmail.com> References: <20201220045535.848591-1-npiggin@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000021, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is the HugePage / THP equivalent for 1b2de5d039c8 ("mm/cow: don't bother write protecting already write-protected pages"). Signed-off-by: Nicholas Piggin --- mm/huge_memory.c | 14 ++++++++++---- mm/hugetlb.c | 2 +- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9237976abe72..87da60c583a9 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1111,8 +1111,11 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, mm_inc_nr_ptes(dst_mm); pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); - pmdp_set_wrprotect(src_mm, addr, src_pmd); - pmd = pmd_mkold(pmd_wrprotect(pmd)); + if (pmd_write(pmd)) { + pmdp_set_wrprotect(src_mm, addr, src_pmd); + pmd = pmd_wrprotect(pmd); + } + pmd = pmd_mkold(pmd); set_pmd_at(dst_mm, addr, dst_pmd, pmd); ret = 0; @@ -1218,8 +1221,11 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, return -EAGAIN; } - pudp_set_wrprotect(src_mm, addr, src_pud); - pud = pud_mkold(pud_wrprotect(pud)); + if (pud_write(pud)) { + pudp_set_wrprotect(src_mm, addr, src_pud); + pud = pud_mkold(pud_wrprotect(pud)); + } + pud = pud_mkold(pud); set_pud_at(dst_mm, addr, dst_pud, pud); ret = 0; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index cbf32d2824fd..c068ae4b2fef 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3813,7 +3813,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, } set_huge_swap_pte_at(dst, addr, dst_pte, entry, sz); } else { - if (cow) { + if (cow && huge_pte_write(entry)) { /* * No need to notify as we are downgrading page * table protection not changing it to point From patchwork Sun Dec 20 04:55:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11983781 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A133C4361B for ; Sun, 20 Dec 2020 04:55:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2169723A5E for ; Sun, 20 Dec 2020 04:55:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2169723A5E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B68ED6B0068; Sat, 19 Dec 2020 23:55:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B19836B006C; Sat, 19 Dec 2020 23:55:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9E2806B006E; Sat, 19 Dec 2020 23:55:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0204.hostedemail.com [216.40.44.204]) by kanga.kvack.org (Postfix) with ESMTP id 83FF76B0068 for ; Sat, 19 Dec 2020 23:55:49 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 422318249980 for ; Sun, 20 Dec 2020 04:55:49 +0000 (UTC) X-FDA: 77612448018.22.anger03_0a05bd32744c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin22.hostedemail.com (Postfix) with ESMTP id 287CF18038E67 for ; Sun, 20 Dec 2020 04:55:49 +0000 (UTC) X-HE-Tag: anger03_0a05bd32744c X-Filterd-Recvd-Size: 6435 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Sun, 20 Dec 2020 04:55:48 +0000 (UTC) Received: by mail-pl1-f174.google.com with SMTP id e2so3822505plt.12 for ; Sat, 19 Dec 2020 20:55:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=S0i+SiU+HSwLByUxXA2fjHZNAtDQPgxviRKysSaSS3w=; b=i8gXCGtTmntMbjJF+dcARSh3FSXoytamZ1YK2mrv8qsRVKw9tHR7Y7Zarxlplra4YD eyYmwItiWgWa1dh6votrLxg3Kgzu0xBlIzbZIQQ8g7T7L6e+90drgkZrcOdoUt4iMd6l NY299oxJc3+kXrncmocritYske70k6woY2N2yg+h3eeqBqgWgwvvyXLWlr3N8OcKYUyp 2qb+vCJ3jfNwCCFV/WS3eGaCdaEED7S1B1h6zyuCPdRuh6JjbZY8h8G5qxkRbg/QGISY daRzvefmTL7tkVAoTY+T/yRDdiLyqguZlhfs71Tag5FE0rBO6MVdwfBnQJfHB0jMl72M svng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=S0i+SiU+HSwLByUxXA2fjHZNAtDQPgxviRKysSaSS3w=; b=BGs0V6cSj7GFdCDBu1l4xIwtRBL12ah6RxyLhY0M+gUKOifvFc8Gs+kbPdJiB2cWHg VeTSEol5zcuxda3XhWCCZmRwkt9Cv2eadija/sFOHyG7ugiF2TLnQwWNlnNVZTQZO2bI zsJUpsGZpqKhfshG8D3ZIVLLIwm6eEjaJ+qoWvj10fmHEBwBu1Y+l4ycg3kyJcLwCexm AXS+KFBR8zHHBxGJDmh5p/2UwG5yEPlMrdeObLOUFvR0PViUrtb+YuG1HikxAsI2oMHY 0Qg9CmJcMInrNTFs+BDjDJnIgBgwSG9/eszptMvUZxqC3UOZbuUo68tQ2sDmIKLiwPJB VL2Q== X-Gm-Message-State: AOAM533rz7HLEjtUILgMXXfuFNyhM5XwwdNXwyQfsvuWiPhu20uWvozB o0+pb26nLS6/4mdVjbn/DMu+SuOd0hA= X-Google-Smtp-Source: ABdhPJx8xhUTo1at8PlNARbDVpR3p/TT1iGAX/C6QjTlYX0wqujyS56OlrJBL5wGh42KW0UWAjF2pQ== X-Received: by 2002:a17:902:6903:b029:da:f458:798c with SMTP id j3-20020a1709026903b02900daf458798cmr11291063plk.68.1608440147810; Sat, 19 Dec 2020 20:55:47 -0800 (PST) Received: from bobo.ibm.com (193-116-97-30.tpgi.com.au. [193.116.97.30]) by smtp.gmail.com with ESMTPSA id l13sm13632529pgq.51.2020.12.19.20.55.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Dec 2020 20:55:47 -0800 (PST) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , Linus Torvalds , Bibo Mao Subject: [PATCH v3 2/3] mm/cow: optimise pte accessed bit handling in fork Date: Sun, 20 Dec 2020 14:55:34 +1000 Message-Id: <20201220045535.848591-3-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201220045535.848591-1-npiggin@gmail.com> References: <20201220045535.848591-1-npiggin@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: fork clears dirty/accessed bits from new ptes in the child. This logic has existed since mapped page reclaim was done by scanning ptes when it may have been quite important. Today with physical based pte scanning, there is less reason to clear these bits, so this patch avoids clearing the accessed bit in the child. Any accessed bit is treated similarly to many, with the difference today with > 1 referenced bit causing the page to be activated, while 1 bit causes it to be kept. This patch causes pages shared by fork(2) to be more readily activated, but this heuristic is very fuzzy anyway -- a page can be accessed by multiple threads via a single pte and be just as important as one that is accessed via multiple ptes, for example. In the end I don't believe fork(2) is a significant driver of page reclaim behaviour that this should matter too much. This and the following change eliminate a major source of faults that powerpc/radix requires to set dirty/accessed bits in ptes, speeding up a fork/exit microbenchmark by about 5% on POWER9 (16600 -> 17500 fork/execs per second). Skylake appears to have a micro-fault overhead too -- a test which allocates 4GB anonymous memory, reads each page, then forks, and times the child reading a byte from each page. The first pass over the pages takes about 1000 cycles per page, the second pass takes about 27 cycles (TLB miss). With no additional minor faults measured due to either child pass, and the page array well exceeding TLB capacity, the large cost must be micro faults caused by setting the accessed bit. Signed-off-by: Nicholas Piggin --- mm/huge_memory.c | 2 -- mm/memory.c | 1 - mm/vmscan.c | 5 +++++ 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 87da60c583a9..f2ca0326b5af 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1115,7 +1115,6 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmdp_set_wrprotect(src_mm, addr, src_pmd); pmd = pmd_wrprotect(pmd); } - pmd = pmd_mkold(pmd); set_pmd_at(dst_mm, addr, dst_pmd, pmd); ret = 0; @@ -1225,7 +1224,6 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, pudp_set_wrprotect(src_mm, addr, src_pud); pud = pud_mkold(pud_wrprotect(pud)); } - pud = pud_mkold(pud); set_pud_at(dst_mm, addr, dst_pud, pud); ret = 0; diff --git a/mm/memory.c b/mm/memory.c index 990e5d704c08..dd1f364d8ca3 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -886,7 +886,6 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, */ if (vm_flags & VM_SHARED) pte = pte_mkclean(pte); - pte = pte_mkold(pte); /* * Make sure the _PAGE_UFFD_WP bit is cleared if the new VMA diff --git a/mm/vmscan.c b/mm/vmscan.c index 257cba79a96d..604ead623842 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1012,6 +1012,11 @@ static enum page_references page_check_references(struct page *page, * Note: the mark is set for activated pages as well * so that recently deactivated but used pages are * quickly recovered. + * + * Note: fork() will copy referenced bit from parent + * to child ptes, despite not having been accessed by + * the child. This is to avoid micro-faults on initial + * access. */ SetPageReferenced(page); From patchwork Sun Dec 20 04:55:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11983783 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 58941C4361B for ; Sun, 20 Dec 2020 04:55:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 03E5D23A5E for ; Sun, 20 Dec 2020 04:55:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 03E5D23A5E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 993F66B006C; Sat, 19 Dec 2020 23:55:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 943D36B006E; Sat, 19 Dec 2020 23:55:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 833BD6B0070; Sat, 19 Dec 2020 23:55:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0104.hostedemail.com [216.40.44.104]) by kanga.kvack.org (Postfix) with ESMTP id 64BBF6B006C for ; Sat, 19 Dec 2020 23:55:52 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 2CEEF180AD82F for ; Sun, 20 Dec 2020 04:55:52 +0000 (UTC) X-FDA: 77612448144.24.boats70_1d108af2744c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin24.hostedemail.com (Postfix) with ESMTP id 0D0211A4A0 for ; Sun, 20 Dec 2020 04:55:52 +0000 (UTC) X-HE-Tag: boats70_1d108af2744c X-Filterd-Recvd-Size: 9494 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Sun, 20 Dec 2020 04:55:51 +0000 (UTC) Received: by mail-pl1-f171.google.com with SMTP id j1so3850935pld.3 for ; Sat, 19 Dec 2020 20:55:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9bX3LR+n1i0jGEGH3dWRU+mPM2rK0/8EWTnEof4Fv40=; b=nY2uBwPdkxF4qOrBcBFqRSED5+iUmsyuIg32bAnbFKN6twHpXeiHZWDaHJ/Ro0hhvf STNGlB7UYx+15oWTWvhDuwZ1aPKbeqGSquegDv4Ne5yPM3HCakGQ5f2BNzZnTuzLW5Y2 XxJrnNlyd3EI4vYVceVKfCEBZ7p77KGNVele9BMP21ynuHZke2K2oRxbzzHoCItP7qfa qNI4/uVioCVc45TOE6iWVhJF2awFVbcqsHhwvaImQxxTHe3jColIJ5iiHYOLlUgaVmVP xhxk5jCA+cnjxVe+lcUwFedYY1NOSMpGuaEDDE+9zR3R+4vLaI3SG0umCyEGiSoSHWIe ov5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9bX3LR+n1i0jGEGH3dWRU+mPM2rK0/8EWTnEof4Fv40=; b=eSj2L0taetPHWTCnocb2uK2fmgCoweqiLz8j17TNYGSMuGdNNlCnS/JfWu0DzwDPBz S4bNk7i/reQAiSsetcPNveU0csGH5x9abNlHqE3viF2Vs5csQNdKqbA1BsHSj5w5UGTJ YuMABb6bW8eY/tmOku4iqQhU2jk4OREpA1R0hTRe2aikqvbn2TF5nvh6zJzOCCTp3EAH WIkFEE5jnFta5uKZA61PKI7qiJBqEz3xvQ/HUVh9KzlCf8SoTUcnEM3UoEwn7SGb4irE UK57ksS1PHLmFK41tQKDNCYNkOYCBUFKNq67I0lTcHZVSayOsypvKsluZ+3Z3luQ8RF/ dVvw== X-Gm-Message-State: AOAM53270NOfdDFYO14yzHRgkTf3XP8DlDOgp1nbIFiaYF1ZIbqDfTke hsrY4pq841+g74CwOftszuu25QMSI4Y= X-Google-Smtp-Source: ABdhPJzwFr+CaiXFxtP03ohueNGOOIdu541URfXMkFXGiKdCmQZQa5qLEKYYWrZ7tTJNpeoa/OLQtw== X-Received: by 2002:a17:90a:9f44:: with SMTP id q4mr11791060pjv.226.1608440150554; Sat, 19 Dec 2020 20:55:50 -0800 (PST) Received: from bobo.ibm.com (193-116-97-30.tpgi.com.au. [193.116.97.30]) by smtp.gmail.com with ESMTPSA id l13sm13632529pgq.51.2020.12.19.20.55.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Dec 2020 20:55:50 -0800 (PST) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , Linus Torvalds , Bibo Mao Subject: [PATCH v3 3/3] mm: optimise pte dirty/accessed bit setting by demand based pte insertion Date: Sun, 20 Dec 2020 14:55:35 +1000 Message-Id: <20201220045535.848591-4-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201220045535.848591-1-npiggin@gmail.com> References: <20201220045535.848591-1-npiggin@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Similarly to the previous patch, this tries to optimise dirty/accessed bits in ptes to avoid access costs of hardware setting them. This tidies up a few last cases where dirty/accessed faults can be seen, and subsumes the pte_sw_mkyoung helper -- it's not just architectures with explicit software dirty/accessed bits that take expensive faults to modify ptes. The vast majority of the remaining dirty/accessed faults on kbuild workloads after this patch are from NUMA migration, due to remove_migration_pte inserting old/clean ptes. Signed-off-by: Nicholas Piggin --- arch/mips/include/asm/pgtable.h | 2 -- include/linux/pgtable.h | 16 ---------------- mm/huge_memory.c | 4 ++-- mm/memory.c | 14 +++++++------- mm/migrate.c | 1 + mm/shmem.c | 1 + mm/userfaultfd.c | 2 +- 7 files changed, 12 insertions(+), 28 deletions(-) diff --git a/arch/mips/include/asm/pgtable.h b/arch/mips/include/asm/pgtable.h index 4f9c37616d42..3275495adccb 100644 --- a/arch/mips/include/asm/pgtable.h +++ b/arch/mips/include/asm/pgtable.h @@ -406,8 +406,6 @@ static inline pte_t pte_mkyoung(pte_t pte) return pte; } -#define pte_sw_mkyoung pte_mkyoung - #ifdef CONFIG_MIPS_HUGE_TLB_SUPPORT static inline int pte_huge(pte_t pte) { return pte_val(pte) & _PAGE_HUGE; } diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 8fcdfa52eb4b..70d04931dff4 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -424,22 +424,6 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres } #endif -/* - * On some architectures hardware does not set page access bit when accessing - * memory page, it is responsibilty of software setting this bit. It brings - * out extra page fault penalty to track page access bit. For optimization page - * access bit can be set during all page fault flow on these arches. - * To be differentiate with macro pte_mkyoung, this macro is used on platforms - * where software maintains page access bit. - */ -#ifndef pte_sw_mkyoung -static inline pte_t pte_sw_mkyoung(pte_t pte) -{ - return pte; -} -#define pte_sw_mkyoung pte_sw_mkyoung -#endif - #ifndef pte_savedwrite #define pte_savedwrite pte_write #endif diff --git a/mm/huge_memory.c b/mm/huge_memory.c index f2ca0326b5af..f6719312dc27 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2151,8 +2151,8 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = maybe_mkwrite(entry, vma); if (!write) entry = pte_wrprotect(entry); - if (!young) - entry = pte_mkold(entry); + if (young) + entry = pte_mkyoung(entry); if (soft_dirty) entry = pte_mksoft_dirty(entry); if (uffd_wp) diff --git a/mm/memory.c b/mm/memory.c index dd1f364d8ca3..4cebba596660 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1639,7 +1639,7 @@ static int insert_page_into_pte_locked(struct mm_struct *mm, pte_t *pte, get_page(page); inc_mm_counter_fast(mm, mm_counter_file(page)); page_add_file_rmap(page, false); - set_pte_at(mm, addr, pte, mk_pte(page, prot)); + set_pte_at(mm, addr, pte, pte_mkyoung(mk_pte(page, prot))); return 0; } @@ -1954,10 +1954,9 @@ static vm_fault_t insert_pfn(struct vm_area_struct *vma, unsigned long addr, else entry = pte_mkspecial(pfn_t_pte(pfn, prot)); - if (mkwrite) { - entry = pte_mkyoung(entry); + entry = pte_mkyoung(entry); + if (mkwrite) entry = maybe_mkwrite(pte_mkdirty(entry), vma); - } set_pte_at(mm, addr, pte, entry); update_mmu_cache(vma, addr, pte); /* XXX: why not for insert_page? */ @@ -2889,7 +2888,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) } flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); entry = mk_pte(new_page, vma->vm_page_prot); - entry = pte_sw_mkyoung(entry); + entry = pte_mkyoung(entry); entry = maybe_mkwrite(pte_mkdirty(entry), vma); /* * Clear the pte entry and flush it first, before updating the @@ -3402,6 +3401,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); dec_mm_counter_fast(vma->vm_mm, MM_SWAPENTS); pte = mk_pte(page, vma->vm_page_prot); + pte = pte_mkyoung(pte); if ((vmf->flags & FAULT_FLAG_WRITE) && reuse_swap_page(page, NULL)) { pte = maybe_mkwrite(pte_mkdirty(pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE; @@ -3545,7 +3545,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) __SetPageUptodate(page); entry = mk_pte(page, vma->vm_page_prot); - entry = pte_sw_mkyoung(entry); + entry = pte_mkyoung(entry); if (vma->vm_flags & VM_WRITE) entry = pte_mkwrite(pte_mkdirty(entry)); @@ -3821,7 +3821,7 @@ vm_fault_t alloc_set_pte(struct vm_fault *vmf, struct page *page) flush_icache_page(vma, page); entry = mk_pte(page, vma->vm_page_prot); - entry = pte_sw_mkyoung(entry); + entry = pte_mkyoung(entry); if (write) entry = maybe_mkwrite(pte_mkdirty(entry), vma); /* copy-on-write page */ diff --git a/mm/migrate.c b/mm/migrate.c index ee5e612b4cd8..d33b2bfc846b 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2963,6 +2963,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, } } else { entry = mk_pte(page, vma->vm_page_prot); + entry = pte_mkyoung(entry); if (vma->vm_flags & VM_WRITE) entry = pte_mkwrite(pte_mkdirty(entry)); } diff --git a/mm/shmem.c b/mm/shmem.c index 7c6b6d8f6c39..4f23b16d6baf 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2420,6 +2420,7 @@ static int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, goto out_release; _dst_pte = mk_pte(page, dst_vma->vm_page_prot); + _dst_pte = pte_mkyoung(_dst_pte); if (dst_vma->vm_flags & VM_WRITE) _dst_pte = pte_mkwrite(pte_mkdirty(_dst_pte)); else { diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 9a3d451402d7..56c44aa06a7e 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -99,7 +99,7 @@ static int mcopy_atomic_pte(struct mm_struct *dst_mm, if (mem_cgroup_charge(page, dst_mm, GFP_KERNEL)) goto out_release; - _dst_pte = pte_mkdirty(mk_pte(page, dst_vma->vm_page_prot)); + _dst_pte = pte_mkdirty(pte_mkyoung(mk_pte(page, dst_vma->vm_page_prot))); if (dst_vma->vm_flags & VM_WRITE) { if (wp_copy) _dst_pte = pte_mkuffd_wp(_dst_pte);