From patchwork Tue Aug 28 11:20:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10578279 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 807F414E1 for ; Tue, 28 Aug 2018 11:20:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 717BE28D71 for ; Tue, 28 Aug 2018 11:20:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 659BF299CF; Tue, 28 Aug 2018 11:20:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0696228D71 for ; Tue, 28 Aug 2018 11:20:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C6FE26B45EA; Tue, 28 Aug 2018 07:20:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BF5596B45EC; Tue, 28 Aug 2018 07:20:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A1EC56B45ED; Tue, 28 Aug 2018 07:20:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 5C8316B45EA for ; Tue, 28 Aug 2018 07:20:51 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id a26-v6so968558pgw.7 for ; Tue, 28 Aug 2018 04:20:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=PqIy7bFKrUvh3N2mZhaZbdyHPW+hgwvBa4i30Nw8Ybc=; b=qdTKCDjwEb+DgTaqDtTh5JwsunZEzcpo+xxAJDz2tabeCC+6lQkq9wu0TtzlmmrR/d xmhfY26y6tRpxVu5WHp4PmtwVmN187at175oEG1kM79bPVhcDm6BFEL3gc8lPhekuyoY KtT6KR0ZWrBp2CHySaFRkm4DDjpv//ILZCrqARDronv7QTyxqx3cBssWRZ6cEjqwKV3x wRCxROaQtAuOiK//Pl3akBa/vu2YvzHBpNl4uYuwmG5rAiEKVVxkBva2cq9mwRUxQ7am GsI+lTcVjNg8EPAhPA21yGs0CPcg8eO3hSokw6arOhnS/etmFWJXoVFG3MRxkyyHgpbv nwXg== X-Gm-Message-State: APzg51Bu1xr2y9S4xS6PG1FRBx08t+6bO8rap2g4PeOW1uluKxpSpyse EPur+YGVZDoqops2BQOdG2s9CqDv6F/QSKCYyMoXufAPy9B894JbiQmvEGRXjQQf56kLeKjlVhI NXo/Rbjd2vqj6vgOQXJ0dnz2guUktRV5iFPiBWTLWcpTzicTpOlI92aT0RznJp9cjDgjnR+/Qwz 6hSIv6kNFpVoF8Hskpa4v2KRatqAdQsZZcejLbrrJIC7PEKAucMpFDVkOGDmZUPHHGf3X9Fq8WY BlxKuwpOFIK05z1YEWhBYf7lAhyqI8Lrvsebf4ysfX03gipbvxkqd6UDZyqf9jx5sO6D+F8UsSp qRQ6JmH/TB6FqYzcboDXd4Jsi5/YrammUvCiicePhhfa7hS5GwmjPzy1tEFjJRMyYtFNOQ6Q7+Q 8 X-Received: by 2002:a17:902:82c9:: with SMTP id u9-v6mr1122274plz.138.1535455251043; Tue, 28 Aug 2018 04:20:51 -0700 (PDT) X-Received: by 2002:a17:902:82c9:: with SMTP id u9-v6mr1122206plz.138.1535455250020; Tue, 28 Aug 2018 04:20:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535455250; cv=none; d=google.com; s=arc-20160816; b=DzBqR7MBZcyvahB+gvl57h8L+zJn92sP3Jh9tWb0Xs7eczrWLQyEhbE+V8vNz6yPMe VycZQEQOdFzh6bdxix6YDrphv3DDmwYV2AomKbDeR0ufPqXghotGx4YFqlcoDgRZmlf8 pGEJizJ0aVmrVlNBZ5YmlZLwoC/fW/S0YBJzWqmXQ23fZVoOR9Q+WxWuIngFqRiE0V3B cNtVR7SMjGDOhT8LFdrFKTyxpxr8ZwESNrBDia2viGCbdfCAqXyHtJEimebpueskVjMi fWIBsMVi0Ftlb33tUmcsaaB+iBk5VmIkP7YSsnJbk3TJLaqCl2ztjgND115fI8P2NbHo NzqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=PqIy7bFKrUvh3N2mZhaZbdyHPW+hgwvBa4i30Nw8Ybc=; b=WtaBl6/0mViEuEOSs4M6zMZJA65GyLxPGhGYY6/PiQv2ZWM6VCpC9TEhAaZL9kqlvO Sy8A6lESHsG/BYWU2LyTUdLW9bK0cD54kF6e3G2JZPS+KKPPqdpTxEVFrIDx3/OlYPNX H/iUM87fGpGxgXBFGwW7suZATdEsAtVSOPBHMrTtrlI3vCPJhFxnUdKXvUC2rPFZVS8G QW3vp4B+kNFZ0bQ0rlRt15prxSnHqB51pKAeAPEKKklW92fO8Pxg+z7zWBwrt5hKh2uv 9qOUOG72pVgx/h9gBVrsJtv10Vb1mTMxrztZ5c4noV3w1PEiZcmBpU6xwDsGjDT4wn+3 dslA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=BZHp6QbN; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id u189-v6sor179715pgd.416.2018.08.28.04.20.49 for (Google Transport Security); Tue, 28 Aug 2018 04:20:49 -0700 (PDT) Received-SPF: pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=BZHp6QbN; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=PqIy7bFKrUvh3N2mZhaZbdyHPW+hgwvBa4i30Nw8Ybc=; b=BZHp6QbN59dTJzQoscwjj/i43FAP0Gjci3ePWXPKsGC8C1C6frC6pwM7ZO9Og1acP6 kalOnMjZXknjI8Nx28NzpbS+Bf3PiNWazSfTSbEb5Tfl+239vryJxYCtq1UZm1KWzn0H LErHEmA5WLAWDKAGAxnCjAXNSt84PWxsr1fyECPVZJQ/WqwNnCqAHpj1BOcu3dssf1Xt ri+KTIc8FfxXuQLLx0wAF3iLM12t2IOTD2KNAaa3tt6ARchgJ8OtW2orWdvTnAmDuRT0 34rrzhoTIK0LlmWUaEsu/2lFprRk1Z7eQ/jprd4BLoyNtsG+l4id/94KNevuGHZiRKdN ojKA== X-Google-Smtp-Source: ANB0Vda1rxfWQ20CAFltZX59amKlHwbHH1qDC49g8kZHwadojwF7gpVndoanEo69eLkZduJDG3UbpQ== X-Received: by 2002:a63:5751:: with SMTP id h17-v6mr1085425pgm.423.1535455249468; Tue, 28 Aug 2018 04:20:49 -0700 (PDT) Received: from roar.au.ibm.com (59-102-81-67.tpgi.com.au. [59.102.81.67]) by smtp.gmail.com with ESMTPSA id s3-v6sm3287917pgj.84.2018.08.28.04.20.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Aug 2018 04:20:48 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Andrew Morton , Linus Torvalds Subject: [PATCH 1/3] mm/cow: don't bother write protectig already write-protected huge pages Date: Tue, 28 Aug 2018 21:20:32 +1000 Message-Id: <20180828112034.30875-2-npiggin@gmail.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180828112034.30875-1-npiggin@gmail.com> References: <20180828112034.30875-1-npiggin@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This is the THP equivalent for 1b2de5d039c8 ("mm/cow: don't bother write protecting already write-protected pages"). Explicit hugetlb pages don't get the same treatment because they don't appear to have the right accessor functions. Signed-off-by: Nicholas Piggin --- mm/huge_memory.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9592cbd8530a..d9bae12978ef 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -973,8 +973,11 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, mm_inc_nr_ptes(dst_mm); pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); - pmdp_set_wrprotect(src_mm, addr, src_pmd); - pmd = pmd_mkold(pmd_wrprotect(pmd)); + if (pmd_write(pmd)) { + pmdp_set_wrprotect(src_mm, addr, src_pmd); + pmd = pmd_wrprotect(pmd); + } + pmd = pmd_mkold(pmd); set_pmd_at(dst_mm, addr, dst_pmd, pmd); ret = 0; @@ -1064,8 +1067,11 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, /* No huge zero pud yet */ } - pudp_set_wrprotect(src_mm, addr, src_pud); - pud = pud_mkold(pud_wrprotect(pud)); + if (pud_write(pud)) { + pudp_set_wrprotect(src_mm, addr, src_pud); + pud = pud_wrprotect(pud); + } + pud = pud_mkold(pud); set_pud_at(dst_mm, addr, dst_pud, pud); ret = 0; From patchwork Tue Aug 28 11:20:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10578281 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 911DF174C for ; Tue, 28 Aug 2018 11:20:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 822D428D71 for ; Tue, 28 Aug 2018 11:20:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 758A7299CC; Tue, 28 Aug 2018 11:20:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E8B2728D71 for ; Tue, 28 Aug 2018 11:20:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 78C036B45EC; Tue, 28 Aug 2018 07:20:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 738846B45EE; Tue, 28 Aug 2018 07:20:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 62A056B45EF; Tue, 28 Aug 2018 07:20:55 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 1BD756B45EC for ; Tue, 28 Aug 2018 07:20:55 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id i68-v6so762787pfb.9 for ; Tue, 28 Aug 2018 04:20:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=rCMUeVqKxlpR8mAw1QaqwgN5G/ui6F18sxW16mRgzbg=; b=aK1S50usuoG5DhnC4VjB4EG5YEP6i1WRx4LVAgZNxlJhDpSJh5dYvUDBAzi+r2B8T/ V5BYXAmokKwph5E840IguIxRURJdp5OrzGXvmVufgClHi7wRu1DSB22vTrku2DuQFP0e XaleSYo0q10IXmU7JOwL3gjZkbLWUln7flhifFFWpvRJpDmZzEi/htLJQCdkjmfNgGKA d2a6fhWMm7yZtuccuvGSrRuoblajCqH7fwDSouFNDZXtX3JpknE6fmxEA5Hl79wlwqnJ euHYo/MxBdBGXzt/JLVGiU5EERWjwDkbDxJawE8bD5b8CjR5B6Lxl8vbkITeBsrXi7DW 17cg== X-Gm-Message-State: APzg51By0+KWRtzzh7VBAF6B5bxz4opIrAdWuVhR6YAufs/Wy3Njc1SA maw43wMwR3QVIYm2LRN9G9a+cjEKL+tC/VgKTB2DiJMv3oLML1qTfYrS6/YIPUcNdpTH+7MOyXz PXDUyg4wiSJ/x0qjWYVtt/Xo9yEhFxO+kUGuYyHNxxhHItZEBdCXhAVtQVo8KEnWwGNTyhiHkoL LNSYhdLkZds4RDWUHwN/HN0DoEZ1Sa800ZrbjrzFkJXPsxDeRB2S3kc3Efs9tjgnRe5GSwr1BP/ D690q2Ak5xAPLyAZLqPjP2ma2+XoM5UJUJmY8ak9cTkljiANhUO+af7FX+WjTvOCGbAGaSz4rkC bF8Tiy60iCOdn/zmR9fkwATGUXbrNN8Aa1Rok+VX5RcfnEC0spgve0bc0pbGW+eE10Tv5rr5G4j O X-Received: by 2002:a17:902:558f:: with SMTP id g15-v6mr1142985pli.38.1535455254771; Tue, 28 Aug 2018 04:20:54 -0700 (PDT) X-Received: by 2002:a17:902:558f:: with SMTP id g15-v6mr1142929pli.38.1535455253780; Tue, 28 Aug 2018 04:20:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535455253; cv=none; d=google.com; s=arc-20160816; b=IT9etu6j5bAR7FNDmfvelKYs87C/GGf3rATyqF9wurh1daAobDl8K5NrVVAVHro2lX H5NJT1Dw7Lo0DmPsXxpWnHn+osVysBVm4Yq6XRse1K5dcHMrWgFz7M3aHPn4pkOy5eCJ cM9gJJntIcJYS2Z1bZhRZgVFEHbKujfITAltPFpmBup6mUSazkjj4UX9sjYze7CLgUDj CFHNVAVsKWtdYl4XyxKFBEkg+YcCS4BVlG1yo6ynnxInvDy2nR4klfmt0tF+CXcXydx7 ex+tl+nH+7/sj+Hty8cAbz6r01bMQt8KmoQSqy9en40t4FZ+RwsWNm80nxpRx3vhxxp5 cVEA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=rCMUeVqKxlpR8mAw1QaqwgN5G/ui6F18sxW16mRgzbg=; b=Ypr2n/axfU4BA15e8+ulBKw5plmbVFJ4sQnthT9Mf5VI/1c1X9k92xjrzyDm6oGGtN jwaetCptprYXdkqEfwjLazhunUR/XLZmmh3ZMZOtFAAZeV4Phafkl1PYysdL5xyWn16e 9jGcc4BC0jxa9Rvd4+MfpDZnVYofKi3HW0qTUxKgLxN2SCmpA+EH8FWXFIc5hlNkKaw4 bq+G42a9EVBffj3Z7fRhGaF1uTq5G0w0sfGBvPfz706pc1K4gbjTo3ZtsJQk3OHO7zr/ 6ReUpNil9bETvxYlB1Vqj/0MDbx2/Vc7JE+asaFGE7f2MZ6kFbSPqtkov0jdg/8XBNLj +y/g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=rOguRM3J; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id b69-v6sor240052plb.139.2018.08.28.04.20.53 for (Google Transport Security); Tue, 28 Aug 2018 04:20:53 -0700 (PDT) Received-SPF: pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=rOguRM3J; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=rCMUeVqKxlpR8mAw1QaqwgN5G/ui6F18sxW16mRgzbg=; b=rOguRM3J3lcM/kPZaSPcAEQbaVyrefutmBcfZrlssC+KfOFnsXJ6Vjui5heOyza9Xt iK7+x8ng89Bjapc7F0EsHJVJL5mB4ONxLruzuwpj/fUVU1WeBA6YVerZ2bDzK8nAemv0 uiBq3kX8C2IYkp9o+e1KW8Ac4R6HS4UvGcSAbaickQf+2TuqDmHDAX2cpKOajfYDYmOA VAm6IQKvxoIOHHydgXN+4A1KtQN/jt1SfMGDmY9E5mZE9DQgP2DXejhn/yB6yWhb8JwG 0TUf+aWp/CUuaqFWPjc6qjO1WRuCZukxDNIRvlCHNhD7UCpEn6x9ZTVKVRx+zG6Q4Ttp u1YA== X-Google-Smtp-Source: ANB0Vdbt9K3kz71sMSBn8c+JLrglp1376ALqJAIKdR2WDl+whCCjXliR4Y+1Pyz04+N2UEXNEBrAKQ== X-Received: by 2002:a17:902:6b4c:: with SMTP id g12-v6mr1111608plt.159.1535455253350; Tue, 28 Aug 2018 04:20:53 -0700 (PDT) Received: from roar.au.ibm.com (59-102-81-67.tpgi.com.au. [59.102.81.67]) by smtp.gmail.com with ESMTPSA id s3-v6sm3287917pgj.84.2018.08.28.04.20.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Aug 2018 04:20:52 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Andrew Morton , Linus Torvalds Subject: [PATCH 2/3] mm/cow: optimise pte dirty/accessed bits handling in fork Date: Tue, 28 Aug 2018 21:20:33 +1000 Message-Id: <20180828112034.30875-3-npiggin@gmail.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180828112034.30875-1-npiggin@gmail.com> References: <20180828112034.30875-1-npiggin@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP fork clears dirty/accessed bits from new ptes in the child. This logic has existed since mapped page reclaim was done by scanning ptes when it may have been quite important. Today with physical based pte scanning, there is less reason to clear these bits. Dirty bits are all tested and cleared together and any dirty bit is the same as many dirty bits. Any young bit is treated similarly to many young bits, but not quite the same. A comment has been added where there is some difference. This eliminates a major source of faults powerpc/radix requires to set dirty/accessed bits in ptes, speeding up a fork/exit microbenchmark by about 5% on POWER9 (16600 -> 17500 fork/execs per second). Skylake appears to have a micro-fault overhead too -- a test which allocates 4GB anonymous memory, reads each page, then forks, and times the child reading a byte from each page. The first pass over the pages takes about 1000 cycles per page, the second pass takes about 27 cycles (TLB miss). With no additional minor faults measured due to either child pass, and the page array well exceeding TLB capacity, the large cost must be caused by micro faults caused by setting accessed bit. Signed-off-by: Nicholas Piggin --- mm/huge_memory.c | 2 -- mm/memory.c | 10 +++++----- mm/vmscan.c | 8 ++++++++ 3 files changed, 13 insertions(+), 7 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d9bae12978ef..5fb1a43e12e0 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -977,7 +977,6 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmdp_set_wrprotect(src_mm, addr, src_pmd); pmd = pmd_wrprotect(pmd); } - pmd = pmd_mkold(pmd); set_pmd_at(dst_mm, addr, dst_pmd, pmd); ret = 0; @@ -1071,7 +1070,6 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, pudp_set_wrprotect(src_mm, addr, src_pud); pud = pud_wrprotect(pud); } - pud = pud_mkold(pud); set_pud_at(dst_mm, addr, dst_pud, pud); ret = 0; diff --git a/mm/memory.c b/mm/memory.c index b616a69ad770..3d8bf8220bd0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1038,12 +1038,12 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, } /* - * If it's a shared mapping, mark it clean in - * the child + * Child inherits dirty and young bits from parent. There is no + * point clearing them because any cleaning or aging has to walk + * all ptes anyway, and it will notice the bits set in the parent. + * Leaving them set avoids stalls and even page faults on CPUs that + * handle these bits in software. */ - if (vm_flags & VM_SHARED) - pte = pte_mkclean(pte); - pte = pte_mkold(pte); page = vm_normal_page(vma, addr, pte); if (page) { diff --git a/mm/vmscan.c b/mm/vmscan.c index 7e7d25504651..52fe64af3d80 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1021,6 +1021,14 @@ static enum page_references page_check_references(struct page *page, * to look twice if a mapped file page is used more * than once. * + * fork() will set referenced bits in child ptes despite + * not having been accessed, to avoid micro-faults of + * setting accessed bits. This heuristic is not perfectly + * accurate in other ways -- multiple map/unmap in the + * same time window would be treated as multiple references + * despite same number of actual memory accesses made by + * the program. + * * Mark it and spare it for another trip around the * inactive list. Another page table reference will * lead to its activation. From patchwork Tue Aug 28 11:20:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10578283 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AB9F7174C for ; Tue, 28 Aug 2018 11:21:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9B80A299CC for ; Tue, 28 Aug 2018 11:21:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8DD29297DB; Tue, 28 Aug 2018 11:21:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0A801297DB for ; Tue, 28 Aug 2018 11:21:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 010156B45EE; Tue, 28 Aug 2018 07:20:59 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id F01066B45F0; Tue, 28 Aug 2018 07:20:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA2DE6B45F1; Tue, 28 Aug 2018 07:20:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id 90B2E6B45EE for ; Tue, 28 Aug 2018 07:20:58 -0400 (EDT) Received: by mail-pl1-f200.google.com with SMTP id 2-v6so584008plc.11 for ; Tue, 28 Aug 2018 04:20:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=u5LLVwzqsd0W3khip77eczbmhBS8m0M7MliDfum5fbo=; b=lpJE/cCxp+swSmhNUjRHWD7Gw30dIpiwG1IDLi+yqcCdVHHaqDwaJjy2EXfsoT2jF6 GjznfnMX59yNuG2VfX51mu3P1jTWWj1aDsaokfM5xK3/cW6KtyJVvOX+/L+ZF+ZBIsi6 2iqdYmDkL9Q0JmN+0uNSdgylYEhbpFMJYRXw2TWMnK6nXMaOrPxBhj7a7/NTJrB10TtW v1hRRnXssyA+DvWrIWBMEz+okKOZNJiZmnTlvssz7g9DOOS8C2cs1+3KoiLgHeLN6Jiu JJUdJr7114IY5TOdxQPtOL9p2Piousuja3LjoWK4hRYkoqCi9VT1GmuKRNV9LDl0Hyzo ++dg== X-Gm-Message-State: APzg51DKUCvFJhCqOO8juPwyNu4wiE0bddHGJm7SOjcI7RoLZqN4a5Uz SrvTc7XAv8xEsvRCSnu5+H8pAjDPfhZKPdhOmmmIoHJqxjWjFD7KCy6KRun5TW8q4qtLqiJf3Z+ MEmOlB+cfZVRGSOhXs0fyeX7vxOMQNHwArAFKfMvwcyNIe/bhtXIW1Hnt90vpLCP39f6IXLvTd8 embUtSDovc7AvTf4HeqYqFMcF8UwgqOKMrkfKixWFLzTYNJVH3Z5cgb4rTBc/O/OHQgepPi/4cr +cONzku2fWLYrT/w/ZLlw/+l60Ho3phhB//pPZkwWgPv3UB0ebStlyEYlYaMjjxEEHtHefsE/Ic foH+djOg7AC0FL0Kwu2xo1ohGKc0mAdr3XuBtZaNDHEOQIBP7A3lj9oqLZRPb0Egf6YwRHi0/zk R X-Received: by 2002:a17:902:48c8:: with SMTP id u8-v6mr1143130plh.152.1535455258266; Tue, 28 Aug 2018 04:20:58 -0700 (PDT) X-Received: by 2002:a17:902:48c8:: with SMTP id u8-v6mr1143078plh.152.1535455257491; Tue, 28 Aug 2018 04:20:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535455257; cv=none; d=google.com; s=arc-20160816; b=UcIgFewQBUdPV3qNqVKLdfGUcFmSDEagS7698QAWhgMH+f1c3Cv0HGOYGxUyVWVw3r lcyyFnZ+CW1uRnr7EbOgIXuwYddIEmXB7tOXUMyM5EmcXJUujuYrI0YEAo19BSsn3Knn OKGJL/vI081rPWaByqRmvKAHg4KkxiAYj3e1r42AEHZ71n41sT9qkt7S8tsOyr8xbasp QcZZZx9XIQdRXnX1y1z7T3jyds04xJ7ZkKUEP4VJqza6VCCAXUdRRjfwYC0VnyeYFCkQ g0hSvCYSqCSoQyaxoYIZ/Cr4YS2qx7oFEPHwhTtoowg59u55TR5U1MQcz+npTRiQUduB 0m8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=u5LLVwzqsd0W3khip77eczbmhBS8m0M7MliDfum5fbo=; b=k3XRGxdpMvNf9gTvuxFRExcIbVntpogAni1QJ0kn6ZrpSIovr3YIaBwTEUpyHjrWTl rY6navx5dCy/aJLVbJlzxjJjd57pqAevW2satIxKYySLrwdsKejzONME0B2s0mZHee9j rN5eyDZSLU76rSHfJBzJCBrld6GX53Be/QwWUeYig6P3Y05bVR/Xff0MED+tC/3aO/11 ZfncvGPgDOTvejgrn9v2lNtP65lkpyNyyVvq8KTO6Lh3m5j+so61x46zRxFwhZo3ZcdQ BmKleMQhBoLgS70Ghy2AHQVJaA0/LLrzYTYj3WdKQt+YA/Q7hfA0mm962jPYN0fRN84i 1aNg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=JwRKkOpx; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id e2-v6sor263779pls.43.2018.08.28.04.20.57 for (Google Transport Security); Tue, 28 Aug 2018 04:20:57 -0700 (PDT) Received-SPF: pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=JwRKkOpx; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=u5LLVwzqsd0W3khip77eczbmhBS8m0M7MliDfum5fbo=; b=JwRKkOpxwjV2ovQ7J2Ho5+UlDBygiVUre+hfgsY9ifuXf7lMftdoyq0EjG5Y4wV9jI edNRMwartMVSqXKLDu9KmlMzCUzsjm1SEnGMe2GKjqwu5rEkfsiRHxxqwBpQMsqp84uM 61qFr8Q4SUNZ52mt4/gmik+I7401vIvRBgcknLsZKC4trEXjn7k+zwZSc1V1G9+jFkyv BoW03rpPR4WSm6aaQuOxSQqpT2H0MJDPANu/Bs7ctr23p5f5jUTAAiVEgXGHIM0Ktpv1 1j6HNaUmy1Rnx0kpiaEMO7Ut3drfbl2yP3OpVKcWN5uUo30RDYFEChjPVH3nS0A+Qxzg 7TOw== X-Google-Smtp-Source: ANB0VdYR8Iq6a5/JlwAWF9iIuyrZ3FY50H+M9B7BfYKPAMTl0ScSRzxrBp5pb47wFpOFK+WGxxiTKg== X-Received: by 2002:a17:902:6845:: with SMTP id f5-v6mr1121540pln.17.1535455257066; Tue, 28 Aug 2018 04:20:57 -0700 (PDT) Received: from roar.au.ibm.com (59-102-81-67.tpgi.com.au. [59.102.81.67]) by smtp.gmail.com with ESMTPSA id s3-v6sm3287917pgj.84.2018.08.28.04.20.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Aug 2018 04:20:56 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Andrew Morton , Linus Torvalds Subject: [PATCH 3/3] mm: optimise pte dirty/accessed bit setting by demand based pte insertion Date: Tue, 28 Aug 2018 21:20:34 +1000 Message-Id: <20180828112034.30875-4-npiggin@gmail.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180828112034.30875-1-npiggin@gmail.com> References: <20180828112034.30875-1-npiggin@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Similarly to the previous patch, this tries to optimise dirty/accessed bits in ptes to avoid access costs of hardware setting them. Signed-off-by: Nicholas Piggin --- mm/huge_memory.c | 12 +++++++----- mm/memory.c | 8 +++++--- 2 files changed, 12 insertions(+), 8 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 5fb1a43e12e0..2c169041317f 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1197,6 +1197,7 @@ static vm_fault_t do_huge_pmd_wp_page_fallback(struct vm_fault *vmf, for (i = 0; i < HPAGE_PMD_NR; i++, haddr += PAGE_SIZE) { pte_t entry; entry = mk_pte(pages[i], vma->vm_page_prot); + entry = pte_mkyoung(entry); entry = maybe_mkwrite(pte_mkdirty(entry), vma); memcg = (void *)page_private(pages[i]); set_page_private(pages[i], 0); @@ -2067,7 +2068,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, struct page *page; pgtable_t pgtable; pmd_t old_pmd, _pmd; - bool young, write, soft_dirty, pmd_migration = false; + bool young, write, dirty, soft_dirty, pmd_migration = false; unsigned long addr; int i; @@ -2145,8 +2146,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, page = pmd_page(old_pmd); VM_BUG_ON_PAGE(!page_count(page), page); page_ref_add(page, HPAGE_PMD_NR - 1); - if (pmd_dirty(old_pmd)) - SetPageDirty(page); + dirty = pmd_dirty(old_pmd); write = pmd_write(old_pmd); young = pmd_young(old_pmd); soft_dirty = pmd_soft_dirty(old_pmd); @@ -2176,8 +2176,10 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = maybe_mkwrite(entry, vma); if (!write) entry = pte_wrprotect(entry); - if (!young) - entry = pte_mkold(entry); + if (young) + entry = pte_mkyoung(entry); + if (dirty) + entry = pte_mkdirty(entry); if (soft_dirty) entry = pte_mksoft_dirty(entry); } diff --git a/mm/memory.c b/mm/memory.c index 3d8bf8220bd0..d205ba69918c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1830,10 +1830,9 @@ static int insert_pfn(struct vm_area_struct *vma, unsigned long addr, entry = pte_mkspecial(pfn_t_pte(pfn, prot)); out_mkwrite: - if (mkwrite) { - entry = pte_mkyoung(entry); + entry = pte_mkyoung(entry); + if (mkwrite) entry = maybe_mkwrite(pte_mkdirty(entry), vma); - } set_pte_at(mm, addr, pte, entry); update_mmu_cache(vma, addr, pte); /* XXX: why not for insert_page? */ @@ -2560,6 +2559,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) } flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); entry = mk_pte(new_page, vma->vm_page_prot); + entry = pte_mkyoung(entry); entry = maybe_mkwrite(pte_mkdirty(entry), vma); /* * Clear the pte entry and flush it first, before updating the @@ -3069,6 +3069,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); dec_mm_counter_fast(vma->vm_mm, MM_SWAPENTS); pte = mk_pte(page, vma->vm_page_prot); + pte = pte_mkyoung(pte); if ((vmf->flags & FAULT_FLAG_WRITE) && reuse_swap_page(page, NULL)) { pte = maybe_mkwrite(pte_mkdirty(pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE; @@ -3479,6 +3480,7 @@ vm_fault_t alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg, flush_icache_page(vma, page); entry = mk_pte(page, vma->vm_page_prot); + entry = pte_mkyoung(entry); if (write) entry = maybe_mkwrite(pte_mkdirty(entry), vma); /* copy-on-write page */