From patchwork Tue Oct 16 13:13:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10643533 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6405D13B0 for ; Tue, 16 Oct 2018 13:14:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5120F28C31 for ; Tue, 16 Oct 2018 13:14:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4524929654; Tue, 16 Oct 2018 13:14:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CA62B28C31 for ; Tue, 16 Oct 2018 13:14:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 84E6C6B0006; Tue, 16 Oct 2018 09:14:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7FF596B0007; Tue, 16 Oct 2018 09:14:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6EDD96B0008; Tue, 16 Oct 2018 09:14:02 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id 30A9C6B0006 for ; Tue, 16 Oct 2018 09:14:02 -0400 (EDT) Received: by mail-pl1-f198.google.com with SMTP id g6-v6so18199436plo.0 for ; Tue, 16 Oct 2018 06:14:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=og3nps762l1wehgPrVvLZ4tOt6tgHDhsP/lk8/fPWl8=; b=PRfKbDD7BmJuMWtTbCeIiW7mp9S8KEBv1AybCPNJPSb+DGWAMh52R14Zz91j4pqsTv b6gxKgsE4Pkdv9pWJGzdXV0NclQ62XqR7VxDuMaWisegDJTAfG960tnqi8o6PsLcnG/J C8m98eU7fE1I+ItPaTNRuJFLUSaYcqZ2AOTZClVJrshtuwxSRiFgrQAmHB7hT5uo2if0 8rp8lLkSEqqlUGEzIUjNqcEStAZITJLJBjgNhz+BUk/lVK3+XFinUIpbAYqSvYvT1/qr ojeSh7nNkDlVmAny6oaPS0IQnhoSg8y43CJjCoKgpcH3jqiYXi80zs+GA3UaJy7Rs4hO Rndg== X-Gm-Message-State: ABuFfojsHbXLdUxC6qutEr1bEV1FuiGq/RqFis6Vi8//l3lGYd8Z7TIj qmpoOOOoD09V2utO/SxM+thh8whWD1gfcfkpVjw/YzS6y+cdeSuCWCWakQfMteK1oNTrSqs6/wg cFZ65N6UzvFZIq4kRI1PifALgNhmXY6NUFrdXdkVW6jwjeNgXvPawsVSt/lQZdpRkFSzstkdHBR 6sjuf5RTuILXahpAWRG6sX8wuPtiwhL9Rs+EY30o5fXpp8ApxiAfkvcOSEuXXsHbtyk/jYITNni kUF8Mhhi/GPQaA2h7umaUsFb2qvUkjx3JB8BkDE35BHm8QH+jZ5j6Yx+Leb5B5dQ8m86i/knOim XVaKIP4BnniIxUJ3eAHcNn+JeUpNCReESmybybE2fuymd6yjQsaH1xUKwWPwiccgIPpbn8k8GvB d X-Received: by 2002:a62:507:: with SMTP id 7-v6mr22093262pff.80.1539695641873; Tue, 16 Oct 2018 06:14:01 -0700 (PDT) X-Received: by 2002:a62:507:: with SMTP id 7-v6mr22093212pff.80.1539695641037; Tue, 16 Oct 2018 06:14:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539695641; cv=none; d=google.com; s=arc-20160816; b=W3bM4nc2ROMGIfZ5Oeeaglk2iSXOpnq31p1tfXvK7xH/vWPByGgyZ9nidHUqLfVnn2 S07kdrIN3BocJ3PMjsIS5vEh8Kdm+p9XSp4RIyOOl+vneqEquigKW7nz/kqKhHI2vZGA TDpBeaLnd+B4+BaFGtVOBeKACFAZ7VoT5Qf5oZbcmcg4lbh5lH6Re3Z2kvDrFWSQ06mS DAdCu5ONvhglmE5Vz21Srsuh3ZXgoEygILhKAs/Lm8w7DTbM5/T0KRRz2A2zveWYcLc9 l4P8cF13ZI44b81/LOLDA4gcZ7rYX4nG5wWleo9lIIbP8hLuk190nL2piu3zBck/Ke7i rJZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=og3nps762l1wehgPrVvLZ4tOt6tgHDhsP/lk8/fPWl8=; b=k21w1OqoZ4pPnRFIMjlUXNcnMMvsX0wprl6UgWK72n+SSZjZVPivWjfXrV4SxVS/gK GHivFvRkN/6hyzD7L9DCpxtuLw7CBhU3TcJcbWOeB04t9LzW9QCQysSa5lMmsdUOv4Rv 8GaH4oh2FzH5HOosjdPGlWQdw+P+sQhd0f60lO+D2I/1oglJql4JHU5hfd1Dgf7mzlXl qURzei6a5zniDlfn91VLyQGttiZYNMSvyAdQxxFTr3rk4WB+XTvbTfimDL53GrWPKO/7 u+pj5Bqn2i83WAz7D5IEHFn59GynWdBIbQmD2N78+mRbCVfrOaes4Z1CrhRgZmaOKkD2 uKNw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=Nm0YzlJB; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id i69-v6sor4614927pge.17.2018.10.16.06.14.00 for (Google Transport Security); Tue, 16 Oct 2018 06:14:01 -0700 (PDT) Received-SPF: pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=Nm0YzlJB; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=og3nps762l1wehgPrVvLZ4tOt6tgHDhsP/lk8/fPWl8=; b=Nm0YzlJBr/tFpHAsamNBQRtcflUZ1a3Fhs7jRJoXnpFPig4gbEgKXSD8GJg5pTuCzZ flZP8osFYXA9rcYPLY1PyxKzJemBVuQjRz30z6FwFB4TaoUDz2o5xuG/CMsPKZ2UFyqs RZ+GE2BV0OhCu0HksSYWPZDL+bdF8YLGRJ14FGakPYH/sq2hk0gVDXssBQ1yLJUjEr7Q Xs9PwkS8+UAIHo28AreMJSU7LdlhogWZ2UT0D0x7OwKMjaCFxzh/mt7sv551VBgwuR39 L/QfwPOstPLfmSkiEpLRxyhfsrZnq5zTvdQQsfeW8zv5Cuo1DGQSTzvpgO2MSayvrNPU O72Q== X-Google-Smtp-Source: ACcGV633qW2OBNPwm3Of34MQ0PiCfw8SZWKbv1d8VPnZh7InlVEXQra/DC+Bn4ZaPxLg1AladyddzA== X-Received: by 2002:a63:4f20:: with SMTP id d32-v6mr18957422pgb.231.1539695640808; Tue, 16 Oct 2018 06:14:00 -0700 (PDT) Received: from roar.local0.net ([60.240.252.156]) by smtp.gmail.com with ESMTPSA id j62-v6sm16043423pgd.40.2018.10.16.06.13.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Oct 2018 06:14:00 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Linus Torvalds , linux-mm , linux-arch , Linux Kernel Mailing List , ppc-dev , Ley Foon Tan Subject: [PATCH v2 1/5] nios2: update_mmu_cache clear the old entry from the TLB Date: Tue, 16 Oct 2018 23:13:39 +1000 Message-Id: <20181016131343.20556-2-npiggin@gmail.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20181016131343.20556-1-npiggin@gmail.com> References: <20181016131343.20556-1-npiggin@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Fault paths like do_read_fault will install a Linux pte with the young bit clear. The CPU will fault again because the TLB has not been updated, this time a valid pte exists so handle_pte_fault will just set the young bit with ptep_set_access_flags, which flushes the TLB. The TLB is flushed so the next attempt will go to the fast TLB handler which loads the TLB with the new Linux pte. The access then proceeds. This design is fragile to depend on the young bit being clear after the initial Linux fault. A proposed core mm change to immediately set the young bit upon such a fault, results in ptep_set_access_flags not flushing the TLB because it finds no change to the pte. The spurious fault fix path only flushes the TLB if the access was a store. If it was a load, then this results in an infinite loop of page faults. This change adds a TLB flush in update_mmu_cache, which removes that TLB entry upon the first fault. This will cause the fast TLB handler to load the new pte and avoid the Linux page fault entirely. Reviewed-by: Ley Foon Tan Signed-off-by: Nicholas Piggin --- arch/nios2/mm/cacheflush.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/nios2/mm/cacheflush.c b/arch/nios2/mm/cacheflush.c index 506f6e1c86d5..d58e7e80dc0d 100644 --- a/arch/nios2/mm/cacheflush.c +++ b/arch/nios2/mm/cacheflush.c @@ -204,6 +204,8 @@ void update_mmu_cache(struct vm_area_struct *vma, struct page *page; struct address_space *mapping; + flush_tlb_page(vma, address); + if (!pfn_valid(pfn)) return; From patchwork Tue Oct 16 13:13:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10643535 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7514113B0 for ; Tue, 16 Oct 2018 13:14:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6228228C31 for ; Tue, 16 Oct 2018 13:14:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 561DA29654; Tue, 16 Oct 2018 13:14:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EA59328C31 for ; Tue, 16 Oct 2018 13:14:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ABF696B0008; Tue, 16 Oct 2018 09:14:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A70126B000A; Tue, 16 Oct 2018 09:14:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 985A86B000C; Tue, 16 Oct 2018 09:14:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id 5954B6B0008 for ; Tue, 16 Oct 2018 09:14:07 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id w15-v6so17112611pge.2 for ; Tue, 16 Oct 2018 06:14:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=vbOXilnyHM7Y3HZf42CmhT0al3ZotZPzM5/9KSS+Wl0=; b=Cyz6B0uf4Chxg+YStY94gn+i27VddluFcmul9JcsuH4nkmkUtaEsWobaNtxGO6L8Sr agvI9e8Iv8lYowS/aplO5r/OVrhV41WxAOoYjh4JR4kg3p0Khz6hA91avPLlr10gEchK hW2UJhLv3h4fRZ1qlhsdRpshpLpZTuiIOgJviyQqTaxRiGgPztleXNOwDsomRbB7rr1H V8oWbtrqnpvvXWpsK2Mf1UQj/EbnGhY99Z1vvJxHV2VspHzpdTc1VY6JyAkrAddW27yG OJ48FZCaPJFXiIicr3jzQTpFiR5VfhWt7mN/th+u7SbK15y4jkiX2kcpzWEt4qQ7muKw 6r8Q== X-Gm-Message-State: ABuFfoiOPBvbKqBKIWv22BP18MIj3F6lJk8LlknIZV0Cjama9gw+fqKQ lqNJ1BGnWbW3TID2TjzhXeMu6YwrTUtMrtaOG/1v1Db5vS/kffK+3gK2/Nzlr7ttIAUbVPKzgSF EitAjg+kaY8OhUJMpsroBgYXFI9Q+1l43f5nLyypCOY/8Dz+H6MB53AtAvKmgDlrky7co43xZM2 A0o+d1LMJOKzENIo0QxzhFvCe3SmTrVp1PR5t0AvhCk+ueuU/MVnYJkZCbALm4TxX78m7ACC6uo OxylN1T3ikifd7dDmyLKYLK9Nr+mdFU39YcM68f4W870KvA1uV7XdtvrQ2L7pYD+Z2J8gsW6lsd 5RGiZAtkd4uYwS6LQuyMgvJUgfmaFlnabdqmzpahnwacykKEcmzmGLk1wpdOK5/BKAJBWynVyuG p X-Received: by 2002:a63:e943:: with SMTP id q3-v6mr19998831pgj.42.1539695647054; Tue, 16 Oct 2018 06:14:07 -0700 (PDT) X-Received: by 2002:a63:e943:: with SMTP id q3-v6mr19998779pgj.42.1539695646303; Tue, 16 Oct 2018 06:14:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539695646; cv=none; d=google.com; s=arc-20160816; b=GxKtgR/TjlutoTjcGrqlyx7muaM1m76dj6/BM0ceOaQrHVcuQaH3WDg265cYozlGdB Lw3GT1DHtQLjeI/Bfqx25Ry71kil+W1IRq/KCcxhuu9m/6qT4nSvZqtqHImzis4Ur+7H sawt2vSyraJwb66CFY4AIitdjtCra94Z0Vq/IKJWSi/ViOCfvWVeTyN5p57hL76q/aqF k7c3Zj+J7st8NYqbZZqRHeZ8J3UILnKLyJMRvH4kHsppLhagNBjU8Dm16Mgj5c8APcIq H8m35m2v6kyQJO4aEGwskQuw9VAjZJg+SFo6xH2/3WXhCNvWuWqIzEOz+1fTQXNyGuME XsvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vbOXilnyHM7Y3HZf42CmhT0al3ZotZPzM5/9KSS+Wl0=; b=Rf5UP3bJo14pNYw0Yz3/jg/h9D8N86W6YrimuMWXNpy+JHSMw5n06Zfe3TQFwfUNyn BEKKg+CqBS1Yfp+txwjxTNgy7o0TaoXc67OvYhqoVF4SqnyuAgghlD2IATyuonn69zzB iIPkKW6A+Dx2ZIKOq11OeBqNtnKd9ncXTx9SYD/iNJHVbDwa3AKW9u188IXRxPK+IjCL GGLqj2TA6nfj4YAB5mle1ySzLbaw10C25ci3a+5A2AbrqiVQifXvbuMzYXS7oCVLw/pI +wUaE3EF4HlwtMM6f3cSNWGCvuM50CEs16fP7+Tag3xcAJgkU+jnof8g34SIID34wn3Z m3WA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="A/SSJVZx"; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id k33-v6sor4225591pld.54.2018.10.16.06.14.06 for (Google Transport Security); Tue, 16 Oct 2018 06:14:06 -0700 (PDT) Received-SPF: pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="A/SSJVZx"; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=vbOXilnyHM7Y3HZf42CmhT0al3ZotZPzM5/9KSS+Wl0=; b=A/SSJVZxVxROnzdtTJGd3Rl0mtQnn0aeP262cO0PG37AB5aJFbY1kcl0Q9RVU+AsSN hnGdUUZtKTqK8l4cdbSjbZ9BUzn/hPxOFdEPN4vlGKIB25R+NFVzhjzkDFWSAaFBLaSt xrSM6/l6UKnNfWzsSJg7ulrXaZrJXIUqdvEaGukSxIoLvU3Es6tmC1I+PECGOfrldZKg NLn/cJoSesrvsErHZ4uFkC2dmA03+a8M23DZATqifHAalHpK02AeaGxYq3X86uOI4su5 c7wNfOzT0sZv7rpzm5n9vvCGcZMKbl4VkQEOnVbCI02AICWJXxrRFSVG1IEGRym457Rs WgRA== X-Google-Smtp-Source: ACcGV625ozmXlwQNgWm1xWhrstSjyDp7scuMStIV7BLFGcBHbvseOYBjb3efegUxYSoldpVrEtpcww== X-Received: by 2002:a17:902:9b84:: with SMTP id y4-v6mr21817926plp.332.1539695646081; Tue, 16 Oct 2018 06:14:06 -0700 (PDT) Received: from roar.local0.net ([60.240.252.156]) by smtp.gmail.com with ESMTPSA id j62-v6sm16043423pgd.40.2018.10.16.06.14.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Oct 2018 06:14:05 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Linus Torvalds , linux-mm , linux-arch , Linux Kernel Mailing List , ppc-dev , Ley Foon Tan Subject: [PATCH v2 2/5] mm/cow: don't bother write protecting already write-protected huge pages Date: Tue, 16 Oct 2018 23:13:40 +1000 Message-Id: <20181016131343.20556-3-npiggin@gmail.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20181016131343.20556-1-npiggin@gmail.com> References: <20181016131343.20556-1-npiggin@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This is the HugePage / THP equivalent for 1b2de5d039c8 ("mm/cow: don't bother write protecting already write-protected pages"). Signed-off-by: Nicholas Piggin --- mm/huge_memory.c | 14 ++++++++++---- mm/hugetlb.c | 2 +- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 58269f8ba7c4..0fb0e3025f98 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -973,8 +973,11 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, mm_inc_nr_ptes(dst_mm); pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); - pmdp_set_wrprotect(src_mm, addr, src_pmd); - pmd = pmd_mkold(pmd_wrprotect(pmd)); + if (pmd_write(pmd)) { + pmdp_set_wrprotect(src_mm, addr, src_pmd); + pmd = pmd_wrprotect(pmd); + } + pmd = pmd_mkold(pmd); set_pmd_at(dst_mm, addr, dst_pmd, pmd); ret = 0; @@ -1064,8 +1067,11 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, /* No huge zero pud yet */ } - pudp_set_wrprotect(src_mm, addr, src_pud); - pud = pud_mkold(pud_wrprotect(pud)); + if (pud_write(pud)) { + pudp_set_wrprotect(src_mm, addr, src_pud); + pud = pud_wrprotect(pud); + } + pud = pud_mkold(pud); set_pud_at(dst_mm, addr, dst_pud, pud); ret = 0; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 5c390f5a5207..54a4dcb6ee21 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3287,7 +3287,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, } set_huge_swap_pte_at(dst, addr, dst_pte, entry, sz); } else { - if (cow) { + if (cow && huge_pte_write(entry)) { /* * No need to notify as we are downgrading page * table protection not changing it to point From patchwork Tue Oct 16 13:13:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10643537 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5729E13AD for ; Tue, 16 Oct 2018 13:14:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4511228C31 for ; Tue, 16 Oct 2018 13:14:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 393D329654; Tue, 16 Oct 2018 13:14:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A8F2A28C31 for ; Tue, 16 Oct 2018 13:14:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 659016B000C; Tue, 16 Oct 2018 09:14:13 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5DB356B000D; Tue, 16 Oct 2018 09:14:13 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4CC0A6B000E; Tue, 16 Oct 2018 09:14:13 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id 243906B000C for ; Tue, 16 Oct 2018 09:14:13 -0400 (EDT) Received: by mail-pl1-f198.google.com with SMTP id j9-v6so14358943plt.3 for ; Tue, 16 Oct 2018 06:14:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=b8n7ekTz4I5WPtItNykHetIxRP9150gP8kPb0BQtO90=; b=YMqwycpxj2hKkNk3AqW7/cmKYmmDH0UoH4Sm7EcZLrmwRa24l+brZxAf7Kh9RrV6nR aOZMCZg4swgDvFfEiwgIpBMWpjcBVHSekRB7+fBZodTK6di0KnSP1C4vZUAb4Z1F6Nmb fpjod4Rf1Ifuot8NDTlGtl+1uZPNzOSTLmdIxUUUWdY4BASUyxrjyM3DI5F56Ks3ADXq c+0wqoSuEzx0XUyAXdt5QAxM+Uw2uHCfXKyWvHRUcpZ3//jC/+Fnwyt9MnwG2VABdVxY 6GRwoEq5jNITP/hONgpSNc2UVUAfcRqz2i2dzgVgg+frOziG/9fFejklZUoiSrT63acv hvlw== X-Gm-Message-State: ABuFfoiiNXnkgrN93DTQo63Pjja3TrSYo4HaIyUVqKDG/cokkZEQsiwW WMJJc1On0Ma/GzyEkQoOhTtI2NrsIb8bf2WIbMuge2WVpXJ2Uqq6d9GDxDmSkeqXEDn8wkuHXv4 hr5gYOVZClyluJxLm29GEjtH7RQ/cX9iJkqWi3pzhwW5hVV/SN/B6NFvnNBXhembvduEOh7FwJU E16AFfclo80ePxY1EqUKDug1cIOQLF0rDlH0Sv+8fXBIabAQHmmTWeKuyDhO9vklkwx8nNXQbGl CNusxVchySp36Kqve6Pm/LG0cx1GcICp5U59tL57ts2IFVKk6Onkw7lrdwxTbWvqRTP9100I/zU ce/HjDow2KQdm7O5FtedU1L2zzVl9xva8Za3bWHak9id8Ul2bLSNuWszK6vTQ2CCEmdGqoIm3k9 C X-Received: by 2002:a63:f711:: with SMTP id x17-v6mr20075982pgh.322.1539695652785; Tue, 16 Oct 2018 06:14:12 -0700 (PDT) X-Received: by 2002:a63:f711:: with SMTP id x17-v6mr20075928pgh.322.1539695651955; Tue, 16 Oct 2018 06:14:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539695651; cv=none; d=google.com; s=arc-20160816; b=RVlGdw0JobaRHdaTt7U3Q4MO37QyJmc8gaGfFBmDRH5vq7jILy4EWUNLxoVcVGylIz 0H8sU80mZ3YFQ+3lVaoaa24ERe1nUwngHgV37bYCJB8hYCo1rhIxtgi3iGzayU3x2zYy WKFZH63nTRY//GPxqTt6FSWmvGniOuf+l0UNtyroFDKPzVlKnyLmcRBWUNFLO6ECLXnT yUQHF5kB6CvZKMPKwuZpkqkiag5nMqejEOOcUnLL3Mfa8rMj/iXXFPp8x+cmhMDzqaAN +PrYiU38fotSMb3PofQrnQQnFrx0X0KIqKJJRQP1W5jqUWxoKbw3upB+6AdsSAnR+JeC bZxg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=b8n7ekTz4I5WPtItNykHetIxRP9150gP8kPb0BQtO90=; b=JYbqdI1dD/KqH2vvJc7SxMgLuk2VPhxPzuZA4nPA82lNM/eLHa9q+Xqb0qCCJDjytl 0lHz9XZ9gpM4JFI7wfoLwg8qB29E192XhNgcdBywqj33EF1rLkWVfqVwQ6VydG31/hy3 9OizZ2Hqm36e2+tCALeLI9tcYmVR2FY4q6nrdoAUdfdEps0wDKQR9LtoqJnuL4uXmPNT 9Zt9mgPY0YXCEBl6d9oxYar+L4e+k5rof34CnIyBp/imVAzlEddMECjH0ofniIf6Eo8l fPLkx8yXZme0pLvWwYQ6EH5IdEOqkEwKlZuMEiHkemMy/E7DRTuUBe5NKZD+yITsvFQi Ry6Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="WYUM7y/k"; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id s23-v6sor4222653plr.15.2018.10.16.06.14.11 for (Google Transport Security); Tue, 16 Oct 2018 06:14:11 -0700 (PDT) Received-SPF: pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="WYUM7y/k"; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=b8n7ekTz4I5WPtItNykHetIxRP9150gP8kPb0BQtO90=; b=WYUM7y/kL8/+heRnnym3+/pjQXjKvyJ+4vExqnWSUKAt5OCcPWLDKAyjYXXX+3tTNR T6b0g4t9dmXMbmx3FIvcAqX7O329x5zhDruzeal6OmBiza0FOgjbU1RMeljPcpkbKp0/ TFo8sX1VGUzkmoHiR1PFKoUwS+KpHNocpFVRrcfhk+rx6fduiPzvcVAXqnIL0cBl6THr hbVSz2bbtVDrUnEsJdSAy25sFuItTMKF0xRIJvWJnmBDgjGpCixgOlx4aG9rDmg6RsAx /uuAKEyJO8ldmQhR9iNJHXbr4spuqd6ROpfB/V469ofmS6CbDMXdpP6hxlUKVxCe5x+R 3iTQ== X-Google-Smtp-Source: ACcGV63ooqapdfib9rvEhAj6eX85EavZn0yGkIHSWlSmi1u/6UNdDgdZJZkJZsL5NL3zvWoE/2y3vA== X-Received: by 2002:a17:902:4681:: with SMTP id p1-v6mr13592768pld.97.1539695651671; Tue, 16 Oct 2018 06:14:11 -0700 (PDT) Received: from roar.local0.net ([60.240.252.156]) by smtp.gmail.com with ESMTPSA id j62-v6sm16043423pgd.40.2018.10.16.06.14.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Oct 2018 06:14:11 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Linus Torvalds , linux-mm , linux-arch , Linux Kernel Mailing List , ppc-dev , Ley Foon Tan Subject: [PATCH v2 3/5] mm/cow: optimise pte accessed bit handling in fork Date: Tue, 16 Oct 2018 23:13:41 +1000 Message-Id: <20181016131343.20556-4-npiggin@gmail.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20181016131343.20556-1-npiggin@gmail.com> References: <20181016131343.20556-1-npiggin@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP fork clears dirty/accessed bits from new ptes in the child. This logic has existed since mapped page reclaim was done by scanning ptes when it may have been quite important. Today with physical based pte scanning, there is less reason to clear these bits, so this patch avoids clearing the accessed bit in the child. Any accessed bit is treated similarly to many, with the difference today with > 1 referenced bit causing the page to be activated, while 1 bit causes it to be kept. This patch causes pages shared by fork(2) to be more readily activated, but this heuristic is very fuzzy anyway -- a page can be accessed by multiple threads via a single pte and be just as important as one that is accessed via multiple ptes, for example. In the end I don't believe fork(2) is a significant driver of page reclaim behaviour that this should matter too much. This and the following change eliminate a major source of faults that powerpc/radix requires to set dirty/accessed bits in ptes, speeding up a fork/exit microbenchmark by about 5% on POWER9 (16600 -> 17500 fork/execs per second). Skylake appears to have a micro-fault overhead too -- a test which allocates 4GB anonymous memory, reads each page, then forks, and times the child reading a byte from each page. The first pass over the pages takes about 1000 cycles per page, the second pass takes about 27 cycles (TLB miss). With no additional minor faults measured due to either child pass, and the page array well exceeding TLB capacity, the large cost must be caused by micro faults caused by setting accessed bit. Signed-off-by: Nicholas Piggin --- mm/huge_memory.c | 2 -- mm/memory.c | 1 - mm/vmscan.c | 8 ++++++++ 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 0fb0e3025f98..1f43265204d4 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -977,7 +977,6 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmdp_set_wrprotect(src_mm, addr, src_pmd); pmd = pmd_wrprotect(pmd); } - pmd = pmd_mkold(pmd); set_pmd_at(dst_mm, addr, dst_pmd, pmd); ret = 0; @@ -1071,7 +1070,6 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, pudp_set_wrprotect(src_mm, addr, src_pud); pud = pud_wrprotect(pud); } - pud = pud_mkold(pud); set_pud_at(dst_mm, addr, dst_pud, pud); ret = 0; diff --git a/mm/memory.c b/mm/memory.c index c467102a5cbc..0387ee1e3582 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1033,7 +1033,6 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, */ if (vm_flags & VM_SHARED) pte = pte_mkclean(pte); - pte = pte_mkold(pte); page = vm_normal_page(vma, addr, pte); if (page) { diff --git a/mm/vmscan.c b/mm/vmscan.c index c5ef7240cbcb..e72d5b3336a0 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1031,6 +1031,14 @@ static enum page_references page_check_references(struct page *page, * to look twice if a mapped file page is used more * than once. * + * fork() will set referenced bits in child ptes despite + * not having been accessed, to avoid micro-faults of + * setting accessed bits. This heuristic is not perfectly + * accurate in other ways -- multiple map/unmap in the + * same time window would be treated as multiple references + * despite same number of actual memory accesses made by + * the program. + * * Mark it and spare it for another trip around the * inactive list. Another page table reference will * lead to its activation. From patchwork Tue Oct 16 13:13:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10643539 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 12BDA13AD for ; Tue, 16 Oct 2018 13:14:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F00E528C31 for ; Tue, 16 Oct 2018 13:14:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E3BA129654; Tue, 16 Oct 2018 13:14:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7A1EA28C31 for ; Tue, 16 Oct 2018 13:14:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1224C6B000E; Tue, 16 Oct 2018 09:14:18 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0D2776B0010; Tue, 16 Oct 2018 09:14:18 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F2A326B0266; Tue, 16 Oct 2018 09:14:17 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id B36D46B000E for ; Tue, 16 Oct 2018 09:14:17 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id r16-v6so16927666pgv.17 for ; Tue, 16 Oct 2018 06:14:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=Uwz2MhBiZj2/RLBte8wwxNWfHSbBq1uBQUZRjpspgrw=; b=VB7ZdcT5iOjsyQVCkTmcAJnr4ZP2jICz5jr2JuRjvZelj4chNFt6e5+OBgs19cH1zj 6/6sx1EUURliZOj45Ev6fPJZvftHJWKuvBT6oweuOaXzNd2XCBCITDz1Se1wHMCRCk2o 9PHXtIooPu+XKjlAWXU6Aww1Jd9IjCQ2w/TUY5Tfgr/b+e2lIDnm60Wu3LS9rrfANFyu Ggj9A27rAcMV/R7kqNWa+pHfKJEtl9BHl2v31CzxKLYTsCDrCxJhCdxW9d7ZeWXcoARA 87xyJ9sPJtrWk4Bxv7FoKkrf/F5F45kVRAfYiJE31HEJHsqd0wvQ4O4rkIBpIT6Luz1k skyQ== X-Gm-Message-State: ABuFfohazoVa62EhTBIF7iwXzYnWzfuFnMevSGnmcrur0Dm5Y9nVfg3D OVLgEMGZ44rnTPwqAmNQem8fc/iXnKOpNxxTOFz6FrMsSA5GaBwepFM9XGpqsGyALhF2Z2U/UD8 XdgBadJCP+ERjZTbglb5Q/DpVklsk5pToZrXmKhVnEtHxbWjuUYG4PFbv4obEAIVBMKn98B1PyJ sw7ZtrAQs0bfQT4m0WjjT2aNmZs0k+/Agsu83aJrxKo3JSllXPOyxFvx+JPCDZDIUHcLIsQbA8M X3h4ehViK7XLWTlHUaaWi/hO6rQsc3gpJ32SM18rbMAwhH38XvR5YZGV8xmiWUZp3//rkAR7j8w 4TG0wYZPiXyneGLa1VU2EnZSHlLJ6lNB/Fk8oGCr5fY2t5J0NAxu9xwRh9lKO1/7J1IV9l7rxba 6 X-Received: by 2002:a63:1302:: with SMTP id i2-v6mr20580604pgl.380.1539695657410; Tue, 16 Oct 2018 06:14:17 -0700 (PDT) X-Received: by 2002:a63:1302:: with SMTP id i2-v6mr20580565pgl.380.1539695656736; Tue, 16 Oct 2018 06:14:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539695656; cv=none; d=google.com; s=arc-20160816; b=PP7dxMq6JEcFpv8rjH7KU0RaaX8VexsFeXF79qOEpp9soyVKVWROgI8iiNv4MBfhx2 Jpsrf/9yY065vPQ2JsbMHkIjEMapdcQK7AzlMFQlCcZ4i4e9HHeCA5GjP0apJTbx4wE/ jq9OApf/8ez+/QlgJ62kWlDCjdxXFP/CTsqQdLT9/dix8bYSIWsLL5FvNrKDJUZjZev7 g4otjxrcP/NP+fowAwVJNDOoNhl41csFduWY7OZzD9N5j08pdTVAAGttMeWuqIinEdbw oUcA/nSPd/OIu3QQKd6EqATpw5Yx2qC4EdFWgK0AcCF8+eOn2KBgfoauNhMbFfuvhYgh J5ew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Uwz2MhBiZj2/RLBte8wwxNWfHSbBq1uBQUZRjpspgrw=; b=gACZf+vGa35yZopFrlkUZXeAVDz3QaSBf8lPnv3aybHlyIOUS444JZO3557jTkAneT wQ6GSxka1mevLaciuTRBGUPGE3XsOf82Zj8NX4u5JtWYLD4IrVD/Ew0nqVGDyGBHbnB8 txbGaK/RFTLnOgt0Q+ImYqLy/Li8nkfaVHZKIR80HI0l+ofSqX2oXyuCZlABCPw4bDpq 9uLa5V654PaO9h3r/6LcEME9GLs+KMfE2mJqFaont3zkRvhqq4hEI1xx4heN1gU03FSS xNBiyDE6MuvUYSjnb4ykaAQ4PRYyTwU4SVYxRfmZdQH+2zvNi0XZAmrJ+1GJlLiY6PQA w+oA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=e11cs6jE; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id j33-v6sor4465352pld.51.2018.10.16.06.14.16 for (Google Transport Security); Tue, 16 Oct 2018 06:14:16 -0700 (PDT) Received-SPF: pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=e11cs6jE; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Uwz2MhBiZj2/RLBte8wwxNWfHSbBq1uBQUZRjpspgrw=; b=e11cs6jEoeMmwoBTYdNdNJksH7qI2wwNhzL8dHOfMYBv61ZXmE7ZyV/m0HuwprSE8P LFyAsDmOM0lbAcgAwqRMqM/kSCDNmRysuTjQBcOTsE+FYIuV6YatjeYM9khiyj/R3/AY QCOLgFdpTNNuFTtQ/BUGeOh8mtqwWkMaj1cDEQfLz/cYm4GL6lZaJ74Hjstyi58GeYBt wOKkZeW4oBUFptEPdrDwkxiVepZCC+JvI0CR0+o9yZckDgcyrP91npD4G3d5GNYQsbQN Zhnb8I+DxMlfZAf2J/KOOMUmZ9HMuxpolqtPk2C9td0SKgrYsShSDKNx8nkIWCHWN94Z 0Q0A== X-Google-Smtp-Source: ACcGV62O4GEomn4Rji7i3zI4ZPz6+o/83UivhZoWG06/NphsLTdj3gWE9y8+Ptx4lChGWFwrT5esfA== X-Received: by 2002:a17:902:34a:: with SMTP id 68-v6mr21440986pld.39.1539695656502; Tue, 16 Oct 2018 06:14:16 -0700 (PDT) Received: from roar.local0.net ([60.240.252.156]) by smtp.gmail.com with ESMTPSA id j62-v6sm16043423pgd.40.2018.10.16.06.14.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Oct 2018 06:14:16 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Linus Torvalds , linux-mm , linux-arch , Linux Kernel Mailing List , ppc-dev , Ley Foon Tan Subject: [PATCH v2 4/5] mm/cow: optimise pte dirty bit handling in fork Date: Tue, 16 Oct 2018 23:13:42 +1000 Message-Id: <20181016131343.20556-5-npiggin@gmail.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20181016131343.20556-1-npiggin@gmail.com> References: <20181016131343.20556-1-npiggin@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP fork clears dirty/accessed bits from new ptes in the child. This logic has existed since mapped page reclaim was done by scanning ptes when it may have been quite important. Today with physical based pte scanning, there is less reason to clear these bits, so this patch avoids clearing the dirty bit in the child. Dirty bits are all tested and cleared together, and any dirty bit is the same as many dirty bits, so from a correctness and writeback bandwidth point-of-view it does not matter if the child gets a dirty bit. Dirty ptes are more costly to unmap because they require flushing under the page table lock, but it is pretty rare to have a shared dirty mapping that is copied on fork, so just simplify the code and avoid this dirty clearing logic. Signed-off-by: Nicholas Piggin --- mm/memory.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 0387ee1e3582..9e314339a0bd 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1028,11 +1028,12 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, } /* - * If it's a shared mapping, mark it clean in - * the child + * Child inherits dirty and young bits from parent. There is no + * point clearing them because any cleaning or aging has to walk + * all ptes anyway, and it will notice the bits set in the parent. + * Leaving them set avoids stalls and even page faults on CPUs that + * handle these bits in software. */ - if (vm_flags & VM_SHARED) - pte = pte_mkclean(pte); page = vm_normal_page(vma, addr, pte); if (page) { From patchwork Tue Oct 16 13:13:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10643541 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 20A8013AD for ; Tue, 16 Oct 2018 13:14:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0E6BB28C31 for ; Tue, 16 Oct 2018 13:14:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 00A3929654; Tue, 16 Oct 2018 13:14:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 76CD828C31 for ; Tue, 16 Oct 2018 13:14:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CBB7F6B0266; Tue, 16 Oct 2018 09:14:22 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C6AC46B0269; Tue, 16 Oct 2018 09:14:22 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B5A8E6B026A; Tue, 16 Oct 2018 09:14:22 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id 77E4C6B0266 for ; Tue, 16 Oct 2018 09:14:22 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id r134-v6so12031819pgr.19 for ; Tue, 16 Oct 2018 06:14:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=Wozx3I6t1Z6fZ+0xttJ6rllGkcnzPt2KAsF0ALUBnnI=; b=aeovy2OfkPVRdw/IW6YRdNMYZdE2hBI7ky+lvfZXHAwB7dJDgay6fYzsq1UHoKV1Ic c5M30L5bXxrr35sUhMO2p/QMHCOAGH5yjErzRwoBaL/mD1TgrpaSItxosOxF3at1hZjW 3BXpwbss/zlZxyUB1yezP/34quWrPskX9fF1qcL5MYnDk3GTlr1jHENBOKNSNnq8hzyb kz3539UOui3E3llDz/+qu6tZpdsq+BGtPDz6lvH7iCjA+OaZjrIQ4cm1P63urBkIkKFi 1bKPzgw3pBSTZI3bo+03JT9wMH43sshiTJn8avqR2+Ij/LEmrcqlBfHJtXvDis7iMoYR tZhw== X-Gm-Message-State: ABuFfogfo8DiKwp2MFDlKa1ijIpVPAYdZVdtbho8F7/j99Q/+M7RX455 d95+Gl5MvM5R1gE87wFLsd+pzbXNRl85O8htn+I8XDILpZwsyyxozqQuwFwL51lOxB+jULy9e3+ orUiRuvsi1rAhgr/78C4xuLh+OTmBc/4hBH4OPqhx54K958GpZ7kDF/1OZHu7IKR8vU+UC5hk3E 5YevsfUkPPIEsm77qaDbbIcLkq6roeZXZqcG/Ypvvbby8rlc35cE7ivE/fOfS/vqWdlCUOIcr5A iZtrXeIz4j31F4kZANn0JiDaQ5GVFGZ/oSLsbOsyzG+zg4PdWEkP9O9OidAZitRYVanLzetGokW GfmHmTyBDeMozpwKcwrvVvPv2QqwIyOHNTvD5IsG35z/S9RVhU3n/W/G0tRoL53j9axnysBiybz V X-Received: by 2002:a17:902:8d82:: with SMTP id v2-v6mr22100167plo.9.1539695662165; Tue, 16 Oct 2018 06:14:22 -0700 (PDT) X-Received: by 2002:a17:902:8d82:: with SMTP id v2-v6mr22100125plo.9.1539695661379; Tue, 16 Oct 2018 06:14:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539695661; cv=none; d=google.com; s=arc-20160816; b=WNRsaq6Bw8reEtLAilb4NpmjZC9orF8bbqKGVKuTACaf0mE2d5xksbCgoa+MzTQ7iK 2hxyfRDI5Dt4VqmYqm16ly9xfgnaFuqhqF5IuU0MiBi+9KxrUtrbd8+X43enyNjG0GAY W7lkD5GXVoujKmnZTs24ekxyG5kJtss+63YhjqgftdFA1nac1To3h4XFg4nGEYooL8pf lBodeCTzjetwEtIEU/dbYObt5C5Sc9Ri4DlrQfeRZPA7qvAMi3A4BCpp76xQFFieY42F +RUIK+UwQw3f095ANgE9dr7jVstqKD69QtYviSnnSSq7r9lCC5S4X4Z7t7kHWW2g6Nm3 ubvA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Wozx3I6t1Z6fZ+0xttJ6rllGkcnzPt2KAsF0ALUBnnI=; b=hvVGfcOnxVD0MduCpbcvBtcA1okLqDsvap9l0bDhpAZym17oC6Bj99qr7URWtAMcXE N0vi1h8kzRZZbRUHjWxJTPzuVmdkhMTkEwKfKtSQLkhm5aHZLQBHD6rlSq5ghzJW3Nac skWTDL2xcsoWFwNgxpNvky0Ea2SyhgrN/lRNXgBEC6tUBvfNJ/Ci4xSKpRIAorSLsZjj LP8B0LYgxrfz6yQFqenbRporla0/BjkjQlQ9vqows3GfQCWw5CLa+6P9fnDR5lY3dvB5 rWMgF/RLKaqhTOowosP0DfZJ5Jilbt9+/WVs/P1SeH5jFsSq3yb4LuCxU/mKuu4GCEIh 0dVg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=QRnnDkdr; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id l20-v6sor4443475pgh.86.2018.10.16.06.14.21 for (Google Transport Security); Tue, 16 Oct 2018 06:14:21 -0700 (PDT) Received-SPF: pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=QRnnDkdr; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Wozx3I6t1Z6fZ+0xttJ6rllGkcnzPt2KAsF0ALUBnnI=; b=QRnnDkdrX4fTuAStV0TssQmvg1tQGZDM8vwrwyePkqEPHeenfDLv/cItj0JX95n69G z/jVDqEaBiBrP+MCux6xsHe8ceWLckKJyYmwS2L38fAkinSn2GD/WWuGpyEw95WtG0ub V/Uhabhj24big4D6uo/4ZkeWohj7BQHqqv7c09ZBuAuEm34YxZgn5plnw1psItwCJGoL zzuBpbBoeYk0qesuFnzoPrdViEUg4i9rMh1GV/bP3wUAan7fBuk9DdbLzpFF2Vy1cosG 0uMbbR8trBZmW0mfb/w+Wc24TkIyRYwl39mgvpMJYP4uPJnprMkcGzs4cPlwH4A4lbMW Hs7Q== X-Google-Smtp-Source: ACcGV62PowkKGuB1+DTkoNK49/BnGHkMHGcEPghSygGdFtY1eXOZ5nEe+W/s+bLONLOvO07moh8WRA== X-Received: by 2002:a63:db55:: with SMTP id x21-v6mr19632152pgi.365.1539695661103; Tue, 16 Oct 2018 06:14:21 -0700 (PDT) Received: from roar.local0.net ([60.240.252.156]) by smtp.gmail.com with ESMTPSA id j62-v6sm16043423pgd.40.2018.10.16.06.14.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Oct 2018 06:14:20 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Linus Torvalds , linux-mm , linux-arch , Linux Kernel Mailing List , ppc-dev , Ley Foon Tan Subject: [PATCH v2 5/5] mm: optimise pte dirty/accessed bit setting by demand based pte insertion Date: Tue, 16 Oct 2018 23:13:43 +1000 Message-Id: <20181016131343.20556-6-npiggin@gmail.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20181016131343.20556-1-npiggin@gmail.com> References: <20181016131343.20556-1-npiggin@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Similarly to the previous patch, this tries to optimise dirty/accessed bits in ptes to avoid access costs of hardware setting them. Signed-off-by: Nicholas Piggin --- mm/huge_memory.c | 12 ++++++++---- mm/memory.c | 9 ++++++--- 2 files changed, 14 insertions(+), 7 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 1f43265204d4..38c2cd3b4879 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1197,6 +1197,7 @@ static vm_fault_t do_huge_pmd_wp_page_fallback(struct vm_fault *vmf, for (i = 0; i < HPAGE_PMD_NR; i++, haddr += PAGE_SIZE) { pte_t entry; entry = mk_pte(pages[i], vma->vm_page_prot); + entry = pte_mkyoung(entry); entry = maybe_mkwrite(pte_mkdirty(entry), vma); memcg = (void *)page_private(pages[i]); set_page_private(pages[i], 0); @@ -2067,7 +2068,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, struct page *page; pgtable_t pgtable; pmd_t old_pmd, _pmd; - bool young, write, soft_dirty, pmd_migration = false; + bool young, write, dirty, soft_dirty, pmd_migration = false; unsigned long addr; int i; @@ -2145,7 +2146,8 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, page = pmd_page(old_pmd); VM_BUG_ON_PAGE(!page_count(page), page); page_ref_add(page, HPAGE_PMD_NR - 1); - if (pmd_dirty(old_pmd)) + dirty = pmd_dirty(old_pmd); + if (dirty) SetPageDirty(page); write = pmd_write(old_pmd); young = pmd_young(old_pmd); @@ -2176,8 +2178,10 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = maybe_mkwrite(entry, vma); if (!write) entry = pte_wrprotect(entry); - if (!young) - entry = pte_mkold(entry); + if (young) + entry = pte_mkyoung(entry); + if (dirty) + entry = pte_mkdirty(entry); if (soft_dirty) entry = pte_mksoft_dirty(entry); } diff --git a/mm/memory.c b/mm/memory.c index 9e314339a0bd..f907ea7a6303 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1804,10 +1804,9 @@ static int insert_pfn(struct vm_area_struct *vma, unsigned long addr, entry = pte_mkspecial(pfn_t_pte(pfn, prot)); out_mkwrite: - if (mkwrite) { - entry = pte_mkyoung(entry); + entry = pte_mkyoung(entry); + if (mkwrite) entry = maybe_mkwrite(pte_mkdirty(entry), vma); - } set_pte_at(mm, addr, pte, entry); update_mmu_cache(vma, addr, pte); /* XXX: why not for insert_page? */ @@ -2534,6 +2533,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) } flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); entry = mk_pte(new_page, vma->vm_page_prot); + entry = pte_mkyoung(entry); entry = maybe_mkwrite(pte_mkdirty(entry), vma); /* * Clear the pte entry and flush it first, before updating the @@ -3043,6 +3043,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); dec_mm_counter_fast(vma->vm_mm, MM_SWAPENTS); pte = mk_pte(page, vma->vm_page_prot); + pte = pte_mkyoung(pte); if ((vmf->flags & FAULT_FLAG_WRITE) && reuse_swap_page(page, NULL)) { pte = maybe_mkwrite(pte_mkdirty(pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE; @@ -3185,6 +3186,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) __SetPageUptodate(page); entry = mk_pte(page, vma->vm_page_prot); + entry = pte_mkyoung(entry); if (vma->vm_flags & VM_WRITE) entry = pte_mkwrite(pte_mkdirty(entry)); @@ -3453,6 +3455,7 @@ vm_fault_t alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg, flush_icache_page(vma, page); entry = mk_pte(page, vma->vm_page_prot); + entry = pte_mkyoung(entry); if (write) entry = maybe_mkwrite(pte_mkdirty(entry), vma); /* copy-on-write page */