From patchwork Mon Oct 15 20:27:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10642451 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7F0FB15E2 for ; Mon, 15 Oct 2018 20:27:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7059D28D52 for ; Mon, 15 Oct 2018 20:27:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6456029A75; Mon, 15 Oct 2018 20:27:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D4ABA28D52 for ; Mon, 15 Oct 2018 20:27:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB7F36B026D; Mon, 15 Oct 2018 16:27:33 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B67726B026E; Mon, 15 Oct 2018 16:27:33 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A57506B026F; Mon, 15 Oct 2018 16:27:33 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 618586B026D for ; Mon, 15 Oct 2018 16:27:33 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id r134-v6so10450139pgr.19 for ; Mon, 15 Oct 2018 13:27:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=7Qe2JxVHT4wv/fjv1yZD5Agxrtv5cmXm5Gl9QXkvW3g=; b=MeP9dJSy3zEN/8Iwi+0A/bv/Lr+Zto6XeIToze/ePw3BS8ylfxNDSmkQMae4hVanxj AAogPozATyUKjdVNUQDC0KLVtGm7HrHvizJlsFQ5jqUjb9d1py5Ua9NJ3G9wzkQlOpOb Wed8F4gsCtsGF2RZdNnaksRaI8U+chg73YVu9EMKluHE0Aj/iOfFvwgwFhScAUQ5GUip L1NHqwbltHjtMy90XK5n1NzUyOZV/Ahu0VmMNo2qYy9923s57jGYf1mfPfO1w/VGlxkC ShBdwTYmgLV48JuSKxP+Ow5wYoL38s+MKmYk1Xa7eFyjzVCkP7hjOC8P3ZKLxKoVob3m /acw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfoi5PEu7jtO/o235KAzo3ufQsRydQwywuEg/AjuMPpRFVPmlkdel sKWQVYp3xTA9q9S8Iv3FqrI4qssZU66m1p7OQbVMYQ/A2nugalFj1f5eVA981emjU06WSL0WHjU n8xw9ZfA5+v37nVGYrdBE9fb0KgbF5uT77luO9GXksdaiCzfcC8OH8Roq6SYhCi0LzA== X-Received: by 2002:a63:6a86:: with SMTP id f128-v6mr17635921pgc.165.1539635253073; Mon, 15 Oct 2018 13:27:33 -0700 (PDT) X-Google-Smtp-Source: ACcGV63YZyMQf6J4bEV6mirwab20n+a+IFxU7Fx/soPllZHEU6fmHcG2NUVz9hQ/WMoQKEZWyVIk X-Received: by 2002:a63:6a86:: with SMTP id f128-v6mr17635893pgc.165.1539635252298; Mon, 15 Oct 2018 13:27:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539635252; cv=none; d=google.com; s=arc-20160816; b=0Ki2CSPFXAjVYDErjM8eoLvkrq3zGEsOlrcO3Z0uoo/fUv66Fhoo4cai9rHwB/vxTK w/khVxaX68vnxVoHRZnBI007lmbdhA6AYlxxiYTvtIKAkZKZQnMvl3PaswvqPGFs1bwi bswkhh4IMhaEhNX4cs+Vdw3zrSXP2G6TOmLRCq3HhbmMeau3TAUabwGJDAydLeoXEThd iI94cFM6r8iVWJtnlF04OOfGBEtFh8pWiYLG4yVKwU/IkdP+arWnMEl1FwihhJuH6g3y FFuDPkTJiBKcUirRnI5b/Q2tyW2re2X9SQoSSDpOrRJ1cbbSg/DEE1YKYXLhoZCylk8o GoNQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=7Qe2JxVHT4wv/fjv1yZD5Agxrtv5cmXm5Gl9QXkvW3g=; b=ykuGqs6xqGgcrvLKK1hkNjMAICD3z9zCBUxC5POpY5/kHtetmgOe+NxWYNYmiebUA8 7Yy5PR3c2vrzdJ9V51A4aSaK6GKmCIun7NLddKhQC5ApwuNEo3N6tcfnprWtXNC8IjnS YNGpDP4jjEr0hwmG2JDcejw9z0j8FPPca3JssgI9i8dSnRkvBlnXfPX9rWpRpTEjpOJE evGRXBSbekITA0M4aV3oYAyZ4ggWiYQ7iUEpV5vrT2B3OdKcpVP/KcKUexP0XMD/w8mx eN4aKHnp/ZmSndYN/NTjulc7eThGxvLjnOIEAl3nHVYxdfHgpVUryRZfFVj6BR/ymEFS W3Yw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga14.intel.com (mga14.intel.com. [192.55.52.115]) by mx.google.com with ESMTPS id 207-v6si11120856pgb.298.2018.10.15.13.27.32 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 15 Oct 2018 13:27:32 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 192.55.52.115 as permitted sender) client-ip=192.55.52.115; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 15 Oct 2018 13:27:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,385,1534834800"; d="scan'208";a="92202560" Received: from ahduyck-mobl.amr.corp.intel.com (HELO localhost.localdomain) ([10.7.198.154]) by orsmga003.jf.intel.com with ESMTP; 15 Oct 2018 13:27:30 -0700 Subject: [mm PATCH v3 6/6] mm: Add reserved flag setting to set_page_links From: Alexander Duyck To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: pavel.tatashin@microsoft.com, mhocko@suse.com, dave.jiang@intel.com, alexander.h.duyck@linux.intel.com, linux-kernel@vger.kernel.org, willy@infradead.org, davem@davemloft.net, yi.z.zhang@linux.intel.com, khalid.aziz@oracle.com, rppt@linux.vnet.ibm.com, vbabka@suse.cz, sparclinux@vger.kernel.org, dan.j.williams@intel.com, ldufour@linux.vnet.ibm.com, mgorman@techsingularity.net, mingo@kernel.org, kirill.shutemov@linux.intel.com Date: Mon, 15 Oct 2018 13:27:30 -0700 Message-ID: <20181015202730.2171.53682.stgit@localhost.localdomain> In-Reply-To: <20181015202456.2171.88406.stgit@localhost.localdomain> References: <20181015202456.2171.88406.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch modifies the set_page_links function to include the setting of the reserved flag via a simple AND and OR operation. The motivation for this is the fact that the existing __set_bit call still seems to have effects on performance as replacing the call with the AND and OR can reduce initialization time by as much as 4 seconds, ~26s down to ~22s, per NUMA node on a 4 socket system with 3TB of persistent memory per node. Looking over the assembly code before and after the change the main difference between the two is that the reserved bit is stored in a value that is generated outside of the main initialization loop and is then written with the other flags field values in one write to the page->flags value. Previously the generated value was written and then then btsq instruction was issued after testing the is_reserved value. Signed-off-by: Alexander Duyck --- include/linux/mm.h | 9 ++++++++- mm/page_alloc.c | 14 +++++--------- 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index ec6e57a0c14e..31d374279b90 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1152,11 +1152,18 @@ static inline void set_page_node(struct page *page, unsigned long node) page->flags |= (node & NODES_MASK) << NODES_PGSHIFT; } +static inline void set_page_reserved(struct page *page, bool reserved) +{ + page->flags &= ~(1ul << PG_reserved); + page->flags |= (unsigned long)(!!reserved) << PG_reserved; +} + static inline void set_page_links(struct page *page, enum zone_type zone, - unsigned long node, unsigned long pfn) + unsigned long node, unsigned long pfn, bool reserved) { set_page_zone(page, zone); set_page_node(page, node); + set_page_reserved(page, reserved); #ifdef SECTION_IN_PAGE_FLAGS set_page_section(page, pfn_to_section_nr(pfn)); #endif diff --git a/mm/page_alloc.c b/mm/page_alloc.c index f145063615a7..5c2f545c7669 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1179,7 +1179,7 @@ static void __meminit __init_single_page(struct page *page, unsigned long pfn, unsigned long zone, int nid) { mm_zero_struct_page(page); - set_page_links(page, zone, nid, pfn); + set_page_links(page, zone, nid, pfn, false); init_page_count(page); page_mapcount_reset(page); page_cpupid_reset_last(page); @@ -1232,20 +1232,16 @@ static void __meminit __init_pageblock(unsigned long start_pfn, * call because of the fact that the pfn number is used to * get the section_nr and this function should not be * spanning more than a single section. + * + * We can use a non-atomic operation for setting the + * PG_reserved flag as we are still initializing the pages. */ - set_page_links(page, zone, nid, start_pfn); + set_page_links(page, zone, nid, start_pfn, is_reserved); init_page_count(page); page_mapcount_reset(page); page_cpupid_reset_last(page); /* - * We can use the non-atomic __set_bit operation for setting - * the flag as we are still initializing the pages. - */ - if (is_reserved) - __SetPageReserved(page); - - /* * ZONE_DEVICE pages union ->lru with a ->pgmap back * pointer and hmm_data. It is a bug if a ZONE_DEVICE * page is ever freed or placed on a driver-private list.