From patchwork Wed May 9 08:54:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aaron Lu X-Patchwork-Id: 10388743 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 71A4460170 for ; Wed, 9 May 2018 08:53:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 61F70288CD for ; Wed, 9 May 2018 08:53:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5672228E3F; Wed, 9 May 2018 08:53:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9019A288CD for ; Wed, 9 May 2018 08:53:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 69C5A6B04BD; Wed, 9 May 2018 04:53:33 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 672816B04BF; Wed, 9 May 2018 04:53:33 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5136A6B04C0; Wed, 9 May 2018 04:53:33 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl0-f70.google.com (mail-pl0-f70.google.com [209.85.160.70]) by kanga.kvack.org (Postfix) with ESMTP id 0840A6B04BD for ; Wed, 9 May 2018 04:53:33 -0400 (EDT) Received: by mail-pl0-f70.google.com with SMTP id p19-v6so948594plo.14 for ; Wed, 09 May 2018 01:53:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=bpXmFeTrGIkK+Ao7LOP6lwgav0Qa8fPC410Jymu2IKQ=; b=I/BuRswtI0kgUG6s/FqzzUFIPZpial9UmDVHNHIzCqcO8smqzZDXFRTdNA3kbZYzfz ivv0XEUVnn7MiqcsE2IldxTz+0fj0h+0Yfp6lVzFNQaMD0U80d3zXl8QShrQe0PwNdRA spBRLhehgz6sSzqveXs2v9YENSwLHM2119Wm1T1Qsv+1Wh6/1Bke42MWWssPHWAErcto MjeFJvvY6U23EllwxoBo+5oDsf6kipZWAPWI26DrrFJ85fnFgTmEQtfR6tWMqvzoBDfJ lMWdCu7LbCD0sNdgAlZ7IUkXJ4LPJ49D7QJdM8+/y4OBHH116bZqkjAaGUQiUTVyPQ92 Z64w== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=aaron.lu@intel.com X-Gm-Message-State: ALQs6tCDX8ddl3wFZ0PkVRM++9WLZ/shcxJa6SuXzxmbSSquw9deT+Rf WovBUaAGdxxAg2wYJG+sZa+cmOclAA4NieyanHSCp2oz+NlGJghdvHihwsP36UGbkPEq9v4Yp20 qqbeUSPXb4JAFeKVnWkts4YkxAKf83P0FjlQcyQYwYqVZjbK9nFjMMYEOFlvcHJZqBg== X-Received: by 2002:a63:90c4:: with SMTP id a187-v6mr35047329pge.189.1525856012600; Wed, 09 May 2018 01:53:32 -0700 (PDT) X-Google-Smtp-Source: AB8JxZqO+j7yzULbmmELayH2BsdqV8ifdngvj0peDJVgFWHs0OXfzMUx0ZVtWY+Xe++Ra4OEPv91 X-Received: by 2002:a63:90c4:: with SMTP id a187-v6mr35047285pge.189.1525856011383; Wed, 09 May 2018 01:53:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1525856011; cv=none; d=google.com; s=arc-20160816; b=RFosuRCX8XUgKKwfyNlRSNrbBlK+A9Gapf2GMGj2P0WoFK6KaRNIqE55gwSjYjU8fu 1QuR6mI8j4wiRg4rDI/f0JoriSFG1CKo7P3bYYMV1YDG279lZpq2T7n7Ega8kkDDg4Do z0gw0XYZqgH+eDjBWMbcHbaEixx6Gcmf+VCnF7oA/6fpUSETXZKhECYIAQtMoUjC37eE GE7KVtvKJIllxZWA/jTZvuoMwa0JbwWb8I27AAzvl+znerF5B+yci3ORImAa9Gad4W23 phjvv9zYxzregLXkhQTzSKmJgaxTzQKmPvlhty0PI0tbrCuP8/KasPfVKAhy3wh2+Ak+ Enpg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=bpXmFeTrGIkK+Ao7LOP6lwgav0Qa8fPC410Jymu2IKQ=; b=I9DhhID5GDatlg07v6O6/DWsyIZXlQfAZ3xfji7LJZSwsAyt1k9YgimAlU8iGCIG8v dhBvpD6C4GBq3IIbj5NTWyrYmXR0PwRjHBammVGcvg4Y2Bhzdlo2myiAaXaWvdViwo6f FYAnxXsDo2YoprE8HA5XthQHV/gSTsroU4SRAiKa1cNfE+K7GBMF04okj9WK3SGecAXq vbXVwuxAbaZndShL/W1SDNyz/BNzICMNw7s6otJ8FxJr7a0Fp38923/Kr9M/bjfq8P/k N+5VPMfKVSWhiX1PfuYBzzEc9B7l/C0yRNaOUt3TdTXL0ZBrUzoxkVhyZ5FkVbcU4qyQ pJzw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=aaron.lu@intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id x1-v6si25257590plv.520.2018.05.09.01.53.31 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 09 May 2018 01:53:31 -0700 (PDT) Received-SPF: pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=aaron.lu@intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 09 May 2018 01:53:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.49,381,1520924400"; d="scan'208";a="227055155" Received: from aaronlu.sh.intel.com ([10.239.159.135]) by fmsmga006.fm.intel.com with ESMTP; 09 May 2018 01:53:28 -0700 From: Aaron Lu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , Huang Ying , Dave Hansen , Kemi Wang , Tim Chen , Andi Kleen , Michal Hocko , Vlastimil Babka , Mel Gorman , Matthew Wilcox , Daniel Jordan , Tariq Toukan Subject: [RFC v3 PATCH 2/5] mm/__free_one_page: skip merge for order-0 page unless compaction failed Date: Wed, 9 May 2018 16:54:47 +0800 Message-Id: <20180509085450.3524-3-aaron.lu@intel.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180509085450.3524-1-aaron.lu@intel.com> References: <20180509085450.3524-1-aaron.lu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Running will-it-scale/page_fault1 process mode workload on a 2 sockets Intel Skylake server showed severe lock contention of zone->lock, as high as about 80%(42% on allocation path and 35% on free path) CPU cycles are burnt spinning. With perf, the most time consuming part inside that lock on free path is cache missing on page structures, mostly on the to-be-freed page's buddy due to merging. One way to avoid this overhead is not do any merging at all for order-0 pages. With this approach, the lock contention for zone->lock on free path dropped to 1.1% but allocation side still has as high as 42% lock contention. In the meantime, the dropped lock contention on free side doesn't translate to performance increase, instead, it's consumed by increased lock contention of the per node lru_lock(rose from 5% to 37%) and the final performance slightly dropped about 1%. Though performance dropped a little, it almost eliminated zone lock contention on free path and it is the foundation for the next patch that eliminates zone lock contention for allocation path. Suggested-by: Dave Hansen Signed-off-by: Aaron Lu --- include/linux/mm_types.h | 1 + mm/compaction.c | 13 ++++++- mm/internal.h | 27 ++++++++++++++ mm/page_alloc.c | 94 +++++++++++++++++++++++++++++++++++++++++------- 4 files changed, 121 insertions(+), 14 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 21612347d311..17c5604e6ec0 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -93,6 +93,7 @@ struct page { pgoff_t index; /* Our offset within mapping. */ void *freelist; /* sl[aou]b first free object */ /* page_deferred_list().prev -- second tail page */ + bool buddy_merge_skipped; /* skipped merging when added to buddy */ }; union { diff --git a/mm/compaction.c b/mm/compaction.c index 028b7210a669..004416312092 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -777,8 +777,19 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, * potential isolation targets. */ if (PageBuddy(page)) { - unsigned long freepage_order = page_order_unsafe(page); + unsigned long freepage_order; + /* + * If this is a merge_skipped page, do merge now + * since high-order pages are needed. zone lock + * isn't taken for the merge_skipped check so the + * check could be wrong but the worst case is we + * lose a merge opportunity. + */ + if (page_merge_was_skipped(page)) + try_to_merge_page(page); + + freepage_order = page_order_unsafe(page); /* * Without lock, we cannot be sure that what we got is * a valid page order. Consider only values in the diff --git a/mm/internal.h b/mm/internal.h index 62d8c34e63d5..eeec12740dc2 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -541,4 +541,31 @@ static inline bool is_migrate_highatomic_page(struct page *page) void setup_zone_pageset(struct zone *zone); extern struct page *alloc_new_node_page(struct page *page, unsigned long node); + +static inline bool page_merge_was_skipped(struct page *page) +{ + return page->buddy_merge_skipped; +} + +void try_to_merge_page(struct page *page); + +#ifdef CONFIG_COMPACTION +static inline bool can_skip_merge(struct zone *zone, int order) +{ + /* Compaction has failed in this zone, we shouldn't skip merging */ + if (zone->compact_considered) + return false; + + /* Only consider no_merge for order 0 pages */ + if (order) + return false; + + return true; +} +#else /* CONFIG_COMPACTION */ +static inline bool can_skip_merge(struct zone *zone, int order) +{ + return false; +} +#endif /* CONFIG_COMPACTION */ #endif /* __MM_INTERNAL_H */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a92afa362e1f..0a7988d9935d 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -691,6 +691,16 @@ static inline void clear_page_guard(struct zone *zone, struct page *page, unsigned int order, int migratetype) {} #endif +static inline void set_page_merge_skipped(struct page *page) +{ + page->buddy_merge_skipped = true; +} + +static inline void clear_page_merge_skipped(struct page *page) +{ + page->buddy_merge_skipped = false; +} + static inline void set_page_order(struct page *page, unsigned int order) { set_page_private(page, order); @@ -700,6 +710,13 @@ static inline void set_page_order(struct page *page, unsigned int order) static inline void add_to_buddy_common(struct page *page, struct zone *zone, unsigned int order) { + /* + * Always clear buddy_merge_skipped when added to buddy because + * buddy_merge_skipped shares space with index and index could + * be used as migratetype for PCP pages. + */ + clear_page_merge_skipped(page); + set_page_order(page, order); zone->free_area[order].nr_free++; } @@ -730,6 +747,7 @@ static inline void remove_from_buddy(struct page *page, struct zone *zone, list_del(&page->lru); zone->free_area[order].nr_free--; rmv_page_order(page); + clear_page_merge_skipped(page); } /* @@ -800,7 +818,7 @@ static inline int page_is_buddy(struct page *page, struct page *buddy, * -- nyc */ -static inline void __free_one_page(struct page *page, +static inline void do_merge(struct page *page, unsigned long pfn, struct zone *zone, unsigned int order, int migratetype) @@ -812,16 +830,6 @@ static inline void __free_one_page(struct page *page, max_order = min_t(unsigned int, MAX_ORDER, pageblock_order + 1); - VM_BUG_ON(!zone_is_initialized(zone)); - VM_BUG_ON_PAGE(page->flags & PAGE_FLAGS_CHECK_AT_PREP, page); - - VM_BUG_ON(migratetype == -1); - if (likely(!is_migrate_isolate(migratetype))) - __mod_zone_freepage_state(zone, 1 << order, migratetype); - - VM_BUG_ON_PAGE(pfn & ((1 << order) - 1), page); - VM_BUG_ON_PAGE(bad_range(zone, page), page); - continue_merging: while (order < max_order - 1) { buddy_pfn = __find_buddy_pfn(pfn, order); @@ -894,6 +902,61 @@ static inline void __free_one_page(struct page *page, add_to_buddy_head(page, zone, order, migratetype); } +void try_to_merge_page(struct page *page) +{ + unsigned long pfn, buddy_pfn, flags; + struct page *buddy; + struct zone *zone; + + /* + * No need to do merging if buddy is not free. + * zone lock isn't taken so this could be wrong but worst case + * is we lose a merge opportunity. + */ + pfn = page_to_pfn(page); + buddy_pfn = __find_buddy_pfn(pfn, 0); + buddy = page + (buddy_pfn - pfn); + if (!PageBuddy(buddy)) + return; + + zone = page_zone(page); + spin_lock_irqsave(&zone->lock, flags); + /* Verify again after taking the lock */ + if (likely(PageBuddy(page) && page_merge_was_skipped(page) && + PageBuddy(buddy))) { + int mt = get_pageblock_migratetype(page); + + remove_from_buddy(page, zone, 0); + do_merge(page, pfn, zone, 0, mt); + } + spin_unlock_irqrestore(&zone->lock, flags); +} + +static inline void __free_one_page(struct page *page, + unsigned long pfn, + struct zone *zone, unsigned int order, + int migratetype) +{ + VM_BUG_ON(!zone_is_initialized(zone)); + VM_BUG_ON_PAGE(page->flags & PAGE_FLAGS_CHECK_AT_PREP, page); + + VM_BUG_ON(migratetype == -1); + if (likely(!is_migrate_isolate(migratetype))) + __mod_zone_freepage_state(zone, 1 << order, migratetype); + + VM_BUG_ON_PAGE(pfn & ((1 << order) - 1), page); + VM_BUG_ON_PAGE(bad_range(zone, page), page); + + if (can_skip_merge(zone, order)) { + add_to_buddy_head(page, zone, 0, migratetype); + set_page_merge_skipped(page); + return; + } + + do_merge(page, pfn, zone, order, migratetype); +} + + /* * A bad page could be due to a number of fields. Instead of multiple branches, * try and check multiple fields with one check. The caller must do a detailed @@ -1151,9 +1214,14 @@ static void free_pcppages_bulk(struct zone *zone, int count, * can be offset by reduced memory latency later. To * avoid excessive prefetching due to large count, only * prefetch buddy for the first pcp->batch nr of pages. + * + * If merge can be skipped, no need to prefetch buddy. */ - if (prefetch_nr++ < pcp->batch) - prefetch_buddy(page); + if (can_skip_merge(zone, 0) || prefetch_nr > pcp->batch) + continue; + + prefetch_buddy(page); + prefetch_nr++; } while (--count && --batch_free && !list_empty(list)); }