From patchwork Thu Sep 3 07:01:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Shi X-Patchwork-Id: 11752517 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C9E90109B for ; Thu, 3 Sep 2020 07:01:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9F94120716 for ; Thu, 3 Sep 2020 07:01:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9F94120716 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 316A66B005A; Thu, 3 Sep 2020 03:01:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 24EB56B005D; Thu, 3 Sep 2020 03:01:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C7296B0062; Thu, 3 Sep 2020 03:01:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0221.hostedemail.com [216.40.44.221]) by kanga.kvack.org (Postfix) with ESMTP id E214C6B005A for ; Thu, 3 Sep 2020 03:01:43 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id A3A8B180AD80F for ; Thu, 3 Sep 2020 07:01:43 +0000 (UTC) X-FDA: 77220854886.26.lake13_0b1336e270a7 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin26.hostedemail.com (Postfix) with ESMTP id 76A7C1804B655 for ; Thu, 3 Sep 2020 07:01:43 +0000 (UTC) X-Spam-Summary: 1,0,0,e6f3ea9e61318d78,d41d8cd98f00b204,alex.shi@linux.alibaba.com,,RULES_HIT:41:355:379:421:541:800:960:966:973:988:989:1260:1261:1345:1359:1431:1437:1534:1543:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:2693:2740:3138:3139:3140:3141:3142:3354:3865:3867:3868:3870:3871:3872:4321:4385:4605:5007:6261:6630:8531:8603:9592:10004:11026:11232:11473:11658:11914:12043:12048:12296:12297:12438:12555:12895:12986:13161:13229:13846:14181:14394:14721:21060:21080:21222:21451:21627:21966:30054,0,RBL:115.124.30.130:@linux.alibaba.com:.lbl8.mailshell.net-64.201.201.201 62.20.2.100;04yru666tongwrcjmfh8ymj6zis95ocoaqrb5m4p6rxe7qxydb4f8q67t6m1qop.o3dhd4unbyytyngkphtagcjmomrk3p6mbiwe3d4pryn7da77fpibm439d4qwh7x.a-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: lake13_0b1336e270a7 X-Filterd-Recvd-Size: 4636 Received: from out30-130.freemail.mail.aliyun.com (out30-130.freemail.mail.aliyun.com [115.124.30.130]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Thu, 3 Sep 2020 07:01:41 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R211e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e01355;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=9;SR=0;TI=SMTPD_---0U7n-ifu_1599116494; Received: from aliy80.localdomain(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0U7n-ifu_1599116494) by smtp.aliyun-inc.com(127.0.0.1); Thu, 03 Sep 2020 15:01:35 +0800 From: Alex Shi To: Anshuman Khandual , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Alexander Duyck Cc: Andrew Morton , Mel Gorman , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v4 2/4] mm/pageblock: remove false sharing in pageblock_flags Date: Thu, 3 Sep 2020 15:01:21 +0800 Message-Id: <1599116482-7410-2-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1599116482-7410-1-git-send-email-alex.shi@linux.alibaba.com> References: <1599116482-7410-1-git-send-email-alex.shi@linux.alibaba.com> X-Rspamd-Queue-Id: 76A7C1804B655 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Current pageblock_flags is only 4 bits, so it has to share a char size in cmpxchg when get set, the false sharing cause perf drop. If we incrase the bits up to 8, false sharing would gone in cmpxchg. and the only cost is half char per pageblock, which is half char per 128MB on x86, 4 chars in 1 GB. Signed-off-by: Alex Shi Cc: Andrew Morton Cc: Mel Gorman Cc: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org --- include/linux/page-isolation.h | 2 +- include/linux/pageblock-flags.h | 2 +- mm/page_alloc.c | 10 +++------- 3 files changed, 5 insertions(+), 9 deletions(-) diff --git a/include/linux/page-isolation.h b/include/linux/page-isolation.h index 572458016331..baa2e1ecc134 100644 --- a/include/linux/page-isolation.h +++ b/include/linux/page-isolation.h @@ -35,7 +35,7 @@ static inline bool is_migrate_isolate(int migratetype) struct page *has_unmovable_pages(struct zone *zone, struct page *page, int migratetype, int flags); -void set_pageblock_migratetype(struct page *page, int migratetype); +void set_pageblock_migratetype(struct page *page, unsigned char migratetype); int move_freepages_block(struct zone *zone, struct page *page, int migratetype, int *num_movable); diff --git a/include/linux/pageblock-flags.h b/include/linux/pageblock-flags.h index d189441568eb..f785c9d6d68c 100644 --- a/include/linux/pageblock-flags.h +++ b/include/linux/pageblock-flags.h @@ -25,7 +25,7 @@ enum pageblock_bits { * Assume the bits will always align on a word. If this assumption * changes then get/set pageblock needs updating. */ - NR_PAGEBLOCK_BITS + NR_PAGEBLOCK_BITS = BITS_PER_BYTE }; #ifdef CONFIG_HUGETLB_PAGE diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 81e96d4d9c42..3688e6b83318 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -517,19 +517,15 @@ void set_pfnblock_flags_mask(struct page *page, unsigned long flags, unsigned long bitidx, byte_bitidx; unsigned char old_byte, byte; - BUILD_BUG_ON(NR_PAGEBLOCK_BITS != 4); + BUILD_BUG_ON(NR_PAGEBLOCK_BITS != BITS_PER_BYTE); BUILD_BUG_ON(MIGRATE_TYPES > (1 << PB_migratetype_bits)); bitmap = get_pageblock_bitmap(page, pfn); bitidx = pfn_to_bitidx(page, pfn); byte_bitidx = bitidx / BITS_PER_BYTE; - bitidx &= (BITS_PER_BYTE-1); VM_BUG_ON_PAGE(!zone_spans_pfn(page_zone(page), pfn), page); - mask <<= bitidx; - flags <<= bitidx; - byte = READ_ONCE(bitmap[byte_bitidx]); for (;;) { old_byte = cmpxchg(&bitmap[byte_bitidx], byte, (byte & ~mask) | flags); @@ -539,13 +535,13 @@ void set_pfnblock_flags_mask(struct page *page, unsigned long flags, } } -void set_pageblock_migratetype(struct page *page, int migratetype) +void set_pageblock_migratetype(struct page *page, unsigned char migratetype) { if (unlikely(page_group_by_mobility_disabled && migratetype < MIGRATE_PCPTYPES)) migratetype = MIGRATE_UNMOVABLE; - set_pfnblock_flags_mask(page, (unsigned long)migratetype, + set_pfnblock_flags_mask(page, migratetype, page_to_pfn(page), MIGRATETYPE_MASK); }